Tool Enables Scientists to Uncover Patterns in Vast Data Sets
Relationships discovered in data will shed light on vexing problems and increase human understanding
With support from the National Science Foundation, researchers from the Broad Institute and Harvard University recently developed a tool that can uncover patterns in large data sets in a way that no other software program can.
Called Maximal Information Coefficient or MIC, the tool can can tease out multiple, recurring events or sets of data hidden in health information from around the globe, or in the changing bacterial landscape of the gut or even in statistics amassed from a season of competitive sports--and much more. The researchers report their findings in the Dec. 16th issue of the journal Science.
Part of a suite of statistical tools called MINE for Maximal Information-based Nonparametric Exploration, MIC has the ability to sort through today's mass of research variables--from attempts to track hurricanes, efforts to model earthquakes, endeavors to identify the Higgs Boson and efforts to glean insights from affecting the world economy and social networking interaction.
Researchers currently use advanced technology to gather big, complex, data sets, which may be incredibly useful in enhancing system understanding, if, in fact, vast amounts of data can be organized so that telling information may be extracted. Sophisticated computer programs research these data sets with great speed, but fall short in even-handedly detecting different kinds of patterns in large data collections, essential for more sophisticated analysis.
One of the greatest strengths of this newly discovered tool within MINE is its ability to detect and analyze a broad spectrum of patterns and characterize them according to a number of different parameters a researcher might be interested in. Other statistical tools work well for searching for a specific pattern in a large data set, but cannot score and compare different kinds of possible relationships. Researchers can also use MINE to generate new ideas and connections.
Learn more about MINE, MIC and patterns identified in biological and health data, as well as statistics from the 2008 baseball season by visiting the Broad Institute website. A video about this work also is available on the website.
The National Science Foundation (NSF) is an independent federal agency that supports fundamental research and education across all fields of science and engineering. In fiscal year (FY) 2017, its budget is $7.5 billion. NSF funds reach all 50 states through grants to nearly 2,000 colleges, universities and other institutions. Each year, NSF receives more than 48,000 competitive proposals for funding and makes about 12,000 new funding awards.
Useful NSF Web Sites: