KDI Home    Contact Us   

The DEEP2 Redshift Survey Web site

Computational Challenges in Cosmology

In the past 15 years cosmology has undergone a renaissance, transforming from a data-starved science to a data-driven one. The COBE satellite and subsequent observations of the cosmic microwave background have begun to provide a detailed picture of the early Universe. Telescopes have found galaxies at distances corresponding to the Universe at one-tenth of its present age, while large-scale redshift surveys have begun to map out the structure of the nearby Universe.

At the same time, according to an NSF award abstract, "the size of these datasets threatens to leave cosmology data-swamped. Realizing our scientific goals depends on meeting the qualitatively new computational challenges set by quantitatively new data."

This multi-disciplinary research project, led by the University of California at Berkeley and funded under the National Science Foundation's KDI (Knowledge and Distributed Intelligence) program, has involved a focused collaboration between astrophysicists, statisticians, and computer scientists to develop computational tools, techniques, and technologies to cope with the new challenges posed by these massive datasets.

According to Professor Marc Davis of the Astronomy Department at Berkeley, a major part of the KDI project has dealt with handling data from Phase 2 of the DEEP Survey (Deep Extragalactic Evolutionary Probe). DEEP is a multi-year program that uses the twin 10-meter Keck Telescopes in Hawaii, the world's largest, and the Hubble Space Telescope to conduct a large-scale survey of distant, faint, field galaxies. The goals include investigating the formation and evolution of galaxies, the origin of large structure, the nature of dark matter, and the geometry of the Universe.

The DEEP2 Redshift Survey utilizes a new DEIMOS spectrograph—featuring very high resolution and efficiency—to obtain spectra of about 65,000 faint galaxies. It is a collaboration between astronomers at Berkeley, the University of California at Santa Cruz, and the University of Hawaii.

"The instrument was completed last year, and we started taking data last summer," Davis says. "It's going extremely well. The data are magnificent. We think we can set powerful constraints on the dark energy of the Universe, among other things—a whole host of cosmological issues will be addressed by our data."

The Berkeley astrophysicist notes that "KDI support was used to help us cope with this huge data onslaught, because the machine, this new spectrograph, generates data at approximately 1 gigabyte per hour, and the processing of the data is pretty complicated. We had to build an automated pipeline to cope with it, because it's just overwhelming."

Davis says he believes the research team has effectively met the computational challenges posed by DEEP2. "When we started the project many years ago, we really were afraid we wouldn't be able to deal with the data at all—that it would just pile up on the floor and we'd be just swamped in it, it'd be hopeless. But it turned out we did meet that challenge."

Full datasets from the DEEP2 Redshift Survey will be made available to the rest of the research community by depositing them with a data-archiving service, Davis says, while summary datasets will be publicly available on the Internet.

The KDI grant also was utilized in connection with work on AMANDA, a project at the South Pole that has involved drilling deep holes in the icecap and installing in them strings of photomultiplier tubes. Most of the photomultiplier tubes are pointed downward. The aim is to detect light signals emitted when extremely high-energy neutrinos, which have come from deep space and passed through the Earth, interact with the ice.

According to Davis, "funds from the KDI grant were used to support AMANDA research—some postdoctoral fellows who were working on data processing and understand how to cope with large datasets."

Other principal investigators involved with the NSF grant were George F. Smoot, Lawrence Berkeley National Laboratory; Philip B. Stark, Department of Statistics, UC Berkeley; Andrew H. Jaffe, formerly at Berkeley's Center for Particle Astrophysics and now at Imperial College, London; and Joseph I. Silk, formerly with the Astronomy Department, UC Berkeley, and now at Oxford University, U.K.

Back to Top of Page