SOLVING EINSTEIN'S EQUATIONS ON SUPERCOMPUTERS
Courtesy of December 1999 issue of Institute of Electrical & Electronics Enginners'
Computer Society Journal
In 1916, Albert Einstein published his famous general theory of relativity, which contains the rules of gravity and provides the basis for modern theories of astrophysics and cosmology. It describes phenomena on all scales in the universe, from compact objects such as black holes, neutron stars, and supernovae to large-scale structure formation such as that involved in creating the distribution of clusters of galaxies. For many years, physicists, astrophysicists, and mathematicians have striven to develop techniques for unlocking the secrets contained in Einstein’s theory of gravity; more recently, computational-science research groups have added their expertise to the endeavor.
Those who study these objects face a daunting challenge: The equations are among the most complicated seen in mathematical physics. Together, they form a set of 10 coupled, nonlinear, hyperbolic-elliptic partial differential equations that contain many thousands of terms. Despite more than 80 years of intense analytical study, these equations have yielded only a handful of special solutions relevant for astrophysics and cosmology, giving only tantalizing snapshots of the dynamics that occur in our universe.
Scientists have gradually realized that numerical studies of Einstein’s equations will play an essential role in uncovering the full picture. Realizing that this work requires new tools, developers have created computer programs to investigate gravity, giving birth to the field of numerical relativity. Progress here has been initially slow, due to the complexity of the equations, the lack of computer resources, and the wide variety of numerical algorithms, computational techniques, and underlying physics needed to solve these equations.
A realistic 3D simulation based on the full Einstein equations is an enormous task: A single simulation of coalescing neutron stars or black holes would require more than a teraflop per second for reasonable performance, and a terabyte of memory. Even if the computers needed to perform such work existed, we must still increase our understanding of the underlying physics, mathematics, numerical algorithms, high-speed networking, and computational science. These cross-disciplinary requirements ensure that no single group of researchers has the expertise to solve the full problem. Further, many traditional relativists trained in more mathematical aspects of the problem lack expertise in computational science, as do many astrophysicists trained in the techniques of numerical relativity. We need an effective community framework for bringing together experts from disparate disciplines and geographically distributed locations that will enable mathematical relativists, astrophysicists, and computer scientists to work together on this immensely difficult problem.
Because the underlying scientific project provides such a demanding and rich system for computational science, techniques developed to solve Einstein’s equations will apply immediately to a large family of scientific and engineering problems. We have developed a collaborative computational framework that allows remote monitoring and visualization of simulations, at the center of which lies a community code called Cactus. Many researchers in the general scientific computing community have already adopted Cactus, as have numerical relativists and astrophysicists. This past June, an international team of researchers at various sites around the world ran, over several weeks, some of the largest such simulations in numerical relativity yet undertaken, using a 256-processor SGI Origin 2000 supercomputer at NCSA.
The scientific visualization of Einstein’s equations as numerical simulations presents us with many challenges. The fundamental objects that describe the gravitational field form components of high-rank, four-dimensional tensors. Not only do the tensor components depend on the coordinates in which they are written, and thus have no direct physical meaning themselves, they are also numerous: The quantity describing space-time curvature has 256 components. Further, interpreting these components is not straightforward: Pure gravity is nothing more than curved “vacuum,” or empty space without matter. Standard intuition in terms of, say, fluid quantities such as density or velocity, do not carry over easily to relativity. Thus, new paradigms for visualization must be developed. The many quantities needed to describe the gravitational field require an enormous volume of data that must be output, visualized, and analyzed.
Figure 1 shows one of the first detailed simulations depicting a full 3D merger of two spinning and orbiting black holes of unequal mass. (The “Black Holes and Gravitational Waves” sidebar describes black holes in greater detail.) We define the black holes by the surfaces of their respective horizons—in our case, and for rather complex reasons, we compute the apparent horizon. The horizon forms the critical 2D surface that separates everything trapped forever inside from everything that could possibly escape. This surface responds to local variations in the gravitational field by distorting and oscillating. In Figure 1, the two black holes have just merged in a grazing collision; the merged single black hole horizon, with its two black holes inside, is at the center. The collision emits a burst of gravitational waves that radiates away toward infinity.
Figure 1. Gravitational waves from a full 3D grazing merger of two black holes. The image shows the objects immediately after the coalescence, when the two holes (seen just inside) have merged to form a single, larger hole at the center. The distortions of the horizon (the Gaussian curvature of the surface) appear as a color map, while the resulting burst of gravitational waves (the even-parity polarization or real part of Y4) appears in red and yellow.
The content of gravitational radiation in the system can be calculated from the 10 independent components of the so-called Riemann curvature tensor (in turn represented by five complex numbers Y0, … Y4). For simplicity, we refer loosely to the real and imaginary parts of the quantity as the two independent physical polarizations of the gravitational waves. In Figure 1, the real part of Y4 shows how some energy from the cataclysm radiates away. (Most of the visualizations in this article, as well as the cover image, were produced with Amira, described in the “Amira” sidebar.)
To fully evaluate the science in this complex simulation, we must examine all aspects of the gravitational waves and other quantities. Figure 2 shows the other polarization of the wave field, the imaginary part of Y4. This additional component of the gravitational radiation does not occur in previously computed axisymmetric collisions and shows interesting phenomena uniquely associated with 3D calculations.
Figure 2. Additional polarization of gravitational waves (imaginary part of Y4) from a 3D merging collision of two black holes. The merged single black-hole horizon can just be seen through the cloud of radiation emitted in the process.
Studying the horizon surface of the black hole by itself provides further insights. The coordinate location or coordinate shape of the horizon does not reflect its physical properties; those properties change with different coordinate choices. However, the (Gaussian) curvature of the surface is an intrinsic property of the horizon’s true geometry and thus does reflect its physical properties. We can compute this quantity and color-map it to the surface of the horizon to understand how deformed it becomes during the collision. Figure 3 shows the results.
Figure 3. Close-up of the horizon—with Gaussian curvature to show the distorted nature of the surface—of a black hole formed by the collision of two black holes. The two individual horizon surfaces can just be seen inside the larger horizon formed during the collision process.
The fabric of space-time may also change depending on the initial concentration of pure gravitational waves. A weak concentration lets the evolving gravitational radiation just “flow away,” like shallow ripples on a pond. However, a large-enough energy concentration in the gravitational wave can implode violently to create a black hole with, again, small ripples flowing away. At this point, comparing the gravitational waves’ field indicators and the oscillations of the Gaussian curvature of the newly formed black hole can be especially informative. For the simulation presented here, the black hole assumes for its final state a static spherical “Schwarzschild black hole,” but the initial black hole that forms will be highly distorted. These distortions radiate away during the evolution. We thus expect to see gravitational radiation correlated with the structures of the Gaussian curvature. By displaying the positive and negative values of the Y4 scalar in full 3D volume, using special volume rendering techniques and overlaying the apparent horizon, we can make this correlation visible, as Figure 4 and Figure 5 show.
Figure 4. Gravitational waves and horizon of a newly formed black hole, caused by massive collapse of a strong gravitational wave on itself. The dotted surface shows the horizon, where green colors indicate high curvature and yellow means zero curvature. The highest curvature indicates the largest gravitational radiation. The “distortion” (the non-sphericity) of the black hole radiates away over time, in accordance with mathematical theorems about black holes.
Figure 5. Horizon of a gravitational wave that implodes to form a black hole, with leftover waves escaping, shown at a later time in the same evolution presented in Figure 4. We see that the horizon curvature is affected by, and correlated with, the evolving gravitational wave.
The foregoing synopsis briefly describes what is going on in one "state of the art" field of astrophysics. I thought that sharing it with those would be astrophysicists of you would be fun and games.