DataCenter design model; photo courtesy of UW News Service

Advanced Computational Methods for Fluid Dynamics with an Eye on the NCAR Supercomputer Center

March 12, 2007 - Researchers in the Mechanical Engineering department, under the direction of Professor Dimitri Mavriplis, are developing the next generation of advanced computational methods for simulating complex fluid dynamic problems. These methods will enable highly accurate simulations on complex engineering problems using state-of-the-art massively parallel computer hardware. The goals of the research are to advance existing capabilities in computational fluid dynamics (CFD) simulations by several orders of magnitude over the next five to ten years, turning currently intractable problems in CFD into everyday production calculations. These objectives are being met by a combination of new algorithms rooted in applied mathematics and fluid mechanics, and the development of new software, which runs efficiently on the emerging class of massively parallel computers. Such a system is the planned NCAR Super computer to be built in Wyoming, which will harness the power of over 100,000 cpus or cores to work in unison.

Computational fluid dynamics (CFD) is a branch of the broader field of Computational Science and Engineering (CSE), in which important physical problems in science and engineering are simulated by solving the equations governing these processes numerically, using computers. In a typical CFD simulation, the volume occupied by the fluid (or air) is divided up into a large number of small cells (triangles or tetrahedra) and the values of the fluid velocity, pressure, density, and temperature are computed for each cell. The effect of the fluid flowing from one cell into its neighbors implies that the values in each cell are related to the values in the neighboring cells. The calculation leads to the requirement of solving a system of simultaneous equations with many unknowns or degrees of freedom. Because there are so many degrees of freedom, the entire problem is too large to be solved by a single desktop computer, and must be partitioned on many computers or cpus, with each cpu being responsible for a portion of the problem. Once each cpu has computed its portion, it must communicate these results to the neighboring partitions which reside on other cpus. The goal of the CFD software is to solve the large set of simultaneous equations with as few calculations as possible, while running efficiently on large numbers of communicating cpus.

A state-of-the-art aerodynamic calculation was performed by Prof. Mavriplis on an aircraft geometry similar to that shown in the above illustration, using a total of 255 million degrees of freedom, running on up to 4000 cpus of the NASA Columbia supercomputer. This calculation required under 10 minutes of total computer time, and formed the basis of a paper presented at the Supercomputing 2005 conference in Seattle WA. The paper was awarded the ”best paper” prize of the conference. However, even for this large problem, which stretches the limit of currently available resources, the delivered accuracy is only a fraction of what is ideally desired in the aircraft design process, and thus simulations of much higher resolution (orders of magnitude more cells or degrees of freedom) are desired. Furthermore, much more complicated problems need to be solved, including problems with moving bodies, such as a maneuvering aircraft, or a helicopter with moving rotors, as well as problems involving multiple physical processes, such as structural deformations, chemically reacting flows, and heat transfer. Such problems can be found in diverse areas of science and engineering, such as gas turbine flows, high-speed re-entry flows for space missions, nuclear reactor core simulations, and weather and climate simulations for atmospheric flows. All these requirements lead to very high requirements in computational power, making all but the simplest of these problems currently intractable.

In order to advance the state-of-the-art, ever larger supercomputers with larger numbers of cpus or cores are being developed and deployed, requiring further research in areas of computer science for enabling efficient implementation of current algorithms on these architectures. At the same time, the development of smarter or more efficient algorithms is also required in order to achieve similar or superior results with fewer overall calculations. One example of an approach in this direction is the development of higher-order Discontinuous Galerkin (DG) methods, under development in the research group of Prof. Mavriplis. High-order DG methods are mathematically more sophisticated than most current-day methods, but enable the calculation of very high accuracy results with many fewer unknowns or degrees of freedom. Another approach to smarter algorithms involves the development and use of adjoint methods for sensitivity and error estimation. For example, adjoint methods can be used to calculate regions in the domain that are most responsible for errors or inaccuracies in the simulation results of interest. Thus, increased accuracy can be achieved at reduced computational expense by using adjoint methods to guide decisions on where best to increase the resolution, as opposed to increasing the cell resolution in all areas of the domain, which quickly becomes impractical when dealing with limited computational resources. Adjoint methods are also important for data-assimilation techniques, often used in the simulation of atmospheric flows, where new experimental data which becomes available as the simulation progresses need to be incorporated or assimilated into the simulation procedure in order to improve the simulation accuracy. The importance of developing such methods is that they can lead to one or more orders of magnitude reduction in the computational expense for a given problem.

The fundamental nature of these algorithms means that they have equally important applications in diverse application areas such as aerodynamics, nuclear reactor core simulations, and climate simulation for atmospheric flows. This overlap with other disciplines has prompted collaboration between researchers in the group of Prof. Mavriplis with researchers at various National Laboratories including the National Center for Atmospheric Research (NCAR) in Boulder CO. It is through this collaboration that discussions on the planned NCAR Supercomputer center were initiated between NCAR and UW, which eventually lead to the selection of UW and Wyoming as primary partners in the NCAR Supercomputer center. This new formal partnership between NCAR and UW will not only greatly enhance our access in Mechanical Engineering and across the UW campus to state-of-the-art computational facilities, but will foster a strong collaboration between UW, NCAR and other Front Range Institutions in the important multidisciplinary area of Computational Science and Engineering.

Prof. Mavriplis’ research program is supported by the NASA Aeronautics Mission Directorate, the Air Force Office of Scientific Research, the Office of Naval Research, the Army Research Office, and the National Science Foundation. Significant contributions have been made by postdoctoral researchers Zhi Yang and Cris Nastase, as well as by the doctoral students Li Wang, Karthik Mani, Pramod Singh and Jing Liu.