November 28, 2012
University of Tsukuba
RIKEN
Tokyo Institute of Technology
A research group of University of Tsukuba, RIKEN, and Tokyo Institute of Technology announced that the ACM Gordon Bell Prize (*1) was awarded to that group for the large scale parallel simulation using the “K computer” (*2) at SC12, the International Conference for High Performance Computing, Networking, Storage and Analysis held in Salt Lake City, on November 15 (US Pacific Standard Time). The group performed extremely large simulations with an unprecedentedly high level of efficiency.
The target of the award-winning simulation, presented at SC12, was the gravitational evolution of dark matter (*3) in the early Universe. The number of dark matter particles simulated was two trillions, which is the world’s largest dark matter simulation at present. The execution performance (*4) was 5.67 Pflops on the 98% resources of the K computer system.
This year, the Gordon Bell Prize was awarded singly to the Tsukuba group. A USA group achieved a 14 Pflops execution performance on Sequoia of LLNL for a similar dark matter simulation. Although the peak performance of Sequoia is twice that of K computer, the calculation speed achieved by the Tsukuba group is 2.4 times faster than that by the USA group. This is because the numerical algorithm of the Tsukuba group is more advanced than that of the USA group.
Background
As part of the Strategic Programs for Innovative Research (SPIRE) Field 5 “The origin of matter and the universe” research group led by T. Ishiyama, a research associate in University of Tsukuba, has been developing high performance software for use on the K computer. The group has been developing a massively parallel computing code (*5) based on the TreePM algorithm for large astrophysical N-body simulations and simulated the evolution of dark matter distribution in the early Universe on the 98% resources of the K computer system.
Our Universe is primarily composed of dark matter and dark energy. Dark matter exists about five times as much as baryonic matter. Baryonic matter is the usual one which we can see as atoms or molecules. Since dark matter dominates the gravitational evolution of the Universe, it is important to understand the evolution of dark matter in order to reveal the structure formation of the Universe.
This simulation is still running on the K computer. If this simulation is completed, the fine structure of dark matter existing in the Milky Way and its formation processes can be partly revealed. Finally, it makes great improvement of the strategy for indirect detection of dark matter.
Fig.1 The distribution of dark matter in a 16.8G particles simulation at redshift z=400 (initial), 70, 40, and 31. The redshift is an index for time or distance used in astrophysics. Higher redshift value means earlier Universe. The distribution is almost uniform at the beginning (z=400, about two million years after the Big Bang, the width of the image corresponds to about five light years).
Gradually, dark matter concentrates via the gravity and forms large structures. Bottom-left and bottom-middle images are enlargement of the image of z=31 (about 100 million years after the Big Bang, the width corresponds to about 65 light years).
Research Methods and Results
The research group simulated the evolution of two trillion dark matter particles in the early Universe using the 98% resources (81408 nodes) of the K computer system. They achieved 5.67 Pflops execution performance, which corresponds to 55% of the theoretical peak.
The TreePM method adopted in the software is one algorithm to accelerate the calculation of gravitational interactions. In this method, the short-range force is computed with the tree algorithm while the long-range force with the particle mesh (PM) algorithm. This method has been well approved in this century, and so, several groups have been developing software based on the TreePM method. Until now, parallelization techniques for about one thousand processes were reported.
The Tsukuba group has developed a novel communication algorithm for the long-range part and a highly tuned gravity kernel for the short-range part. The group has succeeded in developing the TreePM software that works with an extremely high level of efficiency on a hundred thousands processes at least.
This software enables us to perform gravitational N-body simulations with more than one trillion particles within a practical period of time. It is the largest gravitational N-body simulation in the world for which the number of particles exceeds one trillion. The simulation is expected to complete within about three days. If we use a single personal computer, it will take more than a few hundred years.
Such simulations enable us to reveal finer and more accurate structures of the Milky Way dark matter as compared to previous simulations. It is important for the detection experiments of dark matter.
The group of Argonne National Laboratory of USA performed similar simulations to those of the Tsukuba group. The Argonne group achieved about 14 Pflops sustained performance on IBM Sequoia with about 20 Pflops theoretical peak performance, which is about twice the peak Performance of K computer. Nevertheless, the software of the Tsukuba group can solve the gravity much faster than that of the Argonne group.
The calculation speed for a dark matter particle is about 2.4 times faster. This means that the software of the Tsukuba group is more advanced in solving large scale simulations. For this reason, the Gordon Bell Prize was awarded to the Tsukuba group in the end.
Regarding the past Gordon Bell Prizes, the group that could access the fastest supercomputers at that time, had the superiority. However, it resulted differently this year. This means that the software of the Tsukuba group is evaluated as the state-of-the-art and leading position in the world.
Future Expectations
The research group has been studying continually the fine structures of dark matter. By combining this software with the SPH (smoothed particle hydrodynamics) algorithm for compressive fluids, the group is addressing early scientific achievements such as star formation and galaxy formation simulated on the K computer.
Paper awarded the Gordon Bell Prize
Tomoaki Ishiyama, Keigo Nitadori, and Junichiro Makino, 2012, “4.45 Pflops astrophysical N-body simulation on K computer: the gravitational trillion-body problem”, in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC ’12), IEEE Computer Society Press, Los Alamitos, CA, USA, Article 5
Glossary and Notes
1. ACM Gordon Bell prize
The ACM Gordon Bell Prize, sponsored by the Association of Computing Machinery, is awarded to promote advancements in parallel computing technology, and is presented every November at the International Conference for High Performance Computing, Networking, Storage and Analysis (SC, Supercomputing Conference) to the papers that show the most outstanding results in hardware and application development.
2. K computer
The K computer, jointly developed by RIKEN and Fujitsu, is part of the High-Performance Computing Infrastructure (HPCI) initiative led by Japan’s Ministry of Education, Culture, Sports, Science and Technology (MEXT). This newly developed supercomputer commenced full operation at the end of September 2012. The “K computer” is the registered trademark of RIKEN. “K” comes from the Japanese Kanji letter “Kei” which means ten peta or 10 to the 16th power. In its original sense, “Kei” expresses a large gateway, and it is hoped that the system will be a new gateway to computational science.
3. Dark matter
It is the matter to interact via the gravity only. The nature of dark matter particle is still unknown. The initial primordial density fluctuations of dark matter grow via the gravity and form dark matter structures with various sizes everywhere. Then, the baryonic matter condenses in the gravitational potential of these dark matter structures, and forms first stars and galaxies. This is the cold dark matter theory, which is widely accepted as the standard theory of the formation and evolution of the Universe.
4. Execution performance
In contrast to theoretical performance, this is the computational performance obtained in running a given program, and acts as a practical measure of a computer’s performance.
5. Massively parallel code
An application program that can run efficiently on a massively parallel computer.
Press Contacts
Center for Computational Sciences, University of Tsukuba
Public Relations Office
Tel: +81-29-853-6260 E-Mail: pr [at] ccs.tsukuba.ac.jp
RIKEN Advanced Institute for Computational Science
Office for Research Communications
Tel: +81-78-940-5623, 5624
Tokyo Institute of Technology
Center for Public Information
Tel: +81- 3-5734-2976
About University of Tsukuba
The University of Tsukuba aims to establish free exchange and close relationship in both basic and applied sciences with educational and research organizations and academic communities in Japan and overseas. The university makes a contribution to the world through its educational system that seeks to make the most of students’ creativity and individuality.
http://www.tsukuba.ac.jp/english/ http://www.ccs.tsukuba.ac.jp/
About RIKEN
RIKEN is Japan’s flagship research institute devoted to basic and applied research. Over 2500 papers by RIKEN researchers are published every year in reputable scientific and technical journals, covering topics ranging across a broad spectrum of disciplines including physics, chemistry, biology, medical science and engineering. RIKEN’s advanced research environment and strong emphasis on interdisciplinary collaboration has earned itself an unparalleled reputation for scientific excellence in Japan and around the world. For more information, please see: http://www.riken.jp/
About Tokyo Institute of Technology
Tokyo Institute of Technology, also known as Tokyo Tech, is one of Japan’s leading research universities, providing a focused, scientifically and technologically based education to about 10,000 undergraduate and graduate students. Tokyo Tech has many nationally recognized programs, all top-ranked by peers and publications alike, and is ranked in the 65th in overall rankings and 19th in Engineering & Technology in the QS 2012 World University Rankings. As a leading scientific and technological university, Tokyo Tech has three undergraduate schools, six graduate schools, six research laboratories, and many research and service centers that consistently contribute vital research and innovation to industry and business. For more information, please see: http://www.titech.ac.jp/english/