Parallel Cross Section Reader for Abeille
High Performance Computing
As part of my Parallel Computing course work, I conducted a scalability and performance study of Abeille, an open-source Monte Carlo neutron transport code, on NERSC’s Perlmutter supercomputer. The goal was to understand how well Abeille scales to leadership-class systems and to remove the primary bottlenecks that limit large-scale simulations.
We found that while Abeille’s transport kernel scales well, reading dozens of small nuclear data files (ACE format) from the parallel filesystem becomes a significant bottleneck at scale — when run on many MPI ranks. To address this, we developed and evaluated two strategies:
- Consolidating ACE files into a single file using ADIOS2 or HDF5 (using HighFive) for parallel reading, and
- Reading the data on a single MPI rank and broadcasting it to all ranks.
Key Findings
- The PowerIterator (particle transport) phase shows strong scaling up to 64–128 nodes when particle counts per thread are sufficient.
- Overall weak scaling degrades beyond ~16 nodes due to filesystem I/O saturation during ACE parsing.
- Using ADIOS2 or HighFive to read from a single file reduces parsing time from >25 seconds to <1–2 seconds, a 25x–50x speedup.
- ADIOS2 (BP5) delivered the best performance and slightly reduced data size (~28 GB to ~26 GB).
- At higher MPI counts, disk contention reappears; the single-rank read + MPI broadcast approach scales best, achieving ~4 seconds parsing time up to 256 ranks on Perlmutter.
This study demonstrates that consolidating nuclear data into a single file and limiting I/O to a small number of MPI ranks — rather than having all ranks read independently — can significantly improve scalability on large HPC systems. While the results are presented using Abeille, the underlying issue and solution are broadly applicable to other Monte Carlo transport codes such as OpenMC.
These improvements are especially important for coupled multiphysics simulations that require many repeated Monte Carlo solves, as well as for depletion calculations where large numbers of cross-section files must be read at every step.
This work was carried out on NERSC’s Perlmutter system with guidance from Prof. Jacob Merson and Prof. Hunter Belanger.