Palabos is a general software library for computational fluid dynamics using the lattice Boltzmann method. In biomedical research, the main domains of applications are cardiovascular, including the simulation of blood flow in arteries, the investigation of the effect of medical devices such as stents, and cell-level blood simulations to investigate fundamental blood properties. Principal target users are researchers in the biomedical domain and R&D developing new medical devices.
The code is open-source and is distributed under the terms of the software license AGPLv3. It can be downloaded from the Web site www.palabos.unige.ch and is straightforward to install, as it depends on no external library (except for an implementation of the MPI communication layer).
Technical specifications
The Palabos library is written fully in C++ and runs on CPU clusters. It is parallelized with MPI for multi-core CPUs and for CPU clusters, using one MPI task per core (single-run parallelism). Typical memory requirements are 2 GB per core. Pre- and post-processing capabilities are built-in to the Palabos software library. A recommended data visualization tool for Palabos output is the ParaView software.
HPC usage and parallel performance
The need for HPC is particularly motivated in the field of cell-level blood simulations. Here, an exascale or pre-exascale system is required to simulate a macroscopically significant number of blood cells (of the order of 1 billion cells) while maintaining a detailed description of the physics of each cell. Cell-level simulations are achieved with the Palabos-npFEM coupling. A multi-scale description, i.e., simulation of a large blood volume (a section of an artery) while maintaining the cell-level description, is achieved in the limit of weak scaling, as the blood volume (and the total number of blood cells) scales with the number of available nodes.
The figure above reports the scaling behaviour of the Palabos-npFEM code, as it simulates a coupled fluid – Red Blood Cell system on a heterogeneous platform (fluid on CPU, blood cells on GPU). The graph shows weak scaling, and the size of the simulated blood volume (in micro-meters) is indicated for each data point.
Current research focuses on scaling the Palabos-npFEM code to a larger number of nodes, and on porting the full Palabos library to GPU for more efficient large-scale simulations of arterial flows. Porting of the Palabos library to single GPU has been achieved. Current and future efforts focus on multi-GPU performance of the full Palabos library.
The multi-GPU performance of Palabos was among others improved by participating in the hackathon Eurohack21 in September 2021, organized by the Swiss supercomputing centre CSCS. On this occasion, an exchange between the Palabos development team and HPC experts from CSCS and from NVIDIA took place. This allowed the Palabos team to make suggestions for extensions to the NVIDIA HPC SDK that would allow HPC codes like the Palabos one to be ported to an exascale capable machine more efficiently.
