Alya

The code is written in modern Fortran language, with some user tools in Python and Perl. The memory is dynamically allocated and strongly depends on the problem solved. The code has no external third-party high-level library dependency, the only dependencies are of low-level types, such as MPI.

The code has been compiled in several platforms such as ARM, Intel, NVIDIA, or IBM Power processors. The parallelization strategy is a hybrid one, MPI+OpenMP. Part of the code has been ported to GPUs to offload solvers or matrix assembly. The code has examples of use as monolithic, coupled and in ensembles, and for replica computing. Being portable and efficient in both large and small architectures, it has records on different running scenarios. The code relies mostly on internally developed tools. There is no need for containers or workflow managers. Several pre-processing tools have been used in combination with Alya: Ansa, IcemCFD, GMesh, or GiD. As for post-processing tools, typically used applications include ParaView, VisIt, Ensight, or GiD.

The code is used in complex very large, coupled problems and it is designed and continuously improved to run efficiently in parallel for coupled problems. If not carefully done, coupling can strongly punish performance: a code can be very efficient solving a fluid or a solid mechanics problem, but this does not preclude that a fluid-solid interaction case could become extremely inefficient. We use a multiple instance point-to-point MPI communication strategy for coupled problems, allowing us to run the individual instances in different architectures with a heavy communication pattern. We also combine this with load balancing schemes by linking Alya to the BSC's DLB tool for dynamic load balance. For all this, the use of HPC combined with an efficient yet very advanced programming and running strategy becomes important. 

The strong scalability of Alya has been established in PRACE, Alya being part of the Unified European Applications Benchmark Suite (UEABS), as well as through continuous testing on the main supercomputers in Europe and the US. The continuous development and improvement of our HPC capabilities have been carried on since the early days of the Alya Development Team, in 2005, through several collaboration projects. Since then, we have focused on all the aspects of the problem, and lately with the exascale horizon in sight. Our current strategy is a tick-tock model, in which we dedicate alternative efforts, on one hand, efficiency at system level in large core counts (weak/strong scalability, communication bottlenecks, partition algorithms, system level load balance, etc.), and, on the other hand, efficiency at node level (vectorization, hybrid models, porting to new architectures, co-execution, node level load balance, etc.). Of course, on the road to exascale all levels are accountable. All phases, when completed, contribute to a general strategy. On top of that, for a few years, we have been working on combining Machine Learning with modelling and simulation, all in the context of High Performance Computing and exascale. At this moment, according to our long-term development plan, we are finishing one of our node level efficiency phases. As mentioned above, the CompBioMed2 applications which Alya is tackling are all large and tightly coupled. Alya consists of many different modules, each accounting for different physics and solving for a particular set of equations. Some of them, including the one of interest in CoE RAISE (https://www.coe-raise.eu/), have been ported to GPGPUs using OpenACC pragmas. In addition, a co-execution model has been developed to enhance the resources usage. The idea of co-execution is to have the same computational kernels running at the same time on both CPUs and GPGPUs and therefore to fully exploit hardware heterogeneity [1,2]. Some of the kernels of Alya have also been ported to FPGAs in the context of the LEGaTO European project (https://legato-project.eu/). Alya is general purpose engineering application used and developed within several CoEs working in different scientific domains. Since its conception in 2004, Alya has being designed and extended using advanced High Performance Computing programming techniques to solve coupled problems on supercomputers efficiently, making it one of the most promising European scientific application which is expected to be taken to the exascale. Several co-design activities have been focusing around Alya in dedicated EU projects such as the European Processor Initiative (EPI, https://www.european-processor-initiative.eu/) and the Excellerat CoE (https://www.excellerat.eu/) where the code is one of the main supported applications. From a co-design prospective, Alya can indeed efficiently exploit both latency-oriented CPUs and throughput-oriented GPUs. Those are the two parallelization models required on the upcoming pre-Exascale EU supercomputers. The portability to GPUs has implications both in the programming language and on the data structuring. It is being carried progressively in the distinct modules of Alya following a common strategy. In particular, the N-S solver, critically used in many biomedical applications, was completely ported to GPUs and run on a POWER9+NVIDIA system [1] and we aim to port all the modules involved in multiphase combustion simulations to GPUs within The Center of Excellence in Combustion (CoEC, https://coec-project.eu/). Additionally, it is worth remarking that partner BSC is leading the MareNostrum Exascale Experimental Project (MEEP, https://meep-project.eu/) which is a flexible FPGA-based emulation platform that will explore hardware/software co-design for Exascale Supercomputers and other hardware targets, based on European-developed IP (Intellectual Property). Within this project, BSC is working to foresee the behaviour of Alya's computational kernels in FPGA-based emulators of future exascale computers, especially with those architectures proposed by the EPI initiative.

Cited references:

[1] R. Borrell, D. Dosimont, M. Garcia-Gasulla, G. Houzeaux, O. Lehmkuhl, V. Mehta, H. Owen, M. Vázquez, and G. Oyarzun. Heterogeneous CPU/GPU co-execution of CFD simulations on the POWER9 architecture: Application to airplane aerodynamics. Future Generation Computer Systems, 107:31-48, 2020.

[2] G. Oyarzun, M. Avila, R. Borrell, G. Houzeaux, O. Lehmkuhl, H. Owen. Explicit/Implicit Navier-Stokes Solver on CPU/GPU heterogeneous Supercomputers, PARCFD2020, Paris, May 2021.