Full waveform inversion : GPU development of modeling and inversion algorithms

Postdoc offer

Frame

Full waveform inversion (FWI) is a high resolution seismic imaging technique based on an iterative data fitting procedure (Virieux & al., 2017). The match between observed data and synthetic data computed through the numerical solution of partial differential equations is iteratively improved following local optimization methods. Since 2010 and a spectacular application on 3D field data from the North sea (Sirgue et al., 2010), FWI has become a standard velocity building tool in the exploration industry. It has been also adopted for imaging the Earth’s structure at various scales, from global-regional tomography, to near surface characterization for the detection/monitoring of fluid contents.

One of the key limitation for applying FWI is its intrinsic computational cost. Basically, one wave propagation problem has to be solved for each seismic source at each iteration of the inversion process. Such wave propagation problem exhibits a computational complexity in O(ω4) where ω corresponds to the higher end of the frequency spectrum of the propagated signal. For realistic size problems, the number of sources can reach several thousands.

The resolution power of FWI being driven by the frequency, there is a constant interest to increase the considered frequency (with applications reaching 100 Hz in the acoustic approximation). Another ongoing trend consists also in considering a more complex wave propagation physics, from acoustic propagation towards visco-elastic propagation. Such a richer physics can in turn provide the tools to better characterize the subsurface through multi-parameter estimation (basically P-wave and S-wave velocities in the elastic approximation instead of only P-wave velocity estimation).

Within the SEISCOPE project, we have developed a FORTRAN90 code based on a spectral element discretization of the elastodynamics equation named SEM46 (Trinh et al., 2019 ; Cao et al., 2022b). This package is dedicated to full waveform modeling and inversion in the visco-elastic approximation. Thanks to a double level MPI parallelization (sources + domain decomposition), SEM46 exhibits very good scaling properties for problems involving up to several tens of billions degrees of freedom on up to several tens of thousands of cores.

The current implementation is however CPU based. The newest exascale machines available in the national computing centers however rely more and more on hybrid architectures, involving CPU and GPU (and/or even ARM architecture), the reason being the energetic efficiency of GPU and ARM is higher than that of CPU. The main objective of this project is thus to port the SEM46 main kernels on GPU architectures.

Project

The main kernels of SEM46 which need to be ported on GPU are related with

• computing the incident field in the (visco-)elastic approximation
• computing the adjoint field in the (visco-)elastic approximation
• building the gradient from the incident and adjoint field with an appropriate strategy
• incorporate the fluid-solid coupling in the GPU implementation

Step 1 has already been performed by a former post-doc of our group using a CUDA-based GPU implementation. A smart partitioning strategy is implemented to accelerate the computation of this kernel on GPU, and a CUDA-MPI language is used for an efficient multi-GPU parallelization not requiring to go through the CPU to communicate between GPU (Cao et al., 2022a). The project will start by taking the hand on these development and port them on available NVIDIA GPU architecture (for instance Jean Zay GPU partition at Idris center).

Step 2, 3 and 4 remain to be done and will constitute the next steps of the project. Once the kernels will be implemented, the implementation will be tested and validated against the CPU version of SEM46 and models of increasing complexity will be considered. We have in mind to test performance scalabilities with multiple GPU on the challenging SEAMII foothill models, as well as on a specifically designed 3D models by one of our sponsor.

Additional investigation could be performed along the porting towards AMD GPU architectures. The newest available GPU machines are AMD-based, therefore there is an interest to have a versatile implementation able to run on both NVIDIA and AMD GPU. Another possibility of investigation is to extend the GPU porting to the other main kernels already developed in CPU in SEM46 : the Bessel smoothing kernel (regularization technique for our FWI algorithm) and the homogeneization kernel (computation of smooth anisotropic equivalent media, Cao et al. (2023))