Full waveform inversion : towards exascale computing and uncertainty quantification

Postdoc offer

Frame

Full waveform inversion (FWI) is a high resolution seismic imaging technique based on an iterative data fitting procedure \citepVirieux_2017_FWI. The match between observed data and synthetic data computed through the numerical solution of partial differential equations is iteratively improved following local optimization methods. Since 2010 and a spectacular application on 3D field data from the North sea (Sirgue_2010_FWI), FWI has become a standard velocity building tool in the exploration industry. It has been also adopted for imaging the Earth’s structure at various scales, from global-regional tomography, to near surface characterization for the detection/monitoring of fluid contents.

One of the key limitation for applying FWI is its intrinsic computational cost. Basically, one wave propagation problem has to be solved for each seismic source at each iteration of the inversion process. Such wave propagation problem exhibits a computational complexity in O(w4) where w corresponds to the higher end of the frequency spectrum of the propagated signal. For realistic size problems, the number of sources can reach several thousands.

The resolution power of FWI being driven by the frequency, there is a constant interest to increase the considered frequency (with applications reaching 100 Hz in the acoustic approximation). Another ongoing trend consists also in considering a more complex wave propagation physics, from acoustic propagation towards visco-elastic propagation. Such a richer physics can in turn provide the tools to better characterize the subsurface through multi-parameter estimation (basically P-wave and S-wave velocities in the elastic approximation instead of only P-wave velocity estimation).

Finally, FWI being a nonlinear ill-posed inverse problem, relying on a single estimate of its solution seems not a good option. Uncertainty quantification strategies are thus required to better characterize its solution. Recently, we have shown how this issue can be targeted using Ensemble Transform Kalman Filters (ETKF) from the data assimilation community (Thurin_2019_UQC,Hoffmann_2022_UQ3).

All these factors put together show how demanding FWI is going to be in the coming years in terms of computing strategies. This calls for algorithm able to exploit the newest exascale HPC machines which start to be available to research groups. This is the topic of this project.

Project

Two main FORTRAN90 codes have been developed within the SEISCOPE project. The first is named TOYxDAC_TIME. It is based on a visco-acoustic modeling engine, discretized with a 4th order finite-difference technique. Its implementation relies on a double level of parallelism : MPI for the source distribution, and OpenMP for the spatial discretization (a prototype version implements three level : MPI for sources, then domain decomposition with MPI and OpenMP on each subdomain). The second is named SEM46. It is based on a visco-elastic modeling engine discretized with a spectral element technique. Its implementation relies on double level of parallelism : MPI for the source distribution and MPI for an inner level of domain decomposition.

For both codes, the double (even triple) level of rigid parallelization through MPI and/or OpenMP is not flexible enough for an efficient use of computational resources beyond 10,000 cores. Beyond this number, the probability of failure on a given core starts to increase too drastically for such a technique to be sufficiently resilient/robust. The first aim of the project is to re-design our codes structure using a lower level of granularity, at the one-gradient computation level, for one source. Then, the objective will be to use/develop a task-scheduling manager (in a language to be defined, Python, Shell, others) interacting with the job-scheduler of clusters such that the FWI code can be deployed to be adapted dynamically to the available computational resources, managing possible failure/errors of some jobs to relaunch them.

Once this algorithm will have been tested and validated, the second part of the project will consist in embedding this task scheduling algorithm in a higher level framework to implement uncertainty quantification using our ETKF approach. Within this approach, instead of solving 1 FWI problem from a given initial model, an ensemble of M initial models is considered, and M FWI problems are solved in parallel, with statistics to be inferred from the M parallel evolving models at known in advance meeting points. Another task scheduling could thus be also used to implement this ETKF strategy, overlaying the previous task scheduling algorithm setup in the first phase.

The SEISCOPE group being granted access to French national computing center, it is expected to test these approaches on real scale during the project, and to validate the development so that they can benefit to the ensemble of actors of the SEISCOPE group. The machines at TGCC (Irene Rome) and IDRIS (Jean Zay) are the primal targets, but the new machine at CINES (Adastra) could also be considered.