Based on a November 18, 2022 article by Patrick Riley, Berkeley Lab Computing Sciences

An international team of scientists led by Jean-Luc Vay, head of ATAP’s Accelerator Modeling Program (AMP), has been honored with the 2022 ACM Gordon Bell Prize. Vay’s team—which includes Axel Huebl and Remi Lehe from AMP, Andrew Myers and Weiqun Zhang from Berkeley Lab’s Applied Mathematics & Computational Research Division, and Kevin Gott from NERSC—collaborated with a team led by Henri Vincenti, physicist and head of the Theory and Modelling Physics Group in the High Intensity Program at CEA (the French Alternative Energies and Atomic Energy Commission) and other key organizations,

 

The prestigious award, presented to the team on November 17, 2022, comes with $10,000 in prize money and recognizes outstanding achievements in high-performance computing applied to challenges in science, engineering, and large-scale data analytics.

The research described in the winning paper, “Pushing the Frontier in the Design of Laser-Based Electron Accelerators with Groundbreaking Mesh-Refined Particle-In-Cell Simulations on Exascale-Class Supercomputers,” could help revolutionize radiotherapy treatments and, eventually, lead to major improvements for the next generation of particle accelerators devoted to high-energy physics. Their new simulation code (computer program), called WarpX, could see application in several areas of research.

“We are so excited to receive the award,” said Vay,” and is recognition of all of our hard work in supercomputing as well as pushing new ideas in advanced algorithms and physics.”

Pushing the boundaries of laser-based electron accelerators… and of codes

The team led by Jean-Luc Vay, head of ATAP’s Accelerator Modeling Program, collaborated with the team of Henri Vincenti, physicist and research director at CEA (the French Alternative Energies and Atomic Energy Commission) and other key organizations.

Vay said the research, which builds on and extends earlier work by his and Vincenti’s teams, presents a first-of-kind mesh-refined massively parallel particle-in-cell (PIC) code for kinetic plasma simulations, optimized on a quartet of supercomputers that includes Frontier, Fugaku, Summit, and Berkeley Lab’s Perlmutter. “In addition to melding mesh refinement with PIC codes, developing such a simulation code that runs efficiently at scale is completely new.”

“We’re really pushing here – in three directions,” Vay added. Those directions are improving the state of the art in particle accelerators, notably including laser-driven machines like those being developed at ATAP’s BELLA Center, while at the same time operating on the cutting edge of algorithms and developing a versatile code that is compatible with different types of supercomputers.

WarpX “enabled 3D simulations of laser-matter interactions on Frontier, Fugaku, and Summit, which have so far been out of the reach of standard codes,” the researchers write in the paper’s abstract. “These simulations helped remove a major limitation of compact laser-based electron accelerators, which are promising candidates for next-generation high-energy physics experiments, ultra-high dose rate FLASH radiotherapy, and other applications.”

Improved radiotherapy in a FLASH

In FLASH radiotherapy, a therapeutic dose is delivered in a very short time (a few seconds) at a much higher dose-rate than in conventional treatment protocols. After decades of anecdotal reports on the FLASH radiotherapy effect, more recent experiments have demonstrated a strong difference in sensitivity of healthy and unhealthy tissues to ionizing radiation delivered within short and bright pulses. However, to date, mechanisms behind the benefits of ultra-high dose rate radiotherapy have not been elucidated. This understanding requires a deeper insight into the basis of radiation toxicity on biological samples at disparate timescales, ranging from the femtoseconds (molecule excitation) to the hour (cellular response) and beyond.

Toward this end, the teams led by Vay and Vincenti used complex WarpX simulations to model a novel conceptual design of a laser-based electron injector and accelerator. This design can produce highly charged and very short particle electron beams of high quality that have the potential to deposit a large enough therapeutic dose at ultra-high peak dose rates in tumor cells without damaging other cells.

By using plasma mirrors, the team’s accelerators can deliver a much larger dose of energy in a much shorter time than current conventional methods. Moreover, in order to penetrate deeper into the human body and treat tumors, very large and costly accelerating structures are currently necessary.

“With laser plasma accelerators,” Vincenti said, “we can shrink the distance and enable very high-energy electrons to … democratize, let’s say, FLASH radiotherapy in medical centers at a cheaper cost and at a smaller scale.”

Other potential applications include the building of miniature x-ray free electron lasers for the science of ultrafast phenomena, as well as the development of particle colliders that could be more compact and — the hope is — cheaper than the current generation. To be sure, putting the HEP applications into practice may still take decades.

“This is, I would say, really at the research level of maturity, and computer simulations are key to establishing that it can be done,” Vay said. “It’s very promising.”

Labwide and international synergy of expertise for better modeling

Modeling the project presented a challenge. The main obstacle in modeling the plasma mirror design stemmed from the huge interval of spatial and temporal scales, the scientists noted. To help overcome that hurdle, the team used WarpX, a code developed via their ECP project, Vay noted. WarpX combines the adaptive mesh refinement (AMR) framework AMReX, developed at Berkeley Lab, with novel computational techniques. AMR is akin to a computational microscope, allowing researchers to zoom in on specific regions of space. Berkeley Lab’s Andrew Myers and Weiqun Zhang, both key members of the AMReX development team, are co-authors on the WarpX paper.

Introductory shot of Axel Huebl video

This short video celebrating the unveiling of the National Energy Research Supercomputing Center’s Perlmutter system features Axel Huebl and WarpX simulations of particle accelerators. The Accelerator Modeling Program’s WarpX code had the honor of being among the first batch of jobs in the inaugural run of Perlmutter at its May 27, 2021 dedication.

“AMReX is a numerical library that helps us implement physical block-structured mesh refinement algorithms,” said Axel Huebl, a computational physicist at Berkeley Lab who is now a lead developer of WarpX and co-lead author of the Gordon Bell paper. “Mesh refinement enables us to focus the computational power on the most interesting parts of a simulation, while staying effective for the larger, macroscopic evolution of, for instance, the plasma physics we model.”

Though work on this paper began in the spring of last year, its foundation was laid through the group’s affiliation with the ECP. “It’s really years of hard work from many people,” Vay said, adding that the team recently also had to pull long shifts to make efficient use of the various supercomputers when they gained access for brief windows of time.

Thanks to the ongoing collaboration through CEA and RIKEN, the researchers had full access to Fugaku, located in Japan, for 12-hour spans. “We had to stay connected on the machine to follow our simulations,” said lead author Luca Fedeli, a plasma physicist, WarpX developer, and currently a post-doc at CEA. “So we had like 12-hour Zoom calls at weird times.”

A broad spectrum of emerging and potential applications

WarpX is a cutting-edge software platform developed under Vay’s leadership within DOE’s Exascale Computing Project (ECP) for the modeling of plasma accelerators. It is now being leveraged by the Collaboration for Advanced Modeling of Particle Accelerators (CAMPA)—also led by Vay, and part of DOE’s Scientific Discovery through Advanced Computing (SciDAC) research program—to speed up the modeling of all types of accelerators. Ongoing support of the capabilities developed in WarpX, and its wider ecosystem beyond the ECP project, will be vital to the research and development of future accelerators. According to Vay, the list of research areas that could benefit from WarpX is growing rapidly.

“It is already being used by our collaborators in the computational physics division to simulate the physics of pulsars,” he said.

There is also a spin-off of WarpX, called the Adaptive mesh Refinement Time-domain ElectrodynaMIcs Solver), or ARTEMIS, he noted, which couples the Maxwell’s equations implementation in WarpX with classical equations to simulate the behavior of quantum materials behavior and could lead to next-generation microelectronics.

ATAP researchers are working with AMReX to build a software stack called Beam, Plasma & Accelerator Simulation Toolkit (BLAST) comprising flexible modules that can be used for different applications, said Huebl. “Some of these modules, for example, are now leveraged in the ImpactX project.” The project is part of the Laboratory Directed Research and Development program and aims to improve the performance of linear accelerators and accelerator rings.

The team plans to investigate how WarpX could be used in simulations involving 20 to 50 consecutive laser plasma accelerators stages. The work will aim to create better electron beams for applications in radiotherapy and the new colliders.

“We also are also looking to extend the method beyond laser plasma accelerators to investigate ion acceleration and solid target interaction, for example, as used in fusion research,” said Rémi Lehe, a researcher at Berkeley Lab and a co-author of the Gordon Bell Prize winning paper.

Mesh refinement, he continued, would be “extremely useful not only in conventional accelerators but also in nuclear fusion technologies,” which is a key area of research for the DOE and “for which WarpX is also well positioned.”

The paper’s other co-authors include France Boillod-Cerneux, deputy at the CEA’s Fundamental Research Department regarding numerical simulations; Thomas Clark, a Ph.D. student at CEA; Kevin Gott, a NERSC staff member working in the User Engagement Group; Conrad Hillairet, with ARM; Stephan Jaure, with Atos; Adrien Leblanc, with ENSTA Paris; Myers, a member of the Center for Computational Sciences and Engineering at Berkeley Lab; Christelle Piechurski, the Chief HPC project officer for GENCI; Mitsuhisa Sato, deputy director of RIKEN Center for Computational Science; Neil Zaïm, a postdoc at CEA; and Weiqun Zhang, a senior computer systems engineer at Berkeley Lab.

Learn More
Tags: