Over recent decades, physicists and materials scientists globally have focused on crafting computer codes to simulate essential material properties. This burgeoning field now boasts a wide array of tools, enabling researchers to publish tens of thousands of scientific articles annually.
Artistic rendition of the ability of the AiiDA workflow engine (whose logo is shown on the mug) to seamlessly compute materials properties with multiple quantum-mechanical simulation codes, thanks to the "AiiDA common workflow" interface. The image in the tablet is a simplified version, where the results of 11 different computational approaches and codes are compared (in particular, the figure compares the parameters of the equation of state: equilibrium volume, bulk modulus and its derivative, for all 960 materials and chemical elements considered in the paper)
These codes primarily rely on density-functional theory (DFT), a modeling approach employing various approximations. DFT helps mitigate the otherwise daunting complexity inherent in calculating the behavior of each electron based on the principles of quantum mechanics.
Indeed, the discrepancies between results obtained from various codes stem from the numerical approximations employed and the specific numerical parameters chosen to suit particular material classes or to calculate properties vital for specific applications—such as conductivity for potential battery materials.
Due to the intricacy of these codes, ensuring their absolute freedom from coding errors or overly coarse numerical approximations is immensely challenging. However, it remains crucial for the scientific community to ascertain that results from different codes are comparable, exhibit consistency, and are reproducible.
This verification process ensures reliability and fosters confidence in the outcomes generated across different computational methodologies.
In a recent publication in Nature Review Physics, a substantial consortium of scientists undertook the most extensive validation endeavor to date concerning solid-state density-functional theory (DFT) codes. They furnished their peers with tools and a comprehensive set of guidelines aimed at evaluating and enhancing both current and forthcoming codes.
This endeavor builds upon a prior study featured in Science in 2016, which compared 40 computational approaches. Each approach calculated energies for a test set comprising 71 crystals, each representing an element on the periodic table. This earlier study concluded that mainstream codes demonstrated notably strong agreement with one another.
That work was reassuring, but it did not really explore enough chemical diversity. In this study we considered 96 elements, and for each of them we simulated ten possible crystal structures.
Giovanni Pizzi, Corresponding Author and Leader, Materials Software and Data Group, Paul Scherrer Institute
The study focused on investigating four distinct unary crystals for each of the initial 96 elements of the periodic table. Additionally, they examined six various oxides, compounds that incorporate oxygen atoms alongside the element itself.
Consequently, they compiled a dataset comprising 960 materials and their corresponding properties. These calculations were conducted using two independent, cutting-edge DFT codes known as FLEUR and WIEN2k. Both codes are classified as “all-electron” (AE), indicating their capability to explicitly consider all electrons within the atoms being analyzed.
This dataset stands as an open benchmark for assessing the accuracy of other codes, especially those reliant on pseudopotentials. Unlike the all-electron (AE) codes, pseudopotential-based codes simplify the treatment of non-participating electrons in chemical bonds, allowing for lighter computational processes.
We actually have already started to improve nine such codes in our paper, comparing their results to those in our dataset, measuring the discrepancies, and adjusting their numerical parameters (such as the pseudopotentials) accordingly.
Giovanni Pizzi, Corresponding Author and Leader, Materials Software and Data Group, Paul Scherrer Institute
Within the study, there are outlined recommendations for users of DFT codes. These guidelines aim to ensure the reproducibility of computational studies, provide insight into leveraging the reference dataset for forthcoming verification studies, and offer strategies to broaden its scope. This expansion includes incorporating additional families of codes and diverse material properties into the dataset.
“We hope our dataset will be a reference for the field for years to come,” adds Pizzi, who is one of the nine MARVEL investigators who authored the study, along with Marnik Bercx, Kristjan Eimre, Sebastiaan Huber, Matthias Krack, Nicola Marzari, Aliaksandr Yakutovich, Jusong Yu, Austin Zadoks.
Additionally, the study establishes a platform for forthcoming verification studies using AiiDA, an open-access computational framework developed by the National Centre for Competence in Research (NCCR) MARVEL.
This framework, supported by the project led by Pizzi, along with the European Centre of Excellence MaX, facilitated the study's execution and provided an avenue for future investigations.
AiiDA allows us to write the same instruction in the same way for 11 different codes, for example, the request to compute a specific structure. It can then run the calculation for you and select the right numerical parameters for each.
Giovanni Pizzi, Corresponding Author and Leader, Materials Software and Data Group, Paul Scherrer Institute
Expanding the reference dataset with more structures is a primary goal. Pizzi envisions a future where assessments consider not only the accuracy of different codes but also their computational cost in terms of time and resources. This approach aims to aid scientists in identifying the most cost-efficient parameters for their calculations.
Journal Reference
Bosoni, E., et al. (2023). How to verify the precision of density-functional-theory implementations via reproducible and universal workflows. Nature Reviews Physics. doi.org/10.1038/s42254-023-00655-3.