ESM-Tools is a software product developed and maintained at AWI as part of the ESM project. The aim of the software is to unify model infrastructure, giving a common framework for downloading, compiling, running and organizing coupled or standalone models.
The ESM Tool 5.0 is open Source, written in Python, and highly modular. It has a high degree of Information splitting (see Information Splitting subsection), and a plugin and recipe manager, so you can insert your own Python code at any point of the tool’s workflow.
Features in version 5.0
Experiment directory clean-up: Clean up the experiment directory from run folders, with different levels of customization
Forcing and input file linking: ESM-Tools is now capable of not only copying files into the experiment directory and keeping them organized, but also linking files, so that copying time is reduced to minimal.
Message of the day: every time you run esm_master or esm_rumscripts ESM-Tool checks the installed version and can return messages informing about possible updates and bugfixes
Support for new Models and setups:
Cookbook: a cookbook has been added to the documentation including recipes for the most common actions performed by users
Advantages of using ESM-Tools:
- Getting into a running model in less time
- Eases consistent workflows and experiment reproducibility
- Easy to implement new models / coupled setups
- Identical simulation-folder structure across different models / simulations
- Supports standalone models and coupled setups
- One software to rule them all
- Adequate for educational purposes (easy-to-run ESMs) and research (highly modular and flexible)
Downloads, configures and compiles:
- A wide range of ESM models and coupled setups
- For broad selection of HPC systems
- Supported models, setups and HPCs easily expandable through the writing of easy-to-read YAML files
- The user only needs to prepare a brief YAML runscript
File management and postprocessing:
- Automatically organizes files in folders
Calendar and math operations, switches, list and dictionary expansion, …
Splitting the information across different files and programming languages / file formats allows us to keep ESM-Tools stable through the development process, and contributes to its modularity and versatility. We consider three levels of Information:
ESM-Tools is coded in Python and its functionalities are distributed along several packages including, among others, the esm_master (to download, configure and compile models), esm_runscripts (to run simulations) and the esm_parser (to read the YAML files and interpret the user input). The code itself contains no information about models, coupled setups or HPC systems. Development: ESM-Tool is open source, meaning that everyone can easily access and contribute to the tools. However, the ESM Tools Core Development Team, based at the Alfred Wegener Institute (AWI), takes the lean on the development, maintenance of the code and user support.
Config files (model developer and HPC information)
The information necessary to compile and run models under a given HPC system is all written in easy-to-read files, in YAML format. These files are distributed with the main package (esm_tools), that also take care of the installation of the other packages. There are three types of config files:
· Machine files: information specific to the HPC systems
· Model files: information specific to the models
· Setup files: Information specific to the setup of several models
ESM-Tools stablishes an overwriting hierarchy for the information contained in these files: whatever variable / list / dictionary defined in the model file will be overwritten by the same variable / list / dictionary in the setup file, or merged with priority to the setup file information (Fig. 1) This allows for a stabler model config files (e.g. the setup files contain the changes needed for the model to be part of the coupled setup, that override the default standalone model config file).
· Development: config files are written by model and coupled setup developers who want to distribute their models through ESM-Tool. ESM-Tools Core Development Team offers support for this process (ESM-Tools user manual, GitHub issues, workshops, one-to-one assistance)
Front-end user’s runscript
The runscript is written in YAML format, by the front-end user and it contains only the necessary changes to the defaults to run the desired simulation (paths, scenarios, resolution, …). User information overwrites every other information (Fig. 1)
· Development: each model / coupled setup comes together with some template runscript that the user can easily edit. Support for this process is given by model / coupled setups developers and the Core Development Team (ESM-Tools user manual, GitHub issues and workshops).
Implementing a new model / coupled setup
Why would you want to implement you model / coupled setup in ESM-Tools?
As a model developer, project manager or university lecturer, you’ll find yourself many times overburden under many help requests / user questions. Most of this support requests occur because of the large number of steps involved in the compilation and running of a model or coupled setup. ESM-Tools helps developers to have a consistent and reproducible way of building their model and running it, so that they can spend more time in development.
We presented a poster in AGU 2020 Fall Meeting. If you missed it, don’t worry! Posters will still be available online until the end of January. If you were registered to the meeting you can check the poster here.