1 of 3

Fireworks & Atomate

This page describes the setup procedure for using fireworks and atomate on high performance computing systems. These instructions are intended for first time users.

Please see the following subsections on fireworks/atomate setup and usage:

name: NERSC_fworker
category: ''
query

name: savio_fworker
category: ''
query

name: lrc_fworker
category: ''
query

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch

Setup

This page describes how to set up fireworks and atomate.

This page is intended to help you get set-up the first time you are using fireworks and atomate so you can learn how these softwares work. This set-up is not ideal for most research applications. Please refer to if you are looking for practical advice and tips on how to run calculations once you understand the basics of fireworks.

Note: These instructions are based on the original version of atomate. atomate2 is a rising code that is still in development. Generally these research codes are under continuous improvement, so if something isn't working here please ask! It is likely the documentation needs to be updated.

Set up an environment:

Fireworks: Advanced Usage

This page builds upon the Fireworks + Atomate Setup page to provide practical tips and advice for more advanced use cases which may be helpful once your start running a larger number of calculations.

Note: atomate2 is a rising code that is still in development. Generally these research codes are under continuous improvement, so if something isn't working here please ask! It is likely the documentation needs to be updated.

Running VASP Best Practices:

For more information on best practices for running VASP on multiple nodes (i.e. how to set vasp_cmd in my_fworker.yaml based on the number of nodes requested in my_qadapter) see the NERSC vasp training.

Choose the appropriate number of nodes, processes, and cores/process:

Follow this to decide how many nodes to request, the number of parallel processes to use, and the number of cores to use for each parallel process. The rule of thumb to remember is that you should use all of the cores for every node that you request. For example, if there are 256 cores per node, you request N nodes, n processes, and c cores/process, then the following relationship must hold:

Specify in my_qadapter.yaml and specify and in my_fworker.yaml within your configuration folder.

Multiple Fireworks Configurations:

In research you may need the ability to run different types of calculations where you will want to easily switch between various set-ups. This section builds upon the previous content to extend it for setting-up running fireworks with multiple configurations.

For demonstrations purposes, here we will illustrate how to set-up two fireworks configurations for switching between running calculations with 1 node vs. 4 nodes. We will make a few other assumptions based on common best practices:

You have applied tags ("1N" or "4N") to specify which fireworks should be run with the 1N vs. 4N firework configuration
You have created separate python environments (cms_1N and cms_4N) for running each type of calculation

These basic steps can be adapted for making many more different custom fireworks configurations for every type of calculation you need!

Fireworks Config Directory

Based on the section you should have a directory for your fireworks configuration files named fw_config in your home directory.

The db.json file in your fw_config directory set-up in the previous section can be used as is assuming you want all your results to be saved in the same database for your various fireworks configurations.

We will need to create a directory for each firework configuration with the specific files that will outline the settings for that configuration (e.g. FW_config.yaml, my_fworker.yaml, my_launchpad.yaml, my_qadapter.yaml). We will have a directory named 1N for the single node configuration and another directory named 4N for the four node configuration. So the fw_config directory in your home directory should contain the following:

db.json (file)
1N (directory)
4N (directory)

Now within the 1N and 4N directories you will need to create the following firework configuration files with the specific settings you want for each fireworks configuration. Note given the variations between different supercomputers, specifics such as file paths, VASP commands, etc. have been omitted here and the format for NERSC has been used. If needed, see the section for examples of appropriate firework configuration files (FW_config.yaml, my_fworker.yaml, my_launchpad.yaml, my_qadapter.yaml) for each specific supercomputer system.

1N Configuration Files

4N Configuration Files

Switching Between Firework Configurations

Now that you have your firework configuration files set-up, we need the ability to specify which firework configuration we want to use upon launching queue jobs. This is done based on the value of the bash environment variable FW_CONFIG_FILE. You can print out what is the value of this variable by typing the following command...

echo $FW_CONFIG_FILE

Note that in the instructions, you were instructed to modify your .bashrc or .bashrc.ext file located in your home directory to automatically activate a fireworks configuration by default. Here you should modify that file so you can type a command to activate a specific fireworks configuration such as 1N_fw_config or 4N_fw_config by adding the following lines:

Now you should be ready to use either firework configuration to launch jobs into the supercomputer queue! Here are some examples of what you might do...

Continuous Job Submission:

tmux

We recommend using a tmux or screen when submitting jobs to preserve queue submission when the main ssh session is terminated. This allows one to keep the queue saturated with jobs. For example, for tmux:

To exit the tmux session, press the “control” and “b” keys together, followed by “d”.

To re-enter the tmux session:

For more tmux commands, see the .

qlaunch commands

Use qlaunch command to reserve jobs with SLURM’s scheduler. Qlaunch has 3 modes; singleshot, rapidfire, and multi. You will want to use rapidfire. Some useful flags to set are:

-m to specify maximum # of jobs in queue at any given time
--nlaunches to specify how many fireworks to run.

Tips for High-Throughput Computing

Fireworks Left in Running State

When you are running many fireworks in rapidfire mode, jobs may hang and not complete. This is referred to as "lost runs" in fireworks. To check for lost runs, use the command:

To rerun these jobs, add the --rerun flag to the command.

Fizzled Fireworks

You can easily reset all fizzled fireworks back to the ready state to be run again with this command in terminal.

When you are running many fireworks, sometimes a calculation that fizzled the first time will successfully complete if you just rerun it. You can check the error message for why a firework fizzled through the MongoDB launches collection through the following path in the document:

action -> stored_data -> _exception -> _stacktrace

Checking the error can help you decided whether simply rerunning the firework may work.

Job Packing

If you have hundreds or thousands of calculations to run, you may be interested in "job packing" to take advantage of the or improve your queue wait times by reducing the number of jobs you submit to slurm (e.g. 1x 1024 node job will be scheduled faster than 1024x single node jobs). The group has python scripts that will automatically distribute running fireworks when multiple nodes are requested for a job. Consult Jason Munro for access to these and an introduction if you are interested in setting up job packing for running calculations.

Fireworks: Advanced Usage

This page builds upon the Fireworks + Atomate Setup page to provide practical tips and advice for more advanced use cases which may be helpful once your start running a larger number of calculations.

Running VASP Best Practices:

Choose the appropriate number of nodes, processes, and cores/process:

Specify in my_qadapter.yaml and specify and in my_fworker.yaml within your configuration folder.

Multiple Fireworks Configurations:

You have applied tags ("1N" or "4N") to specify which fireworks should be run with the 1N vs. 4N firework configuration
You have created separate python environments (cms_1N and cms_4N) for running each type of calculation

These basic steps can be adapted for making many more different custom fireworks configurations for every type of calculation you need!

Fireworks Config Directory

Based on the section you should have a directory for your fireworks configuration files named fw_config in your home directory.

db.json (file)
1N (directory)
4N (directory)

1N Configuration Files

4N Configuration Files

Switching Between Firework Configurations

echo $FW_CONFIG_FILE

Now you should be ready to use either firework configuration to launch jobs into the supercomputer queue! Here are some examples of what you might do...

Continuous Job Submission:

tmux

To exit the tmux session, press the “control” and “b” keys together, followed by “d”.

To re-enter the tmux session:

For more tmux commands, see the .

qlaunch commands

Use qlaunch command to reserve jobs with SLURM’s scheduler. Qlaunch has 3 modes; singleshot, rapidfire, and multi. You will want to use rapidfire. Some useful flags to set are:

-m to specify maximum # of jobs in queue at any given time
--nlaunches to specify how many fireworks to run.

Tips for High-Throughput Computing

Fireworks Left in Running State

When you are running many fireworks in rapidfire mode, jobs may hang and not complete. This is referred to as "lost runs" in fireworks. To check for lost runs, use the command:

To rerun these jobs, add the --rerun flag to the command.

Fizzled Fireworks

You can easily reset all fizzled fireworks back to the ready state to be run again with this command in terminal.

action -> stored_data -> _exception -> _stacktrace

Checking the error can help you decided whether simply rerunning the firework may work.

Job Packing

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
walltime: '24:00:00'
account: matgen
job_name: knl_launcher
signal: SIGINT@60
qos: regular
constraint: 'cpu'
pre_rocket: |
            source activate cms
            module load vasp-tpc/5.4.4-knl
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/home/users/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
walltime: '24:00:00'
account: co_lsdi
job_name: knl_launcher
queue: savio2_knl
qos: lsdi_knl2_normal
ntasks: 64
pre_rocket: |
            source activate cms
            module load vasp
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
            post_rocket: null

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/home/users/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
ntasks: 64
queue: cf1
walltime: '06:00:00'
account: lr_mp
job_name: lrc_cpu_launcher
signal: SIGINT@60
qos: condo_mp_cf1
pre_rocket: |
            source activate cms
            module load vasp/6.prerelease
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/perlmutter/my_fworker.yaml singleshot
nodes: 1
walltime: '06:00:00'
account: matgen_g
job_name: gpu_launcher
signal: SIGINT@60
qos: regular
constraint: 'gpu'
pre_rocket: |
            source activate cms
            module load vasp/6.2.1-gpu
            export SLURM_CPU_BIND="cores"
            unset MPICH_GPU_SUPPORT_ENABLED

_fw_q_type: SLURM
rocket_launch: rlaunch -w /home/YOUR_USERNAME/fw_config/my_fworker.yaml rapidfire --max_loops 5
nodes: 1
ntasks_per_node: 128
walltime: '12:28:00'
account: PROJECT_NAME
job_name: job_name_var
exclude_nodes: 'c5-17,c1-39,c5-27,c1-61,c12-6,c1-37,c10-27,c10-18,c1-36,c9-13,c4-6,c1-58,c6-43,c9-15,c4-48,c5-55,c1-38,c7-8,c1-63,c10-49,c4-22'
signal: SIGINT@60
queue: standard
pre_rocket: |
            export ATOMATE2_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/atomate2/config/atomate2.yaml"
            export JOBFLOW_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/atomate2/config/jobflow.yaml"
            export FW_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/FW_config.yaml"
            # load conda environment below if needed
            source activate cms
            module load vasp/cpu_6.4.2-intel
            module load gcc
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
            export KMP_AFFINITY=balanced

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /home/USERNAME/fw_config/my_fworker.yaml rapidfire --max_loops 5
nodes: 1
walltime: '24:00:00'
job_name: vasp_launcher
signal: SIGINT@60
queue: standard
account: drxkp
pre_rocket: |
                conda activate at2
                module load vasp/6.4.2
                module load craype-x86-spr  #specifies sapphire rapids architecture
                export OMP_NUM_THREADS=1
                export KMP_AFFINITY=balanced # for codes built with intel compilers

                ulimit -Ss unlimited #specific to resolve for SIGSEGV error

Fireworks & Atomate

Setup

hashtagSet up an environment:

Fireworks: Advanced Usage

hashtagRunning VASP Best Practices:

hashtagChoose the appropriate number of nodes, processes, and cores/process:

hashtagMultiple Fireworks Configurations:

hashtagFireworks Config Directory

hashtag1N Configuration Files

hashtag4N Configuration Files

hashtagSwitching Between Firework Configurations

hashtagContinuous Job Submission:

hashtagtmux

hashtagqlaunch commands

hashtagTips for High-Throughput Computing

hashtagFireworks Left in Running State

hashtagFizzled Fireworks

hashtagJob Packing

Fireworks: Advanced Usage

hashtagRunning VASP Best Practices:

hashtagChoose the appropriate number of nodes, processes, and cores/process:

hashtagMultiple Fireworks Configurations:

hashtagFireworks Config Directory

hashtag1N Configuration Files

hashtag4N Configuration Files

hashtagSwitching Between Firework Configurations

hashtagContinuous Job Submission:

hashtagtmux

hashtagqlaunch commands

hashtagTips for High-Throughput Computing

hashtagFireworks Left in Running State

hashtagFizzled Fireworks

hashtagJob Packing

Setup

hashtagSet up an environment:

hashtagGetting Started Fireworks Configuration:

hashtagCreate Fireworks configuration files

hashtagFW_config.yaml (fireworks configuration file)

hashtagdb.json (Database file)

hashtagmy_launchpad.yaml (Fireworks LaunchPad file)

hashtagCreate Fireworker and Queue Launchers:

hashtagFireworker file (my_fworker.yaml)

hashtagQueue Adapter (my_qadapter.yaml):

hashtagConfigure Bash profile:

hashtagSetup API key and Add POTCAR Directory to pymatgen

hashtagUsing Fireworks and Atomate:

hashtagInitializing Fireworks Database:

hashtagUsing Atomate to Create a Workflow

hashtagLaunching a Workflow

hashtagMonitoring Your Launchpad (Fireworks Database)

hashtagTerminal Commands

hashtagLaunchpad Web GUI

hashtagViewing Database and Outputs:

hashtagSetup connection using mongodb credentials

Fireworks & Atomate

Set up an environment:

Running VASP Best Practices:

Choose the appropriate number of nodes, processes, and cores/process:

Multiple Fireworks Configurations:

Fireworks Config Directory

1N Configuration Files

4N Configuration Files

Switching Between Firework Configurations

Continuous Job Submission:

tmux

qlaunch commands

Tips for High-Throughput Computing

Fireworks Left in Running State

Fizzled Fireworks

Job Packing

Running VASP Best Practices:

Choose the appropriate number of nodes, processes, and cores/process:

Multiple Fireworks Configurations:

Fireworks Config Directory

1N Configuration Files

4N Configuration Files

Switching Between Firework Configurations

Continuous Job Submission:

tmux

qlaunch commands

Tips for High-Throughput Computing

Fireworks Left in Running State

Fizzled Fireworks

Job Packing

Set up an environment:

Getting Started Fireworks Configuration:

Create Fireworks configuration files

FW_config.yaml (fireworks configuration file)

db.json (Database file)

my_launchpad.yaml (Fireworks LaunchPad file)

Create Fireworker and Queue Launchers:

Fireworker file (my_fworker.yaml)

Queue Adapter (my_qadapter.yaml):

Configure Bash profile:

Setup API key and Add POTCAR Directory to pymatgen

Using Fireworks and Atomate:

Initializing Fireworks Database:

Using Atomate to Create a Workflow

Launching a Workflow

Monitoring Your Launchpad (Fireworks Database)

Terminal Commands

Launchpad Web GUI

Viewing Database and Outputs:

Setup connection using mongodb credentials