# Setup

This page is intended to help you get set-up the first time you are using fireworks and atomate so you can learn how these softwares work. This set-up is not ideal for most research applications. Please refer to [Fireworks: Advanced Usage](https://materialsproject.gitbook.io/persson-group-handbook/computing/atomate-setup/atomate-setup) if you are looking for practical advice and tips on how to run calculations once you understand the basics of fireworks.

**Note:** These instructions are based on the original version of atomate. atomate2 is a rising code that is still in development. Generally these research codes are under continuous improvement, so if something isn't working here please ask! It is likely the documentation needs to be updated.

## Set up an environment:

**Create the conda environment:**

{% tabs %}
{% tab title="Perlmutter (NERSC)" %}

```
module load python
conda create -n cms python=3.10
```

{% endtab %}

{% tab title="Savio (UCB)" %}

```
module load python/3.8
conda create -n cms python=3.9
```

{% endtab %}

{% tab title="Lawrencium (UCB)" %}

```
module load python/3.9.12
conda create -n cms python=3.9
```

{% endtab %}

{% tab title="Swift (NREL)" %}

```
module load python
conda create -n cms python=3.10
```

{% endtab %}

{% tab title="Kestrel (NREL)" %}

```
module load python
conda create -n cms python=3.10
```

{% endtab %}
{% endtabs %}

**Note:** *cms* is an abbreviation for "computational materials science," but feel free to pick your own environment name!

Activate the environment and install the base libraries:

```bash
source activate cms
conda install numpy scipy matplotlib pandas
```

**To install atomate2 on the environment, use one of the following:**

(For Most People) If you are just using atomate and fireworks, with no plans to develop code/ workflows, use the following:

```bash
pip install atomate2 fireworks
pip install ruamel.yaml==0.17.40
```

For developers:

```bash
source activate cms
cd ~/.conda/envs/cms
mkdir code
cd code
git clone https://github.com/materialsproject/pymatgen.git
git clone https://github.com/materialsproject/pymatgen-db.git
git clone https://github.com/materialsproject/fireworks.git
git clone https://github.com/materialsproject/custodian.git
git clone https://github.com/hackingmaterials/atomate.git

cd pymatgen
python setup.py develop
pip install -r requirements.txt
cd ..
cd pymatgen-db
python setup.py develop
pip install -r requirements.txt
cd ..
cd fireworks
python setup.py develop
pip install -r requirements.txt
cd ..
cd custodian
python setup.py develop
pip install -r requirements.txt
cd ..
cd atomate
python setup.py develop
pip install -r requirements.txt
cd ..
```

## Getting Started Fireworks Configuration:

### Create Fireworks configuration files

**Note: You will want to follow the steps in this section on both your local computer and the supercomputer where you will be running your calculations** (e.g. making a fw\_config folder in your home directory that contains the 3 files: FW\_config.yaml, db.json, and my\_launchpad.yaml) so you can easily interface with fireworks from both your local computer and the supercomputer.

Make directory in your home directory called fw\_config:

```bash
cd ~
mkdir fw_config
cd fw_config
```

Now create 3 files: FW\_config.yaml, db.json, and my\_launchpad.yaml with the following contents. Make sure you replace YOUR\_USERNAME with your username; on NERSC you will also need to find your home folder, which is typically under another higher folder with the name of the FIRST LETTER of your username.  **Note**: you can view your [filesystem online](https://my.nersc.gov/filebrowser.php). Also, consider setting up access to your NERSC filesystem locally on your computer: Check out "Mounting NERSC's file system locally" here:[#running-jobs-on-nersc](https://materialsproject.gitbook.io/persson-group-handbook/high-performance-computing#running-jobs-on-nersc "mention") 

#### FW\_config.yaml (fireworks configuration file)

{% tabs %}
{% tab title="Perlmutter (NERSC)" %}

```yaml
CONFIG_FILE_DIR: /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"
```

{% endtab %}

{% tab title="Savio + Lawrencium (UCB)" %}

```yaml
CONFIG_FILE_DIR: /global/home/users/YOUR_USERNAME/fw_config
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"
```

{% endtab %}

{% tab title="Local" %}

```
CONFIG_FILE_DIR: /HOME_DIRECTORY_PATH/fw_config
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"
```

{% endtab %}

{% tab title="Kestrel + Swift (NREL)" %}

```
CONFIG_FILE_DIR: /home/YOUR_USERNAME/fw_config/
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"
```

{% endtab %}
{% endtabs %}

#### db.json (Database file)

```
{
    "host": "mongodb07.NERSC.gov",
    "port": 27017,
    "database": "fw_db_name",
    "collection": "tasks",
    "admin_password": "<admin_password>",
    "admin_user": "<admin_username>",    
    "readonly_password": "<readonly_password>",
    "readonly_user": "<readonly_username>",
    "aliases_config": null
}
```

#### my\_launchpad.yaml (Fireworks LaunchPad file)

```
host: mongodb07.NERSC.gov
port: 27017
name: "fw_db_name"
username: "<admin_username>"
password: "<admin_password>"
logdir: null
Istrm_lvl: DEBUG
user_indices: []
wf_user_indices: []
```

### Create Fireworker and Queue Launchers:

#### Fireworker file (my\_fworker.yaml)

{% tabs %}
{% tab title="Perlmutter (NERSC)" %}

```yaml
name: NERSC_fworker
category: ''
query: '{}'
env:
    db_file: /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'srun -n 64 -c 4 --cpu_bind=cores vasp_std'
    gamma_vasp_cmd: 'srun -n 64 -c 4 --cpu_bind=cores vasp_gam'
    scratch_dir: /global/cscratch1/sd/YOUR_USERNAME
    incar_update:
```

{% endtab %}

{% tab title="Savio" %}

```yaml
name: savio_fworker
category: ''
query: '{}'
env:
    db_file: /global/home/users/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'mpirun --bind-to core vasp_std'
    gamma_vasp_cmd: 'mpirun --bind-to core vasp_gam'
    scratch_dir: /global/scratch/YOUR_USERNAME/
    incar_update:
```

{% endtab %}

{% tab title="Lawrencium" %}

```yaml
name: lrc_fworker
category: ''
query: '{}'
env:
    db_file: /global/home/users/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'mpirun --bind-to core vasp_std'
    gamma_vasp_cmd: 'mpirun --bind-to core vasp_gam'
    scratch_dir: /global/scratch/users/YOUR_USERNAME/
    incar_update:
```

{% endtab %}

{% tab title="NERSC-Perlmutter-GPU" %}

```yaml
name: gpu_fworker
category: ''
query: '{}'
env:
    db_file: /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'srun -n 4 -c 16 --gpu-bind=single:1 vasp_std'
    gamma_vasp_cmd: 'srun -n 4 -c 16 --gpu-bind=single:1 vasp_gam'
    scratch_dir: /pscratch/sd/FIRST_LETTER/YOUR_USERNAME
    incar_update:
```

{% endtab %}

{% tab title="Kestrel + Swift (NREL)" %}

```
name: NREL_fworker
category: ''
query: '{}'
env:
    db_file: /home/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'srun -n 64 -c 1 --cpu_bind=cores vasp_std'
    gamma_vasp_cmd: 'srun -n 64 -c 1 --cpu_bind=cores vasp_gam'
    scratch_dir: /projects/PROJECT_NAME/YOUR_USERNAME
    incar_update:
```

{% endtab %}
{% endtabs %}

#### Queue Adapter (my\_qadapter.yaml):

{% tabs %}
{% tab title="Perlmutter (NERSC)" %}

```yaml
_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
walltime: '24:00:00'
account: matgen
job_name: knl_launcher
signal: SIGINT@60
qos: regular
constraint: 'cpu'
pre_rocket: |
            source activate cms
            module load vasp-tpc/5.4.4-knl
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
```

{% endtab %}

{% tab title="Savio" %}

```yaml
_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/home/users/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
walltime: '24:00:00'
account: co_lsdi
job_name: knl_launcher
queue: savio2_knl
qos: lsdi_knl2_normal
ntasks: 64
pre_rocket: |
            source activate cms
            module load vasp
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
            post_rocket: null
```

{% endtab %}

{% tab title="Lawrencium" %}

<pre class="language-yaml"><code class="lang-yaml"><strong>_fw_name: CommonAdapter
</strong>_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/home/users/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
ntasks: 64
queue: cf1
walltime: '06:00:00'
account: lr_mp
job_name: lrc_cpu_launcher
signal: SIGINT@60
qos: condo_mp_cf1
pre_rocket: |
            source activate cms
            module load vasp/6.prerelease
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
</code></pre>

{% endtab %}

{% tab title="NERSC-Perlmutter-GPU" %}

```yaml
_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/perlmutter/my_fworker.yaml singleshot
nodes: 1
walltime: '06:00:00'
account: matgen_g
job_name: gpu_launcher
signal: SIGINT@60
qos: regular
constraint: 'gpu'
pre_rocket: |
            source activate cms
            module load vasp/6.2.1-gpu
            export SLURM_CPU_BIND="cores"
            unset MPICH_GPU_SUPPORT_ENABLED
```

{% endtab %}

{% tab title="Swift (NREL)" %}

```
_fw_q_type: SLURM
rocket_launch: rlaunch -w /home/YOUR_USERNAME/fw_config/my_fworker.yaml rapidfire --max_loops 5
nodes: 1
ntasks_per_node: 128
walltime: '12:28:00'
account: PROJECT_NAME
job_name: job_name_var
exclude_nodes: 'c5-17,c1-39,c5-27,c1-61,c12-6,c1-37,c10-27,c10-18,c1-36,c9-13,c4-6,c1-58,c6-43,c9-15,c4-48,c5-55,c1-38,c7-8,c1-63,c10-49,c4-22'
signal: SIGINT@60
queue: standard
pre_rocket: |
            export ATOMATE2_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/atomate2/config/atomate2.yaml"
            export JOBFLOW_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/atomate2/config/jobflow.yaml"
            export FW_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/FW_config.yaml"
            # load conda environment below if needed
            source activate cms
            module load vasp/cpu_6.4.2-intel
            module load gcc
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
            export KMP_AFFINITY=balanced 
```

{% endtab %}

{% tab title="Kestrel (CPU)" %}

```
_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /home/USERNAME/fw_config/my_fworker.yaml rapidfire --max_loops 5
nodes: 1
walltime: '24:00:00'
job_name: vasp_launcher
signal: SIGINT@60
queue: standard
account: drxkp
pre_rocket: |
                conda activate at2
                module load vasp/6.4.2
                module load craype-x86-spr  #specifies sapphire rapids architecture
                export OMP_NUM_THREADS=1
                export KMP_AFFINITY=balanced # for codes built with intel compilers

                ulimit -Ss unlimited #specific to resolve for SIGSEGV error
```

{% endtab %}
{% endtabs %}

Available queues, partitions, and *qos* descriptions can be found at the following links:

* [NERSC](https://docs.nersc.gov/jobs/policy/)
* [Savio](https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/)
* [Lawrencium](https://sites.google.com/a/lbl.gov/high-performance-computing-services-group/lbnl-supercluster/lawrencium)

**Note:** specifying singleshot in the queue adapter will limit each reserved job to running only one firework (even if other fireworks are ready and could run with your remaining wall time). This can be changed to rapidfire but this may result in lost runs (fireworks that do not complete because they run out of wall time).

### Configure Bash profile:

Append the following lines to the .bashrc or .bashrc.ext file, which is located in your home directory, e.g. /global/homes/s/sivonxay:

{% tabs %}
{% tab title="NERSC" %}

```bash
module load python
export PMG_VASP_PSP_DIR='/global/cfs/cdirs/matgen/POTCARs'
export FW_CONFIG_FILE='/global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'
```

{% endtab %}

{% tab title="Savio" %}

```bash
module load python/3.8
export PMG_VASP_PSP_DIR=/clusterfs/cloudcuckoo/software/POTCARs
export FW_CONFIG_FILE='/global/home/users/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'
```

{% endtab %}

{% tab title="Lawrenicum" %}

```bash
module load python/3.9.12
export PMG_VASP_PSP_DIR=/clusterfs/mp/software/POTCARs
export FW_CONFIG_FILE='/global/home/users/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'
```

{% endtab %}

{% tab title="Local" %}

```
export FW_CONFIG_FILE='/HOME_DIRECTORY_PATH/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'
```

{% endtab %}

{% tab title="NERSC-Perlmutter-GPU" %}

```bash
module load python
export PMG_VASP_PSP_DIR='/global/cfs/cdirs/matgen/POTCARs'
export FW_CONFIG_FILE='/global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'
```

{% endtab %}
{% endtabs %}

**Note:** the line `alias cdconfig='cd ~/fw_config'` is optional but is recommended so you can more easily locate the directory with your configuration files stored by typing cdconfig into the command line.

### Setup API key and Add POTCAR Directory to pymatgen

Go to materialsproject.org and get an API key. Make sure you are using the environment setup above to run these commands in terminal with your cms python environment activated.

{% tabs %}
{% tab title="NERSC" %}

```
pmg config --add PMG_MAPI_KEY <USER_API_KEY>
pmg config --add PMG_VASP_PSP_DIR /global/cfs/cdirs/matgen/POTCARs
```

{% endtab %}

{% tab title="Savio" %}

```
pmg config --add PMG_MAPI_KEY <USER_API_KEY>
pmg config --add PMG_VASP_PSP_DIR /clusterfs/cloudcuckoo/POTCARs
```

{% endtab %}

{% tab title="Lawrencium" %}

```
pmg config --add PMG_MAPI_KEY <USER_API_KEY>
pmg config --add PMG_VASP_PSP_DIR /clusterfs/mp/software/POTCARs
```

{% endtab %}
{% endtabs %}

Running these commands will create a .pmgrc.yaml file (if it doesn’t exist already) containing these configuration settings in your home directory

## Using Fireworks and Atomate:

The Materials Project Workshop has some great sessions introducing how to use fireworks and atomate. It might be helpful to watch the youtube recordings before trying out launching your own calculations!

[2019 MP Workshop: Atomate Basics](https://www.youtube.com/watch?v=-MalOMJt34U\&list=PLTjFYVNE7LThQriSxXUIr_Wkg9tUcoax5\&index=9)

[2021 MP Workshop: Automating DFT](https://www.youtube.com/watch?v=C4Qagw1Izt0\&list=PLTjFYVNE7LTi6kGvPAF7DDQYj0KDL-vQL\&index=9)

### Initializing Fireworks Database:

First, let's confirm that you have your fireworks configuration set-up correctly to point to the right MongoDB database for your launchpad. Check your default fireworks configuration file with echo.

```
echo $FW_CONFIG_FILE
```

Copy the file path printed out and check the contents of the specified FW\_config.yaml file to find your `FW_CONFIG_DIR` using cat.&#x20;

```
cat /HOME_DIRECTORY_PATH/fw_config/FW_config.yaml
```

You should see something like this printed out. Copy the file path specified by `CONFIG_FILE_DIR`.

> CONFIG\_FILE\_DIR: /HOME\_DIRECTORY\_PATH/fw\_config
>
> QUEUE\_UPDATE\_INTERVAL: 5
>
> ECHO\_TEST: "cms fw\_config activated"

Print out the contents of the `my_launchpad.yaml` file in your `CONFIG_FILE_DIR` using cat.

```
cat YOUR_FW_CONFIG_FILE_PATH/my_launchpad.yaml
```

If you do not see something that resembles your `my_launchpad.yaml` file written earlier, there may be something wrong with your fireworks configuration that needs to be resolved before moving forward.

<mark style="color:red;">**BE EXTREMELY CAREFUL WHEN RUNNING THIS RESET COMMAND.**</mark> It will wipe all existing entries in your fireworks database in the fireworks, workflows, and launches collections. You must activate your python environment where fireworks is installed before running this command in terminal.

```bash
lpad reset
```

### Using Atomate to Create a Workflow

The following code blocks are python code that should be run in jupyter or via python from the terminal. Make sure you have the environment setup above activated. The computer running the code should have access to mongodb07.NERSC.gov; this can be disregarded when running directly on NERSC or when connected to to the LBNL intranet. For computers outside of LBNL, a VPN will need to be used.

Run the following python code in a jupyter notebook or by creating a file named `make_workflow.py` and running the command `python make_workflow.py` in terminal:

```python
from pymatgen.ext.matproj import MPRester
from fireworks import LaunchPad
from atomate.vasp.workflows.presets.core import wf_static
from atomate.vasp import powerups

mpr = MPRester()

structure = mpr.get_structure_by_material_id("mp-145")
wf = wf_static(structure)
wf = powerups.add_common_powerups(wf, {"scratch_dir":">>scratch_dir<<"})

lp = LaunchPad.from_file('YOUR_FW_CONFIG_FILE_PATH/fw_config/my_lp.yaml')
lp.add_wf(wf)
```

This adds a static energy calculation for Hexagonal Lu to your job database.

To check if this workflow has been added to your launch pad, run this command in terminal:

```bash
lpad get_fws -s READY
```

You can also check if this workflow has been added successfully by looking at your MongoDB `fireworks` collection or the [launchpad webgui](#launchpad-web-gui).

### Launching a Workflow

Use qlaunch command to reserve jobs with SLURM’s scheduler. qlaunch has 3 modes: singleshot, rapidfire, and multi. Singleshot is used to launch one job, rapidfire is used to launch multiple jobs in quick succession, and multi creates one job with multiple fireworks runs. You’ll probably want to use rapidfire (where it is important to add the --nlaunches flag to specify how many fireworks to run). Here is example code for launching one job from the command line.

**Note: Make sure you are in the directory you want your calculations to be run from before you run these commands, you may want to create a new folder for this. Do not run these commands from your home directory.**

{% tabs %}
{% tab title="Singleshot" %}

```
conda activate cms
echo $FW_CONFIG_FILE
qlaunch singleshot
```

{% endtab %}

{% tab title="Rapidfire" %}

```
conda activate cms
echo $FW_CONFIG_FILE
qlaunch rapidfire --nlaunches 1
```

{% endtab %}
{% endtabs %}

### Monitoring Your Launchpad (Fireworks Database)

#### Terminal Commands

Status of running fireworks can be determined by using the command:

```
lpad get_fws -s RUNNING
```

If a firework has failed or fizzled, you can rerun it by returning its state to ready to be launched again.

{% tabs %}
{% tab title="Rerun a Specific Firework" %}

```bash
lpad rerun_fws -i <fw_id>
```

{% endtab %}

{% tab title="Rerun All Fizzled Fireworks" %}

```bash
lpad rerun_fws -s FIZZLED
```

{% endtab %}
{% endtabs %}

#### Launchpad Web GUI

Many prefer to view their jobs in the web GUI using the following command on their local computer in terminal with your python environment activated. Note this requires having your `FW_CONFIG_FILE` variable set-up to point correctly to your MongoDB database through the `my_launchpad.yaml` file. Otherwise you must run this command in a folder contain your `my_launchpad.yaml` file.

```bash
lpad webgui
```

If you have not yet made a local copy your `my_launchpad.yaml` file, you can download it.

```bash
scp <username>@dtn01.NERSC.gov:~/fw_config/my_launchpad.yaml .
```

## Viewing Database and Outputs:

Download and install Robo 3T from the following [link](https://robomongo.org/download)

### Setup connection using mongodb credentials

![Connection settings in Robo 3T](https://3149419256-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkhcGa_w8TQp85JHqyt%2Fuploads%2Fp9K5ERJIDQXeBGlUwoPT%2Fmongodb_1.png?alt=media\&token=9cd9da8c-63e0-4860-8ed7-b3f15dc41a3e)

![Authentication settings in Robo 3T](https://3149419256-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MkhcGa_w8TQp85JHqyt%2Fuploads%2F4GcgCfipWmEVVRd2IjV7%2Fmongodb_2.png?alt=media\&token=a899bb3e-a368-439c-a4fb-eed860c31ab1)

For security reasons, if you want to connect to the database, you will either:

* Have to connect via the NERSC network, e.g. enter your details in the "SSH" tab and connect via a data transfer node such as dtn01.nersc.gov.
* Have to connect via the LBL network, e.g. by being on-site or using the LBL VPN. In this case you will have to add `-ext` to your database addressed, for example `mongodb03-ext.nersc.gov`

**Special thanks to the original authors of this page**: Eric Sivonxay, Julian Self, Ann Rutt, and Oxana Andriuc