# NREL (Kestrel)

If you encounter an error with NREL, there is most likely documentation on <https://nrel.github.io/HPC/Documentation/Systems/Kestrel/>.

## Setting up NREL account

Visit website <https://www.nrel.gov/hpc/user-accounts.html> to get started on making a user account. You will need to setup a MFA before proceeding.

Send an email to <HPC-Help@nrel.gov> to get added on the correct projects. Ask Kristin or the NREL coordinators of the group to see which projects you belong to.&#x20;

## Logging on

For the purposes of this example, assume your NREL account username is: `johndoe`

#### Kestrel login:

```
ssh johndoe@kestrel.nrel.gov
```

Once in, the command line will prompt you to type in your account password+OTP.&#x20;

#### General NREL HPC login:&#x20;

```
ssh johndoe@hpcsh.nrel.gov
```

Use this for loggin into the NREL HPC systems. From there, you can connect to any other NREL HPC system like Swift, Eagle, etc.&#x20;

## Conda Environment Setup&#x20;

I personally would recommend setting up miniconda instead of using their default conda modules. If you would like to use their default conda module, execute `module load conda` and proceed to the conda install steps.&#x20;

#### Miniconda

You can find these install instructions on <https://docs.anaconda.com/miniconda/#quick-command-line-install>

```
cd ~/ #ensures you start off in your home dir

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

source ~/.bashrc
```

You should now see a `(base)` or `(miniconda)` on the lefthand side of your command prompt.&#x20;

#### Setting up your conda environment

Warning: This is different for every user. Please check with your mentor before proceeding.&#x20;

The following conda setup is designed for VASP users who will be using atomate2 with fireworks.&#x20;

```
conda create -n cms python=3.9 pandas seaborn numpy scipy matplotlib
conda activate cms 

pip install git+https://github.com/materialsproject/atomate2.git #replace with your desired way to downloading atomate2
pip install fireworks
pip install pydantic==2.4.2 #to avoid some pydantic bugs with fireworks...
```

Please refer to <https://materialsproject.gitbook.io/persson-group-handbook/computing/atomate-setup/setup#set-up-an-environment> for setting up the fireworks yaml files and the fw\_config directory.&#x20;

In addition, also setup these two additional files:&#x20;

#### atomate2.yaml

{% tabs %}
{% tab title="Kestre CPU" %}

```
VASP_CMD: srun -n 104 -c 1 vasp_std 
VASP_GAMMA_CMD: srun -n 104 -c 1 vasp_gam
```

{% endtab %}

{% tab title="Kestrel GPU" %}

```
VASP_CMD: srun -n 104 -c 1 --cpu-bind=cores --gpu-bind=single:1 -G 4 vasp_std
VASP_GAMMA_CMD: srun -n 104 -c 1 --cpu-bind=cores --gpu-bind=single:1 -G 4 vasp_std
```

{% endtab %}

{% tab title="Kestrel CPU 2 Nodes" %}

```
VASP_CMD: srun -n 104 -c 2 vasp_std 
VASP_GAMMA_CMD: srun -n 104 -c 2 vasp_gam
```

{% endtab %}
{% endtabs %}

#### jobflow\.yaml

Please use trajectory store version if you need to run AIMD/MD type calculations with atomate2

{% tabs %}
{% tab title="General" %}

```
JOB_STORE:
  docs_store:
    type: MongoStore
    database: johndoe_general
    host: mongodb03.nersc.gov
    port: 27017
    username: johndoe_general_admin
    password: ***PASSWORD***
    collection_name: kestrel_outputs
  additional_stores:
    data:
      type: GridFSStore
      database: johndoe_general
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: ***PASSWORD***
      collection_name: kestrel_outputs_blobs
```

{% endtab %}

{% tab title="With trajectory store" %}

```
JOB_STORE:
  docs_store:
    type: MongoStore
    database: johndoe_general
    host: mongodb03.nersc.gov
    port: 27017
    username: johndoe_general_admin
    password: ***PASSWORD***
    collection_name: kestrel_outputs
  additional_stores:
    data:
      type: GridFSStore
      database: johndoe_general
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: ***PASSWORD***
      collection_name: kestrel_outputs_blobs
    trajectory:
      type: GridFSStore
      database: johndoe_general_admin
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: s***PASSWORD***
      collection_name: kestrel_trajectory_blobs
```

{% endtab %}
{% endtabs %}