NREL (Kestrel)

This page describes how to get setup at the National Renewable Energy Laboratory (NREL) on Kestrel.

If you encounter an error with NREL, there is most likely documentation on https://nrel.github.io/HPC/Documentation/Systems/Kestrel/.

Setting up NREL account

Visit website https://www.nrel.gov/hpc/user-accounts.html to get started on making a user account. You will need to setup a MFA before proceeding.

Send an email to [e-mail] to get added on the correct projects. Ask Kristin or the NREL coordinators of the group to see which projects you belong to.

Logging on

For the purposes of this example, assume your NREL account username is: johndoe

Kestrel login:

ssh [e-mail]

Once in, the command line will prompt you to type in your account password+OTP.

General NREL HPC login:

ssh [e-mail]

Use this for loggin into the NREL HPC systems. From there, you can connect to any other NREL HPC system like Swift, Eagle, etc.

Conda Environment Setup

I personally would recommend setting up miniconda instead of using their default conda modules. If you would like to use their default conda module, execute module load conda and proceed to the conda install steps.

Miniconda

You can find these install instructions on https://docs.anaconda.com/miniconda/#quick-command-line-install

cd ~/ #ensures you start off in your home dir

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

source ~/.bashrc

You should now see a (base) or (miniconda) on the lefthand side of your command prompt.

Setting up your conda environment

Warning: This is different for every user. Please check with your mentor before proceeding.

The following conda setup is designed for VASP users who will be using atomate2 with fireworks.

conda create -n cms python=3.9 pandas seaborn numpy scipy matplotlib
conda activate cms 

pip install git+https://github.com/materialsproject/atomate2.git #replace with your desired way to downloading atomate2
pip install fireworks
pip install pydantic==2.4.2 #to avoid some pydantic bugs with fireworks...

Please refer to https://materialsproject.gitbook.io/persson-group-handbook/computing/atomate-setup/setup#set-up-an-environment for setting up the fireworks yaml files and the fw_config directory.

In addition, also setup these two additional files:

atomate2.yaml

Kestre CPU

VASP_CMD: srun -n 104 -c 1 vasp_std 
VASP_GAMMA_CMD: srun -n 104 -c 1 vasp_gam

Kestrel GPU

VASP_CMD: srun -n 104 -c 1 --cpu-bind=cores --gpu-bind=single:1 -G 4 vasp_std
VASP_GAMMA_CMD: srun -n 104 -c 1 --cpu-bind=cores --gpu-bind=single:1 -G 4 vasp_std

Kestrel CPU 2 Nodes

VASP_CMD: srun -n 104 -c 2 vasp_std 
VASP_GAMMA_CMD: srun -n 104 -c 2 vasp_gam

jobflow.yaml

Please use trajectory store version if you need to run AIMD/MD type calculations with atomate2

General

JOB_STORE:
  docs_store:
    type: MongoStore
    database: johndoe_general
    host: mongodb03.nersc.gov
    port: 27017
    username: johndoe_general_admin
    password: ***PASSWORD***
    collection_name: kestrel_outputs
  additional_stores:
    data:
      type: GridFSStore
      database: johndoe_general
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: ***PASSWORD***
      collection_name: kestrel_outputs_blobs

With trajectory store

JOB_STORE:
  docs_store:
    type: MongoStore
    database: johndoe_general
    host: mongodb03.nersc.gov
    port: 27017
    username: johndoe_general_admin
    password: ***PASSWORD***
    collection_name: kestrel_outputs
  additional_stores:
    data:
      type: GridFSStore
      database: johndoe_general
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: ***PASSWORD***
      collection_name: kestrel_outputs_blobs
    trajectory:
      type: GridFSStore
      database: johndoe_general_admin
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: s***PASSWORD***
      collection_name: kestrel_trajectory_blobs