Conda Environments

Using Conda environments on NERSC systems

What are conda environments?

Conda envs are a great way to manage package/library versions. Frequently we need a specific configuration and package versions for one project's needs can conflic with another project's needs. Conda envs allow us to create separate "environments" where you can be free to install any package version you like without it affecting anything outside of the environment.

Reminder: Put conda on PATH

By default, conda is not on PATH and you may get an error when trying to call `conda ...`. Get it on PATH by:

$ module load python

Note: you can that line to your ~/.bash_profle to prevent you having to do this each time you log in.

Create Conda env

To create a new named conda enviornment, use the following commands. In this example we create an enviornment named my_env with python 3.8

conda create -n my_env python=3.8

Use a Conda env you've created

To enter your environment, activated it using its name:

conda activate my_env

To list your available enviornments:

conda env list

Change default conda settings (optional)

Change environment directory

By default, if we create a new enviroment, it will be stored in the $HOME directory (e.g. /global/homes/m/<username>/.conda/envs). Each of us has a quota of 40G for $HOME, and sometimes conda environments can get quite big, which can cause out-of-quota problem. So, let's change the default environment directory to avoid this.

You should have access to /global/common/software/matgen/ (or /global/common/software/jcesr, depending on the account you have access to). Create a directory under your username (to store all your software), e.g.

$ cd /global/common/software/matgen   # or "cd /global/common/software/jcesr"  
$ mkdir <username>   # change <username> to your NERSC user name 
$ chmod g-w <username>   # remove group write access to avoid others changing it 

Within your directory, create a directory to store conda environemts (assuming we want to store it at .../<username>/conda/envs):

$ cd <username> 
$ mkdir conda && mkdir conda/envs

Then, config conda to prepend to envs_dirs what we've created:

$ conda config --prepend envs_dirs /global/common/software/matgen/<username>/conda/envs

This is all you need to do.

To ensure it's successful, you can view conda settings by

$ conda config --show

You will find something like

envs_dirs:
  - /global/common/software/matgen/<username>/conda/envs
  - /global/homes/m/<username>/.conda/envs
  - /global/common/software/nersc/pm-2021q4/sw/python/3.9-anaconda-2021.11/envs

Alternatively, you can open ~/.condarc to see all the changes you've made. You can even directly edit it to remove the changes or add new ones.

Change package directory

When you install a package, the package will first be downloaded to $HOME, (e.g. /global/homes/m/<username>/.conda/pkgs). You can change the default package storage directory as well:

$ mkdir /global/common/software/matgen/<username>/conda/pkgs
$ conda config --prepend pkgs_dirs /global/common/software/matgen/<username>/conda/pkgs

Agian, you may need to change matgen to the accout you have access to, and, of course, change <username> to your username.