This website serves as the group handbook for the Persson Group at the University of California, Berkeley and Lawrence Berkeley National Laboratory.

The purpose of this handbook is to:

1) serve as a living repository for knowledge within the group

2) assist new group members in onboarding

While the Onboarding Guides and the Group Librarian have the ultimate responsibility of overseeing the handbook (see Group Jobs), maintaining and improving the group handbook is the responsibility of all group members. Our newest group members often have the most valuable insight in improving the onboarding process and where there may be critical gaps in the handbook. Please see Contributing for guidelines on making contributions. Note that the handbook can be dynamically searched from within the group Slack by writing /gitbook SEARCH_QUERY.

We would like to especially thank Anubhav Jain, whose wonderful group handbook (Jain Group Handbook) inspired much of the content you see here.

About the Group

Our group studies the physics and chemistry of materials using atomistic computational methods and high-performance computing technology. Most of our research is focused on materials for energy applications, such as battery electrode materials, electrolytes, photocatalysts, thermoelectrics, etc. We are co-located at Lawrence Berkeley National Laboratory (LBNL) and the University of California, Berkeley, just down the hill from the Lab. Our main workspaces are located at LBNL, but we have a number of additional desks available on campus for students to use in between classes, which are located in the Graduate Student Bay of Hearst Memorial Mining Building (HMMB).

Group events

Some of the regular things we do as a group are:

• Happy Hour. We meet up for food & drinks most Thursday evenings.

• Group retreats/outings. Past activities include "beach olympics", Halloween costume parties, paintball, cardboard boat races in Kristin's pool, picnics, trips to SF, and more!

• Group Time. A meeting every other week hosted by members of the group that covers important topics within our Group Agreements.

• Birthday celebrations. We celebrate birthdays in the group at our big group meeting.

• Farewell dinners. We celebrate group members who are graduating or leaving the group.

• Winter holiday party. This is often a joint party with the Ceder Group.

Before arriving onsite, please be sure you completed the Human Resources Orientation checklist in Before you Arrive.

Obtaining/Renewing LBNL Badge

To get onsite access at LBNL, you will need to have your LBNL Badge. If you need to obtain it for the first time or if it is expired, you can email siteaccess@lbl.gov to set up an appointment to get one, or set up an appointment using this link. On the day of your appointment, hop on an LBNL shuttle and show the driver your temporary badge (obtained from an email from siteaccess@lbl.gov). You will be dropped off at B65, the badge office.

Pandemic Policies at LBNL

LBNL has constantly changing and adapting policies in response to COVID-19 based on the current situation. Please refer to https://covid.lbl.gov/home to get up-to-date information on Covid Lab Safety Requirements.

UC Berkeley Campus

The Cal 1 card can gives you after-hours and weekend access to the Hearst Memorial Mining Building. Be sure to email Ariana Castro and/or Catalina Estrada for site access.

LBNL Shuttle System

• The most up to date information on using the shuttle system can be found here. Note there is a helpful app for tracking the shuttles in real time. The shuttles also have bike racks so it is common for people to use the shuttles to come onsite and then bike home.

• The Blue route is best for getting to and leaving from B33. The closest stop to B33 for coming onsite is B26. To leave from B33, the best stop is B48 (the firehouse) but B54 (the cafeteria) is also an option.

• For getting to the Molecular Foundry (B76), the closest stop is B62, 66, 67. This stop is accessible through both the Blue route and Orange route.

Lunchtime!

• If you are onsite at LBL in B33, people in the group often gather to eat lunch around 12 or 12:30pm. There are picnic tables outside B33 for outdoor eating.

• Many people bring food from home (B33 has a kitchen including a fridge, microwave, and toaster oven) but you can also buy food at the LBNL cafeteria in B54.

• The cafeteria is currently closed (2023) but there are food trucks at B91!

LBNL Funded Members

• Use LBNL Ebuy (not Ebay) wherever possible - you need to be on the lab network (onsite via an ethernet cable) or be connected via the VPN

• Use Amazon, etc. to buy various components if not available via EBuy

• The laptop is government property; you are expected to return it to the group when you are done working at LBNL. Note that Mac computers make it very simple to transfer everything over to your next computer.

• You are free to take your laptop home, on trips, etc., unless you are an intern in which case other restrictions may apply from the internship program.

• The lab receives your computer and tags it before sending it over to you.

• You must back up your computer very regularly (at least once per week, ideally continuously). This is simple using the Time Machine app. Just plug your backup drive into your monitor so when you connect to your monitor, you also back up. If there are (for some reason) errors in backing up, fix that issue immediately. There are zero excuses for not doing this.

Making the purchase

1. Provide all the details of your selections in an email and send to Shannon Dowling. If all looks okay, she will give you a project and activity ID.

2. Go to eBuy, and for items available there, add them to your cart and submit the requisition with the project and activity ID. Ask Shannon about which SAS approver to list if you are unsure (the SAS approver can vary by project and activity ID).

3. For items not available on eBuy, contact esdradmin@lbl.gov (and cc Kristin and Shannon Dowling) to obtain a procurement form. Fill it out with item details (Vendor, website, price, etc.) and send it back.

4. If you select the overnight shipping option (ask Kristin about this and the related extra costs) most parts, except the computer, will arrive within a week to 10 days. The computer needs to be tagged by the lab, so with overnight shipping, it should arrive within 2 weeks. Ideally, you will select your computer well before arriving at the lab and won’t need overnight shipping.

UCB Funded Members

• Use BearBuy

• More details coming to this documentation soon.

Conferences

There are a lot of great reasons to go to scientific conferences:

• you keep up to date on developments in the field

• you will get to know the people in the field

• you are broadcasting your work to the research community. Many if not most people learn about new research by hearing about it at a conference. Thus, if you want people to know about your work, you must be willing to tell people about it.

If you are first-year student, you can simply attend a conference (no poster/talk required). After your first year, you are expected to present a poster/talk.

You should identify conferences you’d like to attend several months (ideally >6 months) in advance.

Abstract deadlines often come early. For example, the deadline for MRS Spring 2023 (April 10-14, 2023) was almost a full six months before: October 27, 2022!

Once you have identified a conference you’d like to attend, please take the following actions:

• Tell Kristin about the conference and what project you’d like to present as soon as possible

• Submit a conference travel request form. This form is a very basic and takes a few minutes to fill out. If you do not submit the travel request form several months in advance, you may not receive LBNL approval to attend.

• Work with Kristin to submit an abstract. You should send her the proposed abstract (with all details - title, authors, text, figures, etc.) with at least 3 days advance notice.

Once you have received approval to attend the conference, you can begin taking the normal steps to book travel.

Travel Funded through LBNL

If you haven’t done so already, make sure your travel profile (e.g., your frequent flier programs) are completed for the lab. E-mail esdradmin@lbl.gov if you don’t have a travel profile yet. You will need to set this up to get access to Concur.

Flights, rental cars, and hotels can be booked easily through Concur (see below).

Note: your username for Concur is YOUR_EMPLOYEE_ID@lbl.

You should not have to pay upfront for anything booked Concur. Flights will be automatically ticketed (and paid for) by the lab. For rental car and hotels, you will need to pay out of pocket and be reimbursed later.

If you are planning to combine vacation and travel, remember the lab’s policy of taking only one vacation day per two work days. Note that days spent traveling to and from the conference count as work days.

In terms of travel receipts and reimbursement:

You do not need to save receipts for meals. You will receive a per diem instead. You also do not need receipts for taxi rides under $75, although you may want to submits them anyway when you have them.

If you are traveling with outside funding (e.g., the conference organizers are going to reimburse you), save all receipts and tickets as they may be needed for reimbursement.

The proper way to request reimbursements for trips within the US is through the esdradmin site’s “Travel:Domestic” tab. If you have trouble, you can email esdradmin@lbl.gov.

MSD point of contact: Jasmin Harris (jaharris@lbl.gov)

As for international trips including Canada, you should get in touch with the ESDR admin person that you work with. That person will provide you a corresponding form and help you through the (more complex) process of international-travel reimbursement.

For those funded by MSD, select Yuan Ting Lin Hou (Tracy Hou) as your travel approver in TREX. Guidelines for using TREX and creating an expense report:

https://ocfo.lbl.gov/trex-information/trex-guidance/

https://drive.google.com/file/d/1jwsbc3fQZsOrADmL4oUYL5Wa-L4vHuSC/view.

Travel Funded through UC Berkeley

ERSO handles administrative matters for the College of Engineering at UC Berkeley. Individuals can pay out of pocket for travel expenses and then submit a reimbursement request through the ERSO intranet. Unlike LBL, approval for attending a conference is not required with UC Berkeley.

Travel Policy Key Points: https://travel.berkeley.edu/understand-policy/key-points-travel-policy

ERSO Reimbursement: https://www.erso.berkeley.edu/web/before-you-submit#travel-1

ERSO Travel Website: https://www.erso.berkeley.edu/web/travel

Welcome to Berkeley and the Persson Group! Here is a checklist to help you get started once you have recieved your access to LBL either as an employee or affiliate.

Useful Quick Links

Persson Group Website: People

Google Drive

lbl.gov A-Z Index

UC Berkeley ERSO Intranet

UC Berkeley UCPath Payroll, benefits, and HR system

First Day Checklist

Very Basics & High Priority Administrative Tasks

• The Onboarding Team will communicate with you by email to coordinate having a group member meet you the first time you arrive in Building 33. They will help you find your desk and provide a short walk through to become familiar with the building and introduce you to other group members. Once you make your way to Building 33, ask around to find the group member you are looking for if you are having trouble.

• Follow the instructions received during your orientation or by email to activate your Berkeley Lab Identity Account (aka LDAP). This log-in will be used for most LBL online resources. Connect to the internet using the lbnl-visitor WiFi network (it is open access).

• Try out your newly created LDAP log-in. Connect to the lbnl-employee WiFi network, open your @lbl.gov email, google calendar, etc.

• Send out an email introducing yourself to perssongroup@lbl.gov (this is the email list for everyone in the Persson group) from your @lbl.gov email address. This is a tradition to help welcome new group members since it can take some time to meet everyone in person. This also serves as a notice for the group to help initiate various onboarding tasks. This works for graduates (Ph.D. and Master's students), postdocs, undergraduates, short-term visiting scholars, and summer interns. Here is an example introduction email (note permissions to view these documents are tied to your @lbl.gov google account).

• Request access to building 33 for your LBL badge by using this self-serving website. You should select "33 Perimeter" and "33 1st floor Office / lab area." You will also need access to "LBNL Business Need Access" and "LBNL Gates/50 Complex/Library," but these should be assigned to you by default and no specific request should be needed.

  By default, you will NOT have off-hours site access to Building 33 (i.e. from 5:30pm to 7am on weekdays, on weekends, holidays) unless a request is submitted.

• Prioritize completing the GERT (General Employee Radiological Training) before the end of your first day.

• Get your project/funding specific information for making purchases:

  • Purchasing with LBL Funds: Ask Sonia Dominguez for the project id, activity id, and SAS Approver associated with your funding to use Ebuy.

  • Purchasing with UC Berkeley Funds: To make purchases using UC Berkeley funds, you will need the chart string associated with that funding source which can be obtained from UC Berkeley Research Administration Staff for the College of Engineering. Email Miriam Kader including that your PI is Kristin Persson and ask for the chart string associated with your funding source.

• Begin setting up your desk (ideally before completing the ergonomic self-assessment). Order a computer as soon as possible if you have not done so already. See Getting a Computer and Workspace Set-up for guidelines on selecting a computer, needed accessories, and other equipment.

  • When you are ready, check-out Purchasing for more information on placing your order.

  • Ask Bryant Li (or the Onboarding Team) about checking out the the group's inventory of unclaimed items.

  • Any personal orders under $300 can be made without approval. Orders over $300 that aren't for a group role (like buying ink for the printers) should be approved by Sonia and Shannon.

First Week Checklist

Also start working through the other checklists and information below when possible

• Send an email to Theo Jaffrelot Inizan (theo.jaf@lbl.gov) requesting to be added to the Persson Group email lists and team google drive (if you do not have access already). Use the subject line "Onboarding: [YOUR NAME]".

• Complete all LBL/UC Berkeley training courses including the ergonomic self-assessment. Note the GERT must be completed on your first day. Also note that UC Berkeley required training courses will not be assigned until after your UCPath profile is set up, which can take 1-2 weeks after your start date.

• Orient yourself with LBL's calendar system (Google Calendar). Update your LBL calendar with your schedule so other group members can view your availability.

• Check to ensure your calendar is populated with needed meetings (group meetings, subgroup meetings, etc.) Contact a meeting organizer to invite you to needed calendar events.

• Join the Berkeley Theory Slack workspace using your @lbl.gov email. This Slack workspace provides a way for multiple groups and researchers at LBL to communicate about computational materials research, The Materials Project, and other collaborative efforts. Feel free to introduce yourself in the #introductions channel.

• Join the Persson Group Huddle Slack workspace using your @lbl.gov email. This workspace is just for the Persson group.

• Send an email to Vir Karan (virkaran@lbl.gov) with a headshot and tiny bio for the group website (bit of humor in the tiny bio is very welcome).

• Explore the group’s Team Google Drive. Note some of these documents populate the group handbook where access is limited to group members.

• Add your name and information to the Persson Group roster

• Print out an updated name tag for your desk. If needed, work with Shannon to update the name tag for your office.

Research Introduction Checklist

• Meet with Kristin to plan an appropriate introduction for your specific research, project management, subgroup, etc.

• Ask to read the proposal that funds your work. This will help explain the impact of your project, the long-term plans and goals, and how your project fits in with other efforts.

• Get familiar with your subgroup and any other individuals you will be working closely with for your research.

• Generally take the time to learn about other group member's research focuses and areas of expertise to get a sense of who may be your best resource for different questions and who you may be supporting if you are a more senior group member.

Computer & IT Checklist

• Install the lab VPN for connecting to the lab network from home. For example, this lets you download research articles from home. See https://software.lbl.gov for instructions on installation.

• Contact Howard Li about getting user accounts set-up for any computing resources you may be using (NERSC, Savio, VASP, etc.).

• Get connected to the printer. The printer is located in 143C and instructions for how to connect can be found on the internal group site. Julian Self is the current printer czar and can advise you if you are having any difficulties.

• Begin setting up your research computer. See setting up you computer and our internal group site for some installation tutorials and recommended softwares.

Payroll Basics

LBL Employees

LBL employees (staff, postdocs, and graduate students) must enter their time on a regular basis using the LETS payroll system (total hours for a single month are reported). You will need the project ID and activity ID associated with the funding source for your income/stipend. Sonia Dominguez sends time entering instructions before each payroll deadline and can address any related questions.

LBL Payroll Resources

UC Berkeley Employees / LBL Affiliates

UC Berkeley employees (staff, postdocs, and graduate students) must enter their time monthly using the CalTime payroll system.

Note: Sometimes your initial leave balances (24 PTO days and 12 Sick Days) do not populate correctly in CalTime until after the 2nd paycheck processes. If this is an issue (i.e., if you need to use leave during your first pay period), contact ersopayroll@erso.berkeley.edu.

CalTime

Information on UCPath Payroll and Benefits System

Graduate Students: UC Berkeley's ERSO (Engineering Resrouce Support Organization) handles processing GSR appointment requests. These appointments are used to issue stipends (no need to report hours in a payroll system) so keep an eye out for email reminders each Fall, Spring, and Summer.

Purchasing a new computer is budgeted for when hiring long-term appointments (graduate student, postdoc, staff). Short-term appointments (undergraduate students, internships, etc.) will not involve a computer purchase unless otherwise stated as a computer can be borrowed from the group’s inventory instead.

It typically takes 2-3 weeks (and in Fall 2024, it took 20 days ) for a computer to arrive so it is recommended to prioritize placing an order either before your arrival or as soon as possible after you start.

If you are planning to purchase a computer through LBNL, please find the estimated timeline and some general guidelines here. You do not need to use this computer request form. See below on purchasing the laptop.

Why Purchase a Macbook Pro?

Note: The points in this section come mainly from the Hacking Materials Group handbook.

Buying a high end Macbook Pro is highly advised although not mandatory. In our experience, Macs offer the best systems for our work and having the same operating system within our group makes it easier to troubleshoot and share code. Using computers with similar hardware can also make it easier to share and pass down computer accessories and equipment. While we want to be conscious of cost and avoid wastefulness, the value of using well performing equipment that better supports our productivity, collaborative group culture, and reduces wasted time adds value which greatly outweighs the initial financial cost. If you would like to get anything other than a Macbook Pro, please consult more senior group members first.

You can use the Apple website to browse details and see what is available in through the supplier, Anacapa, in Ebuy and Bearbuy. Many group members use a 14” or 16" Macbook Pro with an M2 Pro or M3 Pro chip. The 14" is as powerful as the 16" and has improved mobility. However, the battery life of the 16'' laptop is slightly longer.

Making the Purchase

If you are ordering a computer before your arrival, you will need to coordinate with Shannon Dowling (srdowling@lbl.gov) to handle the purchase. Please send an email with the details of your selections. You may also need to ask Kristin for a project funding source.

Otherwise, if you are ordering a computer after you have started, you can make arrangements yourself. See purchasing for more information.

Overview

Our group’s main computing resources are:

• (the LBNL supercomputing center, one of the biggest in the world)

• / Berkeley Research Computing (BRC)

• / Berkeley Lab Research Computing (LRC)

• (the NREL supercomputing center)

• Argonne Leadership Computing Facility (sometimes)

• Oak Ridge Leadership Computing Facility (sometimes)

At any time, if you feel you are computing-limited, please contact Kristin so she can work with you on finding solutions.

Setup Guides (see subpages)

Back-Up Data Frequently

Mongo DB

You should back-up your Mongo DB data frequently. The Mongo DB NERSC offers is not backed-up automatically. It's important to run regular backups during the course of your research. For Mongo DB you can:

• use the “mongodump” command line tool, tutorials available online — it’s a one line command

module load mongodb; mongodump --host=<host e.g. mongo01.nersc.gov> --port=27017 --username=<username> --password="<password>" --db="<db_name>" --authenticationDatabase="<db_name (same as --db flag)>" --out="<path to directory>"

NERSC Cori, BRC Savio, LRC Lawrencium

You should back-up your "scratch" directory data frequently.

Each cluster has different methods of deleting old data on your scratch directory:

You can use one of the following popular tools for backing up raw data in our group:

Both of these tools can help backup your raw calculation files to your Google Drive, or Box Drive, or even external hard drives.

Additional resources

Other Persson group members and the NERSC website are both excellent resources for getting additional help. If that fails, you can reach out to the NERSC Operations staff:

• 1-800-666-3772 (or 1-510-486-8600)

• Computer Operations = menu option 1 (24/7)

• Account Support = menu option 2, accounts@nersc.gov

• HPC Consulting = menu option 3, or consult@nersc.gov

Special thanks to the original authors of this page: Kara Fong, Eric Sivonxay, and John Dagdelen

A member of the Persson Group Onboarding Team will reach out before your expected start date to share this group handbook and coordinate other onboarding efforts to welcome you to the Persson Group. Although many things can only be taken care of after arriving, here are a few simple things you can do in advance.

If you have not been contacted by the Persson Group Onboarding Team at least one week before your arrival, please reach out to the relevant Onboarding Guide (see ). We try our best to reach out proactively, but there may have been a miscommunication or oversight on our end if we haven't reached out to you before your arrival.

Work Scope and Hiring Details

Several administrative and research details in onboarding are unique to each group member. Some of the following information may have been communicated during your hiring process, but please reach out to Shannon Dowling (srdowling@lbl.gov) to set up a meeting with Kristin if you are unsure. You can also contact members of the Onboarding Team.

• What project(s) will I be working on?

• What funding source will support my work?

• Will I be hired as a UC Berkeley employee (and LBL affiliate) or LBL employee?

• Which subgroup(s) will I be joining? Who will I be working most closely with?

You might not have answers to any or all of these questions when you arrive! That's OK! In the meantime, you can also browse the '' tab in group resources for helpful tutorials and additional information about software / codes frequently used within our group.

Many administrative matters (HR orientation, payroll, purchasing, etc.) will vary depending on if you are a UC Berkeley or LBL employee. This information will help you navigate joining the Persson Group such as understanding which administrative systems you will use and who can help you become orientated with your research/work.

If you are an undergraduate research assistant and are getting course credits for your research work, please check with Kristin and your department about the course requirements (workload, final deliverable, etc.) at the beginning of the semester.

Human Resources Orientation

As mentioned above, the HR processes and orientation will vary depending on whether your financial support will come from UC Berkeley or LBL-controlled funding sources. Please refer to the relevant information below depending on your situation.

LBL Employees

• There are a few steps that must be completed before your LBL new hire orientation can be scheduled (completing a formal application, background check, etc.). Contact Sonia Dominguez (scdominguez@lbl.gov) with any questions or status update requests for matters on the Persson Group end during this time.

• Once everything has been successfully processed, you should receive an email from LBL HR regarding an in person new hire appointment with a general national lab orientation and to complete new hire paperwork. This appointment is held at the national lab and further instructions will be sent regarding location, accessing the lab, and getting your badge afterwards.

• Quick links:

  • - try to complete the General Employee Radiological Training (GERT) on your first day

  • - timekeeping software (still needed if you're paid weekly / monthly!)

  • - register for and view healthcare benefits

UC Berkeley Employees

• Once your hiring paperwork with UC Berkeley has been processed, you should receive an email regarding an in person new hire appointment with UC Berkeley HR to complete new hire paperwork. After this appointment, it may take 1-2 weeks for your information to be
• Proceed to follow the instructions for LBL Affiliates Due to Kristin's joint appointment, all graduate students and postdocs hired through UC Berkeley also have LBL Affiliate status to access the group's offices and resources at the national lab.

LBL Affiliates (i.e graduate students on fellowship)

• On the Persson Group end, a LBL Affiliate appointment must be processed by our group's assistant, Shannon Dowling, before you can obtain your LBL badge.

• Once your appointment has been processed, you will receive email instructions from LBL for getting your LBL badge and setting up your LDAP account (used for authentification for national lab resources). Unlike LBL employees, no in person orientation with LBL is required. Note that your LBL Affiliate appointment does not depend on having a Calnet ID or Cal 1 Card; this can be processed immediately after you complete your UC Berkeley hiring paperwork.

• If your start date is coming up soon, but you have not received any email instructions from LBL, contact Shannon Dowling asking for an update on the status of your affiliate appointment.

Order a Computer

Short-term appointments (undergraduate students, internships, etc.) will not involve a computer purchase unless otherwise stated. Instead, an excellent computer can be borrowed from the group’s inventory.

Onboarding Points of Contact

Postdoc & Staff Onboarding: Theo Jaffrelot Inizan, 33-143B (theo.jaf@lbl.gov)

Graduate Student Onboarding: Bryant Li, 33-143F (bryantli@lbl.gov)

LBL Administrative Assistance: Shannon Dowling, 33-122B (srdowling@lbl.gov)

LBL Funding & Payroll Assistance: Sonia Dominguez, 33-122B (scdominguez@lbl.gov)

UC Berkeley Administrative Assistance: David Anderssen, Krystle Bartholomew, or Vincent Ianniello, ERSO HR Operations, 199ME Cory Hall (ersohrops@erso.berkeley.edu)

UC Berkeley Funding & Payroll Assistance: Miriam Kader, 353 Cory Hall (mkader@berkeley.edu)

Desk Assignments: Bryant Li, 33-143F (bryantli@lbl.gov)

Onboarding buddy: the onboarding guide will assign an onboarding buddy to the incoming postdoc/graduate student. The onboarding buddy can help the onboarding guide to answer questions and help our new members get familiar with the workplace.

Writing your paper

At some point in every project, you will have gotten far enough along and gathered enough results to be able to start the process of writing a paper. Assuming Kristin has given you the go-ahead, it is a good idea to start your writing process with an outline or a "skeleton" draft of what you anticipate the paper will look like. For a paper skeleton, it's helpful to have most of these things covered:

1. An idea for a title

2. A list of expected coauthors

3. Sketches or first attempts at most of your figures

4. Topic sentences for each paragraph in the paper

5. Background ideas and key papers that you plan to reference

6. Information about your methods (e.g., VASP parameters)

7. Key discussion points

8. Some conclusions and takeaways

9. An idea of what will go into the Supplementary Information (SI)

10. Ideas for which journal(s) you may want to submit to

Once you have a paper skeleton, you should contact Kristin for preliminary feedback before you undertake the process of writing the full draft. This step could save you a lot of time later down the road. It is important that you both agree on what the narrative of the paper will be before you start writing.

You may choose to write your draft in your preferred software, but we recommend using Microsoft Word or LaTeX to make it easier to collaboratively edit and format later for journal submission. Note that as a UC Berkeley student or LBL employee you can get free access to Overleaf Pro, an online LaTeX editor that allows for collaborative live editing and automatically tracks your changes.

Getting feedback on your draft

When your paper is nearing completion, and assuming you were already in agreement with Kristin about the paper skeleton, make sure to provide all coauthors with a significant amount of time to read through your paper and make edits. Ideally, you will provide at least a few weeks to do so. Depending on your coauthors' schedules, it may be possible to accelerate this step (e.g., 1 week), but you should not count on this. Always make sure to provide ample time for feedback -- your coauthors will thank you later! :)

Which journal(s) to submit to?

Before submitting to any journals, you should first speak with Kristin (and your coauthors) about their recommendations for where to submit your work. Journal selection can often be a matter of personal preference, but certain journals are often associated with different types of projects and different levels of impact. As you go throughout your career, you will gain intuition on the world of scientific publishing and which journal(s) make the most sense for your work. Kristin, as well as older members in the group, can help guide you in getting started.

A major point of consideration in selecting journals is whether or not that journal is open-access. As outlined in our Group Agreements (#10), we believe that science should be open and accessible to all. Journals that are open-access do not have paywalls and will allow anyone to access the published version of the article. Some journals which are normally behind paywalls may allow you to make your article open-access for an extra fee. You should discuss with Kristin if it would be feasible (and sensible) to pay extra for publishing in an open-access format. Note that it is still possible to share your work openly; please see the following section on preprints.

Here is a selection of scientific journals where some of our group's work has been published, in no particular order:

• Chemistry of Materials

• Nature Communications

• Science Advances

• Computational Materials Science

• Inorganic Chemistry

• ACS Applied Materials & Interfaces

• ACS Energy Letters

• Journal of the American Chemical Society

• Chemical Reviews

• Chemical Science

• Journal of the Electrochemical Society

• npj Computational Materials

• Journal of Materials Chemistry A

• Phys. Rev. Materials

• RSC Advances

• Materials Today

• Journal of Physical Chemistry C

• Scientific Reports

• Scientific Data

• Nature Reviews Materials

• Advanced Materials

• Nature

Should you submit a preprint?

A preprint is a copy of your paper before it enters the peer-review process. A preprint may follow the style/format of a particular journal, but it will not contain any of the special formatting that occurs upon publishing in a journal (i.e., the publisher's version). As author of a paper, you own the copyright to your own preprint, but once the paper has entered the peer-review process at a journal, your work is subject to the copyright policy of that journal. Make sure to check out the specific copyright policies of the journal you are interested in publishing in!

Sharing a preprint to websites like ArXiv or ChemRxiv is a great way to quickly distribute your work to a wide audience, and helps contribute to our group's mission to make science accessible and open to all (Group Agreements #10).

Cover Art

For special (and potentially high-impact) papers, you may want to consider hiring a professional artist to create cover art illustration. Kristin recommends the following studio:

Ella Maru Studio

If you are interested, please contact Kristin to discuss art ideas/options for your paper.

After your article is published

Congratulations on successfully getting through this lengthy process! Now that your article is published, make sure to:

1. Alert the group site Webmaster(s) about your recently published journal article (and provide them with a copy)

2. Contact the group's Social Media Chair (currently John) so that we can share your work on Twitter!

3. Feel free to share your article on your own social media, including retweeting the group and publisher

4. Consider contacting the MSE department (or your specific department) about the publication if you think it is worth sharing on one of the departmental news websites

5. Celebrate your success and pass on what you've learned to other group members as they go through this process themselves :)

What are conda environments?

Conda envs are a great way to manage package/library versions. Frequently we need a specific configuration and package versions for one project's needs can conflic with another project's needs. Conda envs allow us to create separate "environments" where you can be free to install any package version you like without it affecting anything outside of the environment.

Reminder: Put conda on PATH

By default, conda is not on PATH and you may get an error when trying to call `conda ...`. Get it on PATH by:

$ module load python

Note: you can that line to your ~/.bash_profle to prevent you having to do this each time you log in.

Create Conda env

To create a new named conda enviornment, use the following commands. In this example we create an enviornment named my_env with python 3.8

conda create -n my_env python=3.8

Use a Conda env you've created

To enter your environment, activated it using its name:

conda activate my_env

To list your available enviornments:

conda env list

Change default conda settings (optional)

Change environment directory

By default, if we create a new enviroment, it will be stored in the $HOME directory (e.g. /global/homes/m/<username>/.conda/envs). Each of us has a quota of 40G for $HOME, and sometimes conda environments can get quite big, which can cause out-of-quota problem. So, let's change the default environment directory to avoid this.

You should have access to /global/common/software/matgen/ (or /global/common/software/jcesr, depending on the account you have access to). Create a directory under your username (to store all your software), e.g.

$ cd /global/common/software/matgen   # or "cd /global/common/software/jcesr"  
$ mkdir <username>   # change <username> to your NERSC user name 
$ chmod g-w <username>   # remove group write access to avoid others changing it 

Within your directory, create a directory to store conda environemts (assuming we want to store it at .../<username>/conda/envs):

$ cd <username> 
$ mkdir conda && mkdir conda/envs

Then, config conda to prepend to envs_dirs what we've created:

$ conda config --prepend envs_dirs /global/common/software/matgen/<username>/conda/envs

This is all you need to do.

To ensure it's successful, you can view conda settings by

$ conda config --show

You will find something like

envs_dirs:
  - /global/common/software/matgen/<username>/conda/envs
  - /global/homes/m/<username>/.conda/envs
  - /global/common/software/nersc/pm-2021q4/sw/python/3.9-anaconda-2021.11/envs

Alternatively, you can open ~/.condarc to see all the changes you've made. You can even directly edit it to remove the changes or add new ones.

Change package directory

When you install a package, the package will first be downloaded to $HOME, (e.g. /global/homes/m/<username>/.conda/pkgs). You can change the default package storage directory as well:

$ mkdir /global/common/software/matgen/<username>/conda/pkgs
$ conda config --prepend pkgs_dirs /global/common/software/matgen/<username>/conda/pkgs

Agian, you may need to change matgen to the accout you have access to, and, of course, change <username> to your username.

An optimal workspace set-up is essential for making sure you can consistently contribute your best efforts. Remember, just as experimental groups invest their resources in chemicals, new lab instruments, equipment time, etc. to do their research, as computationalists, we invest in office items, software, computers, and other related tools that will make our group members more effective. So if you will use it, buy it. Don't worry so much about if you "need" vs. "want" it.

Personalizing Your Workspace

Please feel free to decorate your workspace with photos, posters, or other personal touches.

Basic Ergonomic Workstation Accessories

An effective workspace should help maximize your productivity. You will spend a great deal of time working in this space so it is worth investing in any equipment needed for you to work more efficiently and comfortably. Order any peripherals and ergonomic devices (e.g. wrist pads) such as:

• An external monitor: One big display screen is usually better from an ergonomic perspective than dual monitors. Note that Mac OS include features such as "Split View" (split your display between showing two different windows side by side) and “Spaces” (quickly shift between desktops displaying different sets of windows and open apps).

• A keyboard: The Apple Wireless Keyboard is a popular option. Some prefer mechanical keyboards. If you prefer a larger or ergonomic keyboard, you can certainly get that.

• A mouse/trackpad: The Apple Magic Trackpad can be a popular option for those who value consistency between their laptop and desk workstation. And it also offers additional functionality and customization built in for Mac OS (e.g. gestures). After a while you get used to doing everything on your trackpad even if you were previously very productive/accurate with a mouse on Windows. However, many group members get by just fine with a mouse (especially the Apple Magic Mouse, which has some gesture support.)

Additional Recommended Computer Accessories

• An external hard drive (compatible with Time Machine for backing-up your computer). 4TB is a good size.

  • Some recommend a USB-C G-Drive that also doubles as a charger.

• An extra computer charger

• An HDMI adapter dongle

• USB to USB-C adaptor (if you have personal electronics that rely on USB connections)

• Computer docking station (optional)

• An ethernet cable adapter dongle

• A presentation tool or clicker, e.g., Logitech R800 (optional)

Remember to be mindful of selecting cables and devices that have compatible ports/connectors and purchasing appropriate adaptors when necessary. There are also many spare adapters/monitors/keyboards etc. in the offices, so please see if you can reuse them before purchasing one, in order to avoid waste.

There is a separate handbook for members of the group who are working on the Materials Project (MP), which includes info about production, builders, the API, etc.

Please find this handbook here: https://app.gitbook.com/@materialsproject/s/handbook/

The MP Handbook has restricted access but it is accessible to anybody working on MP, including collaborators outside the Persson Group. Please contact an MP staff member for access.

SLURM is rejecting my jobs because my home directory is full

e.g. "home directory over quota" ,

We are limited to 40 GB of files in our home directories. This error indicates you have to many files.

Steps to diagnose:

1. Run showquota or myquota to see your file system space usage. If 'home' is 40GB or greater that's your issue.

2. Run du -sh * in your home directory and look for any large directories.

3. Run du -sh .[^.]* in your home directory and look for any large dot directories. Common issues are large .cache directory and large .conda directory

4. Alternatively, you can use the Ncurses Disk Utility tool via shifter to view your home directory usage with an ncurses GUI:

What to do if Conda env dirs are too large?

• Quick solution: conda clean --all

What to do if .cache is too large?

• Quick solution: Delete it! rm -rf .cache

To get started with calculations at NERSC:

1. Ask Kristin about whether you will be running at NERSC and, if so, under what account / repository to charge.

2. Request a NERSC account through the NERSC homepage (Google “NERSC account request”).

3. A NERSC Liason or PI Proxy will validate your account and assign you an initial allocation of computing hours.

4. At this point, you should be able to log in, check CPU-hour balances, etc. through “NERSC NIM” and “My NERSC” portals

5. In order to log in and run jobs on the various machines at NERSC, review the NERSC documentation.

6. In order to load and submit scripts for various codes (VASP, ABINIT, Quantum Espresso), NERSC has lots of information to help. Try Google, e.g. [“NERSC VASP”]().

   ... * Note that for commercial codes such as VASP, there is an online form that allows you to enter your VASP license, which NERSC will confirm and then allow you access to. Log in to , select "Open Request", and fill out the "VASP License Confirmation Request" form.

7. Please make a folder inside your project directory and submit all your jobs there, as your home folder has only about 40GB of space. For example, for matgen project, your work folder path should be something like the following:

   /global/cfs/projectdirs/matgen/YOUR_NERSC_USERNAME

8. You can also request a mongo database for your project to be hosted on NERSC. Google [“MongoDB on NERSC”]() for instructions. Patrick Huck can also help you get set up and provide you with a preconfigured database suited for running Materials Project style workflows.

9. (Optional)

Running Jobs on NERSC

This tutorial provides a brief overview of setting yourself up to run jobs on NERSC. If any information is unclear or missing, feel free to edit this document or contact Kara Fong.

Setting up a NERSC account:

Contact the group’s NERSC Liaison (currently Rohith Srinivaas Mohanakrishnan and Howard Li, see [Group Jobs list]()). They will help you create an account and allocate you computational resources. You will then receive an email with instructions to fill out the Appropriate Use Policy form, set up your password, etc.

Logging on (Setup):

1. Download the script from NERSC to your home folder

2. At the terminal type ./sshproxy.sh -u <nersc_username>

3. Enter your password and OTP

You should now be able to log in without authenticating for 24 hours!

Logging on to Perlmutter

You can set up an alias for Perlmutter, or you can ssh into Perlmutter by running the following command in the terminal

ssh perlmutter-p1.nersc.gov

Transferring files to/from NERSC:

For small files, you can use SCP (secure copy). To get a file from NERSC, use:

To send a file to NERSC, use:

Running and monitoring jobs:

The following instructions are for running on Perlmutter.

Most jobs are run in batch mode, in which you prepare a shell script telling the batch system how to run the job (number of nodes, time the job will run, etc.). NERSC’s batch system software is called SLURM. Below is a simple batch script example, copied from the NERSC website:

To submit your batch script, use sbatch myscript.sl in the directory containing the script file.

Below are some useful commands to control and monitor your jobs:

For Perlmutter GPU, the job scripts will look similar:

For more options in the executable, please refer to NERSC documentation. To work with the high-throughput infrastructure, please refer to "Fireworks & Atomate" in this handbook.

Choosing a QOS (quality of service):

You specify which queue to use in your batch file. Use the debug queue for small, short test runs, and the regular queue for production runs.

Automatic job submission on NERSC: scrontab

In atomate, by using --maxloop 3 for example when setting rocket_launch in your my_qadapter.yaml, after 3 trials in each minute if there are still no READY jobs available in your Launchpad Fireworks would stop the running job on NERSC to avoid wasting computing resources. On the other hand, if you have Fireworks available with the READY state and you have been using crontab for a few days, even if the jobs you submitted a few days ago start running on NERSC, they would pull any READY Fireworks and start RUNNING them reducing the turnaround from a few days to a few hours! So how to setup crontab? Please follow the instructions here: 1. ssh to the node where you want to setup the crontab; try one that is easy to remember such as cori01 or edison01; for logging in to a specific node just do for example “ssh cori01” after you log in to the system (Cori in this example).

1. Type and enter: scrontab -e

2. Now you can setup the following command in the opened vi editor. What it does is basically running the SCRIPT.sh file every 120 minutes of every day of every week of every month of every year (or simply /120 *):
3. Setup your SCRIPT.sh like the following: (as a suggestion, you can simply put this file and the log file which keeps a log of submission states in your home folder):

Running Jupyter Notebooks on perlmutter

Automatic Job Packing with FireWorks

DISCLAIMER: Only use job packing if you have trouble with typical job submission. The following tip is not 100% guaranteed to work., and is based on limited, subjective experience on Cori. Talk to Alex Dunn (ardunn@lbl.gov) for help if you have trouble.

The Cori queue system can be unreasonably slow when submitting many (e.g., hundreds, thousands) of small (e.g., single node or 2 nodes) jobs with qos-normal priority on Haswell. In practice, we have found that the Cori job scheduler will give your jobs low throughput if you have many jobs in queue, and you will often only be able to run 5-30 jobs at a time, while the rest wait in queue for far longer than originally expected (e.g., weeks). While there is no easy way to increase your queue submission rate (AFAIK), you can use FireWorks job-packing to “trick” Cori’s SLURM scheduler into running many jobs in serial on many parallel compute nodes with a single queue submission, vastly increasing throughput.

You can use job packing with the “multi” option to rlaunch. This command launches N parallel python processes on the Cori scheduling node, each which runs a job using M compute nodes.

The steps to job packing are: 1. Edit your my_qadapter.yaml file to reserve N * M nodes for each submission. For example, if each of your jobs takes M = 2 nodes, and you want a N = 10 x speedup, reserve 20 nodes per queue submission. 2. Change your rlaunch command to:

To have each FireWorks process run as many jobs as possible in serial before the walltime, use the --nlaunches 0 option. To prevent FireWorks from submitting jobs with little walltime left (causing jobs to frequently get stuck as “RUNNING”), set the --timeout option. Make sure --timeout is set so that even a long running job submitted at the end of your allocation will not run over your walltime limit. Your my_qadapter.yaml should then have something similar to the following lines:

Typically, setting N <= 10 will give you a good N-times speedup with no problems. There are no guarantees, however, when N > 10-20. Use N > 50 at your own risk!

Jobs on Premium QOS

By default, premium QOS access is turned off for everyone in the group. When there is a scientific emergency (for example, you need to complete a calculation ASAP for a meeting with collaborators the next day), premium QOS can be utilized. In such cases, please contact Howard (hli98@lbl.gov or on Slack) to request premium QOS access. The access will be then turned off automatically after three weeks or after the emergency has been dealt with.

Lawrence Berkeley National Laboratory's Laboratory Research Computing (LRC) hosts the Lawrencium supercomputing cluster. LRC operates on a condo computing model, in which many PI's and researchers purchase nodes to add to the system. Nodes are accessible to all who have access to the system, though priority access will be given to contributors of the specific nodes. BRC provides 3 types of allocations: Condo - Priority access for nodes contributed by the condo group. PI Computing Allowance (PCA) - Limited computing time provided to each PI member using Lawrencium.

Persson Group ES1 GPU Node Specs:

Setting up an LRC account

Please make sure you will actually be performing work on Lawrencium before requesting an account. To get an account on Lawrencium, navigate to , register an account and wait for approval form the LRC team. and the to a one-time password token generator and your account. You will also need to .

Before logging on (setup)

You must use the SSH protocol to connect to Lawrencium. Make sure you have SSH installed on your local computer (you can check this by typing which ssh). Make sure you have a directory named $HOME/.ssh on your local computer (if not, make it).

After your account is made, you'll need to set up 2-factor authentication. We recommend using Google Authenticator, although any OTP manager will work.

You should now be ready to log on!

Logging on to LRC

To access your shiny new Lawrencium account, you'll want to SSH onto the system from a terminal.

You will be prompted to enter your pin+OTP (e.g. <your_pin><OTP> without any spaces). This will take you to your home directory. You may also find it useful to set up an alias for signing on to HPC resources. To do this, add the following line to your bash_profile:

Now you will be able to initialize a SSH connection to Savio just by typing lawrencium in the command line and pressing enter.

Running on LRC

Under the condo accounts condo_mp_cf1 (56 cf1 nodes) and condo_mp_es1 (1 gpu node), we have exclusive access to certain Lawrencium nodes. If you do not know which of these node groups you are supposed to be running on, you probably shouldn't be running on Lawrencium. Additionally, we have the ability to run on ES1 GPU nodes at low priority mode (es_lowprio).

Accessing Software binaries

Software within LRC is managed through modules. You can access precompiled, preinstalled software by loading the desired module.

To view a list of currently installed programs, use the following command:

To view the currently loaded modules use the command:

Software modules can be removed by using either of the following two commands:

Using Persson Group Owned LRC nodes

To run on the nodes, use the following job script, replacing with the desired job executable name.

Using Non-Persson Owned LRC Nodes

In addition to using Persson owned nodes (lower wait times, limited capacity), you can also submit directly to the main LRC queue. For jobs that aren't on a time crunch low-turnaround time, this can be a great option because it will not saturate our condo nodes. All of the instructions are identical to above except the accoutn should be set to pc_mp instead of lr_mp.

Interactive Jobs on the Group GPU Condo Node

To run an interactive session on the GPU node, use the following two commands to provision and log in to the node: salloc --time=24:00:00 --nodes=1 -p es1 --gres=gpu:2 --cpus-per-task=8 --qos=condo_mp_es1 --account=lr_mp srun --pty -u bash -i

Berkeley Research Computing (BRC) hosts the Savio supercomputing cluster. Savio operates on a condo computing model, in which many PI's and researchers purchase nodes to add to the system. Nodes are accessible to all who have access to the system, though priority access will be given to contributors of the specific nodes. BRC provides 3 types of allocations: Condo - Priority access for nodes contributed by the condo group. Faculty Computing Allowance (FCA) - Limited computing time provided to each Faculty member using Savio.

Setting up a BRC account

Please make sure you will actually be performing work on Savio before requesting an account. To get an account on Savio, navigate to the BRC portal, register an account, make sure to select the appropriate allocation, and wait for approval from the BRC team. Typically, most students and postdocs will be running on co_lsdi. For more information, visit (Berkeley Research Computing)[http://research-it.berkeley.edu/services/high-performance-computing]

After your account is made, you'll need to set up 2-factor authentication. This will allow you to generate "one time passwords" (OTPs). You will need append a OTP to the end of your NIM password each time you log on to a NERSC cluster. We recommend using Google Authenticator, although any OTP manager will work.

Logging on (Setup):

You must use the SSH protocol to connect to BRC. Make sure you have SSH installed on your local computer (you can check this by typing which ssh). Make sure you have a directory named $HOME/.ssh on your local computer (if not, make it).

We also advise you to configure a ssh socket so that you only have to log into BRC with a OTP only once per session (helpful if you are scp-ing things). To do this:

1. Create the directory ~/.ssh/sockets if it doesn't already exist.

2. Open your ssh config file /.ssh/config (or create one if it doesn't exist) and add the following:

   Copy

   Host *.brc.berkeley.edu
   ControlMaster auto
   ControlPath ~/.ssh/sockets/%r@%h-%p
   ControlPersist 600

After your account is made, you'll need to set up 2-factor authentication. We recommend using Google Authenticator, although any OTP manager will work.

You should now be ready to log on!

Logging on to BRC

To access your shiny new savio account, you'll want to SSH onto the system from a terminal.

ssh your_username@hpc.brc.berkeley.edu

You will be prompted to enter your passphrase+OTP (e.g. <your_password><OTP> without any spaces). This will take you to your home directory. You may also find it useful to set up an alias for signing on to HPC resources. To do this, add the following line to your bash_profile:

alias savio="ssh your_username@hpc.brc.berkeley.edu"

Now you will be able to initialize a SSH connection to Savio just by typing savio in the command line and pressing enter.

Running on BRC

Under the condo account co_lsdi, we have exclusive access to 28 KNL nodes. Additionally, we have the ability to run on other nodes at low priority mode.

Accessing Software binaries

Software within BRC is managed through modules. You can access precompiled, preinstalled software by loading the desired module.

module load <module_name>

To view a list of currently installed programs, use the following command:

module avail

To view the currently loaded modules use the command:

module list

Software modules can be removed by using either of the following two commands:

module unload <module_name>
module purge

Accessing In-House software packages

The Persson Group maintains their own versions of popular codes such as VASP, GAUSSIAN, QCHEM and LAMMPS. To access these binaries, ensure that you have the proper licenses and permissions, then append the following line to the .bashrc file in your root directory:

export MODULEPATH=${MODULEPATH}:/global/home/groups/co_lsdi/sl7/modfiles

Using Persson Group Owned KNL nodes

To run on the KNL nodes, use the following job script, replacing with the desired job executable name. To run vasp after loading the proper module, use vasp_std, vasp_gam, or vasp_ncl.

#!bin/bash -l
#SBATCH --nodes=1                 #Use 1 node
#SBATCH --ntasks=64               #Use 64 tasks for the job
#SBATCH --qos=lsdi_knl2_normal    #Set job to normal qos
#SBATCH --time=01:00:00           #Set 1 hour time limit for job
#SBATCH --partition=savio2_knl    #Submit to the KNL nodes owned by the Persson Group
#SBATCH --account=co_lsdi         #Charge to co_lsdi accout
#SBATCH --job-name=savio2_job     #Name for the job

mpirun --bind-to core <executable>

Running on Haswell Nodes(on Low Priority)

To run on Haswell nodes, use the following slurm submission script:

#!bin/bash -l
#SBATCH --nodes=1                 #Use 1 node
#SBATCH --ntasks_per_core=1       #Use 1 task per core on the node
#SBATCH --qos=savio_lowprio       #Set job to low priority qos
#SBATCH --time=01:00:00           #Set 1 hour time limit for job
#SBATCH --partition=savio2        #Submit to the haswell nodes
#SBATCH --account=co_lsdi         #Charge to co_lsdi accout
#SBATCH --job-name=savio2_job     #Name for the job

mpirun --bind-to core <executable>

You should coordinate the specific days of vacation and personal time off with Kristin, especially for an extended absence.

Graduate Students

Graduate students researchers are allotted 12 days of personal time off per year (1 per/month) and graduate student instructors are allotted 3 days of personal time off per semester, as given by the UAW 2865 contracts. Additionally, graduate students are entitled to take off all holidays observed by LBNL. You should email Kristin well in advance (at least one month) with the days you plan to take off.

After discussing with Kristin, please add your PTO to the Persson Group Vacation Calendar. If you do not have access to this calendar, email Shannon for access and add it to your list of calendars.

LBL Postdocs

Postdocs will receive a set number of vacation / personal time off (PTO) days that will be outlined in your hiring package. For union postdocs, the union has currently negotiated 24 PTO days per year along with other benefits.

Berkeley Postdocs

As a new postdoc, you should submit a Postdoc time sheet each month to the Payroll team at 197M Cory Hall, ersopayroll@erso.berkeley.edu. This time sheet is used only to record the Personal Time Off and Sick Leave that you use. According to UC Postdoc policy, you are granted 24 days of Personal Time Off and 12 days of Sick Leave per 12-month appointment.

More information to come

Talk to Sam Blau, Ryan, or Evan if you have Q-Chem questions!

Lawrencium

module load /clusterfs/mp/modules/qchem/5.4.2

Cori

Coming soon!

Perlmutter

Coming soon!

If you encounter an error with NREL, there is most likely documentation on https://nrel.github.io/HPC/Documentation/Systems/Kestrel/.

Setting up NREL account

Visit website https://www.nrel.gov/hpc/user-accounts.html to get started on making a user account. You will need to setup a MFA before proceeding.

Send an email to HPC-Help@nrel.gov to get added on the correct projects. Ask Kristin or the NREL coordinators of the group to see which projects you belong to.

Logging on

For the purposes of this example, assume your NREL account username is: johndoe

Kestrel login:

ssh johndoe@kestrel.nrel.gov

Once in, the command line will prompt you to type in your account password+OTP.

General NREL HPC login:

ssh johndoe@hpcsh.nrel.gov

Use this for loggin into the NREL HPC systems. From there, you can connect to any other NREL HPC system like Swift, Eagle, etc.

Conda Environment Setup

I personally would recommend setting up miniconda instead of using their default conda modules. If you would like to use their default conda module, execute module load conda and proceed to the conda install steps.

Miniconda

You can find these install instructions on https://docs.anaconda.com/miniconda/#quick-command-line-install

cd ~/ #ensures you start off in your home dir

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

source ~/.bashrc

You should now see a (base) or (miniconda) on the lefthand side of your command prompt.

Setting up your conda environment

Warning: This is different for every user. Please check with your mentor before proceeding.

The following conda setup is designed for VASP users who will be using atomate2 with fireworks.

conda create -n cms python=3.9 pandas seaborn numpy scipy matplotlib
conda activate cms 

pip install git+https://github.com/materialsproject/atomate2.git #replace with your desired way to downloading atomate2
pip install fireworks
pip install pydantic==2.4.2 #to avoid some pydantic bugs with fireworks...

Please refer to https://materialsproject.gitbook.io/persson-group-handbook/computing/atomate-setup/setup#set-up-an-environment for setting up the fireworks yaml files and the fw_config directory.

In addition, also setup these two additional files:

atomate2.yaml

VASP_CMD: srun -n 104 -c 1 vasp_std 
VASP_GAMMA_CMD: srun -n 104 -c 1 vasp_gam

VASP_CMD: srun -n 104 -c 1 --cpu-bind=cores --gpu-bind=single:1 -G 4 vasp_std
VASP_GAMMA_CMD: srun -n 104 -c 1 --cpu-bind=cores --gpu-bind=single:1 -G 4 vasp_std

VASP_CMD: srun -n 104 -c 2 vasp_std 
VASP_GAMMA_CMD: srun -n 104 -c 2 vasp_gam

jobflow.yaml

Please use trajectory store version if you need to run AIMD/MD type calculations with atomate2

JOB_STORE:
  docs_store:
    type: MongoStore
    database: johndoe_general
    host: mongodb03.nersc.gov
    port: 27017
    username: johndoe_general_admin
    password: ***PASSWORD***
    collection_name: kestrel_outputs
  additional_stores:
    data:
      type: GridFSStore
      database: johndoe_general
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: ***PASSWORD***
      collection_name: kestrel_outputs_blobs

JOB_STORE:
  docs_store:
    type: MongoStore
    database: johndoe_general
    host: mongodb03.nersc.gov
    port: 27017
    username: johndoe_general_admin
    password: ***PASSWORD***
    collection_name: kestrel_outputs
  additional_stores:
    data:
      type: GridFSStore
      database: johndoe_general
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: ***PASSWORD***
      collection_name: kestrel_outputs_blobs
    trajectory:
      type: GridFSStore
      database: johndoe_general_admin
      host: mongodb03.nersc.gov
      port: 27017
      username: johndoe_general_admin
      password: s***PASSWORD***
      collection_name: kestrel_trajectory_blobs

We generally run 2 versions of VASP - 5.4.4 and 6. VASP 6 is used primarily on nodes operating on the Knights Landing (KNL) CPU architecture, while VASP 5.4.4 is generally used on all other CPUs.

VASP on NERSC

NERSC maintains compiled, and optimized versions of the vasp binaries. To get access to these binaries, you must be added to the group VASP license. If/when you determine that your research requires the use of VASP, please contact Kristin and request to be added to the VASP License. We currently have two sets of licenses i) UC Berkeley Affiliated (Purchased through VASP Software GmbH) and ii) LBNL Affiliated (Purchased through Materials Design). Once you have been added to one of the licenses, fill out the VASP License Confirmation Request to get access to the binaries on NERSC machines.

More information can be found in the NERSC documentation

The potcar files can be found in the following directory.

/project/projectdirs/matgen/POTCARs

*Note*: Users with only jcesr access will need to request access to matgen to access these POTCARs. Contact the NERSC Resource Manager and request to be added to the 'matgen' UNIX group.

VASP on Berkeley Research Computing (savio)

When running vasp on savio, we have access to our own compilations. To access the vasp binaries on savio, add the following line to your .bashrc file.

export MODULEPATH=${MODULEPATH}:/global/home/groups/co_lsdi/sl7/modfiles

POTCARs are located at:

/clusterfs/cloudcuckoo/POTCARs

For issues with these compilations, please contact Eric Sivonxay (esivonxay@lbl.gov)

Lawrencium

When running vasp on Lawrencium, we have access to our own compilations. To access the vasp binaries on Lawrencium, add the following line to your .bashrc file.

export MODULEPATH=${MODULEPATH}:/global/home/groups-sw/lr_mp/modules/vasp

Pre-release versions of the VASP modules are located at export /clusterfs/mp/temp_modfiles

POTCARs are located at:

/clusterfs/mp/software/POTCARs

For issues with these compilations, please contact Eric Sivonxay (esivonxay@lbl.gov)

Special thanks to the original authors of this page: Eric Sivonxay

Installing Python development environment

The best way to manage Python installations these days is a “conda env”. This will allow you to manage different Python “environments”, where each environment is a set of libraries that you have installed. For example, you can have one environment that uses Python 2.7 and has certain library versions installed, and another environment that uses Python 3.5 and has other libraries installed. Another advantage of conda environments is that you can apply the same procedure on NERSC and other computing centers that support conda.

To do this, follow the online instructions on installing a conda environment and see modifications below:

• https://docs.conda.io/en/latest/miniconda.html

• https://docs.conda.io/projects/conda/en/latest/user-guide/index.html

• (probably) prefer to install the “miniconda” version rather than anaconda

• (probably) prefer to install “miniconda 3” rather than “miniconda 2”. Both will work fine and allow you to do everything the other one does so don’t stress too much about this decision.

• When creating environments, use a command like this (note that this also installs recommended libraries):

  Copy

  conda create --name py3 python=3 numpy matplotlib seaborn plotly pandas flask pymongo scipy sympy scikit-learn jupyter

• If you want a reference guide to conda commands, try: https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html

Install high-throughput computation environment

Our group has a set of base codebases used for performing high-throughput calculations. Note that if your project does not involve high-throughput calculation, you may need only one or two of these libraries installed – ask your subgroup head if you are unsure.

After activating a conda environment, install the following packages using a combination of git clone >>REPO_NAME<< and python setup.py develop. Start with:

git clone https://www.github.com/materialsproject/fireworks

You might need to generate an ssh key for the git clone command to work: ssh-keygen -t rsa -b 4096 no password is probably OK unless you are security conscious add your SSH key to your Github profile

Then:

cd fireworks; python setup.py develop; cd ..

Repeat the process above for the remaining libraries:

git clone https://www.github.com/materialsproject/pymatgen
cd pymatgen; python setup.py develop; cd ..
git clone https://www.github.com/materialsproject/pymatgen-db
cd pymatgen-db; python setup.py develop; cd ..
git clone https://www.github.com/materialsproject/custodian
cd custodian; python setup.py develop; cd ..

Repeat the same process for a couple of other libraries on the hackingmaterials github site:

git clone https://www.github.com/hackingmaterials/atomate
cd atomate; python setup.py develop; cd ..
git clone https://www.github.com/hackingmaterials/matminer
cd matminer; python setup.py develop; cd ..

If you want, you can automatically source activate your environment in your .bash_profile file. This will automatically load your environment when you open a Terminal. Otherwise, you will start off in your default Mac Python.

Other things to do

• Bring the now-empty box to 143C, write your name on it, and add it to the tower.

• Set up your Time Machine backup (make sure you have purchased or received an external hard disk).

  https://support.apple.com/en-us/HT204412

• You can also set up an online backup plan (e.g., Crashplan or Backblaze) to provide you with a second backup.

• Install MongoDB.

• Install Docker.

• Purchase Microsoft office from LBNL software distribution; if you are a UC Berkeley student/employee you can get this, and lots of other software, for free: UC Berkeley Software Catalog

Graduate Students

New Students Moving from Another State/Country

Although the deadline for filing your Statement of Legal Residence (SLR) is in June, you must complete all necessary conditions for residency (getting your CA drivers license, registering your car in CA, etc) before the end of your first semester.

Postdocs

: Postdocs have the option to join the union which aims to improve wages, hours, benefits, and other employment aspects for its members. You are welcome to ask other postdocs for their opinions based on their experiences with the union as well.

Join the postdoc Slack for general discussions for postdoc life at LBL

For Affiliates

LBL sends out daily emails and offers other notification options through . In the past, some affiliates were not automatically signed-up for Elements announcements and missed important time sensitive information contained in these updates.

Please see the following subsections on fireworks/atomate setup and usage:

DEPRECATION WARNING

Use of pymatgen.io.openmm is no longer recommended. It has been superceded by workflows in atomate2. Once the tutorials are complete, someone ought to update this page :) Points of contact for MD questions: 1. Orion (leaving August 2024) 2. Nessa 3. Rohith

OpenMM

Our group has a robust infrastructure for setting up and running molecular MD simulations. The core of this infrastructure is in the Pymatgen . Once you have installed that package and its dependencies, you should be able to use the following code. For a quick install of the dependencies, execute these commands:

First, you should import all of the functions and objects that we'll use below.

Generators

A pymatgen InputSet contains all of the files needed to set up and run a simulation. Writing those files by hand would be incredibly tedious, so we use a Generator to create the InputSet. The Generator is instantiated with simulation properties as input, like temperature and force field. Once it is created, it can generated many InputSets with different sets of molecules.

Input Set

An InputSet is created by calling generator.get_input_set() and providing the molecules (as smiles) and the density of the solution. When this happens, a ton of code is run behind the scenes that sets up the simulation.

We might not want to run our simulation immediately! For example, we might want to upload it to a supercomputer. We can easily do this by saving all of the files with input_set.write_input.

It is equally easy to load an InputSet from a directory.

Simulations

An InputSet can create a dynamic OpenMM Simulation object. This simulation object can be run and modified using the OpenMM software package.

Once we have a simulation, we will first need to minimize the energy, otherwise the simulation box will explode. Once that's done, we can start the simulation!

Congratulations! You ran your first OpenMM simulation!

Reporters

By default, an OpenMM Simulation will not output any information. If we want data, we will need to add OpenMM Reporters to write out the trajectory and simulation state. In the next cell, I attach the most useful reporters, StateData, which writes state information, DCDReporter, which saves the trajectory, and PDBReporter, which saves the topology.

200 is an extremely short time interval, in reality, you would want to use something like 1,000 for the StateDataReporter and 10,000 for the DCD reporter. Next we can run the simulation and see that it ran. We are using a short time interval here because this is just a demonstration.

Running a real simulation

A real molecular dynamics simulation is slightly more complex than the toy examples we've covered here. We first need to equilibrate the pressure of the simulation, which will change the volume until the simulation contents are at atmospheric pressure. Then we need to anneal the simulation by heating it to 400K to make sure we find a true equilibrium. Fortunately, the Pymatgen code has very convenient functions for doing both of these things. A real simulation would look something like below (though in reality we would run it for many more timesteps).

In 2020, the Persson Group came together and drafted a list of Group Agreements. These are now listed on the official group website:

The agreements are a set of values that all members of the Persson Group, as it stood in 2020, agreed upon. The agreements are not intended as "rules", but as things that were important to us at the time and that we wanted to prioritize moving forward. However, the agreements list is a living document; future iterations of the group may decide to edit and establish different group agreements!

Our software stack

We contribute to and maintain many Python packages. Many of these can be found under the materialsproject organization on GitHub:

• - for representing and analyzing crystal structures, as well as setting up/performing manual calculations

• - code for the Materials Project API

• - for executing and managing calculation workflows on supercomputing resources. This helps you keep a calculation dashboard

• - Python library for writing computational workflows

• - instead of directly running an executable like VASP, one can wrap the executable in custodian to detect and fix errors

• - repository for materials science workflows.

• - new successor to atomate, written using jobflow. We recommend this package when possible.

• - for building modular, scalable data pipelines

• - builders and API for Materials Project data (built upon maggma)

• - for data mining / ML on materials science data

Additional software used by many members of our group include:

• - Molecular visualization software especially good for crystals

• - Molecular visualization software especially good for molecular dynamics trajectories. We have a group license for the pro version. If you feel you'll be actively using OVITO Pro, please contact Qian Chen for the group license key.

• - graphical molecule builder and viewer that integrates tightly with Q-Chem

• - graphical molecule builder and viewre

• - You can obtain a free academic license . There is also a free version with more limited capabilities, but it works fine for simple use cases.

• - molecular visualization program

• - Modern, flexible and extensible text editor from Microsoft. Compatible with Github Copilot!

• - Python text editor from JetBrains. Free with Github Student Developer Pack

• - Alternative to native Terminal. Has more features and allows for greater customization.

If you are a Windows user, consider making use of the following:

To learn how to use the software stack, you can consult the documentation of the individual codebases as well as review the following resources:

If you have a specific question, sometimes the easiest solution is to post it to the Slack group and crowdsource the answer!

Software help groups

If you have problems with software, and in particular the software maintained by our group and our collaborators, you should contact the appropriate help group. The documentation for the software will list what that channel is; if not, try the Github Issues page. If you are reaching out for help, try to provide everything needed to quickly reproduce and debug the problem (files, test code, etc).

Two other ways to get software help that are more self-guided are:

• If you are having trouble using a particular class or function, look for unit tests within the code, which often demonstrate how to use the class or function

• If the class or function has a unique name (e.g., MaterialsProjectCompatibility), another option is to both Google and search on github.com for the particular class/function. The github.com search will often reveal code snippets from users all around the world.

Perssongroup organization (Github)

Looking for a place to host your code? Our group has an organization on Github here:

Please contact Hui Zheng to be added to the organization.

Github Student Developer Pack (sign up for this!)

As someone who is affiliated with UC Berkeley, did you know that you can get free Github Pro and a ton of free software (e.g., GitKraken, Pycharm, Github Copilot)?

This is totally worth the time to sign up. You can find the application below:

-------

Special thanks to the original authors of this page: John Dagdelen and Eric Sivonxay

More Information to come

Lawrencium "Condo" Nodes

The group has some deidicated GPU nodes on Lawrencium that you can use. More information here.

ML with GPUs on Perlmutter

This page gives a brief guidance on setting up a new conda environment for running machine learning jobs on perlmutter. But you may find general info on:

• how to change default conda settings (e.g. environment directory and package directory to avoid home directory out-of-quota problem)

• how to clone an existing conda environment

• example slurm script to submit a job to perlmutter

Put conda on PATH

First, log in to perlmutter (without typing the $, same below)

$ ssh <username>@cori.nersc.gov   # replace <username> by your NERSC user name 
$ ssh perlmutter 

By default, conda is not on PATH. Get it on PATH by:

$ module load python

Clone an existing conda environment

Let's create an environment called matml, short for materials machine learning.

We can do $ conda create --name matml to create it, and then $ conda install python <other_package_name> to install packages. But if you intend to use PyTorch or TensorFlow, there are pre-installed versions on perlmutter that are built from source. These could be optimized for the hardware, so it would be better to use them. NERSC folks have already put them in a conda environment, and we just need to clone it.

First, check what versions are available (using pytorch as an example, but it is similar for tensorflow)

$ module spider pytorch

Then pick the version we want to use (e.g. pytorch/1.10.0, jot it down, you will use it later) and load it

$ module load pytorch/1.10.0
$ which python 

The purpose of which python is to find out the path to the pytorch conda environment, and you will see something like /global/common/software/nersc/shasta2105/pytorch/1.10.0/bin/python. This means the conda environment is at /global/common/software/nersc/shasta2105/pytorch/1.10.0 (NOTE, /bin/python is excluded).

Finally, clone the pytorch conda environment using its path to create the matml environment,

$ conda create --name matml --clone /global/common/software/nersc/shasta2105/pytorch/1.10.0

Submit a job

Test it works

Assume you have finished the above steps, and logged out of perlmutter.

The next time you log back in, the first thing is to get the necesary modules loaded:

$ module load python
$ module load pytorch/1.10.0 

and activate your environment:

$ conda activate matml

Note, the order of the above three commands matters here. The purpose is to ensure the correct python on your PATH. Make sure to load python first, then the torch environment (e.g. pytorch/1.10.0), and finally activate your own conda environment. You can check it by $ which python, and make sure it is from the matml enviroment (e.g. /global/common/software/matgen/<username>/conda/envs/matml/bin/python).

To test that everying works

$ python
>>> import torch 
>>>
>>> torch.cuda.is_available()
True
>>>
>>> device = torch.device("cuda")  # use gpu
>>> a = torch.tensor([1., 1.], device=device)
>>> b = torch.tensor([2., 2.], device=device)
>>> a+b
tensor([3., 3.], device='cuda:0')

Submit a batch job

Create a python script, named, e.g. my_first_torch_job.py

import torch 

device = torch.device("cuda")  # use gpu
a = torch.tensor([1., 1.], device=device)
b = torch.tensor([2., 2.], device=device)

print(a + b)

Create a slurm batch script for it, named, e.g. submit.sh

#!/bin/bash -l

#SBATCH --job-name=test_job
#SBATCH --time=0-00:10:00  
#SBATCH --ntasks=1      
#SBATCH --cpus-per-task=2     # 2 cpus for the job
#SBATCH --gpus=1              # 1 gpu for the job 
#SBATCH --constraint=gpu
#SBATCH --qos=regular         # use the `regular` queue
#SBATCH --account=matgen_g    # don't forget the `_g`; you may want to use `jcesr_g`

module load python
module load pytorch/1.10.0

conda activate matml

python my_first_torch_job.py

Submit your job (Ensure no conda environment is activated, otherwise module load python in submit.sh won't work. Use $ conda deactivate to deactivate if you are in an environment. Alternatively, you can log out and log in again before running the below command):

$ sbatch submit.sh

Contact

If you find something wrong, please fix it. If you are unsure, let Mingjian Wen know and he can probably help.

Talk to Rishabh, Jingyang, or Ryan if you have LAMMPS questions!

More Information to come

Creating a LAMMPS data file with relevant force field parameters

1. Create the molecule in Avogadro or wherever you want to

2. Then use antechamber to get the partial charges. Antechamber is part of AmberTools. Installation instructions are available in their manual

3. The relevant command:- antechamber -fi pdb -fo prepi -i <infile.pdb> -o output.prepi -c bcc -j 4 -at gaff -nc -1 The -nc flag is the net charge in the molecule. It will error out if not invoked for ions.

   Sometimes it might be advisable to take partial charges and LJ pair coefficients directly from papers. This is especially important if you are trying to benchmark your results against previous publications. For instance, in case of TFSI-, antechamber partial charges are significantly different than partial charges available in previous publications - https://pubs.acs.org/doi/full/10.1021/jp0686893

   In short, antechamber is a good starting point, but it is always advisable to tread carefully with partial charges in MD. Also remember, that these charges are not scaled. Why is charge scaling important? Go here – ‘Kirby, B. J.; Jungwirth, P. Charge Scaling Manifesto: A Way of Reconciling the Inherently Macroscopic and Microscopic Natures of Molecular Simulations. J. Phys. Chem. Lett. 2019, 10 (23), 7531–7536. https://doi.org/10.1021/acs.jpclett.9b02652.’

4. Open the .pdb file in VMD. Use Extensions -> TK console to open the TCL scripting menu

5. Then use this sequence of commands to get the bonds, angles, dihedrals and most importantly, the box size topo retype bonds topo guessangles topo guessdihedrals set sel [atomselect top all]

   set center [measure center $sel weight none]

   $sel moveby [vecscale -1.0 $center]

   set minmax [measure minmax $sel -withradii]

   set box [vecscale 1.1 [vecsub [lindex $minmax 1] [lindex $minmax 0]]]

   pbc set $box

6. Once all these TCL commands execute without any errors, we can get the lammps data file using: - topo writelammpsdata <lammps.data> full [Note: we can change the atom type to something else like atomic depending on the system, but mostly for molecular systems, full will be the way to go]

7. This is just a skeletal data file with the relevant bonds, angles, and dihedrals. No parameters are currently in this datafile. Sadly, the grunt work starts here. On paper, we can use the moltemplate package to get all the parameters, but it has never worked reliably (at least for me) . The work around is to visit the relevant force field parameter page on the moltemplate GitHub. For instance, for GAFF go to, https://github.com/jewettaij/moltemplate/blob/master/moltemplate/force_fields/gaff.lt and extract all the parameters (LJ, bond, angle, dihedral) from there. Remember to benchmark your LJ parameters.

8. Here is what a finished file will look like

LAMMPS Description 
8 atoms
7 bonds
12 angles
9 dihedrals
0 impropers

4 atom types
3 bond types
4 angle types
1 dihedral types
0 improper types

-3.443375 3.376625  xlo xhi
-3.591275 3.749025  ylo yhi
-2.930725 3.049975  zlo zhi

Masses

1 12.010700 # C
2 18.998404 # F
3 15.999400 # O
4 32.064999 # S

Atoms # full

1 1 3 -0.63000 -1.613375 -0.108625 -1.138875 # O UNL
2 1 4 1.02000 -0.839375 0.519375 -0.059875 # S UNL
3 1 3 -0.63000 -1.476375 0.389375 1.258125 # O UNL
4 1 1 0.35000 0.748625 -0.462625 0.053125 # C UNL
5 1 2 -0.16000 1.596625 -0.000625 1.027125 # F UNL
6 1 2 -0.16000 1.473625 -0.444625 -1.110875 # F UNL
7 1 2 -0.16000 0.539625 -1.787625 0.340125 # F UNL
8 1 3 -0.63000 -0.429375 1.895375 -0.368875 # O UNL

Bonds

1 3 1 2
2 3 2 8
3 2 2 4
4 3 2 3
5 1 4 6
6 1 4 7
7 1 4 5

Angles

1 4 1 2 8
2 1 1 2 4
3 4 1 2 3
4 1 4 2 8
5 4 3 2 8
6 1 3 2 4
7 3 2 4 6
8 3 2 4 7
9 3 2 4 5
10 2 6 4 7
11 2 5 4 6
12 2 5 4 7

Dihedrals

1 1 1 2 4 6
2 1 1 2 4 7
3 1 1 2 4 5
4 1 8 2 4 6
5 1 8 2 4 7
6 1 8 2 4 5
7 1 3 2 4 6
8 1 3 2 4 7
9 1 3 2 4 5

Pair Coeffs #lj/charmm/coul/long

1 0.0660 3.50000000000 #C Kelkar Maginn 
2 0.0530 2.95000000000 #F Kelkar Maginn
3 0.2100 2.95992190114 #O Kelkar Maginn
4 0.2500 3.56359487256 #S Kelkar Maginn

Bond Coeffs #harmonic

1 356.9 1.3497 # SOURCE1_SOURCE5 2188 0.0139, this is c3-f
2 233.5 1.8075 # SOURCE1_SOURCE5 644 0.0143, this is c3-s6
3 512.7 1.4533 # SOURCE1_SOURCE5 2456 0.0105, this is o-s6

Angle Coeffs #harmonic

1 65.4 108.61   # SOURCE3_SOURCE5 1062 1.0758, this is c3-s6-o
2 70.9 107.36   # SOURCE2_SOURCE5 1178 0.5429, this is f-c3-f
3 63.1 109.68   # SOURCE4_SOURCE5 57 0.4273, this is f-c3-s6
4 73.6 120.05   # SOURCE4_SOURCE5 971 1.8153, this is o-s6-o

Dihedral Coeffs #fourier

1 1 0.14445 3 0.0 #F-C-S-O used same parameters as X-c3-s6-X

Lawrencium

module load ms/lammps/3Mar20-mpi

Compiling LAMMPs

author: Bryant Li

The GPU parallelized versions that works with KOKKOS is still a work in process. Most of the time, there is a default module load lammps on each super computer. This is specifically designed for those we needs to compile LAMMPs with an external package. Please refer to https://docs.lammps.org/Build_extras.html for what LAMMPs can accommodate.

Kestrel

Below is an example with PyACE:

Useful links:

https://docs.lammps.org/Build_extras.html#ml-pace https://github.com/ICAMS/lammps-user-pace/ https://pacemaker.readthedocs.io/en/latest/pacemaker/quickstart/#lammps Note: if doesn't work/not fast enough, try to speed it up with this: https://docs.nersc.gov/development/programming-models/kokkos/

#substitute this line for whichever type of lammps is configured to your code
git clone https://github.com/ICAMS/lammps-user-pace.git


cd lammps
mkdir build
cd build

cmake -DCMAKE_BUILD_TYPE=Release -D BUILD_MPI=ON -DPKG_ML-PACE=ON ../cmake

make

After executing the last line in the command prompt, you will have an lmp binary in the lammps/build. directory. You can set up a submit script like below:

#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --time=24:00:00
#SBATCH --partition=standard
#SBATCH --account=bmrkp
#SBATCH --job-name=lmp_launcher
#SBATCH --output=lmp_launcher-%j.out
#SBATCH --signal=SIGINT@60
#SBATCH --mail-type=all
#SBATCH --mail-user=<<YOUR_EMAIL@lbl.gov>> #remove this line if you want

exe="/PATH_TO_LAMMPSBUILD/lammps/build/lmp"
input="-in in.lammps"

export OMP_PROC_BIND=spread
export OMP_PLACES=threads

command="srun -n 96 --cpu-bind=cores $exe $input" 

echo $command

$command

Perlmutter

Below is an example with pair_allegro. WARNING!!!! ONLY WORKS FOR CPU.

Prereqs:

cd repo
git clone -b stable_29Sep2021_update2 --depth 1 git@github.com:lammps/lammps
git clone git@github.com:mir-group/pair_allegro
cd pair_allegro
./patch_lammps.sh ../lammps/

Conda env setup:

conda create -n pair_allegro python=3.9 numpy matplotlib pandas plotly seaborn
conda activate pair_allegro 
pip install torch==1.12.1+cpu torchvision==0.13.1+cpu torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cpu
pip install mkl-include (apparently already within it...?)

Make lammps:

cd lammps
mkdir build
cd build

cmake ../cmake -DCMAKE_PREFIX_PATH=`python -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DMKL_INCLUDE_DIR=`python -c "import sysconfig;from pathlib import Path;print(Path(sysconfig.get_paths()[\"include\"]).parent)"`

make -j16

Module list at the time:

Currently Loaded Modules:
  1) craype-x86-milan                        6) cray-dsmml/0.2.2        11) xalt/2.10.2              16) gpu/1.0
  2) libfabric/1.15.2.0                      7) cray-libsci/22.11.1.2   12) Nsight-Compute/2022.1.1  17) evp-patch
  3) craype-network-ofi                      8) craype/2.7.19           13) Nsight-Systems/2022.2.1  18) python/3.9-anaconda-2021.11
  4) xpmem/2.5.2-2.4_3.20__gd0f7936.shasta   9) perftools-base/22.09.0  14) cudatoolkit/11.7         19) gcc/11.2.0
  5) PrgEnv-gnu/8.3.3                       10) cpe/22.11               15) craype-accel-nvidia80    20) cray-mpich/8.1.22

GPU Setup (THIS HASN'T WORKED FOR pair_allegro; due to a PyTorch error. You may have better luck if you use a different external code).

module list:

Currently Loaded Modules:
  1) craype-x86-milan                        6) cray-dsmml/0.2.2        11) xalt/2.10.2              16) gpu/1.0
  2) libfabric/1.15.2.0                      7) cray-libsci/22.11.1.2   12) Nsight-Compute/2022.1.1  17) evp-patch
  3) craype-network-ofi                      8) craype/2.7.19           13) Nsight-Systems/2022.2.1  18) python/3.9-anaconda-2021.11
  4) xpmem/2.5.2-2.4_3.20__gd0f7936.shasta   9) perftools-base/22.09.0  14) cudatoolkit/11.7         19) gcc/11.2.0
  5) PrgEnv-gnu/8.3.3                       10) cpe/22.11               15) craype-accel-nvidia80    20) cray-mpich/8.1.22

Prereqs:

cd pair_allegro_lammps
git clone -b stable_29Sep2021_update2 --depth 1 git@github.com:lammps/lammps
git clone git@github.com:mir-group/pair_allegro
cd pair_allegro
./patch_lammps.sh ../lammps/

Conda env setup:

conda create -n pair_allegro_gpu python=3.9 numpy matplotlib pandas plotly seaborn
conda activate pair_allegro_gpu 
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
pip install mkl-include (apparently already installed)

Make lammps:

cd lammps
mkdir build
cd build

module load cudnn/8.7.0


cmake ../cmake -DCMAKE_PREFIX_PATH=`python -c 'import torch;print(torch.utils.cmake_prefix_path)'` -DMKL_INCLUDE_DIR=`python -c "import sysconfig;from pathlib import Path;print(Path(sysconfig.get_paths()[\"include\"]).parent)" -DCUDA_TOOLKIT_ROOT_DIR=/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7

make -j16

The group meeting schedule is available on the group's Google doc.

Gaussian is available on Savio. Anyone wanting to use Gaussian on Savio must sign Exhibit A. Print out two copies of the form and sign both (i.e. they should not be signed electronically). Then provide both physical copies to Alice. If you are remote, an e-mailed copy will suffice for the time being.

GSRs, GSIs, and post-docs are all eligible to join the union, UAW 4811, and are protected by union contracts. However, a contract is only ever as strong as our ability to enforce it---we must actively keep the University accountable to its workers by ensuring that, if and when our rights guaranteed by the contract are violated, we work together to protect each other.

If you are facing any challenges in your workplace, whether that be with your research lab or a class you are teaching or otherwise, know that you do not have to go through it alone. If you have any issues in your workplace or think your working rights are being violated, and you would like to discuss what options for recourse you might have, you can always contact a union representative at organize@uc-uaw.org or your MSE department stewards (Mackinzie Farnell and Yoshi Chiu at mackinzie_farnell@berkeley.edu and yoshi_chiu@berkeley.edu).

Late pay

GSIs and GSRs have the right to timely, correct pay. If there are any issues with your pay, whether it is late or the incorrect amount, you have the right to receive your full paycheck, in addition to damages of $200 plus 25% of the paycheck amount.

Workload

For GSIs employed at 50% FTE, workload must be limited to 340 hours per semester. Additionally, during any given week, GSIs cannot be required to work more than 40 hours or more than 8 hours per day. The number of hours worked in excess of twenty (20) hours per week may not total more than 77 hours per semester. If you are expecting to exceed this workload maximum, you should communicate to your faculty supervisor prior to exceeding the maximums. Remedies to workload violations include increasing your appointment percentage (i.e., paying more) or modifying the work assignment to be commensurate with your appointment.

For GSRs, workload must be reasonable and related to the program's research needs, and commensurate to the appointment percentage. For GSRs employed at 50% FTE, this is 20 hours of work outside of your academic obligations. In MSE, GSRs are hired at 100% FTE during the summer--any work in excess of 40 hours per week is a definite workload violation. If you are working significantly more than 40 hours per week on research obligations, you are likely exceeding appropriate workloads for your position. Violations of workload are subject to an expedited grievance process.

Paid time-off

In addition to holidays observed by the University, which you cannot be required to work on:

• GSRs have up to 12 work days per year (1 per month) of personal time off

• GSIs have up to 3 work days per semester of personal time off

• GSIs and GSRs have up to 8 weeks of paid leave, which can be parental leave, for their own health, or to take care of health of a family member

• GSIs and GSRs have up to 5 days of bereavement leave, due to death of a family member

If you are required to work on a University observed holiday, for example, to monitor a critical experiment, you have the right to take an alternative day off.

Protections against bullying and harassment

For GSIs and GSRs, you have the right to a workplace environment that is safe, respectful, and free of any harassment or abusive conduct. The types of abusive conduct which we are protected against is expansive--if you feel that you are being treated poorly, or are made uncomfortable or unsafe at work, you have the right to be protected. Remedies for violations of our right to a respectful work environment are equally expansive, and could include eliminating 1:1 meeetings, having a union representative present at all meetings, having third party reviews of recommendation letters, non-contact orders, facilitated placements into new labs, and more. Protecting yourself from abusive behavior will not jeopardize your academic career--please reach out to a union representative if you need help.

For any meetings in which a worker could reasonably expect disciplinary action, the worker has the right to have a union representative present with them during the meeting.

Workplace safety

Workers have the right to safe working conditions, and the right to refuse working in unsafe working conditions. For example, labs and workspaces must have available and functioning safety equipment and have any maintenance issues be promptly addressed.

Access needs

Workers have the right to secure appropriate accomodations that address their equitable access to work place. When appropriate accomodations are not yet in place, workers have the right to interim accomodations as they engage in an interactive process with the University to develop appropriate accomodations. Medical documentation is not needed to secure accomodations; the University may request that the worker make a good-faith attempt to secure documentation for a particular accomodation, and the documentation can be procured from any relevant medical provider.

Appointment security

Any GSI or GSR positions offered to you, informally or formally, verbal or in writing, must be kept available to you. Positions cannot be taken away from you--if you have a position taken away, you have the right to be employed in a commensurate position or for the wages in damages.

Who can I talk to?

The central email for the GSI and GSR unions at Berkeley is organize@uc-uaw.org-- any issue, question or concern you might have could be sent there, and a fellow member of our union will get in touch with you to help address whatever you might be facing. In the MSE department, your union stewards are Mackinzie Farnell (mackinzie_farnell@berkeley.edu) and Yoshi Chiu (yoshi_chiu@berkeley.edu).

What happens when I have a rights violation?

If there is an issue to be resolved, we can help file a grievance to remedy any issues and obligate the University to enforce the contract appropriately. The beginning of the grievance process is informal, and ideally, workplaces issues are addressed quickly at the informal stage. Should the informal stage not be sufficient to resolve a work place issue you might face, the grievance process proceeds more formally between the Union and the University. The grievance process is centered on the worker--any remedies suggested are designed specifically for the worker, to the extent that they seek a remedy, and to the extent to which they are comfortable filing a formal grievance. If you feel that your rights have been violated and are interested in pursuing the grievance process, reach out to a union representative; it could also be helpful for you to take personal notes and documentation concerning the rights violation.

This page is intended to help you get set-up the first time you are using fireworks and atomate so you can learn how these softwares work. This set-up is not ideal for most research applications. Please refer to Fireworks: Advanced Usage if you are looking for practical advice and tips on how to run calculations once you understand the basics of fireworks.

Note: These instructions are based on the original version of atomate. atomate2 is a rising code that is still in development. Generally these research codes are under continuous improvement, so if something isn't working here please ask! It is likely the documentation needs to be updated.

Set up an environment:

Create the conda environment:

module load python
conda create -n cms python=3.10

module load python/3.8
conda create -n cms python=3.9

module load python/3.9.12
conda create -n cms python=3.9

module load python
conda create -n cms python=3.10

module load python
conda create -n cms python=3.10

Note: cms is an abbreviation for "computational materials science," but feel free to pick your own environment name!

Activate the environment and install the base libraries:

source activate cms
conda install numpy scipy matplotlib pandas

To install atomate2 on the environment, use one of the following:

(For Most People) If you are just using atomate and fireworks, with no plans to develop code/ workflows, use the following:

pip install atomate2 fireworks
pip install ruamel.yaml==0.17.40

For developers:

source activate cms
cd ~/.conda/envs/cms
mkdir code
cd code
git clone https://github.com/materialsproject/pymatgen.git
git clone https://github.com/materialsproject/pymatgen-db.git
git clone https://github.com/materialsproject/fireworks.git
git clone https://github.com/materialsproject/custodian.git
git clone https://github.com/hackingmaterials/atomate.git

cd pymatgen
python setup.py develop
pip install -r requirements.txt
cd ..
cd pymatgen-db
python setup.py develop
pip install -r requirements.txt
cd ..
cd fireworks
python setup.py develop
pip install -r requirements.txt
cd ..
cd custodian
python setup.py develop
pip install -r requirements.txt
cd ..
cd atomate
python setup.py develop
pip install -r requirements.txt
cd ..

Getting Started Fireworks Configuration:

Create Fireworks configuration files

Note: You will want to follow the steps in this section on both your local computer and the supercomputer where you will be running your calculations (e.g. making a fw_config folder in your home directory that contains the 3 files: FW_config.yaml, db.json, and my_launchpad.yaml) so you can easily interface with fireworks from both your local computer and the supercomputer.

Make directory in your home directory called fw_config:

cd ~
mkdir fw_config
cd fw_config

Now create 3 files: FW_config.yaml, db.json, and my_launchpad.yaml with the following contents. Make sure you replace YOUR_USERNAME with your username; on NERSC you will also need to find your home folder, which is typically under another higher folder with the name of the FIRST LETTER of your username. Note: you can view your filesystem online. Also, consider setting up access to your NERSC filesystem locally on your computer: Check out "Mounting NERSC's file system locally" here:#running-jobs-on-nersc

FW_config.yaml (fireworks configuration file)

CONFIG_FILE_DIR: /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"

CONFIG_FILE_DIR: /global/home/users/YOUR_USERNAME/fw_config
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"

CONFIG_FILE_DIR: /HOME_DIRECTORY_PATH/fw_config
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"

CONFIG_FILE_DIR: /home/YOUR_USERNAME/fw_config/
QUEUE_UPDATE_INTERVAL: 5
ECHO_TEST: "cms fw_config activated"

db.json (Database file)

{
    "host": "mongodb07.NERSC.gov",
    "port": 27017,
    "database": "fw_db_name",
    "collection": "tasks",
    "admin_password": "<admin_password>",
    "admin_user": "<admin_username>",    
    "readonly_password": "<readonly_password>",
    "readonly_user": "<readonly_username>",
    "aliases_config": null
}

my_launchpad.yaml (Fireworks LaunchPad file)

host: mongodb07.NERSC.gov
port: 27017
name: 'fw_db_name'
username: '<admin_username>'
password: ‘<admin_password>’
logdir: null
Istrm_lvl: DEBUG
user_indices: []
wf_user_indices: []

Create Fireworker and Queue Launchers:

Fireworker file (my_fworker.yaml)

name: NERSC_fworker
category: ''
query: '{}'
env:
    db_file: /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'srun -n 64 -c 4 --cpu_bind=cores vasp_std'
    gamma_vasp_cmd: 'srun -n 64 -c 4 --cpu_bind=cores vasp_gam'
    scratch_dir: /global/cscratch1/sd/YOUR_USERNAME
    incar_update:

name: savio_fworker
category: ''
query: '{}'
env:
    db_file: /global/home/users/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'mpirun --bind-to core vasp_std'
    gamma_vasp_cmd: 'mpirun --bind-to core vasp_gam'
    scratch_dir: /global/scratch/YOUR_USERNAME/
    incar_update:

name: lrc_fworker
category: ''
query: '{}'
env:
    db_file: /global/home/users/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'mpirun --bind-to core vasp_std'
    gamma_vasp_cmd: 'mpirun --bind-to core vasp_gam'
    scratch_dir: /global/scratch/users/YOUR_USERNAME/
    incar_update:

name: gpu_fworker
category: ''
query: '{}'
env:
    db_file: /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'srun -n 4 -c 16 --gpu-bind=single:1 vasp_std'
    gamma_vasp_cmd: 'srun -n 4 -c 16 --gpu-bind=single:1 vasp_gam'
    scratch_dir: /pscratch/sd/FIRST_LETTER/YOUR_USERNAME
    incar_update:

name: NREL_fworker
category: ''
query: '{}'
env:
    db_file: /home/YOUR_USERNAME/fw_config/db.json
    vasp_cmd: 'srun -n 64 -c 1 --cpu_bind=cores vasp_std'
    gamma_vasp_cmd: 'srun -n 64 -c 1 --cpu_bind=cores vasp_gam'
    scratch_dir: /projects/PROJECT_NAME/YOUR_USERNAME
    incar_update:

Queue Adapter (my_qadapter.yaml):

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
walltime: '24:00:00'
account: matgen
job_name: knl_launcher
signal: SIGINT@60
qos: regular
constraint: 'cpu'
pre_rocket: |
            source activate cms
            module load vasp-tpc/5.4.4-knl
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/home/users/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
walltime: '24:00:00'
account: co_lsdi
job_name: knl_launcher
queue: savio2_knl
qos: lsdi_knl2_normal
ntasks: 64
pre_rocket: |
            source activate cms
            module load vasp
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
            post_rocket: null

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/home/users/YOUR_USERNAME/fw_config/my_fworker.yaml singleshot
nodes: 1
ntasks: 64
queue: cf1
walltime: '06:00:00'
account: lr_mp
job_name: lrc_cpu_launcher
signal: SIGINT@60
qos: condo_mp_cf1
pre_rocket: |
            source activate cms
            module load vasp/6.prerelease
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/perlmutter/my_fworker.yaml singleshot
nodes: 1
walltime: '06:00:00'
account: matgen_g
job_name: gpu_launcher
signal: SIGINT@60
qos: regular
constraint: 'gpu'
pre_rocket: |
            source activate cms
            module load vasp/6.2.1-gpu
            export SLURM_CPU_BIND="cores"
            unset MPICH_GPU_SUPPORT_ENABLED

_fw_q_type: SLURM
rocket_launch: rlaunch -w /home/YOUR_USERNAME/fw_config/my_fworker.yaml rapidfire --max_loops 5
nodes: 1
ntasks_per_node: 128
walltime: '12:28:00'
account: PROJECT_NAME
job_name: job_name_var
exclude_nodes: 'c5-17,c1-39,c5-27,c1-61,c12-6,c1-37,c10-27,c10-18,c1-36,c9-13,c4-6,c1-58,c6-43,c9-15,c4-48,c5-55,c1-38,c7-8,c1-63,c10-49,c4-22'
signal: SIGINT@60
queue: standard
pre_rocket: |
            export ATOMATE2_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/atomate2/config/atomate2.yaml"
            export JOBFLOW_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/atomate2/config/jobflow.yaml"
            export FW_CONFIG_FILE="/home/YOUR_USERNAME/fw_config/FW_config.yaml"
            # load conda environment below if needed
            source activate cms
            module load vasp/cpu_6.4.2-intel
            module load gcc
            export OMP_PROC_BIND=true
            export OMP_PLACES=threads
            export OMP_NUM_THREADS=1
            export KMP_AFFINITY=balanced 

_fw_name: CommonAdapter
_fw_q_type: SLURM
rocket_launch: rlaunch -w /home/USERNAME/fw_config/my_fworker.yaml rapidfire --max_loops 5
nodes: 1
walltime: '24:00:00'
job_name: vasp_launcher
signal: SIGINT@60
queue: standard
account: drxkp
pre_rocket: |
                conda activate at2
                module load vasp/6.4.2
                module load craype-x86-spr  #specifies sapphire rapids architecture
                export OMP_NUM_THREADS=1
                export KMP_AFFINITY=balanced # for codes built with intel compilers

                ulimit -Ss unlimited #specific to resolve for SIGSEGV error

Available queues, partitions, and qos descriptions can be found at the following links:

• NERSC

• Savio

• Lawrencium

Note: specifying singleshot in the queue adapter will limit each reserved job to running only one firework (even if other fireworks are ready and could run with your remaining wall time). This can be changed to rapidfire but this may result in lost runs (fireworks that do not complete because they run out of wall time).

Configure Bash profile:

Append the following lines to the .bashrc or .bashrc.ext file, which is located in your home directory, e.g. /global/homes/s/sivonxay:

module load python
export PMG_VASP_PSP_DIR='/global/cfs/cdirs/matgen/POTCARs'
export FW_CONFIG_FILE='/global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'

module load python/3.8
export PMG_VASP_PSP_DIR=/clusterfs/cloudcuckoo/software/POTCARs
export FW_CONFIG_FILE='/global/home/users/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'

module load python/3.9.12
export PMG_VASP_PSP_DIR=/clusterfs/mp/software/POTCARs
export FW_CONFIG_FILE='/global/home/users/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'

export FW_CONFIG_FILE='/HOME_DIRECTORY_PATH/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'

module load python
export PMG_VASP_PSP_DIR='/global/cfs/cdirs/matgen/POTCARs'
export FW_CONFIG_FILE='/global/homes/FIRST_LETTER/YOUR_USERNAME/fw_config/FW_config.yaml'
alias cdconfig='cd ~/fw_config'

Note: the line alias cdconfig='cd ~/fw_config' is optional but is recommended so you can more easily locate the directory with your configuration files stored by typing cdconfig into the command line.

Setup API key and Add POTCAR Directory to pymatgen

Go to materialsproject.org and get an API key. Make sure you are using the environment setup above to run these commands in terminal with your cms python environment activated.

pmg config --add PMG_MAPI_KEY <USER_API_KEY>
pmg config --add PMG_VASP_PSP_DIR /global/cfs/cdirs/matgen/POTCARs

pmg config --add PMG_MAPI_KEY <USER_API_KEY>
pmg config --add PMG_VASP_PSP_DIR /clusterfs/cloudcuckoo/POTCARs

pmg config --add PMG_MAPI_KEY <USER_API_KEY>
pmg config --add PMG_VASP_PSP_DIR /clusterfs/mp/software/POTCARs

Running these commands will create a .pmgrc.yaml file (if it doesn’t exist already) containing these configuration settings in your home directory

Using Fireworks and Atomate:

The Materials Project Workshop has some great sessions introducing how to use fireworks and atomate. It might be helpful to watch the youtube recordings before trying out launching your own calculations!

2019 MP Workshop: Atomate Basics

2021 MP Workshop: Automating DFT

Initializing Fireworks Database:

First, let's confirm that you have your fireworks configuration set-up correctly to point to the right MongoDB database for your launchpad. Check your default fireworks configuration file with echo.

echo $FW_CONFIG_FILE

Copy the file path printed out and check the contents of the specified FW_config.yaml file to find your FW_CONFIG_DIR using cat.

cat /HOME_DIRECTORY_PATH/fw_config/FW_config.yaml

You should see something like this printed out. Copy the file path specified by CONFIG_FILE_DIR.

CONFIG_FILE_DIR: /HOME_DIRECTORY_PATH/fw_config

QUEUE_UPDATE_INTERVAL: 5

ECHO_TEST: "cms fw_config activated"

Print out the contents of the my_launchpad.yaml file in your CONFIG_FILE_DIR using cat.

cat YOUR_FW_CONFIG_FILE_PATH/my_launchpad.yaml

If you do not see something that resembles your my_launchpad.yaml file written earlier, there may be something wrong with your fireworks configuration that needs to be resolved before moving forward.

BE EXTREMELY CAREFUL WHEN RUNNING THIS RESET COMMAND. It will wipe all existing entries in your fireworks database in the fireworks, workflows, and launches collections. You must activate your python environment where fireworks is installed before running this command in terminal.

lpad reset

Using Atomate to Create a Workflow

The following code blocks are python code that should be run in jupyter or via python from the terminal. Make sure you have the environment setup above activated. The computer running the code should have access to mongodb07.NERSC.gov; this can be disregarded when running directly on NERSC or when connected to to the LBNL intranet. For computers outside of LBNL, a VPN will need to be used.

Run the following python code in a jupyter notebook or by creating a file named make_workflow.py and running the command python make_workflow.py in terminal:

from pymatgen.ext.matproj import MPRester
from fireworks import LaunchPad
from atomate.vasp.workflows.presets.core import wf_static
from atomate.vasp import powerups

mpr = MPRester()

structure = mpr.get_structure_by_material_id("mp-145")
wf = wf_static(structure)
wf = powerups.add_common_powerups(wf, {"scratch_dir":">>scratch_dir<<"})

lp = LaunchPad.from_file('YOUR_FW_CONFIG_FILE_PATH/fw_config/my_lp.yaml')
lp.add_wf(wf)

This adds a static energy calculation for Hexagonal Lu to your job database.

To check if this workflow has been added to your launch pad, run this command in terminal:

lpad get_fws -s READY

You can also check if this workflow has been added successfully by looking at your MongoDB fireworks collection or the launchpad webgui.

Launching a Workflow

Use qlaunch command to reserve jobs with SLURM’s scheduler. qlaunch has 3 modes: singleshot, rapidfire, and multi. Singleshot is used to launch one job, rapidfire is used to launch multiple jobs in quick succession, and multi creates one job with multiple fireworks runs. You’ll probably want to use rapidfire (where it is important to add the --nlaunches flag to specify how many fireworks to run). Here is example code for launching one job from the command line.

Note: Make sure you are in the directory you want your calculations to be run from before you run these commands, you may want to create a new folder for this. Do not run these commands from your home directory.

conda activate cms
echo $FW_CONFIG_FILE
qlaunch singleshot

conda activate cms
echo $FW_CONFIG_FILE
qlaunch rapidfire --nlaunches 1

Monitoring Your Launchpad (Fireworks Database)

Terminal Commands

Status of running fireworks can be determined by using the command: