3. Compute Platform

Simulations like WRF often require greater computing resources that you may have on your personal computer, but High Performance Computing (HPC) systems provide the needed computing power. I-WRF can be run on user workstations or on select HPC platforms and details about gaining access to them are provided below.

3.1. NSF NCAR

The NSF NCAR computing resources are available for projects large or small; for researchers, instructors, and students at U.S. academic institutions; and for modeling, machine learning, and analysis activities.

Instructions
Get Access to NCAR HPC

To get started with running I-WRF on NCAR HPC systems, you will need access to one of the supported platforms (Derecho, Casper, or other NCAR computing resources). If you do not already have access, you will need to request an account through the NCAR Computing and Information Systems Laboratory (CISL).

NCAR HPC access is typically provided to researchers affiliated with NCAR, university collaborators, or those with approved allocations. If you need to request access:

  • Visit the Getting Started with NSF NCAR HPC Resources page to create an account

  • Provide information about your research project and institutional affiliation

  • Wait for account approval, which typically takes 1-2 business days

  • Once approved, you can log in to the HPC systems using SSH

Most NCAR HPC systems use a batch job scheduling system for computational work. While the I-WRF exercises can be run interactively for demonstration purposes, production runs should be submitted through the job scheduler. Refer to the NCAR HPC documentation for specific guidance on your target system. Once you have access to an NCAR HPC system, you can log in using SSH from your local machine or through NCAR’s web-based interfaces where available.

Log in to NCAR HPC

To log in, start your terminal or Secure Shell client and run an ssh command:

ssh -X username@derecho.hpc.ucar.edu

The -X is optional and requests simple X11 graphics forwarding to your client. You can omit username in the command above if your Derecho username is the same as your username on your local computer.

With your successful login to Derecho through SSH, you can now create the run folders, install Docker software and download the data to run the simulation and verification.

3.2. Jetstream2

The Jetstream2 cloud computing platform is a flexible, user-friendly cloud computing environment available to U.S.-based researchers and educators at no cost through support from the National Science Foundation’s Advanced Cyberinfrastructure Coordination.

It is recommended that you follow the instructions in each section in the order presented to avoid encountering issues during the process. Most sections refer to external documentation to provide details about the necessary steps and to offer additional background information.

Instructions
Get an ACCESS Account

To get started with Jetstream2, you will need to create an account with the National Science Foundation (NSF)’s ACCESS program. If you do not already have one, register for an ACCESS account. When registering your account, you can either choose to associate your existing University/Organizational account or create an entirely new ACCESS account when registering. The Jetstream2 team strongly recommends that you create a new ACCESS account, as your organizational affiliation may change in the future.

Add ACCESS Account Allocation

Once you have an account you will need to have a computational “allocation” added to that account. Allocations provide the credits you will spend when running instances on Jetstream2. If there are no allocation owners who can add you to their allocation and you are interested in obtaining your own, you may request an allocation that will allow you to use an ACCESS-affiliated cyberinfrastructure resource. Note that allocations may only be requested by faculty, staff and graduate researchers, so undergraduates must work with a faculty sponsor to requeste an allocation.

Be sure to read all of the information on the request page so that you make a suitable request. Note that you will need to describe the project for which the allocation is intended and provide a CV or resume for the principal investigator. An “Explore” project (400,000 credits) will be much more than enough to work with this exercise. You will want to work with the resource “Indiana Jetstream2 CPU” (not GPU). The typical turnaround time for allocation requests is one business day.

Log in to Jetstream2’s Exosphere Web Site

Once you have an ACCESS account and an allocation has been added to it, you can log in to Jetstream’s Exosphere web dashboard. The process of identifying your allocation and ACCESS ID to use Jetstream2 is described on this page of the Introduction to Jetstream2 Cornell Virtual Workshop, and on this page of the Jetstream2 documentation.

While adding an allocation to your account, it is recommended that you choose the “Indiana University” region of Jetstream2 for completing this exercise.

Create a Cloud Instance on Jetstream2

After you have logged in to Jetstream2 and added your allocation to your account, you are ready to create the cloud instance where you will run the simulation and verification. If you are not familiar with the cloud computing terms “image” and “instance”, it is recommended that you read about them before proceeding.

For this tutorial, you will be able to log in to your instance using Exosphere’s Web Desktop or Web Shell functionalities. If you would rather log in using the ssh command from a shell on your own computer, you will need to create an SSH key pair and upload it to Jetstream2 before creating your instance. Optional information about doing those things is available here:

Creating an SSH Key and uploading it to Jetstream2 (Optional)

You may choose to upload a public SSH key to Jetstream2 before creating your instance. Jetstream2 will inject that public key into an instance’s default user account, and you will need to provide the matching private SSH key to log in to the instance. If you are not familiar with “SSH key pairs”, you should read about them before continuing.

  • First, create a Jetstream2 SSH Key on your computer using the ssh-keygen command. That command allows you to specify the name and location of the private key file it creates, with the default being “id_rsa”. The matching public key file is saved to the same location and name with .pub appended to the filename. Later instructions will assume that your private key file is named id_rsa, but you may choose a different name now and use that name in those later instructions.

  • Then, upload the public key to Jetstream2 through the Exosphere web interface.

The Cornell Virtual Workshop topic Creating an Instance provides detailed information about creating a Jetstream2 instance. While following those steps for this tutorial, be sure to make the configuration choices specific to the I-WRF use case. For beginners, we recommend the Hurricane Matthew use case (Note: a new window will open).

After clicking the “Create” button, wait for the instance to enter the “Ready” state (it takes several minutes). Note that the instance will not only be created, but will be running so that you can log in right away.

Log in to the Instance

The Exosphere web dashboard provides two easy-to-use methods for logging in to your instance through a web browser. The “Web Shell” button will open a terminal to your instance, and the “Web Desktop” button will open a view of the instance’s graphical desktop (if enabled). Both views open in a new browser tab, and Exosphere automatically logs you in to the instance. For this tutorial you should open a Web Desktop so that you will be able to view the plots (if any) that are generated from the simulation output.

If you wish to log in to the instance from a shell on your computer, you can do so following the information in this optional content:

Logging in to a Jetstream2 Instance using SSH From a Shell

You can use the SSH command to access your instance from a shell on your computer. The instructions for connecting to Jetstream2 using SSH can be executed in the Command Prompt on Windows (from the Start menu, type “cmd” and select Command Prompt) or from the Terminal application on a Mac.

In either case you will need to know the location and name of the private SSH key created on your computer (see SSH section, above), the IP address of your instance (found in the Exosphere web dashboard) and the default username on your instance, which is “exouser”.

Once you are logged in to the instance, your shell prompt will have the form exouser@instance-name:~$, which indicates your username, the instance name, and your current working directory, followed by “$”.

If your shell ever becomes unresponsive or disconnected from the instance, you can recover from that situation by opening a new Web Desktop (if available) or rebooting the instance. In the Exosphere dashboard page for your instance, in the Actions menu, select “Reboot”. The process takes several minutes, after which the instance status will return to “Ready”.

Managing a Jetstream2 Instance

In order to use cloud computing resources efficiently, you must know how to manage your Jetstream2 instances. Instances incur costs whenever they are running (on Jetstream2, this is when they are “Ready”). “Shelving” an instance stops it from using the cloud’s CPUs and memory, and therefore stops it from incurring any charges against your allocation.

When you are through working on an I-WRF exercise, you should shelve your instance. Note that any programs that are running when you shelve the instance will be terminated, but the contents of the disk are preserved when shelving.

To shelve, you need to be in the details page for your instance (with the “Actions” menu in the upper right). If you are on the Instances page, click and instance’s name to be taken to its details page. From the Actions menu, select Shelve. You will be prompted in that location to confirm the shelve action - click Yes to complete the action. In the Instances page your instance will briefly be listed as “Shelving”, and then as “Shelved” when the operation is complete.

When you later return to the dashboard and want to use the instance again, use the Action menu’s “Unshelve” option to start the instance up again. You can also use the “Resize” action to change the flavor (number of CPUs and amount of RAM) of the instance. Increasing the number of CPUs can make your computations finish more quickly, but doubling the number of CPUs doubles the cost per hour to run the instance, so Shelving as soon as you are done becomes even more important!

3.3. Red Cloud

The Red Cloud cloud computing platform, provided by Cornell Center for Advanced Computing (CAC), is a subscription-based Infrastructure as a Service cloud that provides root access to virtual servers and on-demand storage to Cornell researchers.

It is recommended that you follow the instructions in each section in the order presented to avoid encountering issues during the process. Most sections refer to external documentation to provide details about the necessary steps and to offer additional background information.

Instructions
Prepare to Use Red Cloud

To get started with Red Cloud, you will need to:

  • Go to the CAC portal and log in. The instructions to log in are on the CAC TechDocs page: Portal Login.

  • Get access to Red Cloud by doing one of the following options:

    1. Start a new project (only available for Cornell Faculty and Staff).

    2. Join an existing project.

    3. Request an exploratory account.

  • For new projects and existing projects, make sure that the project has Red Cloud subscriptions.

  • Once you join a project with Red Cloud subscription, log in to Red Cloud Horizon web interface.

The sections below will guide you through this process. For an overview of Red Cloud, read Cornell TechDocs Red Cloud documentation.

Option 1: Start a Project

One way to get a CAC account is to request a project. Note that you must be a Cornell faculty member or staff member to view the pages below and start a project. You may submit a project request at the CAC portal. Thoroughly review the rates (login required) page to understand the Red Cloud subscription service.

Once your project is approved, you can manage your project on the CAC portal. Read the Portal Overview to learn how to manage a project. Detailed instructions to start a project are available at the CAC TechDocs page: As a Cornell Faculty or Staff, How Do I Start a New Project?

Option 2: Join a Project

To join an existing project, submit a request to join on the CAC portal. You should only do this if your PI has requested you to submit the request. Once the PI of the project approves the request, an email is sent to you with the login information. For the full instructions, navigate to the CAC TechDocs page: How Do I Join an Existing Project?

Option 3: Open an Exploratory Account

You may also request an exploratory account if you have not made one already. This account has limited computing hours and storage. This is sufficient for the Hurricane Matthew exercise, but does not have enough compute hours to complete the LULC exercise. To request an exploratory account, follow the instructions on the CAC TechDocs page: How Do I Request an Exploratory Project? You are also given one hour of free consulting for any help you may need.

Log in to Red Cloud Horizon Interface

Once you join a project with Red Cloud subscription, you can log into the Red Cloud Horizon web interface.

Create a Cloud Instance and Log In

After you have logged in to the Red Cloud Horizon interface, you are ready to create the cloud instance where you will run the I-WRF simulation. If you are not familiar with the cloud computing terms “image” and “instance”, it is recommended that you read about them here before proceeding: Red Cloud: Images and Red Cloud: Run Linux Instance.

Create an SSH Key

You can either upload a public SSH key to Red Cloud or generate an SSH key pair on Red Cloud before creating your instance. Red Cloud injects the uploaded public key or generated public key into the instance’s default user account, and you will need to provide the matching private SSH key to log in to the instance. If you are not familiar with “SSH key pairs”, you should read about them before continuing.

  • It’s highly recommended that you create a key pair on Red Cloud. Be sure to follow the steps and save the private key it generated with the correct format and permission before proceeding. This is the easiest way to generate a key pair.

  • Alternatively, create an SSH Key on your computer using the “ssh-keygen” command. That command allows you to specify the name of the private key file it creates, with the default being “id_rsa”. The matching public key file is saved and named with “.pub” appended to the filename. Then, import the public key to Red Cloud through the Red Cloud web interface.

Create a Security Group

Security groups are firewalls that control inbound and outbound network traffic to your instances. For an instance to be accessible, its security group must have port 22 (SSH) enabled. You can read more about them at Red Cloud: Security Groups.

If you will access the instance from a Cornell Network (eduroam Wi-Fi, Ethernet, Cornell VPN, etc.), it’s sufficient to use the security group that already exists on your project: “campus-only-ssh”. This security group is already configured to enable SSH traffic from anywhere in the Cornell Network.

If you cannot use any of the options above, you will need to create a security group and add an SSH rule for an IP address you frequently use. Follow the steps below to create a security group:

Note that once you put your IP address in the CIDR field, you may connect to the instance from that IP address. If your IP address changes for any reason, you will need to remove and update the rule.

Create an Instance

The Cornell TechDocs Creating a New Linux Instance provides detailed information about creating a Linux instance on Red Cloud. While following those steps for this tutorial, be sure to make the configuration choices specific to the I-WRF use case. For beginners, we recommend the Hurricane Matthew use case (Note: a new window will open).

When all the required options are selected, click on the “Launch Instance” button, and wait for the instance to enter the “Active” state. Note that the instance will not only be created, but will be running so that you can log in right away.

Log in to the Instance

The instructions for connecting to Red Cloud Linux instances using SSH can be executed in the Command Prompt on Windows (from the Start menu, type “cmd” and select Command Prompt or search for PowerShell) or from the Terminal application on a Mac.

In either case, you will need to know the location and name of the private SSH key created on your computer or downloaded from Red Cloud (see above), the IP address of your instance (found in the Red Cloud Horizon Web Interface) and the default username on your instance, which is “ubuntu”.

You will know that your login has been successful when the prompt has the form ubuntu@instance-name:~$, which indicates your username, the instance name, and your current working directory, followed by “$”

Managing a Red Cloud Instance

In order to use cloud computing resources efficiently, you must know how to manage your Red Cloud instances. Instances incur costs whenever they are running (on Red Cloud, this is when they are “Active”). “Shelving” an instance stops it from using the cloud’s CPUs and memory, and therefore stops it from incurring any charges against your project.

When you are finished running I-WRF, be sure to use the instance’s dropdown menu in the web interface to “Shelve” the instance so that it is no longer spending your computing hours. If you later return to the web interface and want to use the instance again, Use the dropdown menu’s “Unshelve Instance” option to start the instance up again. Note that any programs that were running when you shelve the instance will be lost, but the contents of the disk are preserved when shelving.

You may also want to try the “Resize” action to change the number of CPUs of the instance. Decreasing the number of CPUs may slow down your computations, but it will also reduce the cost per hour to run the instance. Increasing the number of CPUs can make your computations finish more quickly. Doubling the number of CPUs doubles the cost per hour to run the instance. Nonetheless, it’s important to shelve the instance as soon as you are done.

3.4. Windows (Intel CPU)

Simulations like WRF often require significant computing resources, so it is recommended that the computer you use have at least four cores, 32 GB of RAM, and 50 GB of available disk space. This exercise runs the I-WRF programs as Docker “containers”, which simplifies the set-up work needed to run the simulation and verification. However, the code used to build those Docker containers was compiled expressly for use on Intel CPUs, so the Windows 10 or 11 computer you use must contain an Intel processor (note that these instructions are not intended for use on a system running Windows Server). Your Windows account will also need to have administrative privileges in order to perform all necessary steps.

Instructions

During the I-WRF exercises, you will create the run folders, install the software and download the data that are needed to run the simulation and verification. You will only need to perform these steps once. The I-WRF exercises will instruct you to issue numerous DOS commands in a Windows “Command Prompt” shell. To open such a shell:

  • Click the Start icon and then type cmd to display matching commands.

  • Right click on the “Command Prompt” option that is shown and select “Run as administrator”.

  • A black shell window should open.