6.2. Land Use/Land Cover Change

6.2.1. Scientific Objective

The Land Use/Land Cover Change (LULC) use case examines how changes in land surface characteristics impact weather patterns and atmospheric dynamics through idealized modeling scenarios. This science use case focuses on a deep convection system that passed over the Dallas-Fort Worth (DFW) metropolitan region on July 4th, 2017, and the simulations follow Zhou et al. 2024. This exercise provides an introduction to running computationally complex simulations and analyses using containerized applications. METplus provides statistical analysis and evaluation tools to quantify these meteorological impacts, making this verification workflow valuable for studies in urban meteorology, land-atmosphere interactions, and climate change impact assessment.

Reference

Zhou, X., Letson, F., Crippa, P. and Pryor, S.C., 2024. Urban effect on precipitation and deep convective systems over Dallas-Fort Worth. Journal of Geophysical Research: Atmospheres, 129(10), p.e2023JD039972.

6.2.2. Version Added

0.3

6.2.3. Datasets

This use case runs the Weather Research & Forecasting (WRF) and WRF Pre-Processing System (WPS) models with data from The High-Resolution Rapid Refresh (HRRR) and modified meteorological data.

6.2.4. Running This I-WRF Use Case

Instructions a provided below for running the Land Use/Land Cover Change use case for each Compute Platform on which it has been tested.

6.2.4.1. On NSF NCAR HPC

Follow the compute platform instructions for NSF NCAR to secure access to and log in to NSF NCAR HPC.

The following instructions can be used to run METplus verification for the LULC use case of the I-WRF weather simulation framework from the National Center for Atmospheric Research (NCAR) on NCAR High Performance Computing (HPC) platforms. The steps below configure and execute the METplus verification framework using applications containerized with Apptainer (formerly Singularity) to analyze I-WRF LULC simulation output.

NCAR HPC systems provide the computational resources needed for efficient METplus processing of large I-WRF datasets. This exercise uses containerized METplus applications, which simplifies the setup work needed to run the verification and ensures consistency across different HPC environments.

It is recommended that you follow the instructions in each section in the order presented to avoid encountering issues during the process. This guide assumes that I-WRF LULC simulations have already been completed and focuses specifically on the METplus verification workflow. The following sections instruct you to issue numerous Linux commands in your shell. The commands in each section can be copied and pasted into your shell terminal.

With your NCAR HPC account active and you logged in to the system, you can now set up the environment and create the necessary directories to run METplus verification.

Instructions
Load Required Modules

NCAR HPC systems use environment modules to manage software. Load the Apptainer module which provides the containerization software needed to run METplus:

module load apptainer
Confirm Bash Shell

The following commands are formatted to run in the Linux Bash shell, which you may or may not be running by default. Copy and paste the following command to print your default shell:

echo $SHELL

If needed, switch to the bash shell by running:

bash
${BASH_ENV}

Note that you can modify your default login shell through the Systems Account Manager (SAM) but changes can take up to several hours to take effect.

Define Environment Variables

We will be using environment variables throughout this exercise to ensure consistent file paths and resource names. Copy and paste the definitions below into your shell before proceeding:

IWRF_WORK_DIR=${SCRATCH}/iwrf_work
LOCAL_OUTPUT_DIR=${IWRF_WORK_DIR}/metplus_out
export APPTAINER_TMPDIR=${TMPDIR}

Any time you open a new shell session on the HPC system, you will need to reload the apptainer module, switch shells, if needed, and redefine these variables before executing the commands that follow.

Create Working Directories

The METplus verification process requires specific directory structures to organize input data, configuration files, and output results. Create the main working directory in your scratch space:

mkdir -p ${IWRF_WORK_DIR}

Create a directory to store the METplus verification output:

mkdir -p ${LOCAL_OUTPUT_DIR}

Create a directory for temporary Apptainer files. The $TMPDIR variable is automatically set on NCAR HPC systems to an appropriate temporary storage location:

mkdir -p ${APPTAINER_TMPDIR}
Download Configuration Files

METplus requires configuration files to direct its verification behavior. These are available in the I-WRF GitHub repository. Clone the repository to access the LULC use case configuration:

git clone https://github.com/NCAR/i-wrf ${IWRF_WORK_DIR}/i-wrf

This creates a local copy of all I-WRF configuration files, including the METplus settings needed for the LULC verification workflow.

Get the METplus and Data Container Images

As mentioned above, the METplus software is provided as a containerized image that will run using Apptainer on your NCAR HPC system. Apptainer (formerly Singularity) is the preferred containerization technology on HPC systems, as it provides secure container execution without requiring root privileges. Unlike cloud environments that use Docker directly, NCAR HPC systems use Apptainer to run containerized applications.

The METplus image contains all the necessary software and dependencies to perform verification of I-WRF LULC simulation output. You can “pull” (download) the METplus image from the container registry to your HPC system’s storage.

You must pull the METplus software container and the input data containers that contain the observational and WRF simulation data for the LULC use case:

apptainer pull ${IWRF_WORK_DIR}/iwrf-metplus.sif docker://ncar/iwrf-metplus:latest
apptainer pull ${IWRF_WORK_DIR}/data-lulc-input-obs.sif oras://registry-1.docker.io/ncar/iwrf-data:lulc-input-obs-d03.apptainer
apptainer pull ${IWRF_WORK_DIR}/data-lulc-input-wrf.sif oras://registry-1.docker.io/ncar/iwrf-data:lulc-input-wrf-d03.apptainer

These commands download three container images: the METplus software, the observational data, and the WRF simulation data. Processing the METplus software can take up to 15 minutes and the data containers can take several minutes each.

Configure Container Data Bindings

After the container images have been downloaded, you can run the METplus verification to compare the I-WRF LULC simulation results against observational data and generate statistical verification results and visualization plots. This process involves configuring the data bindings and executing the verification workflow for two meteorological variables.

METplus requires access to input data, configuration files, and output directories. Apptainer uses bind mounts to make local directories and container images available inside the running container. Set up the environment variables that define these data bindings.

First, define the local directory paths for configuration and visualization scripts:

LOCAL_METPLUS_CONFIG_DIR=${IWRF_WORK_DIR}/i-wrf/use_cases/Land_Use_Land_Cover/METplus
LOCAL_PLOT_SCRIPT_DIR=${IWRF_WORK_DIR}/i-wrf/use_cases/Land_Use_Land_Cover/Visualization

Next, configure the Apptainer bind mounts. This environment variable tells Apptainer how to map local directories and container images to paths inside the running container:

export APPTAINER_BIND="${IWRF_WORK_DIR}/data-lulc-input-obs.sif:/data/input/obs:image-src=/,${LOCAL_METPLUS_CONFIG_DIR}:/config,${IWRF_WORK_DIR}/data-lulc-input-wrf.sif:/data/input/wrf:image-src=/,${LOCAL_OUTPUT_DIR}:/data/output,${LOCAL_PLOT_SCRIPT_DIR}:/plot_scripts,${APPTAINER_TMPDIR}:${APPTAINER_TMPDIR}"

This configuration provides the container with access to:

  • Observational data from the data-lulc-input-obs.sif container image at /data/input/obs

  • WRF simulation data from the data-lulc-input-wrf.sif container image at /data/input/wrf

  • METplus configuration files from the I-WRF repository at /config

  • Visualization script files for generating plots at /plot_scripts

  • Output directory for writing verification results at /data/output

  • Temporary directory for Apptainer operations

Run METplus

The LULC use case includes verification for two meteorological variables: accumulated precipitation and radar reflectivity. Each verification is run separately using its own METplus configuration file.

Run the accumulated precipitation verification:

apptainer exec ${IWRF_WORK_DIR}/iwrf-metplus.sif /metplus/METplus/ush/run_metplus.py /config/GridStat_apcp_lulc.conf

This process compares simulated precipitation accumulation against observational data and generates statistical metrics. While it may take up to 20 minutes to run, progress information is displayed while the verification is performed.

Run the reflectivity verification:

apptainer exec ${IWRF_WORK_DIR}/iwrf-metplus.sif /metplus/METplus/ush/run_metplus.py /config/GridStat_refc_lulc.conf

This process evaluates the model’s ability to simulate radar reflectivity patterns compared to observed radar data. This step may take up to 10 minutes to run.

Both verification processes use GridStat, which computes grid-to-grid verification statistics. The tools generate comprehensive statistical output including bias, correlation, and skill scores that quantify the model’s performance.

Verify Output Generation

After both METplus runs complete successfully, you can verify that the output files were created properly.

Check that the GridStat verification output was generated:

ls -1 ${LOCAL_OUTPUT_DIR}/grid_stat/*

This should show directories containing statistical output files in text format that can be viewed and analyzed.

Check that the METplotpy visualization plots were created locally:

ls -1 ${LOCAL_OUTPUT_DIR}/met_plot/*/*.png

This should display a list of PNG image files containing plots and graphics that visualize the verification results. These plots provide graphical representations of the statistical comparisons between the I-WRF LULC simulations and observational data.

Visualize the Results

In the near future, this exercise will be extended to include instructions to visualize the model output directly.

6.2.4.2. On Red Cloud

Follow the compute platform instructions for Red Cloud to secure access to and log in to Red Cloud.

Instructions
Instance Configuration

Make the following choices when creating your instance:

  • When choosing an image as the instance source:

    • Select Boot from Source is “Image”

    • Volume Size (GB) is 1000

    • Delete Volume on Instance Delete is “Yes”

    • Select the “ubuntu-24.04-LTS” image

  • In Flavor, choose the “Flavor” c64.m120 (64 Virtual CPUs) to provide a faster simulation run-time. Note that this will consume Red Cloud subscriptions very fast.

  • In Network, select “public”.

  • In Security Groups, select “campus-only-ssh” or the security group you created.

  • In Key Pair, select the SSH public key that you created or uploaded previously.

Preparing the Environment

With your instance created and running and you logged in to it through SSH, you can now create the run folders, install Docker software and download the data to run the simulation and verification. You will only need to perform these steps once, as they essentially change the contents of the instance’s disk and those changes will remain even after the instance is shelved and unshelved.

The following sections instruct you to issue numerous Linux commands in your shell. If you are not familiar with Linux, you may want to refer to An Introduction to Linux when working through these steps. The commands in each section can be copied using the button in the upper right corner and then pasted into your shell by right-clicking.

Get the Docker Image

You must pull the correct versions of the image that will be used in this exercise onto your instance:

sudo docker pull ncar/iwrf:lulc-2024-10-04
Install and Enable CephFS

You need to access the data used in this exercise. In total, the full data are close to 90 GB in size. Usually, such large datasets cannot be shared easily. However, Red Cloud now has a Ceph cluster, a distributed file system that stores the data locally at Cornell CAC. Any Linux machine on the Cornell network can access this data using the following steps.

First, update the package list:

sudo apt update

Install CephFS client:

sudo apt install ceph-common

The CephFS mounting steps are slightly more complicated. When a CephFS share is created, access rules must be set for writing or reading the data. This credential is called a keyring, which consists of an entity name (accessTo) and a key (accessKey). For this exercise, copy and paste the credentials for read-only access:

accessTo="iwrf-lulc-read-only"
accessKey="AQBsg0lozVQiDxAAZYNFvpyD9lqdzYD1ouv/Wg=="

The following commands set up the keyring:

mkdir -p /etc/ceph
echo -e "[client.${accessTo}]\n    key = ${accessKey}" | sudo tee /etc/ceph/ceph.client.${accessTo}.keyring

The keyring file must be only readable to root:

sudo chown root:root /etc/ceph/ceph.client.${accessTo}.keyring
sudo chmod 600 /etc/ceph/ceph.client.${accessTo}.keyring

Choose the mount point for the CephFS share, which will be in the home directory:

cephfsPath="128.84.20.11:6789,128.84.20.12:6789,128.84.20.15:6789,128.84.20.13:6789,128.84.20.14:6789:/volumes/_nogroup/e91d7ccd-9845-4d2a-acc6-d40e572ee796/937df611-6035-47bc-92ed-ad09fb225715"
mountPoint="/home/ubuntu/lulc_input"

Mount to the location:

echo "${cephfsPath} ${mountPoint} ceph name=${accessTo},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0 2" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
mkdir -p ${mountPoint}
sudo mount ${mountPoint}

You might encounter some errors during the mount step. Disregard these errors, and run the following command to test if mount is successful:

df -h ${mountPoint}

If the CephFS share is mounted correctly, the following output is shown:

Filesystem                                                                                                                                                                             Size  Used Avail Use% Mounted on
128.84.20.11:6789,128.84.20.12:6789,128.84.20.15:6789,128.84.20.13:6789,128.84.20.14:6789:/volumes/_nogroup/e91d7ccd-9845-4d2a-acc6-d40e572ee796/937df611-6035-47bc-92ed-ad09fb225715  100G   85G   16G  85% /home/ubuntu/lulc_input
Download Configuration Files

Both WPS and WRF require some configuration files to direct their behavior, and those are downloaded from the I-WRF GitHub repository. Those configuration files are then copied into a folder. These commands perform the necessary operations:

git clone https://github.com/NCAR/i-wrf ~/i-wrf
cp -r ~/i-wrf/use_cases/Land_Use_Land_Cover ~/lulc_configs
chmod 777 ~/lulc_configs/run_full.sh
Using Screen in Linux

This exercise has two options: simulate the Dallas-Fort Worth area over a 3 hour period or over a 36 hour period. In either case, your simulation may run for several hours or for several days. During this time, any disconnects from the instance will interrupt the simulation. For this reason, it’s almost necessary to use the Linux command screen. By using screen, you create and enter a screen session. Within it, you may run commands as if you were in a normal terminal. You can disconnect from the screen session or the instance, and any ongoing process will continue in the background. At any time, SSH back into the instance and connect to the screen session to check the progress. Disconnecting from and connecting to a screen session is called “detaching” and “attaching”. In this exercise, we will only use part of the functionalities of screen. You may see the full documentation of screen at GNU Screen.

To start a screen session with lulc as the session name, enter the following into your terminal:

screen -S lulc

To show all running screen sessions and see if you are attached to any screen sessions, enter the following (if you started a screen session, it displays that you are attached to one):

screen -ls

Inside a screen session, if you want to detach from it, you would need to press a combination of keys:

Ctrl+A, D

To attach to the screen session lulc, enter the following:

screen -r lulc
Set Input, Output, and Configs Paths

Copy and paste the following lines to set up paths of the input, output, and configuration files:

mkdir -p ~/lulc_output
WRF_INPUT=~/lulc_input
WRF_OUTPUT=~/lulc_output
WRF_CONFIGS=~/lulc_configs
Run WPS and WRF Manually

The instructions below will run WPS and WRF manually. It runs WPS and WRF and simulates the weather between 12:00, July 3 2017 to 15:00, July 3 2017, just a window of three hours over the Dallas-Fort Worth area. This simulation will take about 8 to 9 hours total to run. If you would like to try the full simulation (between 12:00, July 3 2017 to 00:00, July 5 2017), read “Start WPS and WRF with a Script (Full Simulation)” below.

You are ready to run the Docker container that will perform the simulation. First, make sure you are in a screen session. The command below starts the container and provides a shell prompt, which places you inside the container as the user “root”. From there, we will run each command one by one:

sudo docker run --shm-size 100G -it \
-v $WRF_INPUT:/home/wrfuser/lulc_input \
-v $WRF_OUTPUT:/home/wrfuser/lulc_output \
-v $WRF_CONFIGS:/home/wrfuser/lulc_configs \
ncar/iwrf:lulc-2024-10-04 bash

The command has numerous arguments and options, which do the following:

  • docker run creates the container if needed and then runs it.

  • --shm-size 100G -it tells the command how much shared memory to use, and to run interactively in the shell.

  • The -v options map folders in your cloud instance to paths within the container.

  • ncar/iwrf:lulc-2024-10-04 is the Docker image to use when creating the container.

Setting Up

Set up the container environment, ensure all required executables are in $PATH, and address memory limits. First, source /etc/bashrc to load the environment, then allow unlimited stack size:

source /etc/bashrc
ulimit -s unlimited

And define some environment variables for input and output paths:

WPS_DIR=/home/wrfuser/WPS
WRF_DIR=/home/wrfuser/WRF
WPS_INPUT=/home/wrfuser/lulc_input/WPS_input
WRF_INPUT=/home/wrfuser/lulc_input/WRF_input
CONFIGS=/home/wrfuser/lulc_configs
OUTPUT=/home/wrfuser/lulc_output
Run WPS

The first half of the instructions is to run WRF Preprocessing Systems (WPS) on geographic data and meteorological data. The WPS software is located at /home/wrfuser/WPS and the geographic data and meteorological data are in /home/wrfuser/lulc_input/WPS_input, as WPS_GEOG and HRRR_0703, respectively.

In WPS, the program geogrid.exe creates terrestrial data from static geographic data and defines the simulation domains. The section &geogrid in the namelist.wps directs geogrid.exe to read domain configuration parameters from WPS_GEOG:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_geogrid.wps ${WPS_DIR}/namelist.wps
ln -fs ${WPS_INPUT}/WPS_GEOG $WPS_DIR
./geogrid.exe

Next, the program ungrib.exe unpacks the meteorological data into WRF intermediate format. Vtable is used to specify which fields to unpack, by linking the Vtable file to ${WPS_DIR}/Vtable. The meteorological data consists of two formats, wrfprs and wrfnat, which are linked and unpacked separately. The &ungrib section in namelist.wps specifies which files to use. Link the files and run ungrib.exe on wrfprs files to generate files with “HRRR_PRS” headers:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_ungrib_prs.wps ${WPS_DIR}/namelist.wps
cp ${CONFIGS}/WPS/Vtable/Vtable.hrrr.modified ${WPS_DIR}/ungrib/Variable_Tables/
ln -sf ${WPS_DIR}/ungrib/Variable_Tables/Vtable.hrrr.modified ${WPS_DIR}/Vtable
./link_grib.csh ${WPS_INPUT}/HRRR_0703/hrrr.*.wrfprs
./ungrib.exe

Link the files and run ungrib.exe on wrfnat files to generate files with “HRRR_NAT” headers using a new namelist containing a different &ungrib section:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_ungrib_nat.wps ${WPS_DIR}/namelist.wps
./link_grib.csh ${WPS_INPUT}/HRRR_0703/hrrr.*.wrfnat
./ungrib.exe

The last step is to call metgrid.exe to interpolate the meteorological data onto the simulation domain, and the outputs of metgrid.exe are used as inputs to WRF. This process is guided by the &metgrid section of namelist.wps:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_metgrid.wps ${WPS_DIR}/namelist.wps
./metgrid.exe
Run WRF

The latter half of the exercise involves running two WRF simulations to investigate the impact of land use and land cover (LULC) on simulated deep convection over different sizes of the Dallas-Fort Worth (DFW) area. The first simulation is a control simulation using data generated from the previous WPS steps. The second simulation is a perturbed simulation with modified data, where the DFW area is expanded to four times its original size.

Control Simulation

The control simulation runs WRF with the outputs generated from the previous WPS steps. Copy the relevant namelist, define environment variables, and link the met_em files from WPS:

cd $WRF_DIR
ln -sf ${WRF_DIR}/run/* ${WRF_DIR}
cp ${CONFIGS}/WRF/namelist/namelist.input $WRF_DIR
cp ${CONFIGS}/WRF/ctl/wrfvar_lulc_*.txt $WRF_DIR
ln -sf ${WPS_DIR}/met_em* $WRF_DIR

The WRF software is located at /home/wrfuser/WRF, which contains two programs, real.exe and wrf.exe. real.exe vertically interpolates the outputs of metgrid.exe and generates boundary and initial conditions: wrfbdy_d01, wrfinput_d01, wrfinput_d02, and wrfinput_d03:

cd $WRF_DIR
./main/real.exe

Create a directory named wrfdata in the WRF directory to store the output from WRF and run WRF simulation with 60 CPU cores. If you had used a different flavor on this instance, adjust the CPU core count to a suitable number:

cd $WRF_DIR
mkdir -p ${WRF_DIR}/wrfdata
mpiexec -n 60 -ppn 60 ./main/wrf.exe

This step will take about 4 hours to run. When it is finished, move the outputs from wrfdata to the output directory:

mv ${WRF_DIR}/wrfdata ${OUTPUT}/ctl
DFW4X Simulation

The perturbed simulation will modify the inputs such that the DFW area is four times its original size. Instead of making modifications on our own, the modified data is provided.

First, remove the files used for the control simulation:

cd $WRF_DIR
rm met_em*
rm wrfbdy_d01
rm wrfinput*

Link the appropriate files for DFW4X simulation:

ln -sf ${WRF_DIR}/run/* $WRF_DIR
ln -sf ${WRF_INPUT}/dfw4x/wrfbdy_d01 $WRF_DIR
ln -sf ${WRF_INPUT}/dfw4x/wrfinput* $WRF_DIR
ln -sf ${WRF_INPUT}/dfw4x/met_em* $WRF_DIR

Create a directory named wrfdata in the WRF directory to store the output from WRF and run WRF simulation with 60 CPU cores. This step will take slightly more than 4 hours to run. If you had used a different flavor on this instance, adjust the CPU core count to a suitable number:

cd $WRF_DIR
mkdir -p ${WRF_DIR}/wrfdata
mpiexec -n 60 -ppn 60 ./main/wrf.exe

When it is finished, move the outputs from wrfdata to the output directory:

mv ${WRF_DIR}/wrfdata ${OUTPUT}/dfw4x

After moving the outputs, you may exit the container by entering exit.

Start WPS and WRF with a Script (Full Simulation)

If you would like to run WPS and WPS for the entire duration, from 12:00, July 3 2017 to 00:00, July 5 2017, you can use the script run_full.sh provided in the ~/lulc_configs directory.

First, make sure you are in a screen session. If you would like to run the entire process in one command, you just have to run the script. If you had used a different flavor than c64.m120 on this instance, adjust the CPU core count to a suitable number in the script (e.g. mpiexec -n 60 -ppn 60 ./main/wrf.exe to mpiexec -n 28 -ppn 28 ./main/wrf.exe for the c28.m224 flavor).

The script runs inside the container, prints lots of status information, and creates output files in the output directory you created. Execute this command to start a container with the image we pulled:

sudo docker run --shm-size 100G -it \
-v $WRF_INPUT:/home/wrfuser/lulc_input \
-v $WRF_OUTPUT:/home/wrfuser/lulc_output \
-v $WRF_CONFIGS:/home/wrfuser/lulc_configs \
ncar/iwrf:lulc-2024-10-04 /home/wrfuser/lulc_configs/run_full.sh

The command has numerous arguments and options, which do the following:

  • docker run creates the container if needed and then runs it.

  • --shm-size 100 -it tells the command how much shared memory to use, and to run interactively in the shell.

  • The -v options map folders in your cloud instance to paths within the container.

  • ncar/iwrf:lulc-2024-10-04 is the Docker image to use when creating the container.

The simulation will take about 4 days to run. When complete, the terminal will become available again. The output files will be in the lulc_output directory in the home directory. See the section below for instructions on how to view the outputs.

View Outputs

To view the outputs in the lulc_output directory, you must get read permission:

sudo chmod -R a+r $WRF_OUTPUT

Use the ls command to list the files in the ctl or dfw4x directory:

ls $WRF_OUTPUT/ctl
ls $WRF_OUTPUT/dfw4x
View Full WRF Output

If you do not have the resource to run the entire simulation but would like to see the results, paste the following commands to access the full output Ceph share:

accessTo="iwrf-lulc-output-read-only"
accessKey="AQCe60lo0kUiJBAAkf9bYacxnfjVM4zcku67Xw=="
cephfsPath="128.84.20.11:6789,128.84.20.12:6789,128.84.20.15:6789,128.84.20.13:6789,128.84.20.14:6789:/volumes/_nogroup/4686628e-540f-4e99-8cd1-9e53dcb9f97d/686fafb3-94a2-4547-b33a-178f1f59ff8f"
mountPoint="/home/ubuntu/lulc_full_output"

mkdir -p /etc/ceph
echo -e "[client.${accessTo}]\n    key = ${accessKey}" | sudo tee /etc/ceph/ceph.client.${accessTo}.keyring

sudo chown root:root /etc/ceph/ceph.client.${accessTo}.keyring
sudo chmod 600 /etc/ceph/ceph.client.${accessTo}.keyring

echo "${cephfsPath} ${mountPoint} ceph name=${accessTo},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0 2" | sudo tee -a /etc/fstab

sudo systemctl daemon-reload

mkdir -p ${mountPoint}
sudo mount ${mountPoint}

df -h ${mountPoint}

The full output should be in /home/ubuntu/lulc_full_output.

6.2.4.3. On Jetstream2

Follow the compute platform instructions for Jetstream2 to secure access to and log in to Jetstream2.

Instructions
Instance Configuration

Make the following choices when creating your instance:

  • When choosing an image as the instance source, if viewing “By Type”, select the “Ubuntu 24.04” image. If viewing “By Image”, choose the “Featured-Ubuntu24” image.

  • Choose the “Flavor” m3.2xl (64 CPUs) to provide a faster simulation run-time.

  • Select a custom disk size of 1000 GB, which is large enough to hold this exercise’s data and results.

  • For “Enable web desktop?”, select Yes.

  • For “Choose an SSH public key”, select None unless you want to use your own SSH key that you uploaded previously.

  • You do not need to set any of the Advanced Options.

Preparing the Environment

With your instance created and running and you logged in to it through SSH, you can now create the run folders, install Docker software and download the data to run the simulation and verification. You will only need to perform these steps once, as they essentially change the contents of the instance’s disk and those changes will remain even after the instance is shelved and unshelved.

The following sections instruct you to issue numerous Linux commands in your shell. If you are not familiar with Linux, you may want to refer to An Introduction to Linux when working through these steps. The commands in each section can be copied using the button in the upper right corner and then pasted into your shell by right-clicking.

Recovering Unresponsive Instance

If your shell ever becomes unresponsive or disconnected from the instance, you can recover from that situation by opening a new Web Desktop (if available) or rebooting the instance. In the Exosphere dashboard page for your instance, in the Actions menu, select “Reboot”. The process takes several minutes, after which the instance status will return to “Ready”.

Get the Docker Image

You must pull the correct versions of the image that will be used in this exercise onto your instance:

sudo docker pull ncar/iwrf:lulc-2024-10-04
Install and Enable CephFS

You need to access the data used in this exercise. In total, the full data are close to 90 GB in size. Usually, such large datasets cannot be shared easily. However, Jetstream2 has a Ceph cluster, a distributed file system that stores the data locally at Jetstream2. Any Linux machine on Jetstream2 can access this data using the following steps.

First, update the package list:

sudo apt update

Install CephFS client:

sudo apt install ceph-common

The CephFS mounting steps are slightly more complicated. When a CephFS share is created, access rules must be set for writing or reading the data. This credential is called a keyring, which consists of an entity name (accessTo) and a key (accessKey). For this exercise, copy and paste the credentials for read-only access:

accessTo="iwrf-lulc-read-only"
accessKey="AQCLixNooPVSGBAASckRTu+xrJeDzaoQQEv6SQ=="

The following commands set up the keyring:

mkdir -p /etc/ceph
echo -e "[client.${accessTo}]\n    key = ${accessKey}" | sudo tee /etc/ceph/ceph.client.${accessTo}.keyring

The keyring file must be only readable to root:

sudo chown root:root /etc/ceph/ceph.client.${accessTo}.keyring
sudo chmod 600 /etc/ceph/ceph.client.${accessTo}.keyring

Choose the mount point for the CephFS share, which will be in the home directory:

cephfsPath="149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/6e81fe46-b69e-4d33-be08-a2580b420b81/6cc28fc1-35f3-41b4-8652-f14555097810"
mountPoint="/home/exouser/lulc_input"

Mount to the location:

echo "${cephfsPath} ${mountPoint} ceph name=${accessTo},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0 2" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
mkdir -p ${mountPoint}
sudo mount ${mountPoint}

Run the following command to test if mount is successful:

df -h ${mountPoint}

If the CephFS share is mounted correctly, the following output is shown:

Filesystem                                                                                                                                                                                       Size  Used Avail Use% Mounted on
149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/6e81fe46-b69e-4d33-be08-a2580b420b81/6cc28fc1-35f3-41b4-8652-f14555097810  100G   85G   16G  85% /home/exouser/lulc_input
Download Configuration Files

Both WPS and WRF require some configuration files to direct their behavior, and those are downloaded from the I-WRF GitHub repository. Those configuration files are then copied into a folder. These commands perform the necessary operations:

git clone https://github.com/NCAR/i-wrf ~/i-wrf
cp -r ~/i-wrf/use_cases/Land_Use_Land_Cover ~/lulc_configs
chmod 777 ~/lulc_configs/run_full.sh
Using Screen in Linux

This exercise has two options: simulate the Dallas-Fort Worth area over a 3 hour period or over a 36 hour period. In either case, your simulation may run for several hours or for several days. During this time, any disconnects from the instance will interrupt the simulation. For this reason, it’s almost necessary to use the Linux command screen. By using screen, you create and enter a screen session. Within it, you may run commands as if you were in a normal terminal. You can disconnect from the screen session or the instance, and any ongoing process will continue in the background. At any time, SSH back into the instance and connect to the screen session to check the progress. Disconnecting from and connecting to a screen session is called “detaching” and “attaching”. In this exercise, we will only use part of the functionalities of screen. You may see the full documentation of screen at GNU Screen.

To start a screen session with lulc as the session name, enter the following into your terminal:

screen -S lulc

To show all running screen sessions and see if you are attached to any screen sessions, enter the following (if you started a screen session, it displays that you are attached to one):

screen -ls

Inside a screen session, if you want to detach from it, you would need to press a combination of keys:

Ctrl+A, D

To attach to the screen session lulc, enter the following:

screen -r lulc
Set Input, Output, and Configs Paths

Copy and paste the following lines to set up paths of the input, output, and configuration files:

mkdir -p ~/lulc_output
WRF_INPUT=~/lulc_input
WRF_OUTPUT=~/lulc_output
WRF_CONFIGS=~/lulc_configs
Run WPS and WRF Manually

The instructions below will run WPS and WRF manually. It runs WPS and WRF and simulates the weather between 12:00, July 3 2017 to 15:00, July 3 2017, just a window of three hours over the Dallas-Fort Worth area. This simulation will take about 8 to 9 hours total to run. If you would like to try the full simulation (between 12:00, July 3 2017 to 00:00, July 5 2017), read “Start WPS and WRF with a Script (Full Simulation)” below.

You are ready to run the Docker container that will perform the simulation. First, make sure you are in a screen session. The command below starts the container and provides a shell prompt, which places you inside the container as the user “root”. From there, we will run each command one by one:

sudo docker run --shm-size 100G -it \
-v $WRF_INPUT:/home/wrfuser/lulc_input \
-v $WRF_OUTPUT:/home/wrfuser/lulc_output \
-v $WRF_CONFIGS:/home/wrfuser/lulc_configs \
ncar/iwrf:lulc-2024-10-04 bash

The command has numerous arguments and options, which do the following:

  • docker run creates the container if needed and then runs it.

  • --shm-size 100G -it tells the command how much shared memory to use, and to run interactively in the shell.

  • The -v options map folders in your cloud instance to paths within the container.

  • ncar/iwrf:lulc-2024-10-04 is the Docker image to use when creating the container.

Setting Up

Set up the container environment, ensure all required executables are in $PATH, and address memory limits. First, source /etc/bashrc to load the environment, then allow unlimited stack size:

source /etc/bashrc
ulimit -s unlimited

And define some environment variables for input and output paths:

WPS_DIR=/home/wrfuser/WPS
WRF_DIR=/home/wrfuser/WRF
WPS_INPUT=/home/wrfuser/lulc_input/WPS_input
WRF_INPUT=/home/wrfuser/lulc_input/WRF_input
CONFIGS=/home/wrfuser/lulc_configs
OUTPUT=/home/wrfuser/lulc_output
Run WPS

The first half of the instructions is to run WRF Preprocessing Systems (WPS) on geographic data and meteorological data. The WPS software is located at /home/wrfuser/WPS and the geographic data and meteorological data are in /home/wrfuser/lulc_input/WPS_input, as WPS_GEOG and HRRR_0703, respectively.

In WPS, the program geogrid.exe creates terrestrial data from static geographic data and defines the simulation domains. The section &geogrid in the namelist.wps directs geogrid.exe to read domain configuration parameters from WPS_GEOG:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_geogrid.wps ${WPS_DIR}/namelist.wps
ln -fs ${WPS_INPUT}/WPS_GEOG $WPS_DIR
./geogrid.exe

Next, the program ungrib.exe unpacks the meteorological data into WRF intermediate format. Vtable is used to specify which fields to unpack, by linking the Vtable file to ${WPS_DIR}/Vtable. The meteorological data consists of two formats, wrfprs and wrfnat, which are linked and unpacked separately. The &ungrib section in namelist.wps specifies which files to use. Link the files and run ungrib.exe on wrfprs files to generate files with “HRRR_PRS” headers:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_ungrib_prs.wps ${WPS_DIR}/namelist.wps
cp ${CONFIGS}/WPS/Vtable/Vtable.hrrr.modified ${WPS_DIR}/ungrib/Variable_Tables/
ln -sf ${WPS_DIR}/ungrib/Variable_Tables/Vtable.hrrr.modified ${WPS_DIR}/Vtable
./link_grib.csh ${WPS_INPUT}/HRRR_0703/hrrr.*.wrfprs
./ungrib.exe

Link the files and run ungrib.exe on wrfnat files to generate files with “HRRR_NAT” headers using a new namelist containing a different &ungrib section:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_ungrib_nat.wps ${WPS_DIR}/namelist.wps
./link_grib.csh ${WPS_INPUT}/HRRR_0703/hrrr.*.wrfnat
./ungrib.exe

The last step is to call metgrid.exe to interpolate the meteorological data onto the simulation domain, and the outputs of metgrid.exe are used as inputs to WRF. This process is guided by the &metgrid section of namelist.wps:

cd $WPS_DIR
cp ${CONFIGS}/WPS/namelist/namelist_metgrid.wps ${WPS_DIR}/namelist.wps
./metgrid.exe
Run WRF

The latter half of the exercise involves running two WRF simulations to investigate the impact of land use and land cover (LULC) on simulated deep convection over different sizes of the Dallas-Fort Worth (DFW) area. The first simulation is a control simulation using data generated from the previous WPS steps. The second simulation is a perturbed simulation with modified data, where the DFW area is expanded to four times its original size.

Control Simulation

The control simulation runs WRF with the outputs generated from the previous WPS steps. Copy the relevant namelist, define environment variables, and link the met_em files from WPS:

cd $WRF_DIR
ln -sf ${WRF_DIR}/run/* ${WRF_DIR}
cp ${CONFIGS}/WRF/namelist/namelist.input $WRF_DIR
cp ${CONFIGS}/WRF/ctl/wrfvar_lulc_*.txt $WRF_DIR
ln -sf ${WPS_DIR}/met_em* $WRF_DIR

The WRF software is located at /home/wrfuser/WRF, which contains two programs, real.exe and wrf.exe. real.exe vertically interpolates the outputs of metgrid.exe and generates boundary and initial conditions: wrfbdy_d01, wrfinput_d01, wrfinput_d02, and wrfinput_d03:

cd $WRF_DIR
./main/real.exe

Create a directory named wrfdata in the WRF directory to store the output from WRF and run WRF simulation with 60 CPU cores. If you had used a different flavor on this instance, adjust the CPU core count to a suitable number:

cd $WRF_DIR
mkdir -p ${WRF_DIR}/wrfdata
mpiexec -n 60 -ppn 60 ./main/wrf.exe

This step will take about 4 hours to run. When it is finished, move the outputs from wrfdata to the output directory:

mv ${WRF_DIR}/wrfdata ${OUTPUT}/ctl
DFW4X Simulation

The perturbed simulation will modify the inputs such that the DFW area is four times its original size. Instead of making modifications on our own, the modified data is provided.

First, remove the files used for the control simulation:

cd $WRF_DIR
rm met_em*
rm wrfbdy_d01
rm wrfinput*

Link the appropriate files for DFW4X simulation:

ln -sf ${WRF_DIR}/run/* $WRF_DIR
ln -sf ${WRF_INPUT}/dfw4x/wrfbdy_d01 $WRF_DIR
ln -sf ${WRF_INPUT}/dfw4x/wrfinput* $WRF_DIR
ln -sf ${WRF_INPUT}/dfw4x/met_em* $WRF_DIR

Create a directory named wrfdata in the WRF directory to store the output from WRF and run WRF simulation with 60 CPU cores. This step will take slightly more than 4 hours to run. If you had used a different flavor on this instance, adjust the CPU core count to a suitable number:

cd $WRF_DIR
mkdir -p ${WRF_DIR}/wrfdata
mpiexec -n 60 -ppn 60 ./main/wrf.exe

When it is finished, move the outputs from wrfdata to the output directory:

mv ${WRF_DIR}/wrfdata ${OUTPUT}/dfw4x

After moving the outputs, you may exit the container by entering exit.

Start WPS and WRF with a Script (Full Simulation)

If you would like to run WPS and WPS for the entire duration, from 12:00, July 3 2017 to 00:00, July 5 2017, you can use the script run_full.sh provided in the ~/lulc_configs directory.

First, make sure you are in a screen session. If you would like to run the entire process in one command, you just have to run the script. If you had used a different flavor than c64.m120 on this instance, adjust the CPU core count to a suitable number in the script (e.g. mpiexec -n 60 -ppn 60 ./main/wrf.exe to mpiexec -n 28 -ppn 28 ./main/wrf.exe for the c28.m224 flavor).

The script runs inside the container, prints lots of status information, and creates output files in the output directory you created. Execute this command to start a container with the image we pulled:

sudo docker run --shm-size 100G -it \
-v $WRF_INPUT:/home/wrfuser/lulc_input \
-v $WRF_OUTPUT:/home/wrfuser/lulc_output \
-v $WRF_CONFIGS:/home/wrfuser/lulc_configs \
ncar/iwrf:lulc-2024-10-04 /home/wrfuser/lulc_configs/run_full.sh

The command has numerous arguments and options, which do the following:

  • docker run creates the container if needed and then runs it.

  • --shm-size 100 -it tells the command how much shared memory to use, and to run interactively in the shell.

  • The -v options map folders in your cloud instance to paths within the container.

  • ncar/iwrf:lulc-2024-10-04 is the Docker image to use when creating the container.

The simulation will take about 4 days to run. When complete, the terminal will become available again. The output files will be in the lulc_output directory in the home directory. See the section below for instructions on how to view the outputs.

View Outputs

To view the outputs in the lulc_output directory, you must get read permission:

sudo chmod -R a+r $WRF_OUTPUT

Use the ls command to list the files in the ctl or dfw4x directory:

ls $WRF_OUTPUT/ctl
ls $WRF_OUTPUT/dfw4x
View Full WRF Output

If you do not have the resources to run the entire simulation but would like to see the results, paste the following commands to access the full output Ceph share:

accessTo="iwrf-lulc-output-read-only"
accessKey="AQCv7EloaSlPERAAlXaru8qHfl6d+/3u+yx36g=="
cephfsPath="149.165.158.38:6789,149.165.158.22:6789,149.165.158.54:6789,149.165.158.70:6789,149.165.158.86:6789:/volumes/_nogroup/83cfc802-c288-4727-991d-e33da52b36e4/4fd211a1-c611-4948-8444-bb4ec166b7a7"
mountPoint="/home/exouser/lulc_full_output"

mkdir -p /etc/ceph
echo -e "[client.${accessTo}]\n    key = ${accessKey}" | sudo tee /etc/ceph/ceph.client.${accessTo}.keyring

sudo chown root:root /etc/ceph/ceph.client.${accessTo}.keyring
sudo chmod 600 /etc/ceph/ceph.client.${accessTo}.keyring

echo "${cephfsPath} ${mountPoint} ceph name=${accessTo},x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,rw 0 2" | sudo tee -a /etc/fstab

sudo systemctl daemon-reload

mkdir -p ${mountPoint}
sudo mount ${mountPoint}

df -h ${mountPoint}

The full output should be in /home/exouser/lulc_full_output.