6.1. Hurricane Matthew

6.1.1. Scientific Objective

In October 2016, Hurricane Matthew made landfall in Haiti, Cuba, and then the Bahamas before paralleling the eastern U.S. coast, causing widespread damage. For more information, see NHC Report - Hurricane Matthew.

6.1.2. Version Added

0.1

6.1.3. Datasets

Update section content to describe the datasets used to run WRF and METplus.

6.1.4. Running This I-WRF Use Case

With your instance created and running and you logged in to it, you can now install the necessary software and download the data to run the simulation.

Instructions a provided below for running the Hurricane Matthew use case for each Compute Platform on which it has been tested.

6.1.4.1. On NSF NCAR HPC

Follow the compute platform instructions for NSF NCAR to secure access to and log in to NSF NCAR HPC.

These instructions are currently limited to running the METplus verification software and assume that WRF output is already available in a local directory.

Instructions

Log into Derecho

From a terminal window, log into Derecho HPC using your login credentials, passing the -Y argument to enable X window forwarding be able to view images:

ssh -Y {username}@derecho.hpc.ucar.edu

Load Required Modules

NCAR HPC systems use environment modules to manage software. Load the Apptainer module which provides the containerization software needed to run WRF and METplus:

module load charliecloud apptainer gcc cuda ncarcompilers

Define Working Directory

Set an environment variable called WORKING_DIR to a directory to store all of the input and output files for the use case. Change this path to your preference:

WORKING_DIR=${SCRATCH}/iwrf_work

Define Environment Variables

We will be using some environment variables throughout this exercise to make sure that we refer to the same resource names and file paths wherever they are used. Copy and paste the definitions below into your shell to define the variables before proceeding:

WRF_IMAGE=ncar/iwrf:latest
METPLUS_IMAGE=ncar/iwrf-metplus:latest
WRF_TOP_DIR=${WORKING_DIR}/wrf
WRF_DATE_DIR=${WRF_TOP_DIR}/20161006_00
METPLUS_DIR=${WORKING_DIR}/metplus_out
WRF_CONFIG_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/WRF
METPLUS_CONFIG_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/METplus
PLOT_SCRIPT_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/Visualization
OBS_DATA_VOL=matthew-input-obs

Any time you open a new shell on your instance, you will need to perform this action to redefine the variables before executing the commands that follow.

Set Apptainer Temp Directory

Set the $APPTAINER_TMPDIR environment variable to $TMPDIR to ensure that the correct temp directory is used by Apptainer. $TMPDIR is set automatically upon log in to the NCAR HPC:

export APPTAINER_TMPDIR=${TMPDIR}

Create Environment File

The environment variables set in the above instructions will not be available inside the compute node, so create an environment file that can be sourced.

echo export WORKING_DIR=${WORKING_DIR} > ${WORKING_DIR}/env_matthew.sh
echo export WRF_TOP_DIR=${WRF_TOP_DIR} >> ${WORKING_DIR}/env_matthew.sh
echo export WRF_DATE_DIR=${WRF_DATE_DIR} >> ${WORKING_DIR}/env_matthew.sh
echo export METPLUS_CONFIG_DIR=${METPLUS_CONFIG_DIR} >> ${WORKING_DIR}/env_matthew.sh
echo export PLOT_SCRIPT_DIR=${PLOT_SCRIPT_DIR} >> ${WORKING_DIR}/env_matthew.sh
echo export METPLUS_DIR=${METPLUS_DIR} >> ${WORKING_DIR}/env_matthew.sh
echo export OBS_DATA_VOL=${OBS_DATA_VOL} >> ${WORKING_DIR}/env_matthew.sh
echo export APPTAINER_TMPDIR=${APPTAINER_TMPDIR} >> ${WORKING_DIR}/env_matthew.sh

Create Working Directories

The METplus verification process requires specific directory structures to organize input data, configuration files, and output results. Create the main working directory in your scratch space:

mkdir -p ${WORKING_DIR}

Create a directory to store the METplus verification output:

mkdir -p ${METPLUS_DIR}

Create a directory to store the WRF inputs and outputs:

mkdir -p ${WRF_DATE_DIR}

Create a directory for temporary Apptainer files. The $TMPDIR variable is automatically set on NCAR HPC systems to an appropriate temporary storage location:

mkdir -p ${APPTAINER_TMPDIR}

Download Configuration Files

Both WRF and METplus require some configuration files to direct their behavior, and those are downloaded from the I-WRF GitHub repository. Some of those configuration files are then copied into the run folders. These commands perform the necessary operations:

git clone https://github.com/NCAR/i-wrf ${WORKING_DIR}/i-wrf
cp ${WRF_CONFIG_DIR}/namelist.* ${WRF_DATE_DIR}
cp ${WRF_CONFIG_DIR}/vars_io.txt ${WRF_DATE_DIR}
cp ${WRF_CONFIG_DIR}/run.sh ${WRF_DATE_DIR}

Pull The Docker Image As A Singularity Image Files (.sif)

Pull the WRF software, METplus software image and observation data from the container registry to your HPC system’s storage. This will create a files ending in .sif in the ${WORKING_DIR} directory:

apptainer pull ${WORKING_DIR}/iwrf_latest.sif docker://${WRF_IMAGE}
apptainer pull ${WORKING_DIR}/iwrf-metplus.sif docker://${METPLUS_IMAGE}
apptainer pull ${WORKING_DIR}/data-${OBS_DATA_VOL}.sif oras://registry-1.docker.io/ncar/iwrf-data:${OBS_DATA_VOL}.apptainer

Note

If an error is displayed when attempting to pull the images, creating a DockerHub account and authenticating through apptainer may be necessary:

apptainer remote login --username {USERNAME} docker://docker.io

where {USERNAME} is your DockerHub username.

Check that there are files named iwrf_latest.sif, iwrf-metplus.sif, and data-matthew-input-obs.sif in the ${WORKING_DIR} directory to confirm that the images were pulled successfully:

ls ${WORKING_DIR}

Download Data for WRF

To run WRF on the Hurricane Matthew data set, you need to have several data sets to support the computation. The commands in these sections download archive files containing that data, then uncompress the archives into folders. The geographic data is large and takes several minutes to acquire, while the other two data sets are smaller and are downloaded directly into the WRF run folder, rather than the user’s home directory.

Get the geographic data representing the terrain in the area of the simulation:

wget https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_high_res_mandatory.tar.gz -O ${WORKING_DIR}/geog_high_res_mandatory.tar.gz
tar -xvzf ${WORKING_DIR}/geog_high_res_mandatory.tar.gz -C ${WORKING_DIR}
rm -f ${WORKING_DIR}/geog_high_res_mandatory.tar.gz

Get the case study data (GRIB2 files):

wget https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_1deg.tar.gz -O ${WRF_DATE_DIR}/matthew_1deg.tar.gz
tar -xvzf ${WRF_DATE_DIR}/matthew_1deg.tar.gz -C ${WRF_DATE_DIR}
rm -f ${WRF_DATE_DIR}/matthew_1deg.tar.gz

Get the SST (Sea Surface Temperature) data:

wget https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_sst.tar.gz -O ${WRF_DATE_DIR}/matthew_sst.tar.gz
tar -xzvf ${WRF_DATE_DIR}/matthew_sst.tar.gz -C ${WRF_DATE_DIR}
rm -f ${WRF_DATE_DIR}/matthew_sst.tar.gz

Gain Interactive Access To A Compute Node

Tasks that are resource intensive should not be run on the login nodes, so a compute node should be accessed through Derecho’s job queue before starting the container. Change directory to the $WORKING_DIR, then run the following command to submit an interactive job in the develop queue.:

cd ${WORKING_DIR}
qsub -l select=1:ncpus=8:mpiprocs=8 -A <account_id> -l walltime=01:00:00 -I -q develop

The above command should be modified with your specific account ID for charging computing time. The number of processors needed can also be specified here. The full documentation for the qsub command can be found on Adaptive Computing’s website.

This may take a few minutes. When it completes, the terminal prompt will change to something like user@decNNNN. Once you see this new prompt, your interactive job is active and you can move on to the next step in the instructions.

Source Environment File

Source the environment file that was created earlier:

source env_matthew.sh

Configure Container Data Bindings for WRF

Set environment variable to bind directories to the containers (note: this can also be accomplished by passing the value on the command line using the –bind argument)

• Terrestrial Data:

  Data required by Geogrid

  • Local: ${WORKING_DIR}

  • Container: /home/wrfuser/terrestrial_data

• WRF:

  WRF configuration files and run script

  • Local: ${WRF_DATE_DIR}

  • Container: /tmp/hurricane_matthew

• Job Queue Information:

  Make the job queue information available to the container, which provides the available hosts and number of compute cores. This information is required by the mpirun command in the script.

  • Local: /var/spool/pbs

  • Container: /var/spool/pbs

• Apptainer temp directory

  • Local: ${APPTAINER_TMPDIR}

  • Container: ${APPTAINER_TMPDIR}

export APPTAINER_BIND="${WORKING_DIR}:/home/wrfuser/terrestrial_data,${WRF_DATE_DIR}:/tmp/hurricane_matthew,/var/spool/pbs:/var/spool/pbs,${APPTAINER_TMPDIR}:${APPTAINER_TMPDIR}"

Running WRF In The Container

Once the interactive job has started, the run script can be called inside the container to run WRF:

module load charliecloud apptainer gcc cuda ncarcompilers
apptainer exec ${WORKING_DIR}/iwrf_latest.sif /tmp/hurricane_matthew/run.sh

Once WRF begins to execute, there will not be any log updates for a few minutes until WRF completes and a new terminal prompt appears. After the script finishes running the WRF output data will be in ${WRF_DATE_DIR}/wrfout_d01*. If these files exist, it indicates that the WRF run was successful. If these files do not appear, you can check the ${WRF_DATE_DIR}/rsl.error.* files for errors.

ls ${WRF_DATE_DIR}/wrfout_d01*

Configure Container Data Bindings for METplus

Set environment variable to bind directories to the containers (note: this can also be accomplished by passing the value on the command line using the –bind argument)

• Input data directories for WRF, raob, and metar input data

  • WRF:

    • Local: ${WRF_TOP_DIR}

    • Container: /data/input/wrf

  • RAOB:

    • Local: From data-matthew-input-obs.sif

    • Container: /data/input/obs/raob

  • METAR:

    • Local: From data-matthew-input-obs.sif

    • Container: /data/input/obs/metar

  • Config directory containing METplus use case configuration file

    • Local: ${METPLUS_CONFIG_DIR}

    • Container: /config

  • Plot script directory containing WRF plotting scripts

    • Local: ${PLOT_SCRIPT_DIR}

    • Container: /plot_scripts

  • Output directory to write output

    • Local: ${METPLUS_DIR}

    • Container: /data/output

• Apptainer temp directory

  • Local: ${APPTAINER_TMPDIR}

  • Container: ${APPTAINER_TMPDIR}

export APPTAINER_BIND="${WORKING_DIR}/data-${OBS_DATA_VOL}.sif:/data/input/obs:image-src=/,${METPLUS_CONFIG_DIR}:/config,${WRF_TOP_DIR}:/data/input/wrf,${METPLUS_DIR}:/data/output,${PLOT_SCRIPT_DIR}:/plot_scripts,${APPTAINER_TMPDIR}:${APPTAINER_TMPDIR}"

Run METplus

Execute the run_metplus.py command inside the container to run the use case:

apptainer exec ${WORKING_DIR}/iwrf-metplus.sif /metplus/METplus/ush/run_metplus.py /config/PointStat_matthew.conf

Check that the output data was created locally:

ls -1  ${WORKING_DIR}/metplus_out/point_stat

Exit the Interactive Compute Node

Be sure to run exit when you are done to stop the compute node:

exit

Examine the Plots

The images generated by this use case can be viewed using the display command if X-window forwarding has been enabled.

MET line plot:

display ${WORKING_DIR}/metplus_out/met_plot/line_T2_RMSE_BCMSE.png

A number of plots of the WRF output can also be found under ${WORKING_DIR}/metplus_out/wrf/20161006_00/plots. For example:

display ${WORKING_DIR}/metplus_out/wrf/20161006_00/plots/map_wrf_d01_RAIN_20161008_0000.png

6.1.4.2. On Jetstream2

Follow the compute platform instructions for Jetstream2 to secure access to and log in to a Jetstream2 instance.

The following sections instruct you to issue numerous Linux commands in your shell. If you are not familiar with Linux, you may want to want to refer to An Introduction to Linux when working through these steps. The commands in each section can be copied using the button in the upper right corner and then pasted into your web shell by right-clicking.

Instructions

Instance Configuration

Make the following choices when creating your instance:

• When choosing an image as the instance source, if viewing “By Type”, select the “Ubuntu 22.04” image. If viewing “By Image”, choose the “Featured-Ubuntu22” image.

• Choose the “Flavor” m3.quad (4 CPUs) to provide a faster simulation run-time.

• Select a custom disk size of 100 GB, which is large enough to hold this exercise’s data and results.

• For “Enable web desktop?”, select Yes.

• For “Choose an SSH public key”, select None unless you want to use your own SSH key that you uploaded previously.

• You do not need to set any of the Advanced Options.

Define Working Directory

Set an environment variable called WORKING_DIR to a directory to store all of the input and output files for the use case. Change this path to your preference:

WORKING_DIR=/home/exouser

Define Environment Variables

We will be using some environment variables throughout this exercise to make sure that we refer to the same resource names and file paths wherever they are used. Copy and paste the definitions below into your shell to define the variables before proceeding:

WRF_IMAGE=ncar/iwrf:latest
METPLUS_IMAGE=ncar/iwrf-metplus:latest
WRF_TOP_DIR=${WORKING_DIR}/wrf
WRF_DATE_DIR=${WRF_TOP_DIR}/20161006_00
METPLUS_DIR=${WORKING_DIR}/metplus_out
WRF_CONFIG_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/WRF
METPLUS_CONFIG_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/METplus
PLOT_SCRIPT_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/Visualization
OBS_DATA_VOL=matthew-input-obs

Any time you open a new shell on your instance, you will need to perform this action to redefine the variables before executing the commands that follow.

Create the WRF and METplus Run Folders

The simulation is performed using a script that expects to run in a folder where it can create result files. The first command below creates a folder (named “wrf”) under the user’s home directory, and a sub-folder within “wrf” to hold the output of this simulation. The subfolder is named “20161006_00”, which is the beginning date and time of the simulation. Similarly, a run folder named “metplus_out” must be created for the METplus process to use:

mkdir -p ${WRF_DATE_DIR}
mkdir -p ${METPLUS_DIR}

Download Configuration Files

Both WRF and METplus require some configuration files to direct their behavior, and those are downloaded from the I-WRF GitHub repository. Some of those configuration files are then copied into the run folders. These commands perform the necessary operations:

git clone https://github.com/NCAR/i-wrf ${WORKING_DIR}/i-wrf
cp ${WRF_CONFIG_DIR}/namelist.* ${WRF_DATE_DIR}
cp ${WRF_CONFIG_DIR}/vars_io.txt ${WRF_DATE_DIR}
cp ${WRF_CONFIG_DIR}/run.sh ${WRF_DATE_DIR}

Pull Docker Objects

As mentioned above, the WRF and METplus software are provided as Docker images that will run as a “container” on your cloud instance. To run a Docker container, the Docker Engine must be installed on your instance. You can then “pull” (download) the WRF and METplus images that will be run as containers. The Ubuntu instance you created already has the Docker Engine installed and running.

To get the WRF and METplus Docker Images and the observed weather data, you must pull the correct versions of the WRF and METplus images onto your instance:

docker pull ${WRF_IMAGE}
docker pull ${METPLUS_IMAGE}

METplus is run to perform verification of the results of the WRF simulation against observations gathered during Hurricane Matthew. We download that observation data by pulling a Docker volume that holds it, and then referencing that volume when we run the METplus Docker container. The commands to pull and create the volume are:

docker pull ncar/iwrf-data:${OBS_DATA_VOL}.docker
docker create --name ${OBS_DATA_VOL} ncar/iwrf-data:${OBS_DATA_VOL}.docker

Download Data for WRF

To run WRF on the Hurricane Matthew data set, you need to have several data sets to support the computation. The commands in these sections download archive files containing that data, then uncompress the archives into folders. The geographic data is large and takes several minutes to acquire, while the other two data sets are smaller and are downloaded directly into the WRF run folder, rather than the user’s home directory.

Get the geographic data representing the terrain in the area of the simulation:

wget https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_high_res_mandatory.tar.gz -O ${WORKING_DIR}/geog_high_res_mandatory.tar.gz
tar -xvzf ${WORKING_DIR}/geog_high_res_mandatory.tar.gz -C ${WORKING_DIR}
rm -f ${WORKING_DIR}/geog_high_res_mandatory.tar.gz

Get the case study data (GRIB2 files):

wget https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_1deg.tar.gz -O ${WRF_DATE_DIR}/matthew_1deg.tar.gz
tar -xvzf ${WRF_DATE_DIR}/matthew_1deg.tar.gz -C ${WRF_DATE_DIR}
rm -f ${WRF_DATE_DIR}/matthew_1deg.tar.gz

Get the SST (Sea Surface Temperature) data:

wget https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_sst.tar.gz -O ${WRF_DATE_DIR}/matthew_sst.tar.gz
tar -xzvf ${WRF_DATE_DIR}/matthew_sst.tar.gz -C ${WRF_DATE_DIR}
rm -f ${WRF_DATE_DIR}/matthew_sst.tar.gz

Run WRF

With everything in place, you are now ready to run the Docker container that will perform the simulation. The downloaded script runs inside the container, prints lots of status information, and creates output files in the run folder you created. Execute this command to run the simulation in your shell:

docker run --shm-size 14G -it \
  -v ${WORKING_DIR}:/home/wrfuser/terrestrial_data \
  -v ${WRF_DATE_DIR}:/tmp/hurricane_matthew \
  ${WRF_IMAGE} /tmp/hurricane_matthew/run.sh

The command has numerous arguments and options, which do the following:

• docker run creates the container if needed and then runs it.

• --shm-size 14G -it tells the command how much shared memory to use, and to run interactively in the shell.

• The -v options map folders in your cloud instance to paths within the container.

• ncar/iwrf:latest is the Docker image to use when creating the container.

• /tmp/hurricane_matthew/run.sh is the location within the container of the script that it runs.

The simulation initially prints lots of information while initializing things, then settles in to the computation. The provided configuration simulates 48 hours of weather and takes about 12 minutes to finish on an m3.quad Jetstream2 instance. Once completed, you can view the end of an output file to confirm that it succeeded:

tail ${WRF_DATE_DIR}/rsl.out.0000

The output should look something like this:

Timing for main: time 2016-10-07_23:50:00 on domain 1: 0.25548 elapsed seconds
Timing for main: time 2016-10-07_23:52:30 on domain 1: 0.25495 elapsed seconds
Timing for main: time 2016-10-07_23:55:00 on domain 1: 0.25066 elapsed seconds
Timing for main: time 2016-10-07_23:57:30 on domain 1: 0.25231 elapsed seconds
Timing for main: time 2016-10-08_00:00:00 on domain 1: 0.25795 elapsed seconds
Timing for Writing wrfout_d01_2016-10-08_00:00:00 for domain 1: 0.68666 elapsed seconds
Timing for Writing wrfout_zlev_d01_2016-10-08_00:00:00 for domain 1: 0.47411 elapsed seconds
Timing for Writing wrfout_plev_d01_2016-10-08_00:00:00 for domain 1: 0.47619 elapsed seconds
Timing for Writing restart for domain 1: 1.54598 elapsed seconds
d01 2016-10-08_00:00:00 wrf: SUCCESS COMPLETE WRF

Run METplus

After the WRF simulation has finished, you can run the METplus verification to compare the simulated results to the actual weather observations during the hurricane and generate plots of the simulation. This process takes about nine minutes to complete. We use command line options to tell the METplus container several things, including where the observed data is located, where the METplus configuration can be found, where the plotting scripts can be found, where the WRF output data is located, and where it should create its output files:

docker run --rm -it \
  --volumes-from ${OBS_DATA_VOL} \
  -v ${METPLUS_CONFIG_DIR}:/config \
  -v ${PLOT_SCRIPT_DIR}:/plot_scripts \
  -v ${WRF_TOP_DIR}:/data/input/wrf \
  -v ${METPLUS_DIR}:/data/output ${METPLUS_IMAGE} \
  /metplus/METplus/ush/run_metplus.py /config/PointStat_matthew.conf

Progress information is displayed while the verification is performed. WARNING log messages are expected because observations files are not available for every valid time and METplus is configured to allow some missing inputs. An ERROR log message indicates that something went wrong. METplus first converts the observation data files to a format that the MET tools can read using the MADIS2NC wrapper. Point-Stat is run to generate statistics comparing METAR observations to surface-level model fields and RAOB observations to “upper air” fields. METplus will print its completion status when the processing finishes.

The results of the METplus verification can be found in ${WORKING_DIR}/metplus_out/point_stat. These files contain tabular output that can be viewed in a text editor. Turn off word wrapping for better viewing. Refer to the MET User’s Guide for more information about the Point-Stat output.

View the Plotted Simulation Results

The METplus container also plots the results of the simulation, outputting them as PNG images. To view these images:

• Find the desktop shortcut “Files” on the left side of the desktop and click it to open a file browser.

• Double-click on the following folders in order: metplus_out, wrf, 20161006_00, then plots.

• Double-click on the first image in the folder, which opens an image viewing application.

• Click the Maximize button in the upper right to increase the viewer to full size.

• Click the button in the middle of the right side of the image to advance to the next image.

• Image legends are shown at the bottom and timeframes are shown in the upper right.

• Each of the six plot sequences contains 16 or 17 images.

When you are finished running simulations and viewing their results, you can close the web browser tab containing your Web Desktop. Then, return to the Exosphere dashboard to manage your instance so it does not incur further charges.

Refer back to the Managing a Jetstream2 Instance section of the Jetstream2 instructions to avoid unneccessary computing costs.

6.1.4.3. On Red Cloud

Follow the compute platform instructions for Red Cloud to secure access to and log in to a Red Cloud instance.

The following sections instruct you to issue numerous Linux commands in your shell. If you are not familiar with Linux, you may want to want to refer to An Introduction to Linux when working through these steps. The commands in each section can be copied using the button in the upper right corner and then pasted into your web shell by right-clicking.

Instructions

Instance Configuration

Make the following choices when creating your instance:

• When choosing an image as the instance source

  • Select Boot from Source is “Image”

  • Volume Size (GB) is 100

  • Delete Volume on Instance Delete is “Yes”

  • Select the “ubuntu-22.04-LTS” image

• In Flavor, choose the “Flavor” c4.m32 (4 Virtual CPUs) to provide a faster simulation run-time.

• In Network, select “public”.

• In Security Groups, select “campus-only-ssh” or the security group you created.

• In Key Pair, select the SSH public key that you created or uploaded previously.

Define Working Directory

Set an environment variable called WORKING_DIR to a directory to store all of the input and output files for the use case. Change this path to your preference:

WORKING_DIR=/home/ubuntu

Define Environment Variables

We will be using some environment variables throughout this exercise to make sure that we refer to the same resource names and file paths wherever they are used. Copy and paste the definitions below into your shell to define the variables before proceeding:

WRF_IMAGE=ncar/iwrf:latest
METPLUS_IMAGE=ncar/iwrf-metplus:latest
WRF_TOP_DIR=${WORKING_DIR}/wrf
WRF_DATE_DIR=${WRF_TOP_DIR}/20161006_00
METPLUS_DIR=${WORKING_DIR}/metplus_out
WRF_CONFIG_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/WRF
METPLUS_CONFIG_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/METplus
PLOT_SCRIPT_DIR=${WORKING_DIR}/i-wrf/use_cases/Hurricane_Matthew/Visualization
OBS_DATA_VOL=matthew-input-obs

Any time you open a new shell on your instance, you will need to perform this action to redefine the variables before executing the commands that follow.

Create the WRF and METplus Run Folders

The simulation is performed using a script that expects to run in a folder where it can create result files. The first command below creates a folder (named “wrf”) under the user’s home directory, and a sub-folder within “wrf” to hold the output of this simulation. The subfolder is named “20161006_00”, which is the beginning date and time of the simulation. Similarly, a run folder named “metplus_out” must be created for the METplus process to use:

mkdir -p ${WRF_DATE_DIR}
mkdir -p ${METPLUS_DIR}

Download Configuration Files

Both WRF and METplus require some configuration files to direct their behavior, and those are downloaded from the I-WRF GitHub repository. Some of those configuration files are then copied into the run folders. These commands perform the necessary operations:

git clone https://github.com/NCAR/i-wrf ${WORKING_DIR}/i-wrf
cp ${WRF_CONFIG_DIR}/namelist.* ${WRF_DATE_DIR}
cp ${WRF_CONFIG_DIR}/vars_io.txt ${WRF_DATE_DIR}
cp ${WRF_CONFIG_DIR}/run.sh ${WRF_DATE_DIR}

Get the WRF and METplus Docker Images and the Observed Weather Data

Once you have confirmed Docker is installed, you must pull the correct versions of the WRF and METplus images onto your instance:

sudo docker pull ${WRF_IMAGE}
sudo docker pull ${METPLUS_IMAGE}

METplus is run to perform verification of the results of the WRF simulation using observations gathered during Hurricane Matthew. We download that data by pulling a Docker volume that holds it, and then referencing that volume when we run the METplus Docker container. The commands to pull and create the volume are:

sudo docker pull ncar/iwrf-data:${OBS_DATA_VOL}.docker
sudo docker create --name ${OBS_DATA_VOL} ncar/iwrf-data:${OBS_DATA_VOL}.docker

Download Data for WRF

To run WRF on the Hurricane Matthew data set, you need to have several data sets to support the computation. The commands in these sections download archive files containing that data, then uncompress the archives into folders. The geographic data is large and takes several minutes to acquire, while the other two data sets are smaller and are downloaded directly into the WRF run folder, rather than the user’s home directory.

Get the geographic data representing the terrain in the area of the simulation:

wget https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_high_res_mandatory.tar.gz -O ${WORKING_DIR}/geog_high_res_mandatory.tar.gz
tar -xvzf ${WORKING_DIR}/geog_high_res_mandatory.tar.gz -C ${WORKING_DIR}
rm -f ${WORKING_DIR}/geog_high_res_mandatory.tar.gz

Get the case study data (GRIB2 files):

wget https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_1deg.tar.gz -O ${WRF_DATE_DIR}/matthew_1deg.tar.gz
tar -xvzf ${WRF_DATE_DIR}/matthew_1deg.tar.gz -C ${WRF_DATE_DIR}
rm -f ${WRF_DATE_DIR}/matthew_1deg.tar.gz

Get the SST (Sea Surface Temperature) data:

wget https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_sst.tar.gz -O ${WRF_DATE_DIR}/matthew_sst.tar.gz
tar -xzvf ${WRF_DATE_DIR}/matthew_sst.tar.gz -C ${WRF_DATE_DIR}
rm -f ${WRF_DATE_DIR}/matthew_sst.tar.gz

Run WRF

With everything in place, you are now ready to run the Docker container that will perform the simulation. The downloaded script runs inside the container, prints lots of status information, and creates output files in the run folder you created. Execute this command to run the simulation in your shell:

sudo docker run --shm-size 14G -it \
  -v ${WORKING_DIR}:/home/wrfuser/terrestrial_data \
  -v ${WRF_DATE_DIR}:/tmp/hurricane_matthew \
  ${WRF_IMAGE} /tmp/hurricane_matthew/run.sh

The command has numerous arguments and options, which do the following:

• docker run creates the container if needed and then runs it.

• --shm-size 14G -it tells the command how much shared memory to use, and to run interactively in the shell.

• The -v options map folders in your cloud instance to paths within the container.

• ncar/iwrf:latest is the Docker image to use when creating the container.

• /tmp/hurricane_matthew/run.sh is the location within the container of the script that it runs.

The simulation initially prints lots of information while initializing things, then settles in to the computation. The provided configuration simulates 48 hours of weather and takes about 26 minutes to finish on a c4.m32 Red Cloud instance. Once completed, you can view the end of an output file to confirm that it succeeded:

tail ${WRF_DATE_DIR}/rsl.out.0000

The output should look something like this:

Timing for main: time 2016-10-07_23:50:00 on domain 1: 0.25548 elapsed seconds
Timing for main: time 2016-10-07_23:52:30 on domain 1: 0.25495 elapsed seconds
Timing for main: time 2016-10-07_23:55:00 on domain 1: 0.25066 elapsed seconds
Timing for main: time 2016-10-07_23:57:30 on domain 1: 0.25231 elapsed seconds
Timing for main: time 2016-10-08_00:00:00 on domain 1: 0.25795 elapsed seconds
Timing for Writing wrfout_d01_2016-10-08_00:00:00 for domain 1: 0.68666 elapsed seconds
Timing for Writing wrfout_zlev_d01_2016-10-08_00:00:00 for domain 1: 0.47411 elapsed seconds
Timing for Writing wrfout_plev_d01_2016-10-08_00:00:00 for domain 1: 0.47619 elapsed seconds
Timing for Writing restart for domain 1: 1.54598 elapsed seconds
d01 2016-10-08_00:00:00 wrf: SUCCESS COMPLETE WRF

Run METplus

After the WRF simulation has finished, you can run the METplus verification to compare the simulated results to the actual weather observations during the hurricane. The verification takes about five minutes to complete. We use command line options to tell the METplus container several things, including where the observed data is located, where the METplus configuration can be found, where the plotting scripts can be found, where the WRF output data is located, and where it should create its output files:

sudo docker run --rm -it \
  --volumes-from ${OBS_DATA_VOL} \
  -v ${METPLUS_CONFIG_DIR}:/config \
  -v ${PLOT_SCRIPT_DIR}:/plot_scripts \
  -v ${WRF_TOP_DIR}:/data/input/wrf \
  -v ${METPLUS_DIR}:/data/output ${METPLUS_IMAGE} \
  /metplus/METplus/ush/run_metplus.py /config/PointStat_matthew.conf

Progress information is displayed while the verification is performed. WARNING log messages are expected because observations files are not available for every valid time and METplus is configured to allow some missing inputs. An ERROR log message indicates that something went wrong. METplus first converts the observation data files to a format that the MET tools can read using the MADIS2NC wrapper. Point-Stat is run to generate statistics comparing METAR observations to surface-level model fields and RAOB observations to “upper air” fields. METplus will print its completion status when the processing finishes.

The results of the METplus verification can be found in ${WORKING_DIR}/metplus_out/point_stat. These files contain tabular output that can be viewed in a text editor. Turn off word wrapping for better viewing. Refer to the MET User’s Guide for more information about the Point-Stat output. In the near future, this exercise will be extended to include instructions to visualize the results.

Refer back to the Managing a Red Cloud Instance section of the Red Cloud instructions to avoid unneccessary computing costs.

6.1.4.4. On Windows (Intel CPU)

Follow the compute platform instructions for Windows (Intel CPU) to launch a “Command Prompt” and an administrator on your Windows workstation.

The following sections instruct you to issue numerous Windows and Linux commands in your shell. If you are not familiar with Linux, you may want to want to refer to An Introduction to Linux when working through these steps. The commands in each section can be copied using the button in the upper right corner and then pasted into your web shell by right-clicking.

Instructions

Define Environment Variables

We will be using some environment variables throughout this exercise to make sure that we refer to the same resource names and file paths wherever they are used. The first variable you need to define will specify the location of the “working directory” for the data and run folders. The example command below specifies that the working directory is the home directory of a hypothetical username “exercise”. You will need to enter a command similar to this that either specifies your user account name instead of “exercise”, or changes the path entirely to use a different location on your computer:

set WORKING_DIR=C:\Users\exercise

Now you can copy and paste the definitions below into your shell to define the other variables before proceeding:

set WRF_IMAGE=ncar/iwrf:latest
set METPLUS_IMAGE=ncar/iwrf-metplus:latest
set WRF_DIR=%WORKING_DIR%\wrf\20161006_00
set METPLUS_DIR=%WORKING_DIR%\metplus_out
set WRF_CONFIG_DIR=%WORKING_DIR%\i-wrf-main\use_cases\Hurricane_Matthew\WRF
set METPLUS_CONFIG_DIR=%WORKING_DIR%\i-wrf-main\use_cases\Hurricane_Matthew\METplus
set PLOT_SCRIPT_DIR=%WORKING_DIR%\i-wrf-main\use_cases\Hurricane_Matthew\Visualization
set OBS_DATA_VOL=matthew-input-obs

Any time you open a new shell on your instance, you will need to perform this action to redefine the variables before executing the commands that follow.

Create the WRF and METplus Run Folders

The simulation is performed using a script that expects to run in a folder where it can create result files. The first command below creates a folder (named “wrf”) under the user’s home directory, and a sub-folder within “wrf” to hold the output of this simulation. The subfolder is named “20161006_00”, which is the beginning date and time of the simulation. Similarly, a run folder named “metplus_out” must be created for the METplus process to use:

mkdir %WRF_DIR%
mkdir %METPLUS_DIR%

Download Configuration Files

Both WRF and METplus require some configuration files to direct their behavior, and those must be downloaded from GitHub:

• In a browser, visit the I-WRF GitHub repository.

• Expand the green button <> Code button, and select Download ZIP.

• After the ZIP file has been downloaded, open it and extract its contents to the working directory you have selected as a folder named “i-wrf-main” (the default). Be careful not to include two levels of “i-wrf-main” folders in the path!

Now, some of the configuration files must be copied into the WRF run folder. These commands perform the necessary operations:

copy /y %WRF_CONFIG_DIR%\namelist.* %WRF_DIR%
copy /y %WRF_CONFIG_DIR%\vars_io.txt %WRF_DIR%
copy /y %WRF_CONFIG_DIR%\run.sh %WRF_DIR%

Install Docker

As mentioned above, the WRF and METplus software are provided as Docker images that will run as a “container” on your cloud instance. To run a Docker container, you must first install the Docker Engine on your instance. You can then “pull” (download) the WRF and METplus images that will be run as containers.

Install Docker Desktop

In order to install Docker on your Windows computer, one or more Windows services must be enabled (these services allow virtualization and running of containers). The process for performing this setup and installation is outlined below. During the setup process your computer may reboot one or more times, so be sure to save all work and close your other applications before beginning the setup.

To install Docker and enable the required components on Windows 10/11, you will install the Docker Desktop for Windows application by following these steps:

• In a web browser, visit Install Docker Desktop on Windows.

• Click on Docker Desktop for Windows - x86_64 to download the installer.

• Run the installer file “Docker Desktop Installer.exe”, which will require a system restart.

• Leave the “Use WSL 2 instead of Hyper-V” option checked in the dialog that appears.

• After the installation is complete, use the Start menu to find and run Docker Desktop, then agree to the terms and complete the other steps in the “first use” wizard.

The Docker Desktop app should now show a green “Engine running” status in the lower left corner. If your engine isn’t running or you encounter any other issues, visit the Troubleshoot Docker Desktop page.

Get the WRF and METplus Docker Images and the Observed Weather Data

Once Docker is running, you must pull the correct versions of the WRF and METplus images onto your instance. Open a Command Prompt shell as done before, execute the commands to define the environment variables, and then issue these commands:

docker pull %WRF_IMAGE%
docker pull %METPLUS_IMAGE%

METplus is run to perform verification of the results of the WRF simulation using observations gathered during Hurricane Matthew. We download that data by pulling a Docker volume on which the data resides, then creating a container from that volume, and then referencing that volume when we run the METplus Docker container. The commands to pull the volume and create a container for it are:

docker pull ncar/iwrf-data:%OBS_DATA_VOL%.docker
docker create --name %OBS_DATA_VOL% ncar/iwrf-data:%OBS_DATA_VOL%.docker

Download Data for WRF

To run WRF on the Hurricane Matthew data, you need to have three data sets to support the computation. The commands in this section download archive files containing that data, then uncompress the archives into folders. The geographic data is large and takes several minutes to acquire, while the other two data sets are smaller and are downloaded directly into the WRF run folder, rather than the main working directory.

The steps to process each data set are the same:

• Visit the data set’s URL in a web browser, which will download the .tar.gz file.

• Unzip the .tar.gz file contents into the destination folder.

• Remove the downloaded .tar.gz file.

Begin by download all of the data sets in this table:

Data Set	URL	Destination
Terrain	https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_high_res_mandatory.tar.gz	%WORKING_DIR%
Case study	https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_1deg.tar.gz	%WRF_DIR%
Sea Surface Temps	https://www2.mmm.ucar.edu/wrf/TUTORIAL_DATA/matthew_sst.tar.gz	%WRF_DIR%

Now, in your command prompt window, change directory (“cd”) to the folder where those files were downloaded. Then, copy/paste the commands below to unzip the data and delete the downloaded files:

tar -xzf geog_high_res_mandatory.tar.gz -C %WORKING_DIR%
del geog_high_res_mandatory.tar.gz

tar -xzf matthew_1deg.tar.gz -C %WRF_DIR%
del -f matthew_1deg.tar.gz

tar -xzf matthew_sst.tar.gz -C %WRF_DIR%
del -f matthew_sst.tar.gz

Run WRF

With everything in place, you are now ready to run the Docker container that will perform the simulation. The downloaded script runs inside the container, prints lots of status information, and creates output files in the run folder you created. Execute this command to run the simulation in your shell:

docker run --shm-size 14G -it ^
  -v %WORKING_DIR%:/home/wrfuser/terrestrial_data ^
  -v %WRF_DIR%:/tmp/hurricane_matthew ^
  %WRF_IMAGE% /tmp/hurricane_matthew/run.sh

The command has numerous arguments and options, which do the following:

• docker run creates the container if needed and then runs it.

• --shm-size 14G -it tells the command how much shared memory to use, and to run interactively in the shell.

• The -v options map folders in your cloud instance to paths within the container.

• ncar/iwrf:latest is the Docker image to use when creating the container.

• /tmp/hurricane_matthew/run.sh is the location within the container of the script that it runs.

The simulation initially prints lots of information while initializing things, then settles in to the computation. The provided configuration simulates 48 hours of weather and should take less than 30 minutes to finish, depending on your CPU’s number of cores and clock speed. Once completed, you can view the end of an output file to confirm that it succeeded:

powershell -command "& {Get-Content %WRF_DIR%\rsl.out.0000 | Select-Object -last 10}"

The output should look something like this:

Timing for main: time 2016-10-07_23:50:00 on domain 1: 0.25548 elapsed seconds
Timing for main: time 2016-10-07_23:52:30 on domain 1: 0.25495 elapsed seconds
Timing for main: time 2016-10-07_23:55:00 on domain 1: 0.25066 elapsed seconds
Timing for main: time 2016-10-07_23:57:30 on domain 1: 0.25231 elapsed seconds
Timing for main: time 2016-10-08_00:00:00 on domain 1: 0.25795 elapsed seconds
Timing for Writing wrfout_d01_2016-10-08_00:00:00 for domain 1: 0.68666 elapsed seconds
Timing for Writing wrfout_zlev_d01_2016-10-08_00:00:00 for domain 1: 0.47411 elapsed seconds
Timing for Writing wrfout_plev_d01_2016-10-08_00:00:00 for domain 1: 0.47619 elapsed seconds
Timing for Writing restart for domain 1: 1.54598 elapsed seconds
d01 2016-10-08_00:00:00 wrf: SUCCESS COMPLETE WRF

Run METplus

After the WRF simulation has finished, you can run the METplus verification to compare the simulated results to the actual weather observations during the hurricane. The verification takes about five minutes to complete. We use command line options to tell the METplus container several things, including where the observed data is located, where the METplus configuration can be found, where the plotting scripts can be found, where the WRF output data is located, and where it should create its output files:

docker run --rm -it ^
  --volumes-from %OBS_DATA_VOL% ^
  -v %METPLUS_CONFIG_DIR%:/config ^
  -v %PLOT_SCRIPT_DIR%:/plot_scripts ^
  -v %WORKING_DIR%\wrf:/data/input/wrf ^
  -v %METPLUS_DIR%:/data/output %METPLUS_IMAGE% ^
  /metplus/METplus/ush/run_metplus.py /config/PointStat_matthew.conf

Progress information is displayed while the verification is performed. WARNING log messages are expected because observations files are not available for every valid time and METplus is configured to allow some missing inputs. An ERROR log message indicates that something went wrong. METplus first converts the observation data files to a format that the MET tools can read using the MADIS2NC wrapper. Point-Stat is run to generate statistics comparing METAR observations to surface-level model fields and RAOB observations to “upper air” fields. METplus will print its completion status when the processing finishes.

The results of the METplus verification can be found in %WORKING_DIR%\metplus_out\point_stat. These files contain tabular output that can be viewed in a text editor. Turn off word wrapping for better viewing. Refer to the MET User’s Guide for more information about the Point-Stat output. In the near future, this exercise will be extended to include instructions to visualize the results.