Code Sleuthing

Yesterday, I discussed a problem that forecasters often have which researchers may be less aware of on a day-to-day basis: the sheer amount of change that can occur within a short time span in the atmosphere. To complement that perspective, today I'll discuss a challenge that researchers often face which forecasters may not deal with as often: the detective work and complexity of eradicating code bugs.

One of the main comparisons of SFE 2016 concerns two dynamic model cores: the ARW and the NMMB. These cores have both been examined daily in the SFE, but the NMMB consistently has produced higher temperatures, lower dewpoints, and subsequently less CAPE than the ARW. This difference was discussed in an earlier blog post, but at the time we were unsure if a bug was causing the differences. However, as each day continued to show a warm, dry bias in the NMMB, it became apparent that something systemic was wrong. This led researchers on a trail of  sleuthing to uncover the latent issue.

In preparing to run the NMMB core during the SFE, facilitators worked closely with scientists at the Environmental Modeling Center (EMC). Two different agencies were running NMMB ensemble members for the SFE: the National Severe Storms Laboratory (NSSL) and the Center for the Analysis and Prediction of Storms (CAPS). The warm, dry bias was noticed in the NSSL runs prior to the experiment, but had been eliminated during a series of changes to the runs. While the issue was fixed, the precise source of the error remained unknown, as did which code change fixed the problem. Therefore, when the issue began showing up in the CAPS runs (used for the afternoon comparisons and evaluations), the fix wasn't clear. First, the source of the error had to be determined. EMC researchers had rerun the NMMB core with the same user-generated input and lateral boundary conditions that CAPS and NSSL had been using when the issue was first noticed, and did not experience the bias in their results. Therefore, the issue was likely not in any of the preprocessing or boundary condition generation. One source of potential error was eliminated and the clues to the actual problem accumulated.

The next step was to consider what could differ between the model parameterizations, since the model code itself was likely fine. An EMC researcher noticed that the afternoon heat fluxes were showing unreasonable values, suggesting a land surface parameter as the root of the issue. The EMC was then able to hone in on the two land surface parameter tables utilized when running the NMMB: the United States Geological Survey (USGS) table and the International Geosphere-Biosphere Programme (IGBP) table. These tables determine the land surface type across the domain. For example, a 10 in the USGS table might be code for a desert, which would interact differently with the atmosphere than a jungle. EMC uses IGBP to determine land surfaces in all of their NMMB runs, and table choice is determined in the configure files. 

Herein lies the problem: when the model is configured to ingest the IGBP table values, it can also ingest USGS table values. However, the mapping of land surface types will be different - a land surface coded as 10 will likely not be the same type of land surface in both tables, causing incorrect heat fluxes. The CAPS runs were configuring with the IGBP table option, but ingesting USGS land surface files to determine the land type. 

Two fixes are being implemented by EMC to prevent future scientists from being tripped up by this nuance. First, the Developmental Testbed Center (DTC) is making a note in its documentation to clarify this aspect of the model. Since they maintain the community code, users who download and experiment with the code will now be aware of this potential problem. Secondly, the model ideally would fail if the files ingested do not match the type of files indicated while configuring the model. The EMC code repository is making this fix, so that the model will provide an error message to inform future users that the tables do not match. 

Clearly, determining the source of this problem required many steps, eliminating potential sources of error before discovering the actual source, and collaboration between a variety of scientists at different agencies. Having multiple people closely examine the forecasts (as we do in the SFEs) helps model developers determine potential errors, particularly insidious ones. Communicating these challenges and recent improvements to forecasters is a key part of the SFEs, dovetailing with enhancing researcher understanding of forecast variability.