Verification (with Low Population)

The target area yesterday was very small, hugging the U.S.-Mexico border from the Big Bend region of Texas northward and westward to New Mexico. This area is sparsely populated, which becomes an issue when trying to verify forecasts of severe weather. The United States is far from the only country to have this problem - one participant gave a talk this week that mentioned how the area with the most severe weather in South America is also sparsely populated. When we're forecasting for an underpopulated area in the experiment we have to examine metrics other than Local Storm Reports (LSRs), particularly because the verification of yesterday's forecasts is the first activity each morning.

One of these metrics is the warnings issued by local NWS offices. By looking at where warnings were issued, we get a better idea of storm locations and where some LSRs may eventually appear, particularly in sparsely populated areas. When we were verifying yesterday's forecasts this morning, there were no reports in Texas. However, there were warnings, and according to the current LSR page for yesterday, some LSRs were called in and recorded after our verifications.

Another method of verification that we consider regarding hail forecasts is the radar-derived Maximum Estimated Size of Hail (MESH) tracks. These can be used to create practically perfect forecasts, but tend to create higher probabilities than we would typically use. However, yesterday it suggested a 15% contour, and did not contour any area of significant hail. Therefore, it matched the general character of our forecasts on both the total severe desk (above) and the individual hazards desk (below):

While these practically perfect forecasts are useful to compare to our forecasts, it's also interesting to see where there are MESH tracks and LSRs of hail overlapping. The overlap can give us ideas about how a storm acts across its lifetime. For example, take a storm producing LSRs in a more populated area, but moving into a less populated area. If a MESH track concurrent with the LSRs extends into the less populated area, we can deduce that the storm continued to produce hail. Yesterday's LSR/MESH map looked like this:

Upon closer examination of our area of concern, there are clearly MESH tracks in the low-population Big Bend area. 

Indeed, the LSRs that came in after this map was generated are in that general Big Bend vicinity. We also see only one small speck of red, which is an estimate of greater than 2" hail, in south Texas. Finally, we can see that the hail LSRs in New Mexico are concurrent with MESH tracks, resulting in two forms of verification indicating severe hail.

While participants can have a difficult time evaluating marginal events with only a few reports, considering warnings and other verification such as MESH can add some information to participant ratings. Allowing them to consider the different ways forecasts can be verified and the strengths and weaknesses of each method allows for multiple aspects of the forecast verification problem to be brought to light and discussed.