A view of Puget Sound from the Washington State Convention Center, home of the 2017 AMS Annual Meeting |

# NSSL/SPC Spring Experiment Blog

Extending the conversation about real-time high-resolution convection-allowing modeling.

## Tuesday, February 07, 2017

### The SFE at the American Meteorological Society's Annual Meeting

During the week before last, over 4500 meteorologists convened in Seattle, Washington for the 97th American Meteorological Society (AMS) annual meeting. As always, I left this meeting with a plethora of new ideas, enthusiasm for the field, and at least a dozen papers added to my to-read pile. However, I also noticed a number of talks which mentioned the Spring Forecasting Experiment, including results from past experiments and hints of what's to come in SFE 2017.

## Friday, December 02, 2016

### A Late November Outbreak

Greetings from the off-season!

While SFE 2017 (!) is a ways off yet, preparations are already underway for many of the collaborators that provide products to the experiment. Development of the ensembles and guidance tested in the SFEs often occurs across a number of years, as tweaks suggested by prior experiments are implemented alongside new product development.

For example, in SFE 2015 four sets of tornado probabilities were evaluated. While all of the probabilities used 2-5 km updraft helicity (UH) from the NSSL-WRF ensemble, they differed in the environmental criteria used to filter the UH (i.e., if a simulated storm from a member was moving into an unfavorable environment, it was less likely to form a tornado and therefore the ensemble probabilities were lowered). These probabilities showed an overforecasting bias in the seasonally aggregated statistics, and the bias was consequential enough to be noted in subjective participant evaluations. The most typical rating for the probabilities was a 5 or 6 on a scale of 1-10, leaving much room for improvement.

To improve these tornado probabilities, a set of climatological tornado frequencies given a right-moving supercell and a significant tornado parameter (STP) value, as calculated by forecasters at the SPC, were brought to bear on the problem. The application of the climatological frequencies grounded the probabilities in reality. For example, in the prior probabilities if 6/10 ensemble members had a simulated storm passing over the same spot, the forecast probability would be 60%. The updated probabilities consider the magnitude of the STP in the environment the storm is moving into in each member. For example, if all of the storms were moving into an environment with an STP of 2.0, each member is assigned the climatological frequency of a storm to produce a tornado in that situation as the probability of a tornado. Then, the probabilities are averaged across each member. Assuming that 6/10 members now have the storm moving into an environment with STP = 2, the probability would be 60% * the climatological frequency of a tornado given STP = 2. This approach lowers the probabilities, and thus reduces overforecasting.

The new set of probabilities will be tested in SFE 2017. However, these probabilities have been worked on for over a year, and are already available daily on the NSSL-WRF ensemble's website.

While the statistics for all of the tornado probabilities discussed herein were aggregated over the peak of tornado season (i.e., April-June), the end of November 2016 brought tornadoes to the southeastern United States, and with them, the chance to test the new probabilities. We'll focus specifically on 29 November 2016, a day that saw 44 filtered tornado local storm reports (LSRs):

The Storm Prediction Center had a good handle on this scenario, showcasing the potential for severe weather across some of the affected region four days in advance. At 0600 UTC on the day of the event, their "enhanced" area covered much of the hardest-hit areas, with the axis of the outlook a bit skewed from the axis of the LSRs. The outlook and LSRs are shown below.

These probabilities have a much lower magnitude, but still encompass most of the tornado reports within the 10% contour. The 2% contour is also extended westward into Louisiana, capturing the tornado report that the prior probabilities missed. Overall, this forecast is more like the SPC's outlook, and better reflects what happened on the 29th.

Will we see the same trends into the spring? Aggregated seasonal statistics from spring 2014-2015 seem to suggest yes. However, the opportunity to get participant reflection and evaluation on these probabilities and this methodology awaits - and I, for one, am excited to see what new insights our participants will bring.

While SFE 2017 (!) is a ways off yet, preparations are already underway for many of the collaborators that provide products to the experiment. Development of the ensembles and guidance tested in the SFEs often occurs across a number of years, as tweaks suggested by prior experiments are implemented alongside new product development.

For example, in SFE 2015 four sets of tornado probabilities were evaluated. While all of the probabilities used 2-5 km updraft helicity (UH) from the NSSL-WRF ensemble, they differed in the environmental criteria used to filter the UH (i.e., if a simulated storm from a member was moving into an unfavorable environment, it was less likely to form a tornado and therefore the ensemble probabilities were lowered). These probabilities showed an overforecasting bias in the seasonally aggregated statistics, and the bias was consequential enough to be noted in subjective participant evaluations. The most typical rating for the probabilities was a 5 or 6 on a scale of 1-10, leaving much room for improvement.

To improve these tornado probabilities, a set of climatological tornado frequencies given a right-moving supercell and a significant tornado parameter (STP) value, as calculated by forecasters at the SPC, were brought to bear on the problem. The application of the climatological frequencies grounded the probabilities in reality. For example, in the prior probabilities if 6/10 ensemble members had a simulated storm passing over the same spot, the forecast probability would be 60%. The updated probabilities consider the magnitude of the STP in the environment the storm is moving into in each member. For example, if all of the storms were moving into an environment with an STP of 2.0, each member is assigned the climatological frequency of a storm to produce a tornado in that situation as the probability of a tornado. Then, the probabilities are averaged across each member. Assuming that 6/10 members now have the storm moving into an environment with STP = 2, the probability would be 60% * the climatological frequency of a tornado given STP = 2. This approach lowers the probabilities, and thus reduces overforecasting.

The new set of probabilities will be tested in SFE 2017. However, these probabilities have been worked on for over a year, and are already available daily on the NSSL-WRF ensemble's website.

While the statistics for all of the tornado probabilities discussed herein were aggregated over the peak of tornado season (i.e., April-June), the end of November 2016 brought tornadoes to the southeastern United States, and with them, the chance to test the new probabilities. We'll focus specifically on 29 November 2016, a day that saw 44 filtered tornado local storm reports (LSRs):

The Storm Prediction Center had a good handle on this scenario, showcasing the potential for severe weather across some of the affected region four days in advance. At 0600 UTC on the day of the event, their "enhanced" area covered much of the hardest-hit areas, with the axis of the outlook a bit skewed from the axis of the LSRs. The outlook and LSRs are shown below.

The 0600 UTC outlook is shown here, because that is when the probabilities computed above become available - our hope is that someday forecasters can look at these probabilities as a "first-guess", encompassing multiple severe storm parameters from the ensemble into one graphic. The SPC's probabilistic tornado forecast from 0600 UTC encompassed all of the tornado reports, but was a bit too far west initially. Ideally, the ensemble tornado forecasts would resemble the SPC's forecast:

When we consider the UH-based probabilities, there's a pocket of high probabilities, between 25-30%, in an area that is close to the highest density of tornado reports. Additionally, all of the reports are not encompassed by the probabilities, and there is an extraneous blob of 5% risk over the DC/Maryland area. The 10% corridor of the probabilities extends further north than the SPC's, but overall, this was a decent forecast, if a bit high in that "bulls-eye" of probabilities.

Let's compare this to the STP-based probabilities:

Will we see the same trends into the spring? Aggregated seasonal statistics from spring 2014-2015 seem to suggest yes. However, the opportunity to get participant reflection and evaluation on these probabilities and this methodology awaits - and I, for one, am excited to see what new insights our participants will bring.

## Wednesday, June 08, 2016

### SFE 2016 Wrap Up

Well, last week concluded SFE 2016. This season was a particularly interesting one. While we always deal with some marginal cases and mesoscale forcing as the mechanism for severe convection, this year seemed to feature many of those cases. Lots of days throughout the experiment were a bit difficult to forecast conceptually, even the high-end days such as 26 May. While the full period forecasts were easier, breaking down the full period into specific four-hour chunks proved challenging, given that these forecasts contained both a forecast of convective initiation/intensification (if the convection was ongoing) of severe storms, as well as the motion and evolution of those storms (i.e., would supercells form and merge into an MCS? Would morning convection reintensify?). Each of those elements is a forecast challenge separately, but we combined them into one.

In a way, it's ideal that we faced so many of these environments. We've seen in past SFEs that when the CAMs are strongly forced, they often do quite well at pinpointing the location and intensity of severe convection. Where do they have the most difficulty? Under weaker forcing, when remnant outflow boundaries and mesoscale details have a large influence on the day's convection. To have a 65-member CAM ensemble in the CLUE operating during these environments may give us unparalleled insight to what CAM ensemble design characteristics perform best under uncertain circumstances, and can augment the deterministic guidance that is already operational. While we may have come into most days looking at only a small area where CAPE, shear, and a lifting mechanism were present, this set of days will provide us with many case studies of realistic, less-than-ideal circumstances.

As always, a huge thanks goes out to our participants, who hailed from multiple countries and states. We gathered a number of subjective impressions from these participants on various subsets of the CLUE, illustrating forecaster and researcher insights about how these CAMs may best be applied. In the case of the isochrones, this year's comments will help design a better, more user-friendly product and introduction to the concept for next year.

Two great challenges lie ahead: the verification and analysis of the massive amount of data generated and collected during SFE 2016, and the planning of SFE 2017. Such is the cycle of an annual experiment - the work is never done. Onward!

In a way, it's ideal that we faced so many of these environments. We've seen in past SFEs that when the CAMs are strongly forced, they often do quite well at pinpointing the location and intensity of severe convection. Where do they have the most difficulty? Under weaker forcing, when remnant outflow boundaries and mesoscale details have a large influence on the day's convection. To have a 65-member CAM ensemble in the CLUE operating during these environments may give us unparalleled insight to what CAM ensemble design characteristics perform best under uncertain circumstances, and can augment the deterministic guidance that is already operational. While we may have come into most days looking at only a small area where CAPE, shear, and a lifting mechanism were present, this set of days will provide us with many case studies of realistic, less-than-ideal circumstances.

As always, a huge thanks goes out to our participants, who hailed from multiple countries and states. We gathered a number of subjective impressions from these participants on various subsets of the CLUE, illustrating forecaster and researcher insights about how these CAMs may best be applied. In the case of the isochrones, this year's comments will help design a better, more user-friendly product and introduction to the concept for next year.

Two great challenges lie ahead: the verification and analysis of the massive amount of data generated and collected during SFE 2016, and the planning of SFE 2017. Such is the cycle of an annual experiment - the work is never done. Onward!

## Wednesday, June 01, 2016

### Chopping the FAR

Afternoons in the SFE are composed of three main parts: A Day 2 forecast, evaluations of various aspects of the CAMs, and updates to the morning forecasts. Sometimes, very little new information contributes to these updates, particularly if convection has not initiated by the time of the update. Other days, convective initiation or intensification has occurred, and we have a much better concept of how the convection will evolve. Yesterday was an excellent example of how the afternoon updates can improve upon the morning forecasts, once we get a sense of the evolution.

## Tuesday, May 31, 2016

### Data Driven

Well, we have arrived at the fifth and final week of SFE 2016. By this point in the experiment, the facilitators are mostly used to the rhythm of the testbed, knowing what observational data and model guidance we'll go over each day. By the end of the week, participants are generally used to the fast pace of the experiment as well. However, the first day of each week provides some reminders as to how much we're throwing at the participants. I thought that tonight, I'd provide a brief rundown of what we consider when making our full period outlooks each day, which run from 16Z of any given day to 12Z the following day.

## Thursday, May 26, 2016

### CLUE Comparisons

Each day, an evaluation takes place on the total severe desk to compare three subensembles of the CLUE. One subensemble contains 10 ARW members, one contains 10 NMMB members, and one contains 5 ARW and 5 NMMB members. Participants look at two fields to evaluate these subsets of the CLUE: the probability of reflectivity greater than 40 dBZ, and the updraft helicity. This week, all ensembles have been having trouble with grasping the complex convective scenario, as have most of the convection-allowing guidance. However, today's comparison highlighted the challenges of these evaluations: each model had different strengths at varying time periods throughout the forecast, but participants had to provide one summary rating for the entire run.

## Tuesday, May 24, 2016

### Model Solutions Galore

This week we've been experiencing broader risk areas in the Spring Forecasting Experiment than previous weeks. The instability has recovered across much of the Great Plains, and the persistent southwesterly flow at upper levels due to a trough in the west has sent steep lapse rates over a wide area. While the trough is still somewhat too far west for the greatest flow to coincide with a broad area of instability, deep-layer shear has been sufficient for severe storms to occur

When we have large areas to consider in conjunction with the huge amount of NWP data we have from the CLUE, deterministic CAMs, and operational large-scale guidance, the number of different scenarios can be overwhelming. Particularly this week, there are multiple solutions for how the day's weather could evolve, according to the NWP. Mesoscale details from prior convection have also played a large role both today and yesterday in making our forecasts, and the variation in the CAM guidance reflects the reliance on those small-scale details. One member's outflow boundary is likely not in the same place as another's, and it's up to participants to determine which solution we think will verify. As an example of the different solutions we saw yesterday, here's a snapshot of five ensemble members whose configuration differs only in the microphysics scheme they're using. Observations are in the lower right hand panel:

*somewhere*each day. Determining the exact location is a daily difficulty for our SFE participants.When we have large areas to consider in conjunction with the huge amount of NWP data we have from the CLUE, deterministic CAMs, and operational large-scale guidance, the number of different scenarios can be overwhelming. Particularly this week, there are multiple solutions for how the day's weather could evolve, according to the NWP. Mesoscale details from prior convection have also played a large role both today and yesterday in making our forecasts, and the variation in the CAM guidance reflects the reliance on those small-scale details. One member's outflow boundary is likely not in the same place as another's, and it's up to participants to determine which solution we think will verify. As an example of the different solutions we saw yesterday, here's a snapshot of five ensemble members whose configuration differs only in the microphysics scheme they're using. Observations are in the lower right hand panel:

Subscribe to:
Posts (Atom)