Tuesday, February 8, 2005
Forecast assessment: Tactics, techniques, and tools
Forecasts serve as a key link between resource managers and hydroclimatic research, because almost all resource management decisions require, whether explicitly or implicitly, some sort of hydroclimatic forecast, and research progress is often judged in terms of improved predictability. However, while advances in predictability are desirable, alone they are insufficient for more extensive application of forecasts or better decision and societal outcomes. Many decision makers have difficulty placing forecast information in appropriate historical and regional contexts. Further, "downstream" forecasts that reflect impacts of climate variability typically have more relevance to decisions than climate products. Finally, pervasive misinterpretation of forecasts and uncertainty about their accuracy present formidable barriers.
Quantitative evaluations of the seasonal forecasts demonstrate that simple approaches, using "hit or miss" criteria (e.g., probability of detection, false alarm rate) or traditional summary statistics (e.g., root mean squared error, correlation) neglect important aspects of forecast performance and can even be misleading, potentially affecting resource management decisions. On the other hand, distributions-oriented evaluation criteria are more informative and allow decision makers to target those aspects of forecast performance that are important for their situation. For example, using the criteria of "discrimination", predictions of seasonal streamflow volumes for Colorado River tributaries are shown to convey useable information not revealed by other criteria, even with lead-times of several months. From a decision maker's perspective, it is important that forecast evaluations be frequently updated and target the regions, seasons, lead times, and criteria important to specific decision making situations. From an operational perspective, more information needs to be archived than has been the traditional practice, especially for probabilistic predictions.
Our work with stakeholders led to development of an interactive forecast assessment tool, accessible over the Internet (http://hydis6.hwr.arizona.edu/ForecastEvaluationTool/). The tool allows users to evaluate forecast performance for the regions, seasons, forecast lead times, and performance criteria relevant to their specific decision making situations and at the conceptual level they are capable of understanding, while offering opportunity to shift to more sophisticated criteria. The tool also allows users to test their forecast interpretation skills, efficiently monitor the time evolution of the climate forecasts and subsequent observations, and place the forecasts in the context of recent and historical observations.
The webtool is presently implemented to evaluate the seasonal temperature and precipitation outlooks issued by the National Weather Service (NWS) Climate Prediction Center , but based on feedback from interested decision makers, we are expanding the webtool to include other seasonal forecasts and supporting information, including water supply outlooks issued by the NWS and Natural Resources Conservation Service. Recognizing that many individuals lack Internet access, our webtool design also includes customized report generation so extension agents or other trusted information intermediaries can provide material to decision makers at meetings or site visits.
Our experience highlights several issues associated with development of nontraditional research products that are fundamentally different than traditional research products or even many decision support tools. Design and implementation of commercial quality websites is fundamentally different than producing traditional research products, requiring highly specialized technical personnel and time frames that can be difficult to accommodate in many federally funded projects. In addition, there are issues of the long-term sustainability of climate products and tools, their transferability, and maintenance. Regardless, from the perspective of decision makers, such products may be more useful than traditional products.
Holly Hartmann is a research scientist in the Department of Hydrology and Water Resources at the University of Arizona . She has been involved with one of the NOAA Regional Integrated Science Assessment (RISA) projects, the Climate Assessment for the Southwest (CLIMAS), since 1998. Holly was formerly a research hydrologist with the NOAA Great Lakes Environmental Research Laboratory.