Predictability and Predictions

Abstract

This essay describes the author’s lifetime experiences with predictability theory and weather predictions.

“Prediction is very difficult, especially if it’s about the future.” [1]

Niels Bohr (Physics Nobel Prize 1922)

Humans have always been fascinated by foretelling the future, and oracles, shamans, priests, and prophets have held prominent positions in all cultures. Various strategies, often occult, form the basis for astrology, palmistry, numerology, soothsaying, prognostication, fortune telling, and Tarot. Horoscopes remain popular today, while weather, economic, and political forecasters hold lucrative positions in society, in spite of often dubious records.

One famous seer, Michel de Nostredame (1503–1566), known as Nostradamas, was a French astrologer and prophet who published a collection of 942 four-lined prophesies in his book Les Prophéties [2]. While mostly foretelling and commenting on political and religious catastrophes, there is one example of his prophesy of a weather and climate disaster:

“For forty years the rainbow will not be seen.

For forty years it will be seen every day.

The dry earth will grow more parched,

and there will be great floods when it is seen.”

The prediction of complex systems based on mathematics and physics rather than the occult also has a long history. In the 17th century, Gottfried Leibnitz (1646–1716) speculated that everything proceeds mathematically, and so someone who had sufficient understanding and could take into account everything, “would be a prophet and see the future in the present as in a mirror” [3]. About a hundred years later, the Marquis de Laplace (1749–1827) dreamed of an intelligent being (an intellect, later dubbed Laplace’s Demon) who knew the positions and velocities of every single atom and used Newton’s equations of motion to predict the future of the entire universe [4].These ideas of Leibniz and Laplace likely inspired the early ideas associated with numerical prediction of the weather, transforming weather prediction from an arcane art to a science.

“One can’t predict the weather more than a few days in advance.” [5]

Stephen Hawking (1942–2018)

While still in high school, I began my career in meteorology as a Student Trainee in the U.S. Weather Bureau. I spent summers at the Weather Bureau Airport Stations in Richmond and Norfolk, Virginia and was introduced formally to the challenges of operational weather forecasting. We did not have any numerical model guidance at this time, and our forecasts were entirely qualitative, limited to a day or two. However, I noticed that lines of thunderstorms would often develop in the early afternoon over the Blue Ridge Mountains and propagate eastward, reaching the coast in the evening, and these events appeared to have some predictability. (Here, I use the word predictability in its common lay meaning, “able to be predicted.” This definition differs somewhat from the generally accepted meaning of predictability in the sense of nonlinear fluid dynamics.) I dreamed of someday being able to quantitatively predict such events well before they occur. I had not yet heard of the concept of numerical weather prediction (NWP), which was just being developed at Princeton under the direction of Jule Charney, based on the early ideas of Vilhelm Bjerknes and Lewis Fry Richardson.

As an undergraduate student at the University of Wisconsin in Madison (Madison, WI, USA) in the early 1960s, I became interested in numerical simulations, and coded a simple nonlinear one-dimensional gravity wave model on punch cards. Given appropriate initial conditions, the model correctly predicted many cycles of a propagating gravity wave. By this time, the Weather Bureau was distributing 36 h forecasts of 500 mb flows using a barotropic model, which at least to us students was of dubious value, but still captured our imagination as we anxiously awaited the latest model run.

While still at Wisconsin, I moved from the Virginia airport stations to the ESSA (Environmental Science Services Administration, now NOAA) National Hurricane Research Laboratory (Miami, FL, USA) and began developing a nonlinear, baroclinic, three-dimensional model of the tropical cyclone. During this development and debugging, I suffered through various forms of numerical and physical instabilities. The numerical instabilities could be controlled by suitable choices of finite difference schemes and various damping or smoothing mechanisms, but physical instabilities persisted and resulted in the evolution of somewhat realistic mesoscale features such as rainbands and eddies on the outflow layer that were not present in the initial conditions [6].

Around this time, in the early 1960s, Edward Lorenz at MIT noted that small differences in a simple numerical model he was working with doubled in size every four days, and in 1963 he published his classic paper in the Journal of Atmospheric Sciences, “Deterministic nonperiodic flow,” which has been the theoretical foundation of modern predictability theory for nonlinear fluid dynamics, including numerical prediction of atmospheres and oceans [7]. In 1972, Lorenz gave a talk on atmospheric predictability with the title “Does the flap of a butterfly’s wing in Brazil set off a tornado in Texas?” This rhetorical and provocative question has intrigued scientists and the public ever sense, and “the butterfly effect” has come to mean chaos and the lack of predictability of chaotic nonlinear systems, including human societies [8].

During my time at the National Hurricane Research Laboratory, I became friends with Fred Sanders, a leading synoptic meteorologist from MIT. I sailed with Fred on his sailboat, the Stillwater, from Miami to Marblehead, Massachusetts in June 1968. One quiet warm night under the stars during our midnight watch, we had a long philosophical discussion of whether the universe was theoretically predictable. I was of the opinion that it might be deterministic, that if someone knew all the physics, chemistry, and biology and had perfect knowledge of the complete state of the universe at a given time, the future of the universe was knowable. I had been captured by Laplace’s demon. Fred must have been aware of Lorenz’ work on chaos theory by this time, and he gently argued against this possibility.

When I moved to Penn State as an Assistant Professor in 1971, my PhD student Tom Warner and I began modifying the hurricane simulation model to accept real data and make forecasts, and the model evolved into multiple generations of what we called the Penn State Mesoscale Model, or MM for short. The fifth-generation of MM, MM5, became widely used internationally as a research and forecast model. There were some prominent scientists in the community that were skeptical of this effort, warning us that trying to forecast small-scale (mesoscale) atmospheric phenomena was a fool’s errand, making various arguments based on turbulence and predictability theories and lack of observations on the mesoscale. Yet, as our model development and testing continued, we saw many examples of realistic and observed mesoscale features developing in the model forecasts, in spite of only large-scale features present in the initial conditions. These included fronts, jet streaks, precipitation systems, and various features associated with terrain variations, surface conditions such as soil moisture variations, and land-water contrasts. In our first comprehensive paper describing the MM model [9], we noted that nonlinear processes are capable of producing smaller scale information in the forecast than is present in the initial conditions, as long waves interact to produce energy in shorter waves. Frontogenesis through horizontal deformation in a large-scale baroclinic field was a prime example [10]. These nonlinear effects starting with realistic large-scale initial conditions, combined with local forcing associated with variations in terrain and surface conditions, and diabatic effects led us to hypothesize “in many synoptic situations, if the local forcing is modeled correctly, the details of the initial perturbations are not particularly important.” Perhaps some predictability of mesoscale features did exist, at least for limited times into the numerical forecast. I was still thinking of those lines of thunderstorms that developed over the Blue Ridge Mountains in summer.

I summarized the above ideas in a 1984 paper, “Predictability of mesoscale meteorological phenomena,” classifying the development of mesoscale weather systems into two types [11]: (1) those resulting from forcing by surface inhomogeneities and (2) those resulting from internal modifications of large-scale flow patterns. Land-sea breezes, mountain-valley breezes, mountain waves, heat island circulations, coastal fronts, and dryline and moist convection are often generated by the first mechanisms. Fronts and jet stream phenomena, generated by shearing and stretching deformation associated with large-scale flows, belong to the second class. After showing examples of these two types of mesoscale predictions at a conference on predictability, I recall Michael Ghil, a theoretical climate dynamicist from UCLA, jumped up and exclaimed, tongue in cheek, “maybe we should stop studying predictability and just start making predictions!”

On longer time scales, forcing associated with anomalies such as ocean temperatures and changing greenhouse gas concentrations offered hope for some predictive skill of climate models, as argued by Jagadish Shukla in his 1998 Science article [12].

After I moved to NCAR in 1981 and began the administrative phase of my career, my active research with MM ended. However, the development of the MM was continued very competently by Tom Warner, Bill Kuo, and others, and as the model resolution and physical parameterizations improved, we continued to see examples of successful predictions of mesoscale features. Mesoscale modeling efforts were now going on in many places, and I was excited more than 20 years later when I became aware of some stimulating results from a mesoscale global model at NASA carried out by Bo-Wen Shen and his colleagues. By 2008, global mesoscale models with horizontal resolutions of ~10 km were possible (in 1978 our MM experiments with real data were over a limited area with horizontal resolutions of ~111 km). The NASA simulations of tropical storm genesis were supporting Tom’s and my hypothesis from 1978! Large-scale forcing associated with the Madden–Julian Oscillation was responsible for the correct prediction of tropical cyclones such as Nargis [13] in the Indian Ocean, many days before any evidence of the tropical cyclone circulation existed [13]. Based in large part on this successful forecast, I wrote a short essay in UCAR Magazine in 2011 with an optimistic view on predictability [14].

In 2014, Bo-Wen moved to San Diego State University’s Department of Mathematics and Statistics (San Diego, CA, USA) where he continued his work on theoretical aspects of atmospheric predictability, extending Lorenz’ revolutionary work in the 1960s and 1970s (Ref. [15], which concludes “the entirety of weather possesses both chaos and order.”) This work, and the success of high-resolution global models, now supported by mesoscale satellite observations assimilated with sophisticated four-dimensional variational schemes, in predicting useful mesoscale phenomena such as tropical cyclones, provide optimism for continuing to increase the forecast lead time of significant weather. Nevertheless, it seems clear to me now that the butterflies will ultimately prevail, and nothing is predictable forever. The demon has been exorcised.