A Culture of Checking: Open Climate Models (4)

By: Dr. Ricky Rood , 4:01 AM GMT on January 25, 2011

Share this Blog
4
+

A Culture of Checking: Open Climate Models (4)

This is a continuation of my series on community approaches to climate science and, specifically, climate models. I am exploring the ideas of open innovation and open source communities. This article will be on verification, testing, evaluation, and validation. I know that this is, perhaps, a bit off the path of what I imagine is my main audience, but these ideas are at the core of climate science, and also part of the politicization (or de-politicization) of science. Before I dig into the arcane, I wanted to reference Jeff Master’s post on the major floods of the past few months. This combination of warm seas, warm air, water vapor and rain and geography and people is difficult to ignore, and I plan to revisit these floods in the context of the recent guest blog by Christine Shearer. Now, deeper into the world of how we build climate models. Here are links to the previous articles in the series: #1 in series, #2 in series, #3 in series

Verification, Testing, Evaluation, Validation: Validation is an important part of the scientific method. The scientific method always relies on observations, in concert with the development of testable hypotheses, and experiments or predictions which provide the tests for the hypotheses. In practice the validation process includes not only a scientist evaluating the results of the experiment or prediction, but providing written documentation which describes their work and their methods of validation. Using this documentation other scientists are able to evaluate the work and design independent experiments or predictions and methods of validation. It is not until there is independent confirmation of a scientist’s work that the work is accepted into the scientific body of knowledge. Once in the body of knowledge, work is not recognized as fact - rather it sits as a contribution that is, often, continuously challenged. It is a harsh, competitive process – not sublime.

By the very definition of scientific practice, a certain level of transparency is required. The transparency allows those who are, essentially, competitors to examine, reproduce, and independently confirm or refute the work of the scientist who initiated the original study. This general process of validation, reporting, and independent certification is a remarkably conservative process. It is slow to interject major shifts and changes into the knowledge base. This conservativeness impacts strongly the way scientists speak to each other and the public; there are always nuances of uncertainty and equivocation (again, see this entry). If you think about this culture of validation, a culture of checking, it has a lot in common with how markets are run and governance. Governance? A set of rules, buy in by participants, and a system of checks and balances. I have a number of previous articles on validation and transparency in a more general sense: Opinions and Anecdotal Evidence, Trust, but Verify, Uncertainty.

Verification, testing, evaluation, validation, I am grouping these words together to describe the culture of checking that is pervasive in the good practice of science. When I went to school, the idea of checking my arithmetic was taught again and again. It had to be taught again and again, because I was not smart enough to understand the value of checking as an abstract concept. As problems get more and more complex, strategies for checking get more and more sophisticated. When I used to hire a lot of new graduates at NASA, during the interviews we explored how they checked their work. During the first year of employment we were often coaching, teaching, and training people in how to check.

How do we translate the skills of checking to climate models? I have been laboring over the words verification, testing, evaluation, and validation. What is the difference? To start, it is important to realize that climate models are computer codes, programs, software. And though many scientists object to the following categorization, the product that climate modelers produce is software. It is software for a purpose – the scientific investigation of climate. As suggested by the previous entries in this series; the software is complex; it represents individual sub-processes (like cloud formation), it represents an approximation to those sub-processes; it is developed by geographically dispersed individuals, who are generally not managed as a coordinated group.

I will start with the easy one. Verification is the process of assuring that the software you have written or implemented is doing what you intended for it to do. Suppose you are writing a simple computer program to calculate how long it will take you to drive from Limon, Colorado to DeKalb, Illinois. This is a simple equation of motion represented by distance traveled equals speed multiplied by time traveled. You might check your program (your model) by seeing how long it takes to drive 100 miles at 20 miles per hour – a simple problem for which you can confidently state the answer. Maybe you try different pairs of speeds and distances. If you are thoroughly scientific, you might collect some data with your car. Of course, that might raise questions of determining the accuracy of your speedometer and your odometer, another requirement for checking. With some confidence you can develop a program that, without driving from Limon to DeKalb, you can make a very good approximation of how long it will take at a given speed. You can perhaps add another question – how fast do I have to drive to make the trip in 24 hours? 18 hours? With this example you can imagine the process of verification, checking that your program is doing what it is supposed to do. You might also say that you are testing your code. Testing is another word in the culture of checking, which takes on more specific meaning in different processes.

Evaluation and validation are more difficult to explain. Both words are linked with a comparison with independent information, specifically, observed information. At the risk of being tedious, when I worked at NASA, there were different sub-cultures represented by those who made instruments and those who made models. Validation at NASA often defined the process by which people who took measurements from space assured that those measurements, say, measured temperature. This would require deploying different types of temperature measuring devices, like thermometers, to take concurrent measurements at the same place. The point here is that a new way to measure temperature was being evaluated with an accepted, established way to measure temperature. Within NASA, it was a widely held belief that models could not be “validated,” because in general there was not such a clean comparison to a standard of accepted knowledge. Hence, the word “evaluation” emerged as the way to state that the model was being compared with observed information.

I will ultimately maintain that models can be validated in a formal sense. This remains an assertion that many of my colleagues disagree with. While, I accept the nuances of evaluation and validation and testing, it is important that climate science embrace the rigor implied by “validation” of models. Before I go on, I provide a link to couple of papers that were provided to me by Doug Post some time ago. These papers drew largely from the experience in the U.S. National Laboratories responsible for assuring the robustness and safety of our nuclear weapons through the use of computational models (think about that application!). These papers generated transient discussion in the climate community: Computational Science Demands a New Paradigm and Software Project Management …..

As happens with my blogs, they sometimes, get a bit long, so in the spirit of the medium, I am going to search for the take away message. The existence of the semantic arguments concerning the words evaluation and validation suggest that defining measures of quality assurance of climate models is a difficult process. It is not uniquely defined. It depends on what you are trying to do, for example, predict El Nino or how the ocean melts the bases of glaciers in Antarctica. The evaluation or validation process also depends on how a modeling system performs. This system is constructed from sub-components developed by individual scientists, in practice, spread all over the world. The migration of individual components is sometimes performed by those reading the literature and reproducing work as described in the literature; local adaptation of algorithms is often performed.

As I stated above, there is a culture of checking in our field. Individuals check their work at multiple levels. But as the components are brought together, the ability to check gets more and more difficult. Remember, these pieces are, themselves, neither unique nor absolutely accurate. As we consider the Earth’s climate, the question becomes, what do we check against? We get to the question of the quality of measurements - just how good are they? And we get to the social problem that as the climate model is built from its pieces developed by individuals how do we define and codify a process that rises to the standard of validation? As this process becomes more and more complex, we are often moved to using the word “art” to describe the process of building models (see The Art of Climate Modeling ).

The final point that I want to make in this entry is that the culture of checking that scientists intuitively accept as individuals extends as an essential ingredient to the collective development of complex software systems. There are, in the development of these complex software systems, tensions that are not rationalized by convergent, deductive reasoning. These tensions might represent the choice of the quality of the oceanic circulation versus the quality of the atmospheric circulation in a computationally and human resource constrained environment. These tensions might abstract to conflicts between oceanographers and meteorologists, perhaps even the program managers that fund oceanographers and meteorologists.

The description and codification of a validation plan for a climate model, therefore, extends far beyond the definition of a set of observations that uniquely or adequately define the Earth’s climate. There are judgments and decisions that need to be made. There are tensions, perhaps conflicts, which need to be reconciled. There are even philosophical discussions about whether or not climate models can be validated. If the open innovation communities I am exploring in this series are to be realized, then the description and codification of the validation process is necessary. Beyond the narrow world of scientists, we need to be able to point to the elements and measurements of validation in order to provide the foundation of the use of models in mitigation, adaptation, and geo-engineering.

r





Figure 1. From Online Math Tutor.


Reader Comments

Comments will take a few seconds to appear.

Post Your Comments

Please sign in to post comments.

or Join

Not only will you be able to leave comments on this blog, but you'll also have the ability to upload and share your photos in our Wunder Photos section.

Display: 0, 50, 100, 200 Sort: Newest First - Order Posted

Viewing: 91 - 41

Page: 1 | 2 | 3Blog Index

91. cyclonebuster
3:12 PM GMT on February 02, 2011
Quoting MichaelSTL:


How about on this map? The one you posted was ONLY for the U.S. - less than 2% of the global surface area:



Also, LOL @ the posting of a WIND CHILL map as if WIND CHILL was meaningful! Wind chill is only how cold it feels to a human being, not how cold it really is (water will not freeze with a wind chill below zero if the air temperature is above zero).


These guys need to chill out! They think this storm is proof of global cooling.
Member Since: January 2, 2006 Posts: 127 Comments: 20470
90. cyclonebuster
3:09 PM GMT on February 02, 2011
Quoting HaloReachFan:
Looks like more negative temperatures on this graph than positive temperatures. I'm not going to count but I'm sure its close.


WOW! If that happens it must be just a few times a year if that!
Member Since: January 2, 2006 Posts: 127 Comments: 20470
88. cyclonebuster
3:08 PM GMT on February 02, 2011
Quoting MichaelSTL:
RECORD EVENT REPORT
NATIONAL WEATHER SERVICE CHICAGO IL
1238 AM CST WED FEB 2 2011

...NEW ALL-TIME DAILY SNOWFALL RECORD FOR THE MONTH OF FEBRUARY...

A RECORD BREAKING 13.6 INCHES OF SNOW FELL AT CHICAGO OHARE
INTERNATIONAL AIRPORT YESTERDAY. THIS NOT ONLY BREAKS THE PREVIOUS
RECORD OF 4.0 INCHES FOR THE DATE SET BACK ON FEBRUARY 1 1967...BUT
ALSO ESTABLISHES A NEW ALL TIME RECORD DAILY SNOWFALL FOR THE MONTH
OF FEBRUARY FOR CHICAGO. THE PREVIOUS CALENDAR DAY SNOWFALL RECORD
FOR CHICAGO FOR FEBRUARY WAS 11.5 INCHES SET BACK ON FEBRUARY 18
1908.


More water vapor available for more snow and ice.
Member Since: January 2, 2006 Posts: 127 Comments: 20470
87. HaloReachFan
2:42 PM GMT on February 02, 2011
Looks like more negative temperatures on this graph than positive temperatures. I'm not going to count but I'm sure its close.
Member Since: September 15, 2010 Posts: 1 Comments: 563
86. HaloReachFan
2:41 PM GMT on February 02, 2011
Member Since: September 15, 2010 Posts: 1 Comments: 563
85. iceagecoming
2:27 PM GMT on February 02, 2011
Link

Only just Feb.?
Member Since: January 27, 2009 Posts: 26 Comments: 1096
84. iceagecoming
1:55 PM GMT on February 02, 2011
Link

Brrr!

Glad I use oil to heat with and a generator.
Member Since: January 27, 2009 Posts: 26 Comments: 1096
83. Patrap
1:29 PM GMT on February 02, 2011
Yeah,,I want to see that here one day,,but we had the last Freeze of note way back Xmas 89.
Member Since: July 3, 2005 Posts: 428 Comments: 129906
80. cyclonebuster
3:52 AM GMT on February 02, 2011
Quoting HaloReachFan:
If you really want to reduce your carbon footprint on this planet then you'll quit posting on here and supporting the making of computers. I think you took one to many hits from the pipe back in the day.


LOL! Jack Wagon here thinks I am doing nothing to solve the problem. Evidentally he knows there is a problem if we need to reduce our footprint.









Member Since: January 2, 2006 Posts: 127 Comments: 20470
79. Patrap
3:40 AM GMT on February 02, 2011
Dats the way it goes..

Calamity misses one City and Bam..

We drown the Pigs..

Squeal,Squeal,Squeal...



Maybe next time.
Member Since: July 3, 2005 Posts: 428 Comments: 129906
75. HaloReachFan
2:22 AM GMT on February 02, 2011
Quoting cyclonebuster:
Quoting cyclonebuster:


Polar Bears are more fit than Man even their cubs are!



That's why we have advanced our lifestyle and they haven't. LOL


You call this advanced?





I didn't do that to them. How bout the billions of animals that aren't affected by us. You don't show them. LOL. Your one sided opinions are humorous.
Member Since: September 15, 2010 Posts: 1 Comments: 563
74. cyclonebuster
2:15 AM GMT on February 02, 2011
Quoting cyclonebuster:


Polar Bears are more fit than Man even their cubs are!



That's why we have advanced our lifestyle and they haven't. LOL


You call this advanced?



Member Since: January 2, 2006 Posts: 127 Comments: 20470
72. HaloReachFan
2:10 AM GMT on February 02, 2011
Quoting cyclonebuster:


Polar Bears are more fit than Man even their cubs are!


That's why we have advanced our lifestyle and they haven't. LOL
Member Since: September 15, 2010 Posts: 1 Comments: 563
71. cyclonebuster
1:56 AM GMT on February 02, 2011
Quoting HaloReachFan:


In your personal opinion.


Polar Bears are more fit than Man even their cubs are!
Member Since: January 2, 2006 Posts: 127 Comments: 20470
70. HaloReachFan
1:38 AM GMT on February 02, 2011
Quoting cyclonebuster:


Neanderthal Man was more fit than man!


In your personal opinion.
Member Since: September 15, 2010 Posts: 1 Comments: 563
69. cyclonebuster
1:36 AM GMT on February 02, 2011
"ONLY YOU CAN REVERSE GLOBAL WARMING"

Member Since: January 2, 2006 Posts: 127 Comments: 20470
68. cyclonebuster
1:32 AM GMT on February 02, 2011
Quoting HaloReachFan:


Survival of the Fittest?


Neanderthal Man was more fit than man!
Member Since: January 2, 2006 Posts: 127 Comments: 20470
67. HaloReachFan
1:28 AM GMT on February 02, 2011
Quoting JFLORIDA:
Polar bear swam 426 miles to find ice in Arctic seas (LA Times)

Its cub didn't survive the swim.


Survival of the Fittest?
Member Since: September 15, 2010 Posts: 1 Comments: 563
66. cyclonebuster
12:42 AM GMT on February 02, 2011
Quoting JFLORIDA:
Polar bear swam 426 miles to find ice in Arctic seas (LA Times)

Its cub didn't survive the swim.


"ONLY YOU CAN PREVENT GLOBAL WARMING"

Member Since: January 2, 2006 Posts: 127 Comments: 20470
64. iceagecoming
9:33 PM GMT on February 01, 2011
# Prepared by the U.S. Dept. of Commerce, NOAA, Space Weather Prediction Center
# Please send comments and suggestions to SWPC.Webmaster@noaa.gov
#
# Last 30 Days Daily Solar Data
#
# Sunspot Stanford GOES15
# Radio SESC Area Solar X-Ray ------ Flares ------
# Flux Sunspot 10E-6 New Mean Bkgd X-Ray Optical
# Date 10.7cm Number Hemis. Regions Field Flux C M X S 1 2 3
#---------------------------------------------------------------------------
2011 01 02 91 38 260 0 -999 A8.8 0 0 0 0 0 0 0
2011 01 03 92 51 320 0 -999 A8.7 1 0 0 1 0 0 0
2011 01 04 91 54 360 0 -999 B1.0 1 0 0 0 0 0 0
2011 01 05 88 50 250 0 -999 A8.4 0 0 0 0 0 0 0
2011 01 06 87 28 230 0 -999 A7.4 0 0 0 0 0 0 0
2011 01 07 86 52 170 1 -999 A7.7 0 0 0 0 0 0 0
2011 01 08 85 52 180 0 -999 A7.2 0 0 0 0 0 0 0
2011 01 09 83 50 150 2 -999 A7.2 0 0 0 0 0 0 0
2011 01 10 83 35 140 1 -999 A7.2 0 0 0 0 0 0 0
2011 01 11 83 26 130 0 -999 A6.5 0 0 0 0 0 0 0
2011 01 12 80 23 80 0 -999 A6.8 0 0 0 0 0 0 0
2011 01 13 80 14 10 0 -999 A7.3 0 0 0 0 0 0 0
2011 01 14 79 11 0 0 -999 A9.3 2 0 0 0 0 0 0
2011 01 15 80 11 120 1 -999 B1.0 1 0 0 0 0 0 0
2011 01 16 80 15 120 0 -999 A7.3 0 0 0 0 0 0 0
2011 01 17 82 36 170 1 -999 A9.0 0 0 0 0 0 0 0
2011 01 18 81 34 190 0 -999 A5.7 0 0 0 0 0 0 0
2011 01 19 81 28 140 0 -999 A5.7 0 0 0 0 0 0 0
2011 01 20 82 32 190 0 -999 A6.8 0 0 0 0 0 0 0
2011 01 21 88 42 160 1 -999 B1.1 7 0 0 3 0 0 0
2011 01 22 88 36 240 0 -999 B1.0 1 0 0 0 0 0 0
2011 01 23 84 38 260 0 -999 A7.7 0 0 0 0 0 0 0
2011 01 24 83 28 270 0 -999 A6.4 1 0 0 0 0 0 0
2011 01 25 81 27 230 0 -999 A5.7 0 0 0 0 0 0 0
2011 01 26 80 25 400 0 -999 A6.1 0 0 0 1 0 0 0
2011 01 27 81 0 0 0 -999 A7.2 2 0 0 0 0 0 0
2011 01 28 81 27 30 2 -999 A9.3 2 1 0 2 0 0 0
2011 01 29 81 27 25 0 -999 A6.1 0 0 0 1 0 0 0
2011 01 30 83 22 50 0 -999 A5.3 0 0 0 0 0 0 0
2011 01 31 81 21 70 0 -999 A5.8 0 0 0 0 0 0 0

These sunspot numbers are really going to destroy the Nasa predictions, what next, science review, admit don't know how it
works? Interesting data at the very least.
Member Since: January 27, 2009 Posts: 26 Comments: 1096
63. atmoaggie
8:01 PM GMT on February 01, 2011
Quoting JFLORIDA:
ECMWF is a public supported and funded yet private company that sells forecasts ???
I *think* it's free within the EU...where the taxes that support it are collected...
Member Since: August 16, 2007 Posts: 6 Comments: 12463
62. atmoaggie
8:00 PM GMT on February 01, 2011
Quoting MichaelSTL:


LOL, Spencer thinks that Gore is a SCIENTIST?!?!

That should tell you all you need to know about Spencer!
?
Quoting martinitony:
He has caused the spread of more pseudo-scientific incompetence on the subject of global warming (I’m sorry — climate change) than any climate scientist could possibly have ever accomplished.
Is the sentence really that unclear?
Member Since: August 16, 2007 Posts: 6 Comments: 12463
58. martinitony
4:36 PM GMT on February 01, 2011
Quoting McBill:

As usual, wrong again, Mr. Martini.

Hot Arctic-Cold Continents



Yeah, you're right McBill. Surface and lower atmospheric temperatures are not really falling and sea surface temperatures are not really falling either. The Earth is getting warming by the minute and all that flooding, which has never happened before, is caused by global warming, manmade global warming.Those winter storms roaring across the plains was caused by global warming as were the snows in Central Park and Great Britain. Thanks for straightening me out.
Member Since: July 29, 2009 Posts: 0 Comments: 970
56. martinitony
4:09 PM GMT on February 01, 2011
Quoting MichaelSTL:


LOL, Spencer thinks that Gore is a SCIENTIST?!?!

That should tell you all you need to know about Spencer!


Michael you really prove that you don't know how to read.
!. "He" refers to Al Gore.
2.The reference to "climate scientist" is not a reference to gore but instead to all climate scientists.
In other words, Gore has spread more misinformation than any in those that are climate scientists has spread misinformation.
Spencer's words in no way whatsoever imply Gore is a scientist.
From the article "Who else but a politician.." is a clear reference to Gore.
Do you now understand, Michael?
Member Since: July 29, 2009 Posts: 0 Comments: 970
51. iceagecoming
2:27 PM GMT on February 01, 2011
Link

COLUMBIA, Mo. (AP) — A major winter storm expected to bring snow and ice, followed by bitter cold, has its eye on one third of the U.S. — from the Rockies to New England.

Looks like warm is a relative word.
(Sound of Chattering teeth)
Member Since: January 27, 2009 Posts: 26 Comments: 1096
Roy Spencer's blog last night:


I really can’t decide whether I should hate Al Gore… or thank him for giving me something to write about.

He has caused the spread of more pseudo-scientific incompetence on the subject of global warming (I’m sorry — climate change) than any climate scientist could possibly have ever accomplished. Who else but a politician could spin so much certainty out of a theory?

As someone who has lived and breathed meteorology and climate for 40 years now, I can assure you that this winter’s storminess in the little 2% patch of the Earth we like to call the ‘United States of America’ has nothing to do with your SUV.

Natural climate variability? Maybe.

But I would more likely chalk it up to something we used to call “WEATHER”.

Let me give you a few factoids:

1) No serious climate researcher — including the ones I disagree with — believes global warming can cause colder weather. Unless they have become delusional as a result of some sort of mental illness. One of the hallmarks of global warming theory is LESS extratropical cyclone activity — not more.

2) If some small region of the Earth is experiencing unusually persistent storminess, you can bet some other region is experiencing unusually quiet weather. You see, in the winter we get these things called ’storm tracks’….

3) Evidence for point #2 is that we now have many years of global satellite measurements of precipitation which shows that the annual amount of precipitation that falls on the Earth stays remarkably constant from year to year. The AREAS where it occurs just happen to move around a whole lot. Again, we used to call that “weather”.

4) Global average temperature anomalies (departures from seasonal norms) have been falling precipitously for about 12 months now. Gee, maybe these snowstorms are from global cooling! Someone should look into that! (I know…cold and snow from global cooling sounds crazy….I’m just sayin’….)

I could go on and on.

Now, I know I’m not going to change the minds of any of the True Believers…those who read all of Reverend Al’s sermons, and say things like, “You know, global warming can mean warmer OR colder, wetter OR drier, cloudier OR sunnier, windier OR calmer, …”. Can I get an ‘amen’??

But I hope I can still save a few of those out there who are still capable of independent reasoning and thought.

NOW can I go to bed?
Member Since: July 29, 2009 Posts: 0 Comments: 970
Quoting cyclonebuster:
What ya'll doing to reverse global warming?


Holding my breathe longer in between breaths.
Member Since: September 15, 2010 Posts: 1 Comments: 563
Quoting NRAamy:
I stopped eating beans....

:)


Don't worry Amy the Tunnels will allow YOU to eat them again!
Member Since: January 2, 2006 Posts: 127 Comments: 20470
I stopped eating beans....

:)
Member Since: Posts: Comments:
"ONLY YOU CAN PREVENT GLOBAL WARMING"



Member Since: January 2, 2006 Posts: 127 Comments: 20470
I don't know what it is but I know what it does!
Member Since: January 2, 2006 Posts: 127 Comments: 20470
"ONLY YOU CAN REVERSE GLOBAL WARMING"
Member Since: January 2, 2006 Posts: 127 Comments: 20470
What ya'll doing to reverse global warming?
Member Since: January 2, 2006 Posts: 127 Comments: 20470
Quoting idontknowforsure:


That won't be necessary Mr. Buster,
Global Cooling


It is if you don't want to commit suicide! You will see how the tunnels work one day I promise!
Member Since: January 2, 2006 Posts: 127 Comments: 20470
Quoting cyclonebuster:


I am going to start a corporation called "Reverse Global Warming Now" I may visit TES again soon.


That won't be necessary Mr. Buster,
Global Cooling
Member Since: Posts: Comments:

Viewing: 91 - 41

Page: 1 | 2 | 3Blog Index

Top of Page

About RickyRood

I'm a professor at U Michigan and lead a course on climate change problem solving. These articles often come from and contribute to the course.

RickyRood's Recent Photos

Clouds in the lee of the Rockies at sunset.
Clouds in the lee of the Rockies at sunset.
Clouds in the lee of the Rockies at sunset.
Clouds in the lee of the Rockies at sunset.