Validation and the Scientific Organization: Organizing U.S. Climate Modeling (4)
Validation and the Scientific Organization: Organizing U.S. Climate Modeling (4)
This entry is the last I will be writing about organizing U.S. climate modeling, software, and open source communities â for a while. At the end of this entry are links to the blogs/articles in a couple of series. I am going to start by quoting a comment from atmoaggie on the previous entry.
âThe difference between all of those (rbr: types of models in a previous comment) and climate models is the ability to study their validity.
I would like to see a climate modeling 10 year forecast of some parameters, such as, maybe, average SST for the month of June 2021. Too specific? How about average global SST for JJA (summer) 2021. Still too specific? Maybe the average global SST for the next 10 years.
I, too, work in modeling. In storm surge modeling, one can very easily tune a model to better match the results for one storm (by adjusting air-sea drag, e.g.) only to find that the model is not useful for forecasting as another parameter or physical calculation is incorrect (the sea floor friction formulation, e.g.)
I bring this up to illustrate what can go wrong when modeling a hindcast, tuning to match observations, and applying that model to forecasts. And climate is far more complex, I think, than tides and TC wind and pressure-forced storm surges.â
I want to bring together two streams of thought that I have pursued over the past few months â validation and the scientific organization. First, I will discuss whether or not climate models can be validated and then argue that the development of a validation plan is at the center of developing a scientific organization.
Validation: As suggested in some of my earlier entries the question about whether or not climate models can be validated is a controversial issue. The controversy lies, first, in philosophy. The formal discussion of whether or not climate models can or cannot be validated often starts with a greatly cited paper by Naomi Oreskes et al. entitled Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. In fact quoting the first two sentences in the abstract:
âVerification and validation of numerical models of natural systems is impossible. This is because natural systems are never closed and because model results are always nonunique.â
Oreskes et al. argues that the performance of the models can be âconfirmedâ by comparison with observations. However, if the metric of âvalidationâ is a measure of absolute truth, then such absolute validation is not possible. By such a definition little of the science of complex systems, which would include most biological science, medical science, and nuclear weapons management, can stand up to formal validation.
I will return to the stream I started with the quote from atmoaggie, which makes reference to storm surge model (see here for excellent discussion of storm surges Resio and Westerink). The point of the comment is that the storm surge model can be tuned and thereby calibrated based on observations of past storm surges and theory, but the model may still fail in future predictions of storm surges. This points out a weakness in the development of models of natural systems, that the adjustments of the models to represent a historical situation does not assure that model correctly represents the physics of cause and effect. In fact, this is a general problem with modeling of complex natural systems, if you get the answer âright,â then that does not mean you get it right for the right reason. Hence, in the spirit of Oreskes et al. validation is not possible â there is no absolute to be had.
Yet, arenât storm surge models useful and usable? The same situation is true for weather models and river forecast models, their correctness cannot be assured in any absolute sense, but arenât they useful and usable? Atmoaggie poses a set of predictions, all of which are reasonable propositions, that may or may not be convincing to him or her. These do not represent a complete set of metrics to evaluate models, and the success or failure of these predictions does not state in any absolute sense whether or not the models have usable information. There are many more elements of model evaluation that determine our level of confidence in the use of models.
It is easy, therefore, to establish that models that cannot be formally validated can be both useful and usable. The results of these models might not be certain, but the degree of confidence that can be attributed to their calculations is very high. This confidence is, in general, established by many forms of model evaluation and additional sources of relevant information, most importantly, observations and basic physical principles.
Validation, verification, evaluation, certification, confirmation, calibration: All of the words in this list have been used in discussions of how to assess the quality of models. For some, there are nuanced differences between the words, but in the general discussion they are all likely to take on the same meaning â some quantitative measure of model quality. The word âvalidation;â however, is special. Within political or philosophical arguments, the statement âmodels cannot be validated,â carries a powerful message, especially if one establishes as a principle that the elimination or the reduction of uncertainty is required prior to taking action (see Shearer and Rood). Many scientists take on the mantra that climate models cannot be validated. When I worked at NASA, the culture was that measurements of temperature (for example) could be validated, but that models could not. But if one is talking about temperatures from satellites over a deep layer of the atmosphere, in the spirit of Oreskes et al., can satellite temperature measurements be validated? We can state with stunning confidence that the satellite temperatures are within a certain closeness of a more intuitive or accepted measure of temperature â like a thermometer on a balloon. This is, to me, more calibration than validation, but in my world at NASA, calibration was done in a lab with standards (and that is why we have NIST). At NASA we talked about models being âevaluated.â
Other arguments I have heard about climate modes defying validation are based on to what do we chose to validate against â what is our standard? Suppose that you are interested in how well the model represents the Pacific Ocean, and I am interested in how well it represents the Arctic Ocean. And the scientist down the hall wants to know how well it represents the ice-age cycles, and another wants to know how well it represents the 20th century temperature variability. There is no absolute way to make these choices. More fundamentally, if it is a climate model then how do we measure âclimate?â
The list goes on â I have frequently heard arguments of one community making critical remarks about the âscienceâ of other communities. The weather forecast community relies strongly on forecast skill scores, but these measures are by no means unique and for a variety of reasons often only indirectly relevant to the quality of climate models. There is no fundamental reason that an excellent climate model would automatically be an excellent weather forecasting model. The opposite is true as well. Over the years of my career there have been criticisms of climate science by other fields of physics. The gist of their arguments is that they donât validate models the same way we do, and since we do a good job, they donât. These arguments make great fuel for political argument and the maintenance of doubt. (Here is an interesting article by Oreskes and Renouf.)
Validation is, therefore, both controversial and important. I pose that validation is at the center of the development of the scientific organization.
Validation and the Scientific Organization: The definition I have posed for the scientific organization is an organization that as a whole functions according to the scientific method. Therefore, if it is a climate modeling organization the model development path, the modeling problems that are being addressed, are determined in a unified way. In that determination, it is required that ways to measure success be identified. This leads to a strategy of evaluation that is determined prior to the development and implementation of model software. With the existence of an evaluation strategy, a group of scientists who are independent of the developers can be formed to serve as the evaluation team.
The development of an evaluation plan requires that a fundamental question be asked? What is the purpose of the model development? What is the application? If the model is being developed to do âscience,â then there is no real constraint that balances the interests of one scientific problem versus another. There is little or no way to set up a ladder of priorities.
Again, I will emphasize that to achieve this, and it can be achieved, is a matter of governance and management. It is a process of developing organizational rather than individual goals. It is a myth to imagine that if a group of individuals are each making the âbestâ scientific decisions, the accumulation of their activities will be the best integrated science. Science and scientists are not immune to the The Tragedy of the Commons. If one wants to achieve scientifically robust results from a unified body of knowledge, then one needs to manage the components of that body of knowledge so that as a whole the scientific method is honored. Enough on that pulpit.
Back to evaluation and validation â Minimally, the arguments about the nuanced meaning of validation and evaluation are a subject about which the climate modeling community needs to develop a standard. By my interpretation, the evaluation of climate models can be structured and quantified as âvalidation.â
When I was at NASA I had a programmatic requirement to develop a validation plan. And, yes, my friends and colleagues would tell me that that validation was âimpossible.â But I am stubborn, and not so smart, so I persisted and still persist with the notion. That old plan can still be found here in Algorithm Theoretical Basis Document for Goddard Earth Observing System Data Assimilation System (GEOS DAS) with a Focus on Version 2.
The software we produced was an amalgam of weather forecasting and climate modeling. For the validation plan the strategy was taken to define a quantitative baseline of model performance for a set of geophysical phenomena. These phenomena were broadly studied and simulated well enough that they described a credibility threshold for system performance. They were chosen to represent the climate system. Important aspects of this validation approach were that it is defined by a specific suite of phenomena, formally separated validation from development, and relied on both quantitative and qualitative analysis.
The validation plan separated "scientific" validation from "systems" validation. It included steps of routine point-by-point monitoring of simulation and observations, formal measures of quality assessment by measure of fit of simulations and observations, and calculation of skill scores to a set of "established forecasts." There was a melding of methodologies of practices of the study of weather and the study of climate. We distinguished the attributes of the scientific validation from the systems validation. The systems validation, focused on the credibility threshold described above, used simulations that were of longer time scales than the established forecasts and brought attention to a wider range of variables important to climate. The scientific validation was a more open-ended process, often requiring novel scientific investigation of new problems. The modeling software system was released for scientific validation and use after a successful systems validation.
The end result of this process was the quantitative description of the modeling system against a standard set of measures over the course of one modeling release to the next. Did it meet the criterion of the absolute validation? No. Did it provide a defensible quantitative foundation for scientific software and its application? Yes.
All told, it does little to base a body of scientific knowledge on the premise that validation is âimpossible.â Rather than following such a premise, which immediately devalues the knowledge base, it is more useful to develop a systematic approach to robust, appropriate validation. This stands to represent the complexity of the Earthâs climate and its investigation that serves not only the scientific method, but the communication of that science to other scientists, and to those with a stake in those scientific results. It sets a standard.
r
Open Climate Modeling:
Greening of the Desert
Stickiness and Climate Models
Open Source Communities, What are the Problems?
A Culture of Checking
Organizing U.S. Climate Modeling:
Something New in the Past Decade?
The Scientific Organization
A Science-Organized Community
Validation and the Scientific Organization
Reader Comments
Page: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 — Blog Index
I wouldn't trust anything from UEA, NOAA, NASA or the like. I'm really sick of seeing that completely fraudulent "hockey schtick" graph. Everyone knows it's wrong yet people keep using it as a reference. Even though every model the warmists produced has been proven to be wrong, they keep hyping them. Even if CO2 were at double the concentration it is now, no harm would befall the planet. In fact, it would probably help crop growth considering we are about to enter another ice age.
Refuseniks commonly trash our government. Are you one of them?
I trash lies and incompetent science benefiting the elite. I don't trust government. I spent part of my younger life serving this country. Have you?
I work for the Army now.
How many chemical weapons have you disposed of in your life time?
Aw come on, that's not fair! We ALL trash the government!
:)
Really now!
Ain't Ready for Marines Yet. What do chemical weapons have to do with trashing the government? You work for the Army or you're in the Army? I don't think there is a plethora of chemical weapons here in the states.
You asked question. I answered it.
that's different.....
:)
You correct. We thermally disposed most of the old ones.The new ones are a different story.
Link
Evidently you can not read the charts I posted from NOAA about how much it has warmed in the upper GLOBAL layers of the oceans and the atmosphere. COMPRENDE?
Evidently many scientists disagree with you. Even some from NOAA are beginning to doubt the veracity of the surface temperature stations.
Global Warming Standstill Confirmed - But How Long Will It Last?
Monday, 04 July 2011 19:10 Dr. David Whitehouse
It is good news that the authors of the PNAS paper Reconciling anthropogenic climate change with observed temperature 1998 %u2013 2008 (Kaufmann et al. 2011) recognise that there has been no global temperature increase since 1998. Even after the standstill appears time and again in peer-reviewed scientific studies, many commentators still deny its reality. We live in the warmest decade since thermometer records began about 150 years ago, but it hasn%u2019t gotten any warmer for at least a decade.
The researchers tweak an out-of-date climate computer model and cherry-pick the outcome to get their desired result. They do not use the latest data on the sun%u2019s influence on the Earth, rendering their results of academic interest only.
They blame China%u2019s increasing coal consumption that they say is adding particles into the atmosphere that reflect sunlight and therefore cool the planet. The effect of aerosols and their interplay with other agents of combustion is a major uncertainty in climate models. Moreover, despite China%u2019s coal burning, data indicate that in the past decade the amount of aerosols in the atmosphere has not increased.
The researchers seek to explain the temperature standstill between 1998 and 2008. They say that the global temperature has increased since then.
This is misleading. There was an El Nino in 2010 (natural cyclic warming) but even that did not raise temperatures above 1998. In fact the standstill has continued to 2010 and 2011 appears to be on course to be a cooler year than any of the preceding ten years.
Tweaking computer models like this proves nothing. The real test is in the real world data. The temperature hasn%u2019t increased for over a decade. For there to be any faith in the underlying scientific assumptions the world has to start warming soon, at an enhanced rate to compensate for it being held back for a decade.
Despite what the authors of this paper state after their tinkering with an out-of-date climate computer model, there is as yet no convincing explanation for the global temperature standstill of the past decade.
Either man-made and natural climatic effects have conspired to completely offset the warming that should have occurred due to greenhouse gasses in the past decade, or our estimation of the %u2018climate sensitivity%u2019 to greenhouse gasses is too large.
This is not an extreme or %u2018sceptic%u2019 position but represents part of the diversity of scientific opinion presented to the IPCC that is seldom reported.
e-mail: david.whitehouse@thegwpf.org
Tell that to the Arctic Ice! It won't listen.
Wait--you state "Even some from NOAA are beginning to doubt the veracity of the surface temperature stations", but then quote an article written by the clueless David Whitehouse* of the ultra-denialist "Global Warming Policy Foundation" (an organization that refuses to disclose its funding source[s] as to do so would be "controversial") that talks bout last week's Kaufmann paper. So, since Whitehouse's article failed to say anything about the "some from NOAA" who "are beginning to doubt the veracity of the surface temperature stations", would you mind please sharing any information you might have?
* - Yes, clueless. Here's his prediction for 2010: "2010 will be remembered for just two warm months [March and June], attributable to the El Nino effect, with the rest of the year being nothing but average, or less than average temperature." Of course, the year went on to be the hottest ever. IOW, clueless.
Ah, a little swift-boating going on, I see. Don't believe everything you hear on Fox or read on WUWT, my friend.
Hansen has never received a cent from George Soros.
The amount Michael Mann is alleged to have received varies depending on which denialist blog you read; I've seen figures that range from $450,000 up to $3 million. I'd say that's a little over exaggerated. His school--Penn State--and the University of Hawaii shared a grant of $770,000 for a project called "Improved Projections of the Climate Response to Anthropogenic Forcing: Combining Paleoclimate Proxy and Instrumental Observations with an Earth System Model". Of that money, Mann received $57,000 (or 7.5%) over a three-year period, which works out to $19,000 a year. Even without a calculator, I can see that's somewhat less than the $1 million
per yearsomeone like Willie Soonis paid by ExxonMobil, et al.has made over the past decade in fossil fuel research grants alone.So, from more than $3.2 million to $19K a year. I wonder: does Glenn Beck ever feel the shame that he should?
(The funniest thing is, the denialist who made the baseless claim against Mann, Tom Borelli, was the manager of the Philip Morris Corporate Scientific Affairs department in the 1990s--that is, right at the heart of the "Cigarettes are GOOD for you!!!" campaign. I guess his anti-science stance then made him a great fit for Bug Energy vis-a-vis global warming.)
perfect for when we live in a Water World.....
Here you go. Read it and weep.
Link
IBD did, indeed, print the original Hansen story. They eventually issued a correction, but not before--surprise--NewsBusters or WND or one of those other whack-job sites picked up the ball and ran away with it.
From what I can tell, the WSJ was not the originator of the Mann story, though they did print it (it is, after all, a Rupert Murdoch publication, and we know how high those standards are). But regardless, it's not true as you've repeated that Mann received $2.5 million in stimulus funds. Please read thoroughly my previous comment.
As with the aforementioned "swift boating", repetition of a lie doesn't make it true.
My real concern is focused on those who are setting policy through their globally published information.
Conflict of interest guidelines for the IPCC
by Judith Curry
My bad; it was late (for me) and I was tired, so I inadvertently posted an error. Willie Soon--officially, at least--doesn't take in a million a year. (I've corrected the original comment.) He has, on the other hand, brought in $1 million in grants alone over the past decade...nearly every single cent of it from the fossil fuel industry. That fact alone should certainly call his climate change credibility into question--and, indeed, it does. (And, of course, Soon also makes tons more speaking on behalf of numerous denialist groups.)
Now, NASA James E. Hansen--again--has never received a cent from George Soros. Yes, he has a salary. I'm not sure what it is, but $180K for a degreed scientist (BA, MS, PhD) with nearly 50 years of experience sounds about right, if not a little low.
If people care to engage in not just smearing Hansen with Glenn Beck fantasies about his taking money from Soros, or by confusing a normal science salary with money earned from polluters, they should go right ahead. None of that changes the science; none of that changes the facts: the planet is warming, and warming rapidly.
So thousands of earth scientists from around the globe--even ones from nations unfriendly to each other--are all involved in a vast decades-long conspiracy to defraud the common people? And as part of their global fraud, they've managed to secretly communicate with one another for decades without a single person slipping up and exposing their scam for what it is? And as part of that same plot, they've even managed to convince the oceans to acidify, glaciers and Arctic ice to melt, precipitation patterns to change, and the very seasons to change their start and end dates? And they've managed to do all this on government salaries, while at the same time avoiding the urge to grab any of the billions in Big Energy money offered to those who'll jump ship?
That's quite a tale, I have to tell you. ;-)
I suppose folks can believe that if they wish. Of course, it makes much more sense that the people making hundreds of billions in profits off the current fossil fuel paradigm would spare no expense and effort to maintain that paradigm as long as possible (a paradigm which, by the way, "redistributes wealth" to a relative few at the top). But, hey, people can believe the former if they wish. Who am I to stop them? ;-)
Enter political slogans and themes and the credibility of one's arguments seem biasedly generic and unoriginal.
Ooops! Looks like you got hold of some bad science there.
1) The following self-explanatory trio of charts --which only goes through 2003--is from a paper authored by Polyakov:
This quote is from another Polyakov paper: "The composite temperature record shows that since 1875 the Arctic has warmed by 1.2C, so that over the entire record the warming trend was 0.094C/decade."
2) On the second part, Grudd is furious over the misuse and distortion of his work. 'Nuff said.
It's warm, and getting warmer.
Man has changed the Earth's seasonsal change start and end dates? Really? So the equinoxes and solstices of the Sun/Earth have somehow been compensated by Big Energy corporations on a planet 93 million miles away? That's astounding! So how did 40 years of co2 increase effect the Earth's rotation around the sun...and at ununiform intervals?
The spool is being unraveled. AGW theory is no more a plight to slow corporate profits as it is a legitimate threat to the world. Science is being used as a weapon once again. this time, its an economic war.
What axe? please explain.
When we stabilize? That's quite a fantasy. But I'll bite: as I've said before, if by some reversal of the known laws of physics the oceans and air miraculously stopped heating from CO2--and that "stabilization" wasn't due to, say, a large impact event, global thermonuclear war, a supervolcano eruption, or some nation's ill-advised decision to intentionally flood the atmosphere with particulate matter to offset the CO2 warming--I will be at the front of the line of those saying, "I was wrong, and I humbly apologize."
But that's not going to happen, of course. I know it. You know it. And, far more importantly, scientists know it. And much as some might constantly and hopefully repeat the mantra that "none are sure it's mad-made", most scientists definitely are sure--and they're just as sure it's getting worse.
It's warm, and getting warmer.
What term would you prefer? "Concerted effort to smear a credible scientist by continued repetition of a bald-faced lie" just seemed too long. But I'll tell you what; if you're truly concerned about rhetoric, I'll make an effort to tone down when you cease writing things like "climate hysteria" (#292), "madness" and "hatred for oil and coal corporations" (#294), "forged disaster" (#256), "spouting off about denialism and ignorance" and "universal healthcare" (#240). How about it? ;-)
Just how is an expensive abrupt change in energy consumption going to stop a planet from warming?
This globe could be warming for another 25 thousand years.
What the heck is the goal of climate science anyway?
I just don't get it.
I mean, I know its all about money and scientists need to eat too, but seriously, if you want to talk about being swift-boated, think about those Big Energy corps that have spent billions of dollars over the last few decades on improving their product and production since 1979.
The same scene sort of played out in the early 20th century between private U.S. oil prospectors and the masters of transit, the railroad mega-tycoons. There is no way railroad firms were going to bow down to a new energy concept, they were just going to acquire the entire supply and reap the profits.
Eventually the private oil man was squeezed to oblivion by Big Railroad. The reason was solely to monopolize and dominate.
The exact same thing is happening today. Cripple and Assimilate. Welcome to Gengis Khan Wind & Solar Enterprises, Inc. Homebase: Dover, Deleware
we have plenty of Big Brothers right here on this board. And if they can't change the past, they most certainly will assassinate the messengers of truth.
I can go either way. What ever is more fun. I actually enjoy the current rhetoric, wit, sarcasm and comedy et al. But I will tone it down, for you and you only, if you think our discourse will continue to be intellectual and entertaining.
Railroad had its money well invested in Capitol Hill
only years later would Congress dispatch Standard Oil's monopoly
but law ruled after the heavily lobbied railroads completed their goals.
if its true history repeats itself, we'll only know the truth of AGW after the well invested, heavy lobbied anti-corporate environmentalists have destroyed the petroleum industry.
laws will rule after the goals are met.
Here is a question for some of you. Where did the majority of the missing Arctic Sea ice actually melt?
A hint for you :)
Link
You must have either read a different comment than the one you noted, or you committed a Freudian slip of sorts; comment 339 doesn't say a thing about either corporations or the wealthy. Can you please highlight which comment you intended? Thanks so much...
Viewing: 301 - 351
Page: 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 — Blog Index