John Holdren is one of President Obama’s science advisors. In 2008, Pulitzer Prize-winning journalist Thomas Friedman quoted Holdren in his best-selling book Hot, Flat and Crowded. The topic was the Intergovernmental Panel on Climate Change (IPCC):
The most important conclusions about global climatic disruption.have not been concocted by the Sierra Club or the enemies of capitalism. They are based on an immense edifice of painstaking studies published in the world’s leading peer-reviewed scientific journals. They have been vetted and documented in excruciating detail by the largest, longest, costliest, most international, most interdisciplinary, and most thorough formal review of a scientific topic ever conducted.
Among those who believe humans are responsible for dangerous climate change, this quote is treated as the silver bullet that settles everything. (See, for example, the end of this blog post, the letter-to-the-editor headlined “Scientific evidence is plentiful” here, and this Presbyterian church sermon.)
This quote also turns up in some disturbingly official places. For example, a 2009 report by the Economics Committee of the Australian Senate quoted Friedman quoting Holdren. So did a 2008 report produced by the state of Washington’s Department of Ecology (see pages 13-14 of this 95-page PDF). A 2010 article published in the Wake Forest Law Review also begins by quoting Friedman quoting Holdren.
But this quote has as many holes as my colander.
PROBLEMS #1, 2, and 3
The IPCC’s conclusions are not based solely on painstaking studies. Nor is the IPCC’s source material all published in peer-reviewed journals. In fact, 30% of the references in the IPCC’s 2007 report (5,587 out of 18,531) are to grey literature – including press releases, magazine articles, and advocacy material produced by groups such as Greenpeace. Furthermore, the genuinely peer-reviewed material cited by the IPCC amounts to several thousand studies. Simple arithmetic indicates that only a fraction could possibly have appeared in the world’s leading journals.
PROBLEM #4
Holdren describes the IPCC process as the costliest review of a scientific topic ever conducted. How can this be true when we are told again and again that IPCC authors work for free, on a volunteer basis?
PROBLEM #5
Holdren says the IPCC has vetted.in excruciating detail the evidence on which its conclusions rest. But that’s not what IPCC insiders said when they answered a questionnaire last year. These people were asked for their views concerning how the IPCC addresses data quality. Here are some of their answers (all bolding added by me):
As far as I can tell, there is no data quality assurance associated with what the IPCC is doing. (p. 99)
This is probably the weakest part of IPCC˜s process. Since the IPCC is a review body, it does not do data assurance or quality control in a systematic fashion. (p. 52)
it was of course impossible to check in detail the reliability of all the information in the vast amount of underlying sources. (p. 92)
Quality assurance and error identification is not existent. (p. 384)
Data quality assurance, per se, is beyond the scope of the work of the IPCC. (p. 203)
The substantial working time is so short and schedule is always behind, and many sentences inserted last moment cannot be well examined. That is the reality. (p. 265)
PROBLEM #6
If the IPCC is making no effort to ensure that the scientific studies it cites are valid, who is? Holdren implies that the mere fact that a study has been published in one of the world’s leading peer-reviewed scientific journals means its conclusions are sound. This assumption appears to be shared by some of those involved in the IPCC. In the answers to this same questionnaire we find the following remarks:
I don’t think IPCC can be held responsible for data quality etc in the published papers it quotes – that is an issue for the journals concerned. (p. 660)
I am not sure data quality control and assurance is an IPCC issue in the first instance. We have a peer-review scientific process for that. (p. 362)
Re [quality assurance/quality control, this] is handled by the peer-review / publication process and it should not be IPCC’s responsibility. (p. 20)
IPCC does not do research, it reviews it, so it is expected that a paper published in an important journal did the analyses of data quality, quality assurance, etc. (p. 58)
Other IPCC insiders, however, recognize that this approach is problematic. There are hundreds (perhaps thousands) of academic journals out there, each with its own way of doing things. Since there is no accreditation process for journals, how smart is it to blindly trust that just because a paper got published means that it has been thoroughly vetted? As one insider observes:
a normal scientific paper put into the journal literature as part of the ongoing intellectual conversation is not necessarily written to the same level of rigour required to justify the weight of an IPCC major conclusion. (p. 332)
At this juncture it’s worth recalling that, when senior climate scientist Phil Jones was asked about the level of scrutiny his work receives prior to being published by some of the world’s leading peer-reviewed scientific journals, even the pro-environment UK Guardian newspaper was shocked:
The most startling observation came when [Jones] was asked how often scientists reviewing his papers for probity before publication asked to see details of his raw data, methodology and computer codes. “They’ve never asked,” he said.
Then there’s the fact that the IPCC relies on huge amounts of temperature data stored in collections that are subject to no independent oversight. Once again, let’s hear from IPCC insiders:
the needs of the IPCC for data quality assurance substantially exceed the needs for journal publications. The quality and uncertainty of key data sets.should undergo a data quality assessment as part of the IPCC assessment process. (p. 99)
One weakness is that we generally do not have peer review processes for major data holdings – just because there is a quality publication from a dataset does not mean all the underlying data are good. (p. 362)
John Holdren is a smart man. He is a professor at Harvard and a former president of the American Association for the Advancement of Science. He surely understands, therefore, that the peer-review process used by academic journals has its shortcomings.
Given that Holdren is currently advising The White House he is also a well-connected individual. He surely could have discovered for himself, via conversations with IPCC authors, that the IPCC does no systematic vetting – not of scientific papers on which it relies and not of data collections upon which the entire idea of a temperature increase in the late 20th century depends.
A drug company would never be permitted to bring a new product to market if it took as much for granted as does the IPCC. Reports written by the IPCC have not had every conclusion quadruple-checked for accuracy. Instead, these reports blithely rely on temperature data that has not been audited by neutral third parties – and on scientific studies whose conclusions may have received only a cursory review prior to publication.
Rather than accepting Holdren’s opinion, government officials should be taking an up-close-and-personal look at the IPCC for themselves.
Because Holdren’s immense edifice is actually a house of cards.
