Canadian blogger Hilary Ostrov and Australian computer programmer Peter B. have given the climate change world a gift this week. Since March they’ve been hyperlinking and annotating the 3,000-page Intergovernmental Panel on Climate Change (IPCC) report released in 2007. The result is AccessIPCC.com.
Those of us who’ve been taking a close look at the 2007 report (also known as AR4) have identified numerous concerns. Now we have a tool to analyze it more comprehensively than ever before.
The more than 18,000 references on which the IPCC builds its case have been coded via an automated process. This means one can now visually scan a chapter and see which citations are of potential concern. To gain a clean designation (and thus remain untagged by AccessIPCC), a reference must meet stringent criteria:
- it is apparently peer-reviewed
- it doesn’t violate the IPCC’s cut-off dates (that issue is discussed here, here, here, and here)
- it wasn’t written by an author of that IPCC chapter (which would make it a self-reference)
- it wasn’t written by an author of another IPCC chapter (an indicator of the incestuous nature of the IPCC-affiliated community)
- it doesn’t base its conclusions on a model or a simulation (versus real-world observations)
- it wasn’t written by a Person of Concern
- it wasn’t published by a Journal of Concern
Persons of Concern are defined here (the actual list of names appears at the “Persons of Concern” tab at the bottom left of your screen). These are a group of activist-oriented scientists known as the Hockey Team – as well as a handful of people closely associated with them in the Climategate emails.
Journals of Concern are those that have published papers authored by Persons of Concern. One of the most disturbing things we learned from Climategate is that academic peer-review can be startlingly superficial. Phil Jones, the director of the University of East Anglia’s Climatic Research Unit (the source of the Climategate e-mails), told a UK parliamentary committee in March that, in all his years of publishing papers in reputable journals such as Nature and Science, no one has ever asked to examine his raw data or his computer code.
In drug trials, raw data and computer codes are inspected as a matter of course. A new kind of aspirin receives orders of magnitude more scrutiny than does the sort of research routinely cited by the IPCC. This means that governments are making trillion-dollar climate decisions based on IPCC reports that rely on data no one has ever double-checked. The mere fact that an academic journal published the research findings seems to be good enough for the IPCC. (This approach then empowers author cliques to highlight papers that support certain arguments while ignoring the existence of contrary ones. The IPCC’s approach to the species extinction question provides a classic illustration of this.)
AccessIPCC doesn’t imply that every article published in a Journal of Concern is flawed. It doesn’t imply that every research paper that depends on modeling is invalid. It merely alerts us to the fact that we should proceed with caution. (If your brain, like my own, benefits from cheat sheets here’s one you can print-out until you become familiar with the AccessIPCC codes.)
In the IPCC version of its report, citations are cryptic and a good deal of messing about is required before one is able to examine a reference in full. AccessIPCC solves this problem. For example, a section in which the IPCC discusses urban air quality looks like this on the IPCC website (click the image to enlarge or click here to go to the original):
Below is a cut-and-paste of the same section from AccessIPCC:
8.4.1.4 Urban air quality
Background levels of ground-level ozone have risen since pre-industrial times because of increasing emissions of methane, carbon monoxide and nitrogen oxides; this trend is expected to continue over the next 50 years ( Fusco and Logan, 2003 [MoS] ; Prather et al., 2003 [JoC] ). Changes in concentrations of ground-level ozone driven by scenarios of future emissions and/or weather patterns have been projected for Europe and North America ( Stevenson et al., 2000 [JoC, MoS] ; Derwent et al., 2001 [JoC] ; Johnson et al., 2001 [JoC, MoS] ; Taha, 2001 [NPR, MoS] ; Hogrefe et al., 2004 [MoS, ARC] ). Future emissions are, of course, uncertain, and depend on assumptions of population growth, economic development, regulatory actions and energy use ( Syri et al., 2002 [MoS] ; Webster et al., 2002a [MoS] ). Assuming no change in the emissions of ozone precursors, the extent to which climate change affects the frequency of future ‘ozone episodes’ will depend on the occurrence of the required meteorological conditions ( Jones and Davies, 2000 [MoS] ;( Sousounis et al., 2002; ) Hogrefe et al., 2004 [MoS, ARC] ;( Laurila et al., 2004; ) Mickley et al., 2004 [JoC, MoS]). Table 8.4 summarises projections of future morbidity and mortality based on current exposure-mortality relationships applied to projected ozone concentrations. An increase in ozone concentrations will affect the ability of regions to achieve air-quality targets. There are no projections for cities in low- or middle-income countries, despite the heavier pollution burdens in these populations.
Hover your cursor over any of the above citations and voilà – full details appear. The red-coloured font tells us that 12 of the 14 citations in this paragraph may be of concern. The MoS after the first one tells us these research conclusions were derived via simulations. Since a simulation’s output cannot be accurate if its inputs are suspect, it’s important to receive this heads-up. With just a glance we now see that 10 of these 14 citations rely on modeling rather than on concrete, real-world data to reach their conclusions.
We also see that 5 citations have been tagged JoC – which means they were published in Journals of Concern. One is tagged NPR – which tells us it is not peer-reviewed. And two are tagged ARC- which indicates this particular research was written by a person who authored another chapter in the same IPCC report.
In a different part of the report, a cut-and-paste from near the top of the AccessIPCC version of the chapter shows us how many cited papers were produced by the people who wrote the IPCC chapter (see it online here):
Coordinating Lead Authors:
Pete Smith (UK) [SRC:15], Daniel Martino (Uruguay) [SRC:2],Lead Authors:
Zucong Cai (PR China) [SRC:8], Daniel Gwary (Nigeria) [SRC:2], Henry Janzen (Canada) [SRC:8], Pushpam Kumar (India) [SRC:3], Bruce McCarl (USA) [SRC:13], Stephen Ogle (USA) [SRC:7], Frank O’Mara (Ireland) [SRC:8], Charles Rice (USA) [SRC:6], Bob Scholes (South Africa), Oleg Sirotenko (Russia) [SRC:3], Mark Howden (Australia) [SRC:2], Tim McAllister (Canada) [SRC:3], Genxing Pan (China) [SRC:5], Vladimir Romanenkov (Russia) [SRC:3], Steven Rose (USA) [SRC:1], Uwe Schneider (Germany) [SRC:5], Sirintornthep Towprayoon (Thailand) [SRC:2], Martin Wattenbach (UK) [SRC:2],
This means that this chapter’s arguments rely on research conducted by 19 out of its 20 authors. Coordinating lead author Pete Smith felt it was appropriate to cite 15 of his own publications. Meanwhile, 13 papers authored by lead author Bruce McCarl are also cited.
As those familiar with academia are aware, one earns brownie points if one’s papers are widely cited elsewhere. In this case, we see that such citations have been orchestrated by the authors themselves. The above results also lend support to Ross McKitrick’s view (see page 3 here) that one needn’t wait for the IPCC to write its report. All one needs to do is look at the people who’ve been assigned to a particular chapter and the outcome can be predicted in advance.
This is Version 1 of AccessIPCC. And remember, it has been produced by volunteers. Hilary cautions us that the project is not yet complete and is “far from perfect.” Some of the codes will no doubt need correcting. (There’s a dedicated blog where discussions of that nature can take place.)
That being said, AccessIPCC represents an enormous leap forward. We now have a more user-friendly – and immeasurably richer – way of evaluating the IPCC’s Nobel-winning report.
Full disclosure: AccessIPCC is the brainchild of Peter B., who contacted me in March while I was coordinating the Citizen Audit of AR4 references. He and Hilary became acquainted and joined forces via that initiative.
