Exaggerated claims, substandard research, and a disservice to truth

ORB's "million Iraqi deaths" survey seriously flawed, new study shows

There have been several survey-based attempts to roughly estimate the number of Iraqis killed as a result of the 2003 invasion and subsequent conflict. It is unfortunate that the most careful and well-resourced survey work in this area (from the UNDP and WHO)1 has been scarcely visible, while the most flawed and inadequate work has dominated public discourse. This has been largely due to the shocking (but ultimately numbing) effect of their hugely exaggerated death toll figures.

Iraq Body Count (IBC) applied an early and so far unanswered set of reality checks2 to the Johns Hopkins survey published in the Lancet in October 2006, a paper which has recently been comprehensively discredited in a new study3 by Prof. Michael Spagat of Royal Holloway University. Even among the generally inexact survey results for deaths in Iraq the "Lancet estimate" was an extreme outlier, asserting 450,000 more deaths from violence than the much larger WHO-funded study that estimated 151,000 such deaths by July 2006. The only evidence that appeared to support the Lancet finding was published by a polling company, Opinion Research Business (ORB), which estimated 1 million violent Iraqi deaths by August 2007.

In a meticulous and detailed analysis4 of ORB's survey, IBC researcher Josh Dougherty and Spagat have laid to rest any notion that ORB's massive estimate is even nominally sound, let alone capable of providing validation for another outlier. The press release to their study, published in Survey Research Methods (Vol. 4, No. 1, 2010) is reproduced below.

In an environment whose most notable feature is a shameful paucity of information, exposing another survey into Iraqi deaths as specious is a regrettable but necessary task. Such inflated, and all too carelessly adopted, estimates trivialise the already documented (and tragically high level of) deaths about which the world still knows far too little. Nor do they contribute, even in broad outline, to the work that lies ahead: the establishment of a comprehensive, locally-verifiable, detailed truth, painstakingly pieced together, death by death, name by name, recording each and every casualty of this conflict. The pressing need is for more truth rooted in real experience, not the manipulation of numbers disconnected from reality.

(AUTHORS' PRESS RELEASE, 5 May 2010)

New study exposes fundamental flaws in poll estimate of one million Iraqis killed since 2003

In September of 2007 a British polling company named ORB released an alarming estimate of “more than 1,000,000 Iraqis murdered” in the Iraq war (updated version here). ORB's poll-based estimate has been cited approvingly in much of the blogosphere, a number of mainstream media outlets and academic publications, and in official statements from the Bloomberg School of Public Health of Johns Hopkins University in support of their own work in this area.

“Conflict Deaths in Iraq: A Methodological Critique of the ORB Survey Estimate” by Michael Spagat and Josh Dougherty, just published in Survey Research Methods, describes in detail how the ORB poll is riddled with critical inconsistencies and methodological shortcomings. This first and only peer-reviewed analysis of the ORB estimate concludes that it is too flawed, exaggerated and ill-founded to contribute to discussion of the human costs of the Iraq war. Survey Research Methods also publishes a reply from Johnny Heald of ORB and our response to Heald’s piece.

An internal validity check of ORB data across three separate polls reveals internal contradictions indicative of compromised data collection practices which greatly exaggerate the resulting estimate (section 4 of the paper). In particular, four governorates in central Iraq account for more than 80% of ORB’s estimated one million deaths. Yet in these governorates a higher percentage of respondents report deaths of household members than report deaths of extended family members in another ORB poll conducted only six months earlier. This pattern can not be seen as credible since extended family networks are far larger than households. The percentage reporting deaths in the entire southern region of Iraq, on the other hand, does show the expected sharp drop between the two polls (from 35 to 7 percent) when ORB switches from its extended-family question to its household question. This more reasonable pattern casts strong doubt on precisely the data (in the four key central governorates) which provided the vast bulk (more than 80%) of deaths in the “million” figure.

The ORB poll is also marred by a number of serious quality problems including (sections 2, 3, 5 and 6):

• a claimed margin of error of plus or minus 8% that is unrealistically narrow;
• critical problems with ORB’s published mortality question that invite respondents to report both non-violent deaths and deaths of extended family members in what is purportedly a household survey measuring “murders”;
• a failure to disclose key methodological information such as the exact wordings of its questions as asked in the field, i.e., in Arabic and Kurdish;
• an inadequate treatment of non-response in which ORB assumes that death rates of non-respondents were identical to those of responding households;

An external validity check exposes the ORB data as inconsistent with a large number of credible sources (section 7). The ORB estimate can find partial company only with the widely criticized Lancet estimate from 2006, although these two outlying sources still differ strongly from one another on the geographical pattern of deaths (see section 1 for sources critical of the Lancet estimate).

Michael Spagat of Royal Holloway, University of London said “the ORB poll is not a serious piece of research and should never have seen the light of day.” Josh Dougherty of Iraq Body Count added that “there has been terrible carnage in the Iraq war but no useful purpose is served by promoting inflated death toll estimates.”

Royal Holloway press release, 7 May 2010