Lancet Surveys of Iraq War Casualties - The Second Study (2006) - Criticisms

Criticisms

The Iraq Body Count project (IBC), who compiles a database of reported civilian deaths, has criticised the Lancet's estimate of 601,000 violent deaths out of the Lancet estimate of 654,965 total excess deaths related to the war. An October 2006 article by IBC argues that the Lancet estimate is suspect "because of a very different conclusion reached by another random household survey, the Iraq Living Conditions Survey 2004 (ILCS), using a comparable method but a considerably better-distributed and much larger sample." IBC also enumerates several "shocking implications" which would be true if the Lancet report were accurate, e.g. "Half a million death certificates were received by families which were never officially recorded as having been issued" and claims that these "extreme and improbable implications" and "utter failure of local or external agencies to notice and respond to a decimation of the adult male population in key urban areas" are some of several reasons why they doubt the study's estimates. IBC states that these consequences would constitute "extreme notions". Later statements in a 2010 article by IBC say that the "hugely exaggerated death toll figures" from the 2006 Lancet report have "been comprehensively discredited" by recently published research.

Jon Pedersen of the Fafo Institute and research director for the ILCS survey, which estimated approximately 24,000 (95% CI 18,000-29,000) war-related deaths in Iraq up to April 2004, expressed reservations about the low pre-war mortality rate used in the Lancet study and about the ability of its authors to oversee the interviews properly as they were conducted throughout Iraq. Pedersen has been quoted saying he thinks the Lancet numbers are "high, and probably way too high. I would accept something in the vicinity of 100,000 but 600,000 is too much."

Debarati Guha-Sapir, director of the Centre for Research on the Epidemiology of Disasters in Brussels, was quoted in an interview for Nature.com saying that Burnham's team have published "inflated" numbers that "discredit" the process of estimating death counts. "Why are they doing this?" she asks. "It's because of the elections.". However, another interviewer a week later paints a more measured picture of her criticisms: "She has some methodological concerns about the paper, including the use of local people — who might have opposed the occupation — as interviewers. She also points out that the result does not fit with any she has recorded in 15 years of studying conflict zones. Even in Darfur, where armed groups have wiped out whole villages, she says that researchers have not recorded the 500 predominately violent deaths per day that the Johns Hopkins team estimates are occurring in Iraq. But overall Guha-Sapir says the paper contains the best data yet on the mortality rate in Iraq." A subsequent article co-authored by Guha-Sapir and Olivier Degomme for CRED reviews the Lancet data in detail. It concludes that the Lancet overestimated deaths and that the war-related death toll was most likely to be around 125,000 for the period covered by the Lancet study, reaching its conclusions by correcting errors in the 2006 Lancet estimate and triangulating with data from IBC and ILCS.

Beth Osborne Daponte, a demographer known for producing death estimates for the first Gulf War, evaluates the Lancet survey and other sources in a paper for the International Review of the Red Cross. Among other criticisms, Daponte questions the reliability of pre-war estimates used in the Lancet study to derive its "excess deaths" estimate, and the ethical approval for the survey. She concludes that the most reliable information available to date is provided by the Iraq Family Health Survey, the Iraq Living Conditions Survey and Iraq Body Count.

Mark van der Lann, professor of biostatistics and statistics at UC Berkeley, disputes the estimates of both Lancet studies on several grounds in a paper co-authored with writer Leon de Winter. The authors argue that the confidence intervals in the Lancet study are too narrow, saying, "our statistical analysis could at most conclude that the total number of violent deaths is more than 100.000 with a 0.95 confidence — but this takes not into account various other potential biases in the original data." Among the main conclusions of their evaluation are that "the estimates based upon these data are extremely unreliable and cannot stand a decent scientific evaluation. It may be that the number of violent deaths is much higher than previously reported, but this specific report, just like the October 2004 report, cannot support the estimates that have been flying around the world on October 29, 2006. It is not science. It is propaganda."

Fred Kaplan of Slate criticized the first Lancet study and has again raised concerns about the second. Kaplan argues that the second study has made some improvements over the first, such as "a larger sample, more fastidious attention to data-gathering procedures, a narrower range of uncertainty", and writes that "this methodology is entirely proper if the sample was truly representative of the entire population—i.e., as long as those households were really randomly selected." He cites the low pre-war mortality estimate and the "main street bias" critique as two reasons for doubting that the sample in this study was truly random. And he concludes saying that the question of the war's human toll is "a question that the Lancet study doesn't really answer".

Dr. Madelyn Hsaio-Rei Hicks published a paper on the 2006 Lancet survey expressing concern over problems presented by the rapid interviewing rate reported by the survey. Her paper concluded that, "In view of the significant questions that remain unanswered about the feasibility of their study’s methods as practiced at the level of field interviews,it is necessary that Burnham and his co-authors provide detailed, data-based evidence that all reported interviews were indeed carried out, and how this was done in a valid manner. In addition, they need to explain and to demonstrate to what degree their published methodology was adhered to or departed from across interviews, and to demonstrate convincingly that interviews were done in accordance with the standards of ethical research."

Borzou Daragahi Iraq correspondent for the Los Angeles Times, in an interview with PBS, questioned the study based on their earlier research in Iraq, saying, "Well, we think—the Los Angeles Times thinks these numbers are too large, depending on the extensive research we've done. Earlier this year, around June, the report was published at least in June, but the reporting was done over weeks earlier. We went to morgues, cemeteries, hospitals, health officials, and we gathered as many statistics as we could on the actual dead bodies, and the number we came up with around June was about at least 50,000. And that kind of jibed with some of the news report that were out there, the accumulation of news reports, in terms of the numbers killed. The U.N. says that there's about 3,000 a month being killed; that also fits in with our numbers and with morgue numbers. This number of 600,000 or more killed since the beginning of the war, it's way off our charts."

The October 2006 Lancet estimate also drew criticism from the Iraqi government. Government spokesman Ali Debbagh said, "This figure, which in reality has no basis, is exaggerated". Iraq's Health Minister Ali al-Shemari gave a similar view in November 2006: "Since three and a half years, since the change of the Saddam regime, some people say we have 600,000 are killed. This is an exaggerated number. I think 150 is OK."

A 2008 article in the National Journal revealed for the first time that the Lancet survey was funded in part by George Soros' Open Society Institute. This led to some concerns regarding the objectivity of the survey, including erroneous claims in the media that Soros may have been directly involved in the survey. Outspoken survey critic Michael Spagat, economics professor at Royal Holloway, University of London stated "The authors should have disclosed the donation and for many people that would have been a disqualifying factor in terms of publishing the research."

John Tirman, executive director of MIT's Center for International Studies, responded to the issue of Soros funding: "My center at MIT used internal funds to underwrite the survey. More than six months after the survey was commissioned, the Open Society Institute, the charitable foundation begun by Soros, provided a grant to support public education efforts of the issue. We used that to pay for some travel for lectures, a web site, and so on. OSI (Open Society Institute), much less Soros himself (who likely was not even aware of this small grant), had nothing to do with the origination, conduct, or results of the survey. The researchers and authors did not know OSI, among other donors, had contributed."

A 2010 paper by Professor Michael Spagat entitled "Ethical and Data-Integrity Problems in the Second Lancet Survey of Mortality in Iraq" was published in the peer reviewed journal Defense & Peace Economics. This paper argues that there were several "ethical violations to the survey's respondents", faults the study authors for "non-disclosure of the survey's questionnaire, data-entry form, data matching anonymised interviewer identifications with households and sample design", and presents "evidence relating to data fabrication and falsification, which falls into nine broad categories." The paper concludes that the Lancet survey, "cannot be considered a reliable or valid contribution towards knowledge about the extent of mortality in Iraq since 2003."

Two articles from August 2012 by Joel Wing, who runs the blog Musings on Iraq, explore many of the major flaws that have been raised by various researchers over the years. Wing notes that, "The two Lancet studies taken at face value seemed like legitimate estimates of the number of Iraqis that might have been killed after the 2003 invasion," but finds that, "There is evidence that the authors overestimated the number of post-invasion Iraq deaths, did not include data that contradicted their findings, and misrepresented other reports about casualties during the Iraq War to bolster their argument. The Iraqi survey teams might not have followed protocol and the methodology, and some may have faked their results. The responses of the Lancet writers to their critics have been a series of convenient and contradictory statements, and their refusal to openly share all of their data only adds to the suspicions that their work is deeply flawed." Wing concludes that other credible studies "should be consulted first, while the two Lancet papers should be dismissed."