Iraq War Logs: Technical appendix to IBC analysis

First published 25 Oct 2010

Rationale and Method

Several factors influenced our analytic approach. We could not analyse all of the Iraq War Logs one by one because we had only a few weeks to prepare our report using very limited resources. Yet we wanted to understand the logs as a whole. In particular, we wanted to get a handle on what these logs will add to the information already in the public domain about deaths in the Iraq war. We were especially interested in civilian deaths since these are what IBC has been recording since the beginning of the war. Given the very limited time, we decided that the best way forward was to draw a random sample of all of the logs and match these up against the IBC database. The question then became how to draw this sample.

We had good reasons to expect that there would be a strong relationship between sizes of events, measured by the number of deaths, and the rate at which these would match against the IBC data, with deaths in larger events matching more frequently than smaller ones.1 We wanted to draw data that would illuminate this relationship with a reasonable degree of statistical reliability. This goal required the sample to adequately cover the full range of different event sizes. If, to the contrary, we had drawn a simple random sample, i.e., we had given every log an equal probability of selection, then our sample would have contained a very large number of small events and very small samples of large events. In this case, our matching-rate estimates for larger events would have been extremely imprecise. To overcome this problem we drew a stratified sample that guaranteed good coverage of a full range of size classifications.

It was impossible to select good samples for every possible event size (1,2,3…). So we combined the events into size ranges that are narrow for the smaller, more common, events and wider for the bigger ones. These ranges were 1, 2-3, 4-9, 10-19 and 20+ (the latter being exhaustively analysed).

A further consideration is that estimate precision improves at a decreasing rate as sample size grows. So, for example, the added precision from sampling an additional 25 logs will be more starting from a base where one has sampled only 25 logs than it will be starting from a base of 100 sampled logs. Doubling our sample size of events with one civilian killed from 100 (our actual size) to 200 would not have been possible in the time available, and would have given only a small boost to precision.

We also wanted to completely understand the large events in the logs, where the determination of the cut-off for “large” would depend mainly on the limited time available to carry out the analysis. We started processing large events and soon realised that there would only be sufficient time to process all the events with twenty of more deaths and still process a good sample of events below this size. This is how we settled on the 20+ cutoff for full analysis.

We judged that there would be enough time to randomly sample about 500 logs after the full census of the large events was complete. Following the principles discussed above we wanted most of these sampled logs to be in the civilians category. This ensured that we sampled each size range reasonably well and did not sample any one size range beyond the point of diminishing returns. We settled on the sample sizes each category, given in the first three tables below, whose confidence intervals, assuming the worse-case spread (at a matching rate of 50%) were sufficient for this preliminary work.

Logs were each given a unique identifier, and the list then shuffled randomly. Each sample category was derived by filtering the shuffled list according to the selected size ranges, and drawing the required number of logs from the top of this shuffled list.

The next step was to carefully match each of the selected logs to the IBC database. The details recorded in the logs were examined by an experienced member of the IBC team. Date and location were used to initially identify potential matching IBC entries. The best match from these incidents, based on numbers killed, and type of weapons used, helped to identify matching incidents. If no match could be found the search was extended to the day before or after the date given in the log, as discrepancies could exist between the two sources on any given variable, including the exact date. Provinces can sometimes be discrepant as well, particularly near the boundaries of two or more provinces, so neighbouring provinces were searched as well when no match could be found. Some difficult cases required reexamination of the original source reports used by IBC.

There are two distinct types of matches determined for each log. The first is whether the deaths were included in IBC. The second was whether the details of the specific incident were recorded as a discrete entry in IBC. Many thousands of deaths in the IBC database are not recorded at the level of detail of a specific violent event or incident, but rather as an aggregated total of multiple events. One such example is noted in our accompanying 'details' article, where IBC has only one entry for 35 bodies found around Baghdad over the course of a day.2 The logs also record 35 bodies found on that day in Baghdad but record each specific incident independently, and include many other details of the deaths. In such a case the deaths are considered matched in the IBC database while the incidents are not. This explains why the matching rates are different in the three deaths tables, and three incident tables below, with deaths generally having a considerably higher matching rate than incidents.

Our matching criteria were fairly strict. For example, if no matching incident was found within a few days of the date recorded in the log it was determined to have no match. This strictness could overstate the number which failed to match in cases where the dates recorded in the two sources differed by more than a couple of days.

The following three tables summarise our estimates of the number of deaths that will be added to public knowledge in the different categories of the Iraq War Logs once all of the new records are processed by IBC researchers and integrated into IBC's database. The confidence intervals reflect only sampling errors, and do not reflect non-sampling errors, such as undiscovered coding errors in the logs or IBC.

Deaths in the 'Civilian' category

In some of the categories for which the estimates of new deaths or incidents are very small the lower limits for the confidence intervals dip slightly below zero. This is because the formula for calculating confidence intervals does not automatically account for the fact that the number of deaths to be added to the IBC database from the logs cannot be a negative number. In these cases we replaced the negative lower limit of deaths to be added to the IBC database with the number of actual (not estimated) new deaths we found in the sampled logs within that category. Clearly the final number of new deaths to add to IBC cannot be lower than these.

Deaths in the 'Host Nation' category

* There is considerable overlap between IBC and the Host Nation category, mainly because IBC includes ordinary (non-paramilitary) police. Most other Host Nation deaths fall outside the IBC definition of civilians so IBC does not record them (see section 3.3 of IBC Methods on ‘Inclusion and exclusions’). This means that Host Nation deaths can fail to match IBC for two reasons: either IBC is simply unaware of those deaths or they are deaths that are not compatible with IBC’s inclusion criteria. For the purpose of estimating the number of new deaths to be added to IBC, deaths of the latter type are effectively equivalent to matched deaths, since neither matched deaths nor non-compatible (e.g. combatant) deaths will add new deaths to IBC. As such, these are treated as equivalent in the table above.

Deaths in the 'Enemy' category

When IBC is unsure of whether deaths are of civilians or combatants it publishes an upper and lower range of figures to reflect this uncertainty. So, for example, if six people of ambiguous status are killed in an incident then that incident will be coded as zero to six civilian deaths. The estimate of 3,485 disputed deaths is based on the upper range of IBC figures which means that IBC concedes that some of these victims may have been combatants. If we instead use IBC’s lower range the estimate for disputed deaths is 3,148 (CI: 2,551-3,745). Since these numbers are fairly close, the discussion in the main text uses only the upper range, mainly to avoid the added complication of having to speak in terms of multiple ranges.

Deaths in Iraq War Logs and IBC compared

The number of 79,187 deaths in the Iraq War Logs is our calculation of the number of deaths contained in the logs which meet the same inclusion/exclusion criteria as IBC’s civilian deaths database. This allows us to look at comparable deaths in the two sources and determine what portion of each dataset is missing from the other. To derive this IBC-comparable deaths total we include the entirety of the ‘Civilian’ category of the logs (66,081). We then add our central estimate of the number of ‘Host Nation’ deaths that fall within IBC’s inclusion criteria (e.g., non-paramilitary Iraqi police) based on the results found in the samples described above. This figure comes to 9,621 (8,558-10,684) out of 15,196 total ‘Host Nation’ deaths. We then also add our central estimate of the number of disputed ‘Enemy’ deaths, as we determined these to be in IBC as civilians. This figure is 3,485 (2,992-3,978) as shown in the table in the previous section.

Combining these three figures results in 79,187 (78,191-80,183) deaths in the logs that can be directly compared to IBC. The remaining deaths in the logs (roughly 30,000) are of combatants (e.g., Iraqi Army, insurgents) which IBC does not include in its database, and must therefore be excluded from this comparison to reveal the number of IBC-recorded deaths that are missing from the logs. (These combatant deaths are discussed in the ‘Total deaths’ section in our main 'numbers' article and below).3

The comparable IBC figure is simply the IBC total for the period of 1 January 2004 through 31 December 2009 (92,092), minus the months of May 2004 (621) and March 2009 (416) since these months are missing from the logs. This leaves a comparable IBC total of 91,055.

We have previously estimated that 15,144 (13,525-16,762) of the comparable log deaths are missing from IBC, thus when we subtract this number from 79,187 the remaining number of comparable log deaths that actually match to IBC is reduced to 64,043. We can then subtract 64,043 from 91,055 and see that the remaining number of IBC-recorded deaths left unaccounted for in the logs would be 27,012.

Here and in the main text we use central estimates only, for the sake of simplicity. The range of error inherent in the parts of these calculations that rely on estimated figures are shown here using 95% confidence intervals.

Total deaths 2003-2010

In three cases, where labelled, the table in the main text of this section uses central estimates derived from our sample. The only one of these not to be discussed elsewhere is “Iraq War Logs ‘Host Nation’ combatant - central estimate”. The 95% confidence interval for this estimate of 5,575 is 4,512-6,638.

Here is a list of, and links for, all sources used in this section for combatant deaths for the missing sections of the logs. These numbers were officially released figures from either the US Department of Defence or the Iraqi government:

1. Insurgents killed June-December 2003
2. Insurgents & Iraqi soldiers killed January-September 2010
   Jan: 20 soldiers, 54 insurgents
   Feb: 45 soldiers, 52 insurgents
   Mar: 101 soldiers, 57 insurgents
   Apr: 15 soldiers, 48 insurgents
   May: 17 soldiers, 55 insurgents
   Jun: 30 soldiers, 44 insurgents (subscription required, NINA 01 July 2010)
   Jul: 50 soldiers, 100 insurgents
   Aug: 54 soldiers, 98 insurgents
   Sep: 33 soldiers, 78 insurgents
   
3. Insurgents killed May 2004
   
4. Insurgents & Iraqi soldiers killed March 2009
   
5. US & Coalition military killed

Analysis at the incident level

The following three tables summarise our estimates of the number of newly detailed incidents that will be added to the IBC database in the different categories of the Iraq War Logs once all of the new records are processed.

Incident details in the 'Civilian' category

20+: There were 239 logs that each recorded 20 or more deaths, for a total of 9,915 deaths. There were a total of 6 logs that were not matched to the IBC database. 4 of these cases were discounted in our analysis for various reasons:

• (IRQ20061229n126) A mortar attack is recorded incorrectly in the Iraq War Logs as having 22 deaths and 1 injury while the accompanying text makes clear that it actually had 1 death and 22 injuries, i.e., deaths and injuries were reversed. The transposition of deaths and injury in this way has been observed in other logs.
• (IRQ20070109n105) A helicopter accident involving deaths of foreign security contractors which are not not within the inclusion criteria of the IBC database.
• (IRQ20080718n56) A report based on “Military intel” refers to an unconfirmed find of 250 bodies in a mass grave in Baghdad. There have been many such finds of mass graves of both pre- and post-war victims during the war but this report provides no indication of whether the alleged find was of pre- or post-war deaths. Moreover, a search of media sources for the period revealed no evidence of such a find. Yet if it occurred, and was post-war deaths, it would be the biggest single find of bodies for the whole war. We judged this log too unreliable for inclusion in the analysis.
• (IRQ20050726n134) This log mistakenly codes 20 deaths while the accompanying text makes clear that there were 8-10 deaths. This is an example of a recurring error in the SIGACT data, where some deaths and injuries described in the text are doubled in the formal columns. In this instance 10 deaths are doubled to 20.

This leaves two logs of 20+ events missing from the IBC database with a total of 45 new deaths. We get 56 new deaths if we add the two incidents above for which the number of deaths are incorrect in the logs.

Incident Details in the 'Host Nation' category

Incident details in the 'Enemy' category

Limitations/Weaknesses:

Limitations and weaknesses of our sample-based analysis include the following:

1. Formal columns in the Iraq War Logs are sometimes inaccurate. Examples include occasional transposition of deaths and injuries and double counting of some deaths/incidents in multiple logs. However, we do not think that such errors are numerous enough to strongly affect our analysis, in part because we managed to correct for some more severe instances, as indicated above.

2. A problem with the sampling is that there is some overlap between the different sampling categories. This means, for example, that an event containing both 'Civilian' and 'Host Nation' deaths is more likely to turn up in our sample then an event with only one or the other type of death. If events with deaths in multiple categories differ systematically from events with deaths in only one category then our estimates are biased. In principle, we can eliminate this bias by using a sufficiently sophisticated weighting scheme. This bias should, however, be fairly small since most events contain only one type of death.

3. The estimation method assumes that the matching rate within any size range of events applies uniformly across that whole size range even though we know that matching rates actually vary with size. This weakness was inevitable because we could not process a big enough sample to yield good estimates for every possible event size. Fortunately, this problem does not apply to the size-1 category which already accounts for more than half of the estimated civilians to be added to IBC. Moreover, it only applies slightly to the 2-3 category which provides more than a third of the civilian estimate. For events killing more than 10 almost all events match so, again, this issue rarely arises. Therefore the potential impact of this limitation is small, and largely confined to the 4-9 category.

Finally, note that all limitations of the present estimates are temporary because eventually IBC will process all the logs and learn the exact numbers.