Shooters' Committee On Political Education

Home

LEGISLATIVE

News

Links

Rewards

JOIN

Acknow1edgements:

Technical Committee
The technical committee was comprised of personnel from the larger law enforcement agencies in the State of California. These personnel had expertise and familiarity with the existing ballistics imaging systems, DRUGFIRE~~ and IBIS©, currently in use.

The following personnel were instrumental in developing the study performance tests, submitting data for the survey and reviewing this study:

Forensic Technology Inc.
The correlation tests were conducted at the facilities of Forensic Technology Inc. (FTI) using their NIBIN system and a FTI operator. Two California Department of Justice representatives who provided the test samples observed them. Without the support of FTI, the study could not have been conducted in a timely manner.

California Highway Patrol Academy - West Sacramento
The availability of the staff of the Weapons Training Unit and and their generous assistance greatly simplified the process of obtaining about 2,000+ fired cartridge cases from 790+ pistols

1.0    EXECUTIVE SUMMARY

Summary
Automated computer matching systems do not provide conclusive results. Rather, a list of potential candidates are presented that must be manually reviewed. When applying this technology to the concept of mass sampling of manufactured firearms, a huge inventory of potential candidates will be generated for manual review. This study indicates that this number of candidate cases will be so large as to be impractical and will likely create complications so great that they cannot be effectively addressed.

1.1 Firearms Identification and Automation
The concept of automated imaging was originally developed to aid the firearms examiner in keeping track of open case files. Open case files refer to those cases in which an evidence cartridge case or bullet could not be linked to any firearms in the possession of law enforcement at the time of examination. In 1994 the Office of National Drug Control Policy (ONDCP) validated the concept of ballistics imaging of firearms evidence in the forensic science community (consisting of bullets and cartridge cases in which they could be automatically compared to evidence specimens for preliminary correlation). There are several issues associated with an automated imaging concept that have to be considered. These relate to issues that impact the efficacy of the use of ballistics imaging when applied to large numbers of commercially produced firearms. These are:

1. Current imaging systems require trained personnel, ideally a firearms examiner, for entry, searching and verification. The use of technicians typically results in higher numbers of false positives that need to be microscopically compared
   .
2. Current systems may not be as efficient for rimfire firearms and are limited to auto loading weapons. Proposed systems will not practically accommodate revolvers, rim fires, certain shotguns and rifles. A large proportion of firearms sold in CA may never make entry into the system.
   
3. It is unknown at this time whether or not the algorithm can successfully ID a cartridge case fired after typical break-in and wear have occurred back to the #1 casing fired at the time of manufacture. Performance Test #7 (See page 8-11) showed that even in a limited database, the ranking of subsequently fired casings could drop enough to fall from a candidate list for consideration. Typically quoted existing research/papers regarding presistence of fired marks on fired cartridge cases were written based on manual comparison by qualified firearms examiners, not automated correlation techniques.
   
4. All potential "hits" selected for further inspection by computer correlation must be confirmed by "hands on" microscopic examination by a qualified firearms examiner.
   
5. Firearms that generate markings on cartridge casings can change with use and can also be readily altered by the user. They are not permanently definedidentifiers like fingerprints or DNA. Hence, images captured when the firearm is produced may not have a fixed relationship to fired cartridge casings subsequently recovered.
   
6. Cartridge casings from different manufacturers of ammunition may be marked differently by a single firearm such that they may not correlate favorably.
   
7. As progressively larger numbers of similarly produced firearms are entered into the database, images with similar signatures should be expected that would make it more difficult to find a link. Therefore, this increase in database size does not necessarily translate to more hits.
8. Fired cartridge casings are much easier to enter, correlate, and review than fired bullets.
   
9. Not all firearms generate markings on cartridge casings that can be identified back to the firearm.

1.2    Current Use of Automated Ballistics Imaging by Law Enforcement Agencies

Automated ballistics imaging systems are currently in use by many law enforcement agencies (LEA). These systems are called DRUGFIRETM and IBIS©, both of which operate under the acronym of National Integrated Ballistics Information Network (NIBIN). A recent MOU between the FBI and the BATF dictates that only the IBIS system will continue under the NIBIN banner. The NIBIN systems provide "cold hits" or a link between two or more crimes. These cold hits are much more frequent in large urban areas. One reason for this is believed to be that firearms used in gang crimes are frequently passed around by gang members, reused in crime and are subsequently available for linkage. Cold hits provide an investigative lead for the investigator; they do not necessarily implicate any one shooter. In six years of operation, the Southern California Database has 433 cold hits from 338 firearms. This area covers all the major metropolitan areas of all Southern California. As an example, the Southern California 9 mm Luger cartridge case evidence database size has 3,422 evidence cartridge cases and

1A "cold hit" is an occurrence in which a match between two separate case ekhibits in a database is achieved which were not previously known to be related.

2 Some of these cold hits may actually be warm hits in that there was an expectation before MBIN analysis
that they came from a common source.

10,532 test-fired cartridge cases at the time this publication was prepared. These databases are very diverse and include numerous manufacturers.

In contrast, the Sacramento County Crime Laboratory has actively developed its ballistics-imaging database since 1996 and has had 14 cartridge case cold hits with no prosecutions. The Oakland Police Department has 37 cold hits to date, one of which led to a conviction. The actual issue of cost effectiveness and related labor cost for each cold hit has not been documented nor researched. There have been no studies published that discuss the significance of a cold hit and its effectiveness.

1.3   Expectations of a California Database

In estimating the size of a potential California database, the figure of 107,791 pistols entries per year has been used. This would only count semi-automatic pistols .25 caliber4 and higher. Revolver will not be entered due to the low frequency of revolver cartridge cases found at crime scenes. After five years there will be an estimated 538,955 registered pistols in this database. Of these, about 242,5Q0 handgun entries will be in the 9 mm Luger cartridge category. This would still represent a very small fraction of all the handguns in circulation.

Recent legislation under California Penal Code section 12125, et seq., (SB 15 Safe Handgun bill) may reduce the variety of manufacturers that can sell handguns. This will cause more uniformity and less diversity in the cartridge case database. Essentially, there will be many more cartridge case specimens for an approved-for-sale model by a particular manufacturer. Even if the database is localized to specific regions of the state, one can readily expect to find 700+ registered handguns from one model and manufacture in a rural county with a population of 550,000 people. Consequently, metropolitan areas are expected to have much higher numbers of similar handguns than is currently found.

1.4  Limitations of the Performance Tests

In order to test a larger database, 792 California Highway Patrol (CHP) Smith & Wesson model 4006~ pistols were test fired with a variety of .40 S&W ammunition. The performance tests in this study were designed to mimic what would happen when a database is substantially increased in size. The results obtained from these tests are not reflective of what is currently obtained by the local LEA's. Current California Law Enforcement Agency (LEA) databases are much smaller and more diverse. Even in the current Southern California database of 3,422 - 9mm cartridge cases, it is extremely unlikely that there would be 792 incidents of evidence cartridge cases fired by one

3 Estimate given by the DOJ Firearms Division based on new sales for the period 1997~2OOO.

4 Rim fires are not considered at this time. Although they can be imaged in NIBIN, the utility of such imaging for unique individual characteristics has not yet been independently verified.
~    These were scheduled to be issued to CHP Cadets and most were in new condition.

caliber, manufacture and model. However, for a database of newly sold handguns, one could expect to see several times this number in the highly populated areas.

For automated imaging, a cartridge case can be preliminarily identified from its breech face markings and/or firing pin impressions. In order to simplify potential identification selections, one would like to have both breech face and firing pin impressions rank high. These tests looked at correlations and positions of fired test cartridge cases compared to specimens known to be in the database. Ideally the test/evidence cartridge case should be in the first rank.

1.5   Results of the Performance Tests

The performance tests have provided some results that indicate both the potential and limitations of a statewide database. Most of these results have not been mixed in a current real-life database. The combination of this test database and a current real-life database would have improved the information about correlation performance.

Computer Capability and Speed
The IBIS system appears to have the potential to be scalable and should be capable of operating with a large California database. This would not be for real time analysis since each search of a hypothetical 100,000-cartridge case database would require 1.5 hours using current hardware.

Effect of Cartridge Case Ranking and Database Size
As a database was increased in size by a factor of 7 (100 to 700), the position or ranking of test-evidence cartridge cases, initially in the ~ ten ranks, would change (with one exception) to undetectable ranks. This change in rank could be sufficient enough that an examiner might not link the test/evidence cartridge case to one in a larger database. if the test/evidence cartridge case was in the first or second rank, it had a tendency to stay in these ranks when there was a four-fold increase in database size. The interpretation of this is that one would like to see a cold hit in the 1st or 2nd position (rank) for large databases.

Comparison of Cartridge Cases from the Same Manufacture
The system looked at 50 duplicate test fired cartridge cases selected at random from the 792 Federal cartridge cases in the database. The results for these same ammunition tests are as follows:

38% were missed and not in the top 15 ranks.
48% with either the breech face or firing pin were in the ~ rank.
62% with either the breech face or firing pin or both were in the top 15 ranks.

Comparison of Cartridge Cases from the Different Manufactures
The system looked at 72 test fired cartridge cases using different ammunition and fired from random CHP guns. The results for the different ammunition tests are as follows:

62.5% were missed and not in the top 15 ranks.
22.2% with either the breech face or firing pin in the 1st rank.
37.5% with either the breech face or firing pin or both in the top 15 ranks.

The reason figures are quoted for 1st rank and the top 15 ranks is that one may want to use the percentages for the 1st rank with large databases in order to more accurately estimate cold hit rates. Database size can become a key issue for potential identifications.

This' performance test illustrates the effect that the change of a cartridge can have on the perceived signature or image of a breech face or firing pin. Different cartridges can have this effect on the apparent signature because the impression may not mark in a similar manner with the same level of detail6. The algorithm is still doing its basic job of identifying similar images, thus the algorithm cannot be faulted for its lack of identification if the apparent image is different. By increasing image quality or correlating images with different illumination methods there exists the potential for improving the algorithm hit rate.

Altered Breech Face
Changing the signature of a breech face or f'irlng pin impression for one of the CHP handguns used in this study was a relatively easy affair. The minor alteration required less than 5 minutes of labor to change the signature of the breech face and firing pin. This change is sufficient to make the cartridge case breech face unrecognizable, by IBIS algorithm, to the first set of cartridge cases test fired from that same pistol. This type of effort has happened in actual laboratory casework.

Lon~evity Study

Two non-CHP handguns were used to determine the effect of multiple test firings on the persistence of cartridge case impression signatures. Six hundred rounds were test fired from each of these two handguns. There is some indication of signature degradation as one compares test #600 to test #1, but no definitive conclusions could be made. In the future, further evaluation of several of the database test handgun cartridge cases that ranked in the upper 10 rankings should be used for such a test.

1.6 Interpretation of Results

It should be noted that this study looked at the cartridge case hits in the first fifteen ranks. In actual practice, when examiners are trained on the IBIS system, they are trained to only look at the first 10 ranks.

Cartndge Case Hit Rate
The California system will be working with large databases. As this is the case, the statistics that should be applied are from Performance Test lC (correlation position and DB size) and Performance Test 3 (different cartridge ammunition).

6 Furthermore, some primers in new cartridges may have similar manufacturing marks on them that could be construed as coming from the breech face of a firearm.

The effect on correlation position, as illustrated in Performance Test IC, Figure 8-7, appears valid for a larger database. Cartridge cases that are not in rank one may not be detected as the database of similar handguns dramatically increased in size. Thus the most meaningful results are those specimens in the first rank.

Using the data from Performance Test 1, Figure 8-1:

• 48% of the cartridge cases ranked in the number one position in either breech face or firing pin.
• 62% of the cartridge cases ranked in the top 15 positions in either breech face or firing pin.

Using the data from Performance Test 3, Figure 8-9:

• 22.2% of the cartridge case ranked in the number one position in either breech face and firing pin.
• 37.5% of the cartridge cases ranked in the top 15 positions in either breech face or firing pin.

78% of the evidence cases that should have a counterpart in the proposed database may not be detected when aifferent cartridges are used. This is based on the performance of the cartridge cases test fired with CHP firearms when different ammunition was used.

A significant deterioration of correlation results was observed when one factor was incorporated into the control performance test (Performance Test 1). Additional factors that may also have detrimental effects to Perfornuance Test 1 include, but are not limited to:

• Human errors - The listed performance test results did not include any effect of human error. Computer correlation
  results will have to be screened by human operators. There exists the potential error of hits being missed by the screening operator.
• Longevity of marks - Persistence of the markings with use and wear of the firearm
• Database size
• Sub-Class marks.
• Altered breech face or firing pin.

Database Size
The proposed database size will be very large with many firearms expected to be made by the same manufacturer. At a presumed rate of 107,791 new handguns per year, from a limited selection of semiautomatic models and calibers, the database will be expected to represent:

• After five years there may be 538,955 handgun cartridge case images.
• These 535,955 handguns will still be a small fraction of the existing firearm in circulation
• About 45% of the handguns will be 9 mm Luger pistols.
• Many samples of a small variety of different handguns. A small rural county, such as San Joaquin, had 800+ handguns of one model, caliber and manufacture.

1.7 Recommendations for Further Action

Any ballistics-imaging program is complex and has many ramifications for the end user. In developing this study, other issues that should have been addressed were found. Furthermore, experimental designs could be improved in order to derive additional meaningful data. A study of this type has many variables that need to be researched and addressed. Some recommended areas for further study are as follows:

• Conduct studies using 1,000 - 9mm Luger pistols that are the same model and make and are in use by local law enforcement agencies. Ideally, they should be new if such a large number can be found.
• Evaluate this data in its own pristine database and in combination with a database from a large regional area that has a variety of different firearms.
• Conduct a longevity study to evaluate persistence and recognition of correlation related marks. Fire multiple rounds using several of the test handguns that initially rank high in the database search. Then fire 600+ rounds and conduct additional comparisons
• Evaluate investigative effectiveness of "hits", what happens to a "hit" at local law enforcement agencies after it is found in the database.
• Review what "hits" enter the system for prosecution.
• Further define what are "cold hits"8, how they are documented, and the actual number of firearms represented by the cold hits.
• Evaluate the associated costs for ballistics imaging systems including telecommunication I data lines, equipment, labor for data entry, and the related costs associated with manual comparisons and verifications by an examiner.
• Evaluate the potential number of evidence cartridge cases seized by local law enforcement agencies that could be entered into a system but are currently not entered due to funding limitations.
• Evaluate the types and sources of firearms that provide the most potential for cold hits.
• Determine the age distribution of those firearms that have been found in cold hits. This distribution can be useful to decide the length of time a state database has to be retained for active searching and if data archiving can occur.

7 An underlying assumption is that the average individual does not fire more than 12 boxes (600 rounds) of ammunition. In a typical practice shoot at a range, it is very common to fire 2 boxes or 100 rounds of ammunition for practice.

8 Actual verification on whether or not a hit is actually "cold" is needed. Many agencies are concerned
with good numbers, rather than accuracy. Are agencies funded by the so-called success of the automated system going to accurately represent their true cold hit statistics?

Related: President's Report: Ballistics Database Program: A Complete Failure