Mixed Messages

# Screening programme evaluation applied to airport security

BMJ 2007; 335 (Published 20 December 2007) Cite this as: BMJ 2007;335:1290

1. Eleni Linos, doctoral student1,
2. Elizabeth Linos, research assistant23,
3. Graham Colditz, associate director4
1. 1Department of Epidemiology, Harvard School of Public Health, Boston, MA 02115, USA
2. 2Department of Economics, Harvard University, Littauer Center, Cambridge, MA, USA
3. 3J-Poverty Action Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139-4307, USA
4. 4Prevention and Control, Siteman Cancer Center, Washington University School of Medicine, Campus Box 8109, St Louis, MO 63110, USA
1. Correspondence to: E Linos elinos{at}hsph.harvard.edu

Eleni Linos, Elizabeth Linos, and Graham Colditz investigate whether airport security screening would pass the National Screening Committee’s criteria for an effective screening test

### The tests and evidence of benefit

We systematically reviewed the literature on airport security screening tools. A systematic search of PubMed, Embase, ISI Web of Science, Lexis, Nexis, JSTOR, and Academic Search Premier (EBSCOhost) found no comprehensive studies that evaluated the effectiveness of x ray screening of passengers or hand luggage, screening with metal detectors, or screening to detect explosives. When research teams requested such information from the US Transportation Security Administration they were told that evaluating new screening programmes might be useful, but it was overshadowed by “time pressures to implement needed security measures quickly.”16 In addition, we noticed that new airport screening protocols were implemented immediately after news reports of terror threats (fig 1).

Fig 1 Timeline of changes to airport screening protocols, costs, and news events related to terrorist threats

The little we do know about airport security screening comes from investigations of the factors that influence the sensitivity of visual screening of x ray images. These studies conclude that sensitivity depends on the screener’s experience, rather than the precision of the machine. Practice improves the screener’s performance, but unfamiliar or rare objects are hard to identify regardless of experience.171819 Mammography radiologists realise this and undergo years of specialised training after medical school.20

Even without clear evidence of the accuracy of testing, the Transportation Security Administration defended its measures by reporting that more than 13 million prohibited items were intercepted in one year.21 Most of these illegal items were lighters. The screening literature shows that length time and lead time bias produce misleading interpretations of screening studies because of earlier detection of more benign cases that would not necessarily become clinically apparent (overdiagnosis). A similar problem arises with the above reasoning—although more than a million knives were seized in 2006, we do not know how many would have led to serious harm.

## The questions

The absence of scientific evaluations of the screening tools currently in place and the vast amount of money spent by governments worldwide on airport security have led us to muse over current airport security protocols and wonder about their optimal implementation. What is the sensitivity of the screening question, “Did you pack all your bags yourself?” and has anyone ever said no? Can you hide anything in your shoes that you cannot hide in your underwear? What are the ethical implications of preselecting high risk groups? Are new technologies that “see” through clothes acceptable? What hazards should we screen for? Guns and explosives certainly, but what about radioactive materials or infectious pathogens? Concerns about cost effectiveness—including the indirect costs of passengers’ time spent in long queues—will be central to future decisions, but first we need solid evidence of benefit.

## An experiment

If we were to evaluate the effectiveness of airport screening, we would start by assessing the accuracy of current tests for illegal objects in passengers’ luggage. This would yield only preliminary information on screening test performance; we would need to reapply for funding to evaluate the overall benefit of security screening on mortality and calculate the number needed to screen to prevent the death of one traveller.22 After informing the airport managers, gaining approval from research ethics committees and police, and registering our trial with one of the acceptable International Committee of Medical Journal Editors trial registries, we would select passengers at random at the check-in desks and give each traveller a small wrapped package to put in their carry-on bags. (We would do this after they have answered the question about anyone interfering with their luggage.) A total of 600 passengers would be randomised to receive a package, containing a 200 ml bottle of a non-explosive liquid, a knife, or a bag of sand of similar weight (control package) in a 1:1:1 ratio. Investigators and passengers would be blinded to the contents of the package. Our undercover investigators would measure how long it takes to get through security queues and record how many of the tagged customers are stopped and how many get through. A passenger who is stopped and asked to open the wrapped box would be classed as a positive test result, and any unopened boxes would be considered a negative test result. We would use the number of true and false positives and true and false negatives to estimate the sensitivity and specificity of the current screening process and pool the waiting times to estimate an average waiting time for each passenger (fig 2).

Fig 2 Study design flow chart for evaluation of current screening test for hand luggage

We have heard rumours that this sort of thing actually goes on—that agents occasionally carry illicit items through airport screening units to “test” them and identify gaps in security. Perhaps the evidence we are searching for is strong, but secret. And of course rigorous airport screening may have other benefits. It certainly deters the transport of any illicit object, such as less dangerous but equally unwanted plants, animals, or drugs. In addition, in the midst of mounting reports of thwarted terrorist attacks on airports, the process is comforting to frequent flyers and their families. Nevertheless, the absence of publicly available evidence to satisfy even the most basic criteria of a good screening programme concerns us.

## Conclusion

Of course, we are not proposing that money spent on unconfirmed but politically comforting efforts to identify and seize water bottles and skin moisturisers should be diverted to research on cancer or malaria vaccines. But what would the National Screening Committee recommend on airport screening? Like mammography in the 1980s, or prostate specific antigen testing and computer tomography for detecting lung cancer more recently, we would like to open airport security screening to public and academic debate. Rigorously evaluating the current system is just the first step to building a future airport security programme that is more user friendly and cost effective, and that ultimately protects passengers from realistic threats.

## Footnotes

• Thanks to Lorelei Mucci, Monica McGrath, Mike Stoto, and Pat Cox for useful discussions.

• Contributors and sources: Eleni L and GC conceived and designed the study. All authors helped collect data and write and edit the manuscript. Eleni L is guarantor. GC has worked extensively on breast and colorectal cancer screening and advises the American Cancer Society on implementation of screening programmes.

• Funding: NIH grant R25 CA098566 provided salary support for Eleni L. The funder had no role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

• Competing interests: None declared.

• Provenance and peer review: Not commissioned; externally peer reviewed.

## References

View Abstract