Can Epidemiology Be Applied to Auto Safety Analysis?

Recent concerns over auto safety incidents prompted three of our pharmaceutical industry experts, Dov Rothman, Mei Sheng Duh, and Paul Greenberg, to consider how epidemiologic analysis used in drug safety investigations might be applied to assess car safety data.

Can auto safety be analyzed like drug safety?

The term "epidemiology" typically refers to the study of the causes, distribution, and control of disease in the population. In the pharmaceutical industry, epidemiologic investigations often involve analysis of large, complex data sets to determine the risk of adverse events associated with the use of specific prescription drugs. The results can affect prescribing decisions, reimbursement strategies, product labels, and product withdrawals.

We have undertaken numerous analyses of adverse events data in the pharmaceutical industry and find that we can apply similar statistical techniques and methodological approaches to automotive safety analysis.

When epidemiologists examine the Food and Drug Administration’s (FDA) MedWatch data, they analyze the voluntary reports of often rare adverse drug reactions sent in by patients with a variety of health conditions and genetic predispositions. The FDA’s spontaneous MedWatch adverse event reporting system is "numerator-based," meaning that the data reflect only reported adverse events – the total universe of all prescription drug users, or the "denominator," is unknown. In addition, MedWatch is subject to reporting biases, including variability in report quality, under-reporting, and reporting rates affected by media coverage and litigations.

Similarly, to infer vehicle safety, investigators with the National Highway Transportation Safety Administration (NHTSA) often rely on data from voluntary complaints filed by transportation safety authorities and car owners with different driving habits and varying levels of interest in reporting complaints. In many ways, NHTSA investigations are subject to the same limitations and reporting inconsistencies as MedWatch investigations. Nonetheless, they offer a useful starting point from which to develop research hypotheses about auto safety data.

How safe are Toyotas?

Reports of unintended acceleration of some Toyota models have prompted consumer safety concerns, wide-ranging product recalls, and Congressional scrutiny. But are Toyotas really less safe than other vehicles? We analyzed the NHTSA publicly available vehicle complaint data to find out. Our study covered 1998 to 2008, prior to the spike in reporting about unintended acceleration. Because it is based on voluntary reports, the data set did not include all vehicle complaints and associated deaths. Furthermore, it did not distinguish the characteristics of Toyota versus non-Toyota drivers on potentially important dimensions, including safe driving habits. Despite these limitations, the data still enabled us to develop a useful perspective.

Toyotas are less safe … except when they’re not

The NHTSA data show that Toyotas appear to be less safe when the analysis focuses exclusively on speed control. As indicated in the table, the ratio of speed control complaints for Toyotas versus all other vehicles exceeded 1.00. In the language of epidemiology, the relative risk of speed control complaints was 14% higher (1.14–1.00) for Toyotas compared with all other types of vehicles. Since the 95% confidence interval (CI) does not include 1.00, the point at which there is no difference on this dimension between Toyotas and other vehicles, this result is statistically significant. For speed control related deaths, the estimated relative risk was 64% higher (1.64–1.00) for Toyotas. However, as this 95% CI includes 1.00, there are two possible interpretations: either there is no difference between Toyotas and other vehicles in terms of speed control deaths, or the small number of such events means a decisive statistical distinction cannot be drawn in this particular case. In contrast, when we account for the full range of possible safety concerns, Toyotas were associated with lower complaint and death rates than other vehicles: 53% (1.00–0.47) and 66% (1.00–0.34) lower, respectively. Each of these results is statistically significant, as the corresponding 95% CIs do not include 1.00.

Analysis of NHTSA Complaints Database
	Toyotas [1]	All Other [2]	Toyota Relative Risk [3] = [1]/[2]	Toyota Relative Risk 95% Confidence Interval
Speed control complaints per 1,000,000 vehicles	73.87	64.71	1.14	[1.08, 1.20]
Speed control deaths per 1,000,000 vehicles	0.46	0.28	1.64	[0.83, 3.26]
All complaints per 1,000,000 vehicles	926.48	1,953.38	0.47	[0.47, 0.48]
All deaths per 1,000,000 vehicles	2.78	8.17	0.34	[0.26, 0.44]

Notes and Sources

 
Of course, there are meaningful differences between safety analyses of prescription drugs and automobiles. For example, pharmaceutical product safety typically is reported in a disaggregated manner at the product level, reflecting that drug safety profiles can differ markedly across therapeutic areas. By contrast, "product" safety in the automobile industry often is discussed at the manufacturer level. Though specific models of cars, trucks, and SUVs likely have very different safety profiles, broad comparisons of vehicle manufacturers’ entire lines often fail to adjust for this composition effect.

Both the FDA and NHTSA consider product safety narrowly in many instances, focusing on specific types of adverse events or complaints versus overall safety profiles. But as we have shown above, it is possible to draw very different conclusions about product safety depending on whether specific or overall safety is the outcome of interest.

Our speed control example illustrates that insights obtained from data analysis are highly dependent on the questions asked. The implications for consumers can be meaningful, as a car buyer who avoids all Toyotas for fear of speed-control injury may choose another vehicle type that actually is less safe overall. Only by studying the data in the appropriate manner does this insight become clear. ■