Deviation from perfect performance measures the diagnostic utility of eyewitness lineups but partial area under the roc curve does not.

Abstract

In this paper, we explain why partial Area Under the ROC Curve often does not inform on which of two lineup procedures has superior diagnostic utility. We then introduce a novel ROC-based measure - deviation from perfect performance - that consistently indicates which of two procedures has superior diagnostic utility. The ROC_Example file available on this page provides a script for calculating the pAUC in R using the pROC package (Xavier et al., 2011) and for using the 'ggroc' function (Wu, 2015) to plot ROC curves with ggplot2 (Wickham, 2009). Hypothetical data corresponding to Figure 1 from Smith, Lampinen, Wells, Smalarz, and Mackovichova (Accepted, JARMAC) are also provided. There are four Deviation from Perfect Performance example files: DPP_Average, DPP_HighConfidence, DPP_min, and DPP_terminal. These functions correspond to different ways that we might define utility (Lampinen, Smith, Wells, Accepted, Law and Human Behavior). Average DPP corresponds to the average utility of an ID procedure and is the most similar of these measures to the partial Area Under the Curve, but instead of providing an index of the average culprit identification rate that can be achieved for some range of innocent-suspect identification rates, this measure indicates the average utility that can be achieved across all levels of eyewitness confidence (or ROC operating points). High-confidence DPP corresponds to the utility that can be achieved at high levels of confidence. Terminal-point DPP corresponds to the utility a procedure can achieve when collapsing over all levels of confidence. Finally, minimum DPP (which is equal to maximum utility) corresponds to the point of maximum utility on the ROC curve. In each of these example files, the script calculates DPP for two different lineup ROC curves and uses the 'boot' function (Canty & Ripley, 2017) to inferentially compare the DPP of the two lineup procedures. See Smith et al. (Accepted, JARMAC) for more information.

Ähnliche Arbeiten