Abstract
As human-managed systems become more complex, automated anomaly detection can provide assistance—but only if it is effective. Rigorous evaluation of automated detection is vital for determining its effectiveness before implementation into systems. We identified recurring issues in evaluation practices limiting the conclusions that can be applied from published studies to broader application. In this paper, we demonstrate the implications of these issues and illustrate solutions. We show how receiver operating characteristic curves can reveal performance tradeoffs masked by reporting of single metric results and how using multiple simulation data examples can prevent biases that result from evaluation using single training and testing examples. We also provide methods for incorporating detection latency into tradeoff analyses. Application of these methods will help to provide researchers, engineers, and decision makers with a more objective basis for anomaly detection performance evaluation, resulting in greater utility, better performance, and cost savings in systems engineering.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Burns, C.M.: Towards proactive monitoring in the petrochemical industry. Saf. Sci. 44, 27–36 (2006)
Russell, E.L., Chiang, L.H., Braatz, R.D.: Fault detection in industrial processes using canonical variate analysis and dynamic principal component analysis. Chemomther. Intell. Lab. Syst. 51, 81–93 (2000)
Sommer, R., Paxson, V.: Outside the closed world: on using machine learning for network intrusion detection. In: 2010 IEEE Symposium on Security and Privacy, pp. 305–316 (2010)
Yin, S., Ding, S.X., Haghani, A., Hao, H., Zhang, P.: A comparison study of basic data-driven fault diagnosis and process monitoring methods on the benchmark Tennessee Eastman process. J. Process Control 22, 1567–1581 (2012)
Mahadevan, S., Shah, S.L.: Fault detection and diagnosis in process data using one-class support vector machines. J. Process Control 19, 1627–1639 (2009)
Rato, T.J., Reis, M.S.: Fault detection in the tennessee eastman benchmark process using dynamic principal components analysis based on decorrelated residuals (DPCA-DR). Chemomther. Intell. Lab. Syst. 125, 101–108 (2013)
Lee, J.-M., Yoo, C., Lee, I.-B.: Statistical process monitoring with independent component analysis. J. Process Control 14, 467–485 (2004)
Cheng, C.-Y., Hsu, C.-C., Chen, M.-C.: Adaptive kernel principal component analysis (KPCA) for monitoring small disturbances of nonlinear processes. Ind. Eng. Chem. Res. 49, 2254–2262 (2010)
Hsu, C.-C., Chen, M.-C., Chen, L.-S.: Integrating Independent Component Analysis and Support Vector Machine for Multivariate Process Monitoring. Comput. Ind. Eng. 59, 145–156 (2010)
Lee, J.-M., Qin, S.J., Lee, I.-B.: Modified independent component analysis for multivariate statistical process monitoring. IFAC Proc. 39, 1133–1138 (2006)
Downs, J.J., Vogel, E.F.: A plant-wide industrial process control problem. Comput. Chem. Eng. 17, 245–255 (1993)
Chiang, L.H., Russell, E.L., Braatz, R.D.: Fault diagnosis in chemical processes using fisher discriminant analysis, discriminant partial least squares, and principal component analysis. Chemomther. Intell. Lab. Syst. 50, 243–252 (2000)
Lindsay, R.C., Wells, G.L.: Improving eyewitness identifications from lineups: simultaneous versus sequential lineup presentation. J. Appl. Psychol. 70, 556 (1985)
Gronlund, S.D., Mickes, L., Wixted, J.T., Clark, S.E.: Conducting an eyewitness lineup: how the research got it wrong. Psychol. Learn. Motiv. 63, 1–43 (2015)
Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27, 861–874 (2006)
Lee, J.-M., Qin, S.J., Lee, I.-B.: Fault detection and diagnosis based on modified independent component analysis. AIChE J. 52, 3501–3514 (2006)
Khalastchi, E., Kalech, M., Kaminka, G.A., Lin, R.: Online data-driven anomaly detection in autonomous robots. Knowl. Inf. Syst. 43, 657–688 (2014)
Fawcett, T., Provost, F.: Activity monitoring: noticing interesting changes in behavior. In: Proceedings of Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. 1, pp. 53–62 (1999)
Kleinman, K.P., Abrams, A.M.: Assessing surveillance using sensitivity, specificity and timeliness. Stat. Methods Med. Res. 15, 445–464 (2006)
Wagenmakers, E.-J., van der Maas, H.L.J., Dolan, C.V., Grasman, R.P.P.P.: EZ does it! Extensions of the EZ-diffusion model. Psychon. Bull. Rev. 15, 1229–1235 (2008)
Lavin, A., Ahmad, S.: Evaluating real-time anomaly detection algorithms–the Numenta anomaly benchmark. In: 2015 IEEE 14th International Conference on Machine Learning and Application (ICMLA), pp. 38–44 (2015)
Heagerty, P.J., Zheng, Y.: Survival model predictive accuracy and ROC curves. Biometrics 61, 92–105 (2005)
Hewitt, P., Lane, M.A.: Human factors engineering delivers ROI. Control Glob. 1–2 (2017)
Tvaryanas, A.P., Thompson, W.T., Constable, S.H.: Human factors in remotely piloted aircraft operations: HFACS analysis of 221 mishaps over 10 years. Aviat. Space Environ. Med. 77, 724–732 (2006)
Sorkin, R., Woods, D.D.: Systems with human monitors: a signal detection analysis. Hum. Comput. Interact. 1, 49–75 (1985)
Acknowledgments
This work was sponsored by the Office of Naval Research, Human & Bioengineered Systems (ONR 341), program officer Dr. Jeffrey G. Morrison under contract N00014-15-C-5003. The authors thank Drs. Harvey Smallman and Shannon Toftely for their advice and assistance. The views expressed are those of the authors and do not reflect the official policy or position of the Office of Naval Research, Department of Defense, or US Government.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Rieth, C.A., Amsel, B.D., Tran, R., Cook, M.B. (2018). Issues and Advances in Anomaly Detection Evaluation for Joint Human-Automated Systems. In: Chen, J. (eds) Advances in Human Factors in Robots and Unmanned Systems. AHFE 2017. Advances in Intelligent Systems and Computing, vol 595. Springer, Cham. https://doi.org/10.1007/978-3-319-60384-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-60384-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-60383-4
Online ISBN: 978-3-319-60384-1
eBook Packages: EngineeringEngineering (R0)