A Systematic Approach to Evaluation of Delivered AI Solutions

Abstract

<div class="section abstract"> <div class="htmlview paragraph">An approach to performance assessment and evaluation of robustness' of delivered AI (artificial intelligence) solutions (e.g., Aided Target Detection and Recognition (AiTDR) or ground vehicle autonomy behaviors) is presented. The initial development assumes the AI solution is delivered as a blackbox solution taking specified inputs and delivering output or outputs per stated requirements. The methods developed seek to not only confirm that requirements are met but to confirm performance in a manner that provides supporting evidence of which input information contributed to the AI output. Methods are developed for both AI solutions that take a single sample input as well as AIs that take a sequence of inputs to produce an output. Examples are included for convolutional neural networks (CNN). Planned extensions to the blackbox methods are discussed. Extensions include adaptations to support Residual Network (ResNet) solutions, evaluation of solution robustness when hidden layer activations are exposed, and development of metrics and methodologies to support AI solution training by ensuring the AI solution is properly trained, not over trained, or undertrained. A process is proposed for assessment of the AI solution with metrics to identify primary contributing inputs to the solution and an approximate indication of where the solution falls along the training scale.</div> </div>

Ähnliche Arbeiten