On the disagreement problem in Human-in-the-Loop federated machine learning

Abstract

The popularity of Artificial Intelligence (AI) has risen sharply in recent years, revolutionizing applications in most sectors with unprecedented functionalities. Milestones and achievements like ChatGPT demonstrate not only the impressive capabilities of AI, but also how accessible such technologies have become in recent times. However, the success of AI applications depends heavily on the underlying information integration processes. Among the most important processes are the training of the AI model at the core of the application and the collection and pre-processing of training data. In particular, the task of collecting high-quality training data can be very costly and resource-intensive, as in many cases large amounts of data have to be annotated manually. Human annotators must have extensive expertise for certain tasks in order to provide high-quality training data. In this paper, we present a framework to maximize the efficiency of human experts in a Machine Learning (ML) scenario, with the aim of optimizing the use of human expertise in active learning. This is done by constantly measuring the quality of human experts’ input, as well as by involving human annotators only when needed. We showcase the benefits of our proposed framework by applying it to a problem in image classification, proving its usefulness to reduce the cost of annotating training data. The source code of the framework is publicly available at https://github.com/human-centered-ai-lab/app-HITL-annotator . • Optimizing human expertise in AI training boosts efficiency in image classification tasks. • Enhancing AI training by efficiently leveraging human expertise in active learning. • Maximizing human input efficiency in AI training through targeted expert involvement. • Streamlining AI training by optimizing the role of human experts in data annotation. • Efficient use of expert input enhances AI training in resource-intensive tasks.

Ähnliche Arbeiten