Synthetic Image Evaluation Towards a Gamified Ranking Tool for Enhancing the Annotation Task

Abstract

While self-supervised learning reduces reliance on labeled data, high-quality annotations remain critical for rare and specialized medical tasks. Manual annotation is costly, subjective, and timeconsuming and often requires expert knowledge. Given these challenges, an innovative solution emerges: a gamified annotation framework that incorporates synthetic data to evaluate annotator performance unobtrusively while enhancing engagement. Gamification can control variability, adjust task difficulty based on user accuracy, and track annotator reliability using measurable indicators such as consistency, number of tasks completed, and accuracy levels. It distinguishes experts from non-experts through a transparent ranking mechanism without penalizing consistent performance. Therefore, we evaluate two generative approaches for synthetic image generation with conditional generative adversarial networks and diffusion models using lung fluid cell images. In a user study with 49 participants, including five experts with specific knowledge of the data provided, we assessed their ability to identify synthetic versus original images in paired and single image classification tasks. Participants more often detected generated images (48%), while diffusion-based samples were more difficult to distinguish. Experts performed slightly (5%) better but faced similar challenges with diffusion outputs. This approach enables scalable assessment of annotation quality and reduces reliance on expert supervision. Our findings reveal insights into the feasibility and challenges of using synthetic images. Future work will refine scoring mechanisms and expand the gamified framework to set a new gold standard.

Ähnliche Arbeiten