AI Powered Multimodal Dermatological Chatbot for Skin Disease Diagnosis and Consultation

Abstract

Skin diseases affect a significant majority of the population of the world, and early diagnosis is crucial for treating them successfully. To meet the needs of on-time diagnosis as well as appropriate communication, we put forward a dialogue system based on LLaMA- 3 vision LLM model. The system shall be capable of processing dermatology images along with user inputs in the form of speech to output correct, medicinally sourced answers. Utilizing large-scale pre-trained language models and embedding medical domain-specific vocabularies, the chatbot provides initial diagnosis, possible medical conditions, and suggestions for prevalent skin disorders. The audio is recorded using an audio recorder, processed with OpenAI Whisper to be converted into text, and images uploaded for examination using the LLaMA-3 Vision model. The image and transcribed text are processed by the model through GroqAPI, producing a response that is synthesized to speech using Eleven Labs or GTTS. Gradio provides an easy interface, allowing for smooth interaction with the system. The system is especially useful in areas where dermatologists are scarce, providing an initial line of consultation that is affordable and scalable. The goal is to augment conventional dermatological procedures, decrease diagnostic delays, and equip users with instant, informative feedback in response to visual and textual inputs.

Ähnliche Arbeiten