احمد شريف

Abstract With the advent of digital health an‎d an increase in the volume of digital health-linked data created, there is an opportunity to use artificial intelligence (AI) in new ways to help improve how clinical decisions are made. However, traditional AI algorithms are limited in their ability to be clinically applied because they have constraints associated with only having one input type. The goal of research is to address the limitations by constructing a Multimodal Clinical AI Framework. This approach allow for the integration of patient data from various sources (such as medical records, physicianʹs notes, x-rays, etc.) into a single location; thus enabling a more comprehensive an‎d clearer understan‎ding of the patient’s diagnostic situation. A Large Language Model (LLM) is integrated into the proposed framework. The LLM produces human-readable, detailed explanations of the prediction so that clinicians can understan‎d how the AI-generated diagnostic reasoning worked. With this information, clinicians can see the reasoning behind the AIʹs predictions an‎d make the best possible decisions about patient care. An assessment of the method implemented in this research took the form of three distinct classification tasks-‎-pathology, clinical findings an‎d anatomical sites-‎-through application to the MIMIC-IV an‎d MIMIC-CXR datasets. The multimodal model provided a good demonstration of high performance by achieving approximately 69% accuracy overall on the three tasks of interest (i.e., moderate to strong predictive performance). In more detail, it yielded high performance levels across three types of classifications (i.e., pathology (accuracy = 74.8%, macro-F1 = 0.670), findings (accuracy = 58.0%, macro-F1 = 0.270), an‎d sites (accuracy = 74.0%, macro-F1 = 0.710)). When assessing the performance of the multimodal model via stepwise analysis, incremental improvements were gained through modality-specific embeddings, latent-space fusion an‎d end-to-end training. The system maintained high levels of meaningfulness for minority classes, demonstrating evidence that the multimodal approach is robust, as confirmed through qualitative levels of clinical relevance an‎d interpretability of LL-M based explanations.

: هوش مصنوعي چندوجهي , پشتيباني از تصميم‌گيري باليني

Multimodal AI , Clinical Decision Support

Ahmed Shareef

Dr. Behrouz Minaei-Bidgoli