محمدرضا زارع نژاد

Capturing and conveying a video’s meaning and critical concepts by analyzing the subtle details and events is a fundamental yet challenging task in video interpretation. Identifying the dominant emotional tone in a video significantly enhances the perception of its atmosphere. Despite a strong emphasis on video captioning, existing models often need to adequately address emotional themes or accurately detect emotions, resulting in suboptimal captioning results. To address these limitations, this paper proposes a novel Semantic Processing and Emotion-informed video-Captioning Through Retrieva‎l and Understanding Modalities (SPECTRUM) framework to empower the generation of emotionally and semantically credible captions. Leveraging our pioneering structure, SPECTRUM discerns multimodal semantics and emotional themes using Visual-Text Attribute Investigation (VTAI). It determines the orientation of descriptive captions through a Holistic Concept-Oriented Theme (HCOT), expressing emotionally-informed and field-acquainted references. Exploiting video-to-text retrieva‎l capabilities and the multifaceted nature of video content, VTAI estimates the emotional probabilities of candidate captions. Next, the dominant theme of the video is determined through an appropriate weighting of embedded attribute vectors and the application of coarse- and fine-grained emotional concepts in HCOT, which defines the video’s contextual alignment. The generated caption of the model results from the final decoding of multimodal features and embedded attribute vectors. Furthermore, using two loss functions, SPECTRUM is optimized to integrate emotional information and minimize prediction errors. Extensive experiments on the EmVidCap, MSVD, and MSR-VTT video captioning datasets demonstrate that SPECTRUM significantly surpasses state-of-the-art methods. Quantitative and qualitative eva‎luations highlight SPECTRUM’s capability to capture and convey video emotions and multimodal attributes accurately.

كپشن گذاري ويدئو , بررسي احساسات , توصيف معنايي محتواي ويدئو

Video Captioning , Emotion Detection , semantic video representation

Mohammadreza Zarenejad

Dr. AliAsghar Beheshti Shirazi