MBZUAI · Significant research
MBZUAI researchers developed Mobile-VideoGPT, a compact and efficient multimodal model for real-time video understanding on edge devices. The system uses keyframe selection, efficient token projection, and a Qwen-2.5-0.5B language model. Testing showed that Mobile-VideoGPT is faster and performs better than other models while being significantly smaller, and the model and code are publicly available. Why it matters: This research enables on-device AI processing for video, reducing reliance on remote servers and addressing privacy concerns, which can accelerate the adoption of AI in mobile and embedded applications.
multimodal · video understanding · edge devices · MBZUAI · Mobile-VideoGPT
Get the weekly digest
Top AI stories from the GCC region, every week.
arXiv ·
Video-ChatGPT is a new multimodal model that combines a video-adapted visual encoder with a large language model (LLM) to enable detailed video understanding and conversation. The authors introduce a new dataset of 100,000 video-instruction pairs for training the model. They also develop a quantitative evaluation framework for video-based dialogue models.
arXiv ·
This paper introduces Adaptive Entropy-aware Optimization (AEO), a new framework to tackle Multimodal Open-set Test-time Adaptation (MM-OSTTA). AEO uses Unknown-aware Adaptive Entropy Optimization (UAE) and Adaptive Modality Prediction Discrepancy Optimization (AMP) to distinguish unknown class samples during online adaptation by amplifying the entropy difference between known and unknown samples. The study establishes a new benchmark derived from existing datasets with five modalities and evaluates AEO's performance across various domain shift scenarios, demonstrating its effectiveness in long-term and continual MM-OSTTA settings.
arXiv ·
A new benchmark, LongShOTBench, is introduced for evaluating multimodal reasoning and tool use in long videos, featuring open-ended questions and diagnostic rubrics. The benchmark addresses the limitations of existing datasets by combining temporal length and multimodal richness, using human-validated samples. LongShOTAgent, an agentic system, is also presented for analyzing long videos, with both the benchmark and agent demonstrating the challenges faced by state-of-the-art MLLMs.