Researchers have introduced BloomBench, a new cognitively human-grounded, bilingual (English-Arabic) multimodal benchmark for Vision-Language Models (VLMs), as part of the Almieyar benchmarking series. Grounded in Bloom's Taxonomy, it systematically evaluates six levels of cognition—Remember, Understand, Apply, Analyze, Evaluate, Create—through carefully designed image-question-answer tasks. A comprehensive study using BloomBench revealed that state-of-the-art VLMs exhibit strong semantic understanding but struggle significantly with factual recall and creative synthesis, alongside a critical performance gap between Arabic and English. Why it matters: This benchmark provides a crucial tool for diagnosing cognitive weaknesses in current VLMs and lays the groundwork for developing more cognitively aligned and inclusive multimodal AI, particularly for cross-lingual applications.
The Technology Innovation Institute (TII) in Abu Dhabi has partnered with Quantinuum to gain access to advanced quantum computing systems, including Quantinuum's upcoming Helios system. This agreement aims to accelerate the development and benchmarking of quantum computing algorithms and commercial applications in the UAE. TII's Quantum Research Center will leverage this access to improve quantum algorithms and train algorithm developers. Why it matters: This partnership strengthens the UAE's position as a global leader in science and technology by providing access to state-of-the-art quantum computing resources.
The Open Arabic LLM Leaderboard (OALL) has been launched to benchmark Arabic language models, addressing the gap in resources for non-English NLP. It incorporates datasets like AlGhafa, ACVA, and translated versions of MMLU and EXAMS from the AceGPT suite. The leaderboard uses normalized log likelihood accuracy for tasks, built around HuggingFace’s LightEval framework. Why it matters: This initiative promotes research and development in Arabic NLP, serving over 380 million Arabic speakers by enhancing the evaluation and improvement of Arabic LLMs.
The Technology Innovation Institute (TII) in Abu Dhabi, in collaboration with NVIDIA, has demonstrated large-scale simulations of the adiabatic quantum annealing (QA) algorithm for problem instances involving up to 500,000 qubits. TII's simulator achieved solution quality exceeding that of all solvers evaluated from the MQLib repository, a library for combinatorial optimization benchmarking. The emulator is accessible to external users via an experimental cloud platform hosted at https://q-inspired.tii.ae. Why it matters: This collaboration expands the range of complex optimization problems that can be investigated using quantum-inspired approaches, beyond those currently achievable with near-term quantum hardware.
The paper introduces ArabicNumBench, a benchmark for evaluating LLMs on Arabic number reading using both Eastern and Western Arabic numerals. It evaluates 71 models from 10 providers on 210 number reading tasks, using zero-shot, zero-shot CoT, few-shot, and few-shot CoT prompting strategies. The results show substantial performance variation, with few-shot CoT prompting achieving 2.8x higher accuracy than zero-shot approaches. Why it matters: The benchmark establishes baselines for Arabic number comprehension and provides guidance for model selection in production Arabic NLP systems.
This paper benchmarks the performance of large language models (LLMs) on Arabic medical natural language processing tasks using the AraHealthQA dataset. The study evaluated LLMs in multiple-choice question answering, fill-in-the-blank, and open-ended question answering scenarios. The results showed that a majority voting solution using Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3 achieved 77% accuracy on MCQs, while other LLMs achieved a BERTScore of 86.44% on open-ended questions. Why it matters: The research highlights both the potential and limitations of current LLMs in Arabic clinical contexts, providing a baseline for future improvements in Arabic medical AI.
Saudi Arabia's national AI authority, SDAIA (Saudi Data and AI Authority), is reportedly developing a National AI Index. This index aims to measure and track the Kingdom's progress and capabilities across various dimensions of artificial intelligence. It is expected to encompass areas such as research and development, talent, infrastructure, and ethical considerations. Why it matters: This initiative provides a crucial framework for evaluating Saudi Arabia's AI ecosystem, guiding future strategic investments, and benchmarking its national AI capabilities against international standards.
MBZUAI researchers introduce VideoMathQA, a new benchmark for evaluating mathematical reasoning in videos, requiring models to interpret visual information, text, and spoken cues. The dataset spans 10 mathematical domains with videos ranging from 10 seconds to over 1 hour, and includes multi-step reasoning annotations. The benchmark aims to evaluate temporal cross-modal reasoning and highlights the limitations of existing approaches in complex video-based mathematical problem solving.