Researchers introduce HalluTruthQA, a new fine-grained benchmark designed for hallucination detection, localization, and explanation in Arabic question answering. This benchmark comprises 2,400 expert-curated examples across Islamic knowledge, history, science, and geography, featuring character-level error spans, human explanations, and various hallucination types. The study evaluated four open-source Arabic LLMs (ALLaM-7B, Falcon-H1R-7B, Qwen3-32B, SILMA) across detection, localization, factual verification, and explanation tasks, revealing no single model outperforms others across all metrics. Why it matters: HalluTruthQA provides a critical tool for advancing the factual accuracy and reliability of Arabic LLMs by enabling more granular and comprehensive hallucination evaluation beyond response-level detection.
Researchers have introduced HalluTruthQA, a new fine-grained benchmark designed for hallucination detection, localization, and explanation in Arabic Question Answering. The benchmark comprises 2,400 expert-curated examples spanning four knowledge-intensive domains: Islamic knowledge, history, science, and geography, with detailed annotations including character-level erroneous spans and human-written explanations. Four open-source LLMs ( extsc{Allam}, extsc{Falcon-H1}, extsc{Qwen32}, and extsc{Silma}) were evaluated, demonstrating varied performance across detection, localization, factual verification, and explanation tasks. Why it matters: This benchmark offers a comprehensive tool for evaluating and enhancing the factual accuracy and trustworthiness of Arabic LLMs, promoting more sophisticated assessment beyond simple hallucination detection.
The paper introduces AraHalluEval, a new framework for evaluating hallucinations in Arabic and multilingual large language models (LLMs). The framework uses 12 fine-grained hallucination indicators across generative question answering and summarization tasks, evaluating 12 LLMs including Arabic-specific, multilingual, and reasoning-based models. Results show factual hallucinations are more common than faithfulness errors, with the Arabic model Allam showing lower hallucination rates. Why it matters: This work addresses a critical gap in Arabic NLP by providing a comprehensive tool for assessing and mitigating hallucination in LLMs, which is essential for reliable AI applications in the Arabic-speaking world.
This paper benchmarks the performance of large language models (LLMs) on Arabic medical natural language processing tasks using the AraHealthQA dataset. The study evaluated LLMs in multiple-choice question answering, fill-in-the-blank, and open-ended question answering scenarios. The results showed that a majority voting solution using Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3 achieved 77% accuracy on MCQs, while other LLMs achieved a BERTScore of 86.44% on open-ended questions. Why it matters: The research highlights both the potential and limitations of current LLMs in Arabic clinical contexts, providing a baseline for future improvements in Arabic medical AI.
This paper introduces MOTOR, a multimodal retrieval and re-ranking approach for medical visual question answering (MedVQA) that uses grounded captions and optimal transport to capture relationships between queries and retrieved context, leveraging both textual and visual information. MOTOR identifies clinically relevant contexts to augment VLM input, achieving higher accuracy on MedVQA datasets. Empirical analysis shows MOTOR outperforms state-of-the-art methods by an average of 6.45%.
This paper introduces two methods for creating Arabic LLM prompts at scale: translating existing English prompt datasets and creating natural language prompts from Arabic NLP datasets. Using these methods, the authors generated over 67.4 million Arabic prompts covering tasks like summarization and question answering. Fine-tuning a 7B Qwen2 model on these prompts outperforms a 70B Llama3 model in handling Arabic prompts. Why it matters: The research provides a cost-effective approach to scaling Arabic LLM training data, potentially improving the performance of smaller, more accessible models for Arabic NLP.
Researchers introduce ArabicaQA, a large-scale dataset for Arabic question answering, comprising 89,095 answerable and 3,701 unanswerable questions. They also present AraDPR, a dense passage retrieval model trained on the Arabic Wikipedia. The paper includes benchmarking of large language models (LLMs) for Arabic question answering. Why it matters: This work addresses a significant gap in Arabic NLP resources and provides valuable tools and benchmarks for advancing research in the field.
The paper introduces MedPromptX, a clinical decision support system using multimodal large language models (MLLMs), few-shot prompting (FP), and visual grounding (VG) for chest X-ray diagnosis, integrating imagery with EHR data. MedPromptX refines few-shot data dynamically for real-time adjustment to new patient scenarios and narrows the search area in X-ray images. The study introduces MedPromptX-VQA, a new visual question answering dataset, and demonstrates state-of-the-art performance with an 11% improvement in F1-score compared to baselines.