Skip to content
GCC AI Research

HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

arXiv · · Significant research

LLM Arabic AI Research NLP Ethics

Summary

Researchers have introduced HalluTruthQA, a new fine-grained benchmark designed for hallucination detection, localization, and explanation in Arabic Question Answering. The benchmark comprises 2,400 expert-curated examples spanning four knowledge-intensive domains: Islamic knowledge, history, science, and geography, with detailed annotations including character-level erroneous spans and human-written explanations. Four open-source LLMs ( extsc{Allam}, extsc{Falcon-H1}, extsc{Qwen32}, and extsc{Silma}) were evaluated, demonstrating varied performance across detection, localization, factual verification, and explanation tasks. Why it matters: This benchmark offers a comprehensive tool for evaluating and enhancing the factual accuracy and trustworthiness of Arabic LLMs, promoting more sophisticated assessment beyond simple hallucination detection.

Keywords

Hallucination detection · Arabic Question Answering · LLMs · Benchmark · Factual verification

Read original article →

Get the weekly digest

Top AI stories from the GCC region, every week.

Related

AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs

arXiv ·

The paper introduces AraHalluEval, a new framework for evaluating hallucinations in Arabic and multilingual large language models (LLMs). The framework uses 12 fine-grained hallucination indicators across generative question answering and summarization tasks, evaluating 12 LLMs including Arabic-specific, multilingual, and reasoning-based models. Results show factual hallucinations are more common than faithfulness errors, with the Arabic model Allam showing lower hallucination rates. Why it matters: This work addresses a critical gap in Arabic NLP by providing a comprehensive tool for assessing and mitigating hallucination in LLMs, which is essential for reliable AI applications in the Arabic-speaking world.