A new methodology emulating fact-checker criteria assesses news outlet factuality and bias using LLMs. The approach uses prompts based on fact-checking criteria to elicit and aggregate LLM responses for predictions. Experiments demonstrate improvements over baselines, with error analysis on media popularity and region, and a released dataset/code at https://github.com/mbzuai-nlp/llm-media-profiling.
MBZUAI student Zain Muhammad Mujahid is researching methods to detect media bias using NLP and LLMs. His approach profiles bias across media outlets using LLMs like ChatGPT to predict bias based on 16 identifiers. The research aims to develop a tool that instantly provides a bias profile for a given media URL. Why it matters: This research has the potential to combat misinformation and enhance media literacy in the region by providing tools to identify biased reporting, and it is expanding to Arabic and other languages.
MBZUAI researchers presented new resources at EMNLP for improving the factuality of LLMs, including a web application for fact-checking LLM-generated text and benchmarks for evaluating automated fact-checkers. They found that current automated fact-checkers miss nearly 40% of false claims generated by LLMs. The study breaks down the fact-checking process into eight tasks, including decomposition and decontextualization, to identify where systems fail. Why it matters: This work addresses a critical challenge in the deployment of LLMs by providing tools and methods for improving their reliability and trustworthiness, which is essential for widespread adoption in sensitive applications.
Researchers at MBZUAI have introduced QRAFT, an LLM-based framework designed to automate the generation of fact-checking articles. The system mimics the writing workflow of human fact-checkers, aiming to bridge the gap between automated fact-checking systems and public dissemination. While QRAFT outperforms existing text-generation methods, it still falls short of expert-written articles, highlighting areas for further research.
Iryna Gurevych from TU Darmstadt presented research on using large language models for real-world fact-checking, focusing on dismantling misleading narratives from misinterpreted scientific publications and detecting misinformation via visual content. The research aims to explain why a false claim was believed, why it is false, and why the alternative is correct. Why it matters: Addressing misinformation, especially when supported by seemingly credible sources, is critical for public health, conflict resolution, and maintaining trust in institutions in the Middle East and globally.