Testing LLMs safety in Arabic from two perspectives | NAACL

Summary

Researchers at MBZUAI presented a new Arabic dataset at NAACL to measure LLM safety, building on a Chinese dataset called 'Do Not Answer'. The dataset includes nearly 5,800 questions with challenges and harmless requests containing sensitive terms to test for over-sensitivity. The team localized cultural concepts and added 3,000 questions specific to Arabic language and culture. Why it matters: This comprehensive benchmark, accounting for the diversity of Arabic dialects and cultures, advances the development of safer and more culturally aligned LLMs for Arabic speakers.