A study investigated the impact of domain-specific pretraining on Transformer models for modeling digital pragmatics in Arabic-English code-switched discourse. It compared MARBERT and XLM-R, using BERT as a baseline, on a dataset of 11695 X posts. MARBERT significantly outperformed XLM-R, achieving a Macro F1 of 0.85 on an independent test set compared to XLM-R's 0.52. The research concludes that Transformer performance in specialized pragmatic classification tasks relies more on a domain-specific pretraining profile than on multilingual coverage alone. Why it matters: This highlights the critical importance of tailored pretraining for achieving optimal performance in complex Arabic NLP tasks, particularly in code-switching contexts prevalent in the region.
Arabic Language Models (LMs) are primarily pretrained on Modern Standard Arabic (MSA), with an expectation of transferring to diverse Arabic dialects for real-world applications. This work explores cross-lingual transfer in Arabic LMs using probing on three Natural Language Processing (NLP) tasks and representational similarity. The findings indicate that transfer is possible but disproportionate across dialects, with some evidence of negative interference in models trained to support all Arabic dialects. Why it matters: This research highlights crucial challenges for building robust Arabic AI systems that effectively handle the significant linguistic diversity of the Arab world.