A Mixture of Experts (MoE) layer is a sparsely activated deep learning layer. It uses a router network to direct each token to one of the experts. Yuanzhi Li, an assistant professor at CMU and affiliated faculty at MBZUAI, researches deep learning theory and NLP. Why it matters: This highlights MBZUAI's engagement with cutting-edge deep learning research, specifically in efficient model design.
An associate professor of Statistics at the University of Toronto gave a talk on how ensemble learning stabilizes and improves the generalization performance of an individual interpolator. The talk focused on bagged linear interpolators and introduced the multiplier-bootstrap-based bagged least square estimator. The multiplier bootstrap encompasses the classical bootstrap with replacement as a special case, along with a Bernoulli bootstrap variant. Why it matters: While the talk occurred at MBZUAI, the content is about ensemble learning which is a core area for improving AI model performance, and is of general interest to the AI research community.
This article discusses a talk by Dr. David Xianfeng Gu at MBZUAI on gaining a geometric understanding of deep learning. The talk addresses questions such as what a DL system learns, how it learns, and how to improve the learning process. Dr. Gu is a professor at SUNY Stony Brook and affiliated with multiple prestigious institutions. Why it matters: Understanding the fundamentals of deep learning is crucial for advancing AI research and development in the region.
A recent study questions the necessity of deep ensembles, which improve accuracy and match larger models. The study demonstrates that ensemble diversity does not meaningfully improve uncertainty quantification on out-of-distribution data. It also reveals that the out-of-distribution performance of ensembles is strongly determined by their in-distribution performance. Why it matters: The findings suggest that larger, single neural networks can replicate the benefits of deep ensembles, potentially simplifying model deployment and reducing computational costs in the region.
The study compares deep learning models trained via transfer learning from ImageNet (TII-models) against those trained solely on medical images (LMI-models) for disease segmentation. Results show that combining outputs from both model types can improve segmentation performance by up to 10% in certain scenarios. A repository of models, code, and over 10,000 medical images is available on GitHub to facilitate further research.