Multi-lingual Scaling Laws for Mixture of Expert Language Models

This project investigates the architectural foundations needed to build fully open, auditable, and regulation-compliant large language models tailored to Europe’s multilingual landscape. Current state-of-the-art models are predominantly non-European and opaque in their training data, methods, and governance, limiting scientific reproducibility and hindering compliance with the EU AI Act. They also systematically underperform on low-resource European languages. To address this, the OpenEuroLLM initiative aims to develop frontier-level foundation models that support all official European languages while ensuring digital sovereignty. A key open question is whether dense transformer architectures, currently used by most European efforts, are inherently more stable for multilingual training than the sparse Mixture-of-Experts (MoE) architectures now favoured by global frontier models. This project will fill that gap by deriving multilingual scaling laws for both dense and MoE models, enabling rigorous comparison of their performance, efficiency, and learning dynamics. Using compute-optimal training approaches and probabilistic hyperparameter optimization, we will train models across a range of sizes and token budgets to identify architecture-specific optima. The results will guide the design of the first OpenEuroLLM flagship model family, planned for release in 2026, and provide a robust, reproducible workflow, along with empirical insights, that will benefit the broader European open-source LLM ecosystem.