Guido-12B-A2B: An Open-Weight Hybrid MoE Language Model with Persistent Neural Memory, Trained End-to-End on European HPC

The researchers request 500,000 GPU hours on MareNostrum 5 ACC to train "Guido-12B-A2B", an open-weight language model that is, to their knowledge, the first to combine four complementary compute pathways in a single production-scale architecture: "Kimi Delta Attention (KDA)" linear attention (O(N)), "Multi-Head Latent Attention (MLA)" (O(N²) with ~95% KV-cache compression), "Titans persistent neural memory" that learns at inference time (O(1) recall), and "fine-grained DeepSeek-style Mixture-of-Experts" (48 routed + 2 shared experts, top-4). The model is 12.2B total parameters but only 2.05B active per token, giving it the inference cost of a ~2B dense model with the capacity of a 12B one. This proposal builds directly on a completed EuroHPC Playground campaign (5,000 GPU-h, project `EHPC-AIF-2026PG01-050`) that validated the architecture and produced the first systematic stabilization study of Titans memory at scale, and on an "active 50,000 GPU-h Fast Lane allocation" (`EHPC-AIF-2026FL01-249`) that is running a three-arm routing-and-systems pilot to lock the final configuration before this production run. Guido-12B-A2B will be trained on ~8.4T tokens of open, permissively-licensed European and open data, post-trained with constitutional-AI safety alignment, fully evaluated, and "released as open weights under Apache 2.0" together with training code, recipe, and complete training logs — a novel-architecture contribution to European AI sovereignty trained entirely on EuroHPC infrastructure.