EuroLoop - Parameter-Efficient Foundation Models for Europe

AI Technology: Natural Language Processing

EuroLoop addresses the need for parameter efficient European AI reasoning models by scaling adaptive looped transformer models up to 7B parameters. 

So far, scaling language models meant scaling the parameters but recently another approach has gained considerable traction. Looped transformers are able to recursively improve their hidden representations by learning to iterate across individual blocks while reusing the same parameters. 

EuroLoop will be one of the first projects that scales this architecture beyond a proof-of-concept, by training a family of open-source loop models at 1.3B and 7B on up to 3 trillion tokens.

For this the team will be the first to establish scaling laws for looped models, which will be used to train the largest looped model to date. All models, data, and code will be released under permissive open-source research licences.