Cross-Lingual Federated Learning of LLM-based ASR models and LLMs

The recent trend in literature of utilising a small connector network as the means for bridging the latent representation spaces of pre-trained foundation models has recently been adopted by several works in literature dealing with multimodal data. The approach typically entails keeping frozen the pre-trained foundation models -- speech encoder and the LLM -- during training while solely training the connector parameters to facilitate a mapping between the speech encoder's embeddings and the lexicon embedding space of the LLM. Within the scope of this research, the objective is to apply the Federation Learning framework to the fine-tuning process of previously developed connectors, with a particular focus on enhancing their performance in a multilingual context. The final LLM-based ASR and LLMs will be employed as core backbone for different pilots and virtual assistant technologies developed within the EU project ELOQUENCE.

This research comprises part of the tasks within the funded EU project ELOQUENCE (GA 101135916). ELOQUENCE aims to develop state-of-the-art multilingual, multimodal, and context-aware conversational AI technologies designed for safety-critical and socially sensitive applications. 

The Team participating in this project is composed of several partners from the ELOQUENCE consortium including: Telefónica Innovación Digital (TID - Large Enterprise, Spain) as PI and Project Coordinator of ELOQUENCE, Barcelona Supercomputing Center (BSC - Research Center Spain), Fondazione Bruno Kessler (FBK - Research Center Italy), University of Essex (UESSEX - Academia, UK) , Brunel University of London (BUL - Academia, UK). 

This development builds on previous work funded by EuroHPC Benchmark Grant EHPC-BEN-2024B12-071 and EuroHPC Development Grant EHPC-DEV-2025D12-133, where testing and measurements and technical feasibility of previous algorithms have been performed on MN5 ACC resources.