Lean Neural Networks

AI Technology: Deep Learning

Neural networks have demonstrated remarkable success in various domains, such as computer vision, natural language processing, protein folding, drug interaction prediction, and many others.

However, most cutting-edge applications rely on large neural networks with billions of parameters, demanding substantial computational resources (e.g., expensive GPUs) for training and inference. 

Large neural networks cannot be easily used in embedded or low-power devices, where the amount of available computational power is limited. 

Running a neural network in offline systems is beneficial in many cases, especially in privacy-sensitive environments.

Even though widespread use of large neural networks is likely inevitable, specific tasks can be handled by much leaner networks that can be deployed on regular computers, mobile devices, etc. This should expand the possibilities for applications and lower deployment costs. 

This project aims to design and develop innovative techniques for training lean neural networks that maintain high accuracy while drastically reducing required computational resources. 

Thus, these networks will be accessible for deployment on regular computers and mobile devices. 

The study will mainly focus on improving the efficiency of sparse neural networks. The research team's primary targets are large language models, but our methods are applicable to any neural network.