Training Large Language Models at Scale Using Modalities
Day 2 | 15:15 – 15:45 | Workshop Room 3
Dr. Mehdi Ali
Lamarr Institute
Dr. Max Lübbering
Fraunhofer IAIS
Abstract
Large Language Models (LLM) have demonstrated impressive results across various applications. However, training these models requires not only vast datasets and substantial computational resources but also efficient frameworks that can operate in a distributed setting on thousands of GPUs. In this workshop, we present Modalities, our open-source framework for training foundation models at scale. In the first part of the workshop, we will introduce the key features of Modalities and share our recent results from our scaling experiments where we trained models with up to 30 billion parameters at one of Europe’s largest high-performance computing centers. In the second part of the workshop, we will train a model hands-on from scratch using Modalities. Key topics covered will include tokenization, training, training efficiency, and monitoring.
Dr. Mehdi Ali
Mehdi Ali is a research scientist at Fraunhofer IAIS and Innovation Group Leader on Foundation Model Research at the Lamarr Institute. His group is strongly involved in national and international projects focused on training Large Language Models, such as OpenGPT-X, TrustLLM, and EuroLingua-GPT.
His research focuses on foundation models and knowledge graph representation learning. Mehdi’s work has been published in top-tier machine learning journals and conferences, including the Journal of Machine Learning Research (JMLR), IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) and the International Semantic Web Conference (ISWC). In 2021, he and his co-authors received the Best Paper Award at ISWC for their paper, “Improving Inductive Link Prediction Using Hyper-Relational Facts.”
Additionally, Mehdi is the founder of PyKEEN, a community project that facilitates the efficient training and evaluation of knowledge graph embedding models. Recently, he co-founded Modalities, an open-source framework for training multimodal foundation models at scale.
Dr. Max Lübbering
Dr. Max Lübbering leads the Machine Learning Engineering Team in the Media Engineering Department at Fraunhofer IAIS and is a core member of the Eurolingua-GPT team. He is also a core developer of Modalities, an open-source framework designed for large-scale foundation model training. During his PhD, Dr. Lübbering proposed novel architectural adaptations to deep neural networks, enabling them to recognize their own limitations in open-world settings.