Publication

epfLLM Megatron-LLM

2023

Abstract

This library enables pre-training and fine-tuning of large language models (LLMs) at scale. Our repository is a modification of the original Megatron-LM codebase by Nvidia.

Added key features include:

architectures supported: Llama, Llama 2, Code Llama, Falcon and Mistral support training of large models (70B Llama 2, 65B Llama 1, 34B Code Llama, 40B Falcon and Mistral) on commodity hardware on multiple nodes 3-way parallelism: tensor parallel, pipeline parallel and data parallel training (inherited from Megatron) full pretraining, finetuning and instruct tuning support Support for special tokens & tokenizers grouped-query attention (GQA) and multi-query attention (MQA) Rotary Position Embeddings (RoPE), RMS layer norm, Lima dropout RoPE scaling for longer attention context support FlashAttention 2 BF16 / FP16 training WandB integration Metrics support: Ease to add custom metrics to evaluate on the validation set while training Conversion to and from Hugging Face hub

Official source

https://infoscience.epfl.ch/entities/publication/a016ec19-2957-4601-9093-0c3c8944d4fb

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

2023

Abstract

This library enables pre-training and fine-tuning of large language models (LLMs) at scale. Our repository is a modification of the original Megatron-LM codebase by Nvidia.

Added key features include:

Official source

https://infoscience.epfl.ch/entities/publication/a016ec19-2957-4601-9093-0c3c8944d4fb

About this result

Related concepts (32)

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Large language model

A large language model (LLM) is a language model characterized by its large size. Their size is enabled by AI accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Related publications (11)

epfLLM Megatron-LLM

Chat with Graph Search

Self-supervised learning for automated avalanche detection from seismic and infrasound data

ProteomeLM: A proteome-scale language model enables accurate and rapid prediction of protein–protein interactions and gene essentiality across taxa

Subseasonal-to-Seasonal strategies for the Earth System Foundation Model (ESFM)