Lecture

Model Compression: Techniques for Efficient NLP Models

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture on model compression in modern NLP covers various techniques such as pruning, quantization, weight factorization, weight sharing, knowledge distillation, and sub-quadratic transformers. It discusses the challenges of large language models, the benefits of compressing models, and the impact on memory footprint and inference time. The lecture also explores the concept of knowledge distillation, leveraging soft labels for student models, and the importance of attention mechanisms in transformer models.

Official source

Ontological neighbourhood

Information engineering

Signal processing: Data compression

Related concepts (32)

Lossless compression

Lossless compression is a class of data compression that allows the original data to be perfectly reconstructed from the compressed data with no loss of information. Lossless compression is possible because most real-world data exhibits statistical redundancy. By contrast, lossy compression permits reconstruction only of an approximation of the original data, though usually with greatly improved compression rates (and therefore reduced media sizes).

Data compression

In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression. Lossy compression reduces bits by removing unnecessary or less important information.

Lossy compression

In information technology, lossy compression or irreversible compression is the class of data compression methods that uses inexact approximations and partial data discarding to represent the content. These techniques are used to reduce data size for storing, handling, and transmitting content. The different versions of the photo of the cat on this page show how higher degrees of approximation create coarser images as more details are removed. This is opposed to lossless data compression (reversible data compression) which does not degrade the data.

Compression artifact

A compression artifact (or artefact) is a noticeable distortion of media (including , audio, and video) caused by the application of lossy compression. Lossy data compression involves discarding some of the media's data so that it becomes small enough to be stored within the desired or transmitted (streamed) within the available bandwidth (known as the data rate or bit rate). If the compressor cannot store enough data in the compressed version, the result is a loss of quality, or introduction of artifacts.

Image compression

Image compression is a type of data compression applied to s, to reduce their cost for storage or transmission. Algorithms may take advantage of visual perception and the statistical properties of image data to provide superior results compared with generic data compression methods which are used for other digital data. Image compression may be lossy or lossless. Lossless compression is preferred for archival purposes and often for medical imaging, technical drawings, clip art, or comics.

Related lectures (64)

Neural Signal Compression

Explores analog-to-digital conversion, neural signal optimization, multichannel architectures, and on-chip compression techniques in neuroengineering.

Model Compression Techniques: Enhancing Neural Networks

Covers model compression techniques to enhance the efficiency of large language models in production settings.

Second-Order Model Compression

Explores second-order model compression for massive deep neural networks, showcasing compression techniques and their impact on model accuracy.

JPEG 2000: Image Compression

Explores image compression principles, focusing on JPEG 2000, covering transform-based coding, quantization, entropy coding, region of interest, error resilience, and software implementations.

MP3 Compression