Lecture

Scaling Language Models

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the scaling of language models, discussing the challenges and benefits of training and deploying large models. It explores the evolution of model sizes over the years, the impact of model size on performance, and the trade-offs between model size, data size, and compute resources. The instructor emphasizes the importance of scaling laws in estimating optimal model and dataset sizes for a fixed compute budget.

Official source

Related concepts (25)

Scale model

A scale model is a physical model which is geometrically similar to an object (known as the prototype). Scale models are generally smaller than large prototypes such as vehicles, buildings, or people; but may be larger than small prototypes such as anatomical structures or subatomic particles. Models built to the same scale as the prototype are called mockups. Scale models are used as tools in engineering design and testing, promotion and sales, filmmaking special effects, military strategy, and hobbies such as rail transport modeling, wargaming and racing; and as toys.

Rail transport modelling scales

Rail transport modelling uses a variety of scales (ratio between the real world and the model) to ensure scale models look correct when placed next to each other. Model railway scales are standardized worldwide by many organizations and hobbyist groups. Some of the scales are recognized globally, while others are less widespread and, in many cases, virtually unknown outside their circle of origin. Scales may be expressed as a numeric ratio (e.g. 1/87 or 1:87) or as letters defined in rail transport modelling standards (e.

Computer performance

In computing, computer performance is the amount of useful work accomplished by a computer system. Outside of specific contexts, computer performance is estimated in terms of accuracy, efficiency and speed of executing computer program instructions. When it comes to high computer performance, one or more of the following factors might be involved: Short response time for a given piece of work. High throughput (rate of processing work). Low utilization of computing resource(s). Fast (or highly compact) data compression and decompression.

Cloud computing

Cloud computing is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user. Large clouds often have functions distributed over multiple locations, each of which is a data center. Cloud computing relies on sharing of resources to achieve coherence and typically uses a pay-as-you-go model, which can help in reducing capital expenses but may also lead to unexpected operating expenses for users.

Performance engineering

Performance engineering encompasses the techniques applied during a systems development life cycle to ensure the non-functional requirements for performance (such as throughput, latency, or memory usage) will be met. It may be alternatively referred to as systems performance engineering within systems engineering, and software performance engineering or application performance engineering within software engineering.

Related lectures (16)

Fixing Serverless Computing with Open-Source Hardware

Explores challenges and solutions in serverless computing, focusing on energy usage, cloud utilization, FaaS architecture, and performance analysis.

Vision-Language-Action Models: Training and Applications

Delves into training and applications of Vision-Language-Action models, emphasizing large language models' role in robotic control and the transfer of web knowledge. Results from experiments and future research directions are highlighted.

Understanding Stereotype Threat

Delves into stereotype threat and its impact on performance due to fear of confirming stereotypes.

Stereotype Threat: Effects and Mitigation

Delves into how negative stereotypes can impact performance and the strategies proposed to mitigate these effects.

Language Models: From Theory to Computation