Lecture

Training Strategies for Transformers

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers training strategies for Transformers, focusing on applications in NLP and Vision. It discusses vanilla Transformer architectures, pre-training strategies, and recent advancements in the field. The instructor emphasizes the rapid evolution of Transformer research and the challenges in scaling up models. Various techniques like BERT, BEIT, and GPT are explained, along with their respective training methodologies. The lecture also touches on the limitations of large-scale models and the computational costs involved. Overall, it provides insights into the key aspects of training Transformers and the current trends in the field.

Official source

Related concepts (32)

Scale model

A scale model is a physical model which is geometrically similar to an object (known as the prototype). Scale models are generally smaller than large prototypes such as vehicles, buildings, or people; but may be larger than small prototypes such as anatomical structures or subatomic particles. Models built to the same scale as the prototype are called mockups. Scale models are used as tools in engineering design and testing, promotion and sales, filmmaking special effects, military strategy, and hobbies such as rail transport modeling, wargaming and racing; and as toys.

Rail transport modelling scales

Rail transport modelling uses a variety of scales (ratio between the real world and the model) to ensure scale models look correct when placed next to each other. Model railway scales are standardized worldwide by many organizations and hobbyist groups. Some of the scales are recognized globally, while others are less widespread and, in many cases, virtually unknown outside their circle of origin. Scales may be expressed as a numeric ratio (e.g. 1/87 or 1:87) or as letters defined in rail transport modelling standards (e.

Architectural model

An architectural model is a type of scale model made to study aspects of an architectural design or to communicate design intent. They are made using a variety of materials including paper, plaster, plastic, resin, wood, glass, and metal. Models are built either with traditional handcraft techniques or via 3D printing technologies such as stereolithography, fused filament fabrication, and selective laser sintering. The use of architectural models dates to pre-history. Some of the oldest standing models were found in Malta at Tarxien Temples.

Computer vision

Computer vision tasks include methods for , , and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context means the transformation of visual images (the input to the retina in the human analog) into descriptions of the world that make sense to thought processes and can elicit appropriate action.

Machine vision

Machine vision (MV) is the technology and methods used to provide -based automatic inspection and analysis for such applications as automatic inspection, process control, and robot guidance, usually in industry. Machine vision refers to many technologies, software and hardware products, integrated systems, actions, methods and expertise. Machine vision as a systems engineering discipline can be considered distinct from computer vision, a form of computer science.

Related lectures (25)

Deep Learning for Question Answering

Explores deep learning for question answering, analyzing neural networks and model robustness to noise.

Deep Learning: Convolutional Neural Networks

Covers Convolutional Neural Networks, standard architectures, training techniques, and adversarial examples in deep learning.

Deep Learning: Edge Detection and Neural Networks

Discusses edge detection techniques and the evolution of deep learning in neural networks.

Neural Networks: Training and Activation

Explores neural networks, activation functions, backpropagation, and PyTorch implementation.

Deep Learning for NLP Log in to Mediaspace to watch this video