Concept

Generative pre-trained transformer

Generative pre-trained transformers (GPT) are a type of large language model (LLM) and a prominent framework for generative artificial intelligence. The first GPT was introduced in 2018 by OpenAI. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs. OpenAI has released very influential GPT foundation models that have been sequentially numbered, to comprise its "GPT-n" series. Each of these was significantly more capable than the previous, due to increased size (number of trainable parameters) and training. The most recent of these, GPT-4, was released in March 2023. Such models have been the basis for their more task-specific GPT systems, including models fine-tuned for instruction followingwhich in turn power the ChatGPT chatbot service. The term "GPT" is also used in the names and descriptions of such models developed by others. For example, other GPT foundation models include a series of models created by EleutherAI, and recently seven models created by Cerebras. Also, companies in different industries have developed task-specific GPTs in their respective fields, such as Salesforce's "EinsteinGPT" (for CRM) and Bloomberg's "BloombergGPT" (for finance). Generative pretraining (GP) was a long-established concept in machine learning applications, but the transformer architecture was not available until 2017 when it was invented by employees at Google. That development led to the emergence of large language models like BERT in 2018 and XLNet in 2019, which were pre-trained transformers (PT) but not designed to be generative (they were "encoder-only"). Also around that time, in 2018, OpenAI published its article entitled "Improving Language Understanding by Generative Pre-Training," in which it introduced the first generative pre-trained transformer (GPT) system ("GPT-1").

Official source

https://en.wikipedia.org/wiki/Generative_pre-trained_transformer

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related courses (18)

CS-461: Foundation models and generative AI

This course covers the principles, architectures, and applications of foundation models and generative AI, including generative methods, tokenization, multi-modal learning, adaptation, prompting, and their use in reasoning, decision-making, and scientific ...

CS-433: Machine learning

Machine learning methods are becoming increasingly central in many sciences and applications. In this course, fundamental principles and methods of machine learning will be introduced, analyzed and practically implemented. ...

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Official source

https://en.wikipedia.org/wiki/Generative_pre-trained_transformer

About this result

Related courses (18)

CS-461: Foundation models and generative AI

CS-433: Machine learning

EE-556: Mathematics of data: from theory to computation

Related lectures (32)

Language Models: From Theory to Computation

Explores the mathematics of language models, covering architecture design, pre-training, and fine-tuning, emphasizing the importance of pre-training and fine-tuning for various tasks.

Chemical Reaction Prediction: Molecular Transformer

Explores chemical reaction prediction using generative models and molecular transformers, emphasizing the importance of molecular language processing and stereochemistry.

Pretraining: Transformers & Models

Explores pretraining models like BERT, T5, and GPT, discussing their training objectives and applications in natural language processing.

Related publications (32)

OH-Grasp: A one-shot human-centric robotic grasping framework for human–robot collaboration in Industry 5.0

The advancement of Industry 5.0 has positioned human–robot collaboration (HRC) as a critical component of smart manufacturing, requiring robots to work seamlessly alongside humans in dynamic, unstructured environments. However, several challenges remain, including the heavy reliance on object-specific data for training dynamic perception models, the reliance on manually prepared CAD models for 6D pose estimation, and the lack of interaction semantics in grasp planning. To this end, a one-shot human-centric

Elsevier BV2027

Generating antimicrobial peptides via genomic transfer learning

We present a generative machine learning pipeline for the design of linear antimicrobial peptides (AMPs). To extend diversity beyond synthetically validated peptide datasets (

\sim

7,000 entries), we apply transfer learning by training a Generative Pre-trained Transformer (GPT) on the genomically derived AMPSphere dataset (

\sim

863,000 entries), before fine-tuning on the Database of Antimicrobial Activity and Structure of Peptides (DBAASP). We assess the filtered sequences with a committee of Minimum Inhib

bioRxiv2026

Modular Framework for 3D Molecular Generation in Computational Chemistry Applications

Three-dimensional molecular generative models construct molecules by assigning atoms explicit Cartesian coordinates. This formulation allows the molecular generation process to be guided by geometric (e.g., steric effects, molecular shape) and physicochemical constraints toward desired objectives in a given chemical problem. The practical adoption of these models, however, remains limited by two key challenges: the high computational cost associated with model training as well as the lack of standardized pl

American Chemical Society (ACS)2026

Related people (32)

Charlotte Cäcilie Gertrud Bunne

Charlotte Bunne is an assistant professor at EPFL in the School of Computer and Communication Sciences (IC) and School of Life Sciences (SV). She is part of the Swiss Institute for Experimental Cancer Research (ISREC) and the EPFL AI Center. Before, she was a PostDoc at Genentech and Stanford working with Aviv Regev and Jure Leskovec and completed a PhD in Computer Science at ETH Zurich working with Andreas Krause and Marco Cuturi. During her graduate studies, she was a visiting researcher at the Broad Inst

David Atienza Alonso

David Atienza Alonso is an associate professor of EE and director of the Embedded Systems Laboratory (ESL) at EPFL, Switzerland. He received his MSc and PhD degrees in computer science and engineering from UCM, Spain, and IMEC, Belgium, in 2001 and 2005, respectively. His research interests include system-level design methodologies for multi-processor system-on-chip (MPSoC) servers and edge AI architectures. Dr. Atienza has co-authored more than 350 papers, one book, and 12 patents in these previous areas. He has also received several recognitions and award, among them, the ICCAD 10-Year Retrospective Most Influential Paper Award in 2020, Design Automation Conference (DAC) Under-40 Innovators Award in 2018, the IEEE TCCPS Mid-Career Award in 2018, an ERC Consolidator Grant in 2016, the IEEE CEDA Early Career Award in 2013, the ACM SIGDA Outstanding New Faculty Award in 2012, and a Faculty Award from Sun Labs at Oracle in 2011. He has also earned two best paper awards at the VLSI-SoC 2009 and CST-HPCS 2012 conference, and five best paper award nominations at the DAC 2013, DATE 2013, WEHA-HPCS 2010, ICCAD 2006, and DAC 2004 conferences. He serves or has served as associate editor of IEEE Trans. on Computers (TC), IEEE Design & Test of Computers (D&T), IEEE Trans. on CAD (T-CAD), IEEE Transactions on Sustainable Computing (T-SUSC), and Elsevier Integration. He was the Technical Program Chair of DATE 2015 and General Chair of DATE 2017. He served as President of IEEE CEDA in the period 2018-2019 and was GOLD member of the Board of Governors of IEEE CASS from 2010 to 2012. He is a Distinguished Member of ACM and an IEEE Fellow.

Sabine Süsstrunk

Prof. Dr. Sabine Süsstrunk leads the Image and Visual Representation Lab in the School of Computer and Communication Sciences (IC) at EPFL since 1999. From 2015-2020, she was also the first Director of the Digital Humanities Institute (DHI), College of Humanities (CdH). Her main research areas are in computational photography, computational imaging, color image processing and computer vision, machine learning, and computational image quality and aesthetics. Sabine has authored and co-authored over 200 publications, of which 7 have received best paper/demo awards, and holds over 10 patents. Sabine served as chair and/or committee member in many international conferences on image processing, computer vision, and image systems engineering. She is President of the Swiss Science Council SSC, Founding Member and Member of the Board (President 2014-2018) of the EPFL-WISH (Women in Science and Humanities) Foundation, Member of the Board of the SRG SSR (Swiss Radio and Television Corporation), and Member of the Board of Largo Films. She received the IS&T/SPIE 2013 Electronic Imaging Scientist of the Year Award for her contributions to color imaging, computational photography, and image quality, and the 2018 IS&T Raymond C. Bowman and the 2020 EPFL AGEPoly IC Polysphere Awards for excellence in teaching. Sabine is a Fellow of IEEE and IS&T.

Pascal Fua

Pascal Fua received an engineering degree from Ecole Polytechnique, Paris, in 1984 and the Ph.D. degree in Computer Science from the University of Orsay in 1989. He then worked at SRI International and INRIA Sophia-Antipolis as a Computer Scientist. He joined EPFL in 1996 where he is now a Professor in the School of Computer and Communication Science and heads the Computer Vision Laboratory. His research interests include shape modeling and motion recovery from images, analysis of microscopy images, and Augmented Reality. His research interests include shape modeling and motion recovery from images, analysis of microscopy images, and machine learning. He has (co)authored over 300 publications in refereed journals and conferences. He is an IEEE Fellow and has been an Associate Editor of IEEE journal Transactions for Pattern Analysis and Machine Intelligence. He often serves as program committee member, area chair, and program chair of major vision conferences and has cofounded three spinoff companies (Pix4D, PlayfulVision, and NeuralConcept).

Amir Roshan Zamir

Amir Zamir is a Tenure-Track Assistant Professor of Computer Science at the Swiss Federal Institute of Technology in Lausanne (EPFL). Prior to EPFL, he spent time at Stanford, UC Berkeley, and UCF. His research interests are broadly in computer vision, machine learning, perception-for-robotics, and AI. He has been recognized with CVPR (2018) Best Paper Award, CVPR (2016) Best Student Paper Award, CVPR (2020) Best Paper Award Nomination, and NVIDIA Pioneering Research Award (2018), among others. His research has been covered by various press outlets, such as The New York Times or Forbes.

Jean-Marc Odobez

Expertise Computer Vision, machine Learning (Bayesian models, deep learning), multimodal processing.Activity analysis, human behavior understanding, human communication, interaction modeling Dr Jean-Marc Odobez (Msc 1990, PhD 1994) received his Ph.D degree from Rennes University in 1994 for his PhD dissertation done at INRIA. After 5 years as an assistant professor at the University du Maine, France, he decided to join Idiap, where he is now the Head of the Perception & Activity Understanding Group and adju

Francesco Craighero

Expertise machine learning, explainable AI, single-cell analysis, genomics

Simon Elias Bibri

Expertise • Sustainable cities• Smarter eco-cities• Sustainable smart cities• Artificial Intelligence of Things (AIoT) • Generative Artificial Intelligence of Things (GAIoT)• Foundation Models (FMs) • Quantum Generative Artificial Intelligence (QGAI)• Urban Digital Twins (UDTs)• Urban Brain (UB) • Smart Circular Metabolism• Urban planning and design• Environmental sustainability and climate resilience• Sustainable smart energy• Sustainable smart transportation• Sustainable smart waste and water management•

Michael Christoph Gastpar

Michael Gastpar is a (full) Professor at EPFL. From 2003 to 2011, he was a professor at the University of California at Berkeley, earning his tenure in 2008. He received his Dipl. El.-Ing. degree from ETH Zürich, Switzerland, in 1997 and his MS degree from the University of Illinois at Urbana-Champaign, IL, USA, in 1999. He defended his doctoral thesis at EPFL on Santa Claus day, 2002. He was also a (full) Professor at Delft University of Technology, The Netherlands. His research interests are in network information theory and related coding and signal processing techniques, with applications to sensor networks and neuroscience. He is a Fellow of the IEEE. He is the co-recipient of the 2013 Communications Society & Information Theory Society Joint Paper Award. He was an Information Theory Society Distinguished Lecturer (2009-2011). He won an ERC Starting Grant in 2010, an Okawa Foundation Research Grant in 2008, an NSF CAREER award in 2004, and the 2002 EPFL Best Thesis Award. He has served as an Associate Editor for Shannon Theory for the IEEE Transactions on Information Theory (2008-11), and as Technical Program Committee Co-Chair for the 2010 International Symposium on Information Theory, Austin, TX.

Florent Gérard Krzakala

Related units (32)

Computer Vision Laboratory

Geographic Information Systems Laboratory

Environmental Computational Science and Earth Observation Laboratory

Related concepts (9)

Foundation models

A foundation model (also called base model) is a large machine learning (ML) model trained on a vast quantity of data at scale (often by self-supervised learning or semi-supervised learning) such that it can be adapted to a wide range of downstream tasks. Foundation models have helped bring about a major transformation in how artificial intelligence (AI) systems are built, such as by powering prominent chatbots and other user-facing AI.

Text-to-image model

A text-to-image model is a machine learning model which takes an input natural language description and produces an image matching that description. Such models began to be developed in the mid-2010s, as a result of advances in deep neural networks. In 2022, the output of state of the art text-to-image models, such as OpenAI's DALL-E 2, Google Brain's , StabilityAI's Stable Diffusion, and Midjourney began to approach the quality of real photographs and human-drawn art.

Large language model

A large language model (LLM) is a language model characterized by its large size. Their size is enabled by AI accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.