Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Boosting: Adaboost Algorithm

In course

MATH-412: Statistical machine learning

A course on statistical machine learning for supervised and unsupervised learning ...

Description

This lecture covers the concept of boosting, focusing on the Adaboost algorithm and forward stagewise additive modeling. It explains the derivation of Adaboost as FSAM on the exponential loss, boosted trees, and interpretations of Adaboost. The lecture also discusses the solutions for B and G, comparing with Adaboost, boosting of trees, and general gradient tree boosting for classification.

Instructor

Yoav Zemel

Official source

Ontological neighbourhood

Information engineering

Machine learning: Decision tree learning

Related concepts (25)

Boosting (machine learning)

In machine learning, boosting is an ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms that convert weak learners to strong ones. Boosting is based on the question posed by Kearns and Valiant (1988, 1989): "Can a set of weak learners create a single strong learner?" A weak learner is defined to be a classifier that is only slightly correlated with the true classification (it can label examples better than random guessing).

Gradient boosting

Gradient boosting is a machine learning technique used in regression and classification tasks, among others. It gives a prediction model in the form of an ensemble of weak prediction models, i.e., models that make very few assumptions about the data, which are typically simple decision trees. When a decision tree is the weak learner, the resulting algorithm is called gradient-boosted trees; it usually outperforms random forest.

AdaBoost

AdaBoost, short for Adaptive Boosting, is a statistical classification meta-algorithm formulated by Yoav Freund and Robert Schapire in 1995, who won the 2003 Gödel Prize for their work. It can be used in conjunction with many other types of learning algorithms to improve performance. The output of the other learning algorithms ('weak learners') is combined into a weighted sum that represents the final output of the boosted classifier. Usually, AdaBoost is presented for binary classification, although it can be generalized to multiple classes or bounded intervals on the real line.

Ensemble learning

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

Loss functions for classification

In machine learning and mathematical optimization, loss functions for classification are computationally feasible loss functions representing the price paid for inaccuracy of predictions in classification problems (problems of identifying which category a particular observation belongs to). Given as the space of all possible inputs (usually ), and as the set of labels (possible outputs), a typical goal of classification algorithms is to find a function which best predicts a label for a given input .

Related lectures (32)

Structured Classifications: Decision Trees and Boosting

Explores decision trees, overfitting elimination, boosting techniques, and their practical applications in predictive modeling.

Decision Trees: Induction & Attributes

Explores decision trees, attribute selection, bias-variance tradeoff, and ensemble methods in machine learning.

Nonlinear Supervised Learning

Explores the inductive bias of different nonlinear supervised learning methods and the challenges of hyper-parameter tuning.

Decision and regression trees

Explores decision and regression trees, impurity measures, learning algorithms, and implementations, including conditional inference trees and tree pruning.

Decision Forests: Structure and Training