Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Crash course on ensemble methods: Bagging and Boosting

In course

EE-411: Fundamentals of inference and learning

This is an introductory course in the theory of statistics, inference, and machine learning, with an emphasis on theoretical understanding & practical exercises. The course will combine, and alternate, between mathematical theoretical foundations and prac ...

Description

This lecture introduces ensemble methods, focusing on Bagging and Boosting. It covers the concept of weak learnability leading to strong learnability, the AdaBoost algorithm for combining weak classifiers, and the process of setting weights at each time step. The instructor explains the iterative process of adding new classifiers, adapting weights, and re-weighting instances based on misclassifications.

Instructor

Florent Gérard Krzakala

Official source

Related concepts (28)

Boosting (machine learning)

In machine learning, boosting is an ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms that convert weak learners to strong ones. Boosting is based on the question posed by Kearns and Valiant (1988, 1989): "Can a set of weak learners create a single strong learner?" A weak learner is defined to be a classifier that is only slightly correlated with the true classification (it can label examples better than random guessing).

Ensemble learning

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

AdaBoost

AdaBoost, short for Adaptive Boosting, is a statistical classification meta-algorithm formulated by Yoav Freund and Robert Schapire in 1995, who won the 2003 Gödel Prize for their work. It can be used in conjunction with many other types of learning algorithms to improve performance. The output of the other learning algorithms ('weak learners') is combined into a weighted sum that represents the final output of the boosted classifier. Usually, AdaBoost is presented for binary classification, although it can be generalized to multiple classes or bounded intervals on the real line.

Naive Bayes classifier

In statistics, naive Bayes classifiers are a family of simple "probabilistic classifiers" based on applying Bayes' theorem with strong (naive) independence assumptions between the features (see Bayes classifier). They are among the simplest Bayesian network models, but coupled with kernel density estimation, they can achieve high accuracy levels. Naive Bayes classifiers are highly scalable, requiring a number of parameters linear in the number of variables (features/predictors) in a learning problem.

Gradient boosting

Gradient boosting is a machine learning technique used in regression and classification tasks, among others. It gives a prediction model in the form of an ensemble of weak prediction models, i.e., models that make very few assumptions about the data, which are typically simple decision trees. When a decision tree is the weak learner, the resulting algorithm is called gradient-boosted trees; it usually outperforms random forest.

Related lectures (33)

Boosting: Adaboost Algorithm

Covers boosting with a focus on the Adaboost algorithm, forward stagewise additive modeling, and gradient tree boosting.

Adaboost: Boosting Methods

Explains Adaboost algorithm for building strong classifiers from weak ones, with a focus on boosting methods and face detection.

Structured Classifications: Decision Trees and Boosting

Explores decision trees, overfitting elimination, boosting techniques, and their practical applications in predictive modeling.

Nearest Neighbor Rules: Part 2

Explores the Nearest Neighbor Rules, k-NN algorithm challenges, Bayes classifier, and k-means algorithm for clustering.

Advanced Machine Learning: Boosting