Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Bagging and Random Forests

In course

MATH-412: Statistical machine learning

A course on statistical machine learning for supervised and unsupervised learning ...

Description

This lecture covers the concepts of ensembling, bagging, and random forests, focusing on variance reduction, OOB samples, variable importance, and the algorithm details. It explains how bagging works for the square loss and its limitations with different predictors. The instructor also discusses the key idea behind random forests, which aim to decrease correlations further by randomizing the tree construction. Additionally, the lecture explores the concept of out-of-bag cross-validation and various measures of variable importance in random forests, such as the Gini VI score and permutation VI score.

Instructor

Yoav Zemel

Official source

Related concepts (17)

Random forest

Random forests or random decision forests is an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the mean or average prediction of the individual trees is returned. Random decision forests correct for decision trees' habit of overfitting to their training set.

Bootstrap aggregating

Bootstrap aggregating, also called bagging (from bootstrap aggregating), is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression. It also reduces variance and helps to avoid overfitting. Although it is usually applied to decision tree methods, it can be used with any type of method. Bagging is a special case of the model averaging approach.

Ensemble learning

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

Supervised learning

Supervised learning (SL) is a paradigm in machine learning where input objects (for example, a vector of predictor variables) and a desired output value (also known as human-labeled supervisory signal) train a model. The training data is processed, building a function that maps new data on expected output values. An optimal scenario will allow for the algorithm to correctly determine output values for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way (see inductive bias).

Machine learning

Machine learning (ML) is an umbrella term for solving problems for which development of algorithms by human programmers would be cost-prohibitive, and instead the problems are solved by helping machines 'discover' their 'own' algorithms, without needing to be explicitly told what to do by any human-developed algorithms. Recently, generative artificial neural networks have been able to surpass results of many previous approaches.

Related lectures (30)

Supervised Learning: k-NN and Decision Trees

Introduces supervised learning with k-NN and decision trees, covering techniques, examples, and ensemble methods.

Decision Trees: Induction & Attributes

Explores decision trees, attribute selection, bias-variance tradeoff, and ensemble methods in machine learning.

Machine Learning Basics: Supervised and Unsupervised Learning

Covers the basics of machine learning, supervised and unsupervised learning, various techniques like k-nearest neighbors and decision trees, and the challenges of overfitting.

Decision Trees and CLT's: Inference and Machine Learning

Explores decision trees, ensembles, CLT, inference, machine learning, diagnostic methods, boosting, and variance estimation.

Nonlinear Supervised Learning