Lecture

Decision Trees and Boosting

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers decision trees as a flexible method for machine learning, illustrating them as a formalization of strategies for decision-making processes. It explains the concept of decision trees, their construction, and their application in classification and regression tasks. The lecture also introduces boosting as a technique to combine multiple predictors sequentially, focusing on AdaBoost and Gradient Boosting algorithms. It discusses the advantages and limitations of decision trees, including their ability to handle mixed data types and the issue of overfitting. The presentation includes examples, algorithms, and practical implementations using Python libraries like scikit-learn.

Official source

Ontological neighbourhood

Information engineering

Machine learning: Decision tree learning

Related concepts (29)

Gradient boosting

Gradient boosting is a machine learning technique used in regression and classification tasks, among others. It gives a prediction model in the form of an ensemble of weak prediction models, i.e., models that make very few assumptions about the data, which are typically simple decision trees. When a decision tree is the weak learner, the resulting algorithm is called gradient-boosted trees; it usually outperforms random forest.

Ensemble learning

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

Boosting (machine learning)

In machine learning, boosting is an ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms that convert weak learners to strong ones. Boosting is based on the question posed by Kearns and Valiant (1988, 1989): "Can a set of weak learners create a single strong learner?" A weak learner is defined to be a classifier that is only slightly correlated with the true classification (it can label examples better than random guessing).

AdaBoost

AdaBoost, short for Adaptive Boosting, is a statistical classification meta-algorithm formulated by Yoav Freund and Robert Schapire in 1995, who won the 2003 Gödel Prize for their work. It can be used in conjunction with many other types of learning algorithms to improve performance. The output of the other learning algorithms ('weak learners') is combined into a weighted sum that represents the final output of the boosted classifier. Usually, AdaBoost is presented for binary classification, although it can be generalized to multiple classes or bounded intervals on the real line.

Early stopping

In machine learning, early stopping is a form of regularization used to avoid overfitting when training a learner with an iterative method, such as gradient descent. Such methods update the learner so as to make it better fit the training data with each iteration. Up to a point, this improves the learner's performance on data outside of the training set. Past that point, however, improving the learner's fit to the training data comes at the expense of increased generalization error.

Related lectures (64)

Regression Trees and Ensemble Methods in Machine Learning

Discusses regression trees, ensemble methods, and their applications in predicting used car prices and stock returns.

Structured Classifications: Decision Trees and Boosting

Explores decision trees, overfitting elimination, boosting techniques, and their practical applications in predictive modeling.

Decision Trees and Random Forests: Concepts and Applications

Discusses decision trees and random forests, focusing on their structure, optimization, and application in regression and classification tasks.

Classification Algorithms: Generative and Discriminative Approaches

Explores generative and discriminative classification algorithms, emphasizing their applications and differences in machine learning tasks.

Machine Learning Basics: Supervised Learning