Lecture

Regression Trees and Ensemble Methods in Machine Learning

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers regression trees, focusing on the CART method and its application in predicting used car prices. The instructor explains the process of selecting optimal cutoffs for predictors, emphasizing the importance of minimizing mean square error (MSE) at each split. The lecture illustrates how to build a regression tree using a single predictor and extends the discussion to multiple predictors. The instructor highlights the risks of overfitting and the necessity of using cross-validation to determine when to stop splitting the tree. Additionally, the lecture introduces ensemble methods, including random forests and boosting, which enhance prediction accuracy by combining multiple trees. The instructor discusses the advantages and disadvantages of these methods, particularly their interpretability and robustness against overfitting. The session concludes with a case study on stock return prediction, demonstrating the practical application of these concepts in finance, and emphasizes the challenges of low signal-to-noise ratios in financial data.

Official source

Ontological neighbourhood

Statistics

Data analysis: Regression analysis, Cross-validation

Information engineering

Machine learning: Decision tree learning

Related concepts (32)

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

Machine learning

Machine learning (ML) is an umbrella term for solving problems for which development of algorithms by human programmers would be cost-prohibitive, and instead the problems are solved by helping machines 'discover' their 'own' algorithms, without needing to be explicitly told what to do by any human-developed algorithms. Recently, generative artificial neural networks have been able to surpass results of many previous approaches.

Mean squared error

In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of an estimator (of a procedure for estimating an unobserved quantity) measures the average of the squares of the errors—that is, the average squared difference between the estimated values and the actual value. MSE is a risk function, corresponding to the expected value of the squared error loss. The fact that MSE is almost always strictly positive (and not zero) is because of randomness or because the estimator does not account for information that could produce a more accurate estimate.

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Predictive modelling

Predictive modelling uses statistics to predict outcomes. Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place. In many cases, the model is chosen on the basis of detection theory to try to guess the probability of an outcome given a set amount of input data, for example given an email determining how likely that it is spam.

Related lectures (64)

Overfitting in Supervised Learning: Case Studies and Techniques

Addresses overfitting in supervised learning through polynomial regression case studies and model selection techniques.

Machine Learning Basics: Supervised and Unsupervised Learning

Covers the basics of machine learning, supervised and unsupervised learning, various techniques like k-nearest neighbors and decision trees, and the challenges of overfitting.

Machine Learning: Supervised and Unsupervised Learning Techniques

Covers supervised and unsupervised learning techniques in machine learning, highlighting their applications in finance and environmental analysis.

Decision Trees and Random Forests: Concepts and Applications

Discusses decision trees and random forests, focusing on their structure, optimization, and application in regression and classification tasks.

Linear Regression: Statistical Inference and Regularization