Lecture

Data Snooping

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture delves into the data snooping phenomenon, where learning from the test set, which should be kept untouched until the end, can lead to subtle yet significant issues in model training. The instructor emphasizes the importance of being cautious to avoid unintentional learning from the test set.

In MOOC

IoT Systems and Industrial Applications with Design Thinking

Internet of Things (IoT) and smart connected devices have radically changed the way our world works and how companies operate and create new businesses. Do you want to learn about IoT fundamentals? This is a course for you! In this course we provide an overview of the underlaying IoT technologies and its business applications. This course is an introduction to IoT and explains why it is a continuously evolving concept with many industrial applications. You will learn the fundamental building blocks of IoT

Official source

Related concepts (10)

Training, validation, and test data sets

In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data. Such algorithms function by making data-driven predictions or decisions, through building a mathematical model from input data. These input data used to build the model are usually divided into multiple data sets. In particular, three data sets are commonly used in different stages of the creation of the model: training, validation, and test sets.

Cross-validation (statistics)

Cross-validation, sometimes called rotation estimation or out-of-sample testing, is any of various similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. Cross-validation is a resampling method that uses different portions of the data to test and train a model on different iterations. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice.

Learning

Learning is the process of acquiring new understanding, knowledge, behaviors, skills, values, attitudes, and preferences. The ability to learn is possessed by humans, animals, and some machines; there is also evidence for some kind of learning in certain plants. Some learning is immediate, induced by a single event (e.g. being burned by a hot stove), but much skill and knowledge accumulate from repeated experiences. The changes induced by learning often last a lifetime, and it is hard to distinguish learned material that seems to be "lost" from that which cannot be retrieved.

Statistical model validation

In statistics, model validation is the task of evaluating whether a chosen statistical model is appropriate or not. Oftentimes in statistical inference, inferences from models that appear to fit their data may be flukes, resulting in a misunderstanding by researchers of the actual relevance of their model. To combat this, model validation is used to test whether a statistical model can hold up to permutations in the data.

Regression validation

In statistics, regression validation is the process of deciding whether the numerical results quantifying hypothesized relationships between variables, obtained from regression analysis, are acceptable as descriptions of the data. The validation process can involve analyzing the goodness of fit of the regression, analyzing whether the regression residuals are random, and checking whether the model's predictive performance deteriorates substantially when applied to data that were not used in model estimation.

Related lectures (64)

Model Assessment and Hyperparameter Tuning

Explores model assessment, hyperparameter tuning, and resampling strategies in machine learning.

Linear Regression and Gradient Descent

Covers linear regression, gradient descent, overfitting, and ridge regression among other concepts.

Lecture 05 (2024)

Model Complexity and Overfitting in Machine Learning

Covers model complexity, overfitting, and strategies to select appropriate machine learning models.

Model Evaluation