Lecture

Performance Evaluation: Bootstrap and Performance Metrics

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the evaluation of machine learning models through techniques like leave-one-out and bootstrap, discussing their advantages and disadvantages. It also explains performance metrics such as confusion matrices, recall, precision, and F-measure. The instructor provides examples to illustrate these concepts, including scenarios related to medical diagnostics. Terminologies like false positives and false negatives are defined, and the importance of recall in model evaluation is emphasized.

Official source

Ontological neighbourhood

Statistics

Data analysis: Statistical classification

Related concepts (20)

False positives and false negatives

A false positive is an error in binary classification in which a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error, where the test result incorrectly indicates the absence of a condition when it is actually present. These are the two kinds of errors in a binary test, in contrast to the two kinds of correct result (a and a ).

Precision and recall

In pattern recognition, information retrieval, object detection and classification (machine learning), precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space. Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances. Written as a formula:. Recall (also known as sensitivity) is the fraction of relevant instances that were retrieved. Written as a formula: . Both precision and recall are therefore based on relevance.

Positive and negative predictive values

The positive and negative predictive values (PPV and NPV respectively) are the proportions of positive and negative results in statistics and diagnostic tests that are true positive and true negative results, respectively. The PPV and NPV describe the performance of a diagnostic test or other statistical measure. A high result can be interpreted as indicating the accuracy of such a statistic. The PPV and NPV are not intrinsic to the test (as true positive rate and true negative rate are); they depend also on the prevalence.

False positive rate

In statistics, when performing multiple comparisons, a false positive ratio (also known as fall-out or false alarm ratio) is the probability of falsely rejecting the null hypothesis for a particular test. The false positive rate is calculated as the ratio between the number of negative events wrongly categorized as positive (false positives) and the total number of actual negative events (regardless of classification). The false positive rate (or "false alarm rate") usually refers to the expectancy of the false positive ratio.

Confusion matrix

In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one; in unsupervised learning it is usually called a matching matrix. Each row of the matrix represents the instances in an actual class while each column represents the instances in a predicted class, or vice versa – both variants are found in the literature.

Related lectures (44)

Diagnostic Tests: Sensitivity and Specificity

Explains Sensitivity, Specificity, ROC curve, and Precision in diagnostic tests.

Performance Criteria: Confusion Matrix, Recall, Precision, Accuracy

Explores performance criteria in supervised learning, emphasizing precision, recall, and specificity in model evaluation.

Machine Learning: Features and Model Selection

Delves into the significance of features, model evolution, labeling challenges, and model selection in machine learning.

Evaluation Protocols

Explores evaluation protocols in machine learning, including recall, precision, accuracy, and specificity, with real-world examples like COVID-19 testing.

Applied Machine Learning