Lecture

Evaluation Protocols

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the evaluation protocols in machine learning, focusing on recall, precision, accuracy, F-measure, and specificity. It discusses the trade-offs between recall and precision, the importance of specificity in testing, and real-world examples like COVID-19 testing methods and cancer screening. The instructor explains the concepts using examples such as perfect recall and precision scenarios, strategies to improve precision, and the F1-score as a harmonic mean of recall and precision. Additionally, it explores the Receiver-Operator Curve (ROC) and decision thresholds in binary classification.

Official source

Ontological neighbourhood

Statistics

Data analysis: Statistical classification

Related concepts (32)

Accuracy and precision

Accuracy and precision are two measures of observational error. Accuracy is how close a given set of measurements (observations or readings) are to their true value, while precision is how close the measurements are to each other. In other words, precision is a description of random errors, a measure of statistical variability. Accuracy has two definitions: More commonly, it is a description of only systematic errors, a measure of statistical bias of a given measure of central tendency; low accuracy causes a difference between a result and a true value; ISO calls this trueness.

Precision and recall

In pattern recognition, information retrieval, object detection and classification (machine learning), precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space. Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances. Written as a formula:. Recall (also known as sensitivity) is the fraction of relevant instances that were retrieved. Written as a formula: . Both precision and recall are therefore based on relevance.

Cancer screening

Cancer screening aims to detect cancer before symptoms appear. This may involve blood tests, urine tests, DNA tests, other tests, or medical imaging. The benefits of screening in terms of cancer prevention, early detection and subsequent treatment must be weighed against any harms. Universal screening, also known as mass screening or population screening, involves screening everyone, usually within a specific age group. Selective screening identifies people who are known to be at higher risk of developing cancer, such as people with a family history of cancer.

Breast cancer screening

Breast cancer screening is the medical screening of asymptomatic, apparently healthy women for breast cancer in an attempt to achieve an earlier diagnosis. The assumption is that early detection will improve outcomes. A number of screening tests have been employed, including clinical and self breast exams, mammography, genetic screening, ultrasound, and magnetic resonance imaging. A clinical or self breast exam involves feeling the breast for lumps or other abnormalities.

Outer measure

In the mathematical field of measure theory, an outer measure or exterior measure is a function defined on all subsets of a given set with values in the extended real numbers satisfying some additional technical conditions. The theory of outer measures was first introduced by Constantin Carathéodory to provide an abstract basis for the theory of measurable sets and countably additive measures.

Related lectures (53)

Performance Criteria: Confusion Matrix, Recall, Precision, Accuracy

Explores performance criteria in supervised learning, emphasizing precision, recall, and specificity in model evaluation.

Quantifying Performance: Misclassification and F-Measure

Covers quantifying performance through true positives, false negatives, and false positives in machine learning.

Information Retrieval Basics: Document Frequency and Precision

Introduces information retrieval basics, emphasizing document frequency and precision in evaluating retrieval quality.

Information Retrieval Basics: Document Length and Normalization

Explores document length, normalization, bias compensation, and retrieval model evaluation in information retrieval.

Performance Evaluation: Bootstrap and Performance Metrics