Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Evaluation in NLP

Description

This lecture covers the NLP evaluation protocol, creation of gold standards, inter-annotator agreement, Cohen’s kappa, precision, recall, and statistical significance. It discusses the importance of separating data into training, validation, and test sets, and the use of evaluation metrics like confusion matrix, F-score, and more.

Official source

Related concepts (32)

Gold standard

A gold standard is a monetary system in which the standard economic unit of account is based on a fixed quantity of gold. The gold standard was the basis for the international monetary system from the 1870s to the early 1920s, and from the late 1920s to 1932 as well as from 1944 until 1971 when the United States unilaterally terminated convertibility of the US dollar to gold, effectively ending the Bretton Woods system. Many states nonetheless hold substantial gold reserves.

Precision and recall

In pattern recognition, information retrieval, object detection and classification (machine learning), precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space. Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances. Written as a formula:. Recall (also known as sensitivity) is the fraction of relevant instances that were retrieved. Written as a formula: . Both precision and recall are therefore based on relevance.

Training, validation, and test data sets

In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data. Such algorithms function by making data-driven predictions or decisions, through building a mathematical model from input data. These input data used to build the model are usually divided into multiple data sets. In particular, three data sets are commonly used in different stages of the creation of the model: training, validation, and test sets.

Cross-validation (statistics)

Cross-validation, sometimes called rotation estimation or out-of-sample testing, is any of various similar model validation techniques for assessing how the results of a statistical analysis will generalize to an independent data set. Cross-validation is a resampling method that uses different portions of the data to test and train a model on different iterations. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice.

Accuracy and precision

Accuracy and precision are two measures of observational error. Accuracy is how close a given set of measurements (observations or readings) are to their true value, while precision is how close the measurements are to each other. In other words, precision is a description of random errors, a measure of statistical variability. Accuracy has two definitions: More commonly, it is a description of only systematic errors, a measure of statistical bias of a given measure of central tendency; low accuracy causes a difference between a result and a true value; ISO calls this trueness.

Related lectures (42)

Heat Pump Tutorial

Explains the steps to define a heat pump and thermal streams.

Python Lists: Manipulation and Comprehension

Covers Python list manipulation and comprehension, emphasizing memory representation and mutability.

Model Evaluation: K-Nearest Neighbor

Explores model evaluation with K-Nearest Neighbor, covering optimal k selection, similarity metrics, and performance metrics for classification models.

Density Operator: Matrix to the System State

Explores density operator matrix transformation in quantum physics and the implications of measuring the system, leading to state collapse.

Working with Text Files: Basics