Lecture

Evaluation in NLP

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the NLP evaluation protocol, creation of gold standards, inter-annotator agreement, Cohen’s kappa, precision, recall, and statistical significance. It discusses the importance of separating data into training, validation, and test sets, and the use of evaluation metrics like confusion matrix, F-score, and more.

Official source

Related concepts (32)

Quality (business)

In business, engineering, and manufacturing, quality – or high quality – has a pragmatic interpretation as the non-inferiority or superiority of something (goods or services); it is also defined as being suitable for the intended purpose (fitness for purpose) while satisfying customer expectations. Quality is a perceptual, conditional, and somewhat subjective attribute and may be understood differently by different people. Consumers may focus on the specification quality of a product/service, or how it compares to competitors in the marketplace.

Computer file

In computing, a computer file is a resource for recording data on a computer storage device, primarily identified by its . Just as words can be written on paper, so can data be written to a computer file. Files can be shared with and transferred between computers and mobile devices via removable media, networks, or the Internet. Different are designed for different purposes. A file may be designed to store an , a written message, a video, a program, or any wide variety of other kinds of data.

Statistical significance

In statistical hypothesis testing, a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true. More precisely, a study's defined significance level, denoted by , is the probability of the study rejecting the null hypothesis, given that the null hypothesis is true; and the p-value of a result, , is the probability of obtaining a result at least as extreme, given that the null hypothesis is true. The result is statistically significant, by the standards of the study, when .

Precision and recall

In pattern recognition, information retrieval, object detection and classification (machine learning), precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space. Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances. Written as a formula:. Recall (also known as sensitivity) is the fraction of relevant instances that were retrieved. Written as a formula: . Both precision and recall are therefore based on relevance.

Protocol stack

The protocol stack or network stack is an implementation of a computer networking protocol suite or protocol family. Some of these terms are used interchangeably but strictly speaking, the suite is the definition of the communication protocols, and the stack is the software implementation of them. Individual protocols within a suite are often designed with a single purpose in mind. This modularization simplifies design and evaluation. Because each protocol module usually communicates with two others, they are commonly imagined as layers in a stack of protocols.

Related lectures (64)

Software Quality: Managing Defects and Testing Strategies

Explores managing defects, testing strategies, and challenges in software quality, with real-world examples and case studies.

Turbulence: Numerical Flow Simulation

Explores turbulence characteristics, simulation methods, and modeling challenges, providing guidelines for choosing and validating turbulence models.

Heat Pump Tutorial

Explains the steps to define a heat pump and thermal streams.

Python Lists: Manipulation and Comprehension

Covers Python list manipulation and comprehension, emphasizing memory representation and mutability.

Matlab Programming: Script and Function