Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Linear Binary Classification: Perceptron, SGD, Fisher's LDA

In course

MATH-412: Statistical machine learning

A course on statistical machine learning for supervised and unsupervised learning ...

Description

This lecture covers the Perceptron model, including the learning algorithm for separable and non-separable cases, Stochastic Gradient Descent, and Fisher's Linear Discriminant. It also discusses the convergence of algorithms, risk minimization, and generative classification.

Instructor

Yoav Zemel

Official source

Ontological neighbourhood

Information engineering

Machine learning: Topics in machine learning

Related concepts (32)

Linear classifier

In the field of machine learning, the goal of statistical classification is to use an object's characteristics to identify which class (or group) it belongs to. A linear classifier achieves this by making a classification decision based on the value of a linear combination of the characteristics. An object's characteristics are also known as feature values and are typically presented to the machine in a vector called a feature vector.

Perceptron

In machine learning, the perceptron (or McCulloch-Pitts neuron) is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to some specific class. It is a type of linear classifier, i.e. a classification algorithm that makes its predictions based on a linear predictor function combining a set of weights with the feature vector.

Statistical classification

In statistics, classification is the problem of identifying which of a set of categories (sub-populations) an observation (or observations) belongs to. Examples are assigning a given email to the "spam" or "non-spam" class, and assigning a diagnosis to a given patient based on observed characteristics of the patient (sex, blood pressure, presence or absence of certain symptoms, etc.). Often, the individual observations are analyzed into a set of quantifiable properties, known variously as explanatory variables or features.

Stochastic gradient descent

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data).

Separable space

In mathematics, a topological space is called separable if it contains a countable, dense subset; that is, there exists a sequence of elements of the space such that every nonempty open subset of the space contains at least one element of the sequence. Like the other axioms of countability, separability is a "limitation on size", not necessarily in terms of cardinality (though, in the presence of the Hausdorff axiom, this does turn out to be the case; see below) but in a more subtle topological sense.

Related lectures (36)

Perceptron: Part 2

Covers the Perceptron algorithm and its application to binary classification problems, including the Pocket Perceptron algorithm.

Linear Binary Classification

Covers the extension of the 0-1 loss to real-valued score functions and logistic regression.

Introduction to Learning by Stochastic Gradient Descent: Simple Perceptron

Covers the derivation of the stochastic gradient descent formula for a simple perceptron and explores the geometric interpretation of classification.

Generalized Linear Regression: Classification

Explores Generalized Linear Regression, Classification, confusion matrices, ROC curves, and noise in data.

Linear Classification: Signed Distance and Perceptron