Lecture

Nonlinear Machine Learning: k-Nearest Neighbors and Feature Expansion

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture discusses the transition from linear to nonlinear machine learning models, focusing on the k-Nearest Neighbors (k-NN) algorithm and feature expansion techniques. The instructor begins with a recap of linear models, explaining their structure and the algorithms that utilize them, such as least squares, logistic regression, and support vector machines. The lecture then introduces the k-NN algorithm, emphasizing its intuitive approach of classifying data based on the proximity of training samples. The instructor illustrates how k-NN can handle nonlinear data and discusses its properties, including sensitivity to outliers and the importance of the distance function used. The concept of feature expansion is also explored, demonstrating how transforming input data into higher dimensions can make it linearly separable. The lecture concludes with a discussion on the challenges of choosing the right polynomial degree for feature expansion and introduces the idea of kernel methods as a solution to these challenges.

Official source

Related concepts (32)

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Logistic regression

In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear combination of one or more independent variables. In regression analysis, logistic regression (or logit regression) is estimating the parameters of a logistic model (the coefficients in the linear combination).

Least squares

The method of least squares is a standard approach in regression analysis to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns) by minimizing the sum of the squares of the residuals (a residual being the difference between an observed value and the fitted value provided by a model) made in the results of each individual equation. The most important application is in data fitting.

Nonlinear regression

In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or more independent variables. The data are fitted by a method of successive approximations. In nonlinear regression, a statistical model of the form, relates a vector of independent variables, , and its associated observed dependent variables, . The function is nonlinear in the components of the vector of parameters , but otherwise arbitrary.

Linear model

In statistics, the term linear model is used in different ways according to the context. The most common occurrence is in connection with regression models and the term is often taken as synonymous with linear regression model. However, the term is also used in time series analysis with a different meaning. In each case, the designation "linear" is used to identify a subclass of models for which substantial reduction in the complexity of the related statistical theory is possible.

Related lectures (45)

Introduction to Machine Learning: Linear Models

Introduces linear models for supervised learning, covering overfitting, regularization, and kernels, with applications in machine learning tasks.

Landscape and Generalisation in Deep Learning

Explores the challenges and insights of deep learning, focusing on loss landscape, generalization, and feature learning.

Linear Models for Classification: Multi-Class Extensions

Covers linear models for multi-class classification, focusing on logistic regression and evaluation metrics.

Statistics for Civil Engineering

Covers the fundamentals of statistics, exploring probabilistic models, variability, uncertainty, and data collection.

Introduction to Machine Learning: Supervised Learning