Multilayer perceptron

A multilayer perceptron (MLP) is a misnomer for a modern feedforward artificial neural network, consisting of fully connected neurons with a nonlinear kind of activation function, organized in at least three layers, notable for being able to distinguish data that is not linearly separable. It is a misnomer because the original perceptron used a Heaviside step function, instead of a nonlinear kind of activation function (used by modern networks). Modern feedforward networks are trained using the backpropagation method and are colloquially referred to as the "vanilla" neural networks. In 1958, a layered network of perceptrons, consisting of an input layer, a hidden layer with randomized weights that did not learn, and an output layer with learning connections, was introduced already by Frank Rosenblatt in his book Perceptron. This extreme learning machine was not yet a deep learning network. In 1965, the first deep-learning feedforward network, not yet using stochastic gradient descent, was published by Alexey Grigorevich Ivakhnenko and Valentin Lapa, at the time called the Group Method of Data Handling. In 1967, a deep-learning network, which used stochastic gradient descent for the first time, able to classify non-linearily separable pattern classes, was published by Shun'ichi Amari. Amari's student Saito conducted the computer experiments, using a five-layered feedforward network with two learning layers. In 1970, modern backpropagation method, an efficient application of a chain-rule-based supervised learning, was for the first time published by the Finnish researcher Seppo Linnainmaa. The term (i.e. "back-propagating errors") itself has been used by Rosenblatt himself, but he did not know how to implement it, although a continuous precursor of backpropagation was already used in the context of control theory in 1960 by Henry J. Kelley. It is known also as a reverse mode of automatic differentiation. In 1982, backpropagation was applied in the way that has become standard, for the first time by Paul Werbos.