Concept

Delta rule

Related publications (9)

On Balanced Representation Learning in Neural Networks

The efficient training of large-scale deep neural networks currently relies on an ecosystem of optimization heuristics, including weight decay, learning rate warmup, and specific parameterizations. Despite their ubiquity, the fundamental mechanisms driving their success remain poorly understood. Through applied analysis and large-scale experimentation, this thesis establishes that these key methods share a common underlying benefit: they balance the speed at which internal representations change across vary

EPFL2026

Differential modulation of positive and negative prediction errors by stimulus variability in the mouse posterior parietal cortex

To function in uncertain environments, we need to constantly update our internal model of the world. Sensory prediction errors are thought to drive these updates. EEG studies indicate that uncertainty affects neural correlates of prediction errors. It is unclear how uncertainty of the environment modulates prediction error responses in individual neurons. Here, we exposed awake head-restrained mice to sounds followed by tactile stimuli that were either fixed or variable in intensity, making them more or les

2025

Biologically plausible deep learning – but how far can we go with shallow networks?

Wulfram Gerstner, Johanni Michael Brea, Bernd Albert Illing

Training deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining s ...

2019

Generalized Bienenstock-Cooper-Munro rule for spiking neurons that maximizes information transmission

Wulfram Gerstner, Jean-Pascal Théodor Pfister

Maximization of information transmission by a spiking-neuron model predicts changes of synaptic connections that depend on timing of pre- and postsynaptic spikes and on the postsynaptic membrane potential. Under the assumption of Poisson firing statistics, the synaptic update rule exhibits all of the features of the Bienenstock-Cooper-Munro rule, in particular, regimes of synaptic potentiation and depression separated by a sliding threshold. Moreover, the learning rule is also applicable to the more realist

2005

Hebbian learning and spiking neurons

Wulfram Gerstner

A correlation-based (

Hebbian'') learning rule at the spike level is formulated, mathematically analyzed, and compared with learning in a firing-rate description. As for spike coding, we take advantage of a

learning window'' that describes the effect of timing of pre- and postsynaptic spikes on synaptic weights. A differential equation for the learning dynamics is derived under the assumption that the time scales of learning and spiking dynamics can be separated. Formation of structured synapses is analy

1999

Discrete All-Positive Multilayer Perceptrons for Optical Implementation

All-optical multilayer perceptrons differ in various ways from the ideal neural network model. Examples are the use of non-ideal activation functions which are truncated, asymmetric, and have a non-standard gain, restriction of the network parameters to non-negative values, and the limited accuracy of the weights. In this paper, a backpropagation-based learning rule is presented that compensates for these non-idealities and enables the implementation of all-optical multilayer perceptrons where learning occu

1998

Discrete All-Positive Multilayer Perceptrons for Optical Implementation

IDIAP1997

The Effects of Optical Thresholding in Backpropagation Neural Networks

Sigmoid-like activation functions implemented in analog hardware differ in various ways from the standard sigmoidal function as they are asymmetric, truncated, and have a non-standard gain. It is demonstrated how one can adapt the backpropagation learning rule to compensate for these non-standard sigmoids as available in hardware. This method is applied to multilayer neural networks with all-optical forward propagation and liquid crystal light valves (LCLV) as optical thresholding devices. In this paper the

EC2 & Cie1995

CMOS analogue VLSI implementation of a Kohonen map

The domain of artificial neural networks has evolved rapidly during the last decade, and many research groups are presently working on new neuronal algorithms and investigating their potential for technological applications. The idea to use biologically inspired models to implement intelligent systems is issue from the fact that animals, through their adaptation to the environment, have evolved towards robust and reliable structures, well adapted to the imperfections or even the destruction of some of their

EPFL1993

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.