Attention (machine learning)Machine learning-based attention is a mechanism mimicking cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. It can do it either in parallel (such as in transformers) or sequentially (such as recursive neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards. Multiple attention heads are used in transformer-based large language models.
AttentionAttention is the concentration of awareness on some phenomenon to the exclusion of other stimuli. It is a process of selectively concentrating on a discrete aspect of information, whether considered subjective or objective. William James (1890) wrote that "Attention is the taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. Focalization, concentration, of consciousness are of its essence.
Transformer (machine learning model)A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.
Natural language processingNatural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.
Self-harmSelf-harm is intentional behavior that is considered harmful to oneself. This is most commonly regarded as direct injury of one's own skin tissues usually without a suicidal intention. Other terms such as cutting, self-injury, and self-mutilation have been used for any self-harming behavior regardless of suicidal intent. The most common form of self-harm is using a sharp object to cut the skin. Other forms include scratching, hitting, or burning body parts.
Defence mechanismIn psychoanalytic theory, a defence mechanism (American English: defense mechanism) is an unconscious psychological operation that functions to protect a person from anxiety-producing thoughts and feelings related to internal conflicts and outer stressors. Defence mechanisms (Abwehrmechanismen) are unconscious psychological processes employed to defend against feelings of anxiety and unacceptable impulses at the level of consciousness.
Attention deficit hyperactivity disorderAttention deficit hyperactivity disorder (ADHD) is a neurodevelopmental disorder characterised by excessive amounts of inattention, hyperactivity, and impulsivity that are pervasive, impairing in multiple contexts, and otherwise age-inappropriate. ADHD symptoms arise from executive dysfunction, and emotional dysregulation is often considered a core symptom. In children, problems paying attention may result in poor school performance.
History of opticsOptics began with the development of lenses by the ancient Egyptians and Mesopotamians, followed by theories on light and vision developed by ancient Greek philosophers, and the development of geometrical optics in the Greco-Roman world. The word optics is derived from the Greek term τα ὀπτικά meaning 'appearance, look'. Optics was significantly reformed by the developments in the medieval Islamic world, such as the beginnings of physical and physiological optics, and then significantly advanced in early modern Europe, where diffractive optics began.
Nobel PrizeThe Nobel Prizes (noʊˈbɛl ; Nobelpriset nʊˈbɛ̂lːˌpriːsɛt; Nobelprisen nʊˈbɛ̀lːˌpriːsn̩) are five separate prizes that, according to Alfred Nobel's will of 1895, are awarded to "those who, during the preceding year, have conferred the greatest benefit to humankind." Alfred Nobel was a Swedish chemist, engineer, and industrialist most famously known for the invention of dynamite. He died in 1896. In his will, he bequeathed all of his "remaining realisable assets" to be used to establish five prizes which became known as "Nobel Prizes.
Neural machine translationNeural machine translation (NMT) is an approach to machine translation that uses an artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model. They require only a fraction of the memory needed by traditional statistical machine translation (SMT) models. Furthermore, unlike conventional translation systems, all parts of the neural translation model are trained jointly (end-to-end) to maximize the translation performance.
Key (cryptography)A key in cryptography is a piece of information, usually a string of numbers or letters that are stored in a file, which, when processed through a cryptographic algorithm, can encode or decode cryptographic data. Based on the used method, the key can be different sizes and varieties, but in all cases, the strength of the encryption relies on the security of the key being maintained. A key's security strength is dependent on its algorithm, the size of the key, the generation of the key, and the process of key exchange.
Alfred NobelAlfred Bernhard Nobel (noʊˈbɛl , ˈǎlfrɛd nʊˈbɛlː; 21 October 1833 – 10 December 1896) was a Swedish chemist, engineer, inventor, businessman, and philanthropist. He is known for creating dynamite as well as having bequeathed his fortune to establish the Nobel Prize, though he also made several important contributions to science, holding 355 patents in his lifetime. Nobel's most famous invention was dynamite, an explosive using nitroglycerin; it was patented in 1867.
Recurrent neural networkA recurrent neural network (RNN) is one of the two broad types of artificial neural network, characterized by direction of the flow of information between its layers. In contrast to uni-directional feedforward neural network, it is a bi-directional artificial neural network, meaning that it allows the output from some nodes to affect subsequent input to the same nodes. Their ability to use internal state (memory) to process arbitrary sequences of inputs makes them applicable to tasks such as unsegmented, connected handwriting recognition or speech recognition.
Public-key cryptographyPublic-key cryptography, or asymmetric cryptography, is the field of cryptographic systems that use pairs of related keys. Each key pair consists of a public key and a corresponding private key. Key pairs are generated with cryptographic algorithms based on mathematical problems termed one-way functions. Security of public-key cryptography depends on keeping the private key secret; the public key can be openly distributed without compromising security.
Nobel Prize in PhysicsThe Nobel Prize in Physics (Nobelpriset i fysik) is a yearly award given by the Royal Swedish Academy of Sciences for those who have made the most outstanding contributions for humankind in the field of physics. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895 and awarded since 1901, the others being the Nobel Prize in Chemistry, Nobel Prize in Literature, Nobel Peace Prize, and Nobel Prize in Physiology or Medicine. Physics is traditionally the first award presented in the Nobel Prize ceremony.
Key sizeIn cryptography, key size, key length, or key space refer to the number of bits in a key used by a cryptographic algorithm (such as a cipher). Key length defines the upper-bound on an algorithm's security (i.e. a logarithmic measure of the fastest known attack against an algorithm), because the security of all algorithms can be violated by brute-force attacks. Ideally, the lower-bound on an algorithm's security is by design equal to the key length (that is, the algorithm's design does not detract from the degree of security inherent in the key length).
Key exchangeKey exchange (also key establishment) is a method in cryptography by which cryptographic keys are exchanged between two parties, allowing use of a cryptographic algorithm. If the sender and receiver wish to exchange encrypted messages, each must be equipped to encrypt messages to be sent and decrypt messages received. The nature of the equipping they require depends on the encryption technique they might use. If they use a code, both will require a copy of the same codebook. If they use a cipher, they will need appropriate keys.
OpticsOptics is the branch of physics that studies the behaviour and properties of light, including its interactions with matter and the construction of instruments that use or detect it. Optics usually describes the behaviour of visible, ultraviolet, and infrared light. Because light is an electromagnetic wave, other forms of electromagnetic radiation such as X-rays, microwaves, and radio waves exhibit similar properties.
Nobel Prize in ChemistryThe Nobel Prize in Chemistry (Nobelpriset i kemi) is awarded annually by the Royal Swedish Academy of Sciences to scientists in the various fields of chemistry. It is one of the five Nobel Prizes established by the will of Alfred Nobel in 1895, awarded for outstanding contributions in chemistry, physics, literature, peace, and physiology or medicine. This award is administered by the Nobel Foundation, and awarded by the Royal Swedish Academy of Sciences on proposal of the Nobel Committee for Chemistry which consists of five members elected by the Academy.
Nobel Prize in Physiology or MedicineThe Nobel Prize in Physiology or Medicine (Nobelpriset i fysiologi eller medicin) is awarded yearly by the Nobel Assembly at the Karolinska Institute for outstanding discoveries in physiology or medicine. The Nobel Prize is not a single prize, but five separate prizes that, according to Alfred Nobel's 1895 will, are awarded "to those who, during the preceding year, have conferred the greatest benefit to humankind". Nobel Prizes are awarded in the fields of Physics, Chemistry, Physiology or Medicine, Literature, and Peace.
