Lecture

Information Theory Basics

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

COM-102: Advanced information, computation, communication II

Text, sound, and images are examples of information sources stored in our computers and/or communicated over the Internet. How do we measure, compress, and protect the informatin they contain? ...

Description

This lecture covers the basics of information theory, including concepts such as entropy, independence of random variables, and the binary entropy function. It explains how information is measured, the relationship between entropy and uncertainty, and the fundamental inequality in information theory. Examples are provided to illustrate the application of these concepts in various scenarios.

Instructors (2)

Bixio Rimoldi

Bixio Rimoldi received the Dipl. El.-Ing degree as well as the Dr. ès Sciences degree from the ETHZ, Switzerland. Since 1997, he holds a full professor position at the School of Computer and Communication Sciences at EPFL and he is the director of EPFL's Mobile Communications Laboratory (LCM). Prior to joining EPFL, he was in the faculty of the Electrical Engineering department of Washington University. In 1993, he received a US National Science Foundation Young Investigator Award. In 2000, he was elected to the grade of Fellow of the IEEE. During the period 2002-2009 he has been on the Board of Governors of the IEEE Information Theory Society, where he served in several offices including President. He was co-chairman with Bruce Hajek of the 1995 IEEE Information Theory Workshop on Information Theory, Multiple Access, and Queueing (St Louis, MO), and co-chairman with Jim Massey of the 2002 IEEE International Symposium in Information Theory (Lausanne, Switzerland). He has been a member of the editorial board of "Foundations and Trends on Communications and Information Theory," and was an editor of the European Transactions on Telecommunications. During 2005 and 2006 he was the Director of EPFL's Bachelor program in Communication Systems. His interests are in various aspects of digital communications, information theory, and software-defined radio.

Michael Christoph Gastpar

Michael Gastpar is a (full) Professor at EPFL. From 2003 to 2011, he was a professor at the University of California at Berkeley, earning his tenure in 2008. He received his Dipl. El.-Ing. degree from ETH Zürich, Switzerland, in 1997 and his MS degree from the University of Illinois at Urbana-Champaign, IL, USA, in 1999. He defended his doctoral thesis at EPFL on Santa Claus day, 2002. He was also a (full) Professor at Delft University of Technology, The Netherlands. His research interests are in network information theory and related coding and signal processing techniques, with applications to sensor networks and neuroscience. He is a Fellow of the IEEE. He is the co-recipient of the 2013 Communications Society & Information Theory Society Joint Paper Award. He was an Information Theory Society Distinguished Lecturer (2009-2011). He won an ERC Starting Grant in 2010, an Okawa Foundation Research Grant in 2008, an NSF CAREER award in 2004, and the 2002 EPFL Best Thesis Award. He has served as an Associate Editor for Shannon Theory for the IEEE Transactions on Information Theory (2008-11), and as Technical Program Committee Co-Chair for the 2010 International Symposium on Information Theory, Austin, TX.

Official source

Ontological neighbourhood

Statistics

Statistical inference: Mathematical statistics

Related concepts (32)

Entropy (information theory)

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable , which takes values in the alphabet and is distributed according to : where denotes the sum over the variable's possible values. The choice of base for , the logarithm, varies for different applications. Base 2 gives the unit of bits (or "shannons"), while base e gives "natural units" nat, and base 10 gives units of "dits", "bans", or "hartleys".

Cross-entropy

In information theory, the cross-entropy between two probability distributions and over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is optimized for an estimated probability distribution , rather than the true distribution . The cross-entropy of the distribution relative to a distribution over a given set is defined as follows: where is the expected value operator with respect to the distribution .

Entropy

Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodynamics, where it was first recognized, to the microscopic description of nature in statistical physics, and to the principles of information theory.

Random variable

A random variable (also called random quantity, aleatory variable, or stochastic variable) is a mathematical formalization of a quantity or object which depends on random events. The term 'random variable' can be misleading as it is not actually random nor a variable, but rather it is a function from possible outcomes (e.g., the possible upper sides of a flipped coin such as heads and tails ) in a sample space (e.g., the set ) to a measurable space (e.g., in which 1 corresponding to and −1 corresponding to ), often to the real numbers.

Noisy-channel coding theorem

In information theory, the noisy-channel coding theorem (sometimes Shannon's theorem or Shannon's limit), establishes that for any given degree of noise contamination of a communication channel, it is possible to communicate discrete data (digital information) nearly error-free up to a computable maximum rate through the channel. This result was presented by Claude Shannon in 1948 and was based in part on earlier work and ideas of Harry Nyquist and Ralph Hartley.

Related lectures (64)

Entropy and Data Compression: Huffman Coding Techniques

Discusses entropy, data compression, and Huffman coding techniques, emphasizing their applications in optimizing codeword lengths and understanding conditional entropy.

Conditional Entropy and Information Theory Concepts

Discusses conditional entropy and its role in information theory and data compression.

Information Measures: Entropy and Information Theory

Explains how entropy measures uncertainty in a system based on possible outcomes.

Information Theory: Source Coding, Cryptography, Channel Coding

Covers source coding, cryptography, and channel coding in communication systems, exploring entropy, codes, error channels, and future related courses.

Lecture: Shannon