Lecture

Lossless Compression: Shannon-Fano and Huffman

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-119(a): Information, Computation, Communication

D'une part, le cours aborde: (1) la notion d'algorithme et de représentation de l'information, (2) l'échantillonnage d'un signal et la compression de données et (3) des aspects liés aux systèmes: ordinateur, mémoire, etc. D'autre part, le cours donne une i ...

Description

This lecture covers the principles of lossless compression, focusing on exploiting data redundancy to shorten sequences efficiently. It introduces the Shannon-Fano algorithm, which divides letters based on their frequency, and the Huffman algorithm, which assigns variable-length codes to optimize compression. By comparing the two methods, the instructor demonstrates how Huffman coding outperforms Shannon-Fano in terms of efficiency and speed. The lecture also delves into the concept of entropy, calculating the entropy of a given sequence to highlight the effectiveness of Huffman coding in minimizing the average number of bits per letter.

In MOOC

Information, Computation, Communication: Introduction to computational thinking

In the first part of the course, we'll look at how algorithms can be used to solve a problem in a very concrete way, leading us on to one of the great questions of modern theoretical computer science. The second part of the course will be devoted to information and communication: we'll be looking at a number of topics, including information representation, signal processing, data compression, communication and cryptography.

Instructors (3)

Olivier Lévêque

Olivier Lévêque was born in Switzerland in 1971. He received the physics diploma from EPFL in 1995 and completed his PhD in mathematics at EPFL in 2001. Since then, he has been with the Laboratory of Information Theory at EPFL. He spent the academical year 2005-2006 at the Electrical Engineering Department of Stanford University, where he was appointed as lecturer. His research interests include stochastic analysis, random matrices, wireless communications and information theory.

Dan-Cristian Tomozei

Mirjana Stojilovic

Mirjana Stojilović received the Dipl. Ing. and Ph.D. degrees from the School of Electrical Engineering, University of Belgrade, in 2006 and 2013, respectively. From 2010 to 2013, she was collaborating with the Processor Architecture Laboratory at EPFL, visiting periodically as a Guest Researcher. From 2013 to 2016, she was working at the University of Applied Sciences Western Switzerland as a senior researcher, and at EPFL as a lecturer. She joined Parallel Systems Architecture Lab at EPFL in October 2016. Mirjana's main research interests include electronic design automation, reconfigurable computing, electromagnetic-compatibility and signal-integrity issues, and hardware security. Mirjana Stojilović serves on the program committee of the FPGA, FPL, and FCCM conferences and as a reviewer for IEEE TCAD, TVLSI, TC, TEMC, IEEE Access and ACM TRETS. She received the Best Paper Award at 2016 International Symposium on Electromagnetic Compatibility (EMC Europe 2016), Young Scientist Award at 33rd International Conference on Lightning Protection (ICLP2016), and the Young Author Best Paper Award at the 20th Telecommunication Forum in Belgrade (TELFOR 2012). In 2015, the EPFL School of Computer and Communication Sciences (IC) presented her with the Teaching Award.

Official source

Quiz

Related concepts (32)

Data compression

In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression. Lossy compression reduces bits by removing unnecessary or less important information.

Huffman coding

In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression. The process of finding or using such a code is Huffman coding, an algorithm developed by David A. Huffman while he was a Sc.D. student at MIT, and published in the 1952 paper "A Method for the Construction of Minimum-Redundancy Codes". The output from Huffman's algorithm can be viewed as a variable-length code table for encoding a source symbol (such as a character in a file).

Lossless compression

Lossless compression is a class of data compression that allows the original data to be perfectly reconstructed from the compressed data with no loss of information. Lossless compression is possible because most real-world data exhibits statistical redundancy. By contrast, lossy compression permits reconstruction only of an approximation of the original data, though usually with greatly improved compression rates (and therefore reduced media sizes).

Image compression

Image compression is a type of data compression applied to s, to reduce their cost for storage or transmission. Algorithms may take advantage of visual perception and the statistical properties of image data to provide superior results compared with generic data compression methods which are used for other digital data. Image compression may be lossy or lossless. Lossless compression is preferred for archival purposes and often for medical imaging, technical drawings, clip art, or comics.

Entropy (information theory)

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable , which takes values in the alphabet and is distributed according to : where denotes the sum over the variable's possible values. The choice of base for , the logarithm, varies for different applications. Base 2 gives the unit of bits (or "shannons"), while base e gives "natural units" nat, and base 10 gives units of "dits", "bans", or "hartleys".

Related lectures (64)

Compression: Prefix-Free Codes

Explains prefix-free codes for efficient data compression and the significance of uniquely decodable codes.

Data Compression and Shannon's Theorem Summary

Summarizes Shannon's theorem, emphasizing the importance of entropy in data compression.

Data Compression and Shannon-Fano Algorithm

Explores the Shannon-Fano algorithm for data compression and its efficiency in creating unique binary codes for letters.

Data Compression and Shannon's Theorem: Lossy Compression

Explores data compression, including lossless methods and the necessity of lossy compression for real numbers and signals.

Entropy and Data Compression: Huffman Coding Techniques