Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Efficient Storage for Analytics: Columnar Storage & Compression Techniques

In course

CS-422: Database systems

This course is intended for students who want to understand modern large-scale data analysis systems and database systems. It covers a wide range of topics and technologies, and will prepare students to be able to build such systems as well as read and und ...

Description

This lecture covers storage technologies for analytics, focusing on columnar storage and compression techniques. It explains the benefits of Decomposition Storage Model (DSM), including DSM properties, page format, and write operations. The lecture also discusses compression methods such as lossless compression, run-length encoding, bit-vector encoding, and dictionary encoding. It explores how operators work over compressed data and introduces the concept of Write-Optimized Storage (WOS). Additionally, it delves into Partition Attributes Across (PAX) and its advantages, as well as the transition from DSM to PAX. The lecture concludes by emphasizing the importance of choosing the right storage layout based on the workload requirements.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Instructor

Anastasia Ailamaki

Official source

Related concepts (18)

Data compression

In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression. Lossy compression reduces bits by removing unnecessary or less important information.

Lossless compression

Lossless compression is a class of data compression that allows the original data to be perfectly reconstructed from the compressed data with no loss of information. Lossless compression is possible because most real-world data exhibits statistical redundancy. By contrast, lossy compression permits reconstruction only of an approximation of the original data, though usually with greatly improved compression rates (and therefore reduced media sizes).

Lossy compression

In information technology, lossy compression or irreversible compression is the class of data compression methods that uses inexact approximations and partial data discarding to represent the content. These techniques are used to reduce data size for storing, handling, and transmitting content. The different versions of the photo of the cat on this page show how higher degrees of approximation create coarser images as more details are removed. This is opposed to lossless data compression (reversible data compression) which does not degrade the data.

Compression artifact

A compression artifact (or artefact) is a noticeable distortion of media (including , audio, and video) caused by the application of lossy compression. Lossy data compression involves discarding some of the media's data so that it becomes small enough to be stored within the desired or transmitted (streamed) within the available bandwidth (known as the data rate or bit rate). If the compressor cannot store enough data in the compressed version, the result is a loss of quality, or introduction of artifacts.

Image compression

Image compression is a type of data compression applied to s, to reduce their cost for storage or transmission. Algorithms may take advantage of visual perception and the statistical properties of image data to provide superior results compared with generic data compression methods which are used for other digital data. Image compression may be lossy or lossless. Lossless compression is preferred for archival purposes and often for medical imaging, technical drawings, clip art, or comics.

Related lectures (48)

Data Compression: Shannon-Fano Algorithm

Explores the Shannon-Fano algorithm for efficient data compression and its applications in lossless and lossy compression techniques.

JPEG 2000: Image Compression

Explores image compression principles, focusing on JPEG 2000, covering transform-based coding, quantization, entropy coding, region of interest, error resilience, and software implementations.

JPEG XS & JPEG XL: Next-Gen Image Compression

Explores the cutting-edge JPEG XS and JPEG XL image compression standards, emphasizing their efficiency and versatility in various applications.

Data Compression: Entropy Definition

Explores data compression through entropy definition, types, and practical examples, illustrating its role in efficient information storage and transmission.

Principal Component Analysis: Dimensionality Reduction