Lecture

Table Storage in Data Lakes

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the concept of table storage in data lakes, focusing on technologies like Delta Lake, Iceberg, and Hudi. It explains how these systems support record updates and the history behind their development. The instructor discusses the design of table storage for mini-batch updates, the separation of incremental data and logs, and the importance of snapshot isolation. Different approaches to record updates in Delta Lake, Iceberg, and Hudi are compared, highlighting their efficiency for different use cases.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Official source

Related concepts (32)

Metadata

Metadata (or metainformation) is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including: Descriptive metadata – the descriptive information about a resource. It is used for discovery and identification. It includes elements such as title, abstract, author, and keywords. Structural metadata – metadata about containers of data and indicates how compound objects are put together, for example, how pages are ordered to form chapters.

Metadata standard

A metadata standard is a requirement which is intended to establish a common understanding of the meaning or semantics of the data, to ensure correct and proper use and interpretation of the data by its owners and users. To achieve this common understanding, a number of characteristics, or attributes of the data have to be defined, also known as metadata. Metadata is often defined as data about data.

File system

In computing, a file system or filesystem (often abbreviated to fs) is a method and data structure that the operating system uses to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one large body of data with no way to tell where one piece of data stopped and the next began, or where any piece of data was located when it was time to retrieve it. By separating the data into pieces and giving each piece a name, the data are easily isolated and identified.

Object storage

Object storage (also known as object-based storage) is a computer data storage that manages data as objects, as opposed to other storage architectures like which manages data as a file hierarchy, and block storage which manages data as blocks within sectors and tracks. Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier. Object storage can be implemented at multiple levels, including the device level (object-storage device), the system level, and the interface level.

Metadata registry

A metadata registry is a central location in an organization where metadata definitions are stored and maintained in a controlled method. A metadata repository is the database where metadata is stored. The registry also adds relationships with related metadata types. A metadata engine collects, stores and analyzes information about data and metadata (data about data) in use within a domain. Metadata registries are used whenever data must be used consistently within an organization or group of organizations.

Related lectures (64)

Gitlab Agent for Kubernetes (`agentk`)

Covers the setup of a Gitlab agent for Kubernetes, focusing on installation, version control, and troubleshooting.

File Organization: Storage and Cache Management

Explores file and page organization in databases, cache management, and log-structured files.

In Silico Neuroscience: Data Reproducibility and Reusability

Emphasizes data reproducibility and reusability in in silico neuroscience, focusing on neuroinformatics tools and methods.

Data Storage and Transmission: Metadata for Disks

Explores the importance of metadata in organizing media libraries and the challenges in retrieving specific works or artists efficiently.

Data, big data, clouds and IoT