Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Spark Ecosystem: Architectural Choices

In course

CS-422: Database systems

This course is intended for students who want to understand modern large-scale data analysis systems and database systems. It covers a wide range of topics and technologies, and will prepare students to be able to build such systems as well as read and und ...

Description

This lecture covers the Spark ecosystem, focusing on architectural choices such as the storage layer, programming models, fault tolerance, and RDDs. It discusses the limitations of MapReduce, the advantages of Spark's Resilient Distributed Datasets (RDDs), and the data flow model. The instructor explains how Spark improves expressiveness, extensibility, and performance by utilizing distributed RAM and enabling additional optimizations. Examples from Apache Spark, Microsoft, and Google are used to illustrate the concepts.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Instructor

Anastasia Ailamaki

Official source

Related concepts (17)

Optimizing compiler

In computing, an optimizing compiler is a compiler that tries to minimize or maximize some attributes of an executable computer program. Common requirements are to minimize a program's execution time, memory footprint, storage size, and power consumption (the last three being popular for portable computers). Compiler optimization is generally implemented using a sequence of optimizing transformations, algorithms which take a program and transform it to produce a semantically equivalent output program that uses fewer resources or executes faster.

Program optimization

In computer science, program optimization, code optimization, or software optimization, is the process of modifying a software system to make some aspect of it work more efficiently or use fewer resources. In general, a computer program may be optimized so that it executes more rapidly, or to make it capable of operating with less memory storage or other resources, or draw less power. Although the word "optimization" shares the same root as "optimal", it is rare for the process of optimization to produce a truly optimal system.

Peephole optimization

Peephole optimization is an optimization technique performed on a small set of compiler-generated instructions; the small set is known as the peephole or window. Peephole optimization involves changing the small set of instructions to an equivalent set that has better performance.

Loop optimization

In compiler theory, loop optimization is the process of increasing execution speed and reducing the overheads associated with loops. It plays an important role in improving cache performance and making effective use of parallel processing capabilities. Most execution time of a scientific program is spent on loops; as such, many compiler optimization techniques have been developed to make them faster. Since instructions inside loops can be executed repeatedly, it is frequently not possible to give a bound on the number of instruction executions that will be impacted by a loop optimization.

Interprocedural optimization

Interprocedural optimization (IPO) is a collection of compiler techniques used in computer programming to improve performance in programs containing many frequently used functions of small or medium length. IPO differs from other compiler optimizations by analyzing the entire program as opposed to a single function or block of code. IPO seeks to reduce or eliminate duplicate calculations and inefficient use of memory and to simplify iterative sequences such as loops.

Related lectures (54)

Optimizing Recursive Queries

Explores optimizing recursive queries in database systems using Datalog and semirings, discussing the challenges and solutions in data analytics.

Gossip Efficiency: Decentralized Systems

Explores gossip efficiency in decentralized systems, covering protocols, interaction needs, and bandwidth optimization, along with search algorithms and optimizations.

Optimization with Constraints: KKT Conditions

Covers the optimization with constraints, focusing on the Karush-Kuhn-Tucker (KKT) conditions.

Introduction to Algorithms

Explores the ingredients and selection of algorithms for different goals.

Energy Systems Optimization