Lecture

General-Purpose Distributed Execution System

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture discusses the design of a general-purpose distributed execution system, focusing on scaling applications with distributed execution becoming the norm. It covers the challenges of building distributed systems, the use of specialized frameworks like Spark and Flink, and the implementation of distributed futures for high-performance and fault-tolerant shuffle. The lecture explores the benefits of decentralized control logic, pass-by-reference strategies, and the use of futures for asynchronous RPCs. It also showcases the results of using Exoshuffle to improve ML training speed and accuracy, as well as the advancements in fault tolerance and ownership models for distributed futures.

Official source

Related concepts (32)

Ownership

Ownership is the state or fact of legal possession and control over property, which may be any asset, tangible or intangible. Ownership can involve multiple rights, collectively referred to as title, which may be separated and held by different parties. The process and mechanics of ownership are fairly complex: one can gain, transfer, and lose ownership of property in a number of ways. To acquire property one can purchase it with money, trade it for other property, win it in a bet, receive it as a gift, inherit it, find it, receive it as damages, earn it by doing work or performing services, make it, or homestead it.

Remote procedure call

In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is written as if it were a normal (local) procedure call, without the programmer explicitly writing the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote.

Distributed data store

A distributed data store is a computer network where information is stored on more than one node, often in a replicated fashion. It is usually specifically used to refer to either a distributed database where users store information on a number of nodes, or a computer network in which users store information on a number of peer network nodes. Distributed databases are usually non-relational databases that enable a quick access to data over a large number of nodes.

Social ownership

Social ownership is the appropriation of the surplus product, produced by the means of production, or the wealth that comes from it, to society as a whole. It is the defining characteristic of a socialist economic system. It can take the form of community ownership, state ownership, common ownership, employee ownership, cooperative ownership, and citizen ownership of equity.

State ownership

State ownership, also called government ownership and public ownership, is the ownership of an industry, asset, or enterprise by the state or a public body representing a community, as opposed to an individual or private party. Public ownership specifically refers to industries selling goods and services to consumers and differs from public goods and government services financed out of a government's general budget.

Related lectures (64)

Big Data Challenges: Scaling to Massive Data

Explores challenges of handling massive data in the era of big data, discussing solutions like MapReduce and Spark.

General Introduction to Big Data

Covers data science tools, Hadoop, Spark, data lake ecosystems, CAP theorem, batch vs. stream processing, HDFS, Hive, Parquet, ORC, and MapReduce architecture.

Memory Safety and Client/Server Architecture: Key Concepts

Covers memory safety principles and the client/server architecture in system design.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Decentralized Systems: Introduction