Lecture

Distributed Execution: Challenges and Strategies

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture discusses the challenges and strategies related to distributed execution, focusing on the implementation of custom hardware, TCP sockets, InfiniBand network, and Arrow. The instructor guides the team through task prioritization, system optimization, and collaborative planning for efficient execution.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Official source

Related concepts (15)

Network socket

A network socket is a software structure within a network node of a computer network that serves as an endpoint for sending and receiving data across the network. The structure and properties of a socket are defined by an application programming interface (API) for the networking architecture. Sockets are created only during the lifetime of a process of an application running in the node.

Berkeley sockets

Berkeley sockets is an application programming interface (API) for Internet sockets and Unix domain sockets, used for inter-process communication (IPC). It is commonly implemented as a library of linkable modules. It originated with the 4.2BSD Unix operating system, which was released in 1983. A socket is an abstract representation (handle) for the local endpoint of a network communication path. The Berkeley sockets API represents it as a () in the Unix philosophy that provides a common interface for input and output to streams of data.

Mathematical optimization

Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries.

Unix domain socket

A Unix domain socket aka UDS or IPC socket (inter-process communication socket) is a data communications endpoint for exchanging data between processes executing on the same host operating system. It is also referred to by its address family AF_UNIX.

Combinatorial optimization

Combinatorial optimization is a subfield of mathematical optimization that consists of finding an optimal object from a finite set of objects, where the set of feasible solutions is discrete or can be reduced to a discrete set. Typical combinatorial optimization problems are the travelling salesman problem ("TSP"), the minimum spanning tree problem ("MST"), and the knapsack problem. In many such problems, such as the ones previously mentioned, exhaustive search is not tractable, and so specialized algorithms that quickly rule out large parts of the search space or approximation algorithms must be resorted to instead.

Related lectures (58)

Heuristic Optimization Methods

Explores heuristic optimization methods to find the global optimum efficiently.

MILP Model and Typical Days by FM

Discusses MILP model, typical days, clustering, and extreme periods analysis in energy systems optimization.

Flexible Guidance Design

Explores the design of flexible guidances, emphasizing the analysis of structures with orthogonal flexible joints and the optimization using flexible columns.

Understanding Generalization: Implicit Bias & Optimization

Explores the trade-off between complexity and risk in machine learning models, the benefits of overparametrization, and the implicit bias of optimization algorithms.

Shortest path: Properties MOOC: Optimization: principles and algorithms - Network and discrete optimization