Lecture

Resource Management: Static vs Resource Containers in YARN

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-422: Database systems

This course is intended for students who want to understand modern large-scale data analysis systems and database systems. It covers a wide range of topics and technologies, and will prepare students to be able to build such systems as well as read and und ...

Description

This lecture delves into resource management in distributed computing frameworks, focusing on static resource allocation and the concept of resource containers. The instructor explains how static allocation can lead to cluster underutilization and introduces the idea of resource containers as a more flexible approach. By using YARN as a case study, the lecture showcases how resource containers centralize resource management, enabling multiple frameworks to coexist harmoniously and scale efficiently.

Instructor

Anastasia Ailamaki

Expertise Database management systems, scientific applications, computer architecture Teaching & PhD PhD Students Hamish Nicholson, Anna Herlihy, Yi Jiang, Eleni Zapridou, Georgiy Konstantinovich Lebedev, Ioanna Tsakalidou Past EPFL PhD Students Farhan Tauheed, Radu Ioan Stoica, Manos Athanassoulis, Pinar Tözün, Adrian Daniel Popescu, Ioannis Alagiannis, Renata Borovica-Gajic, Danica Porobic, Iraklis Psaroudakis, Manos Karpathiotakis, Matthaios Alexandros Olma, Mirjana Pavlovic, Eleni Tzirita Zacharatou,

Official source

Related concepts (10)

Yarn

Yarn is a long continuous length of interlocked fibres, used in sewing, crocheting, knitting, weaving, embroidery, ropemaking, and the production of textiles. Thread is a type of yarn intended for sewing by hand or machine. Modern manufactured sewing threads may be finished with wax or other lubricants to withstand the stresses involved in sewing. Embroidery threads are yarns specifically designed for needlework. Yarn can be made of a number of natural or synthetic materials, and comes in a variety of colors and thicknesses (referred to as "weights").

Cluster analysis

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters). It is a main task of exploratory data analysis, and a common technique for statistical data analysis, used in many fields, including pattern recognition, , information retrieval, bioinformatics, data compression, computer graphics and machine learning.

Thread (yarn)

A thread is a long strand of material, often composed of several filaments or fibres, used for joining, creating or decorating textiles. In early humanity Egyptians were known for creating thread using plant fibers, wool and hair. Today, thread can also be made of many different materials including but not limited to cotton, wool, linen, nylon, silk, polyester etc. Their are also Metal threads, sometimes used in decorative textiles, can be made of fine wire.

Clustering high-dimensional data

Clustering high-dimensional data is the cluster analysis of data with anywhere from a few dozen to many thousands of dimensions. Such high-dimensional spaces of data are often encountered in areas such as medicine, where DNA microarray technology can produce many measurements at once, and the clustering of text documents, where, if a word-frequency vector is used, the number of dimensions equals the size of the vocabulary.

MapReduce

MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a map procedure, which performs filtering and sorting (such as sorting students by first name into queues, one queue for each name), and a reduce method, which performs a summary operation (such as counting the number of students in each queue, yielding name frequencies).

Related lectures (1)

Data Wrangling with Hive: Managing Big Data Efficiently