Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Big Data Challenges: Distributed Computing with Spark

In course

CS-401: Applied data analysis

This course teaches the basic techniques, methodologies, and practical skills required to draw meaningful insights from a variety of data, with the help of the most acclaimed software tools in the data science world (pandas, scikit-learn, Spark, etc.) ...

Description

This lecture covers the challenges posed by big data, the growth of data sources, and the limitations of single-machine processing. It introduces the concept of RDDs in Spark, explaining their distribution over clusters and parallel processing. The instructor discusses the hardware requirements for big data, emphasizing the use of budget hardware and the issues related to failures and network latency. The lecture also explores the MapReduce paradigm, explaining how work is divided across machines and how failures are handled. Additionally, it covers the basics of RDD transformations and actions, as well as the importance of lazy execution and RDD persistence. The use of broadcast variables, accumulators, and Spark DataFrames is also highlighted.

Instructors (2)

Maria Brbic

Maria Brbic is an assistant professor in computer science at EPFL. Prior to joining EPFL, Maria was a postdoctoral researcher in Computer Science at Stanford University working with Jure Leskovec. She received her PhD degree from University of Zagreb in 2019, while also researching at Stanford University and University of Tokyo. Her research was awarded with the Fulbright Scholarship, L’Oreal UNESCO for Women in Science Scholarship, Branimir Jernej award for outstanding publication in biology and biomedicine, and Josip Loncar Silver Plaque award for the best doctoral dissertation. She has been named a Rising Star in EECS by MIT in 2021. Her research is focused on developing new machine learning methods and applying her methods to advance biomedical research.

Robert West

Robert West is a tenure-track assistant professor of computer science at EPFL, where he heads the Data Science Lab. In his research, he develops and applies techniques in machine learning, computational social science, natural language processing, social network analysis, and data mining. Bob also collaborates closely with the Wikimedia Foundation, in his role as a Wikimedia Research Fellow. Bob’s work has won several awards, including best/outstanding paper awards at ICWSM’21, ICWSM’19, and WWW’13, a best-paper runner-up award at WWW’16, a Google Faculty Research Award, a Facebook Research Award, a Hewlett-Packard Graduate Fellowship, and a Facebook Graduate Fellowship. He is actively involved in the research community, e.g., as an Associate Editor of ICWSM and EPJ Data Science and as a co-founder of the Wiki Workshop (held at WWW and ICWSM) and the Applied Machine Learning Days. Bob received his PhD in Computer Science from Stanford University, his MSc from McGill University, Canada, and his undergraduate degree from Technische Universität München, Germany.[Last updated: 25 Aug 2021]

Official source

Related concepts (32)

Big data

Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Though used sometimes loosely partly because of a lack of formal definition, the interpretation that seems to best describe big data is the one associated with a large body of information that we could not comprehend when used only in smaller amounts.

Apache Spark

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.

MapReduce

MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a map procedure, which performs filtering and sorting (such as sorting students by first name into queues, one queue for each name), and a reduce method, which performs a summary operation (such as counting the number of students in each queue, yielding name frequencies).

Data processing

Data processing is the collection and manipulation of digital data to produce meaningful information. Data processing is a form of information processing, which is the modification (processing) of information in any manner detectable by an observer. The term "Data Processing", or "DP" has also been used to refer to a department within an organization responsible for the operation of data processing programs. Data processing may involve various processes, including: Validation – Ensuring that supplied data is correct and relevant.

Data

In common usage and statistics, data (USˈdætə; UKˈdeɪtə) is a collection of discrete or continuous values that convey information, describing the quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted formally. A datum is an individual value in a collection of data. Data is usually organized into structures such as tables that provide additional context and meaning, and which may themselves be used as data in larger structures.

Related lectures (53)

Scaling up: Spark and Big Data

Explores the challenges of big data processing and introduces Spark as a solution.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Big Data Challenges: Scaling to Massive Data

Explores challenges of handling massive data in the era of big data, discussing solutions like MapReduce and Spark.

Advanced Spark Optimization Techniques: Managing Big Data

Discusses advanced Spark optimization techniques for managing big data efficiently, focusing on parallelization, shuffle operations, and memory management.

Big Data: Best Practices and Guidelines