Lecture

Big Data Challenges: Distributed Computing with Spark

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-401: Applied data analysis

This course teaches the basic techniques, methodologies, and practical skills required to draw meaningful insights from a variety of data, with the help of the most acclaimed software tools in the data science world (pandas, scikit-learn, Spark, etc.) ...

Description

This lecture covers the challenges posed by big data, the growth of data sources, and the limitations of single-machine processing. It introduces the concept of RDDs in Spark, explaining their distribution over clusters and parallel processing. The instructor discusses the hardware requirements for big data, emphasizing the use of budget hardware and the issues related to failures and network latency. The lecture also explores the MapReduce paradigm, explaining how work is divided across machines and how failures are handled. Additionally, it covers the basics of RDD transformations and actions, as well as the importance of lazy execution and RDD persistence. The use of broadcast variables, accumulators, and Spark DataFrames is also highlighted.

Instructors (2)

Maria Brbic

Maria Brbic is an assistant professor in computer science at EPFL. Prior to joining EPFL, Maria was a postdoctoral researcher in Computer Science at Stanford University working with Jure Leskovec. She received her PhD degree from University of Zagreb in 2019, while also researching at Stanford University and University of Tokyo. Her research was awarded with the Fulbright Scholarship, L’Oreal UNESCO for Women in Science Scholarship, Branimir Jernej award for outstanding publication in biology and biomedicine, and Josip Loncar Silver Plaque award for the best doctoral dissertation. She has been named a Rising Star in EECS by MIT in 2021. Her research is focused on developing new machine learning methods and applying her methods to advance biomedical research.

Robert West

Robert West is a tenure-track assistant professor of computer science at EPFL, where he heads the Data Science Lab. In his research, he develops and applies techniques in machine learning, computational social science, natural language processing, social network analysis, and data mining. Bob also collaborates closely with the Wikimedia Foundation, in his role as a Wikimedia Research Fellow. Bob’s work has won several awards, including best/outstanding paper awards at ICWSM’21, ICWSM’19, and WWW’13, a best-paper runner-up award at WWW’16, a Google Faculty Research Award, a Facebook Research Award, a Hewlett-Packard Graduate Fellowship, and a Facebook Graduate Fellowship. He is actively involved in the research community, e.g., as an Associate Editor of ICWSM and EPJ Data Science and as a co-founder of the Wiki Workshop (held at WWW and ICWSM) and the Applied Machine Learning Days. Bob received his PhD in Computer Science from Stanford University, his MSc from McGill University, Canada, and his undergraduate degree from Technische Universität München, Germany.[Last updated: 25 Aug 2021]

Official source

Related concepts (32)

Data processing

Data processing is the collection and manipulation of digital data to produce meaningful information. Data processing is a form of information processing, which is the modification (processing) of information in any manner detectable by an observer. The term "Data Processing", or "DP" has also been used to refer to a department within an organization responsible for the operation of data processing programs. Data processing may involve various processes, including: Validation – Ensuring that supplied data is correct and relevant.

MapReduce

MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is composed of a map procedure, which performs filtering and sorting (such as sorting students by first name into queues, one queue for each name), and a reduce method, which performs a summary operation (such as counting the number of students in each queue, yielding name frequencies).

Big data

Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Though used sometimes loosely partly because of a lack of formal definition, the interpretation that seems to best describe big data is the one associated with a large body of information that we could not comprehend when used only in smaller amounts.

Data Preprocessing

Data preprocessing can refer to manipulation or dropping of data before it is used in order to ensure or enhance performance, and is an important step in the data mining process. The phrase "garbage in, garbage out" is particularly applicable to data mining and machine learning projects. Data collection methods are often loosely controlled, resulting in out-of-range values, impossible data combinations, and missing values, amongst other issues. Analyzing data that has not been carefully screened for such problems can produce misleading results.

Computer cluster

A computer cluster is a set of computers that work together so that they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software. The components of a cluster are usually connected to each other through fast local area networks, with each node (computer used as a server) running its own instance of an operating system. In most circumstances, all of the nodes use the same hardware and the same operating system, although in some setups (e.

Related lectures (64)

Big Data Challenges: Scaling to Massive Data

Explores challenges of handling massive data in the era of big data, discussing solutions like MapReduce and Spark.

Scaling up: Spark and Big Data

Explores the challenges of big data processing and introduces Spark as a solution.

Data Wrangling with Hive: Managing Big Data Efficiently

Covers data wrangling techniques using Apache Hive for efficient big data management.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Data Wrangling with Hadoop