Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Scaling up: Spark and Big Data

In course

CS-401: Applied data analysis

This course teaches the basic techniques, methodologies, and practical skills required to draw meaningful insights from a variety of data, with the help of the most acclaimed software tools in the data science world (pandas, scikit-learn, Spark, etc.) ...

Description

This lecture covers the challenges of processing big data, the rise of distributed computing, and the introduction of Spark as a solution. Topics include the big-data problem, hardware considerations, MapReduce, RDDs, transformations, actions, persistence, Spark SQL, and Spark's Machine Learning Toolkit.

Instructors (2)

Robert West

Robert West is a tenure-track assistant professor of computer science at EPFL, where he heads the Data Science Lab. In his research, he develops and applies techniques in machine learning, computational social science, natural language processing, social network analysis, and data mining. Bob also collaborates closely with the Wikimedia Foundation, in his role as a Wikimedia Research Fellow. Bob’s work has won several awards, including best/outstanding paper awards at ICWSM’21, ICWSM’19, and WWW’13, a best-paper runner-up award at WWW’16, a Google Faculty Research Award, a Facebook Research Award, a Hewlett-Packard Graduate Fellowship, and a Facebook Graduate Fellowship. He is actively involved in the research community, e.g., as an Associate Editor of ICWSM and EPJ Data Science and as a co-founder of the Wiki Workshop (held at WWW and ICWSM) and the Applied Machine Learning Days. Bob received his PhD in Computer Science from Stanford University, his MSc from McGill University, Canada, and his undergraduate degree from Technische Universität München, Germany.[Last updated: 25 Aug 2021]

Maria Brbic

Maria Brbic is an assistant professor in computer science at EPFL. Prior to joining EPFL, Maria was a postdoctoral researcher in Computer Science at Stanford University working with Jure Leskovec. She received her PhD degree from University of Zagreb in 2019, while also researching at Stanford University and University of Tokyo. Her research was awarded with the Fulbright Scholarship, L’Oreal UNESCO for Women in Science Scholarship, Branimir Jernej award for outstanding publication in biology and biomedicine, and Josip Loncar Silver Plaque award for the best doctoral dissertation. She has been named a Rising Star in EECS by MIT in 2021. Her research is focused on developing new machine learning methods and applying her methods to advance biomedical research.

Official source

Related concepts (32)

Big data

Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Though used sometimes loosely partly because of a lack of formal definition, the interpretation that seems to best describe big data is the one associated with a large body of information that we could not comprehend when used only in smaller amounts.

Google Docs

Google Docs is an online word processor included as part of the free, web-based Google Docs Editors suite offered by Google, which also includes Google Sheets, Google Slides, Google Drawings, Google Forms, Google Sites and Google Keep. Google Docs is accessible via an internet browser as a web-based application and is also available as a mobile app on Android and iOS and as a desktop application on Google's ChromeOS. Google Docs allows users to create and edit documents online while collaborating with other users in real time.

Google Drive

Google Drive is a and developed by Google. Launched on April 24, 2012, Google Drive allows users to store files in the cloud (on Google's servers), synchronize files across devices, and . In addition to a web interface, Google Drive offers apps with offline capabilities for Windows and macOS computers, and Android and iOS smartphones and tablets. Google Drive encompasses Google Docs, Google Sheets, and Google Slides, which are a part of the Google Docs Editors office suite that permits collaborative editing of documents, spreadsheets, presentations, drawings, forms, and more.

Apache Spark

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.

Google

Google LLC (ˈɡuːɡəl) is an American multinational technology company focusing on artificial intelligence, online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, and consumer electronics. It has been referred to as "the most powerful company in the world" and as one of the world's most valuable brands due to its market dominance, data collection, and technological advantages in the field of artificial intelligence. Alongside Amazon, Apple Inc.

Related lectures (37)

Big Data Challenges: Scaling to Massive Data

Explores challenges of handling massive data in the era of big data, discussing solutions like MapReduce and Spark.

Big Data Challenges: Distributed Computing with Spark

Explores big data challenges, distributed computing with Spark, RDDs, hardware requirements, MapReduce, transformations, and Spark DataFrames.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Advanced Spark Optimization Techniques: Managing Big Data

Discusses advanced Spark optimization techniques for managing big data efficiently, focusing on parallelization, shuffle operations, and memory management.

Logistic Regression: Vegetation Prediction