Lecture

Scaling up: Spark and Big Data

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-401: Applied data analysis

This course teaches the basic techniques, methodologies, and practical skills required to draw meaningful insights from a variety of data, with the help of the most acclaimed software tools in the data science world (pandas, scikit-learn, Spark, etc.) ...

Description

This lecture covers the challenges of processing big data, the rise of distributed computing, and the introduction of Spark as a solution. Topics include the big-data problem, hardware considerations, MapReduce, RDDs, transformations, actions, persistence, Spark SQL, and Spark's Machine Learning Toolkit.

Instructors (2)

Maria Brbic

Maria Brbic is an assistant professor in computer science at EPFL. Prior to joining EPFL, Maria was a postdoctoral researcher in Computer Science at Stanford University working with Jure Leskovec. She received her PhD degree from University of Zagreb in 2019, while also researching at Stanford University and University of Tokyo. Her research was awarded with the Fulbright Scholarship, L’Oreal UNESCO for Women in Science Scholarship, Branimir Jernej award for outstanding publication in biology and biomedicine, and Josip Loncar Silver Plaque award for the best doctoral dissertation. She has been named a Rising Star in EECS by MIT in 2021. Her research is focused on developing new machine learning methods and applying her methods to advance biomedical research.

Robert West

Robert West is a tenure-track assistant professor of computer science at EPFL, where he heads the Data Science Lab. In his research, he develops and applies techniques in machine learning, computational social science, natural language processing, social network analysis, and data mining. Bob also collaborates closely with the Wikimedia Foundation, in his role as a Wikimedia Research Fellow. Bob’s work has won several awards, including best/outstanding paper awards at ICWSM’21, ICWSM’19, and WWW’13, a best-paper runner-up award at WWW’16, a Google Faculty Research Award, a Facebook Research Award, a Hewlett-Packard Graduate Fellowship, and a Facebook Graduate Fellowship. He is actively involved in the research community, e.g., as an Associate Editor of ICWSM and EPJ Data Science and as a co-founder of the Wiki Workshop (held at WWW and ICWSM) and the Applied Machine Learning Days. Bob received his PhD in Computer Science from Stanford University, his MSc from McGill University, Canada, and his undergraduate degree from Technische Universität München, Germany.[Last updated: 25 Aug 2021]

Official source

Related concepts (32)

Google Docs

Google Docs is an online word processor included as part of the free, web-based Google Docs Editors suite offered by Google, which also includes Google Sheets, Google Slides, Google Drawings, Google Forms, Google Sites and Google Keep. Google Docs is accessible via an internet browser as a web-based application and is also available as a mobile app on Android and iOS and as a desktop application on Google's ChromeOS. Google Docs allows users to create and edit documents online while collaborating with other users in real time.

Google Drive

Google Drive is a and developed by Google. Launched on April 24, 2012, Google Drive allows users to store files in the cloud (on Google's servers), synchronize files across devices, and . In addition to a web interface, Google Drive offers apps with offline capabilities for Windows and macOS computers, and Android and iOS smartphones and tablets. Google Drive encompasses Google Docs, Google Sheets, and Google Slides, which are a part of the Google Docs Editors office suite that permits collaborative editing of documents, spreadsheets, presentations, drawings, forms, and more.

Google

Google LLC (ˈɡuːɡəl) is an American multinational technology company focusing on artificial intelligence, online advertising, search engine technology, cloud computing, computer software, quantum computing, e-commerce, and consumer electronics. It has been referred to as "the most powerful company in the world" and as one of the world's most valuable brands due to its market dominance, data collection, and technological advantages in the field of artificial intelligence. Alongside Amazon, Apple Inc.

Logistic regression

In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear combination of one or more independent variables. In regression analysis, logistic regression (or logit regression) is estimating the parameters of a logistic model (the coefficients in the linear combination).

Data processing

Data processing is the collection and manipulation of digital data to produce meaningful information. Data processing is a form of information processing, which is the modification (processing) of information in any manner detectable by an observer. The term "Data Processing", or "DP" has also been used to refer to a department within an organization responsible for the operation of data processing programs. Data processing may involve various processes, including: Validation – Ensuring that supplied data is correct and relevant.

Related lectures (64)

Big Data Challenges: Scaling to Massive Data

Explores challenges of handling massive data in the era of big data, discussing solutions like MapReduce and Spark.

Logistic Regression: Vegetation Prediction

Explores logistic regression for predicting vegetation proportions in the Amazon region through remote sensing data analysis.

Big Data Challenges: Distributed Computing with Spark

Explores big data challenges, distributed computing with Spark, RDDs, hardware requirements, MapReduce, transformations, and Spark DataFrames.

Hadoop Ecosystem: Architectural Choices & MapReduce Programming

Explores the Hadoop ecosystem's architecture and MapReduce programming model, emphasizing strengths and limitations.

Big Data Best Practices and Guidelines