Lecture

2024 - DSLab - Week6 - Introduction to the Spark runtime architecture

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Related lectures (40)

Cache Memory

Explores cache memory design, hits, misses, and eviction policies in computer systems, emphasizing spatial and temporal locality.

Machine learning: Physics and Data

Delves into the intersection of physics and data in machine learning models, covering topics like atomic cluster expansion force fields and unsupervised learning.

Introduction to Spark Runtime Architecture

Introduces Apache Spark, covering its architecture, RDDs, transformations, actions, fault tolerance, deployment options, and practical exercises in Jupyter notebooks.

Big Data Challenges: Scaling to Massive Data

Explores challenges of handling massive data in the era of big data, discussing solutions like MapReduce and Spark.

Introduction to Data Stream Processing: Concepts and Applications

Covers the principles of data stream processing and its applications in real-time data analysis.

CS-200 -- 1c. Instruction Set Architecture

Data Wrangling with Hadoop

Covers data wrangling techniques using Hadoop, focusing on row versus column-oriented databases, popular storage formats, and HBase-Hive integration.

Big Data Challenges: Distributed Computing with Spark

Explores big data challenges, distributed computing with Spark, RDDs, hardware requirements, MapReduce, transformations, and Spark DataFrames.

Handling Data: Data Models and Wrangling

Explores data handling fundamentals, including models, sources, and wrangling, emphasizing the importance of understanding and addressing data problems.

Integrating Scalable Data Storage and Map Reduce Processing with Hadoop

Covers the integration of scalable data storage and map reduce processing using Hadoop, including HDFS, Hive, Parquet, ORC, Spark, and HBase.

Maximum Likelihood: Inference and Model Comparison

Explores maximum likelihood inference, model selection, and comparing models using likelihood ratios.

Multiclass Classification

Covers the concept of multiclass classification and the challenges of linearly separating data with multiple classes.

Neuroscience Data Analysis

Explores neuroscience data analysis, focusing on K+ and Na+ currents, Ca²+ currents, and data reproducibility.

Data Science Fundamentals

Covers the fundamentals of data science, emphasizing breadth over depth and practical application in machine learning and data analysis.

Neural Networks: Training and Activation

Explores neural networks, activation functions, backpropagation, and PyTorch implementation.

Data, big data, clouds and IoT

Explores data representation, databases, cloud computing, and challenges in the cloud environment.

Introduction to Spark runtime architecture

Introduces Apache Spark, covering its key features, history, RDDs, architecture, and distributed computing framework.

General Introduction to Data Science

Offers a comprehensive introduction to Data Science, covering Python, Numpy, Pandas, Matplotlib, and Scikit-learn, with a focus on practical exercises and collaborative work.

Data Wrangling Techniques: HBase and Hive Integration

Covers data wrangling techniques using HBase and Hive, focusing on integration and practical applications.

Machine Learning Basics

Introduces the basics of machine learning, covering supervised and unsupervised learning, linear regression, and data understanding.