Course

COM-490: Large-scale data science for real-world data

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Lectures in this course (40)

Introduction: What do we mean by Data Science?

Introduces the team, provides a crash course on Python, and explores the journey into Data Science and the importance of refining data.

General Introduction to Data Science

Offers a comprehensive introduction to Data Science, covering Python, Numpy, Pandas, Matplotlib, and Scikit-learn, with a focus on practical exercises and collaborative work.

Collaborative Data Science: Tools and Git Workflow

Explores tools like Git and Docker for collaborative data science projects.

Data Wrangling with Hadoop: Storage Formats and Hive

Explores data wrangling with Hadoop, emphasizing storage formats and Hive for big data processing.

Introduction to Spark Runtime Architecture

Introduces Apache Spark, covering its architecture, RDDs, transformations, actions, fault tolerance, deployment options, and practical exercises in Jupyter notebooks.

Spark DataFrames: Basics and Optimization

Covers the basics of Spark DataFrames, their advantages, performance comparison with RDDs, and practical demos.

Advanced Spark Optimizations and Partitioning

Covers advanced Spark optimizations, memory management, shuffle operations, and data partitioning strategies to improve big data processing efficiency.

Advanced Spark Optimizations and Partitioning

Delves into advanced Spark optimizations, partitioning, data skew, persistency, MLlib, and best practices.

Introduction to Data Stream Processing

Covers the fundamentals of data stream processing, including tools like Apache Storm and Kafka, key concepts like event time and window operations, and the challenges of stream processing.

Advanced Data Stream Processing Concepts

Explores advanced data stream processing concepts, including Kafka, Spark stream, joins, and route planning models.

Analytics on Data at Rest and Data in Motion

Explores combining data at rest with data in motion, emphasizing the Lambda architecture complexities and quality assessment of streams and batches.

Data Science Essentials: Python, Numpy, Pandas, and Scikit-learn

Covers the essentials of Data Science using Python, Numpy, Pandas, and Scikit-learn, including DNA sequence analysis and classification.

Elements of Collaborative Data Science

Introduces collaborative data science tools like Jupyter notebooks, Docker, and Git, emphasizing data versioning and containerization.

General Introduction to Big Data

Covers data science tools, Hadoop, Spark, data lake ecosystems, CAP theorem, batch vs. stream processing, HDFS, Hive, Parquet, ORC, and MapReduce architecture.

Data Wrangling with Hadoop

Covers data wrangling techniques using Hadoop, focusing on row versus column-oriented databases, popular storage formats, and HBase-Hive integration.

Introduction to Spark runtime architecture

Introduces Apache Spark, covering its key features, history, RDDs, architecture, and distributed computing framework.

Spark Data Frames

Covers Spark Data Frames, distributed collections of data organized into named columns, and the benefits of using them over RDDs.

Advanced Spark Optimization

Delves into advanced Spark optimization techniques, emphasizing data partitioning, shuffle operations, and memory management.

Data Stream Processing: Apache Kafka and Spark

Covers data stream processing with Apache Kafka and Spark, including event time vs processing time, stream processing operations, and stream-stream joins.

Introduction to Data Stream Processing

Introduces data stream processing, covering batch vs stream processing, real-time insights, applications, challenges, and tools like Apache Kafka and Spark Streaming.