Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Course

COM-490: Large-scale data science for real-world data

Lectures in this course (41)

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Introduction to Spark Runtime Architecture

Introduces Apache Spark, covering its architecture, RDDs, transformations, actions, fault tolerance, deployment options, and practical exercises in Jupyter notebooks.

Spark DataFrames: Basics and Optimization

Covers the basics of Spark DataFrames, their advantages, performance comparison with RDDs, and practical demos.

Advanced Spark Optimizations and Partitioning

Covers advanced Spark optimizations, memory management, shuffle operations, and data partitioning strategies to improve big data processing efficiency.

Advanced Spark Optimizations and Partitioning

Delves into advanced Spark optimizations, partitioning, data skew, persistency, MLlib, and best practices.

Introduction to Data Stream Processing

Covers the fundamentals of data stream processing, including tools like Apache Storm and Kafka, key concepts like event time and window operations, and the challenges of stream processing.

Advanced Data Stream Processing Concepts

Explores advanced data stream processing concepts, including Kafka, Spark stream, joins, and route planning models.

Analytics on Data at Rest and Data in Motion

Explores combining data at rest with data in motion, emphasizing the Lambda architecture complexities and quality assessment of streams and batches.

Data Stream Processing: Apache Kafka and Spark

Covers data stream processing with Apache Kafka and Spark, including event time vs processing time, stream processing operations, and stream-stream joins.

Introduction to Data Stream Processing

Introduces data stream processing, covering batch vs stream processing, real-time insights, applications, challenges, and tools like Apache Kafka and Spark Streaming.

Advanced Spark Optimization

Delves into advanced Spark optimization techniques, emphasizing data partitioning, shuffle operations, and memory management.

Spark Data Frames

Covers Spark Data Frames, distributed collections of data organized into named columns, and the benefits of using them over RDDs.

Introduction to Spark runtime architecture

Introduces Apache Spark, covering its key features, history, RDDs, architecture, and distributed computing framework.

Data Wrangling with Hadoop

Covers data wrangling techniques using Hadoop, focusing on row versus column-oriented databases, popular storage formats, and HBase-Hive integration.

General Introduction to Big Data

Covers data science tools, Hadoop, Spark, data lake ecosystems, CAP theorem, batch vs. stream processing, HDFS, Hive, Parquet, ORC, and MapReduce architecture.

Elements of Collaborative Data Science

Introduces collaborative data science tools like Jupyter notebooks, Docker, and Git, emphasizing data versioning and containerization.

Data Science Essentials: Python, Numpy, Pandas, and Scikit-learn

Covers the essentials of Data Science using Python, Numpy, Pandas, and Scikit-learn, including DNA sequence analysis and classification.

Collaborative Data Science

Covers collaborative data science tools, big data concepts, Spark, and data stream processing, with tips for the final project.

Big Data: Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, typical architecture, challenges, and technologies used to address them.

Data formats and data wrangling with Hadoop

Explores Apache Hive for data warehousing, data formats, and partitioning, with practical exercises in querying and connecting to Hive.