Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Data Wrangling with Hadoop

In course

COM-490: Large-scale data science for real-world data

This hands-on course covers tools and methods used by data scientists, from researching solutions to scaling prototypes on Spark clusters. Students engage with the full data engineering and data science pipeline, from data acquisition to extracting insight ...

Description

This lecture covers data wrangling techniques using Hadoop, focusing on concepts like row versus column-oriented databases, popular HDFS storage formats, and the integration between HBase and Hive. Students will learn about Hive tables, HBase architecture, and the differences between HBase and Hive in big data processing.

Instructors (3)

Olivier Verscheure

Sofiane Sarni

Pamela Isabel Delgado Borda

I am a PhD student in the School of Computer and Communication Sciences at EPFL. I am part of the Operating Systems Laboratory and my advisor is Prof. Willy Zwaenepoel. I received my Bachelor's degree in Systems Engineering from Universidad Catolica Boliviana, Bolivia in 2008 and Master's degree in Computer Science, specialization Foundations of Software from EPFL in 2012.

Official source

Ontological neighbourhood

Computer engineering

Databases: Relational databases

Related concepts (32)

Relational database

A relational database is a (most commonly digital) database based on the relational model of data, as proposed by E. F. Codd in 1970. A system used to maintain relational databases is a relational database management system (RDBMS). Many relational database systems are equipped with the option of using SQL (Structured Query Language) for querying and updating the database. The term "relational database" was first defined by E. F. Codd at IBM in 1970. Codd introduced the term in his research paper "A Relational Model of Data for Large Shared Data Banks".

Object database

An object database or object-oriented database is a database management system in which information is represented in the form of objects as used in object-oriented programming. Object databases are different from relational databases which are table-oriented. A third type, object–relational databases, is a hybrid of both approaches. Object databases have been considered since the early 1980s. Object-oriented database management systems (OODBMSs) also called ODBMS (Object Database Management System) combine database capabilities with object-oriented programming language capabilities.

Apache HBase

HBase is an open-source non-relational distributed database modeled after Google's Bigtable and written in Java. It is developed as part of Apache Software Foundation's Apache Hadoop project and runs on top of or Alluxio, providing Bigtable-like capabilities for Hadoop. That is, it provides a fault-tolerant way of storing large quantities of data (small amounts of information caught within a large collection of empty or unimportant data, such as finding the 50 largest items in a group of 2 billion records, or finding the non-zero items representing less than 0.

Database

In computing, a database is an organized collection of data (also known as a data store) stored and accessed electronically through the use of a database management system. Small databases can be stored on a , while large databases are hosted on computer clusters or cloud storage. The design of databases spans formal techniques and practical considerations, including data modeling, efficient data representation and storage, query languages, security and privacy of sensitive data, and distributed computing issues, including supporting concurrent access and fault tolerance.

Apache Hadoop

Apache Hadoop (pronhəˈdu:p) is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for and processing of big data using the MapReduce programming model. Hadoop was originally designed for computer clusters built from commodity hardware, which is still the common use. It has since also found use on clusters of higher-end hardware.

Related lectures (52)

Data Wrangling with Hadoop: Storage Formats and Hive

Explores data wrangling with Hadoop, emphasizing storage formats and Hive for big data processing.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Data formats and data wrangling with Hadoop

Explores Apache Hive for data warehousing, data formats, and partitioning, with practical exercises in querying and connecting to Hive.

General Introduction to Data Science

Offers a comprehensive introduction to Data Science, covering Python, Numpy, Pandas, Matplotlib, and Scikit-learn, with a focus on practical exercises and collaborative work.

Big Data Ecosystems: Technologies and Challenges