Lecture

Collaborative Data Science: Tools and Git Workflow

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

COM-490: Large-scale data science for real-world data

This hands-on course covers tools and methods used by data scientists, from researching solutions to scaling prototypes on Spark clusters. Students engage with the full data engineering and data science pipeline, from data acquisition to extracting insight ...

Description

This lecture covers the tools and workflows essential for collaborative data science projects, including code management with Git, data versioning with Git LFS, and containerization with Docker. It explores Git commands, decentralized version control, and common Dockerfile directives.

Instructors (3)

Olivier Verscheure

SWISS DATA SCIENCE CENTER Housed at both EPFL and ETH Zurich, the newly created Swiss Data Science Center (SDSC) will address the fragmentation within today's data and analytics landscapes. The SDSC will be composed of a distributed multi-disciplinary team of data scientists and experts in domains including personalized health and personalized medicine, transportation, earth and environmental science, social science and digital humanities, and economics. The center aims to federate data providers, data and

Sofiane Sarni

Teaching & PhD Courses Large-scale data science for real-world data COM-490 This hands-on course covers tools and methods used by data scientists, from researching solutions to scaling prototypes on Spark clusters. Students engage with the full data engineering and data science pipeline, from data acquisition to extracting insights, applied to real-world problems.

Pamela Isabel Delgado Borda

I am a PhD student in the School of Computer and Communication Sciences at EPFL. I am part of the Operating Systems Laboratory and my advisor is Prof. Willy Zwaenepoel. I received my Bachelor's degree in Systems Engineering from Universidad Catolica Boliviana, Bolivia in 2008 and Master's degree in Computer Science, specialization Foundations of Software from EPFL in 2012.

Official source

Related concepts (30)

Git

Git (gɪt) is a distributed version control system that tracks changes in any set of s, usually used for coordinating work among programmers who are collaboratively developing source code during software development. Its goals include speed, data integrity, and support for distributed, non-linear workflows (thousands of parallel branches running on different computers). Git was originally authored by Linus Torvalds in 2005 for development of the Linux kernel, with other kernel developers contributing to its initial development.

Version control

In software engineering, version control (also known as revision control, source control, or source code management) is a class of systems responsible for managing changes to computer programs, documents, large web sites, or other collections of information. Version control is a component of software configuration management. Changes are usually identified by a number or letter code, termed the "revision number", "revision level", or simply "revision". For example, an initial set of files is "revision 1".

File system

In computing, a file system or filesystem (often abbreviated to fs) is a method and data structure that the operating system uses to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one large body of data with no way to tell where one piece of data stopped and the next began, or where any piece of data was located when it was time to retrieve it. By separating the data into pieces and giving each piece a name, the data are easily isolated and identified.

GitHub

GitHub, Inc. (ˈgɪthʌb) is a platform and cloud-based service for software development and version control using Git, allowing developers to store and manage their code. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous integration, and wikis for every project. Headquartered in California, it has been a subsidiary of Microsoft since 2018. It is commonly used to host open source software development projects.

Distributed version control

In software development, distributed version control (also known as distributed revision control) is a form of version control in which the complete codebase, including its full history, is mirrored on every developer's computer. Compared to centralized version control, this enables automatic management branching and merging, speeds up most operations (except pushing and pulling), improves the ability to work offline, and does not rely on a single location for backups.

Related lectures (32)

Collaborative Data Science: Tools and Techniques

Introduces collaborative data science tools like Git and Docker, emphasizing teamwork and practical exercises for effective learning.

Version Control System: Basics and Workflow

Introduces the basics of Version Control Systems, focusing on Git operations and branching strategies.

Logistic Regression: Vegetation Prediction

Explores logistic regression for predicting vegetation proportions in the Amazon region through remote sensing data analysis.

Distributed Version Control with Git

Explores Git's distributed version control, covering conflict resolution, collaboration management, and merging in software projects.

Introduction to Jupyter Notebook

Introduces setting up the environment, Git basics, and Jupyter Notebook features with practical examples.

Related concepts (30)

Distributed version control