Concept

Data lake

A data lake is a system or repository of data stored in its natural/raw format, usually object blobs or files. A data lake is usually a single store of data including raw copies of source system data, sensor data, social data etc., and transformed data used for tasks such as reporting, visualization, advanced analytics and machine learning. A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and binary data (images, audio, video). A data lake can be established "on premises" (within an organization's data centers) or "in the cloud" (using cloud services from vendors such as Amazon, Microsoft, or Google). James Dixon, then chief technology officer at Pentaho, coined the term by 2011 to contrast it with data mart, which is a smaller repository of interesting attributes derived from raw data. In promoting data lakes, he argued that data marts have several inherent problems, such as information siloing. PricewaterhouseCoopers (PwC) said that data lakes could "put an end to data silos". In their study on data lakes they noted that enterprises were "starting to extract and place data for analytics into a single, Hadoop-based repository." Many companies use cloud storage services such as Google Cloud Storage and Amazon S3 or a distributed file system such as Apache Hadoop distributed file system (HDFS). There is a gradual academic interest in the concept of data lakes. For example, Personal DataLake at Cardiff University is a new type of data lake which aims at managing big data of individual users by providing a single point of collecting, organizing, and sharing personal data. An earlier data lake (Hadoop 1.0) had limited capabilities with its batch-oriented processing (Map Reduce) and was the only processing paradigm associated with it. Interacting with the data lake meant one had to have expertise in Java with map reduce and higher-level tools like Apache Pig, Apache Spark and Apache Hive (which by themselves were originally batch-oriented).

Official source

https://en.wikipedia.org/wiki/Data_lake

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related courses (20)

CS-401: Applied data analysis

This course teaches the basic techniques, methodologies, and practical skills required to draw meaningful insights from a variety of data, with the help of the most acclaimed software tools in the data science world (pandas, scikit-learn, Spark, etc.) ...

BIO-369: Randomness and information in biological data

Biology is becoming more and more a data science, as illustrated by the explosion of available genome sequences. This course aims to show how we can make sense of such data and harness it in order to understand biological processes in a quantitative way. ...

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Related lectures (32)

Big Data: Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, typical architecture, challenges, and technologies used to address them.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Data Wrangling: ETL Process and Wrangling Issues

Explores the ETL process, data wrangling stages, and common issues.

Related publications (29)

Designing Practical Federated Learning Systems under Data Heterogeneity

As data governance and privacy concerns increasingly limit centralized data collection, learning from decentralized datasets has become essential. Federated Learning (FL) provides a principled framework for this setting by enabling collaborative training across decentralized data silos. However, deploying FL in practice remains challenging due to communication costs, system heterogeneity, robustness and privacy requirements, and the need to support the full data lifecycle, including unlearning. Many of thes

EPFL2026

Data for The Diocotron Instability in the Trapped Electrons Experiment T-REX and its Relevance to Electron Clouds in Gyrotron Guns

This data are the raw files from the experiments with T-REX and the respective .m file to load, convert and plot them. This is currents, voltages, magnetic fields. Within the .zip files the raw data are contained among with the .m files to run, compile, calculate and create figures.

Zenodo2026

ESO-VLT BlueMuse instrument - conceptual design for phase A

BlueMUSE [1] is a blue-optimised, medium spectral resolution, panoramic integral field spectrograph under development for the ESO’s Very Large Telescope (VLT). The project is now entering preliminary design phase. With an optimised transmission down to 350 nm, spectral resolution of R~3500 on average across the wavelength range, and a large FoV (1 arcmin2), BlueMUSE will open up a new range of galactic and extragalactic science cases facilitated by its specific capabilities. BlueMUSE consists of several sub

SPIE2026

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Official source

https://en.wikipedia.org/wiki/Data_lake

About this result

Related courses (20)

CS-401: Applied data analysis

BIO-369: Randomness and information in biological data

EE-556: Mathematics of data: from theory to computation

Related lectures (32)

Big Data: Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, typical architecture, challenges, and technologies used to address them.

Big Data Best Practices and Guidelines

Covers best practices and guidelines for big data, including data lakes, architecture, challenges, and technologies like Hadoop and Hive.

Data Wrangling: ETL Process and Wrangling Issues

Explores the ETL process, data wrangling stages, and common issues.

Related publications (29)

Designing Practical Federated Learning Systems under Data Heterogeneity

EPFL2026

Data for The Diocotron Instability in the Trapped Electrons Experiment T-REX and its Relevance to Electron Clouds in Gyrotron Guns

Zenodo2026

ESO-VLT BlueMuse instrument - conceptual design for phase A

SPIE2026

Related people (16)

Christoph Koch

Expertise knows everything ;-) Publications My publications in Google ScholarMy publications in DBLPMy papers on EPFL's publications server (InfoScience)Prospective PhD students and postdocs please read this before contacting me. Christoph Koch is a professor of Computer Science at EPFL, specializing in data management. Until 2010, he was an Associate Professor in the Department of Computer Science at Cornell University. Previously to this, from 2005 to 2007, he was an Associate Professor of Computer Scienc

Jérôme Chenal

Expertise Urban DesignUrban PlanningUrban ManagementPublic SpaceHousingDevelopment StudiesVisual Anthropology Mission Director of CEAT Director of Excellence in Africa Center Affiliate professor at UM6P Teaching & PhD Past EPFL PhD Students Stéphanie Hasler, Rémi Jaligot, Armel Firmin Kemajou Mbianda, Martí Bosch Padrós, Andrea Salmi, Vitor Pessoa Colombo, Akuto Akpedze Konou, Rim Mrani Past EPFL PhD Students as codirector Marc Antoine Messer, Alexandre Rigal Courses Urban planning in the South PE

Samy Bengio

Expertise Large Language ModelsReasoningStatistical machine learningSpeech processingPerson authenticationMultimodal processingSequence processingArtificial neural networks, support vector machinesHidden markov models, gaussian mixture modelsetc Expertise Large Language ModelsReasoningStatistical machine learningSpeech processingPerson authenticationMultimodal processingSequence processingArtificial neural networks, support vector machinesHidden markov models, gaussian mixture modelsetc Mission Samy Bengio

Denis Gillet

Denis Gillet received the Diploma in Electrical Engineering from the Swiss Federal Institute of Technology in Lausanne (EPFL) in 1988, and the Ph.D. degree in Information Systems also from the EPFL in 1995. During 1992 he was appointed as Research Fellow at the Information Systems Laboratory of Stanford University in the United States. He is currently Maître d'enseignement et de recherche at the EPFL School of Engineering, where he leads the React research group. His current research interests include Technologies Enhanced Learning (TEL), Human Computer Interaction (HCI), Human Devices Interaction (HDI) and Optimal Coordination of Complex and Distributed Systems. Denis Gillet is affiliated at EPFL with the Center for Intelligent Systems and the Center for Digital Education.

Babak Falsafi

Babak is a Professor in the School of Computer and Communication Sciences and the founding director of the EcoCloud, an industrial/academic consortium at EPFL investigating scalable data-centric technologies. He has made numerous contributions to computer system design and evaluation including a scalable multiprocessor architecture which was prototyped by Sun Microsystems (now Oracle), snoop filters and memory streaming technologies that are incorporated into IBM BlueGene/P and Q and ARM cores, and computer system performance evaluation methodologies that have been in use by AMD, HP and Google PerKit . He has shown that hardware memory consistency models are neither necessary (in the 90's) nor sufficient (a decade later) to achieve high performance in multiprocessor systems. These results eventually led to fence speculation in modern microprocessors. His latest work on workload-optimized server processors laid the foundation for the first generation of Cavium ARM server CPUs, ThunderX. He is a recipient of an NSF CAREER award, IBM Faculty Partnership Awards, and an Alfred P. Sloan Research Fellowship. He is a fellow of IEEE and ACM.

Elias Zsolt Stutz

Thomas La Grange

Expertise Electron Microscopy, Aberration Correct Scanning Transmission Electron Microscopy, Electron Diffraction, Ultrafast Transmission Electron Microscopy, Electron Energy Loss Spectroscopy, Energy Dispersive Spectroscopy, Materials Science, High-speed deformation, Dislocation Dynamics, Skyrmionics, Martensitic Phase Transformation Theory, Solidification Expertise Electron Microscopy, Aberration Correct Scanning Transmission Electron Microscopy, Electron Diffraction, Ultrafast Transmission Electron Micro

Anna Fontcuberta i Morral

2014 Associate Professor at the Institut des Matériaux, EPFL 2008 Assistant Professor Tenure Track at the Institut des Matériaux, EPFL 2009 Habilitation in Physics, Technische Universität München 2005-2010 Marie Curie Excellence Grant Team Leader at Walter Schottky Institut, Technische Universität München, on leave from Centre National de la Recherche Scientifique (CNRS, France) 2004-2005 Visiting Scientist at the California Institute of Technology, on leave from CNRS; Senior Scientist and co-founder of Aonex Technologies (a startup company for large area layer transfer of InP and Ge on foreign substrates for the main application of multi-junction solar cells) 2003 Permanent Research Fellow at CNRS, Ecole Polytechnique, France 2001-2002 Postdoctoral Scholar at the California Institute of Technology Study of wafer bonding and hydrogen-induced exfoliation processes for integration of mismatched materials in views of photovoltaic applications Sponsor: Professor Harry A. Atwater 1998-2001 PhD in Materials Science, Ecole Polytechnique Study of polymorphous silicon: growth mechanisms, optical and structural properties. Application to Solar Cells and Thin Film Transistors Advisor: Pere Roca i Cabarrocas 1997-1998 Diplôme dEtudes Approfondis (D.E.A.) in Materials Science at Université Paris XI, France . 1993-1997 BA in Physics at Universitat de Barcelona

Michaël Unser

Expertise Image Processing, Medical Imaging, Biological Imaging, Wavelets, Splines, Multiresolution Michael Unser is Full Professor with EPFL’s school of Engineering and the academic director of EPFL's Center for Imaging, Lausanne, Switzerland. His primary areas of investigation are signal processing, biomedical imaging and applied functional analysis. He is internationally recognized for his research contributions to sampling theory, wavelets, the use of splines for image processing, stochastic processes,

Daniel Sage

Expertise PublicationsPublications Coordinates Biomedical Imaging GroupAddressEPFL STI-IMT-LIB, BM 4.135, Station 17, CH-1015 Lausanne, SwitzerlandTel.+41 21 693 51 89Fax.+41 21 693 68 10Emaildaniel.sage@epfl.chWeb pagehttp://bigwww.epfl.ch/sage/OfficeBM 4.135 Teaching & PhD Past EPFL PhD Students as codirector Daniel Andreas Schmitter, Anaïs Badoual, Laurène Donati Courses Bioimage informatics BIO-410 The course provides a comprehensive overview of methods, algorithms, and computer tools us

Related units (2)

Distributed Information Systems Laboratory

Data-Intensive Applications and Systems Laboratory

Related concepts (3)

Apache Spark

Apache Spark is an open-source unified analytics engine for large-scale data processing. Spark provides an interface for programming clusters with implicit data parallelism and fault tolerance. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Apache Spark has its architectural foundation in the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.

Extract, transform, load

In computing, extract, transform, load (ETL) is a three-phase process where data is extracted, transformed (cleaned, sanitized, scrubbed) and loaded into an output data container. The data can be collated from one or more sources and it can also be output to one or more destinations. ETL processing is typically executed using software applications but it can also be done manually by system operators. ETL software typically automates the entire process and can be run manually or on reoccurring schedules either as single jobs or aggregated into a batch of jobs.

Data integration

Data integration involves combining data residing in different sources and providing users with a unified view of them. This process becomes significant in a variety of situations, which include both commercial (such as when two similar companies need to merge their databases) and scientific (combining research results from different bioinformatics repositories, for example) domains. Data integration appears with increasing frequency as the volume (that is, big data) and the need to share existing data explodes.