Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Text Processing: Large Digital Text Collections Analysis

In course

DH-405: Foundations of digital humanities

This course introduces the core concepts and methodologies of Digital Humanities, integrating both theoretical and practical perspectives. Students learn to work with large-scale cultural datasets, acquiring the tools and techniques required for their proc ...

Description

This lecture explores the processing of large digital text collections in the field of Digital Humanities. It covers the extraction of hidden regularities and structures from massive textual objects, the distinction between Humanities Computing and Computational Linguistics, the challenges posed by very large textual objects, and the use of text processing pipelines. The lecture delves into the significance of projects like Project Gutenberg and Wikisource, the concept of text reuse, and the application of TF-IDF, Latent Semantic Analysis, and Topic Modeling in analyzing text data.

Instructor

Frédéric Kaplan

Official source

Ontological neighbourhood

Information engineering

Natural language processing: Topics in natural language processing

Related concepts (25)

Word embedding

In natural language processing (NLP), a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.

Linguistics

Linguistics is the scientific study of language. The modern-day scientific study of linguistics takes all aspects of language into account — i.e., the cognitive, the social, the cultural, the psychological, the environmental, the biological, the literary, the grammatical, the paleographical, and the structural. Linguistics is based on a theoretical as well as descriptive study of language, and is also interlinked with the applied fields of language studies and language learning, which entails the study of specific languages.

Word2vec

Word2vec is a technique for natural language processing (NLP) published in 2013. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence. As the name implies, word2vec represents each distinct word with a particular list of numbers called a vector.

Full-text search

In text retrieval, full-text search refers to techniques for searching a single computer-stored document or a collection in a full-text database. Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases (such as titles, abstracts, selected sections, or bibliographical references). In a full-text search, a search engine examines all of the words in every stored document as it tries to match search criteria (for example, text specified by a user).

Project Gutenberg

Project Gutenberg (PG) is a volunteer effort to digitize and archive cultural works, as well as to "encourage the creation and distribution of eBooks." It was founded in 1971 by American writer Michael S. Hart and is the oldest digital library. Most of the items in its collection are the full texts of books or individual stories in the public domain. All files can be accessed for free under an open format layout, available on almost any computer. , Project Gutenberg had reached 50,000 items in its collection of free eBooks.

Related lectures (31)

Vector Space Semantics (and Information Retrieval)

Explores the Vector Space model, Bag of Words, tf-idf, cosine similarity, Okapi BM25, and Precision and Recall in Information Retrieval.

Handling Text: Document Retrieval, Classification, Sentiment Analysis

Explores document retrieval, classification, sentiment analysis, TF-IDF matrices, nearest-neighbor methods, matrix factorization, regularization, LDA, contextualized word vectors, and BERT.

Document Retrieval and Classification

Covers document retrieval, classification, sentiment analysis, and topic detection using TF-IDF matrices and contextualized word vectors like BERT.

Text Handling: Matrix, Documents, Topics

Explores text handling, focusing on matrices, documents, and topics, including challenges in document classification and advanced models like BERT.

Text Processing: Humanities Computing and Linguistics