Extending Capabilities of Attention-based Models

Deep learning has brought incredible progress in areas where traditional algorithms struggle. It can be because they need rules that are sometimes hard to conceptualize, it can also be that the number and manner of combining these rules are too complex to grasp. Learning strategies, on the other hand, automatically leverage the hidden structures present in the data to achieve good performance on these complex tasks.

These progresses have recently reached general public consideration. Text-based assistants using Large language models (LLMs) are the main example, but deep learning models have proven their usefulness in many subfields of language, vision, and speech. While these domains are general purpose and benefit from data sets of colossal size, many more specific areas can similarly benefit from deep learning. This thesis focuses on attention models, which are the most versatile architectures in the field. % In this thesis, we focus on the most versatile deep learning architecture: attention models. We extend their capabilities making them capable of handling data coming from more unexplored application domains with specificities of real data.

First, we introduce the problem of high-altitude wind nowcasting and introduce a smart averaging model, which serves as a baseline. Next, we discuss the limitations of attention models and, consequently, present a sub-quadratic alternative based on multilayer perceptrons (MLPs). In the second part, we present an attention-based model as a solution to the high-altitude wind nowcasting problem. We show that this model is simple and robust and that attention layers help during learning. We also demonstrate that models do not need to have an encoder-decoder structure to do context-conditional inference. We conclude this work with the introduction of a new method for generating more complex signals. It challenges a traditional take: the assumption that the order of autoregression must be similar to the natural order of the data. We realized that during generation, models generally have a good estimate of the rest of the sequence. This allowed us to introduce a generation method with a sub-linear number of steps as a function of the number of elements, by exploiting the fact that it is often possible to generate several elements in parallel and validate those that are self-consistent.

These methods contribute to a current effort to broaden the application domains of deep learning to specific domains and data that are structured differently than those belonging to the canonical fields of vision, speech, and language.