Data-driven decision making requires organizations to extract insights from real-time data. Stream processing must handle high-volume and diverse queries and data. Query volumes grow as businesses seek more insights, while variability arises from different query characteristics and performance requirements. Data volume stems from constant event generation, and variability reflects shifting distributions. To handle such workloads, modern stream processing engines adopt a distributed architecture that relies on query, data, and state partitioning. They dedicate resources per query for performance isolation and scale operators by statically partitioning data and co-located state. However, this architecture is costly and scales poorly under high query concurrency, volatile distributions, and large states: per-query resource reservations inflate infrastructure costs, skewed distributions create stragglers that cap scalability, and coupling state access to the data path puts I/O on the critical path, stalling the CPU and inflating latency.
This thesis enables scalable, resource-efficient processing under evolving data streams by making performance primarily a function of non-redundant work rather than of the query count, data skew, and state size. We adapt execution to the workload along the three main axes of the system architecture: (i) we handle high query volumes by revisiting shared query execution while preserving per-query performance; (ii) process volatile data streams by learning and continuously adjusting the partitioning policy; and (iii) hide expensive state access by decoupling it from the data path.
To improve resource efficiency at high query concurrency without compromising per-query performance, we introduce FunShare, a system that dynamically partitions queries into sharing groups based on sharing opportunities and performance constraints. FunShare minimizes redundant computation while preserving per-query performance through an adaptive framework that monitors execution metrics, accurately estimates computation overlaps, and reconfigures plans on the fly.
To achieve high effective parallelism under volatile streams, we propose Dalton, a lightweight, adaptive, yet scalable partitioning operator that relies on reinforcement learning. By memoizing state and tracking recent experience, Dalton: i) adjusts its policy at runtime to the workload, ii) minimizes per-tuple overhead, and iii) efficiently scales to multiple instances that learn cooperatively and converge to a joint policy.
To hide expensive state access, we propose Keyed Prefetching, which decouples state I/O from the data path by extracting future access keys at upstream lookahead operators and fetching state as soon as the key is known, rather than when the tuple finally reaches the stateful operator. Keyed Prefetching is accurate---hints carry exact keys from lookahead operators---and timely---the runtime selects the lookahead whose hints arrive early enough to hide I/O.
