Database systems have always been designed and optimized to maximize a specific target metric. For over four decades, throughput has been the target metric of choice for Online Transaction Processing engines. Traditionally, OLTP engines were deployed on uniprocessors with high-performance disk arrays. Given the guaranteed improvement in single-threaded performance provided by Moore's law, disk-based OLTP engines focused on optimizing for the then-dominating source of overhead - disk I/O. Research into multitasking and buffer caching techniques steadily provided improvements in throughput and research into concurrency control and recovery techniques guaranteed that such improvements could be achieved without compromising data consistency.
Around mid 2000s, Dennard scaling came to a crushing halt and processor vendors were no longer able to raise clock frequencies without substantially increasing power consumption. This led to the birth of multicore processors, which provide explicit threadlevel parallelism as an alternative to frequency scaling for increasing throughput. Multicore processors, however, also challenge OLTP engine design, as the transition from multitasking-style parallelism to true concurrency requires both thread synchronization to protect data structure consistency and transaction synchronization to protect database consistency. Thus, OLTP research focused on developing scalable synchronization techniques for exploiting parallelism provided by multicore processors [1,2,3].
Towards the late 2000s, DRAM price free-fall reached a level where it was possible for a single server to have terabytes of memory. With the exception of a few rare cases, it is now possible to fit most operational databases entirely in memory. Research from the database community quickly demonstrated the lack of scalability of traditional disk-based OLTP engines in the new main-memory contexts [4]. Subsequent research efforts on scalable main-memory OLTP engines adopt radically different designs when compared to their disk-based counterparts [5,6,7].
Today, state-of-the-art main-memory OLTP engines can handle millions of transactions per second and provide near-linear scalability under most workloads. However, three recent trends indicate an impending change in OLTP engine design once again: changes in application workloads, shifting hardware landscape, and new target metrics. We describe these trends briefly, and then outline the contents of the talk.
In this talk, we discuss the implications of these trends on the design of next-generation transaction processing engines. We revisit old designs, examine current designs, and explore new designs with the twin goal of meeting changing application demands and optimizing for newer metrics by exploiting emerging hardware. We also discuss our ongoing projects to address the issues stemming from the changing hardware and software landscape and adapt engine designs to emerging trends, in order to demonstrate that tr
