Mitigating the Bandwidth Wall via Data-Streaming System-Accelerator Co-Design

Transformers have revolutionized AI in natural language processing and computer vision, but their enormous computation and memory demands pose significant challenges for hardware acceleration. In practice, endto-end throughput is often limited by paged data movement and interconnect bandwidth, not just raw MAC count. This work proposes a unified system-accelerator co-design approach to efficiently accelerate transformer inference by jointly optimizing a novel hardware matrix accelerator and its system integration with paged, streaming dataflows and explicit overlap of compute and transfer. On the hardware side, we introduce MatrixFlow, a loosely-coupled 16×16 systolic-array accelerator featuring a block-based matrix multiplication method that is page-aligned (4 KB tiles), uses only a small (≈20 KB) on-chip buffer, and runs a pipelined schedule of DMA, compute, and DMA-out to fully utilize interconnect bandwidth, emphasizing standard DMA-driven streaming rather than large on-chip reuse. On the system side, we develop Gem5-AcceSys, an extension of the gem5 full system simulator allowing exploration of standard interconnects (PCIe) and configurable memory hierarchies including Direct-Memory (DM), Direct-Cache (DC), and Device-Memory (DevMem) modes with SMMU/TLB effects. Through co-design, MatrixFlow's novel dataflow and the Gem5-AcceSys platform are tuned in tandem to alleviate data-movement bottlenecks without requiring specialized CPUinstruction-set modifications. We validate our approach with gem5 simulations on representative transformer models (BERT and ViT) across multiple data types and system setups. Results demonstrate up to 22× speedup in end-to-end inference over a CPU-only baseline and performance gains of 5×-8× over state-of-the-art loosely-and tightly-coupled accelerators. Furthermore, we show that a standard PCIe-based host memory design can achieve ∼80% of the performance of on-device HBM memory. Overall, paged streaming and pipeline overlap, not large local SRAMs, emerge as the most effective knobs for efficient transformer inference under realistic system constraints.