Modern cloud services and AI inference run on heterogeneous platforms---CPUs, GPUs, and domain-specific accelerators---yet delivered performance and energy efficiency routinely fall far short of what the hardware
can do on paper. The gap is not in the compute engines themselves: it lies in data movement, address translation, interconnect protocols, and runtime control policies that sit between components and between
software layers. As models grow and hardware diversifies, these cross-layer costs become the dominant bottleneck, and no single-layer fix can close the gap alone.
This dissertation tackles this problem from five angles, progressing across the system stack.
We begin by asking where the efficiency is lost. A cross-generational study of cloud CPU workloads on seven Intel Xeon generations reveals that typical services retire only about one-third of available pipeline
slots; the rest is consumed by memory stalls, branch mispredictions, and speculative waste. An Issued-to-Retired Gap (IRG) metric exposes that wider cores do not proportionally help---the mismatch between
microarchitectural scaling and real workload behavior is widening, not closing.
We then address a specific OS-level bottleneck uncovered by this diagnosis: virtual-to-physical address translation. Every memory access passes through the TLB, and misses trigger multi-level page walks costing
hundreds of cycles. We propose Intermediate Address Space (IAS), which splits translation into two decoupled stages and moves the expensive part off the cache-critical path. IAS speeds up cache-resident workloads
by up to 40% on CPUs and 185% on APUs without giving up the virtual-memory abstraction.
Moving down to accelerator integration, we develop Gem5-AcceSys, a full-system framework modeling PCIe links, DMA engines, and IOMMU/SMMU translation inside gem5, showing that host-memory designs over standard
interconnects can reach ~80% of device-attached memory performance when the data path is properly orchestrated. We co-design MatrixFlow, a streaming systolic-array accelerator that tiles matrices to 4 KB pages,
pipelines DMA with compute, and keeps only minimal on-chip buffers. On transformer models (BERT, ViT), MatrixFlow achieves up to 22x speedup over CPU-only execution and 5-8x over prior loosely- and tightly-coupled
accelerators.
At the serving layer, we observe that once compute and data paths are efficient, the remaining energy slack is in how the GPU reacts to workload dynamics. GreenLLM separates LLM inference into prefill and decode
phases and applies per-phase frequency scaling under latency SLOs. On real Alibaba and Azure traces, it cuts GPU energy by up to 34% with no throughput loss and fewer than 3.5% additional SLO violations.
Finally, SigmaQuant assigns per-layer quantization bitwidths under explicit memory and accuracy budgets, guided by distribution statistics rather than exhaustive search. At equal model size it improves accuracy
