Diagnosing Performance Overheads in the Xen Virtual Machine Environment

Virtual Machine (VM) environments (e.g., VMware and Xen) are experiencing a resurgence of interest for diverse uses including server consolidation and shared hosting. An application's performance in a virtual machine environ- ment can differ markedly from its performance in a non- virtualized environment because of interactions with the underlying virtual machine monitor and other virtual ma- chines. However, few tools are currently available to help de- bug performance problems in virtual machine environments. In this paper, we present Xenoprof, a system-wide statis- tical profiling toolkit implemented for the Xen virtual ma- chine environment. The toolkit enables coordinated profil- ing of multiple VMs in a system to obtain the distribution of hardware events such as clock cycles and cache and TLB misses. We use our toolkit to analyze performance overheads in- curred by networking applications running in Xen VMs. We focus on networking applications since virtualizing network I/O devices is relatively expensive. Our experimental re- sults quantify Xen's performance overheads for network I/O device virtualization in uni- and multi-processor systems. Our results identify the main sources of this overhead which should be the focus of Xen optimization efforts. We also show how our profiling toolkit was used to uncover and re- solve performance bugs that we encountered in our experi- ments which caused unexpected application behavior.