One Index to Run Them All: Fast and Accurate Inference with Iso-Indexed Codebooked Ensembles

This paper introduces I2-CE (Iso-Indexed Codebooked Ensembles) as a means for fast, efficient, and robust inference at the edge. Ensemble learning improves generalization by combining the outputs of multiple learners, but its computational and memory requirements grow linearly with the number of learners. We address this challenge by enforcing a single, shared index map across all learners, distinguishing their behaviors solely through compact, learner-specific codebooks. Codebook-based methodologies restrict the admissible values of weights of trained models to a discrete set, which can be stored in lookup tables and indexed by low-bitwidth pointers. By storing only one set of indices, shared between learners, we dramatically reduce memory requirements since the memory overhead for additional learners is limited to that of a (small) codebook. Moreover, such a structure produces a set of highly regular models that greatly benefit from parallel execution. Leveraging this, our framework exploits datalevel parallelism within and across learners, employing Single Instruction Multiple Data (SIMD) instructions for run-time optimization. We explore the performance of iso-indexed codebooks on ensembles of Convolutional Neural Networks (CNNs), evaluating their accuracy and run-time across various configurations of codebook sizes, SIMD sizes, and number of learners. Compared to a floating point, non-codebook, non-ensemble implementation, we achieve speedups of up to 3.27x, while limiting the accuracy drop due to codebooking to 2.5%, for a 2 learners configuration employing 4-values codebooks.