Learning to Correct Cross-Modality Bounding Box Shifts in Drone-Based RGBT Object Detection

Drone-based RGBT object detection plays a crucial role in many around-the-clock applications such as search & rescue. However, real-world drone-viewed RGBT data suffers from the prominent position shift problem, i.e., the position of a tiny object differs greatly in different modalities. Existing methods simply rely on implicit learning to predict offsets and perform alignment. However, it is challenging to obtain reliable offset predictions due to the lack of supervision; moreover, detectors tend to be biased towards the aligned modality since the other modality is learned with confusing ground-truth bounding boxes, making the offset prediction non-trivial. In this paper, we propose to address the prominent position shift with cross-modality bounding box correction, which exploits cross-modal predictions to correct shifted ground-truth bounding boxes, providing explicit supervision and thereby enhancing the multimodal representation learning. Particularly, we leverage cross-modal knowledge by framing cross-modality bounding box correction into an unsupervised domain adaptation setting and design a progressive bounding box correction strategy, effectively improving correction results. With informative representations of both modalities established, we perform shifted window-based cascaded alignment to mine long-range dependencies between spatially unaligned features and effectively align multimodal features. Our method exceeds the state-of-the-art methods by 4.87 and 5.30 mAP50, respectively, on the RGBTDronePerson dataset and the shifted subset of DroneVehicle. Moreover, we devise a metric named aSim to evaluate the correction performance, and we boost the precision of the sensed modality ground truth by 30.04 points on RGBTDronePerson and 10.29 points on the shifted subset of DroneVehicle.