Hardware-in-the-loop training for lensless imaging with a programmable mask

Lensless imaging can drastically relax traditional camera constraints by replacing lenses with optical masks, enabling lighter, cheaper, and thinner systems. However, unlike lenses, there is a lack of clear criteria for optical mask design. Most approaches are heuristic: either selecting a random mask or designing one with desired spectral and/or directional filter properties. Recent work jointly optimizes a phase mask and a task-specific neural network, but in simulation. We propose and demonstrate hardware-in-the-loop (HITL) training for jointly optimizing the mask and reconstruction parameters of a lensless imaging system, using the physical system itself in forward passes and a simulated model for determining updates during backpropagation. As the physical system uses a programmable mask, system updates can be done during training. Results show significant improvements in image quality metrics (2.14 dB in PSNR, 21.4% relative improvement in a perceptual metric) by jointly learning mask and reconstruction parameters. A low-cost prototype (less than 100 USD) is used, with open-source training and measurement code available on GitHub: https://github.com/LCAV/LenslessPiCam