Time-of-flight fluorescence depth mapping using a spatiotemporal deep learning model

Significance Fluorescence-guided surgery (FGS) utilizes molecular contrast agents to highlight critical structures or pathological tissues in real time. The premise of FGS is to enable precise surgical decision-making through accurate visualization and quantitative assessment of fluorophore distribution. However, strong effects of diffusion and absorption of fluorescent light in tissue confound fluorescence images, preventing accurate quantitative assessment of the concentration and distribution of fluorescent markers. These optical artifacts may lead to misinterpretation of tissue boundaries and compromised surgical precision, thereby diminishing the capabilities of FGS. Resolving topological depth maps of fluorophore distribution at the millimeter scale is an important first step in performing quantitative sub-surface fluorescence imaging.

Aim In this study, we present a spatiotemporal deep learning architecture that utilizes picosecond single-photon avalanche diode (SPAD) sensor images to rapidly recover the depth topology of a fluorophore distribution embedded in diffuse media. The network is designed to work with wide-field, epi-illumination geometry and millimeter spatial resolution.

Approach A ConvLSTM-UNet deep learning network was developed for picosecond time-resolved image analysis. This network was trained on 5000 spatiotemporal maps simulated by the optical Monte Carlo method and convolved with the instrument response function (IRF) of the imaging system. The experimental setup utilized a SwissSPAD2 sensor synchronized with a 635 nm picosecond laser diode. Using only 10 selected temporal gates as input, the network could recover depth maps. Reconstruction accuracy was evaluated using mean error metrics across various depths and background concentrations of a fluorophore with a simulated decay time of 100 ps.

Results A total of 75 different test fluorescence video data were evaluated. This set encompassed 15 unique inclusion shapes at five different depths. The network successfully reconstructed fluorescence topography up to 15 mm with a mean absolute error of less than 0.6 mm and mean depth variances below 0.5 mm. The inference time was ∼30 ms.

Conclusions Integrating temporal and spatial deep learning networks enabled depth mapping from time-resolved fluorescence data. Utilizing real IRF proved the applicability of SPAD sensors for sub-surface fluorescence mapping.