Resolution Enhancement with a Task-Assisted GAN to Guide Optical Nanoscopy Image Analysis and Acquisition

We developed a deep learning model that predicts super-resolved versions of diffraction-limited microscopy images. Our model, named Task-Assisted Generative Adversarial Network (TA-GAN), incorporates an auxiliary task (e.g. segmentation, localization) that is closely related to the characterization of the observed biological nanostructures. We evaluate how TA-GAN improves generative accuracy over unassisted methods using images acquired with two modalities: confocal (diffraction-limited) and STimulated Emission Depletion (STED, super-resolved) microscopy. The generated synthetic STED images show an accurate distribution of the F-actin nanostructures, and they replicate the nanoscale morphology of synaptic protein clusters. We expand the applicability of the TA-GAN to online live-cell imaging assistance. Incorporated directly into the acquisition pipeline of the microscope, the TA-GAN informs the user on the nanometric content of the field of view without requiring the acquisition of a super-resolved image. This information is used to automatically select time points of interest for the visualization of biological change, optimizing the acquisition sequence and reducing light exposure. TA-GAN partially replaces super-resolution imaging acquisitions, enables the creation of domain-adapted labeled datasets requiring minimal manual annotation, and assists microscopy users by taking online decisions regarding the choice of imaging modality and region of interest. Fig. 0 Graphical Abstract. The proposed model has two general use cases: TA-GAN, for paired datasets, and TA-CycleGAN, for unpaired datasets. Top-left: The TA-GAN uses a task adapted to each dataset for accurate resolution enhancement. Middle-left: The generated synthetic STED images are used to analyze the distribution of nanostructures that were not resolved in the original confocal image. Top-right: Domain adaptation using the TA-CycleGAN enables the generation of large annotated synthetic image datasets from a new domain. Middle-right: The task network generates the corresponding annotations on the synthetic dataset. Bottom: Both models can be used for microscopy acquisition guidance. The TA-GAN model, trained using a TA-CycleGAN generated dataset, can automatically identify regions and frames of interest from the low-resolution images. Automatic switching between low- and high-resolution imaging modalities is guided by the TA-GAN predictions. Scale bars: 1 μm.