Generative Imaging and Image Processing via Generative Encoder
<p style='text-indent:20px;'>This paper introduces a novel generative encoder (GE) framework for generative imaging and image processing tasks like image reconstruction, compression, denoising, inpainting, deblurring, and super-resolution. GE unifies the generative capacity of GANs and the stability of AEs in an optimization framework instead of stacking GANs and AEs into a single network or combining their loss functions as in existing literature. GE provides a novel approach to visualizing relationships between latent spaces and the data space. The GE framework is made up of a pre-training phase and a solving phase. In the former, a GAN with generator <inline-formula><tex-math id="M1">\begin{document}$ G $\end{document}</tex-math></inline-formula> capturing the data distribution of a given image set, and an AE network with encoder <inline-formula><tex-math id="M2">\begin{document}$ E $\end{document}</tex-math></inline-formula> that compresses images following the estimated distribution by <inline-formula><tex-math id="M3">\begin{document}$ G $\end{document}</tex-math></inline-formula> are trained separately, resulting in two latent representations of the data, denoted as the generative and encoding latent space respectively. In the solving phase, given noisy image <inline-formula><tex-math id="M4">\begin{document}$ x = \mathcal{P}(x^*) $\end{document}</tex-math></inline-formula>, where <inline-formula><tex-math id="M5">\begin{document}$ x^* $\end{document}</tex-math></inline-formula> is the target unknown image, <inline-formula><tex-math id="M6">\begin{document}$ \mathcal{P} $\end{document}</tex-math></inline-formula> is an operator adding an addictive, or multiplicative, or convolutional noise, or equivalently given such an image <inline-formula><tex-math id="M7">\begin{document}$ x $\end{document}</tex-math></inline-formula> in the compressed domain, i.e., given <inline-formula><tex-math id="M8">\begin{document}$ m = E(x) $\end{document}</tex-math></inline-formula>, the two latent spaces are unified via solving the optimization problem</p><p style='text-indent:20px;'><disp-formula> <label/> <tex-math id="FE1"> \begin{document}$ z^* = \underset{z}{\mathrm{argmin}} \|E(G(z))-m\|_2^2+\lambda\|z\|_2^2 $\end{document} </tex-math></disp-formula></p><p style='text-indent:20px;'>and the image <inline-formula><tex-math id="M9">\begin{document}$ x^* $\end{document}</tex-math></inline-formula> is recovered in a generative way via <inline-formula><tex-math id="M10">\begin{document}$ \hat{x}: = G(z^*)\approx x^* $\end{document}</tex-math></inline-formula>, where <inline-formula><tex-math id="M11">\begin{document}$ \lambda>0 $\end{document}</tex-math></inline-formula> is a hyperparameter. The unification of the two spaces allows improved performance against corresponding GAN and AE networks while visualizing interesting properties in each latent space.</p>