tokens: Analogously to generative adversarial networks (Goodfellow et al. 2014 ), the conditional autoencoder is adversarially trained to generate samples that hinder the critic network’s ability to do its job. The joint distribution p ( x , u , z ) can be expressed using Bayes’ rule asThis can be rewritten asif and only if u is statistically independent from z . If the joint distribution p ( u , z ) is not factorizable, the distributions q ( x , u , z ) and p ( x , u , z ) will be different. It follows from this that u and z are statistically independent if and only if the samples ( x , u , z ) drawn according to p ( x , u , z ) are indistinguishable from those sampled from q ( x , u , z ). How can we generate samples from these two distributions? If we consider our autoencoder to be an idealistic autoencoder capable of perfectly reconstructing its inputs, then the encoder and decoder can be viewed as respectively approximately parameterizing p ( z ∣ u , x ) and p ( x ∣ u , z ), which are both deterministic functions. We can thus draw samples from p ( x , u , z ) = p ( z ∣ u , x ) p ( u , x ) by first randomly sampling from the data set to obtain ( u , x ) and then using the encoder to obtain the associated z . We can similarly draw samples from q ( x , u , z ) = p ( x ∣ u , z ) p ( u ) p ( z ) through reusing our samples drawn from p ( x , u , z ). By scrambling the ( u , z ) pairs within a batch, we can effectively remove any joint information between u and z (Belghazi et al. 2018 ), which results in samples drawn from the marginal distribution ( u , z ) ∼ p ( u ) p ( z ). We can then use the decoder that approximates p ( x ∣ u , z ) to obtain approximate samples q ( x , u , z ) drawn from p ( x ∣ u , z ) p ( u ) p ( z ). As stated above, enforcing statistical independence is the same as finding a latent representation, z , such that samples drawn according to these two procedures are indistinguishable. This bears a strong similarity to the training objective of generative adversarial networks that attempt to train a generator such that generated samples are indistinguishable from samples drawn from a data set. As such, we can take inspiration from existing generative adversarial network architectures to solve our disentanglement objective. In generative adversarial networks (Goodfellow et al. 2014 ), a critic network is trained to distinguish between samples drawn from a data set and samples created by a generator network fed samples from a well-understood probability distribution. The generator and critic network are jointly optimized in a minimax game. That is, the critic attempts to maximally distinguish between the two data streams, and the generator attempts to minimize the critic network’s ability at doing so. The global optimum of this two-player game occurs when both the generator and critic network can no longer improve—when the two data streams are identical. For our disentanglement neural network architecture, we take heavy inspiration
ner_tags: O O O O O B-Citation I-Citation I-Citation I-Citation O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model O O O B-Model I-Model O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Citation I-Citation I-Citation I-Citation O O O O O O O O O O B-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula I-Formula O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model I-Model I-Model B-Citation I-Citation I-Citation I-Citation O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B-Model I-Model O O O O O