Class-Splitting Generative Adversarial Networks
Grinblat, G.L.1 , Uzal, L.C.1 and Granitto, P.M.1

arXiv:1709.07359v1 [stat.ML] 21 Sep 2017

1

CIFASIS, French Argentine International Center for Information and Systems
Sciences, UNR-CONICET, Argentina

Figure 1: Samples generated with our Splitting GAN method with supervised training on CIFAR-10
dataset. Each line has samples of one of the original classes. Each side has samples corresponding to
one of the two clusters generated for each class. We use ResNet-B architecture (see text for details).

Abstract

when no class information is available, i.e. in the
unsupervised setup. Our generated samples reach
Generative Adversarial Networks (GANs) produce state-of-the-art Inception scores for CIFAR-10 and
systematically better quality samples when class STL-10 datasets in both supervised and unsuperlabel information is provided., i.e. in the con- vised setup.
ditional GAN setup. This is still observed for
the recently proposed Wasserstein GAN formulaIntroduction
tion which stabilized adversarial training and al- 1
lows considering high capacity network architectures such as ResNet. In this work we show how The irruption of Generative Adversarial Nets
to boost conditional GAN by augmenting available (GAN) [5] produced a great leap for image data
class labels. The new classes come from cluster- generation. Samples were generated simply by aping in the representation space learned by the same plying a neural network transform to an input ranGAN model. The proposed strategy is also feasible dom vector sampled from a uniform distribution.
1

There is no need for any Markov chains or unrolled
approximate inference networks during either training or generation of samples [5]. GAN based generative models did not take long to reach impressive
image quality [13, 14, 17, 10] at least for some specific datasets.
However, current GAN models cannot produce
convincing samples when trained on datasets of images with high variability, even for relatively low
resolution images. On the other hand, it is observed
that sample quality improves when class information is taken into account in a conditional GAN
setup [11, 12]. These findings suggest that it is hard
to learn a multimodal distribution from a smooth
transform of a uniform (or Gaussian) distribution
and that providing categorical class information to
the generator alleviates this problem.
Our proposal is inspired in two observations.
First, as mentioned above, conditioning generation with categorical class labels with high level
of abstraction improves image quality. Second, as
early observed in [5, 13], the adversarial network
pair learns an useful hierarchy of representations
in an unsupervised setup. We propose to exploit
the same representation space learned by the GAN
model in order to generate new class labels with a
high level of abstraction. This is done by applying
a simple clustering method in this representation
space. By conditioning generation with this new
class labels the model is able to generate better
samples. This can be done either when prior class
information is available or not.
The main contributions of the present paper are1 :

Figure 2: Class-conditioned samples generated
with WGAN-GP method [7] over the ResNet-B architecture for CIFAR-10 dataset. Each row has
samples of a given class.

art Inception scores for CIFAR-10 and STL-10
datasets in both tasks.

2

Background

In the original GAN formulation [5], the generator is a neural network that transform noise input
z ∼ p(z) into fake samples and the discriminator
D is a neural network with a single scalar output
with a sigmoid activation. This output is interpreted as the model probability for its input being
a real image from the dataset distribution against
being a fake image sampled by generator G. The
discriminator D is trained using the standard binary classification formulation by minimizing the
binary cross-entropy between fake and real distributions. On the other hand, the generator G is
simultaneously trained to mislead the discriminator. This is accomplished in practice by updating
G parameters minimising the same loss but with
fake samples tagged with a ‘true’ label [5].
In other words, the discriminator is updated by

• We propose a method for increasing the number of class labels during conditional GAN
training based on clustering in the representation space learned by the same GAN model.
We base our implementation on the more stable Wasserstein GAN formulation [2, 7].
• We show an effectively way of adapting networks architecture to handle this increasing
number of classes.
• We show that this splitting GAN strategy improves samples quality both in the supervised
and unsupervised tasks, reaching state-of-the1 The source code will be available at https://github.
com/CIFASIS

2

Figure 3: Real CIFAR-10 samples corresponding to the 20 clusters found by our method. Each line is
divided in the same way as in Figure 1.
gradient descent over a negative log likelihood loss of the generated samples by adding more structure
to the GAN latent space and an auxiliary classiLD = − E [log(D(x̃))]− E [log(1−D(G(z)))], fier. This approach requires the dataset to include
x∼Pr
z∼p(z)
class labels, i.e. to work in a supervised setting.
(1)
The generator receives the noise vector z and also
while the generator minimizes
the selected label c so that the generated sample
LG = − E [log(D(G(z)))].
(2) is a function of both. Furthermore, the discrimiz∼p(z)
nator has, in addition to the usual objective, the
task of correctly classifying the real and generated
The main issue in the original GAN formula- samples (through an auxiliary classifier). The gention was the instability of the training process that erator is optimized not only to deceive the discrimmade very hard to improve architectures and to inator, but also to generate fake samples that minscale up to bigger images. In [13] a deep convo- imize the auxiliary classifier error, i.e. to produce
lutional architecture was proposed for both gen- well class-defined samples.
erator and discriminator which presents some degree of stability for adversarial training. This work
was the first one producing convincing image sam- 2.2 WGAN
ples for datasets with low variability (Bedrooms
and Faces) and relatively low resolution (64x64). In order to address the problem of instability in
However, standard GAN formulation fails to gen- GAN training, Arjovsky et al. in a series of works
erate globally consistent samples when trained on [1, 2] proposed a reformulation of the function to be
optimized. They argue that the original loss funcdatasets with high variability like ImageNet.
tion presents discontinuities and vanishing gradients with respect to generator parameters. Instead,
2.1 AC-GAN
they proposed a distance for distributions known
In order to tackle datasets with high variability, as Earth-Mover distance or Wasserstein-1, which
Odena et al. [12] proposed to improve the quality captures the cost of transporting mass in order to
3

transform one distribution into the other. From
this distance they derive the WGAN loss function
for the minimax objective
min max E [D(x)] −
G D∈D x∼Pr

E [D(G(z))]

(3)

z∼p(z)

where D (called critic in WGAN formulation) is
not anymore a binary classifier and is restricted to
be in the set D of 1-Lipschitz functions. Again, z is
a noise vector sampled from a simple distribution
(uniform or Gaussian distribution). The Lipschitzness of D was imposed by weight clipping in this
first version of WGAN.
The importance of Arjovsky’s contribution lies
on a gain in the robustness of the adversarial
training process and a reduction in the frequency
of the mode collapse phenomenon. Furthermore,
the proposed loss function correlates well with the
observed sample quality as opposed to the original GAN loss which gives little information about
training progress.

Figure 4: Samples generated with the ResNet-B architecture trained with Splitting GAN over CIFAR10 without class labels (unsupervised).

is the most recurrent benchmark for image generation in GAN literature. In the unsupervised setting
(without using any class information) they reach
2.2.1 WGAN-GP
the state-of-the-art, while in the supervised case
An improved version of WGAN was recently pro- (following the strategy of AC-GAN and without
posed by Gulrajani et al. [7]. They found that the tuning any hyperparameter nor architecture) they
2
weight clipping can cause convergence problems in reach the second place behind SGAN [8] . All this
some settings and propose to enforce the Lipschitz advantageous features made –to our knowledge–
constraint on the critic D by penalizing its gradi- WGAN the current standard formulation for adversarial training.
ent’s norm.
The penalty term is computed over a set of random points x̂ uniformly sampled from straight lines
Our Method:
Splitting
between real and fake sample points. Naming as Px̂ 3
the latter distribution, the new loss can be written
GAN
as
The main idea is to generate new artificial classes
L =
E [D(x̃)] − E [D(x)]
based on the representation learned by the last hidx∼Pr
x̃∼Pg
den layer of the critic after enough training itera

+λ E (k∇x̂ D(x̂)k2 − 1)2
(4) tions. This is done by applying k-means to each
x̂∼Px̂
class set in this representation space. We divide
each set in two clusters only when the class has
where the penalty coefficient is set to λ = 10.
This improved WGAN formulation exhibits high more samples than a certain threshold. After that,
robustness against changing model architecture. training is resumed replacing the old labels with
The authors tested six different network designs the new ones for the entire dataset.
With this procedure we need to make two mifor both G and D, which typically break standard
GAN training but show stable WGAN training for nor modifications to the model architecture before
all cases. Furthermore, WGAN formulation helps resuming learning:
achieving better quality samples. Quantitative re2 By simply duplicating the number of feature maps of
sults are reported by the authors in terms of the Gulrajani’s networks we found WGAN outperforms SGAN
Inception score [14] over CIFAR-10 dataset, which score. See Sec. 4
4

1. The auxiliary classifier needs to predict a different number of classes, so we extend the last
layer of this classifier adding a copy of the
weights of the parent class for each child class.

Method
Improved GAN (-L+HA) [14]
EGAN-Ent-VI [4]
DFM [15]
Splitting GAN ResNet-B (ours)
WGAN-GP ResNet-B
WGAN-GP ResNet-A [7]
Splitting GAN ResNet-A (ours)

Inception Score
6.86 ± 0.06
7.07 ± 0.10
7.72 ± 0.13
7.80 ± 0.08
7.81 ± 0.10
7.86 ± 0.07
7.90 ± 0.09

2. In the conditional WGAN-GP implementation
[6], the class labels are injected in each batch
normalization layer of the generative network
by setting a specific gain and bias parameters
(γ and b) for each class. We follow this strategy in our proposal and, for the class splitting,
we set the new pair (γ, b) for each child class as Table 1: Unsupervised Inception scores on CIFARγchild = γf ather +∆γ and bchild = bf ather +∆b, 10
with initialization ∆γ = 0 and ∆b = 0 when
the new classes are created. This formulation
Method
Inception Score
implies that child classes start both with the
father class params and they eventually beImproved GAN [14]
8.09 ± 0.07
come different.
AC-GAN [12]
8.25 ± 0.07
WGAN-GP ResNet-A [7]
8.42 ± 0.10
SGAN [8]
8.59 ± 0.12
4 Results
WGAN-GP ResNet-B
8.67 ± 0.14
Splitting GAN ResNet-A (ours)
8.73 ± 0.08
To demonstrate the effectiveness of our proposal,
Splitting GAN ResNet-B (ours)
8.87 ± 0.09
we conducted several experiments with CIFAR-10
[9], a dataset containing 50000 32x32 images corresponding to 10 different classes, and the unlabeled Table 2: Supervised Inception scores on CIFAR-10
set of STL-10, containing 100000 larger and more
diverse images [3]. We based our model on the improved WGAN algorithm proposed by Gulrajani et
al. [7, 6]. In all cases, during training, we sample ture with the original WGAN-GP algorithm. The
50000 images from the current model to select the results are detailed in Table 1 and Table 2.
best one so far based on the Inception score. FiAlso for comparison, samples obtained with the
nally, we sample another 50000 with the best model WGAN-GP supervised model (Figure 2) and obin order to calculate the reported score, following tained with the proposed method (Figure 1) are
[12].
shown. Figure 3 has real samples of CIFAR-10 corresponding to each of the 20 clusters found.

4.1

CIFAR-10

Figures 4 and 5 show generated images and clusters found in the unsupervised test.

With CIFAR-10, an unsupervised test was performed starting from all the examples considered
as a single class and dividing them into two clusMethod
Inception Score
ters every approximately 10 epochs. This was done
with the Gulrajani’s ResNet architecture without
Original Dataset [15]
26.08 ± 0.26
changes (named ResNet-A) and a modified version
DFM [15]
8.51 ± 0.13
(ResNet-B) doubling the number of maps in each
WGAN-GP
ResNet-A
9.05 ± 0.12
convolutional layer. A supervised test was also conSplitting
GAN
ResNet-A
(ours)
9.50
± 0.13
ducted with these two architectures, starting from
the original 10 classes of CIFAR and dividing them
into two at approximately 20 training epochs. For
comparison we also trained the ResNet-B architec- Table 3: Unsupervised Inception scores for STL-10
5

Figure 6: Images generated by the model trained
on STL-10.

4.2

STL-10

We treat STL-10 in the same way as [15]. That
is, we downsample each dimension by 2, resulting
in 48x48 RGB images. We tested our algorithm
with the ResNet-A architecture, with the minimum changes necessary for the model to generate
48x48 images. Table 3 shows the resulting Inception score. Figures 6 and 7 show the generated
images and the clusters found by the method.

5

Discussion

Several things can be observed from the results presented in the previous section. First, regarding the
obtained clusterization of the real samples (Figure
3 for the supervised case and Figure 5 for the unsupervised one), we can visually find rules that define
the vast majority of samples, for at least several
clusters. As an example, in the supervised case
(Figure 3) we can see in the left side of the fourth
row cats looking forward and in the left side of the
eighth row horse side views. Compare with cats
and horses in several positions corresponding to the
Figure 5: The 25 clusters found in the unsupervised clusters in the right side. In the unsupervised case
case (real CIFAR-10 samples). Each line has two (Figure 5) we can see a tendency to generate clusdifferent clusters.
ters for cars, planes or ships, but in general they
6

are much more mixed.
Regarding the generated samples in the supervised case (Figure 1 for our method and Figure 2 for
WGAN-GP), we can see that the class splits allows
the model to generate better samples. Not only for
the more uniform clusters such as the horse side
views or the cats looking forward, but for the whole
original class. Compare for example the fourth row
(cats) or the eighth row (horses) in Figure 1 with
those rows in Figure 2, corresponding to the same
model trained with WGAN-GP. Note that these
samples do not differ too much from those shown in
[7]. Even in classes where the clustering step does
not seem to have found an obvious separation rule,
such as cars (second row), a better sample quality
can be observed than in the original WGAN-GP.
In the unsupervised case with CIFAR-10 (Figure 4), although the Inception score is similar than
the one obtained by the state-of-the-art so far, the
samples generated seem to be of a higher quality.
Nevertheless, they do not reach the quality of the
generated images in a supervised setting. It is always advisable to start the division into clusters
from the predefined classes, if this information is
available.
In the case of STL-10 (Figure 6), there is a noticeable difference in the Inception score. The reason
of this may be that STL-10 is a much more diverse
dataset, so a division into a large number of classes
can be beneficial. It should be noted that in this
case the state-of-the-art is much further from the
actual dataset score than in the case of CIFAR-10.
The success of our Splitting GAN method suggests that reinjecting high level information from
critic to the generative model improves sampling
quality. This breaks the strictly adversarial training and allows some degree of information sharing
between both networks. We believe that this simFigure 7: The 15 clusters found by the model (real ple (but successful) strategy could inspire a new
STL-10 samples).
and better adversarial training formulation where
a small amount of high level information directly
flows from critic’s last layers to generator input.

6

Conclusions
Work

and

Future

In this work we showed that our Splitting GAN
method allows generating better images. This can
7

be seen in the results on CIFAR-10 and STL-10 for
which clearly better images were obtained. This
is supported by an Inception score well above the
previous state-of-the-art for both datasets.
A future direction of research to improve the current Splitting GAN version is oriented to understand how a given model architecture or dataset
determines the optimal number of clusters (or
classes). Also, clusterization could be enhanced
during adversarial training with the addition of an
extra loss term like in [16].
We are also currently working in a generalization of the Splitting GAN ideas following two paths:
First, making the high level information from the
critic’s representation flows continually to the generator, avoiding special purpose steps at predefined
times in the training process. Second, avoiding the
hard threshold clustering step and replacing it with
some sort of continuous representation capturing
the same amount of information.

A. Courville, and Y. Bengio. Generative
adversarial nets. In Advances in neural information processing systems, pages 2672–2680,
2014.
[6] I. Gulrajani. Improved Training of Wasserstein GANs. https://github.com/igul222/
improved_wgan_training, 2017. [Online; accessed 18-Sep-2017].
[7] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans.
arXiv preprint
arXiv:1704.00028, 2017.
[8] X. Huang, Y. Li, O. Poursaeed, J. Hopcroft,
and S. Belongie.
Stacked generative
adversarial networks.
arXiv preprint
arXiv:1612.04357, 2016.
[9] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.

[10] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang,
and S. P. Smolley. Least squares generative adversarial networks. arXiv preprint
The authors acknowledge grant support from ANArXiv:1611.04076, 2016.
PCyT PICT-2016-4374.
[11] M. Mirza and S. Osindero.
Conditional
generative adversarial nets. arXiv preprint
arXiv:1411.1784, 2014.
References

7

Acknowledgements

[1] M. Arjovsky and L. Bottou. Towards princi- [12] A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier
pled methods for training generative adversarGANs. In D. Precup and Y. W. Teh, editors,
ial networks. In International Conference on
Proceedings of the 34th International ConferLearning Representations 2017, 2017.
ence on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages
[2] M. Arjovsky, S. Chintala, and L. Bot2642–2651, International Convention Centre,
tou.
Wasserstein gan.
arXiv preprint
Sydney, Australia, 06–11 Aug 2017. PMLR.
arXiv:1701.07875, 2017.
[3] A. Coates, A. Ng, and H. Lee. An analysis of [13] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep
single-layer networks in unsupervised feature
convolutional generative adversarial networks.
learning. In Proceedings of the fourteenth inarXiv preprint arXiv:1511.06434, 2015.
ternational conference on artificial intelligence
and statistics, pages 215–223, 2011.
[14] T. Salimans, I. Goodfellow, W. Zaremba,
V. Cheung, A. Radford, X. Chen, and
[4] Z. Dai, A. Almahairi, P. Bachman, E. Hovy,
X. Chen. Improved techniques for training
and A. Courville.
Calibrating energygans. In D. D. Lee, M. Sugiyama, U. V.
based generative adversarial networks. arXiv
Luxburg, I. Guyon, and R. Garnett, editors,
preprint arXiv:1702.01691, 2017.
Advances in Neural Information Processing
[5] I. Goodfellow, J. Pouget-Abadie, M. Mirza,
Systems 29, pages 2234–2242. Curran AssoB. Xu,
D. Warde-Farley,
S. Ozair,
ciates, Inc., 2016.
8

[15] D. Warde-Farley and Y. Bengio. Improving
generative adversarial networks with denoising
feature matching. In International Conference
on Learning Representations 2017, 2017.
[16] Y. Wen, K. Zhang, Z. Li, and Y. Qiao. A
discriminative feature learning approach for
deep face recognition. In European Conference
on Computer Vision, pages 499–515. Springer,
2016.
[17] J. Zhao, M. Mathieu, and Y. LeCun. Energybased generative adversarial network. arXiv
preprint arXiv:1609.03126, 2016.

9

