Adversarial Attacks Beyond the Image Space

arXiv:1711.07183v2 [] 21 Nov 2017

Xiaohui Zeng1 , Chenxi Liu2 , Yu-Siang Wang3 , Weichao Qiu2 , Lingxi Xie2
Yu-Wing Tai4 , Chi Keung Tang1 , Alan L. Yuille2
1
HKUST 2 Johns Hopkins U 3 National Taiwan U 4 Tencent Youtu

Abstract

Object
Classification

Generating adversarial examples is an intriguing problem and an important way of understanding the working
mechanism of deep neural networks. Recently, it has attracted a lot of attention in the computer vision community.
Most existing approaches generated perturbations in image
space, i.e., each pixel can be modified independently. However, it remains unclear whether these adversarial examples
are authentic, in the sense that they correspond to actual
changes in physical properties.
This paper aims at exploring this topic in the contexts
of object classification and visual question answering. The
baselines are set to be several state-of-the-art deep neural
networks which receive 2D input images. We augment these
networks with a differentiable 3D rendering layer in front,
so that a 3D scene (in physical space) is rendered into a 2D
image (in image space), and then mapped to a prediction
(in output space). There are two (direct or indirect) ways
of attacking the physical parameters. The former backpropagates the gradients of error signals from output space
to physical space directly, while the latter first constructs
an adversary in image space, and then attempts to find the
best solution in physical space that is rendered into this
image. An important finding is that attacking physical space
is much more difficult, as the direct method, compared with
that used in image space, produces a much lower success
rate and requires heavier perturbations to be added. On
the other hand, the indirect method does not work out,
suggesting that adversaries generated in image space are
inauthentic. By interpreting them in physical space, most
of these adversaries can be filtered out, showing promise in
defending adversaries.

R: bench 

Visual Question Answering
Q: There is a sphere that is
in front of the large yellow
object, what color is it?



A: red

𝑝 = 3.7 × 10−3

R: chair



Q: What size is the other
red block that is the same
material as the blue cube?

A: large

𝑝 = 0.8 × 10−3

A: cyan



𝑝 = 2.4 × 10−3

A: 0

conf = 89.9%

conf = 54.0%

conf = 64.3%

𝑝 = 4.7 × 10−3

𝑝 = 3.0 × 10−3

𝑝 = 2.7 × 10−3

R: table



conf = 89.9%

A: yes



conf = 45.7%

A: 0
conf = 52.8%

Figure 1. Adversarial examples for object classification and visual
question answering. The first row is the original image. The
middle group shows the perturbations (magnified by a factor of
5 and shifted by 128) and perturbed images by attacking image
space, and the bottom group by attacking physical space. p and
conf are the perceptibility (see Section 3.3.1) and the confidence
score on the predicted class, respectively. Attacking physical
space is more difficult, as we always observe a larger perceptibility
and a lower confidence score.

1. Introduction
Recent years have witnessed a rapid development in the
area of deep learning, in which deep neural networks have
been applied to a wide range of computer vision tasks, such
as image classification [16][12], object detection [9][34],
semantic segmentation [37][6], visual question answer1

ing [2][14], etc. The basic idea is to design a hierarchical
structure to learn visual patterns from labeled data. With
the availability of powerful computational resources and
large-scale image datasets [7], researchers have designed
more and more complicated models and achieved a boost
in visual recognition performance.
Despite the great success of deep learning, there still
lacks an effective method to understand the working mechanism of deep neural networks. An interesting effort is to
generate so-called adversarial perturbations. They are visually imperceptible noise [11] which, after being added to
an input image, changes the prediction results completely,
sometimes ridiculously. These examples can be constructed
in a wide range of vision problems, including image classification [28], object detection and semantic segmentation [45]. Researchers believed that the existence of adversaries implies unknown properties in feature space [43].
This work is motivated by the fact that conventional 2D
adversaries were often generated by modifying each image
pixel individually. Thus, while being strong in attacks,
it remains unclear if they can be generated by perturbing
physical properties in the 3D world. We notice that previous
work found adversarial examples “in the physical world”
by taking photos on the printed perturbed images [17]. But
our work is different and more essential as we only allow
to modify some basic physical parameters such as surface
normals. For this respect, we follow [19] to implement
3D rendering as a differentiable layer, and plug it into the
state-of-the-art neural networks for object classification and
visual question answering. In this way, we build a mapping
function from physical space (a set of physical parameters,
including surface normals, illumination and material), via
image space (a rendered 2D image), to output space (the
object class or the answer to a question).
We aim at answering two questions. (i) Is it possible to
directly generate perturbations in physical space (i.e., modifying basic physical parameters)? (ii) Given an adversary in
image space, is it possible to find an approximate solution
in physical space, so that the re-rendered 2D image preserves the attacking ability? Based on our framework, these
questions correspond to two different ways of generating
perturbations in physical space. The first one, named the
direct method, computes the difference between the current
output and the desired output, back-propagates the gradients to the physics layer directly and makes modifications.
The second one, the indirect method, first constructs an
adversary in image space, and then attempts to find the best
solution in physical space that is rendered into it. Both
methods are implemented by the iterative version of the
Fast Gradient Sign Method (FGSM) [11]. We constrain the
change in image intensities to guarantee the perturbations
to be visually imperceptible. Experiments are performed on
two datasets, i.e., 3D ShapeNet [5] for object classification

and CLEVR [14] for visual question answering.
Our major finding is that attacking physical space is
much more difficult than attacking image space. Although
it is possible to find adversaries using the direct manner
(YES to Question (i)), the success rate is lower and the
perceptibility of perturbations becomes much larger than
required in image space. This is expected, as the rendering
process couples changes in pixel values, i.e., modifying one
physical parameter (e.g., illumination) may cause many pixels to be changed at the same time. This also explains why
we found it almost impossible to generate adversaries using
the indirect manner (conditional NO to Question (ii); it
is possible that currently available optimization algorithms
such as FGSM are not strong enough). An implication of
this research is an effective approach to defend adversaries
generated in image space – finding an approximate solution
in physical space and re-rendering will make them fail.
The remainder of this paper is organized as follows.
Section 2 briefly introduces related work. The approach
of generating adversarial perturbations in physical space is
described in Section 3. After experiments are shown in
Section 4, we conclude our work in Section 5.

2. Related Work
Deep learning is the state-of-the-art machine learning
technique to learn visual representations from labeled data.
The basic methodology is to stack differentiable units in
a hierarchical manner [16]. It is believed that a network
with a sufficient number of layers and neurons can capture
complicated distributions in feature space. With large-scale
image datasets such as ImageNet [7], powerful computational resources such as GPU’s, and the assistance of efficient strategies [27][39][13], it is possible to train very deep
networks in a reasonable period of time. Recent years, deep
neural networks have been widely applied to computer vision problems, including image classification [38][42][12],
object detection [9][34], semantic segmentation [37][6],
visual question answering [2][8][15], etc.
Despite the success of deep learning, it remains a challenging task to explain what is learned by these complicated models. One of the most interesting efforts towards
this goal is to generated adversaries. In terms of adversaries [11], we are talking about small noise that is (i)
imperceptible to humans, and (ii) able to cause deep neural
networks make wrong predictions after being added to the
input image. It was shown that such perturbations can
be easily generated in a white-box environment, i.e., both
the network structure and pre-trained weights are known to
the attacker. Early studies were mainly focused on image
classification [28][26]. But soon, researchers were able to
attack deep networks for detection and segmentation [45],
and also visual question answering [46]. Efforts were also
made in finding universal perturbations which can transfer
2

across images [25], as well as adversarial examples in the
physical world which were produced by taking photos on
the printed perturbed images [17].
Attacking a known network (a.k.a, a white box) started
with setting a goal of prediction. There were generally two
types of goals. The first one (a non-targeted attack) aimed at
reducing the probability of the true class [28], and the second one (a targeted attack) defined a specific class that the
network should predict [20]. After that, the error between
the current and the target predictions was computed, and
gradients back-propagated to the image layer. This idea was
developed into a set of algorithms, including the Steepest
Gradient Descent Method (SGDM) [26] and the Fast Gradient Sign Method (FGSM) [11]. The difference lies in that
SGDM computed accurate gradients, while FGSM merely
kept the sign in every dimension of these gradients. The
latter one, while being less powerful in direct attacks, often
enjoys stronger transfer ability. The iterative version of
these two algorithms were also studied [17]. In comparison,
attacking an unknown network (a.k.a., a black box) is much
more challenging [20], and an effective way is to sum up
perturbations from a set of white-box attacks [45].
In opposite, there exist efforts in protecting deep networks from adversarial attacks. Defensive distillation [31]
proposed to defend the network by distilling robust visual
knowledge, but a stronger attacking method was designed
to beat this defense [4]. It was shown that training deep
networks in a large-scale dataset increases the robustness
against adversarial attacks [18], and a more essential solution was to add adversarial examples to the training
data [44]. Researchers also developed an algorithm to
detect whether an image was attacked by adversarial perturbations [23]. The battle between attackers and defenders
continues, and these ideas are closely related to the bloom
of Generative Adversarial Networks [10][33].

questions. First, is it possible to perturb the 3D physical
parameters so as to attack 2D deep networks? Second, are
the conventional 2D adversaries interpretable by perturbing
some physical parameters of a 3D scene? We answer these
two questions based on one single framework (Section 3.2),
which plugs a rendering module to various deep neural
networks, and thus builds a mapping function from the
physics layer, via the image layer, to the output layer.
Our goal is to generate adversaries on the physics layer.
There are two different ways, i.e., either directly backpropagating errors from the output layer to the physics
layer, or first constructing an adversary in the image layer,
and then finding a set of physical parameters that are rendered into it. We name them as direct and indirect ways
of attacking the physics layer. In experiments, the direct
method works reasonably well, but the indirect method fails
completely. Quantitative results are shown in Section 4.

3.2. From Physical Parameters to Prediction
As the basis of this work, we build an end-to-end framework which receives the physical parameters of a 3D scene,
renders them into a 2D image, and outputs prediction, e.g.,
the class of an object, or the answer to a visual question.
Note that our research involves 3D to 2D rendering as
part of the pipeline, which stands out from previous work
which either worked on rendered 2D images [40][15], or
directly processed 3D data without rendering them into 2D
images [32][36].
We denote the physical space, image space and output
space by X , Y and Z, respectively. Given a 3D scene
X ∈ X , the first step is to render it into a 2D image
Y ∈ Y. For this purpose, we consider three sets of physical
parameters, i.e., surface normals N, illumination L, and
material m. By giving these parameters, we assume that the
camera geometries, e.g., position, rotation, field-of-view,
etc., are known beforehand and will remain unchanged in
each case. The rendering module (Section 3.2.1) is denoted
by Y = r(N, L, m). Then, we make use of Y in two
vision tasks, i.e., object classification and visual question
answering. An individual deep neural network receives Y
as its input, and outputs the prediction Z ∈ Z. The networks for classification
question answering
are
 and visual



C
V
C
C
V
V
denoted by Z = f Y; θ
and Z = f Y, q; θ ,

3. Approach
3.1. Motivation: Inauthenticity of Adversaries
Despite the fast development in generating adversaries
to attack deep neural networks, we note that almost all the
existing algorithms assumed that each pixel of the image
can be modified independently. Although this strategy
was successful in finding strong adversaries, the perturbed
images may be inauthentic, i.e., they may not correspond
to any 3D scenes in the real world. Motivated by this, we
study the possibility of constructing adversaries by directly
modifying the physical parameters, i.e., surface normals and
material of an object, and illumination of a 3D scene. Note
that our goal is more essential than the previous work [17],
which generated adversaries “in the physical world” by
taking photos of the printed perturbed images.
To be concrete, this work is aimed at answering two

respectively. Here, q is the question, ZC and ZV are
the output vectors, and θ C and θ V are the corresponding
parameters, respectively.
3.2.1

3D Object Rendering

We make use of [19], a differentiable algorithm for 3D
object rendering. Note that some other algorithms [3][22]
provide better rendering qualities but cannot be used in this
paper because they are non-differentiable. Differentiability
3

is indispensable, as it enables us to back-propagate errors
from the image layer to the physics layer.
Three sets of parameters are considered. (i) Surface normals are represented by a 2-channel image which has the
same size as the desired output image Y, sized WN × HN ,
where each pixel is encoded by the azimuth and polar
angles of the normal vector at this position. (ii) Illumination is defined by an HDR environment map of dimension
WL × HL , with each pixel storing the intensity of the light
coming from this direction (a spherical coordinate system
is used). (iii) Material impacts image rendering with a set
of bidirectional reflectance distribution functions (BRDFs)
which describe the point-wise light reflection for both diffuse and specular surfaces [29]. The material parameters
used in this paper come from the directional statistics BRDF
model [30], which represents a BRDF as a combination
of Dm distributions with Pm parameters in each. Mathematically, we have N ∈ RWN ×HN ×2 , L ∈ RWL ×HL and
m ∈ RDm ×Pm .
The rendering algorithm [19] is based on some reasonable assumptions, e.g., translation-invariant natural illumination (incoming light depends only on the direction), and
there is no emission and omit complex light interactions
like inter-reflections and subsurface scattering. Then the
intensity of each pixel of Y can be computed by integrating
the reflected light intensity over all incoming light directions [21]. The integral is substituted by a discrete sum over
a set of incoming light directions for numerical computations. In practice, the rendering process is implemented as
a network layer, which is differentiable to input parameters
N, L and m. Please refer to [19] for mathematical equations and technical details.

We make use of the algorithm described in [15]. This
algorithm consists of two components: a program generator
and an execution engine. The goal of the program generator
is to convert the question q written in natural language to
a tree structure (i.e., a program) that describes the order
and types of a series of actions to carry out. Specifically, a
sequence-to-sequence model [41] is used to translate words
in the question to the prefix traversal of the abstract syntax
tree. The execution engine assembles a neural module network [1] according to the predicted program. Each module
is a small convolutional neural network that corresponds to
one predicted action in the tree. The convolutional image
features may be queried at various places in this assembled
network.
In the testing stage, given a question q, the network
structure and its parameters θ V (q) are fixed, and the prediction process is very similar to that in object classification.
Thus, we unify object classification and visual question
answering into the same formulation:

3.2.2

∆Y = r(N + ∆N, L + ∆L, m + ∆m) − r(N, L, m).
(2)
Perceptibility is computed on the perturbations of both
the rendered image and the physical parameters. Following [43][26], the image perceptibility is defined as:
!1/2
WN X
HN
X
1
.
2
k∆yw,h k2
, (3)
p = p(∆Y) =
WN × HN w=1

Z = f (Y; θ) = f ◦ r(N, L, m; θ),
C

where θ denotes network parameters, i.e., θ or θ (q).

3.3. Attacking Physical Space
3.3.1

Perceptibility

The goal of an adversarial attack is to produce a visually
imperceptible perturbation, so that the network makes incorrect predictions after it is added to the original image.
Let the physical parameters be N, L, m, and the rendered
image be Y = r(N, L, m). Denote the perturbations added
to these parameters by ∆N, ∆L and ∆m, respectively. The
perturbation added to the rendered image is:

Object Classification

Based on the rendered 2D images, object classification is
straightforward. Two popular networks (AlexNet [16] and
ResNet [12]) are investigated. We start with two models
pre-trained on the ILSVRC2012 dataset [35], and fine-tune
them using the rendered images in the ShapeNet [5] dataset.
This is necessary, as the rendered images contain quite
different visual patterns from natural images.
C
In the testing stage, the network parameters

 θ is fixed,
and we predict the class by Z = f Y; θ C , where Z ∈

h=1

where yw,h is a 3-dimensional vector representing the RGB
intensities (normalized in [0, 1]) of a pixel. Similarly, we
can also define the perceptibility values for object parameters, i.e., p(∆N), p(∆L) and p(∆m). For example,

K

[0, 1] (K is the number of classes) is the probability
distribution over all classes.
3.2.3

(1)
V

Visual Question Answering

p(∆N) =

WN X
HN
X
1
2
k∆nw,h k2
WN × HN w=1

!1/2
.

(4)

h=1

The visual question answering problem [14] considered in
this paper is essentially a classification task. Given an input
image Y and a question q, the goal is to choose the correct
answer from a pre-defined set (32 choices).

In experiments, p(∆Y) is the major criterion of visual
imperceptibility, but we also guarantee that p(∆N), p(∆L)
and p(∆m) are very small.
4

3.3.2

Setting a Goal of Attacking

the re-rendered image, and any perturbations exceeding a
fixed threshold U is truncated. In practice, U is set to be 18,
the reasonable value that is imperceptible to human eyes.
Truncations cause the inconsistency between the physical
parameters and the rendered image and risk failures in
attacking. To avoid frequent truncations, we set the learning
rate η to be small, which consequently increases the number
of iterations needed to attack the network.

Attacking the physical parameters starts with setting a goal,
which is what we hope the network to predict. In practice, this is done by defining a loss function L(Z), which
determines how far the current output is from the desired
status. In this work, the goal is set in two ways, i.e., either
targeted or non-targeted. A targeted attack specifies a class
c as which the image should be classified, and thus defines a
target output vector 1c using the one-hot encoding scheme.
The Manhattan distance between Z and 1c forms the loss
function:
.
LT (Z) = L(Z; c) = kZ − 1c k1 .

3.3.4

In opposite, the indirect attack first finds a perturbation ∆Y
in image space, then computes perturbations in physical
space, i.e., ∆N, ∆L and ∆m. ∆Y is generated in the same
way of perturbing the physical parameters.
The next step is to find physical perturbations, i.e., ∆N0 ,
∆L0 and ∆m0 , so that the newly rendered image

(5)

On the other hand, a non-targeted attack specified a class c0
as which the image should not be classified, and the goal is
to minimize the c0 -th dimension of the output Z:
.
LN (Z) = L(Z; c0 ) = Zc0 .

Indirect Attacks

Y0 = r(N + ∆N0 , L + ∆L0 , m + ∆m0 )

(6)

is close to Y + ∆Y. Mathematically, we minimize the
following loss function:

Throughout this paper, we make use of the non-target attack, and refer to the loss term as L(Z) = LN (Z).

L(Y0 ) = kY + ∆Y − Y0 k1 .
3.3.3

Direct Attacks

(9)

Similarly, Eqn (9) is expanded in physical space by substituting Y0 with Eqn (8), and optimization can be performed
on physical parameters either jointly or individually.
Note that the indirect way is indeed pursuing interpreting ∆Y in physical space. However, as we will show
in experiments, this way does not work out in any cases.
This suggests that adversaries generated in image space,
despite being strong in attacks, are often inauthentic, and
their approximations in physical space either do not exist,
or cannot be found by this simple optimization.

There are two ways of attacking physical space. The direct attack works by expanding the loss function L(Z),
i.e., L(Z) = L ◦ f ◦ r(N, L, m; θ), and minimizing this
function with respect to the physical parameters N, L and
m. Note that these parameters can also be optimized either jointly or individually. Without loss of generality, we
optimize N individually in the following example.
The optimization starts with an initial (unperturbed) state
.
N0 = N. A total of Tmax iterations are performed. In the
t-th round, we compute the gradient vectors with respect
to Nt−1 , i.e., ∇Nt−1 L ◦ f ◦ r(Nt−1 , L, m; θ), and update
Nt−1 along this direction. We follow the Fast Gradient
Sign Method (FGSM) [11] to only preserve the sign in each
dimension of the gradient vector, as this algorithm does
not need normalization, in which large gradient values may
swallow small ones. We denote the perturbation in the t-th
iteration by
.
∆Nt = ∇Nt−1 L ◦ f ◦ r(Nt−1 , L, m; θ),

(8)

4. Experiments
4.1. 3D Object Classification
4.1.1

Settings

We investigate 3D object recognition in the ShapeNetCorev2 dataset [5], which contains 51 rigid object categories,
each with various 3D models. We randomly sample 125 3D
models from each class, and generate 4 fixed viewpoints for
each object, so that each category has 500 training images.
Similarly, another randomly chosen 50 × 4 objects for each
class are used for testing.
We start with two popular deep neural networks, i.e.,
a 8-layer AlexNet [16] and a 34-layer deep residual network [12]. It is easy to generate our algorithm to other
network structures. Both networks are pre-trained on the
ILSVRC2012 dataset [35], and fine-tuned in our training set
through 40 epochs. Each mini-batch contains 256 samples,
and the learning rate is 0.001 for AlexNet and 0.005 for

(7)

and update Nt−1 by Nt = Nt−1 + η · ∆Nt−1 , where η is
the learning rate. This iterative process is terminated if the
goal of attacking is reached. The accumulated
Pperturbation
T
over all T iterations is denoted by ∆N = η · t=1 ∆Nt for
later reference.
Throughout the attacking process, in order to guarantee
imperceptibility, we constrain the RGB intensity changes
on the image layer. In each iteration, after a new set of
physical perturbations are generated, we check all pixels on
5

Attacking
Perturbations
FGSM on AlexNet
FGSM on ResNet-34

Image
Succ.
p
100.00
7.0
100.00
6.0

Surface N.
Succ.
p
89.02
18.7
87.84
11.0

Illumination
Succ.
p
37.65
43.9
18.04
20.4

Material
Succ.
p
21.57
19.9
3.92
30.3

Combined
Succ.
p
92.55
16.7
92.55
11.0

Table 1. Effect of white-box adversarial attacks to image space or individual elements in physical space. By combined, we allow three
sets of physical parameters to be perturbed jointly. Succ. denotes the success rate of attacks (%, higher is better), and p is the perceptibility
value (unit: 10−3 , lower is better) defined in Section 3.3.1.

GT: car
A: car
R: car




Attacking AlexNet (L) & ResNet (R)

𝑝 = 7.9 × 10−3

A: pillow



conf = 93.5%

𝑝 = 6.7 × 10−3

R: helmet 

conf = 60.9%

GT: tower

GT: m-bike

GT: train

A: tower 

A: m-bike

A: train

R: tower 

R: m-bike

R: train

Attacking AlexNet (L) & ResNet (R)

Attacking AlexNet (L) & ResNet (R)

𝑝 = 10.9 × 10−3

𝑝 = 14.6 × 10−3

𝑝 = 16.4 × 10−3

A: helmet 

R: bookshelf

A: train

conf = 86.9%

conf = 76.8%



conf = 93.9%

𝑝 = 16.7 × 10−3

R: earphone

conf = 74.6%




Attacking AlexNet (L) & ResNet (R)

𝑝 = 9.7 × 10−3

A: vessel



conf = 95.0%

𝑝 = 4.4 × 10−3

R: vessel



conf = 76.6%

Figure 2. Examples of adversaries generated in the 3D object classification task. In each group, the top row shows the original testing
image, which is correctly predicted by both AlexNet (A) and ResNet (R). The following two rows display the perturbations and the
attacked image, respectively. All perturbations are magnified by a factor of 5 and shifted by 128. p is the perceptibility value defined in
Section 3.3.1, and conf is the confidence score of the prediction.

ResNet-34. These networks work quite well on the original testing set (no perturbations are added), i.e., AlexNet
achieves 73.59% top-1 classification accuracy, and ResNet34 reports 79.35%. These numbers are comparable to the
single-view baseline accuracy reported in [40].
For each class, from the correctly classified testing samples, we choose 5 images with the highest classification
probabilities as the targets to generate adversaries. This
target set seems small, but we emphasize that attacking
in physical space is very time-consuming, as we need
to repeatedly re-render the perturbed physical parameters
throughout a total of 120 iterations. Using a Titan-X Pascal
GPU, attacking one image takes 12 minutes on average1 .
4.1.2

normals, illumination and surface) individually or to attack
them jointly. For comparison, we also provide results of
attacking image space directly. For each individual set of
parameters, we set a non-target goal (see Section 3.3.2), use
the SGD optimizer (momentum is 0.9 and weight-decay
is 10−4 ), and set the maximal number of iterations to be
120. Choosing the learning rate is a little bit tricky. If
the learning rate is too large, the truncation in image space
will happen frequently (see Section 3.3.3), and we cannot
guarantee an accurate correspondence between physical and
image spaces. On the other hand, if the learning rate is
too small, the accumulated perturbations are not enough
to change the prediction. We choose the best learning rate
from {0.0008, 0.001, 0.002, 0.005, 0.008, 0.01} for each set
of physical parameters. This is not cheating, as adversarial
attacks assume that we know the labels of all the target
images [11].

Quantitative Results

We apply the iterative version of Fast Gradient Sign Method
(FGSM), and try to attack each physical parameters (surface

Results of direct attacks are summarized in Table 1 (the
indirect method does not work out, see below). First, we
demonstrate that adversaries widely exist in both image
space and physical space. In image space, as researchers

1

This time cost depends on the parameters allowed to be perturbed.
Perturbing surface normals, illumination and material takes around 8, 4
and 10 minutes on each image, respectively, and attacking all of them
jointly takes around 12 minutes.

6

Average Loss Function Value

Loss Curves on AlexNet

adversarial attacks. This property also holds in the scenario
of visual question answering. As a side note, although
perturbing surface normals allows to change each pixel
independently, the rendered RGB intensity is also highly
impacted by other two parameters. This is the reason why
allowing all physical parameters to be jointly optimized
produces the highest success rate. We take this option in
the remaining diagnostic experiments.

Image
Normals
Illumination
Material
Joint

1.0

0.8

0.6

0.4

We plot the curves of the average loss over 255 target
images throughout the iterative process of generating adversaries in Figure 3. The loss values in physical attacks
especially for illumination and material drop much slower
than those in image attacks. Even with a larger number of
iterations, there are still some objects that are not attacked
successfully, especially in the scenarios of perturbing the
illumination and material parameters.

0.2

0.0
0

20

40

60

80

100

120

Average Loss Function Value

Number of Iterations

Loss Curves on ResNet−34
1.0

Finally, as the average perceptibility in physical space
is much larger than that in image space, we conjecture
that adversaries generated in image space are inauthentic,
i.e., using the current optimization approach (FGSM), it
is almost impossible to find physical parameters that are
approximately rendered into them. This is verified using the
indirect method described in Section 3.3.4. In both AlexNet
and ResNet-34, it fails on all 255 target images, regardless
of the learning rate and optimizer (SGD or Adam).

0.8

0.6

Image
Normals
Illumination
Material
Joint

0.4

0.2

0.0
0

20

40

60

80

100

120

Number of Iterations

4.2. Visual Question Answering

Figure 3. Curves of the average loss function value throughout the
iterations of FGSM. The starting point of each curve is the average
prediction confidence on the original images.

4.2.1

We extend our experiments to a more challenging vision
task – visual question answering. Experiments are performed on the recently released CLEVR dataset [14]. This
is an engine that can generate an arbitrary number of 3D
scenes with meta-information (object configuration). Each
scene is also equipped with multiple generated questions,
e.g., asking for the number of specified objects in the scene,
or if the object has a specified property.

have explored before [43][26], it is easy to confuse the
network with small perturbations – in our case, the success
rate is always 100% and the perceptibility does not exceed
10−2 . In physical space, however, generating adversaries
becomes much more difficult – the success rate becomes
lower and large perceptibility values are often observed on
the successful cases. Typical adversarial examples generated in physical space are shown in Figure 2.
4.1.3

Settings

The baseline algorithm is named Inferring and Executing
Programs (IEP) [15]. It applied an LSTM to parse each
question into a tree structure, which is then converted into a
neural module network. We use the released model without
training it by ourselves. We sample a subset of 200 images
from the original testing set, and equip each of them with 3
visual questions. The original model reports an answering
(classification) accuracy of 96.17% on these 200×3 images.

Diagnosis

Among three sets of physical parameters, attacking surface
normals is more effective than the other two. This is as
expected, as using local perturbations is often easier in
attacking deep neural networks [11]. The surface normal
matrix shares the same dimensionality with the image lattice, and changing an element in the matrix only impacts
on a single pixel in the rendered image. In comparison,
illumination and material are both global properties of the
3D scene or the object, so tuning each parameter will cause
a number of pixels to be modified, hence less effective in

We randomly pick up 100 testing images, on which all
3 questions are correctly answered, as the target images.
The settings for generating adversarial perturbations are the
same as in the classification experiments, i.e., the iterative
FGSM are used, and three sets of physical parameters are
attacked either individually or jointly.
7

Attacking
Perturbations
FGSM on IEP

Image
Succ.
p
100.00
2.9

Surface N.
Succ.
p
77.67
9.9

Illumination
Succ.
p
49.67
11.0

Material
Succ.
p
19.00
8.4

Combined
Succ.
p
82.67
9.9

Table 2. Generating white-box attacks for visual question answering with IEP [15]. By combined, we allow three sets of physical
parameters to be perturbed jointly. Succ. denotes the success rate of attacks (%, higher is better) of giving a correct answer, and p is the
perceptibility value (unit: 10−3 , lower is better) defined in Section 3.3.1.
Q1: Is there another small object of the same shape as the tiny
gray matte object?
Q2: What number of other objects are of the same size as the red
block?
Q3: Are there any big red matte balls?



A1: yes

Attacking Q1

Attacking Q2

𝑝 = 4.4 × 10−3

A1: no

A2: 4





A3: no

A2: 3



A1: small

Attacking Q3

𝑝 = 3.4 × 10−3

conf = 52.3%

Q1: What size is the other blue matte thing that is the same
shape as the yellow rubber thing?
Q2: Are there fewer cyan matte objects than tiny green shiny
blocks?
Q3: The large thing right of the big cyan rubber cube has what
shape?



conf = 66.8%

Attacking Q1

𝑝 = 8.3 × 10−3

A3: yes



A1: large

A2: no

Attacking Q2

𝑝 = 6.6 × 10−3

conf = 58.0%





conf = 57.2%



A3: cube

Attacking Q3

𝑝 = 5.5 × 10−3

A2: yes





conf = 58.1%

𝑝 = 5.2 × 10−3

A3: no
conf = 44.6%

Figure 4. Examples of adversaries generated in the 3D visual question answering task. In each group, the top row shows the original testing
image and two questions, both of which are correctly answered. The following two rows display the perturbations and the attacked image,
respectively. All perturbations are magnified by a factor of 5 and shifted by 128. p is the perceptibility value defined in Section 3.3.1, and
conf is the confidence score of choosing this answer. Note that the rightmost column shows a ridiculous answer.

4.2.2

5. Conclusions

Quantitative Results

Results are shown in Table 2. We observe similar phenomena as in the classification experiments. This is as expected,
since after the question is parsed and a neural module network is generated, attacking either image or physical space
is essentially equivalent to that in the classification task.
A side note comes from perturbing the material parameters. Although some visual questions are asking about the
material (e.g., metal or rubber) of an object, the success
rate of this type of questions does not differ from that in
attacking other questions significantly. This is because we
are constraining perceptibility, which does not allow the
material parameters to be modified by a large value.
A significant difference of visual question answering
comes from the so called language prior. With a language
parser, the network is able to clinch a small subset of
answers without looking at the image, e.g., when asked
about the color of an object, it is very unlikely for the
network to answer yes or three. We find that sometimes
the network can make such ridiculous errors, e.g., in the
rightmost column of Figure 4, when asked about the shape
of an object, the network answers no after a non-targeted
attack.

This paper delivers an important message, which is the
difficulty of generating adversaries in physical space. To
study this topic, we plug a differentiable rendering layer
into the state-of-the-art deep networks for object classification and visual question answering. Two methods are used
to attack the physical parameters. Directly constructing
adversaries in physical space is effective, but the success
rate is lower than that in image space, and much heavier
perturbations are required in a successful attack. Second,
based on the current optimization algorithms, e.g., iterative
FGSM, it is almost impossible to generate the adversaries in
image space by perturbing the physical parameters, which
suggests that 2D adversaries in physical space are often not
well explained by physical perturbations.
This work has two potentials. First, the existence of
real physical adversaries may trigger research in more complicated 3D applications, e.g., stereo matching [47] or reinforcement learning [24] in 3D virtual scenes. Second,
in 3D vision scenarios, we can defend the deep neural
networks from 2D adversaries by enforcing an image to be
interpreted in physical space, so that their attacking abilities
are weakened or removed after being re-rendered.
8

Acknowledgements

[15] J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman,
L. Fei-Fei, C. Zitnick, and R. Girshick. Inferring and
Executing Programs for Visual Reasoning. International
Conference on Computer Vision, 2017.
[16] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet
Classification with Deep Convolutional Neural Networks.
Advances in Neural Information Processing Systems, 2012.
[17] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial Examples in the Physical World. Workshop Track, International
Conference on Learning Representations, 2017.
[18] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial
Machine Learning at Scale. International Conference on
Learning Representations, 2017.
[19] G. Liu, D. Ceylan, E. Yumer, J. Yang, and J. Lien. Material Editing Using a Physically Based Rendering Network.
International Conference on Computer Vision, 2017.
[20] Y. Liu, X. Chen, C. Liu, and D. Song. Delving into Transferable Adversarial Examples and Black-Box Attacks. International Conference on Learning Representations, 2017.
[21] S. Lombardi and K. Nishino. Reflectance and Natural Illumination from a Single Image. European Conference on
Computer Vision, 2012.
[22] J. McCormac, A. Handa, S. Leutenegger, and A. Davison.
SceneNet RGB-D: 5M Photorealistic Images of Synthetic
Indoor Trajectories with Ground Truth. International Conference on Computer Vision, 2017.
[23] J. Metzen, T. Genewein, V. Fischer, and B. Bischoff. On Detecting Adversarial Perturbations. International Conference
on Learning Representations, 2017.
[24] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. Ballard,
A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu,
et al. Learning to Navigate in Complex Environments. arXiv
preprint arXiv:1611.03673, 2016.
[25] S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard.
Universal Adversarial Perturbations. Computer Vision and
Pattern Recognition, 2017.
[26] S. Moosavi-Dezfooli, A. Fawzi, and P. Frossard. DeepFool:
A Simple and Accurate Method to Fool Deep Neural Networks. Computer Vision and Pattern Recognition, 2016.
[27] V. Nair and G. Hinton. Rectified Linear Units Improve
Restricted Boltzmann Machines. International Conference
on Machine Learning, 2010.
[28] A. Nguyen, J. Yosinski, and J. Clune. Deep Neural Networks
are Easily Fooled: High Confidence Predictions for Unrecognizable Images. Computer Vision and Pattern Recognition, 2015.
[29] F. Nicodemus, J. Richmond, J. Hsia, I. Ginsberg, and
T. Limperis. Geometrical Considerations and Nomenclature
for Reflectance. Radiometry, pages 94–145, 1992.
[30] K. Nishino. Directional Statistics BRDF Model. International Conference on Computer Vision, 2009.
[31] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami.
Distillation as a Defense to Adversarial Perturbations against
Deep Neural Networks. IEEE Symposium on Security and
Privacy, 2016.
[32] C. Qi, H. Su, K. Mo, and L. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation.
Computer Vision and Pattern Recognition, 2017.

We thank Guilin Liu, Cihang Xie, Zhishuai Zhang and
Yi Zhang for instructive discussions.

References
[1] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural
Module Networks. Computer Vision and Pattern Recognition, 2016.
[2] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra,
C. Lawrence Zitnick, and D. Parikh. VQA: Visual Question
Answering. International Conference on Computer Vision,
2015.
[3] Blender Online Community. Blender – a 3D modelling
and rendering package. https://www.blender.org/,
2017. Blender Foundation, Blender Institute, Amsterdam.
[4] N. Carlini and D. Wagner. Towards Evaluating the Robustness of Neural Networks. IEEE Symposium on Security and
Privacy, 2017.
[5] A. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang,
Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. ShapeNet:
An Information-Rich 3D Model Repository. arXiv preprint
arXiv:1512.03012, 2015.
[6] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and
A. Yuille. DeepLab: Semantic Image Segmentation with
Deep Convolutional Nets, Atrous Convolution, and Fully
Connected CRFs. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 2017.
[7] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. FeiFei. ImageNet: A Large-Scale Hierarchical Image Database.
Computer Vision and Pattern Recognition, 2009.
[8] H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu.
Are You Talking to a Machine? Dataset and Methods for
Multilingual Image Question Answering. Advances in Neural Information Processing Systems, 2015.
[9] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich Feature Hierarchies for Accurate Object Detection and Semantic
Segmentation. Computer Vision and Pattern Recognition,
2014.
[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Nets. Advances in Neural Information
Processing Systems, 2014.
[11] I. Goodfellow, J. Shlens, and C. Szegedy. Explaining and
Harnessing Adversarial Examples. International Conference
on Learning Representations, 2015.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. Computer Vision and Pattern
Recognition, 2016.
[13] S. Ioffe and C. Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate
Shift. International Conference on Machine Learning, 2015.
[14] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei,
C. Zitnick, and R. Girshick. CLEVR: A Diagnostic Dataset
for Compositional Language and Elementary Visual Reasoning. Computer Vision and Pattern Recognition, 2017.

9

[33] A. Radford, L. Metz, and S. Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. International Conference on Learning
Representations, 2016.
[34] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN:
Towards Real-Time Object Detection with Region Proposal
Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017.
[35] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
et al. ImageNet Large Scale Visual Recognition Challenge.
International Journal of Computer Vision, pages 1–42, 2015.
[36] K. Sfikas, T. Theoharis, and I. Pratikakis. Exploiting the
PANORAMA Representation for Convolutional Neural Network Classification and Retrieval. Eurographics Workshop
on 3D Object Retrieval, 2017.
[37] E. Shelhamer, J. Long, and T. Darrell. Fully Convolutional
Networks for Semantic Segmentation. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 39(4):640–651,
2017.
[38] K. Simonyan and A. Zisserman. Very Deep Convolutional
Networks for Large-Scale Image Recognition. International
Conference on Learning Representations, 2015.
[39] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: A Simple Way to Prevent Neural
Networks from Overfitting. Journal of Machine Learning
Research, 15(1):1929–1958, 2014.
[40] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller. Multiview Convolutional Neural Networks for 3D Shape Recognition. International Conference on Computer Vision, 2015.
[41] I. Sutskever, O. Vinyals, and Q. Le. Sequence to Sequence
Learning with Neural Networks. Advances in Neural Information Processing Systems, 2014.
[42] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going Deeper with Convolutions. Computer Vision and
Pattern Recognition, 2015.
[43] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan,
I. Goodfellow, and R. Fergus. Intriguing Properties of Neural
Networks. 2014.
[44] F. Tramer, A. Kurakin, N. Papernot, D. Boneh, and P. McDaniel. Ensemble Adversarial Training: Attacks and Defenses. arXiv preprint arXiv:1705.07204, 2017.
[45] C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, and A. Yuille.
Adversarial Examples for Semantic Segmentation and Object Detection. International Conference on Computer Vision, 2017.
[46] X. Xu, X. Chen, C. Liu, A. Rohrbach, T. Darell, and D. Song.
Can You Fool AI with Adversarial Examples on a Visual
Turing Test? arXiv preprint arXiv:1709.08693, 2017.
[47] Y. Zhang, W. Qiu, Q. Chen, X. Hu, and A. Yuille. UnrealStereo: A Synthetic Dataset for Analyzing Stereo Vision.
arXiv preprint arXiv:1612.04647, 2016.

10

