Accelerating Deep Learning with Memcomputing
Haik Manukian1 , Fabio L. Traversa2 , Massimiliano Di Ventra1

arXiv:1801.00512v2 [cs.LG] 24 Jan 2018

Abstract
Restricted Boltzmann machines (RBMs) and their extensions, called “deep-belief networks”, are powerful neural networks
that have found applications in the fields of machine learning and big data. The standard way to train these models
resorts to an iterative unsupervised procedure based on Gibbs sampling, called “contrastive divergence” (CD), and
additional supervised tuning via back-propagation. However, this procedure has been shown not to follow any gradient
and can lead to suboptimal solutions. In this paper, we show an efficient alternative to CD by means of simulations
of digital memcomputing machines (DMMs). We test our approach on pattern recognition using a modified version
of the MNIST data set. DMMs sample effectively the vast phase space given by the model distribution of the RBM,
and provide a very good approximation close to the optimum. This efficient search significantly reduces the number
of pretraining iterations necessary to achieve a given level of accuracy, as well as a total performance gain over CD.
In fact, the acceleration of pretraining achieved by simulating DMMs is comparable to, in number of iterations, the
recently reported hardware application of the quantum annealing method on the same network and data set. Notably,
however, DMMs perform far better than the reported quantum annealing results in terms of quality of the training. We
also compare our method to advances in supervised training, like batch-normalization and rectifiers, that work to reduce
the advantage of pretraining. We find that the memcomputing method still maintains a quality advantage (> 1% in
accuracy, and a 20% reduction in error rate) over these approaches. Furthermore, our method is agnostic about the
connectivity of the network. Therefore, it can be extended to train full Boltzmann machines, and even deep networks
at once.
Keywords: Deep Learning, Restricted Boltzmann Machines, Memcomputing
1. Introduction
The progress in machine learning and big data driven
by successes in deep learning is difficult to overstate. Deep
learning models (a subset of which are called “deep-belief
networks”) are artificial neural networks with a certain
amount of layers, n, with n > 2 [1]. They have proven
themselves to be very useful in a variety of applications,
from computer vision [2] and speech recognition [3] to
super-human performance in complex games[4], to name
just a few. While some of these models have existed for
some time [5], the dramatic increases in computational
power combined with advances in effective training methods have pushed forward these fields considerably [6].
Successful training of deep-belief models relies heavily
on some variant of an iterative gradient-descent procedure, called back-propagation, through the layers of the
network [7]. Since this optimization method uses only gradient information, and the error landscapes of deep networks are highly non-convex [8], one would at best hope
to find an appropriate local minimum.
However, there is evidence that in these highdimensional non-convex settings, the issue is not getting
1 Department of Physics, University of California, San Diego, La
Jolla, CA 92093
2 MemComputing, Inc., San Diego, CA, 92130 CA

stuck in some local minima but rather at saddle points,
where the gradient also vanishes [9], hence making the
gradient-descent procedure of limited use. A takeaway
from this is that a “good” initialization procedure for assigning the weights of the network, known as pretraining,
can then be highly advantageous.
One such deep-learning framework that can utilize this
pretraining procedure is the Restricted Boltzmann Machine (RBM) [5], and its extension, the Deep Belief Network (DBN) [10]. These machines are a class of neural network models capable of unsupervised learning of a
parametrized probability distribution over inputs. They
can also be easily extended to the supervised learning
case by training an output layer using back-propagation
or other standard methods [1].
Training RBMs usually distinguishes between an unsupervised pretraining, whose purpose is to initialize a good
set of weights, and the supervised procedure. The current most effective technique for pretraining RBMs utilizes an iterative sampling technique called contrastive divergence (CD) [11]. Computing the exact gradient of the
log-likelihood is exponentially hard in the size of the RBM,
and so CD approximates it with a computationally friendly
sampling procedure. While this procedure has brought
RBMs most of their success, CD suffers from the slow
mixing of Gibbs sampling, and is known not to follow the

gradient of any function [12].
Partly due to these shortcomings of pretraining with
CD, much research has gone into making the backpropagation procedure more robust and less sensitive to
the initialization of weights and biases in the network.
This includes research into different non-linear activation
functions (e.g., “rectifiers”) [13] to combat the vanishing
gradient problem and normalization techniques (such as
“batch-normalization”) [14] that make back-propagation
in deep networks more stable and less dependent on initial
conditions. In sum, these techniques make training deep
networks an easier (e.g., more convex) optimization problem for a gradient-based approach like back-propagation.
This, in turn, relegates the standard CD pretraining procedure’s usefulness to cases where the training set is sparse
[1], which is becoming an increasingly rare occurrence.
In parallel with this research into back-propagation, sizable effort has been expended toward improving the power
of the pretraining procedure, including extensions of CD
[15, 16], CD done on memristive hardware [17], and more
recently, approaches based on quantum annealing that try
to recover the exact gradient [18] involved in pretraining.
Some of these methods are classical algorithms simulating
quantum sampling [19], and still others attempt to use a
hardware quantum device in contact with an environment
to take independent samples from its Boltzmann distribution for a more accurate gradient computation. For instance, in a recent work, the state of the RBM has been
mapped onto a commercial quantum annealing processor
(a D-Wave machine), the latter used as a sampler of the
model distribution [20]. The results reported on a reduced
version of the well-known MNIST data set look promising
as compared to CD [20]. However, these approaches require expensive hardware, and cannot be scaled to larger
problems as of yet.
In the present paper, inspired by the theoretical underpinnings [21, 22] and recent empirical demonstrations [23, 24] of the advantages of a new computing
paradigm –memcomputing [25]– on a variety of combinatorial/optimization problems, we seek to test its power
toward the computationally demanding problems in deep
learning.
Memcomputing [25] is a novel computing paradigm that
solves complex computational problems using processing
embedded in memory. It has been formalized by two of
us (FLT and MD) by introducing the concept of universal memcomputing machines[21]. In short, to perform a
computation, the task at hand is mapped to a continuous
dynamical system that employs highly-correlated states
[26] (in both space and time) of the machine to navigate
the phase space efficiently and find the solution of a given
problem as mapped into the equilibrium states of the dynamical system.
In this paper, we employ a subset of these machines
called digital memcomputing machines (DMMs) and, more
specifically, their self-organizing circuit realizations [22].
The distinctive feature of DMMs is their ability to read and

write the initial and final states of the machine digitally,
namely requiring only finite precision. This feature makes
them easily scalable as our modern computers.
From a practical point of view DMMs can be built with
standard circuit elements with and without memory [22].
These elements, however, are non-quantum. Therefore,
the ordinary differential equations of the corresponding circuits can be efficiently simulated on our present computers.
Here, we will indeed employ only simulations of DMMs on
a single Xeon processor to train RBMs. These simulations
show already substantial advantages with respect to CD
and even quantum annealing, despite the latter is executed
on hardware. Of course, the hardware implementation of
DMMs applied to these problems would offer even more
advantages since the simulation times will be replaced by
the actual physical time of the circuits to reach equilibrium. This would then offer a realistic path to real-time
pretraining of deep-belief networks.
In order to compare directly with quantum annealing
results recently reported[20], we demonstrate the advantage of our memcomputing approach by first training on
a reduced MNIST data set as that used in Ref [20]. We
show that our method requires far less pretraining iterations to achieve the same accuracy as CD, as well as an
overall accuracy gain over both CD and quantum annealing. We also train the RBMs on the reduced MNIST data
set without mini-batching, where the quantum annealing
results are not available. Also in this case, we find both
a substantial reduction in pretraining iterations needed as
well a higher level of accuracy of the memcomputing approach over the traditional CD.
Our approach then seems to offer many of the advantages of quantum approaches. However, since it is based
on a completely classical system, it can be efficiently deployed in software (as we demonstrate in this paper) as
well as easily implemented in hardware, and can be scaled
to full-size problems.
Finally, we investigate the role of recent advances in supervised training by comparing accuracy obtained using
only back-propagation with batch-normalization and rectifiers starting from a random initial condition versus the
back-propagation procedure initiated from a network pretrained with memcomputing, with sigmoidal activations
and without batch-normalization. Even without these advantages, namely operating on a more non-convex landscape, we find the network pretrained with memcomputing maintains an accuracy gain over state-of-the-art backpropagation by more than 1% and a 20% reduction in
error rate. This gives further evidence to the fact that
memcomputing pretraining navigates to an advantageous
initial point in the non-convex loss surface of the deep network.
2. RBMs and Contrastive Divergence
An RBM consists of m visible units, vj , j = 1 . . . m, each
fully connected to a layer of n hidden units, hi , i = 1 . . . n,
2

updates from the n-th to the (n + 1)-th iteration:

Output
layer

o2

o1

n+1
n
wij
= αwij
+ [hvi hj iDATA − hvi hj iMODEL ],

o3

(4)

where α is called the “momentum” and  is the “learning
rate”. A similar update procedure is applied to the biases:
h1

h2

h3

bn+1
= αbni + [hvi iDATA − hvi iMODEL ],
i

(5)

cn+1
= αcnj + [hhj iDATA − hhj iMODEL ].
j

(6)

RBM
v2

v1

v3

v4

This form of the weight updates is referred to as
“stochastic gradient optimization with momentum”. Here
α is the momentum parameter and  the learning rate. The
first expectation value on the rhs of Eqs. (4), (5), and (6)
is taken with respect to the conditional probability distribution with the data fixed at the visible layer. This
is relatively easy to compute. Evaluation of the second
expectation on the rhs of Eqs. (4), (5), and (6) is exponentially hard in the size of the network, since obtaining
independent samples from a high-dimensional model distribution easily becomes prohibitive with increasing size
[11]. This is the term that CD attempts to approximate.
The CD approach attempts to reconstruct the difficult
expectation term with iterative Gibbs sampling. This
works by sequentially sampling each layer given the sigmoidal conditional probabilities, namely


X
p(hi = 1|v) = σ 
wij vj + ci  ,
(7)

Figure 1: A sketch of an RBM with four visible nodes, three hidden nodes, and an output layer with three nodes. The value of
each stochastic binary node is represented by vi , hi ∈ {0, 1}, which
are sampled from the probabilities in Eqs. (7), (8). The connections between the layers represent the weights, wij ∈ R (biases not
shown). Note the lack of connections between nodes in the same
layer, which distinguishes the RBM from a Boltzmann machine. The
RBM weights are trained separately from the output layer with generative pretraining, then tuned together via back-propogation (just
as in a feed-forward neural network).

both usually taken to be binary variables. In the restricted
model, no intra-layer connection are allowed, see Fig. 1.
The connectivity structure of the RBM implies that,
given the hidden variables, each input node is conditionally
independent of all the others:
p(vi , vj |h) = p(vi |h)p(vj |h).

j

(1)

for the visible layer, and similarly for the hidden layer
!
X
p(vj = 1|h) = σ
wij hi + bj ,
(8)

The joint probability is given by the Gibbs distribution,
p(v, h) =

1 −E(v,h)
e
,
Z

with an energy function
XX
X
X
E(v, h) = −
wij hi vj −
bj v j −
ci hi ,
i

j

j

i

(2)

with σ(x) = (1 + e−x )−1 . The required expectation values are calculated with the resulting samples. In the limit
of infinite sampling iterations, the expectation value is recovered. However, this convergence is slow and in practice
usually only one iteration, referred to CD-1, is used [27].

(3)

i

where wij is the weight between the i-th hidden neuron
and the j-th visible neuron, and bj , ci are real numbers
indicating the “biases” of the neurons. The value, Z, is
a normalization constant, and is known in statistical mechanics as the partition function. Training an RBM then
amounts to finding a set of weights and biases that maximizes the likelihood (or equivalently minimizes the energy)
of the observed data.
A common approach to training RBMs for a supervised
task is to first perform generative unsupervised learning
(pretraining) to initialize the weights and biases, then run
back-propagation over input-label pairs to fine tune the
parameters of the network. The pretraining is framed as
a gradient ascent over the log-likelihood of the observed
data, which gives a particularly tidy form for the weight

3. Efficient Sampling with Memcomputing
We replace the CD reconstruction with our approach
that utilizes memcomputing to compute a much better approximation to the model expectation than CD.
More specifically, we map the RBM pretraining problem onto DMMs realized by self-organizing logic circuits
(SOLCs) [22]. These electrical circuits define a set of coupled differential equations which are set to random initial conditions and integrated forward toward the global
solution of the given problem[22, 23, 24]. The ordinary
differential equations we solve can be found in Ref. [22]
appropriately adapted to deal with the particular problem
discussed in this paper.
3

(a) 100 back-propagation iterations
20
20

10

16

Mem-QUBO
CD-1

1

15

0

10

20

30

0.8

40

Accuracy

Total Weight

18

14

0.6
0.4

12
0.2

10
35.5

36

36.5

37

37.5

38

38.5

39

39.5

0

40

Simulation Time (arbitrary units)

0

5

10

15

20

25

30

35

40

45

50

45

50

45

50

Pretraining Iterations

Figure 2: Plot of the total weight of a MAX-SAT clause as a function of simulation time (in arbitrary units) of a DMM. A lower weight
variable assignment corresponds directly to a higher probability assignment of the nodes of an RBM. If the simulation has not changed
assignments in some time, we restart with another random (independent) initial condition. The inset shows the full simulation, with all
restarts. The main figure focuses on the last three restarts, signified
by the black box in the inset.

(b) 200 back-propagation iterations

Accuracy

0.8

Within this memcomputing context, we construct a
reinterpretation of the RBM pretraining that explicitly
shows how it corresponds to an NP-hard optimization
problem, which we then tackle using DMMs. We first observe that to obtain a sample near most of the probability
mass of the joint distribution, p(v, h) ∝ e−E(v,h) , one must
find the minimum of the energy of the form Eq. 3, which
constitutes a quadratic unconstrained binary optimization
(QUBO) problem [28].
We can see this directly by considering the visible and
hidden nodes as one vector x = (v, h) and re-writing the
energy of an RBM configuration as


B
0


W
,
C

0.4

0
0

5

10

15

20

25

30

35

40

Pretraining Iterations

(c) 400 back-propagation iterations

Mem-QUBO
CD-1

1
0.8

(9)

where Q is the matrix
Q=

0.6

0.2

Accuracy

E = −xT Qx,

Mem-QUBO
CD-1

1

0.6
0.4
0.2

(10)

0
0

with B and C being the diagonal matrices representing the
biases bj and ci , respectively, while the matrix W contains
the weights.
We then employ a mapping from a general QUBO problem to a weighted maximum satisfiability (weighted MAXSAT) problem, similar to [29], which is directly solved by
the DMM. The weighted MAX-SAT problem is to find an
assignment of boolean variables that minimizes the total
weight of a given boolean expression written in conjunctive
normal form [28].
This problem is a well-known problem in the NP-hard
complexity class [28]. However, it was recently shown in
Ref. [23], that simulations of DMMs show dramatic (exponential) speed-up over the state-of-the-art solvers [23],
when attempting to find better approximations to hard
MAX-SAT instances beyond the inapproximability gap
[30].

5

10

15

20

25

30

35

40

Pretraining Iterations

Figure 3: Memcomputing (Mem-QUBO) accuracy on the test set
of the reduced MNIST problem versus contrastive divergence for
n = 100(a), 200(b), 400(c) iterations of back-propagation with
√ minibatches of 100. The plots show average accuracy with ±σ/ N error
bars calculated across 10 DBNs trained on N = 10 different partitions of the training set. One can see a dramatic acceleration with
the memcomputing approach needing far less iterations to achieve
the same accuracy, as well as an overall performance gap (indicated
by a black arrow) that back-propagation cannot seem to overcome.
Note that some of the error bars for both Mem-QUBO and CD1 are very small on the reported scale for a number of pretraining
iterations larger than about 20.

The approximation to the global optimum of the
weighted MAX-SAT problem given by memcomputing is
then mapped back to the original variables that represent
4

(a) 100 back-propagation iterations

the states of the RBM nodes. Finally, we obtain an approximation to the “ground state” (lowest energy state) of
the RBM as a variable assignment, x∗ , close to the peak
of the probability distribution, where ∇P (x∗ ) = 0. This
assignment is obtained by integration of the ordinary differential equations that define the SOLCs dynamics. In
doing so we collect an entire trajectory, x(t), that begins
at a random initial condition in the phase space of the
problem, and ends at the lowest energy configuration of
the variables (see Fig. 2).
Since the problem we are tackling here is an optimization one, we do not have any guarantee of finding the
global optimum. (This is in contract to a SAT problem
where we can guarantee DMMs do find the solutions of
the problem corresponding to equilibrium points, if these
exist [22, 31, 32].) Therefore, there is an ambiguity about
what exactly constitutes the stopping time of the simulation, since a priori, one cannot know that the simulation
has reached the global minimum.
We then perform a few “restarts” of the simulation (that
effectively correspond to a change of the initial conditions)
and stop the simulation when the machine has not found
any better configuration within that number of restarts.
The restarts are clearly seen in Fig. 2 as spikes in the
total weight of the boolean expression. In this work we
have employed 28 restarts, which is an over-kill since a
much smaller number would have given similar results.
The full trajectory, x(t), together with the above
“restarts” is plotted in Fig. 2. It is seen that this trajectory, in between restarts, spends most of its time in
“low-energy regions,” or equivalently areas of high probability. A time average, hx(t)i, gives a good approximation
to the required expectations in the gradient calculation in
Eqs. (4), (5), and (6). In practice, even using the best assignment found, x∗ , shows a great improvement over CD
in our experience. This is what we report in this paper.
Note also that a full trajectory, as the one shown in Fig. 2,
takes about 0.5 seconds on a single Xeon processor.
As a testbed for the memcomputing advantage in deep
learning, and as a direct comparison to the quantum annealing hardware approaches, we first looked to the reduced MNIST data set as reported in [20] for quantum
annealing using a D-wave machine. Therefore, we have
first applied the same reduction to the full MNIST problem as given in that work, which consists of removing two
pixels around all 28 × 28 grayscale values in both the test
and training sets. Then each 4 × 4 block of pixels is replaced by their average values to give a 6×6 reduced image.
Finally, the four corner pixels are discarded resulting in a
total of 32 pixels representing each image.
We also trained the same-size DBN consisting of two
stacked RBMs each with 32 visible and hidden nodes,
training each RBM one at a time. We put both the CD1 and our Memcomputing-QUBO (Mem-Qubo) approach
through N = 1, · · · , 50 generative pretraining iterations
using no mini-batching.
For the memcomputing approach, we solve one QUBO

1
Mem-QUBO
CD-1

Accuracy

0.8

0.6

0.4

0.2

0

0

5

10

15

20

25

30

35

40

45

50

45

50

45

50

Pretraining iterations

(b) 500 back-propagation iterations
1
Mem-QUBO
CD-1

Accuracy

0.8

0.6

0.4

0.2

0

0

5

10

15

20

25

30

35

40

Pretraining iterations

(c) 800 back-propagation iterations
1
Mem-QUBO
CD-1

Accuracy

0.8

0.6

0.4

0.2

0

0

5

10

15

20

25

30

35

40

Pretraining iterations

Figure 4: Mem-QUBO accuracy on the reduced MNIST test set vs.
CD-1 after n = 100(a), 500(b), 800(c) iterations of back-propagation
with no mini-batching. The resulting pretraining acceleration shown
by the memcomputing approach is denoted by the horizontal arrow.
A performance gap also appears, emphasized by the vertical arrow,
with Mem-QUBO obtaining a higher level of accuracy than CD-1,
even for the highest number of back-propagation iterations. No error
bars appear here since we have trained the full test set.

problem per pretraining iteration to compute the model
expectation value in Eqs. (4), (5), and (6). We pick out
the best variable assignment, x∗ , which gives the ground
state of Eq. (3) as an effective approximation of the required expectation. After generative training, an output
classification layer with 10 nodes was added to the net5

work (see Fig. 1) and 1000 back-propagation iterations
were applied in both approaches using mini-batches of 100
samples to generate Fig. 3. For both pretraining and backpropagation, our learning rate was set to  = 0.1 and momentum parameters were α = 0.1 for the first 5 iterations,
and α = 0.5 for the rest, same as in [20].
Accuracy on the test set versus CD-1 as a function of
the number of pretraining iterations is seen in Fig. 3.
The memcomputing method reaches a far better solution
faster, and maintains an advantage over CD even after
hundreds of back-propagation iterations. Interestingly, our
software approach is even competitive with the quantum
annealing method done in hardware [20] (cf. Fig. 3 with
Figs. 7, 8, and 9 in Ref. [20]). This is quite a remarkable
result, since we integrate a set of differential equations
of a classical system, in a scalable way, with comparable
sampling power to a physically-realized system that takes
advantage of quantum effects to improve on CD.
Finally, we also trained the RBM on the reduced MNIST
data set without mini-batches. We are not aware of
quantum-annealing results for the full data set, but we
can still compare with the CD approach. We follow a similar procedure as discussed above. In this case, however,
no mini-batching was used for a more direct comparison
between the Gibbs sampling of CD and our memcomputing approach. The results are shown in Fig. 4 for different numbers of back-propagation iterations. Even on the
full modified MNIST set, our memcomputing approach requires a substantially lower number of pretraining iterations to achieve a high accuracy and, additionally, shows
a higher level of accuracy over the traditional CD, even
after 800 back-propagation iterations.

1
Mem-QUBO+Sig
BN+ReLU

0.975
0.95

Accuracy

0.925
0.9
0.875

0.96

0.85

0.95

0.825

0.94

0.8

0.93

0.775
0.75

0.92
800

0

100

200

300

400

500

600

900

700

1000

800

900

1000

Backpropagation Iterations

Figure 5: Accuracy on the reduced MNIST test set obtained on a
network pretrained with (blue curve) Mem-QUBO and sigmoidal activation functions (Sig) versus the same size network with (red curve)
no pretraining but with batch normalization (BN) and rectified linear
units (ReLU). Both networks were trained with stochastic gradient
descent with momentum and mini-batches of 100. The inset clearly
shows an accuracy advantage of Mem-QUBO greater than 1% and
an error rate reduction of 20% throughout the training.

we employ the batch-normalization procedure [14] coupled
with rectified linear units (ReLUs) [13]. As anticipated,
batch-normalization smooths out the role of initial conditions, while rectifiers should render the energy landscape
defined by Eq. (3) more convex. Therefore, combined they
indeed seem to provide an advantage compared to the network trained with CD using sigmoidal functions [13].
However, they are not enough to overcome the advantages of our memcomputing approach. In fact, it is obvious
from Fig. 5 that the network pretrained with memcomputing maintains an accuracy advantage (of more than 1% and
a 20% reduction in error rate) on the test set out to more
than a thousand back-propagation iterations. It is key
to note that the network pretrained with memcomputing
contains sigmoidal activations compared to the rectifiers
in the network with no pretraining. Also, the pretrained
network was trained without any batch normalization procedure.
Therefore, considering all this, the pretrained network
should pose a more difficult optimization problem for
stochastic gradient descent. Instead, we found an accuracy advantage of memcomputing throughout the course
of training. This points to the fact that with memcomputing, the pretraining procedure is able to operate close to
the “true gradient” (Eqs. (4), (5), and (6)) during training, and in doing so, initializes the weights and biases of
the network in a advantageous way.

4. The Role of Supervised Training
The computational difficulty of computing the exact
gradient update in pretraining, combined with the inaccuracies of CD, has inspired research into methods which
reduce (or outright eliminate) the role of pretraining deep
models. These techniques include changes to the numerical
gradient procedure itself, like adaptive gradient estimation
[33], changes to the activation functions (e.g., the introduction of rectifiers) to reduce gradient decay and enforce
sparsity [13], and techniques like batch normalization to
make back-propagation less sensitive to initial conditions
[14]. With these new updates, in many contexts, deep networks initialized from a random initial condition are found
to compete with networks pretrained with CD [13].
To complete our analysis we have then compared a
network pretrained with our memcomputing approach
to these back-propagation methods with no pretraining.
Both networks were trained with stochastic gradient descent with momentum and the same learning rates and
momentum parameter we used in Section 3.
In Fig. 5, we see how these techniques fair against a network pretrained with the memcomputing approach on the
reduced MNIST set. In the randomly initialized network,

5. Conclusions
In this paper we have demonstrated how the memcomputing paradigm (and, in particular, its digital realization [22]) can be applied toward the chief bottlenecks in
deep learning today. In this paper, we directly assisted
a popular algorithm to pretrain RBMs and DBNs, which
6

consists of gradient ascent on the log-likelihood. We have
shown that memcomputing can accelerate considerably the
pretraining of these networks far better than what is currently done.
In fact, simulations of digital memcomputing machines
achieve accelerations of pretraining comparable to, in number of iterations, the hardware application of the quantum
annealing method, but with better quality. In addition,
unlike quantum computers, our approach can be easily
scaled on classical hardware to full size problems.
In addition, our memcomputing method retains an advantage also with respect to advances in supervised training, like batch-norming and rectifiers, that have been introduced to eliminate the need of pretraining. We find indeed,
that despite our pretraining done with sigmoidal functions,
hence on a more non-convex landscape than that provided
by rectifiers, we maintain an accuracy advantage greater
than 1% (and a 20% reduction in error rate) throughout
the training.
Finally, the form of the energy in Eq. 3 is quite general
and encompasses full DBNs. In this way, our method can
also be applied to pretraining entire deep-learning models
at once, potentially exploring parameter spaces that are
inaccessible by any other classical or quantum methods.
We leave this interesting line of research for future studies.
Acknowledgments – We thank Forrest Sheldon for useful
discussions and Yoshua Bengio for pointing out the role of
supervised training techniques. H.M. acknowledges support from a DoD-SMART fellowship. M.D. acknowledges
partial support from the Center for Memory and Recording Research at UCSD.

[10]

[11]
[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]
[20]

[21]

References

[22]

[1] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature
521 (7553) (2015) 436–444.
[2] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in
neural information processing systems, 2012, pp. 1097–1105.
[3] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed,
N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath,
et al., Deep neural networks for acoustic modeling in speech
recognition: The shared views of four research groups, IEEE
Signal Processing Magazine 29 (6) (2012) 82–97.
[4] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness,
M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland,
G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature 518 (7540) (2015) 529–533.
[5] P. Smolensky, Information processing in dynamical systems:
foundations of harmony theory, in: Parallel distributed processing: explorations in the microstructure of cognition, vol. 1,
MIT Press, 1986, pp. 194–281.
[6] Y. Bengio, et al., Learning deep architectures for ai, Foundations and trends R in Machine Learning 2 (1) (2009) 1–127.
[7] D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning representations by back-propagating errors, nature 323 (6088) (1986)
533–536.
[8] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, Y. LeCun, The loss surfaces of multilayer networks, in: Artificial Intelligence and Statistics, 2015, pp. 192–204.
[9] Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli,
Y. Bengio, Identifying and attacking the saddle point problem

[23]

[24]

[25]
[26]

[27]
[28]
[29]

[30]
[31]

7

in high-dimensional non-convex optimization, in: Advances in
neural information processing systems, 2014, pp. 2933–2941.
G. E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm
for deep belief nets, Neural computation 18 (7) (2006) 1527–
1554.
G. E. Hinton, Training products of experts by minimizing contrastive divergence, Training 14 (8).
I. Sutskever, T. Tieleman, On the convergence properties of
contrastive divergence, in: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics,
2010, pp. 789–795.
X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural
networks, in: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011, pp. 315–
323.
S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep
network training by reducing internal covariate shift, in:
F. Bach, D. Blei (Eds.), Proceedings of the 32nd International
Conference on Machine Learning, Vol. 37 of Proceedings of Machine Learning Research, PMLR, Lille, France, 2015, pp. 448–
456.
URL http://proceedings.mlr.press/v37/ioffe15.html
T. Tieleman, G. Hinton, Using fast weights to improve persistent contrastive divergence, in: Proceedings of the 26th Annual
International Conference on Machine Learning, ACM, 2009, pp.
1033–1040.
E. Romero Merino, F. Mazzanti Castrillejo, J. Delgado Pin,
D. Buchaca Prats, Weighted Contrastive Divergence, ArXiv eprintsarXiv:1801.02567.
A. M. Sheri, A. Rafique, W. Pedrycz, M. Jeon, Contrastive
divergence for memristor-based restricted boltzmann machine,
Engineering Applications of Artificial Intelligence 37 (2015)
336–342.
J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe,
S. Lloyd, Quantum machine learning, Nature 549 (7671) (2017)
195–202.
N. Wiebe, A. Kapoor, K. M. Svore, Quantum deep learning,
arXiv preprint arXiv:1412.3489.
S. H. Adachi, M. P. Henderson, Application of quantum annealing to training of deep neural networks, arXiv preprint
arXiv:1510.06356.
F. L. Traversa, M. Di Ventra, Universal memcomputing machines, IEEE Trans. on Neural Networks 26 (2015) 2702–2715.
F. L. Traversa, M. Di Ventra, Polynomial-time solution of prime
factorization and np-complete problems with digital memcomputing machines, Chaos: An Interdisciplinary Journal of Nonlinear Science 27 (2) (2017) 023107. arXiv:http://dx.doi.org/
10.1063/1.4975761, doi:10.1063/1.4975761.
URL http://dx.doi.org/10.1063/1.4975761
F. L. Traversa, P. Cicotti, F. Sheldon, M. Di Ventra, Evidence
of an exponential speed-up in the solution of hard optimization
problems, ArXiv e-printsarXiv:1710.09278.
H. Manukian, F. L. Traversa, M. Di Ventra, Memcomputing numerical inversion with self-organizing logic gates, IEEE Transactions on Neural Networks and Learning Systems PP (99) (2017)
1–6. doi:10.1109/TNNLS.2017.2697386.
M. Di Ventra, Y. V. Pershin, The parallel approach, Nature
Physics 9 (2013) 200–202.
M. Di Ventra, F. L. Traversa, I. V. Ovchinnikov, Topological
field theory and computing with instantons, Annalen der Physik
(2017) 1700123doi:10.1002/andp.201700123.
G. Hinton, A practical guide to training restricted boltzmann
machines, Momentum 9 (1) (2010) 926.
S. Arora, B. Barak, Computational Complexity: A Modern Approach, Cambridge University Press, 2009.
Z. Bian, F. Chudak, W. G. Macready, G. Rose, The ising model:
teaching an old problem new tricks, D-wave systems (2010) 1–
32.
J. Håstad, Some optimal inapproximability results, Journal of
the ACM (JACM) 48 (4) (2001) 798–859.
M. Di Ventra, F. L. Traversa, Absence of periodic orbits in

digital memcomputing machines with solutions, Chaos: An Interdisciplinary Journal of Nonlinear Science 27 (2017) 101101.
[32] M. Di Ventra, F. L. Traversa, Absence of chaos in digital memcomputing machines with solutions, Phys. Lett. A 381 (2017)
3255.
[33] D. Kingma, J. Ba, Adam: A method for stochastic optimization,
arXiv preprint arXiv:1412.6980.

8

