Gibbs Sampling with
Low-Power Spiking Digital Neurons
Srinjoy Das† + , Bruno Umbria Pedroni⊥ , Paul Merolla‡ , John Arthur‡ , Andrew S. Cassidy‡ , Bryan L. Jackson‡
Dharmendra Modha‡ , Gert Cauwenberghs⊥ + , Ken Kreutz-Delgado† +
Email: {s2das, bpedroni, kreutz, gert}@ucsd.edu, {pameroll, arthurjo, andrewca, bryanlj, dmodha}@us.ibm.com
†

ECE,

⊥

BioEng. &

+

arXiv:1503.07793v2 [] 27 Mar 2015

‡

Inst. for Neural Computation, UC San Diego, La Jolla, CA 92093
IBM Research Almaden, San Jose, CA 95120

Abstract—Restricted Boltzmann Machines and Deep Belief
Networks have been successfully used in a wide variety of applications including image classification and speech recognition.
Inference and learning in these algorithms uses a Markov Chain
Monte Carlo procedure called Gibbs sampling. A sigmoidal
function forms the kernel of this sampler which can be realized
from the firing statistics of noisy integrate-and-fire neurons on
a neuromorphic VLSI substrate. This paper demonstrates such
an implementation on an array of digital spiking neurons with
stochastic leak and threshold properties for inference tasks and
presents some key performance metrics for such a hardwarebased sampler in both the generative and discriminative contexts.

I. INTRODUCTION AND BACKGROUND
Restricted Boltzmann Machines (RBMs) and Deep Belief
Networks (DBNs) (Fig. 1) are stochastic neural networks
that have been used for a wide variety of generative and
discriminative tasks like image classification, sequence completion, motion synthesis and speech recognition. An RBM is
a stochastic neural network consisting of two symmetrically
interconnected layers composed of neuron-like units — a set
of visible units v and a set of hidden units h. For an RBM
there are no interconnections within a layer. Both inference
and learning in this model use a Markov Chain Monte Carlo
(MCMC) procedure called Gibbs Sampling [1] where each
neuron is sampled based on its total input from other connected
neurons with a sigmoidal activation function. DBNs consist of
one visible layer and multiple layers of hidden units. Learning
in a DBN can be done in a layer-by-layer manner on each
RBM with this Gibbs sampling procedure [2]. Following this
approach the values of the neurons in each layer of a DBN can
be inferred by using the same stochastic sampling procedure.
Neuromorphic computing is an area of Very Large Scale
Integrated Circuit (VLSI) design inspired by the architecture
and function of the brain. Such systems which have been
realized with both analog [3] and digital [4] circuit elements
consist of massively parallel arrays of interconnected spiking
neurons modeled on the basis of neurons and synapses present
in biological neural substrates. In contrast to the traditional von
Neumann computing paradigm, memory and computation are
tightly coupled in this architecture. The principal benefits are
extremely energy efficient computation by spiking neurons in
a highly concurrent fashion.

	
  	
  	
  (a)	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  (b)	
  

	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  

Fig. 1: a) Restricted Boltzmann Machine with 4 visible and 3 hidden units.
b) Deep Belief Network with 3 hidden layers

The majority of RBMs and DBNs described in the literature
currently operate on standard platforms like high performance
CPU (Central Processing Unit) or GPU (Graphical Processing
Unit) and are deployable on cloud and related infrastructures.
However, for ultra low-power, realtime realizations of these
algorithms, hardware substrates provided by neuromorphic
VLSI are naturally amenable to the use of sampling methods for probabilistic computation in the context of highdimensional real world data. In this paper we propose an
MCMC sampling scheme for RBMs and DBNs using the
stochastic leak and threshold properties of digital spiking
neurons on a neuromorphic VLSI substrate. Such a framework
has significant potential for enabling applications which will
benefit from realtime, energy-efficient realizations, such as the
Internet of Things and Brain Computer Interfaces.
II. INFERENCE ON SPIKING SUBSTRATES
The RBM captures a probabilistic generative model of the
input data based on the Boltzmann distribution as below [1]:
(
exp(−E(v,h))
p(v, h) = Σv,h
exp(−E(v,h))
(1)
T
where E(v, h) = −vT Wh − bT
v v − bh h
Here p denotes the Boltzmann probability distribution and
E is an energy function of v and h where v denotes the state
of the visible units which are driven by the input data and
h represents the state of the hidden units. W represents the
weight between v and h, and bv , bh represent the biases of
v and h respectively. A necessary and sufficient condition for
sampling from the Boltzmann distribution given by the above
equation is to sample each neuron with a sigmoid probability

law as a function of the activities of all other connected
neurons [5]. This is as given below :
1
(2)
P(xi = 1|xj , j 6= i) =
1 + e−(Σj wij xj +bi )
This rule forms the kernel of the Gibbs sampling MCMC
procedure for an RBM. Here wij is the weight between
neurons xi , xj and bi denotes the bias of neuron xi . This
equivalence between probability laws at the unit and ensemble
levels is exactly realizable for substrates where explicit synchronization is provided and all samples from the underlying
primitives representing neurons are collected in discrete steps.
This allows the RBM to do alternating parallel sampling to
generate statistics for the MCMC inference process.
The TrueNorth neurosynaptic processor [4] is an array of
4096 cores. Each core consists of a crossbar with 256 axons
and 256 neurons. Communication between neurons occurs
only with all-or-none spikes realized with digital circuitry.
In the interconnection network, the spikes are generated with
asynchronous digital logic, however all spikes are explicitly
aligned at a single clock edge for use in the next step of
computation. On this substrate, the stochastic and dynamical
properties of the neurons coupled with these discrete synchronization steps can be used to generate the sigmoid probability
law used for sampling from the Boltzmann distribution of
interest. A Gibbs sampler for inference in RBMs and DBNs
can thus be constructed on such a substrate for the conditions
outlined in Eqns. (1), (2).
III. DIGITAL GIBBS SAMPLER
TrueNorth [4] is composed of digital integrate-and-fire
neurons (I&F) with both stochastic and deterministic leak
and threshold properties. The dynamical equations for the
membrane potential Vj (t) for neuron j at time t in this case
are shown below [6]:

N−1

Vj (t) = Vj (t − 1) + Σi=0 xi (t)si
(3)
Vj (t) = Vj (t) − λj


If (Vj (t) ≥ αj , SPIKE and set Vj (t) = Rj
Here λj and αj represent the leak and threshold values respectively of neuron j which can be stochastic or deterministic,
xi represents the input from N other neurons, si represents
the synaptic weight between neurons i, j and Rj represents
the reset value for neuron j.
Using similar I&F neurons with stochastic leak and threshold an algorithm for realizing the sigmoidal sampling rule
(Eqn. 2) to perform MCMC sampling in RBMs is given below:

Fig. 2: Noisy and smooth realizations of digital sigmoid using 1000 samples
per v-value

number of discrete time steps T w for sampling, fixed threshold
value V t, the number of bits allowed for the stochastic
threshold variation T M (uniformly distributed discrete random
variable) and the value of leak. For the sampled neuron,
V t rand and V denote its threshold and membrane potential
respectively. After integration, the sampled value of a neuron is
set to 1 if it spikes in any of the allowed number of sampling
intervals T w. Note that T M and leak are both positive. This
method uses the underlying substrate’s dynamical (integration)
and stochastic properties along with spike synchronization
at fixed time steps to generate a sigmoid probability curve.
Random threshold and leak values are realized with on-chip
pseudo random number generators (PRNGs).
IV. WEIGHT AND BIAS SCALING
On a substrate of digital neurons and synapses, weight
and bias values have to be quantized in accordance with
the finite precision available in hardware. To increase the
dynamic range, a multiplicative factor (scale > 1) is applied to
the weights obtained after offline training of the RBM/DBN
and before they are mapped to the hardware for applicable
inference tasks (classification, pattern completion and others).
Therefore the four sigmoid parameters (T w, V t, T M and
leak) should be chosen so that for any neuron i there is a
smooth mapping between the values of vi = Σj wij xj + bi
versus the activation probability P (vi ) and a majority of the
inputs are not mapped to the portion of the curve which
saturates to 0 or 1. This is illustrated in Fig. (2), where
parameter values of (4,0,3,1) provide an insufficient range
for smooth realization of the function. In contrast parameter
values of (4,100,8,90) provide a larger dynamic range, where
the original weight and bias values have been multiplied by a
scale factor (in this case scale = 50) for the mapping. This is
equivalent to scaling the ideal sigmoid by this same factor:
Pscaled (v) =

1
v
1 + e− scale

(4)

V. EFFECT OF STOCHASTIC PARAMETERS
repeat
V = V + leak ∗ (B(0.5)), B is Bernoulli
Vt rand = Vt + floor(U(0, 2TM − 1))
spiked(V ≥ Vt rand) = 1
until Tw steps;
Each neuron in the RBM is mapped to one digital neuron
in this sampling scheme. The algorithm uses 4 parameters: the

A heuristic motivation for the sigmoid realization using the
stochastic leak and threshold over multiple sampling intervals
is provided in Figs. (3a, 3b). In the first case Fig. (3a) only
the stochastic leak is initially applied to sample the inputs
which causes the transfer curve to split into 3 regions: input
< 10 where the neuron will never spike as the membrane
potential is always below threshold, input > 10 and < 100
where the spiking probability is 0.5 (since the leak of 90 can

(a) Effect of noisy threshold

Fig. 6: Classification accuracy for test data without noise

Fig. 7: Classification accuracy for test data with noise
(b) Effect of noisy leak
Fig. 3: Noisy threshold and leak for sigmoid realization

occur with probability 0.5) and input >100 where the neuron
is guaranteed to spike since the threshold is reached. As the
stochastic threshold is applied, the curve becomes piecewiselinear which approximates to a sigmoid over multiple sampling
intervals T w. Similarly when only the stochastic threshold
is initially applied (Fig. 3b), the probability of activation is
proportional to the difference between the input and the fixed
threshold. As the stochastic leak is applied over multiple T w
sampling intervals the curve also attains sigmoidal shape.

classification results are shown in Fig. 6. It is noticeable that
there is no significant difference in classification performance
irrespective of the sampler complexity or the scale factor used
for the digital realization.
Index
P1
P2
P3
P4
P5
P6
P7

(Tw, Vt, TM, leak)
(1, -130, 8, 0)
(1, -80, 8, 102)
(1, -20, 8, 200)
(1, -100, 9, 300)
(16, 50, 9, 15)
(16, 100, 10, 30)
(16, 633, 8, 90)

scale
50
50
75
120
30
50
100

TABLE I: Digital neuron parameters

VI. PERFORMANCE METRICS
Different realizations of the sigmoidal approximation with
varying complexity are possible for performing inference in
RBMs/DBNs using Gibbs sampling. This can be studied in
the context of both discriminative and generative tasks.

Fig. 4: Test data

Fig. 5: Test data with salt noise

A. CLASSIFICATION PERFORMANCE
For classification we study the performance of the digital
sampler on the MNIST dataset which consists of 28x28
grayscale images of handwritten digits [0-9] (Fig. 4) and
their corresponding labels. The RBM with 784 visible, 500
hidden and 10 label neurons is trained offline on such a set
of 5000 digits (training data). Classification performance is
then tested for this RBM with the same number of visible,
hidden and label units using the digital sampler with varying
parameterizations of T w, V t, T M, leak (refer Table I) on a
set of 1000 labeled digits (test data) which are similar to
those seen by the RBM during training. Appropriate scale
factors are applied to the weights and biases as outlined in
Section IV before the digital sigmoid is used for sampling. The

B. GENERALIZATION PERFORMANCE
The classification performance of different realizations of
the digital sampler can also be studied with noisy versions
of the test data. Two types of noise are introduced in the
test dataset: salt noise which corrupts randomly chosen pixels
to white and salt and pepper noise where randomly chosen
pixels can be turned white or dark. In both cases the level of
noise (pixel corruption) is controllable with a noise f actor.
This type of data (refer Fig. 5) is very different from those
that were used for training the RBM/DBN, so this provides
a measure of the generalization performance of the sampler.
The classification results for an RBM with 500 hidden units
are shown in Fig. 7. The sampler resolution (scale factor)
has a significant effect on classification performance for noisy
versions of test data as compared to the original test data
itself. The samplers with the highest scale factor (P4, P7) have
significantly fewer errors for the noisy data sets.
C. GENERATIVE MODEL PERFORMANCE
Inference performed on an RBM in hardware for generative
tasks like pattern completion depends on the quality of the
MCMC samples. That is, how closely these samples reflect the
learned probability distribution. This can be characterized by
the Kullback-Leibler (KL) divergence metric which is a similarity measure between any 2 probability distributions. MCMC
sampling with the ideal and various digital realizations of the

	
  

	
  
Fig. 9: a) Ideal sigmoid vs. TrueNorth realization b) Firing pattern showing
how synaptic weight values ranging from -100 to 100 are used with the
respective connected axons to activate data neurons 0 through 200
(a)	
  

(b)	
  

Fig. 8: a) TrueNorth processor b) Sigmoid generation circuit with TrueNorth
neurons and synapses

sampler can be used to generate the probability distributions
and these can be compared versus the exact distribution given
by Eq. (1). Since it is difficult to calculate this distribution for
high-dimensional inputs like MNIST (784 visible neurons),
this is done for an RBM with 3 visible and 2 hidden neurons
with pre-defined weights and biases. The results are shown
in Table II. The KL-divergence increases for approximate
realizations of the digital sampler (leak = 0) indicating a lower
quality of the sampled generative model in this case.
KL divergence (1e+05 Gibbs iterations)
exact vs ideal sampler
exact vs digital(1,-130,8,0)
exact vs digital(1,-80,8,102)

Trial-1
6.2e-05
0.0218
0.0091

Trial-2
6.06e-05
0.1090
0.0330

Trial-3
5.71e-05
0.0259
0.0083

TABLE II: KL divergence for various digital sampler realizations

VII. TRUENORTH SIGMOID CHARACTERIZATION
We implemented the digital sigmoid algorithm discussed
here on TrueNorth as shown in Fig. 8 using supported parameter values. Deterministic spike inputs are driven on axons
d−K through dK (K=100) every Tw+2 (Tw=16) cycles and
are integrated on the connected neurons n−K through nK .
The synaptic weights are set between -100 to 100 in steps of
1 for these connections (indicated by dots on the crossbar).
Each such neuron is configured with a stochastic threshold α
realized by V t = 50, T M = 9 and leak value λ set to 0. A
single neuron nleak generates a stochastic leak of 0 or 1 for
all the data neurons. This is multiplied by a factor of 15 by
the weight on the connected axons and is the value of leak
used in the algorithm. The number of sampling intervals T w
for a single application of the stimulus (spike input) is set to
16. Spiking probabilities obtained with TrueNorth neurons are
compared versus the ideal sampler in Fig. 9(a). A snapshot
of the firing pattern of the characterized data neurons n−K
through nK is shown in Fig. 9(b).
VIII. CONCLUSIONS
We have demonstrated that approximate realizations of a
Gibbs sampler on a digital neuromorphic substrate are feasible
for classification with low system latency (T w = 1). However,

for generative tasks it may be necessary to use a form of the
sampler that replicates the sigmoid more accurately via the
use of a nonzero stochastic leak. Our proposed method of
realization of the sigmoidal function with low-power, digital
integrate-and-fire neurons is well suited for Gibbs sampling in
RBMs and DBNs with parallel arrays of visible and hidden
neurons in contrast to hardware implementations of sigmoids
on standard Von Neumann computing platforms [7], [8].
Another advantage of our proposed implementation is that
the required noise generation mechanism uses on-chip PRNG
circuits which are easier to realize as compared to sampling
from I&F neurons with Gaussian noise [9]. Given these
advantages such a sampler is ideally suited for a whole range
of inference tasks in practical low-power, realtime applications
which is the subject of ongoing investigation.
ACKNOWLEDGMENTS
The authors would like to thank the team members of
the Brain-Inspired Computing group at IBM Almaden for
supporting this project.
R EFERENCES
[1] S. Haykin, Neural Networks and Learning Machines (3rd Edition).
Prentice Hall, 2008.
[2] G. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for
deep belief nets,” Neural computation, vol. 18, no. 7, pp. 1527–1554,
2006.
[3] G. Indiveri, B. Linares-Barranco, T. J. Hamilton, A. Van Schaik,
R. Etienne-Cummings, T. Delbruck, S.-C. Liu, P. Dudek, P. Häfliger,
S. Renaud et al., “Neuromorphic silicon neuron circuits,” Frontiers in
neuroscience, vol. 5, 2011.
[4] P. A. Merolla, J. V. Arthur, R. Alvarez-Icaza, A. S. Cassidy, J. Sawada,
F. Akopyan, B. L. Jackson, N. Imam, C. Guo, Y. Nakamura et al., “A
million spiking-neuron integrated circuit with a scalable communication
network and interface,” Science, vol. 345, no. 6197, pp. 668–673, 2014.
[5] R. Rojas, Neural networks: a systematic introduction. Springer, 1996.
[6] A. S. Cassidy, P. Merolla, J. V. Arthur, S. K. Esser, B. Jackson, R. Alvarezicaza, P. Datta, J. Sawada, T. M. Wong, V. Feldman, A. Amir, D. B.
dayan Rubin, E. Mcquinn, W. P. Risk, and D. S. Modha, “Cognitive
computing building block: A versatile and efficient digital neuron model
for neurosynaptic cores,” in in International Joint Conference on Neural
Networks (IJCNN). IEEE, 2013.
[7] M. Tommiska, “Efficient digital implementation of the sigmoid function
for reprogrammable logic,” in Computers and Digital Techniques, IEE
Proceedings-, vol. 150, no. 6. IET, 2003, pp. 403–411.
[8] A. Tisan, S. Oniga, D. MIC, and A. Buchman, “Digital implementation of
the sigmoid function for fpga circuits,” ACTA TECHNICA NAPOCENSIS
Electronics and Telecommunications, vol. 50, no. 2, p. 6, 2009.
[9] E. Neftci, S. Das, B. Pedroni, K. Kreutz-Delgado, and G. Cauwenberghs,
“Event-driven contrastive divergence for spiking neuromorphic systems,”
Frontiers in Neuroscience, vol. 7, p. 272, 2013.

