1

An Incremental Self-Organizing Architecture
for Sensorimotor Learning and Prediction

arXiv:1712.08521v2 [] 9 Mar 2018

Luiza Mici, German I. Parisi, and Stefan Wermter

Abstract—During visuomotor tasks, robots must compensate
for temporal delays inherent in their sensorimotor processing
systems. Delay compensation becomes crucial in a dynamic
environment where the visual input is constantly changing, e.g.,
during the interacting with a human demonstrator. For this purpose, the robot must be equipped with a prediction mechanism
for using the acquired perceptual experience to estimate possible
future motor commands. In this paper, we present a novel
neural network architecture that learns prototypical visuomotor
representations and provides reliable predictions on the basis of
the visual input. These predictions are used to compensate for the
delayed motor behavior in an online manner. We investigate the
performance of our method with a set of experiments comprising
a humanoid robot that has to learn and generate visually
perceived arm motion trajectories. We evaluate the accuracy
in terms of mean prediction error and analyze the response of
the network to novel movement demonstrations. Additionally, we
report experiments with incomplete data sequences, showing the
robustness of the proposed architecture in the case of a noisy
and faulty visual sensor.
Index Terms—Self-organized networks, hierarchical learning,
motion prediction

I. I NTRODUCTION

R

EAL-TIME interaction with the environment requires
robots to adapt their motor behavior according to perceived events. However, each sensorimotor cycle of the robot
is affected by an inherent latency introduced by the processing
time of sensors, transmission time of signals, and mechanical
constraints [1][2][3]. Due to this latency, robots exhibit a
discontinuous motor behavior which may compromise the
accuracy and execution time of the assigned task.
For social robots, delayed motor behavior makes humanrobot interaction (HRI) asynchronous and less natural. Synchronization of movements during HRI may increase rapport
and endow humanoid robots with the ability to collaborate
with humans during daily tasks [4]. A possible solution to the
sensorimotor latency is the application of predictive mechanisms which accumulate information from robot’s perceptual
and motor experience and learn an internal model which
estimates possible future motor states [5][6]. The learning of
these models in an unsupervised manner and their adaptation
throughout the acquisition of new sensorimotor information
remains an open challenge.
Latencies between perception and possible motor behavior occur in human beings [7] as well. Such discrepancies
Authors are with the Department of Informatics, Knowledge
Technology, University of Hamburg, Vogt-Koelln-Strasse 30, 22527
Hamburg,
Germany
e-mail:
{mici, parisi, wermter}

@informatik.uni-hamburg.de
Preprint submitted to IEEE Transactions on Cognitive and Developmental
Systems (TCDS)

are caused by neural transmission delays and are constantly
compensated by predictive mechanisms in our sensorimotor
system that account for both motor prediction and anticipation
of the target movement. Miall et al. [8] have proposed that
the human cerebellum is capable of estimating the effects
of a motor command through an internal action simulation
and a prediction model. Furthermore, there are additional
mechanisms for visual motion extrapolation which account
for the anticipation of the future position and movement of
the target [9]. Not only do we predict sensorimotor events in
our everyday tasks, but we also constantly adjust our delay
compensation mechanisms to the sensory feedback [10] and
to the specific task [11].
Recently, there has been a considerable growth of learningbased prediction techniques, which mainly operate in a “learn
then predict” approach, i.e., typical motion patterns are extracted and learned from training data sequences and then
learned patterns are used for prediction [2][12][13][14]. The
main issue with this approach is that the adaptation of the
learned models is interrupted by the prediction stage. However,
it is desirable for a robot operating in natural environments
to be able to learn incrementally, i.e., over a lifetime of
observations, and to refine the accumulated knowledge over
time. Therefore, the development of learning-based predictive
methods accounting for both incremental learning and predictive behavior still need to be fully investigated.
In this work, we propose a novel architecture that learns
sensorimotor patterns and predicts the future motor states in
an on-line manner. We evaluate the architecture in the context
of an imitation task in an HRI scenario. In this scenario,
body motion patterns demonstrated by a human demonstrator
are mapped to trajectories of robot joint angles and then
learned for subsequent imitation by a humanoid robot. We
approach the demonstration of the movements through motion
capture with a depth sensor, which provides us with reliable
estimations and tracking of a 3D human body pose. Thus,
the three-dimensional joint positions of the skeleton model
constitute the input to the architecture. The learning module
captures spatiotemporal dependencies through a hierarchy of
Growing When Required (GWR) [15] networks, which has
been successfully applied for the classification of human
activities [16][17]. The learning algorithm processes incoming robot joint angles and progressively builds a dictionary
of motion segments. Finally, an extended GWR algorithm,
implemented at the last layer of our architecture, approximates
a prediction function and utilizes the learned motion segments
for predicting forthcoming motor commands.
We evaluate our system on a dataset of three subjects performing 10 arm movement patterns. We study the prediction

2

accuracy of our architecture while being continuously trained.
Experimental results show that the proposed architecture can
adapt quickly to an unseen pattern and can provide accurate
predictions albeit continuously incorporating new knowledge.
Moreover, we show that the system can maintain its performance even when training takes place with missing sensory
information.
II. R ELATED W ORK

growing extensions of the SOM such as the GWR algorithm
[15]. The GWR algorithm has the advantage of a nonfixed, but
varying topology that requires no specification of the number
of neurons in advance. Moreover, the prediction capability of
the self-organizing approaches in the case of multidimensional
data sequences has not been thoroughly analyzed in the literature. In the current work, we present experimental results in
the context of a challenging robotic task, whereby real-world
sensorimotor sequences have to be learned and predicted.

A. Motion prediction
Motion analysis and prediction are an integral part of robotic
platforms that counterbalance the imminent sensorimotor latency. Well-known methods for tracking and prediction are the
Kalman Filter models, as well as their extended versions which
assume non-linearity of the system, and the Hidden Markov
Models (HMM). Kalman filter-based prediction techniques
require a precise kinematic or dynamic model that describes
how the state of an object evolves while being subject to a
set of given control commands. HMMs describe the temporal
evolution of a process through a finite set of states and transition probabilities. Predictive approaches based on dynamic
properties of the objects are not able to provide correct longterm predictions of human motion [18] due to the fact that
human motion also depends on other higher-level factors than
kinematic constraints, such as plans or intentions.
There are some alternatives to approaches based on probabilistic frameworks in the literature and neural networks are
probably the most popular ones. Neural networks are known to
be able to learn universal function approximations and thereby
predict non-linear data even though dynamic properties of a
system or state transition probabilities are not known [19][3].
For instance, Multilayer Perceptrons (MLP) and Radial Basis Function (RBF) networks as well as Recurrent Neural
Networks have found successful applications as predictive
approaches [12][20][13][2]. A subclass of neural network
models, namely the Self-Organizing Map (SOM) [21], is able
to perform local function approximation by partitioning the
input space and learning the dynamics of the underlying
process in a localized region. The advantage of the SOMbased methods is their ability to achieve long-term predictions
at much less expensive computational time [22].
Johnson and Hogg [23] first proposed the use of multilayer
self-organizing networks for the motion prediction of a tracked
object. Their model consisted of a bottom SOM layer learning
to represent the object states and the higher SOM layer learning motion trajectories through the leaky integration of neuron
activations over time. Similar approaches were proposed later
by Sumpter and Bulpitt [24] and Hue et al. [25], who modeled
time explicitly by adding lateral connections between neurons
in the state layer, obtaining performances comparable to that
of the probabilistic models.
Several other approaches use SOMs extended with temporal associative memory techniques [20], e.g., associating to
each neuron a linear Autoregressive (AR) model [26][27].
A drawback which is common to these approaches is their
assumption of knowing a priori the number of movement
patterns to be learned. This issue can be mitigated by adopting

B. Incremental learning of motion patterns
In the context of learning motion sequences, an architecture
capable of incremental learning should identify unknown patterns and adapt its internal structure in consequence. This topic
has been the focus of a number of studies on programming
by demonstration (PbD) [28]. Kulić et al. [29] used HMMs
for segmenting and representing motion patterns together with
a clustering algorithm that learns in an incremental fashion
based on intra-model distances. In a more recent approach, the
authors organized motion patterns as leaves of a directed graph
where edges represented temporal transitions [30]. However,
the approach was built upon automatic segmentation which
required observing the complete demonstrated task, thereby
becoming task-dependent. A number of other works have also
adapted HMMs to the problem of incremental learning of
human motion [31][32][33][34]. The main drawback of these
methods is their requirement for knowing a priori the number
of motions to be learned or the number of Markov models
comprising the learning architecture.
Ogata et al. [35] proposed a model that considers the
case of long-term incremental learning. In their work, a
recurrent neural network was used to learn a navigation task
in cooperation with a human partner. The authors introduced
a new training method for the recursive neural network in
order to avoid the problem of memory corruption during new
training data acquisition. Calinon et al. [36] showed that the
Gaussian Mixture Regression (GMR) technique can be successfully applied for encoding demonstrated motion patterns
incrementally through a Gaussian Mixture Model (GMM)
tuned with an expectation-maximization (EM) algorithm. The
main limitation of this method is the need to specify in advance
the number and complexity of tasks in order to find an optimal
number of Gaussian components. Therefore, Khansari-Zadeh
and Billard [37] suggested a learning procedure capable of
modelling demonstrated motion sequences through an adaptive
GMM. Cederborg et al. [38] suggested to perform a local
partitioning of the input space through kd-trees and training
several local GMR models.
However, for high-dimensional data, partitioning of input
space in a real-time system requires additional computational
time. Regarding this issue, it is convenient to adopt selforganized network-based methods that perform in parallel partitioning of the input space through the creation of prototypical
representations as well as the fitting of necessary local models.
The application of a growing self-organizing network, such
as the GWR, allows for the learning of prototypical motion
patterns in an incremental fashion [39].

3

GWR2

m(t)

Vision module

P-GWR

Predict

Compute Angles

GWR1

Visuomotor learning

Nao actuators
m(t+d)

Delay
compensation

Robot control

Fig. 1: Overview of the proposed system for the sensorimotor delay compensation during an imitation scenario. The vision
module acquires motion from a depth sensor and estimates the three-dimensional position of joints. Shoulder and elbow angle
values are extracted and fed to the visuomotor learning algorithm. The robot then receives predicted motor commands processed
by the delay compensation module.

III. M ETHODOLOGY
A. Overview
The proposed learning architecture consists of a hierarchy of
GWR networks [15] which process input data sequences and
learn inherent spatiotemporal dependencies (Fig. 1). The first
layer of the hierarchy learns a set of spatial prototype vectors
which will then encode incoming data samples. The temporal
dependence of the input data is captured as temporally ordered
concatenations of consecutively matched prototypes which
become more complex and of higher dimensionality when
moving towards the last layer. When body motion sequences
are provided, the response of the neurons in the architecture
resembles the neural selectivity towards temporally ordered
body pose snapshots in the human brain [40]. This simple, but
effective data sequence representation is also convenient in a
prediction application due to implicitly mapping past values to
the future ones. The concatenation vector is composed of two
parts: the first part carries information about the input data at
previous time steps and the second part concerns the desired
output of this mapping.
The evaluation of the predictive capabilities of the proposed
architecture for compensating robot sensorimotor delay will
be conducted in an imitation scenario where a simulated Nao
robot imitates a human demonstrator while compensating for
the sensorimotor delay in an on-line manner.
B. Learning with the GWR algorithm
The GWR network is composed of neurons and edges
that link the neurons forming neighborhood relationships. The
network starts with a set of two neurons randomly initialized
and, during the learning iterations, both neurons and edges can
be created, updated, or removed. At each learning iteration,
t, the first and the second best-matching units (BMUs) are
computed as the neurons with the smallest Euclidean distance
with the input sample x(t). The activity of the network, a(t),
is computed as a function of the Euclidean distance between
the weight vector of the first BMU, wb , and the input data
sample x(t):
a = exp(−||x(t) − wb ||).

(1)

Whenever the activity of the network is smaller than a given
threshold aT , a new neuron is added with a weight vector:
wr = 0.5 · (x(t) + wb )

(2)

The activation threshold parameter, aT , modulates the amount
of generalization, i.e., the largest discrepancy between an incoming stimulus and its BMU. Edges are created between the
first and the second BMUs. An edge ageing mechanism takes
care of removing rarely activated edges, i.e., edges exceeding
the age threshold, and unconnected neurons consequently. In
this way, representations of data samples that have been seen
in the far past are eliminated leading to an efficient use
of available resources from the lifelong learning perspective.
Moreover, a firing rate mechanism that measures how often
each neuron has been activated by the input leads to a sufficient
training before new neurons are created. The firing rate is
initially set to one and than decreases every time a neuron
and its neighbors are activated in the following way:
∆hi = ρi · κ · (1 − hi ) − ρi ,

(3)

where ρi and κ are the constants controlling the behaviour of
the decreasing function curve. Typically, the ρ constant is set
higher for the BMU (ρb ) than for its topological neighbors
(ρn ). Given an input data sample x(t), if no new neurons are
added, the weights of the first BMU and its neighbors are
updated as follows:
∆wi = i · hi · (x(t) − wi ),

(4)

where i and hi are the constant learning rate and the firing
counter variable respectively. The learning of the GWR algorithm stops when a given criterion is met, e.g., a maximum
network size or a maximum number of learning epochs.
C. Temporal sequence representations
GWR networks do not encode temporal relationships of
the input. This limitation has been addressed by different
extensions, such as hierarchies of GWRs augmented with a
window in time memory or recurrent connections [41][16][17].
Since our goal is to both encode data sequences and generate
them, we adopt the first approach, in which the relevant
information regarding data samples in a window of time is
always explicitly available. Moreover, the use of a hierarchy
instead of a shallow architecture allows for the encoding of
multiple time-varying sequences through prototype neurons
which can be reused for representing different sequences.
The GWR learning mechanism described in Section III-B
is employed for training the first two layers of the proposed
architecture, namely the GWR1 and GWR2 networks. The

4

o(t)

z-1

z-1

z-1

z-1

this input-output mapping and apply this learning algorithm to
the last layer of our architecture, i.e., to the P-GWR network.
The input samples fed to the P-GWR network are concatenations of the temporally ordered BMUs from the preceding
layer (Eq. 5). We divide the input into two parts: the regressor,
xin (t), and the desired output, i.e., the value to predict xout (t):

z-1

wb(t)

xin (t) = x(t) ⊕ x(t − 1) ⊕ ... ⊕ x(t − p + 1),

x(t)

xout (t) = x(t + 1),

Fig. 2: Schematic description of the output computation for the
first two layers of our learning architecture (not all neurons
and connections are shown). Given an input data sample x(t),
the weight of the best-matching unit is concatenated with the
weights of the previously activated neurons (depicted in fading
yellow) in order to compute the output o(t). The length of the
concatenation vector is a pre-defined constant τ (τ = 3 in this
example). The z −1 blocks denote the time delay.
output of both networks is computed as the concatenation of
the weights of consecutively activated neurons within a predefined temporal window τ (see Fig.2):
o(t) = wb(t) ⊕ wb(t−1) ⊕ ... ⊕ wb(t−τ +1) ,

(5)

where ⊕ denotes the concatenation operation. Moving up in
the hierarchy, the output o(t) will represent the input for the
GWR network of the higher layer. In this way, the GWR1
network learns a dictionary of prototypes of the spatial body
configurations domain, while the GWR2 and P-GWR networks
encode body motion patterns accumulated over a short and a
longer time period respectively.
Following this hierarchical learning scheme, we adapt the
GWR neuron elimination strategy in a layer-wise manner to
address the problem of forgetting rarely encountered, but still
relevant information. For instance, at the level of the GWR1
network, which represents spatial body configurations, it is
more probable that rarely seen input data samples are due
to sensory noise. Therefore, we can set a lower edge age
threshold here, leading to a higher rate of neuron elimination.
For the GWR2 and P-GWR networks, on the other hand, rarely
seen data samples are most probably due to sub-sequences
encountered in the far past. We can set a higher edge age
threshold so that neurons are removed more rarely.
D. The Predictive GWR algorithm
The problem of one-step-ahead prediction can be formalized as a function approximation problem. Given a multidimensional time series denoted by {y(t)}, the function approximation is of the form:
ŷ(t + 1) = fˆ (y(t), y(t − 1), ..., y(t − (p − 1))|Θ) ,

(6)

where the input of the function, or regressor, has an order of
regression p ∈ Z+ , with Θ denoting the vector of adjustable
parameters of the model and y(t + 1) is the predicted value.
In other words, the prediction function maps the past p input
values to the observed value y(t + 1) directly following them.
We adapt the GWR learning algorithm in order to implement

(7)

with p denoting the maximum index of the past values. Each
neuron of the P-GWR network will then have two weight
vectors which we will call the input win and the output wout
weight vectors. During training, the input weight vector will
learn to represent the input data regressor and the output
weight vector will represent the corresponding predicted value.
This learning scheme has been successfully applied to the
Vector-Quantized Temporal Associative Memory (VQTAM)
model [20], shown to perform well on tasks such as time series
prediction and predictive control [42].
The learning procedure for the Predictive GWR algorithm
resembles the original GWR with a set of adaptations for
temporal processing. During training, the first and the second
best-matching units, b and s, at time step t are computed
considering only the regressor part of the input:
b = arg min ||xin (t) − win
n ||,
n∈A

s = arg min ||xin (t) − win
n ||,

(8)

n∈A/{b}

win
n

is the input weight vector of the neuron n and A
where
is the set of all neurons. However, for the weight updates both
xin (t) and xout (t) are considered:
in
in
∆win
i = i · ci · (x (t) − wi ),

∆wout
= i · ci · (xout (t) − wout
i
i ),

(9)

with the learning rates 0 < i < 1 being higher for the
BMUs (b ) than for the topological neighbors (n ). This
learning mechanism guaranties that the regressor space is
vector-quantized while the prediction error is minimized at
each learning iteration.
The Predictive GWR algorithm operates differently from
supervised prediction approaches. In the latter, the prediction
error signal is the factor that guides the learning, whereas in
the Predictive GWR the prediction error is implicitly computed and minimized without affecting the learning dynamics.
Moreover, unlike the SOM-based VQTAM model, the number
of input-output mapping neurons, or local models, is not predefined nor fixed, but instead adapts to the input data.It should
be noted that overfitting does not occur with the growth of
the network due to the fact that neural growth decreases the
quantization error which is proportional to the prediction error.
E. Predicting sequences
Given an input regressor at time step t, xin (t), the one-stepahead estimate is defined as the output weight vector of the
P-GWR best-matching unit:
ŷ(t + 1) = wout
b

(10)

5

where b is the index of the best-matching unit (Eq. 8). In
the case that the desired prediction horizon is greater than
1, the multi-step-ahead prediction can be obtained by feeding
back the predicted values into the regressor and computing
Eq. 8 recursively until the whole desired prediction vector
is obtained. An alternative to the recursive prediction is the
vector prediction which is obtained by increasing the dimension of the xout vector with as many time steps as the desired
prediction horizon h. Thus, the input regressor and the desired
output would have the following form:
xin (t) = x(t) ⊕ x(t − 1) ⊕ ... ⊕ x(t − p + 1),
xout (t) = x(t + 1) ⊕ x(t + 2) ⊕ ... ⊕ x(t + h),

(11)

where p denotes the index of the past values. The same
dimensionality should be defined for the weight vectors win
and wout of the P-GWR neurons as well. This solution requires
the training of the architecture with this setting of the weights.
IV. T HE IMITATION SCENARIO
A. Overview
The scenario consists of a Nao robot incrementally learning
a set of visually demonstrated body motion patterns and
directly imitating them while compensating for the sensorimotor delay. We showcase the predictive capabilities of the
proposed architecture in the context of an imitation scenario
motivated by the fact that it can potentially imply behaviour
synchronization in the human-robot interaction. For humans,
the synchronization of behavior is a fundamental principle for
motor coordination and is known to increase rapport in daily
social interaction [4]. Psychological studies have shown that
during conversation humans tend to coordinate body posture
and gaze direction [43]. This phenomenon is believed to be
connected to the mirror neuron system [44], suggesting a
common neural mechanism for both motor control and action
understanding. Interpersonal coordination is an integral part
of human interaction, thus we assume that applied to HRI
scenarios it may promote the social acceptance of robots.
A schematic description of the proposed system is given in
Fig. 1. The user’s body motion is the input into the model and
the motor commands for the robot are obtained by mapping
the user’s arm skeletal structure to the robot’s arm joint
angles. This direct motion transfer allows for a simple, yet
compact representation of the visuomotor states that does not
require the application of computationally expensive inverse
kinematics algorithms. Demonstrated motion trajectories are
learned incrementally by training our hierarchical GWR-based
learning algorithm. This allows for extracting prototypical
motion patterns which can be used for the generation of robot
movements as well as prediction of future target trajectories in
parallel. In this robot task, the prediction of future visuomotor
states is necessary to compensate for the sensory delay introduced by the vision sensor camera, the signal transmission
delay as well as the robot’s motor latency during motion
generation. The simulated Nao robot is used as the robotic
platform for the experimental evaluation.

Fig. 3: Examples of arm movement patterns. The visual input
data, represented as three-dimensional skeleton sequences, are
mapped to the robots’ joint angles.

B. System description
A general overview of the proposed architecture is depicted
in Fig. 1. The system consists of three main modules: 1) The
vision module, which includes the depth sensor and the tracking of the 3D skeleton through OpenNI/NITE framework; 1
2) The visuomotor learning module, which receives angle
values and provides future motor commands; 3) The robot
control module, which processes motor commands and relays
them to the microcontrollers of the robot, which in our case
is a locally simulated Nao.
Our contribution is the visuomotor learning module which
performs incremental adaptation and early prediction of human
motion patterns. Although the current setup uses a simulated
environment, we will consider a further extension of the
experiments towards the real robot. Therefore, we simulate
the same amount of motor response latency as it has been
quantified in the real Nao robot, being between 30 to 40
ms [2]. This latency could be even higher due to reduced
motor performance, friction or weary hardware. Visual sensor
latency on the other hand, for an RGB and depth resolution of
640x480, together with the computation time required from the
skeleton estimation middleware can peak up to 500 ms [45].
Taking into consideration also possible transmission delays
due to connectivity issues, we assume a maximum of 600 ms
1 OpenNI/NITE:

http://www.openni.org/software

6

of overall sensorimotor latency in order to carry out experiments described in Section V.
C. Data acquisition and representation
The motion sequences were collected with an Asus Xtion
Pro camera at 30 frames per second. This type of sensor
is capable of providing synchronized color information and
depth maps at a reduced power consumption and weight,
making it a more suitable choice than a Microsoft Kinect
for being placed on our small humanoid robot. Moreover, it
offers a reliable and markerless body tracking method [46]
which makes the interface less invasive. The distance of each
participant from the visual sensor was maintained between the
sensor’s operational range, i.e., 0.8 − 3.5 meters. To attenuate
noise, we computed the median value for each body joint every
3 frames resulting in 10 joint position vectors per second [16].
We selected joint angles to represent the demonstrator’s
postures. Joint angles allow a straightforward reconstruction
of the regressed motion without applying inverse kinematics,
which may be difficult due to redundancy and leads to less
natural movements. Nao’s arm kinematic configuration differs
from the human arm in terms of degrees of freedom (DoF).
For instance, the shoulder and the elbow joints have only
two DoFs while human arms have three. For this reason, we
compute only shoulder pitch and yaw and elbow yaw and
roll from the skeletal representation by applying trigonometric
functions and map them to the Nao’s joints by appropriate
rotation of the coordinate frames. Wrist orientations are not
considered since they are not provided by the OpenNI/NITE
framework. Considering the two arms, a frame contains a total
of 8 angle values of body motion, which are given as input to
the visuomotor learning module.
V. E XPERIMENTAL R ESULTS
We conducted experiments with a set of movement patterns
that were demonstrated either with one or with both arms
simultaneously: raise arm(s) laterally, raise arm(s) in front,
wave arm(s), rotate arms in front of the body both clockwise
and counter-clockwise. Some of the movement patterns are
illustrated in Fig. 3. In total, 10 different motion patterns were
obtained, each repeated 10 times by three participants (one
female and two male) who were given no explicit indication of
the purpose of the study nor instructions on how to perform the
arm movements. In total, we obtained 30 demonstrations for
each of the pattern. We first describe the incremental training
procedure, then we assess and analyze in details the prediction
accuracy of the proposed learning method. We focus on the
learning capabilities of the method while simulating a possible
recurring malfunctioning of the visual system leading to loss
of entire data chunks. We conclude with a model for choosing
the optimal predicted value for a system with a variable delay.

TABLE I: Training parameters for each GWR network in
our architecture for the incremental learning of sensorimotor
patterns.
Parameter
Activation Threshold
Firing Threshold
Learning rates
Firing counter behavior
Maximum edge age
Training epochs

Value
aT = 0.98
fT = 0.1
b = 0.1, n = 0.01
ρb = 0.3, ρn = 0.1, κ = 1.05
{100, 200, 300}
50

sees all networks composed of two neurons with random
weight vectors, i.e., carrying no relevant information about
the input data. The GWR1 network is trained in order to
perform spatial vector quantization. Then the current sequence
is gradually encoded as a trajectory of activated neurons as
described in Eq. 5 and given in input to the GWR2 network
of the second layer. The same procedure is then repeated
for the second layer until the training of the full architecture
is performed. The learning of the 30 demonstrations of one
motion pattern from all three subjects constitutes an epoch.
The learning parameters used throughout our experiments
are listed in Table I. The parameters have been empirically
fine-tuned by considering the learning factors of the GWR
algorithm. The firing threshold fT and the parameters ρb , ρn ,
and κ define the decrease function of the firing counter (Eq. 3)
and were set in order to have at least seven trainings of a bestmatching unit before inserting a new neuron. It has been shown
that increasing the number of trainings per neuron does not
affect the performance of a GWR network significantly [15].
The learning rates are generally chosen to yield a faster
training for the BMUs than for their topological neighbors.
However, given that the neurons’ decreasing firing counter
modulates the weights update (Eq. 4), an optimal choice of the
learning rates has little impact on the architecture’s behaviour
in the long run. The training epochs were chosen by analysing
the converging behaviour of the composing GWR networks in
terms of neural growth.
The activation threshold parameter aT , which modulates the
number of neurons, has the largest impact on the architecture’s behaviour. The closer to 1 this value is, the greater
is the number of neurons created and the better is the data
reconstruction during the prediction phase. Therefore, we kept
aT relatively high for all GWR networks. We provide an
analysis of the impact of this parameter on the prediction
performance of our architecture in Section V-B3. Finally, the
maximum edge age parameter, which modulates the removal
of rarely used neurons, was set increasingly higher with each
layer. As discussed in Section III-C, the neurons activated less
frequently in the lower layer may be representing noisy input
data samples, whereas in higher layers the neurons capture
spatiotemporal dependencies which may vary significantly
from sequence to sequence.

A. Hierarchical training
The training of our architecture is carried out in an online manner. This requires that the GWR networks are trained
sequentially one data sample at a time. The initialization phase

B. Predictive behavior
We now assess the predictive capabilities of the proposed
method while the training is occurring continuously. Consid-

7

Nr. of Frames

Fig. 4: Behavior of the proposed architecture during training on an unseen sequence demonstrated by one subject (the sequence
is presented three times to the network). From top to bottom illustrated are: the skeleton model of the visual sequence, the
ground truth data of robot joint angles, the values predicted from the network, and the Euclidean distance between predicted
values and the ground truth over time (red dashed line indicating the statistical trend).

ering that the data sample rate is 10 fps (see Section IV-C), we
set a prediction horizon of 6 frames in order to compensate
for the estimated delay of 600 ms.
1) How fast does the architecture adapt to a new sequence?: An example of the on-line response of the architecture is shown in Fig. 4. We observed that, except in cases
of highly noisy trajectories, the network adapted to an unseen
input already after a few video frames, e.g., ≈ 100 frames
which correspond to 10 seconds of the video sequence, and
refined its internal representation after three presentations of
the motion sequence demonstrated by one subject, i.e., after
30 demonstrations. This can be seen by the statistical trend of
the prediction error.
2) Behaviour analysis and prediction performance during
incremental learning: We presented the movement sequences
one at a time and let the architecture train for 50 epochs
on each new sequence. The training phase was in total of
500 epochs for the whole dataset. Then, we re-ran the same
experiment by varying the presentation order of the sequences
and report the results averaged across all trials. In this way,
the behavior analysis does not depend on the order of the data
given during training. We analyzed the cumulative prediction
error (C.P.E) of the model by computing the mean squared
error (MSE) over all movement sequences learned up to each
training epoch. For comparison, we also computed the MSE
between the values predicted by the model and the sensory
input after being processed by the GWR1 and the GWR2 net-

works. We refer to this performance measure as the prediction
error (P.E.) since it evaluates directly the prediction accuracy
of the P-GWR network while removing the quantization error
propagated from the first two layers.
The flow of the overall MSE during training and the neural
growth of the GWR networks composing the architecture are
reported in Fig. 5. The moment in which we introduce a
new motion sequence is marked by a vertical dashed line. As
expected, the cumulative prediction error increases as soon
as a new sequence is introduced (leading to the high peaks
in Fig. 5.a.), for then decreasing immediately. However, the
error does not grow but stays constant even though new
knowledge is being added every 50 learning epochs. This is
a desirable feature for an incremental learning approach. In
Fig. b., we observe that with the introduction of a new motion
sequence there is an immediate neural growth of the three
GWR networks followed by the stabilisation of the number of
neurons indicating a fast convergence. This neural growth is
an understandable consequence of the fact that the movement
sequences are very different from each other. In fact, the
GWR1 network, performing quantization of the spatial domain,
converges to a much lower number of neurons, whereas the
higher layers, namely the GWR2 and the P-GWR network,
have to capture a high variance of spatiotemporal patterns.
However, the computational complexity of a prediction step is
O(n), where n is the number of neurons. Thus the growth of
the network does not introduce significant computational cost.

Prediction MSE

8

(a)

0.12
0.11
0.10
0.09
0.08
0.07
0.06
0.05
0.04
0

1000

2000

3000 4000 5000
Number of Neurons

6000

7000

8000

Fig. 6: Prediction mean squared error (MSE) versus the
number of neurons in the P-GWR network.

as no surprise since producing accurate long-term predictions
is a challenging task when dealing with human-like motion
sequences. However, it seems that in average the error does
not grow linearly but remains under 0.25 radians.
(b)

C. Learning with missing sensory data
Fig. 5: (a) The cumulative prediction error (C.P.E) averaged
over all learned sequences up to each learning epoch (in blue)
and the prediction error (P.E.) computed between the predicted
sequence and the sequence represented by the architecture (in
red), (b) Average and standard deviation of the neural growth
of the three GWR networks during learning.

3) Impact of the activation threshold: In the described
experiments, we set a relatively high activation threshold
parameter aT which led to a continuous growth of the GWR
networks. Thus, we further investigated how a decreased
number of neurons in the P-GWR network would affect
the overall prediction error. For this purpose, we fixed the
weight vectors of the first two layers after having been
trained on the entire dataset, and ran multiple times the
incremental learning procedure on the P-GWR network, each
time with a different activation threshold parameter aT ∈
{0.5, 0.55, 0.6, ..., 0.9, 0.95, 0.99}. We observed that a lower
number of neurons, obtained through lower threshold values,
led to quite high values of the mean squared error (Fig. 6).
However, due to the hierarchical structure of our architecture,
the quantization error can be propagated from layer to layer. It
is expected that similar performances can be reproduced with
a lower number of neurons in the P-GWR network when a
lower quantization error is obtained in the preceding layers.
4) Sensitivity to the prediction horizon: We now take the
architecture trained on the whole dataset and evaluate its
prediction accuracy while increasing the prediction horizon
up to 20 frames which correspond to 2s of a video sequence.
For achieving multi-step-ahead prediction, we compute the
predicted values recursively as described in Section III-E. In
Fig. 7, we report the mean absolute error and the standard
deviation in radians in order to give a better idea of the error
range. The results show a relatively high magnitude of error for
prediction horizons bigger than 10 frames. This should come

In the following set of experiments, we analyze how the
predictive performance of the network changes when trained
on input data produced by a faulty visual sensor. We simulate
an occurring loss of entire input data chunks in the following
way: during the presentation of a motion pattern, we randomly
choose video frames where a whole second of data samples
(i.e., 10 frames) is eliminated. The network is trained for
50 epochs on a motion sequence, each time with a different
missing portion of information. We repeat the experiment
thereby increasing the occurrence of this event in order to
compromise up to 95% of the data and see how much the
overall prediction error increases. Results are averaged over
epochs and are presented in Fig. 8. As it can be seen, the
prediction MSE stays almost constant up to 30% of data loss.
This means that the network can still learn and predict motion
sequences even under such circumstances.
D. Compensating a variable delay
Experimental results reported so far have accounted for
compensating a fixed time delay which has been measured
empirically by generating motor behavior with the real robot.
However, the proposed architecture can also be used when the
delay varies due to changes in the status of the hardware. In
this case, given the configuration of the robot at time step t
in terms of joint angle values Jξ (t), where ξ is the time delay
estimation, the optimal predicted angle values to execute in
the next step can be chosen in the following way:
P ∗ = arg min ||Jξ (t) − P (t + i)||,

(11)

i∈[0,h]

where P (t+i) are the predictions computed up to a maximum
h of the prediction horizon.
The application of this prediction step requires a method for
the estimation of the time-delay ξ, which is out of the scope
of this work. Current time-delay estimation techniques mainly
cover constant time delays, random delay with a specific noise

9

showed that a possible system malfunction causing loss of data
samples has a relatively low impact on the overall performance
of the system.
B. Growing self-organization and hierarchical learning

Fig. 7: Mean absolute error (in radians) for increasing values
of prediction horizons (expressed in frames). In our case, 20
frames correspond to 2 seconds of a video sequence.

Fig. 8: Prediction MSE averaged over 50 epochs of training
on each motion pattern. For up to 30% of data loss the MSE
does not grow linearly but rather stays almost constant. From
this point on, the increasing percentage of data loss leads to
an inevitable growth of the prediction error.

characteristic, or restricted dynamic time delay [47], which
nonetheless do not address uncertainty affecting real-world
robot applications. Computational models inspired by biology
have also been proposed for the time-delay estimation [47].
However, these models assume knowledge of the sensorimotor
dynamics.
VI. D ISCUSSION
A. Summary
In this paper, we presented a self-organized hierarchical
neural architecture for sensorimotor delay compensation in
robots. In particular, we evaluated the proposed architecture in
an imitation scenario, in which a simulated robot had to learn
and reproduce visually demonstrated arm movements. Visuomotor sequences were extracted in the form of joint angles,
which can be computed from a body skeletal representation in
a straightforward way. Sequences generated by multiple users
were learned using hierarchically-arranged GWR networks
equipped with an increasingly large temporal window.
The prediction of the visuomotor sequences was obtained
by extending the GWR learning algorithm with a mapping
mechanism of input and output vectors lying in the spatiotemporal domain. We conducted experiments with a dataset of 10
arm movement sequences showing that our system achieves
low prediction error values on the training data and can adapt
to unseen sequences in an online manner. Experiments also

The building block of our architecture is the GWR network [15], which belongs to the unsupervised competitive
learning class of artificial neural networks. A widely known
algorithm of this class is the SOM [21]. The main component
of these algorithms are the neurons equipped with weight
vectors of a dimensionality equal to the input size. Through
learning, the neurons become prototypes of the input space
while preserving the input’s topology, i.e., similar inputs are
mapped to neurons that are near to each other. In the case of
SOMs, these neurons are distributed and remain fixed in a 2D
or a 3D lattice which has to be defined a priori and requires an
optimal choice of its size. In the GWR network, the topological
structure of the neurons is dynamic and grows to adapt to the
topological properties of the input space. In this regard, the
GWR network is similar to the GNG algorithm [48], another
widely used growing self-organizing neural network. However,
the neural growth of the GWR algorithm is not constant, as
in the case of the GNG, but rather depends on how well
the current state of the network represents the input data.
Thus, from the perspective of incremental learning, the GWR
algorithm is more suitable than the GNG since new knowledge
can be added to the network as soon as new data become
available.
The hierarchical arrangement of the GWR networks
equipped with a window in time memory is appealing due
to the fact that it can dynamically change the topological
structure in an unsupervised manner and learn increasingly
more complex spatiotemporal dependencies of the data in the
input space. This allows for a reuse of the neurons during
sequence encoding, having learned prototypical spatiotemporal
patterns out of the training sequences. Although this approach
seems to be quite resource-efficient for the task of learning
visuomotor sequences, the extent to which neurons are reused
is tightly coupled with the input domain. In fact, in our
experiments with input data samples represented as multidimensional vectors of both arms’ shoulder and elbow angles,
there was little to no overlap between training sequences. This
led to a significant growth of the network with each sequence
presentation.
The parameters modulating the growth rate of each GWR
network are the activation threshold and the firing counter
threshold. The activation threshold aT establishes the maximum discrepancy between the input and the prototype neurons
in the network. The larger we set the value of this parameter,
the smaller is the discrepancy, i.e., the quantization error
of the network. The firing counter threshold fT is used to
ensure the training of recently added neurons before creating
new ones. Thus, smaller thresholds lead to more training of
existing neurons and the slower creation of new ones, favoring
again better network representations of the input. Intuitively,
the less discrepancy between the input and the network
representations, the smaller the inputs reconstruction error

10

during prediction phase. However, less discrepancy means
also more neurons. This proved to be not the main issue in
our experiments since the number of neurons did not affect
significantly the computation of the predicted values.
A limitation of the sliding time-window technique for the
encoding of temporal sequences is the high computational
cost it introduces due to the data’s higher dimensionality.
However, in our case using angles as body pose features
leads to a low-dimensional input compared to e.g. images.
So, the training with long time windows does not pose a
computational challenge. Furthermore, it has been shown that
long-term predictions based on a sliding window are more
accurate than recurrent approaches [49].
The use of joint angles as visuomotor representations may
seem to be a limitation of the proposed architecture due to the
fact that it requires sensory input and robot actions to share
the same representational space. For instance, in an object
manipulation task, this requirement is not satisfied, since the
visual feedback would be the position information given by
the object tracking algorithm. This issue can be addressed by
including both the position information and the corresponding
robot joint angles as input to our architecture. Due to the
associative nature of self-organizing networks and their capability to function properly when receiving an incomplete input
pattern, only the prediction of the object movement patterns
would trigger the generation of corresponding patterns of the
robot behavior.
C. Future work
An interesting direction for future work is the extension of
the current implementation towards the autonomous generation
of robot movements that account for both delay compensation
as well as reaching a given action goal. For this purpose, the
implementation of bidirectional Hebbian connections would
have to be investigated in order to connect the last layer of
the proposed architecture with a symbolic layer containing
action labels [16][50] and explore how such symbolic layer
can modulate the generation of the movement patterns when
diverging from the final goal.
Future studies with the real robot will address the introduction of overall body configuration constraints for learning
the perceived motion. The visual body tracking framework becomes unreliable in certain conditions, e.g., when the demonstrator is sitting or is touching objects in the background. In
these cases, the provided body configurations may become
unrealistic and cannot be mapped to the robot, or, even
worse, when mapped to the robot may lead to a hardware
break. For this reason, outlier detection mechanisms should
be investigated in order to discard these unrealistic body
configurations during training.
The imitation scenario studied in this paper was carried
out offline, i.e., the synchronization was evaluated on an
acquired data set of motion patterns. However, the successful
application of the proposed learning algorithm in providing
accurate motor commands, thereby compensating the sensorimotor delay, encourages future experiments comprising an
HRI user study in which participants will be able to teach the
motion patterns directly to the robot.

ACKNOWLEDGMENT
The authors gratefully acknowledge partial support by
the EU- and City of Hamburg-funded program ProExzellenzia 4.0, the German Research Foundation DFG under project CML (TRR 169), and the Hamburg Landesforschungsförderungsprojekt.
R EFERENCES
[1] J. Mainprice, M. Gharbi, T. Siméon, and R. Alami, “Sharing effort in
planning human-robot handover tasks,” in Proceedings of IEEE ROMAN, 2012, pp. 764–770.
[2] J. Zhong, C. Weber, and S. Wermter, “A predictive network architecture
for a robust and smooth robot docking behavior,” Paladyn, vol. 3, no. 4,
pp. 172–180, 2012.
[3] R. Saegusa, F. Nori, G. Sandini, G. Metta, and S. Sakka, “Sensory prediction for autonomous robots,” in Proceedings of IEEE-RAS Humanoid
Robots, 2007, pp. 102–108.
[4] T. Lorenz, A. Mörtl, B. Vlaskamp, A. Schubö, and S. Hirche, “Synchronization in a goal-directed task: human movement coordination with
each other and robotic partners,” in Proceedings of IEEE RO-MAN,
2011, pp. 198–203.
[5] A. Bahill, “A simple adaptive smith-predictor for controlling time-delay
systems: A tutorial,” IEEE Control systems magazine, vol. 3, no. 2, pp.
16–22, 1983.
[6] S. Behnke, A. Egorova, A. Gloye, R. Rojas, and M. Simon, “Predicting
away robot control latency,” in Robot Soccer World Cup. Springer,
2003, pp. 712–719.
[7] R. Nijhawan and S. Wu, “Compensating time delays with neural predictions: are predictions sensory or motor?” Philosophical Transactions of
the Royal Society of London A: Mathematical, Physical and Engineering
Sciences, vol. 367, no. 1891, pp. 1063–1078, 2009.
[8] R. Miall, D. J. Weir, D. M. Wolpert, and J. Stein, “Is the cerebellum a
smith predictor?” Journal of motor behavior, vol. 25, no. 3, pp. 203–216,
1993.
[9] D. Kerzel and K. R. Gegenfurtner, “Neuronal processing delays are
compensated in the sensorimotor branch of the visual system,” Current
Biology, vol. 13, no. 22, pp. 1975–1978, 2003.
[10] M. Rohde, L. C. van Dam, and M. O. Ernst, “Predictability is necessary
for closed-loop visual feedback delay adaptation,” Journal of vision,
vol. 14, no. 3, pp. 4–4, 2014.
[11] C. de la Malla, J. López-Moliner, and E. Brenner, “Dealing with delays
does not transfer across sensorimotor tasks,” Journal of Vision, vol. 14,
no. 12, p. 8, 2014.
[12] J. Mainprice and D. Berenson, “Human-robot collaborative manipulation
planning using early prediction of human motion,” in Proceedings of
IEEE/RSJ IROS, 2013, pp. 299–306.
[13] M. Ito and J. Tani, “On-line imitative interaction with a humanoid robot
using a mirror neuron model,” in Proceedings of IEEE ICRA, vol. 2,
2004, pp. 1071–1076.
[14] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye
coordination for robotic grasping with deep learning and large-scale data
collection,” arXiv preprint arXiv:1603.02199, 2016.
[15] S. Marsland, J. Shapiro, and U. Nehmzow, “A self-organising network
that grows when required,” Neural Networks, vol. 15, no. 8, pp. 1041–
1058, 2002.
[16] G. I. Parisi, J. Tani, C. Weber, and S. Wermter, “Emergence of multimodal action representations from neural network self-organization,”
Cognitive Systems Research, 2016.
[17] L. Mici, G. I. Parisi, and S. Wermter, “Recognition of transitive actions
with hierarchical neural network learning,” in Proceedings of ICANN.
Springer International Publishing, 2016, pp. 472–479.
[18] D. Vasquez, T. Fraichard, O. Aycard, and C. Laugier, “Intentional motion
on-line learning and prediction,” Machine Vision and Applications,
vol. 19, no. 5, pp. 411–425, 2008.
[19] A. M. Schaefer, S. Udluft, and H.-G. Zimmermann, “Learning long-term
dependencies with recurrent neural networks,” Neurocomputing, vol. 71,
no. 13, pp. 2481–2488, 2008.
[20] G. A. Barreto, “Time series prediction with the self-organizing map: A
review,” in Perspectives of neural-symbolic integration. Springer, 2007,
pp. 135–158.
[21] T. Kohonen, Self-organization and associative memory. Berlin: Springer
Science & Business Media, 1993.

11

[22] G. Simon, J. A. Lee, M. Cottrell, and M. Verleysen, “Forecasting the
cats benchmark with the double vector quantization method,” Neurocomputing, vol. 70, no. 13, pp. 2400–2409, 2007.
[23] N. Johnson and D. Hogg, “Learning the distribution of object trajectories
for event recognition,” Image and Vision computing, vol. 14, no. 8, pp.
609–615, 1996.
[24] N. Sumpter and A. Bulpitt, “Learning spatio-temporal patterns for
predicting object behaviour,” Image and Vision Computing, vol. 18,
no. 9, pp. 697–704, 2000.
[25] W. Hu, D. Xie, T. Tan, and S. Maybank, “Learning activity patterns using
fuzzy self-organizing neural network,” IEEE Transactions on Systems,
Man, and Cybernetics, Part B (Cybernetics), vol. 34, no. 3, pp. 1618–
1626, 2004.
[26] J. Walter, H. Riter, and K. Schulten, “Nonlinear prediction with selforganizing maps,” in Proceedings of IEEE IJCNN, 1990, pp. 589–594.
[27] J. Vesanto, “Using the SOM and local models in time-series prediction,”
in Proceedings of Workshop on Self-Organizing Maps 1997, 1997, pp.
209–214.
[28] A. G. Billard, S. Calinon, and R. Dillmann, Learning from Humans.
Springer International Publishing, 2016, pp. 1995–2014.
[29] D. Kulić, W. Takano, and Y. Nakamura, “Incremental learning, clustering
and hierarchy formation of whole body motion patterns using adaptive
hidden markov chains,” The International Journal of Robotics Research,
vol. 27, no. 7, pp. 761–784, 2008.
[30] D. Kulić, C. Ott, D. Lee, J. Ishikawa, and Y. Nakamura, “Incremental
learning of full body motion primitives and their sequencing through
human motion observation,” The International Journal of Robotics
Research, vol. 31, no. 3, pp. 330–345, 2012.
[31] W. Takano and Y. Nakamura, “Humanoid robot’s autonomous acquisition of proto-symbols through motion segmentation,” in Proceedings of
IEEE-RAS Humanoid Robots, 2006, pp. 425–431.
[32] A. G. Billard, S. Calinon, and F. Guenter, “Discriminative and adaptive
imitation in uni-manual and bi-manual tasks,” Robotics and Autonomous
Systems, vol. 54, no. 5, pp. 370–384, 2006.
[33] S. Ekvall, D. Aarno, and D. Kragic, “Online task recognition and realtime adaptive assistance for computer-aided machine control,” IEEE
Transactions on Robotics, vol. 22, no. 5, pp. 1029–1033, 2006.
[34] K. R. Dixon, J. M. Dolan, and P. K. Khosla, “Predictive robot programming: Theoretical and experimental analysis,” The International Journal
of Robotics Research, vol. 23, no. 9, pp. 955–973, 2004.
[35] T. Ogata, S. Sugano, and J. Tani, “Open-end human robot interaction
from the dynamical systems perspective: Mutual adaptation and incremental learning,” in Proceedings of IEA-AIE. Springer, 2004, pp. 435–
444.
[36] S. Calinon and A. Billard, “Incremental learning of gestures by imitation
in a humanoid robot,” in Proceedings of ACM/IEEE HRI, 2007, pp. 255–
262.
[37] S. M. Khansari-Zadeh and A. Billard, “Bm: An iterative algorithm
to learn stable non-linear dynamical systems with gaussian mixture
models,” in Proceedings of IEEE ICRA, 2010, pp. 2381–2388.
[38] T. Cederborg, M. Li, A. Baranes, and P.-Y. Oudeyer, “Incremental local
online gaussian mixture regression for imitation learning of multiple
tasks,” in Proceedings of IEEE/RSJ IROS, 2010, pp. 267–274.
[39] G. I. Parisi, S. Magg, and S. Wermter, “Human motion assessment in
real time using recurrent self-organization,” in Proceedings of IEEE ROMAN, Aug 2016, pp. 71–76.
[40] M. A. Giese and T. Poggio, “Neural mechanisms for the recognition of
biological movements,” Nature Reviews Neuroscience, vol. 4, no. 3, pp.
179–192, 2003.
[41] G. I. Parisi, C. Weber, and S. Wermter, “Self-organizing neural integration of pose-motion features for human action recognition,” Frontiers in
Neurorobotics, vol. 9, 2015.
[42] G. D. A. Barreto, A. F. Araújo, and H. J. Ritter, “Self-organizing feature
maps for modeling and control of robotic manipulators,” Journal of
Intelligent and Robotic Systems, vol. 36, no. 4, pp. 407–450, 2003.
[43] K. Shockley, D. C. Richardson, and R. Dale, “Conversation and coordinative structures,” Topics in Cognitive Science, vol. 1, no. 2, pp.
305–319, 2009.
[44] G. Tessitore, R. Prevete, E. Catanzariti, and G. Tamburrini, “From
motor to sensory processing in mirror neuron computational modelling,”
Biological Cybernetics, vol. 103, no. 6, pp. 471–485, 2010.
[45] M. A. Livingston, J. Sebastian, Z. Ai, and J. W. Decker, “Performance
measurements for the microsoft kinect skeleton,” in Proceedings of IEEE
VRW, 2012, pp. 119–120.
[46] J. Han, L. Shao, D. Xu, and J. Shotton, “Enhanced computer vision with
microsoft kinect sensor: A review,” IEEE Transactions on Cybernetics,
vol. 43, no. 5, pp. 1318–1334, Oct 2013.

[47] A. Sargolzaei, M. Abdelghani, K. K. Yen, and S. Sargolzaei, “Sensorimotor control: computing the immediate future from the delayed
present,” BMC Bioinformatics, vol. 17, no. 7, 2016.
[48] B. Fritzke, “A growing neural gas network learns topologies,” Advances
in Neural Information Processing Systems, vol. 7, pp. 625–632, 1995.
[49] J. Bütepage, M. Black, D. Kragic, and H. Kjellström, “Deep representation learning for human motion prediction and classification,” ArXiv
preprint arXiv:1702.07486, Feb. 2017.
[50] F. Schrodt and M. V. Butz, “Just imagine! Learning to emulate and infer
actions with a stochastic generative architecture,” Frontiers in Robotics
and AI, vol. 3, p. 5, 2016.

Luiza Mici received her Bachelor’s and Master’s degree in Computer Engineering from the University of
Siena, Italy. Since 2015, she is a research associate
and PhD candidate in the Knowledge Technology
Group at the University of Hamburg, Germany,
where she was part of the research project CML
(Crossmodal Learning). Her main research interests
include perception and learning, neural network selforganization, and bio-inspired action recognition.

German I. Parisi received his Bachelors and Masters degree in Computer Science from the University
of Milano-Bicocca, Italy. In 2017 he received his
PhD in Computer Science from the University of
Hamburg, Germany, where he was part of the research project CASY (Cognitive Assistive Systems)
and the international PhD research training group
CINACS (Cross-Modal Interaction in Natural and
Artificial Cognitive Systems). In 2015 he was a
visiting researcher at the Cognitive Neuro-Robotics
Lab at the Korea Advanced Institute of Science and
Technology (KAIST), Daejeon, South Korea. Since 2016 he is a research associate of international project Transregio TRR 169 on Crossmodal Learning in
the Knowledge Technology Institute at the University of Hamburg, Germany.
His main research interests include neurocognitive systems for human-robot
assistance, computational models for multimodal integration, neural network
self-organization, and deep learning.

Stefan Wermter is Full Professor at the University
of Hamburg and Director of the Knowledge Technology institute. He holds an MSc from the University of Massachusetts in Computer Science, and
a PhD and Habilitation in Computer Science from
the University of Hamburg. He has been a research
scientist at the International Computer Science Institute in Berkeley, US before leading the Chair in
Intelligent Systems at the University of Sunderland,
UK. His main research interests are in the fields
of neural networks, hybrid systems, neuroscienceinspired computing, cognitive robotics and natural communication. He has
been general chair for the International Conference on Artificial Neural
Networks 2014. He is an associate editor of the journals Transactions of
Neural Networks and Learning Systems, Connection Science, International
Journal for Hybrid Intelligent Systems and Knowledge and Information
Systems and he is on the editorial board of the journals Cognitive Systems
Research, Cognitive Computation and Journal of Computational Intelligence.
Currently he serves as Co-coordinator of the DFG-funded SFB/Transregio
International Collaborative Research Centre on Crossmodal Learning and is
coordinator of the European Training Network SECURE on Safe Robots.

