Ultrafast photonic reinforcement learning based on laser
chaos
Makoto Naruse1, Yuta Terashima2, Atsushi Uchida2 & Song-Ju Kim3

1

Strategic Planning Department, National Institute of Information and Communications Technology,
4-2-1 Nukui-kita, Koganei, Tokyo 184-8795, Japan

2

Department of Information and Computer Sciences, Saitama University, 255 Shimo-Okubo,
Sakura-ku, Saitama city, Saitama 338-8570, Japan

3

WPI Center for Materials Nanoarchitectonics, National Institute for Materials Science, 1-1 Namiki,
Tsukuba, Ibaraki 305-0044, Japan

* Corresponding author. Email: naruse@nict.go.jp

1

ABSTRACT
Reinforcement learning involves decision making in dynamic and uncertain environments, and
constitutes one important element of artificial intelligence (AI). In this paper, we
experimentally demonstrate that the ultrafast chaotic oscillatory dynamics of lasers efficiently
solve the multi-armed bandit problem (MAB), which requires decision making concerning a
class of difficult trade-offs called the explorationexploitation dilemma. To solve the MAB, a
certain degree of randomness is required for exploration purposes. However, pseudo-random
numbers generated using conventional electronic circuitry encounter severe limitations in
terms of their data rate and the quality of randomness due to their algorithmic foundations.
We generate laser chaos signals using a semiconductor laser sampled at a maximum rate of 100
GSample/s, and combine it with a simple decision-making principle called tug-of-war with a
variable threshold, to ensure ultrafast, adaptive and accurate decision making at a maximum
adaptation speed of 1 GHz. We found that decision-making performance was maximized with
an optimal sampling interval, and we highlight the exact coincidence between the negative
autocorrelation inherent in laser chaos and decision-making performance. This study paves the
way for a new realm of ultrafast photonics in the age of AI, where the ultrahigh bandwidth of
photons can provide new value.

2

INTRODUCTION
Physical unique attributes of photons have been utilized in information processing in the literature of
optical computing1. New photonic processing principles have recently emerged to solve complex
time-series prediction problems2-4, and issues in spatiotemporal dynamics5 and combinatorial
optimization6, which coincide with the rapid shift to the age of artificial intelligence (AI). These
novel approaches exploit the ultrahigh bandwidth attributes of photons and their enabling device
technologies2,3,6. This paper experimentally demonstrates the usefulness of ultrafast chaotic
oscillatory dynamics in semiconductor lasers for reinforcement learning, which is among the most
important elements in machine learning.
Reinforcement learning involves adequate decision making in dynamic and uncertain
environments7. It forms the foundation of a variety of applications, such as information
infrastructures8, online advertisements9, robotics10, transportation11, and Monte Carlo tree search12,
which is used in computer gaming13. A fundamental of reinforcement learning is known as the multiarmed bandit problem (MAB), where the goal is to maximize total reward from multiple slot
machines, the reward probabilities of which are unknown7,14,15. To solve the MAB, one needs to
explore better slot machines. However, too much exploration may result in excessive loss, whereas
too quick a decision, or insufficient exploration, may lead to neglect of the best machine. There is a
trade-off, referred to as the explorationexploitation dilemma7. A variety of algorithms for solving

3

the MAB have been proposed in the literature, such as -greedy14, softmax16, and upper confidence
bound17.
These approaches typically involve probabilistic attributes, especially for exploration purposes.
While the implementation and improvements of such algorithms on conventional digital computing
are important for various practical applications, understanding their limitations and investigating
novel approaches are also important based on perspectives from postsilicon computing. For example,
the pseudo-random number generation (RNG) used in conventional algorithmic approaches has
severe limitations, such as its data rate, due to the operating frequencies of digital processors (~ GHz
range). Moreover, the quality of randomness in RNG has serious limitations18. The usefulness of
photonic random processes for machine learning is also discussed by utilizing multiple optical
scattering19.
We consider that directly utilizing physical irregular processes in nature is an exciting approach
with the goal of realizing artificially constructed, physical decision-making machines20. Indeed, the
intelligence of slime moulds or amoebae, a single-cell natural organism, has been used in solution
searches, whereby complex inter-cellular spatiotemporal dynamics play a key role21. This stimulated
the subsequent discovery of a new principle of decision making strategy called tug of war (TOW),
invented by Kim et al.22,23. The principle of the TOW method originated from the observation of
slime moulds: the dynamic expanding and shrinking of their bodies while maintaining a constant
intracellular resource volume allows them to collect environmental information, and the conservation

4

of the volume of their bodies entails a nonlocal correlation within the body. The fluctuation, or
probabilistic behaviour, in the body of amoebae is important for the exploration of better solutions.
The name “TOW” is a metaphor to represent such a nonlocal correlation while accommodating
fluctuation, which enhances decision-making performance23.
This principle can be adapted to photonic processes. In past research, we experimentally
exhibited physical decision making based on near-field-mediated optical excitation transfer at the
nanoscale20,24 and with single photons25. These former studies pursued the ultimate physical
attributes of photons in terms of diffraction limit-free spatial resolutions and energy efficiency by
near-field photons26,27, and the quantum attributes of single-light quanta28. The nonlocal aspect of
TOW is directly physically represented by the wave nature of a single photon or an exciton polariton,
whereas fluctuation is also directly represented by their intrinsic probabilistic attributes. However,
the fluctuations are limited by practical limitations on the measurements and control systems (second
range in the worst case) as well as the single-photon generation rate (kHz range).
The ultrafast, high-bandwidth aspect of photons is another promising physical platform for
TOW-based decision making to complement diffraction-limit-free and low-energy near-field photon
approaches as well as quantum-level single-photon strategies. As demonstrated below, chaotic
oscillatory dynamics of lasers that contains negative autocorrelation experimentally demonstrates 1
GHz decision making. In addition to the resultant speed merit, it should be emphasized that the
technological maturity of ultrafast photonic devices allows for relatively easy and scalable system

5

implementation through commercially available photonic devices. Furthermore, the applications of
the proposed ultrafast photonics-based, and the former near-field/single-photon-based, decision
making are complementary: the former targets high-end, data centre scenarios by highlighting
ultrafast performance, whereas the latter appeals to low-energy, Internet-of-Things-related (IoT)29,
and security30 applications.
In this study, we demonstrate ultrafast reinforcement learning based on chaotic oscillatory
dynamics in semiconductor lasers31-34 that yields adaptation from zero-prior knowledge at a gigahertz
(GHz) range. The randomness is based on complex dynamics in lasers32-34, and its resulting speed is
unachievable in other mechanisms, at least through technologically reasonable means. We
experimentally show that ultrafast photonics has significant potential for reinforcement learning. The
proposed principles using ultrafast temporal dynamics can be matched to applications including an
arbitration of resources at data centres35 and high-frequency trading36, where decision making is
required at least within milliseconds, and other such high-end utilities. Scientifically, this study paves
a way toward the understandings of the physical origin of the enhancement of intelligent abilities
(which is reinforcement learning herein) when natural processes (laser chaos herein) are coupled
with external systems; this is what we call natural intelligence.
Chaotic dynamics in lasers have been examined in the literature32-34, and their applications have
exploited the ultrafast attributes of photonics for secure communication37-39, random number
generation31,40,41, remote sensing42, and reservoir computing2-4. Reservoir computing is a type of

6

neural network similar to deep learning13 that has been intensively studied to provide recognitionand prediction-related functionalities. The reinforcement learning described in this study differs
completely from reservoir computing from the perspective that neither a virtual network nor machine
learning for output weights is required. However, it should be noted that reinforcement learning is
important in complementing the capabilities of neural networks, indicating the potential for the
fusion of photonic reservoir computing with photonic reinforcement learning in future work.
Principle of reinforcement learning
For the simplest case that preserves the essence of solving the MAB, we consider a player who
selects one of two slot machines, called slot machines 1 and 2 hereafter, with the goal of maximizing
reward (known as the two-armed bandit problem). Denoting the reward probabilities of the slot
machines by Pi (i  1, 2) , the problem is to select the machine with the highest reward probability.
The amount of reward dispensed by each slot machine for a play is assumed to be the same in this
study.
The measured chaotic signal s (t ) is subjected to the threshold adjuster (TA), according to the
TOW principle. The output of the TA is immediately the decision concerning the slot machine to
choose. If s (t ) is equal to or greater than the threshold value T (t ) , the decision is made to select slot
machine 1. Otherwise, the decision is made to select slot machine 2. The reward—the win/lose
information of a slot machine play—is fed back to the TA.
The chaotic signal level s (t ) is compared with the threshold value T (t ) denoted by

7

T (t )  k  TA(t ) 

(1)

where TA(t ) is the threshold adjuster value at cycle t, TA(t )  is the nearest integer to TA(t )
rounded to zero, and k is a constant determining the range of the resultant T (t ) . In this study, we
assumed that TA(t )  takes the values  N ,   1, 0,1,  , N , where N is a natural number. Hence
the number of the thresholds is 2 N  1 , referred to as TA’s resolution. The range of T (t ) is limited
between kN and kN by setting T (t )  kN when TA(t )  is greater than N , as well as
T (t )   kN when TA(t )  is smaller than  N .

If the selected slot machine yields a reward at cycle t (in other words, wins the slot machine
play), the TA value is updated at cycle t  1 based on

TA(t  1)    TA(t ) if slot machine 1 wins
TA(t  1)    TA(t ) if slot machine 2 wins

(2)

where  is referred to as the forgetting (memory) parameter20, and  is the constant increment (in
this experiment,   1 and   0.999 ). In this study, the initial TA value was zero. If the selected
machine does not yield a reward (or loses in the slot machine play), the TA value is updated by

TA(t  1)    TA(t ) if slot machine 1 fails
TA(t  1)    TA(t ) if slot machine 2 fails

(3)

where  is the increment parameter defined below. Intuitively speaking, the TA takes a smaller
value if slot machine 1 is considered more likely to win, and a greater value if slot machine 2 is
considered more likely to earn the reward. This is as if the TA value is being pulled by the two slot
machines at both ends, which coincides with the notion of a tug of war.
8

The fluctuation, necessary for exploration, is realized by associating the TA value with the
threshold of digitization of the chaotic signal train. If the chaotic signal level s (t ) is equal to or
greater than the assumed threshold T (t ) , the decision is immediately made to choose slot machine 1;
otherwise, the decision is made to select slot machine 2. Initially, the threshold is zero; hence, the
probability of choosing either slot machine 1 or 2 is 0.5. As time elapses, the TA value shifts
(becomes positive or negative) towards the slot machine with the higher reward probability based on
the dynamics shown in Eqs. (2) and (3). We should note that due to the irregular nature of the
incoming chaotic signal, the possibility of choosing the opposite machine is not zero, which is a
critical feature of exploration in reinforcement learning. For example, even when the TA value is
sufficiently small (meaning that slot machine 1 seems highly likely to be the better machine), the
probability of the decision to choose slot machine 2 is not zero.
In TOW-based decision making, the increment parameter  in Eq. (3) is determined based
on the history of betting results. Let the number of times when slot machine i is selected in cycle t be

Si and the number of wins in selecting slot machine i be Li . The estimated reward probabilities of
slot machines 1 and 2 are given by

L
L
Pˆ1  1 , Pˆ2  2 .
S1
S2

(4)

 is then given by

 

Pˆ1  Pˆ2
.
2  ( Pˆ1  Pˆ2 )

(5)

9

The initial  is assumed to be unity, and a constant value is assumed when the denominator of Eq.
(5) is zero. The detailed derivation of Eq. (5) is shown in Ref. 23.

RESULTS
The architecture of laser chaos-based reinforcement learning is schematically shown in Fig. 1a. A
semiconductor laser is coupled with a polarization-maintaining (PM) coupler. The emitted light is
incident on a variable fibre reflector, by which a delayed optical feedback is supplied to the laser,
leading to laser chaos40. The output light at the other end of the PM coupler is detected by a highspeed, AC-coupled photodetector through an optical isolator (ISO) and an attenuator, and is sampled
by a high-speed digital oscilloscope at a rate of 100 GSample/s (a 10-ps sampling interval). The
detailed specifications of the experimental apparatus are described in the Methods section.
Figure 1b shows an example of the chaotic signal train. Figures 1c and 1d show the optical and
radio frequency (RF) spectra of laser chaos measured by optical and RF spectrum analysers,
respectively. The semiconductor laser was operated at a centre wavelength of 1547.782 nm. The
standard bandwidth31 of the RF spectrum was estimated as 10.8 GHz. Figure 1e summarizes the
histogram of the signal levels of the chaotic trains, which spanned from 0.2 to 0.2, with the level
zero at its maximum incidence, and slightly skewed between the positive and negative sides. A
remark is that the small incidence peak at the lowest measured amplitude is due to our experimental
apparatus (not by the laser), and it does not critically effect in the present study. The rounded TA

10

values, TA(t )  , are also schematically illustrated at the right-hand side of Fig. 1b and the upper side
of Fig. 1e, assuming N  10 . This means that TA(t )  ranges from  1 0 to 10 . Signal value s (t )
spans between approximately 0.2 and 0.2. The particular example of TA values shown in Figs. 1b
and 1e shows the case when the constant k in Eq. (1) is given by 0.02, so that the actual threshold
T (t ) spans from 0.2 to 0.2. In the experimental demonstration of this study, the TA and the slot

machines were emulated in offline processing, whereas online processing was technologically
feasible due to the simple procedure of the TA (Remarks are given in the Discussion section).

[EXPERIMENT-1] Adaptation to sudden environmental changes
We first solved the two-armed bandit problems given by the following two cases, where the reward
probabilities {P1 , P2 } were given by {0.8, 0.2} and {0.6, 0.4} . We assumed that the sum of the reward
probabilities P1  P2 was known prior to slot machine plays. With this knowledge,  is unity by Eq.
(5).
The slot machine was consecutively played 4,000 times, and this play was repeated 100 times.
The red and blue curves in Fig. 2a show the evolution of the correct decision rate (CDR), defined by
the ratio of the number of selections of the machines that yielded the higher reward probability at
cycle t in 100 trials, with respect to the probability combination of {0.8, 0.2} and {0.6, 0.4} . The
chaotic signal was sampled every 10 ps; hence, the total duration of the 4,000 plays of the slot
machine was 40 ns. In order to represent sudden environmental changes (or uncertainty), the reward

11

probability was forcibly interchanged every 10 ns, or every 1,000 plays (For example,
{P1 , P2 }  {0.8, 0.2} was reconfigured to {P1 , P2 }  {0.2, 0.8} ). The resolution of TA was set to 5 (or
N  2 ).

We observed that the CDR curves quickly approached unity, even after sudden changes in the
reward probabilities, showing successful decision making. The adaptation was steeper in the case of
a reward probability combination of {0.8, 0.2} than that of {0.6, 0.4} , since the difference in the
reward probability was greater in the former case ( 0.8  0.2  0.6 ) than the latter ( 0.6  0.4  0.2 ).
This meant that decision making was easier. The red and blue curves in Fig. 2b represent the
evolution of TA values ( TA(t ) ) in the case of {0.8, 0.2} and {0.6, 0.4} , respectively, where the TA
value became greater than 2 or 2; hence, it took long for the TA values to be inverted to the other
polarity following environmental change. This is the origin of some delay in the CDR in responding
to the environmental changes observed in Fig. 2a. One simple method of improving adaptation is to
limit the maximum and minimum values of TA(t ) . The forgetting parameter  is also crucial to
improving adaptation speed.
Figures 2c, 2d and 2e characterize decision-making performance dependencies with respect to
the configuration of TA. Figure 2c concerns TA resolutions demonstrated by the seven curves therein,
with corresponding TA resolutions of 5 to 255, where the adaptation was quicker in fewer
resolutions than not. Figure 2d considers TA range dependencies: while keeping the centre of the TA
value at zero and the TA resolution at 5, the three curves in Fig. 2d compare CDRs by a TA range of

12

0.1, 0.2 and 0.4. We observed that full coverage (0.4) of the chaotic signal yields the best
performance. Figure 2e shows the TA’s centre value dependencies while maintaining TA ranges of
0.4 and a resolution of 5, where we can clearly observe the deterioration of CDRs by the shift in the
centre of the TA from zero.

[EXPERIMENT-2] Adaptation from zero prior knowledge
Here we considered decision-making problems without any prior knowledge of the slot machines.
Hence, parameter  needed to be updated. The reward probabilities of the slot machines were set to
{P1 , P2 }  {0.5, 0.1} , and 50 consecutive plays were executed. The colour curves in Fig. 3a depict
CDRs at six sampling intervals of chaotic signal trains from 10 to 400 ps. Hence, the time needed to
complete 50 consecutive slot machine plays differed among these; it ranges from
10 ps  50  500 ps to 400 ps  50  20 ns . Moreover, the black curve shows the CDR obtained by

uniformly distributed pseudo-random numbers generated by the Mersenne Twister for the random
signal source, instead of experimentally observed chaotic signals. We observed that CDRs based on
chaotic signals exhibited more rapid adaptation than the uniformly distributed pseudo-random
numbers.
The most prompt adaptation, or the optimal performance, was obtained at a particular sampling
interval. Figure 3b compares CDRs at the cycle t  3 ; the sampling interval of 50 ps yielded the best
performance, indicating that the original chaotic dynamics of the laser could physically be optimized

13

such that the most prompt decision-making is realized. Indeed, the autocorrelation of the laser chaos
signal trains was evaluated as shown in Fig. 3c, its negative maximum value is taken when the time
lag is given by 5 or 5, corresponding exactly to the sampling interval of 50 ps ( 5  10 ps ). In other
words, the negative correlation of chaotic dynamics enhanced the exploration ability for decision
making. Furthermore, this finding suggests that optimal performance is obtained at the maximum
sampling rate (or data rate) by physically tuning the dynamics of the original laser chaos, which will
be an important and exciting topic for future investigation. The adaptation speed of decision making
was estimated as 1 GHz in this optimal case, where CDR was larger than 0.95 at 20 cycles with a 50ps sampling interval (20 GSamples/s) in Fig. 3a (1 GHz = (50 ps  20 cycles)1).
Furthermore, we characterized CDRs with normally distributed random numbers (referred to as
RANDN) in order to ensure that the statistical incidence patterns of the laser chaos, which were
similar to a normal distribution shown in Fig. 1e, were not the origins of the fast adaptation of the
decision making. By keeping the mean value of RANDN at zero, the standard deviation (  ) was
configured as 0.2, 0.1 and 0.01. As shown in Fig. 4, the CDRs at cycle t  3 by RANDN were
inferior to chaotic signals and the uniformly distributed random numbers (denoted by RAND).
Moreover, the CDR was evaluated by surrogating the chaotic signal time series sampled at 50 ps
intervals, which resulted in poorer performance than the original, as shown in Fig. 4. These
evaluations support the claim that laser chaos is beneficial to the performance of reinforcement
learning, in addition to its ultrafast data rate for random signals.

14

DISCUSSION
The performance enhancement of decision making by chaotic laser dynamics is demonstrated and
the impacts of negative autocorrelation is clearly suggested. Further understanding between chaotic
oscillatory dynamics and decision making is a part of important future research. The first regards to
physical insights. Toomey et al. recently showed that the complexity of laser chaos varies within the
coherence collapse region in the given system43. The level of the optical feedback, injection current
of the laser becomes an important parameter in determining the complexity of chaos, which is the
entrance to thorough insights. We also consider the use of the bandwidth enhancement technique31
with optically injected lasers to improve the adaptation speed of decision making over tens of GHz.
Meanwhile, besides negative autocorrelation inherent in laser chaos, other perspectives could address
the underlying mechanism such as diffusivity,44 Hurst exponents,45 etc.
In the experimental demonstration, the nonlocal aspect of the TOW principle was not directly,
physically relevant to the chaotic oscillatory dynamics of the lasers. By combining the chaotic
dynamics with the threshold adjustor, the nonlocal and fluctuation properties of the TOW principle
emerged, which have not been completely realized in the literature nor in our past experimental
studies24,25. Such a hybrid realization of nonlocality in TOW leads to higher likelihood of
technological implementability, and better scalability and extension to higher-grade problems.
Online post-processing can become feasible through electric circuitry, as already demonstrated for a

15

random-bit generator41,46 since the post-processing is very simple as described in Eqs. (1) to (5). For
scalability, a variety of approaches can be considered, such as time-domain multiplication, which
exploits the ultrafast attributes of chaotic lasers, and is a frequently used strategy in ultrafast photonic
systems. Introducing multi-threshold values31,41 is another simple extension of our proposed scheme.
Whole-photonic realization is an interesting issue to explore, and has already been implied by
the analysis where the time-domain correlation of laser chaos strongly influences decision-making
performances. The mode dynamics of multi-mode lasers47 are very promising for the implementation
of nonlocal properties of fully photonic systems required for decision making. Synchronization and
its clustering properties in coupled laser networks48,49 are also interesting approaches to physically
realizing the nonlocality of the TOW. These systems can automatically tune the optimal settings for
decision making (e.g., negative autocorrelation properties), which can lead to autonomous photonic
intelligence.
We also make note of the extension of the principle demonstrated in this paper to highergrade machine learning problems. The competitive MAB50 with multiple players is an exciting topic
for photonic intelligence research as it involves the so-called Nash equilibrium, and is the foundation
of such important applications as resource allocation and social optimization. Investigating the
possibilities of extending the present method and utilizing ultrafast laser dynamics for competitive
MAB is highly interesting.

16

Conclusion
We experimentally established that laser chaos provides ultrafast reinforcement learning and
decision making. The adaptation speed of decision making reached 1 GHz in the optimal case with
the sampling rate of 20 GSample/s (50-ps decision-making intervals) using the ultrafast dynamics
inherent in laser chaos. The maximum adaptation performance coincided with the negative
maximum of the autocorrelation of the original time-domain laser chaos sequences, demonstrating
the strong impact of chaotic lasers on decision making. The origin of the performance was also
validated by comparing with uniformly and normally distributed pseudo-random numbers as well as
surrogated arrangements of original chaotic signal trains. This study is the first demonstration of
ultrafast photonic reinforcement learning or decision making, to the best of our knowledge, and
paves the way for research on photonic intelligence and new applications of chaotic lasers in the
realm of artificial intelligence.

METHODS
Optical system
The laser used in the experiment was a distributed-feedback (DFB) semiconductor laser mounted on
a butterfly package with optical fibre pigtails (NTT Electronics, KELD1C5GAAA). The injection
current of the semiconductor laser was set to 58.5 mA (5.85 Ith), where the lasing threshold Ith was
10.0 mA. The relaxation oscillation frequency of the laser was 6.5 GHz. The temperature of the

17

semiconductor laser was set to 294.86 K. The laser output power was 13.2 mW. The laser was
connected to a variable fibre reflector, which reflected a fraction of light back into the laser, inducing
high-frequency chaotic oscillations of optical intensity32-34. 1.9 % of the laser output power was fed
back to the laser cavity from the reflector. The fibre length between the laser and the reflector was
4.55 m, corresponding to the feedback delay time (round trip) of 43.8 ns. Polarization-maintaining
fibres were used for all optical fibre components. The optical output was converted to an electronic
signal by a photodetector (New Focus, 1474-A, 35 GHz bandwidth) and sampled by a digital
oscilloscope (Tektronics, DPO73304D, 33 GHz bandwidth, 100 GSample/s, eight-bit vertical
resolution). The RF spectrum of the laser was measured by an RF spectrum analyser (Agilent,
N9010A-544, 44 GHz bandwidth). The optical wavelength of the laser was measured by an optical
spectrum analyser (Yokogawa, AQ6370C-20).
Data analysis
[EXPERIMENT-1] A chaotically oscillated signal train was sampled at a rate of 100 GSample/s by
9,999,994 points, which lasted approximately 10 s. As described in the main text, 4,000
consecutive plays were repeated 100 times; hence, the total number of slot machine plays was
400,000. With a 10-ps interval sampling, the initial 400,000 points of the chaotic signal were used
for the decision-making experiments. The post-processing required for 400,000 iterations of slot
machine plays was approximately 1.2 s (or 3.0 s/decision), on a normal-grade personal computer
(Panasonic, CF-SX3, 16 GB RAM, Windows 7, MATLAB R2011b).

18

[EXPERIMENT-2]
(1) Sampling methods: A chaotic signal train was sampled at 10-ps intervals with 9,999,994
sampling points. Such a train was measured 120 times. Each chaotic signal train was referred to
as chaosi , and there were 120 kinds of such trains: i  1,  ,120 . In demonstrating 10  M ps
sampling intervals, where M was a natural number ranging from 1 to 40 (namely, the sampling
intervals were 10 ps, 20 ps, , and 400 ps), we chose one of every M samples from the original
sequence.
(2) Evaluation of CDR regarding a specific chaos sequence: For every chaotic signal train

chaosi , 50 consecutive plays were repeated 200 times. Consequently, 10,000 points were used
from chaosi . Such evaluations were repeated 100 times. Hence, 1,000,000 slot machine plays
were conducted in total. These CDRs were calculated for all signal trains chaosi ( i  1,  ,120 ).
(3) Evaluation of CDR of all chaotic sequences: We evaluated the average CDR of all
chaotic signal trains ( i  1,  ,120 ) derived in (2) above, which were the results discussed in the
main text.
(4) Autocorrelation of chaotic signals: The autocorrelation was computed based on all
9,999,994 sampling points of chaosi , and was evaluated for all chaosi ( i  1,  ,120 ). The
autocorrelation demonstrated in Fig. 3c was evaluated as the average of these 120 kinds of
autocorrelations.
(5) Surrogate methods: The surrogate time series of original chaotic sequences were generated
by the randperm function in MATLAB which is based on the sorting of pseudorandom
numbers generated by Mersenne Twister.

19

Data availability. The data sets generated during the current study are available from the
corresponding author on reasonable request.

References
1. Jahns, J. & Lee, S. H. Optical Computing Hardware. (Academic Press, San Diego, 1994).
2. Larger, L., et al. Photonic information processing beyond Turing: an optoelectronic 3
implementation of reservoir computing. Opt. Express 20, 3241–3249 (2012).
3. Brunner, D., Soriano, M. C., Mirasso, C. R. & Fischer, I. Parallel photonic information
processing at gigabyte per second data rates using transient states. Nat. Commun. 4, 1364 (2013).
4. Vandoorne, K., et al. Experimental demonstration of reservoir computing on a silicon photonics
chip. Nat. Commun. 5, 3541 (2014).
5. Tsang,

M.

&

Psaltis,

D.

Metaphoric

optical

computing

of

fluid

dynamics.

arXiv:physics/0604149v1 (2006).
6. Inagaki, T. et al. A coherent Ising machine for 2000-node optimization problems. Science
10.1126/science.aah4243 (2016).
7. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (The MIT Press,
Massachusetts, 1998).
8. Awerbuch, B. & Kleinberg, R. Online linear optimization and adaptive routing. J. Comput. Syst.

Sci. 74, 97–114 (2008).

20

9. Agarwal, D., Chen, B. -C. & Elango, P. Explore/exploit schemes for web content optimization.

Proc. of ICDM 1–10 (2009). DOI: 10.1109/ICDM.2009.52
10. Kroemer, O. B., Detry, R., Piater, J. & Peters, J. Combining active learning and reactive control
for robot grasping. Robot. Auton. Syst. 58, 1105–1116 (2010).
11. Cheung, M. Y., Leighton, J. & Hover, F. S. Multi-armed bandit formulation for autonomous
mobile acoustic relay adaptive positioning. In 2013 IEEE Intl. Conf. Robot. Auto. 4165–4170
(2013).
12. Kocsis, L. & Szepesvári, C. Bandit based Monte Carlo planning. Machine Learning: ECML
(2006), LNCS 4212, 282–293 (2006). DOI: 10.1007/11871842_29
13. Silver, D., et al. Mastering the game of Go with deep neural networks and tree search. Nature
529, 484–489 (2016).
14. Robbins, H. Some aspects of the sequential design of experiments. B. Am. Math. Soc. 58, 527–
535 (1952).
15. Lai, T. L. & Robbins, H. Asymptotically efficient adaptive allocation rules. Adv. Appl. Math. 6,
4–22 (1985).
16. Daw, N., O’Doherty, J., Dayan, P., Seymour, B. & Dolan, R. Cortical substrates for exploratory
decisions in humans. Nature 441, 876–879 (2006).
17. Auer, P., Cesa-Bianchi, N. & Fischer, P. Finite-time analysis of the multi-armed bandit problem.

Machine Learning 47, 235–256 (2002).

21

18. Murphy, T. E. & Roy, R., The world’s fastest dice, Nat. Photon. 2, 714–715 (2008).
19. Saade, A., et al., Random projections through multiple optical scattering: Approximating Kernels
at the speed of light. In IEEE International Conference on Acoustics, Speech and Signal

Processing, March 20–25, 2016, Shanghai, China 6215–6219 (IEEE, 2016).
20. Kim, S. -J., Naruse, M., Aono, M., Ohtsu, M. & Hara, M. Decision Maker Based on Nanoscale
Photo-Excitation Transfer. Sci Rep. 3, 2370 (2013).
21. Nakagaki, T., Yamada, H., & Tóth, Á. Intelligence: Maze-solving by an amoeboid organism.

Nature 407, 470–470 (2000).
22. Kim, S. -J., Aono, M. & Hara, M. Tug-of-war model for the two-bandit problem:
Nonlocallycorrelated parallel exploration via resource conservation. BioSystems 101, 29–36
(2010).
23. Kim, S. -J., Aono, M. & Nameda, E. Efficient decision-making by volume-conserving physical
object. New J. Phys. 17, 083023 (2015).
24. Naruse, M., et al. Decision making based on optical excitation transfer via near-field interactions
between quantum dots. J. Appl. Phys. 116, 154303 (2014).
25. Naruse, M., et al. Single-photon decision maker. Sci. Rep. 5, 13253 (2015).
26. Pohl, D. W. & Courjon, D. Near Field Optics (Kluwer Kluwer, The Netherlands, 1993).
27. Naruse, M., Tate, N., Aono, M. & Ohtsu, M. Information physics fundamentals of
nanophotonics. Rep. Prog. Phys. 76, 056401 (2013).

22

28. Eisaman, M. D., Fan, J., Migdall, A. & Polyakov, S. V. Single-photon sources and detectors.

Rev. Sci. Instrum. 82, 071101 (2011).
29. Kato, H., Kim, S.-J., Kuroda, K., Naruse, M. & Hasegawa, M. The Design and Implementation
of a Throughput Improvement Scheme based on TOW Algorithm for Wireless LAN. In Proc. 4th

Korea-Japan Joint Workshop on Complex Communication Sciences, January 12-13, Nagano,
Japan J13 (IEICE, 2016).
30. Naruse, M., Tate, N. & Ohtsu, M. Optical security based on near-field processes at the nanoscale.

J. Optics 14, 094002 (2012).
31. Sakuraba, R., Iwakawa, K., Kanno, K. & Uchida, A. Tb/s physical random bit generation with
bandwidth-enhanced chaos in three-cascaded semiconductor lasers. Opt. Express 23, 1470–1490
(2015).
32. Soriano, M. C., García-Ojalvo, J., Mirasso, C. R. & Fischer, I. Complex photonics: Dynamics
and applications of delay-coupled semiconductors lasers. Rev. Mod. Phys. 85, 421–470 (2013).
33. Ohtsubo, J. Semiconductor lasers: stability, instability and chaos (Springer, Berlin, 2012).
34. Uchida, A. Optical communication with chaotic lasers: applications of nonlinear dynamics and

synchronization (Wiley-VCH, Weinheim, 2012).
35. Bilal, K., Malik, S. U. R., Khan, S. U. & Zomaya, A. Y. Trends and challenges in cloud
datacenters. IEEE Cloud Computing 1, 10–20 (2014).

23

36. Brogaard, J., Hendershott, T. & Riordan, R. High-frequency trading and price discovery. Rev.

Financ. Stud. 27, 2267–2306 (2014).
37. Colet, P. & Roy, R. Digital communication with synchronized chaotic lasers. Opt. Lett. 19,
2056–2058 (1994).
38. Argyris, A., et al. Chaos-based communications at high bit rates using commercial fibre-optic
links. Nature 438, 343–346 (2005).
39. Annovazzi-Lodi, V., Donati, S. & Scire, A. Synchronization of chaotic injected-laser systems
and its application to optical cryptography. IEEE J Quantum Electron. 32, 953–959 (1996).
40. Uchida, A., et al. Fast physical random bit generation with chaotic semiconductor lasers. Nat.

Photon. 2, 728–732 (2008).
41. Kanter, I., Aviad, Y., Reidler, I., Cohen, E. & Rosenbluh, M. An optical ultrafast random bit
generator. Nat. Photon. 4, 58–61 (2010).
42. Lin, F.-Y. & Liu, J.-M. Chaotic lidar. IEEE J. Sel. Top. Quantum Electron. 10, 991-997 (2004).
43. Toomey, J. P. & Kane, D. M. Mapping the dynamic complexity of a semiconductor laser with
optical feedback using permutation entropy. Opt. Express 22, 1713–1725 (2014).
44. Kim, S. -J., Naruse, M., Aono, M., Hori, H. & Akimoto, T. Random walk with chaotically driven
bias. Sci. Rep. 6, 38634 (2016).
45. Lam, W. S., Ray, W., Guzdar, P. N. & Roy, R. Measurement of Hurst exponents for
semiconductor laser phase dynamics. Phys. Rev. Lett. 94, 010602. (2005).

24

46. Honjo, T., et al. Differential-phase-shift quantum key distribution experiment using fast physical
random bit generator with chaotic semiconductor lasers. Opt. Express 17, 9053–9061 (2009).
47. Aida, T. & Davis, P. Oscillation mode selection using bifurcation of chaotic mode transitions in a
nonlinear ring resonator. IEEE J. Quantum Electron. 30, 2986–2997 (1994).
48. Nixon, M., Fridman, M., Ronen, E., Friesem, A. A., Davidson, N. & Kanter, I. Controlling
synchronization in large laser networks. Phys. Rev. Lett. 108, 214101 (2012).
49. Williams, C. R. S., Murphy, T. E., Roy, R., Sorrentino, F., Dahms, T. & Schöll, E. Experimental
observations of group synchrony in a system of chaotic optoelectronic oscillators. Phys. Rev.

Lett. 110, 064104 (2013).
50. Kim, S. -J., Naruse, M. & Aono, M. Harnessing the Computational Power of Fluids for
Optimization of Collective Decision Making. Philosophies Special Issue “Natural Computation:

Attempts in Reconciliation of Dialectic Oppositions” 1, 245–260 (2016).

Acknowledgements
This work was supported in part by the Core-to-Core Program, A. Advanced Research Networks
from the Japan Society for the Promotion of Science and Grants-in-Aid for Scientific Research from
Japan Society for the Promotion of Science.

25

Figure 1 | Architecture of photonic reinforcement learning based on laser chaos. (a)
Architecture and experimental configuration of laser chaos-based reinforcement learning. Ultrafast
chaotic optical signal is subjected to the tug-of-war (TOW) principle that determines the selection of
slot machines. ISO: optical isolator. (b) An example of chaotic signal trains sampled at 100
GSample/s. The signal level is subjected to a threshold adjustment (TA) for decision making. (c)
Optical spectrum and (d) RF spectrum of the laser chaos used in the experiment. (e) Incidence
statistics (histogram) of the signal level of the laser chaos signal.

26

Figure 2. Reinforcement learning in dynamically changing environments. (a) Evolution
of the correct decision rate (CDR) when the reward probabilities of the two slot machines are {0.8,
0.2} and {0.6, 0.4}. The knowledge that the sum of the reward probability, which is unity in these
cases, is supposed to be given. The reward probability is intentionally swapped every 10 ns in order
to represent sudden environmental changes or uncertainty. Rapid and adequate adaptation is
observed in both cases. (b) Evolution of the threshold adjuster (TA) value underlying correct
decision making. (c-e) CDR performance dependency on the setting of TA. (c) TA resolution
dependency. (d) TA range dependency. (e) TA centre value dependency.
27

Figure 3 | Reinforcement learning from zero prior knowledge. (a) Evolution of CDR with
different sampling intervals of the laser chaos signal (10, 20, 30, 40, 50, and 400 ps) and uniformly
distributed pseudo-random numbers. CDR exhibits prompt adaptation when the sampling interval is
50 ps. (b) CDR is evaluated as a function of the sampling interval from 10 ps to 400 ps, where the
maximum performance is obtained at 50 ps. (c) Autocorrelation of the laser chaos signals exhibits its
negative maximum when the time lag is 5 or 5, which exactly coincides with the fact that the
optimal adaptation is realized at 50 ps (10 ps

 5) sampling intervals.

28

Figure 4 | Comparison of learning performance in laser chaos, uniformly and
normally distributed pseudo-random numbers, and surrogate laser chaos signals.
The laser chaos sampled at 50 ps interval exhibits the best performance compared with other cases,
indicating that the dynamics of laser chaos affects reinforcement learning ability.

29

