\section{Introduction}
Metal nanoparticles (NPs) are used in a multitude of applications, but perhaps the oldest is as a colourizing agent when dispersed in a host dielectric, as in the (dichroic) Lycurgus cup\cite{Freestone2007,Barch2015}. Exposed to optical radiation, metal NPs exhibit scattering properties due to excited plasmons, which depend on their shape, size, composition and the host medium\cite{Mie,Doyle1989,Murray2007,Liz2006}.\\
Producing colours by exploiting plasmonic effects on metal nanostructures is of interest because the colours can last a long time (e.g., the Lycurgus cup), can be rendered down to the diffraction limit\cite{Kumar2012}, and can be used in any metal colouring or marking application where inks, paints or pigments should be avoided for environmental, health, cost or other reasons.
\\
Fabrication techniques for plasmonic colouring of metal surfaces include laser interference lithography (LIL)\cite{Gallinet2015}, electron beam lithography (EBL)\cite{Roberts2014,Tan2014}, ion beam lithography (IBL) or milling (IBM)\cite{Cheng2015}, and hot embossing or nanoimprint lithography (NIL) \cite{Gallinet2015,Clausen2014}. Kumar et al.\cite{Kumar2012} fabricated coloured images via EBL having a resolution as high as ≈ 100 000 pixels/inch$^{2}$. However, the production of large coloured surfaces is incompatible with the demands of low-cost mass-manufacturing. Furthermore, such processes generally require flat surfaces, and with the exception of periodic structures below the diffraction limit of visible light \cite{Clausen2014,Wu2013}, colours produced are generally angle-dependent.
Femtosecond (fs) lasers have been considered ideal for metal colourization due to their ablation characteristics, e.g., the tendency to preferentially release NPs, compared to the large clusters and chunks produced by nanosecond pulses \cite{Tillack2004,Perriere2007,Balling2013}. Guo et al. \cite{Vorobyev2013a} showed that exposing metals to fs pulses could yield highly absorptive surfaces (so called ‘black metals’), or surfaces producing a specific colour. However, only one set of laser parameters was reported for each colour \cite{Fan2014,Vorobyev2008,Fan2013b}, the colour palette was limited and, due to the underlying regular structure, in the case of femtosecond lasers, were angle-dependent \cite{Vorobyev2008c}. In addition, the low pulse energy of fs lasers restricts their use to low repetition rates, making the colouring process very time-consuming.

Picosecond (ps) lasers have lower costs and higher pulse energies. Fan et al. showed a limited colour palette on copper and each colour was associated to one set of laser parameters \cite{Fan2014}. Thermal effects for ps pulses are likely to play a role in the creation of colours, unlike femtosecond lasers, whose pulses are shorter than the thermal expansion time \cite{Nolte1997}.
Here, we report on a universal, high-throughput and deterministic process for producing a complete angle-independent colour palette composed of thousands of colours using a picosecond laser. This is achieved on unpolished noble metal surfaces, including surfaces with frosting and millimetre-scale topographical features. We demonstrate the process by colouring silver coins produced at the Royal Canadian Mint, set to be released to the general public at the end of 2016. Scanning electron microscopy (SEM) reveals that a full colour palette can be obtained by controlling the particle density on the surface.
Our experiments demonstrate that a very large set of laser parameter combinations can produce a given colour, as long as the total accumulated fluence remains the same, making the process scalable and time-efficient.
\begin{figure}[H]
\centering
\begin{tabular}{@{}p{1\linewidth}@{}}
\subfigimg[width=\linewidth]{\textbf{\textcolor{white}{}}}{Figure1abcd.png}\\
\subfigimg[width=\linewidth]{\textbf{\textcolor{white}{}}}{Figure1ef.png}
\end{tabular}
\caption{25 mm$^2$ coloured squares obtained by 1 $\mu$m changing $L_s$  (marked above) for (a) a laser fluence of $\phi=1.67$ J/cm$^{2}$ with a laser marking speed of $v=50$ mm/s, and (b) $\phi=29.75$ J/cm$^{2}$ at $v=1000$ mm/s. (c) Photograph of blue angle-independent colouring of a flat silver coin observed at various angles. (d) Hue versus total accumulated fluence for ($\begingroup\color{blue}\bullet\endgroup$) $\phi=1.12$ J/cm$^{2}$ at $v=11$ mm/s, ($\begingroup\color{red}\star\endgroup$) $\phi=1.67$ J/cm$^{2}$ at $v=50$ mm/s, ($\begingroup\color{green}\filledtriangledown\endgroup$) $\phi=2.59$ J/cm$^{2}$ at $v=100$ mm/s, ($\begingroup\color{black}\filledtriangleleft\endgroup$) $\phi=6.26$ J/cm$^{2}$ at $v=250$ mm/s, ($\begingroup\color{mycolor1}\pentagram\endgroup$) $\phi=13.23$ J/cm$^{2}$ at $v=500$ mm/s, ($\begingroup\color{cyan}\filledsquare\endgroup$) $\phi=20.58$ J/cm$^2$ at $v=750$ mm/s, and ($\begingroup\color{mycolor2}\filledtriangleright\endgroup$) $\phi=29.75$ J/cm$^2$ at $v=1000$ mm/s; Hue can be seen to make a full 360$^{\circ}$ rotation (dashed line). (e) Polar plot representation of (d) with Hue plotted azimuthally and the logarithm of the total accumulated fluence plotted radially; a full rotation in Hue can be observed with increasing total accumulated fluence.  (f) CIE diagram of all the colors obtained using the different laser parameters from (d). }
\label{fig1}
\end{figure}
To theoretically understand the colour formation, we used large-scale computational electrodynamics to simulate the scattering from periodic distributions of different sized nanoparticles, with geometrical parameters based on statistical analysis of SEM images. Simulations show that several NP arrangements can produce the same colour, similar to what is observed experimentally, and that narrow band absorption due to plasmonic resonances in heterogeneous nanoclusters are critical for colour production.

\section{Results and Discussion}
\subsection{Angle-independent laser colouring of silver}
The exposure of pure silver to different laser parameters is observed to produce vivid angle-independent colours, Fig. 1 (a,b,c). The same process was used to produce similar colour palettes on copper and gold. These colours are highly reproducible and found to depend on the density of NPs covering the metal surface.
\\
During laser ablation by raster scanning the sample surface, the number of particles re-deposited and accumulated from one line to the next is dictated by the inter-line spacing $L_s$, i.e., the spacing between two consecutive laser lines. The change in particle density with increasing distance from the laser ablated line can be seen in the supplementary material, Fig. S. 1.\\
Figs. 1 (a,b) show the colours produced on silver by raster scanning the surface, where different colours are produced by simply increasing Ls. In particular, increasing Ls from 1 to 12 $\mu$m with a laser marking speed of v = 50 mm/s results in a production rate of $\eta$ =0.05 to 0.8 mm$^2$/s (Fig. 1 (a)), while augmenting the laser marking speed to v = 1000 mm/s with Ls ranging from 1 to 25 $\mu$m (Fig. 1 (b)) results in production rates of $\eta$ =1 to 25 mm$^2$/s. The production rate is defined as $\eta$ = L$_s$v. The colour palettes in Figs. 1 (a,b) can be extended by simply reducing the line spacing between each successive line (i.e., increasing total accumulated fluence). The angle-independent colours produced on silver, Fig. 1 (c), cover the spectral and the nonspectral regions (e.g., Magenta) through a 360$^{\circ}$ rotation in Hue, Fig. 1 (d-f).
\\
The total accumulated fluence is defined as
\begin{equation}
\Phi = \phi N_{eff} =\frac{a^{2}Ef}{vL_s},
\end{equation}
where the laser fluence is given by
\begin{equation}
\phi = \frac{2E}{\pi w_o^2},
\end{equation}
the number of effective laser shots is
\begin{equation}
N_{eff}= \underbrace{\sqrt{\frac{\pi}{2}}\frac{aw_of}{v}}_\text{intra-line}\underbrace{\sqrt{\frac{\pi}{2}}\frac{aw_o}{L_s}}_\text{inter-line},
\end{equation}
and $E$ is the laser pulse energy, ωo the beam waist radius, $f$ the laser repetition rate, and $a$ a correction factor due to the larger modified region when using a higher pulse energy. This correction factor was determined from semi-logarithmic plots used to determine the laser spot size, following the procedure detailed in \cite{Jandeleit1996}. The intra-line component of $N_{eff}$ encompasses the shot overlap within a local region, where $f/v$ is the distance traveled between successive laser pulses in a single laser line. The inter-line component, in comparison, considers the geometrical overlap, $a\omega_o/L_s$, between successive laser lines. At large spacings (\textit{i.e.}, lower total accumulated fluence and $a\omega_o/L_s$ $\rightarrow$ 1) the colours are observed to converge to yellow due to the absence of overlap between consecutive laser lines. Further evidence of this accumulation process can be observed from the absence of colours, other than yellow, in the last line of each of the coloured squares. The last line does not undergo the process of particle accumulation as there are no subsequent lasered lines. In addition, in the absence of overlap and for large distances between successive lines, LIPSS structures can be observed to form next to the laser ablated lines causing the yellow colors to be angle-dependent. In the overlap case no LIPSS structures are created or they are destroyed and the surface remains relatively smooth, Fig. S. 2 in supplementary materials.
\begin{figure}[H]
\centering
\begin{tabular}{@{}p{1\linewidth}@{}}
\subfigimg[width=\linewidth]{\textbf{\textcolor{white}{}}}{Figure2.png}
\end{tabular}
\caption{Photographs of silver topographical butterfly coins (a) before and (b) after laser colouring; the topography has an overall height of $\sim$ 2 mm (surfaces as high as 4 mm were also coloured) without adjusting focus. The distinctive white areas on the butterfly are frosted ($\it{i.e}$., melted) finish; (c) optical microscope close up of the frosted regions, height of $\sim$ 1 to 2 $\mu$m. (d) Photograph of a silver eagle coin. The eagle is frosted and its surface is similar to that shown in (c). The difference in elevation is observed to create a colour gradient on the eagle wings (e) and tail (f). The eagles in (d-f) originate from different coins coloured using different laser parameters. (g) Blue Jay colouring on a flat silver coin, with permission from National Geographic. (h) Colouring of a butterfly, $\sim$ 10 mm wide, on a flat silver coin}
\label{fig2}
\end{figure}
Previous work on picosecond and femtosecond laser colouring of metals \cite{Vorobyev2013a, Fan2014} only report a single set of laser parameters to obtain a given colour (less than 10 angle-independent colours). We show that there are in fact a very large number of laser parameter combinations that could be used to obtain a given Hue, as long as the total accumulated fluence remains the same -- as highlighted by the trend (master curve) in Fig. 1 (d) -- under the condition that the repetition rate remains fixed. Fig. 1 (e) shows a polar plot of Fig. 1 (d) with Hue plotted azimuthally and the logarithm of the total accumulated fluence plotted radially; a full 360$^{\circ}$ rotation in Hue is observed. The strength of the process and the master curve can also be seen in the aesthetic control of a single colour. For example, not only are we able to produce the colour blue, but we can also control the type of blue (\textit{e.g.}, navy blue, sky blue, etc...), as the Lightness of the colours, for the same Hue, scales up with laser fluence, $\phi$, as seen in Figs. 1 (b). The process allows us to generate thousands of colours, well-beyond what has been previously reported, enabling this technique to rival paint based processes currently used at the Royal Canadian Mint. Chroma, however, is observed to remain unaffected for each Hue when using different laser fluences, $\phi$. Figure 1 (f) is a CIE xy chromaticity diagram of the colours obtained via the different laser parameters in Fig. 1 (d). The rotation and overlapping of the data points on the diagram shows explicitly that the xy values of CIE XYZ colour space can be recovered for a fixed laser fluence, $\phi$, by simply adjusting the marking speed or line spacing. Deviations from the master curve in Fig. 1 (d), are attributed to the increasingly chaotic nature of the surface with increasing total accumulated fluence. While we are able to render reds reproducibly, there is less variety in the reds than for other colours due to the high slope in the red region on the master curve (Hue: 345 to 15). 	From equation (1), $v$, $L_s$ and $E$ can be changed independently to obtain a specific total accumulated fluence corresponding to a desired colour, Fig. 1 (d). \\

However, changing the repetition rate, $f$, with either $v$, $L_s$ and $E$ to obtain a desired total accumulated fluence resulted in colours other than what was expected based on the master curve of Fig. 1(d). To study the effect of the laser repetition rate on the colours, $f$ was changed proportionally with speed, $v$, to maintain a fixed total accumulated fluence with $E$ and $L_s$ kept constant. The colour palette obtained was similar to that shown in Fig. 1(a) but with fewer colours. Increasing $f$ and keeping it fixed while changing $v$, $L_s$ and $E$ would only create reduced colour palettes. The baseline in Fig. 1(d) increased with increasing repetition rate gradually cutting off the lower colours (\textit{i.e.}, blues, purples and reds). No colours other than yellow were observed at repetition rates above 400 kHz suggesting a local thermal accumulation effect. \\

Figs. 2 (a) and (b) show before and after images of the colouring process applied to silver coins of varying topography (up to 2 mm in height). The different colours were selected prior to laser colouring using the corresponding total accumulated fluence values from the master curve (Fig. 1 (d)). Precise colouring of the coin features was done using vision alignment software with pattern recognition. The vectors were obtained from an artist at the Royal Canadian Mint. The process is capable of uniform and viewing angle-independent colouring of elevated and complex surfaces that could not be coloured via traditional paint-based manufacturing processes. Moreover, compared to top-down fabrication techniques, this colouring process is efficient regardless of surface quality, allowing for the uniform colouring of frosted (\textit{i.e.}, melted) surfaces, Figs. 2 (a-f).\\

In addition, due to the dependence of the colours on the total accumulated fluence, colour gradients can also be produced by focusing the laser beam slightly above the surface, outside the confocal volume, making the colouring process sensitive to topography - see the eagle wings (e) and tail (f) in Fig. 2. The extent of the number of colours in the colour gradient is governed by the distance between the surface of the coin and the laser focus. At small distances (while still out of the confocal region) the colour gradient is observed to have a smooth transition in colours over a large area. Flat Figs. 2(g,h) and topographical (Fig. 2 (b)) surfaces situated within the confocal volume are, however, coloured uniformly. White and perfect black were also achieved using the appropriate laser parameter combinations, as shown in Fig. 2 (g).
\\
The long term stability of the colours was observed to depend on the colour. However, passivation coatings formed via 	atomic layer deposition on coloured silver surfaces protected the colours during aggressive humidity and tarnish tests carried 		out at the Royal Canadian Mint. The colours were, however, slightly red-shifted due to the passivation layer, an effect that 	can be pre-compensated by altering the write laser parameters.
\subsection{Surface Analysis}
\begin{figure}[H]
\centering
\begin{tabular}{@{}p{1\linewidth}@{}}
\subfigimg[width=\linewidth]{\textbf{\textcolor{black}{}}}{Figure3.png}
\end{tabular}
\caption{(a,b,c) Low-magnification optical microscope images of coloured surfaces; (d,e,f) low- and (g,h,i) high-magnification SEM images of corresponding surfaces. Surfaces processed using $\phi=1.12$ J/cm$^2$ at $v=11$ mm/s for (a,d,g) $L_s=5$ $\mu$m (Hue = 216.5, Cyan), (b,e,h) $L_s=10$ $\mu$m (Hue = 269.3, Blue), and (c,f,i) $L_s=30$ $\mu$m (Hue = 17.2, Red). The number of medium NPs is observed to decrease with increasing $L_s$ (d,e,f -- scale 1 $\mu$m), and the number of small NPs to increase with $L_s$ (g,h,i -- scale 0.2 $\mu$m).}
 \label{fig3}
\end{figure}

Extensive SEM analyses of regions exhibiting different colours on silver reveal 3 distinct classes of particles differentiated by size: large ($r\geq75$  nm), medium ($10.7\leq r$ $\textless$ 75 nm) and small ($r$ $\textless$ 10.7 nm) where $r$ is the radius of a particle.
These particle classes were obtained from statistical analysis of SEM images which produced histograms with well-defined ranges of particle sizes (see Fig. S. 4(a,b,c), supplementary material).

Figs. \ref{fig3}(a-c) show three coloured surfaces with corresponding low- and high-magnification SEM images. The particle density is found to change significantly with line spacing, as shown in Figs. \ref{fig3}(d,g).
In Figs. \ref{fig3}(d-i), the small and medium particles are seen to form random networks with the medium-sized particles sparsely covering the surface and the small ones more densely distributed across the irradiated region. The formation of NPs is believed to come from the combination of thermal effects\cite{Che2008, Galhenage2013, Roque2005, Fazio2014} and the re-deposition of particles following laser ablation\cite{Perriere2007, Tillack2004a, Tillack2004}.

Upon close examination of the SEM images, it appears that the small particles are in reality approximately hemispherical, an observation supported by a cross-section of a coloured silver sample obtained through focused ion beam milling and imaging. The small particles can be viewed as spheres partially embedded into the silver surface.
\\
SEM analyses of coloured regions reveal that the number density of the small particles, produced using different laser parameters, follows its own clear distinctive trend with total accumulated fluence, (Fig. S. 3 (a), supplementary materials), similar to that of Fig. 1 (d), whereas the number density of the medium particles does not (Fig. S. 3 (b), supplementary materials). This observation suggests that the small particles play a major role in the colours perceived, even though they have not been considered in previous works. The mean radius of small and medium particles was found to remain approximately constant as a function of line spacing (Figs. S. 3 (c,d), determined from the analysis of three SEM images per line spacing or colour). However, the mean inter-particle distance (wall-to-wall) changes with line spacing, Figs. S. 3 (e,f), suggesting that the colours are affected by near-field interactions between nanoparticles in close proximity \cite{Rechberger2003,Romero2006,Jain2010,Liz2006},  particularly the associated surface plasmon resonance frequency \cite{Liz2006}.
\\
Wavelength-dispersive spectroscopy (WDS) analysis of the different colours in Figure 1, showed no difference in the amount of oxidation measuring 2.8 $\pm$ 0.4$\%$ oxygen content for all Hue values tested. Monte Carlo simulations (WinXray) of silver oxide layers, under the same WDS operating conditions, gave an equivalent oxide thickness of $\sim$ 2 nm. The same conclusion was gathered from the invariability of the oxygen content in separate energy dispersive spectroscopy (EDS) and X-ray photoelectron spectroscopy (XPS) measurements of the different coloured surfaces. The XPS analyses of the coloured surfaces also showed no sulphur content.

\subsection{FDTD Simulations}
Our detailed numerical simulations show that colour formation can be viewed as a selective absorption process occurring due to plasmonic resonances. White light incident on a nanostructured metallic surface is not fully reflected, as it would be for a smooth surface. Rather, narrow-band spectral components (colours) are subtracted due resonant and absorptive processes in nanoclusters (homogeneous and/or heterogeneous), single particles, lattice resonant modes, and absorption in the medium, consistent with another study \cite{Ng2015}. What survives is reflected (there is no transmittance).
\begin{figure}[H]
\centering
\begin{tabular}{@{}p{.8\linewidth}@{}}
\subfigimg[width=\linewidth]{\textbf{\textcolor{black}{a)}}}{Figure4a.pdf}\\
\subfigimg[width=\linewidth]{\textbf{\textcolor{black}{b)}}}{Figure4b.pdf}
\end{tabular}
\begin{tabular}{@{}p{.7\linewidth}@{}}
\subfigimg[width=\linewidth]{\textbf{\textcolor{black}{c)}}}{Fig4c.png}
\end{tabular}
\caption{(a) Computed reflectance spectra considering only small NPs embedded by $R_s$/2, only medium NPs embedded by $R_m$/2, and medium and small NPs embedded by $R_m$/2 and $R_s$/2, respectively. (b) Reflectance spectra computed by varying the embedding of the small NPs in steps of 0.5 nm (the medium NPs are embedded $R_m$/2). (c) Graph of Hue values vs. total accumulated fluence comparing the measured and computed values for small NPs embedding of 1.5-2.5 nm, and medium NPs not embedded.}
\label{fig5}
\end{figure}
The analysis of SEM images produced statistics of the distributions, \textit{i.e.}, average radii and average inter-particle distances for small, medium and large NPs. We carried out simulations considering only small and medium Ag NPs uniformly distributed on a Ag substrate. This is supported by the two discernible bumps (bimodal distribution) noted in the histograms (Fig. S. 4 (a,b,c), supporting information). The large particles are neglected due to their low density. The radii of small and medium sized NPs are $R_s$ and $R_m$, and the inter-particle distances (centre-to-centre) are $D_s$ and $D_m$, respectively.
In each case the NPs were periodically arranged in a hexagonal configuration, which produces a hexamer unit cell. The hexamer has $D_{6h}$ symmetry, and based on group theory, exhibits polarization independence (isotropy), as verified by simulations \cite{Hent2010}. We chose $D_m$ as an integer multiple of $D_s$ to allow the application of periodic boundary conditions (PBCs). The periodicity of the unit cell is small enough to preclude coupling by diffraction into surface plasmon waves on the Ag surface.
\\
By changing the surface density of small and medium sized NPs, their radii, and their level of embedding into the substrate, several types of heterogeneous nanoclusters can be formed. Embedding transforms the NPs gradually from spherical to hemispherical. A nanocluster has a central nanoparticle (cNP) with nearby nanoparticles arranged in a ring (rNPs). Plasmonic nanoclusters embedded in a homogeneous medium or on a dielectric have been extensively studied (dimer, trimer, quadrumer, tetramer, hexamer, heptamer)\cite{Luk2010,Mirin2009,Hent2010}, but not on a metal substrate, and not arranged and embedded as inspired by SEM images of laser-processed metal surfaces. Plasmonic nanoclusters exhibit resonances which can lead to selective absorption\cite{Mirin2009}.
We have observed different types of resonant modes in our simulations; we will refer to them as cluster (or collective) resonances.

In the simulations that follow, we consider the case of line spacing $L_s=5$ $\mu$m marked at $v=50$ mm/s with a laser fluence $\phi=1.67$ J/cm$^2$, having statistical average parameters $R_s=4$ nm, $R_m=34.3$ nm, $D_s=13.5$ nm and $D_m=108$ nm.

In Fig. 4 (a) we show the computed reflectance spectrum considering small NPs only, medium NPs only, and the combination of both NP sizes; the NPs are half-embedded, \textit{i.e.}, embedded by half their radius. The medium size NPs are observed to produce colours in the absence of the small ones, but changing $D_m$ and the level of embedding does not reproduce the range of colours observed experimentally. Alternatively, considering only small NPs produced colours that are too light to be discernible. However, the interaction between small and medium NPs drastically alters the computed reflection spectrum, as noted by the appearance of a deep dip at $\lambda$ = 650 nm in Fig. 4 (a), which is due to a new cluster resonance. This will produce a discernible final colour. In our simulations we observed a very high sensitivity to embedding, which changes dramatically the geometry of the nanoclusters, and consequently, their resonance characteristics. In particular, the embedding of the small NPs increases the size of the nano-gaps between the cNP and the rNPs. In Fig. 4 (b) we show the reflectance by embedding the small NPs in steps of 0.5 nm. A blue-shift in the cluster resonance is observed when the embedding of small NPs is increased. A similar blue-shift due to cluster expansion was reported in \cite{Hent2010}.
In Fig. 4 (b) we also see other smaller dips in the reflectance curves. These are due to nanocluster resonances arising from aggregates of small NPs. In Fig. 4 (c) we compare the experimental results with the FDTD simulations. A qualitative agreement was found for medium NPs not embedded and small NPs embedded into the substrate by 2.5 nm. We were able to reproduce by simulations the full colour palette, supporting the role of plasmons in the colour rendition. We found that a large set of geometries can produce the same reflectance, in the same way as a large set of laser parameters can produce the same colour.

In Figs. \ref{fig6}(a-d) we show the electric field distribution at the free-space optical wavelengths of $390$ nm (a,c) and $650$ nm (b,d), which are the wavelengths at which we observe absorption dips in Fig. \ref{fig5}(a).
We show $xz$ planes cut 2 nm above the silver surface for medium NPs only (a,b), and medium and small NPs (c,d).
In Fig. \ref{fig6}(a) we observe the resonance of the medium sized NP at $390$ nm, as predicted by Mie theory for the same sphere in air.
In our case, the resonance produces a dip in the reflectance due to the presence of the substrate.
Fig. \ref{fig6}(b) shows that the medium NPs only do not produce any absorption at $650$ nm, and the reflectance is very high.

In Fig. \ref{fig6}(c) we observe that the near-field intensity of the medium NPs at $390$ nm is reduced by the presence of the small NPs, which act as plasmonic chain waveguides \cite{Maier2002}, coupling the medium size NPs.
\\
In Figs. 5 (a-d) we show the electric field distribution at the free-space optical wavelengths of $\lambda$ = 390 nm (a,c) and $\lambda$ = 650 nm, which are the wavelengths at which we observe absorption dips in Fig. 4 (a). We show xz planes cut 2 nm above the silver surface for medium NPs only (Figs. 5 (a,b)), and medium and small NPs (Figs. 5 (c,d)).  In Fig. 5 (a) we observe the resonance of the medium sized NPs at $\lambda$ = 390 nm, as predicted by Mie theory for the same sphere in air. In our case, the resonance produces a dip in the reflectance due to the presence of the substrate. Fig. 5 (b) shows that medium NPs only do not produce any absorption at $\lambda$ = 650 nm, and the reflectance is very high. In Fig. 5 (c) we observe that the near-field intensity of the medium NPs at $\lambda$ = 390 nm is reduced by the presence of the small NPs, which act as plasmonic waveguides \cite{Maier2002}, coupling the medium size NPs. When small and medium NPs are illuminated at $\lambda$ = 650 nm (Fig. 5(d)), a heterogeneous cluster resonance is excited, producing field enhancement in the nano-gaps, and ultimately the strong absorption observed in Fig. 5 (a). In Fig. 5 (e) we show a snapshot of the time-domain simulation in a yz plane cut through the centre of the medium NPs. The plane wave excitation pulse has just hit the nanostructures inducing localized surface plasmons in all NPs which are clearly visible. The time-evolution of the excitation in the xz and yz planes is shown in Movies S. 1 and S. 2 (supporting information), respectively.
In Fig. 5 (f) we show an SEM image for the case of $\phi$ = 1.67 J/cm2 at $v$ = 50 mm/s with $L_s$ = 8 $\mu$m, which further justifies our simulation approach. We observe the presence of two sets of particles (small and medium sizes) having a random distribution. In our simulations we considered perfect periodicity of the NPs, which gives isotropy with respect to the incident polarization, narrowband resonances, and ultimately colour selectivity. This can be considered a good qualitative approximation. In \cite{Nishijima2012} random clusters were investigated, and the authors reported a broadening of the resonances. This matches our experiments which show broader spectra with respect to simulations.
\begin{figure}[H]
  \centering
  \begin{tabular}{@{}p{.8\linewidth}@{}}
	\subfigimg[width=\linewidth]{\textbf{\textcolor{white}{}}}{Figure5abcd.pdf}
	\end{tabular}
	\begin{tabular}{@{}p{.8\linewidth}@{}}
	\subfigimg[width=\linewidth]{\textbf{\textcolor{black}{e)}}}{Figure5e.pdf}
  \end{tabular}
	\begin{tabular}{@{}p{.7\linewidth}@{}}
	\subfigimg[width=\linewidth]{\textbf{\textcolor{white}{f)}}}{Figure5f.png}
  \end{tabular}
  \caption{FDTD simulations showing the electric field distribution for only medium size NPs at (a) $\lambda$ = 390 nm and (b) $\lambda$ = 650 nm, and for medium and small NPs at (c) $\lambda$ = 390 nm and (d) $\lambda$ = 650 nm. (e) Time-domain snapshot of medium and small nanoparticles embedded by $R_{m}$/2 and $R_{s}$/2, respectively. (f) SEM image for $\phi$=1.67 J/cm$^2$ at $v=50$ mm/s with $L_s=8$ $\mu$m.}
  \label{fig6}
\end{figure}

\section{Conclusion}
We described a universal and deterministic process for colouring noble metals with and without topography. Each individual colour can be linked to a total accumulated fluence. The colours originate from random distributions of small and medium nanoparticles embedded into the surface, induced and controlled by laser exposure.  The randomness of the nanoparticle networks is modelled effectively by assuming a periodic structure defined by statistical averages of nanoparticle size and separation. We have demonstrated that plasmonic effects arising in heterogeneous nanoclusters explain the full palette of experimental colours. The small nanoparticles in particular, which have always been neglected in post-surface analyses following laser exposure, play a fundamental role in the colour formation. In fact, they change the geometry of the nanoclusters, the cluster resonance, and ultimately the perceived colour. The new proposed method decreases the colouring time of large metal surfaces, and opens the door to large-scale industrial applications for anti-counterfeiting, bio-sensing, bio-compatibility, and the decoration of consumer products such as jewels, art, architectural elements and fashion items.

\section{Methods}
In our experiments, 1064 nm light from a 15 W Duetto (Nd:YVO$_{4}$, Time-Bandwidth Product) mode-locked MOPA laser, operating at a repetition rate of 50 kHz and producing 10 ps pulses, was focused on the metal surface using an F-theta lens (f=163 mm, Rodenstock). The pulse energy for the presented results ranged between 3.4 and 91.4 $\mu$J. The laser was fully electronically integrated and enclosed by a third party for industrial applications (GPC-PSL, FOBA). For accurate focusing, the surface of the samples was located using a touch probe system. The silver samples were of 99.99$\%$ purity and not polished prior to machining to meet requirements of reproducibility in industrial applications. For machining, the samples were placed on a 3-axis translation stage with a resolution of 1 $\mu$m in both the lateral and axial directions. The samples were raster scanned using galvanometric XY mirrors (Turboscan 10, Raylase) displacing the beam in a top to bottom fashion with a mechanical shutter blocking the beam between successive lines. The laser polarization was parallel to the marking direction. The laser power was computer controlled via a laser interface and calibrated using a power meter (3A-P-QUAD, OPHIR). A spot size of 14 $\mu$m was obtained from semi-logarithmic plot of the square diameter of the modified region, measured with a scanning electron microscope (SEM), as a function of energy, following the procedure described in \cite{Jandeleit1996}. High resolution SEM (JSM-7500F FESEM, JEOL) images were obtained using secondary electron imaging (SEI) mode. Colours were quantified using a Chroma meter (CR-241, Konica Minolta) with the CIELCH colour space, 2 observer and illuminant C (North sky daylight); where L is colour Lightness, C is Chroma (colour saturation) and H is Hue (colour value associated with a 360 polar scale). For analysis of the SEM images, a Matlab program was  written to locate the position of each particle and record its diameter and the wall-to-wall inter-distance spacing to its nearest-neighbours.

Three dimensional finite-difference time-domain (FDTD) simulations\cite{Taflove2005, Taflove2013} have been performed to determine the origin of the colour formation process. We used in-house 3D-FDTD parallel code \cite{Lesina2015,Vaccari2014} on an IBM BlueGene/Q supercomputer (64k cores) part of the Southern Ontario Smart Computing Innovation Platform (SOSCIP).

The nanoparticles are arranged on the $xz$-plane. The system is excited by a $z$-polarized plane wave. This is a broadband electromagnetic pulse propagating along the $y$-direction from air and impinging on the nanostructured surface.
The analysis was performed over the wavelength range 350-750 nm in a single run of the code by in-line discrete Fourier transform (DFT). A space-step of $0.25$ nm was used for the simulations, and $0.125$ nm for visualization quality (Figs. \ref{fig6}(a-e) and Movies).
The dispersion of silver was introduced by the Drude+2CP model \cite{Vial2011}. This model was implemented in FDTD by the auxiliary differential equation (ADE) technique \cite{Prokopidis2013}. The simulation domain in the direction of the plane wave propagation is truncated by convolutional perfectly matched layers (CPML) absorbing boundary conditions \cite{Roden2000}. The simulations required up to 16k cores.
The theoretical reflectance spectrum is calculated by integration of the Poynting vector in the backward far-field region.
The experimental reflectance spectra (colours) were reconstructed using an in-house Matlab code, weighting each frequency composing the spectra to the spectral sensitivity of the eye.


\section{Introduction}
	\label{sec:intro}
	\vspace{-0.25cm}
	The freely available multispectral satellite imagery and advancement in modern machine learning have paved the way for a wide variety of applications. These include disaster assessment~\cite{dao2015object}, crop classification~\cite{csillik2019object, kobayashi2020crop}, urbanization~\cite{deepCountISPRS19} and environment monitoring~\cite{donlon2012global}.
	The increasing quality and resolution of available remote sensing imagery have made it possible to perform robust crop monitoring and yield estimation over large areas~\cite{battude2016estimating}. For instance, the Sentinel-2 satellite imagery with a spatial resolution of 10m and Landsat-8 imagery with a spatial resolution of 30m is available every five days and 16 days, respectively. Thus, enabling us to perform detailed spatio-temporal analysis~\cite{vuolo2018much}.

	\par
	Supervised learning is considered to be the state-of-the-art approach to produce crop maps. Some traditional machine learning algorithms like Support Vector Machines (SVM) \cite{saini2018crop}, K-nearest neighbors (KNN) \cite{thanh2018comparison}, Cart \cite{sonobe2017experimental}, and Random Forest \cite{rodriguez2012assessment} have been widely applied for crop classification. Among these methods, those using a series of images at different time stamps~\cite{gomez2016optical} have shown better results than the one using a single image \cite{saini2018crop}.

	\par
	Due to the popularity of deep learning, recently, the problem of crop classification has also been attempted using different deep learning architectures. In earlier work, a 1D convolutional neural network model has been proposed~\cite{zhou2018crops} which stacked features of different time stamps as in the case of random forest.  To use the temporal information more efficiently, ~\cite{pelletier2019temporal} has proposed a new 1D CNN model. This method feeds multivariate time series to 1D CNN instead of stacked values of features. Similarly, a new 1D CNN architecture based on the idea of the Inception network has been introduced~\cite{zhong2019deep}.  It has shown improved performance on traditional machine learning algorithms, including Random Forest and XGboost~\cite{abdi2020land}.
	\par
	The 2D convolutional neural networks show outstanding image classification performance, and many architectures, including Alexnet~\cite{krizhevsky2012imagenet}, Inception, Resnets~\cite{he2016deep}, etc., have been introduced. A new dataset comprising Sentinel-2 images of different landcover types from around the world has been developed~\cite{helber2019eurosat} and different 2D CNN architectures, including Resnet50 and GoogleNet~\cite{szegedy2015going}, have been compared to classify the multispectral images. Recently, an ensemble of 2D CNN has been used for crop classification using multi-temporal Sentinel-1 and Landsat-8 imagery~\cite{kussul2017deep}. This study shows the significance of using spatial information in addition to temporal one.
	\par
	The recurrent neural networks are very popular for applications where time-series data is used. The idea of using LSTM \cite{zhou2019long} has also been applied to landcover classification using multi-temporal SAR imagery~\cite{zhong2019deep}. This work has claimed that LSTM performs worse than Random Forest classifier for crop classification using multispectral imagery. The recurrent convolutional neural network (R-CNN)~\cite{rcnn} uses LSTM followed by 2D CNN, but unlike~\cite{kussul2017deep}, this architecture uses the information of only one pixel. The idea of combining CNN and LSTM has also been introduced for landcover classification~\cite{kwak2019combining}\cite{russwurm2018multi}.
	\par
	The 3D convolutional neural networks are widely considered for applications related to video, medical imaging, and remote sensing~\cite{BhimraICASSP2019}. In \cite{ji20183d}, a 3D CNN architecture has been presented showing better performance than 2D CNN as it uses the temporal information better than the 2D CNN where timestamps of all features are stacked on one axis.

	\begin{figure*}[t]
		\centering
		\includegraphics[width = 18cm]{images/model.png}
		\caption{Proposed model architectures showing combination of spatio-temporal and temporal convolution blocks. Spatio-temporal block via 3D CNN are shown in blue whereas the temporal only block via 1D convolution are shown in green.}
		\vspace{-0.4cm}
		\label{fig:model}
	\end{figure*}

	Existing literature suggests various ways to exploit available spatio-temporal data~\cite{BhimraICASSP2019,ji20183d} however, these methods are either good at utilizing spatial information or temporal information but not both. We instead propose a novel DCNN based architecture that combines both spatial as well as temporal analysis. In the first stage of our architecture, we propose to use 3D convolutions that perform spatio-temporal analysis without collapsing the temporal dimension. Once the spatio-temporal features are extracted via multiple 3D CNN layers, we introduce temporal only analysis to further extract important information from the temporal dimension only. This spatio-temporal followed by temporal only analysis helps eliminate noise, usually present in the temporal only analysis. Thus the proposed approach outperforms classical as well as state-of-art-methods on benchmark datasets.

	\section{Methodology}
	\label{sec:method}
	\vspace{-0.25cm}

	\subsection{Data Pre-processing}
	\vspace{-0.2cm}
	We have divided the pre-processing of data into five steps. The first step is to select all the images of a multispectral satellite for the region of interest in a suitable period, covering every crop's cropping season under consideration. For this purpose, the NDVI time series for different crops have been examined. The second step is to select the least cloudy images in that period. In our case, the images with cloud cover less than $10\%$ have been selected. The third step is the cloud masking of the chosen images. The fourth step is to take the median of images chosen for each month. This step is for larger areas, as in our case, because the region can't be covered in a single image of a satellite. In the fifth and final step, the missing pixels due to the cloud or other reasons have been filled, which can be done using simple interpolation or linear regression. All these pre-processing steps have been performed on the Google Earth Engine Platform.

	%

	\vspace{-0.2cm}
	\subsection{Feature Selection}
	\vspace{-0.2cm}
	For each image in our dataset, $13$ features have been selected. The selected features have six multi-spectral satellite imagery bands, including Blue, Green, Red, Near Infrared (NIR), and two Short Wave Infrared (SWIR) bands. Table~\ref{tab:bands} shows the selected Sentinel-2 and Landsat-8 bands. The other seven features are indices derived from these bands, including Normalized Difference Vegetation Index (NDVI), Enhanced Vegetation Index, Green Normalized Difference Vegetation Index (GNDVI), Soil Adjusted Vegetation Index (SAVI), Bare Soil Index (BSI), Normalized Difference Water Index (NDWI) and Normalized Difference Buildup Index (NDBI). These indices~\cite{pal2017comparison,vermote2016preliminary} are calculated as follows:
	%
	\begin{table}[b]
		\centering
		\caption{Selected Bands of Landsat-8 and Sentinel-2\Bstrut}
		\begin{tabular}{>{\centering}p{1.6cm} >{\centering} p{0.5cm}>{\centering}p{0.8cm}>{\centering}p{0.6cm}>{\centering}p{0.6cm}p{0.8cm}p{0.8cm}}
			\hline
			\textbf{Bands}&\textbf{Blue}&\textbf{Green}&\textbf{Red}&\textbf{NIR}&\textbf{SWIR1}& \textbf{SWIR2}\Tstrut\Bstrut\\
			\hline\hline
			\textbf{Sentinel-2}&B2&B3&B4&B8&B11&B12\Tstrut\\
			\textbf{Landsat-8}&B2&B3&B4&B5&B6&B7\\
			\hline
		\end{tabular}
		\label{tab:bands}
	\end{table}
	%

	\begin{center}
		$\text{NDVI} = \frac{NIR - RED}{NIR + RED}~~~~,~~\text{GNDVI} = \frac{NIR - GREEN}{NIR + GREEN}$
	\end{center}
	\begin{center}
		$\text{EVI} = 2.5\times\frac{NIR - RED}{NIR + 6\times RED-7.5\times BLUE + 1}$
	\end{center}
	\begin{center}
		$\text{SAVI} = 1.5\times\frac{NIR - RED}{NIR + RED + 0.5}$
	\end{center}
	\begin{center}
		$\text{BSI} = \frac{(SWIR + RED) - (NIR + BLUE)}{(SWIR + RED) + (NIR + BLUE)}$
	\end{center}
	\begin{center}
		$\text{NDBI} = \frac{SWIR1 - NIR}{SWIR1 + NIR}~~,~~~\text{NDWI}= \frac{GREEN - NIR}{GREEN + NIR}$
	\end{center}
	%
	\subsection{Model Architecture}
	\vspace{-0.2cm}
	The model architecture is shown in Fig.~\ref{fig:model}. The model consists of three parts concatenated to each other. The first part shown in blue performs spatio-temporal convolution using 3D CNN. The second part shown in green performs the temporal convolution. The output of 3D CNN is squeezed before feeding to 1D CNN. These two parts extract the features from input. The third part consists of a fully connected neural network, which predicts the label from multi-temporal input images.
	\par
	The model input shape is $(r, r, t, c)$ where $r$ is the spatial dimension of input, $t$ is the number of timestamps, and $c$ is the number of channels. In our implementation, spatial dimension $r$ is $7$, timestamps $t$ are $9$ and channels $c$ are $13$. Each convolutional block has a batch normalization layer before activation.  It has been observed that with batch-normalization, the model converges faster as well as shows better accuracy.

	\section{RESULTS AND EVALUATION}
	\label{sec:results}
	\vspace{-0.2cm}
	\subsection{Experimental Setup}
	\vspace{-0.2cm}
	The experiments have been performed on two different California counties, including Imperial County and Yolo County. For both regions, the crop survey data provided by the California Department of Water Resources (CDWR) \cite{dataset} for the year 2016 have been used as ground truth information.
	%
	\begin{figure}[t]
		\centering
		\includegraphics[width=8.5cm]{images/mapsamples.png}
		\caption{The regions of interest showing training (red) and test (blue) data distribution for Yolo (left) and Imperial Counties (right) distribution.}
		\vspace{-0.2cm}
		\label{fig:mapsamples}
	\end{figure}
	%
	\par
	For Imperial County, $10$ major classes have been considered, while $14$ classes have been taken into account for Yolo County. The "Others" class contains either minor crop types, fallow land, or build up. There are about $7256$ and $7340$ total parcels for Imperial County and Yolo County, respectively, as mentioned in Table~\ref{tab:yolo_samples}. These parcels have been split into $60\%$ training data and $40\%$ test data. The fields have been divided into training and test sets before sampling to ensure that no sample from the training region is included in the test set. The Fig.~\ref{fig:mapsamples} shows the distribution of training and testing parcels. The imagery of Sentinel-2 and Landsat-8 has been used for Imperial County and Yolo County, respectively. In both cases, nine scenes from March to November 2016 have been used. The same dataset has been used to train all the classifiers, including proposed and state-of-the-art methods, in both pixel and parcel-based analysis.

	\begin{table}[ht]
		\centering
		\small
		\caption{Summary of survey data from Imperial and Yolo counties provided by California Department of Water Resources (CDWR)}
		\begin{tabular}{p{1.6cm} p{0.8cm} p{0.8cm} | p{1.6cm} p{0.8cm} p{0.8cm}}

			\hline
			\multicolumn{3}{c}{\textbf{Imperial County}} & \multicolumn{3}{c}{\textbf{Yolo County}}\Tstrut\Bstrut\\
			\hline
			\multirow{2}{*}{\textbf{Class}} &\textbf{Parcels} & \textbf{\%age} & \multirow{2}{*}{\textbf{Class}} &\textbf{Parcels} & \textbf{\%age}\Tstrut\\
			&  \textbf{Count} & \textbf{area} & &  \textbf{Count} & \textbf{area}\Bstrut\\

			\hline \hline
			Alfalfa & 2156 & 31.3\% & Rice & 350 & 11.2\%\Tstrut\\
			Pastures & 1150 & 15.4\%& Safflower & 161 & 2.4\%\\
			Lettuce& 619 & 8.2\%& Corn & 150 & 2.1\%\\
			Wheat & 315 & 4.3\%& Field crops & 407 & 6.6\%\\
			Onions & 229& 3.0\% & Alfalfa & 575 & 8.7\%\\
			Truck Crops & 1026 & 11.6\% & Pastures & 440 & 4.3\%\\
			Corn & 232& 3.5\% & Cucurbits & 168 & 1.8\%\\
			Field Crops & 192 & 2.9\% & Tomatoes & 496 & 9.8\%\\
			Subtropical &271 & 1.6\% & Truck Crops & 289 & 1.3\%\\
			Others & 1066 & 18.2\%& Almonds & 799 & 8.9\%\\
			& & & Deciduous & 867 & 5.3\%\\
			& & & Subtropical & 176 & 1.1\%\\
			& & & Vineyard & 688 & 5.6\%\\
			& & & Others & 1874 & 30.9\%\\\hline
		\end{tabular}
		\vspace{-0.4cm}
		\label{tab:yolo_samples}
	\end{table}

	\subsection{Comparative Analysis}
	\vspace{-0.2cm}
	We compared our proposed hybrid model with four state-of-the-art methods, namely Random Forest, 1D CNN \cite{zhong2019deep}, 2D CNN\cite{kussul2017deep} and 3D CNN\cite{ji20183d}

	\begin{table*}[t]
		\centering
		\small
		\caption{Table showing comparison of our proposed 3D$\rightarrow$1D CNN with state-of-the-art methods on Imperial and Yolo counties. The comparison is performed using pixel-wise and parcel-wise strategies (provides single classification within parcel boundary)}
		\resizebox{0.9\textwidth}{!}{
			\begin{tabular}
				{c|cc|cc|cc|cc}

				\hline
				\multirow{3}{*}{\textbf{Classifier}}&\multicolumn{4}{c}{\textbf{Imperial County}} & \multicolumn{4}{c}{\textbf{Yolo County}}\Tstrut\Bstrut\\
				\cline{2-9}
				&\multicolumn{2}{c|}{\textbf{Pixel-wise}} & \multicolumn{2}{c|}{\textbf{Parcel-wise}}&\multicolumn{2}{c|}{\textbf{Pixel-wise}} & \multicolumn{2}{c}{\textbf{Parcel-wise}}\Tstrut\Bstrut\\
				\cline{2-9}
				& \textbf{Accuracy}&\textbf{F1-Score}&\textbf{Accuracy}&\textbf{F1-Score}& \textbf{Accuracy}&\textbf{F1-Score}&\textbf{Accuracy}&\textbf{F1-Score}\Tstrut\Bstrut\\
				\hline \hline
				Random Forest & 78.96\% & 78.60\% & 80.61\% & 80.39\% & 86.68\% & 86.25\%& 82.23\% & 81.21\%\Tstrut\\
				1D CNN \cite{zhong2019deep} & 79.46\% & 79.05\%& 80.33\% & 80.24\%& 87.94\% & 87.80\%& 84.45\% & 83.90\%\\
				2D CNN \cite{kussul2017deep} & 79.71\% & 79.48\%& 80.54\% & 80.40\% & 88.50\% & 88.36\%& 84.55\% & 84.18\%\\
				3D CNN \cite{ji20183d} & 79.95\% & 79.70\%& 80.80\% & 80.72\%&88.01\% & 87.82\%& 83.17\% & 82.74\%\\
				3D$\rightarrow$1D CNN  (Ours)& \textbf{81.23\%} & \textbf{81.16\%}& \textbf{81.76\%} & \textbf{81.76\%} & \textbf{90.23\%} & \textbf{90.08\%} & \textbf{85.87\%} & \textbf{85.51\%}\\
				\hline

			\end{tabular}
		}
		\label{tab:results}
	\end{table*}

	\begin{figure*}[ht]
		\centering
		\begin{minipage}[b]{0.3\linewidth}
			\centering
			\centerline{\includegraphics[width=5cm]{images/originalimperialmap3.png}}

			\centerline{}\medskip
		\end{minipage}
		%
		\begin{minipage}[b]{0.3\linewidth}
			\centering
			\centerline{\includegraphics[width=5cm]{images/imperialpapermap3.png}}

			\centerline{}\medskip
		\end{minipage}
		\begin{minipage}[b]{0.3\linewidth}
			\centering
			\centerline{\includegraphics[width=5cm]{images/imperial3d1dmap3.png}}

			\centerline{}\medskip
		\end{minipage}
		\begin{minipage}[b]{0.3\linewidth}
			\centering
			\centerline{\scalebox{-1}[1]{\includegraphics[width=5cm]{images/originalyolomap3.png}}}

			\centerline{(a) Ground Truth}\medskip
		\end{minipage}
		%
		\begin{minipage}[b]{0.3\linewidth}
			\centering
			\centerline{\scalebox{-1}[1]{\includegraphics[width=5cm]{images/yolopapermap3.png}}}

			\centerline{(b) 3D CNN\cite{ji20183d}}\medskip
		\end{minipage}
		\begin{minipage}[b]{0.3\linewidth}
			\centering
			\centerline{\scalebox{-1}[1]{\includegraphics[width=5cm]{images/yolo3d1dmap3.png}}}

			\centerline{(c) 3D$\rightarrow$1D CNN (Ours)}\medskip
		\end{minipage}\vspace{-0.2cm}

		\caption{Ground truth and maps produced for Imperial County (Top) and Yolo County (Bottom) by 3D CNN and our network.\label{fig:maps}}
		\vspace{-0.2cm}

	\end{figure*}

	For random forest, the number of trees has been set to $100$, the minimum number of samples required to split a node is two, and the maximum number of features is set to the square root of the number of features. For the 1D Convolutional Neural Network, the architecture proposed in \cite{zhong2019deep} has been used except that more features have been used instead of just EVI. The idea given by \cite{pelletier2019temporal} has been applied to consider more features in 1D CNN. For 2D Convolutional Neural Network, five ensembles of 2D CNN have been used as proposed in \cite{kussul2017deep}. The number of filters in the network ensembles is 160, 170, 180, 190, and 200. Due to the larger number of input features, more filters have to be used. For 3D Convolutional Neural Network, the model proposed in \cite{ji20183d} using the concept of VGG has been used, but the number of filters has been increased to 64 and 128 in both layers. Our proposed method has the same architecture as explained in Sec.~\ref{sec:method}.
	\par
	All the deep learning networks have been trained for about $50$ epochs using Adam optimizer with a default learning rate of $0.001$. The batch size of $128$ has been used, and each network's best results have been recorded. The accuracy and weighted F1-score for each classifier are mentioned in the Tables~\ref{tab:results}. The table shows the pixel-wise accuracy and F1-score as well as the parcel-wise results. It is evident from both tables that our proposed model has outperformed all the other classifiers in both experiments. Also, other deep learning models don't show much better accuracy than the random forest, and parcel-wise accuracy for the Imperial county of random forest is greater than other deep learning models except for our proposed one. On the other hand, our 3D$\rightarrow$1D CNN model improves the accuracy and F1-score by $2\%$. The parcel-wise results have been calculated by assigning each parcel a label based on the majority voting of pixels in it.
	\par
	Contrary to our expectation, the parcel-wise accuracy is $4-5\%$ less than pixel-wise accuracy for each classifier in the case of Yolo County. The Yolo County survey data analysis revealed that $3.5\%$ parcels in test data were tagged as multi-use, i.e., more than one crop was sown in those fields. In contrast, Imperial County data have no field labeled as multi-use. Therefore, accuracy increased in the case of Imperial County while decreased in the case of Yolo County. Another reason is the presence of larger polygons in the test set of Yolo County.
	\par

	The \textbf{Qualitative Analysis} is shown in Fig.~\ref{fig:maps} which shows the original labels from CDWR data and results produced by 3D CNN \cite{ji20183d} and our proposed 3D$\rightarrow$1D CNN model. In the figure, there are some blank spaces where ground truth was not available, and it can be observed that those blank spaces have been predicted as "Others" represented by black color. The zoomed portion of the results in Fig.~\ref{fig:maps} further illustrates that the maps produced by our hybrid model have lesser noise compared to the ones produced by vanilla 3D CNN.

	\begin{table}[tbh]
		\centering
		\caption{Performance Comparison of State of the art models}
		\resizebox{0.9\columnwidth}{!}{
			\begin{tabular}{c c c}
				\hline
				\textbf{Classifier} & \textbf{Parameters} & \textbf{Inference time (ms)}\Tstrut\Bstrut\\
				\hline \hline
				1D CNN \cite{zhong2019deep} & 1,083,970  & 6.39\Tstrut\\
				2D CNN \cite{kussul2017deep}  & 4,278,282 & 7.11\\
				3D CNN \cite{ji20183d}  & 394,050  & 4.72\\
				3D$\rightarrow$1D CNN (Ours) & 361,406 &  4.58\\
				\hline
			\end{tabular}
		}
		\label{tab:performance}
	\end{table}

	\vspace{-0.4cm}
	\subsection{Computational Cost}
	\par
	Table~\ref{tab:performance} shows the comparison of the number of parameters in different deep learning models and their inference time. Our model has a lesser number of parameters as well as the fastest inference time. This is because, instead of flattening the output of 3D CNN, we fed it to 1D CNN, which reduced the number of parameters and improved the performance.

	\vspace{-0.2cm}
	\section{Conclusion}
	\vspace{-0.4cm}
	In this paper, a new architecture and a method for large-area crop classification have been presented. Our hybrid model combines the spatio-temporal representation via 3D CNN with temporal only representation via 1D CNN. Different neural network architectures proposed in the past for crop mapping have been implemented and compared with our designed model.  Experiments have been performed using multispectral imagery of different satellites to show the generality of our method. The experimentation has revealed that the proposed hybrid network performs better on benchmark CDWR datasets.

	In the future, we aim to extend our approach to fields that have been marked as multi-use (having more than one crop in a small parcel) as multi-use is a more commonly observed pattern in agriculture-based economies, particularly among developing countries.

	\small
	\bibliographystyle{IEEEbib}


\section{Context}
\label{sec:context}

Maritime traffic surveillance has played an important role, not only in navigation  safety, but also in other Maritime Situational Awareness (MSA) aspects. Among others, the early detection of abnormal behaviours helps identify suspicious activities, enforce law, perform efficiently search and rescue, etc.

In recent years, the world has experienced an unprecedented development of Big Data and Artificial Intelligence (AI), of which Deep Learning~\cite{lecun_deep_2015} is the core. People have tried to leverage Big Data and AI in many domains, and maritime surveillance is not an exception. The rich information provided by the Automatic Identification System (AIS) makes it a very appealing research topic. In~\cite{mantecon_deep_2019}, Airbus Defence and Space used a ResNet-based architecture to re-identify the navigation status of vessels. Similarly, convolutional neural networks were used to discover mobility modes in~\cite{chen_use_2018}. Some work went into detecting fishing pattern using a data mining approach~\cite{souza_improving_2016}. NATO STO Centre for Maritime Research and Experimentation used recurrent neural networks to predict vessel position in~\cite{forti_prediction_nodate}. Our previous work~\cite{nguyen_multi-task_2018,nguyen_geotracknet-maritime_2019} takes advantage of the high modeling capacity of deep neural network to create a probabilistic presentation of AIS tracks for anomaly detection.

Although those models have given impressive experimental results, applying them to operational systems is still called into question: (i) the black (or gray) box nature of Deep Learning raises concern about their reliability, (ii) the massive volume of AIS data requires a suitable deployment strategy. Besides that, the absence of reference groudtruthed dataset also questions the ability to evaluate the operational usefulness of such learning-based strategies.

In this paper, we report our progress in making a research prototype, namely the \textit{GeoTrackNet } model presented in \cite{nguyen_multi-task_2018} and \cite{nguyen_geotracknet-maritime_2019}, reach operational needs.
\revise{Specifically, we analyse the types of anomalies detected by the model, and evaluate scalability as well as the possibility of deploy it in real-time a big data platform.}

The paper is organised as follows: in Section~\ref{sec:geoTrackNet}, we give a brief introduction of \textit{GeoTrackNet}. The report on the validation of the output of the model is presented in Section~\ref{sec:validation}. Section~\ref{sec:onlineDetection} demonstrates the ability to deploy \textit{GeoTrackNet } in a real-time distributed system. We end the paper by the conclusions and perspectives in Section~\ref{sec:conclusions}.
\section{GeoTrackNet}
\label{sec:geoTrackNet}

In this section, we summary the principles of \textit{GeoTrackNet}. For more details, readers are encouraged to refer to~\cite{nguyen_geotracknet-maritime_2019}.

\textit{GeoTrackNet } is a probabilistic model to detect abnormal behaviours in maritime traffic. The model is based on the hypothesis that most of vessels' behaviours in the training set are normal and the model can learn the dynamics of those tracks. In the training phase, \textit{GeoTrackNet } learns a distribution representing the data (see Fig.~\ref{fig:GeoTrackNet}). This distribution is then used in the test phase to state how likely a new AIS track is.
Any track that does not follow this distribution will be flagged as abnormal. To capture the high complexity of AIS data, \textit{GeoTrackNet } uses a Variational Recurrent Neural Network (VRNN)~\cite{chung_recurrent_2015}, which is the state of the art in text and speech modeling.

Because vessel density and vessels' behaviours are not geographically homogeneous, \textit{GeoTrackNet } also exploits a geographically-dependent \textit{a contrario} detection rule. The Region Of Interest (ROI) is divided into small cells, then a local threshold is applied to state the normality of each AIS message in this cell.

\begin{figure}
    \centering
    \includegraphics[width=85mm]{figures/geoTrackNet_principle.png}
    \caption{\textit{GeoTrackNet } principle. In the training phase, the model learns a distribution representing AIS tracks in the ROI. Any track that does not follow this distribution will be associated with a very low likelihood and will be considered as abnormal in the test phase.}
    \label{fig:GeoTrackNet}
\end{figure}

\section{Analysis of the performance of GeoTrackNet}
\label{sec:validation}

Bringing such research prototype to an operational system requires, \textit{inter alia}, to address its validation in terms of relevance of the unusual trajectories that are detected, the explainability of the results and the scalability of the model.

\subsection{Relevance of the detected anomalies}
\label{sec:analysis_relevance}

Since \textit{GeoTrackNet } is a probabilistic model, it detects what is probabilistically unusual. These anomalies may not correspond to suspicious activities. As mentioned above, maritime anomaly detection is an intermediate step in MSA. The final purpose is to detect whether actions are needed when an anomaly happens, e.g. enforcing law when a smuggling activity is committed. It is thus operationally important to better understand the types of unusual trajectories raised by the system together with its limitations\footnote{\revise{The aim of this paper is to analyse the types of vessels' behaviours flagged as abnormal by \textit{GeoTrackNet}. For the comparison of the performance of of \textit{GeoTrackNet} and other state-of-the-art models', please refer to \cite{nguyen_geotracknet-maritime_2019}}}.

\revise{It is also important to note that maritime traffic anomaly detection is an ill-defined problems. No universal definitions of abnormal behaviours or groundtruthed datasets are available for this problem, hence the validation of the outcomes of the detector is subjective. We focus on understanding the types of vessels' behaviours will be flagged by \textit{GeoTrackNet}, rather than calculating quantitative measure.}

This analysis involved a manual inspection of the results of \textit{GeoTrackNet } over AIS data acquired around the Ushant Traffic Separation Scheme (TSS) by maritime domain experts
from CLS Group  (Collecte Localisation Satellites).
Those experts are in charge of the operational monitoring of maritime traffic using multiple sources of data (AIS, Earth Observation, Open Source Intelligence, ELINT, etc). For this analysis they took into consideration their knowledge of the maritime activities in the area, the AIS tracks, the log of the vessels, the weather conditions, etc. This analysis allows to highlight the types of trajectories that are raised (or not raised) as unusual.

\subsubsection{Unusual trip}
\label{sec:unusual_trip}

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/running_in.png}
    \caption{Running-in. This vessel made a test voyage after being repaired at the port of Brest.}
    \label{fig:running-in}
\end{figure}

Fig.~\ref{fig:running-in} shows the trajectory of a vessel of type cargo raised by \textit{GeoTrackNet }, steaming to sea and then turning back to the departing port. The visual inspection of the vessels tracks in the dataset shows that this behaviour is indeed unusual. One could consider that this voyage might involve a transshipment. However, after checking other sources of information, we figured out that this vessel had been repaired in the port of Brest (it was berthed in the shipyard), and most probably performed a test trip before going back to the port.
Even if finally not suspicious, we consider relevant to report such tracks as statistically unusual in this location.

\subsubsection{Effect of weather conditions}
\label{sec:weather_conditions}

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/wind.png}
    \caption{Anomaly due to extreme weather condition. Because of a strong wind blew from the opposite direction, this vessel slowed down (in the red zone). Some similar cases were detected at the same night.}
    \label{fig:wind}
\end{figure}

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/wind20170325.png}
    \caption{Wind speed and wind direction on 2017-03-25 at 21:00 UTC from NCEP GFS weather forecast. From this model the wind was blowing from North East in the opposite direction of vessels entering the English Channel with a speed of around 30 knots.}
    \label{fig:wind_speed}
\end{figure}

\textit{GeoTrackNet } does not take into account the environmental conditions.
In extreme weather/ocean conditions, vessels' behaviours change.
By nature, extreme conditions are rare events. Hence, the model did not see enough of these types of behaviours in the training set to include them into the normalcy model.
The changes of behaviour due to specific weather condition may then be flagged  as abnormal behaviours.

An example of such cases is shown in Fig.~\ref{fig:wind}. On the night of March 25, 2017, there was a strong wind blowing from the North East. Some vessels in the area were highly affected by this wind at the entrance of the English Channel just after turning right after the Ushant TSS, while the wind was blowing in the opposite direction with a speed of around of 30 knots.

Again, this behaviour is not suspicious, but raised as statistically unusual. We consider relevant to report such tracks as in such specific weather conditions. Some vessels may be impacted differently by wind drag.

\subsubsection{Weak changes of trajectories}
\label{sec:weak}

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/stop1.png}
    \caption{Example of a weak change in trajectory detected by \textit{GeoTrackNet }.}
    \label{fig:stop}
\end{figure}

As expected \textit{GeoTrackNet } succeeds to raise anomalies on small changes of trajectories mixing both unusual changes of speed and location. Such weak change of trajectory is illustrated in Fig.~\ref{fig:stop}. A vessel left the English channel, slowed down slightly off the usual route, and then started again. Similar behaviours were observed near the TSS in the opposite track (vessel stopping briefly before entering the Channel).

We associated this type of behaviours to the \textit{MARPOL NOx Tier III Regulation before entering/leaving Sulphur Emission Control Areas (SECA)}~\cite{international_maritime_organization_imo_international_nodate}.
Such SECA has been in place since 2015 for the North Sea including the English Channel with a limit at the 5°W meridian. The reporting of such behaviours is relevant as it is likely that the vessels stop their engine before changing their fuel. This may relate to  higher risks of engine failure (not restart after stop) with vessels potentially drifting in the traffic lane or to the coast.

\subsubsection{Unreported anomalies}
\label{sec:unreported}

As \textit{GeoTrackNet } is designed to report statistically unusual tracks, it does not report those that are not sufficiently represented in the training dataset. Without surprise, it thus fails to report vessels crossing the traffic lanes and entering the separation area. This TSS is heavily monitored by the local authorities and the Maritime Rescue Co-ordination Centre (MRCC) and the separation areas can easily be seen for instance in Fig. \ref{fig:new_results}. However, such trajectories can easily be reported by simpler techniques, by just checking the position of AIS messages with respect to those forbidden areas.

\subsection{Scalability of the model}
\label{sec:scalability}

Former results were demonstrated on an area of interest with a limited geographic coverage based on training over terrestrial AIS data~\cite{nguyen_geotracknet-maritime_2019}. Using \textit{GeoTrackNet } on different scenarios raises the following questions: Does it provide similar results when trained and applied over a larger geographic area? Does it provide similar performance using a combination of both terrestrial and satellite AIS data? Does it provide relevant results on areas where maritime traffic is less structured? The evaluation of these questions is ongoing. However, we have preliminary results related to the scalability of the model.

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/new_ROI.png}
    \caption{New dataset. The ROI of the new dataset is 3.4 times bigger than the original. It contains both S-AIS and T-AIS, while the original training set contains T-AIS only.}
    \label{fig:new_ROI}
\end{figure}

We tested the model on a new dataset, which contains both satellite AIS messages (S-AIS) and terrestrial AIS messages (T-AIS)\footnote{The original dataset contains T-AIS only.}, over the same period (January to March 2017). The ROI of the new dataset is 3.4 times bigger than the original one. We used the same hyper-parameters as those used in the previous setting, except the dimensions of the ROI.

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/new_results.png}
    \caption{AIS tracks detected as abnormal by \textit{GeoTrackNet } trained on the new dataset. Blue: AIS tracks in the training set; other colours: detected tracks.}
    \label{fig:new_results}
\end{figure}

The result is shown in Fig.~\ref{fig:new_results}. Almost all the tracks flagged as abnormal by the previous model are detected again by the new model. There are new abnormal tracks because they do not exist in the original dataset (S-AIS tracks or outside of the previous ROI).

The combination of S-AIS and T-AIS seems to improve the performance of \textit{GeoTrackNet}. As shown in Fig.~\ref{fig:s-ais}, a vessel which was falsely detected by the original model is correctly flagged as normal by the new model thanks to the improved training dataset.

\begin{figure}
    \centering
    \includegraphics[width=80mm]{figures/trainingdata_issue.png}
    \caption{An AIS track flagged as abnormal by the original model but considered as normal by the new model. This track is actually normal. However, in the original training set, there are not many tracks of this type (because they are outside of the coverage zone of the terrestrial AIS station), hence it was flagged as abnormal. The coverage area of the new dataset is better, \textit{GeoTrackNet } then succeeded to tag this track as normal.}
    \label{fig:s-ais}
\end{figure}

\section{Online Detection}
\label{sec:onlineDetection}

Within an operational surveillance system of a given area of interest, it is mandatory
to detect suspicious behaviours as early as possible in order to be able to
react accordingly if necessary. For this application, \textit{GeoTrackNet } could be a
tool helping the analysts to sort through the ever increasing flow of AIS data
they have to monitor in the area of interest. In this context, the ability of \textit{GeoTrackNet}, possibly implemented within a distributed framework, to process real AIS data streams and raise alerts in real time is a key issue.

In this section, we first present the proposed framework for the implementation of  \textit{GeoTrackNet } in a streaming context. Then, we report numerical experiments to evaluate scale-up performance.

\subsection{High level principles}
\label{subsec:highLevelPrinciples}

In a streaming context, \textit{GeoTrackNet } is wrapped in a streaming operator: the
\emph{GeoTrackNet Operator}\xspace. This operator is responsible for (a) preprocessing the stream of AIS
messages and (b) triggering the calls to the anomaly detection
function. The preprocessing phase consists in building incrementally the
tracks from the stream of AIS messages. A track is composed of AIS messages
belonging to the same ship (same MMSI) where erroneous positions ({e.g.,}\xspace not in
the ROI) or erroneous speed messages ({e.g.,}\xspace greater than 30 knots) have been
removed. The time difference between consecutive messages in a track is
assumed to be less than a threshold (here 4 hours). Otherwise, they will be
part of two different tracks. Note that tracks are re-sampled to a resolution
of 10 minutes using a linear interpolation.

The \emph{GeoTrackNet Operator}\xspace must then keep hold of the currently active tracks until a minimum
track duration is reached before triggering the anomaly detection. To cope with a
potentially high velocity of the AIS messages stream, the \emph{GeoTrackNet Operator}\xspace can be
replicated. In this situation the stream of input messages needs to be
partitioned such that two messages belonging to the same ship are sent to the
same instance of the \emph{GeoTrackNet Operator}\xspace. We choose to partition the stream of messages
according to the value of the MMSI field to achieve this requirement.

The current implementation of the \emph{GeoTrackNet Operator}\xspace is based on the
Faust\footnote{https://faust.readthedocs.io} library. This library implements
the Kafka\footnote{https://kafka.apache.org} protocol. The Kafka protocol
allows for partitioning the stream of AIS messages regardless of the number of
\emph{GeoTrackNet Operators}\xspace actually deployed. Consequently, the infrastructure can get scaled up and
down when necessary to keep up with the message rate.

\subsection{Synthetic evaluation}

The performance of \textit{GeoTrackNet } is critical as it allows both early anomaly
detection and low resource usage. In this section, a quantitative
evaluation of the computational complexity of \textit{GeoTrackNet } is proposed. For this purpose, our benchmarking experiments rely on
a synthetic set of tracks built from the original dataset~\cite{nguyen_geotracknet-maritime_2019} but covering the
period from July 2011 to January 2018 \footnote{The code to reproduce the experiment in this Section is available at: https://github.com/msimonin/MultitaskAIS/tree/online\_detection/bench}.
In total, $237\,863$ tracks were built, $147\,786$ did not pass the preprocessing
step while $90\,077$ were actually tested for normality. For each tested track the execution
time of \textit{GeoTrackNet } was recorded. Note that this time includes the preprocessing
time. The evaluation has been carried on the Grid'5000 testbed~\cite{balouek_adding_2013}, specifically
on a Dell PowerEdge C6220 of the \textit{Paravance} cluster.
\textit{GeoTrackNet } was given one CPU core and the execution time
statistics are depicted in Fig.~\ref{fig:synthetic}.
\begin{figure}
\begin{center}
    \begin{tabular}{lr}
    mean  & $2.07$ \\
    \hline
    std   & $0.22$ \\
    \hline
    min   & $1.49$ \\
    \hline
    $q_1$ &	$1.93$ \\
    \hline
    median& $2.04$ \\
    \hline
    $q_3$ & $2.19$ \\
    \hline
    max   & $3.74$ \\
    \hline
    \end{tabular}
    \caption{Statistics of the execution time of \textit{GeoTrackNet } on $90\,077$
    tracks. On average, it takes $2,07$s to test the normality of a track.
    Only the tracks that passed the preprocessing phase are accounted here.}
    \label{fig:synthetic}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
    \includegraphics[width=\linewidth]{figures/uniq_mmsi.pdf}
    \caption{CDF of the number of unique MMSIs received in a 10 minutes time window for the
original dataset (2011-2018). Reading: 80\% of the time less than 80 unique
MMSIs are received on a 10 minutes time window.}
    \label{fig:synthetic:cdf}
\end{center}
\end{figure}
In our setup, \textit{GeoTrackNet } can process on average 0.5 tracks per second using one
CPU core. Additionally, due to the preprocessing phase, points are added to a
track every 10 minutes if the rate of messages received for this MMSI is at some point lower than that. Fig.~\ref{fig:synthetic:cdf} depicts the amount of unique MMSI for which at least one message is received in a fixed window. More specifically, according to the graph, a maximum of 400 unique MMSIs are received in a time window of 10 minutes. In other words, for this dataset and ROI, the peak rate at which the \emph{GeoTrackNet Operator}\xspace has to be called is 400 times per 10 minutes, but most of the time, it is significantly lower that that (as can be seen on the picture, 90\% of the time, less than 100 unique MMSI are received in a time window of 10 minutes). This leads us to the conclusion that 2 CPU cores are sufficient to treat all the tracks of the ROI.

\section{Conclusions and Perspectives}
\label{sec:conclusions}

\textit{GeoTrackNet } is a computer-aided system designed to report statistically unusual vessels' tracks. We analysed the validity of this system in terms of relevance of the reported anomalies, scalability of the model, computing performance and possibility to process data streams.

We showed that the model is able to report unusual behaviour that may be difficult to detect with simple rule-based geofencing and limits on course over ground (COG) or speed over ground (SOG). As expected, it fails to detect rare events in area of low traffic. However, most of them can be easily detected by other techniques such as geofencing. It is thus a promising complement in an operational system.

The current version of \textit{GeoTrackNet } does not take into account environmental information. A post-processing step where the ocean current, the wind speed, etc. are used to detect unusual vessels' behaviours due to extreme weather can be added in the future.

\section{Acknowledgment}
\label{sec:Acknowledgment}

This work was supported by public funds (Minist\`ere de l'Education Nationale, de l'Enseignement Sup\'erieur et de la Recherche, FEDER, R\'egion Bretagne, Conseil G\'en\'eral du Finist\`ere, Brest M\'etropole) and by Institut Mines T\'el\'ecom, received in the framework of the VIGISAT program managed by ``Groupement Bretagne T\'el\'ed\'etection'' (BreTel). The authors acknowledge the support of DGA (Direction G\'en\'erale de l'Armement) and ANR (French Agence Nationale de la Recherche) under reference ANR-16-ASTR-0026 (SESAME initiative) and AI Chair OceaniX, the labex Cominlabs, the Brittany Council and the GIS BRETEL (CPER/FEDER framework).

The dataset used in this paper is provided by CLS and Erwan Guegueniat.

The analysis of abnormal tracks in this paper was carried out using the Maritime Awareness System (MAS) of CLS.

Experiments in Section \ref{sec:onlineDetection} were carried out using the Grid'5000
testbed, supported by a scientific interest group hosted by Inria and including
CNRS, RENATER and several Universities as well as other organizations.