\section{Introduction}

	High-resolution linear spectropolarimetry measures the change in linear polarization across a spectral line and is a useful probe of circumstellar environments at small spatial scales.  Circumstellar disks, rotationally distorted winds, magnetic fields, asymmetric radiation fields (optical pumping), and in general, any scattering asymmetry can produce a change in linear polarization across a spectral line such as $H\alpha$.  These signatures can directly constrain the density and geometry of the circumstellar material.  Typical spectropolarimetric signals are small, often a few tenths of a percent change in polarization across a spectral line.  Measuring these signals requires very high signal to noise observations and careful control of systematics to measure signals at the 0.1\% level.

	In this letter, we present the spectropolarimetric variability, as well as spectroscopic variability, of the Herbig Ae/Be stars AB Aurigae, MWC480, MWC158, MWC120, and HD58647.  To date, only a few detections of spectropolarimetric signals in Herbig AeBe's have been reported, and the variability of these signatures has not been studied in detail (Vink et al. 2002, 2005  Mottram et al. 2007).  We show that variability is significant, and show how it can provide information about the near-star environment with future modeling.

	The H$\alpha$ line in these stars is very strong, having line/continuum ratios of roughly 3 to 12, typically with P-Cygni profiles or central reversals.  Our observations of AB Aurigae, MWC480 and MWC120 show P-Cygni profiles with strong variability of the blueshifted absorption component, often over 10-minute timescales.  This is entirely consistent with other studies (Catala et al. 1999, Beskrovnaya et al. 1995).  The H$\alpha$ lines of MWC158 and HD58647 showed strong central reversals and were much more stable, though we had fewer observations.

	 The amplitude of the change in linear polarization across the H$\alpha$ line is roughly 1\% for AB Aurigae, MWC480, and MWC158, while HD58647 and MWC120 show smaller, but still significant signatures.  These polarization changes are all centered on the absorption component, not on line center, and almost always have a single-loop trajectory in QU space, a so-called QU-loop.

	Many types of polarization effects are known in optical astronomy, some related to scattering and others relating to atomic and molecular processes.  Polarization effects can be seen in broad-band continuum polarization, or as changes in polarization resolved across a spectral line.  Early analytical studies showed the possibility of spectropolarimetric effects from scattering very close to the central star (McLean 1979, Wood et al. 1993 \& 1994).  Recent monte carlo modelling of scattering by circumstellar materials has shown a wealth of possible polarimetric line-effects from disks, winds, and envelopes (Vink et al. 2005, Harries 2000, Ignace 2004).  For example, unpolarized line emission that forms over broad stellar envelopes can produce a depolarization in the line core relative to the stellar continuum.  Small clumps in a stellar wind that scatter and polarize significant amounts of light can enhance the polarization at that clump's specific velocity and orientation.

	This technique probes small spatial scales, being sensitive to the geometry and density of the material near the central star.  Even for the closest young stars (150pc), these spatial scales are smaller than 0.1 milliarcseconds across and will not be imaged directly, even by 100m telescopes.  Since the circumstellar material is involved in accretion, outflows, winds and disks, with many of these phenomena happening simultaneously, spectropolarimetry can put unique constraints on the types of densities and geometries of the material involved in these processes.

	A preliminary study of the H$\alpha$ line at medium spectral resolution (R$\sim$8500, rebinned heavily) in Herbig Ae/Be stars showed many different morphologies and amplitudes (Vink et al. 2002).  Some showed polarization changes as high as 2\%, while others showed none at all.  Since the models predicting polarization across spectral lines are not currently invertable and predict spectropolarimetric effects centered on the emissive core, we wanted to do an in-depth study of a few sources to see if the variability of the spectropolarimetric line profiles could shed some light on the nature of the near-star environment.

\section{Targets}

	Since spectropolarimetry is a photon-hungry technique, we wanted to apply this technique to bright, well-studied stars that had previously-detected spectropolarimetric signatures to monitor their variability and the nature of the polarimetric signatures.  We chose AB Aurigae and MWC480 for close study, and MWC120, MWC158, and HD58647 as other bright observable targets.

	The Herbig Ae star, AB Aurigae (HD31293, HIP22910) is the brightest of the Northern hemisphere Herbig Ae stars (V=7.1) and is one of the best studied intermediate-mass young stars.  It has a near face-on circumstellar disk resolved in many wavelengths (eg: Grady et al. 2005, Fukagawa et al. 2004).  It also has an active stellar wind with it's strong emission lines often showing strong P-Cygni profiles.  Spectroscopic measurements put AB Aurigae somewhere between late B and early A spectral types (B9 in Th\'{e} et al. 1994, B9Ve in Beskrovnaya et al. 1995, A0 to A1 Fernandez et al. 1995) with an effective temperature of around 10000K.  The star has a wind that is not spherically symmetric with a mass loss rate of order $10^{-8} {M_\odot}$ per year, and an extended chromosphere reaching $T_{max}\sim$17000K  at 1.5$R_\ast$ (Catala \& Kuasz 1987,  Catala et al. 1999).  A short-term variability study done by Catala et al. (1999) showed that an equatorial wind with a variable opening angle, or a disk-wind originating 1.6$R_\ast$ out with a similar opening angle could explain the variability.

	There were only two previous high-resolution spectropolarimetric observations of AB Aurigae.  One was a single data set taken in 1999 and published with two different papers (Vink et al. 2002, Pontefract et al. 2000), another was taken in 2004 (Mottram et al. 2007).  The polarization varied by roughly 0.4\% to 0.7\% across the line, but to achieve the required S/N in each resolution element, the polarization spectra were rebinned to constant flux with a lower effective resolution of 2700 (25 elements over 60$\AA$ but with varying spectral coverage).  We are uncertain why the QU loops changed shape between the Pontefract et al. 2000 and Vink et al. 2002 papers, but there is clearly a shape change between Vink et al. 2002 and Mottram et al. 2007 showing evidence of moderate variability.

		  MWC 480 and MWC120 also showed strong blue-shifted absorption components.  The H$\alpha$ line and the continuum polarization of MWC480 had been studied in detail by Beskrovnaya \& Pogodin (2004) who concluded that MWC480 also had an inhomogenious wind which was variable on short timescales.  MWC480 had a large amplitude signature ($\sim$1.8\%) in Vink et al. (2002), but showed a less significant signature in Mottram et al. 2007 again pointing to variability.

	  MWC158 and HD58647 showed strong absorption near line-center.  MWC158 is a mid-B type star, and was previously studied for spectroscopic variability as well as low-resolution spectropolarimetry (Bjorkman et al. 1998, Pogodin 1998, Jaschek \& Andrillat 1998).  HD58647 is a late B type star (B9 in Th\'e et al. 1994).  H$\alpha$ line spectropolarimetry for MWC120 and MWC158 was presented by Oudmaijer et al. (1999) showing line effects.  All stars had signatures in Vink et al. 2002.  Clearly the signatures are at least mildly variable long timescales, partly motivating this study.

\section{Observations}

	We observed our targets on 8 nights during the engineering of the HiVIS spectropolarimeter (with 5 more lost to weather) in 2004 and on 27 nights over the fall and winter (with 13 more lost to weather) of 2006-2007.  We observed AB Aurigae or MWC480 continuously for several hours on some nights, and all 5 targets intermittently on others with a focus on AB Aurigae and MWC480.  We have a total of 148 polarization measurements for AB Aurigae, 58 for MWC480, 24 for MWC120, 39 for MWC158, 19 for HD58647, plus 33 unpolarized standard star observations taken over the 40 nights in 2006-2007.  We achieved a continuum S/N of typically 500 or better for our observations.  The polarization data were subsequently binned-by-flux to a nearly constant S/N for each resolution-element, typically 800-1000, accounting for the 0.1\%-0.2\% noise seen in the continuum polarization measurements.

	The H$\alpha$ line for all stars showed significant variability in intensity, width, and profile shape that is entirely consistent with other spectroscopic variability studies (Beskrovnaya et al. 1995, Beskrovnaya \& Pogodin 2004, Catala et al. 1999).  Figure \ref{fig:lprof} shows the average H$\alpha$ line for AB Aurigae for each night to illustrate the nightly variations.  Figure \ref{fig:pcyg} shows the absorption-trough of AB Aurigae on five of the most variable nights.  On some nights, AB Aurigae and MWC480 showed dramatic spectroscopic variation on a timescale of minutes to hours, mostly in the blueshifted absorption trough.  Some general line width and line strength variability was also seen on short timescales.  All nights showed much smaller but significant variation.

	Each star showed a significant change in linear polarization above 0.2\%.  Figure \ref{fig:rawpol} shows examples of the spectropolarimetry for AB Aurigae before any flux-dependent binning or frame-rotation, illustrating the quality of the data.  The full spectropolarimetry for all five targets and many unpolarized standards is shown in figure \ref{fig:swap-all}.  The continuum has been removed and they have been de-rotated to a common instrumental QU frame (but not to the celestial sphere).  It should be noted that a full polarization-calibration of the telescope is not yet complete.  The telescope induces continuum polarization at the few percent level and can rotate the plane of polarization, as described in Harrington et al. 2006.  We suspect this rotation to be a large contributor to the aparent variability of the polarization in figure 3.  However, these telescope properties have a weak wavelength dependence and are not expected to cause significant polarization effects across a single spectral line, and only amount to rotations, attenuations, and translations of the QU loops in QU space.

	The change in polarization occurred in the absorption component, either central or P-Cygni, for all five stars.  The change was strongest ($\sim$1\%) for the stars with the strongest P-Cygni absorptions (AB Aurigae and MWC480).  There was also a significant change of about 0.5\% in the central reversal of MWC158.  The change was much weaker in MWC120 and HD58647, which showed a P-Cygni absorption and strong central reversal respectively.

	AB Aurigae exhibits some intrinsic spectropolarimetric variability.  Even though we don't have a full model for the telescope's polarization effects yet, observations at a single pointing over many different nights show changes in the shapes and widths of the polarization spectra.  The overall width and structure of the polarization spectra certainly change from night to night, regardless of what telescope calibration we will apply to this data in the future.

\section{Discussion}

	In the analytic studies of McLean (1979) and Wood et al. (1993 and 1994), and in subsequent monte-carlo models of Vink et al. (2005), the simple "disk-scattering'' polarization models predict spectropolarimetric signatures centered on the emissive core. The models use the stellar surface (chromosphere) as the source of a broad unpolarized emission line.  This line flux is then scattered by the circumstellar material, which doppler-shifts and polarizes the scattered flux.  This scattered light causes the polarimetric effects when added to the original unpolarized line.  McLean (1979) also mentions a depolarization effect, where the stellar continuum is polarized, and unpolarized H$\alpha$ emission depolarizes the starlight across the line.  This effect would also be strongest in the emissive core.  In our stars, the change in polarization occurred in and around the absorptive component, whether central or blue-shifted, and the polarization near the emission peak was nearly identical to the continuum polarization.  McLean (1979) did mention another effect in a P-Cygni absorption trough, but only when there is a signature in the emissive component.  The lack of models explaining signatures only in the absorptive component led us to explore other explanations that would require the absorbing material to also be the polarizing material.

	We are developing a new model where an anisotropic radiation field causes the absorbing material to polarize the transmitted light.  The polarization originates from the anisotropy in the lower level populations of the n=2 to n=3 H$\alpha$ transition in the intervening gas.  Anisotropic radiation from the star excites the intervening gas and leads to a population anisotropy in the n=2 substates (called optical pumping).  The anisotropy causes the absorbing material to absorb different incident polarizations by different amounts.  The main difference between this model and the scattering model is that only the absorbing material, the material occulting the photosphere and chromosphere, is responsible for the changing polarization, whereas the scattering models integrate scattered light from the entire circumstellar region with each part contributing to the polarization change.

	In AB Aurigae, the H$\alpha$ photons are thought to come from an extended chromosphere, out to 1.5$R_\ast$ with the P-Cygni absorption occuring further out where the incident radiation anisotropy is significant (Catala et al. 1999).  The optical pumping model would produce polarization where there is absorption of the underlying H$\alpha$ emission.  The optical pumping model we are developing shows good promise of giving a direct constraint on the density and geometry of the absorbing material, and thus a possible way of determining the circumstellar material's physical properties (Kuhn et al. submitted).  While the depolarization explanation and the electron-disk scattering models of Vink et al. (2005) could explain many of their observations, they do not explain polarization effects isolated in the absorptive component.  The optical pumping model can explain these absorption-only polarization effects.  We compiled this very large high-precision data set to show the diversity of spectripolarimetric effects and hope that future modeling efforts will allow us to use such data to constrain the density and geometry of circumstellar material.

\acknowledgements
This work was partially supported by the NSF AST-0123390 grant.  We wish to thank Katie Whitman for help during the engineering observations and for discussions about data reduction.  We also wish to thank Don Mickey for many stimulating discussions about telescope polarization.


\section{Introduction}
 The study of high energy scattering near the horizon of a black hole has lead to an improved perspective on quantum chaos \cite{LO, Dray:1984ha, SS, KitaevTalks, MSS}. The scrambling of information near the horizon of the black hole is related to the chaotic spread of information on the boundary quantum system.

 The signature of quantum chaos used in this context is given by the exponential growth of the square of double commutators $\langle |[A(0),B(t)]|^2 \rangle_\beta$ evaluated at a thermal state with inverse temperature $\beta$,  \cite{ KitaevTalks,LO}. The main object needed to compute this double commutator is the real part of the out-of-time-ordered (OTOC) correlator $\langle A^\dagger(0) B^\dagger(t) A(0) B(t) \rangle_\beta$. The connected part of the OTOC grows exponentially with time in a chaotic system, with a rate defined as the Lyapunov exponent $\lambda$. This growth happens long after the dissipation time $t_d$ controlled by the thermalization scale of time-ordered correlators. At a much larger scrambling time $t_{\rm sc}$ the connected part of the OTOC, and also the commutator, saturate.

 In reference \cite{SS} it was explained in detail how, in holographic theories, the bulk computation of an OTOC is given by a high energy scattering near the black hole horizon \cite{Dray:1984ha}. The result is a convolution between wavefunctions (bulk-boundary propagators), that evolve the particles from the boundary to the near horizon region, and a local high energy S-matrix near the horizon. A high energy gravitational scattering is equivalent to a classical shockwave interaction \cite{Dray:1984ha}. This gives a Lyapunov exponent $\lambda = 2 \pi / \beta$ which was shown in \cite{MSS} to be maximal. When doing this calculation in string theory, and assuming inelastic effects are subleading, reference \cite{SS}, building upon \cite{Brower:2006ea}, explains how the sum over all stringy modes are equivalent to an effective elastic Pomeron which produces a perturbative correction $\lambda = \frac{2\pi }{\beta} (1 - \mathcal{O}(\alpha'))$, where $\alpha'$ is related to the string tension, which is analogous to the flat space Regge asymptotics. On the other hand, the saturation of the OTOC at the scrambling time is related to the decay of the bulk-boundary propagators and therefore to the quasinormal modes of the black hole.

So far the attention has focused on OTOC that appear on double commutators, since they are more directly related to the definition of chaos of \cite{LO} as explained above. In this note we will analyze general OTOC between four arbitrary operators. For holographic theories, the importance of these quantities is more evident in the bulk. When the operators are different, the bulk scattering is completely inelastic and the Pomeron controlling these OTOC does not necessarily have the quantum numbers of the vacuum, for example. In particular, gravity plays no role in the local near horizon bulk interaction, and the growing piece of these correlators probe interactions that are not universal. Therefore it is interesting to study them to get more information about the bulk.

In this paper we will extend the chaos bound and constrain arbitrary out-of-time ordered correlators. The assumptions we will use are the same as in the original chaos bound of \cite{MSS}. In particular, we will focus for simplicity on hermitian operators although we will relax this assumption later. For the exponential ansatz of an OTOC we will use the notation
\begin{eqnarray}\label{intro:eq:}
{\rm Re}~{\rm Tr} [ y A(0) y B(t) y C(0) y D(t) ] \approx F_d -  \varepsilon_{ABCD} e^{\lambda_{ABCD} t},
\end{eqnarray}
where by $F_d$ we denote the order one factorized approximation (which implicitly also depends on the choice of operators) and $\varepsilon$ is a small correction. Following \cite{MSS} we define $y$ such that $y^4=e^{-\beta H}/Z$.   For systems with a large number of degrees of freedom $N$ the amplitude of the growing piece is of order $\varepsilon\sim N^{-2}$ while the factorized piece is generically of order one. Unless all four operators $A$, $B$, $C$ and $D$ are all different, the correlator in \eqref{intro:eq:} is real. Correlators involving combinations $ABAB$, $ABCB$ or $ABAD$ are all real.

We will refer to configurations such as $ABAB$ that appear on double commutators as `diagonal' or `elastic' OTOC, while we will refer to the generic OTOC as `off-diagonal' or `inelastic' correlators.

In this notation, the chaos bound of \cite{MSS} is a statement about the positivity of the prefactor $\varepsilon_{ABAB}>0$ and a bound on the growth rate $\lambda_{ABAB} \leq 2\pi/\beta$. This is valid for any choice of $A$ and $B$, even though in most examples the Lyapunov exponent is independent on the choice of operators. We will show (see details in section \ref{sec:proof}) that the inelastic OTOC for four different operators is also similarly constrianed
\begin{equation}
\lambda_{ABCD} \leq \lambda_{\rm diag} \leq \frac{2\pi}{\beta},
\end{equation}
where $\lambda_{\rm diag} \equiv {\rm min} (\lambda_{ABAB},\lambda_{ADAD},\lambda_{CBCB}, \lambda_{CDCD})$. This means a generic off-diagonal OTOC cannot grow faster than diagonal ones. From the gravity side, this puts a bound on the spin of the effective mode controlling this interaction, it cannot be bigger than $2$.

It is reasonable to expect all diagonal or off-diagonal OTOC for arbitrary $A$, $B$, $C$, $D$ to grow with the same rate $\lambda_L$, even if not maximal \footnote{For example in the SYK model \cite{Sachdev:1992fk, KitaevTalks,Maldacena:2016hyu,Kitaev:2017awl,Gu:2018jsv} the exponentially growing piece always comes from the same set of ladder diagrams, regardless of how we glue it to external operators.}. With this assumption, we can also bound the amplitude of the growing piece. In the general case of four different operators the constraint is presented in section \ref{sec:proof}. If we take two of the operators to be the same then we can write a simpler version
\begin{equation}\label{ec:introbound}
(\varepsilon_{ABCB})^2 \leq \varepsilon_{ABAB} \varepsilon_{CBCB},
\end{equation}
and similarly for $\varepsilon_{ABAD}$. The same structure is present for the case of an OTOC between four different operators, $\varepsilon_{ABCD}$  is bounded by the prefactors appearing in diagonal correlators. From the gravity side this puts a bound on how strongly can matter couple to the effective mode controlling this interaction.

In the context of holographic theories, the coefficient in the left hand side of the inequality \eqref{ec:introbound} is given by an inelastic scattering between particles in the bulk, which in general does not involve graviton exchange. It is interesting to see that we can bound such a process by the right hand side, which is universally fixed by gravitational interactions and the equivalence principle. Even though the inelastic coupling $\varepsilon_{ABCD}$ does not necessarily have a definite sign, its magnitude cannot be bigger than a mean of the gravitational couplings. In this context, this analysis suggests that gravity is the highest spin interaction, and the strongest with that spin.

In section \ref{sec:nonh} we generalize the chaos bound to non-hermitian operators. Then similar constraints on an OTOC between four non-hermitian operators can be derived.

In section \ref{sec:2dCFT} we make some comments regarding the behavior of inelastic OTOC for 2d CFT.  In \cite{Roberts:2014ifa} the authors show how the maximal chaos exponent is controlled by the dominance of the identity Virasoro block. For an OTOC between different operators the identity channel does not appear in the OPE expansion. Using semiclassical expressions for non-vacuum blocks at large central charge $c$ we study the behavior of off-diagonal OTOC. In particular, we see that before the scrambling time, Virasoro descendants are not important in the second sheet. This shows how gravity naively plays no role in the physics of these OTOCs. After the scrambling time at which exponential growth stops, we show how quasinormal modes dictate the decay of the OTOC.

We conclude in section \ref{Sec:conc} with open questions and future directions.

\section{Constraints on generic OTOC}\label{sec:proof}

In this section we will show how to bound general OTOC between arbitrary operators. The argument is simple but requires some notation. In order to do that, we will begin by stating the chaos bound from \cite{MSS}, which we will refer to as the elastic chaos bound.

In \cite{MSS} the authors focus on a particular correlator
\begin{equation}
F(t) \equiv {\rm Tr} [ y A(0) y B(t) y A(0) y B(t) ],
\end{equation}
between hermitian operators $A$ and $B$, where divergences are regularized by placing them symmetrically along the euclidean circle of size $\beta$. This is implemented by inserting the operators $y$ defined as $y^4=e^{-\beta H}/Z$. The motivation for considering such correlators comes from its relation to commutators square between operators $A(0)$ and $B(t)$.

We will consider times that are much larger than the dissipation time $t_d$ but smaller than the scrambling time $t_{sc}$, which we assume to be parametrically larger (as in, for example, large $N$ theories). In this regime the OTOC is almost constant and given by its factorized contribution $F_d$ to leading order, where
\begin{equation}
F_d \approx {\rm Tr} [ y^2 A y^2 A] {\rm Tr} [ y^2 B y^2 B]
\end{equation}
For chaotic systems we expect the subleading behavior to have an exponential behavior
\begin{equation}\label{eq:ans}
F(t) = F_d - \varepsilon ~e^{\lambda t} +\ldots
\end{equation}
where $\lambda$ is the Lyapunov exponent of the system. The parameter $\varepsilon$ is a small constant which controls the scrambling time at which the OTOC decays. For a large $N$ system it is of order $\varepsilon \sim N^{-2}$. From now on we will denote these prefactors of exponentially growing terms by $\varepsilon$ to denote they are small compared to the factorized term.

The chaos bound from \cite{MSS} states that the quantity $F(t)$ is bounded by the right hand side of equation \eqref{eq:ans} with both
\begin{equation}
\varepsilon \geq 0 ~~~{\rm and}~~~\lambda \leq \frac{2\pi}{\beta}
\end{equation}
We will take this as our starting point for the generalizations below. Therefore we will implicitly use the same assumptions and caveats as in \cite{MSS}.

\subsection{An inelastic chaos bound}

In this section we will prove the bound stated in the introduction regarding OTOC between four different operators. We will focus first on hermitian operators. The upshot is that the growing piece of a general OTOC  cannot grow faster than exponentially, with the maximal rate $\lambda = 2\pi/\beta$. We will also see how to put a bound on the magnitude of the growing piece.

To simplify the presentation, we will go over the argument in steps. We will first generalize the chaos bound to a correlator ${\rm Tr} [ y A(0) y B(t) y C(0) y B(t) ]$. This OTOC is real for arbitrary operators since
\begin{equation}\label{eq:realABCB}
{\rm Tr} [ y A y B(t) y C y B(t) ]^\dagger = {\rm Tr} [ B(t) y C y B(t) y A y ]={\rm Tr} [ y A y B(t) y C y B(t) ],
\end{equation}
where we used that the operators are hermitian. In the first line we applied the hermitian conjugation inside the trace and in the second one used the cyclic property of the trace\footnote{Similarly, one can show that $ {\rm Tr} [ y A y B(t) y A y D(t) ]$ is real and symmetric under exchange of $B$ and $D$. }. Moreover, the OTOC is also symmetric under the exchange of $A$ and $C$
\begin{equation}\label{eq:symABCB}
{\rm Tr} [ y A y B(t) y C y B(t) ]={\rm Tr} [ y C y B(t) y A y B(t) ]
\end{equation}
which follows from the cyclic property of the trace.

To bound ${\rm Tr} [ y A(0) y B(t) y C(0) y B(t) ]$ we will analyze a diagonal correlator of the form
\begin{eqnarray}\label{eq:corrvwvw}
F(t)= {\rm Tr} [ y V y B(t) y V y B(t) ]  ,~~~ V=\alpha_1 A + \alpha_2 C,
\end{eqnarray}
for arbitrary real coefficients $\alpha_1$ and $\alpha_2$. To simplify the notation, we omit the time argument when the operator is inserted at $t=0$. Expanding each term in the right hand side of equation \eqref{eq:corrvwvw} gives
\begin{eqnarray}\label{eq:ABCBexpand}
F(t)  &=& \alpha_1^2  {\rm Tr} [ y A y B(t) y A y B(t) ]+\alpha_2^2  {\rm Tr} [ y C y B(t) y C y B(t) ] \nn
&& + 2 \alpha_1 \alpha_2 {\rm Tr} [ y A y B(t) y C y B(t) ].
\end{eqnarray}
This contains the correlator we want to bound. There for by using the information we learn from the chaos bound on diagonal OTOC we can bound off-diagonal OTOC such as ${\rm Tr} [ y A y B(t) y C y B(t) ]$.

Before we move on we can write an ansatz for these OTOC similar to equation \eqref{eq:ans}. For concretenes and to set notation we write
\begin{eqnarray}
{\rm Tr} [ y A y B(t) y A y B(t) ] &=& F^{AA}_d - \varepsilon_{AA}~ e^{\lambda_{AA} t},\\
{\rm Tr} [ y C y B(t) y C y B(t) ] &=& F^{CC}_d-\varepsilon_{CC} ~e^{\lambda_{CC} t},\\
{\rm Tr} [ y A y B(t) y C y B(t) ] &=&F^{AC}_d- \varepsilon_{AC} ~e^{\lambda_{AC} t}.
\end{eqnarray}
Where we indicate the dependence on the operators of the factorized leading contribution $F_d$, the amplitude of growing piece $\varepsilon$ and rate $\lambda$. We leave the dependence on the operator $B$ implicit. From \eqref{eq:realABCB} and \eqref{eq:symABCB} we know that $F_d^{AC}=F_d^{CA}$, $\varepsilon_{AC}=\varepsilon_{CA}$ and $\lambda_{AC}=\lambda_{CA}$ are real.

To leading order, the right hand side of equation \eqref{eq:ABCBexpand} above is approximately constant in time, order one, and given by
\begin{equation}
F_d \approx \alpha_1^2 F_d^{AA} + \alpha_2^2 F_d^{CC} + 2 \alpha_1 \alpha_2 F_d^{AC}.
\end{equation}
This quantity is positive for any choice of $\alpha_1$ and $\alpha_2$. This can be shown using Cauchy-Schwarz or more directly by starting from expression \eqref{eq:corrvwvw} in terms of $V$. Moreover, we might also diagonalize the $2\times 2$ matrix of two-point functions between $A$ and $C$ such that $F_d^{AC}=0$, making the equation above manifestly positive.

Next, we will focus on the subleading piece growing in time. We will consider first the most general case where all rates exponents are allowed to be different. Using this ansatz for the maximal growth we can write the subleading part of the OTOC as
\begin{equation}\label{eq:ABCBsublead}
F_d - F(t)= \alpha_1^2 \varepsilon_{AA} e^{\lambda_{AA} t} + \alpha_2^2 \varepsilon_{CC}  e^{\lambda_{CC} t} + 2 \alpha_1 \alpha_2 \varepsilon_{AC} e^{\lambda_{AC} t}.
\end{equation}
 Since $\alpha_1$ and $\alpha_2$ are arbitrary coefficients, and by the elastic chaos bound, we can conclude that $\lambda_{AA}, \lambda_{CC}, \lambda_{AC} \leq 2\pi/\beta$. Otherwise we could form a linear combination of $A$ and $C$ such that $F(t)$ could violate the chaos bound. Moreover we can also argue that $\lambda_{AC} \leq {\rm min}(\lambda_{AA},\lambda_{CC})$. Otherwise eventually the mixed term would dominate and we could choose a sign of the $\alpha$'s which would give a negative prefactor, also violating the chaos bound. In other words, $\lambda_{AC} > {\rm min}(\lambda_{AA},\lambda_{CC})$ would imply $\varepsilon_{AC}=0$.

Above we considered the most general case. Now we will assume that all OTOC grow with the same rate $\lambda$. In this case the chaos bound on the prefactor sign gives  us a bound
\begin{equation}
\alpha_1^2 \varepsilon_{AA} + \alpha_2^2 \varepsilon_{CC}  + 2 \alpha_1 \alpha_2 \varepsilon_{AC} \geq 0, ~~~~\forall ~\alpha_1,\alpha_2
\end{equation}
coming from the diagonal chaos bound applied to the right hand side of equation \eqref{eq:ABCBsublead}. This condition is equivalent to the following constraint
\begin{equation}\label{eq:boundabac}
\varepsilon_{AC}^2 \leq \varepsilon_{AA} \varepsilon_{CC}
\end{equation}
From this condition we see that even though we can constrain the growth of $\langle A B CB \rangle$, the chaos bound does not constrain the sign of the correction, which could be positive or negative but with a magnitude bounded by $\sqrt{\varepsilon_{AA} \varepsilon_{CC}}$. This is also analogous to the ANEC case studied in \cite{CMT} (see also \cite{Meltzer:2017rtf}).

Having done this, the obvious next step is to consider other linear combinations. An option is
\begin{equation}\label{eq:ABCDlc}
F(t)= {\rm Tr} [ y A y W(t) y A y W(t) ]  ,~~~ W=\alpha_1 B + \alpha_2 D.
\end{equation}
The chaos bound applied to this correlator gives analogous bounds as the previous analysis for the (real) correlator $ {\rm Tr} [ y A y B(t) y A y D(t) ]$. Namely, the growing piece cannot grow too fast and the amplitude cannot be bigger than diagonal one. Instead, to obtain new bounds, we will consider
\begin{equation}\label{eq:ABCDlc}
F(t)= {\rm Tr} [ y A y W(t) y C y W(t) ]  ,~~~ W=\alpha_1 B + \alpha_2 D
\end{equation}
with real coefficients $\alpha_1$ and $\alpha_2$. Then we can use the inelastic chaos bound derive above to constrain ${\rm Tr} [ y A y B(t) y C y D(t) ]$. We again assume an exponential ansatz on each term. A new feature of the most general case is that the mixed term now is not real anymore since
\begin{equation}
{\rm Tr} [ y A y B(t) y C y D(t) ]^\dagger = {\rm Tr} [ y A y D(t) y C y B(t) ] = {\rm Tr} [ y C y B(t) y A y D(t) ].
\end{equation}
This means that exchanging $A \leftrightarrow C$ or $B \leftrightarrow D$ are related by complex conjugation. Only a simultaneous exchange of $A \leftrightarrow C$ and $B \leftrightarrow D$ is a symmetry. From expanding the right hand side of \eqref{eq:ABCDlc} we see it is only sensitive to the real part of ${\rm Tr} [ y A y B(t) y C y D(t) ]$.

To set notation we write the exponential ansatz for the correlators as
\begin{eqnarray}
{\rm Tr} [ y A y B(t) y A y B(t) ] &=& F_d^{ABAB} - \varepsilon_{ABAB} ~e^{\lambda_{ABAB} t}, \\
{\rm Tr} [ y A y B(t) y C y B(t) ] &=& F_d^{ABCB} - \varepsilon_{ABCB} ~e^{\lambda_{ABCB} t},\\
{\rm Re}~{\rm Tr} [ y A y B(t) y C y D(t) ] &=& F_d^{ABCD} - \varepsilon_{ABCD} ~e^{\lambda_{ABCD} t}.
\end{eqnarray}
In these expressions all quantities on the right hand side are real. In the third line, after taking the real part, the quantities are symmetric under independently exchanging $A$ and $C$ or $B$ and $D$.

Now we can expand the right hand side of \eqref{eq:ABCDlc}. Again, the factorized contributions $F_d$ give some leading constant piece for the correlator in \eqref{eq:ABCDlc} and we will focus on the subleading growing piece. From the constraint on the growth rate in time we obtain the following bound quoted in the introduction
\begin{equation}
\lambda_{ABCD} \leq \lambda_{\rm diag} \leq  \frac{2\pi}{\beta},
\end{equation}
where $\lambda_{\rm diag} \equiv {\rm min} (\lambda_{ABAB},\lambda_{ADAD},\lambda_{CBCB}, \lambda_{CDCD})$. Namely, if the rate of growth are different for each term, we can say that $\lambda_{ABCD}$ is smaller than the minimum of $\lambda_{ABAB}$, $\lambda_{CBCB}$, etc, which are all smaller than $2\pi/\beta$ by the chaos bound.

Similarly to the previous case, we can assume all OTOC have the same rate of growth, and then we can also bound the amplitude $\varepsilon$ of the growing piece. The bound we obtain from the previous analysis, equation \eqref{eq:boundabac}, is
\begin{eqnarray}\label{gencond}
&&\hspace{-0.5cm}(\alpha_1^2 \varepsilon_{ABAB} +\alpha_2^2 \varepsilon_{ADAD} + 2 \alpha_1 \alpha_2 \varepsilon_{ABAD} )(\alpha_1^2 \varepsilon_{CBCB} +\alpha_2^2 \varepsilon_{CDCD} + 2 \alpha_1 \alpha_2 \varepsilon_{CBCD} )\nn
&&~~~-(\alpha_1^2 \varepsilon_{ABCB} +\alpha_2^2 \varepsilon_{ADCD} + 2 \alpha_1 \alpha_2 \varepsilon_{ABCD} )^2\geq 0,
\end{eqnarray}
which should be satisfied for any choice of $\alpha_1$, $\alpha_2$. Since this condition is invariant under a rescaling of $\alpha_i \to \lambda \alpha_i$, we can fix $\alpha_1=1$. Then this condition \eqref{gencond} is equivalent to the positivity of a quartic polynomial on the variable $\alpha_2$ with coefficients depending on the $\varepsilon$'s. \footnote{For a general quartic polynomial $P(x)=a x^4 + b x^3 + c x^2 + d x + e $ it should have $a,e>0$ and the condition having four complex roots is to have a positive discriminant $\Delta(P)\geq0$, and a positive $8 ac-3b^2\geq0$.} When these conditions are written in terms of the amplitudes $\varepsilon$'s they look algebraically complicated and not very enlightening.

To simplify the discussion we can use the previous bound $\varepsilon_{ABAD}^2 \leq \varepsilon_{ABAB} \varepsilon_{ADAD}$ and $ \varepsilon_{CBCD}^2 \leq \varepsilon_{CBCB} \varepsilon_{CDCD}$ to complete the square in the first line of equation \eqref{gencond}. Then we can derive a non-optimal bound on the most generic $\varepsilon_{ABCD}$ as
\begin{equation}
\varepsilon_{ABCD} ^2 \leq  4( \sqrt{ \varepsilon_{ADAD}\varepsilon_{CBCB}} +\sqrt{ \varepsilon_{ABAB}\varepsilon_{CDCD}})^2.
\end{equation}
Even though this bound is not optimal it shows in a more transparent way how the prefactor of the off-diagonal OTOC is bounded by the diagonal ones. This is the main conceptual point, in a holographic setting this shows hows a generic interaction is bounded by the gravitational interactions.

As a final comment,we can also consider a correlator of the type
\begin{equation}
F(t)= {\rm Tr} [ y V y W(t) y V y W(t) ]  ,~~~ V=\alpha_1 A + \alpha_2 C,~~{\rm and}~~W=\alpha_3 B + \alpha_4 D
\end{equation}
One might wonder whether the $\varepsilon>0$ constraint for $F(t)$ by varying all $\alpha$'s independently we can derive a bound on $\varepsilon_{ABCD}$ stronger than the one above which was derived by steps. It is easy to see that this is not the case, and considering the most general linear combination does not generate new constraints compared to equation \eqref{gencond}.

\subsection{Non-Hermitian Operators}\label{sec:nonh}
So far we have discussed OTOC between arbitrary hermitian operators. Some of the steps for the bound on chaos argument from \cite{MSS} do not directly work for non-hermitian operators. We will show here that the bound on hermitian operators is enough to prove this generalization.

Consider the following OTOC between general non-hermitian operators
\begin{equation}\label{eq:vdwdvw}
F(t)= {\rm Tr} [ y V^\dagger y W^\dagger(t) y V y W(t) ]
\end{equation}
We will show how the bound on ${\rm Re}~F(t)$ derives from the bound on hermitian operators. This quantity is related to the double commutator between non-hermitian $V$ and $W$ that appear in the definition of chaos.

We will expand a general non-hermitian operator $\mathcal{O}$ in two hermitian components $\mathcal{O}_R = (\mathcal{O}+\mathcal{O}^\dagger)/2$ and $\mathcal{O}_I = (\mathcal{O}-\mathcal{O}^\dagger)/(2i)$, for $\mathcal{O}=V,W$. To simplify the expressions we write below, we will use a shorthand for the OTOC defining $\langle ABCD\rangle \equiv {\rm Tr} [ y A(0) y B(t) y C(0) y D(t)]$. Starting from the right hand side of \eqref{eq:vdwdvw}, expanding and using the cyclic property of the trace it is easy to show that
\begin{eqnarray}\label{eq:nonhermtotl}
{\rm Re} ~F(t)&=&{\rm Re}~{\rm Tr} [ y (V_R-i V_I) y W^\dagger(t) y (V_R+i V_I) y W(t) ]  \nn
&=&{\rm Re}~[ \langle V_R W^\dagger V_R W\rangle + \langle V_I W^\dagger V_I W\rangle- i (\langle V_I W^\dagger V_R W\rangle-\langle V_R W^\dagger V_I W\rangle)]\nn
&=&\langle V_R W^\dagger V_R W\rangle + \langle V_I W^\dagger V_I W\rangle
\end{eqnarray}
where we used that $ \langle V_I W^\dagger V_R W\rangle^* = \langle W^\dagger V_R W V_I \rangle = \langle V_I W^\dagger V_R W\rangle$ and similarly for $\langle V_R W^\dagger V_I W\rangle$, implying they are both real, and therefore the last term in the right hand side of the equation above is purely imaginary.

Now we can expand $W$ and use that $\langle A B C B\rangle$ is real for hermitian operators, to show
\begin{equation}\label{eqnonh}
{\rm Re} ~F(t) = \langle V_R W_R V_R W_R \rangle + \langle V_R W_I V_R W_I\rangle +\langle V_I W_R V_I W_R \rangle + \langle V_I W_I V_I W_I\rangle,
\end{equation}
Then if we write ${\rm Re} ~F(t) = F_d - \varepsilon ~e^{\lambda t}$, the chaos bound on the growth rate automatically applies to each term individually on the right hand side implying $\lambda \leq 2\pi/\beta$. Moreover since all terms appear with a plus sign, the bound on the sign of the prefactor still applies, implying $\varepsilon\geq0$.

Taking the usual chaos bound for non-hermitian operators as a starting point we can derive analogous results as in the previous section for general non-hermitian operators.

\section{An Example: 2d CFTs}\label{sec:2dCFT}

In the context of 2d CFTs one can show that at large $c$, a large gap in the twist is enough to obtain maximal chaos by using results from semiclassical limits of Virasoro conformal blocks \cite{Roberts:2014ifa} \footnote{Other studies of chaos in 2d CFT from different perspectives can be found in \cite{Jackson:2014nla, Turiaci:2016cvo} (see also \cite{Perlmutter:2016pkf}). }. This is given purely by a product of stress tensors acting on the identity and therefore can be interpreted as coming in the bulk from a purely gravitational interaction.

In this section we want to study in a simple setup which role the inelastic version of the chaos bound plays for large $c$ 2d CFTs with a sparse spectrum. Under some assumptions, we will study the general behavior of off-diagonal OTOC. We propose that a resummation of intermediate channels can be written as a single non-vacuum block corresponding to an operator with an effective dimension and an effective spin.

From a bulk perspective this constrains matter interaction (OPE coefficients of arbitrary operators) using the chaos bound.

\subsection{Kinematics}
In any 2d CFT an arbitrary four point function can be written as
\begin{equation}\label{eq:4pftnction}
\langle W_1(z_1,\bar{z}_1)W_2 (z_2,\bar{z}_2) V_3 (z_3, \bar{z}_3) V_4 (z_4,\bar{z}_4) \rangle = \frac{1}{z_{12}^{h_1+h_2} z_{34}^{h_3+h_4} }\frac{1}{\bar{z}_{12}^{h_1+h_2} \bar{z}_{34}^{h_3+h_4} } G(z,\bar{z})
\end{equation}
where $G(z,\bar{z})$ can be expanded in Virasoro conformal blocks and the cross-ratio is defined as $z=\frac{z_{12}z_{34}}{z_{13}z_{24}}$ and a similar anti-holomorphic version. The operators are arbitrary but we use the letters $V$ and $W$ to indicate which ones will be at time $0$ ($V$'s) and which at time $t$ ($W$'s). Schematically we expand the four-point function as
\begin{equation}\label{eq:OPEdef}
G(z,\bar{z}) = \sum_{p} C_{12p}C_{34p}~ \mathcal{F} \big[{}^{h_1}_{h_2}{}^{h_3}_{h_4}\big](h_p, z)~ \mathcal{F}\big[{}^{\bar{h}_1}_{\bar{h}_2}{}^{\bar{h}_3}_{\bar{h}_4}\big](\bar{h}_p,\bar{z})
\end{equation}
where $\mathcal{F}_{h_p} \big[{}^{h_1}_{h_2}{}^{h_3}_{h_4}\big](z)$ are the Virasoro blocks corresponding to a Virasoro primary operator $\mathcal{O}_p$ with (anti)holomorphic weights $h_p$($\bar{h}_p$), dimension $\Delta = h_p + \bar{h}_p$ and spin $s=|h_p-\bar{h}_p|$. The blocks are normalized such that $\mathcal{F}_h(z) = z^h(1+\ldots)$ for a small $z$ expansion.

In going from Euclidean to Lorenzian signature, different orders of operators are encoded in the monodromy of the blocks \cite{Rehren:1987ur}. Following \cite{Roberts:2014ifa} we choose the kinematics of the correlator to be
\begin{eqnarray}
&&z_1 = e^{\frac{2\pi}{\beta} ( t+i\epsilon_1)},~~~~\hspace{0.08cm}z_2 = e^{\frac{2\pi}{\beta} ( t+i\epsilon_2)},~~~~\hspace{0.08cm}z_3 = e^{\frac{2\pi}{\beta} ( x+i\epsilon_3)},~~~z_4 = e^{\frac{2\pi}{\beta} ( x+i\epsilon_4)},\nn
&&\bar{z}_1 = e^{-\frac{2\pi}{\beta} ( t+i\epsilon_1)},~~~\bar{z}_2 = e^{-\frac{2\pi}{\beta} ( t+i\epsilon_2)},~~~\bar{z}_3 = e^{\frac{2\pi}{\beta} ( x-i\epsilon_3)},~~~\bar{z}_4 = e^{\frac{2\pi}{\beta} ( x-i\epsilon_4)}
\end{eqnarray}
where $\epsilon_1<\epsilon_3<\epsilon_2<\epsilon_4$ and as we raise the time from $0$ to $t$ the cross-ratio $z$ goes once around $z=1$ while $\bar{z}$ remains in the first sheet. For times larger than the dissipation time, which is of order $\beta$, we can evaluate the blocks on the second sheet with cross-ratios $z \approx - \epsilon_{12}^\star \epsilon_{34} e^{\frac{2\pi}{\beta}(x-t)}$ and $\bar{z} \approx - \epsilon_{12}^\star \epsilon_{34} e^{-\frac{2\pi}{\beta}(x+t)}$, where $\epsilon_{ij} = i (e^{\frac{2\pi}{\beta} i \epsilon_i} - e^{\frac{2\pi}{\beta} i \epsilon_j})$. For a configuration with operators equally spaced on the thermal circle, $\epsilon_{12}^\star \epsilon_{34}=4i$ and $z = -4 i  e^{\frac{2\pi}{\beta}(x-t)}$ and $\bar{z} = - 4i e^{-\frac{2\pi}{\beta}(x+t)}$.

If we normalize operators by their 2pt function on the plane $\langle O (x) O(0) \rangle =(x)^{-2\Delta_O}$, then each term in the factorized answer for the four-point function gives a factor of ${\rm Tr} [ y^2 O y^2 O] = (\pi/\beta )^{2\Delta_O}$. This also appears from the position dependent prefactor in the right hand side of equation \eqref{eq:4pftnction} when the operators are all different.

\subsection{Elastic case}

In this case we take the two operators at $t=0$ and $t$ to be the same $V_3 = V_4 = V$ and $W_1 = W_2 = W$. Then the identity appears on the intermediate channel in the OPE written above in equation \eqref{eq:OPEdef}. Assuming a large twist gap we can approximate the full correlator by the vacuum block. Following \cite{Roberts:2014ifa} we will take a large $c$ limit with $h_1/c\ll1$ fixed but small and $h_3\gg 1$. Then we can make use of heavy-light semiclassical blocks found in \cite{Fitzpatrick:2014vua}. The final answer is given by
\begin{eqnarray}\label{eq:vacuumblockapprox}
\frac{ {\rm Tr} [ y V y W(t) y V y W(t)]}{ {\rm Tr}[y^2 V y^2 V]{\rm Tr}[y^2 W y^2 W]} &\approx& \big( 1+\frac{6 \pi  h_1}{c} e^{\frac{2 \pi}{\beta}(t-x)}  \big)^{-2h_3}\nn
&\approx&1 - \frac{12 \pi  h_1  h_3}{c}  e^{\frac{2 \pi}{\beta}(t-x)} + \ldots
\end{eqnarray}
which saturates the chaos bound. In the first line we wrote the chaos limit of the identity Virasoro block, while in the second line we focus on times $\beta^{-1} \ll t \ll t_{\rm sc}=\frac{\beta}{2\pi} \log c$. This was formally done at infinite gap. Corrections from finite gap and how the correlator Reggeize was recently studied in reference  \cite{Chang:2018nzm}. Within this approximation, for times larger than the scrambling time $t\gg t_{\rm sc} $ this OTOC goes to zero exponentially at a rate related by the quasinormal modes.

\subsection{Inelastic case}
In the inelastic case we can take four arbitrary operators $W_1$, $W_2$, $V_3$ and $V_4$. In order to have analytic control over the Virasoro conformal blocks we will take large $c$ with $1 \ll h_3,h_4$ and fixed $h_1/c$, $h_2/c$ but small. Moreover we also take $h_{12}=(h_1-h_2)/2$ and $h_{34}=(h_3 -h_4)/2$ to be of order one such that the results of  \cite{Fitzpatrick:2014vua} apply.

We can take a basis of operators such that the identity block does not appear on the intermediate channel (this is automatic if the dimensions are different). Instead within a similar approximation as in the elastic case, we need to consider light intermediate operators of low twist. The semiclassical Virasoro block was also computed in this case, when the channel dimension $h_p$ is of order one \cite{Fitzpatrick:2014vua}. After going to the second sheet and using the chaos kinematics we get
\begin{equation}\label{eq:nonvacblock}
\mathcal{F} \big[{}^{h_1}_{h_2}{}^{h_3}_{h_4}\big](h, z) = \left(\frac{1}{ 1-\frac{12 \pi i  (h_1+h_2)}{cz}  }\right)^{h_3+h_4-h} z^h {}_2 F_1 (h - h_{12}, h+ h_{34}, 2h, z)|_{{\rm 2nd~sheet}} ,
\end{equation}
where the hypergeometric function comes from $SL(2)$ descendants and it is evaluated on the second sheet.

For times between dissipation and scrambling, the first term in the right hand side of \eqref{eq:nonvacblock} is constant and the exponential growth comes from the hypergeometric function
\begin{equation}
{\rm Tr} [ y V_3 y W_1(t) y V_4 y W_2(t)] = N_\beta \sum_{p} C_{12p}C_{34p}~d_p~e^{ \frac{2\pi}{\beta}(s_p-1)t}e^{- \frac{2\pi}{\beta}(\Delta_p-1) x},
\end{equation}
where $\Delta_p$, $s_p$ is the dimension and spin of the intermediate channel operator $O_p$. The normalization coming from the prefactor of \eqref{eq:4pftnction} is $N_\beta = (\pi/\beta)^{\Delta_1 +\Delta_2 + \Delta_3 + \Delta_4}$. $d_p$ is a coefficient
\begin{equation}
d_p = \frac{8\pi }{(i4)^{h_p-\bar{h}_p }(2h-1)}\frac{\Gamma(2h_p)}{\Gamma(h_p\pm h_{12})}\frac{\Gamma(2h_p)}{\Gamma(h_p\pm h_{34})},
\end{equation}
where $\Gamma(a\pm b) =\Gamma(a+b)\Gamma(a-b)$. This factor depends on the dimension of the intermediate channel and comes from the evaluation of the hypergeometric function in \eqref{eq:nonvacblock} on the second sheet.

The growing part of elastic OTOC for holographic CFTs are dominated by the vacuum block, dual to gravitational interactions in the bulk. The growing part of inelastic OTOCs is not related to gravitational interactions. The fact that Virasoro descendants are irrelevant for the calculation of inelastic OTOCs is a manifestation of this fact. This is not completely obvious and we give the details in the appendix. The amplitude of the growing piece of the inelastic OTOC is small due to the fact that the OPE coefficients are subleading in $N$ and in the gap.

Lets first assume that the spins are bounded. Then by looking at this expression assuming a large twist gap we can conclude that in the low-lying part of the spectrum all particles must have spin $s<2$. If a primary happens to have $s=2$ then its interactions with other particles cannot be stronger than gravity (bounds OPE coefficient).

If there happens to be a light particle with $s>2$, then its contribution should be Reggeize among the low-lying spectrum to give an effective spin $s_{\rm eff} <2$. Then we can see the statement in the previous paragraph as a statement about inelastic Pomerons in the theory.
\begin{figure}[t!]
\begin{center}
\begin{tikzpicture}[scale=1]
\node[inner sep=0pt] (russell) at (2.5,1.5)
    {\includegraphics[width=.25\textwidth]{curve2.pdf}};
    \node[inner sep=0pt] (russell) at (2.5,1.5)
    {\includegraphics[width=.25\textwidth]{curve.pdf}};
\draw[thick, ->] (0,0) -- (5,0);
\draw[thick, ->] (0,0) -- (0,3);
\draw[thick] (3.5,0-0.1)--(3.5,0+0.1);
\draw (3.5,-0.5) node {$t_{\rm sc}$};
\draw (-1,2) node {$F(t)$};
    \end{tikzpicture}
    \end{center}
    \vspace{-0.85cm}
    \caption{\small  Sketch of a typical behavior of inelastic OTOC $F(t)$ in 2d CFT (black curve) as a function of time, assuming its approximated by a non-vacuum block with effective dimension $\Delta_{\rm eff}$ and spin $s_{\rm eff}$. Initially the OTOC grows exponentially with rate $\lambda = \frac{2\pi}{\beta} (s_{\rm eff}-1)$. For late times the fast decay is controlled by the quasi-normal modes. In blue we show a typical elastic OTOC. }
    \label{fig:temp-check}
    \end{figure}

The calculation required in the previous paragraph to evaluate the inelastic OTOC is complicated, even in the case of elastic OTOC \cite{Chang:2018nzm}. We can conjecture that the result of summing over infinite spins is equivalent to a single non-vacuum block with effective $h_{\rm eff}$, $\bar{h}_{\rm eff}$ and effective dimension $\Delta_{\rm eff}=h_{\rm eff} + \bar{h}_{\rm eff}$ and spin $s_{\rm eff}=|h_{\rm eff} - \bar{h}_{\rm eff}|$. With this assumption the inelastic OTOC is given by
\begin{equation}
N^{-1}_\beta~{\rm Tr} [ y V_3 y W_1(t) y V_4 y W_2(t)] = ~C_{12}C_{34}~\mathcal{F} \big[{}^{h_1}_{h_2}{}^{h_3}_{h_4}\big](h_{\rm eff}, z) ~\bar{z}^{\bar{h}_{\rm eff}},
\end{equation}
where the holomorphic block is given by equation \eqref{eq:nonvacblock}, and $C_{12}$ ($C_{34}$) are effective couplings between operators $W_1$, $W_2$ ($V_3$, $V_4$) and the effective Pomeron state $h_{\rm eff}, \bar{h}_{\rm eff}$. Depending on the effective spin, $C_{12}C_{34}$ might be bounded by the dimensions of the external operators, following \eqref{ec:introbound}.

With the proposal of the previous paragraph, we can analyze times longer than scrambling $t_{\rm sc} \lesssim t$. In this case the situation changes and the part of the block coming from the Virasoro descendants dominate. Namely, the first factor in the right hand side of \eqref{eq:nonvacblock} decays exponentially. Assuming the behavior of the correlator is equivalent to a single block of dimension $\Delta_{\rm eff}$ of order one and spin $s_{\rm eff}<2$, the OTOC decays exponentially as $\sim e^{- \frac{2\pi}{\beta} (h_3+h_4)t} $. Under the assumption of the previous paragraph, we show in figure \ref{fig:temp-check} the behavior of a typical OTOC between different operators.

To summarize, inelastic OTOCs grow exponentially in a way controlled by a Pomeron exchange (unrelated to gravity), until the scrambling time $t_{\rm sc}$ at which the correlator begins to decay according to the quasinormal modes frequencies. This is expected since this decay is due to the bulk-boundary propagators appearing in the bulk calculation of the OTOC. The picture and the proposal that emerges from this analysis should be worked out in more detail following \cite{Chang:2018nzm} and \cite{Collier:2018exn} (see also \cite{Liu:2018iki} and \cite{Hampapura:2018otw}), but we leave it for future work.

\section{Open Questions}\label{Sec:conc}

To conclude we would like to state some open questions.

It would be nice to compute the off-diagonals OTOC introduced here for SYK models \footnote{In particular, the approach of \cite{Berkooz:2018jqr} might be very useful for this.}. Following the notation of \cite{Kitaev:2017awl} and \cite{Gu:2018jsv}, we can write a generic OTOC as a convolution between form factors describing the coupling of external operators to an effective `scramblon' mode, and the scramblon propagator which grows exponentially with time. In this perspective the bound stated here constraints the behavior of the general form factors appearing in these models. Their rate of growth in time is bounded by the growth of the scramblon mode through the elastic OTOC.

Moreover, the mode responsible to the Lyapunov behavior of inelastic OTOC might not be the same as the one for elastic case (for example, it might not have the quantum numbers of the vacuum). Therefore there must be a mode which similarly to the Schwarzian mode generates exponential growth\footnote{From the perspective of \cite{MTV} the problem is analogous to finding a generalization of Liouville theory that allows primary operators with spin.}. A simple proposal would be a similar mode living on ${\rm Diff}(S^1)/U(1)$ instead of ${\rm Diff}(S^1)/SL(2)$, but this requires further study.

This question can be extended to higher dimensions. In 2d CFTs the maximal chaos behavior coming from the identity block can be understood as coming from a Goldstone mode of broken reparametrization invariance \cite{Turiaci:2016cvo}. It would be nice to find a description of a similar soft mode producing exponential growth of off-diagonal OTOC, related to non-vacuum representations. To analyze this problem 2d versions of SYK might be useful as explicit examples \cite{Gu:2016oyy, Turiaci:2017zwd, Murugan:2017eto, Berkooz:2017efq}. In the particular case of 2d CFTs it would be interesting to repeat the analysis of \cite{Chang:2018nzm} for a general OTOC.

The analysis of this paper can be extended to higher order OTOC with more than four operators. The diagonal version of these correlators was studied in \cite{Haehl:2017pak} (see also \cite{Haehl:2018izb}). Results in this direction were derived in \cite{Basu:2018akv}.

Finally, after understanding how correlators Reggeize in the chaos limit, it would be nice to recast the bound derived in this paper as a bound on OPE data. This might also help sharpen the statement about gravity being the highest spin, strongest, interaction.

\bigskip

\begin{center}
{\bf Acknowledgements}
\end{center}

\vspace{-2mm}

We want to thank G. Horowitz, J. Maldacena, M. Rangamani and D. Stanford for discussions and comments on the draft. This work was supported by a Fundamental Physics Fellowship. We also benefited from the workshop ``Chaos
and Order: From strongly correlated systems to black holes" at KITP, supported in part by the National Science Foundation under Grant No. NSF PHY-1748958 to the KITP.

\begin{appendix}
\section{Semiclassical Virasoro Blocks}\label{app:blocks}
\end{appendix}
In this paper we used a Virasoro conformal block with two external light operators of weights $h_L\pm \delta_L$ and two external heavy operators of weights $h_H \pm \delta_H$. In the large $c$ limit with $h_H/c$ fixed these were obtained in \cite{Fitzpatrick:2014vua} for a light intermediate channel $h$. Within this approximation it is given by
\begin{equation}\label{app:fitzkapblock}
\mathcal{F} \big[{}^{H}_{H}{}^{L}_{L}\big](h, z)= (1-w)^{(h_L+\delta_L)(1-\alpha^{-1})}\left( \frac{w}{\alpha ~z} \right)^{h-2h_L} z^h {}_2 F_1 \left( h - \frac{\delta_H}{\alpha} , h+\delta_L , 2 h , w \right)
\end{equation}
where $\alpha=\sqrt{1-24 h_H/c}$ and $w(z)=1-(1-z)^\alpha$. In the case of pairwise identical operators, $\delta_H=\delta_L=0$, and for the vacuum channel, this expression gives
\begin{equation}
\mathcal{F} \big[{}^{H}_{H}{}^{L}_{L}\big](0, z) =  (1-w)^{(h_L+\delta_L)(1-\alpha^{-1})}\left( \frac{w}{\alpha~z} \right)^{-2h_L}.
\end{equation}
In the limit for which $h_H/c \ll 1$ ($\alpha\approx 1-12h_H/c$), for small $z$ in the second sheet the block is approximately $ \mathcal{F} \big[{}^{H}_{H}{}^{L}_{L}\big](0, z) \approx \left(\frac{z}{1-(1-z)^\alpha} \right)^{2h_L}$. This expression reproduces equation \eqref{eq:vacuumblockapprox}, which is the main result in \cite{Roberts:2014ifa}.

For the case of generic intermediate channel we can use the general formula, go to the second sheet and evaluate in the chaos limit. This gives the same answer as doing the analytic continuation of the expression
\begin{equation}\label{app:eq3}
\mathcal{F} \big[{}^{H}_{H}{}^{L}_{L}\big](h, z)= \left( \frac{z}{1-(1-z)^\alpha} \right)^{2h_L-h} z^h {}_2 F_1 \left( h - \frac{\delta_H}{\alpha} , h+\delta_L , 2 h , w \right)
\end{equation}
The first term is equivalent to the vacuum block with shifted dimensions. This term is entirely due to Virasoro descendants. The evaluation of this term in the second sheet is therefore
\begin{equation}\label{eq:prefactorapp}
\left( \frac{z}{1-(1-z)^\alpha} \right)^{2h_L-h} \Bigg|_{2{\rm nd~sheet}} \approx \left( \frac{1}{1-\frac{24 \pi i h_H}{cz} } \right)^{2h_L -h}
\end{equation}
In the chaos limit, and for times smaller than the scrambling time $c z \gg 1$ ($t\ll \frac{\beta}{2\pi} \log c$), this gives approximately a constant $1$. After the scrambling time $c z \ll 1$, this term decays and controls the decay of correlators. The second term of \eqref{app:eq3} evaluated on the second sheet and for small $z$ gives
\begin{equation}
z^h{}_2 F_1 (h - \frac{h_{H}}{\alpha}, h+ h_{L}, 2h, w)|_{{\rm 2nd~sheet}}=  \frac{2\pi i ~e^{i \pi (h_{L}-h_{H})}}{2h-1}\frac{\Gamma(2h)}{\Gamma(h\pm \frac{h_{H}}{\alpha})}\frac{\Gamma(2h)}{\Gamma(h\pm h_{L})} \frac{z}{(\alpha z)^h}
\end{equation}
For $h_H/c$ with $\alpha \sim 1$ this gives the same as the evaluation of the global block $z^h{}_2 F_1 (h - h_{H}, h+ h_{L}, 2h, z)$. Therefore we can approximate the Virasoro block by
\begin{equation}
\mathcal{F} \big[{}^{H}_{H}{}^{L}_{L}\big](h, z)\approx \left( \frac{z}{1-(1-z)^{1-12 h_H/c}} \right)^{2h_L-h} z^h {}_2 F_1 \left( h - \delta_H , h+\delta_L , 2 h , z \right).
\end{equation}
The evaluation of this expression above in the second sheet for small $z$ gives the same answer as applied on the original \eqref{app:fitzkapblock}. Therefore all the effects of Virasoro descendants in the chaos limit come from the prefactor in the equation above.

\begingroup\raggedright


\section{Discussion}
\label{sec:conclusion-future}

In this work we have introduced affinity-VAE, a neural network capable of organising the latent representation based on the similarity of the object. While a $\beta$-VAE is capable of cluster separation and latent disentanglement, we have little control over the learned factorised representations and captured semantics. We have shown that with guidance from an automatically generated affinity matrix we can create more homogeneous, rotationally-invariant clusters that could improve the classification accuracy. Furthermore, the affinity metric can be tailored to the data or domain of interest, improving the generality of the method. Since the affinity metric is only utilised during the training phase (Equation \ref{eq-shape-vae-loss}), we can infer the similarity of unseen objects in the encoded latent representation. A pre-trained network can therefore be easily applied to discovery of new classes. We have demonstrated that the learned latent spaces can be continuous, which enables the potential use of the method with unseen data (e.g. discovery of new species).

We have demonstrated the potential of the method in a scientific application using the example of subtomogram target identification in volumetric cryo-ET data. While the results are promising and show the potential of the method for discovery of new species in experimental data, more experiments are required to test the effectiveness of the method on a full tomogram. This comes with new challenges, notably the presence of noise, crowding, interactions, multiple conformations and translation.

\section{Methods}
\label{sec:Methods}

\subsection{$\beta$-VAE}
\label{sec:method_VAE}

The Variational Autoencoder (VAE) is a class of generative neural network aiming to learn the joint distribution of inputs $x$ and their low-dimensional latent generative factors $z$ \cite{Kingma2014}. Through maximising the probability of reconstruction, i.e. minimising the difference between the input $x$ and the decoded output $y$ (reconstruction term), we effectively organise the latent space. Concurrently, the VAE regularises the latent space by minimising the difference between real and estimated distributions (variational term)
encouraging the approximate posterior to reflect the prior. To aid the optimisation, these are both set to Gaussian and the prior is set to normal allowing for the ``reparametrisation trick'' \cite{Higgins2017}.

$\beta$-VAE is an iteration of the VAE framework introducing a hyperparameter $\beta$ which modulates the learning constraints applied to the model \cite{Higgins2017}. Values of $\beta>1$ put effective constraints on the capacity of the latent $z$ bottleneck encouraging factorised representations \cite{burgess2018}. The $\beta$-VAE approach is therefore most commonly used for unsupervised factorised representation learning.

The goal of the training is to minimise the objective function $L$,

\begin{equation}
    \underset{x}{\mathrm{argmin}}\ L =  ||x - y||^2 + \beta \times D_{KL}\left[ \mathcal{N}(\mu_{z}, \sigma_{z}),  \mathcal{N}(0, 1)\right]
\end{equation}
Where $x$ and $y = d(e(x))$ denote the input data and the output reconstruction respectively, $z = e(x)$ is the encoded latent representation and $e$ and $d$ are the encoder and decoder parts of the neural network, respectively. $D_{KL}$ refers to the Kullback-Leibler divergence between the prior $ \mathcal{N}(\mu, \sigma)$ and  posterior $ \mathcal{N}(0, 1)$ distributions. The first term of the equation is the reconstruction loss minimising the difference between the inputs and decoded outputs. The second term, parameterised by $\beta$, is the variational term regularising the latent space.

\subsection{affinity-VAE}

In this section we introduce the affinity-VAE neural network,  which is characterised by addition of an affinity-based regularisation to the existing formulation of the $\beta$-VAE. To facilitate this, we also introduce some architectural changes to the network, notably an additional fully-connected layer to denote the pose of the object.

\subsubsection{Affinity-based loss component}
In addition to the reconstruction and KL divergence terms of the standard VAE loss function, we introduced a new shape regularisation term $S(z)$. The hyperparameter $\gamma$ provides fine control of the influence of this regularisation term (in a similar manner to $\beta$):

\begin{equation}
    L = ||x - y||^2 + \beta \times D_{KL}\left[N(\mu_{z}, \sigma_{z}), N(0, 1)\right] + \gamma \times S(z)
    \label{eq-shape-vae-loss}
\end{equation}

where $S(z)$ is the L1 norm of the difference between a pre-calculated affinity matrix $\mathbf{A}$ and the cosine similarity of the latent representations:

\begin{equation}
    S(z) = \frac{
        \sum_{i, j}^{N}\
        \left|\left|
            \mathbf{A}_{i j}
            -
            \frac{
                z_i \cdot z_j
            }{
                ||z_i|| \cdot ||z_j||
            }
        \right|\right|
    }{
        N
    }
    \label{eq-shape}
\end{equation}
with $z$ denoting the latent variables, $i, j$ the indices of the vectors in the batch and $N$  the batch size.

\begin{wrapfigure}[13]{r}{0.6\textwidth}
    \begin{center}
    \includegraphics[width=0.6\textwidth]{figures/3-B-shape-VAE-noin.png}
    \caption{Architecture of the affinity-VAE.}
    \label{fig:shape-VAE}
    \end{center}
\end{wrapfigure}

 The cosine similarity measures the distance between two latent points, whereas the pre-computed affinity matrix ($\mathbf{A}$) provides feedback on their actual pairwise similarity. This effectively organises the latent space so that similar objects (regardless of their pose), as described by the similarity descriptor in the affinity matrix, are placed close together in the latent space. The pre-calculated affinity matrix is generated automatically by computing pairwise similarity scores for the entire training dataset with a target function. In our case this is SOAP for the 2D alphanumeric data and FSC for 3D protein data  (more on data generation in section \ref{sec:Methods-data}), but different metrics could be chosen to facilitate different datasets, or to organise the latent space by different factors (see section \ref{sec:similarity-metrics}). Furthermore, the affinity matrix is only used in the training stage, therefore a pre-trained network can be easily applied to discovery of new classes/species.

\subsubsection{Pose channel}
The latent space is represented in terms of distributions expressed through learnable fully connected layers for the mean ($\mu$) and the variance ($\sigma$). We have introduced a third learnable fully connected layer to represent the $n$-D pose of the object, in a similar manner to rotation in an rVAE~\cite{Ziatdinov2021, ziatdinov_kalinin_2021}. By feeding the same shape similarity values (matrix identity) for the same object regardless of its orientation, we force the latent space and the additional pose parameter to learn to represent the pose of the object.

\subsection{Latent map}

To ensure that our method is generalising to new data/tomograms so that the network does not need retraining every time it analyses new data, we introduce the "latent map" approach. Instead of  training the network on the whole tomogram, we create a latent "map" pre-seeded with existing proteins. Once pre-trained with a set of objects that is non-redundant (i.e. no duplicates) but exhaustive (i.e. considering all morphologies), the latent space can be used as a dictionary or a reference space for both seen and unseen data. The seen data would form clusters within or near the existing cluster centres in the latent map while the unseen data would do so  near or between groups of similar morphology. This brings in a promise of applying affinity-VAE as an unsupervised template-and-label-free approach to the classification and detection of unseen samples or novel structures in cryo-ET tomograms.

\subsection{Latent embeddings, clustering and KNN}

The encoded samples are represented in the latent space through their generative factors, however the dimensionality of that latent space ($z$) is a hyperparameter. We are therefore analysing $n$-D latent spaces such that $n$ is high enough to capture the complexity in the original image, but low enough to only capture the true semantics. Therefore, even with the simplest 3D datasets $n > 2$. To visualise and interpret any relationships between the latent codes corresponding to different semantics, and visually assess the degree to which identical or similar objects form homogeneous clusters in the latent space, we produce latent embeddings in which each latent vector is given a location in a lower-dimensional (2D) manifold. We have explored the following methods for visualisation of latent embeddings:
\begin{itemize}
    \item UMAP \cite{umap} (Uniform Manifold Approximation and Projection) -- used for our 2D alphanumeric data;
    \item t-SNE \cite{tsne} (t-Distributed Stochastic Neighbour Embedding) -- used for our 3D protein and tetramino data.
\end{itemize}

The accuracy of the method was evaluated in three different ways. To assess whether the object was classified correctly we explored the following approaches:
\begin{itemize}
    \item means for the clusters are calculated and each point is classified to belong in a cluster if it lies within +/- $2\sigma$ of the mean - used for 2D alphanumeric data;
    \item $k = 5$ nearest neighbors (KNN) \cite{knn} were selected for each encoded data point and a class was assigned based on majority voting (producing hard assignments) -- used for 3D protein and tetramino data;
    \item $k=100$ nearest neighbors \cite{knn} were selected and assignment fractions for each class were used as likelihood of belonging to that class (producing soft probabilistic assignments) --  used for 3D protein and tetramino data.
\end{itemize}
For quick evaluation of the 3D data we used hard assignments because they are faster, however soft assignments provide more insight into the classification, which is particularly useful for evaluating unseen data or novel structures.

\subsection{Data simulation}
\label{sec:Methods-data}
Three datasets were used in the evaluation of the methods: 1) 2D alphanumeric data, 2) 3D tetramino data, and 3) simulated 3D protein data.

\begin{figure}[h]
    \centering
    \includegraphics[width=1\textwidth]{figures/3-E-sample_data.pdf}
    \caption{Sample data, left:  alphanumeric, middle : tetramino blocks and right:  protein volumes from the list in \cite{ecoli}.}
    \label{fig:sample_data}
\end{figure}

Images in the alphanumeric dataset are constructed from letters and digits ($x \in \{ \texttt{a, e, b, d, p, i, j, z, 2, k, x, u}\}$), rotated at various angles (Figure \ref{fig:sample_data} left). Images are created using the Pillow library in Python \cite{clark2015pillow}, rotated to a defined angle $\theta$ where $\left\{\theta \in \mathbb{Z}|-45<\theta<45 \right\}$, and converted to a binary two-dimensional array. We use an $80/20$ split for training and testing.

The tetramino data (Figure \ref{fig:sample_data} middle) was designed to emulate an easier, but similar, 3D scenario to real cryo-ET data. Tetraminos were made of combinations of four identical cubes connected wall-to-wall. They were generated on-the-fly with controllable parameters such as rotation and translation. Other controllable aspects include morphological adjustments (e.g. elongation) to test the semantic disentangling power of the method. This allows us to generate any desired shape combinations to fully test the performance of our method. The training data was constructed from 6 morphologically different tetraminos which were used to pre-seed the latent map, while for evaluation we constructed a new, previously unseen class that was morphologically similar to two other classes from the training set. All data was augmented using rotations randomly sampled at $\theta = \{10, 20, ..., 360\}$ in 3 different planes. All images were resized prior to training to 32x32x32 voxels. Test and validation data constituted 10\% and 20\% of the training set respectively.

The protein data (Figure \ref{fig:sample_data} right) was generated from the list of 50 most abundant \emph{E. coli} proteins \cite{ecoli}. Chimera \cite{chimera} was used to generate a synthetic 3D density map from each protein on the list. The maps were generated at 10 Å resolution without taking atomic $B$-factors into account. The training data was constructed from $n$ randomly selected classes (protein types), which were used to pre-seed the latent map, and a different, previously unseen class was selected for evaluation. All data was augmented with rotations randomly sampled at $\theta = \{10, 20, ..., 360\}$ in 3 different planes. All images were resized prior to training to 64x64x64 voxels. Test and validation data constituted 10\% and 20\% of the whole dataset respectively.

\subsection{Affinity metrics} \label{sec:similarity-metrics}

Regularisation of the latent space ensures that semantically similar objects are encoded near to each other (i.e. at similar positions in latent space) while dissimilar objects are encoded far apart. To achieve this regularisation we require knowledge of the similarity between all pairs of training examples. The affinity between two structures can be described in accordance with the intrinsic properties of the data. The choice of affinity descriptor (shape, or indeed any other metric) should therefore be made with respect to the property of the data intended to achieve the desired data separation.

\subsubsection{2D alpha-numeric data}

In order to investigate the similarity between two structures in the alphanumeric data, we employ the Smooth Overlap of Atomic Positions (SOAP) descriptor \cite{Bartok2010,Bartok2013}. This shape descriptor uses a combination of radial and spherical harmonics. In our model, we are treating every pixel that is not background as an “atom”. The SOAP descriptor places a Gaussian density distribution at the location of each selected pixel. The SOAP kernel is then defined as the overlap of the two local nearest neighbouring densities integrated over all three-dimensional rotations.

\subsubsection{3D tetramino and protein data}

Fourier shell correlation (FSC) \cite{fsc} curves are the standard metric for resolution estimation of cryo-EM maps. The method calculates the similarity of two images as a function of spatial frequency,  by calculating the correlation between the Fourier coefficients of each image in thin spherical shells:

\begin{equation}
    FSC(k) = \frac{
        \sum{F_1(k)  F_2(k)^*}
    }{
        \sqrt{
            \sum{|F_1(k)|^2}\sum{|F_2(k)|^2}
        }
    }
    \label{eq-fsc}
\end{equation}

where $F_1(k)$ and $F_2(k)$ are the (complex) coefficients of the Fourier transforms of the two structures in a spherical shell at radius $k$. In this work, to obtain a single value for use as an affinity metric, we take an average of the FSC across all spatial frequency shells, weighted by the number of Fourier coefficients in each shell according to the method described by Brown \textit{et al.}~\cite{avgfsc}. This gives a measure of similarity between the two 3D objects with a value between $+1$ and $-1$, with the former indicating a strong agreement.

\section{Results}
\label{sec:Results}

\begin{figure}[t]
    \centering
    \includegraphics[width = 1\textwidth]{figures/4-A-latent-alphanumeric.pdf}
   \caption{Left: UMAP embedding of latent space trained with 10000 randomly rotated samples of  alphanumeric data. Right: The confusion matrix build from the prediction for 200 samples of seen (a, b, d, e, i, j, z and k) and unseen data (2, x and p). For this calculation the choice of hyperparameters include $\gamma = 8$ and $\beta=5$.}
    \label{fig:4-A-latent_alphanumeric}
\end{figure}

\subsection{Latent clustering for 2D alpha-numeric data}

Given the appropriate choice of hyper-parameters, data classes are clustered together for the alphanumeric data in the latent space and the clusters are correlated in accordance to their shape. The left panel of Figure \ref{fig:4-A-latent_alphanumeric} illustrates the UMAP embedding of 10,000 rotations from a set of samples from the alphanumeric data. For this calculation the choice of hyperparameters include $\gamma = 8$ and $\beta=5$. The right panel shows the confusion matrix constructed from the prediction for 200 samples of the seen data (a, b, d, e, i, j, z and k) and unseen data (2, x and p). The confusion matrix shows that the model predicts a strong affinity between letters with higher shape similarity for the seen and unseen data (for example, the unseen numeral 2 shows the closest match to the letter z from the training set).

\subsection{Latent clustering for 3D tetramino and protein data}
\begin{figure}
    \centering
    \includegraphics[width = 1\textwidth]{figures/4-A-latent-tetraminos.png}
    \caption{Left: t-SNE embedding of 1000 (rotationally augmented) tetraminos from a set of  morphologies. Different colours correspond to different morphologies and opacity indicates train/validation/test set. Right: Morphology of different tetramino types positioned in areas corresponding to their latent location. Colors between left and right panel are corresponding. Tetramino unseen during training and validation indicated with red arrow.}
    \label{fig:4-A-latent-tetraminos}
\end{figure}

The results on the tetramino data are illustrated in the latent embedding in Figure \ref{fig:4-A-latent-tetraminos} and on the protein data in the latent embedding in Figure \ref{fig:4-A-latent-proteins}. In both datasets, the cluster separation between different morphologies was very good. Rotated objects of the same morphology were placed in homogeneous clusters regardless of their orientation. Additionally, in the tetramino data the clusters were arranged so that morphologically similar objects (e.g. E and T, L and I) were closer together in the latent space than dissimilar objects (e.g. I and E). A similar trend was observed in the protein data, where dimeric (two subunit) proteins were all arranged close together and ordered by the size of the protein, whereas elongated monomeric (single-subunit) proteins were placed separately forming a more homogeneous area in the latent space.

\begin{figure}
    \centering
    \includegraphics[width = 1\textwidth]{figures/4-A-latent-proteins.png}
    \caption{Left: t-SNE embedding of 1000 (rotationally augmented) proteins from a set of 8 randomly chosen morphologies. Different colours correspond to different morphologies and opacity indicates train/validation/test set. Right: Morphology of different protein types positioned in areas corresponding to their latent location. Colors between left and right panel are corresponding. Proteins unseen during training and validation indicated with red arrows.}
    \label{fig:4-A-latent-proteins}
\end{figure}

When the network was presented with samples previously unseen during training, they formed separate clusters in the embedding positioned close to similar morphologies, which suggests that the learned latent spaces are continuous (see more in subsection \ref{sec:latent-continuity}) and offers potential use of the method for discovery of new morphologies. At the same time, cluster homogeneity was preserved within the unseen clusters, regardless of the object orientation. In the case of the tetramino data, we introduced a new morphology during evaluation that was a fusion of two similar morphologies existing in training (EL), which, as expected, clustered between the two similar classes (E and L). In the case of the protein data, the two introduced morphologies were selected randomly from a list containing an exhaustive set of proteins from the \textit{E. coli} cytoplasm. The dimeric protein (blue) was placed near other dimers and positioned in a roughly correct position arranged by size (increasing left to right). The monomeric protein with an elongated domain was placed overlapping with another cluster of proteins with elongated domains.

The accuracy of classification as measured by KNN on the protein test set was up to 90\% with the optimal set of hyperparameters.

\subsection{Latent and pose interpolations} \label{sec:latent-continuity}

We performed interpolations across the outputs of the pose channel on alphanumeric data. The results of the interpolations are illustrated in Figure \ref{fig:4-B_pose_angle}. We observed a linear correlation between the pose value and the angle of the rotation, which demonstrates that pose channel does indeed capture information about the pose of the object.

We also explored the extent of disentanglement present in the generated latent spaces. Upon visual inspection we were able to identify morphological semantics across different dimensions (Figure \ref{fig:4-B-latent-continuity} left). In this example using the protein data, dimension 2 appeared to capture the size, whereas dimension 6 described whether the protein was a dimer (two subunits) or a monomer (single unit). Other dimensions supported other morphological features, including elongation (dim 1), smoothness (dim 3 and 4), and toroidal geometry (dim 5).

Additionally, we performed latent interpolations across all dimensions between four existing (encoded) data points in the latent space (Figure \ref{fig:4-B-latent-continuity} right). Non-existing (not encoded) points from the latent space generated realistic reconstructions and there was a smooth transition between different morphologies, including when transitioning between the number of protein subunits. This shows that the generated latent spaces are continuous and suitable to discovery of new morphologies unseen during training.

\begin{wrapfigure}[22]{r}{0.5\textwidth}
    \begin{center}
        \includegraphics[width=0.5\textwidth]{figures/pose_angle_images_interpolation.png}
        \caption{Linear relationship between encoded pose and the associated angle of rotation of the input. The nested panel shows the reconstruction using a given latent encoding ($z$) and varying the value of the pose corresponding to 5 degree intervals.}
        \label{fig:4-B_pose_angle}
    \end{center}
\end{wrapfigure}

\begin{figure}[b]
    \begin{center}
    \includegraphics[width = 1\textwidth]{figures/4-B-latent-interpolations.png}
    \caption{Left: Latent interpolations on protein data in a 6 dimensional latent space. Rows correspond to dimensions and columns correspond to interpolated values, where the central column is the encoded input. Right: Latent interpolations between 4 existing encoded proteins (corners). Decoded reconstructions in between the 4 corners are taken from the latent space at evenly sampled multi-dimensional strides between the encodings of existing proteins. }
    \label{fig:4-B-latent-continuity}
    \end{center}
\end{figure}

\subsection{Affinity-based loss component and the influence of $\gamma$}

Figure \ref{fig:4-C_VAE_shapeVAE} illustrates a comparison between affinity-VAE and a standard $\beta$-VAE ($\gamma=0$ and no pose channel) on alphanumeric data, including the latent space representations (top row) as well as the proximity matrix where the distances between the centres of the clusters are displayed (bottom row). Inspection of the two latent space representations shows that affinity-VAE ($\gamma=10$) is more successful at relating the clusters with higher affinity than the $\beta$-VAE framework ($\gamma=0$). This is confirmed in the proximity matrix where the distances between different clusters (i.e. the off-diagonal elements) are generally much higher in affinity-VAE than in the $\beta$-VAE, which would be expected to improve classification rates due to less cluster proximity contamination. Secondly, letters with similar morphology (e.g. b and d) are in closer proximity in the latent map in affinity-VAE compared with $\beta$-VAE. Another example of this is the proximity of cluster i and j to cluster z in $\beta$-VAE, which have been pushed apart in affinity-VAE due to the low affinity.

When $\gamma$ was set to 0 and pose turned off on protein data, the resulting latent spaces generated by the standard $\beta$-VAE were still organised, however the organisation was by an offset from the centre (radially) and an average intensity (within each cluster) of the object (Figure \ref{fig:4-C_VAE_shapeVAE-tetramino}) rather than morphological affinity (as previously demonstrated in Figure \ref{fig:4-A-latent-proteins}). Since $\beta$-VAE is completely data-agnostic, we have little control over the learned factorised representations and the semantics we want to capture.
\begin{figure
    \begin{center}
    \includegraphics[width =0.7\textwidth]{figures/4-C-bVAE_vs_aVAE.pdf}
    \caption{Latent space representation for the alphanumeric dataset using VAE where $\gamma=0$ and pose channel is off (left column) and affinity-VAE where $\gamma=10$ (right column). Bottom row: latent space proximity matrix displaying the distance between  cluster centres. The values for the distances in the are normalised to be between 0-100 in both matrices. The colour map is used as a guide for the eye to emphasise the largest and smallest numbers displayed on the matrix.}
    \label{fig:4-C_VAE_shapeVAE}
    \end{center}
\end{figure}
Introducing the affinity matrix and a similarity-based loss component organises the latent space by the quality captured through the similarity metric.

\begin{figure}[h
    \begin{center}
    \includegraphics[width=1\textwidth]{figures/4-C-gamma=0-tetramino.png}
    \caption{Latent embedding on a tetramino data from a hyperparameter-tuned $\beta$-VAE ($\gamma=0).$ The latent space is organised by centre offset radially (left) and average intensity within clusters (right).}
    \label{fig:4-C_VAE_shapeVAE-tetramino}
    \end{center}
\end{figure}

\begin{figure}
    \begin{center}
    \includegraphics[width = 1\textwidth]{figures/4-D-VAE_beta_variation.pdf}
    \caption{The latent space is presented for $\beta=1,2,3,4$ and $5$ using original VAE framework ($\gamma=0$). A more effective categorisation of classes is observed for values $\beta>1$.}
    \label{fig:4-D-VAE-beta}
    \end{center}
\end{figure}

\begin{figure}
    \begin{center}
    \includegraphics[ width = 1\textwidth]{figures/4-D-gamma_beta5.pdf}
    \caption{For $\beta = 5$ we have explored the effect of shape-affinity regularisation of latent space for $\gamma =0, 1,3,4,6,10$ and $20$. The colour code is the same as Figure \ref{fig:4-D-VAE-beta}.}
    \label{fig:4-D-AVAE_gamma}
    \end{center}
\end{figure}
\newpage

\subsection{The influence of $\beta$, $\gamma$ and $z$}

\begin{wrapfigure}{r}{0.5\textwidth}
    \begin{center}
    \includegraphics[width=0.5\textwidth, trim={1.25cm 6.3cm 2cm 4.2cm},clip]{figures/4-F-AN_SOAP_FSC.pdf}
    \caption{The left and right panel show a comparison of the latent space representation and the reconstruction of the alphanumeric data for the SOAP and FSC shape descriptors respectively.}
    \label{fig:AN_SOAP_FSC}
    \end{center}
\end{wrapfigure}

In order to achieve an appropriately disentangled latent representation, the value of $\beta$ should be chosen carefully. Since $\beta=1$ corresponds to the original VAE method, a stronger emphasis on the $D_{KL}$ term ($\beta>1$ in our case) encourages learning of a more disentangled latent representation as shown in Figure \ref{fig:4-D-VAE-beta}. This is in agreement with existing literature \cite{Higgins2017, burgess2018}. Within the affinity-VAE framework, the value of $\gamma$ should be chosen in a similar range to that of $\beta$. This is due to the competing effect of the two terms in Equation \ref{eq-shape-vae-loss}.

Figure \ref{fig:4-D-AVAE_gamma} explores the effect of affinity regularisation on the latent space. By switching on the shape-affinity ($\gamma>0$) the clusters are instantly grouped together based on their shape-similarity unlike the $\beta$-VAE framework where the letters do form clusters of their own, but are not closer in the latent space if they are similar in shape (Figure \ref{fig:4-D-VAE-beta}).  As the emphasis on the affinity increases, the latent space becomes increasingly sparse pushing the clusters further apart. The suitability of this empty space for specific studies can be explored by investigating the continuity through latent interpolations and classification of unseen data.

For tetraminos, the optimal range of latent dimensions was between 5 and 9. We also observed a relationship between the number of dimensions and degrees of freedom in the data (e.g. 3 rotations,  translations + semantics). Small values of $\beta$ (but $>1$) gave best accuracy. Increasing the affinity regularisation $\gamma$ produced better KNN accuracy (saturated around value of 1000), however, it also increased reconstruction loss. This is expected, as latent regularisation vs. reconstruction quality is always a trade-off, and in our case latent disentanglement (and therefore accurate classification and recognition) is more important than faithful reconstructions.

\subsection{Similarity function}

To illustrate the influence of the choice of similarity metric used to calculate the affinity matrix on the overall cluster separation we compared various metrics across different datasets.

While the SOAP metric was used for 2D alphanumeric data and FSC for 3D data, a comparison between the two descriptors for the alphanumeric data is provided in Figure \ref{fig:AN_SOAP_FSC}. SOAP provided better cluster separation as well as increased certainty of reconstruction on the alphanumeric data.

We also explored the choice of similarity metric in 3D tetramino data. Figure \ref{fig:4-E-MD_FSC} shows a comparison between no affinity descriptor, mean difference and FSC metrics. Mean difference, unlike avg FSC, is a real space descriptor which is not frequency weighted (Equation \ref{eq-md}).
\begin{equation}
    \frac{1}{N}\sum_{i=0}^{i=N} x_i - y_i
    \label{eq-md}
\end{equation}
where $x_i$ and $y_i$ are voxels in comparison, and $N$ stands for the number of voxels (image size).

While mean difference also improved the cluster separation over a standard $\beta$-VAE, some clusters (e.g. L and I) still remained joint. On the other hand, after employing the FSC as a similarity
metric not only did the cluster separation improve, but also the organisation of clusters was morphologically aligned (e.g. elongated shapes like L, I and T clustered near each other).

\section*{Acknowledgement}

This work was supported by the Medical Research Council [grant number MR/V000403/1] and Ada Lovelace Centre.

This work was supported by Wave 1 of The UKRI Strategic Priorities Fund under the EPSRC Grant EP/W006022/1, particularly the “AI for Science” theme within that grant \& The Alan Turing Institute.

The authors would like to thank James Parkhurst and Joel Greer for data simulation support.

\bibliographystyle{unsrtnat}

\section{Introduction}
\label{sec:Introduction}

Lying at the core of machine learning research, representation learning is one of the most important problems in our data-driven world. Through the recent decades the performance of a task has been largely driven by an appropriately encoded (crafted or learned) representation of an input \cite{Kingma2014}. Furthermore, data, and especially scientific or image data, is often overly redundant and, given it is composed of dependent signals, it could be meaningfully described with a compressed representation. Additionally, the interpretability of the factorised representations plays an important role especially in scientific data. Finally, unlike the standard benchmark data in machine learning, real-life or scientific scenarios are often open problems with no existing annotations or ground truth. The development of such methods for interpretable, factorised representations of data without supervision is therefore crucial to future scientific discoveries.

Many visual recognition, detection, or classification tasks have benefited from disentangled $\beta$-VAE \cite{Higgins2017} representations in the recent years. However, despite significant developments \cite{Bepler2019explicitly, Ziatdinov2021}, learning continuous latent representations that capture translation and rotation along with object semantics remains an open problem to this date. In addition, as such representations were designed to be completely data-agnostic, the approach makes no assumptions about the data resulting in little control over the learned semantics. In certain domains, however, such as visual tasks, common characteristics, such as objects' affinity or similarity, could aid the process of representation learning \cite{neuro, neuro2}.

We illustrate the problem on an open scientific challenge: the identification of molecules in volumetric cryogenic Electron Tomography (cryo-ET) image data. Cryo-ET is an emerging high resolution imaging technique that has the potential to revolutionize our understanding of molecular and cellular biology. Although powerful in their own right, structures of isolated and purified proteins convey little to no information on spatial distribution and interactions between the cellular systems in their native environments. Cryo-ET is uniquely capable of 3D \textit{in situ} imaging, spanning molecular to cellular scales. The main promise of cryo-ET is to deliver such spatial mapping of a cellular landscape \cite{tomo}, otherwise known as visual proteomics \cite{proteo, proteomics}.

Cryo-ET tomograms are generated by collecting a tilt series of a frozen specimen in a transmission electron microscope (TEM). The individual 2D projection images are aligned and back-projected to generate the 3D tomogram \cite{Turk2020}. It enables resolution of the entire proteome of molecules inside whole cells in 3D~\cite{plates,detectors}, with further promise of increasing the precision of this method to 20 Å \cite{resolution-revolution, resolution}.  However, recent advances in instrumentation have not been matched by equivalent methodological developments for extraction of contextual information from reconstructed volumes \cite{proteo}. Such analysis routinely includes recognition and classification of particles of the same class, followed by subtomogram averaging within a class to obtain structures with higher local resolution and signal-to-noise ratios \cite{resolution, Castano2019, Wan2016}. However, particle localisation, recognition and classification are inherently challenging for several reasons including low signal-to-noise ratios (SNR),  molecular crowding, compositional and conformational heterogeneity, the random orientation of molecules and the abundance of different protein types. Many existing computational strategies have been developed to enable subtomogram target identification.

\textbf{Template matching:} This method most commonly used to localise particles in cryo-electron tomograms is a computationally expensive algorithm and relies upon the availability of high-resolution template libraries to match against. Furthermore, it disregards the compositional and conformational heterogeneity of structures in the cellular environment. The development of template-free multiple classification techniques is a significant step towards visual proteomics, a system-wide analysis of the cell’s proteome in 3D.

\textbf{CNNs for multi-class classification:} Faster than template matching owing to their ability to identify textures and abstract image primitives, Convolutional Neural Networks (CNNs) are widely used to determine the probability of a given class member in an image \cite{Foo:22,Kavitha2019}. They have also been widely used in the context of cryo-EM and cryo-ET \cite{Bepler2019, Wagner2019, Palovcak5015, deeptomo}. Furthermore, some modified versions of the CNN framework have achieved rotational invariance \cite{Chidester2019,delchevalerie2021}. However, the dependency of CNNs on manually-labelled data urges further developments.

\textbf{Template-free and label-free unsupervised approaches:} \textit{De novo} visual proteomics in single cells through pattern mining \cite{miningtomo} employs the Fourier Shell Correlation \cite{fsc} score to detect similar patterns from training subtomogram sets and therefore introduces the first template-free and label-free subtomogram classification method. However, the main limitation of this method is the reliance on favourable abundances of a given object for successful classification. On the other hand, authors in \cite{Rice2022} use pattern mining to organise a lower-dimensional embedding generated with CNNs.

\textbf{Variational AutoEncoders (VAE):} VAEs and $\beta$-VAEs (which we build upon in this work) are a class of CNNs which effectively reduce the dimensionality of 3D volume data while encoding it into a low-dimensional latent representation. During training, they organise the latent space encouraging factorised representations of semantic patterns. VAE frameworks have been widely used on cryo-EM \cite{Bepler2019explicitly, Zhong2021} and cryo-ET data. A great example of the latter is the work done by Zeng \textit{et. al} where features of macromolecules and membranes as well as non-cellular features have been coarsely characterised \cite{vaetomo}. This work is an important  validation for the use of VAEs in tomograms. More recently, there has been an increased interest in rotationally invariant VAEs, which we are also exploring in this work. Some examples include rVAE/jrVAE \cite{Ziatdinov2021, ziatdinov_kalinin_2021} and spatial-VAE \cite{Bepler2019explicitly}. Also related to our work are contrastive learning frameworks \cite{contrastive} maximizing the agreement between augumentations of the same data.

In this work, we introduce Affinity Variational Autoencoder (affinity-VAE), a  deep neural network for automatic clustering and classification of multidimensional objects based on their similarity -- in our case, their morphological similarity. We focus on affinity-based latent space regularisation in addition to a standard $\beta$-VAE loss function. We introduce an affinity-based $\gamma$ parameterised loss component along with the addition of a pose channel and an automatically generated affinity matrix to learn the pose of the sample in an unsupervised manner during training. The performance of this method is first investigated on a 2D alphanumeric dataset and then the application of the method is demonstrated  on a simulated 3D cryo-ET example. The preliminary success of this novel approach in comparison with the existing $\beta$-VAE framework demonstrates its potential for application with experimental cryo-ET data.