\section{Conclusion}
In this work we presented a method for multi-instrument transcription, from easily attainable unaligned supervision. We have demonstrated the method's strength for in-the-wild transcription, including cross-dataset evaluation.
We have also showed the simplicity of collecting data for our framework, which generates annotation on its own in a fully-automated process. Our work presents unprecedented transcription quality on a wide variety of instruments and genres. This work's capabilities open several new lines of research.

Besides extending to human voices, additional effects could be added to the detection, including echo, velocity, etc. Beside the added functionality, this would probably improve the basic detection as well. In addition, adding a musical prior, driving predictions to only make sense musically (in a similar manner to a NLP) would also probably boost performance.
Another central direction for future work is generative models. DNN based models that synthesize realistic music, although producing realistic timbre, cannot produce coherent music without conditioning on notes. Generating realistic-sounding music conditioned on notes is ideal for musicians as it enables full control over the content of the produced music. We believe the transcriptions produced using our approach can be used as a conditioning signal for training generative models, by learning the reverse mapping from transcriptions to original audio.
Finally, additional E-M iterations on small data or specific performances, even during inference, would also be an interesting avenue for future research, which we hope this work would inspire.

{\small
\bibliographystyle{IEEEtranN}

\section{Introduction}
\label{section:introduction}
Automatic Music Transcription (AMT) is the task of decoding musical notes from an audio signal, and is one of the most central tasks in Music Information Retrieval (MIR). It benefits musicology and music education, musical search, and could even aid in realistic music synthesis.
AMT is challenging due to several reasons, such as notes sharing partial frequencies, polyphony (simultaneous notes played together, analogous to occlusions in computer vision), echo effects,
and multi-instrument performances, escalating complexity.

Unsurprisingly, similarly to fields such as Computer Vision and Natural Language Processing, deep neural networks have contributed to AMT as well. However, as DNNs require massive amounts of training data, progress is limited. The main bottleneck is that manual annotation is severely infeasible, even if done by experts, as it requires highly precise timing. For this reason, for most instruments no datasets of highly accurate annotation have been collected. Collection efforts have concentrated mainly on two instruments. Guitar \cite{DBLP:conf/ismir/XiBPYB18, DBLP:conf/ismir/WigginsK19} annotations are done semi-automatically with human verification, in a difficult to scale process. For the piano, unique equipment (the Disklavier) logs key activity during performance, making annotation trivial and data collection simpler. Indeed, the guitar dataset we use for evaluation~\cite{DBLP:conf/ismir/XiBPYB18} (which is practically the only available one) consists of only $\sim$3 hours of recordings, compared to $\sim$140 hours of piano material~\cite{DBLP:conf/iclr/HawthorneSRSHDE19}. It is therefore not surprising that most AMT literature concentrates on the latter, where supervision and evaluation are clean and readily available~\cite{DBLP:conf/ismir/HawthorneESRSRE18, DBLP:conf/iclr/HawthorneSRSHDE19, DBLP:conf/ismir/HawthorneSSME21}.

As it turns out, even within the case of the piano, supervised detectors struggle to generalize to variances in the instrument or environment, let alone from synthetic to real data. For this reason, for example, the accuracy of SOTA methods degrades in cross-dataset evaluation~\cite{DBLP:journals/corr/abs-2111-03017} (e.g., training on the piano recordings of the MAESTRO dataset~\cite{DBLP:conf/iclr/HawthorneSRSHDE19}, and testing on those of MAPS~\cite{unknown}).
To mitigate these data intensive requirements, a popular approach seeks to annotate existing recordings through alignment of real performances to their corresponding musical score. In other words, an easily obtainable digitized performance (or MIDI) of a musical piece is aligned to a real recorded performance. After the MIDI is warped to best match the recording, it is used as annotation. This is how, for example, the well known MusicNet dataset was constructed (with the support of human verification)~\cite{DBLP:conf/iclr/ThickstunHK17}. While promising, the alignment quality this approach demonstrates is not high enough to be used as labeling for network training. Indeed, the aforementioned dataset is notorious for its labeling inaccuracies~\cite{DBLP:conf/ismir/HawthorneESRSRE18, DBLP:journals/corr/abs-2111-03017}.

In this work, we observe that the alignment process could be intertwined with the training of the transcriber, through the \textit{Expectation Maximization} (EM) framework.
We introduce {Note$_{EM}$}, a framework that supports \textit{unaligned supervision}, based on easy-to-obtain musical scores to supervise in-the-wild recordings. The process comprises three steps (see Figure \ref{fig:overview}): first, we take an off-the-shelf architecture proposed for transcription, and bootstrap its training on synthetic data. Second, for the E-step, we use the resulting network to predict the transcription of unlabeled recordings. The unaligned score is then warped based on the predictions as likelihood terms, and used as labeling. For the M-step, the transcriber itself is trained on the new generated labels. Depending on the metric, best results were obtained when performing one or two such E-M iterations. In any case, alignment based on network predicted likelihoods is considerably more accurate than alignment based on spectral features~\cite{DBLP:conf/iclr/ThickstunHK17} (see Section~\ref{section:experiments}). It also enables better handling of inconsistencies between the audio and the score, which are inevitable.

Using this scheme, we achieve transcription accuracy that outperforms all existing methods on cross-dataset evaluations by a large margin for both the note- and frame-level metrics. For example, we reach 89.7\% note-level and 77.0\% frame-level F1 score on the MAESTRO test set (without using MAESTRO training data), where ~\citet{DBLP:journals/corr/abs-2111-03017} reach 28\% and 60\% when not including MAESTRO in the train set.
Furthermore, we report note-level accuracy that compares or even surpasses fully supervised piano/guitar-specific transcription methods.
This is despite our method being trained on synthetic data and unaligned supervision alone.

{Note$_{EM}$} also enables simple and convenient training on different instruments and genres. To demonstrate this, we train our network on other instruments, such as violin, clarinet, harpsichord, and many others - between 11-22 instruments, depending on the configuration. Furthermore, to evaluate the method's usability, we train it using a small-scale self-collected set of musical performances and corresponding unaligned supervision, and observe similar accuracy. We even generate alternative labeling to the aforementioned MusicNet dataset, which we denote MusicNet$_{EM}$, and demonstrate it is more accurate.  Finally, we also witness satisfying generalization capabilities, through the high quality transcription of unseen instruments and genres such as rock or pop (in which case transcription is pitch only).

Our contributions are as follows:
\begin{itemize}
    \item {Note$_{EM}$} -- A general framework for training polyphonic (multi-instrument) transcribers using unaligned supervision, allowing the use of in-the-wild recordings for training.

    \item A new SOTA note-level F1-score on the MAPS dataset of $87.3\%$ (vs. $86.4\%$ of supervised~\cite{DBLP:conf/iclr/HawthorneSRSHDE19}), and considerable improvement for cross-dataset evaluations. This is even though training is done using less supervision and less data ($\sim$34 vs. $\sim140$ hours).

    \item unprecedented generalization to unseen instruments and musical genres. Results on these genres are unfortunately only qualitative due to lack of ground truth, but they are unmistakably favorable non-the-less.

    \item Alternative annotation for MusicNet, denoted MusicNet$_{EM}$, which is shown to be more accurate.

\end{itemize}

\section{Related Work}

The two common forms of transcription are \textit{note-level}, where start (\textit{onset}) / end (\textit{offset}) note events are detected,
and \textit{frame-level} transcription, where pitches are predicted at every given time,  implicitly determining the duration of notes.
Other forms of transcription include stream-level, where the performance is segmented into different streams or voices.
Segmentation can be according to instrument~\cite{DBLP:journals/taslp/WuC020, DBLP:journals/corr/abs-2111-03017}, but can also be between instances of the same instrument.

While early works reduced the task of transcription to detection of active notes per-frame, later works~\cite{DBLP:conf/ismir/HawthorneESRSRE18, DBLP:conf/iclr/HawthorneSRSHDE19, DBLP:journals/taslp/WuC020} show the advantage of breaking down the detection into two components: onsets - beginning of notes, and frames - presence of notes. This is based on the observation that the more important and distinguished part of a note event is its onset.

In multi-instrument transcription, the simpler form ignores instrument classes, assigning a single class for each pitch~\cite{DBLP:conf/icassp/WuCS19, DBLP:conf/mm/CheukHS21}. Only a handful of works also address, as we do, the problem of note-with-instrument transcription
\cite{DBLP:journals/taslp/WuC020, DBLP:conf/icassp/ManilowSP20, DBLP:journals/corr/abs-2111-03017}. As we demonstrate (Section~\ref{section:experiments}), our approach provides cleaner and more attainable labeling, thus clearly surpassing the performance of these works.

For piano transcription, the main benchmarks are MAPS~\cite{unknown} and MAESTRO~\cite{DBLP:conf/iclr/HawthorneSRSHDE19}. The MAPS dataset consists of synthetic and real piano performances, where usually the real performances are used for testing. MAESTRO is a large-scale dataset containing 140 hours of classical western piano performances, with fine and accurate annotation, generated using a Disklavier. The accurate annotation allows outstanding transcription quality~\cite{DBLP:conf/iclr/HawthorneSRSHDE19, DBLP:conf/ismir/HawthorneSSME21, DBLP:journals/corr/abs-2111-03017}. However, the main drawback of this dataset is the lack of variety: It contains only piano recordings, which prevents generalization to other musical instruments, and even to varieties in recording environments and pianos. Thus, transcription quality degrades significantly even when testing the model on other piano test sets, such as MAPS.

For annotation of guitar transcription, \citet{DBLP:conf/ismir/XiBPYB18} rely on hexaphonic pickup (separated to 6 strings), breaking the problem down into annotation of monophonic music which is simpler than polyphonic.
Unfortunately, this approach still requires manual labor, which limits broad data collection. This results in a small dataset - 3 hours in total. Hence, this dataset can be used for evaluation but is less effective for training in-the-wild transcribers.

For other instruments, or multi-instrument transcription, the main existing dataset is MusicNet~\cite{DBLP:conf/iclr/ThickstunHK17}, which contains 34 hours of classical western music, performed on various instruments. The annotation was obtained by aligning separate-sourced (i.e. by other performers) MIDI performances, rendered into audio, with the real recordings, according to low frequencies. This dataset has the clear advantage of variety, both in instruments and in recording environments, as recordings were gathered from many different sources. However, despite being verified by musicians, the alignment is of poor quality, and timing of notes is not precise, significantly inhibiting learning and performance, as we show. Similar datasets exist - SU~\cite{DBLP:conf/cmmr/SuY15}, extended SU~\cite{DBLP:journals/taslp/WuC020}, and URMP~\cite{URMP} datasets, which suffer from similar limitations and are small.

On the task of instrument-sensitive transcription (note-with-instrument), few works have been done, because
of the aforementioned limitations of multi-instrument datasets.~\citet{DBLP:journals/taslp/WuC020} train and test on MusicNet for this task, but reported note-level accuracies are very low, below 51\% on all instruments except for piano and violin, on which the accuracies are $\sim$69\% and $\sim$61\% respectively.~\citet{DBLP:journals/corr/abs-2111-03017} train on a mixture of datasets - MAESTRO, GuitarSet, MusicNet and Slakh2100 (Synthetic). They map the spectrogram into a sequence of semantic midi events, taking an NLP seq2seq approach. This setting is flexible and allows to easily represent multi-instrument transcription. However, the performance on the cross-dataset, or zero-shot task, is low (below 33\% on note-level F1), and performance on MusicNet is low, even when training on MusicNet (50\% note-level F1 at most).

It is important to note, that none of the latter works propose any framework or method for weakly- or self-supervised transcription. \citet{DBLP:conf/mm/CheukHS21} train instrument-insensitive transcription without supervision using a reconstruction loss and Virtual Adversarial Training~\cite{DBLP:journals/pami/MiyatoMKI19}, but as we show, our framework performs much better, and also allows instrument-sensitive transcription. To our knowledge, our work is the first to propose such a framework for multi-instrument polyphonic music, including instrument-sensitive transcription.

\section{Method}\label{section:method}
\begin{algorithm}[tb]
   \caption{Transcription EM}
   \label{alg:em}
\begin{algorithmic}
  \State {\bfseries Input:} audio $a_1, \dots a_N$, unaligned MIDI $m_1,\dots m_N$
   \State {\bfseries Output:} transcriber $f_\Theta$, labels $y_1, \dots y_N$
   \State pre-train $f_\Theta$ (synthetic)
   \State $y_i, d_i = None, \infty\quad i=1,\dots,N$
   \Repeat
   \For{$i=1$ {\bfseries to} $N$}
   \State $y_i^{temp}, d_i^{temp}= DTW(f_\Theta(a_i), m_i)$
   \If {$d_i^{temp} < d_i$}
   \State $y_i, d_i = y_i^{temp}, d_i^{temp}$
   \EndIf
   \EndFor
   \State $\Theta = argmin\frac{1}{N}\sum_{i=1}^N L(f_\Theta, a_i, y_i)$
   \Until{$\frac{1}{N}\sum_{i=1}^N d_i$ converges}
\State \textbf{return} $f_\Theta$, $y_1, \dots y_N$
\end{algorithmic}
\end{algorithm}

\begin{figure*}
\includegraphics[width=0.99\textwidth]{unaligned_supervision_overview.pdf}
\caption{{Note$_{EM}$} system overview. Left: the overall EM approach. Given a synthetic or otherwise supervised dataset, and an unaligned domain, we start by training the transcriber $\mathcal{T}$ on the synthetic data. Next, we use the transcriber to \textit{label} the domain (E-step, middle). We use this as supervision for further training, resulting in a stronger $\mathcal{T}$ model (M-step, right). Middle: At the core of our unaligned supervision scheme is the alignment step. Probabilities for each note at each timestep are computed using $\mathcal{T}$. Then, the unaligned labels are warped using DTW to maximize said logits. Right: the warped results are accumulated into the aligned dataset, which can be used to retrain $\mathcal{T}$. During training we use pitch shift augmentation, to improve robustness and performance.}
\label{fig:overview}
\end{figure*}

The key observation of our method is that a weak transcriber can still produce accurate predictions if the global content of the outcome is known up to a warping function. These accurate predictions, in turn, can be used as labels to further improve the transcriber itself. As we demonstrate (see Section~\ref{section:experiments}), this approach is more accurate than that of pseudo-labels (see Section~\ref{subsection:pseudo}), due to the unaligned known global content. The weak transcriber thus transforms weak supervision into full supervision and refines itself.

Our method, described in pseudo-code Algorithm~\ref{alg:em}, relies on \textit{Expectation Maximization (EM)} (see Section~\ref{sec:em}), and involves three components (see Figure~\ref{fig:overview} left): (I) Synthetic data initial training (Section ~\ref{section:synth}), (II) aligning real recordings with separate-source MIDI (Section~\ref{subsection:alignment}), including deciding which frames to use and which not to (Section~\ref{subsection:pseudo}). (III) transcriber refinement, including pitch-shift equivariance augmentations (Section~\ref{equivariance}).

\subsection{Expectation Maximization (EM)}
\label{sec:em}
Expectation Maximization (EM) is a paradigm for unsupervised or weakly-supervised learning, where labels are unknown, and are assigned according to maximum likelihood. It can be formulated as an optimization problem:
\[
\Theta^* = \underset{\Theta}{argmax}\;\underset{y_1,\dots, y_n}{max}\;P_\Theta(a_1, \dots, a_n, y_1, \dots, y_n)
\]
where $a_1,\dots,a_n$ are data samples, and $y_1,\dots,y_n$ are their unknown labels. The optimization problem can be solved by alternating steps, repeated iteratively until convergence (assuming some pre-training or bootstrapping of $\Theta$):
\begin{align}
&y_1,\dots,y_n = \underset{y_1,\dots,y_n}{argmax}\; P_\Theta(a_1, \dots, a_n, y_1, \dots, y_n)\label{line1}\\
&\Theta = \underset{\Theta}{argmax}\;P_\Theta(a_1, \dots, a_n, y_1, \dots, y_n)\label{line2}
\end{align}
which are referred to as the \textit{E-step} (\ref{line1}) and the \textit{M-step} (\ref{line2}).

In our scenario, the data samples $a_1, \dots, a_n$ are the unlabelled audio recordings, and $y_1, \dots, y_n$ are the unknown per-frame labels. We assume that the recordings are performances of pre-defined musical pieces $m_1, \dots, m_n$, such as in classical music, in the form of MIDI from other performers. We perform the E-step by aligning $m_1, \dots, m_n$ with the predicted probabilities over $a_1, \dots, a_n$ using dynamic time warping (DTW)~\cite{dtw_cite}.
We initialize $\Theta$ by training on synthetic data which is (trivially) supervised.

\subsection{Initial training}
\label{section:synth}

We use synthetic data (see Section~\ref{sec:data} for details) to train the architecture proposed by~\citet{DBLP:conf/iclr/HawthorneSRSHDE19}. Of course, our training scheme can also be applied to other architectures, but this one has proven to be effective for supervised piano transcription, reaching 95\% note-level and 90\% frame-level F1 scores. It has separate detection heads for onsets, offsets, and frames, allowing to perform alignment according to semantic information. As we show (see Supplementary), onset information is the most effective for alignment.
This initial network is trained to detect only pitch, without instrument, but it can also be further trained to detect instrument as well (see section~\ref{subsubsection:instrument_sensitive}).

\subsection{Labeling}
We label real data using dynamic time warping between the initial network's predicted probabilities and the corresponding MIDIs. This is contrary to \cite{DBLP:conf/iclr/ThickstunHK17}, who compute the dynamic time warping in the frequency space. As can be seen in the Supplementary, MIDI guided alignment yields more accurate labels than simple thresholding. It also provides instrument information.

The alignment process is depicted in Figure~\ref{fig:overview} middle, and essentially relies on Dynamic Time Warping. Using DTW, we search for a chronologically monotonic mapping between the unaligned labeling and its corresponding recording, such that for each selected note the probability, as predicted by the transcription model, is maximized.

We argue that using the network's predicted probabilities as local descriptors for DTW
has the following advantages:

\textbf{(i) Inconsistencies} -- For a separate-source MIDI (i.e., originating from a different performer), inconsistencies between the performances in inevitable. This includes repetitions of cadenzas, and more subtle nuances, such as trills, or in-chord order changing. Precise onset timing can be adjusted locally for each note independently according to predicted likelihoods. Failed detection, whether false positive or false negative, can be avoided based on network's probabilities, i.e., pseudo-labels can also be leveraged in addition to the alignment.

\textbf{(ii) Label refinement} - the labeling process can be repeated during training, thus refining the labels, since the network has improved.

\textbf{(iii) DTW computation speed} - for DTW descriptors, we project the 88 pitches into a single octave (12 pitches) using maximum activation across octave, hence representation length for DTW is 12 rather than 50~\cite{DBLP:conf/iclr/ThickstunHK17}. This has an impact on computation speed because DTW requires quadratic time. After projection, for an audio recording of $\sim$2:30 minutes, DTW takes $\sim$1 second.

\paragraph{Pseudo Labels}\label{subsection:pseudo}
As aforementioned, the alignment can produce false detections, whether positive or negative.
To avoid this false detection automatically, and still leverage all data, we label classes with predicted confidence above a threshold $T_{pos}$ as positive, and classes with predicted confidence beneath a threshold $T_{neg}$ as negative, regardless of the alignment. Classes with probability $0.5<p<T_{pos}$ which were not marked positive are considered unknown and we do not back-propagate loss through them. We do this to allow detection of onsets undetected by the labeling. We do not do the same for negative detection (i.e., $T_{neg}<p<0.5$) as there is already a strong bias against onset detection, as onsets are very sparse (an onset lasts a single frame).

In our experiments we use thresholds $T_{pos} = 0.75$ and $T_{neg} = 0.01$ for all classes - onsets, frames and offsets. We can use a low negative threshold since the MIDI performance already constrains the labels, and activations (whether onset, frame, or offset) are sparse, thus mode collapse is less of an issue.

\subsection{Tonality - Pitch Shift Equivariance}\label{equivariance}
Music transcription has a unique inherent structure, where a pitch shift on the waveform induces a corresponding predetermined translation of the labels.
We leverage this structure by enforcing consistency across pitch shift: We create 11 additional pitch shifted copies of our data, with pitch shifts (in semitones):
$s_i~=~i~+~\alpha_i,\; -5~\le~i~\le~5,\;  \alpha_i~\sim~\mathcal{U}(-0.1, 0.1)$,
where $\mathcal{U}(-0.1, 0.1)$ is the uniform distribution on the interval $[0,1]$, as suggested by Thickstun et al.~\cite{DBLP:conf/icassp/ThickstunHFK18}. \textbf{We compute the labels only for the original copy}, and for each copy shift labels accordingly. This not only augments the data by an order of magnitude, but also implicitly enforces consistency across pitch shift, serving as a regularization, forcing the model to learn tonality.

\subsection{Instrument-Sensitive Transcription (note-with-instrument)}
In this setting, we define a distinct class for each combination of pitch and instrument, i.e., the number of classes $C$ is (number of pitches)$\cdot$(number of instruments).

We start with instrument-insensitive training on synthetic data. To adjust the transcriber to the new task of detecting also instrument, we duplicate the weights of the final linear layer of the onset stack $I + 1$ times: once for each instrument, and one copy to maintain instrument-insensitive prediction. This redundancy serves as regularization and improves learning. Thus, at the beginning of instrument-sensitive training, upon detection of a note, the transcriber will detect the note as active on all instruments. During training the transcriber will learn to separate instruments, according to the labels. We apply the same labelling process to this scenario as well - the difference only being more classes. We maintain the low representation length of $12$ for DTW computation by maximizing activation both across octave and instrument. To allow the transcriber (which is initially insensitive to instrument) to learn instrument separation, we do not use pseudo-labels in the initial labelling, only from the second labelling iteration.

\section{Experiments}\label{section:experiments}
\begin{table*}
\input{tables/table_piano}
\end{table*}
\begin{table*}
\input{tables/table_musicnet_v3}
\end{table*}
\begin{table}
\input{tables/table_guitar_v2}
\end{table}

For all our experiments, we use an architecture similar to the one proposed by \citet{DBLP:conf/iclr/HawthorneSRSHDE19}, but wider, to handle variety in instruments: We use LSTM layers of size 384, convolutional filters of size 64/64/128, and linear layers of size 1024.

We re-sampled all recordings to $16kHz$ sample rate, and used the log-mel spectrogram with 229 bins as the input representation. We used hop length 512. We used the mean BCE loss, with an Adam optimizer, with gradient clipped to norm $3$, and batch size $8$. The initial synthetic model was trained for
$350K$
steps. This took 65 hours on a pair of Nvidia GeForce RTX 2080 Ti GPUs. Further training on real data was done for $90 * |Dataset|$ steps.
In the case of MusicNet$_{EM}$, this is $\sim 90 * 310 = 28K$ iterations. For most experiments, labeling is performed twice: once after sythetic training, and once after $45 * |Dataset|$ steps. Training on MusicNet$_{EM}$, for 28K iterations including 2 labelling iterations which require DTW, took 16 hours on a pair of Nvidia GeForce RTX 2080 Ti GPUs.

In the following, we discuss the data we have used during our evaluations (Section~\ref{sec:data}), we report quantitative results (Section~\ref{sec:eval}), and compare to previous work (throughout the evaluations of Section~\ref{sec:eval}). The affects of the pitch-shift augmentations can be seen in Tables~\ref{transcription_table}, and~\ref{guitarset_table}. Further ablations studies, considering various steps, such as the pseudo-labeling, EM iterations, alignment quality, and others can be found in the supplementary material (Section~\ref{sec:ablations}).
\subsection{Data \& Instrument Distribution}
\label{sec:data}

In our experiments, we use three datasets:

\textbf{MIDI Pop Dataset}~\cite{midiDataset} is a large collection of MIDI files. The data consists of almost $80,000$ songs, from which we used $\sim8,500$ randomly selected ones. $\sim4500$ of the performances, of length 278:09:01 hours, are \textit{mp3} compressed, and the rest with lossless \textit{flac} compression. In total 501:11:30 hours of audio were synthesized from MIDI. We use this dataset to bootstrap the process, by training the system to transcribe the rendered audio according to the original MIDI. Note that for flexibility, we only use pitch labels from this data, without instrument specific labels. \textbf{We use this dataset only for pre-training}.

\textbf{MusicNet}~\cite{DBLP:conf/iclr/ThickstunHK17} comprises recordings of multiple instruments in an unbalanced mix. Labels for this dataset are of lower quality, as they are generated by alignment to musical scores, but in preprocess. Most recordings are of a piano ($\sim$15 out of $\sim$ 34 hours are piano solo, and $\sim$7 other hours include the piano). We use the recordings of this dataset, and their provided unaligned corresponding musical scores.  Instead of the provided labels (or aligned scores), we offer MusicNet$_{EM}$ (in the supplementary material) -- alternative labeling generated by our framework -- and demonstrate their superiority (Section~\ref{section:experiments}).

Our \textbf{Self-Collected dataset} demonstrates the simplicity of collecting data for our method. We gather $74$ additional hours of recordings,
including over $30$ hours of orchestra, $5$ hours of solo guitar (pieces by Albeniz, Sor, and Tarrega), $11$ hours of harpsichord ($6$ hours solo), and more. We use this data to supplement or replace MusicNet in our experiments.

Qualitative results in the accompanying video are from a model trained on all three datasets (the \textbf{MIDI Pop} dataset used only for pre-train).
Our generated annotation for MusicNet, and our code, together with qualitative examples for various genres and instruments, are available at \url{https://benadar293.github.io}.

\subsection{Evaluation}\label{sec:eval}
For our experiments,
we train only on MusicNet$_{EM}$ and/or self collected in-the-wild data, where the model is pre-trained on synthetic data. \textbf{We do not use MAESTRO, MAPS, or GuitarSet for training}. We evaluate our method on piano, guitar, strings, and wind instruments, in an \textit{instrument-sensitive} (i.e., note-with-instruments, see Table~\ref{table:instrument_sensitive_v2}), or an \textit{instrument-insensitive} (see Tables~\ref{transcription_table} (piano),~\ref{table:musicnet_combined} (MusicNet test), and~\ref{guitarset_table} (GuitarSet)) manner. For the latter, only MusicNet is used, while for the former we also train an additional model using the Self-Collected data.

For instrument-insensitive transcription (Tables~\ref{transcription_table},~\ref{guitarset_table},~\ref{table:musicnet_combined}) we report the metrics \textbf{note} (onset detection within 50ms or less) and \textbf{frame} (accuracy in detecting if a note is active/not). \textbf{Note-with-offset}, for varying thresholds, can be found in the Supplementary material. For instrument-sensitive transcription (Table~\ref{table:instrument_sensitive_v2}), we report the \textbf{note-with-instrument} metric, which uses the same 50ms timing rule, but only for notes with the correctly predicted instrument.

\subsubsection{Piano}
For piano transcription, we evaluate on the MAPS and MAESTRO test sets.
Results can be seen in Tables~\ref{transcription_table} (instrument-insensitive) and~\ref{table:instrument_sensitive_v2} (instrment-sensitive). It can be seen that note-level accuracy is near-supervised level, even surpassing supervised-level on MAPS. This is despite training on different datasets and no direct supervision, let alone precise labeling of the exact same instrument. For frame-level accuracy, the task is more challenging, since note endings are typically weak and thus harder to decipher. While this expectedly induces lower $F1$ score for the MAESTRO dataset, we also see near-supervised performance on MAPS. Note that the same training procedure done using original MusicNet annotations yields much lower accuracy. This strongly indicates our annotation is more accurate. Similar results are achieved with self-collected data of $\sim$30 hours of piano and guitar.

\subsubsection{Guitar}
For guitar transcription, we evaluate on the GuitarSet dataset. Table~\ref{guitarset_table} demonstrates \textbf{generalization to a new instrument}, since MusicNet$_{EM}$ does not contain guitar performances. For guitar training data in Table~\ref{table:instrument_sensitive_v2} we use the self-collected $\sim5$ hours of guitar recordings together with MusicNet$_{EM}$.
Results are consistent with the piano experiments, indicating significant improvements.

\subsubsection{String \& Wind Instruments}\label{section:invariant}
As mentioned, existing annotation of the dataset is notoriously inaccurate, and Tables~\ref{transcription_table},~\ref{guitarset_table} indicate our annotation method is more accurate. To further demonstrate this for other instruments, we evaluate on the MusicNet test set using both the original annotation and ours
(Table~\ref{table:musicnet_combined}). Test annotation is done as described in Section~\ref{section:method}, but without the pseudo-labels step. Results can be seen in Tables~\ref{table:musicnet_combined} (instrument-insensitive) and~\ref{table:instrument_sensitive_v2} (instrument-sensitive).

As can be seen in Table~\ref{table:musicnet_combined}, on the note-level, we have conclusive results, that our generated annotation used for training performs significantly better than training on the original annotation (over 20\% difference) on both test annotations. This indicates the method can flexibly extend to novel material with cheap labeling.

\subsubsection{Instrument-Sensitive Transcription}\label{subsubsection:instrument_sensitive}
\begin{table*}
\input{tables/table_note_with_instrument}
\end{table*}

\paragraph{Training \& evaluation} For Quantitative evaluation, we use the 11 instrument classes of MusicNet, with the addition of guitar, together 12 instrument classes. We evaluate on the MusicNet test set, on GuitarSet, and on MAPS. In the instrument-sensitive setting, a note is considered correct only if its predicted instrument is correct (note-with-instrument). We train on MusicNet$_{EM}$ together with the self-collected guitar data (to allow guitar detection since guitar data does not exist in the MusicNet recordings). Similar to Table~\ref{table:musicnet_combined}, we report MusicNet test results both according to our annotation, and the original annotation. Results can be seen in Table~\ref{table:instrument_sensitive_v2}. Metrics are unsurprisingly lower than Table~\ref{table:musicnet_combined}, since instrument detection is required, and confusions can occur e.g. between violin and viola.

We believe the metrics on the original MusicNet test annotation \textbf{are far from reflecting real performance}. We provide a qualitative comparison to~\citet{DBLP:journals/taslp/WuC020} in the Video, clearly demonstrating the better performance of our approach.

\section{You \emph{can} have an appendix here.}
\section{Supplementary Material for "Unaligned Supervision for Automatic Music Transcription in The Wild"}
\subsection{Aligning real data with MIDI from a different source}
\subsubsection{Avoiding Singular Points}\label{section:labelling}
Since we align real recordings with external MIDI (i.e., from a different performer), alignment can fail at points with a contradiction in content between the two performances. This can happen when (i) one sequence has a repeated candenza while the other does not, or (ii) because of subtle nuances, and differences in precise timing of adjacent notes (e.g. in trills, or timing of individual notes within a chord). In such cases, the alignment will collapse a long segment of one sequence into a single frame in the other sequence. The long segment can be e.g. $1$ minute in case (i), or e.g. $1$ second in case (ii). Such frames that are mapped to long segments of the other sequence are called singular points. This issue is discussed by \citet{DBLP:conf/iclr/ThickstunHK17}. Their solution is to verify alignment by experts, and to exclude recordings where this occurs. This prevents the process from being fully automatic, and is less desired. Our solution is to only assign labels to non-singular points, and mask the loss from singular points. We still might assign pseudo-labels to singular points, see Subsection~\ref{subsection:pseudo} in the paper. This allows us to avoid failed alignment and also leverage all data, in a fully-automated process.

In more detail, given an audio performance with frames $1,\dots,T$, and an unaligned midi performance of the same piece with frames $1,\dots T_{target}$, the initial network predicts for each frame $1\le t\le T$ and pitch $1\le f\le 88$ probabilities for onset, frame, and offset. We denote these predictions: $P_{on}, P_{fr}, P_{off} \in[0, 1]^{T\times88}$. Similarly, we denote by $Q_{on}, Q_{fr}, Q_{off} \in\{0, 1\}^{T\times88}$ the onset, frame, and offset activations in the corresponding target midi. As local descriptors $X, Y$ for frames of the audio recording and the midi performance respectively, we use a weighted sum:
\begin{align}
&X = A * P_{on} + B* P_{fr} + C * P_{off} \\
&Y = A * Q_{on} + B* Q_{fr} + C * Q_{off}\label{Q}\\
&X\in\mathbb{R}^{T\times88},\quad Y\in\mathbb{R}^{T_{target}\times88}
\end{align}
where $A >> B >> C$, i.e., the alignment is based mainly on the \textbf{onset information}. In our experiments we used values $A = 100, B = 0.01, C = 0.001$. See Table~\ref{alignment_table} for the significant difference in accuracy, in both note- and frame-level, when aligning according to onset information, compared to aligning according to frame information.

Given a pair of sequences $X, Y$ The DTW algorithm returns an optimal alignment in the form of monotone multi-valued mappings (an index in the source can be mapped to multiple indices in the target):
\[M: X\to Y,\quad M^{-1}: Y\to X\]
where monotonicity  implies
\[i\le j \implies k\le k'\quad\forall k\in M(i),\quad k'\in M(j).\]
and similarly for $M^{-1}$. We define the set of singular points $S = S_1\cup S_2$ where
\begin{align*}
&S_1 = \{i\,:\, |M(i)| > w\}\quad S_2 = \cup_{j: |M'(j)| > w'} M'(j)
\end{align*}
$S_1$ is the set of indices mapped to more than $w$ indices in the target domain (interval of length $>w$ in the target collapses into a single frame in the source), and $S_2$ is the set of indices mapped to indices in the target domain that cover more than $w'$ indices in the source domain (interval of length $>w'$ in the source collapses into a single frame in the target). These window sizes control a tradeoff between precision and recall. We used values $3\le w\le 9$, $w'=100$. Results in Tables~\ref{transcription_table},~\ref{table:musicnet_combined},~\ref{guitarset_table} in the paper were obtained using $w=3$, and Table~\ref{table:instrument_sensitive_v2} using $w=7$. Larger values of $w$ cause noise as they allow imprecise onset timing, and small values of $w'$ (e.g., $w'=3$) result in transcriptions that are entirely staccato.

We then assign labels to non-singular points in the following manner: Each non-singular frame $t$ in the source sequence, is mapped to a set of frames $M(t)$ in the target sequence, where $|M(t)|\le w$. We define the label $\hat{X}(t, p)$ of frame $t$ at pitch $p$ to be the maximum activation of the pitch $p$ across all frames in $M(t)$. Since we have multiple kinds of activations - onset, frame, offset, and none - we use the hierarchy: onset $>$ frame $>$ offset $>$ none.

We then assign labels only to non-singular points, in the following manner: The possible labels are: $3$ - onset, $2$ - frame, $1$ offset, and $0$ - none. We assign labels $\hat{X}$:
\begin{align}\hat{X}_t = elem\_wise\_max_{s\in M(t)}Z_s\quad i\in [T]\setminus S\label{xhat_label}\end{align}
Where $Z$ is the target label, and is defined as follows:
\[
Z = max\{3 * Q_{on}, 2 * Q_{fr}, 1 * Q_{off}\}\in [3]^{T_{target}\times 88}
\]
where $Q_{on}, Q_{fr}, Q_{off}$ are defined as in line \ref{Q} in the equation in the previous section.
Note that \[Z_s\in[3]^{88}\quad 1\le s\le T_{target}\] and the maximum over $s$ in~\ref{xhat_label} is performed entry-wise.

We back-propagate loss only from non-singular points (unless they were marked positive/negative by the pseudo-labeling which we perform afterwards). This enables us to leverage all data, and prevents the need to discard whole pieces because they contain singular points.

\subsubsection{Local-Max Adjustment}
Because of the aforementioned slight differences in precise onset timing between the real recording and its corresponding MIDI, the alignment can produce small errors in onset timing. We further refine the labels for each note independently by adjusting each note onset to be a local maximum across time (according to the predicted probabilities), which allows labeling with accurate onset timing. We do the same for note offsets. Still, offsets require further investigation since they are harder to detect.
This adjustment of onset timing is not possible when aligning spectral features of polyphonic music, as in \citet{DBLP:conf/iclr/ThickstunHK17}. A similar local-max adjustment is performed by \citet{DBLP:conf/ismir/XiBPYB18} for annotation of guitar performances, according to flux novelty (similar to spectral features) rather than a network's predicted probabilities. This however is only possible because the different guitar strings are separated, therefore the annotation is fact of monophonic music.

\section{Data \& Instrument Distribution}
\input{tables/table_data}

As we mention in the paper, the MusicNet dataset provides recordings of multiple instruments, however, the
dataset is imbalanced. Most recordings are of solo piano ($\sim$15 out of $\sim$ 34 hours are piano solo, and $\sim$7 other hours include piano). We demonstrate the simplicity of collecting data for our method, by gathering $74$ additional hours of recordings. The full distribution of instruments can be seen in Table~\ref{table_distrib}. Transcriptions in the video are by a model trained on all data, both MusicNet and the self-collected.

\subsection{Further Experiments \& Ablation Studies}
\label{sec:ablations}
\subsubsection{Alignment Evaluation}\label{subsection:alignment}
\begin{table*}
\input{tables/table_alignment}
\end{table*}
We measure the accuracy of our labeling process on the Maestro validation dataset, for which precise annotation exists. For 46 out of the 105 pieces in the validation dataset, we were able to find additional unaligned midis (to be used instead of those offered with the dataset). We report the note and frame metrics of the alignment w.r.t the ground truth annotation, when aligment is done over precidtions on the model trained on synthetic data. We compare the results to simple thresholding. We also show the higher accuracy of aligning according to onset information rather than frame information, even for the frame-level accuracy. We show results for other parameters as well. Unless otherwise stated, we use local-max adjustment of onset timing with a window size of 7 frames. We do this in an inclusive manner: after the initial alignment, if a neighbor of an onset has a higher onset prediction, we mark it as an onset instead, and repeat this 3 times. We do this for both left and right neighbors, hence the small decrease in precision. All results can be seen in Table~\ref{alignment_table}. We also measure the accuracy on these 46 pieces, after training on them with the labels computed by the alignment (not the ground truth labels), and evaluate the accuracy of the network on them using the ground truth labels (last row in Table~\ref{alignment_table}). Main points to note in the table are: (i) Alignment according to onset information yields much more accurate annotations than aligning according to frame information, even in the frame-level metric. (ii) While annotation according to alignment alone yields slightly better annotation than thresholding with threshold 0.5, the combination of alignment, with thresholding with a higher threshold of 0.75, performs significantly better, with improvement of 4\%. (iii) The window size parameters $w, w'$ control a tradeoff between precision and recall. (iv) Local max adjustment significantly increases note-level recall, also increases frame-level recall, and gives a slight improvement in note- and frame-level F1 score. (v) The actual performance of the network on the 46 pieces after training on them with the computed annotation, is higher than the annotation's accuracy.

\subsubsection{Alignment Vs. Pseudo-Labels}
\begin{table*}
\input{tables/table_pseudo}
\end{table*}
To evaluate the contribution of each of the components - alignment with midi and pseudo-labels, we train two additional models - one where we label the real audio recordings only using pseudo-labels obtained by thresholding with a 0.5 threshold, and one where we label only using alignment. Results can be seen in Table~\ref{table:pseudo}. On the note-level, alignment alone performs better than psuedo-labels on all evaluation sets - MAPS, MAESTRO and GuitarSet. On the frame-level, alignment performs better on the MAPS test set and GuitarSet, while psuedo-labels perform better on the MAESTRO test. Our method which combines both, performs best on all three test sets, on both the note- and frame-level.

\subsubsection{Pitch Shift}
\begin{table}
\input{tables/table_pitch_shift}
\end{table}
An ablation study measuring the effect of pitch shift augmentation can be seen in Table~\ref{table:pitch_shift}: we train an additional model without pitch shift augmentation. We train both models for the same time to compensate for the smaller amount of data when training without pitch shift. For piano transcription, this augmentation gives $\sim$2\% of improvment in both note- and frame-level F1 score, increasing both precision and recall. For guitar, the improvement is 7.5\% note-level and almost 4\% frame-level. We perform the same experiment for the guitar dataset (see Section~\ref{section:experiments}).

\subsubsection{Label Update Rate}
\begin{table*}
\input{tables/table_em}
\end{table*}
To evaluate the effect of repeated updates of annotation (repeating the E-step), we train 3 models with different policies: (i) We compute the labels once only, and train on this annotation. (ii) We update the labels 12 times during training in equal intervals. (iii) We update the labels once, in the middle of training. Single labelling had the highest precision, but lower recall. Results can be seen in Table~\ref{table_em}. Policy (iii) produced the best note-level results, while policy (i) gave the best frame-level results.

\subsubsection{Velocity}\label{section:velocity}
\begin{table}
\input{tables/table_velocity}
\end{table}
Dynamics and velocity are key components of any musical performance, and are a central part of the expressivity. ~\cite{DBLP:conf/ismir/HawthorneESRSRE18, DBLP:conf/iclr/HawthorneSRSHDE19} incorporate velocity into their model, i.e., the model predicts the intensity in which each note was played. The designated equipment they use for data annotation (Disklavier) also provides velocity information. However, in a weakly supervised setting such as ours, velocity becomes a challenge, since there is no direct way to recover the original note velocities from the training data, since the audio recording and the midi performance are from different sources, moreover, velocity is not necessarily well-defined. There might be some correlation between the real performances and the corresponding midi performances, but this is not guarantied. Note that velocity annotation only exists for piano datasets (MAESTRO and MAPS) but neither for GuitarSet nor MusicNet.

When evaluating on the MAESTRO an MAPS test sets, The best velocity predictions were made by the initial model trained on synthetic data, as it was trained with full supervision over the velocity. I.e., the real data did not improve velocity prediction - see Table~\ref{velocity_table}. We tried using velocities from the midi (Table~\ref{velocity_table} AL), and using velocities predicted by the initial model as labels (Table~\ref{velocity_table} PL), but this did not improve velocity prediction. Since accurate velocity information cannot be derived from separate-source midi, we believe self-supervision is the main direction for training velocity detection, and we leave this to future work.

\subsubsection{GuitarSet Full Metrics}
\begin{table}
\input{tables/table_guitar}
\end{table}
Results can be seen in Table~\ref{guitarset_table_full}.

\subsubsection{Frame \& Offset Detection}
\begin{table*}
\input{tables/table_note_with_offset}
\end{table*}
Onsets by definition are the initial appearance, or beginning of notes, and their lengths do not vary between notes - long notes and short notes have an onset with the same length, which is typically defined to be a single frame. Thus, there is a strict correspondence between onsets in a real performance and its corresponding midi, up to a warping function. However, frame activation determines the duration of a note, which lasts several frames and can significantly vary between different notes. The musical score of a piece has instructions for note duration, which provides approximate information that enables learning frame-level transcription in the weakly supervised setting. However, small discrepancies can exist between the real and the midi performances, even after warping, as the exact time of offset can slightly vary between performances. Therefore, although there is improvement in frame-level accuracy gained through weak supervision, it is moderate. These small discrepancies in performance explain the gap between supervised and weakly supervised learning in the frame-level accuracy in Table~\ref{transcription_table} (79.6-81.4\% vs. 84.9\%) and between note-level accuracy and frame-level accuracy in the weakly supervised setting (79.6-81.4\% vs. 87.3\%). However, as we've explained in Section~\ref{section:introduction}, the human ear is sensitive mainly to the onset time, and less to the notes' precise duration and offset time, assuming note duration is approximately correct.

To measure the accuracy of our trained model in detecting note offsets, we compute the note-with-offset level metrics for different thresholds. The standard tolerance for offset detection is 50 milliseconds, or \%20 of the note length, whichever is greater. Results can be seen in Table~\ref{table:offset}. It can be seen that the contribution of unaligned supervision to offset detection is small, and increases as the offset tolerance thresholds are increased.

We believe frame-level detection, together with offset detection, can be further improved through time-stretching consistency, and this is an important direction for future work.

\subsubsection{MAESTRO with unaligned supervision}
\begin{table}
\input{tables/table_train_accuracy}
\end{table}
An important question that arises is what is the accuracy on the test set, when some samples from the test domain, or samples similar to the test domain, are seen during training, but without labels, only unaligned supervision. To evaluate this, we searched for midi performances of pieces in the MAESTRO dataset, unaligned and by other performers. We were able to find such performances for 46 pieces from the MAESTRO validation set, of total time 6:57:22. We denote this by MAESTRO$_{EM}$. We conduct two experiments: (i) We train on MAESTRO$_{EM}$ alone using our method, without the ground truth labels, and then measure accuracy on MAESTRO$_{EM}$ w.r.t. the ground truth labels. (ii) In another experiment, we add MAESTRO$_{EM}$ to MusicNet$_{EM}$ to measure the effect on the MAESTRO test set. Results can be seen in Table~\ref{train_table}, rows 1-2.


\section{INTRODUCTION}
\label{sec:intro}

\subsection{Presentation of the project}

The Provence Adaptive-optics PYramid RUn System (PAPYRUS) is an adaptive optics system to be installed
at the Coude focus of the T152 telescope (diameter of 1.52 m) at Observatoire de Haute Provence (OHP).
The main specificity of Papyrus is to use a pyramid wavefront sensor, which shows better SNR
performances than the usual Shack-Hartmann wavefront sensor. The pyramid wavefront sensor has
widely been described theoretically \cite{fauvarque2017,fauvarque2019kernel} and tested on bench \cite{janin2019adaptive}. PAPYRUS is a demonstrator of feasibility on sky
and a TRL maturation step in between the current tests performed at LAM and the future generation of
pyramid wavefront sensors such as ELT/HARMONI.\\

The specificity of PAPYRUS is to be a low cost and student led project. Indeed PAPYRUS will use
components already available at LAM and ONERA in order to drastically reduce hardware costs. The
PAPYRUS project encompasses all the following steps:
\begin{itemize}
    \item System specification and requirements
    \item System optical and mechanical design
    \item Assembly and testing at LAM
    \item Installation and testing at OHP
    \item Maintenance and eventual upgrades at OHP
\end{itemize}

\subsection{PAPYRUS project objectives}

As stated above, PAPYRUS is a demonstrator of capability to place a pyramid wavefront sensor on
sky. The LAM/GRD and ONERA teams will gain a valuable knowledge on the pyramid behaviour,
especially regarding the development of specific control laws. The impact of the PSF modulation
on the pyramid will be a major topic of study thanks to a dedicated modulation mirror and to a dedicated
pyramid focal plane camera \cite{chambouleyron2021focal} (“gain scheduling camera”). Management of the pyramid optical gains and non-linearities is still a subject under active research. PAPYRUS might be upgraded in the future to test new techniques or components for astronomy with AO.\\

Moreover, PAPYRUS data will be used to check image post-processing possibilities from pyramid AO systems. PSF analysis, PSF estimation \cite{fetick2020blind} or deconvolution \cite{mugnier2004mistral} is envisaged.\\

Finally, PAPYRUS will be a pedagogical on-sky AO bench located near Marseille (France) to teach on AO techniques students coming from different laboratories or summer schools at OHP. Observing proposals may be issued by astronomers once the system performances and sky coverage have been validated on sky.

\subsection{The Observatoire de Haute Provence (OHP) site}

The astronomical site of OHP, located inthe South-East of France, has been chosen for accessibility reasons. It is the closest observatory to the development laboratory at LAM, the OHP T152 telescope has available space for installing an AO bench and available observation time. The telescope observation time will have to be shared with the AURELIE instrument, the latter being used only 50 (clear) nights per year approximately. The OHP suffers from approximately 60-80 cloudy nights per year (figure 3), it remains around 220-240 available clear nights per year for PAPYRUS.\\

The OHP is supporting the PAPYRUS effort. OHP will allow installation of the AO bench on the telescope and teaching some PAPYRUS people to use the telescope in autonomy. The T152 telescope is made of four mirrors of 88\% reflectivity each, providing a total transmission of 52\%. Recent coating on the primary mirror might slightly improve this transmission factor. The main drawback is the T152 important pointing error of $\pm 2$~arcmin. Once the star has been located in the T152 within this error range, a manual search would have to be performed to make it appear on the PAPYRUS camera. This manual searching phase will reduce slightly the effective AO lock time of PAPYRUS but is perfectly manageable according to OHP astronomers and our own usage of the telescope.

\section{Adaptive Optics Design}

\subsection{Atmospheric conditions at OHP}

The adaptive optics design and performances depends on the atmospheric conditions at the observing site. Data from OHP \cite{OHPwebsite} in Fig.\ref{fig:ohpseeing} show $\sim 60$ nights of good seeing $s<2"$. The median of the data is in between $s=2"$ and $s=4"$, representing approximately $200$ nights per year. These values give a typical Fried parameter $4<r_0<6$ cm at a wavelength of $600$ nm.\\

The median wind speed at OHP is about $5$ m/s, with bursts up to $7$ m/s.

\begin{figure}[h]
\centering
	\includegraphics[width=0.6\columnwidth]{images/ohp_seeing_2018.PNG}
    \caption{Statistics of seeing at OHP in 2018. Image taken and adpted from OHP website.}
    \label{fig:ohpseeing}
\end{figure}

\subsection {Adaptive optics error budget}

The performances of the system may be stated in term of Strehl ratio between the AO corrected long exposure and the T152 diffraction limit. Using the Marechal approximation, this Strehl ratio is related to the residual electromagnetic phase variance over the pupil as
\begin{equation}
    S_r = e^{-\sigma^2}
\end{equation}
The usual way of balancing an AO error budget\cite{rigaut1998} is to write the phase variance $\sigma^2$ as the quadratic sum of different contributions
\begin{equation}
    \sigma^2 = \sigma_\text{fitting}^2 + \sigma_\text{aliasing}^2 + \sigma_\text{noise}^2 + \sigma_\text{temporal}^2 + \sigma_\text{others}^2
\end{equation}
The definition and computations of these terms is given below. These expressions being chromatic, we choose to give all values in the visible, at a wavelength of $635$ nm.

\subsubsection{Fitting error}

The so-called fitting error is due to the limited number of actuators of the deformable mirror. Thus the high spatial frequencies of the turbulent phase cannot be corrected by the AO system and are present in the residuals. The expression of this error is
\begin{equation}
    \sigma_\text{fitting}^2 = 0.27 \left(\frac{d}{r_0}\right)^{5/3}
\end{equation}
with $d$ the pitch between two actuators and $r_0$ the Fried parameter of the turbulence. For our $17 \times 17$ deformable mirror on a $D=1.52$ m pupil, the pitch is $9.5$ cm, giving an error of
\begin{equation}
    \sigma_\text{fitting}^2 = 0.6 \text{\;rad}^2
\end{equation}

\subsubsection{Aliasing error}

The wavefront measurement is sampled for detection, producing aliasing of the high frequencies onto the low frequencies. This information from the high frequencies enters in the AO loop and generates a deformable mirror response. However the aliasing is smaller on a pyramid wavefront sensor than ona Shack-Hartmann wavefront sensor. It is then neglected in our AO budget
\begin{equation}
    \sigma_\text{aliasing}^2 = 0 \text{\;rad}^2
\end{equation}

\subsubsection{Noise error}

The photon noise and detector noise both produce intensity fluctuations on the wavefront sensor and consequently in the estimated wavefront. The advantage of our AO design is to use a low-noise OCAM$^2$K detector for wavefront sensing. Detector noise is thus neglected.\\

Since PAPYRUS will be targeting bright stars in its early design, photon noise will be mitigated. In order to achieve a correct SNR $=10$ on each measurement, one needs $100$ photons per point of measure per frame. This flux on the detector translates into flux of the AO guiding target as
\begin{equation}
    N_\text{ph/m2/s} = N_\text{ph/meas/frame}\frac{F}{\eta}\left(\frac{n_{meas}}{D}\right)^2
\end{equation}
where $N_\text{ph/m2/s}$ is the flux of the guiding target in photons per m$^2$ per second, $N_\text{ph/meas/frame}=100$ is the desired flux on the detector in photons per point of measurement per frame, $F=300$ Hz is the frame frequency, $\eta\simeq 10\%$ is the global photon efficiency from telescope to the WFS detector, $n_{meas}=16$ is the number of measurements in the diameter, $D=1.52$ m is the telescope diameter. One finds
\begin{equation}
    N_\text{ph/m2/s} = 33\times 10^6
\end{equation}
For a wavelength of $700$ nm and a filter width of $20$ nm, this flux is obtained for stars of magnitude $V\leq 4$. With this condition on the star magnitude the noise error propagated in the loop is neglected
\begin{equation}
    \sigma_\text{noise}^2 = 0 \text{\;rad}^2
\end{equation}

\subsubsection{Temporal error}

The temporal error comes from the delay between the actual phase present on the pupil and its correction by the deformable mirror. The temporal error writes
\begin{equation}
    \sigma_\text{temporal}^2 = 0.04 \left(\frac{V}{DB}\right)^2 \left(\frac{D}{r_0}\right)^{5/3} \sum_{n=1}^{N_r}(n+1)^{-2/3}
\end{equation}
with $V$ the turbulence equivalent wind speed (Taylor frozen flow hypothesis), $B$ the AO loop bandwidth and $N_r$ the number of radial modes corrected. For a control law including an integrator, the equivalent bandwidth writes
\begin{equation}
    B = \frac{F}{2\pi}\sqrt{\frac{g}{1+2\tau F}}
\end{equation}
with $g$ the integrator gain and $\tau$ the loop delay. A reasonable choice of gain ensuring both performances and robustness is
\begin{equation}
    g = \frac{1}{1+\tau F}
\end{equation}
Considering $\tau F = 2$ frames delay for PAPYRUS and a wind speed $V = 7$ m/s, we deduce the bandwidth and then the temporal error
\begin{equation}
    \sigma_\text{temporal}^2 = 5.9 \text{\;rad}^2
\end{equation}

\subsubsection{Other sources of error}

Other sources of error include, but are not limited to, scintillation on the pupil, non-common path errors, and calibration errors. These errors are not taken into account, especially in front of the important temporal error.

\subsection{Conclusions on the budget}

The error budget above is highly dominated by the temporal error. This error is established with the highest wind speed at OHP, and thus corresponds to a worst case of PAPYRUS usage. It is planned to mitigate this error by implementing more complex control laws to increase the bandwidth, and using a better RTC. Indeed we plan to replace our current computer (run on Matlab) by a dedicated ALPAO high-speed RTC.\\

Adaptive-optics corrected PSF are simulated using OOMAO\cite{conan2014} end-to-end simulations with the PAPYRUS system parameters, see Fig. \ref{fig:oomaopsf}. The temporal error being the dominant one, we use three different AO loop frequencies for the simulations. It shows that the current $F=300$ Hz is quite limitating and incline us to go for a better RTC at $F=500$ Hz at least.

\begin{figure}[h]
\centering
	\includegraphics[width=0.7\columnwidth]{images/papyrus_oomao_psf.PNG}
    \caption{PSF from end-to-end simulations. Three cases of loop frequency are considered to simulate the PAPYRUS PSF.}
    \label{fig:oomaopsf}
\end{figure}

\section{Optical Design}

\subsection{Initial data and design challenges}
The optical design is built to work together with the T152 telescope at OHP. It represents an \textit{F/28.56} Cassegrain system with the focal length of 43.4 m and the maximum field of view (FoV) of $\pm 3^{\circ}$.
The telescope has residual aberrations, which don't exceed $16.8 \mu m$,while the Airy disk radius is $13.95 \mu m$.

Also, we had certain cameras available for the scientific and wavefront sensing branches as well as the commercial deformable and tip-tilt mirrors. Their parameters were also driving the design.

Thus, except of the telescope parameters the initial data for the optical design development included:
\begin{enumerate}
    \item Diameter of off-the-shelf mirror used for slow tip/tilt =25.4 mm;
    \item Diameter of the existing DM = 37.5 mm;
    \item Max diameter of the existing fast tip/tilt mirror = 12 mm;
    \item Format of ORCA science camera = $13.312 \times13.312 mm^2$ with $2048 \times 2048$ pixels;
    \item Format of OCAM WFS camera= $5.76 \times 5.7 mm^2$ with $240\times240$ pixels;
    \item The measured  parameters of existing glass pyramid: facet angle = $8.9^{\circ}$, material LF5, leading to deflection angle of $\pm 5.44^{\circ}$;
    \item Science camera FoV = $\pm 1"$;
    \item WFS FoV = $\pm 0.25:$;
    \item Working wavebands =  400-1000 nm (Science camera) and  630-690 nm (WFS with filter).
\end{enumerate}

In addition to these initial data there were some limitations, which made the development challenging. First, the project budget and the development time were limited. Therefore we had to rely only on off-the-shelf components and iteratively repeat the design to find compromise solutions. Second, since our demonstrator is targeting multiple tasks, its' design should be flexible with a possibility to replace some components,  to connect testing and alignment modules or to feed other instruments through optical ports, to get an access to any par of it for educational purposes.  Third, the optical design of the bench should provide a high image quality to minimise the instrumental WFE and the difference between the science and WFS branches. Finally, there were limitations in overall dimension and optical interfaces because the bench will be installed
on a telescope among the existing instruments.

\subsection{Optical design overview and analysis}

With all of the above-listed initial values and limitations we developed the following optical design (see Figure~\ref{fig:layout}).
In order to provide flexible connection with the telescope and compensate the height difference the movable periscope unit 2, 3 is used to pick up the beam from the  telescope focal plane 1. Hereafter we use two optical relays to form intermediate focal and pupil planes and place key components there. The relays are built on the basis of off-axis parabolic (OAP) mirrors. use of OAP allows to exclude chrmatic aberrations and significantly reduce the overall dimensions, although makes the alignment more difficult. The beam is collimated (4) and the pupil is imaged on the slow tip-tilt mirror 5. Further the beam is focused again (6) with $-0.5^x$ magnification and the pupil is re-imaged to the infinity. Folding mirror 7 represents an optical port, which can be used to couple the telescope simulator optical to the intermediate focus 8. The beam is collimated again (9) and the pupil is imaged onto DM 10. Then it is focused (11) in the same way with $–1^x$ magnification and the pupil projection to the infinity. The image formed in focal plane is detected by science camera 13. Before the science camera a beamsplitter 12 is mounted to feed the WFS branch. Note, that the WFS branch uses lens optics, because the components there should have short focal lenghts and work with relatively large angular fields. The WFS $1^{st}$ intermediate focus 14 can be used to set a diaphragm. The beam is re-collimated (15) and the pupil is projected to the modulation mirror 17. In the formed parallel beam after 15 a filter 16 can be mounted to minimize the WFS chromatism. It should be noted that some chromatic aberration is inherent to the pyramid itself regardless of the auxiliary optics properties. The beam is focused (18), while the pupil is projected to the infinity. The focused beam is split (19) to form two identical images at the tracking camera 20 and the tip of pyramid 21. The pyramid splits the beam in quadrants with $5.44^\circ$ deflection. The resultant beams are collimated (22) and the pupil is projected to the WFS camera 23.

\begin{figure}[H]
\centering
	\includegraphics[width=10 cm]{papyrus_optics.png}
    \caption{General view of the PAPYRUS optical design  (see the text for the elements notation). }
    \label{fig:layout}
\end{figure}

Thus, the optical design consists of a series of functional elements, each served with a collimating and focusing relay. Since each focusing component should project the pupil to the infinity, the pupils coincide with the focusing components focal planes. For the same reason, the collimating components form the pupil images in their back focal planes.
The actual focal lengths and magnifications in the system were choosen to fit the initial design conditions and to obtain the focal lengths, $F/\#$ and field values close to those of available commercial components.\\

The Figure \ref{fig:sci cam} shows the image quality in the field of view of science camera. The spot diagram root mean square (RMS) radii vary between 5.3 and 14.9 $\mu m$, while the Airy disk radius is $10.1 \mu m$. So, the image quality is diffraction limited on-axis and slightly degrades towards the field edge. The WFE RMS value is $0.048 \lambda$ with the maximum of $0.16 \lambda$ at 700 nm. The on sky quality will  however be dominated by AO residuals and anisoplanatism, so this design is completely accepted for PAPYRUS.

\begin{figure}[H]
\centering
	\includegraphics[width=16 cm]{Sci_CAm_quality.png}
    \caption{Image quality at the scientific camera: A -- spot diagrams over the FoV (the colors correspond to wavelengths in $\mu m$); B -- wavefront error in waves $@ 0.7 \mu m$  referred to the exit pupil plane. }
    \label{fig:sci cam}
\end{figure}

The image quality assessment for the WFS branch is shown in Figure~\ref{fig:track cam}. The geometrical spot size remains stable across the field, since the nominal pyramid FoV is only $\pm14.4"$. It has the RMS radius of 5.1-6.0 $\mu m$ , while the Airy radius is 12.1 $\mu m$. The WFE for a single pyramid facet is $0.046 \lambda$ RMS at 700 nm. It becomes possible to reach the diffraction limit only with use of a red band filter. Otherwise the chromatic aberrations inherent to this system would blur the spot images by factor of 4.

\begin{figure}[H]
\centering
	\includegraphics[width=16 cm]{Track_Cam_quality.png}
    \caption{WFS branch image quality at the tracking camera: A -- spot diagrams over the FoV (the colors correspond to wavelengths in $\mu m$ with filter); B -- wavefront error in waves $@ 0.7 \mu m$  referred to the exit pupil plane. }
    \label{fig:track cam}
\end{figure}

We would like to note separately the design issue related to the last WFS lens, which forms the image on the camera after the pyramid. It should work with a relatively low $F/\#$ and extended field of view, but also in an unordinary setup, where the pupil is shifted and doesn't coincide with the nominal aperture stop or its' images. In the meantime this lens should be an off-the-shelf component and provide a high image quality. A $Canon^{TM}$ EF-S 24mm F/2.8 STM pancake photographic lens was chosen, since it is has small total length and provide high image quality. However, to account for possible vignetting and aberrations growth is this setup we had to reconstruct an approximate optical design of the lens. An approximate algorithm of this reverse-engineering is given below. We hope that it can be useful for those, who are going to use commercial lenses in non-standard laboratory setups.
\begin{enumerate}
    \item The general data about the lens like the number of lenses and groups as well as the composition of positive and negative components were taken from the manufacturer\cite{CANON}.
    \item The closest patent was found\cite{Ohtake97}. The lens composition is similar to the known one and the publication date is the closest ot the lens commercial release.
    \item The patent lens was scaled to fit the focal length and the first lens diameter of the commercial lens.
    \item The lens was optimized to fit such criteria as the MTF values and vignetting known from open sources. The radii of curvature and thicknesses were used as free variables. In addition to standard boundary conditions defining the edge and center thicknesses, the overall system length and diameters and optical powers of individual components were restricted. The lens design after optimization is shown in Figure~\ref{fig:canon},
\end{enumerate}

\begin{figure}[H]
\centering
	\includegraphics[width=16 cm]{fake_canon.png}
    \caption{Reconstructed design of the WFS lens: A -- general view, B -- MTF in comparison with the reference curve from open sources.}
    \label{fig:canon}
\end{figure}

The figure \ref{fig:papyrusview} shows the opto-mechanical implementation of PAPYRUS with the different elements already presented before. The optomechanical design uses off-the-shelf parts for all the adjustments and holding the components and has as less as possible custom parts only as mechanical interface adapters. The mechnaical mounts provide the necessary number of degrees of freedom for every unit. The entire bench is assembled on a single breadboard.
The optical path is divided into straight segments and all the components of such a segment are mounted on a single optical rail. Position of each of the optical rails is given by a couple (line+corner) flat templates. The periscope assembly and the cameras are mounted via custom adapters, which can be 3d-printed or milled in aluminium. All the OAP mirrors have adapters to compensate the vertex lateral shift. Finally, the WFS camera together with the pyramid are mounted in a custom housing with adjustment rings and attached directly to the WFS camera C-mount.
This optomechanical implementation allows some room for future upgades of the AO bench, and visiting scientific instrument to be adapted to the bench.

\begin{figure}[H]
\centering
	\includegraphics[width=\columnwidth]{images/papyrus_view.PNG}
    \caption{Opto-mechanical model of PAPYRUS in 3D view (left) and top view (right). GSC: gain scheduling camera. DM: deformable mirror. OAP: off-axis parabola. Slow TT: slow tip-tilt mirror.}
    \label{fig:papyrusview}
\end{figure}

\section{Current Status of papyrus}

\subsection{Assembly Integration and Tests}

The AIT of PAPYRUS will be organized following this strategy:
\begin{itemize}
    \item component validation at LAM
    \item global bench alignement and validation at LAM
    \item final coupling and commissioning at T152 OHP.
\end{itemize}

\subsubsection{Test of the pyramid}

The critical components of PAPYRUS have been tested individually. The glass pyramid is obviously one of these critical components. Pictures of the pyramid tip (Fig. \ref{fig:pyramid}) show a "rooftop" configuration due to manufacture precision. The rooftop size is approximately the diffraction size that would prevent WFS sensing in one direction if non modulated. However the modulation of the PSF around the pyramid tip solves this issue.

\begin{figure}[H]
\centering
	\includegraphics[width=0.5\columnwidth]{images/pyramid.png}
    \caption{Picture of the PAPYRUS pyramid tip, showing a "rooftop" configuration.}
    \label{fig:pyramid}
\end{figure}

\subsection{Testing the WFS branch}

The WFS branch is the core of the AO instrument. This branch has already been aligned at LAM (Fig. \ref{fig:wfsbranch}) to test the critical components to work together: the pyramid, the WFS camera, the DM and the modulation mirror. We processed the raw intensities and were able to compute the calibration matrix of the system by sending push-pull commands on the DM.\\

Once the tests performed, the WFS branch has been unmounted. The alignment of the full bench is currently under process at LAM.

\begin{figure}[H]
\centering
	\includegraphics[width=0.9\columnwidth]{images/wfs_branch.PNG}
    \caption{Tests of the WFS branch at LAM. Left: picture of the WFS branch. Middle: image seen on the WFS camera when a poke is sent to the DM. Right: computed slopes maps.}
    \label{fig:wfsbranch}
\end{figure}

\subsection{On-sky measurements at OHP}

Measurements at the Coude focus of the T152 telescope have been performed on July 20th and 21st, 2021. The objective of these measurements is to characterise the telescope pupil plane and focal plane. The possible high-amplitude movements of these planes is problematic for the AO system. However the pupil plane has shown stability better than $1\%$ of the diameter during a long-exposure observation. The movement of the pupil is consequently less than a fraction of point of measurement of the phase (equivalent to a sub-aperture for a Shack-Hartmann WFS).\\

The star position is targeted within a $\pm 2$ arcmin uncertainty diameter in the sky due to the telescope pointing lack of accuracy. However once the star is found it can be correctly centred manually. Then the PSF suffers a small drift of $\sim 20$ arcsec/hour. This drift is supposed to come from the telescope weight unbalance that might accelerate or slow down the Earth rotation compensation. However this drift is small and will be managed by the PAPYRUS slow tip-tilt mirror.

\begin{figure}[H]
\centering
	\includegraphics[width=0.7\columnwidth]{images/T152_pupil_psf.PNG}
    \caption{Characterisation of the OHP/T152 telescope. Left: image of the pupil. The dashed red ellipse shows presence of localised amplitude fluctuations at the edge of the pupil. Right: long-exposure PSF obtained on Vega.}
    \label{fig:T152test}
\end{figure}

\subsection{Schedule}

The schedule of the PAPYRUS project is given on Fig. \ref{fig:schedule}. Integration at OHP is planned for end 2021, and PAPYRUS will be available for the astronomical community beginning or mid 2022.

\begin{figure}[H]
\centering
	\includegraphics[width=0.9\columnwidth]{images/papyrus_schedule.PNG}
    \caption{Schedule of the PAPYRUS project.}
    \label{fig:schedule}
\end{figure}

\section{Conclusion}

As it was presented here PAPYRUS will soon allow LAM to improved its operational expertise on Pyramids and deliver an AO corrected image at OHP. The system has been designed, the components have been characterised, and the first integration is underway with preparations for arrival at OHP also well advanced. The project will soon be opened to the community and we will welcome any new idea and collaboration to add new capabilities to the bench.

\section{Acknowledgement}

This project has been supported by ONERA, LAM and ALPAO in particular the WOLF ANR. We also thanks First Light and the OHP for their contribution this project.