

Overview
The Pop2Piano model was proposed in Pop2Piano : Pop Audio-based Piano Cover Generation by Jongho Choi and Kyogu Lee.
Piano covers of pop music are widely enjoyed, but generating them from music is not a trivial task. It requires great 
expertise with playing piano as well as knowing different characteristics and melodies of a song. With Pop2Piano you 
can directly generate a cover from a song's audio waveform. It is the first model to directly generate a piano cover 
from pop audio without melody and chord extraction modules. 
Pop2Piano is an encoder-decoder Transformer model based on T5. The input audio 
is transformed to its waveform and passed to the encoder, which transforms it to a latent representation. The decoder 
uses these latent representations to generate token ids in an autoregressive way. Each token id corresponds to one of four 
different token types: time, velocity, note and 'special'. The token ids are then decoded to their equivalent MIDI file.
The abstract from the paper is the following:
Piano covers of pop music are enjoyed by many people. However, the
task of automatically generating piano covers of pop music is still
understudied. This is partly due to the lack of synchronized
{Pop, Piano Cover} data pairs, which made it challenging to apply
the latest data-intensive deep learning-based methods. To leverage
the power of the data-driven approach, we make a large amount of
paired and synchronized {Pop, Piano Cover} data using an automated
pipeline. In this paper, we present Pop2Piano, a Transformer network
that generates piano covers given waveforms of pop music. To the best
of our knowledge, this is the first model to generate a piano cover
directly from pop audio without using melody and chord extraction
modules. We show that Pop2Piano, trained with our dataset, is capable
of producing plausible piano covers.
This model was contributed by Susnato Dhar.
The original code can be found here.
Usage tips