# Compose & Embellish: Piano Performance Generation Pipeline
Trained model weights and training datasets for the paper:
  * Shih-Lun Wu and Yi-Hsuan Yang  
    "[Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach](https://arxiv.org/abs/2209.08212)."  
    _Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_, 2023

**Note:** Materials here should be used in conjunction with our [model implementation Github repo](https://github.com/slSeanWU/Compose_and_Embellish).

## Model characteristics
### Stage 1: "Compose" model
Generates **melody and chord progression** from scratch.

  - Model backbone: 12-layer Transformer w/ relative positional encoding
  - Num trainable params: 41.3M
  - Token vocabulary: [Revamped MIDI-derived events](https://arxiv.org/abs/2002.00212) (**REMI**) w/ slight modifications
  - Pretraining dataset: subset of [Lakh MIDI full](https://colinraffel.com/projects/lmd/) (**LMD-full**), 14934 songs
    - melody extraction (and data filtering) done by **matching lyrics to tracks**: https://github.com/gulnazaki/lyrics-melody/blob/main/pre-processing/create_dataset.py
    - structural segmentation done with **A\* search**: https://github.com/Dsqvival/hierarchical-structure-analysis
  - Finetuning dataset: subset of [AILabs.tw Pop1K7](https://github.com/YatingMusic/compound-word-transformer) (**Pop1K7**), 1591 songs
    - melody extraction done with **skyline algorithm**: https://github.com/wazenmai/MIDI-BERT/blob/CP/melody_extraction/skyline/analyzer.py
    - structural segmentation done in the same way as pretraining dataset
  - Training sequence length: 2400
### Stage 2: "Embellish" model
Generates **accompaniment, timing and dynamics** conditioned on Stage 1 outputs.
  - Model backbone: 12-layer **Performer** ([paper](https://arxiv.org/abs/2009.14794), [implementation](https://github.com/idiap/fast-transformers))
  - Num trainable params: 38.2M
  - Token vocabulary: [Revamped MIDI-derived events](https://arxiv.org/abs/2002.00212) (**REMI**) w/ slight modifications
  - Training dataset: [AILabs.tw Pop1K7](https://github.com/YatingMusic/compound-word-transformer) (**Pop1K7**), 1747 songs
  - Training sequence length: 3072

## BibTex
If you find the materials useful, please consider citing our work:
```
@inproceedings{wu2023compembellish,
  title={{Compose \& Embellish}: Well-Structured Piano Performance Generation via A Two-Stage Approach},
  author={Wu, Shih-Lun and Yang, Yi-Hsuan},
  booktitle={Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2023},
  url={https://arxiv.org/pdf/2209.08212.pdf}
}
```
{"license": "mit", "tags": ["music-generation", "transformer", "pytorch", "audio", "music", "piano"]}
music-generation transformer pytorch audio music piano arxiv:2209.08212 arxiv:2002.00212 arxiv:2009.14794 license:mit region:us
TAGS
#music-generation #transformer #pytorch #audio #music #piano #arxiv-2209.08212 #arxiv-2002.00212 #arxiv-2009.14794 #license-mit #region-us 

# Compose & Embellish: Piano Performance Generation Pipeline
Trained model weights and training datasets for the paper:
  * Shih-Lun Wu and Yi-Hsuan Yang  
    "Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach."  
    _Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_, 2023

Note: Materials here should be used in conjunction with our model implementation Github repo.

## Model characteristics
### Stage 1: "Compose" model
Generates melody and chord progression from scratch.

  - Model backbone: 12-layer Transformer w/ relative positional encoding
  - Num trainable params: 41.3M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Pretraining dataset: subset of Lakh MIDI full (LMD-full), 14934 songs
    - melody extraction (and data filtering) done by matching lyrics to tracks: URL
    - structural segmentation done with A\* search: URL
  - Finetuning dataset: subset of URL Pop1K7 (Pop1K7), 1591 songs
    - melody extraction done with skyline algorithm: URL
    - structural segmentation done in the same way as pretraining dataset
  - Training sequence length: 2400
### Stage 2: "Embellish" model
Generates accompaniment, timing and dynamics conditioned on Stage 1 outputs.
  - Model backbone: 12-layer Performer (paper, implementation)
  - Num trainable params: 38.2M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Training dataset: URL Pop1K7 (Pop1K7), 1747 songs
  - Training sequence length: 3072

## BibTex
If you find the materials useful, please consider citing our work:

# Compose & Embellish: Piano Performance Generation Pipeline
Trained model weights and training datasets for the paper:
  * Shih-Lun Wu and Yi-Hsuan Yang  
    "Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach."  
    _Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_, 2023

Note: Materials here should be used in conjunction with our model implementation Github repo. ## Model characteristics ### Stage 1: "Compose" model
Generates melody and chord progression from scratch.

  - Model backbone: 12-layer Transformer w/ relative positional encoding
  - Num trainable params: 41.3M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Pretraining dataset: subset of Lakh MIDI full (LMD-full), 14934 songs
    - melody extraction (and data filtering) done by matching lyrics to tracks: URL
    - structural segmentation done with A\* search: URL
  - Finetuning dataset: subset of URL Pop1K7 (Pop1K7), 1591 songs
    - melody extraction done with skyline algorithm: URL
    - structural segmentation done in the same way as pretraining dataset
  - Training sequence length: 2400 ### Stage 2: "Embellish" model
Generates accompaniment, timing and dynamics conditioned on Stage 1 outputs.
  - Model backbone: 12-layer Performer (paper, implementation)
  - Num trainable params: 38.2M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Training dataset: URL Pop1K7 (Pop1K7), 1747 songs
  - Training sequence length: 3072 ## BibTex
If you find the materials useful, please consider citing our work:
TAGS
#music-generation #transformer #pytorch #audio #music #piano #arxiv-2209.08212 #arxiv-2002.00212 #arxiv-2009.14794 #license-mit #region-us 
 # Compose & Embellish: Piano Performance Generation Pipeline
Trained model weights and training datasets for the paper:
  * Shih-Lun Wu and Yi-Hsuan Yang  
    "Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach."  
    _Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_, 2023

Note: Materials here should be used in conjunction with our model implementation Github repo. ## Model characteristics ### Stage 1: "Compose" model
Generates melody and chord progression from scratch.

  - Model backbone: 12-layer Transformer w/ relative positional encoding
  - Num trainable params: 41.3M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Pretraining dataset: subset of Lakh MIDI full (LMD-full), 14934 songs
    - melody extraction (and data filtering) done by matching lyrics to tracks: URL
    - structural segmentation done with A\* search: URL
  - Finetuning dataset: subset of URL Pop1K7 (Pop1K7), 1591 songs
    - melody extraction done with skyline algorithm: URL
    - structural segmentation done in the same way as pretraining dataset
  - Training sequence length: 2400 ### Stage 2: "Embellish" model
Generates accompaniment, timing and dynamics conditioned on Stage 1 outputs.
  - Model backbone: 12-layer Performer (paper, implementation)
  - Num trainable params: 38.2M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Training dataset: URL Pop1K7 (Pop1K7), 1747 songs
  - Training sequence length: 3072 ## BibTex
If you find the materials useful, please consider citing our work:
TAGS
#music-generation #transformer #pytorch #audio #music #piano #arxiv-2209.08212 #arxiv-2002.00212 #arxiv-2009.14794 #license-mit #region-us 
# Compose & Embellish: Piano Performance Generation Pipeline
Trained model weights and training datasets for the paper:
  * Shih-Lun Wu and Yi-Hsuan Yang  
    "Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach."  
    _Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)_, 2023

Note: Materials here should be used in conjunction with our model implementation Github repo.## Model characteristics### Stage 1: "Compose" model
Generates melody and chord progression from scratch.

  - Model backbone: 12-layer Transformer w/ relative positional encoding
  - Num trainable params: 41.3M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Pretraining dataset: subset of Lakh MIDI full (LMD-full), 14934 songs
    - melody extraction (and data filtering) done by matching lyrics to tracks: URL
    - structural segmentation done with A\* search: URL
  - Finetuning dataset: subset of URL Pop1K7 (Pop1K7), 1591 songs
    - melody extraction done with skyline algorithm: URL
    - structural segmentation done in the same way as pretraining dataset
  - Training sequence length: 2400### Stage 2: "Embellish" model
Generates accompaniment, timing and dynamics conditioned on Stage 1 outputs.
  - Model backbone: 12-layer Performer (paper, implementation)
  - Num trainable params: 38.2M
  - Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
  - Training dataset: URL Pop1K7 (Pop1K7), 1747 songs
  - Training sequence length: 3072## BibTex
If you find the materials useful, please consider citing our work: