|
--- |
|
language: en |
|
tags: |
|
- audio |
|
- music-generation |
|
- sample-generation |
|
- piano |
|
- fine-tuning |
|
- stable-audio |
|
datasets: |
|
- RoyalCities/Vocal_Textures_Dataset |
|
model_name: Royal Cities Vocal Textures (SAO Finetune) |
|
base_model: stabilityai/stable-audio-open-1.0 |
|
license: other |
|
license_name: stabilityai-community-license |
|
license_link: https://stability.ai/license |
|
--- |
|
|
|
|
|
<center><img src="https://i.imgur.com/MJvcnnn.jpeg" alt="Header Logo" width="100%"></center> |
|
|
|
<center> |
|
<h2 style="font-size: 30px;"><u>Royal Cities Vocal Textures (SAO Finetune)</u></h2> |
|
</center> |
|
<center> |
|
<h2 style="font-size: 19px;">Introduction</h2> |
|
</center> |
|
This finetuned Stable Audio Open model specializes in Vocal / Operatic Chord Progressions to support granular music production workflows. Capable of creating an infinite variety of chord progressions, all output is BPM-synced and key-locked to any note within the 12-tone chromatic scale, in both major and minor keys. This model was trained on a custom dataset crafted within FL Studio and features three distinct voicings: |
|
|
|
- **Male Vocals** |
|
- **Female Vocals** |
|
- **Ensemble Vocals** (Combination of Male and Female) |
|
|
|
|
|
<center> |
|
<h2 style="font-size: 19px;">Model Features</h2> |
|
</center> |
|
|
|
- **Multiple Types of Stem Generation:** Outputs three types of voicings with a focus on Chord Progressions only, |
|
- **Tonal Versatility:** Generates stems in any key across the 12-tone chromatic scale, in both major and minor scales. |
|
- **Audio-To-Audio:** Generates interesting vocal timbres when paired with vocal stems. |
|
- **Simplified Scale Notation:** Scales are written using <b><i>sharps only</i></b> in the following format: |
|
|
|
<pre> |
|
<b>Minor Scales</b> |
|
A minor, A# minor, B minor, C minor, C# minor, D minor, D# minor, |
|
E minor, F minor, F# minor, G minor, G# minor |
|
|
|
<b>Major Scales</b> |
|
A major, A# major, B major, C major, C# major, D major, D# major, |
|
E major, F major, F# major, G major, G# major |
|
</pre> |
|
|
|
|
|
For more details on the VSTs and gear used in sample creation, refer to the Gear section below. |
|
|
|
<center> |
|
<h2 style="font-size: 19px;">Training Methodology</h2> |
|
</center> |
|
|
|
This model was designed in conjunction with the Vocal Textures dataset with a focus on only generating vocal chord progressions across the 12-tone chromatic scale in either Female, Male or Ensemble outputs. |
|
|
|
Vocals / Choirs have very long attacks so it was trained with metadata simply classifying each sample as "Chord Progression" along with the necessary key, bpm and bar information. More details can be found in the dataset section. |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>Usage Guide</u></h2> |
|
</center> |
|
|
|
<center> |
|
<h2 style="font-size: 19px;">Supported GitHub Interfaces</h2> |
|
<p style="font-size: 16px;"> |
|
This model works in both the <a href="https://github.com/RoyalCities/RC-stable-audio-tools">RC Stable Audio Gradio</a> and the <a href="https://github.com/Stability-AI/stable-audio-tools">original Stable Audio GitHub</a>. |
|
</p> |
|
</center> |
|
|
|
|
|
<b> |
|
<p align="center" style="font-size: 18px;"> |
|
You have 2 choices of model - A Full Version and a Smaller Quantized Version which uses less VRAM. |
|
</b> |
|
|
|
<b> |
|
<p align="center" style="font-size: 18px;"> |
|
To use the model simply place both the .ckpt file and the config .json inside their own sub-folder within the "models" folder and launch the gradio. |
|
</b> |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>VST Support</u></h2> |
|
</center> |
|
|
|
<b> |
|
<p align="center" style="font-size: 18px;"> |
|
This model has direct VST compatibility in the <a href="https://audialab.com/products/deep-sampler-2/" style="font-size: 20px;">Audialab Engine.</a> |
|
</b> |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>Prompt Structure</u></h2> |
|
</center> |
|
|
|
To ensure the best results, use the following format for your prompts: |
|
<pre><b> |
|
[Vocal Type], Chord Progression, [Key], [BPM], [Bar Count] |
|
</b></pre> |
|
|
|
Vocal Type Prompts |
|
```python |
|
["Male Vocal Texture"] |
|
["Female Vocal Texture"] |
|
["Ensemble Vocal Texture"] |
|
``` |
|
<center> |
|
<p style="font-size: 16px;"> <b>Examples</b> |
|
</p> |
|
</center> |
|
|
|
##### Male vocal prompt with model output. |
|
"Male Vocal Texture, chord progression, D# minor, 128BPM, 8 bars," |
|
<audio controls src="https://huggingface.co/RoyalCities/Vocal_Textures_Main/resolve/main/example_1.mp3"></audio> |
|
|
|
##### Female vocal prompt with model output. |
|
"Female Vocal Texture, chord progression, B major, 120BPM, 8 bars," |
|
<audio controls src="https://huggingface.co/RoyalCities/Vocal_Textures_Main/resolve/main/example_2.mp3"></audio> |
|
|
|
|
|
##### Ensemble Vocal Prompt with model output. |
|
"Ensemble Vocal Texture, chord progression, C major, 140BPM, 8 bars," |
|
<audio controls src="https://huggingface.co/RoyalCities/Vocal_Textures_Main/resolve/main/example_3.mp3"></audio> |
|
|
|
<center> |
|
<p style="font-size: 16px;"> <b>BPMs/Bars</b> |
|
</p> |
|
</center> |
|
|
|
The BPMs ranged from as low as 100BPM up to 150BPM. The main denominations are **100BPM, 110BPM, 120BPM, 128BPM, 130BPM, 140BPM, 150BPM**. |
|
|
|
There are 2 bar settings: **4 bars** and **8 bars**. |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>Dataset Breakdown</u></h2> |
|
</center> |
|
|
|
<center> |
|
<h2 style="font-size: 19px;">Overview</h2> |
|
</center> |
|
|
|
- **Total .wav files**: 5040 |
|
- **Total duration**: 980.64 minutes |
|
- **Average Length**: 11.67 seconds |
|
- **Total Size**: 14.50 GB |
|
- **Sample Rate**: 44100 Hz |
|
|
|
This dataset was meticulously designed to maintain a perfect balance between major and minor chord progressions, ensuring high accuracy in key generation. |
|
|
|
By including an equal number of samples for all 12 tones in both major and minor scales (across 4 and 8 bars), the model achieves precise results. It reliably produces chord progressions in either major or related minor scales, effectively handling the nuances where major and minor scales share the same notes. |
|
|
|
For example, in music theory, the F major scale and D minor scale share the same notes, but the tonal center of the chord progression drives the classification. After training, the model accurately generates the correct scale based solely on the prompt. |
|
|
|
##### "Male Vocal Texture, chord progression, D minor, 150BPM, 8 bars," |
|
<audio controls src="https://huggingface.co/RoyalCities/Vocal_Textures_Main/resolve/main/example_4a.mp3"></audio> |
|
|
|
##### "Male Vocal Texture, chord progression, F major, 150BPM, 8 bars," |
|
<audio controls src="https://huggingface.co/RoyalCities/Vocal_Textures_Main/resolve/main/example_4b.mp3"></audio> |
|
|
|
|
|
<center> |
|
<h2 style="font-size: 19px;">Dataset Details</h2> |
|
</center> |
|
|
|
<table align="center" style="width: 80%; border-collapse: collapse;"> |
|
<thead> |
|
<tr> |
|
<th style="border: 1px solid black; padding: 8px;">Vocal Type</th> |
|
<th style="border: 1px solid black; padding: 8px;">Major Progression (4 bars)</th> |
|
<th style="border: 1px solid black; padding: 8px;">Major Progression (8 bars)</th> |
|
<th style="border: 1px solid black; padding: 8px;">Minor Progression (4 bars)</th> |
|
<th style="border: 1px solid black; padding: 8px;">Minor Progression (8 bars)</th> |
|
</tr> |
|
</thead> |
|
<tbody> |
|
<tr> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;"><b>Male</b></td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
</tr> |
|
<tr> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;"><b>Female</b></td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
</tr> |
|
<tr> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;"><b>Ensemble</b></td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
<td style="border: 1px solid black; padding: 8px; text-align: center;">420</td> |
|
</tr> |
|
</tbody> |
|
</table> |
|
|
|
<center> |
|
<p style="font-size: 16px;"> <b>The dataset will be released in the future for researchers and music producers who want to get started in finetuning. </b> |
|
</p> |
|
</center> |
|
|
|
<center> |
|
<h2 style="font-size: 19px;">Technical Specifications</h2> |
|
</center> |
|
|
|
- **Platform**: Runpod |
|
- **Monitoring Tool**: Weights and Biases |
|
- **Epoch**: 23 |
|
- **Steps**: 1800 |
|
- **Learning Rate**: 5e-5 |
|
- **Optimizer**: AdamW |
|
- **Scheduler**: InverseLR |
|
- **Batch Size**: 32 |
|
- **Hardware**: 2x NVIDIA A40 GPUs |
|
|
|
See config file for further details. |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>Limitations and Biases</u></h2> |
|
</center> |
|
|
|
The Model has high accuracy when it comes to staying in key due to the balance in the dataset. The metadata however was designed in such a way that the model is mainly designed to generate chord progressions only - as opposed to the piano model which also generated melodies. This is due to Vocal choir samples often having very long attacks so often sit at the back of a mix or to fill out the frequency space. |
|
|
|
I have noticed some light noise in the outputs - in particular the ensemble progressions. It almost sounds like the model is trying to add other instrumentation in. I think this may be due to the base model primarily being trained on music stems rather than vocals but I cannot say for certain - this will need to be corrected in a future model or when there is far more vocal data. |
|
|
|
Best use case may be to add some light reverb or post-processing for best results / use in a song. |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>Gear Used</u></h2> |
|
</center> |
|
|
|
- **DAW:** FL Studio (Image-Line) |
|
- **Vocals:** Multiple Choir Libraries |
|
- **EQ:** PRO-Q3 (FabFilter) |
|
- **Additional Gear:** Fruity Reeverb 2, Shaperbox 3 (Compressor), Soothe2 |
|
|
|
<center> |
|
<h2 style="font-size: 24px;"><u>License</u></h2> |
|
</center> |
|
|
|
|
|
This model is licensed under the Stability AI Community License. It is available for non-commercial use or limited commercial use by entities with annual revenues below USD $1M. For revenues exceeding USD $1M, please refer to the [LICENSE](./LICENSE.md) for detailed terms. |
|
|
