
======================================

**!important**: Be careful when using `caption_attribute_prediction` (We don't recommend to use)!

======================================

# Dataset Card for LP-MusicCaps-MC

## Dataset Description

- **Repository:** [LP-MusicCaps repository](https://github.com/seungheondoh/lp-music-caps)
- **Paper:** [ArXiv](https://arxiv.org/abs/2307.16372)

## Dataset Summary

**LP-MusicCaps** is a Large Language Model based Pseudo Music Caption dataset for `text-to-music` and `music-to-text` tasks. We construct the music-to-caption pairs with tag-to-caption generation (using three existing multi-label tag datasets and four task instructions). The data sources are MusicCaps, Magnatagtune, and Million Song Dataset ECALS subset.

- [LP-MusicCaps MSD](https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MSD): 0.5M Audio with 2.2M Caption
- [LP-MusicCaps MTT](https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MTT): 22k Audio with 88k Caption
- **LP-MusicCaps MC (This Repo)**: 5521 Audio with 22084 Caption. We utilize 13,219 unique aspects used by 10 musicians in the [MusicCaps dataset](https://huggingface.co/datasets/google/MusicCaps) to perform tag-to-caption generation through LLM.


## Data Instances

Each instance in LP-MusicCaps MC (This Repo) represents multiple image-text pair information with meta-attributes:

```
{
  'fname': '[-0Gj8-vB1q4]-[30-40]',
  'ytid': '-0Gj8-vB1q4',
  'aspect_list': ['low quality',
    'sustained strings melody',
    'soft female vocal',
    'mellow piano melody',
    'sad',
    'soulful',
    'ballad'
  ],
  'caption_ground_truth': 'The low quality recording features a ballad song that contains sustained strings, mellow piano melody and soft female vocal singing over it. It sounds sad and soulful, like something you would hear at Sunday services.',
  'caption_writing': 'This heartfelt ballad showcases a soulful and sad low-quality sustained strings melody intertwined with a mellow piano melody, and a soft female vocal, resulting in an emotionally charged and sonically rich experience for listeners.',
  'caption_summary': 'A melancholic and soulful ballad with low-quality sustained strings, a mellow piano melody, and soft female vocals.',
  'caption_paraphrase': 'A melancholic ballad of soulful sadness featuring a low quality sustained strings melody complemented by a soft, mellow piano melody accompanied by a plaintive, soothing female vocal.',
  'caption_attribute_prediction': 'This soulful ballad features a sustained strings melody that tugs at your heartstrings, accompanied by a mellow piano melody and gentle percussion. The soft, emotionally-charged female vocal delivers poetic and poignant lyrics that speak to the sadness and pain of lost love. The addition of a beautiful string arrangement adds to the melodic depth of the song, making it a truly moving listening experience. With its slow tempo, this track exudes a mellow and introspective vibe, perfect for those moments when you need a moment to sit and reflect on the past.',
  'pseudo_attribute': ['emotional lyrics',
    'slow tempo',
    'gentle percussion',
    'string arrangement'
  ],
  'is_crawled': True,
  'author_id': 4,
  'start_s': 30,
  'end_s': 40,
  'audioset_positive_labels': '/m/0140xf,/m/02cjck,/m/04rlf',
  'is_balanced_subset': False,
  'is_audioset_eval': True
}
```

## Pseudo Caption Example:

Input Tags: 

*"video game theme, no singer, instrumental, analog sounding, small keyboard, beatboxing, playful, cheerful, groovy"*

Output Pseudo Captions

*"instrumental track has a joyful and playful vibe, perfect for a video game theme. With no singer, the analog-sounding music features a small keyboard and beatboxing, creating a groovy and cheerful atmosphere"*

[More Information for pseudo caption generation](https://github.com/seungheondoh/lp-music-caps/blob/main/lpmc/llm_captioning/generate.py)
 

## Data Fields

| Name                         | Type            | Description                                                         |
|------------------------------|-----------------|---------------------------------------------------------------------|
| fname                        | string          | File name of the data                                               |
| ytid                         | string          | YouTube ID of the data                                              |
| aspect_list                  | list of strings | List of unique aspects used by musicians in the MusicCaps dataset   |
| caption_ground_truth         | string          | Ground truth caption for the data                                   |
| caption_writing              | string          | Pseudo Caption generated through a writing instruction              |
| caption_summary              | string          | Pseudo Caption generated through a summary instruction              |
| caption_paraphrase           | string          | Pseudo Caption generated through a paraphrase instruction           |
| caption_attribute_prediction | string          | Pseudo Caption generated through a attribute_prediction instruction |
| pseudo_attribute             | list of strings | List of pseudo-attributes using in caption_attribute_prediction     |
| is_crawled                   | boolean         | Indicates whether the data is crawled or not                        |
| author_id                    | int64           | ID of the author                                                    |
| start_s                      | int64           | Start time in seconds                                               |
| end_s                        | int64           | End time in seconds                                                 |
| audioset_positive_labels     | string          | Positive labels from the AudioSet dataset                           |
| is_balanced_subset           | boolean         | Indicates whether the data is part of a balanced subset             |
| is_audioset_eval             | boolean         | Indicates whether the data is for AudioSet evaluation               |

## Considerations for Using the Data

The LP-MusicCaps dataset is recommended to be used for research purposes. Due to the wrong labeling issue, we recommend not using caption_attribute_prediction and pseudo_attribute unless it is specifically for large-scale pretraining. Additionally, the field "is_crawled" indicates the samples used in the reference paper mentioned below.

## Discussion of Biases

It will be described in a paper to be released soon.

## Other Known Limitations

It will be described in a paper to be released soon.
size_categories:1K<n<10K language:en license:mit music text-to-music music-to-text art arxiv:2307.16372 region:us
{"language": ["en"], "license": "mit", "size_categories": ["1K<n<10K"], "pretty_name": "LP-MusicCaps-MC", "tags": ["music", "text-to-music", "music-to-text", "art"]}
TAGS
#size_categories-1K<n<10K #language-English #license-mit #music #text-to-music #music-to-text #art #arxiv-2307.16372 #region-us 

======================================


!important: Be careful when using 'caption\_attribute\_prediction' (We don't recommend to use)!


======================================


Dataset Card for LP-MusicCaps-MC
================================


Dataset Description
-------------------


* Repository: LP-MusicCaps repository
* Paper: ArXiv


Dataset Summary
---------------


LP-MusicCaps is a Large Language Model based Pseudo Music Caption dataset for 'text-to-music' and 'music-to-text' tasks. We construct the music-to-caption pairs with tag-to-caption generation (using three existing multi-label tag datasets and four task instructions). The data sources are MusicCaps, Magnatagtune, and Million Song Dataset ECALS subset.


* LP-MusicCaps MSD: 0.5M Audio with 2.2M Caption
* LP-MusicCaps MTT: 22k Audio with 88k Caption
* LP-MusicCaps MC (This Repo): 5521 Audio with 22084 Caption. We utilize 13,219 unique aspects used by 10 musicians in the MusicCaps dataset to perform tag-to-caption generation through LLM.


Data Instances
--------------


Each instance in LP-MusicCaps MC (This Repo) represents multiple image-text pair information with meta-attributes:


Pseudo Caption Example:
-----------------------


Input Tags:


*"video game theme, no singer, instrumental, analog sounding, small keyboard, beatboxing, playful, cheerful, groovy"*


Output Pseudo Captions


*"instrumental track has a joyful and playful vibe, perfect for a video game theme. With no singer, the analog-sounding music features a small keyboard and beatboxing, creating a groovy and cheerful atmosphere"*


More Information for pseudo caption generation


Data Fields
-----------


Name: fname, Type: string, Description: File name of the data
Name: ytid, Type: string, Description: YouTube ID of the data
Name: aspect\_list, Type: list of strings, Description: List of unique aspects used by musicians in the MusicCaps dataset
Name: caption\_ground\_truth, Type: string, Description: Ground truth caption for the data
Name: caption\_writing, Type: string, Description: Pseudo Caption generated through a writing instruction
Name: caption\_summary, Type: string, Description: Pseudo Caption generated through a summary instruction
Name: caption\_paraphrase, Type: string, Description: Pseudo Caption generated through a paraphrase instruction
Name: caption\_attribute\_prediction, Type: string, Description: Pseudo Caption generated through a attribute\_prediction instruction
Name: pseudo\_attribute, Type: list of strings, Description: List of pseudo-attributes using in caption\_attribute\_prediction
Name: is\_crawled, Type: boolean, Description: Indicates whether the data is crawled or not
Name: author\_id, Type: int64, Description: ID of the author
Name: start\_s, Type: int64, Description: Start time in seconds
Name: end\_s, Type: int64, Description: End time in seconds
Name: audioset\_positive\_labels, Type: string, Description: Positive labels from the AudioSet dataset
Name: is\_balanced\_subset, Type: boolean, Description: Indicates whether the data is part of a balanced subset
Name: is\_audioset\_eval, Type: boolean, Description: Indicates whether the data is for AudioSet evaluation


Considerations for Using the Data
---------------------------------


The LP-MusicCaps dataset is recommended to be used for research purposes. Due to the wrong labeling issue, we recommend not using caption\_attribute\_prediction and pseudo\_attribute unless it is specifically for large-scale pretraining. Additionally, the field "is\_crawled" indicates the samples used in the reference paper mentioned below.


Discussion of Biases
--------------------


It will be described in a paper to be released soon.


Other Known Limitations
-----------------------


It will be described in a paper to be released soon.



TAGS
#size_categories-1K<n<10K #language-English #license-mit #music #text-to-music #music-to-text #art #arxiv-2307.16372 #region-us 

passage: TAGS
#size_categories-1K<n<10K #language-English #license-mit #music #text-to-music #music-to-text #art #arxiv-2307.16372 #region-us 
