
## Dataset Summary

In [CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval](https://ai-muzic.github.io/clamp/), we introduce WikiMusicText (WikiMT), a new dataset for the evaluation of semantic search and music classification. It includes 1010 lead sheets in ABC notation sourced from Wikifonia.org, each accompanied by a title, artist, genre, and description. The title and artist information is extracted from the score, whereas the genre labels are obtained by matching keywords from the Wikipedia entries and assigned to one of the 8 classes (Jazz, Country, Folk, R&B, Pop, Rock, Dance, and Latin) that loosely mimic the GTZAN genres. The description is obtained by utilizing BART-large to summarize and clean the corresponding Wikipedia entry. Additionally, the natural language information within the ABC notation is removed.

WikiMT is a unique resource to support the evaluation of semantic search and music classification. However, it is important to acknowledge that the dataset was curated from publicly available sources, and there may be limitations concerning the accuracy and completeness of the genre and description information. Further research is needed to explore the potential biases and limitations of the dataset and to develop strategies to address them.

## How to Access Music Score Metadata for ABC Notation

To access metadata related to ABC notation music scores from the WikiMT dataset, follow these steps:

1. **Locate the Wikifonia MusicXML Data Link:** Start by visiting the discussion thread on the forum to find the download link for the Wikifonia dataset in MusicXML format (with a .mxl extension). You can find the discussion here: [Download for Wikifonia all 6,675 Lead Sheets](http://www.synthzone.com/forum/ubbthreads.php/topics/384909/Download_for_Wikifonia_all_6,6).

2. **Run the Provided Code:** Once you have found the Wikifonia MusicXML data link, execute the provided Python code below. This code will handle the following tasks:
   - Automatically download the "wikimusictext.jsonl" dataset, which contains metadata associated with music scores.
   - Automatically download the "xml2abc.py" conversion script, with special thanks to the author, Willem (Wim).
   - Prompt you for the Wikifonia data URL, as follows:

   ```python
   Enter the Wikifonia URL: [Paste your URL here]
   ```

   Paste the URL pointing to the Wikifonia.zip file and press Enter.

The below code will take care of downloading, processing, and extracting the music score metadata, making it ready for your research or applications.

```python
import subprocess
import os
import json
import zipfile
import io

# Install the required packages if they are not installed
try:
    from unidecode import unidecode
except ImportError:
    subprocess.check_call(["python", '-m', 'pip', 'install', 'unidecode'])
    from unidecode import unidecode

try:
    from tqdm import tqdm
except ImportError:
    subprocess.check_call(["python", '-m', 'pip', 'install', 'tqdm'])
    from tqdm import tqdm

try:
    import requests
except ImportError:
    subprocess.check_call(["python", '-m', 'pip', 'install', 'requests'])
    import requests

def filter(lines):
    # Filter out all lines that include language information
    music = ""
    for line in lines:
        if line[:2] in ['A:', 'B:', 'C:', 'D:', 'F:', 'G', 'H:', 'I:', 'N:', 'O:', 'R:', 'r:', 'S:', 'T:', 'W:', 'w:', 'X:', 'Z:'] \
        or line=='\n' \
        or (line.startswith('%') and not line.startswith('%%score')):
            continue
        else:
            if "%" in line and not line.startswith('%%score'):
                line = "%".join(line.split('%')[:-1])
                music += line[:-1] + '\n'
            else:
                music += line + '\n'
    return music

def load_music(filename):
    # Convert the file to ABC notation
    p = subprocess.Popen(
        f'python xml2abc_145/xml2abc.py -m 2 -c 6 -x "{filename}"',
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        shell=True
    )
    out, err = p.communicate()

    output = out.decode('utf-8').replace('\r', '')  # Capture standard output
    music = unidecode(output).split('\n')
    music = filter(music).strip()

    return music

def download_and_extract(url):
    print(f"Downloading {url}")

    # Send an HTTP GET request to the URL and get the response
    response = requests.get(url, stream=True)

    if response.status_code == 200:
        # Create a BytesIO object and write the HTTP response content into it
        zip_data = io.BytesIO()
        total_size = int(response.headers.get('content-length', 0))
        
        with tqdm(total=total_size, unit='B', unit_scale=True) as pbar:
            for data in response.iter_content(chunk_size=1024):
                pbar.update(len(data))
                zip_data.write(data)

        # Use the zipfile library to extract the file
        print("Extracting the zip file...")
        with zipfile.ZipFile(zip_data, "r") as zip_ref:
            zip_ref.extractall("")
        
        print("Done!")

    else:
        print("Failed to download the file. HTTP response code:", response.status_code)

# URL of the JSONL file
wikimt_url = "https://huggingface.co/datasets/sander-wood/wikimusictext/resolve/main/wikimusictext.jsonl"

# Local filename to save the downloaded file
local_filename = "wikimusictext.jsonl"

# Download the file and save it locally
response = requests.get(wikimt_url)
if response.status_code == 200:
    with open(local_filename, 'wb') as file:
        file.write(response.content)
    print(f"Downloaded '{local_filename}' successfully.")
else:
    print(f"Failed to download. Status code: {response.status_code}")

# Download the xml2abc.py script (special thanks to Wim Vree for creating this script)
download_and_extract("https://wim.vree.org/svgParse/xml2abc.py-145.zip")

# Download the Wikifonia dataset
wikifonia_url = input("Enter the Wikifonia URL: ")
download_and_extract(wikifonia_url)

wikimusictext = []
with open("wikimusictext.jsonl", "r", encoding="utf-8") as f:
    for line in f.readlines():
        wikimusictext.append(json.loads(line))

updated_wikimusictext = []

for song in tqdm(wikimusictext):
    filename = song["artist"] + " - " + song["title"] + ".mxl"
    filepath = os.path.join("Wikifonia", filename)
    song["music"] = load_music(filepath)
    updated_wikimusictext.append(song)

with open("wikimusictext.jsonl", "w", encoding="utf-8") as f:
    for song in updated_wikimusictext:
        f.write(json.dumps(song, ensure_ascii=False)+"\n")
```

By following these steps and running the provided code, you can efficiently access ABC notation music scores from the WikiMT dataset. Just ensure you have the metadata, the `xml2abc.py` script, and the correct download link before starting. Enjoy your musical journey!


## Copyright Disclaimer
WikiMT was curated from publicly available sources, and all rights to the original content and data remain with their respective copyright holders. The dataset is made available for research and educational purposes, and any use, distribution, or modification of the dataset should comply with the terms and conditions set forth by the original data providers.

## BibTeX entry and citation info
```
@misc{wu2023clamp,
      title={CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval}, 
      author={Shangda Wu and Dingyao Yu and Xu Tan and Maosong Sun},
      year={2023},
      eprint={2304.11029},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}
```
task_categories:text-classification task_categories:text2text-generation size_categories:1K<n<10K language:en license:mit music arxiv:2304.11029 region:us
{"language": ["en"], "license": "mit", "size_categories": ["1K<n<10K"], "task_categories": ["text-classification", "text2text-generation"], "pretty_name": "wikimt", "tags": ["music"]}
TAGS
#task_categories-text-classification #task_categories-text2text-generation #size_categories-1K<n<10K #language-English #license-mit #music #arxiv-2304.11029 #region-us 


## Dataset Summary

In CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval, we introduce WikiMusicText (WikiMT), a new dataset for the evaluation of semantic search and music classification. It includes 1010 lead sheets in ABC notation sourced from URL, each accompanied by a title, artist, genre, and description. The title and artist information is extracted from the score, whereas the genre labels are obtained by matching keywords from the Wikipedia entries and assigned to one of the 8 classes (Jazz, Country, Folk, R&B, Pop, Rock, Dance, and Latin) that loosely mimic the GTZAN genres. The description is obtained by utilizing BART-large to summarize and clean the corresponding Wikipedia entry. Additionally, the natural language information within the ABC notation is removed.

WikiMT is a unique resource to support the evaluation of semantic search and music classification. However, it is important to acknowledge that the dataset was curated from publicly available sources, and there may be limitations concerning the accuracy and completeness of the genre and description information. Further research is needed to explore the potential biases and limitations of the dataset and to develop strategies to address them.

## How to Access Music Score Metadata for ABC Notation

To access metadata related to ABC notation music scores from the WikiMT dataset, follow these steps:

1. Locate the Wikifonia MusicXML Data Link: Start by visiting the discussion thread on the forum to find the download link for the Wikifonia dataset in MusicXML format (with a .mxl extension). You can find the discussion here: Download for Wikifonia all 6,675 Lead Sheets.

2. Run the Provided Code: Once you have found the Wikifonia MusicXML data link, execute the provided Python code below. This code will handle the following tasks:
   - Automatically download the "URL" dataset, which contains metadata associated with music scores.
   - Automatically download the "URL" conversion script, with special thanks to the author, Willem (Wim).
   - Prompt you for the Wikifonia data URL, as follows:

   

   Paste the URL pointing to the URL file and press Enter.

The below code will take care of downloading, processing, and extracting the music score metadata, making it ready for your research or applications.



By following these steps and running the provided code, you can efficiently access ABC notation music scores from the WikiMT dataset. Just ensure you have the metadata, the 'URL' script, and the correct download link before starting. Enjoy your musical journey!


## Copyright Disclaimer
WikiMT was curated from publicly available sources, and all rights to the original content and data remain with their respective copyright holders. The dataset is made available for research and educational purposes, and any use, distribution, or modification of the dataset should comply with the terms and conditions set forth by the original data providers.

## BibTeX entry and citation info

## Dataset Summary

In CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval, we introduce WikiMusicText (WikiMT), a new dataset for the evaluation of semantic search and music classification. It includes 1010 lead sheets in ABC notation sourced from URL, each accompanied by a title, artist, genre, and description. The title and artist information is extracted from the score, whereas the genre labels are obtained by matching keywords from the Wikipedia entries and assigned to one of the 8 classes (Jazz, Country, Folk, R&B, Pop, Rock, Dance, and Latin) that loosely mimic the GTZAN genres. The description is obtained by utilizing BART-large to summarize and clean the corresponding Wikipedia entry. Additionally, the natural language information within the ABC notation is removed.

WikiMT is a unique resource to support the evaluation of semantic search and music classification. However, it is important to acknowledge that the dataset was curated from publicly available sources, and there may be limitations concerning the accuracy and completeness of the genre and description information. Further research is needed to explore the potential biases and limitations of the dataset and to develop strategies to address them. ## How to Access Music Score Metadata for ABC Notation

To access metadata related to ABC notation music scores from the WikiMT dataset, follow these steps:

1. Locate the Wikifonia MusicXML Data Link: Start by visiting the discussion thread on the forum to find the download link for the Wikifonia dataset in MusicXML format (with a .mxl extension). You can find the discussion here: Download for Wikifonia all 6,675 Lead Sheets.

2. Run the Provided Code: Once you have found the Wikifonia MusicXML data link, execute the provided Python code below. This code will handle the following tasks:
   - Automatically download the "URL" dataset, which contains metadata associated with music scores.
   - Automatically download the "URL" conversion script, with special thanks to the author, Willem (Wim).
   - Prompt you for the Wikifonia data URL, as follows:

   

   Paste the URL pointing to the URL file and press Enter.

The below code will take care of downloading, processing, and extracting the music score metadata, making it ready for your research or applications.



By following these steps and running the provided code, you can efficiently access ABC notation music scores from the WikiMT dataset. Just ensure you have the metadata, the 'URL' script, and the correct download link before starting. Enjoy your musical journey! ## Copyright Disclaimer
WikiMT was curated from publicly available sources, and all rights to the original content and data remain with their respective copyright holders. The dataset is made available for research and educational purposes, and any use, distribution, or modification of the dataset should comply with the terms and conditions set forth by the original data providers. ## BibTeX entry and citation info
TAGS
#task_categories-text-classification #task_categories-text2text-generation #size_categories-1K<n<10K #language-English #license-mit #music #arxiv-2304.11029 #region-us 
 ## Dataset Summary

In CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval, we introduce WikiMusicText (WikiMT), a new dataset for the evaluation of semantic search and music classification. It includes 1010 lead sheets in ABC notation sourced from URL, each accompanied by a title, artist, genre, and description. The title and artist information is extracted from the score, whereas the genre labels are obtained by matching keywords from the Wikipedia entries and assigned to one of the 8 classes (Jazz, Country, Folk, R&B, Pop, Rock, Dance, and Latin) that loosely mimic the GTZAN genres. The description is obtained by utilizing BART-large to summarize and clean the corresponding Wikipedia entry. Additionally, the natural language information within the ABC notation is removed.

WikiMT is a unique resource to support the evaluation of semantic search and music classification. However, it is important to acknowledge that the dataset was curated from publicly available sources, and there may be limitations concerning the accuracy and completeness of the genre and description information. Further research is needed to explore the potential biases and limitations of the dataset and to develop strategies to address them. ## How to Access Music Score Metadata for ABC Notation

To access metadata related to ABC notation music scores from the WikiMT dataset, follow these steps:

1. Locate the Wikifonia MusicXML Data Link: Start by visiting the discussion thread on the forum to find the download link for the Wikifonia dataset in MusicXML format (with a .mxl extension). You can find the discussion here: Download for Wikifonia all 6,675 Lead Sheets.

2. Run the Provided Code: Once you have found the Wikifonia MusicXML data link, execute the provided Python code below. This code will handle the following tasks:
   - Automatically download the "URL" dataset, which contains metadata associated with music scores.
   - Automatically download the "URL" conversion script, with special thanks to the author, Willem (Wim).
   - Prompt you for the Wikifonia data URL, as follows:

   

   Paste the URL pointing to the URL file and press Enter.

The below code will take care of downloading, processing, and extracting the music score metadata, making it ready for your research or applications.



By following these steps and running the provided code, you can efficiently access ABC notation music scores from the WikiMT dataset. Just ensure you have the metadata, the 'URL' script, and the correct download link before starting. Enjoy your musical journey! ## Copyright Disclaimer
WikiMT was curated from publicly available sources, and all rights to the original content and data remain with their respective copyright holders. The dataset is made available for research and educational purposes, and any use, distribution, or modification of the dataset should comply with the terms and conditions set forth by the original data providers. ## BibTeX entry and citation info