This dataset is the Open Assistant dataset https://huggingface.co/datasets/OpenAssistant/oasst1, and formatted to be easier to convert to whatever finetune data format you want, removes 75 instances of alignment and 81 dupes.

oasst_clean_format_dedupe.py was first ran on 2023-04-12_oasst_all.trees.jsonl from OpenAssistant/oasst1

Inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

Credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to oasst_clean_data_format.py

I converted to trees so each object in the output has a messages list, each message has a role and content. Each starting prompt in the oasst dataset will appear in the output n times where n is the number of replies the prompt had (unless it had a duplicate).

so if the oasst tree looked like

```
user: aaa
    assistant: bbb
       user: ccc
            assistant: ddd
    assistant: eee
        user: fff
    assistant: ggg
        user: hhh
            assistant: iii
```

then the output would look like

```
{
    messages: [
        { "role": "user", "content": "aaa" },
        { "role": "model", "content": "bbb" },
        { "role": "user", "content": "ccc" },
        { "role": "model", "content": "ddd" }
    ]
}
{
    messages: [
        { "role": "user", "content": "aaa" },
        { "role": "model", "content": "eee" }
        // if tree ends with a user msg then it is not included in the output, this can be changed by passing --save-user-ends to the script
    ]
}
{
    messages: [
        { "role": "user", "content": "aaa" },
        { "role": "model", "content": "ggg" },
        { "role": "user", "content": "hhh" },
        { "role": "model", "content": "iii" }
    ]
}
```
region:us
{}
TAGS
#region-us 

This dataset is the Open Assistant dataset URL and formatted to be easier to convert to whatever finetune data format you want, removes 75 instances of alignment and 81 dupes.

oasst_clean_format_dedupe.py was first ran on 2023-04-12_oasst_all.URL from OpenAssistant/oasst1

Inspired by URL

Credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to oasst_clean_data_format.py

I converted to trees so each object in the output has a messages list, each message has a role and content. Each starting prompt in the oasst dataset will appear in the output n times where n is the number of replies the prompt had (unless it had a duplicate).

so if the oasst tree looked like



then the output would look like


TAGS
#region-us 
