This dataset is HuggingFaceH4/asss unfiltered and deduped, removing 29 instances of blatant alignment and 21 duplicates.
398 instructions remain.

clean.py was first ran on https://huggingface.co/datasets/HuggingFaceH4/asss/blob/122c06526f06a25c54614c1f5c63b4c3f51e96fb/surge-scale-si-aws-dataset.jsonl and then dedupe.py was ran on it. renamed to .json not .jsonl

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py
region:us
{}
TAGS
#region-us 

This dataset is HuggingFaceH4/asss unfiltered and deduped, removing 29 instances of blatant alignment and 21 duplicates.
398 instructions remain.

URL was first ran on URL and then URL was ran on it. renamed to .json not .jsonl

inspired by URL

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to URL
TAGS
#region-us 

passage: TAGS
#region-us 
