This dataset is https://github.com/teknium1/GPTeacher unfiltered, removing 1489 instances of blatant alignment.
23073 instructions remain.

https://github.com/teknium1/GPTeacher/blob/8afcaaa7a11dd980162d861bd6be970f95eb7174/Codegen/codegen-instruct.json

https://github.com/teknium1/GPTeacher/blob/e3b7aba886c6c0c8ad30a650edfa7a3093fbf57c/Instruct/gpt4-instruct-dedupe-only-dataset.json

https://github.com/teknium1/GPTeacher/blob/5b040645528a38bfa81a258e7646f8c92ad7d0dd/Roleplay/roleplay-simple-deduped-roleplay-instruct.json

i combined all of these files above into gpteacher.json and ran clean.py

normal dedupe.py script didn't find any dupes here.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py
region:us
{}
TAGS
#region-us 

This dataset is URL unfiltered, removing 1489 instances of blatant alignment.
23073 instructions remain.

URL

URL

URL

i combined all of these files above into URL and ran URL

normal URL script didn't find any dupes here.

inspired by URL

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to URL
TAGS
#region-us 

passage: TAGS
#region-us 
