
This dataset is the LLaMini from MBZUAI/LaMini-instruction, removing instances of blatant alignment and removes duplicates.
2290278 instructions remain.

i merged the parquets from original repo with parquet2json then ran clean_format_dedupe.py on the resulting jsonl

credit to ehardford for his contains_unwanted_words function

size_categories:1M<n<10M region:us
{"size_categories": ["1M<n<10M"], "pretty_name": "LaMini Instruct Unfiltered Deduped"}
TAGS
#size_categories-1M<n<10M #region-us 


This dataset is the LLaMini from MBZUAI/LaMini-instruction, removing instances of blatant alignment and removes duplicates.
2290278 instructions remain.

i merged the parquets from original repo with parquet2json then ran clean_format_dedupe.py on the resulting jsonl

credit to ehardford for his contains_unwanted_words function

TAGS
#size_categories-1M<n<10M #region-us 
