# Dataset Card for "oasst1_dense_flat"



[OASST1 dataset](https://huggingface.co/datasets/OpenAssistant/oasst1)
But where with retrieved parent_text, and where we only keep messages with dense annotations (all labels have 2 annotators)

```python
from datasets import Dataset, DatasetDict

d={}
for split in ['train','validation']:
    df=load_dataset("OpenAssistant/oasst1")[split].to_pandas()
    m2t=df.set_index("message_id")['text'].to_dict()
    df['parent_text']=df.parent_id.map(lambda x: m2t.get(x,''))

    df=df[df.labels.map(lambda x:x!=None)]
    df=df[df.labels.map(lambda x:x['count'].min()>2)]

    labels=df.labels.map(lambda x:list(x['name'])).value_counts().index[0]
    df=df[df.labels.map(lambda x:x!=None)]
    df=df[df.labels.map(lambda x:list(x['name'])==labels)]
    for label in labels:
        df[label]=df.labels.map(lambda x: x['value'][list(x['name']).index(label)])
    d[split]=Dataset.from_pandas(df,preserve_index=False)
    
DatasetDict(d).push_to_hub('oasst1_dense_flat')
```
https://github.com/LAION-AI/Open-Assistant

```
@article{kopf2023openassistant,
  title={OpenAssistant Conversations--Democratizing Large Language Model Alignment},
  author={K{\"o}pf, Andreas and Kilcher, Yannic and von R{\"u}tte, Dimitri and Anagnostidis, Sotiris and Tam, Zhi-Rui and Stevens, Keith and Barhoum, Abdullah and Duc, Nguyen Minh and Stanley, Oliver and Nagyfi, Rich{\'a}rd and others},
  journal={arXiv preprint arXiv:2304.07327},
  year={2023}
}
```
license:apache-2.0 region:us
{"license": "apache-2.0", "dataset_info": {"features": [{"name": "message_id", "dtype": "string"}, {"name": "parent_id", "dtype": "string"}, {"name": "user_id", "dtype": "string"}, {"name": "created_date", "dtype": "string"}, {"name": "text", "dtype": "string"}, {"name": "role", "dtype": "string"}, {"name": "lang", "dtype": "string"}, {"name": "review_count", "dtype": "int32"}, {"name": "review_result", "dtype": "bool"}, {"name": "deleted", "dtype": "bool"}, {"name": "rank", "dtype": "float64"}, {"name": "synthetic", "dtype": "bool"}, {"name": "model_name", "dtype": "null"}, {"name": "detoxify", "struct": [{"name": "identity_attack", "dtype": "float64"}, {"name": "insult", "dtype": "float64"}, {"name": "obscene", "dtype": "float64"}, {"name": "severe_toxicity", "dtype": "float64"}, {"name": "sexual_explicit", "dtype": "float64"}, {"name": "threat", "dtype": "float64"}, {"name": "toxicity", "dtype": "float64"}]}, {"name": "message_tree_id", "dtype": "string"}, {"name": "tree_state", "dtype": "string"}, {"name": "emojis", "struct": [{"name": "count", "sequence": "int32"}, {"name": "name", "sequence": "string"}]}, {"name": "labels", "struct": [{"name": "count", "sequence": "int32"}, {"name": "name", "sequence": "string"}, {"name": "value", "sequence": "float64"}]}, {"name": "parent_text", "dtype": "string"}, {"name": "spam", "dtype": "float64"}, {"name": "fails_task", "dtype": "float64"}, {"name": "lang_mismatch", "dtype": "float64"}, {"name": "pii", "dtype": "float64"}, {"name": "not_appropriate", "dtype": "float64"}, {"name": "hate_speech", "dtype": "float64"}, {"name": "sexual_content", "dtype": "float64"}, {"name": "quality", "dtype": "float64"}, {"name": "toxicity", "dtype": "float64"}, {"name": "humor", "dtype": "float64"}, {"name": "helpfulness", "dtype": "float64"}, {"name": "creativity", "dtype": "float64"}, {"name": "violence", "dtype": "float64"}], "splits": [{"name": "train", "num_bytes": 59657796, "num_examples": 34059}, {"name": "validation", "num_bytes": 3164029, "num_examples": 1816}], "download_size": 25173939, "dataset_size": 62821825}}
TAGS
#license-apache-2.0 #region-us 

# Dataset Card for "oasst1_dense_flat"



OASST1 dataset
But where with retrieved parent_text, and where we only keep messages with dense annotations (all labels have 2 annotators)


URL


# Dataset Card for "oasst1_dense_flat"



OASST1 dataset
But where with retrieved parent_text, and where we only keep messages with dense annotations (all labels have 2 annotators)


URL
TAGS
#license-apache-2.0 #region-us 
 # Dataset Card for "oasst1_dense_flat"



OASST1 dataset
But where with retrieved parent_text, and where we only keep messages with dense annotations (all labels have 2 annotators)


URL