
This dataset contains vi subsets (first 191 examples) and auto-translation from en to vi subsets (the rest, 38346 examples) from [OASST1](https://huggingface.co/datasets/OpenAssistant/oasst1). All auto-translation examples are generated using [VietAI envit5-translation](https://huggingface.co/VietAI/envit5-translation).
The vi subsets have the same features as the original dataset. Meanwhile, the auto-translation subsets introduce two new features:   
- `"text_chunks"` is a list that contains chunked text split from `"text"`, each chunk has no more than 300 tokens. The sent_tokenizer and word_tokenzier used are from spacy en_core_web_sm model.
- `"text_translation"` contains merged of all translated chunks. Due to the auto-translation model, all new-line symbols (`\n`) are removed.   
The translation script can be found at `translate_en_to_vi.py`
task_categories:conversational size_categories:10K<n<100K language:vi license:apache-2.0 region:us
{"language": ["vi"], "license": "apache-2.0", "size_categories": ["10K<n<100K"], "task_categories": ["conversational"], "dataset_info": {"features": [{"name": "message_id", "dtype": "string"}, {"name": "parent_id", "dtype": "string"}, {"name": "user_id", "dtype": "string"}, {"name": "created_date", "dtype": "string"}, {"name": "text", "dtype": "string"}, {"name": "role", "dtype": "string"}, {"name": "lang", "dtype": "string"}, {"name": "review_count", "dtype": "int32"}, {"name": "review_result", "dtype": "bool"}, {"name": "deleted", "dtype": "bool"}, {"name": "rank", "dtype": "int32"}, {"name": "synthetic", "dtype": "bool"}, {"name": "model_name", "dtype": "string"}, {"name": "detoxify", "struct": [{"name": "toxicity", "dtype": "float64"}, {"name": "severe_toxicity", "dtype": "float64"}, {"name": "obscene", "dtype": "float64"}, {"name": "identity_attack", "dtype": "float64"}, {"name": "insult", "dtype": "float64"}, {"name": "threat", "dtype": "float64"}, {"name": "sexual_explicit", "dtype": "float64"}]}, {"name": "message_tree_id", "dtype": "string"}, {"name": "tree_state", "dtype": "string"}, {"name": "emojis", "sequence": [{"name": "name", "dtype": "string"}, {"name": "count", "dtype": "int32"}]}, {"name": "labels", "sequence": [{"name": "name", "dtype": "string"}, {"name": "value", "dtype": "float64"}, {"name": "count", "dtype": "int32"}]}, {"name": "text_chunks", "sequence": "string"}, {"name": "text_translation", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 59922108.85834358, "num_examples": 38537}], "download_size": 39428167, "dataset_size": 59922108.85834358}}
TAGS
#task_categories-conversational #size_categories-10K<n<100K #language-Vietnamese #license-apache-2.0 #region-us 


This dataset contains vi subsets (first 191 examples) and auto-translation from en to vi subsets (the rest, 38346 examples) from OASST1. All auto-translation examples are generated using VietAI envit5-translation.
The vi subsets have the same features as the original dataset. Meanwhile, the auto-translation subsets introduce two new features:   
- '"text_chunks"' is a list that contains chunked text split from '"text"', each chunk has no more than 300 tokens. The sent_tokenizer and word_tokenzier used are from spacy en_core_web_sm model.
- '"text_translation"' contains merged of all translated chunks. Due to the auto-translation model, all new-line symbols ('\n') are removed.   
The translation script can be found at 'translate_en_to_vi.py'
TAGS
#task_categories-conversational #size_categories-10K<n<100K #language-Vietnamese #license-apache-2.0 #region-us 
