region:us
---
dataset_info:
  dataset_size: 5061937.0
  download_size: 1510086
  features:
  - name: input_ids
    sequence: int32
  - name: attention_mask
    sequence: int8
  - name: labels
    sequence: int64
  - dtype: string
    name: preview
  splits:
  - name: train
    num_bytes: 3405190.0
    num_examples: 500
  - name: test
    num_bytes: 1656747.0
    num_examples: 140

---
# zh-tw-llm-dev-sample-ta8k-d40d11-only_embeddings-tr_wiki_sg_alp-c6795a-c2048

This dataset is a part of the `zh-tw-llm-dev` project.

* Tokenizer: `zh-tw-llm-dev-tokenizer-a8k-d40d11`
* Built with: `translations`, `wikipedia`, `sharegpt`, `alpaca`
* Rows: `train` `500`, `test` `140`
* Max length: `2048`
* Full config:
  ```json
  {"build_with": ["translations", "wikipedia", "sharegpt", "alpaca"], "preview_length": 256, "translations_settings": {"source_dataset": "zetavg/coct-en-zh-tw-translations-twp-300k", "lang_1_key": "en", "lang_2_key": "ch", "templates": ["English: {lang_1}\nChinese: {lang_2}", "Chinese: {lang_2}\nEnglish: {lang_1}"], "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}, "wikipedia_settings": {"source_dataset": "zetavg/zh-tw-wikipedia", "exclude": [{"content_length_longer_than": 512}, {"match": "小行星", "in": "markdown", "in_range": [0, 40]}, {"match": "是中華人民共和國", "in": "markdown", "in_range": [0, 80]}], "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}, "sharegpt_settings": {"source_dataset": "zetavg/ShareGPT-Processed", "train_on_inputs": false, "languages": [{"en": 0.4}, "zh_Hant"], "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}, "alpaca_settings": {"source_dataset": "zetavg/traditional-chinese-alpaca-en-align", "template": "short", "train_on_inputs": false, "rows_limit": 100, "test_size": 0.1, "test_split_seed": 42, "test_rows_limit": 10}}
  ```