# ShareGPT-Processed

The [RyokoAI/ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) dataset, converted to Markdown and labeled with the language used.

## Acknowledgements

* [vinta/pangu.js](https://github.com/vinta/pangu.js) — To insert whitespace between CJK (Chinese, Japanese, Korean) and half-width characters (alphabetical letters, numerical digits and symbols).
* [matthewwithanm/python-markdownify](https://github.com/matthewwithanm/python-markdownify) — Provides a starting point to convert HTML to Markdown.
* [BYVoid/OpenCC](https://github.com/BYVoid/OpenCC) — Conversions between Traditional Chinese and Simplified Chinese.
* [aboSamoor/polyglot](https://github.com/aboSamoor/polyglot) — For language detection.
* [RyokoAI/ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K) — The data source.

## Data Structure

Every row in this dataset is a conversation session, consisting of:

* `id` — An ID.
* `lang` — The language code of the conversation, detected with `polyglot`.
* `conversations` — A list of messages, with each message consisting of:
	* `from` — `human` or `gpt`.
	* `value` — The original message content. HTML.
	* `markdown` — The message converted into Markdown format.
	* `opencc_converted_markdown` — The Markdown-format message converted into Traditional Chinese using `opencc`. Will be `null` for non-Chinese conversations.

Example:

```json
{
  "id": "AZirHBD",
  "conversations": [
    {
      "from": "human",
      "markdown": "請問英文過去完成式的用法？",
      "opencc_converted_markdown": "請問英文過去完成式的用法？",
      "value": "請問英文過去完成式的用法？"
    },
    {
      "from": "gpt",
      "markdown": "英文的過去完成式是用來表示一個已經發生的動作或事件，發生在過去的某個時間，並且在這個時間之前，也發生了另一個動作或事件。在語法上，過去完成式是由助動詞 \"had\" 和過去分詞形式的動詞組成。下面是一些常見的用法：\n\n1. 表示一個已經完成的動作在另一個過去的時間點之前發生。\n\n  * e.g. By the time I arrived, she had already left.\n2. 表示在過去某個時間之前就已經完成的動作。\n\n  * e.g. I had studied English for five years before I moved to the United States.\n3. 表示兩個已經完成的動作中，其中一個發生在另一個之前。\n\n  * e.g. After I had finished my homework, I watched TV.\n4. 用來描述一個條件或假設在過去的時間點之前已經發生。\n\n  * e.g. If she had known about the party, she would have come.\n5. 用來描述過去已經完成的行動對現在的影響。\n\n  * e.g. He had worked as a lawyer for many years, so he knew a lot about the law.\n\n在使用過去完成式時，需要注意時態的一致性和上下文的清晰性，以確保表達的意思清晰明確。",
      "opencc_converted_markdown": "英文的過去完成式是用來表示一個已經發生的動作或事件，發生在過去的某個時間，並且在這個時間之前，也發生了另一個動作或事件。在語法上，過去完成式是由助動詞 \"had\" 和過去分詞形式的動片語成。下面是一些常見的用法：\n\n1. 表示一個已經完成的動作在另一個過去的時間點之前發生。\n\n  * e.g. By the time I arrived, she had already left.\n2. 表示在過去某個時間之前就已經完成的動作。\n\n  * e.g. I had studied English for five years before I moved to the United States.\n3. 表示兩個已經完成的動作中，其中一個發生在另一個之前。\n\n  * e.g. After I had finished my homework, I watched TV.\n4. 用來描述一個條件或假設在過去的時間點之前已經發生。\n\n  * e.g. If she had known about the party, she would have come.\n5. 用來描述過去已經完成的行動對現在的影響。\n\n  * e.g. He had worked as a lawyer for many years, so he knew a lot about the law.\n\n在使用過去完成式時，需要注意時態的一致性和上下文的清晰性，以確保表達的意思清晰明確。",
      "value": "<div class=\"markdown prose w-full break-words dark:prose-invert light\"><p>英文的過去完成式是用來表示一個已經發生的動作或事件，發生在過去的某個時間，並且在這個時間之前，也發生了另一個動作或事件。在語法上，過去完成式是由助動詞 \"had\" 和過去分詞形式的動詞組成。下面是一些常見的用法：</p><ol><li><p>表示一個已經完成的動作在另一個過去的時間點之前發生。</p><ul><li>e.g. By the time I arrived, she had already left.</li></ul></li><li><p>表示在過去某個時間之前就已經完成的動作。</p><ul><li>e.g. I had studied English for five years before I moved to the United States.</li></ul></li><li><p>表示兩個已經完成的動作中，其中一個發生在另一個之前。</p><ul><li>e.g. After I had finished my homework, I watched TV.</li></ul></li><li><p>用來描述一個條件或假設在過去的時間點之前已經發生。</p><ul><li>e.g. If she had known about the party, she would have come.</li></ul></li><li><p>用來描述過去已經完成的行動對現在的影響。</p><ul><li>e.g. He had worked as a lawyer for many years, so he knew a lot about the law.</li></ul></li></ol><p>在使用過去完成式時，需要注意時態的一致性和上下文的清晰性，以確保表達的意思清晰明確。</p></div>"
    },
    {
      "from": "human",
      "markdown": "英文現在式的用法",
      "opencc_converted_markdown": "英文現在式的用法",
      "value": "英文現在式的用法"
    },
    {
      "from": "gpt",
      "markdown": "英文的現在式表示的是現在正在進行或習慣性發生的動作，也可以表示客觀事實或普遍真理。下面是一些常見的用法：\n\n1. 表示現在正在進行的動作。\n\n  * e.g. I am typing on my computer right now.\n2. 表示習慣性的行為或狀態。\n\n  * e.g. I usually drink coffee in the morning.\n3. 表示事實、真相或存在的狀態。\n\n  * e.g. The earth orbits around the sun.\n4. 用於引述、註解或解釋現在發生的事情。\n\n  * e.g. He says he is feeling better today.\n5. 表示未來已經計劃好的事件或行動。\n\n  * e.g. I am leaving for Paris tomorrow.\n\n在使用現在式時，需要注意動詞的時態和主語的人稱和數，以確保表達的意思清晰明確。",
      "opencc_converted_markdown": "英文的現在式表示的是現在正在進行或習慣性發生的動作，也可以表示客觀事實或普遍真理。下面是一些常見的用法：\n\n1. 表示現在正在進行的動作。\n\n  * e.g. I am typing on my computer right now.\n2. 表示習慣性的行為或狀態。\n\n  * e.g. I usually drink coffee in the morning.\n3. 表示事實、真相或存在的狀態。\n\n  * e.g. The earth orbits around the sun.\n4. 用於引述、註解或解釋現在發生的事情。\n\n  * e.g. He says he is feeling better today.\n5. 表示未來已經計劃好的事件或行動。\n\n  * e.g. I am leaving for Paris tomorrow.\n\n在使用現在式時，需要注意動詞的時態和主語的人稱和數，以確保表達的意思清晰明確。",
      "value": "<div class=\"markdown prose w-full break-words dark:prose-invert light\"><p>英文的現在式表示的是現在正在進行或習慣性發生的動作，也可以表示客觀事實或普遍真理。下面是一些常見的用法：</p><ol><li><p>表示現在正在進行的動作。</p><ul><li>e.g. I am typing on my computer right now.</li></ul></li><li><p>表示習慣性的行為或狀態。</p><ul><li>e.g. I usually drink coffee in the morning.</li></ul></li><li><p>表示事實、真相或存在的狀態。</p><ul><li>e.g. The earth orbits around the sun.</li></ul></li><li><p>用於引述、註解或解釋現在發生的事情。</p><ul><li>e.g. He says he is feeling better today.</li></ul></li><li><p>表示未來已經計劃好的事件或行動。</p><ul><li>e.g. I am leaving for Paris tomorrow.</li></ul></li></ol><p>在使用現在式時，需要注意動詞的時態和主語的人稱和數，以確保表達的意思清晰明確。</p></div>"
    }
  ],
  "lang": "zh_Hant"
}
```

## Languages

```json
{
  "en": 63940,
  "zh_Hant": 3201,
  "zh": 6394,
  "es": 2080,
  "ja": 1525,
  "unknown": 4212,
  "pt": 778,
  "it": 512,
  "ko": 2529,
  "nl": 195,
  "ro": 63,
  "fr": 1835,
  "vi": 245,
  "de": 800,
  "cs": 172,
  "iw": 123,
  "ru": 448,
  "id": 369,
  "pl": 194,
  "no": 65,
  "ar": 78,
  "tr": 150,
  "da": 68,
  "sa": 10,
  "sv": 87,
  "ia": 2,
  "fo": 7,
  "sq": 4,
  "el": 14,
  "fa": 22,
  "bs": 13,
  "rm": 2,
  "ms": 45,
  "ca": 14,
  "hr": 26,
  "sk": 23,
  "uk": 106,
  "th": 33,
  "fi": 32,
  "tlh": 1,
  "hu": 46,
  "gl": 21,
  "bg": 4,
  "sr": 18,
  "is": 2,
  "ts": 9,
  "la": 27,
  "sl": 6,
  "uz": 5,
  "qu": 3,
  "ay": 7,
  "mi": 3,
  "ceb": 1,
  "gu": 1,
  "oc": 8,
  "aa": 2,
  "haw": 6,
  "xh": 4,
  "ny": 4,
  "hmn": 2,
  "tk": 2,
  "sco": 6,
  "zzp": 4,
  "so": 2,
  "mg": 1,
  "to": 1,
  "mk": 1,
  "ha": 2,
  "ur": 2,
  "nn": 4,
  "lv": 2,
  "mt": 2,
  "gn": 2,
  "et": 2,
  "ie": 3,
  "tl": 3,
  "lb": 4,
  "bn": 1,
  "rw": 4,
  "bi": 1,
  "ga": 1,
  "war": 1,
  "sw": 2,
  "eo": 2,
  "eu": 2,
  "lt": 1,
  "af": 2,
  "ht": 3,
  "fj": 1,
  "st": 1,
  "na": 1,
  "sd": 1,
  "fy": 1,
  "jw": 1
}
```

Detected with `polyglot`.
task_categories:text-generation size_categories:10K<n<100K language:en language:zh language:es language:ja language:fr license:cc0-1.0 conversation rlhf chatgpt gpt-3.5 region:us
{"language": ["en", "zh", "es", "ja", "fr"], "license": "cc0-1.0", "size_categories": ["10K<n<100K"], "task_categories": ["text-generation"], "dataset_info": {"features": [{"name": "id", "dtype": "string"}, {"name": "conversations", "list": [{"name": "from", "dtype": "string"}, {"name": "markdown", "dtype": "string"}, {"name": "opencc_converted_markdown", "dtype": "string"}, {"name": "value", "dtype": "string"}]}, {"name": "lang", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 2772036550, "num_examples": 90665}], "download_size": 1075261393, "dataset_size": 2772036550}, "tags": ["conversation", "rlhf", "chatgpt", "gpt-3.5"]}
TAGS
#task_categories-text-generation #size_categories-10K<n<100K #language-English #language-Chinese #language-Spanish #language-Japanese #language-French #license-cc0-1.0 #conversation #rlhf #chatgpt #gpt-3.5 #region-us 

# ShareGPT-Processed

The RyokoAI/ShareGPT52K dataset, converted to Markdown and labeled with the language used.

## Acknowledgements

* vinta/URL — To insert whitespace between CJK (Chinese, Japanese, Korean) and half-width characters (alphabetical letters, numerical digits and symbols).
* matthewwithanm/python-markdownify — Provides a starting point to convert HTML to Markdown.
* BYVoid/OpenCC — Conversions between Traditional Chinese and Simplified Chinese.
* aboSamoor/polyglot — For language detection.
* RyokoAI/ShareGPT52K — The data source.

## Data Structure

Every row in this dataset is a conversation session, consisting of:

* 'id' — An ID.
* 'lang' — The language code of the conversation, detected with 'polyglot'.
* 'conversations' — A list of messages, with each message consisting of:
	* 'from' — 'human' or 'gpt'.
	* 'value' — The original message content. HTML.
	* 'markdown' — The message converted into Markdown format.
	* 'opencc_converted_markdown' — The Markdown-format message converted into Traditional Chinese using 'opencc'. Will be 'null' for non-Chinese conversations.

Example:



## Languages



Detected with 'polyglot'.
# ShareGPT-Processed

The RyokoAI/ShareGPT52K dataset, converted to Markdown and labeled with the language used. ## Acknowledgements

* vinta/URL — To insert whitespace between CJK (Chinese, Japanese, Korean) and half-width characters (alphabetical letters, numerical digits and symbols).
* matthewwithanm/python-markdownify — Provides a starting point to convert HTML to Markdown.
* BYVoid/OpenCC — Conversions between Traditional Chinese and Simplified Chinese.
* aboSamoor/polyglot — For language detection.
* RyokoAI/ShareGPT52K — The data source. ## Data Structure

Every row in this dataset is a conversation session, consisting of:

* 'id' — An ID.
* 'lang' — The language code of the conversation, detected with 'polyglot'.
* 'conversations' — A list of messages, with each message consisting of:
	* 'from' — 'human' or 'gpt'.
	* 'value' — The original message content. HTML.
	* 'markdown' — The message converted into Markdown format.
	* 'opencc_converted_markdown' — The Markdown-format message converted into Traditional Chinese using 'opencc'. Will be 'null' for non-Chinese conversations.

Example: ## Languages



Detected with 'polyglot'.
TAGS
#task_categories-text-generation #size_categories-10K<n<100K #language-English #language-Chinese #language-Spanish #language-Japanese #language-French #license-cc0-1.0 #conversation #rlhf #chatgpt #gpt-3.5 #region-us 
 # ShareGPT-Processed

The RyokoAI/ShareGPT52K dataset, converted to Markdown and labeled with the language used. ## Acknowledgements

* vinta/URL — To insert whitespace between CJK (Chinese, Japanese, Korean) and half-width characters (alphabetical letters, numerical digits and symbols).
* matthewwithanm/python-markdownify — Provides a starting point to convert HTML to Markdown.
* BYVoid/OpenCC — Conversions between Traditional Chinese and Simplified Chinese.
* aboSamoor/polyglot — For language detection.
* RyokoAI/ShareGPT52K — The data source. ## Data Structure

Every row in this dataset is a conversation session, consisting of:

* 'id' — An ID.
* 'lang' — The language code of the conversation, detected with 'polyglot'.
* 'conversations' — A list of messages, with each message consisting of:
	* 'from' — 'human' or 'gpt'.
	* 'value' — The original message content. HTML.
	* 'markdown' — The message converted into Markdown format.
	* 'opencc_converted_markdown' — The Markdown-format message converted into Traditional Chinese using 'opencc'. Will be 'null' for non-Chinese conversations.

Example: ## Languages



Detected with 'polyglot'.
passage: TAGS
#task_categories-text-generation #size_categories-10K<n<100K #language-English #language-Chinese #language-Spanish #language-Japanese #language-French #license-cc0-1.0 #conversation #rlhf #chatgpt #gpt-3.5 #region-us 
# ShareGPT-Processed

The RyokoAI/ShareGPT52K dataset, converted to Markdown and labeled with the language used.## Acknowledgements

* vinta/URL — To insert whitespace between CJK (Chinese, Japanese, Korean) and half-width characters (alphabetical letters, numerical digits and symbols).
* matthewwithanm/python-markdownify — Provides a starting point to convert HTML to Markdown.
* BYVoid/OpenCC — Conversions between Traditional Chinese and Simplified Chinese.
* aboSamoor/polyglot — For language detection.
* RyokoAI/ShareGPT52K — The data source.## Data Structure

Every row in this dataset is a conversation session, consisting of:

* 'id' — An ID.
* 'lang' — The language code of the conversation, detected with 'polyglot'.
* 'conversations' — A list of messages, with each message consisting of:
	* 'from' — 'human' or 'gpt'.
	* 'value' — The original message content. HTML.
	* 'markdown' — The message converted into Markdown format.
	* 'opencc_converted_markdown' — The Markdown-format message converted into Traditional Chinese using 'opencc'. Will be 'null' for non-Chinese conversations.

Example:## Languages



Detected with 'polyglot'.