license:cc-by-4.0 region:us
---
license: cc-by-4.0
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
dataset_info:
  features:
  - name: ppl
    dtype: float64
  - name: len
    dtype: int64
  - name: len_group
    dtype: int64
  - name: cluster
    dtype: int64
  - name: group
    dtype: string
  - name: instruction
    dtype: string
  - name: output
    dtype: string
  splits:
  - name: train
    num_bytes: 3762418
    num_examples: 4006
  download_size: 2059932
  dataset_size: 3762418
---
# Dataset Card for "KoQuality"


## Dataset Summary
Korean instruction dataset with only 1.04% selected from several insturction datasets for an autoregressive language model.
The final 4,006 sentences were selected out of 384,067 data from the integrated datasets.


## Dataset Description
- repository: [nayohan/HCLT](https://github.com/nayohan/KoQuality)
- huggingface: [DILAB-HYU/KoQuality](https://huggingface.co/datasets/DILAB-HYU/KoQuality)



### Selection method
1. Grouping according to length of instruction (5 length group)

2. Clustering dataset with KoSimCSE model embedding  <br>
Clustering datasets with 'instruction length' and 'K-means Clustering' using  *KoAlpaca, KULLM*, and *OIG*. (384,067 instructions)

3. PPL sampling <br>
Select 1% of data based on PPL of each cluster. (4006 instructions)



### Used datasets
| Dataset | count | links |
| --- | --- | --- |
| KoAlpaca-v1.1a | 21155 | [https://huggingface.co/datasets/beomi/KoAlpaca-v1.1a](https://huggingface.co/datasets/beomi/KoAlpaca-v1.1a)
| kullm-v2 | 152630 | [https://huggingface.co/datasets/nlpai-lab/kullm-v2](https://huggingface.co/datasets/nlpai-lab/kullm-v2)
| OIG-small-chip2-ko | 210282  | [https://huggingface.co/datasets/heegyu/OIG-small-chip2-ko](https://huggingface.co/datasets/heegyu/OIG-small-chip2-ko)


### How to use
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("DILAB-HYU/KoQuality", split="train")
>>> ds
Dataset({
    features: ['instruction', 'output'],
    num_rows: 4006
})
```
<br>


- Instruction Tuned Polyglot-ko Model: [DILAB-HYU/KoQuality-Polyglot-5.8b](https://huggingface.co/DILAB-HYU/KoQuality-Polyglot-5.8b)
- More details can be found here: [github.com/nayohan/KoQuality](https://github.com/nayohan/KoQuality)