region:us
---
configs:
- config_name: default
  data_files:
  - split: validation
    path: data/validation-*
  - split: dev
    path: data/dev-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: id
    dtype: string
  - name: answer
    dtype: int64
  - name: question
    dtype: string
  - name: choices
    sequence: string
  splits:
  - name: validation
    num_bytes: 890402
    num_examples: 1456
  - name: dev
    num_bytes: 140819
    num_examples: 271
  - name: test
    num_bytes: 7615124
    num_examples: 13062
  download_size: 4415183
  dataset_size: 8646345
---
References: https://huggingface.co/datasets/cais/mmlu

# MMLU (Vietnamese translation version)

## Install

To install `lm-eval` from the github repository main branch, run:

```bash
git clone https://github.com/hieunguyen1053/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
```

## Basic Usage

> **Note**: When reporting results from eval harness, please include the task versions (shown in `results["versions"]`) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the [Task Versioning](#task-versioning) section for more info.

### Hugging Face `transformers`

To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. vlsp-2023-vllm/hoa-1b4) on `mmlu` you can use the following command:


```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=vlsp-2023-vllm/hoa-1b4 \
    --tasks mmlu_vi \
    --device cuda:0
```

Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

```bash
python main.py \
    --model hf-causal \
    --model_args pretrained=vlsp-2023-vllm/hoa-1b4,revision=step100000,dtype="float" \
    --tasks mmlu_vi \
    --device cuda:0
```

To evaluate models that are loaded via `AutoSeq2SeqLM` in Huggingface, you instead use `hf-seq2seq`. *To evaluate (causal) models across multiple GPUs, use `--model hf-causal-experimental`*

> **Warning**: Choosing the wrong model may result in erroneous outputs despite not erroring.