README.md

---

license: mit
---

---
license: mit
datasets:
- allenai/MADLAD-400
language:
- en
- ru
- bg
- uk
- kk
base_model:

- meta-llama/Llama-2-7b-hf

---

VocADT is a solution for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model’s weights fixed.

VocADT offers a flexible and scalable solution without requiring external resources or language constraints.





## New Vocabulary Adapted Models

Only the input/output embeddings are replaced, while all other original weights of base model remain fixed.

These are the merged version: after training the adapters, we merge the original embeddings with the adapter to generate the new embeddings.

| Name | Adapted Model | Base Model | New Vocab Size | Focused Languages |

|---|---|---|---|---|

| VocADT-Latin-Mistral | [h-j-han/Mistral-7B-VocADT-50k-Latin](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Latin) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Swahili (sw), Indonesian (id), Estonian (et), Haitian Creole (ht), English (en)|

| VocADT-Mixed-Mistral | [h-j-han/Mistral-7B-VocADT-50k-Mixed](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Mixed) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Korean (ko), Greek (el), Russian (ru), Bulgarian (bg), English (en) |

| VocADT-Cyrillic-Mistral | [h-j-han/Mistral-7B-VocADT-50k-Cyrillic](https://huggingface.co/h-j-han/Mistral-7B-VocADT-50k-Cyrillic) | [Mistral](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 50k | Russian (ru), Bulgarian (bg), Ukrainian (uk), Kazakh (kk), English (en) |

|||||

| VocADT-Latin-LLama | [h-j-han/Llama2-7B-VocADT-50k-Latin](https://huggingface.co/h-j-han/Llama2-7B-VocADT-50k-Latin) | [Llama](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 50k | Swahili (sw), Indonesian (id), Estonian (et), Haitian Creole (ht), English (en)|

| VocADT-Mixed-LLama | [h-j-han/Llama2-7B-VocADT-50k-Mixed](https://huggingface.co/h-j-han/Llama2-7B-VocADT-50k-Mixed) | [Llama](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 50k | Korean (ko), Greek (el), Russian (ru), Bulgarian (bg), English (en) |

| VocADT-Cyrillic-LLama | [h-j-han/Llama2-7B-VocADT-50k-Cyrillic](https://huggingface.co/h-j-han/Llama2-7B-VocADT-50k-Cyrillic) | [Llama](https://huggingface.co/meta-llama/Llama-2-7b-hf) | 50k | Russian (ru), Bulgarian (bg), Ukrainian (uk), Kazakh (kk), English (en) |





## Quick Start

```python

from transformers import AutoModelForCausalLM, AutoTokenizer



# model_name = "meta-llama/Llama-2-7b-hf" # Base Model
model_name = "h-j-han/Llama2-7B-VocADT-50k-Cyrillic" # Vocabulary Adapted Model

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

prefix = "\nEnglish: Hello!\nUkrainian: Добрий день!\nEnglish: How are you?\nUkrainian: Як справи?\nEnglish: "
line = "Do you speak English?"
suffix = f"\nUkrainian:"
prompt = prefix + line + suffix

inputs = tokenizer(prompt, return_tensors="pt")

for item in inputs:

    inputs[item] = inputs[item].cuda()

outputs = model.generate(**inputs, max_new_tokens=6)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

```



## Reference

We provide code in Github repo: https://github.com/h-j-han/VocADT  

Also, please find details in this paper:

```

@misc{han2024vocadt,

      title={Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?}, 

      author={HyoJung Han and Akiko Eriguchi and Haoran Xu and Hieu Hoang and Marine Carpuat and Huda Khayrallah},

      year={2024},

      eprint={2410.09644},

      archivePrefix={arXiv},

      primaryClass={cs.CL},

      url={https://arxiv.org/abs/2410.09644}, 

}

```