|
---
|
|
language:
|
|
- multilingual
|
|
- en
|
|
- de
|
|
- nl
|
|
- sv
|
|
- da
|
|
- af
|
|
- fr
|
|
- es
|
|
- it
|
|
- pt
|
|
- ro
|
|
- ru
|
|
- cs
|
|
- pl
|
|
- bg
|
|
- uk
|
|
- id
|
|
- jv
|
|
- ms
|
|
- tl
|
|
- ja
|
|
- zh
|
|
- ko
|
|
- vi
|
|
|
|
license: mit
|
|
pipeline_tag: translation
|
|
---
|
|
# MITRE 466M
|
|
|
|
## Description
|
|
MITRE (Multilingual Translation with Registers) is a multilingual, decoder-only model designed for many-to-many translation tasks.
|
|
The technology, i.e., registering, is introduced in our [paper](https://arxiv.org/abs/2501.02979).
|
|
This repository allows you employ our pre-trained model for inference. If you want to reproduce the data mining and training, please refer to this [repository](https://github.com/zhiqu22/mitre).
|
|
|
|
The model supports direct translation across 552 directions for 24 languages spanning over 5 language families.
|
|
You can use our models directly via the `transformers` libs.
|
|
An alternative version of MITRE with 913M parameters is also available in this [repository](https://huggingface.co/naist-nlp/mitre_913m).
|
|
|
|
|
|
## Usages
|
|
Before get tokenizer, you should run `pip install sentencepiece` at first.
|
|
You can simply call the tokenizer and the model by
|
|
```python
|
|
from transformers import AutoModel, AutoTokenizer
|
|
|
|
# you can switch the name to "naist-nlp/mitre_913m"
|
|
tokenizer = AutoTokenizer.from_pretrained("naist-nlp/mitre_466m", trust_remote_code=True, use_fast=False)
|
|
model = AutoModel.from_pretrained("naist-nlp/mitre_466m", trust_remote_code=True)
|
|
```
|
|
|
|
To locally use this model and check the codes, you can clone this hub, then
|
|
```python
|
|
from mitre_466m.tokenization_mitre import MitreTokenizer
|
|
from mitre_466m.modeling_mitre import MitreForConditionalGeneration
|
|
|
|
tokenizer = MitreTokenizer.from_pretrained("mitre_466m")
|
|
model = MitreForConditionalGeneration.from_pretrained("mitre_466m")
|
|
```
|
|
|
|
After get the objects of the model and the tokenizer, we can do translation.
|
|
```python
|
|
english_text = "I have a red apple."
|
|
chinese_text = "我有一个红苹果。"
|
|
model.half() # recommended
|
|
model.eval()
|
|
|
|
# Translating from one or several sentences to a 'target_language'
|
|
src_tokens = tokenizer.encode_source_tokens_to_input_ids([english_text, ], target_language="zh")
|
|
# Translating from one or several sentences to given languages
|
|
# src_tokens = tokenizer.encode_source_tokens_to_input_ids_with_different_tags([english_text, english_text, ], target_languages_list=["de", "zh", ])
|
|
|
|
generated_tokens = model.generate(src_tokens.cuda())
|
|
results = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
|
|
print(results)
|
|
# results
|
|
# de: Ich habe einen roten Apfel.
|
|
# zh: 我有一个红苹果。
|
|
|
|
# For training
|
|
# 1. The difference between tgt_tokens and labels is that the eos_tokens are moved to the right side.
|
|
# 2. We recommend using 'tokenizer.encode_target_tokens_to_labels' instead of modifying tgt_tokens,
|
|
# because 'tokenizer.encode_target_tokens_to_input_ids' has pads.
|
|
# 3. You can refer to our code for detailed implementation.
|
|
# tgt_tokens = tokenizer.encode_target_tokens_to_input_ids(chinese_text)
|
|
# labels = tokenizer.encode_target_tokens_to_labels(chinese_text)
|
|
```
|
|
|
|
## Notes
|
|
We basically follow the style of [M2M](https://huggingface.co/facebook/m2m100_418M), however, we make some necessary improvements to reduce cost in generation.
|
|
You can refer to the codes of 'generate()' in [modeling_mitre.py](https://huggingface.co/naist-nlp/mitre_466m/blob/main/modeling_mitre.py) for much more details.
|
|
Moreover, we have a plan to implement FlashAttention V2 to further boost our models, which will be updated as soon as possible.
|
|
|
|
## Languages covered
|
|
Germanic: English (en), German (de), Dutch; Flemish (nl), Swedish (sv), Danish (da), Afrikaans (af)
|
|
Romance: French (fr), Spanish (es), Italian (it), Portuguese (pt), Romanian; Moldavian; Moldovan (ro)
|
|
Slavic: Russian (ru), Czech (cs), Polish (pl), Bulgarian (bg), Ukrainian (uk)
|
|
Malayo-Polynesian: Indonesian (id), Malay (ms), Javanese (jv), Tagalog;Filipino (tl)
|
|
Asian*: Chinese (zh), Japanese (ja), Korean (ko), Vietnamese (vi)
|
|
|
|
## BibTeX entry and citation info
|
|
```
|
|
@misc{qu2025registeringsourcetokenstarget,
|
|
title={Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation},
|
|
author={Zhi Qu and Yiran Wang and Jiannan Mao and Chenchen Ding and Hideki Tanaka and Masao Utiyama and Taro Watanabe},
|
|
year={2025},
|
|
eprint={2501.02979},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CL},
|
|
url={https://arxiv.org/abs/2501.02979},
|
|
}
|
|
``` |
