
<p align="center">
  <img src="seammm_2.png" width="200" />
</p>

> SeaLLM will be able to "see"!

# *SeaLMMM-7B* - Large Multilingual Multimodal Models for Southeast Asia

<p align="center">
<a href="https://huggingface.co/SeaLLMs/SeaLMMM-7b/" target="_blank" rel="noopener"> 🤗 Tech Memo</a>
&nbsp;&nbsp;
<a href="https://huggingface.co/spaces/SeaLLMs/SeaLMMM-7b" target="_blank" rel="noopener"> 🤗 DEMO</a>
&nbsp;&nbsp;
<a href="https://github.com/DAMO-NLP-SG/SeaLLMs" target="_blank" rel="noopener">Github</a>
&nbsp;&nbsp;
<a href="https://arxiv.org/pdf/2312.00738.pdf" target="_blank" rel="noopener">Technical Report</a>
</p>


We introduce and [showcase](https://huggingface.co/spaces/SeaLLMs/SeaLMMM-7b) an **Early** iteration of SeaLMMM-7B | A unified multilingual and multimodal that excel in both text-only and vision tasks in multiple languages spoken in Southeast Asia.


### SeaLMMM-7B-early abilities
* SeaLMMM-7B is one of the most ouperforming 7B vision-language models at **TEXT-ONLY tasks**, with performance similar to [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2). It is a text-first-vision-second model.
* SeaLMMM-7B **willI** be able to handle most SEA languages, making it more multilingual than En-only LLava, Bilingual (En+Zh) Qwen-VL or Yi-VL. 
* Unlike LLava or specialized VLMs, which demand only one image at the begining, SeaLMMM-7B can seamlessly handle text-only conversations at the begining and visual instructions in the middle of the conversations and support topic and language switching.


### Release and DEMO

- DEMO: [SeaLLMs/SeaLMMM-7b](https://huggingface.co/spaces/SeaLLMs/SeaLMMM-7b).
- Technical report and model weights: Comming soon.

<blockquote style="color:red">
<p><strong style="color: red">Terms of Use and License</strong>: 
By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b/edit/main/LICENSE" target="_blank" rel="noopener">SeaLLMs Terms Of Use</a>.
</blockquote>


## Multimodal Showcases

* **SeaLMMM-7B-early** will have better vision understanding and solving abilities in languages beyond English and Chinese.

![two_cat.png](two_cat.png)


Image: find "x" in Vietnamese. Left: Llava-1.6-34B. Right: SeaLMMM-7B-early. 
<div class="row" style="display: flex; clear: both;">
    <img src="llava_1.6_34b_find_x_vi.png" alt="Forest" style="float: left; width: 39%">
    <img src="find_x_vi.png" alt="Snow" style="float: left; width: 59%">
</div>



## Text-only Evaluation


### Zero-shot CoT Multilingual Math Reasoning

**SeaLLM-7B-early** performs relatively similar to [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2), with 76.2 on GSM8K and 28.0 on MATH.

![fig_sea_math_side_by_side_sealmmm.png](fig_sea_math_side_by_side_sealmmm.png)


<details>
<summary>See details on English and translated GSM8K and MATH with zero-shot reasoning</summary>
<br>

| Model | GSM8K<br>en | MATH<br>en | GSM8K<br>zh | MATH<br>zh | GSM8K<br>vi | MATH<br>vi | GSM8K<br>id | MATH<br>id | GSM8K<br>th | MATH<br>th
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-3.5 | 80.8 | 34.1 | 48.2 | 21.5 | 55 | 26.5 | 64.3 | 26.4 | 35.8 | 18.1
| Qwen-14B-chat | 61.4 | 18.4 | 41.6 | 11.8 | 33.6 | 3.6 | 44.7 | 8.6 | 22 | 6
| Vistral-7b-chat | 48.2 | 12.5 |  |  | 48.7 | 3.1 |  |  |  | 
| Qwen1.5-7B-chat  | 56.8 | 15.3 | 40 | 2.7 | 37.7 | 9 | 36.9 | 7.7 | 21.9 |
| SeaLLM-7B-v2     | 78.2 | 27.5 | 53.7 | 17.6 | 69.9 | 23.8 | 71.5 | 24.4 | 59.6 | 22.4
| SeaLMMM-7B-Early | 76.7 | 28.0 | 51.6 | 17.5 | 67.1 | 24.0 | 67.3 | 26.2 | 53.3 | 21.6
</details>

Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Vistral](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)).


### Multilingual World Knowledge

We evaluate models on 3 benchmarks following the recommended default setups: 5-shot MMLU for En, 3-shot [M3Exam](https://arxiv.org/pdf/2306.05179.pdf) (M3e) for En, Zh, Vi, Id, Th, and zero-shot [VMLU](https://vmlu.ai/) for Vi.

**SeaLMMM-7B-early** even outperforms SeaLLM-7B-v2 in Vi, Id, demonstrating that vision understanding capability does not degrade text-only performance, given the same model size.

| Model | Langs | En<br>MMLU | En<br>M3e | Zh<br>M3e | Vi<br>M3e | Vi<br>VMLU | Id<br>M3e | Th<br>M3e
|-----| -----  | --- |  -- | ----- | ---- | --- | --- | --- |
| GPT-3.5         | Multi | 68.90 | 75.46 | 60.20 | 58.64 | 46.32 | 49.27 | 37.41
| Vistral-7B-chat | Mono  | 56.86 | 67.00 | 44.56 | 54.33 | 50.03 | 36.49 | 25.27
| Qwen1.5-7B-chat | Multi | 61.00 | 52.07 | 81.96 | 43.38 | 45.02 | 24.29 | 20.25
| SeaLLM-7B-v2    | Multi | 61.89 | 70.91 | 55.43 | 51.15 | 45.74 | 42.25 | 35.52
| SeaLMMM-7B-early| Multi | 61.02 | 70.91 | 53.23 | 52.15 | 46.11 | 42.50 | 35.47


VMLU reproduce script [here](https://github.com/DAMO-NLP-SG/SeaLLMs/blob/main/evaluation/vmlu/vmlu_run.py). Lm-eval was used to evaluate MMLU.
0-shot VMLU scores for baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json)).


### SeaLMMM-7B-early has **NOT finished training yet** and will change! Expect certain limitations in a number of area:
* It's multi-modal capabilities on the [DEMO](https://huggingface.co/spaces/SeaLLMs/SeaLMMM-7b) support mainly English & Vietnamese, though other SEA languages should work fine on easy tasks.
* For OCR, it can only read English.
* The early version has not finished training, so it has poorer VQA performances than SOTA VLMs like LLava-1.6.
* SeaLMMM-7B-early sometimes still think it cannot process image due to existing text-only data, future version should fix this.
* Multi-modal multi-turn capabilities are still limited.
We're are working hard to resolve these known issues and open-source models later. Meantime, feedback is welcome.



## Acknowledgement to Our Linguists

We would like to express our special thanks to our professional and native linguists, Tantong Champaiboon, Nguyen Ngoc Yen Nhi and Tara Devina Putri, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.

## Citation

If you find our project useful, we hope you would kindly star our repo and cite our work as follows: Corresponding Author: [l.bing@alibaba-inc.com](mailto:l.bing@alibaba-inc.com)

**Author list and order will change!**

* `*` and `^` are equal contributions.

```
@article{damonlpsg2023seallm,
  author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*,
            Zhiqiang Hu, Chenhui Shen^, Yew Ken Chia^, Xingxuan Li, Jianyu Wang,
            Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
            Chaoqun Liu, Hang Zhang, Lidong Bing},
  title = {SeaLLMs - Large Language Models for Southeast Asia},
  year = 2023,
  Eprint = {arXiv:2312.00738},
}
```


{"language": ["en", "zh", "vi", "id", "th"], "license": "other", "tags": ["multilingual", "sea"], "license_name": "seallms", "license_link": "https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE"}
multilingual sea en zh vi id th arxiv:2312.00738 arxiv:2306.05179 license:other has_space region:us