File size: 1,413 Bytes
8d1df11 0188bf6 8d1df11 c2841d5 8d1df11 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
---
license: apache-2.0
datasets:
- SkelterLabsInc/JaQuAD
language:
- ja
---
MambaSan-130m-instruct 🐍
**MambaSan-instruct is the first chat Japanese language model based on a state-space model architecture (Mamba), not a transformer.**
The model is based on Albert Gu's and Tri Dao's work *Mamba: Linear-Time Sequence Modeling with Selective State Spaces* ([paper](https://arxiv.org/pdf/2312.00752.pdf)) as well as their [model implementation](https://github.com/state-spaces/mamba).
This work was also inspired by heavenq's mamba-chat implementation in English.
Mamba-Chat is based on MambaSan-130m and was fine-tuned on 31,7k examples samples of the [SkelterLabsInc/JaQuAD](https://huggingface.co/datasets/SkelterLabsInc/JaQuAD) dataset. To learn more, you can:
- Take a look at the model on [Huggingface](https://huggingface.co/loiccabannes/MambaSan-130m-instruct) 🤗
- Talk to Mamba-Chat on [Google Colab](https://colab.research.google.com/drive/1oDM071iXTLxiuDMzQtZVgyNzCi22xupy?usp=sharing)
The Code used for pretraining and finetuning will soon be published on my github: https://github.com/lcabannes
<br>
## Citation
```
bibtex
@misc{lcabannes2024MambaSan-130m-instruct,
title = {MambaSan-130-instruct},
author = {Loïc Cabannes},
year = {2024},
howpublished = {HuggingFace},
url = {https://huggingface.co/loiccabannes/MambaSan-130m-instruct/}
}
``` |