|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- SkelterLabsInc/JaQuAD |
|
language: |
|
- ja |
|
--- |
|
|
|
MambaSan-130m-instruct 🐍 |
|
|
|
**MambaSan-instruct is the first chat Japanese language model based on a state-space model architecture (Mamba), not a transformer.** |
|
|
|
The model is based on Albert Gu's and Tri Dao's work *Mamba: Linear-Time Sequence Modeling with Selective State Spaces* ([paper](https://arxiv.org/pdf/2312.00752.pdf)) as well as their [model implementation](https://github.com/state-spaces/mamba). |
|
This work was also inspired by heavenq's mamba-chat implementation in English. |
|
|
|
Mamba-Chat is based on MambaSan-130m and was fine-tuned on 31,7k examples samples of the [SkelterLabsInc/JaQuAD](https://huggingface.co/datasets/SkelterLabsInc/JaQuAD) dataset. To learn more, you can: |
|
|
|
- Take a look at the model on [Huggingface](https://huggingface.co/loiccabannes/MambaSan-130m-instruct) 🤗 |
|
- Talk to Mamba-Chat on [Google Colab](https://colab.research.google.com/drive/1oDM071iXTLxiuDMzQtZVgyNzCi22xupy?usp=sharing) |
|
|
|
|
|
The Code used for pretraining and finetuning will soon be published on my github: https://github.com/lcabannes |
|
<br> |
|
|
|
|
|
## Citation |
|
|
|
``` |
|
bibtex |
|
@misc{lcabannes2024MambaSan-130m-instruct, |
|
title = {MambaSan-130-instruct}, |
|
author = {Loïc Cabannes}, |
|
year = {2024}, |
|
howpublished = {HuggingFace}, |
|
url = {https://huggingface.co/loiccabannes/MambaSan-130m-instruct/} |
|
} |
|
``` |
