# Mamba 2.8b Slim Pyjama - bf16 (16-bit)

This is a 16 bit version of [Mamba-2.8b-slimpj](https://huggingface.co/state-spaces/mamba-2.8b-slimpj/)

Mamba-2.8b-slimpj is a model using the [Mamba](https://arxiv.org/abs/2312.00752) architecture, with 2.8B parameters, trained for 600B tokens on the SlimPajama dataset. 

Model code: https://github.com/state-spaces/mamba/tree/main

To load the model, follow the installation instruction in the code repo, and then:
```
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
model = MambaLMHeadModel.from_pretrained("state-spaces/mamba-2.8b-slimpj")
```

## Inference Notebook (Colab)
- [Notebook here](https://colab.research.google.com/drive/1GsDbbkDTDpia_Dc8s-7bwEn_GrpkBVO4?usp=sharing)
{"language": ["en"], "license": "apache-2.0", "tags": ["mamba", "bf16", "16bit"], "datasets": ["cerebras/SlimPajama-627B"], "pipeline_tag": "text-generation", "inference": true}
transformers pytorch mamba bf16 16bit text-generation en dataset:cerebras/SlimPajama-627B arxiv:2312.00752 license:apache-2.0 endpoints_compatible region:us