## Approach
This model of [Mamba architecture](https://arxiv.org/abs/2312.00752) has been pre-trained on approximately 400B tokens of Chinese and English corpora.

## Usage
```python
import torch

from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
from transformers import AutoTokenizer

repo_id = 'mamba-1.4b-aquila-400b'
device = f"cuda:0"
model = MambaLMHeadModel.from_pretrained(repo_id, dtype=torch.bfloat16, device=device)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(repo_id)
prompt = "The Spring Festival is"
tokens = tokenizer.encode_plus(prompt, truncation=False)["input_ids"]
tokens = torch.tensor(tokens)[None,].to(device)
with torch.no_grad():
    input_length = len(tokens[0])
    out_ids = model.generate(input_ids=tokens, max_length=input_length+200, temperature=1.0, top_p=0.95, eos_token_id=tokenizer.eos_token_id, cg=True, top_k=15)
    out_ids = out_ids[0][input_length:].cpu().numpy()
    out_text = tokenizer.decode(out_ids.tolist())
    print(out_text)
```
> the most important festival of the year for the Chinese people. It usually comes in January or February and it takes about 15 days to prepare for it. 

## References

The Mamba architecture was introduced in [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://arxiv.org/abs/2312.00752).

The official implementation is here: https://github.com/state-spaces/mamba/tree/main