

The following implementation details are shared with Mistral AI's first model Mistral-7B:
- Sliding Window Attention - Trained with 8k context length and fixed cache size, with a theoretical attention span of 128K tokens
- GQA (Grouped Query Attention) - allowing faster inference and lower cache size.
- Byte-fallback BPE tokenizer - ensures that characters are never mapped to out of vocabulary tokens.
For more details refer to the release blog post.
License
Mixtral-8x7B is released under the Apache 2.0 license.
Usage tips
The Mistral team has released 2 checkpoints:
- a base model, Mixtral-8x7B-v0.1, which has been pre-trained to predict the next token on internet-scale data.
- an instruction tuned model, Mixtral-8x7B-Instruct-v0.1, which is the base model optimized for chat purposes using supervised fine-tuning (SFT) and direct preference optimization (DPO).
The base model can be used as follows:
thon