transformers safetensors mistral text-generation FastLanguageModel unsloth en license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference 4-bit bitsandbytes region:us
---
library_name: transformers
license: apache-2.0
language:
- en
tags:
- FastLanguageModel
- unsloth
---

# Model Card for TinyMistral-248M-v2-4bit

A model to use with unsloth for running on cheap hardware, a la Kaggle and Colab free-tier.

[Unsloth](https://unsloth.ai/introducing) has some great ideas about optimization and integration. Very easy to get started with and use on commodity hardware.

[Locutusque](https://huggingface.co/Locutusque) and [Felladrin](https://huggingface.co/Felladrin) in particular are building some wonderful and quite useful small models.

Unsloth has focused on tinyllama, thus this focus on a great alternative, the TinyMistral series. 

## Model Details

Can be loaded through unsloth - rather than over-explaining, check out these resources:
- [Unsloth integration with TRL](https://github.com/huggingface/blog/blob/main/unsloth-trl.md)
- [Unsloth integration with DPO](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)
- [Reproduce Zephyr on a T4](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing)

You get the picture, small models make for better optimization, but this level of optimizing is needed for adoption, distillation, and reduced environmental impact. 

### Model Description

This isn't a useful model on its own - Uses unsloth's FastLanguageModel model loader, which handles a whole bunch of behind the scenes organization and complexity.

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

- **Developed by:** All credit to [Locutusque](https://huggingface.co/Locutusque)
- **Funded by:** Copious pocket lint
- **Shared by:** jtatman
- **Model type:** Peft/BitsnBytes 4-bit ready adapter
- **Language(s) (NLP):** None 
- **License:** Apache 2.x
- **Finetuned from model [optional]:** Locutusque/TinyMistral-248M-v2
- Not Finetuned, a direct save of a 4bit (obviously, not 4bit on disk as this stream format not yet implemented in BnB) adapter load for memory requirements

## Uses

Use with [unsloth library only](https://github.com/unslothai/unsloth)


### Out-of-Scope Use

Will not load on inference on demand or through native transformers - yet.

