
# Mixtral-8x7B-SQL

- Model creator: [Mistral AI](https://huggingface.co/mistralai)
- Original model: [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)


## Description

This repo contains LoRA finetuned model files for [Mistral AI's Mixtral 8x7B v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1).


<!-- prompt-template start -->
## Prompt template: Mixtral

```
<s>[INST] {prompt} [/INST]
```

<!-- prompt-template end -->

<!-- prompt-template start -->


## Instruction format

In order to leverage instruction fine-tuning, your prompt should be surrounded by `[INST]` and `[/INST]` tokens. The very first instruction should begin with a sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.

For example:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model = "machinists/Mixtral-8x7B-SQL"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
    device_map="auto",
)

table_schema = "CREATE TABLE head (age INTEGER)"

question = "How many heads of the departments are older than 56 ?"

system_msg = f" Generate a correct SQL query from the following database schema. \n {table_schema} "

prompt = f"<s>[INST] {system_msg} \n{question} [/INST]"

sequences = pipeline(
    prompt,
    max_length=1000,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")

```

## Model Architecture
The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.

  
## Finetuning

Finetuning technique involving rank of matrices to reduce computational and memory requirements.
Read more: [LoRA Hugging Face article](https://huggingface.co/docs/diffusers/main/en/training/lora)


Epoch - 01

Dataset Name - Clinton/Text-to-sql-v1

No. of records - 262,208

Model Loading - bf16 and flash attention2

Finetuning Technique - Using LoRA on all linear layers

MaxSeqLength - 3078

Effective Batch Size - 16

Mixed Precision Training - tf32 

Learning Rate - 2e-4


## Hardware and Software

- **Training Hardware**: 4 X Nvidia A100 80GB GPU
- **Training Duration**: 16 hours


## The Machinists Team

Manish Kumar - manish.kumar@machinists.tech

Aakash Sarin - aakash.sarin@machinists.tech
{"language": ["en"], "license": "apache-2.0", "tags": ["sql", "finetuned", "lora", "46.7B", "mixtral"], "datasets": ["Clinton/Text-to-sql-v1"], "model_name": "machinists/Mixtral-8x7B-SQL", "base_model": "mistralai/Mixtral-8x7B-v0.1", "model_creator": "Machinists", "model_type": "mixtral", "pipeline_tag": "text-generation", "finetuned_by": "masterfury, pegaison"}
transformers safetensors mixtral text-generation sql finetuned lora 46.7B en dataset:Clinton/Text-to-sql-v1 base_model:mistralai/Mixtral-8x7B-v0.1 license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us