README.md

---
base_model: meta-llama/meta-llama-3.1-8b-instruct
language:
- am
- ha
- ig
- rw
- st
- sn
- so
- sw
- xh
- yo
- zu
- en
- fr
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
datasets:
- masakhane/african-ultrachat
---
# Llama-3.1-8B Instruct African Ultrachat

- **Developed by:** vutuka
- **License:** apache-2.0
- **Finetuned from model :** meta-llama/meta-llama-3.1-8b-instruct
- **Max Content Length :** `8192`
- **Max Steps :**  `800`
- **Training Time :** `02h-22min-08s`
- **Setup :**
  - `1 x RTX A6000`
  - `16 vCPU`
  - `58 GB RAM`
  - `150 GB Storage`
- **Fine Tuned Language :**
  - `Amharic`
  - `Hausa`
  - `Igbo`
  - `Kinyarwanda`
  - `Southern Sotho`
  - `Shona`
  - `Somali`
  - `Swahili`
  - `Xhosa`
  - `Yoruba`
  - `Zulu`
  - `English`
  - `French`
## Introducing Llama 3.1-8B Instruct Fine-Tuned on the Masakhane African UltraChat Dataset

We are excited to announce the fine-tuned version of the Llama 3.1-8B Instruct model, which has been trained on the Masakhane African UltraChat dataset. This fine-tuning leverages the robust architecture of the Llama 3.1 model, designed for high-performance multilingual tasks and long context processing, to enhance its capabilities in understanding and generating responses in African languages.

#### Model Overview

**Llama 3.1-8B Instruct** is part of the Llama 3 family, developed by Meta. It features an optimized transformer architecture and supports multiple languages, including English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. This model variant is particularly suited for instruction-tuned tasks, making it ideal for dialogue and assistant-like applications.

#### Training and Fine-Tuning

The model was fine-tuned using the Masakhane African UltraChat dataset, which is a diverse and extensive collection of conversational data aimed at promoting and enhancing NLP capabilities for African languages. The fine-tuning process involved supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), ensuring the model aligns well with human preferences for helpfulness and safety.

```py
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = shuffled_dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    dataset_num_proc = 2,
    packing = False, # Can make training 5x faster for short sequences.
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 800,
        do_eval=True,
        learning_rate = 3e-4,
        log_level="debug",
        #fp16 = not is_bfloat16_supported(),
        bf16 = True,
        logging_steps = 10,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
        report_to='wandb',
        warmup_ratio=0.3,
    ),
)
```

#### Performance and Capabilities

The fine-tuned Llama 3.1-8B model demonstrates improved performance in understanding and generating text in African languages, providing accurate and contextually appropriate responses. It is designed to handle various conversational tasks, from casual dialogue to more complex inquiries, making it a valuable tool for applications targeting African language users.

#### Key Features

- **Multilingual Support**: Enhanced capabilities in multiple languages, including African languages.
- **Long Context Handling**: Supports up to 128k tokens, making it suitable for long-form conversations.
- **Instruction-Tuned**: Optimized for generating accurate and helpful responses based on user instructions.
- **High Performance**: Utilizes advanced techniques like Grouped-Query Attention (GQA) for improved scalability and efficiency.


## Tokenizer & Chat Format

```py
from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template = "llama-3", # Supports zephyr, chatml, mistral, llama, alpaca, vicuna, vicuna_old, unsloth
    mapping={
        "role": "role",
        "content": "content",
        "user": "",
        "assistant": "",
    }
)

def formatting_prompts_func(examples):
    convos = examples["messages"]
    texts = [tokenizer.apply_chat_template(convo, tokenize = False, add_generation_prompt = False) for convo in convos]
    return { "text" : texts, }
pass
```


## Inference with Unsloth

```py
def chat_llama3_african_ultrachat(message: str, context: str):
    FastLanguageModel.for_inference(model) # Enable native 2x faster inference
    
    messages = [
        {"role": "system", "content": context},
        {"role": "user", "content": message},
    ]
    inputs = tokenizer.apply_chat_template(
        messages,
        tokenize = True,
        add_generation_prompt = True, # Must add for generation
        return_tensors = "pt",
    ).to("cuda")
    
    from transformers import TextStreamer
    text_streamer = TextStreamer(tokenizer)
    #_ = model.generate(input_ids = inputs, streamer = text_streamer, max_new_tokens = 1024, use_cache = True)
    output = model.generate(input_ids = inputs, max_new_tokens = 1024, use_cache = True)
    generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
    
    # Extract the assistant's message
    user_marker = "user"
    assistant_marker = "assistant"
    
    response_start = generated_text.find(assistant_marker) + len(assistant_marker)
    response_end = generated_text.find(user_marker, response_start)
    
    if response_end == -1:
        response = generated_text[response_start:].strip()
    else:
        response = generated_text[response_start:response_end].strip()
    
    return response
```

```py
chat_llama3_african_ultrachat(
    message="Habari !",
    context="Wewe ni wakala wa mtandaoni anayesaidia ambaye hujibu maswali kwa upole na heshima."
)
```

```txt
<|begin_of_text|><|start_header_id|>user<|end_header_id|>

Wewe ni wakala wa mtandaoni anayesaidia ambaye hujibu maswali kwa upole na heshima.<|eot_id|><|start_header_id|>user<|end_header_id|>

Habari!<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Habari yako? Je, unatafuta ushauri au maelezo kuhusu jambo maalum? Ni furaha yangu kusaidia.<|eot_id|>
```


## Inference with Unsloth Chat (`new`)

- Run our code in a `T4` and try the model.

```sh
#@title ↙️ Press ▶ to start 🦥 Unsloth Studio Chat for Gemma-2 2b Instruct

# Unsloth Studio
# Copyright (C) 2024-present the Unsloth AI team. All rights reserved.

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published
# by the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.

# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.

# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.
!git clone https://github.com/unslothai/studio > /dev/null 2>&1
with open("studio/unsloth_studio/chat.py", "r") as chat_module:
    code = chat_module.read().replace(
        'MODEL_NAME = "vutuka/Llama-3.1-8B-Instruct-African-Ultrachat"',
        'MODEL_NAME = "unsloth/gemma-2-2b-it-bnb-4bit"',
    )
exec(code)
```

This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)