File size: 6,634 Bytes

---
license: apache-2.0
tags:
- moe
- frankenmoe
- merge
- mergekit
- lazymergekit
model-index:
- name: Lumina-3.5
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 71.59
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Ppoyaa/Lumina-3.5
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 88.82
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Ppoyaa/Lumina-3.5
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 64.48
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Ppoyaa/Lumina-3.5
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 75.66
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Ppoyaa/Lumina-3.5
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 83.98
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Ppoyaa/Lumina-3.5
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 67.93
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Ppoyaa/Lumina-3.5
      name: Open LLM Leaderboard
---

# Lumina-3.5

Lumina-3.5 is a Mixture of Experts (MoE) made with [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing) with improved reasoning from [Lumina-3](https://huggingface.co/Ppoyaa/Lumina-3). This model uses a context window of up to 32k.

# Response Examples
``` Math
Input: 25 - 4 * 2 + 3 = ?
------------------------------------
Output: To solve this equation, we will follow the order of operations (PEMDAS/BODMAS). This means performing any multiplications and divisions first from left to right, and then additions and subtractions from left to right.

The given equation is: 25 - 4 * 2 + 3

First, let's perform the multiplication:
4 * 2 = 8

Now, our equation looks like: 25 - 8 + 3

Next, perform the subtraction:
25 - 8 = 17

Finally, do the addition:
17 + 3 = 20

So, the result of the equation 25 - 4 * 2 + 3 is 20.
```

``` Reasoning
Input: Jane is faster than Joe. Joe is faster than Sam. Is Sam faster than Jane? Explain your reasoning step by step.
------------------------------------
Output: To answer this question, let's analyze the given information step by step:

We are provided with a comparison between three individuals - Jane, Joe, and Sam - in terms of their relative speeds. The statements tell us the following about their ranks:
Jane > Joe (Jane is faster than Joe)
Joe > Sam (Joe is faster than Sam)
Now we need to determine if the opposite relationship holds true: whether Sam is faster than Jane or not. Based on the given comparisons, we know that both Jane and Joe have faster speeds compared to Sam. There isn't any piece of information suggesting otherwise.

Since the original premise established an ordered ranking among these individuals (Jane > Joe > Sam), and the direct opposite of the relationship between Sam and the other two is not present, we cannot conclude that Sam is faster than Jane.

Thus, based on the provided information, our reasoning confirms that Sam is not faster than Jane.
```

## 🏆 Open LLM Leaderboard Evaluation Results 
|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |75.41|
|AI2 Reasoning Challenge (25-Shot)|71.59|
|HellaSwag (10-Shot)              |88.82|
|MMLU (5-Shot)                    |64.48|
|TruthfulQA (0-shot)              |75.66|
|Winogrande (5-shot)              |83.98|
|GSM8k (5-shot)                   |67.93|

# Quants

Special thanks to GGUFs made by [mradermacher](https://huggingface.co/mradermacher)
* [mradermacher/Lumina-3.5-GGUF](https://huggingface.co/mradermacher/Lumina-3.5-GGUF)

## 💻 Usage

```python
!pip install -qU transformers bitsandbytes accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "Ppoyaa/Lumina-3.5"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)

messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Ppoyaa__Lumina-3.5)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |75.41|
|AI2 Reasoning Challenge (25-Shot)|71.59|
|HellaSwag (10-Shot)              |88.82|
|MMLU (5-Shot)                    |64.48|
|TruthfulQA (0-shot)              |75.66|
|Winogrande (5-shot)              |83.98|
|GSM8k (5-shot)                   |67.93|