Hugging Face's logo

Felladrin
/
Minueza-32M-Base

Text Generation
Transformers
Safetensors
English
mistral
pretrained
conversational
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
3
Minueza-32M-Base
File size: 5,513 Bytes 
aaf2464
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
 
---
license: apache-2.0
pipeline_tag: text-generation
language:
  - en
tags:
  - pretrained
datasets:
  - Skylion007/openwebtext
  - c4
  - wikimedia/wikipedia
  - tiiuae/falcon-refinedweb
  - izumi-lab/open-text-books
  - togethercomputer/RedPajama-Data-V2
  - databricks/databricks-dolly-15k
  - euclaise/reddit-instruct-curated
  - CohereForAI/aya_dataset
widget:
  - text: "Let me tell you a short story:\nOnce upon a time,"
  - text: "Chapter 1: About Quantum Computing\n\nQuantum computing has the potential to"
  - text: "Hi! I'm"
  - text: "The story takes place in"
  - text: "Reducing waste generation is essential to"
  - text: "The best way to improve your health is"
inference:
  parameters:
    max_new_tokens: 64
    do_sample: true
    temperature: 0.72
    top_p: 0.73
    top_k: 50
    repetition_penalty: 1.176
---

# Minueza-32M-Base

## Summary

Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch.

The following versions of this model are available:

- [Minueza-32M-Base](https://huggingface.co/Felladrin/Minueza-32M-Base): The base model, trained from scratch on a large corpus of text in English.
- [Minueza-32M-Chat](https://huggingface.co/Felladrin/Minueza-32M-Chat): A version of the base model fine-tuned on conversational datasets.

## Intended Uses

This model was created with the following objectives in mind:

- Run on mobile web browsers via [Transformers.js](https://huggingface.co/docs/transformers.js).
- Run fast on machines without GPU.
- Serve as a base for fine-tunes using ChatML format, hence the two additional special tokens (`<|im_start|>` and `<|im_end|>`) with `<|im_end|>` as default EOS token.
  - ChatML works great for both instruction and chat models, so if all fine-tunes are made following the ChatML pattern, other users might benefit from the easiness of creating merges.

## Datasets

The model was trained on a subset of each of the following non-synthetic datasets:

- [Skylion007/openwebtext](https://huggingface.co/datasets/Skylion007/openwebtext)
- [c4](https://huggingface.co/datasets/c4)
- [wikimedia/wikipedia - 20231101.simple](https://huggingface.co/datasets/wikimedia/wikipedia/viewer/20231101.simple)
- [tiiuae/falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)
- [izumi-lab/open-text-books](https://huggingface.co/datasets/izumi-lab/open-text-books)
- [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2)
- [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
- [euclaise/reddit-instruct-curated](https://huggingface.co/datasets/euclaise/reddit-instruct-curated)
- [CohereForAI/aya_dataset - original english annotations](https://huggingface.co/datasets/CohereForAI/aya_dataset/viewer/default/train?f[language_code][value]=%27eng%27)

The subsets were interleaved to form the final training corpus of approximately 650 million tokens.

## Usage

This is a pre-trained foundation model. For your task, you will likely want to perform application-specific fine-tuning.

Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care.

Having that said, here's how you can run it:

```python
from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base")

prompt = "The best way to improve your health is"

output = generate(
    prompt,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.72,
    top_p=0.73,
    top_k=50,
    repetition_penalty=1.176,
)

print(output[0]["generated_text"])
```

## Model Architecture

Trained on a context window of 2048 tokens, this is a transformer model with the Mistral architecture, which includes Grouped-Query Attention, Sliding-Window Attention, and Byte-fallback BPE tokenizer.

| Configuration           | Value |
| :---------------------- | :---- |
| max_position_embeddings | 2048  |
| hidden_size             | 312   |
| intermediate_size       | 1092  |
| num_attention_heads     | 12    |
| num_hidden_layers       | 10    |
| num_key_value_heads     | 4     |
| vocab_size              | 32002 |

## Training

| Hyperparameter              | Value                                         |
| :-------------------------- | :-------------------------------------------- |
| learning_rate               | 5e-05                                         |
| train_batch_size            | 1                                             |
| eval_batch_size             | 1                                             |
| seed                        | 42                                            |
| gradient_accumulation_steps | 8                                             |
| total_train_batch_size      | 8                                             |
| optimizer                   | Adam with betas=(0.9,0.999) and epsilon=1e-08 |
| lr_scheduler_type           | linear                                        |
| num_epochs                  | 1.0                                           |

| Framework    | Version     |
| :----------- | :---------- |
| Transformers | 4.38.0.dev0 |
| Pytorch      | 2.1.2       |
| Datasets     | 2.16.1      |
| Tokenizers   | 0.15.1      |

## License

This model is licensed under the [Apache License 2.0](https://huggingface.co/Felladrin/Minueza-32M-Base/resolve/main/license.txt).