|
--- |
|
license: apache-2.0 |
|
pipeline_tag: text-generation |
|
language: |
|
- en |
|
tags: |
|
- pretrained |
|
datasets: |
|
- Skylion007/openwebtext |
|
- c4 |
|
- wikimedia/wikipedia |
|
- tiiuae/falcon-refinedweb |
|
- izumi-lab/open-text-books |
|
- togethercomputer/RedPajama-Data-V2 |
|
- databricks/databricks-dolly-15k |
|
- euclaise/reddit-instruct-curated |
|
- CohereForAI/aya_dataset |
|
widget: |
|
- text: "Let me tell you a short story:\nOnce upon a time," |
|
- text: "Chapter 1: About Quantum Computing\n\nQuantum computing has the potential to" |
|
- text: "Hi! I'm" |
|
- text: "The story takes place in" |
|
- text: "Reducing waste generation is essential to" |
|
- text: "The best way to improve your health is" |
|
inference: |
|
parameters: |
|
max_new_tokens: 64 |
|
do_sample: true |
|
temperature: 0.72 |
|
top_p: 0.73 |
|
top_k: 50 |
|
repetition_penalty: 1.176 |
|
--- |
|
|
|
# Minueza-32M-Base |
|
|
|
## Summary |
|
|
|
Minueza-32M-Base is a foundation model with 32 million parameters trained from scratch. |
|
|
|
The following versions of this model are available: |
|
|
|
- [Minueza-32M-Base](https://huggingface.co/Felladrin/Minueza-32M-Base): The base model, trained from scratch on a large corpus of text in English. |
|
- [Minueza-32M-Chat](https://huggingface.co/Felladrin/Minueza-32M-Chat): A version of the base model fine-tuned on conversational datasets. |
|
|
|
## Intended Uses |
|
|
|
This model was created with the following objectives in mind: |
|
|
|
- Run on mobile web browsers via [Transformers.js](https://huggingface.co/docs/transformers.js). |
|
- Run fast on machines without GPU. |
|
- Serve as a base for fine-tunes using ChatML format, hence the two additional special tokens (`<|im_start|>` and `<|im_end|>`) with `<|im_end|>` as default EOS token. |
|
- ChatML works great for both instruction and chat models, so if all fine-tunes are made following the ChatML pattern, other users might benefit from the easiness of creating merges. |
|
|
|
## Datasets |
|
|
|
The model was trained on a subset of each of the following non-synthetic datasets: |
|
|
|
- [Skylion007/openwebtext](https://huggingface.co/datasets/Skylion007/openwebtext) |
|
- [c4](https://huggingface.co/datasets/c4) |
|
- [wikimedia/wikipedia - 20231101.simple](https://huggingface.co/datasets/wikimedia/wikipedia/viewer/20231101.simple) |
|
- [tiiuae/falcon-refinedweb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) |
|
- [izumi-lab/open-text-books](https://huggingface.co/datasets/izumi-lab/open-text-books) |
|
- [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) |
|
- [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) |
|
- [euclaise/reddit-instruct-curated](https://huggingface.co/datasets/euclaise/reddit-instruct-curated) |
|
- [CohereForAI/aya_dataset - original english annotations](https://huggingface.co/datasets/CohereForAI/aya_dataset/viewer/default/train?f[language_code][value]=%27eng%27) |
|
|
|
The subsets were interleaved to form the final training corpus of approximately 650 million tokens. |
|
|
|
## Usage |
|
|
|
This is a pre-trained foundation model. For your task, you will likely want to perform application-specific fine-tuning. |
|
|
|
Also note that this model was trained on internet text data, which may contain biases, offensive or inappropriate content, and may produce incorrect or irrelevant responses. No evaluation has been conducted, so use with care. |
|
|
|
Having that said, here's how you can run it: |
|
|
|
```python |
|
from transformers import pipeline |
|
|
|
generate = pipeline("text-generation", "Felladrin/Minueza-32M-Base") |
|
|
|
prompt = "The best way to improve your health is" |
|
|
|
output = generate( |
|
prompt, |
|
max_new_tokens=256, |
|
do_sample=True, |
|
temperature=0.72, |
|
top_p=0.73, |
|
top_k=50, |
|
repetition_penalty=1.176, |
|
) |
|
|
|
print(output[0]["generated_text"]) |
|
``` |
|
|
|
## Model Architecture |
|
|
|
Trained on a context window of 2048 tokens, this is a transformer model with the Mistral architecture, which includes Grouped-Query Attention, Sliding-Window Attention, and Byte-fallback BPE tokenizer. |
|
|
|
| Configuration | Value | |
|
| :---------------------- | :---- | |
|
| max_position_embeddings | 2048 | |
|
| hidden_size | 312 | |
|
| intermediate_size | 1092 | |
|
| num_attention_heads | 12 | |
|
| num_hidden_layers | 10 | |
|
| num_key_value_heads | 4 | |
|
| vocab_size | 32002 | |
|
|
|
## Training |
|
|
|
| Hyperparameter | Value | |
|
| :-------------------------- | :-------------------------------------------- | |
|
| learning_rate | 5e-05 | |
|
| train_batch_size | 1 | |
|
| eval_batch_size | 1 | |
|
| seed | 42 | |
|
| gradient_accumulation_steps | 8 | |
|
| total_train_batch_size | 8 | |
|
| optimizer | Adam with betas=(0.9,0.999) and epsilon=1e-08 | |
|
| lr_scheduler_type | linear | |
|
| num_epochs | 1.0 | |
|
|
|
| Framework | Version | |
|
| :----------- | :---------- | |
|
| Transformers | 4.38.0.dev0 | |
|
| Pytorch | 2.1.2 | |
|
| Datasets | 2.16.1 | |
|
| Tokenizers | 0.15.1 | |
|
|
|
## License |
|
|
|
This model is licensed under the [Apache License 2.0](https://huggingface.co/Felladrin/Minueza-32M-Base/resolve/main/license.txt). |
|
|
