
![image/png](https://cdn-uploads.huggingface.co/production/uploads/61bf0e11c88f3fd22f654059/EMSrPEzAFkjY9nvbaJoC3.png)

# polka-1.1b


`polka-1.1b` takes the [TinyLlama-1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) model and enhances it by continuing pretraining on an additional **5.7 billion Polish tokens**, primarily sourced from the [MADLAD-400](https://arxiv.org/abs/2309.04662) dataset. The tokens were sampled in a 10:1 ratio between Polish and English shards using [DSIR](https://github.com/p-lambda/dsir). Furthermore, Polka extends the TinyLlama tokenizer's vocabulary to 43,882 tokens, improving its efficiency for generating Polish text.

The training took 680 GPU hours on a single 8 x RTX 4090 machine with DeepSpeed ZeRO-2.

Context size: 2,048 tokens.

## Notes

This base model was initially developed as the foundation for instruction tuning, which resulted in [polka-1.1b-chat](https://huggingface.co/eryk-mazus/polka-1.1b-chat). Nonetheless, I'm sharing it with the community because I see potential value in its combination of relatively good performance and an efficient bilingual tokenizer.

The model is capable of producing coherent Polish text, but due to its size, it is likely to suffer from hallucination.

## Sample code

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "eryk-mazus/polka-1.1b"

tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side="left")
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", load_in_8bit=True)

prompt = """Przykładowe zapytanie do modelu"""

model_inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
with torch.no_grad():
  generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512,
    do_sample=True,
    penalty_alpha=0.6,
    top_k=5
  )

output = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(output)
```

{"language": ["pl", "en"], "license": "apache-2.0", "datasets": ["allenai/MADLAD-400", "eryk-mazus/polka-pretrain-en-pl-v1"], "base_model": "eryk-mazus/tinyllama-with-custom-tokenizer", "pipeline_tag": "text-generation", "widget": [{"text": "Wied\u017amin 3 to fabularna gra akcji wyprodukowana", "output": {"text": " przez studio CD Projekt RED. Akcja rozgrywa si\u0119 w \u015bwiecie fantasy, a jej bohaterem jest Geralt z Rivii,"}}, {"text": "Gdy ju\u017c b\u0119dziecie w Warszawie, miejscem, kt\u00f3re koniecznie musicie odwiedzi\u0107 jest", "output": {"text": " Muzeum Powstania Warszawskiego. To jedyne tego typu muzeum w Europie"}}]}
transformers pytorch llama text-generation pl en dataset:allenai/MADLAD-400 dataset:eryk-mazus/polka-pretrain-en-pl-v1 arxiv:2309.04662 base_model:eryk-mazus/tinyllama-with-custom-tokenizer license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us