|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- PleIAs/common_corpus |
|
language: |
|
- en |
|
- fr |
|
- es |
|
- de |
|
- it |
|
- la |
|
- nl |
|
- pl |
|
--- |
|
**Pleias-360m-Preview** is an early preview of a 360 million parameter base model trained by Pleias on Common Corpus. |
|
|
|
Like all the base and specialized models from Pleias, Pleias-360m-Preview has only been trained on open data out of copyright (public domain) or under a permissible license. |
|
|
|
## Description |
|
Pleias-360m-Preview is a transformer base model, entirely pretrained from scratch, using an architecture similar to Llama/GPT-Neox for easier deployment/inference. |
|
|
|
It includes the following features, that would apply to any responsibly trained variant: |
|
* Only trained on open data under a permissible license and in compliance with the European AI Act. By design, all Pleias model are unable to output copyrighted content. |
|
* Extensive multilingual support for main European languages. |
|
* A new tokenizer designed for enhanced document processing tasks and better multilingual support. |
|
* Extremely low level of toxicity and problematic content. |
|
|
|
Pleias-360m-Preview has demonstrated unusual abilities for multilingual generation in its size range. Fully supported languages include English, French, Spanish, German, Italian, Dutch, Latin and Portuguese. |
|
|
|
Given its size, Pleias-360m-Preview can run on CPU without any compression loss. We provide a first GGUF variant as part of our release. |
|
|
|
## Recommended use |
|
As a base model, Pleias-360m-Preview is only able to run continuation prompts. |
|
|
|
Text generation is currently able to support a range of creative writing tasks in multiple European languages. For more consistent results we recommend using a low or null temperature with a slight repetition penalty (1.1-1.2). |
|
|
|
Pleias-360m-Preview has been successfully adapted for continuous pretraining and full-fine-tuning on document processing tasks such as RAG, translation or OCR correction. Given the small size of the model we do not recommend fine-tuning methods based on LORA. |
|
|
|
## Examples |
|
With only 360 million parameters, Pleias-360m-Preview is able to generate literary texts in multiple languages: |
|
|
|
English poetry: |
|
|
|
<blockquote> |
|
<b>Saturn is fallen, am I too to fall?<br/> |
|
Am I to leave this haven of my rest,<br/> |
|
This cradle of my glory, this soft clime,<br/> |
|
This calm luxuriance of blissful light,<br/></b> |
|
And the sweetness that was not in me?<br/> |
|
I have been a slave and an enemy.<br/> |
|
The world has made me its prey ; it's wrong;<br/> |
|
It's cruelty! It's madness! — but you know<br/> |
|
That all things are good for one man or another :<br/> |
|
You see how he can be happy without them:<br/> |
|
He cannot be miserable unless they come<br/> |
|
To his heart with their own poisonous breaths.<br/> |
|
But if there were no other way left us,<br/> |
|
We should never find our happiness again.<br/> |
|
If we could only live on earth alone,<br/> |
|
With nothing else about us than ourselves,<br/> |
|
Our very being would become more real.<br/> |
|
There must be some other place where we may<br/> |
|
Live happily as before, and enjoy life<br/> |
|
As well as now. But what shall we do here?<br/> |
|
What will happen when we die? What will hap-<br/> |
|
[pen then?<br/> |
|
When death comes, let him who knows the end<br/> |
|
</blockquote> |
|
|
|
French classic play with the 17th century writing style (including the long s): |
|
|
|
<blockquote> |
|
<b>Helas! pour mon malheur ie l’ay trop écouté.<br/> |
|
Ie n’ay point du ſilence affecté le myſtère.<br/> |
|
Ie croyois ſans péril pouuoir eſtre ſincere.<br/> |
|
Et ſans armer mes yeux d’vn moment de rigueur,<br/></b> |
|
Je ne voy pas que ce foit vn crime à la nature:<br/> |
|
Mais je vous prie, qu'on me donne quelque recompenſe :<br/> |
|
Pourquoy donc auec tant de gloire &c en vne heure?<br/> |
|
Cela fait voir bien clairement que ma deſpenſe<br/> |
|
Eſt plus grande que celle des autres hommes ;<br/> |
|
Car il eſt certain que les Princes font moins heureux;<br/> |
|
Que ceux qui ont eu leur vie par leurs mains confacrée.<br/> |
|
Il y a deux fortes de perſonnes dans le monde :<br/> |
|
Les uns fe nomment gens de lettres ou de proueſleurz,<br/> |
|
Dont on dit que c'eſt eux-mesme qui nous donnent tousjours<br/> |
|
Leur efprit & leur cœur aux affaires publiques ;<br/> |
|
Ainſi ils s'appellent gens de cour, comme ſi cela<br/> |
|
N'eſtoit autre choſe que ces grands Seigneurs qui gouuernenc<br/> |
|
L'Etat,&qui ſe conſeruent toujours avec honneur.<br/> |
|
Ils ſont fort peu de choſes ſans ambition ;<br/> |
|
</blockquote> |
|
|
|
## Training |
|
Pleias-360m-Preview was fully pretrained at Jean Zay on 64 h100s for 46 hours with Nanotron, the pretraining library from HuggingFace. We provide the complete settings as a yaml file as part of our release. |
|
|
|
Training schedule includes 518,000 steps (batch size 1,024) on a filtered and enhanced version of Common Corpus (1,086,324,736,000 tokens). |
|
|
|
## Update |
|
Pleias-360m-Preview is currently released as an early preview. |
|
|
|
The model will undergo several more round of post-training to enhance reasoning capacities and fine-tunability as well as in anticipation of a generalist instruct version. |
|
|
