text
Hi everyone,
Hi everyone,
new_line
text
For my first post, I'm announcing a big release (in multiple ways): probably the largest open corpus in French to date, with 85 billion words in the public domain.
For my first post, I'm announcing a big release (in multiple ways): probably the largest open corpus in French to date, with 85 billion words in the public domain.
new_line
text
The dataset has been prepared in collaboration with Benoît de Courson and Benjamin Azoulay from Gallicagram (
The dataset has been prepared in collaboration with Benoît de Courson and Benjamin Azoulay from Gallicagram (
link
https://shiny.ens-paris-saclay.fr/app/gallicagram
https://shiny.ens-paris-saclay.fr/app/gallicagram
text
). Gallicagram is a major cultural analytics project in French, the open and better version of ngram viewer for large scale search of word and ngram occurrences.
). Gallicagram is a major cultural analytics project in French, the open and better version of ngram viewer for large scale search of word and ngram occurrences.
new_line
text
The corpus is made of two different dataset for monographs (16B words) 
The corpus is made of two different dataset for monographs (16B words) 
resource
https://huggingface.co/datasets/PleIAs/French-PD-Newspapers
dataset
PleIAs/French-PD-Newspapers
https://huggingface.co/datasets/PleIAs/French-PD-Newspapers
text
 and newspapers/periodicals (69B) 
 and newspapers/periodicals (69B) 
resource
https://huggingface.co/datasets/PleIAs/French-PD-Newspapers
dataset
PleIAs/French-PD-Newspapers
https://huggingface.co/datasets/PleIAs/French-PD-Newspapers
text
 Along with the full text it also includes core provenance metadata.
 Along with the full text it also includes core provenance metadata.
new_line
text
Beyond research in digital humanities, the corpus can also be used to training open and reproducible LLMs. Being in the public domain means it can be released everywhere in any shape without restrictions.
Beyond research in digital humanities, the corpus can also be used to training open and reproducible LLMs. Being in the public domain means it can be released everywhere in any shape without restrictions.
new_line
text
The corpus is not perfect: digitization of cultural heritage is challenging and, especially for newspapers, we tackle with layout issues and a significant rate of optical character recognition mistake. Our conviction is that releasing corpus as a commons is the best way to improve on this. Sharing is caring.
The corpus is not perfect: digitization of cultural heritage is challenging and, especially for newspapers, we tackle with layout issues and a significant rate of optical character recognition mistake. Our conviction is that releasing corpus as a commons is the best way to improve on this. Sharing is caring.
Hi everyone,
For my first post, I'm announcing a big release (in multiple ways): probably the largest open corpus in French to date, with 85 billion words in the public domain.
The dataset has been prepared in collaboration with Benoît de Courson and Benjamin Azoulay from Gallicagram (https://shiny.ens-paris-saclay.fr/app/gallicagram). Gallicagram is a major cultural analytics project in French, the open and better version of ngram viewer for large scale search of word and ngram occurrences.
The corpus is made of two different dataset for monographs (16B words) https://huggingface.co/datasets/PleIAs/French-PD-Newspapers and newspapers/periodicals (69B) https://huggingface.co/datasets/PleIAs/French-PD-Newspapers Along with the full text it also includes core provenance metadata.
Beyond research in digital humanities, the corpus can also be used to training open and reproducible LLMs. Being in the public domain means it can be released everywhere in any shape without restrictions.
The corpus is not perfect: digitization of cultural heritage is challenging and, especially for newspapers, we tackle with layout issues and a significant rate of optical character recognition mistake. Our conviction is that releasing corpus as a commons is the best way to improve on this. Sharing is caring.
❤️
anastasiastasenko davanstrien julien-c osanseviero clem victor Tonic Pclanglais blanchon bgowan philschmid kramp Wauplin clefourrier afrideva samusenps KaiKimera kheopss yjernite sbrandeis
🤯
davanstrien julien-c osanseviero clem kramp vicgalle Tonic blanchon philschmid damerajee danielus
👍
neovalle julien-c osanseviero clem Tonic samusenps damerajee
🤗
Tonic samusenps
https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg
Daniel van Strien
davanstrien
user