text
🚀🚀 Exciting times for the document AI community! 
🚀🚀 Exciting times for the document AI community! 
new_line
new_line
text
We're thrilled to announce the release of some of the largest OCR datasets available to the public.
We're thrilled to announce the release of some of the largest OCR datasets available to the public.
new_line
text
🔥 With over 26 million pages , 18 billion text tokens, and 6TB of data, these resources are a significant leap forward for document AI research.
🔥 With over 26 million pages , 18 billion text tokens, and 6TB of data, these resources are a significant leap forward for document AI research.
new_line
new_line
text
Here's how to access these datasets quickly:
Here's how to access these datasets quickly:
new_line
new_line
code_fence
```
from datasets import load_dataset

pdfa_dataset = load_dataset('pixparse/pdfa-eng-wds', streaming=True)
IDL_dataset = load_dataset('pixparse/idl-wds', streaming=True)
```
from datasets import load_dataset

pdfa_dataset = load_dataset('pixparse/pdfa-eng-wds', streaming=True)
IDL_dataset = load_dataset('pixparse/idl-wds', streaming=True)
new_line
text
This enables you to stream them directly, integrating seamlessly with your projects using the Hugging Face datasets library. On the hub, you can find them here:
This enables you to stream them directly, integrating seamlessly with your projects using the Hugging Face datasets library. On the hub, you can find them here:
new_line
new_line
resource
https://huggingface.co/datasets/pixparse/pdfa-eng-wds
dataset
pixparse/pdfa-eng-wds
https://huggingface.co/datasets/pixparse/pdfa-eng-wds
new_line
resource
https://huggingface.co/datasets/pixparse/idl-wds
dataset
pixparse/idl-wds
https://huggingface.co/datasets/pixparse/idl-wds
text
new_line
new_line
text
For lean data loading, the new [chug](
For lean data loading, the new [chug](
link
https://github.com/huggingface/chug
https://github.com/huggingface/chug
text
) library offers a solution with pdf decoding:
) library offers a solution with pdf decoding:
new_line
new_line
new_line
code_fence
```
import chug

task_cfg = chug.DataTaskDocReadCfg(
    page_sampling='all',
)
data_cfg = chug.DataCfg(
    source='pixparse/pdfa-eng-wds',
    split='train',
    batch_size=None,
    format='hfids',
    num_workers=0,
)
data_loader = chug.create_loader(
    data_cfg,
    task_cfg,
)
sample = next(iter(data_loader))
```
import chug

task_cfg = chug.DataTaskDocReadCfg(
    page_sampling='all',
)
data_cfg = chug.DataCfg(
    source='pixparse/pdfa-eng-wds',
    split='train',
    batch_size=None,
    format='hfids',
    num_workers=0,
)
data_loader = chug.create_loader(
    data_cfg,
    task_cfg,
)
sample = next(iter(data_loader))
new_line
new_line
new_line
text
We owe a huge thank you to Peter Wyatt, Kate Tasker, Rachel Taketa, Ali Furkan Biten, Ruben Tito, and their colleagues for their contributions. Their work putting these datasets together has been invaluable. 🤗
We owe a huge thank you to Peter Wyatt, Kate Tasker, Rachel Taketa, Ali Furkan Biten, Ruben Tito, and their colleagues for their contributions. Their work putting these datasets together has been invaluable. 🤗
new_line
new_line
text
Looking Ahead:
Looking Ahead:
new_line
new_line
text
We're on a mission to enhance document AI capabilities, and these datasets are just the beginning. With your engagement and innovation, we're confident in the community's ability to develop robust OCR solutions. We encourage you to explore these datasets, experiment with the code, and contribute to the collective progress in document AI.
We're on a mission to enhance document AI capabilities, and these datasets are just the beginning. With your engagement and innovation, we're confident in the community's ability to develop robust OCR solutions. We encourage you to explore these datasets, experiment with the code, and contribute to the collective progress in document AI.
new_line
new_line
text
For detailed information on usage and licensing, please refer to the dataset cards on the Hugging Face hub.
For detailed information on usage and licensing, please refer to the dataset cards on the Hugging Face hub.
new_line
🚀🚀 Exciting times for the document AI community! 

We're thrilled to announce the release of some of the largest OCR datasets available to the public.
🔥 With over 26 million pages , 18 billion text tokens, and 6TB of data, these resources are a significant leap forward for document AI research.

Here's how to access these datasets quickly:

```
from datasets import load_dataset

pdfa_dataset = load_dataset('pixparse/pdfa-eng-wds', streaming=True)
IDL_dataset = load_dataset('pixparse/idl-wds', streaming=True)
```
This enables you to stream them directly, integrating seamlessly with your projects using the Hugging Face datasets library. On the hub, you can find them here:

https://huggingface.co/datasets/pixparse/pdfa-eng-wds
https://huggingface.co/datasets/pixparse/idl-wds 

For lean data loading, the new [chug](https://github.com/huggingface/chug) library offers a solution with pdf decoding:


```
import chug

task_cfg = chug.DataTaskDocReadCfg(
    page_sampling='all',
)
data_cfg = chug.DataCfg(
    source='pixparse/pdfa-eng-wds',
    split='train',
    batch_size=None,
    format='hfids',
    num_workers=0,
)
data_loader = chug.create_loader(
    data_cfg,
    task_cfg,
)
sample = next(iter(data_loader))
```


We owe a huge thank you to Peter Wyatt, Kate Tasker, Rachel Taketa, Ali Furkan Biten, Ruben Tito, and their colleagues for their contributions. Their work putting these datasets together has been invaluable. 🤗

Looking Ahead:

We're on a mission to enhance document AI capabilities, and these datasets are just the beginning. With your engagement and innovation, we're confident in the community's ability to develop robust OCR solutions. We encourage you to explore these datasets, experiment with the code, and contribute to the collective progress in document AI.

For detailed information on usage and licensing, please refer to the dataset cards on the Hugging Face hub.

🔥
abidlabs VictorSanh lhoestq Dlbk thomwolf manishiitg clefourrier IlyasMoutawwakil osanseviero samusenps DmitryRyumin ajibawa-2023 Pclanglais loubnabnl rwightman marioparreno Chunte NeuralNovel AIAdventurist JoshuaGu antfraia mmhamdy callmeraxz qinluo pankaj-munde Zyn123 andrewrreed ABX-AI pcuenq giux78 victor clem tonnirobo mohammedbriman Wauplin julien-c lunarflu Tom-Neverwinter nikgr kramp francisrafal MoritzLaurer aceeee
🚀
rwightman abidlabs VictorSanh lhoestq Dlbk thomwolf IlyasMoutawwakil osanseviero loubnabnl Chunte mmhamdy andrewrreed ABX-AI pcuenq clem Wauplin julien-c lunarflu Tom-Neverwinter
🤗
rwightman abidlabs VictorSanh lhoestq Dlbk thomwolf IlyasMoutawwakil osanseviero loubnabnl Chunte andrewrreed ABX-AI clem Jakaline Wauplin julien-c lunarflu
🤯
lhoestq Dlbk thomwolf IlyasMoutawwakil osanseviero Chunte femboysLover andrewrreed ABX-AI Wauplin julien-c lunarflu prithivMLmods
👍
shiv2050 Chunte ssml2050 visisco andrewrreed ABX-AI Wauplin julien-c lunarflu Tom-Neverwinter
❤️
samusenps Chunte andrewrreed ABX-AI Wauplin julien-c lunarflu Tom-Neverwinter Tonic
https://cdn-avatars.huggingface.co/v1/production/uploads/1667002643224-604a5184dca2c7ac7508b849.jpeg
Ross Wightman
rwightman
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1621947938344-noauth.png
Abubakar Abid
abidlabs
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1642598610696-noauth.jpeg
Ilyas Moutawwakil
IlyasMoutawwakil
user
/avatars/a514f0d2b2f9937dd6fd97560f8319a8.svg
Ryan Chesler
emelryan
user