
# Dataset Card for Dataset Name


This dataset contains historic newspapers from [Europeana](https://pro.europeana.eu/page/iiif#download). In total the collection has ~32 Billion tokens. Documentation for this dataset is a WIP. 



This dataset card aims to be a base template for new datasets. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md?plain=1).

## Dataset Details

### Dataset Description


- **Curated by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]

### Dataset Sources [optional]

<!-- Provide the basic links for the dataset. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the dataset is intended to be used. -->

### Direct Use

To download the full dataset using the `Datasets` library you can do the following

```python
from datasets import load_dataset

dataset = load_dataset("biglam/europeana_newspapers")
```

You can also access a subset based on language or decade ranges using the following function. 

```python
from typing import List, Optional, Literal, Union
from huggingface_hub import hf_hub_url, list_repo_files

LanguageOption = Literal[
    "et",
    "pl",
    "sr",
    "ru",
    "sv",
    "no_language_found",
    "ji",
    "hr",
    "el",
    "uk",
    "fr",
    "fi",
    "de",
    "multi_language",
]


def get_files_for_lang_and_years(
    languages: Union[None, List[LanguageOption]] = None,
    min_year: Optional[int] = None,
    max_year: Optional[int] = None,
):
    files = list_repo_files("biglam/europeana_newspapers", repo_type="dataset")
    parquet_files = [f for f in files if f.endswith(".parquet")]
    parquet_files_filtered_for_lang = [
        f for f in parquet_files if any(lang in f for lang in ["uk", "fr"])
    ]
    filtered_files = [
        f
        for f in parquet_files
        if (min_year is None or min_year <= int(f.split("-")[1].split(".")[0]))
        and (max_year is None or int(f.split("-")[1].split(".")[0]) <= max_year)
    ]
    return [
        hf_hub_url("biglam/europeana_newspapers", f, repo_type="dataset")
        for f in filtered_files
    ]

```

This function takes a list of language codes, and a min, max value for decades you want to include. You can can use this function to get the URLs for files you want to download from the Hub:

```python
ds = load_dataset("parquet", data_files=get_files_for_lang_and_years(['fr']), num_proc=4)
```

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

[More Information Needed]

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

[More Information Needed]

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

[More Information Needed]

#### Who are the source data producers?

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

[More Information Needed]

### Annotations [optional]

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

#### Annotation process

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

[More Information Needed]

#### Who are the annotators?

<!-- This section describes the people or systems who created the annotations. -->

[More Information Needed]

#### Personal and Sensitive Information

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

## Citation [optional]

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

[More Information Needed]

**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed]

## Dataset Card Contact

[More Information Needed]
task_categories:text-generation task_ids:language-modeling annotations_creators:no-annotation language_creators:machine-generated multilinguality:multilingual size_categories:1M<n<10M language:de language:fr language:el language:et language:fi language:hr language:ji language:pl language:ru language:sr language:sv language:uk newspapers lam OCR region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["machine-generated"], "language": ["de", "fr", "el", "et", "fi", "hr", "ji", "pl", "ru", "sr", "sv", "uk"], "multilinguality": ["multilingual"], "size_categories": ["1M<n<10M"], "source_datasets": [], "task_categories": ["text-generation"], "task_ids": ["language-modeling"], "pretty_name": "Europeana Newspapers ", "tags": ["newspapers", "lam", "OCR"]}
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-machine-generated #multilinguality-multilingual #size_categories-1M<n<10M #language-German #language-French #language-Modern Greek (1453-) #language-Estonian #language-Finnish #language-Croatian #language-ji #language-Polish #language-Russian #language-Serbian #language-Swedish #language-Ukrainian #newspapers #lam #OCR #region-us 


# Dataset Card for Dataset Name


This dataset contains historic newspapers from Europeana. In total the collection has ~32 Billion tokens. Documentation for this dataset is a WIP. 



This dataset card aims to be a base template for new datasets. It has been generated using this raw template.

## Dataset Details

### Dataset Description


- Curated by: 
- Funded by [optional]: 
- Shared by [optional]: 
- Language(s) (NLP): 
- License: 

### Dataset Sources [optional]



- Repository: 
- Paper [optional]: 
- Demo [optional]: 

## Uses



### Direct Use

To download the full dataset using the 'Datasets' library you can do the following



You can also access a subset based on language or decade ranges using the following function. 



This function takes a list of language codes, and a min, max value for decades you want to include. You can can use this function to get the URLs for files you want to download from the Hub:





### Out-of-Scope Use





## Dataset Structure





## Dataset Creation

### Curation Rationale





### Source Data



#### Data Collection and Processing





#### Who are the source data producers?





### Annotations [optional]



#### Annotation process





#### Who are the annotators?





#### Personal and Sensitive Information





## Bias, Risks, and Limitations





### Recommendations



Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

[optional]



BibTeX:



APA:



## Glossary [optional]





## More Information [optional]



## Dataset Card Authors [optional]



## Dataset Card Contact


# Dataset Card for Dataset Name


This dataset contains historic newspapers from Europeana. In total the collection has ~32 Billion tokens. Documentation for this dataset is a WIP. 



This dataset card aims to be a base template for new datasets. It has been generated using this raw template. ## Dataset Details ### Dataset Description


- Curated by: 
- Funded by [optional]: 
- Shared by [optional]: 
- Language(s) (NLP): 
- License: ### Dataset Sources [optional]



- Repository: 
- Paper [optional]: 
- Demo [optional]: ## Uses ### Direct Use

To download the full dataset using the 'Datasets' library you can do the following



You can also access a subset based on language or decade ranges using the following function. 



This function takes a list of language codes, and a min, max value for decades you want to include. You can can use this function to get the URLs for files you want to download from the Hub: ### Out-of-Scope Use ## Dataset Structure ## Dataset Creation ### Curation Rationale ### Source Data #### Data Collection and Processing #### Who are the source data producers? ### Annotations [optional] #### Annotation process #### Who are the annotators? #### Personal and Sensitive Information ## Bias, Risks, and Limitations ### Recommendations



Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

[optional]



BibTeX:



APA: ## Glossary [optional] ## More Information [optional] ## Dataset Card Authors [optional] ## Dataset Card Contact
TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-machine-generated #multilinguality-multilingual #size_categories-1M<n<10M #language-German #language-French #language-Modern Greek (1453-) #language-Estonian #language-Finnish #language-Croatian #language-ji #language-Polish #language-Russian #language-Serbian #language-Swedish #language-Ukrainian #newspapers #lam #OCR #region-us 
 # Dataset Card for Dataset Name


This dataset contains historic newspapers from Europeana. In total the collection has ~32 Billion tokens. Documentation for this dataset is a WIP. 



This dataset card aims to be a base template for new datasets. It has been generated using this raw template. ## Dataset Details ### Dataset Description


- Curated by: 
- Funded by [optional]: 
- Shared by [optional]: 
- Language(s) (NLP): 
- License: ### Dataset Sources [optional]



- Repository: 
- Paper [optional]: 
- Demo [optional]: ## Uses ### Direct Use

To download the full dataset using the 'Datasets' library you can do the following



You can also access a subset based on language or decade ranges using the following function. 



This function takes a list of language codes, and a min, max value for decades you want to include. You can can use this function to get the URLs for files you want to download from the Hub: ### Out-of-Scope Use ## Dataset Structure ## Dataset Creation ### Curation Rationale ### Source Data #### Data Collection and Processing #### Who are the source data producers? ### Annotations [optional] #### Annotation process #### Who are the annotators? #### Personal and Sensitive Information ## Bias, Risks, and Limitations ### Recommendations



Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

[optional]



BibTeX:



APA: ## Glossary [optional] ## More Information [optional] ## Dataset Card Authors [optional] ## Dataset Card Contact
passage: TAGS
#task_categories-text-generation #task_ids-language-modeling #annotations_creators-no-annotation #language_creators-machine-generated #multilinguality-multilingual #size_categories-1M<n<10M #language-German #language-French #language-Modern Greek (1453-) #language-Estonian #language-Finnish #language-Croatian #language-ji #language-Polish #language-Russian #language-Serbian #language-Swedish #language-Ukrainian #newspapers #lam #OCR #region-us 
# Dataset Card for Dataset Name


This dataset contains historic newspapers from Europeana. In total the collection has ~32 Billion tokens. Documentation for this dataset is a WIP. 



This dataset card aims to be a base template for new datasets. It has been generated using this raw template.## Dataset Details### Dataset Description


- Curated by: 
- Funded by [optional]: 
- Shared by [optional]: 
- Language(s) (NLP): 
- License:### Dataset Sources [optional]



- Repository: 
- Paper [optional]: 
- Demo [optional]:## Uses### Direct Use

To download the full dataset using the 'Datasets' library you can do the following



You can also access a subset based on language or decade ranges using the following function. 



This function takes a list of language codes, and a min, max value for decades you want to include. You can can use this function to get the URLs for files you want to download from the Hub:### Out-of-Scope Use## Dataset Structure## Dataset Creation### Curation Rationale### Source Data#### Data Collection and Processing#### Who are the source data producers?### Annotations [optional]#### Annotation process#### Who are the annotators?#### Personal and Sensitive Information## Bias, Risks, and Limitations