# Dataset Card for Heritage Made Digital Newspapers

## Table of Contents
- [Dataset Card for Heritage Made Digital Newspapers](#dataset-card-for-heritage-made-digital-newspapers)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
    - [Languages](#languages)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:**
- **Repository:** https://bl.iro.bl.uk/?locale=en
- **Paper:**
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

This dataset contains text extracted at the article level from historic digitised newspapers from the [Heritage Made Digital](https://bl.iro.bl.uk/collections/9a6a4cdd-2bfe-47bb-8c14-c0a5d100501f?locale=en) newspaper digitisation program at the [British Library](https://www.bl.uk/). The newspapers in the dataset were published between 1800 and 1896. This dataset contains ~2.5 billion tokens and 3,065,408 articles. 

The dataset contains text generated from Optical Character Recognition software on digitised newspaper pages. This dataset includes the plain text from the OCR alongside some minimal metadata associated with the newspaper from which the text is derived and OCR confidence score information generated from the OCR software. 



### Supported Tasks and Leaderboards

This dataset can be used for:
- historical research and digital humanities research
- training language models
- training historic language models

Whilst this dataset can be used for all of these tasks, it is important to understand that the dataset was not constructed in a representative way so it contains biases in terms of the newspapers and articles that are included (more on this below). 

### Languages

The text in this dataset is in English that has been recognised by the OCR software. The OCR software used is generic commercial OCR software that has not been trained on historic newspapers. There are therefore many errors in the text. Some of the OCR in this text will be of such poor quality that is is incomprehensible to a human reader.

## Dataset Structure

### Data Instances

Each row in the dataset is an article from a newspaper as recognised by an OLR (Optical Layout Recognition) step in the digitisation process.

### Data Splits

There is one split in this dataset, the training split.

## Dataset Creation

### Curation Rationale

This dataset consists of public-domain newspapers published in the UK during the 19th Century. The majority of newspapers digitised in the UK are not freely available (even if they are out of copyright). The newspapers in this dataset were digitised specifically to be freely available but also to meet preservation goals for newspapers in poor condition. As a result, the newspapers chosen for digitisation are biased toward poor quality physical newspapers. This may in turn result in worse OCR.

### Source Data

The source data for this dataset is the digitised newspapers from the [Heritage Made Digital](https://bl.iro.bl.uk/collections/9a6a4cdd-2bfe-47bb-8c14-c0a5d100501f?locale=en) newspaper digitisation program. The newspapers in the dataset were published between 1800 and 1870.

### Dataset Curators

The original digitisation was carried out by the British Library. The dataset was created by the British Library in partnership with Findmypast. 

This dataset was created by [@davanstrien](https://huggingface.co/davanstrien). 

### Licensing Information

The newspapers in this dataset are in the public domain. The dataset is licensed under a [Creative Commons Zero v1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/) license.

### Citation Information

[More Information Needed]

### Contributions

Thanks to [@github-username](https://github.com/<github-username>) for adding this dataset.

task_categories:text-generation size_categories:1M<n<10M language:en license:cc0-1.0 newspapers region:us
{"language": ["en"], "license": "cc0-1.0", "size_categories": ["1M<n<10M"], "task_categories": ["text-generation"], "pretty_name": "Heritage Made Digital Newspapers", "dataset_info": {"features": [{"name": "source", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "location", "dtype": "string"}, {"name": "date", "dtype": "timestamp[s]"}, {"name": "item_type", "dtype": "string"}, {"name": "word_count", "dtype": "int32"}, {"name": "ocr_quality_mean", "dtype": "float64"}, {"name": "ocr_quality_sd", "dtype": "float64"}, {"name": "text", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 14304741164, "num_examples": 3065408}], "download_size": 9682476047, "dataset_size": 14304741164}, "configs": [{"config_name": "default", "data_files": [{"split": "train", "path": "data/train-*"}]}], "tags": ["newspapers"]}
TAGS
#task_categories-text-generation #size_categories-1M<n<10M #language-English #license-cc0-1.0 #newspapers #region-us 

# Dataset Card for Heritage Made Digital Newspapers

## Table of Contents
- Dataset Card for Heritage Made Digital Newspapers
  - Table of Contents
  - Dataset Description
    - Dataset Summary
    - Supported Tasks and Leaderboards
    - Languages
  - Dataset Structure
    - Data Instances
    - Data Splits
  - Dataset Creation
    - Curation Rationale
    - Source Data
    - Dataset Curators
    - Licensing Information
    - Citation Information
    - Contributions

## Dataset Description

- Homepage:
- Repository: URL
- Paper:
- Leaderboard:
- Point of Contact:

### Dataset Summary

This dataset contains text extracted at the article level from historic digitised newspapers from the Heritage Made Digital newspaper digitisation program at the British Library. The newspapers in the dataset were published between 1800 and 1896. This dataset contains ~2.5 billion tokens and 3,065,408 articles. 

The dataset contains text generated from Optical Character Recognition software on digitised newspaper pages. This dataset includes the plain text from the OCR alongside some minimal metadata associated with the newspaper from which the text is derived and OCR confidence score information generated from the OCR software. 



### Supported Tasks and Leaderboards

This dataset can be used for:
- historical research and digital humanities research
- training language models
- training historic language models

Whilst this dataset can be used for all of these tasks, it is important to understand that the dataset was not constructed in a representative way so it contains biases in terms of the newspapers and articles that are included (more on this below). 

### Languages

The text in this dataset is in English that has been recognised by the OCR software. The OCR software used is generic commercial OCR software that has not been trained on historic newspapers. There are therefore many errors in the text. Some of the OCR in this text will be of such poor quality that is is incomprehensible to a human reader.

## Dataset Structure

### Data Instances

Each row in the dataset is an article from a newspaper as recognised by an OLR (Optical Layout Recognition) step in the digitisation process.

### Data Splits

There is one split in this dataset, the training split.

## Dataset Creation

### Curation Rationale

This dataset consists of public-domain newspapers published in the UK during the 19th Century. The majority of newspapers digitised in the UK are not freely available (even if they are out of copyright). The newspapers in this dataset were digitised specifically to be freely available but also to meet preservation goals for newspapers in poor condition. As a result, the newspapers chosen for digitisation are biased toward poor quality physical newspapers. This may in turn result in worse OCR.

### Source Data

The source data for this dataset is the digitised newspapers from the Heritage Made Digital newspaper digitisation program. The newspapers in the dataset were published between 1800 and 1870.

### Dataset Curators

The original digitisation was carried out by the British Library. The dataset was created by the British Library in partnership with Findmypast. 

This dataset was created by @davanstrien. 

### Licensing Information

The newspapers in this dataset are in the public domain. The dataset is licensed under a Creative Commons Zero v1.0 Universal license.





### Contributions

Thanks to @github-username for adding this dataset.

# Dataset Card for Heritage Made Digital Newspapers ## Table of Contents
- Dataset Card for Heritage Made Digital Newspapers
  - Table of Contents
  - Dataset Description
    - Dataset Summary
    - Supported Tasks and Leaderboards
    - Languages
  - Dataset Structure
    - Data Instances
    - Data Splits
  - Dataset Creation
    - Curation Rationale
    - Source Data
    - Dataset Curators
    - Licensing Information
    - Citation Information
    - Contributions ## Dataset Description

- Homepage:
- Repository: URL
- Paper:
- Leaderboard:
- Point of Contact: ### Dataset Summary

This dataset contains text extracted at the article level from historic digitised newspapers from the Heritage Made Digital newspaper digitisation program at the British Library. The newspapers in the dataset were published between 1800 and 1896. This dataset contains ~2.5 billion tokens and 3,065,408 articles. 

The dataset contains text generated from Optical Character Recognition software on digitised newspaper pages. This dataset includes the plain text from the OCR alongside some minimal metadata associated with the newspaper from which the text is derived and OCR confidence score information generated from the OCR software. ### Supported Tasks and Leaderboards

This dataset can be used for:
- historical research and digital humanities research
- training language models
- training historic language models

Whilst this dataset can be used for all of these tasks, it is important to understand that the dataset was not constructed in a representative way so it contains biases in terms of the newspapers and articles that are included (more on this below). ### Languages

The text in this dataset is in English that has been recognised by the OCR software. The OCR software used is generic commercial OCR software that has not been trained on historic newspapers. There are therefore many errors in the text. Some of the OCR in this text will be of such poor quality that is is incomprehensible to a human reader. ## Dataset Structure ### Data Instances

Each row in the dataset is an article from a newspaper as recognised by an OLR (Optical Layout Recognition) step in the digitisation process. ### Data Splits

There is one split in this dataset, the training split. ## Dataset Creation ### Curation Rationale

This dataset consists of public-domain newspapers published in the UK during the 19th Century. The majority of newspapers digitised in the UK are not freely available (even if they are out of copyright). The newspapers in this dataset were digitised specifically to be freely available but also to meet preservation goals for newspapers in poor condition. As a result, the newspapers chosen for digitisation are biased toward poor quality physical newspapers. This may in turn result in worse OCR. ### Source Data

The source data for this dataset is the digitised newspapers from the Heritage Made Digital newspaper digitisation program. The newspapers in the dataset were published between 1800 and 1870. ### Dataset Curators

The original digitisation was carried out by the British Library. The dataset was created by the British Library in partnership with Findmypast. 

This dataset was created by @davanstrien. ### Licensing Information

The newspapers in this dataset are in the public domain. The dataset is licensed under a Creative Commons Zero v1.0 Universal license. ### Contributions

Thanks to @github-username for adding this dataset.
TAGS
#task_categories-text-generation #size_categories-1M<n<10M #language-English #license-cc0-1.0 #newspapers #region-us 
 # Dataset Card for Heritage Made Digital Newspapers ## Table of Contents
- Dataset Card for Heritage Made Digital Newspapers
  - Table of Contents
  - Dataset Description
    - Dataset Summary
    - Supported Tasks and Leaderboards
    - Languages
  - Dataset Structure
    - Data Instances
    - Data Splits
  - Dataset Creation
    - Curation Rationale
    - Source Data
    - Dataset Curators
    - Licensing Information
    - Citation Information
    - Contributions ## Dataset Description

- Homepage:
- Repository: URL
- Paper:
- Leaderboard:
- Point of Contact: ### Dataset Summary

This dataset contains text extracted at the article level from historic digitised newspapers from the Heritage Made Digital newspaper digitisation program at the British Library. The newspapers in the dataset were published between 1800 and 1896. This dataset contains ~2.5 billion tokens and 3,065,408 articles. 

The dataset contains text generated from Optical Character Recognition software on digitised newspaper pages. This dataset includes the plain text from the OCR alongside some minimal metadata associated with the newspaper from which the text is derived and OCR confidence score information generated from the OCR software. ### Supported Tasks and Leaderboards

This dataset can be used for:
- historical research and digital humanities research
- training language models
- training historic language models

Whilst this dataset can be used for all of these tasks, it is important to understand that the dataset was not constructed in a representative way so it contains biases in terms of the newspapers and articles that are included (more on this below). ### Languages

The text in this dataset is in English that has been recognised by the OCR software. The OCR software used is generic commercial OCR software that has not been trained on historic newspapers. There are therefore many errors in the text. Some of the OCR in this text will be of such poor quality that is is incomprehensible to a human reader. ## Dataset Structure ### Data Instances

Each row in the dataset is an article from a newspaper as recognised by an OLR (Optical Layout Recognition) step in the digitisation process. ### Data Splits

There is one split in this dataset, the training split. ## Dataset Creation ### Curation Rationale

This dataset consists of public-domain newspapers published in the UK during the 19th Century. The majority of newspapers digitised in the UK are not freely available (even if they are out of copyright). The newspapers in this dataset were digitised specifically to be freely available but also to meet preservation goals for newspapers in poor condition. As a result, the newspapers chosen for digitisation are biased toward poor quality physical newspapers. This may in turn result in worse OCR. ### Source Data

The source data for this dataset is the digitised newspapers from the Heritage Made Digital newspaper digitisation program. The newspapers in the dataset were published between 1800 and 1870. ### Dataset Curators

The original digitisation was carried out by the British Library. The dataset was created by the British Library in partnership with Findmypast. 

This dataset was created by @davanstrien. ### Licensing Information

The newspapers in this dataset are in the public domain. The dataset is licensed under a Creative Commons Zero v1.0 Universal license. ### Contributions

Thanks to @github-username for adding this dataset.