
# Dataset Card for British Library Books

## Table of Contents

- [Dataset Card for British Library Books](#dataset-card-for-British-Library-Books)
  - [Table of Contents](#table-of-contents)
  - [Dataset Description](#dataset-description)
    - [Dataset Summary](#dataset-summary)
    - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
      - [Language model training](#language-model-training)
      - [Supervised tasks](#supervised-tasks)
    - [Languages](#languages)
      - [Language change](#language-change)
      - [Optical Character Recognition](#optical-character-recognition)
        - [OCR word confidence](#ocr-word-confidence)
  - [Dataset Structure](#dataset-structure)
    - [Data Instances](#data-instances)
    - [Data Fields](#data-fields)
    - [Data Splits](#data-splits)
  - [Dataset Creation](#dataset-creation)
    - [Curation Rationale](#curation-rationale)
    - [Source Data](#source-data)
      - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization)
        - [Date normalization](#date-normalization)
        - [Metadata included](#metadata-included)
      - [Who are the source language producers?](#who-are-the-source-language-producers)
    - [Annotations](#annotations)
      - [Annotation process](#annotation-process)
      - [Who are the annotators?](#who-are-the-annotators)
    - [Personal and Sensitive Information](#personal-and-sensitive-information)
  - [Considerations for Using the Data](#considerations-for-using-the-data)
    - [Social Impact of Dataset](#social-impact-of-dataset)
    - [Discussion of Biases](#discussion-of-biases)
      - [Colonialism](#colonialism)
    - [Other Known Limitations](#other-known-limitations)
  - [Additional Information](#additional-information)
    - [Dataset Curators](#dataset-curators)
    - [Licensing Information](#licensing-information)
    - [Citation Information](#citation-information)
    - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://www.bl.uk/collection-guides/digitised-printed-books
- **Repository:** https://doi.org/10.21250/db14
- **Paper:**
- **Leaderboard:**
- **Point of Contact:** labs@bl.uk

### Dataset Summary

This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were published in the 18th and 19th Century, but the collection also consists of a smaller number of books from earlier periods. Items within this collection cover a wide range of subject areas, including geography, philosophy, history, poetry and literature and are published in various languages.

While the books are predominately from the 18th and 19th Centuries, there are fewer books from earlier periods. The number of pages in the corpus by decade:

|      | page count |
| ---- | ---------- |
| 1510 | 94         |
| 1520 | 32         |
| 1540 | 184        |
| 1550 | 16         |
| 1580 | 276        |
| 1590 | 540        |
| 1600 | 1117       |
| 1610 | 1132       |
| 1620 | 1856       |
| 1630 | 9274       |
| 1640 | 4232       |
| 1650 | 2944       |
| 1660 | 5858       |
| 1670 | 11415      |
| 1680 | 8348       |
| 1690 | 13756      |
| 1700 | 10160      |
| 1710 | 9556       |
| 1720 | 10314      |
| 1730 | 13282      |
| 1740 | 10778      |
| 1750 | 12001      |
| 1760 | 21415      |
| 1770 | 28490      |
| 1780 | 32676      |
| 1790 | 50014      |
| 1800 | 307806     |
| 1810 | 478008     |
| 1820 | 589419     |
| 1830 | 681212     |
| 1840 | 1113473    |
| 1850 | 1726108    |
| 1860 | 1725407    |
| 1870 | 2069089    |
| 1880 | 2585159    |
| 1890 | 3365031    |

[More Information Needed]

### Supported Tasks and Leaderboards

This collection has been previously used across various digital history and humanities projects since being published.

The dataset consists of text and a range of metadata associated with this text. This metadata includes:

- date of publication
- place of publication
- country of publication
- language
- OCR quality
- physical description of the original physical item

#### Language model training

As a relatively large dataset, `blbooks` provides a source dataset for training language models. The presence of this metadata also offers interesting opportunities to use this dataset as a source for training language models based on:

- specific time-periods
- specific languages
- certain OCR quality thresholds

The above is not an exhaustive list but offer some suggestions of how the dataset can be used to explore topics such as the impact of OCR quality on language models, the ‘transferability’ of language models across time or the impact of training multilingual language models on historical languages.

#### Supervised tasks

Whilst this dataset does not have annotations for a specific NLP task, such as Named Entity Recognition, it does include a wide variety of metadata. This metadata has the potential to be used for training and/or evaluating a variety of supervised tasks predicting this metadata.

### Languages

This dataset consists of books published in several languages. The breakdown of the languages included (at the page level) is:

| Language              | Pages    |
| --------------------- | -------- |
| English               | 10039463 |
| French                | 1442929  |
| German                | 1172793  |
| Spanish               | 286778   |
| Italian               | 214255   |
| Dutch                 | 204759   |
| Russian               | 193347   |
| Danish                | 93366    |
| Hungarian             | 88094    |
| Swedish               | 76225    |
| Polish                | 58901    |
| Greek, Modern (1453-) | 26104    |
| Latin                 | 25611    |
| Portuguese            | 25410    |
| Czech                 | 20160    |
| Bulgarian             | 7891     |
| Finnish               | 5677     |
| Irish                 | 2743     |
| Serbian               | 1975     |
| Romanian              | 1544     |
| Norwegian Nynorsk     | 1398     |
| Croatian              | 1306     |
| Norwegian             | 1227     |
| Icelandic             | 902      |
| Slovak                | 840      |
| Lithuanian            | 714      |
| Welsh                 | 580      |
| Slovenian             | 545      |
| Indonesian            | 418      |
| Cornish               | 223      |

This breakdown was derived from the first language in the associated metadata field. Some books include multiple languages. Some of the languages codes for this data were also derived using computational methods. Therefore, the language fields in the dataset should be treated with some caution (discussed in more detail below).

#### Language change

The publication dates of books in the data cover a broad period of time (1500-1900). For languages in the dataset with broad temporal coverage, significant [language change](https://en.wikipedia.org/wiki/Language_change) might be found. The ability to study this change by taking reasonably large samples of languages covering different time periods is one of the opportunities offered by this dataset. The fact that the text in this dataset was produced via Optical Character Recognition (OCR) causes some challenges for this type of research (see below).

#### Optical Character Recognition

The digitised books in this collection were transformed into machine-readable text using Optical Character Recognition (OCR) software. The text produced via OCR software will usually include some errors. These errors include; mistakes at the character level; for example, an `i` is mistaken for an `l`, at the word level or across significant passages of text.

The books in this dataset can pose some additional challenges for OCR software. OCR errors can stem from:

- the quality of the original printing: printing technology was a developing technology during the time period covered by this corpus; some of the original book text will include misprints, blurred or faded ink that is hard to read
- damage to the page: some of the books will have become damaged over time, this can obscure all or parts of the text on a page
- poor quality scans: scanning books can be challenging; for example, if the book has tight bindings, it can be hard to capture text that has fallen into the [gutter](https://www.abaa.org/glossary/entry/gutter) of the book.
- the language used in the books may differ from the languages OCR software is predominantly trained to recognise.

##### OCR word confidence

Many OCR engines produce some form of confidence score alongside the predicted text. These confidence scores are usually at the character or word level. The word confidence score was given for each word in the original ALTO XML versions of the text in this dataset in this dataset. The OCR confidence scores should be treated with some scepticism. For historical text or in a lower resource language, for example, a low confidence score may be more likely for words not included in a modern dictionary but may be accurate transcriptions of the original text. With that said, the confidence scores do give some sense of the OCR quality.

An example of text with a high (over 90% mean word confidence score):

```
8 direction to the Conduit, round which is a wide open space, and a good broad pavement called the Parade. It commands a pleasant peep of the slopes and terrace throughout its entire length. The street continuing from the Conduit, in the same general direction, was known anciently as Lodborne Lane, and is now named South Street. From the Conduit two other streets, at right angles to these, are Long Street, leading Eastwards, and Half-Moon Street (formerly Lodborne), leading to Westbury, Trendle Street, and the Horsecastles Road.
```

An example of text with a score below 40%:

```
Hannover. Schrift und Druck von Fr. CultniTmn,',
 "LeMNs'utluirui.",
 'ü 8u«llim» M^äalßwi 01de!lop 1<M.',
 'p^dnalmw vom Xr^u/e, lpiti>»**Kmm lie« !»^2!M kleine lii!<! (,«>* ttünee!<»e^ v»n tndzt Lievclum, 1872,
```

The quality of OCR - as measured by mean OCR confidence for a page - across the dataset correlates with other features. A groupby of publication decade and mean word confidence:

| decade | mean_wc_ocr |
| ------ | ----------- |
| 1510   | 0.499151    |
| 1520   | 0.544818    |
| 1540   | 0.511589    |
| 1550   | 0.4505      |
| 1580   | 0.321858    |
| 1590   | 0.461282    |
| 1600   | 0.467318    |
| 1610   | 0.495895    |
| 1620   | 0.501257    |
| 1630   | 0.49766     |
| 1640   | 0.512095    |
| 1650   | 0.528534    |
| 1660   | 0.521014    |
| 1670   | 0.592575    |
| 1680   | 0.583901    |
| 1690   | 0.567202    |
| 1700   | 0.575175    |
| 1710   | 0.61436     |
| 1720   | 0.627725    |
| 1730   | 0.658534    |
| 1740   | 0.64214     |
| 1750   | 0.657357    |
| 1760   | 0.6389      |
| 1770   | 0.651883    |
| 1780   | 0.632326    |
| 1790   | 0.664279    |
| 1800   | 0.682338    |
| 1810   | 0.708915    |
| 1820   | 0.730015    |
| 1830   | 0.730973    |
| 1840   | 0.713886    |
| 1850   | 0.697106    |
| 1860   | 0.696701    |
| 1870   | 0.717233    |
| 1880   | 0.733331    |
| 1890   | 0.762364    |

As might be expected, the earlier periods have lower mean word confidence scores. Again, all of this should be treated with some scepticism, especially as the size of the data grows over time.

As with time, the mean word confidence of the OCR software varies across languages:

| Language_1            | mean_wc_ocr |
| --------------------- | ----------- |
| Croatian              | 0.755565    |
| Welsh                 | 0.7528      |
| Norwegian Nynorsk     | 0.751648    |
| Slovenian             | 0.746007    |
| French                | 0.740772    |
| Finnish               | 0.738032    |
| Czech                 | 0.737849    |
| Hungarian             | 0.736076    |
| Dutch                 | 0.734977    |
| Cornish               | 0.733682    |
| Danish                | 0.733106    |
| English               | 0.733037    |
| Irish                 | 0.732658    |
| Portuguese            | 0.727746    |
| Spanish               | 0.725111    |
| Icelandic             | 0.724427    |
| Italian               | 0.715839    |
| Swedish               | 0.715633    |
| Polish                | 0.715133    |
| Lithuanian            | 0.700003    |
| Bulgarian             | 0.694657    |
| Romanian              | 0.692957    |
| Latin                 | 0.689022    |
| Russian               | 0.685847    |
| Serbian               | 0.674329    |
| Slovak                | 0.66739     |
| Greek, Modern (1453-) | 0.632195    |
| German                | 0.631457    |
| Indonesian            | 0.6155      |
| Norwegian             | 0.597987    |

Again, these numbers should be treated sceptically since some languages appear very infrequently. For example, the above table suggests the mean word confidence for Welsh is relatively high. However, there isn’t much Welsh in the dataset. Therefore, it is unlikely that this data will be particularly useful for training (historic) Welsh language models.

[More Information Needed]

## Dataset Structure

The dataset has a number of configurations relating to the different dates of publication in the underlying data:

- `1500_1899`: this configuration covers all years
- `1800_1899`: this configuration covers the years between 1800 and 1899
- `1700_1799`: this configuration covers the years between 1700 and 1799
- `1510_1699`: this configuration covers the years between 1510 and 1699

### Configuration option

All of the configurations have an optional keyword argument `skip_empty_pages` which is set to `True` by default. The underlying dataset includes some pages where there is no text. This could either be because the underlying book page didn't have any text or the OCR software failed to detect this text.

For many uses of this dataset it doesn't make sense to include empty pages so these are skipped by default. However, for some uses you may prefer to retain a representation of the data that includes these empty pages. Passing `skip_empty_pages=False` when loading the dataset will enable this option.

### Data Instances

An example data instance:

```python
{'Country of publication 1': 'England',
'Language_1': 'English',
'Language_2': None,
'Language_3': None,
'Language_4': None,
'Physical description': None,
'Publisher': None,
'all Countries of publication': 'England',
'all names': 'Settle, Elkanah [person]',
'date': 1689,
'empty_pg': True,
'mean_wc_ocr': 0.0,
'multi_language': False,
'name': 'Settle, Elkanah',
'pg': 1,
'place': 'London',
'raw_date': '1689',
'record_id': '001876770',
'std_wc_ocr': 0.0,
'text': None,
‘title’: ‘The Female Prelate: being the history and the life and death of Pope Joan. A tragedy [in five acts and in verse] . Written by a Person of Quality [i.e. Elkanah Settle]’}

```

Each instance in the dataset represents a single page from an original digitised book.

### Data Fields

Included in this dataset are:

| Field                        | Data Type | Description                                                                                                   |
| ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------- |
| record_id                    | string    | British Library ID for the item                                                                               |
| date                         | int       | parsed/normalised year for the item. i.e. 1850                                                                |
| raw_date                     | string    | the original raw date for an item i.e. 1850-                                                                  |
| title                        | string    | title of the book                                                                                             |
| place                        | string    | Place of publication, i.e. London                                                                             |
| empty_pg                     | bool      | whether page contains text                                                                                    |
| text                         | string    | OCR generated text for a page                                                                                 |
| pg                           | int       | page in original book the instance refers to                                                                  |
| mean_wc_ocr                  | float     | mean word confidence values for the page                                                                      |
| std_wc_ocr                   | float     | standard deviation of the word confidence values for the page                                                 |
| name                         | string    | name associated with the item (usually author)                                                                |
| all names                    | string    | all names associated with a publication                                                                       |
| Publisher                    | string    | publisher of the book                                                                                         |
| Country of publication 1     | string    | first country associated with publication                                                                     |
| all Countries of publication | string    | all countries associated with a publication                                                                   |
| Physical description         | string    | physical description of the item (size). This requires some normalisation before use and isn’t always present |
| Language_1                   | string    | first language associated with the book, this is usually present                                              |
| Language_2                   | string    |                                                                                                               |
| Language_3                   | string    |                                                                                                               |
| Language_4                   | string    |                                                                                                               |
| multi_language               | bool      |                                                                                                               |

Some of these fields are not populated a large proportion of the time. You can get some sense of this from this [Pandas Profiling](https://github.com/pandas-profiling/pandas-profiling) [report](https://davanstrien.github.io/BL-datasets-pandas-profile-reports/pandas_profile_report_MS_digitised_books_2021-01-09.html)

The majority of these fields relate to metadata about the books. Most of these fields were created by staff working for the British Library. The notable exception is the “Languages” fields that have sometimes been determined using computational methods. This work is reported in more detail in [Automated Language Identification of Bibliographic Resources](https://doi.org/10.1080/01639374.2019.1700201). It is important to note that metadata is neither perfect nor static. The metadata associated with this book was generated based on export from the British Library catalogue in 2021.

[More Information Needed]

### Data Splits

This dataset contains a single split `train`.

## Dataset Creation

**Note** this section is a work in progress.

### Curation Rationale

The books in this collection were digitised as part of a project partnership between the British Library and Microsoft. [Mass digitisation](https://en.wikipedia.org/wiki/Category:Mass_digitization), i.e. projects intending to quickly digitise large volumes of materials shape the selection of materials to include in several ways. Some considerations which are often involved in the decision of whether to include items for digitisation include (but are not limited to):

- copyright status
- preservation needs
- the size of an item, very large and very small items are often hard to digitise quickly

These criteria can have knock-on effects on the makeup of a collection. For example, systematically excluding large books may result in some types of book content not being digitised. Large volumes are likely to be correlated to content to at least some extent, so excluding them from digitisation will mean that material is underrepresented. Similarly, copyright status is often (but not only) determined by publication date. This can often lead to a rapid fall in the number of items in a collection after a certain cut-off date.

All of the above is largely to make clear that this collection was not curated to create a representative sample of the British Library’s holdings. Some material will be over-represented, and others under-represented. Similarly, the collection should not be considered a representative sample of what was published across the period covered by the dataset (nor that the relative proportions of the data for each time period represent a proportional sample of publications from that period). Finally, and this probably does not need stating, the language included in the text should not be considered representative of either written or spoken language(s) from that time period.

[More Information Needed]

### Source Data

The source data (physical items) includes a variety of resources (predominantly monographs) held by the [British Library](bl.uk/](https://bl.uk/). The British Library is a [Legal Deposit](https://www.bl.uk/legal-deposit/about-legal-deposit) library. “Legal deposit requires publishers to provide a copy of every work they publish in the UK to the British Library. It’s existed in English law since 1662.” [source](https://www.bl.uk/legal-deposit/about-legal-deposit).

The source data for this version of the data is derived from the original ALTO XML files and a recent metadata export #TODO add links

[More Information Needed]

#### Initial Data Collection and Normalization

This version of the dataset was created using the original ALTO XML files and, where a match was found, updating the metadata associated with that item with more recent metadata using an export from the British Library catalogue. The process of creating this new dataset is documented here #TODO add link.

There are a few decisions made in the above processing steps worth highlighting in particular:

##### Date normalization

The metadata around date of publication for an item is not always exact. It often is represented as a date range e.g. `1850-1860`. The `date` field above takes steps to normalise this date to a single integer value. In most cases, this is taking the mean of the values associated with the item. The `raw_date` field includes the unprocessed date string.

##### Metadata included

The metadata associated with each item includes most of the fields available via the ALTO XML. However, the data doesn’t include some metadata fields from the metadata export file. The reason fields were excluded because they are frequently not populated. A cut off of 50% was chosen, i.e. values from the metadata which are missing above 50% of the time were not included. This is slightly arbitrary, but since the aim of this version of the data was to support computational research using the collection it was felt that these fields with frequent missing values would be less valuable.

#### Who are the source language producers?

[More Information Needed]

### Annotations

This dataset does not include annotations as usually understood in the context of NLP. The data does include metadata associated with the books.

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

There a range of considerations around using the data. These include the representativeness of the dataset, the OCR quality and the language used. Depending on your use case, these may be more or less important. For example, the impact of OCR quality on downstream tasks will depend on the target task. It may also be possible to mitigate this negative impact from OCR through tokenizer choice, Language Model training objectives, oversampling high-quality OCR, etc.

[More Information Needed]

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

The text in this collection is derived from historical text. As a result, the text will reflect this time period's social beliefs and attitudes. The books include both fiction and non-fiction books.

Examples of book titles that appear in the data (these are randomly sampled from all titles):

- ‘Rhymes and Dreams, Legends of Pendle Forest, and other poems’,
- “Précis of Information concerning the Zulu Country, with a map. Prepared in the Intelligence Branch of the Quarter-Master-General’s Department, Horse Guards, War Office, etc”,
- ‘The fan. A poem’,
- ‘Grif; a story of Australian Life’,
- ‘Calypso; a masque: in three acts, etc’,
- ‘Tales Uncle told [With illustrative woodcuts.]’,
- 'Questings',
- 'Home Life on an Ostrich Farm. With ... illustrations’,
- ‘Bulgarya i Bulgarowie’,
- 'Εἰς τα βαθη της Ἀφρικης [In darkest Africa.] ... Μεταφρασις Γεωρ. Σ. Βουτσινα, etc',
- ‘The Corsair, a tale’,
  ‘Poems ... With notes [With a portrait.]’,
- ‘Report of the Librarian for the year 1898 (1899, 1901, 1909)’,
- “The World of Thought. A novel. By the author of ‘Before I began to speak.’”,
- 'Amleto; tragedia ... recata in versi italiani da M. Leoni, etc']

While using titles alone is insufficient to integrate bias in this collection, it gives some insight into the topics covered by books. Further, the tiles highlight some particular types of bias we might find in the collection. This should in no way be considered an exhaustive list.

#### Colonialism

Even in the above random sample of titles examples of colonial attitudes, we can see examples of titles. We can try and interrogate this further by searching for the name of places that were part of the British Empire when many of these books were published.

Searching for the string `India` in the titles and randomly sampling 10 titles returns:

- “Travels in India in the Seventeenth Century: by Sir Thomas Roe and Dr. John Fryer. Reprinted from the ‘Calcutta Weekly Englishman.’”,
- ‘A Winter in India and Malaysia among the Methodist Missions’,
- “The Tourist’s Guide to all the principal stations on the railways of Northern India [By W. W.] ... Fifth edition”,
- ‘Records of Sport and Military Life in Western India ... With an introduction by ... G. B. Malleson’,
- "Lakhmi, the Rájpút's Bride. A tale of Gujarát in Western India [A poem.]”,
- ‘The West India Commonplace Book: compiled from parliamentary and official documents; shewing the interest of Great Britain in its Sugar Colonies’,
- “From Tonkin to India : by the sources of the Irawadi, January’ 95-January ’96”,
- ‘Case of the Ameers of Sinde : speeches of Mr. John Sullivan, and Captain William Eastwick, at a special court held at the India House, ... 26th January, 1844’,
- ‘The Andaman Islands; their colonisation, etc. A correspondence addressed to the India Office’,
- ‘Ancient India as described by Ptolemy; being a translation of the chapters which describe India and Eastern Asia in the treatise on Geography written by Klaudios Ptolemaios ... with introduction, commentary, map of India according to Ptolemy, and ... index, by J. W. McCrindle’]

Searching form the string `Africa` in the titles and randomly sampling 10 titles returns:

- ['De Benguella ás Terras de Iácca. Descripção de uma viagem na Africa Central e Occidental ... Expedição organisada nos annos de 1877-1880. Edição illustrada',
- ‘To the New Geographical Society of Edinburgh [An address on Africa by H. M. Stanley.]’,
- ‘Diamonds and Gold in South Africa ... With maps, etc’,
- ‘Missionary Travels and Researches in South Africa ... With notes by F. S. Arnot. With map and illustrations. New edition’,
- ‘A Narrative of a Visit to the Mauritius and South Africa ... Illustrated by two maps, sixteen etchings and twenty-eight wood-cuts’,
- ‘Side Lights on South Africa ... With a map, etc’,
- ‘My Second Journey through Equatorial Africa ... in ... 1886 and 1887 ... Translated ... by M. J. A. Bergmann. With a map ... and ... illustrations, etc’,
- ‘Missionary Travels and Researches in South Africa ... With portrait and fullpage illustrations’,
- ‘[African sketches.] Narrative of a residence in South Africa ... A new edition. To which is prefixed a biographical sketch of the author by J. Conder’,
- ‘Lake Ngami; or, Explorations and discoveries during four years wandering in the wilds of South Western Africa ... With a map, and numerous illustrations, etc’]

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

The books are licensed under the [CC Public Domain Mark 1.0](https://creativecommons.org/publicdomain/mark/1.0/) license.

### Citation Information

```bibtext
@misc{bBritishLibraryBooks2021,
  author = {British Library Labs},
  title = {Digitised Books. c. 1510 - c. 1900. JSONL (OCR derived text + metadata)},
  year = {2021},
  publisher = {British Library},
  howpublished={https://doi.org/10.23636/r7w6-zy15}

```

### Contributions

Thanks to [@davanstrien](https://github.com/davanstrien) for adding this dataset.
task_categories:text-generation task_categories:fill-mask task_categories:other task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:machine-generated multilinguality:multilingual size_categories:100K<n<1M source_datasets:original language:de language:en language:es language:fr language:it language:nl license:cc0-1.0 digital-humanities-research region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["machine-generated"], "language": ["de", "en", "es", "fr", "it", "nl"], "license": ["cc0-1.0"], "multilinguality": ["multilingual"], "size_categories": ["100K<n<1M"], "source_datasets": ["original"], "task_categories": ["text-generation", "fill-mask", "other"], "task_ids": ["language-modeling", "masked-language-modeling"], "pretty_name": "British Library Books", "tags": ["digital-humanities-research"], "dataset_info": [{"config_name": "all", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "int32"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 30394267732, "num_examples": 14011953}], "download_size": 10486035662, "dataset_size": 30394267732}, {"config_name": "1800s", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "int32"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 30020434670, "num_examples": 13781747}], "download_size": 10348577602, "dataset_size": 30020434670}, {"config_name": "1700s", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "int32"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 266382657, "num_examples": 178224}], "download_size": 95137895, "dataset_size": 266382657}, {"config_name": "1510_1699", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "timestamp[s]"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 107667469, "num_examples": 51982}], "download_size": 42320165, "dataset_size": 107667469}, {"config_name": "1500_1899", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "timestamp[s]"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 30452067039, "num_examples": 14011953}], "download_size": 10486035662, "dataset_size": 30452067039}, {"config_name": "1800_1899", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "timestamp[s]"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 30077284377, "num_examples": 13781747}], "download_size": 10348577602, "dataset_size": 30077284377}, {"config_name": "1700_1799", "features": [{"name": "record_id", "dtype": "string"}, {"name": "date", "dtype": "timestamp[s]"}, {"name": "raw_date", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "place", "dtype": "string"}, {"name": "empty_pg", "dtype": "bool"}, {"name": "text", "dtype": "string"}, {"name": "pg", "dtype": "int32"}, {"name": "mean_wc_ocr", "dtype": "float32"}, {"name": "std_wc_ocr", "dtype": "float64"}, {"name": "name", "dtype": "string"}, {"name": "all_names", "dtype": "string"}, {"name": "Publisher", "dtype": "string"}, {"name": "Country of publication 1", "dtype": "string"}, {"name": "all Countries of publication", "dtype": "string"}, {"name": "Physical description", "dtype": "string"}, {"name": "Language_1", "dtype": "string"}, {"name": "Language_2", "dtype": "string"}, {"name": "Language_3", "dtype": "string"}, {"name": "Language_4", "dtype": "string"}, {"name": "multi_language", "dtype": "bool"}], "splits": [{"name": "train", "num_bytes": 267117831, "num_examples": 178224}], "download_size": 95137895, "dataset_size": 267117831}]}
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-other #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-machine-generated #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-original #language-German #language-English #language-Spanish #language-French #language-Italian #language-Dutch #license-cc0-1.0 #digital-humanities-research #region-us 

Dataset Card for British Library Books
======================================


Table of Contents
-----------------


* Dataset Card for British Library Books
	+ Table of Contents
	+ Dataset Description
		- Dataset Summary
		- Supported Tasks and Leaderboards
			* Language model training
			* Supervised tasks
		- Languages
			* Language change
			* Optical Character Recognition
				+ OCR word confidence
	+ Dataset Structure
		- Data Instances
		- Data Fields
		- Data Splits
	+ Dataset Creation
		- Curation Rationale
		- Source Data
			* Initial Data Collection and Normalization
				+ Date normalization
				+ Metadata included
			* Who are the source language producers?
		- Annotations
			* Annotation process
			* Who are the annotators?
		- Personal and Sensitive Information
	+ Considerations for Using the Data
		- Social Impact of Dataset
		- Discussion of Biases
			* Colonialism
		- Other Known Limitations
	+ Additional Information
		- Dataset Curators
		- Licensing Information
		- Citation Information
		- Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper:
* Leaderboard:
* Point of Contact: labs@URL


### Dataset Summary


This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were published in the 18th and 19th Century, but the collection also consists of a smaller number of books from earlier periods. Items within this collection cover a wide range of subject areas, including geography, philosophy, history, poetry and literature and are published in various languages.


While the books are predominately from the 18th and 19th Centuries, there are fewer books from earlier periods. The number of pages in the corpus by decade:



### Supported Tasks and Leaderboards


This collection has been previously used across various digital history and humanities projects since being published.


The dataset consists of text and a range of metadata associated with this text. This metadata includes:


* date of publication
* place of publication
* country of publication
* language
* OCR quality
* physical description of the original physical item


#### Language model training


As a relatively large dataset, 'blbooks' provides a source dataset for training language models. The presence of this metadata also offers interesting opportunities to use this dataset as a source for training language models based on:


* specific time-periods
* specific languages
* certain OCR quality thresholds


The above is not an exhaustive list but offer some suggestions of how the dataset can be used to explore topics such as the impact of OCR quality on language models, the ‘transferability’ of language models across time or the impact of training multilingual language models on historical languages.


#### Supervised tasks


Whilst this dataset does not have annotations for a specific NLP task, such as Named Entity Recognition, it does include a wide variety of metadata. This metadata has the potential to be used for training and/or evaluating a variety of supervised tasks predicting this metadata.


### Languages


This dataset consists of books published in several languages. The breakdown of the languages included (at the page level) is:



This breakdown was derived from the first language in the associated metadata field. Some books include multiple languages. Some of the languages codes for this data were also derived using computational methods. Therefore, the language fields in the dataset should be treated with some caution (discussed in more detail below).


#### Language change


The publication dates of books in the data cover a broad period of time (1500-1900). For languages in the dataset with broad temporal coverage, significant language change might be found. The ability to study this change by taking reasonably large samples of languages covering different time periods is one of the opportunities offered by this dataset. The fact that the text in this dataset was produced via Optical Character Recognition (OCR) causes some challenges for this type of research (see below).


#### Optical Character Recognition


The digitised books in this collection were transformed into machine-readable text using Optical Character Recognition (OCR) software. The text produced via OCR software will usually include some errors. These errors include; mistakes at the character level; for example, an 'i' is mistaken for an 'l', at the word level or across significant passages of text.


The books in this dataset can pose some additional challenges for OCR software. OCR errors can stem from:


* the quality of the original printing: printing technology was a developing technology during the time period covered by this corpus; some of the original book text will include misprints, blurred or faded ink that is hard to read
* damage to the page: some of the books will have become damaged over time, this can obscure all or parts of the text on a page
* poor quality scans: scanning books can be challenging; for example, if the book has tight bindings, it can be hard to capture text that has fallen into the gutter of the book.
* the language used in the books may differ from the languages OCR software is predominantly trained to recognise.


##### OCR word confidence


Many OCR engines produce some form of confidence score alongside the predicted text. These confidence scores are usually at the character or word level. The word confidence score was given for each word in the original ALTO XML versions of the text in this dataset in this dataset. The OCR confidence scores should be treated with some scepticism. For historical text or in a lower resource language, for example, a low confidence score may be more likely for words not included in a modern dictionary but may be accurate transcriptions of the original text. With that said, the confidence scores do give some sense of the OCR quality.


An example of text with a high (over 90% mean word confidence score):


An example of text with a score below 40%:


The quality of OCR - as measured by mean OCR confidence for a page - across the dataset correlates with other features. A groupby of publication decade and mean word confidence:



As might be expected, the earlier periods have lower mean word confidence scores. Again, all of this should be treated with some scepticism, especially as the size of the data grows over time.


As with time, the mean word confidence of the OCR software varies across languages:



Again, these numbers should be treated sceptically since some languages appear very infrequently. For example, the above table suggests the mean word confidence for Welsh is relatively high. However, there isn’t much Welsh in the dataset. Therefore, it is unlikely that this data will be particularly useful for training (historic) Welsh language models.


Dataset Structure
-----------------


The dataset has a number of configurations relating to the different dates of publication in the underlying data:


* '1500\_1899': this configuration covers all years
* '1800\_1899': this configuration covers the years between 1800 and 1899
* '1700\_1799': this configuration covers the years between 1700 and 1799
* '1510\_1699': this configuration covers the years between 1510 and 1699


### Configuration option


All of the configurations have an optional keyword argument 'skip\_empty\_pages' which is set to 'True' by default. The underlying dataset includes some pages where there is no text. This could either be because the underlying book page didn't have any text or the OCR software failed to detect this text.


For many uses of this dataset it doesn't make sense to include empty pages so these are skipped by default. However, for some uses you may prefer to retain a representation of the data that includes these empty pages. Passing 'skip\_empty\_pages=False' when loading the dataset will enable this option.


### Data Instances


An example data instance:


Each instance in the dataset represents a single page from an original digitised book.


### Data Fields


Included in this dataset are:


Field: record\_id, Data Type: string, Description: British Library ID for the item
Field: date, Data Type: int, Description: parsed/normalised year for the item. i.e. 1850
Field: raw\_date, Data Type: string, Description: the original raw date for an item i.e. 1850-
Field: title, Data Type: string, Description: title of the book
Field: place, Data Type: string, Description: Place of publication, i.e. London
Field: empty\_pg, Data Type: bool, Description: whether page contains text
Field: text, Data Type: string, Description: OCR generated text for a page
Field: pg, Data Type: int, Description: page in original book the instance refers to
Field: mean\_wc\_ocr, Data Type: float, Description: mean word confidence values for the page
Field: std\_wc\_ocr, Data Type: float, Description: standard deviation of the word confidence values for the page
Field: name, Data Type: string, Description: name associated with the item (usually author)
Field: all names, Data Type: string, Description: all names associated with a publication
Field: Publisher, Data Type: string, Description: publisher of the book
Field: Country of publication 1, Data Type: string, Description: first country associated with publication
Field: all Countries of publication, Data Type: string, Description: all countries associated with a publication
Field: Physical description, Data Type: string, Description: physical description of the item (size). This requires some normalisation before use and isn’t always present
Field: Language\_1, Data Type: string, Description: first language associated with the book, this is usually present
Field: Language\_2, Data Type: string, Description: 
Field: Language\_3, Data Type: string, Description: 
Field: Language\_4, Data Type: string, Description: 
Field: multi\_language, Data Type: bool, Description: 


Some of these fields are not populated a large proportion of the time. You can get some sense of this from this Pandas Profiling report


The majority of these fields relate to metadata about the books. Most of these fields were created by staff working for the British Library. The notable exception is the “Languages” fields that have sometimes been determined using computational methods. This work is reported in more detail in Automated Language Identification of Bibliographic Resources. It is important to note that metadata is neither perfect nor static. The metadata associated with this book was generated based on export from the British Library catalogue in 2021.


### Data Splits


This dataset contains a single split 'train'.


Dataset Creation
----------------


Note this section is a work in progress.


### Curation Rationale


The books in this collection were digitised as part of a project partnership between the British Library and Microsoft. Mass digitisation, i.e. projects intending to quickly digitise large volumes of materials shape the selection of materials to include in several ways. Some considerations which are often involved in the decision of whether to include items for digitisation include (but are not limited to):


* copyright status
* preservation needs
* the size of an item, very large and very small items are often hard to digitise quickly


These criteria can have knock-on effects on the makeup of a collection. For example, systematically excluding large books may result in some types of book content not being digitised. Large volumes are likely to be correlated to content to at least some extent, so excluding them from digitisation will mean that material is underrepresented. Similarly, copyright status is often (but not only) determined by publication date. This can often lead to a rapid fall in the number of items in a collection after a certain cut-off date.


All of the above is largely to make clear that this collection was not curated to create a representative sample of the British Library’s holdings. Some material will be over-represented, and others under-represented. Similarly, the collection should not be considered a representative sample of what was published across the period covered by the dataset (nor that the relative proportions of the data for each time period represent a proportional sample of publications from that period). Finally, and this probably does not need stating, the language included in the text should not be considered representative of either written or spoken language(s) from that time period.


### Source Data


The source data (physical items) includes a variety of resources (predominantly monographs) held by the British Library. The British Library is a Legal Deposit library. “Legal deposit requires publishers to provide a copy of every work they publish in the UK to the British Library. It’s existed in English law since 1662.” source.


The source data for this version of the data is derived from the original ALTO XML files and a recent metadata export #TODO add links


#### Initial Data Collection and Normalization


This version of the dataset was created using the original ALTO XML files and, where a match was found, updating the metadata associated with that item with more recent metadata using an export from the British Library catalogue. The process of creating this new dataset is documented here #TODO add link.


There are a few decisions made in the above processing steps worth highlighting in particular:


##### Date normalization


The metadata around date of publication for an item is not always exact. It often is represented as a date range e.g. '1850-1860'. The 'date' field above takes steps to normalise this date to a single integer value. In most cases, this is taking the mean of the values associated with the item. The 'raw\_date' field includes the unprocessed date string.


##### Metadata included


The metadata associated with each item includes most of the fields available via the ALTO XML. However, the data doesn’t include some metadata fields from the metadata export file. The reason fields were excluded because they are frequently not populated. A cut off of 50% was chosen, i.e. values from the metadata which are missing above 50% of the time were not included. This is slightly arbitrary, but since the aim of this version of the data was to support computational research using the collection it was felt that these fields with frequent missing values would be less valuable.


#### Who are the source language producers?


### Annotations


This dataset does not include annotations as usually understood in the context of NLP. The data does include metadata associated with the books.


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


There a range of considerations around using the data. These include the representativeness of the dataset, the OCR quality and the language used. Depending on your use case, these may be more or less important. For example, the impact of OCR quality on downstream tasks will depend on the target task. It may also be possible to mitigate this negative impact from OCR through tokenizer choice, Language Model training objectives, oversampling high-quality OCR, etc.


### Social Impact of Dataset


### Discussion of Biases


The text in this collection is derived from historical text. As a result, the text will reflect this time period's social beliefs and attitudes. The books include both fiction and non-fiction books.


Examples of book titles that appear in the data (these are randomly sampled from all titles):


* ‘Rhymes and Dreams, Legends of Pendle Forest, and other poems’,
* “Précis of Information concerning the Zulu Country, with a map. Prepared in the Intelligence Branch of the Quarter-Master-General’s Department, Horse Guards, War Office, etc”,
* ‘The fan. A poem’,
* ‘Grif; a story of Australian Life’,
* ‘Calypso; a masque: in three acts, etc’,
* ‘Tales Uncle told [With illustrative woodcuts.]’,
* 'Questings',
* 'Home Life on an Ostrich Farm. With ... illustrations’,
* ‘Bulgarya i Bulgarowie’,
* 'Εἰς τα βαθη της Ἀφρικης [In darkest Africa.] ... Μεταφρασις Γεωρ. Σ. Βουτσινα, etc',
* ‘The Corsair, a tale’,
‘Poems ... With notes [With a portrait.]’,
* ‘Report of the Librarian for the year 1898 (1899, 1901, 1909)’,
* “The World of Thought. A novel. By the author of ‘Before I began to speak.’”,
* 'Amleto; tragedia ... recata in versi italiani da M. Leoni, etc']


While using titles alone is insufficient to integrate bias in this collection, it gives some insight into the topics covered by books. Further, the tiles highlight some particular types of bias we might find in the collection. This should in no way be considered an exhaustive list.


#### Colonialism


Even in the above random sample of titles examples of colonial attitudes, we can see examples of titles. We can try and interrogate this further by searching for the name of places that were part of the British Empire when many of these books were published.


Searching for the string 'India' in the titles and randomly sampling 10 titles returns:


* “Travels in India in the Seventeenth Century: by Sir Thomas Roe and Dr. John Fryer. Reprinted from the ‘Calcutta Weekly Englishman.’”,
* ‘A Winter in India and Malaysia among the Methodist Missions’,
* “The Tourist’s Guide to all the principal stations on the railways of Northern India [By W. W.] ... Fifth edition”,
* ‘Records of Sport and Military Life in Western India ... With an introduction by ... G. B. Malleson’,
* "Lakhmi, the Rájpút's Bride. A tale of Gujarát in Western India [A poem.]”,
* ‘The West India Commonplace Book: compiled from parliamentary and official documents; shewing the interest of Great Britain in its Sugar Colonies’,
* “From Tonkin to India : by the sources of the Irawadi, January’ 95-January ’96”,
* ‘Case of the Ameers of Sinde : speeches of Mr. John Sullivan, and Captain William Eastwick, at a special court held at the India House, ... 26th January, 1844’,
* ‘The Andaman Islands; their colonisation, etc. A correspondence addressed to the India Office’,
* ‘Ancient India as described by Ptolemy; being a translation of the chapters which describe India and Eastern Asia in the treatise on Geography written by Klaudios Ptolemaios ... with introduction, commentary, map of India according to Ptolemy, and ... index, by J. W. McCrindle’]


Searching form the string 'Africa' in the titles and randomly sampling 10 titles returns:


* ['De Benguella ás Terras de Iácca. Descripção de uma viagem na Africa Central e Occidental ... Expedição organisada nos annos de 1877-1880. Edição illustrada',
* ‘To the New Geographical Society of Edinburgh [An address on Africa by H. M. Stanley.]’,
* ‘Diamonds and Gold in South Africa ... With maps, etc’,
* ‘Missionary Travels and Researches in South Africa ... With notes by F. S. Arnot. With map and illustrations. New edition’,
* ‘A Narrative of a Visit to the Mauritius and South Africa ... Illustrated by two maps, sixteen etchings and twenty-eight wood-cuts’,
* ‘Side Lights on South Africa ... With a map, etc’,
* ‘My Second Journey through Equatorial Africa ... in ... 1886 and 1887 ... Translated ... by M. J. A. Bergmann. With a map ... and ... illustrations, etc’,
* ‘Missionary Travels and Researches in South Africa ... With portrait and fullpage illustrations’,
* ‘[African sketches.] Narrative of a residence in South Africa ... A new edition. To which is prefixed a biographical sketch of the author by J. Conder’,
* ‘Lake Ngami; or, Explorations and discoveries during four years wandering in the wilds of South Western Africa ... With a map, and numerous illustrations, etc’]


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


The books are licensed under the CC Public Domain Mark 1.0 license.


### Contributions


Thanks to @davanstrien for adding this dataset.



### Dataset Summary


This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were published in the 18th and 19th Century, but the collection also consists of a smaller number of books from earlier periods. Items within this collection cover a wide range of subject areas, including geography, philosophy, history, poetry and literature and are published in various languages.


While the books are predominately from the 18th and 19th Centuries, there are fewer books from earlier periods. The number of pages in the corpus by decade: ### Supported Tasks and Leaderboards


This collection has been previously used across various digital history and humanities projects since being published.


The dataset consists of text and a range of metadata associated with this text. This metadata includes:


* date of publication
* place of publication
* country of publication
* language
* OCR quality
* physical description of the original physical item #### Language model training


As a relatively large dataset, 'blbooks' provides a source dataset for training language models. The presence of this metadata also offers interesting opportunities to use this dataset as a source for training language models based on:


* specific time-periods
* specific languages
* certain OCR quality thresholds


The above is not an exhaustive list but offer some suggestions of how the dataset can be used to explore topics such as the impact of OCR quality on language models, the ‘transferability’ of language models across time or the impact of training multilingual language models on historical languages. #### Supervised tasks


Whilst this dataset does not have annotations for a specific NLP task, such as Named Entity Recognition, it does include a wide variety of metadata. This metadata has the potential to be used for training and/or evaluating a variety of supervised tasks predicting this metadata. ### Languages


This dataset consists of books published in several languages. The breakdown of the languages included (at the page level) is:



This breakdown was derived from the first language in the associated metadata field. Some books include multiple languages. Some of the languages codes for this data were also derived using computational methods. Therefore, the language fields in the dataset should be treated with some caution (discussed in more detail below). #### Language change


The publication dates of books in the data cover a broad period of time (1500-1900). For languages in the dataset with broad temporal coverage, significant language change might be found. The ability to study this change by taking reasonably large samples of languages covering different time periods is one of the opportunities offered by this dataset. The fact that the text in this dataset was produced via Optical Character Recognition (OCR) causes some challenges for this type of research (see below). #### Optical Character Recognition


The digitised books in this collection were transformed into machine-readable text using Optical Character Recognition (OCR) software. The text produced via OCR software will usually include some errors. These errors include; mistakes at the character level; for example, an 'i' is mistaken for an 'l', at the word level or across significant passages of text.


The books in this dataset can pose some additional challenges for OCR software. OCR errors can stem from:


* the quality of the original printing: printing technology was a developing technology during the time period covered by this corpus; some of the original book text will include misprints, blurred or faded ink that is hard to read
* damage to the page: some of the books will have become damaged over time, this can obscure all or parts of the text on a page
* poor quality scans: scanning books can be challenging; for example, if the book has tight bindings, it can be hard to capture text that has fallen into the gutter of the book.
* the language used in the books may differ from the languages OCR software is predominantly trained to recognise. ##### OCR word confidence


Many OCR engines produce some form of confidence score alongside the predicted text. These confidence scores are usually at the character or word level. The word confidence score was given for each word in the original ALTO XML versions of the text in this dataset in this dataset. The OCR confidence scores should be treated with some scepticism. For historical text or in a lower resource language, for example, a low confidence score may be more likely for words not included in a modern dictionary but may be accurate transcriptions of the original text. With that said, the confidence scores do give some sense of the OCR quality.


An example of text with a high (over 90% mean word confidence score):


An example of text with a score below 40%:


The quality of OCR - as measured by mean OCR confidence for a page - across the dataset correlates with other features. A groupby of publication decade and mean word confidence:



As might be expected, the earlier periods have lower mean word confidence scores. Again, all of this should be treated with some scepticism, especially as the size of the data grows over time.


As with time, the mean word confidence of the OCR software varies across languages:



Again, these numbers should be treated sceptically since some languages appear very infrequently. For example, the above table suggests the mean word confidence for Welsh is relatively high. However, there isn’t much Welsh in the dataset. Therefore, it is unlikely that this data will be particularly useful for training (historic) Welsh language models.


Dataset Structure
-----------------


The dataset has a number of configurations relating to the different dates of publication in the underlying data:


* '1500\_1899': this configuration covers all years
* '1800\_1899': this configuration covers the years between 1800 and 1899
* '1700\_1799': this configuration covers the years between 1700 and 1799
* '1510\_1699': this configuration covers the years between 1510 and 1699 ### Configuration option


All of the configurations have an optional keyword argument 'skip\_empty\_pages' which is set to 'True' by default. The underlying dataset includes some pages where there is no text. This could either be because the underlying book page didn't have any text or the OCR software failed to detect this text.


For many uses of this dataset it doesn't make sense to include empty pages so these are skipped by default. However, for some uses you may prefer to retain a representation of the data that includes these empty pages. Passing 'skip\_empty\_pages=False' when loading the dataset will enable this option. ### Data Instances


An example data instance:


Each instance in the dataset represents a single page from an original digitised book. ### Data Fields


Included in this dataset are:


Field: record\_id, Data Type: string, Description: British Library ID for the item
Field: date, Data Type: int, Description: parsed/normalised year for the item. i.e. 1850
Field: raw\_date, Data Type: string, Description: the original raw date for an item i.e. 1850-
Field: title, Data Type: string, Description: title of the book
Field: place, Data Type: string, Description: Place of publication, i.e. London
Field: empty\_pg, Data Type: bool, Description: whether page contains text
Field: text, Data Type: string, Description: OCR generated text for a page
Field: pg, Data Type: int, Description: page in original book the instance refers to
Field: mean\_wc\_ocr, Data Type: float, Description: mean word confidence values for the page
Field: std\_wc\_ocr, Data Type: float, Description: standard deviation of the word confidence values for the page
Field: name, Data Type: string, Description: name associated with the item (usually author)
Field: all names, Data Type: string, Description: all names associated with a publication
Field: Publisher, Data Type: string, Description: publisher of the book
Field: Country of publication 1, Data Type: string, Description: first country associated with publication
Field: all Countries of publication, Data Type: string, Description: all countries associated with a publication
Field: Physical description, Data Type: string, Description: physical description of the item (size). This requires some normalisation before use and isn’t always present
Field: Language\_1, Data Type: string, Description: first language associated with the book, this is usually present
Field: Language\_2, Data Type: string, Description: 
Field: Language\_3, Data Type: string, Description: 
Field: Language\_4, Data Type: string, Description: 
Field: multi\_language, Data Type: bool, Description: 


Some of these fields are not populated a large proportion of the time. You can get some sense of this from this Pandas Profiling report


The majority of these fields relate to metadata about the books. Most of these fields were created by staff working for the British Library. The notable exception is the “Languages” fields that have sometimes been determined using computational methods. This work is reported in more detail in Automated Language Identification of Bibliographic Resources. It is important to note that metadata is neither perfect nor static. The metadata associated with this book was generated based on export from the British Library catalogue in 2021. ### Data Splits


This dataset contains a single split 'train'.


Dataset Creation
----------------


Note this section is a work in progress. ### Curation Rationale


The books in this collection were digitised as part of a project partnership between the British Library and Microsoft. Mass digitisation, i.e. projects intending to quickly digitise large volumes of materials shape the selection of materials to include in several ways. Some considerations which are often involved in the decision of whether to include items for digitisation include (but are not limited to):


* copyright status
* preservation needs
* the size of an item, very large and very small items are often hard to digitise quickly


These criteria can have knock-on effects on the makeup of a collection. For example, systematically excluding large books may result in some types of book content not being digitised. Large volumes are likely to be correlated to content to at least some extent, so excluding them from digitisation will mean that material is underrepresented. Similarly, copyright status is often (but not only) determined by publication date. This can often lead to a rapid fall in the number of items in a collection after a certain cut-off date.


All of the above is largely to make clear that this collection was not curated to create a representative sample of the British Library’s holdings. Some material will be over-represented, and others under-represented. Similarly, the collection should not be considered a representative sample of what was published across the period covered by the dataset (nor that the relative proportions of the data for each time period represent a proportional sample of publications from that period). Finally, and this probably does not need stating, the language included in the text should not be considered representative of either written or spoken language(s) from that time period. ### Source Data


The source data (physical items) includes a variety of resources (predominantly monographs) held by the British Library. The British Library is a Legal Deposit library. “Legal deposit requires publishers to provide a copy of every work they publish in the UK to the British Library. It’s existed in English law since 1662.” source.


The source data for this version of the data is derived from the original ALTO XML files and a recent metadata export #TODO add links #### Initial Data Collection and Normalization


This version of the dataset was created using the original ALTO XML files and, where a match was found, updating the metadata associated with that item with more recent metadata using an export from the British Library catalogue. The process of creating this new dataset is documented here #TODO add link.


There are a few decisions made in the above processing steps worth highlighting in particular: ##### Date normalization


The metadata around date of publication for an item is not always exact. It often is represented as a date range e.g. '1850-1860'. The 'date' field above takes steps to normalise this date to a single integer value. In most cases, this is taking the mean of the values associated with the item. The 'raw\_date' field includes the unprocessed date string. ##### Metadata included


The metadata associated with each item includes most of the fields available via the ALTO XML. However, the data doesn’t include some metadata fields from the metadata export file. The reason fields were excluded because they are frequently not populated. A cut off of 50% was chosen, i.e. values from the metadata which are missing above 50% of the time were not included. This is slightly arbitrary, but since the aim of this version of the data was to support computational research using the collection it was felt that these fields with frequent missing values would be less valuable. #### Who are the source language producers? ### Annotations


This dataset does not include annotations as usually understood in the context of NLP. The data does include metadata associated with the books. #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


There a range of considerations around using the data. These include the representativeness of the dataset, the OCR quality and the language used. Depending on your use case, these may be more or less important. For example, the impact of OCR quality on downstream tasks will depend on the target task. It may also be possible to mitigate this negative impact from OCR through tokenizer choice, Language Model training objectives, oversampling high-quality OCR, etc. ### Social Impact of Dataset ### Discussion of Biases


The text in this collection is derived from historical text. As a result, the text will reflect this time period's social beliefs and attitudes. The books include both fiction and non-fiction books.


Examples of book titles that appear in the data (these are randomly sampled from all titles):


* ‘Rhymes and Dreams, Legends of Pendle Forest, and other poems’,
* “Précis of Information concerning the Zulu Country, with a map. Prepared in the Intelligence Branch of the Quarter-Master-General’s Department, Horse Guards, War Office, etc”,
* ‘The fan. A poem’,
* ‘Grif; a story of Australian Life’,
* ‘Calypso; a masque: in three acts, etc’,
* ‘Tales Uncle told [With illustrative woodcuts.]’,
* 'Questings',
* 'Home Life on an Ostrich Farm. With ... illustrations’,
* ‘Bulgarya i Bulgarowie’,
* 'Εἰς τα βαθη της Ἀφρικης [In darkest Africa.] ... Μεταφρασις Γεωρ. Σ. Βουτσινα, etc',
* ‘The Corsair, a tale’,
‘Poems ... With notes [With a portrait.]’,
* ‘Report of the Librarian for the year 1898 (1899, 1901, 1909)’,
* “The World of Thought. A novel. By the author of ‘Before I began to speak.’”,
* 'Amleto; tragedia ... recata in versi italiani da M. Leoni, etc']


While using titles alone is insufficient to integrate bias in this collection, it gives some insight into the topics covered by books. Further, the tiles highlight some particular types of bias we might find in the collection. This should in no way be considered an exhaustive list. #### Colonialism


Even in the above random sample of titles examples of colonial attitudes, we can see examples of titles. We can try and interrogate this further by searching for the name of places that were part of the British Empire when many of these books were published.


Searching for the string 'India' in the titles and randomly sampling 10 titles returns:


* “Travels in India in the Seventeenth Century: by Sir Thomas Roe and Dr. John Fryer. Reprinted from the ‘Calcutta Weekly Englishman.’”,
* ‘A Winter in India and Malaysia among the Methodist Missions’,
* “The Tourist’s Guide to all the principal stations on the railways of Northern India [By W. W.] ... Fifth edition”,
* ‘Records of Sport and Military Life in Western India ... With an introduction by ... G. B. Malleson’,
* "Lakhmi, the Rájpút's Bride. A tale of Gujarát in Western India [A poem.]”,
* ‘The West India Commonplace Book: compiled from parliamentary and official documents; shewing the interest of Great Britain in its Sugar Colonies’,
* “From Tonkin to India : by the sources of the Irawadi, January’ 95-January ’96”,
* ‘Case of the Ameers of Sinde : speeches of Mr. John Sullivan, and Captain William Eastwick, at a special court held at the India House, ... 26th January, 1844’,
* ‘The Andaman Islands; their colonisation, etc. A correspondence addressed to the India Office’,
* ‘Ancient India as described by Ptolemy; being a translation of the chapters which describe India and Eastern Asia in the treatise on Geography written by Klaudios Ptolemaios ... with introduction, commentary, map of India according to Ptolemy, and ... index, by J. W. McCrindle’]


Searching form the string 'Africa' in the titles and randomly sampling 10 titles returns:


* ['De Benguella ás Terras de Iácca. Descripção de uma viagem na Africa Central e Occidental ... Expedição organisada nos annos de 1877-1880. Edição illustrada',
* ‘To the New Geographical Society of Edinburgh [An address on Africa by H. M. Stanley.]’,
* ‘Diamonds and Gold in South Africa ... With maps, etc’,
* ‘Missionary Travels and Researches in South Africa ... With notes by F. S. Arnot. With map and illustrations. New edition’,
* ‘A Narrative of a Visit to the Mauritius and South Africa ... Illustrated by two maps, sixteen etchings and twenty-eight wood-cuts’,
* ‘Side Lights on South Africa ... With a map, etc’,
* ‘My Second Journey through Equatorial Africa ... in ... 1886 and 1887 ... Translated ... by M. J. A. Bergmann. With a map ... and ... illustrations, etc’,
* ‘Missionary Travels and Researches in South Africa ... With portrait and fullpage illustrations’,
* ‘[African sketches.] Narrative of a residence in South Africa ... A new edition. To which is prefixed a biographical sketch of the author by J. Conder’,
* ‘Lake Ngami; or, Explorations and discoveries during four years wandering in the wilds of South Western Africa ... With a map, and numerous illustrations, etc’] ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


The books are licensed under the CC Public Domain Mark 1.0 license. ### Contributions


Thanks to @davanstrien for adding this dataset.
TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-other #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-machine-generated #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-original #language-German #language-English #language-Spanish #language-French #language-Italian #language-Dutch #license-cc0-1.0 #digital-humanities-research #region-us 
 ### Dataset Summary


This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were published in the 18th and 19th Century, but the collection also consists of a smaller number of books from earlier periods. Items within this collection cover a wide range of subject areas, including geography, philosophy, history, poetry and literature and are published in various languages.


While the books are predominately from the 18th and 19th Centuries, there are fewer books from earlier periods. The number of pages in the corpus by decade: ### Supported Tasks and Leaderboards


This collection has been previously used across various digital history and humanities projects since being published.


The dataset consists of text and a range of metadata associated with this text. This metadata includes:


* date of publication
* place of publication
* country of publication
* language
* OCR quality
* physical description of the original physical item #### Language model training


As a relatively large dataset, 'blbooks' provides a source dataset for training language models. The presence of this metadata also offers interesting opportunities to use this dataset as a source for training language models based on:


* specific time-periods
* specific languages
* certain OCR quality thresholds


The above is not an exhaustive list but offer some suggestions of how the dataset can be used to explore topics such as the impact of OCR quality on language models, the ‘transferability’ of language models across time or the impact of training multilingual language models on historical languages. #### Supervised tasks


Whilst this dataset does not have annotations for a specific NLP task, such as Named Entity Recognition, it does include a wide variety of metadata. This metadata has the potential to be used for training and/or evaluating a variety of supervised tasks predicting this metadata. ### Languages


This dataset consists of books published in several languages. The breakdown of the languages included (at the page level) is:



This breakdown was derived from the first language in the associated metadata field. Some books include multiple languages. Some of the languages codes for this data were also derived using computational methods. Therefore, the language fields in the dataset should be treated with some caution (discussed in more detail below). #### Language change


The publication dates of books in the data cover a broad period of time (1500-1900). For languages in the dataset with broad temporal coverage, significant language change might be found. The ability to study this change by taking reasonably large samples of languages covering different time periods is one of the opportunities offered by this dataset. The fact that the text in this dataset was produced via Optical Character Recognition (OCR) causes some challenges for this type of research (see below). #### Optical Character Recognition


The digitised books in this collection were transformed into machine-readable text using Optical Character Recognition (OCR) software. The text produced via OCR software will usually include some errors. These errors include; mistakes at the character level; for example, an 'i' is mistaken for an 'l', at the word level or across significant passages of text.


The books in this dataset can pose some additional challenges for OCR software. OCR errors can stem from:


* the quality of the original printing: printing technology was a developing technology during the time period covered by this corpus; some of the original book text will include misprints, blurred or faded ink that is hard to read
* damage to the page: some of the books will have become damaged over time, this can obscure all or parts of the text on a page
* poor quality scans: scanning books can be challenging; for example, if the book has tight bindings, it can be hard to capture text that has fallen into the gutter of the book.
* the language used in the books may differ from the languages OCR software is predominantly trained to recognise. ##### OCR word confidence


Many OCR engines produce some form of confidence score alongside the predicted text. These confidence scores are usually at the character or word level. The word confidence score was given for each word in the original ALTO XML versions of the text in this dataset in this dataset. The OCR confidence scores should be treated with some scepticism. For historical text or in a lower resource language, for example, a low confidence score may be more likely for words not included in a modern dictionary but may be accurate transcriptions of the original text. With that said, the confidence scores do give some sense of the OCR quality.


An example of text with a high (over 90% mean word confidence score):


An example of text with a score below 40%:


The quality of OCR - as measured by mean OCR confidence for a page - across the dataset correlates with other features. A groupby of publication decade and mean word confidence:



As might be expected, the earlier periods have lower mean word confidence scores. Again, all of this should be treated with some scepticism, especially as the size of the data grows over time.


As with time, the mean word confidence of the OCR software varies across languages:



Again, these numbers should be treated sceptically since some languages appear very infrequently. For example, the above table suggests the mean word confidence for Welsh is relatively high. However, there isn’t much Welsh in the dataset. Therefore, it is unlikely that this data will be particularly useful for training (historic) Welsh language models.


Dataset Structure
-----------------


The dataset has a number of configurations relating to the different dates of publication in the underlying data:


* '1500\_1899': this configuration covers all years
* '1800\_1899': this configuration covers the years between 1800 and 1899
* '1700\_1799': this configuration covers the years between 1700 and 1799
* '1510\_1699': this configuration covers the years between 1510 and 1699 ### Configuration option


All of the configurations have an optional keyword argument 'skip\_empty\_pages' which is set to 'True' by default. The underlying dataset includes some pages where there is no text. This could either be because the underlying book page didn't have any text or the OCR software failed to detect this text.


For many uses of this dataset it doesn't make sense to include empty pages so these are skipped by default. However, for some uses you may prefer to retain a representation of the data that includes these empty pages. Passing 'skip\_empty\_pages=False' when loading the dataset will enable this option. ### Data Instances


An example data instance:


Each instance in the dataset represents a single page from an original digitised book. ### Data Fields


Included in this dataset are:


Field: record\_id, Data Type: string, Description: British Library ID for the item
Field: date, Data Type: int, Description: parsed/normalised year for the item. i.e. 1850
Field: raw\_date, Data Type: string, Description: the original raw date for an item i.e. 1850-
Field: title, Data Type: string, Description: title of the book
Field: place, Data Type: string, Description: Place of publication, i.e. London
Field: empty\_pg, Data Type: bool, Description: whether page contains text
Field: text, Data Type: string, Description: OCR generated text for a page
Field: pg, Data Type: int, Description: page in original book the instance refers to
Field: mean\_wc\_ocr, Data Type: float, Description: mean word confidence values for the page
Field: std\_wc\_ocr, Data Type: float, Description: standard deviation of the word confidence values for the page
Field: name, Data Type: string, Description: name associated with the item (usually author)
Field: all names, Data Type: string, Description: all names associated with a publication
Field: Publisher, Data Type: string, Description: publisher of the book
Field: Country of publication 1, Data Type: string, Description: first country associated with publication
Field: all Countries of publication, Data Type: string, Description: all countries associated with a publication
Field: Physical description, Data Type: string, Description: physical description of the item (size). This requires some normalisation before use and isn’t always present
Field: Language\_1, Data Type: string, Description: first language associated with the book, this is usually present
Field: Language\_2, Data Type: string, Description: 
Field: Language\_3, Data Type: string, Description: 
Field: Language\_4, Data Type: string, Description: 
Field: multi\_language, Data Type: bool, Description: 


Some of these fields are not populated a large proportion of the time. You can get some sense of this from this Pandas Profiling report


The majority of these fields relate to metadata about the books. Most of these fields were created by staff working for the British Library. The notable exception is the “Languages” fields that have sometimes been determined using computational methods. This work is reported in more detail in Automated Language Identification of Bibliographic Resources. It is important to note that metadata is neither perfect nor static. The metadata associated with this book was generated based on export from the British Library catalogue in 2021. ### Data Splits


This dataset contains a single split 'train'.


Dataset Creation
----------------


Note this section is a work in progress. ### Curation Rationale


The books in this collection were digitised as part of a project partnership between the British Library and Microsoft. Mass digitisation, i.e. projects intending to quickly digitise large volumes of materials shape the selection of materials to include in several ways. Some considerations which are often involved in the decision of whether to include items for digitisation include (but are not limited to):


* copyright status
* preservation needs
* the size of an item, very large and very small items are often hard to digitise quickly


These criteria can have knock-on effects on the makeup of a collection. For example, systematically excluding large books may result in some types of book content not being digitised. Large volumes are likely to be correlated to content to at least some extent, so excluding them from digitisation will mean that material is underrepresented. Similarly, copyright status is often (but not only) determined by publication date. This can often lead to a rapid fall in the number of items in a collection after a certain cut-off date.


All of the above is largely to make clear that this collection was not curated to create a representative sample of the British Library’s holdings. Some material will be over-represented, and others under-represented. Similarly, the collection should not be considered a representative sample of what was published across the period covered by the dataset (nor that the relative proportions of the data for each time period represent a proportional sample of publications from that period). Finally, and this probably does not need stating, the language included in the text should not be considered representative of either written or spoken language(s) from that time period. ### Source Data


The source data (physical items) includes a variety of resources (predominantly monographs) held by the British Library. The British Library is a Legal Deposit library. “Legal deposit requires publishers to provide a copy of every work they publish in the UK to the British Library. It’s existed in English law since 1662.” source.


The source data for this version of the data is derived from the original ALTO XML files and a recent metadata export #TODO add links #### Initial Data Collection and Normalization


This version of the dataset was created using the original ALTO XML files and, where a match was found, updating the metadata associated with that item with more recent metadata using an export from the British Library catalogue. The process of creating this new dataset is documented here #TODO add link.


There are a few decisions made in the above processing steps worth highlighting in particular: ##### Date normalization


The metadata around date of publication for an item is not always exact. It often is represented as a date range e.g. '1850-1860'. The 'date' field above takes steps to normalise this date to a single integer value. In most cases, this is taking the mean of the values associated with the item. The 'raw\_date' field includes the unprocessed date string. ##### Metadata included


The metadata associated with each item includes most of the fields available via the ALTO XML. However, the data doesn’t include some metadata fields from the metadata export file. The reason fields were excluded because they are frequently not populated. A cut off of 50% was chosen, i.e. values from the metadata which are missing above 50% of the time were not included. This is slightly arbitrary, but since the aim of this version of the data was to support computational research using the collection it was felt that these fields with frequent missing values would be less valuable. #### Who are the source language producers? ### Annotations


This dataset does not include annotations as usually understood in the context of NLP. The data does include metadata associated with the books. #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


There a range of considerations around using the data. These include the representativeness of the dataset, the OCR quality and the language used. Depending on your use case, these may be more or less important. For example, the impact of OCR quality on downstream tasks will depend on the target task. It may also be possible to mitigate this negative impact from OCR through tokenizer choice, Language Model training objectives, oversampling high-quality OCR, etc. ### Social Impact of Dataset ### Discussion of Biases


The text in this collection is derived from historical text. As a result, the text will reflect this time period's social beliefs and attitudes. The books include both fiction and non-fiction books.


Examples of book titles that appear in the data (these are randomly sampled from all titles):


* ‘Rhymes and Dreams, Legends of Pendle Forest, and other poems’,
* “Précis of Information concerning the Zulu Country, with a map. Prepared in the Intelligence Branch of the Quarter-Master-General’s Department, Horse Guards, War Office, etc”,
* ‘The fan. A poem’,
* ‘Grif; a story of Australian Life’,
* ‘Calypso; a masque: in three acts, etc’,
* ‘Tales Uncle told [With illustrative woodcuts.]’,
* 'Questings',
* 'Home Life on an Ostrich Farm. With ... illustrations’,
* ‘Bulgarya i Bulgarowie’,
* 'Εἰς τα βαθη της Ἀφρικης [In darkest Africa.] ... Μεταφρασις Γεωρ. Σ. Βουτσινα, etc',
* ‘The Corsair, a tale’,
‘Poems ... With notes [With a portrait.]’,
* ‘Report of the Librarian for the year 1898 (1899, 1901, 1909)’,
* “The World of Thought. A novel. By the author of ‘Before I began to speak.’”,
* 'Amleto; tragedia ... recata in versi italiani da M. Leoni, etc']


While using titles alone is insufficient to integrate bias in this collection, it gives some insight into the topics covered by books. Further, the tiles highlight some particular types of bias we might find in the collection. This should in no way be considered an exhaustive list. #### Colonialism


Even in the above random sample of titles examples of colonial attitudes, we can see examples of titles. We can try and interrogate this further by searching for the name of places that were part of the British Empire when many of these books were published.


Searching for the string 'India' in the titles and randomly sampling 10 titles returns:


* “Travels in India in the Seventeenth Century: by Sir Thomas Roe and Dr. John Fryer. Reprinted from the ‘Calcutta Weekly Englishman.’”,
* ‘A Winter in India and Malaysia among the Methodist Missions’,
* “The Tourist’s Guide to all the principal stations on the railways of Northern India [By W. W.] ... Fifth edition”,
* ‘Records of Sport and Military Life in Western India ... With an introduction by ... G. B. Malleson’,
* "Lakhmi, the Rájpút's Bride. A tale of Gujarát in Western India [A poem.]”,
* ‘The West India Commonplace Book: compiled from parliamentary and official documents; shewing the interest of Great Britain in its Sugar Colonies’,
* “From Tonkin to India : by the sources of the Irawadi, January’ 95-January ’96”,
* ‘Case of the Ameers of Sinde : speeches of Mr. John Sullivan, and Captain William Eastwick, at a special court held at the India House, ... 26th January, 1844’,
* ‘The Andaman Islands; their colonisation, etc. A correspondence addressed to the India Office’,
* ‘Ancient India as described by Ptolemy; being a translation of the chapters which describe India and Eastern Asia in the treatise on Geography written by Klaudios Ptolemaios ... with introduction, commentary, map of India according to Ptolemy, and ... index, by J. W. McCrindle’]


Searching form the string 'Africa' in the titles and randomly sampling 10 titles returns:


* ['De Benguella ás Terras de Iácca. Descripção de uma viagem na Africa Central e Occidental ... Expedição organisada nos annos de 1877-1880. Edição illustrada',
* ‘To the New Geographical Society of Edinburgh [An address on Africa by H. M. Stanley.]’,
* ‘Diamonds and Gold in South Africa ... With maps, etc’,
* ‘Missionary Travels and Researches in South Africa ... With notes by F. S. Arnot. With map and illustrations. New edition’,
* ‘A Narrative of a Visit to the Mauritius and South Africa ... Illustrated by two maps, sixteen etchings and twenty-eight wood-cuts’,
* ‘Side Lights on South Africa ... With a map, etc’,
* ‘My Second Journey through Equatorial Africa ... in ... 1886 and 1887 ... Translated ... by M. J. A. Bergmann. With a map ... and ... illustrations, etc’,
* ‘Missionary Travels and Researches in South Africa ... With portrait and fullpage illustrations’,
* ‘[African sketches.] Narrative of a residence in South Africa ... A new edition. To which is prefixed a biographical sketch of the author by J. Conder’,
* ‘Lake Ngami; or, Explorations and discoveries during four years wandering in the wilds of South Western Africa ... With a map, and numerous illustrations, etc’] ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information


The books are licensed under the CC Public Domain Mark 1.0 license. ### Contributions


Thanks to @davanstrien for adding this dataset.
passage: TAGS
#task_categories-text-generation #task_categories-fill-mask #task_categories-other #task_ids-language-modeling #task_ids-masked-language-modeling #annotations_creators-no-annotation #language_creators-machine-generated #multilinguality-multilingual #size_categories-100K<n<1M #source_datasets-original #language-German #language-English #language-Spanish #language-French #language-Italian #language-Dutch #license-cc0-1.0 #digital-humanities-research #region-us 
### Dataset Summary


This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were published in the 18th and 19th Century, but the collection also consists of a smaller number of books from earlier periods. Items within this collection cover a wide range of subject areas, including geography, philosophy, history, poetry and literature and are published in various languages.


While the books are predominately from the 18th and 19th Centuries, there are fewer books from earlier periods. The number of pages in the corpus by decade:### Supported Tasks and Leaderboards


This collection has been previously used across various digital history and humanities projects since being published.


The dataset consists of text and a range of metadata associated with this text. This metadata includes:


* date of publication
* place of publication
* country of publication
* language
* OCR quality
* physical description of the original physical item passage: #### Language model training


As a relatively large dataset, 'blbooks' provides a source dataset for training language models. The presence of this metadata also offers interesting opportunities to use this dataset as a source for training language models based on:


* specific time-periods
* specific languages
* certain OCR quality thresholds


The above is not an exhaustive list but offer some suggestions of how the dataset can be used to explore topics such as the impact of OCR quality on language models, the ‘transferability’ of language models across time or the impact of training multilingual language models on historical languages.#### Supervised tasks


Whilst this dataset does not have annotations for a specific NLP task, such as Named Entity Recognition, it does include a wide variety of metadata. This metadata has the potential to be used for training and/or evaluating a variety of supervised tasks predicting this metadata.### Languages


This dataset consists of books published in several languages. The breakdown of the languages included (at the page level) is:



This breakdown was derived from the first language in the associated metadata field. Some books include multiple languages. Some of the languages codes for this data were also derived using computational methods. Therefore, the language fields in the dataset should be treated with some caution (discussed in more detail below).#### Language change


The publication dates of books in the data cover a broad period of time (1500-1900). For languages in the dataset with broad temporal coverage, significant language change might be found. The ability to study this change by taking reasonably large samples of languages covering different time periods is one of the opportunities offered by this dataset. The fact that the text in this dataset was produced via Optical Character Recognition (OCR) causes some challenges for this type of research (see below). passage: #### Optical Character Recognition


The digitised books in this collection were transformed into machine-readable text using Optical Character Recognition (OCR) software. The text produced via OCR software will usually include some errors. These errors include; mistakes at the character level; for example, an 'i' is mistaken for an 'l', at the word level or across significant passages of text.


The books in this dataset can pose some additional challenges for OCR software. OCR errors can stem from:


* the quality of the original printing: printing technology was a developing technology during the time period covered by this corpus; some of the original book text will include misprints, blurred or faded ink that is hard to read
* damage to the page: some of the books will have become damaged over time, this can obscure all or parts of the text on a page
* poor quality scans: scanning books can be challenging; for example, if the book has tight bindings, it can be hard to capture text that has fallen into the gutter of the book.
* the language used in the books may differ from the languages OCR software is predominantly trained to recognise.##### OCR word confidence


Many OCR engines produce some form of confidence score alongside the predicted text. These confidence scores are usually at the character or word level. The word confidence score was given for each word in the original ALTO XML versions of the text in this dataset in this dataset. The OCR confidence scores should be treated with some scepticism. For historical text or in a lower resource language, for example, a low confidence score may be more likely for words not included in a modern dictionary but may be accurate transcriptions of the original text. With that said, the confidence scores do give some sense of the OCR quality.


An example of text with a high (over 90% mean word confidence score):


An example of text with a score below 40%:


The quality of OCR - as measured by mean OCR confidence for a page - across the dataset correlates with other features. A groupby of publication decade and mean word confidence:



As might be expected, the earlier periods have lower mean word confidence scores. Again, all of this should be treated with some scepticism, especially as the size of the data grows over time.


As with time, the mean word confidence of the OCR software varies across languages:



Again, these numbers should be treated sceptically since some languages appear very infrequently. For example, the above table suggests the mean word confidence for Welsh is relatively high. However, there isn’t much Welsh in the dataset. Therefore, it is unlikely that this data will be particularly useful for training (historic) Welsh language models.


Dataset Structure
-----------------


The dataset has a number of configurations relating to the different dates of publication in the underlying data:


* '1500\_1899': this configuration covers all years
* '1800\_1899': this configuration covers the years between 1800 and 1899
* '1700\_1799': this configuration covers the years between 1700 and 1799
* '1510\_1699': this configuration covers the years between 1510 and 1699 passage: ### Configuration option


All of the configurations have an optional keyword argument 'skip\_empty\_pages' which is set to 'True' by default. The underlying dataset includes some pages where there is no text. This could either be because the underlying book page didn't have any text or the OCR software failed to detect this text.


For many uses of this dataset it doesn't make sense to include empty pages so these are skipped by default. However, for some uses you may prefer to retain a representation of the data that includes these empty pages. Passing 'skip\_empty\_pages=False' when loading the dataset will enable this option.### Data Instances


An example data instance:


Each instance in the dataset represents a single page from an original digitised book. passage: ### Data Fields


Included in this dataset are:


Field: record\_id, Data Type: string, Description: British Library ID for the item
Field: date, Data Type: int, Description: parsed/normalised year for the item. i.e. 1850
Field: raw\_date, Data Type: string, Description: the original raw date for an item i.e. 1850-
Field: title, Data Type: string, Description: title of the book
Field: place, Data Type: string, Description: Place of publication, i.e. London
Field: empty\_pg, Data Type: bool, Description: whether page contains text
Field: text, Data Type: string, Description: OCR generated text for a page
Field: pg, Data Type: int, Description: page in original book the instance refers to
Field: mean\_wc\_ocr, Data Type: float, Description: mean word confidence values for the page
Field: std\_wc\_ocr, Data Type: float, Description: standard deviation of the word confidence values for the page
Field: name, Data Type: string, Description: name associated with the item (usually author)
Field: all names, Data Type: string, Description: all names associated with a publication
Field: Publisher, Data Type: string, Description: publisher of the book
Field: Country of publication 1, Data Type: string, Description: first country associated with publication
Field: all Countries of publication, Data Type: string, Description: all countries associated with a publication
Field: Physical description, Data Type: string, Description: physical description of the item (size). This requires some normalisation before use and isn’t always present
Field: Language\_1, Data Type: string, Description: first language associated with the book, this is usually present
Field: Language\_2, Data Type: string, Description: 
Field: Language\_3, Data Type: string, Description: 
Field: Language\_4, Data Type: string, Description: 
Field: multi\_language, Data Type: bool, Description: 


Some of these fields are not populated a large proportion of the time. You can get some sense of this from this Pandas Profiling report


The majority of these fields relate to metadata about the books. Most of these fields were created by staff working for the British Library. The notable exception is the “Languages” fields that have sometimes been determined using computational methods. This work is reported in more detail in Automated Language Identification of Bibliographic Resources. It is important to note that metadata is neither perfect nor static. The metadata associated with this book was generated based on export from the British Library catalogue in 2021.### Data Splits


This dataset contains a single split 'train'.


Dataset Creation
----------------


Note this section is a work in progress.### Curation Rationale


The books in this collection were digitised as part of a project partnership between the British Library and Microsoft. Mass digitisation, i.e. projects intending to quickly digitise large volumes of materials shape the selection of materials to include in several ways. Some considerations which are often involved in the decision of whether to include items for digitisation include (but are not limited to):


* copyright status
* preservation needs
* the size of an item, very large and very small items are often hard to digitise quickly


These criteria can have knock-on effects on the makeup of a collection. For example, systematically excluding large books may result in some types of book content not being digitised. Large volumes are likely to be correlated to content to at least some extent, so excluding them from digitisation will mean that material is underrepresented. Similarly, copyright status is often (but not only) determined by publication date. This can often lead to a rapid fall in the number of items in a collection after a certain cut-off date.


All of the above is largely to make clear that this collection was not curated to create a representative sample of the British Library’s holdings. Some material will be over-represented, and others under-represented. Similarly, the collection should not be considered a representative sample of what was published across the period covered by the dataset (nor that the relative proportions of the data for each time period represent a proportional sample of publications from that period). Finally, and this probably does not need stating, the language included in the text should not be considered representative of either written or spoken language(s) from that time period. passage: ### Source Data


The source data (physical items) includes a variety of resources (predominantly monographs) held by the British Library. The British Library is a Legal Deposit library. “Legal deposit requires publishers to provide a copy of every work they publish in the UK to the British Library. It’s existed in English law since 1662.” source.


The source data for this version of the data is derived from the original ALTO XML files and a recent metadata export #TODO add links#### Initial Data Collection and Normalization


This version of the dataset was created using the original ALTO XML files and, where a match was found, updating the metadata associated with that item with more recent metadata using an export from the British Library catalogue. The process of creating this new dataset is documented here #TODO add link.


There are a few decisions made in the above processing steps worth highlighting in particular:##### Date normalization


The metadata around date of publication for an item is not always exact. It often is represented as a date range e.g. '1850-1860'. The 'date' field above takes steps to normalise this date to a single integer value. In most cases, this is taking the mean of the values associated with the item. The 'raw\_date' field includes the unprocessed date string.##### Metadata included


The metadata associated with each item includes most of the fields available via the ALTO XML. However, the data doesn’t include some metadata fields from the metadata export file. The reason fields were excluded because they are frequently not populated. A cut off of 50% was chosen, i.e. values from the metadata which are missing above 50% of the time were not included. This is slightly arbitrary, but since the aim of this version of the data was to support computational research using the collection it was felt that these fields with frequent missing values would be less valuable.#### Who are the source language producers?### Annotations


This dataset does not include annotations as usually understood in the context of NLP. The data does include metadata associated with the books.#### Annotation process#### Who are the annotators?### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


There a range of considerations around using the data. These include the representativeness of the dataset, the OCR quality and the language used. Depending on your use case, these may be more or less important. For example, the impact of OCR quality on downstream tasks will depend on the target task. It may also be possible to mitigate this negative impact from OCR through tokenizer choice, Language Model training objectives, oversampling high-quality OCR, etc.### Social Impact of Dataset passage: ### Discussion of Biases


The text in this collection is derived from historical text. As a result, the text will reflect this time period's social beliefs and attitudes. The books include both fiction and non-fiction books.


Examples of book titles that appear in the data (these are randomly sampled from all titles):


* ‘Rhymes and Dreams, Legends of Pendle Forest, and other poems’,
* “Précis of Information concerning the Zulu Country, with a map. Prepared in the Intelligence Branch of the Quarter-Master-General’s Department, Horse Guards, War Office, etc”,
* ‘The fan. A poem’,
* ‘Grif; a story of Australian Life’,
* ‘Calypso; a masque: in three acts, etc’,
* ‘Tales Uncle told [With illustrative woodcuts.]’,
* 'Questings',
* 'Home Life on an Ostrich Farm. With ... illustrations’,
* ‘Bulgarya i Bulgarowie’,
* 'Εἰς τα βαθη της Ἀφρικης [In darkest Africa.] ... Μεταφρασις Γεωρ. Σ. Βουτσινα, etc',
* ‘The Corsair, a tale’,
‘Poems ... With notes [With a portrait.]’,
* ‘Report of the Librarian for the year 1898 (1899, 1901, 1909)’,
* “The World of Thought. A novel. By the author of ‘Before I began to speak.’”,
* 'Amleto; tragedia ... recata in versi italiani da M. Leoni, etc']


While using titles alone is insufficient to integrate bias in this collection, it gives some insight into the topics covered by books. Further, the tiles highlight some particular types of bias we might find in the collection. This should in no way be considered an exhaustive list.