# Dataset Card for Project Gutenber - English Language eBooks

A collection of non-english language eBooks (48284 rows, 80%+ of all english language books available on the site) from the Project Gutenberg site with metadata removed. 

Originally colected for https://github.com/LAION-AI/Open-Assistant (follows the OpenAssistant training format)

The METADATA column contains catalogue meta information on each book as a serialized JSON:

| key | original column |
|----|----|
| language | - |
| text_id | Text# unique book identifier on Prject Gutenberg as *int* |
| title | Title of the book as *string* |
| issued | Issued date as *string* |
| authors | Authors as *string*, comma separated sometimes with dates |
| subjects | Subjects as *string*, various formats |
| locc | LoCC code as *string* |
| bookshelves | Bookshelves as *string*, optional |

## Source data

**How was the data generated?**

- A crawler (see Open-Assistant repository) downloaded the raw HTML code for 
  each eBook based on **Text#** id in the Gutenberg catalogue (if available)
- The metadata and the body of text are not clearly separated so an additional
  parser attempts to split them, then remove transcriber's notes and e-book 
  related information from the body of text (text clearly marked as copyrighted or 
  malformed was skipped and not collected)
- The body of cleaned TEXT as well as the catalogue METADATA is then saved as
  a parquet file, with all columns being strings

**Copyright notice:**

- Some of the books are copyrighted! The crawler ignored all books
  with an english copyright header by utilizing a regex expression, but make
  sure to check out the metadata for each book manually to ensure they are okay
  to use in your country! More information on copyright:
  https://www.gutenberg.org/help/copyright.html and
  https://www.gutenberg.org/policy/permission.html
- Project Gutenberg has the following requests when using books without
  metadata: _Books obtianed from the Project Gutenberg site should have the
  following legal note next to them: "This eBook is for the use of anyone
  anywhere in the United States and most other parts of the world at no cost and
  with almost" no restrictions whatsoever. You may copy it, give it away or
  re-use it under the terms of the Project Gutenberg License included with this
  eBook or online at www.gutenberg.org. If you are not located in the United
  States, you will have to check the laws of the country where you are located
  before using this eBook."_
task_categories:text-generation size_categories:10K<n<100K language:en license:mit project gutenberg e-book gutenberg.org region:us
{"language": ["en"], "license": "mit", "size_categories": ["10K<n<100K"], "task_categories": ["text-generation"], "pretty_name": "Project Gutenberg eBooks in English", "dataset_info": {"features": [{"name": "TEXT", "dtype": "string"}, {"name": "SOURCE", "dtype": "string"}, {"name": "METADATA", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 18104255935, "num_examples": 48284}], "download_size": 10748877194, "dataset_size": 18104255935}, "tags": ["project gutenberg", "e-book", "gutenberg.org"]}
TAGS
#task_categories-text-generation #size_categories-10K<n<100K #language-English #license-mit #project gutenberg #e-book #gutenberg.org #region-us 

Dataset Card for Project Gutenber - English Language eBooks
===========================================================


A collection of non-english language eBooks (48284 rows, 80%+ of all english language books available on the site) from the Project Gutenberg site with metadata removed.


Originally colected for URL (follows the OpenAssistant training format)


The METADATA column contains catalogue meta information on each book as a serialized JSON:



Source data
-----------


How was the data generated?


* A crawler (see Open-Assistant repository) downloaded the raw HTML code for
each eBook based on Text# id in the Gutenberg catalogue (if available)
* The metadata and the body of text are not clearly separated so an additional
parser attempts to split them, then remove transcriber's notes and e-book
related information from the body of text (text clearly marked as copyrighted or
malformed was skipped and not collected)
* The body of cleaned TEXT as well as the catalogue METADATA is then saved as
a parquet file, with all columns being strings


Copyright notice:


* Some of the books are copyrighted! The crawler ignored all books
with an english copyright header by utilizing a regex expression, but make
sure to check out the metadata for each book manually to ensure they are okay
to use in your country! More information on copyright:
URL and
URL
* Project Gutenberg has the following requests when using books without
metadata: *Books obtianed from the Project Gutenberg site should have the
following legal note next to them: "This eBook is for the use of anyone
anywhere in the United States and most other parts of the world at no cost and
with almost" no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included with this
eBook or online at URL. If you are not located in the United
States, you will have to check the laws of the country where you are located
before using this eBook."*



# id in the Gutenberg catalogue (if available)
* The metadata and the body of text are not clearly separated so an additional
parser attempts to split them, then remove transcriber's notes and e-book
related information from the body of text (text clearly marked as copyrighted or
malformed was skipped and not collected)
* The body of cleaned TEXT as well as the catalogue METADATA is then saved as
a parquet file, with all columns being strings


Copyright notice:


* Some of the books are copyrighted! The crawler ignored all books
with an english copyright header by utilizing a regex expression, but make
sure to check out the metadata for each book manually to ensure they are okay
to use in your country! More information on copyright:
URL and
URL
* Project Gutenberg has the following requests when using books without
metadata: *Books obtianed from the Project Gutenberg site should have the
following legal note next to them: "This eBook is for the use of anyone
anywhere in the United States and most other parts of the world at no cost and
with almost" no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included with this
eBook or online at URL. If you are not located in the United
States, you will have to check the laws of the country where you are located
before using this eBook."*
TAGS
#task_categories-text-generation #size_categories-10K<n<100K #language-English #license-mit #project gutenberg #e-book #gutenberg.org #region-us 
 # id in the Gutenberg catalogue (if available)
* The metadata and the body of text are not clearly separated so an additional
parser attempts to split them, then remove transcriber's notes and e-book
related information from the body of text (text clearly marked as copyrighted or
malformed was skipped and not collected)
* The body of cleaned TEXT as well as the catalogue METADATA is then saved as
a parquet file, with all columns being strings


Copyright notice:


* Some of the books are copyrighted! The crawler ignored all books
with an english copyright header by utilizing a regex expression, but make
sure to check out the metadata for each book manually to ensure they are okay
to use in your country! More information on copyright:
URL and
URL
* Project Gutenberg has the following requests when using books without
metadata: *Books obtianed from the Project Gutenberg site should have the
following legal note next to them: "This eBook is for the use of anyone
anywhere in the United States and most other parts of the world at no cost and
with almost" no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included with this
eBook or online at URL. If you are not located in the United
States, you will have to check the laws of the country where you are located
before using this eBook."*