size_categories:100M<n<1B license:other arxiv:2310.07875 region:us
---
license: other
pretty_name: TabLib
size_categories:
- 100M<n<1B
extra_gated_prompt: >-
  Accessing the full TabLib dataset requires permission from Approximate Labs. To request access, please provide the following details and Approximate Labs will process your request as soon as possible.
extra_gated_fields:
 Full Name: text
 Email: text
 Country: text
 Organization: text
 What do you intend to do with this dataset?: text
 I agree to abide by the license requirements of the data contained in TabLib: checkbox
---

[![](https://dcbadge.vercel.app/api/server/kW9nBQErGe?compact=true&style=flat)](https://discord.gg/kW9nBQErGe)

<img src="https://approximatelabs.com/tablib.png" width="800" />

# TabLib

A minimally-preprocessed dataset of 627M tables (69 TiB) extracted from HTML, PDF, CSV, TSV, Excel, and SQLite files from GitHub and Common Crawl.

This includes 867B tokens of "context metadata": each table includes provenance information and table context such as filename, text before/after, HTML metadata, etc.

A smaller 0.1% sample of this dataset can be found [here](https://huggingface.co/datasets/approximatelabs/tablib-v1-sample).

For more information, read the [paper](https://arxiv.org/abs/2310.07875) & [announcement blog](https://approximatelabs.com/blog/tablib).

# Dataset Details

## Sources
* **GitHub**: nearly all public GitHub repositories
* **Common Crawl**: the `CC-MAIN-2023-23` crawl

## Reading Tables
Tables are stored as serialized Arrow bytes in the `arrow_bytes` column. To read these, you will need to deserialize the bytes:

```python
import datasets
import pyarrow as pa

# load a single file of the dataset
ds = datasets.load_dataset(
    'approximatelabs/tablib-v1-full',
    data_files='tablib/job=github_000005/batch=000001/part=000001/manifest.parquet',
    token='...',
)

df = ds['train'].to_pandas()

tables = [pa.RecordBatchStreamReader(b).read_all() for b in df['arrow_bytes']]
```

## Licensing
This dataset is intended for research use only.

For specific licensing information, refer to the license of the specific datum being used.

# Contact
If you have any questions, comments, or concerns about licensing, pii, etc. please contact using [this form](https://forms.gle/C74VTWP7L78QDVR67).

# Approximate Labs
TabLib is a project from Approximate Labs. Find us on [Twitter](https://twitter.com/approximatelabs), [Github](https://github.com/approximatelabs), [Linkedin](https://www.linkedin.com/company/approximate-labs), and [Discord](https://discord.gg/kW9nBQErGe).

# Citations
If you use TabLib for any of your research, please cite the TabLib paper:

```
@misc{eggert2023tablib,
      title={TabLib: A Dataset of 627M Tables with Context}, 
      author={Gus Eggert and Kevin Huo and Mike Biven and Justin Waugh},
      year={2023},
      eprint={2310.07875},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```