# YFCC100M subset from OpenAI

Subset of [YFCC100M](https://arxiv.org/abs/1503.01817) used by OpenAI for [CLIP](https://github.com/openai/CLIP/blob/main/data/yfcc100m.md), filtered to contain only the images that we could retrieve.

| Split | train | validation |
| --- | --- | --- |
| Number of samples | 14,808,859 | 16,374 |
| Size | 1.9 TB | 2.1 GB |

Features:
* from the original dataset: `title`, `description`, `photoid`, `uid`, `unickname`, `datetaken`, `dateuploaded`, `capturedevice`, `usertags`, `machinetags`, `longitude`, `latitude`, `accuracy`, `pageurl`, `downloadurl`, `licensename`, `licenseurl`, `serverid`, `farmid`, `secret`, `secretoriginal`, `ext`, `marker`, `key`
* `img`: image content, can be loaded with `PIL.Image.open(io.BytesIO(item['img']))`
* `title_clean` and `description_clean`: derived from `title` and `description` using `clean_text` function detailed below

```python
def clean_text(text):
    # decode url
    text = urllib.parse.unquote_plus(text)
    # remove html tags
    text = re.sub('<[^<]+?>', '', text)
    # remove multiple spaces + "\r" + "\n" + "\t"
    text = " ".join(text.split())
    return text
```
arxiv:1503.01817 region:us
{}
TAGS
#arxiv-1503.01817 #region-us 

YFCC100M subset from OpenAI
===========================


Subset of YFCC100M used by OpenAI for CLIP, filtered to contain only the images that we could retrieve.


Split: Number of samples, train: 14,808,859, validation: 16,374
Split: Size, train: 1.9 TB, validation: 2.1 GB


Features:


* from the original dataset: 'title', 'description', 'photoid', 'uid', 'unickname', 'datetaken', 'dateuploaded', 'capturedevice', 'usertags', 'machinetags', 'longitude', 'latitude', 'accuracy', 'pageurl', 'downloadurl', 'licensename', 'licenseurl', 'serverid', 'farmid', 'secret', 'secretoriginal', 'ext', 'marker', 'key'
* 'img': image content, can be loaded with 'URL(io.BytesIO(item['img']))'
* 'title\_clean' and 'description\_clean': derived from 'title' and 'description' using 'clean\_text' function detailed below



TAGS
#arxiv-1503.01817 #region-us 
