
# CleanVid Map (15M) 🎥
### TempoFunk Video Generation Project

CleanVid-15M is a large-scale dataset of videos with multiple metadata entries such as:
- Textual Descriptions 📃
- Recording Equipment 📹
- Categories 🔠
- Framerate 🎞️
- Aspect Ratio 📺

CleanVid aim is to improve the quality of WebVid-10M dataset by adding more data and cleaning the dataset by dewatermarking the videos in it.

This dataset includes only the map with the urls and metadata, with 3,694,510 more entries than the original WebVid-10M dataset.
Note that the videos are low-resolution, ranging from 240p to 480p. But this shouldn't be a problem as resolution scaling is difficult in Text-To-Video models.
More Datasets to come for high-res use cases.

CleanVid is the foundation dataset for the TempoFunk Video Generation project.

Built from a crawl of Shutterstock from June 25, 2023.

## Format 📊

- id: Integer (int64) - Shutterstock video ID
- description: String - Description of the video
- duration: Float(64) - Duration of the video in seconds
- aspectratio: String - Aspect Ratio of the video separated by colons (":")
- videourl: String - Video URL for the video in the entry, MP4 format. WEBM format is also available most of the times (by changing the extension at the end of the URL.).
- author: String - JSON-String containing information of the author such as `Recording Equipment`, `Style`, `Nationality` and others.
- categories: String - JSON-String containing the categories of the videos. (Values from shutterstock, not by us.)
- framerate: Float(64) - Framerate of the video
- r18: Bit (int64) - Wether the video is marked as mature content. 0 = Safe For Work; 1 = Mature Content

## Code 👩‍💻

If you want to re-create this dataset on your own, code is available here:

https://github.com/chavinlo/tempofunk-scrapper/tree/refractor1/sites/shutterstock

Due to rate-limitations, you might need to obtain a proxy. Functionality for proxies is included in the repository.

## Sample 🧪

```json
{
  "id": 1056934082,
  "description": "Rio, Brazil - February 24, 2020: parade of the samba school Mangueira, at the Marques de Sapucai Sambodromo",
  "duration": 9.76,
  "aspectratio": "16:9",
  "videourl": "https://www.shutterstock.com/shutterstock/videos/1056934082/preview/stock-footage-rio-brazil-february-parade-of-the-samba-school-mangueira-at-the-marques-de-sapucai.mp4",
  "author": {
    "accountsId": 101974372,
    "contributorId": 62154,
    "bio": "Sempre produzindo mais",
    "location": "br",
    "website": "www.dcpress.com.br",
    "contributorTypeList": [
      "photographer"
    ],
    "equipmentList": [
      "300mm f2.8",
      "24-70mm",
      "70-200mm",
      "Nikon D7500 ",
      "Nikon Df",
      "Flashs Godox"
    ],
    "styleList": [
      "editorial",
      "food",
      "landscape"
    ],
    "subjectMatterList": [
      "photographer",
      "people",
      "nature",
      "healthcare",
      "food_and_drink"
    ],
    "facebookUsername": "celso.pupo",
    "googlePlusUsername": "celsopupo",
    "twitterUsername": "celsopupo",
    "storageKey": "/contributors/62154/avatars/thumb.jpg",
    "cdnThumbPath": "/contributors/62154/avatars/thumb.jpg",
    "displayName": "Celso Pupo",
    "vanityUrlUsername": "rodrigues",
    "portfolioUrlSuffix": "rodrigues",
    "portfolioUrl": "https://www.shutterstock.com/g/rodrigues",
    "instagramUsername": "celsopupo",
    "hasPublicSets": true,
    "instagramUrl": "https://www.instagram.com/celsopupo",
    "facebookUrl": "https://www.facebook.com/celso.pupo",
    "twitterUrl": "https://twitter.com/celsopupo"
  },
  "categories": [
    "People"
  ],
  "framerate": 29.97,
  "r18": 0
}
```

## Credits 👥

### Main

- Lopho - Part of TempoFunk Video Generation
- Chavinlo - Part of TempoFunk Video Generation & CleanVid Crawling, Scraping and Formatting

```
@InProceedings{Bain21,
  author       = "Max Bain and Arsha Nagrani and G{\"u}l Varol and Andrew Zisserman",
  title        = "Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval",
  booktitle    = "IEEE International Conference on Computer Vision",
  year         = "2021",
}
```

### Extra

- Salt - Base Threading Code (2022)
task_categories:text-to-video task_categories:video-classification size_categories:10M<n<100M language:en license:cc-by-4.0 captions metadata region:us
{"language": ["en"], "license": "cc-by-4.0", "size_categories": ["10M<n<100M"], "task_categories": ["text-to-video", "video-classification"], "pretty_name": "CleanVid Map (15M)", "dataset_info": {"features": [{"name": "id", "dtype": "int64"}, {"name": "description", "dtype": "string"}, {"name": "duration", "dtype": "float64"}, {"name": "aspectratio", "dtype": "string"}, {"name": "videourl", "dtype": "string"}, {"name": "author", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "framerate", "dtype": "float64"}, {"name": "r18", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 16755833083, "num_examples": 14394510}], "download_size": 5410262648, "dataset_size": 16755833083}, "tags": ["captions", "metadata"]}
TAGS
#task_categories-text-to-video #task_categories-video-classification #size_categories-10M<n<100M #language-English #license-cc-by-4.0 #captions #metadata #region-us 


# CleanVid Map (15M) 
### TempoFunk Video Generation Project

CleanVid-15M is a large-scale dataset of videos with multiple metadata entries such as:
- Textual Descriptions 
- Recording Equipment 
- Categories 
- Framerate ️
- Aspect Ratio 

CleanVid aim is to improve the quality of WebVid-10M dataset by adding more data and cleaning the dataset by dewatermarking the videos in it.

This dataset includes only the map with the urls and metadata, with 3,694,510 more entries than the original WebVid-10M dataset.
Note that the videos are low-resolution, ranging from 240p to 480p. But this shouldn't be a problem as resolution scaling is difficult in Text-To-Video models.
More Datasets to come for high-res use cases.

CleanVid is the foundation dataset for the TempoFunk Video Generation project.

Built from a crawl of Shutterstock from June 25, 2023.

## Format 

- id: Integer (int64) - Shutterstock video ID
- description: String - Description of the video
- duration: Float(64) - Duration of the video in seconds
- aspectratio: String - Aspect Ratio of the video separated by colons (":")
- videourl: String - Video URL for the video in the entry, MP4 format. WEBM format is also available most of the times (by changing the extension at the end of the URL.).
- author: String - JSON-String containing information of the author such as 'Recording Equipment', 'Style', 'Nationality' and others.
- categories: String - JSON-String containing the categories of the videos. (Values from shutterstock, not by us.)
- framerate: Float(64) - Framerate of the video
- r18: Bit (int64) - Wether the video is marked as mature content. 0 = Safe For Work; 1 = Mature Content

## Code ‍

If you want to re-create this dataset on your own, code is available here:

URL

Due to rate-limitations, you might need to obtain a proxy. Functionality for proxies is included in the repository.

## Sample 



## Credits 

### Main

- Lopho - Part of TempoFunk Video Generation
- Chavinlo - Part of TempoFunk Video Generation & CleanVid Crawling, Scraping and Formatting



### Extra

- Salt - Base Threading Code (2022)
# CleanVid Map (15M) ### TempoFunk Video Generation Project

CleanVid-15M is a large-scale dataset of videos with multiple metadata entries such as:
- Textual Descriptions 
- Recording Equipment 
- Categories 
- Framerate ️
- Aspect Ratio 

CleanVid aim is to improve the quality of WebVid-10M dataset by adding more data and cleaning the dataset by dewatermarking the videos in it.

This dataset includes only the map with the urls and metadata, with 3,694,510 more entries than the original WebVid-10M dataset.
Note that the videos are low-resolution, ranging from 240p to 480p. But this shouldn't be a problem as resolution scaling is difficult in Text-To-Video models.
More Datasets to come for high-res use cases.

CleanVid is the foundation dataset for the TempoFunk Video Generation project.

Built from a crawl of Shutterstock from June 25, 2023. ## Format 

- id: Integer (int64) - Shutterstock video ID
- description: String - Description of the video
- duration: Float(64) - Duration of the video in seconds
- aspectratio: String - Aspect Ratio of the video separated by colons (":")
- videourl: String - Video URL for the video in the entry, MP4 format. WEBM format is also available most of the times (by changing the extension at the end of the URL.).
- author: String - JSON-String containing information of the author such as 'Recording Equipment', 'Style', 'Nationality' and others.
- categories: String - JSON-String containing the categories of the videos. (Values from shutterstock, not by us.)
- framerate: Float(64) - Framerate of the video
- r18: Bit (int64) - Wether the video is marked as mature content. 0 = Safe For Work; 1 = Mature Content ## Code ‍

If you want to re-create this dataset on your own, code is available here:

URL

Due to rate-limitations, you might need to obtain a proxy. Functionality for proxies is included in the repository. ## Sample ## Credits ### Main

- Lopho - Part of TempoFunk Video Generation
- Chavinlo - Part of TempoFunk Video Generation & CleanVid Crawling, Scraping and Formatting ### Extra

- Salt - Base Threading Code (2022)
TAGS
#task_categories-text-to-video #task_categories-video-classification #size_categories-10M<n<100M #language-English #license-cc-by-4.0 #captions #metadata #region-us 
 # CleanVid Map (15M) ### TempoFunk Video Generation Project

CleanVid-15M is a large-scale dataset of videos with multiple metadata entries such as:
- Textual Descriptions 
- Recording Equipment 
- Categories 
- Framerate ️
- Aspect Ratio 

CleanVid aim is to improve the quality of WebVid-10M dataset by adding more data and cleaning the dataset by dewatermarking the videos in it.

This dataset includes only the map with the urls and metadata, with 3,694,510 more entries than the original WebVid-10M dataset.
Note that the videos are low-resolution, ranging from 240p to 480p. But this shouldn't be a problem as resolution scaling is difficult in Text-To-Video models.
More Datasets to come for high-res use cases.

CleanVid is the foundation dataset for the TempoFunk Video Generation project.

Built from a crawl of Shutterstock from June 25, 2023. ## Format 

- id: Integer (int64) - Shutterstock video ID
- description: String - Description of the video
- duration: Float(64) - Duration of the video in seconds
- aspectratio: String - Aspect Ratio of the video separated by colons (":")
- videourl: String - Video URL for the video in the entry, MP4 format. WEBM format is also available most of the times (by changing the extension at the end of the URL.).
- author: String - JSON-String containing information of the author such as 'Recording Equipment', 'Style', 'Nationality' and others.
- categories: String - JSON-String containing the categories of the videos. (Values from shutterstock, not by us.)
- framerate: Float(64) - Framerate of the video
- r18: Bit (int64) - Wether the video is marked as mature content. 0 = Safe For Work; 1 = Mature Content ## Code ‍

If you want to re-create this dataset on your own, code is available here:

URL

Due to rate-limitations, you might need to obtain a proxy. Functionality for proxies is included in the repository. ## Sample ## Credits ### Main

- Lopho - Part of TempoFunk Video Generation
- Chavinlo - Part of TempoFunk Video Generation & CleanVid Crawling, Scraping and Formatting ### Extra

- Salt - Base Threading Code (2022)
passage: TAGS
#task_categories-text-to-video #task_categories-video-classification #size_categories-10M<n<100M #language-English #license-cc-by-4.0 #captions #metadata #region-us 
# CleanVid Map (15M)### TempoFunk Video Generation Project

CleanVid-15M is a large-scale dataset of videos with multiple metadata entries such as:
- Textual Descriptions 
- Recording Equipment 
- Categories 
- Framerate ️
- Aspect Ratio 

CleanVid aim is to improve the quality of WebVid-10M dataset by adding more data and cleaning the dataset by dewatermarking the videos in it.

This dataset includes only the map with the urls and metadata, with 3,694,510 more entries than the original WebVid-10M dataset.
Note that the videos are low-resolution, ranging from 240p to 480p. But this shouldn't be a problem as resolution scaling is difficult in Text-To-Video models.
More Datasets to come for high-res use cases.

CleanVid is the foundation dataset for the TempoFunk Video Generation project.

Built from a crawl of Shutterstock from June 25, 2023.## Format 

- id: Integer (int64) - Shutterstock video ID
- description: String - Description of the video
- duration: Float(64) - Duration of the video in seconds
- aspectratio: String - Aspect Ratio of the video separated by colons (":")
- videourl: String - Video URL for the video in the entry, MP4 format. WEBM format is also available most of the times (by changing the extension at the end of the URL.).
- author: String - JSON-String containing information of the author such as 'Recording Equipment', 'Style', 'Nationality' and others.
- categories: String - JSON-String containing the categories of the videos. (Values from shutterstock, not by us.)
- framerate: Float(64) - Framerate of the video
- r18: Bit (int64) - Wether the video is marked as mature content. 0 = Safe For Work; 1 = Mature Content