
# Scalable 3D Captioning with Pretrained Models



## Dataset Description

- **Paper:** [Scalable 3D Captioning with Pretrained Models](https://arxiv.org/abs/2306.07279)
- **Repository**: [Github](https://github.com/crockwell/Cap3D)
- **Project**: [Project](https://cap3d-um.github.io/)



This repository hosts data for [Scalable 3D Captioning with Pretrained Models](https://cap3d-um.github.io/), including descriptive **captions** for 3D objects in [Objaverse](https://arxiv.org/abs/2212.08051) and [ABO](https://arxiv.org/abs/2110.06199) ([example captioning](https://tiangeluo.github.io/projectpages/imgs/Cap3D/000.html)). This repo also includes **point clouds** and **rendered images with camera, depth, and MatAlpha information** of Objaverse objects, as well as their Shap-E latent codes. All the captions and data provided by our paper are released under
ODC-By 1.0 license. 

## Usage
Please download and unzip files from [**Page**](https://huggingface.co/datasets/tiange/Cap3D/tree/main) according to your usage. Below is a table listing fiels descriptions, followed by example Python scripts for data loading.



| Filename                               | Description                                                  |
| -------------------------------------- | ------------------------------------------------------------ |
|**Cap3D_automated_Objaverse_full.csv**  | Combine the text descriptions generated by Cap3D, resulting in **785,150** 3D-caption pairs for the Objaverse dataset.
|**Cap3D_automated_Objaverse_highquality.csv** |In Objaverse, we apply three criteria to filter out low-quality object-caption pairs: (1) captions containing potential inaccuracies; (2) objects scanned within indoor scenes; and (3) objects with potential ethical-issues. Resulting in **549,922** 3D-caption pairs.|
|**Cap3D_automated_Objaverse_no3Dword.csv** | Our current 3D captions are densely packed with "3D-model" terminology, potentially limiting their utility in applications like embodied AI. As such, we've created a version with minimized 3D-related words. For example, "A 3D model of a black and yellow samurai sword." ➡️ "a black and yellow samurai sword."|
| Cap3D_automated_Objaverse.csv          | Combine the text descriptions generated by Cap3D, resulting in **661,577** 3D-caption pairs for the Objaverse dataset. All captions and related 3D objects here have commercial-friendly licenses (including CC-BY 4.0, CC-BY-SA 4.0, and CC0 1.0). We also filter out potential ethical-issues objects (e.g., identifiable face scans, NSFW, etc). 
| Cap3D_automated_ABO.csv                | Text descriptions generated by Cap3D. **6,440** 3D-caption (structural descriptive) pairs for ABO dataset. |
| Cap3D_human_{Objaverse, ABO}.pkl       | Our collected human-authored text descriptions (Mostly English, very few non-English). The pickle file with `shape_id` as key. |
| **PointCloud_zips**                    | **785,150** PointClouds (16,384 colorful points) extracted from Objaverse objects. Saved as `.ply` file. |
| PointCloud_pt_zips                     | PointClouds saved as torch.Tensor `.pt` files, providing faster loading speed than `.ply`. |
| **RenderedImage_perobj_zips**          | Rendered images for Objaverse objects. Once unzip it will have multiple zip files which consists of 20 rendering images along with depth, camera (intrinsic, extrinsic), and MatAlpha information.|

| Filename                               | Description                                                  |
| -------------------------------------- | ------------------------------------------------------------ |
| ShapELatentCode_zips                   | Extracted latent codes for Objaverse objects by using pretrained [Shap-E transmitter](https://github.com/openai/shap-e/blob/main/model-card.md). |
| RenderedImage_zips                     | Rendered images for Objaverse objects: eight unique zip files each contain `661,577` JPEG images, representing different views. |
| our_finetuned_models                   | Model checkpoints for our finetuned models (point-E, shape-E, etc). Our also provide our finetune codes in [Github](https://github.com/crockwell/Cap3D/tree/main/text-to-3D). |
| blender.zip                            | The blender we used in this project.                         |
| compressed_imgs_ABO.zip                | Renderings of ABO objects without textures, used in captioning and evaluation.                        |
| Cap3D_automated_Objaverse_allviews.pkl | Objaverse text descriptions of  before putting into CLIP and GPT4. `pickle.load(open('Cap3D_automated_Objaverse_allviews.pkl','rb'))['view0'][UID]` |
| text-to-3D_test                        | Including files that used to evaluate text-to-3D performance.            |



```
# load our captions
import pandas as pd
captions = pd.read_csv('Cap3D_automated_Objaverse_no3Dword.csv', header=None)

## captions:
##                                        0                                                  1
## 0       ed51a51909ee46c780db3a85e821feb2                           A green and white rifle.
## 1       9110b606f6c547b2980fcb3c8c4b6a1c  a small building with a roof, accompanied by b...
## 2       80d9caaa1fa04502af666135196456e1  a pair of purple and black swords with white h...
## 3       28d43a218cd8466a8c1f82b29b71e314  a small house, island, road with trash, trash ...
## 4       75582285fab442a2ba31733f9c8fae66                              a small, grassy hill.
## ...                                  ...                                                ...
## 661572  ccee95eac2fb48ec92d357e3d853f2bd  a tall, white tower featuring a door, stairs, ...
## 661573  f02f574e85e94b879c1b54f4d3aa4b35           A yellow disc with a hole in the middle.
## 661574  f1d3d36114d34d29a18a8ed1516bf355        pink and white ball with a face and spikes.
## 661575  32d1927d6c0e4d9b9c445fc5988ec6c6  a white and pink bedroom and bathroom, featuri...
## 661576  2fa12ba0af5442c9af8f9bead1a7d020                  Monstera plant with green leaves.

## if u want to obtain the caption for specific UID
caption = captions[captions[0] == '80d9caaa1fa04502af666135196456e1'][1].values[0]


# load point clouds (unzip https://huggingface.co/datasets/tiange/Cap3D/tree/main/PointCloud_pt_zips)
import torch
pts = torch.load('Cap3D_pcs_pt/80d9caaa1fa04502af666135196456e1.pt')

## pts.shape == torch.Size([6, 16384])


```

If you have any questions, please contact [Tiange](mailto:tiange.cs@gmail.com) or [Chris](mailto:cnris@umich.edu).



## Citation Information

If you find our data or code useful, please consider citing:

```bibtex
@article{luo2023scalable,
      title={Scalable 3D Captioning with Pretrained Models},
      author={Luo, Tiange and Rockwell, Chris and Lee, Honglak and Johnson, Justin},
      journal={arXiv preprint arXiv:2306.07279},
      year={2023}
}
```

Please cite ***Objaverse*** and ***ABO*** paper accordingly, if you use related data.

```
@inproceedings{deitke2023objaverse,
  title={Objaverse: A universe of annotated 3d objects},
  author={Deitke, Matt and Schwenk, Dustin and Salvador, Jordi and Weihs, Luca and Michel, Oscar and VanderBilt, Eli and Schmidt, Ludwig and Ehsani, Kiana and Kembhavi, Aniruddha and Farhadi, Ali},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13142--13153},
  year={2023}
}
```

```
@inproceedings{collins2022abo,
  title={Abo: Dataset and benchmarks for real-world 3d object understanding},
  author={Collins, Jasmine and Goel, Shubham and Deng, Kenan and Luthra, Achleshwar and Xu, Leon and Gundogdu, Erhan and Zhang, Xi and Vicente, Tomas F Yago and Dideriksen, Thomas and Arora, Himanshu and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21126--21136},
  year={2022}
}
```

task_categories:text-to-3d license:odc-by arxiv:2306.07279 arxiv:2212.08051 arxiv:2110.06199 region:us
{"license": "odc-by", "task_categories": ["text-to-3d"], "viewer": false}
TAGS
#task_categories-text-to-3d #license-odc-by #arxiv-2306.07279 #arxiv-2212.08051 #arxiv-2110.06199 #region-us 

Scalable 3D Captioning with Pretrained Models
=============================================


Dataset Description
-------------------


* Paper: Scalable 3D Captioning with Pretrained Models
* Repository: Github
* Project: Project


This repository hosts data for Scalable 3D Captioning with Pretrained Models, including descriptive captions for 3D objects in Objaverse and ABO (example captioning). This repo also includes point clouds and rendered images with camera, depth, and MatAlpha information of Objaverse objects, as well as their Shap-E latent codes. All the captions and data provided by our paper are released under
ODC-By 1.0 license.


Usage
-----


Please download and unzip files from Page according to your usage. Below is a table listing fiels descriptions, followed by example Python scripts for data loading.




If you have any questions, please contact Tiange or Chris.


If you find our data or code useful, please consider citing:


Please cite *Objaverse* and *ABO* paper accordingly, if you use related data.



TAGS
#task_categories-text-to-3d #license-odc-by #arxiv-2306.07279 #arxiv-2212.08051 #arxiv-2110.06199 #region-us 
