rupal009 commited on
Commit
e20d361
·
verified ·
1 Parent(s): b958020

Delete README.md

Browse files
Files changed (1) hide show
  1. README.md +0 -300
README.md DELETED
@@ -1,300 +0,0 @@
1
- ## ___***ToonCrafter: Generative Cartoon Interpolation***___
2
- <!-- ![](./assets/logo_long.png#gh-light-mode-only){: width="50%"} -->
3
- <!-- ![](./assets/logo_long_dark.png#gh-dark-mode-only=100x20) -->
4
- <div align="center">
5
- <img src='assets/logo/logo2.png' style="height:100px"></img>
6
-
7
- <a href='https://arxiv.org/abs/2405.17933'><img src='https://img.shields.io/badge/arXiv-2405.17933-b31b1b.svg'></a> &nbsp;
8
- <a href='https://doubiiu.github.io/projects/ToonCrafter/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> &nbsp;
9
- <a href='https://www.youtube.com/watch?v=u3F35do93_8'><img src='https://img.shields.io/badge/Youtube-Video-b31b1b.svg'></a><br>
10
- <a href='https://replicate.com/fofr/tooncrafter'><img src='https://img.shields.io/badge/replicate-Demo-blue'></a>&nbsp;&nbsp;
11
- <a href='https://github.com/camenduru/ToonCrafter-jupyter'><img src='https://img.shields.io/badge/Colab-Demo-Green'></a>&nbsp;
12
- <a href='https://huggingface.co/spaces/Doubiiu/tooncrafter'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face%20ToonCrafter-Demo-blue'></a>
13
-
14
-
15
- _**[Jinbo Xing](https://doubiiu.github.io/), [Hanyuan Liu](https://github.com/hyliu), [Menghan Xia](https://menghanxia.github.io), [Yong Zhang](https://yzhang2016.github.io), [Xintao Wang](https://xinntao.github.io/), [Ying Shan](https://scholar.google.com/citations?hl=en&user=4oXBp9UAAAAJ&view_op=list_works&sortby=pubdate), [Tien-Tsin Wong](https://ttwong12.github.io/myself.html)**_
16
- <br><br>
17
- From CUHK and Tencent AI Lab.
18
-
19
- <strong>at SIGGRAPH Asia 2024, Journal Track</strong>
20
-
21
-
22
- </div>
23
-
24
- ## 🔆 Introduction
25
-
26
- ⚠️ Please check our [disclaimer](#disc) first.
27
-
28
- 🤗 ToonCrafter can interpolate two cartoon images by leveraging the pre-trained image-to-video diffusion priors. Please check our project page and paper for more information. <br>
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
- ### 1.1 Showcases (512x320)
37
- <table class="center">
38
- <tr style="font-weight: bolder;text-align:center;">
39
- <td>Input starting frame</td>
40
- <td>Input ending frame</td>
41
- <td>Generated video</td>
42
- </tr>
43
- <tr>
44
- <td>
45
- <img src=assets/72109_125.mp4_00-00.png width="250">
46
- </td>
47
- <td>
48
- <img src=assets/72109_125.mp4_00-01.png width="250">
49
- </td>
50
- <td>
51
- <img src=assets/00.gif width="250">
52
- </td>
53
- </tr>
54
-
55
-
56
- <tr>
57
- <td>
58
- <img src=assets/Japan_v2_2_062266_s2_frame1.png width="250">
59
- </td>
60
- <td>
61
- <img src=assets/Japan_v2_2_062266_s2_frame3.png width="250">
62
- </td>
63
- <td>
64
- <img src=assets/03.gif width="250">
65
- </td>
66
- </tr>
67
- <tr>
68
- <td>
69
- <img src=assets/Japan_v2_1_070321_s3_frame1.png width="250">
70
- </td>
71
- <td>
72
- <img src=assets/Japan_v2_1_070321_s3_frame3.png width="250">
73
- </td>
74
- <td>
75
- <img src=assets/02.gif width="250">
76
- </td>
77
- </tr>
78
- <tr>
79
- <td>
80
- <img src=assets/74302_1349_frame1.png width="250">
81
- </td>
82
- <td>
83
- <img src=assets/74302_1349_frame3.png width="250">
84
- </td>
85
- <td>
86
- <img src=assets/01.gif width="250">
87
- </td>
88
- </tr>
89
- </table>
90
-
91
- ### 1.2 Sparse sketch guidance
92
- <table class="center">
93
- <tr style="font-weight: bolder;text-align:center;">
94
- <td>Input starting frame</td>
95
- <td>Input ending frame</td>
96
- <td>Input sketch guidance</td>
97
- <td>Generated video</td>
98
- </tr>
99
- <tr>
100
- <td>
101
- <img src=assets/72105_388.mp4_00-00.png width="200">
102
- </td>
103
- <td>
104
- <img src=assets/72105_388.mp4_00-01.png width="200">
105
- </td>
106
- <td>
107
- <img src=assets/06.gif width="200">
108
- </td>
109
- <td>
110
- <img src=assets/07.gif width="200">
111
- </td>
112
- </tr>
113
-
114
- <tr>
115
- <td>
116
- <img src=assets/72110_255.mp4_00-00.png width="200">
117
- </td>
118
- <td>
119
- <img src=assets/72110_255.mp4_00-01.png width="200">
120
- </td>
121
- <td>
122
- <img src=assets/12.gif width="200">
123
- </td>
124
- <td>
125
- <img src=assets/13.gif width="200">
126
- </td>
127
- </tr>
128
-
129
-
130
- </table>
131
-
132
-
133
- ### 2. Applications
134
- #### 2.1 Cartoon Sketch Interpolation (see project page for more details)
135
- <table class="center">
136
- <tr style="font-weight: bolder;text-align:center;">
137
- <td>Input starting frame</td>
138
- <td>Input ending frame</td>
139
- <td>Generated video</td>
140
- </tr>
141
-
142
- <tr>
143
- <td>
144
- <img src=assets/frame0001_10.png width="250">
145
- </td>
146
- <td>
147
- <img src=assets/frame0016_10.png width="250">
148
- </td>
149
- <td>
150
- <img src=assets/10.gif width="250">
151
- </td>
152
- </tr>
153
-
154
-
155
- <tr>
156
- <td>
157
- <img src=assets/frame0001_11.png width="250">
158
- </td>
159
- <td>
160
- <img src=assets/frame0016_11.png width="250">
161
- </td>
162
- <td>
163
- <img src=assets/11.gif width="250">
164
- </td>
165
- </tr>
166
-
167
- </table>
168
-
169
-
170
- #### 2.2 Reference-based Sketch Colorization
171
- <table class="center">
172
- <tr style="font-weight: bolder;text-align:center;">
173
- <td>Input sketch</td>
174
- <td>Input reference</td>
175
- <td>Colorization results</td>
176
- </tr>
177
-
178
- <tr>
179
- <td>
180
- <img src=assets/04.gif width="250">
181
- </td>
182
- <td>
183
- <img src=assets/frame0001_05.png width="250">
184
- </td>
185
- <td>
186
- <img src=assets/05.gif width="250">
187
- </td>
188
- </tr>
189
-
190
-
191
- <tr>
192
- <td>
193
- <img src=assets/08.gif width="250">
194
- </td>
195
- <td>
196
- <img src=assets/frame0001_09.png width="250">
197
- </td>
198
- <td>
199
- <img src=assets/09.gif width="250">
200
- </td>
201
- </tr>
202
-
203
- </table>
204
-
205
-
206
-
207
-
208
-
209
-
210
-
211
- ## 📝 Changelog
212
- - [ ] Add sketch control and colorization function.
213
- - __[2024.05.29]__: 🔥🔥 Release code and model weights.
214
- - __[2024.05.28]__: Launch the project page and update the arXiv preprint.
215
- <br>
216
-
217
-
218
- ## 🧰 Models
219
-
220
- |Model|Resolution|GPU Mem. & Inference Time (A100, ddim 50steps)|Checkpoint|
221
- |:---------|:---------|:--------|:--------|
222
- |ToonCrafter_512|320x512| ~24G & 24s (`perframe_ae=True`)|[Hugging Face](https://huggingface.co/Doubiiu/ToonCrafter/blob/main/model.ckpt)|
223
-
224
- We get the feedback from issues that the model may consume about 24G~27G GPU memory in this implementation, but the community has lowered the consumption to ~10GB.
225
-
226
- Currently, our ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.
227
-
228
-
229
-
230
- ## ⚙️ Setup
231
-
232
- ### Install Environment via Anaconda (Recommended)
233
- ```bash
234
- conda create -n tooncrafter python=3.8.5
235
- conda activate tooncrafter
236
- pip install -r requirements.txt
237
- ```
238
-
239
-
240
- ## 💫 Inference
241
- ### 1. Command line
242
-
243
- Download pretrained ToonCrafter_512 and put the `model.ckpt` in `checkpoints/tooncrafter_512_interp_v1/model.ckpt`.
244
- ```bash
245
- sh scripts/run.sh
246
- ```
247
-
248
-
249
- ### 2. Local Gradio demo
250
-
251
- Download the pretrained model and put it in the corresponding directory according to the previous guidelines.
252
- ```bash
253
- python gradio_app.py
254
- ```
255
-
256
-
257
-
258
-
259
-
260
-
261
- ## 🤝 Community Support
262
- 1. ComfyUI and pruned models (fp16): [ComfyUI-DynamiCrafterWrapper](https://github.com/kijai/ComfyUI-DynamiCrafterWrapper) (Thanks to [kijai](https://twitter.com/kijaidesign))
263
-
264
- |Model|Resolution|GPU Mem. |Checkpoint|
265
- |:---------|:---------|:--------|:--------|
266
- |ToonCrafter|512x320|12GB |[Hugging Face](https://huggingface.co/Kijai/DynamiCrafter_pruned/blob/main/tooncrafter_512_interp-fp16.safetensors)|
267
-
268
- 2. ComfyUI. [ComfyUI-ToonCrafter](https://github.com/AIGODLIKE/ComfyUI-ToonCrafter) (Thanks to [Yorha4D](https://github.com/Yorha4D))
269
-
270
- 3. Colab. [Code](https://github.com/camenduru/ToonCrafter-jupyter) (Thanks to [camenduru](https://github.com/camenduru)), [Code](https://gist.github.com/0smboy/baef995b8f5974f19ac114ec20ac37d5) (Thanks to [0smboy](https://github.com/0smboy))
271
-
272
- 4. Windows platform support: [ToonCrafter-for-windows](https://github.com/sdbds/ToonCrafter-for-windows) (Thanks to [sdbds](https://github.com/sdbds))
273
-
274
- 5. Sketch-guidance implementation: [ToonCrafter_with_SketchGuidance](https://github.com/mattyamonaca/ToonCrafter_with_SketchGuidance) (Thanks to [mattyamonaca](https://github.com/mattyamonaca))
275
-
276
- ## 😉 Citation
277
- Please consider citing our paper if our code is useful:
278
- ```bib
279
- @article{xing2024tooncrafter,
280
- title={ToonCrafter: Generative Cartoon Interpolation},
281
- author={Xing, Jinbo and Liu, Hanyuan and Xia, Menghan and Zhang, Yong and Wang, Xintao and Shan, Ying and Wong, Tien-Tsin},
282
- journal={arXiv preprint arXiv:2405.17933},
283
- year={2024}
284
- }
285
- ```
286
-
287
-
288
- ## 🙏 Acknowledgements
289
- We would like to thank [Xiaoyu](https://engineering.purdue.edu/people/xiaoyu.xiang.1) for providing the [sketch extractor](https://github.com/Mukosame/Anime2Sketch), and [supraxylon](https://github.com/supraxylon) for the Windows batch script.
290
-
291
- <a name="disc"></a>
292
- ## 📢 Disclaimer
293
- We have not set up any official profit-making projects or web applications. Please be cautious.
294
-
295
- Calm down. Our framework opens up the era of generative cartoon interpolation, but due to the variaity of generative video prior, the success rate is not guaranteed.
296
-
297
- ⚠️This is an open-source research exploration, instead of commercial products. It can't meet all your expectations.
298
-
299
- This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.
300
- ****