|
--- |
|
language: |
|
- "en" |
|
tags: |
|
- video |
|
license: apache-2.0 |
|
pipeline_tag: text-to-video |
|
library_name: diffusers |
|
--- |
|
|
|
<p align="center"> |
|
<img src="assets/logo.jpg" height=30> |
|
</p> |
|
|
|
# FastMochi Model Card |
|
|
|
## Model Details |
|
|
|
<div align="center"> |
|
<table style="margin-left: auto; margin-right: auto; border: none;"> |
|
<tr> |
|
<td> |
|
<img src="assets/mochi-demo.gif" width="640" alt="Mochi Demo"> |
|
</td> |
|
</tr> |
|
<tr> |
|
<td style="text-align:center;"> |
|
Get 8X diffusion boost for Mochi with FastVideo |
|
</td> |
|
</tr> |
|
</table> |
|
</div> |
|
|
|
FastMochi is an accelerated [Mochi](https://huggingface.co/genmo/mochi-1-preview) model. It can sample high quality videos with 8 diffusion steps. That brings around 8X speed up compared to the original Mochu with 64 steps. |
|
|
|
- **Developed by**: [Hao AI Lab](https://hao-ai-lab.github.io/) |
|
- **License**: Apache-2.0 |
|
- **Distilled from**: [Mochi](https://huggingface.co/genmo/mochi-1-preview) |
|
- **Github Repository**: https://github.com/hao-ai-lab/FastVideo |
|
|
|
## Usage |
|
|
|
- Clone [Fastvideo](https://github.com/hao-ai-lab/FastVideo) repository and follow the inference instructions in the README. |
|
- You can also run FastMochi using the official [Mochi repository](https://github.com/Tencent/HunyuanVideo) with the script below and this [compatible weight](https://huggingface.co/FastVideo/FastMochi). |
|
|
|
<details> |
|
<summary>Code</summary> |
|
|
|
```python |
|
from genmo.mochi_preview.pipelines import ( |
|
DecoderModelFactory, |
|
DitModelFactory, |
|
MochiMultiGPUPipeline, |
|
T5ModelFactory, |
|
linear_quadratic_schedule, |
|
) |
|
from genmo.lib.utils import save_video |
|
import os |
|
|
|
with open("prompt.txt", "r") as f: |
|
prompts = [line.rstrip() for line in f] |
|
|
|
pipeline = MochiMultiGPUPipeline( |
|
text_encoder_factory=T5ModelFactory(), |
|
world_size=4, |
|
dit_factory=DitModelFactory( |
|
model_path=f"weights/dit.safetensors", model_dtype="bf16" |
|
), |
|
decoder_factory=DecoderModelFactory( |
|
model_path=f"weights/decoder.safetensors", |
|
), |
|
) |
|
# read prompt line by line from prompt.txt |
|
|
|
|
|
output_dir = "outputs" |
|
os.makedirs(output_dir, exist_ok=True) |
|
for i, prompt in enumerate(prompts): |
|
video = pipeline( |
|
height=480, |
|
width=848, |
|
num_frames=163, |
|
num_inference_steps=8, |
|
sigma_schedule=linear_quadratic_schedule(8, 0.1, 6), |
|
cfg_schedule=[1.5] * 8, |
|
batch_cfg=False, |
|
prompt=prompt, |
|
negative_prompt="", |
|
seed=12345, |
|
)[0] |
|
save_video(video, f"{output_dir}/output_{i}.mp4") |
|
``` |
|
|
|
</details> |
|
|
|
|
|
## Training details |
|
|
|
FastMochi is consistency distillated on the [MixKit](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main) dataset with the following hyperparamters: |
|
- Batch size: 32 |
|
- Resulotion: 480X848 |
|
- Num of frames: 169 |
|
- Train steps: 128 |
|
- GPUs: 16 |
|
- LR: 1e-6 |
|
- Loss: huber |
|
|
|
## Evaluation |
|
We provide some qualitative comparisons between FastMochi 8 step inference v.s. the original Mochi with 8 step inference: |
|
|
|
|
|
| FastMochi 6 steps | Mochi 6 steps | |
|
| --- | --- | |
|
|  |  | |
|
|  |  | |
|
|  |  | |
|
|  |  | |
|
|
|
|
