HunyuanVideo: A Systematic Framework For Large Video Generation Model Training

This repo contains PyTorch model definitions, pre-trained weights and inference/sampling code for our paper exploring HunyuanVideo. You can find more visualizations on our project page.

HunyuanVideo: A Systematic Framework For Large Video Generation Model Training

🔥🔥🔥 News!!

Dec 3, 2024: 🤗 We release the inference code and model weights of HunyuanVideo.

📑 Open-source Plan

HunyuanVideo (Text-to-Video Model)
- Inference
- Checkpoints
- Penguin Video Benchmark
- Web Demo (Gradio)
- ComfyUI
- Diffusers
HunyuanVideo (Image-to-Video Model)
- Inference
- Checkpoints

HunyuanVideo: A Systematic Framework For Large Video Generation Model Training

Abstract

We present HunyuanVideo, a novel open-source video foundation model that exhibits performance in video generation that is comparable to, if not superior to, leading closed-source models. HunyuanVideo features a comprehensive framework that integrates several key contributions, including data curation, image-video joint model training, and an efficient infrastructure designed to facilitate large-scale model training and inference. Additionally, through an effective strategy for scaling model architecture and dataset, we successfully trained a video generative model with over 13 billion parameters, making it the largest among all open-source models.

We conducted extensive experiments and implemented a series of targeted designs to ensure high visual quality, motion diversity, text-video alignment, and generation stability. According to professional human evaluation results, HunyuanVideo outperforms previous state-of-the-art models, including Runway Gen-3, Luma 1.6, and 3 top performing Chinese video generative models. By releasing the code and weights of the foundation model and its applications, we aim to bridge the gap between closed-source and open-source video foundation models. This initiative will empower everyone in the community to experiment with their ideas, fostering a more dynamic and vibrant video generation ecosystem.

HunyuanVideo Overall Architechture

HunyuanVideo is trained on a spatial-temporally compressed latent space, which is compressed through Causal 3D VAE. Text prompts are encoded using a large language model, and used as the condition. Gaussian noise and condition are taken as input, our generate model generates an output latent, which is decoded to images or videos through the 3D VAE decoder.

🎉 HunyuanVideo Key Features

Unified Image and Video Generative Architecture

HunyuanVideo introduces the Transformer design and employs a Full Attention mechanism for unified image and video generation. Specifically, we use a "Dual-stream to Single-stream" hybrid model design for video generation. In the dual-stream phase, video and text tokens are processed independently through multiple Transformer blocks, enabling each modality to learn its own appropriate modulation mechanisms without interference. In the single-stream phase, we concatenate the video and text tokens and feed them into subsequent Transformer blocks for effective multimodal information fusion. This design captures complex interactions between visual and semantic information, enhancing overall model performance.

MLLM Text Encoder

Some previous text-to-video model typically use pretrainednCLIP and T5-XXL as text encoders where CLIP uses Transformer Encoder and T5 uses a Encoder-Decoder structure. In constrast, we utilize a pretrained Multimodal Large Language Model (MLLM) with a Decoder-Only structure as our text encoder, which has following advantages: (i) Compared with T5, MLLM after visual instruction finetuning has better image-text alignment in the feature space, which alleviates the difficulty of instruction following in diffusion models; (ii) Compared with CLIP, MLLM has been demonstrated superior ability in image detail description and complex reasoning; (iii) MLLM can play as a zero-shot learner by following system instructions prepended to user prompts, helping text features pay more attention to key information. In addition, MLLM is based on causal attention while T5-XXL utilizes bidirectional attention that produces better text guidance for diffusion models. Therefore, we introduce an extra bidirectional token refiner for enhacing text features.

3D VAE

HunyuanVideo trains a 3D VAE with CausalConv3D to compress pixel-space videos and images into a compact latent space. We set the compression ratios of video length, space and channel to 4, 8 and 16 respectively. This can significantly reduce the number of tokens for the subsequent diffusion transformer model, allowing us to train videos at the original resolution and frame rate.

Prompt Rewrite

To address the variability in linguistic style and length of user-provided prompts, we fine-tune the Hunyuan-Large model as our prompt rewrite model to adapt the original user prompt to model-preferred prompt.

We provide two rewrite modes: Normal mode and Master mode, which can be called using different prompts. The Normal mode is designed to enhance the video generation model's comprehension of user intent, facilitating a more accurate interpretation of the instructions provided. The Master mode enhances the description of aspects such as composition, lighting, and camera movement, which leans towards generating videos with a higher visual quality. However, this emphasis may occasionally result in the loss of some semantic details.

The Prompt Rewrite Model can be directly deployed and inferred using the Hunyuan-Large original code. We release the weights of the Prompt Rewrite Model here.

📈 Comparisons

To evaluate the performance of HunyuanVideo, we selected five strong baselines from closed-source video generation models. In total, we utilized 1,533 text prompts, generating an equal number of video samples with HunyuanVideo in a single run. For a fair comparison, we conducted inference only once, avoiding any cherry-picking of results. When comparing with the baseline methods, we maintained the default settings for all selected models, ensuring consistent video resolution. Videos were assessed based on three criteria: Text Alignment, Motion Quality and Visual Quality. More than 60 professional evaluators performed the evaluation. Notably, HunyuanVideo demonstrated the best overall performance, particularly excelling in motion quality.

Model	Open Source	Duration	Text Alignment	Motion Quality	Visual Quality	Overall	Ranking
Model	Open Source	HunyuanVideo (Ours)	✔	5s	Visual Quality	Overall	Ranking	61.8%	66.5%	95.7%	41.3%	1
CNTopA (API)	✘	5s	62.6%	61.7%	95.6%	37.7%	2
CNTopB (Web)	✘	5s	60.1%	62.9%	97.7%	37.5%	3
GEN-3 alpha (Web)	✘	6s	47.7%	54.7%	97.5%	27.4%	4
Luma1.6 (API)	✘	5s	57.6%	44.2%	94.1%	24.8%	6
CNTopC (Web)	✘	5s	48.4%	47.2%	96.3%	24.6%	5

📜 Requirements

The following table shows the requirements for running HunyuanVideo model (batch size = 1) to generate videos:

Model	Setting (height/width/frame)	GPU Peak Memory
HunyuanVideo	720px1280px129f	60GB
HunyuanVideo	544px960px129f	45GB

An NVIDIA GPU with CUDA support is required.
- The model is tested on a single 80G GPU.
- Minimum: The minimum GPU memory required is 60GB for 720px1280px129f and 45G for 544px960px129f.
- Recommended: We recommend using a GPU with 80GB of memory for better generation quality.
Tested operating system: Linux

🛠️ Dependencies and Installation

Begin by cloning the repository:

git clone https://github.com/tencent/HunyuanVideo
cd HunyuanVideo

Installation Guide for Linux

We provide an environment.yml file for setting up a Conda environment. Conda's installation instructions are available here.

We recommend CUDA versions 11.8 and 12.0+.

# 1. Prepare conda environment
conda env create -f environment.yml

# 2. Activate the environment
conda activate HunyuanVideo

# 3. Install pip dependencies
python -m pip install -r requirements.txt

# 4. Install flash attention v2 for acceleration (requires CUDA 11.8 or above)
python -m pip install git+https://github.com/Dao-AILab/[email protected]

Additionally, HunyuanVideo also provides a pre-built Docker image: docker_hunyuanvideo.

# 1. Use the following link to download the docker image tar file (For CUDA 12).
wget https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/hunyuan_video_cu12.tar

# 2. Import the docker tar file and show the image meta information (For CUDA 12).
docker load -i hunyuan_video.tar

docker image ls

# 3. Run the container based on the image
docker run -itd --gpus all --init --net=host --uts=host --ipc=host --name hunyuanvideo --security-opt=seccomp=unconfined --ulimit=stack=67108864 --ulimit=memlock=-1 --privileged  docker_image_tag

🧱 Download Pretrained Models

The details of download pretrained models are shown here.

🔑 Inference

We list the height/width/frame settings we support in the following table.

Resolution	h/w=9:16	h/w=16:9	h/w=4:3	h/w=3:4	h/w=1:1
540p	544px960px129f	960px544px129f	624px832px129f	832px624px129f	720px720px129f
720p (recommended)	720px1280px129f	1280px720px129f	1104px832px129f	832px1104px129f	960px960px129f

Using Command Line

cd HunyuanVideo

python3 sample_video.py \
    --video-size 720 1280 \
    --video-length 129 \
    --infer-steps 30 \
    --prompt "a cat is running, realistic." \
    --flow-reverse \
    --seed 0 \
    --use-cpu-offload \
    --save-path ./results

More Configurations

We list some more useful configurations for easy usage:

Argument	Default	Description
`--prompt`	None	The text prompt for video generation
`--video-size`	720 1280	The size of the generated video
`--video-length`	129	The length of the generated video
`--infer-steps`	30	The number of steps for sampling
`--embedded-cfg-scale`	6.0	Embeded Classifier free guidance scale
`--flow-shift`	9.0	Shift factor for flow matching schedulers
`--flow-reverse`	False	If reverse, learning/sampling from t=1 -> t=0
`--neg-prompt`	None	The negative prompt for video generation
`--seed`	0	The random seed for generating video
`--use-cpu-offload`	False	Use CPU offload for the model load to save more memory, necessary for high-res video generation
`--save-path`	./results	Path to save the generated video

🔗 BibTeX

If you find HunyuanVideo useful for your research and applications, please cite using this BibTeX:

@misc{kong2024hunyuanvideo,
      title={HunyuanVideo: A Systematic Framework For Large Video Generative Models}, 
      author={Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Junkun Yuan, Kai Wang, Mengyang Liu, Pengyu Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yanxin Long, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, and Jie Jiang, along with Caesar Zhong},
      year={2024},
      archivePrefix={arXiv preprint arXiv:2412.03603},
      primaryClass={cs.CV}
}

Acknowledgements

We would like to thank the contributors to the SD3, FLUX, Llama, LLaVA, Xtuner, diffusers and HuggingFace repositories, for their open research and exploration. Additionally, we also thank the Tencent Hunyuan Multimodal team for their help with the text encoder.

tencent
/

HunyuanVideo