Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation

Mingwang Xu^1* Hui Li^1* Qingkun Su^1* Hanlin Shang¹ Liwei Zhang¹ Ce Liu³

Jingdong Wang² Yao Yao⁴ Siyu Zhu¹

¹Fudan University ²Baidu Inc ³ETH Zurich ⁴Nanjing University

## 📸 Showcase https://github.com/fudan-generative-vision/hallo/assets/17402682/9d1a0de4-3470-4d38-9e4f-412f517f834c ### 🎬 Honoring Classic Films

Devil Wears Prada	Green Book	Infernal Affairs

Patch Adams	Tough Love	Shawshank Redemption

Explore [more examples](https://fudan-generative-vision.github.io/hallo). ## 📰 News - **`2024/06/21`**: 🚀🚀🚀 Cloned a Gradio demo on [🤗Huggingface space](https://huggingface.co/spaces/fudan-generative-ai/hallo). - **`2024/06/20`**: 🌟🌟🌟 Received numerous contributions from the community, including a [Windows version](https://github.com/sdbds/hallo-for-windows), [ComfyUI](https://github.com/AIFSH/ComfyUI-Hallo), [WebUI](https://github.com/fudan-generative-vision/hallo/pull/51), and [Docker template](https://github.com/ashleykleynhans/hallo-docker). - **`2024/06/15`**: ✨✨✨ Released some images and audios for inference testing on [🤗Huggingface](https://huggingface.co/datasets/fudan-generative-ai/hallo_inference_samples). - **`2024/06/15`**: 🎉🎉🎉 Launched the first version on 🫡[GitHub](https://github.com/fudan-generative-vision/hallo). ## 🤝 Community Resources Explore the resources developed by our community to enhance your experience with Hallo: - [Demo on Huggingface](https://huggingface.co/spaces/multimodalart/hallo) - Check out this easy-to-use Gradio demo by [@multimodalart](https://huggingface.co/multimodalart). - [hallo-webui](https://github.com/daswer123/hallo-webui) - Explore the WebUI created by [@daswer123](https://github.com/daswer123). - [hallo-for-windows](https://github.com/sdbds/hallo-for-windows) - Utilize Hallo on Windows with the guide by [@sdbds](https://github.com/sdbds). - [ComfyUI-Hallo](https://github.com/AIFSH/ComfyUI-Hallo) - Integrate Hallo with the ComfyUI tool by [@AIFSH](https://github.com/AIFSH). - [hallo-docker](https://github.com/ashleykleynhans/hallo-docker) - Docker image for Hallo by [@ashleykleynhans](https://github.com/ashleykleynhans). - [RunPod Template](https://runpod.io/console/deploy?template=aeyibwyvzy&ref=2xxro4syy) - Deploy Hallo to RunPod by [@ashleykleynhans](https://github.com/ashleykleynhans). Thanks to all of them. Join our community and explore these amazing resources to make the most out of Hallo. Enjoy and elevate their creative projects! ## 🔧️ Framework ![abstract](assets/framework_1.jpg) ![framework](assets/framework_2.jpg) ## ⚙️ Installation - System requirement: Ubuntu 20.04/Ubuntu 22.04, Cuda 12.1 - Tested GPUs: A100 Create conda environment: ```bash conda create -n hallo python=3.10 conda activate hallo ``` Install packages with `pip` ```bash pip install -r requirements.txt pip install . ``` Besides, ffmpeg is also needed: ```bash apt-get install ffmpeg ``` ## 🗝️️ Usage The entry point for inference is `scripts/inference.py`. Before testing your cases, two preparations need to be completed: 1. [Download all required pretrained models](#download-pretrained-models). 2. [Prepare source image and driving audio pairs](#prepare-inference-data). 3. [Run inference](#run-inference). ### 📥 Download Pretrained Models You can easily get all pretrained models required by inference from our [HuggingFace repo](https://huggingface.co/fudan-generative-ai/hallo). Clone the pretrained models into `${PROJECT_ROOT}/pretrained_models` directory by cmd below: ```shell git lfs install git clone https://huggingface.co/fudan-generative-ai/hallo pretrained_models ``` Or you can download them separately from their source repo: - [hallo](https://huggingface.co/fudan-generative-ai/hallo/tree/main/hallo): Our checkpoints consist of denoising UNet, face locator, image & audio proj. - [audio_separator](https://huggingface.co/huangjackson/Kim_Vocal_2): Kim\_Vocal\_2 MDX-Net vocal removal model. (_Thanks to [KimberleyJensen](https://github.com/KimberleyJensen)_) - [insightface](https://github.com/deepinsight/insightface/tree/master/python-package#model-zoo): 2D and 3D Face Analysis placed into `pretrained_models/face_analysis/models/`. (_Thanks to deepinsight_) - [face landmarker](https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task): Face detection & mesh model from [mediapipe](https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker#models) placed into `pretrained_models/face_analysis/models`. - [motion module](https://github.com/guoyww/AnimateDiff/blob/main/README.md#202309-animatediff-v2): motion module from [AnimateDiff](https://github.com/guoyww/AnimateDiff). (_Thanks to [guoyww](https://github.com/guoyww)_). - [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse): Weights are intended to be used with the diffusers library. (_Thanks to [stablilityai](https://huggingface.co/stabilityai)_) - [StableDiffusion V1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5): Initialized and fine-tuned from Stable-Diffusion-v1-2. (_Thanks to [runwayml](https://huggingface.co/runwayml)_) - [wav2vec](https://huggingface.co/facebook/wav2vec2-base-960h): wav audio to vector model from [Facebook](https://huggingface.co/facebook/wav2vec2-base-960h). Finally, these pretrained models should be organized as follows: ```text ./pretrained_models/ |-- audio_separator/ | |-- download_checks.json | |-- mdx_model_data.json | |-- vr_model_data.json | `-- Kim_Vocal_2.onnx |-- face_analysis/ | `-- models/ | |-- face_landmarker_v2_with_blendshapes.task # face landmarker model from mediapipe | |-- 1k3d68.onnx | |-- 2d106det.onnx | |-- genderage.onnx | |-- glintr100.onnx | `-- scrfd_10g_bnkps.onnx |-- motion_module/ | `-- mm_sd_v15_v2.ckpt |-- sd-vae-ft-mse/ | |-- config.json | `-- diffusion_pytorch_model.safetensors |-- stable-diffusion-v1-5/ | `-- unet/ | |-- config.json | `-- diffusion_pytorch_model.safetensors `-- wav2vec/ `-- wav2vec2-base-960h/ |-- config.json |-- feature_extractor_config.json |-- model.safetensors |-- preprocessor_config.json |-- special_tokens_map.json |-- tokenizer_config.json `-- vocab.json ``` ### 🛠️ Prepare Inference Data Hallo has a few simple requirements for input data: For the source image: 1. It should be cropped into squares. 2. The face should be the main focus, making up 50%-70% of the image. 3. The face should be facing forward, with a rotation angle of less than 30° (no side profiles). For the driving audio: 1. It must be in WAV format. 2. It must be in English since our training datasets are only in this language. 3. Ensure the vocals are clear; background music is acceptable. We have provided [some samples](examples/) for your reference. ### 🎮 Run Inference Simply to run the `scripts/inference.py` and pass `source_image` and `driving_audio` as input: ```bash python scripts/inference.py --source_image examples/reference_images/1.jpg --driving_audio examples/driving_audios/1.wav ``` Animation results will be saved as `${PROJECT_ROOT}/.cache/output.mp4` by default. You can pass `--output` to specify the output file name. You can find more examples for inference at [examples folder](https://github.com/fudan-generative-vision/hallo/tree/main/examples). For more options: ```shell usage: inference.py [-h] [-c CONFIG] [--source_image SOURCE_IMAGE] [--driving_audio DRIVING_AUDIO] [--output OUTPUT] [--pose_weight POSE_WEIGHT] [--face_weight FACE_WEIGHT] [--lip_weight LIP_WEIGHT] [--face_expand_ratio FACE_EXPAND_RATIO] options: -h, --help show this help message and exit -c CONFIG, --config CONFIG --source_image SOURCE_IMAGE source image --driving_audio DRIVING_AUDIO driving audio --output OUTPUT output video file name --pose_weight POSE_WEIGHT weight of pose --face_weight FACE_WEIGHT weight of face --lip_weight LIP_WEIGHT weight of lip --face_expand_ratio FACE_EXPAND_RATIO face region ``` ## 📅️ Roadmap | Status | Milestone | ETA | | :----: | :---------------------------------------------------------------------------------------------------- | :--------: | | ✅ | **[Inference source code meet everyone on GitHub](https://github.com/fudan-generative-vision/hallo)** | 2024-06-15 | | ✅ | **[Pretrained models on Huggingface](https://huggingface.co/fudan-generative-ai/hallo)** | 2024-06-15 | | 🚧 | **[Optimizing Performance on images with a resolution of 256x256.]()** | 2024-06-23 | | 🚀 | **[Improving the model's performance on Mandarin Chinese]()** | 2024-06-25 | | 🚀 | **[Releasing data preparation and training scripts]()** | 2024-06-28 |

Other Enhancements

- [x] Enhancement: Test and ensure compatibility with Windows operating system. [#39](https://github.com/fudan-generative-vision/hallo/issues/39) - [x] Bug: Output video may lose several frames. [#41](https://github.com/fudan-generative-vision/hallo/issues/41) - [ ] Bug: Sound volume affecting inference results (audio normalization). - [ ] ~~Enhancement: Inference code logic optimization~~. This solution doesn't show significant performance improvements. Trying other approaches. - [ ] Enhancement: Enhancing performance on low resolutions(256x256) to support more efficient usage.

## 📝 Citation If you find our work useful for your research, please consider citing the paper: ``` @misc{xu2024hallo, title={Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animation}, author={Mingwang Xu and Hui Li and Qingkun Su and Hanlin Shang and Liwei Zhang and Ce Liu and Jingdong Wang and Yao Yao and Siyu zhu}, year={2024}, eprint={2406.08801}, archivePrefix={arXiv}, primaryClass={cs.CV} } ``` ## 🌟 Opportunities Available Multiple research positions are open at the **Generative Vision Lab, Fudan University**! Include: - Research assistant - Postdoctoral researcher - PhD candidate - Master students Interested individuals are encouraged to contact us at [siyuzhu@fudan.edu.cn](mailto://siyuzhu@fudan.edu.cn) for further information. ## ⚠️ Social Risks and Mitigations The development of portrait image animation technologies driven by audio inputs poses social risks, such as the ethical implications of creating realistic portraits that could be misused for deepfakes. To mitigate these risks, it is crucial to establish ethical guidelines and responsible use practices. Privacy and consent concerns also arise from using individuals' images and voices. Addressing these involves transparent data usage policies, informed consent, and safeguarding privacy rights. By addressing these risks and implementing mitigations, the research aims to ensure the responsible and ethical development of this technology. ## 🤗 Acknowledgements We would like to thank the contributors to the [magic-animate](https://github.com/magic-research/magic-animate), [AnimateDiff](https://github.com/guoyww/AnimateDiff), [ultimatevocalremovergui](https://github.com/Anjok07/ultimatevocalremovergui), [AniPortrait](https://github.com/Zejun-Yang/AniPortrait) and [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone) repositories, for their open research and exploration. If we missed any open-source projects or related articles, we would like to complement the acknowledgement of this specific work immediately. ## 👏 Community Contributors Thank you to all the contributors who have helped to make this project better!