SVFR: A Unified Framework for Generalized Video Face Restoration

🔥 Overview

SVFR is a unified framework for face video restoration that supports tasks such as BFR, Colorization, Inpainting, and their combinations within one cohesive system.

🎬 Demo

BFR

Case1	Case2

BFR+Colorization

Case3	Case4

BFR+Colorization+Inpainting

Case5	Case6

🎙️ News

[2025.01.02]: We released the initial version of the inference code and models. Stay tuned for continuous updates!
[2024.12.17]: This repo is created!

🚀 Getting Started

Note: It is recommended to use a GPU with 16GB or more VRAM.

Setup

Use the following command to install a conda environment for SVFR from scratch:

conda create -n svfr python=3.9 -y
conda activate svfr

Install PyTorch: make sure to select the appropriate CUDA version based on your hardware, for example,

pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2

Install Dependencies:

pip install -r requirements.txt

Download checkpoints

Download the Stable Video Diffusion

conda install git-lfs
git lfs install
git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt models/stable-video-diffusion-img2vid-xt

Download SVFR

You can download checkpoints manually through link on Google Drive.

Put checkpoints as follows:

└── models
    ├── face_align
    │   ├── yoloface_v5m.pt
    ├── face_restoration
    │   ├── unet.pth
    │   ├── id_linear.pth
    │   ├── insightface_glint360k.pth
    └── stable-video-diffusion-img2vid-xt
        ├── vae
        ├── scheduler
        └── ...

Inference

Inference single or multi task

python3 infer.py \
 --config config/infer.yaml \
 --task_ids 0 \
 --input_path ./assert/lq/lq1.mp4 \
 --output_dir ./results/

task_id:

0 -- bfr
1 -- colorization
2 -- inpainting
0,1 -- bfr and colorization
0,1,2 -- bfr and colorization and inpainting
...

Inference with additional inpainting mask

# For Inference with Inpainting
# Add '--mask_path' if you need to specify the mask file.

python3 infer.py \
 --config config/infer.yaml \
 --task_ids 0,1,2 \
 --input_path ./assert/lq/lq3.mp4 \
 --output_dir ./results/ 
 --mask_path ./assert/mask/lq3.png

License

The code of SVFR is released under the MIT License. There is no limitation for both academic and commercial usage.

The pretrained models we provided with this library are available for non-commercial research purposes only, including both auto-downloading models and manual-downloading models.

Acknowledgments

This work is built on the architecture of Sonic.

BibTex

@misc{wang2025svfrunifiedframeworkgeneralized,
      title={SVFR: A Unified Framework for Generalized Video Face Restoration}, 
      author={Zhiyao Wang and Xu Chen and Chengming Xu and Junwei Zhu and Xiaobin Hu and Jiangning Zhang and Chengjie Wang and Yuqi Liu and Yiyi Zhou and Rongrong Ji},
      year={2025},
      eprint={2501.01235},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.01235}, 
}