Spaces:
Running
on
L40S
Running
on
L40S
| <!-- # SVFR: A Unified Framework for Generalized Video Face Restoration --> | |
| <div> | |
| <h1>SVFR: A Unified Framework for Generalized Video Face Restoration</h1> | |
| </div> | |
| [](https://arxiv.org/pdf/2501.01235) | |
| [](https://wangzhiyaoo.github.io/SVFR/) | |
| ## 🔥 Overview | |
| SVFR is a unified framework for face video restoration that supports tasks such as **BFR, Colorization, Inpainting**, and **their combinations** within one cohesive system. | |
| <img src="assert/method.png"> | |
| ## 🎬 Demo | |
| ### BFR | |
| <!-- | |
| <div style="display: flex; gap: 10px;"> | |
| <video controls width="360"> | |
| <source src="https://wangzhiyaoo.github.io/SVFR/static/videos/wild-test/case1_bfr.mp4" type="video/mp4"> | |
| </video> | |
| <video controls width="360"> | |
| <source src="https://wangzhiyaoo.github.io/SVFR/static/videos/wild-test/case4_bfr.mp4" type="video/mp4"> | |
| </video> | |
| </div> --> | |
| <!-- <div style="display: flex; gap: 10px;"> | |
| <video src="https://github.com/user-attachments/assets/49f985f3-a2db-4b9f-aed0-e9943bae9c17" controls width=45%></video> | |
| <video src="https://github.com/user-attachments/assets/8fcd1dd9-79d3-4e57-b98e-a80ae2badfb5" controls width="45%"></video> | |
| </div> --> | |
| | Case1 | Case2 | | |
| |--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | |
| |<video src="https://github.com/user-attachments/assets/49f985f3-a2db-4b9f-aed0-e9943bae9c17" /> | <video src="https://github.com/user-attachments/assets/8fcd1dd9-79d3-4e57-b98e-a80ae2badfb5" /> | | |
| <!-- <video src="https://wangzhiyaoo.github.io/SVFR/bfr"> --> | |
| <!-- https://github.com/user-attachments/assets/49f985f3-a2db-4b9f-aed0-e9943bae9c17 | |
| https://github.com/user-attachments/assets/8fcd1dd9-79d3-4e57-b98e-a80ae2badfb5 --> | |
| ### BFR+Colorization | |
| <!-- <div style="display: flex; gap: 10px;"> | |
| <video controls width="360"> | |
| <source src="https://wangzhiyaoo.github.io/SVFR/static/videos/wild-test/case10_bfr_colorization.mp4" type="video/mp4"> | |
| </video> | |
| <video controls width="360"> | |
| <source src="https://wangzhiyaoo.github.io/SVFR/static/videos/wild-test/case12_bfr_colorization.mp4" type="video/mp4"> | |
| </video> | |
| </div> --> | |
| <!-- https://github.com/user-attachments/assets/795f4cb1-a7c9-41c5-9486-26e64a96bcf0 | |
| https://github.com/user-attachments/assets/6ccf2267-30be-4553-9ecc-f3e7e0ca1d6f --> | |
| | Case3 | Case4 | | |
| |--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | |
| |<video src="https://github.com/user-attachments/assets/795f4cb1-a7c9-41c5-9486-26e64a96bcf0" /> | <video src="https://github.com/user-attachments/assets/6ccf2267-30be-4553-9ecc-f3e7e0ca1d6f" /> | | |
| ### BFR+Colorization+Inpainting | |
| <!-- <div style="display: flex; gap: 10px;"> | |
| <video controls width="360"> | |
| <source src="https://wangzhiyaoo.github.io/SVFR/static/videos/wild-test/case14_bfr+colorization+inpainting.mp4" type="video/mp4"> | |
| </video> | |
| <video controls width="360"> | |
| <source src="https://wangzhiyaoo.github.io/SVFR/static/videos/wild-test/case15_bfr+colorization+inpainting.mp4" type="video/mp4"> | |
| </video> | |
| </div> --> | |
| <!-- https://github.com/user-attachments/assets/6113819f-142b-4faa-b1c3-a2b669fd0786 | |
| https://github.com/user-attachments/assets/efdac23c-0ba5-4dad-ab8c-48904af5dd89 | |
| --> | |
| | Case5 | Case6 | | |
| |--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | |
| |<video src="https://github.com/user-attachments/assets/6113819f-142b-4faa-b1c3-a2b669fd0786" /> | <video src="https://github.com/user-attachments/assets/efdac23c-0ba5-4dad-ab8c-48904af5dd89" /> | | |
| ## 🎙️ News | |
| - **[2025.01.02]**: We released the initial version of the [inference code](#inference) and [models](#download-checkpoints). Stay tuned for continuous updates! | |
| - **[2024.12.17]**: This repo is created! | |
| ## 🚀 Getting Started | |
| > Note: It is recommended to use a GPU with 16GB or more VRAM. | |
| ## Setup | |
| Use the following command to install a conda environment for SVFR from scratch: | |
| ```bash | |
| conda create -n svfr python=3.9 -y | |
| conda activate svfr | |
| ``` | |
| Install PyTorch: make sure to select the appropriate CUDA version based on your hardware, for example, | |
| ```bash | |
| pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 | |
| ``` | |
| Install Dependencies: | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ## Download checkpoints | |
| <li>Download the Stable Video Diffusion</li> | |
| ``` | |
| conda install git-lfs | |
| git lfs install | |
| git clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt models/stable-video-diffusion-img2vid-xt | |
| ``` | |
| <li>Download SVFR</li> | |
| You can download checkpoints manually through link on [Google Drive](https://drive.google.com/drive/folders/1nzy9Vk-yA_DwXm1Pm4dyE2o0r7V6_5mn?usp=share_link). | |
| Put checkpoints as follows: | |
| ``` | |
| └── models | |
| ├── face_align | |
| │ ├── yoloface_v5m.pt | |
| ├── face_restoration | |
| │ ├── unet.pth | |
| │ ├── id_linear.pth | |
| │ ├── insightface_glint360k.pth | |
| └── stable-video-diffusion-img2vid-xt | |
| ├── vae | |
| ├── scheduler | |
| └── ... | |
| ``` | |
| ## Inference | |
| ### Inference single or multi task | |
| ``` | |
| python3 infer.py \ | |
| --config config/infer.yaml \ | |
| --task_ids 0 \ | |
| --input_path ./assert/lq/lq1.mp4 \ | |
| --output_dir ./results/ | |
| ``` | |
| <li>task_id:</li> | |
| > 0 -- bfr | |
| > 1 -- colorization | |
| > 2 -- inpainting | |
| > 0,1 -- bfr and colorization | |
| > 0,1,2 -- bfr and colorization and inpainting | |
| > ... | |
| ### Inference with additional inpainting mask | |
| ``` | |
| # For Inference with Inpainting | |
| # Add '--mask_path' if you need to specify the mask file. | |
| python3 infer.py \ | |
| --config config/infer.yaml \ | |
| --task_ids 0,1,2 \ | |
| --input_path ./assert/lq/lq3.mp4 \ | |
| --output_dir ./results/ | |
| --mask_path ./assert/mask/lq3.png | |
| ``` | |
| ## License | |
| The code of SVFR is released under the MIT License. There is no limitation for both academic and commercial usage. | |
| **The pretrained models we provided with this library are available for non-commercial research purposes only, including both auto-downloading models and manual-downloading models.** | |
| ## Acknowledgments | |
| This work is built on the architecture of [Sonic](https://github.com/jixiaozhong/Sonic). | |
| ## BibTex | |
| ``` | |
| @misc{wang2025svfrunifiedframeworkgeneralized, | |
| title={SVFR: A Unified Framework for Generalized Video Face Restoration}, | |
| author={Zhiyao Wang and Xu Chen and Chengming Xu and Junwei Zhu and Xiaobin Hu and Jiangning Zhang and Chengjie Wang and Yuqi Liu and Yiyi Zhou and Rongrong Ji}, | |
| year={2025}, | |
| eprint={2501.01235}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CV}, | |
| url={https://arxiv.org/abs/2501.01235}, | |
| } | |
| ``` | |