AIGC Algorithm Intern
Hithink RoyalFlush Cloud Software Co., Ltd.
Challenge
Reduce the long inference time and high GPU memory consumption of diffusion-based lip-sync models.
Approach
Built an end-to-end, two-stage GAN framework for frontal-view lip synchronization, facial reconstruction, and targeted hard-case fine-tuning.
Data & robustness
Trained on large video-generation model outputs and real-world datasets, with classifiers identifying profile views, infants, and occlusions.
Result
Designed uniform sampling for extreme angles, improving profile-view quality without compromising the model's original performance.
