STIDNet: Identity-Aware Face Forgery Detection with Spatiotemporal Knowledge Distillation

Fang, Mingqi; Yu, Lingyun; Xie, Hongtao; Tan, Qingfeng; Tan, Zhiyuan; Hussain, Amir; Wang, Zezheng; Li, Jiahong; Tian, Zhihong

Abstract

The impressive development of facial manipulation techniques has raised severe public concerns. Identity-aware methods, especially suitable for protecting celebrities, are seen as one of promising face forgery detection approaches with additional reference video. However, without in-depth observation of fake video’s characteristics, most existing identity-aware algorithms are just naive imitation of face verification model and fail to exploit discriminative information. In this article, we argue that it is necessary to take both spatial and temporal perspectives into consideration for adequate inconsistency clues and propose a novel forgery detector named SpatioTemporal IDentity network (STIDNet). To effectively capture heterogeneous spatiotemporal information in a unified formulation, our STIDNet is following a knowledge distillation architecture that the student identity extractor receives supervision from a spatial information encoder (SIE) and a temporal information encoder (TIE) through multiteacher training. Specifically, a regional sensitive identity modelling paradigm is proposed in SIE by introducing facial blending augmentation but with uniform identity label, thus encourage model to focus on spatial discriminative region like outer face. Meanwhile, considering the strong temporal correlation between audio and talking face video, our TIE is devised in a cross-modal pattern that the audio information is introduced to supervise model exploiting temporal personalized movements. Benefit from knowledge transfer from SIE and TIE, STIDNet is able to capture individual’s essential spatiotemporal identity attributes and sensitive to even subtle identity deviation caused by manipulation. Extensive experiments indicate the superiority of our STIDNet compared with previous works. Moreover, we also demonstrate STIDNet is more suitable for real-world implementation in terms of model complexity and reference set size.

• Journal Article Type: Article
• Acceptance Date: Jan 12, 2024
• Online Publication Date: Feb 12, 2024
• Deposit Date: Feb 16, 2024
• Publicly Available Date: Feb 19, 2024
• Journal: IEEE Transactions on Computational Social Systems
• Electronic ISSN: 2329-924X
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• DOI: https://doi.org/10.1109/tcss.2024.3356549
• Keywords: Face Forgery Detection, Knowledge Distillation, Video Forensics, Deep Learning
• Public URL: http://researchrepository.napier.ac.uk/Output/3510279