An audio-visual imposture scenario by talking face animation

Abstract

We describe a system that allows an impostor to lead an audio-visual telephone conversation, and sign data electronically on behalf of an authorized client. During the conversation, audio and video of the impostor are altered so as to mimic the client. The voice of an impostor is processed and used to reproduce the voice of the authorized client. Speech segments obtained from clients recordings are used to synthesize new sentences that the client never pronounced. On the visual side, the imposters talking face is detected and facial features are extracted and used to animate a synthetic talking face. The texture of the impersonated face is mapped onto the talking head and coded for transmission over the phone, along with the synthesized voice. Audio-visual coding and synthesis is realized by indexing in a memory containing audio-visual sequences. Stochastic models (coupled HMM) of characteristic segments are used to drive the memory search.