Concept
This project began as an attempt to create a completely offline, privacy-focused AI voice assistant that could interact naturally, both vocally and visually.
It merges speech recognition, natural language understanding, text-to-speech synthesis, and avatar video animation — all running locally on consumer hardware.
My core motivation was to remove cloud dependencies from the AI interaction pipeline. Every response, every facial animation, every word spoken is processed on-device.
Design and Architecture
The project is divided into multiple modular subsystems:
- Whisper ASR for speech-to-text transcription
- Qwen Chat Model for local language understanding
- Edge TTS / Coqui TTS for speech synthesis
- SadTalker / Wav2Lip for avatar lip-sync video generation
- Gradio Web Interface for user interaction
All these modules communicate through asynchronous tasks, managed within a unified Python environment.
Each component is optimized for efficiency — quantized models, GPU acceleration, and parallel pipelines minimize latency without sacrificing realism.
Interface and Experience
The web interface, built with Gradio, serves as the interaction layer between the user and the AI.
Its layout is designed for focus and flow — from microphone activation to avatar playback, all operations are visible in one unified workspace.
Visual Design
- Minimal color palette for high contrast
- Monospaced and modern sans-serif typography
- Responsive grid layout with modular panels
- Animated transitions during avatar video rendering
Technical Deep Dive
Audio Processing
Audio is captured and chunked in real-time using sounddevice, then transcribed via Whisper.
Processing runs in a dedicated thread to avoid blocking the response loop.
Language Model
The conversation engine is powered by Qwen 1.5B Chat, providing context-aware responses without relying on remote APIs.
Voice Generation
The system integrates Edge TTS for fast speech synthesis and Coqui TTS for more expressive, locally generated voices.
Avatar Rendering
Two backends are supported:
- SadTalker — photorealistic animation and head motion.
- Wav2Lip — precise lip synchronization for minimal motion setups.
Both are automatically configured at runtime, with fallback handling for missing checkpoints.
Development Log
- April 2025: Initial prototype integrating Whisper + Qwen
- May 2025: Added TTS pipeline and conversation memory
- June 2025: Integrated SadTalker with optimized FFmpeg handling
- August 2025: Implemented real-time audio capture and thread-safe buffering
- October 2025: UI refinements and modular configuration system completed
Performance and Optimization
Testing was done on a Lenovo LOQ laptop (Intel i5-13450HX, RTX 3050 6GB VRAM, 16GB RAM).
Results:
- ASR latency: ~1.8s average
- Response generation: <2s with 1.5B model
- TTS synthesis: ~1.3s average
- Avatar rendering: 4–6s (depending on backend)
Current Progress
- Local conversational engine
- Whisper ASR integration
- Dual TTS backends (Edge + Coqui)
- SadTalker and Wav2Lip support
- Web interface with Gradio
- Avatar customization system
- Memory persistence
- Mobile-friendly UI
Future Goals
- Expand to multi-character avatars
- Integrate facial emotion control
- Add offline translation pipeline
- Create system tray background service
- Support for personal voice cloning
Acknowledgements
This project stands on the work of open communities and frameworks such as:
Snapshot Gallery
License
Released under the MIT License.
Author: KC
Website: kc.codes
Email: contact@kc.codes
About this Post
This post is written by Kaustubh Chauhan, licensed under CC BY-NC 4.0.