LEWIN AI AGENT

A talking avatar you can run on a laptop. Type or speak, Grok answers out loud, and the mouth follows the voice in the browser — no GPU lipsync model, no cloud viseme service.

The lipsync stack is intentionally small: Grok TTS already returns character timestamps, those characters map to a 15-shape viseme set, and the browser’s Web Audio analyser only supplies jaw energy. That is enough for 2D Rive mascots and 3D GLB heads.

You  ──text or mic──►  Grok chat
                           │
                           ▼
                      Grok TTS (with_timestamps)
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
           MP3 audio    visemes     amplitude
              │            │            │
              └────────────┼────────────┘
                           ▼
                    LEWIN avatar
                 (Rive or Three.js)

Why the lipsync stays light

Most talking-head demos ship a neural lipsync network, a forced aligner, or a paid viseme API. LEWIN does not need those on the hot path.

Timestamps, not a model. Grok TTS with_timestamps already says when each character is spoken. server/visemes.mjs maps English, Russian, and Chinese (via pinyin) onto the Oculus / wawa viseme set: PP FF TH DD kk CH SS nn RR aa E I O U sil.
One analyser node. wawa-lipsync is a few hundred lines of Web Audio FFT. It is used for amplitude (how open the jaw is), not for guessing phonemes.
Hybrid player. HybridPlayer clocks visemes against audio.currentTime and reads volume on requestAnimationFrame. No WASM lipsync runtime, no extra network hop.
Tiny mouth alphabet. 2D Rive files expose one input per viseme; 3D GLBs use morph targets of the same names. Switching avatar does not change the pipeline.
Optional extras stay optional. Montreal Forced Aligner can refine Mandarin phones. Local MLX speech-to-text can drive the mic. Neither is required to chat and lipsync.

Features

Grok LLM replies, spoken with Grok TTS
English, Russian, and Chinese viseme mapping, including spoken forms of digit runs
2D Rive mascots and 3D Ready Player Me / VRoid / AvatarSDK heads
Voice input with barge-in (interrupt the avatar by talking)
Background presets and a compact chat panel

Quick start

You need Node 20+, Yarn 1, and an xAI API key.

git clone https://github.com/robofighter/lewin-ai-agent.git
cd lewin-ai-agent
cp .env.example .env

Put your key in .env:

XAI_API_KEY=xai-...

Then:

yarn
yarn dev

Open http://localhost:5173. That starts the API on 127.0.0.1:8000 and the Vite UI together.

Command	What it does
`yarn dev`	API + UI
`yarn server`	API only
`yarn demo`	UI only (expects the API)
`yarn test`	viseme / STT / VAD unit tests

Typing a message is enough. Live microphone needs the optional STT setup below.

Optional: voice input (Apple Silicon)

Local STT uses mlx-audio and mlx-community/nemotron-3.5-asr-streaming-0.6b-8bit. Skip this on non-Apple machines — chat and lipsync still work.

python3 -m venv server/.venv-stt
server/.venv-stt/bin/pip install -r server/stt-requirements.txt

Uncomment in .env:

STT_PYTHON=server/.venv-stt/bin/python

Restart yarn dev, then press Live. ffmpeg on your PATH helps convert the mic clip to WAV.

Optional: Mandarin MFA

If Montreal Forced Aligner is installed, Chinese replies can swap character visemes for phone-level timing. Leave MFA_ENABLED=0 unless you have MFA; the pinyin fallback is the default.

Environment

Variable	Default	Purpose
`XAI_API_KEY`	—	xAI API key (required)
`GROK_MODEL`	`grok-4.20-0309-non-reasoning`	Chat model
`GROK_VOICE`	`eve`	TTS voice
`GROK_TTS_LANGUAGE`	`auto`	TTS language (`ru`, `en`, `zh`, or `auto`)
`PORT`	`8000`	API port
`MFA_ENABLED`	`0`	Mandarin MFA aligner
`STT_PYTHON`	`python3`	Interpreter for the optional STT worker

Repository layout

server/                 Grok chat, TTS, visemes, optional STT
packages/wawa-lipsync   Browser FFT lipsync (amplitude)
examples/lipsync-demo   React UI, Rive + Three.js avatars

Credits

Maintained by @robofighter.

LEWIN builds on wawa-lipsync by Wassim Samad (MIT), via NoelClay/wawa-lipsync. The viseme alphabet is the Oculus LipSync set. Chat and speech come from xAI Grok.

lewin-ai-agent

Links

README