Parlor On-Device AI Skill by ara.so — Daily 2026 Skills collection. Parlor is a real-time, on-device multimodal AI assistant. It combines Gemma 4 E2B (via LiteRT-LM) for speech and vision understanding with Kokoro TTS for voice output. Everything runs locally — no API keys, no cloud calls, no cost per request. Architecture Browser (mic + camera) │ │ WebSocket (audio PCM + JPEG frames) ▼ FastAPI server ├── Gemma 4 E2B via LiteRT-LM (GPU) → understands speech + vision └── Kokoro TTS (MLX on Mac, ONNX on Linux) → speaks back │ │ WebSocket (streamed audio chunks) ▼ Browser (playback + transcript) Key features: Silero VAD in browser — hands-free, no push-to-talk Barge-in — interrupt AI mid-sentence by speaking Sentence-level TTS streaming — audio starts before full response is ready Platform-aware TTS — MLX backend on Apple Silicon, ONNX on Linux Requirements Python 3.12+ macOS with Apple Silicon or Linux with a supported GPU ~3 GB free RAM uv package manager Installation git clone https://github.com/fikrikarim/parlor.git cd parlor

Install uv if needed

curl -LsSf https://astral.sh/uv/install.sh | sh cd src uv sync uv run server.py Open http://localhost:8000 , grant camera and microphone permissions, and start talking. Models download automatically on first run (~2.6 GB for Gemma 4 E2B, plus TTS models). Configuration Set environment variables before running:

Use a pre-downloaded model instead of auto-downloading

export MODEL_PATH = /path/to/gemma-4-E2B-it.litertlm

Change server port (default: 8000)

export PORT = 9000 uv run server.py Variable Default Description MODEL_PATH auto-download from HuggingFace Path to local .litertlm model file PORT 8000 Server port Project Structure src/ ├── server.py # FastAPI WebSocket server + Gemma 4 inference ├── tts.py # Platform-aware TTS (MLX on Mac, ONNX on Linux) ├── index.html # Frontend UI (VAD, camera, audio playback) ├── pyproject.toml # Dependencies └── benchmarks/ ├── bench.py # End-to-end WebSocket benchmark └── benchmark_tts.py # TTS backend comparison Key Components server.py — FastAPI WebSocket Server The server handles two WebSocket connections: one for receiving audio/video from the browser, one for streaming audio back.

Simplified pattern from server.py

from fastapi import FastAPI , WebSocket import asyncio app = FastAPI ( ) @app . websocket ( "/ws" ) async def websocket_endpoint ( websocket : WebSocket ) : await websocket . accept ( ) async for data in websocket . iter_bytes ( ) :

data contains PCM audio + optional JPEG frame

response_text

await run_gemma_inference ( data ) audio_chunks = await run_tts ( response_text ) for chunk in audio_chunks : await websocket . send_bytes ( chunk ) tts.py — Platform-Aware TTS Kokoro TTS selects backend based on platform:

tts.py uses platform detection

import platform def get_tts_backend ( ) : if platform . system ( ) == "Darwin" :

Apple Silicon: use MLX backend for GPU acceleration

from kokoro_mlx import KokoroMLX return KokoroMLX ( ) else :

Linux: use ONNX backend

from kokoro import KokoroPipeline return KokoroPipeline ( lang_code = 'a' ) tts = get_tts_backend ( )

Sentence-level streaming — yields audio as each sentence is ready

async def synthesize_streaming ( text : str ) : for sentence in split_sentences ( text ) : audio = tts . synthesize ( sentence ) yield audio Gemma 4 E2B Inference via LiteRT-LM

LiteRT-LM inference pattern

from litert_lm import LiteRTLM import os model_path = os . environ . get ( "MODEL_PATH" , None )

Auto-downloads if MODEL_PATH not set

model

LiteRTLM . from_pretrained ( "google/gemma-4-E2B-it" , local_path = model_path ) async def run_gemma_inference ( audio_pcm : bytes , image_jpeg : bytes = None ) : inputs = { "audio" : audio_pcm } if image_jpeg : inputs [ "image" ] = image_jpeg response = "" async for token in model . generate_stream ( ** inputs ) : response += token return response Running Benchmarks cd src

End-to-end WebSocket latency benchmark

uv run benchmarks/bench.py

Compare TTS backends (MLX vs ONNX)

uv run benchmarks/benchmark_tts.py Performance Reference (Apple M3 Pro) Stage Time Speech + vision understanding ~1.8–2.2s Response generation (~25 tokens) ~0.3s Text-to-speech (1–3 sentences) ~0.3–0.7s Total end-to-end ~2.5–3.0s Decode speed: ~83 tokens/sec on GPU. Common Patterns Extending the System Prompt Modify the prompt in server.py to change the AI's persona or task: SYSTEM_PROMPT = """You are a helpful language tutor. Respond conversationally in 1-3 sentences. If the user makes a grammar mistake, gently correct them. You can see through the user's camera and discuss what you observe.""" Adding a New Language for TTS Kokoro supports multiple language codes. Set lang_code in tts.py :

Language codes: 'a' = American English, 'b' = British English

'e' = Spanish, 'f' = French, 'z' = Chinese, 'j' = Japanese

pipeline

KokoroPipeline ( lang_code = 'e' )

Spanish

Customizing VAD Sensitivity (index.html) The Silero VAD threshold can be tuned in the frontend: // In index.html — lower positiveSpeechThreshold = more sensitive const vad = await MicVAD . new ( { positiveSpeechThreshold : 0.6 , // default ~0.8, lower = triggers more easily negativeSpeechThreshold : 0.35 , // how quickly it stops detecting speech minSpeechFrames : 3 , onSpeechStart : ( ) => { / UI feedback / } , onSpeechEnd : ( audio ) => sendAudioToServer ( audio ) , } ) ; Sending Frames Programmatically (WebSocket Client Example) import asyncio import websockets import json import base64 async def send_audio_frame ( audio_pcm_bytes : bytes , jpeg_bytes : bytes = None ) : uri = "ws://localhost:8000/ws" async with websockets . connect ( uri ) as ws : payload = { "audio" : base64 . b64encode ( audio_pcm_bytes ) . decode ( ) , } if jpeg_bytes : payload [ "image" ] = base64 . b64encode ( jpeg_bytes ) . decode ( ) await ws . send ( json . dumps ( payload ) )

Receive streamed audio response

async for message in ws : audio_chunk = message

raw PCM bytes

play or save audio_chunk

Troubleshooting Model download fails

Pre-download manually via huggingface_hub

uv run python -c " from huggingface_hub import hf_hub_download path = hf_hub_download('google/gemma-4-E2B-it', 'gemma-4-E2B-it.litertlm') print(path) " export MODEL_PATH = /path/shown/above uv run server.py Microphone/camera not working in browser Must access via http://localhost (not IP address) — browsers block media APIs on non-localhost HTTP Check browser permissions: address bar → lock icon → reset permissions TTS not loading on Linux

Ensure ONNX runtime is installed

uv add onnxruntime

Or for GPU:

uv add onnxruntime-gpu High latency or slow inference Verify GPU is being used: check for Metal (Mac) or CUDA (Linux) in startup logs Close other GPU-heavy applications On Linux, confirm CUDA drivers match installed onnxruntime-gpu version Port already in use export PORT = 8080 uv run server.py

Or kill the existing process:

lsof -ti:8000 | xargs kill uv sync fails — Python version mismatch

Parlor requires Python 3.12+

python3 --version

Install 3.12 via pyenv or system package manager, then:

uv python pin 3.12 uv sync Dependencies (pyproject.toml) Key packages installed by uv sync : litert-lm — Google AI Edge inference runtime for Gemma fastapi + uvicorn — async web/WebSocket server kokoro — Kokoro TTS ONNX backend kokoro-mlx — Kokoro TTS MLX backend (Mac only) silero-vad — voice activity detection (browser-side via CDN) huggingface-hub — model auto-download