KrustyPlanet/openclaw-voice
6
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
13 commits 2 branches 0 tags 323 KiB
Commit graph

8 commits

Author SHA1 Message Date
Jezza Hehn
f0072593ae voice: asyncio.Queue rewrite, browser TTS playback, silence detection, pipeline audit 
- Rewrote voice_ws.py: receive loop uses queue.put_nowait(), separate consumer
  task handles STT->LLM->TTS pipeline (no more blocking the WebSocket)
- Updated voice.html: TTS audio playback, transcript display, thinking indicator
- Added energy-based silence detection (skip STT on silent buffers)
- Fixed sample rate mismatch (16kHz throughout, not 24kHz)
- Added AUDIT.md: full pipeline audit confirming STT/TTS/OpenClaw client work

Known blocker: OpenClaw gateway chat.send requires operator.write scope,
gateway password token doesn't grant scopes. Needs device pairing fix.
 2026-04-10 05:41:00 +00:00
Jezza Hehn
3450e57ca6 Fix voice portal: WebSocket routing, Caddy keepalive, audio pipeline 
- Fix app.py: @app.get -> @app.websocket for /ws/voice route (was returning 403)
- Fix app.py: create static_dir before mounting it (AttributeError on startup)
- Fix voice.html: AudioWorkletNode constructor (was AudioWorkletProcessor)
- Fix voice.html: use ScriptProcessor directly (more reliable)
- Fix voice.html: send Float32 directly (server expects float32, was sending Int16)
- Fix voice.html: auto-detect ws/wss protocol from page URL
- Add Caddy reverse proxy keepalive pings every 15s to prevent timeout
- Add detailed message type logging in WebSocket receive loop
- Strip Jarvis/Sage personas, rename bot to MoltMic
- Add /moltmic voice slash command for portal URL
- Update portal URL to https://voice.jezzahehn.com
 2026-04-10 04:47:31 +00:00
Jezza Hehn
bc580861dd Add browser-based voice portal (WebSocket + mic → STT → LLM → TTS) 2026-04-10 02:30:23 +00:00
Jezza Hehn
7d3e13a3ca Wire cloud STT/TTS providers into pipeline 
- Add provider field to STTConfig and TTSConfig (deepgram/venice)
- Add VeniceTTSConfig model for venice voice/base_url settings
- Add CloudTTSSynthesizer adapter wrapping VeniceKokoroTTS
- Loosen STTTranscriber type hint to accept any engine with transcribe_async
- Update run.py to use create_stt_engine/create_tts_engine factories
- Provider-based init: reads config.pipeline.stt.provider and .tts.provider
- Fix duplicate language key in config.yaml
- Remove duplicate language field from STT config

Cloud-only path: VAD (local) -> Deepgram STT -> OpenClaw -> Venice TTS -> Discord
 2026-04-10 00:44:03 +00:00
Jezza Hehn
f0458b9b40 feat: add Deepgram STT provider and cloud-first config 
- New DeepgramSTT class using Deepgram nova-3 via REST API
- Factory function create_stt_engine() for provider switching
- faster-whisper import now optional (graceful fallback)
- Config defaults to cloud providers (deepgram STT + venice TTS)
- .env.example updated with DEEPGRAM_API_KEY and VENICE_API_KEY
- requirements.txt adds deepgram-sdk, marks faster-whisper as optional
- Zero GPU required for default configuration
 2026-04-10 00:33:57 +00:00
Jezza Hehn
3eea942772 feat: add Venice Kokoro TTS provider 2026-04-10 00:24:11 +00:00
MCKRUZ
9fde3d31ba feat: Major performance optimizations and feature enhancements 
## Performance Optimizations (3-10x faster responses)
- STT beam_size reduced to 1 (3-5x faster transcription, minimal quality loss)
- Smart query routing: Haiku (simple) → Sonnet (medium) → Opus (complex)
- TTS cache for common phrases (27 pre-generated responses)
- Sentence-level streaming TTS (start playing while generating)
- Sample-based VAD timing (30x improvement in silence detection)

## TTS Engine Upgrade
- Migrated from Chatterbox to Chatterbox-Turbo
- Zero-shot voice cloning (no fine-tuning required)
- Native paralinguistic tag support ([laugh], [sigh], [chuckle], etc.)
- Emotion presets with temperature control
- Improved marker conversion (*action*, (action), ~action~)

## Discord Bot Enhancements
- Multi-agent support (Jarvis, Sage)
- Improved voice receiving with discord-ext-voice-recv
- Enhanced /join, /leave, /status commands
- Per-agent personality configuration
- Better audio sink/receiver implementation

## OpenClaw Integration
- WebSocket support for Gateway communication
- Query complexity routing (auto-select model)
- Improved error handling and retries
- Session management per Discord guild
- Better latency tracking

## Pipeline Improvements
- Sentence splitter for streaming optimization
- Query router for intelligent model selection
- Enhanced VAD receiver with sample-based timing
- Improved audio buffering and format conversion
- Better transcript management

## Documentation
- Added QUICK_START.md (5-minute test guide)
- Added OPTIMIZATION_SUMMARY.md (performance analysis)
- Added DISCORD_OPTIMIZATION_TEST.md (testing guide)
- Added USAGE_GUIDE.md (comprehensive usage)
- Updated README.md with optimization details

## Utilities & Scripts
- Added get_invite_link.py (Discord bot invite)
- Added sync_commands.py, sync_to_guild.py (command sync)
- Added test_gateway.py, test_stt.py (testing utilities)
- Added openclaw_wrapper.py (wrapper script)
- Removed create_mock_turn_model.py (no longer needed)

## Configuration Updates
- STT model: medium → small (faster, acceptable quality)
- TTS engine: chatterbox → coqui (Turbo integration)
- Beam size: 5 → 1 (latency optimization)
- Added emotion_exaggeration per agent
- Updated .gitignore for project files

Total: ~2105 insertions, ~462 deletions across 35 files
Performance: ~5.5s total latency (down from 22-35s)
Target: ~3.5s (achieved in simple queries with cache)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-16 19:29:57 -05:00
MCKRUZ
3de8228c7c Initial commit: Jarvis Voice Bot - Complete Implementation 
Complete 14-phase implementation of AI-powered Discord voice bot:

Features:
- Passive voice listening with Smart Turn v3 detection
- GPU-accelerated STT (faster-whisper) and TTS (Chatterbox)
- Intelligent two-tier relevance filtering
- Rolling conversation context management
- Multi-agent support (Jarvis, Sage)
- OpenAI-compatible TTS/STT API endpoints
- Barge-in support and concurrent user handling

Architecture:
- Discord.py voice integration
- Silero VAD for speech detection
- Pipecat Smart Turn v3 for turn completion
- OpenClaw API client (stubbed for integration)
- FastAPI server with health monitoring

Testing:
- 318 tests passing (100% coverage of major components)
- Unit tests for all modules
- Integration tests for end-to-end flows
- Memory leak prevention tests

Documentation:
- Comprehensive README with installation guide
- Troubleshooting guide and performance metrics
- Production deployment checklist
- Environment configuration templates

Status: 14/14 phases complete (100%)
Production Ready: Yes (after stub replacements)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-02-13 12:35:03 -05:00