{"slug": "stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix", "title": "Stt On discord Vc For My ai vtuber (how do i fix?)", "summary": "A Discord bot developer is seeking help to fix speech-to-text functionality for their AI VTuber in a voice channel, encountering issues with audio processing and integration.", "body_md": "\n\n``` python\nimport time\nimport asyncio\nimport threading\nfrom socket import socket, AF_INET, SOCK_STREAM\nfrom json import dumps, loads\nfrom collections import deque\nfrom typing import Optional\nimport discord\nfrom discord.ext import voice_recv\nfrom discord.ext import commands\nimport numpy as np\nimport logging\nimport pyaudio\nlogging.getLogger(\"discord.ext.voice_recv.reader\").setLevel(logging.ERROR)\n\n# Local audio capture (optional)\nlocal_audio_module = None\ntry:\n    # Add current directory to Python path for local modules\n    import sys\n    import os\n    sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))\n    \n    import local_audio_capture\n    local_audio_module = local_audio_capture\n    print(\"[OK] Local audio capture module loaded\")\nexcept ImportError as e:\n    print(f\"[WARN] Local audio capture module not available: {e}\")\n    # Additional debugging\n    try:\n        \n        import pyaudio\n        print(\"[OK] PyAudio is available\")\n    except Exception as pyaudio_error:\n        print(f\"[ERROR] PyAudio error: {pyaudio_error}\")\n\nvoice_recv_available = True\nvoice_recv_imported = voice_recv\n\nconnecting = False\n# =============================\n# CONFIG\n# =============================\nCORE_HOST = os.getenv(\"CORE_HOST\", \"127.0.0.1\")\nCORE_PORT = int(os.getenv(\"CORE_PORT\", \"8765\"))\nDISCORD_TOKEN = os.getenv(\"DISCORD_TOKEN\")\nif not DISCORD_TOKEN:\n    raise RuntimeError(\"DISCORD_TOKEN environment variable not set\")\n\n# Optional: set MONITORED_USER_IDS=\"123,456\" to only react to specific users\nMONITORED_USER_IDS = {\n    int(x.strip())\n    for x in os.getenv(\"MONITORED_USER_IDS\", \"\").split(\",\")\n    if x.strip().isdigit()\n}\n\n# Local audio capture settings\nLOCAL_AUDIO_ENABLED = os.getenv(\"LOCAL_AUDIO_ENABLED\", \"false\").lower() == \"true\"  # Disabled by default to avoid conflicts\nLOCAL_AUDIO_DEVICE_INDEX = os.getenv(\"LOCAL_AUDIO_DEVICE_INDEX\")\nif LOCAL_AUDIO_DEVICE_INDEX is not None:\n    LOCAL_AUDIO_DEVICE_INDEX = int(LOCAL_AUDIO_DEVICE_INDEX)\nelse:\n    # Default to Yeti microphone if not specified\n    LOCAL_AUDIO_DEVICE_INDEX = 1\n\n# =============================\n# AUDIO / STATE\n# =============================\nlibrosa = None\n\ndef get_librosa():\n    global librosa\n    if librosa is None:\n        import librosa as _librosa\n        librosa = _librosa\n    return librosa\n\nvc_buffer = deque()\nbuffer_lock = threading.Lock()\n\nlast_audio_time = 10\nSILENCE_TIMEOUT = 1.5\ntts_playing_flag = False\ntts_cooldown_time = 10  # Time when TTS cooldown ends\nTTS_COOLDOWN_DURATION = 1.5  # Seconds to ignore audio after TTS finishes (echo protection)\n\ncurrent_voice_client: Optional[discord.VoiceClient] = None\ncurrent_sink = None\nbot_loop: Optional[asyncio.AbstractEventLoop] = None\nlocal_audio_pa = None  # PyAudio instance for local audio\nlocal_audio_stream = None  # Stream for local audio\nlocal_audio_thread = None  # Thread for local audio processing\n\nvoice_connect_lock = asyncio.Lock()\nvoice_action_lock = asyncio.Lock()\n\n# =============================\n# PATCH OPUS DECODER\n# =============================\nimport discord.opus\n\n_real_decode = discord.opus.Decoder.decode\n\ndef safe_decode(self, *args, **kwargs):\n    try:\n        return _real_decode(self, *args, **kwargs)\n    except discord.opus.OpusError:\n        try:\n            return _real_decode(self, None, fec=False)\n        except Exception:\n            return b\"\"\n\ndiscord.opus.Decoder.decode = safe_decode\n\n# =============================\n# HELPERS\n# =============================\ndef is_allowed_member(member: discord.Member) -> bool:\n    # Handle None member\n    if member is None:\n        return False\n    if member.bot:\n        return False\n    if MONITORED_USER_IDS:\n        return member.id in MONITORED_USER_IDS\n    return True\n\n_raw_pcm_counter = 0\n_debug_counter = 0\n_vc_debug_wav_dir = \"debug_audio\"\nos.makedirs(_vc_debug_wav_dir, exist_ok=True)\n\ndef save_debug_wav(audio, sample_rate: int, filename: str):\n    \"\"\"Save audio to debug_audio/ as a WAV file for STT debugging.\"\"\"\n    global _debug_counter\n    _debug_counter += 1\n    audio = np.asarray(audio, dtype=np.float32)\n    audio = np.clip(audio, -1.0, 1.0)\n    path = os.path.join(_vc_debug_wav_dir, filename)\n    try:\n        import soundfile as sf\n        sf.write(path, audio, sample_rate)\n    except Exception as e:\n        print(f\"[DEBUG] Save failed ({filename}): {e}\")\n\ndef discord_pcm_to_whisper(pcm_bytes: bytes) -> np.ndarray:\n    \"\"\"Convert Discord PCM (48kHz, 2ch, s16le, interleaved) → 16kHz mono float32.\"\"\"\n    global _raw_pcm_counter\n    _raw_pcm_counter += 1\n\n    raw = np.frombuffer(pcm_bytes, dtype=np.int16)\n    n = raw.size\n\n    # Warn about clipping in source audio (int16 at extremes)\n    if _raw_pcm_counter <= 5 or _raw_pcm_counter % 50 == 0:\n        clipped = np.sum((raw == 32767) | (raw == -32768))\n        if clipped > n * 0.05:\n            print(f\"[CLIP] #{_raw_pcm_counter}: {clipped}/{n} samples at int16 extremes — source audio too loud!\")\n\n    # Always log first 5, then periodically\n    if _raw_pcm_counter < 5 or _raw_pcm_counter % 50 == 0:\n        print(f\"[PCM] #{_raw_pcm_counter}: {len(pcm_bytes)}B, {n} int16, first 10: {raw[:10].tolist()}\")\n\n    # Stage 1 debug: save raw PCM (first 5 frames only — special format with subtype)\n    if _raw_pcm_counter <= 5:\n        try:\n            import soundfile as sf\n            sf.write(\n                f\"{_vc_debug_wav_dir}/raw{_raw_pcm_counter}_stereo.wav\",\n                raw.reshape(-1, 2),\n                48000,\n                subtype='PCM_16',\n            )\n        except Exception as e:\n            print(f\"[PCM] Save err: {e}\")\n\n    # --- Downmix stereo to mono ---\n    # Discord Opus decodes to 48kHz stereo s16le interleaved.\n    # Each Opus frame has N int16 values = N/2 stereo pairs.\n    if n >= 2 and n % 2 == 0:\n        mono_48k = raw.astype(np.float32).reshape(-1, 2).mean(axis=1)\n    else:\n        mono_48k = raw.astype(np.float32)\n\n    # Normalize int16 → float32 [-1, 1]\n    mono_48k /= 32768.0\n    mono_48k = np.clip(mono_48k, -1.0, 1.0)\n\n    # Stage 2 debug: after mono downmix at 48kHz\n    if _raw_pcm_counter <= 5:\n        save_debug_wav(mono_48k, 48000, f\"stage2_{_raw_pcm_counter}_48k_mono.wav\")\n\n    # --- Resample 48kHz → 16kHz with anti-aliasing ---\n    # Attenuate by 0.9 before resampling to give the anti-alias filter headroom\n    # and prevent clipping from filter overshoot on loud/transient audio.\n    try:\n        from scipy.signal import resample_poly\n        mono_16k = resample_poly(mono_48k * 0.9, up=1, down=3)\n    except ImportError:\n        librosa_local = get_librosa()\n        mono_16k = librosa_local.resample(mono_48k * 0.9, orig_sr=48000, target_sr=16000)\n\n    mono_16k = np.clip(mono_16k, -1.0, 1.0)\n\n    # Stage 3 debug: final 16kHz output (first 5 frames)\n    if _raw_pcm_counter <= 5:\n        save_debug_wav(mono_16k, 16000, f\"stage3_{_raw_pcm_counter}_16k_mono.wav\")\n\n    return mono_16k.astype(np.float32)\n\ndef send_audio_to_core(audio_np: np.ndarray) -> Optional[str]:\n    MAX_SECONDS = 8\n    max_samples = 16000 * MAX_SECONDS\n\n    if len(audio_np) > max_samples:\n        audio_np = audio_np[-max_samples:]\n\n    for attempt in range(3):\n        try:\n            print(f\"🔌 Connecting to vtuber_core at {CORE_HOST}:{CORE_PORT} (attempt {attempt+1}/3)...\")\n            sock = socket(AF_INET, SOCK_STREAM)\n            sock.settimeout(30.0)\n            sock.connect((CORE_HOST, CORE_PORT))\n            print(\"✅ Connected to vtuber_core\")\n\n            payload = {\"samples\": audio_np.tolist()}\n            data = dumps(payload).encode(\"utf-8\")\n\n            sock.sendall(len(data).to_bytes(4, \"big\") + data)\n\n            response = b\"\"\n            while True:\n                chunk = sock.recv(4096)\n                if not chunk:\n                    break\n                response += chunk\n\n            sock.close()\n\n            if not response:\n                print(\"⚠️ No response from vtuber_core\")\n                return None\n\n            result = loads(response.decode(\"utf-8\"))\n            wav_path = result.get(\"wav_path\")\n\n            if wav_path and os.path.exists(wav_path):\n                return wav_path\n\n            print(f\"⚠️ TTS file not found: {wav_path}\")\n            return None\n\n        except (ConnectionRefusedError, TimeoutError, OSError) as e:\n            print(f\"❌ Core not ready (attempt {attempt+1}/3): {e}\")\n            time.sleep(5)\n        except Exception as e:\n            print(f\"❌ send_audio_to_core error: {e}\")\n            import traceback\n            traceback.print_exc()\n            return None\n\n    return None\n\ndef play_wav_in_vc(vc: discord.VoiceClient, wav_path: str):\n    \"\"\"Play WAV file in Discord VC.\"\"\"\n    global tts_playing_flag\n\n    if not vc or vc.channel is None or not os.path.exists(wav_path):\n        return\n\n    try:\n        if hasattr(vc, \"is_playing\") and vc.is_playing():\n            vc.stop()\n    except Exception:\n        pass\n\n    tts_playing_flag = True\n\n    def after_play(err):\n        global tts_playing_flag, tts_cooldown_time\n        tts_playing_flag = False\n        tts_cooldown_time = time.time() + TTS_COOLDOWN_DURATION  # Set cooldown\n\n        try:\n            if os.path.exists(wav_path):\n                os.unlink(wav_path)\n                print(f\"🗑️ Cleaned up: {wav_path}\")\n        except Exception as e:\n            print(f\"⚠️ Failed to clean up {wav_path}: {e}\")\n\n        if err:\n            print(f\"❌ Playback error: {err}\")\n\n    try:\n        vc.play(\n            discord.FFmpegPCMAudio(\n                wav_path,\n                executable=\"ffmpeg\"\n            ),\n            after=after_play\n        )\n        print(f\"▶️ Playing: {wav_path}\")\n    except Exception as e:\n        print(f\"❌ Error playing audio: {e}\")\n        tts_playing_flag = False\n        print(f\"❌ Error playing audio: {e}\")\n        tts_playing_flag = False\n\ndef local_audio_player(wav_path: str):\n    \"\"\"Player function for local audio capture responses.\"\"\"\n    global current_voice_client\n    if current_voice_client and os.path.exists(wav_path):\n        if bot_loop and bot_loop.is_running():\n            bot_loop.call_soon_threadsafe(lambda: play_wav_in_vc(current_voice_client, wav_path))\n        else:\n            play_wav_in_vc(current_voice_client, wav_path)\n\ndef stop_voice_listener(vc: Optional[\"discord.VoiceProtocol\"] = None):\n    \"\"\"Stop any existing voice receive sink safely.\"\"\"\n    if vc is None or not isinstance(vc, discord.VoiceClient):\n        return\n    for name in (\"stop_listening\", \"stop\"):\n        fn = getattr(vc, name, None)\n        if callable(fn):\n            try:\n                fn()\n            except Exception:\n                pass\n            break\n\ndef stop_local_audio_capture():\n    \"\"\"Stop local audio capture if running.\"\"\"\n    global local_audio_pa, local_audio_stream, local_audio_thread\n    \n    if local_audio_module and local_audio_pa:\n        try:\n            local_audio_module.stop_local_audio_capture(local_audio_pa, local_audio_stream)\n            local_audio_pa = None\n            local_audio_stream = None\n            local_audio_thread = None\n            print(\"🛑 Local audio capture stopped\")\n        except Exception as e:\n            print(f\"❌ Error stopping local audio capture: {e}\")\n\nasync def connect_or_move_to_channel(channel: discord.VoiceChannel):\n    \"\"\"Single safe path for connecting/moving the voice client.\"\"\"\n    global current_voice_client, current_sink, local_audio_pa, local_audio_stream, local_audio_thread\n\n    async with voice_connect_lock:\n        guild = channel.guild\n        vc = guild.voice_client\n\n        if vc:\n            if vc.channel is None:\n                try:\n                    await vc.disconnect(force=True)\n                except Exception:\n                    pass\n                vc = None\n            elif isinstance(vc, discord.VoiceClient) and vc.channel != channel:\n                await vc.move_to(channel)\n                print(f\"🔊 Moved to {channel.name}\")\n                current_voice_client = vc\n                return vc\n\n        # Fresh connect\n        print(f\"🔌 Connecting to {channel.name}...\")\n        if not voice_recv_available:\n            raise RuntimeError(\"voice_recv not available\")\n            \n        vc = await channel.connect(\n            cls=voice_recv_imported.VoiceRecvClient,\n            timeout=30.0,\n            self_deaf=False,\n            self_mute=False\n        )\n        print(\"✅ Connected with VoiceRecvClient\")\n        \n        stop_voice_listener(vc)\n        \n        # Attach proper AudioSink for voice_recv\n        sink = DiscordAudioSink()\n        vc.listen(sink)\n        current_sink = sink\n        \n        current_voice_client = vc\n        print(f\"🎧 VC connected: {channel.name} (voice_recv sink attached)\")\n        \n        # Start local audio capture if enabled and available\n        if LOCAL_AUDIO_ENABLED and local_audio_module:\n            try:\n                print(\"🎤 Starting local audio capture...\")\n                local_audio_pa, local_audio_stream, local_audio_thread = local_audio_module.start_local_audio_capture(\n                    device_index=LOCAL_AUDIO_DEVICE_INDEX,\n                    discord_vc_player=local_audio_player\n                )\n                print(\"✅ Local audio capture started\")\n            except Exception as e:\n                print(f\"❌ Failed to start local audio capture: {e}\")\n                # Try without specific device index\n                try:\n                    print(\"🔄 Retrying with default device...\")\n                    local_audio_pa, local_audio_stream, local_audio_thread = local_audio_module.start_local_audio_capture(\n                        discord_vc_player=local_audio_player\n                    )\n                    print(\"✅ Local audio capture started with default device\")\n                except Exception as e2:\n                    print(f\"❌ Failed to start local audio capture with default device: {e2}\")\n        elif LOCAL_AUDIO_ENABLED:\n            print(\"⚠️ Local audio capture enabled but module not available\")\n\n        return vc\n\n# =============================\n# AUDIO SINK - Proper voice_recv subclass\n# =============================\nclass DiscordAudioSink(voice_recv_imported.AudioSink):\n    \"\"\"Proper voice_recv.AudioSink for PCM capture -> Whisper\"\"\"\n    def __init__(self):\n        super().__init__()\n        self.rolling = []\n        self.rolling_max = 50  # ~1 second (50 chunks × ~320 samples @16kHz)\n        self.speech_active = False\n        self.post_speech_seen = 0\n        self.hold_chunks = int(12000 / 320)  # ~0.75s end-of-turn silence hold\n\n    def wants_opus(self) -> bool:\n        return False\n\n    def write(self, user, data):\n        global last_audio_time, vc_buffer, buffer_lock, tts_cooldown_time\n\n        if user is None or not data.pcm:\n            return\n        if not is_allowed_member(user):\n            return\n        if tts_playing_flag or time.time() < tts_cooldown_time:\n            return\n\n        try:\n            audio_np = discord_pcm_to_whisper(data.pcm)\n            if audio_np.size < 80:\n                return\n\n            rms = np.sqrt(np.mean(audio_np ** 2))\n            amp = np.max(np.abs(audio_np))\n            is_speech = amp >= 0.003 and rms >= 0.0006\n\n            # Always maintain rolling buffer (all audio, no filtering)\n            self.rolling.append(audio_np)\n            if len(self.rolling) > self.rolling_max:\n                self.rolling.pop(0)\n\n            if not is_speech and not self.speech_active:\n                return\n\n            with buffer_lock:\n                if not self.speech_active and is_speech:\n                    print(f\"🔊 Speech onset ({amp:.5f} rms={rms:.5f}) prepending {len(self.rolling)} rolling chunks\")\n                    for chunk in self.rolling[:-1]:\n                        vc_buffer.append(chunk)\n                    self.rolling.clear()\n                    self.speech_active = True\n                    self.post_speech_seen = 0\n\n                if is_speech:\n                    self.post_speech_seen = 0\n                else:\n                    self.post_speech_seen += 1\n\n                vc_buffer.append(audio_np)\n                last_audio_time = time.time()\n\n                if self.post_speech_seen >= self.hold_chunks:\n                    self.speech_active = False\n\n        except Exception as e:\n            print(f\"⚠️ Audio sink write error: {e}\")\n\n    def cleanup(self):\n        print(\"🧹 Audio sink cleanup\")\n\n# =============================\n# SPEECH PROCESSOR THREAD\n# =============================\ndef vc_speech_processor():\n    global last_audio_time, tts_playing_flag, current_voice_client\n\n    print(\"[REFRESH] Speech processor thread started\")\n    print(\"💡 Make sure vtuber_core.py is running\")\n\n    while True:\n        time.sleep(0.1)\n\n        try:\n            with buffer_lock:\n                if not vc_buffer:\n                    continue\n\n                time_since_audio = time.time() - last_audio_time\n                total_samples = sum(len(chunk) for chunk in vc_buffer)\n\n                # audio_np is already 16kHz mono after discord_pcm_to_whisper()\n                buffer_full = total_samples >= 16000 * 2  # ~2s utterance\n                silence_long = time_since_audio >= 1.0 and total_samples >= 14000  # ~0.9s minimum length\n\n                max_wait = time_since_audio >= 5.0 and total_samples >= 14000\n\n                if not buffer_full and not silence_long and not max_wait:\n                    continue\n\n                chunks = list(vc_buffer)\n                vc_buffer.clear()\n\n            if not chunks:\n                continue\n\n            audio_np = np.concatenate(chunks)\n\n            if audio_np.size < 8000:\n                duration_short = audio_np.size / 16000.0\n                print(f\"⚠️ Audio too short ({audio_np.size} samples = {duration_short:.2f}s), skipping\")\n                continue\n\n            current_time = time.time()\n            if current_time < tts_cooldown_time:\n                continue\n\n            if tts_playing_flag:\n                continue\n\n            # audio_np is already converted to 16kHz mono in discord_pcm_to_whisper()\n            duration = audio_np.size / 16000.0\n            print(f\"📊 Audio buffer: {len(chunks)} chunks, {len(audio_np)} total samples ({duration:.2f}s)\")\n            print(f\"📤 Sending {duration:.2f}s audio to core...\")\n\n            # Save exact STT input for debugging\n            save_debug_wav(audio_np, 16000, f\"stt_input_{_debug_counter}.wav\")\n\n            wav_path = send_audio_to_core(audio_np)\n            if not wav_path:\n                continue\n\n            if current_voice_client and isinstance(current_voice_client, discord.VoiceClient) and wav_path:\n                print(f\"🔊 AI reply: {wav_path}\")\n                if bot_loop and bot_loop.is_running():\n                    bot_loop.call_soon_threadsafe(lambda: play_wav_in_vc(current_voice_client, wav_path))\n                else:\n                    play_wav_in_vc(current_voice_client, wav_path)\n\n            with buffer_lock:\n                last_audio_time = 0.0\n\n        except Exception as e:\n            print(f\"❌ Speech processor error: {e}\")\n\nthreading.Thread(target=vc_speech_processor, daemon=True).start()\n\n# =============================\n# DISCORD BOT\n# =============================\nintents = discord.Intents.default()\nintents.voice_states = True\nintents.message_content = True\nintents.members = True\n\nbot = commands.Bot(command_prefix=\"!\", intents=intents)\n\n@bot.event\nasync def on_ready():\n    global bot_loop\n    bot_loop = asyncio.get_running_loop()\n    print(f\"[OK] Bot logged in as {bot.user}\")\n    print(\"[BOT] Voice bot ready!\")\n\n@bot.event\nasync def on_voice_state_update(member, before, after):\n    \"\"\"Single clean voice handler\"\"\"\n    global connecting\n\n    if connecting or member.bot or not is_allowed_member(member) or after.channel is None or before.channel == after.channel:\n        return\n    \n    connecting = True\n    try:\n        await asyncio.sleep(2)\n        await connect_or_move_to_channel(after.channel)\n    except Exception as e:\n        print(f\"❌ Voice error: {e}\")\n    finally:\n        connecting = False\n\n@bot.command()\nasync def join(ctx):\n    \"\"\"Join voice channel\"\"\"\n    if not ctx.author.voice:\n        await ctx.send(\"❌ Join voice first!\")\n        return\n    await connect_or_move_to_channel(ctx.author.voice.channel)\n    await ctx.send(\"✅ Joined!\")\n\n@bot.command()\nasync def leave(ctx):\n    \"\"\"Leave voice\"\"\"\n    vc = ctx.guild.voice_client\n    if vc:\n        stop_voice_listener(vc)\n        stop_local_audio_capture()  # Stop local audio capture\n        await vc.disconnect()\n    await ctx.send(\"👋 Left\")\n\nbot.run(DISCORD_TOKEN)\n```\n\nother file:\n\n``` python\nimport socket\n\nimport json\n\nimport numpy as np\n\nimport tempfile\n\nimport soundfile as sf\n\nimport os\n\nimport sys\n\nimport argparse\n\nimport threading\n\nfrom rich.console import Console\n\nimport nltk\n\nimport re\n\nnltk.download('punkt', quiet=True)\n\nEXCLUDED_PHRASES = set()\n\nimport numpy as np\n\nfrom scipy.signal import resample_poly\n\ndef pcm_s16le_48k_stereo_to_16k_mono_float32(pcm_bytes: bytes) -> np.ndarray:\n\naudio_i16 = np.frombuffer(pcm_bytes, dtype=np.int16)\n\nif audio_i16.size == 0:\n\nreturn np.zeros(0, dtype=np.float32)\n\n# Drop incomplete stereo frame if needed.\n\nusable = (audio_i16.size // 2) * 2\n\naudio_i16 = audio_i16[:usable]\n\n# frames x channels\n\nstereo = audio_i16.reshape(-1, 2)\n\n# stereo → mono (mean, not sum!)\n\nmono = stereo.astype(np.float32).mean(axis=1)\n\n# int16 → float32\n\nmono = mono / 32768.0\n\n# safety clamp — this is what keeps your amplitude in [-1, 1]\n\nmono = np.clip(mono, -1.0, 1.0)\n\n# 48 kHz → 16 kHz\n\nmono_16k = resample_poly(mono, up=1, down=3)\n\nreturn mono_16k.astype(np.float32)\n\n# Ensure HF_HOME doesn't point to stale cache (overrides external tool env vars)\n\nif \"HF_HOME\" in os.environ and \"TEST OMNIVOICE\" in os.environ.get(\"HF_HOME\", \"\"):\n\ndel os.environ[\"HF_HOME\"]\n\nif \"HF_HUB_CACHE\" not in os.environ:\n\nos.environ[\"HF_HUB_CACHE\"] = os.path.join(os.path.expanduser(\"~\"), \".cache\", \"huggingface\", \"hub\")\n\n# Parse arguments\n\nparser = argparse.ArgumentParser(description=\"Standalone VTuber Core Server\")\n\nparser.add_argument(\"--voice\", type=str, default=\"meuro-enhanced-v2.wav\")\n\nparser.add_argument(\"--cfg-weight\", type=float, default=0.5)\n\nargs = parser.parse_args()\n\nconsole = Console()\n\nfrom improved_local_stt import process_audio_chunk\n\ndef is_valid_text(text):\n\ntext = text.strip()\n\nif not text:\n\nreturn False\n\n# Check for repetitive patterns that indicate feedback\n\nwords = text.lower().split()\n\nif len(words) > 3:\n\nunique_words = set(words)\n\nrepetition_ratio = len(words) / len(unique_words) if len(unique_words) > 0 else 0\n\nif repetition_ratio > 3.0:  # High repetition likely means feedback\n\nprint(f\"🚫 Likely feedback detected (repetition ratio: {repetition_ratio:.2f}): '{text}'\")\n\nreturn False\n\n# repeated spam like \"the the the\" - increased threshold to allow more variety\n\nif len(words) > 3:  # Only check repetition if we have more than 3 words\n\nif len(set(words)) <= 1:  # Only reject if all words are identical\n\nprint(f\"🚫 All words identical: '{text}'\")\n\nreturn False\n\nreturn True\n\nHALLUCINATED_PHRASES = {\n\n\"thanks for watching\", \"subscribe\", \"thank you\", \"you\",\n\n\"thank you for watching\", \"please subscribe\", \"like and subscribe\",\n\n\"thanks\", \"bye\", \"goodbye\", \"see you\", \"thank\",\n\n\"thanks for listening\", \"thank you for listening\",\n\n\"the\", \"a\", \"and\", \"i\", \"we\", \"you\", \"it\",\n\n}\n\ndef is_valid_transcription(text):\n\n\"\"\"Check if transcription is valid (not garbage/hallucination)\"\"\"\n\nif not text or not text.strip():\n\nreturn False\n\ncleaned = re.sub(r'[.!?,]', '', text.lower()).strip()\n\nif cleaned in HALLUCINATED_PHRASES:\n\nprint(f\"🚫 Hallucination filtered: '{text}'\")\n\nreturn False\n\nif cleaned in EXCLUDED_PHRASES:\n\nreturn False\n\nwords = text.split()\n\nif len(words) < 2:\n\nprint(f\"🚫 Too short: '{text}'\")\n\nreturn False\n\nunique_words = len(set(words))\n\nif len(words) > 3 and unique_words / len(words) < 0.3:\n\nprint(f\"🚫 Repetitive: '{text}'\")\n\nreturn False\n\nreturn True\n\nAUDIO_DEBUG_DIR = \"debug_audio\"\n\nos.makedirs(AUDIO_DEBUG_DIR, exist_ok=True)\n\n_audio_counter = 0\n\ndef save_debug_wav(audio, sample_rate: int, filename: str):\n\n\"\"\"Save audio to debug_audio/ as a WAV file for STT debugging.\"\"\"\n\naudio = np.asarray(audio, dtype=np.float32)\n\naudio = np.clip(audio, -1.0, 1.0)\n\npath = os.path.join(AUDIO_DEBUG_DIR, filename)\n\ntry:\n\nsf.write(path, audio, sample_rate)\n\nexcept Exception as e:\n\nprint(f\"[DEBUG] Save failed ({filename}): {e}\")\n\ndef safe_transcribe(audio_np_or_pcm_bytes, *, input_sr: int = 48000, input_channels: int = 2):\n\n\"\"\"Accept either:\n\n- float32 numpy audio at 16kHz mono (current behavior)\n\n- raw PCM bytes: S16LE stereo at 48kHz (will convert to 16k mono float32)\n\n\"\"\"\n\nglobal _audio_counter\n\n# Convert bytes -> float32 mono @16kHz16kHz\n\nif isinstance(audio_np_or_pcm_bytes, (bytes, bytearray)):\n\nif input_sr != 48000 or input_channels != 2:\n\nprint(f\"[WARN] PCM conversion assumes 48kHz stereo S16LE; got sr={input_sr}, ch={input_channels}\")\n\naudio_np = pcm_s16le_48k_stereo_to_16k_mono_float32(bytes(audio_np_or_pcm_bytes))\n\ndebug_in_sr = 16000\n\nprint(f\"[PCM] Converted PCM bytes -> float32 mono {audio_np.size} samples @16k\")\n\nelse:\n\naudio_np = audio_np_or_pcm_bytes\n\ndebug_in_sr = 16000  # bot-side discord_pcm_to_whisper already produced 16kHz mono\n\nif audio_np is None or getattr(audio_np, \"size\", 0) == 0:\n\nreturn \"\"\n\nprint(f\"[SEARCH] Audio analysis - size: {audio_np.size}, max_amplitude: {np.max(np.abs(audio_np)):.6f}\")\n\n# Limit audio length\n\nmax_samples = 48000\n\nif len(audio_np) > max_samples:\n\naudio_np = audio_np[-max_samples:]\n\nprint(f\"[TRIM] Trimmed to last {max_samples} samples\")\n\nmax_amp = np.max(np.abs(audio_np))\n\nprint(f\"[MIC] Incoming audio | amp={max_amp:.6f} | samples={len(audio_np)}\")\n\nprint(f\"[PROCESS] Processing audio: {audio_np.size} samples, {max_amp:.6f} max amplitude\")\n\n# Save debug WAV (exact STT input)\n\n_audio_counter += 1\n\nsave_debug_wav(audio_np, debug_in_sr, f\"debug_{_audio_counter}.wav\")\n\nprint(f\"[DEBUG] Saved: debug_{_audio_counter}.wav\")\n\nduration_s = len(audio_np) / 16000.0\n\nif duration_s < 0.5:\n\nprint(f\"🚫 Rejected audio: {duration_s:.2f}s too short\")\n\nreturn \"\"\n\nif max_amp < 0.001:\n\nprint(f\"🚫 Rejected audio: amp={max_amp:.6f}\")\n\nreturn \"\"\n\nif max_amp > 1.05:\n\nprint(f\"🛑 Clipped audio: amp={max_amp:.6f} (should be ≤1.0)\")\n\ntry:\n\nprint(\"[STT] Transcribing with improved local STT (fixed)...\")\n\ntext = process_audio_chunk(\"discord_user\", audio_np) or \"\"\n\nprint(f\"[RAW] RAW STT: '{text}'\")\n\nprint(f\"[TEXT] Transcribed text: '{text}'\")\n\nreturn text  # Always str\n\nexcept Exception as stt_error:\n\nprint(f\"[STT-ERROR] STT failed: {stt_error}\")\n\nreturn \"\"\n\nimport requests\n\nSYSTEM_PROMPT = \"You are a helpful AI.\"\n\ndef simple_llm(text):\n\ntry:\n\nprompt = f\"{SYSTEM_PROMPT}\\n\\nUser: {text}\\nAI:\"\n\nprint(f\"📤 Sending to Ollama: '{text[:50]}...'\")\n\nresponse = requests.post(\n\n\"http://127.0.0.1:11434/api/generate\",\n\nheaders={\"Content-Type\": \"application/json\"},\n\njson={\n\n\"model\": \"drivedenpadev/deepseek-v3.2\",\n\n\"prompt\": prompt,\n\n\"stream\": False\n\n},\n\ntimeout=90\n\n)\n\nprint(f\"📥 Ollama response status: {response.status_code}\")\n\nif response.status_code != 200:\n\nprint(f\"⚠️ Ollama returned status {response.status_code}: {response.text}\")\n\nreturn \"Sorry, something went wrong.\"\n\n# safer JSON handling\n\ntry:\n\ndata = response.json()\n\nexcept Exception:\n\nprint(\"❌ Failed to parse JSON:\", response.text)\n\nreturn \"Invalid response from LLM.\"\n\nprint(f\"📄 Ollama response data: {data}\")\n\nresult = data.get(\"response\", \"\").strip()\n\nif not result:\n\nprint(\"⚠️ Empty response from model\")\n\nreturn \"No response from AI.\"\n\nprint(f\"💬 Extracted response: '{result}'\")\n\nreturn result\n\nexcept requests.exceptions.ConnectionError:\n\nprint(\"❌ Cannot connect to Ollama - is it running?\")\n\nreturn \"Sorry, LLM service unavailable.\"\n\nexcept requests.exceptions.Timeout:\n\nprint(\"❌ Ollama request timed out\")\n\nreturn \"Sorry, taking too long to think.\"\n\nexcept Exception as e:\n\nprint(f\"❌ Ollama error: {e}\")\n\nimport traceback\n\ntraceback.print_exc()\n\nreturn \"Sorry, something went wrong.\"\n\nSYSTEM_PROMPT = \"\"\"\n\nYou are a funny VTuber. Keep replies under 14 words. No emojis. Act human\n\n\"\"\"\n\n# TTS using tts.py (standalone)\n\nsys.path.insert(0, os.path.dirname(_file_))\n\nfrom tts import TextToSpeechService, sanitize_tts_text\n\ntry:\n\nprint(\"[ROCKET] Initializing TTS service...\")\n\ntts = TextToSpeechService()\n\nprint(f\"[OK] TTS service initialized (device: {tts.device})\")\n\n# Check if voice file exists\n\nif args.voice and os.path.exists(args.voice):\n\nprint(f\"[OK] Voice model found: {args.voice}\")\n\nelif args.voice:\n\nprint(f\"[WARN] Voice model not found: {args.voice}\")\n\nelse:\n\nprint(\"[WARN] No voice model specified\")\n\nexcept Exception as e:\n\nprint(f\"[ERROR] Failed to initialize TTS service: {e}\")\n\nimport traceback\n\ntraceback.print_exc()\n\ntts = None\n\nHOST = os.getenv(\"CORE_HOST\", \"127.0.0.1\")\n\nPORT = int(os.getenv(\"CORE_PORT\", \"8765\"))\n\ndef handle_audio(audio_np_or_pcm_bytes):\n\ntry:\n\nprint(\"📝 Transcribing...\")\n\ntext = safe_transcribe(audio_np_or_pcm_bytes)\n\nprint(f\"📝 Transcribed text: '{text}'\")\n\nprint(f\"👤 You: {text}\")\n\nif not is_valid_transcription(text):\n\nprint(f\"⏭️ Skipping Ollama (invalid transcript)\")\n\nreturn None, text\n\nresponse = simple_llm(text)\n\nprint(f\"🤖 AI: {response}\")\n\n# Trim response to reasonable length\n\nif len(response) > 500:\n\nresponse = response[:500] + \"...\"\n\nprint(f\"✂️ Trimmed long response: {len(response)} chars\")\n\nsafe_text = sanitize_tts_text(response)\n\nprint(f\"🔤 Sanitized text: '{safe_text}'\")\n\ntry:\n\nif tts is None:\n\nprint(\"❌ TTS service is not initialized\")\n\nreturn None, text\n\nprint(\"🎵 Generating audio...\")\n\nsr, audio = tts.synthesize(\n\nsafe_text,\n\naudio_prompt_path=args.voice,\n\ncfg_weight=args.cfg_weight\n\n)\n\nprint(f\"🎵 TTS result: sr={sr}, audio_shape={getattr(audio, 'shape', 'N/A')}\")\n\nif audio is None or len(audio) == 0:\n\nprint(\"🔇 TTS generated empty audio, skipping\")\n\nreturn None, text\n\ntmp = tempfile.NamedTemporaryFile(suffix=\".wav\", delete=False)\n\ntmp.close()\n\nsf.write(tmp.name, audio, sr)\n\nprint(f\"✅ TTS generated successfully: {tmp.name}\")\n\nreturn tmp.name, text\n\nexcept Exception as tts_error:\n\nprint(f\"❌ TTS generation failed: {tts_error}\")\n\nimport traceback\n\ntraceback.print_exc()\n\nreturn None, text\n\nexcept Exception as e:\n\nprint(f\"❌ Error: {e}\")\n\nimport traceback\n\ntraceback.print_exc()\n\nreturn None\n\ndef server_loop():\n\nsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n\nsock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n\nsock.bind((HOST, PORT))\n\nsock.listen(1)\n\nprint(f\"[SERVER] Standalone Core on {HOST}:{PORT}\")\n\nwhile True:\n\nconn, addr = sock.accept()\n\nprint(f\"[CONNECT] {addr}\")\n\n# Read length + data\n\nsize_bytes = conn.recv(4)\n\nsize = int.from_bytes(size_bytes, \"big\")\n\ndata = b\"\"\n\nwhile len(data) < size:\n\ndata += conn.recv(4096)\n\npayload = json.loads(data)\n\n# Incoming payload audio formats (choose one):\n\n# 1) float32 mono/whatever samples: payload[\"samples\"] as list[float]\n\n# 2) raw PCM bytes (S16LE stereo 48k): payload[\"pcm_s16le\"] as base64 str or list[int]\n\n#    - If using base64: payload[\"pcm_s16le\"] must be a base64-encoded S16LE byte stream.\n\nif \"pcm_s16le\" in payload:\n\npcm_val = payload[\"pcm_s16le\"]\n\nif isinstance(pcm_val, str):\n\nimport base64\n\naudio_input = base64.b64decode(pcm_val)\n\nelse:\n\n# assume list[int] of int16 bytes\n\naudio_input = bytes(pcm_val)\n\nelse:\n\naudio_list = payload.get(\"samples\", [])\n\naudio_input = np.array(audio_list, dtype=np.float32)\n\nwav_path, transcription = handle_audio(audio_input)\n\nresponse = json.dumps({\"wav_path\": wav_path or \"\", \"transcription\": transcription}).encode()\n\nconn.sendall(response)\n\nconn.close()\n\nif _name_ == \"_main_\":\n\nserver_loop()\n```\n\nwhat am i doing wrong help i’ve tried everything and nothing works to fix stt on discord i’m trying to do it locally because i am losing my mind after months of trying to make this work", "url": "https://wpnews.pro/news/stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix", "canonical_source": "https://discuss.huggingface.co/t/stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix/177600#post_1", "published_at": "2026-07-09 02:12:15+00:00", "updated_at": "2026-07-09 03:20:37.493022+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "developer-tools"], "entities": ["Discord", "VTuber"], "alternates": {"html": "https://wpnews.pro/news/stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix", "markdown": "https://wpnews.pro/news/stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix.md", "text": "https://wpnews.pro/news/stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix.txt", "jsonld": "https://wpnews.pro/news/stt-on-discord-vc-for-my-ai-vtuber-how-do-i-fix.jsonld"}}