Show HN: TrackLLM: Are the LLM APIs you rely on stable?

TrackLLM, a monitoring project posted to Hacker News, reports that 839 of 1,743 cataloged LLM API endpoints are being tracked and that 191 changes occurred across 150 endpoints, with probe-prompt outputs shifting as recently as 11 Sep at total variation values of 0.20 and 0.23. The dashboard lists changed endpoints including Tencent's hy3 on atlas-cloud/fp8 (3 changes, last change 28 Aug), Anthropic's claude-sonnet-4.5 on claude-on-aws (2 changes, 9 Sep), DeepSeek's deepseek-v4-flash-0731 on streamlake/fp8 (2 changes, 1 Sep), and Z.ai's glm-5.2 across cloudflare, gmicloud/fp8, together, and z-ai/fp8 (2 changes each). The tracking data matters because it gives developers a public record of how much hosted model behavior drifts from baseline over time.

11 Sep Probe-prompt outputs shifted total variation 0.20 B3IT 11 Sep Probe-prompt outputs shifted total variation 0.23 B3IT 10 Sep Probe-prompt outputs shifted total variation 0.38 B3IT 9 Sep Probe-prompt outputs shifted total variation 0.21 B3IT 7 Sep Probe-prompt outputs shifted total variation 0.36 B3IT 5 Sep Probe-prompt outputs shifted total variation 0.28 B3IT 4 Sep Probe-prompt outputs shifted total variation 0.34 B3IT 3 Sep Probe-prompt outputs shifted total variation 0.55 B3IT 2 Sep Probe-prompt outputs shifted total variation 0.34 B3IT More… changes.html More… providers.html Endpoints endpoints.html 839 endpoints monitored, of 1,743 in the catalog · 191 changes across 150 endpoints All endpoints → endpoints.html | Endpoint | Status | Changes | Methods | Last change | Drift from baseline | |---|---|---|---|---|---| | hy3 endpoints/tencent2fhy323atlas-cloud2ffp8.html atlas-cloud/fp8 providers/atlas-cloud.html | changed | 3 | B3IT | 28 Aug | | | claude-sonnet-4.5 endpoints/anthropic2fclaude-sonnet-4.523claude-on-aws.html claude-on-aws providers/claude-on-aws.html | changed | 2 | B3IT | 9 Sep | | | deepseek-v4-flash-0731 endpoints/deepseek2fdeepseek-v4-flash-073123streamlake2ffp8.html streamlake/fp8 providers/streamlake.html | changed | 2 | LTB3IT | 1 Sep | | | glm-5.2 endpoints/z-ai2fglm-5.223cloudflare.html cloudflare providers/cloudflare.html | changed | 2 | B3IT | 14 Aug | | | glm-5.2 endpoints/z-ai2fglm-5.223gmicloud2ffp8.html gmicloud/fp8 providers/gmicloud.html | changed | 2 | LTB3IT | 29 Aug | | | glm-5.2 endpoints/z-ai2fglm-5.223together.html together providers/together.html | changed | 2 | B3IT | 24 Aug | | | glm-5.2 endpoints/z-ai2fglm-5.223z-ai2ffp8.html z-ai/fp8 providers/z-ai.html | changed | 2 | B3IT | 19 Aug | | | gpt-4o-2024-05-13 endpoints/openai2fgpt-4o-2024-05-1323azure.html azure providers/azure.html | changed | 2 | LT | 21 May | | | gpt-4o-mini endpoints/openai2fgpt-4o-mini23azure.html azure providers/azure.html | changed | 2 | LTB3IT | 23 May | | | hy3 endpoints/tencent2fhy323deepinfra2ffp8.html deepinfra/fp8 providers/deepinfra.html | changed | 2 | B3IT | 20 Aug | | More… endpoints.html