Show HN: ShadowDiff – Shadow testing and SQL diffing for AI-migrated back ends

A developer released ShadowDiff, a network proxy and analysis engine that mirrors live HTTP traffic to both a legacy backend and its AI-refactored replacement to isolate regressions before production deployment. ShadowDiff targets the roughly 5% of hidden edge cases that AI code migration engines miss, including serialization discrepancies such as Jackson omitting empty lists versus Go serializing null, timezone truncations, float precision drift, and database state regressions like missing audit writes and N+1 query loops. The tool integrates with Envoy Proxy and Istio service mesh via request_mirror_policies, filters non-deterministic noise by comparing twin baseline instances, and uses Google Gemini to generate Go and Jest tests from detected differences.

Eliminating the 5% risk that stops enterprise CTOs from deploying AI-modernized backends to production. 🌐 Interactive Live Cloud Sandbox : https://shadowdiff-demo.fly.dev https://shadowdiff-demo.fly.dev/ Self-driving dual-traffic simulator active Enterprises are actively modernizing mission-critical legacy backends Java 7/8 Spring monoliths, Python 2 Django, PHP 5.6 backends into modern cloud-native services Go, TypeScript/Node, FastAPI using AI code migration engines. While AI generates ~95% of refactored code accurately, the remaining 5% of hidden edge cases make production deployment an unacceptable risk : - Subtle Serialization Discrepancies : Jackson omitting empty lists vs Go serializing null , datetime timezone truncations, float precision drift. - Implicit Framework Quirks : Legacy servlet filter quirks, undocumented fallback headers, HTTP 200 responses with embedded error payloads. - Database State Regressions : Silent differences in database transaction boundaries, missing audit writes, or N+1 query loop explosions. ShadowDiff is a high-performance network proxy and analysis engine that duplicates real staging or production HTTP traffic, sending identical payloads to both the legacy service and the AI-refactored service simultaneously. Live Inbound Traffic │ ▼ ┌─────────────────────────────┐ │ Envoy Gateway / Mesh │ │ request mirror policies │ └──────────────┬──────────────┘ │ ┌───────────────────┴───────────────────┐ ▼ ▼ ┌───────────────────┐ ┌───────────────────┐ │ Legacy Service │ │ ShadowDiff Core │ │ Target │ │ Analysis Engine │ └─────────┬─────────┘ └─────────┬─────────┘ │ │ │ Live Response to Client ▼ │ ┌───────────────────┐ │ │ AI-Refactored │ │ │ Candidate S │ │ └─────────┬─────────┘ │ │ └───────────────────┬───────────────────┘ ▼ ┌─────────────────────────────┐ │ 3-Way Diff & SQL Comparator │ │ - P1 ⊕ P2 Noise Filter │ │ - SQL Mutation Interceptor │ │ - N+1 Query Loop Detector │ └──────────────┬──────────────┘ │ Pure Regressions Isolated ▼ ┌─────────────────────────────┐ │ Automated AI Test Generator │ │ - Powered by Google Gemini │ │ - Synthesizes Go/Jest Tests │ │ - Root Cause Code Fixes │ └─────────────────────────────┘ ShadowDiff integrates directly with Envoy Proxy and Istio Service Mesh using request mirror policies . Traffic is cloned asynchronously at line rate in the network layer with zero application latency penalty . Automatically filters out non-deterministic runtime noise timestamps, UUIDs, session nonces by evaluating twin baseline instances: Captures and correlates SQL statements executed by both services for each HTTP trace: - Missing Writes : Flags when legacy wrote to audit logs or ledger tables that modern code omitted. - Transaction Leaks : Warns if legacy used BEGIN...COMMIT and modern ran autocommit statements. - N+1 Query Explosions : Detects when modern ORMs loop and execute$\ge 3\times$ more queries than legacy. Every isolated edge case is analyzed by Google Gemini gemini-2.0-flash , which writes a turnkey integration test Go httptest or Vitest and diagnoses the code fix. Eliminates manual triaging by directly integrating with the GitHub API: - Automatic Branching : Spins up shadowdiff/fix-<trace id from the repository's default branch. - Test File Commitment : Automatically commits the synthesized regression test directly to the repository e.g. tests/regression/test drift <trace id .go . - Rich Pull Request Generation : Opens a detailed PR with side-by-side JSON diff tables, SQL mutation waterfall analysis, N+1 query warnings, and local verification commands. - 1-Click Triage in Console : Developers can inspect the incident in the web console and click "🚀 Open GitHub PR" to immediately open the PR on GitHub. A native deployment blocker that evaluates traffic quality during pre-release soak tests or integration pipelines: - Automated Evaluation : Assesses semantic parity, database mutations, and N+1 query loops over a specified evaluation window. - Zero False Failure : Honors Diffy noise cancellation $\mathcal{N} = P 1 \oplus P 2$ , only failing on genuine regressions. - Pipeline Integration : Returns standard exit codes 0 for PASS, 1 for FAIL/violation, 2 for error with high-impact ANSI colored terminal summaries and optional JSON format. Evaluate quality gate in your CI/CD pipeline: shadowdiff gate \ -dashboard-url=http://localhost:8083 \ -duration=15s \ -min-parity=100.0 \ -min-requests=50 \ -max-drifts=0 \ -allow-n-plus-one=false \ -allow-db-drift=false Allows safe shadow mirroring of mutating HTTP methods POST , PUT , DELETE , PATCH with zero database pollution : - Transactional Savepoint Rollbacks : Injects boundary headers to ensure shadow writes execute within an isolated savepoint and automatically roll back. - Ephemeral Schema Sandboxing : Dynamically creates disposable schemas per test run and drops them upon completion. - Serverless CoW Branching : Leverages Neon / Supabase serverless branching to clone database state instantly via API for mutating candidate runs. Incorporate ShadowDiff directly into your automated deployment pipeline .github/workflows/shadowdiff.yml : name: Dual-Traffic Shadow Gating on: pull request, push jobs: shadow-gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: go-version: '1.22' - name: Build & Start Services run: | go build -o bin/shadowdiff ./cmd/shadowdiff/main.go ./bin/legacy-service -port=:8081 & ./bin/refactored-service -port=:8082 & ./bin/shadowdiff -proxy-port=:8080 -primary-url=http://localhost:8081 -shadow-url=http://localhost:8082 -sandbox-mode=rollback & sleep 2 - name: Replay Traffic run: | for i in {1..25}; do curl -s "http://localhost:8080/api/users?id=$i" /dev/null curl -s -X POST "http://localhost:8080/api/orders" -d '{"amount":99.50}' /dev/null done - name: Run ShadowDiff Quality Gate run: | ./bin/shadowdiff gate \ -dashboard-url=http://localhost:8083 \ -min-parity=100.0 \ -min-requests=20 \ -allow-n-plus-one=false \ -allow-db-drift=false Launch the complete 5-container service mesh testbed Envoy Gateway + ShadowDiff + Legacy Target + Twin Replica + AI Candidate : 1. Clone the repository git clone https://github.com/vishalrajofficial/ShadowDiff.git cd ShadowDiff 2. Launch the Envoy Service Mesh topology docker compose -f docker-compose.envoy.yml up -d Send test traffic to the Envoy Gateway mirrored to ShadowDiff automatically : curl http://localhost:10000/api/users/1 curl -X POST http://localhost:10000/api/orders Open the telemetry dashboard at http://localhost:8083 to inspect the live 3-way diffs, SQL mutation timeline, and Gemini-generated tests For production Kubernetes clusters running Istio or native Envoy, deploy the included EnvoyFilter to mirror traffic dynamically: kubectl apply -f deploy/k8s/envoy-shadow-sidecar.yaml See docs/ARCHITECTURE.md https://github.com/vishalrajofficial/ShadowDiff/blob/main/docs/ARCHITECTURE.md for full architectural specifications. This project is licensed under the MIT License https://github.com/vishalrajofficial/ShadowDiff/blob/main/LICENSE .