Show HN: ShadowDiff – Shadow testing and SQL diffing for AI-migrated back ends

Eliminating the 5% risk that stops enterprise CTOs from deploying AI-modernized backends to production.

🌐 Interactive Live Cloud Sandbox: https://shadowdiff-demo.fly.dev (Self-driving dual-traffic simulator active)

Enterprises are actively modernizing mission-critical legacy backends (Java 7/8 Spring monoliths, Python 2 Django, PHP 5.6 backends) into modern cloud-native services (Go, TypeScript/Node, FastAPI) using AI code migration engines.

While AI generates ~95% of refactored code accurately, the remaining 5% of hidden edge cases make production deployment an unacceptable risk:

• Subtle Serialization Discrepancies : Jackson omitting empty lists vs Go serializingnull , datetime timezone truncations, float precision drift.
• Implicit Framework Quirks : Legacy servlet filter quirks, undocumented fallback headers, HTTP 200 responses with embedded error payloads.
• Database State Regressions : Silent differences in database transaction boundaries, missing audit writes, or N+1 query loop explosions.

ShadowDiff is a high-performance network proxy and analysis engine that duplicates real staging or production HTTP traffic, sending identical payloads to both the legacy service and the AI-refactored service simultaneously.

                           [ Live Inbound Traffic ]
                                      │
                                      ▼
                       ┌─────────────────────────────┐
                       │    Envoy Gateway / Mesh     │
                       │  (request_mirror_policies)  │
                       └──────────────┬──────────────┘
                                      │
                  ┌───────────────────┴───────────────────┐
                  ▼                                       ▼
        ┌───────────────────┐                   ┌───────────────────┐
        │  Legacy Service   │                   │  ShadowDiff Core  │
        │     (Target)      │                   │  (Analysis Engine)│
        └─────────┬─────────┘                   └─────────┬─────────┘
                  │                                       │
                  │ (Live Response to Client)             ▼
                  │                             ┌───────────────────┐
                  │                             │   AI-Refactored   │
                  │                             │   Candidate (S)   │
                  │                             └─────────┬─────────┘
                  │                                       │
                  └───────────────────┬───────────────────┘
                                      ▼
                       ┌─────────────────────────────┐
                       │ 3-Way Diff & SQL Comparator │
                       │ - P1 ⊕ P2 Noise Filter      │
                       │ - SQL Mutation Interceptor  │
                       │ - N+1 Query Loop Detector   │
                       └──────────────┬──────────────┘
                                      │
                         (Pure Regressions Isolated)
                                      ▼
                       ┌─────────────────────────────┐
                       │ Automated AI Test Generator │
                       │ - Powered by Google Gemini  │
                       │ - Synthesizes Go/Jest Tests │
                       │ - Root Cause Code Fixes     │
                       └─────────────────────────────┘

ShadowDiff integrates directly with Envoy Proxy and Istio Service Mesh using request_mirror_policies. Traffic is cloned asynchronously at line rate in the network layer with zero application latency penalty.

Automatically filters out non-deterministic runtime noise (timestamps, UUIDs, session nonces) by evaluating twin baseline instances:

Captures and correlates SQL statements executed by both services for each HTTP trace:

Missing Writes : Flags when legacy wrote toaudit_logs or ledger tables that modern code omitted. #

Transaction Leaks : Warns if legacy usedBEGIN...COMMIT and modern ran autocommit statements. #

N+1 Query Explosions : Detects when modern ORMs loop and execute$\ge 3\times$ more queries than legacy.

Every isolated edge case is analyzed by Google Gemini (gemini-2.0-flash), which writes a turnkey integration test (Go httptest or Vitest) and diagnoses the code fix.

Eliminates manual triaging by directly integrating with the GitHub API:

• Automatic Branching : Spins upshadowdiff/fix-<trace_id> from the repository's default branch.
• Test File Commitment : Automatically commits the synthesized regression test directly to the repository (e.g.tests/regression/test_drift_<trace_id>.go ).
• Rich Pull Request Generation : Opens a detailed PR with side-by-side JSON diff tables, SQL mutation waterfall analysis, N+1 query warnings, and local verification commands.
• 1-Click Triage in Console : Developers can inspect the incident in the web console and click**"🚀 Open GitHub PR"** to immediately open the PR on GitHub.

A native deployment blocker that evaluates traffic quality during pre-release soak tests or integration pipelines:

Automated Evaluation : Assesses semantic parity, database mutations, and N+1 query loops over a specified evaluation window. #

Zero False Failure : Honors Diffy noise cancellation ($\mathcal{N} = P_1 \oplus P_2$ ), only failing on genuine regressions. #

Pipeline Integration : Returns standard exit codes (0 for PASS,1 for FAIL/violation,2 for error) with high-impact ANSI colored terminal summaries and optional JSON format.

shadowdiff gate \
  -dashboard-url=http://localhost:8083 \
  -duration=15s \
  -min-parity=100.0 \
  -min-requests=50 \
  -max-drifts=0 \
  -allow-n-plus-one=false \
  -allow-db-drift=false

Allows safe shadow mirroring of mutating HTTP methods (POST, PUT, DELETE, PATCH) with zero database pollution:

• Transactional Savepoint Rollbacks : Injects boundary headers to ensure shadow writes execute within an isolated savepoint and automatically roll back.
• Ephemeral Schema Sandboxing : Dynamically creates disposable schemas per test run and drops them upon completion.
• Serverless CoW Branching : Leverages Neon / Supabase serverless branching to clone database state instantly via API for mutating candidate runs.

Incorporate ShadowDiff directly into your automated deployment pipeline (.github/workflows/shadowdiff.yml):

name: Dual-Traffic Shadow Gating
on: [pull_request, push]

jobs:
  shadow-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: '1.22'

      - name: Build & Start Services
        run: |
          go build -o bin/shadowdiff ./cmd/shadowdiff/main.go
          ./bin/legacy-service -port=:8081 &
          ./bin/refactored-service -port=:8082 &
          ./bin/shadowdiff -proxy-port=:8080 -primary-url=http://localhost:8081 -shadow-url=http://localhost:8082 -sandbox-mode=rollback &
          sleep 2

      - name: Replay Traffic
        run: |
          for i in {1..25}; do
            curl -s "http://localhost:8080/api/users?id=$i" > /dev/null
            curl -s -X POST "http://localhost:8080/api/orders" -d '{"amount":99.50}' > /dev/null
          done

      - name: Run ShadowDiff Quality Gate
        run: |
          ./bin/shadowdiff gate \
            -dashboard-url=http://localhost:8083 \
            -min-parity=100.0 \
            -min-requests=20 \
            -allow-n-plus-one=false \
            -allow-db-drift=false

Launch the complete 5-container service mesh testbed (Envoy Gateway + ShadowDiff + Legacy Target + Twin Replica + AI Candidate):

git clone https://github.com/vishalrajofficial/ShadowDiff.git
cd ShadowDiff

docker compose -f docker-compose.envoy.yml up -d
curl http://localhost:10000/api/users/1
curl -X POST http://localhost:10000/api/orders

Open the telemetry dashboard at http://localhost:8083 to inspect the live 3-way diffs, SQL mutation timeline, and Gemini-generated tests!

For production Kubernetes clusters running Istio or native Envoy, deploy the included EnvoyFilter to mirror traffic dynamically:

kubectl apply -f deploy/k8s/envoy-shadow-sidecar.yaml

See docs/ARCHITECTURE.md for full architectural specifications.

This project is licensed under the MIT License.