Hi everyone,
For a long time, the AI industry has pushed the narrative that advanced safety research, data poisoning auditing, and model alignment require massive clusters and millions of dollars. I wanted to test if that's true. Over the last few months, using a 2006 Toshiba Satellite (2GB RAM) for orchestration and free Kaggle T4 GPUs ($0 cost), I built and ran a 16-experiment series (EXP08–EXP16) exploring epistemic collapse and data poisoning in LLMs.
Today, I'm open-sourcing the first phase of this work: Beatriz Epistemic Gate.
What problem does it solve?
Malicious data poisoning during fine-tuning is often surgical and invisible to aggregate metrics (like standard perplexity). An attacker can destroy specific facts down to an exact indifference point while global metrics actually improve because of generic fine-tuning on fluent prose. This makes silent poisoning a stealthy threat for small teams and startups fine-tuning local models.
How does Beatriz work?
Instead of invading or rewriting complex training loops, Beatriz acts as a lightweight, non-invasive defensive proxy (~0.1ms decision latency) that sits between a generative source and a training student model. It verifies generated text against an immutable anchor corpus using cosine similarity in embedding spaces, applying a composite loss function:
• LceLce: Preserves linguistic fluency.
• LcontrastivaLcontrastiva (Softplus): Keeps active pressure on the model to anchor to truth even after cross-entropy saturates.
Key Results (EXP08–EXP16):
• Multi-architecture validation: Tested across GPT-2 (124M), Qwen-2.5-0.5B, TinyLlama-1.1B, Pythia-1.4B, and Phi-3-mini-4k-instruct (3.8B).
• Ablation (EXP15): Pure filtering (GATE_ONLY) provides 65% of the defensive benefit out of the box. The contrastive term adds the remaining 35%.
• Scaled Held-Out (EXP16): Validated on a rigorous 30 multi-domain unattacked facts benchmark, consolidating a robust truth margin of +4.19 ± 0.08 with 0.93 Precision and 0.80 Recall on the gate.
What's in the repo right now?
To avoid overwhelming the community, I'm releasing the project in phases. Today you'll find: