Quantifying Prompt Drift: A Zero-Dependency CLI Tool for LLM Prompt Engineering

A developer released Agentic-Local-Prompt-Semantic-Diffuser, a zero-dependency Python CLI tool that statically quantifies semantic drift between two LLM prompts in under 10 seconds. The tool combines term-frequency vectorization with cosine similarity to score prompt similarity, plus structural static analysis that flags formatting risks such as a lost JSON hint, avoiding slow LLM-in-the-loop evaluations for minor prompt iterations.

Managing prompts for Large Language Models LLMs often feels like a dark art. A seemingly minor tweak to a system prompt can inadvertently cause unintended behavioral shifts or output formatting failures—a phenomenon known as semantic drift . To address this, we need a way to statically analyze and quantify the impact of prompt changes before deploying them, without relying on slow and expensive LLM-in-the-loop evaluations for every minor iteration. Below is the implementation of Agentic-Local-Prompt-Semantic-Diffuser , a lightweight, zero-dependency Python CLI tool designed to evaluate the semantic impact of prompt modifications in under 10 seconds. It utilizes term frequency vectorization and cosine similarity to measure semantic drift, alongside structural static analysis to detect formatting risks. Save the following script as prompt diffuser.py . - - coding: utf-8 - - """ Agentic-Local-Prompt-Semantic-Diffuser A one-shot CLI tool to automatically evaluate the semantic impact of prompt changes for local LLMs in under 10 seconds. Execution Example python prompt diffuser.py \ --old-prompt "You are a helpful assistant that outputs JSON." \ --new-prompt "You are a strict assistant that outputs strict JSON format." \ --test-inputs "Hello" "What is the weather?" """ import sys import json import argparse import math from typing import List, Dict, Any def tokenize text: str - List str : """Tokenizes the input text into a list of lowercase words.""" return text.lower .split def get vector text: str, vocabulary: List str - List float : """Generates a term frequency vector for the given text based on the vocabulary.""" tokens = tokenize text return float tokens.count word for word in vocabulary def cosine similarity v1: List float , v2: List float - float: """Calculates the cosine similarity between two vectors.""" dot product = sum a b for a, b in zip v1, v2 norm1 = math.sqrt sum a a for a in v1 norm2 = math.sqrt sum a a for a in v2 if norm1 == 0.0 or norm2 == 0.0: return 0.0 return dot product / norm1 norm2 def analyze structure text: str - Dict str, Any : """Performs static analysis on the prompt to extract structural metadata.""" return { "length": len text , "has json hint": "json" in text.lower , "has markdown": " " in text or " " in text, "line count": text.count "\n" + 1 } def main : parser = argparse.ArgumentParser description="Agentic-Local-Prompt-Semantic-Diffuser" parser.add argument "--old-prompt", required=False, help="Path to old prompt or prompt string" parser.add argument "--new-prompt", required=False, help="Path to new prompt or prompt string" parser.add argument "--test-inputs", required=False, nargs="+", help="Representative test inputs" args = parser.parse args Fallback to default sample inputs if no arguments are provided old p = args.old prompt if args.old prompt else "You are a helpful assistant that outputs JSON." new p = args.new prompt if args.new prompt else "You are a strict assistant that outputs strict JSON format." inputs = args.test inputs if args.test inputs else "Hello", "What is the weather?" Build a unified vocabulary space vocab = list set tokenize old p + tokenize new p for inp in inputs: vocab = list set vocab + tokenize inp Vectorization and global semantic drift calculation old vec = get vector old p, vocab new vec = get vector new p, vocab prompt similarity = cosine similarity old vec, new vec semantic drift = 1.0 - prompt similarity Structural risk evaluation old struct = analyze structure old p new struct = analyze structure new p structural risk = "LOW" Escalate risk if critical structural hints like JSON formatting are altered if old struct "has json hint" = new struct "has json hint" : structural risk = "HIGH" Moderate risk for significant changes in prompt verbosity elif abs old struct "length" - new struct "length" 200: structural risk = "MEDIUM" Estimate the impact on individual test inputs test evaluations = for inp in inputs: inp vec = get vector inp, vocab sim old = cosine similarity old vec, inp vec sim new = cosine similarity new vec, inp vec test evaluations.append { "input": inp, "old alignment": round sim old, 4 , "new alignment": round sim new, 4 , "drift delta": round sim new - sim old, 4 } report = { "status": "SUCCESS", "semantic drift score": round semantic drift, 4 , "structural risk": structural risk, "details": { "prompt similarity": round prompt similarity, 4 , "old structure": old struct, "new structure": new struct, "test evaluations": test evaluations } } print json.dumps report, ensure ascii=False, indent=2 if name == " main ": main If executed without any arguments, the script will run an evaluation using the default sample prompts and test inputs, acting as a quick sanity check. python prompt diffuser.py The tool outputs a structured JSON report. The semantic drift score provides a normalized delta between the two prompts, while the structural risk flag alerts you to potentially breaking changes such as accidentally removing a JSON formatting instruction . { "status": "SUCCESS", "semantic drift score": 0.4377, "structural risk": "LOW", "details": { "prompt similarity": 0.5623, "old structure": { "length": 54, "has json hint": true, "has markdown": false, "line count": 1 }, "new structure": { "length": 68, "has json hint": true, "has markdown": false, "line count": 1 }, "test evaluations": { "input": "Hello", "old alignment": 0.0, "new alignment": 0.0, "drift delta": 0.0 }, { "input": "What is the weather?", "old alignment": 0.0, "new alignment": 0.0, "drift delta": 0.0 } } } To evaluate your own prompt iterations against representative user queries, pass them via command-line arguments: python prompt diffuser.py \ --old-prompt "Summarize the text." \ --new-prompt "Provide a detailed bullet-point summary in Japanese." \ --test-inputs "Machine learning is a subset of artificial intelligence." Quantifying prompt drift statically provides a vital first line of defense before committing prompt changes to your application. While vector-based similarity cannot replace semantic evaluation by an LLM such as LLM-as-a-Judge paradigms , it excels as a rapid, zero-cost CI/CD gatekeeper. By integrating this lightweight script into your workflow, you can proactively detect unintended regressions, missing format instructions, or excessive scope creep in your prompts without incurring API overhead. If this engineering log saved your production server and your sanity , consider supporting our architecture on GitHub Sponsors.