{"slug": "githubs-model-for-secret-detection-finding-more-secrets-with-less-noise", "title": "GitHub’s model for secret detection: Finding more secrets with less noise", "summary": "GitHub trained a context-aware ModernBERT classifier for its secret scanning that raised owner-weighted alert acceptance by 11.5% and cut median post-push scan execution time by 86% versus its previous LLM-based pipeline, the company reported. The two-stage system pairs high-recall regex candidate generation with a fine-tuned classifier that uses file path, secret type, variable names and nearby code to separate real credentials from lookalikes such as test fixtures and build hashes. GitHub said the compact model also lowered serving cost by 98% compared with remote LLM calls, making it practical to extend generic secret detection to push protection before code is committed.", "body_md": "GitHub’s model for secret detection: Finding more secrets with less noise\n\nGitHub’s context-aware secret classifier improves detection quality, raises weighted acceptance by 11.5%, and reduces median scan execution time by 86%.\n\nSmarter secret scanning, at a fraction of the cost\n\n11.5%\n\nhigher acceptance\n\nOwner-weighted alert precision\n\n86%\n\nlower median scan execution time\n\nPost-push scanning\n\n98%\n\nlower serving cost\n\nCompact model vs. remote LLM calls\n\nSecret scanning works best when developers are confident in the findings. Catch too little, and real credentials can remain in a repository. Alert too often, and developers may spend time investigating strings that aren’t secrets.\n\nGeneric secrets make that balance especially difficult. The same string could be a real credential or a documentation placeholder, so detecting it requires understanding the surrounding code and text.\n\nTo address this challenge, we trained a secret classifier specifically for GitHub’s product needs: distinguishing real credentials from lookalikes while meeting the speed and efficiency requirements of scanning at scale.\n\nIn our companion post, Secret protection must scale with software, we explained why secret protection must keep pace with AI-accelerated development. Here, we go behind the scenes of the context-aware classifier powering that effort: how we designed, trained, and evaluated it to find more real secrets with less noise.\n\nWhat we enabled\n\nGitHub secret scanning operates at two key moments:\n\nPush protection helps prevent secrets from entering a repository by scanning changes before a push is accepted, giving developers an opportunity to address the finding.\n\nPost-push scanning works after code has been committed, continuously scanning repository content and creating alerts when potential secrets are detected so developers and security teams can investigate and remediate them.\n\nWe developed a fast classifier that reduced median post-push scan execution time by 86% compared with the previous LLM-based pipeline. That efficiency made it practical to extend generic secret detection to push protection, returning results before developers push changes, not only after code is committed.\n\nWhy secrets are hard to detect\n\nStructured credentials often include a distinctive prefix, length, alphabet, or checksum. Generic secrets may not. Their syntax overlaps with build identifiers, checksums, asset hashes, sample values, and other non-sensitive strings. Context is the deciding signal.\n\nLikely real credential\n\nLikely harmless lookalike\n\nA password assigned beside a production database URL\n\nThe same password-shaped value in a test fixture\n\nA connection string in deployment configuration\n\nA localhost connection string in a setup guide\n\nA high-entropy value bound to an API credential\n\nA high-entropy checksum, build ID, or asset hash\n\nThese cases are hard negatives: strings that match a credential pattern but are not usable credentials. Adding more rules can suppress some of them, but each exception risks hiding a real secret. A learned classifier can instead use signals such as file path, secret type, variable names, and nearby code or text.\n\nThat is why we shipped a new scanner using a two-stage pipeline:\n\nCandidate generation: Regex filters scan content for potential secrets with high recall. This stage is optimized to catch broad families of credential-like patterns quickly and deterministically.\n\nContext-aware classification: A fine-tuned ModernBERT classifier evaluates each candidate and its surrounding context. It is trained to recognize the signals that separate real secrets from false positives, including placeholder values, examples, documentation snippets, generated content, and non-sensitive identifiers. Because the model is small and efficient to run on our inference infrastructure, it costs much less to serve and is simpler to scale than an LLM-based pipeline.\n\nImpact:\n\n11.5% user acceptance improvement, reducing false positives.\n\nMore than 98% cost savings compared with the LLM-based pipeline.\n\nHow our pipeline works\n\nAs content is scanned, regex patterns cast a wide net and flag anything that looks like a credential, so real secrets aren’t missed at the start. Each candidate then goes to a fine-tuned ModernBERT classifier, which reads the value together with its file path, variable names, and surrounding code. The classifier estimates how likely it is to be a real secret. High-confidence findings become alerts, and lookalikes such as placeholders, test fixtures, and checksums are filtered out before they reach developers.\n\nHOW OUR PIPELINE WORKS\n\nBroad first, careful second\n\nSTEP 1\n\nScan\n\nRepository content, commits, pull requests, and other scanned surfaces.\n\nSTEP 2\n\nGenerate candidates\n\nRegex patterns identify possible tokens, keys, and credential-like strings.\n\nSTEP 3\n\nClassify\n\nFine-tuned ModernBERT evaluates candidate + context and assigns likelihood of a true secret.\n\nSTEP 4\n\nAlert or suppress\n\nHigh-confidence findings move forward; likely false positives are filtered.\n\nThe important design choice is that the model does not replace deterministic scanning. Instead, it complements it. Regexes remain the right tool for finding candidate strings quickly. The classifier is used where machine learning provides the most leverage: distinguishing “looks like a secret” from “is likely a real secret.”\n\nA hard-negative problem: secrets that almost look real\n\nMost false positives are not random. They are hard negatives: strings that match a credential pattern but are not usable credentials. Examples include documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs, and generated fixtures. These cases are easy for a regex to match and hard for a purely rule-based system to suppress without accidentally losing real secrets.\n\nModernBERT helps because it can read the candidate together with nearby code and text providing contextual understanding comparable to an LLM without the latency and computational overhead of generating natural-language output. For example, a candidate surrounded by words like “example,” “dummy,” “fake,” or “replace-me” may be lower risk than a candidate assigned to an environment variable in a deployment file. The classifier learns these contextual differences from labeled examples rather than requiring engineers to encode every exception as another rule.\n\nWe chose ModernBERT because it provides a strong fit for this workload: it is compact, efficient, and well suited for classification over code-adjacent text. For secret scanning, the model needs to make a calibrated decision over short candidate windows with high throughput and low latency.\n\nThat makes a fine-tuned encoder classifier a better serving shape than a general-purpose LLM for this stage of the pipeline. The model can be trained directly on our decision boundary, and evaluated using the same recall and precision metrics that matter to secret scanning quality.\n\nHow we trained\n\nThe challenge was finding representative data for cases in which syntax alone is misleading. The training and evaluation strategy combines three sources:\n\nProduct-derived examples from GitHub repositories, using feedback from the secret detection experience and additional human validation.\n\nHigh-recall regex candidates from GitHub repository data, including human-annotated examples and weak labels used to expand coverage.\n\nSynthetic examples targeted at sparse and difficult cases, such as documentation secrets, test fixtures, public keys, checksums, structured lookalikes, and near-miss pairs.\n\nSynthetic rows pass deterministic validation and model-based quality review before inclusion. Training emphasizes real secrets, realistic false positives, and hard negatives that frequently confuse regex-only approaches. Each model input includes the candidate value, file path, secret type, and a bounded amount of surrounding context. We select thresholds through offline evaluation and then validate behavior on production-like traffic before broader rollout.\n\nBefore splitting the data, we grouped rows that share a repository, candidate value, or example so near-duplicates could not appear in both training and evaluation.\n\nTraining focused on the cases that matter most for alert quality: real secrets, realistic false positives, and hard negatives that regexes frequently match. The input representation includes the candidate secret string, file path, type and a bounded amount of surrounding context. The output is a classification score used by the scanner to decide whether the candidate should proceed.\n\nWe optimized for the product tradeoff: preserve or improve recall while reducing false positives and serving cost. In practice, that means the classifier threshold is selected using offline evaluation, then validated against production-like traffic before rollout.\n\nThe most important training principle was to make the model see the confusing examples. We included candidates that are syntactically similar to real secrets but semantically different, so the model learns not only what a secret looks like, but also what a false positive looks like in context.\n\nThe trained model is exported to ONNX format with 8-bit quantization, enabling fast inference.\n\nEvaluation results\n\nOnline result\n\nChange vs. previous LLM pipeline\n\nWhat it measures\n\nRepository owner-weighted acceptance\n\n11.5% higher\n\nCustomer acceptance of detected findings\n\nMedian post-push scan execution time\n\n86% lower\n\nProduction scan execution time\n\nServing cost\n\nMore than 98% lower\n\nCost of the compact classifier compared with remote LLM calls\n\nCustomer A/B evaluation results for our fine-tuned model compared to the previous LLM pipeline.\n\nWe evaluated the new pipeline’s impact in production in two stages: first against the production baseline on the same public traffic and then through a customer A/B experiment.\n\nOffline evaluation: On post-push scanning, the new pipeline achieved 13% higher precision and 19% higher recall than the previous LLM pipeline when evaluated on the same public traffic.\n\nOnline evaluation: We then validated whether these quality improvements translated to customer outcomes through an A/B experiment against the previous LLM pipeline. Repository owner-weighted acceptance increased by 11.5%. We weight acceptance by repository owner to reduce the influence of customers who bulk-resolve large volumes of alerts at once, which can otherwise distort a simple average.\n\nTake this with you\n\nThe best security tool is the one developers trust enough to use. Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught. Lower cost means we can keep improving coverage without making scanning prohibitively expensive.\n\nFor developers and security teams, this should translate into fewer false positives, and faster feedback in the places where secrets are most likely to be introduced: commits, pull requests, and everyday development workflows.\n\nHybrid systems are powerful. Regex gives us speed, transparency, and broad candidate coverage. ModernBERT gives us learned context understanding and false-positive reduction. The combination outperforms either approach used alone for this product constraint.\n\nWe also learned that model size is not the same as model usefulness. A smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical.\n\nThis is one step toward making secret scanning more accurate, scalable, and easy to use. Looking ahead, we are focused on two areas:\n\nBuilding a continuous improvement flywheel. We will use production feedback loops to identify new false negatives and false positives, turning real-world signals into higher-quality training and evaluation data.\n\nExploring new model architectures. We are investigating approaches that can perform secret detection and extraction in a single step, reducing pipeline complexity while preserving accuracy and scale.\n\nAcknowledgments\n\nHuge thanks to the engineering, science and product teams across GitHub and Microsoft who helped build the training data, evaluation suite, serving stack, and rollout plan for this new secret scanner.\n\nAs a Senior Research Scientist in Microsoft CoreAI, I develop agentic model orchestration and routing techniques for GitHub Copilot. My research focuses on adaptive model selection, multi-model coordination, long-horizon agentic workflows, and developer security.\n\nAs a Partner Applied Science Manager in Microsoft CoreAI, I lead an applied research team that measurably boosts developer productivity with ML and AI. We evolve GitHub Copilot in Completion, Chat, Code Review, and agentic workflows by continuously refining models and applying in-context learning for deeper, context-aware assistance.\n\nWriting code got faster. The work around it didn’t. See how you can plan, write code, review changes, and investigate production issues with GitHub Copilot, and what work still needs your judgment.\n\nAs Cybersecurity Awareness Month continues, the GitHub Bug Bounty team spotlights @r31n, diving into their research approach, favorite bug classes, and advice for aspiring hackers.", "url": "https://wpnews.pro/news/githubs-model-for-secret-detection-finding-more-secrets-with-less-noise", "canonical_source": "https://github.blog/ai-and-ml/githubs-model-for-secret-detection-finding-more-secrets-with-less-noise/", "published_at": "2026-10-13 17:56:33+00:00", "updated_at": "2026-11-09 23:50:51.328154+00:00", "lang": "en", "topics": ["ai-safety", "machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["GitHub", "ModernBERT"], "also_reported_by": [], "takeaway": "GitHub's fine-tuned ModernBERT classifier over regex-generated candidates raised owner-weighted alert acceptance 11.5% and cut median post-push scan time 86% at 98% lower serving cost than remote LLM calls.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/githubs-model-for-secret-detection-finding-more-secrets-with-less-noise", "markdown": "https://wpnews.pro/news/githubs-model-for-secret-detection-finding-more-secrets-with-less-noise.md", "text": "https://wpnews.pro/news/githubs-model-for-secret-detection-finding-more-secrets-with-less-noise.txt", "jsonld": "https://wpnews.pro/news/githubs-model-for-secret-detection-finding-more-secrets-with-less-noise.jsonld"}}