Lessons from GitHub’s Secret Scanning

Luisa Crawford
Aug 26, 2026 22:14

GitHub shares best practices for evaluating LLMs before production, focusing on precision, recall, and real-world workflows to improve security tools.

Evaluating large language models (LLMs) before deploying them to production is a complex challenge. In a detailed post on August 25, 2026, GitHub’s Mariko Wakabayashi and Zixiao Chen shared insights from their work on using LLMs to improve secret scanning—a feature that identifies sensitive credentials in code repositories. The lessons extend broadly to AI systems in security, developer tools, and other production workflows.

One critical takeaway? Benchmark scores alone are insufficient. While curated datasets and standardized benchmarks can help during prototyping, they often fail to capture the messy, ambiguous reality of production environments. GitHub’s team emphasized the need to test LLM performance with production-like data, accounting for edge cases, inconsistent inputs, and real-world constraints like latency and cost.

Shifting from Benchmarks to Real-World Metrics

Wakabayashi and Chen outlined a structured framework for evaluation, prioritizing precision, recall, and operational feasibility. For their secret-scanning use case, the goal was to reduce false-positive alerts without jeopardizing recall—a critical constraint in security workflows where missed credentials could pose significant risks. The team defined three core evaluation criteria:

• Primary outcome: Metrics like false-positive reduction and precision to gauge the user benefit.
• Safety constraint: Ensuring recall stays within acceptable limits to avoid security gaps.
• Operational guardrails: Practical considerations such as latency, cost, reliability, and production compatibility.

For instance, an experiment that improved precision but fell below recall guardrails was discarded. Similarly, changes that introduced significant latency or made the system too expensive to run in production were deemed failures, even if they improved model quality.

Evaluation as an Iterative Process

GitHub treated offline evaluation as a continuous process, akin to integration testing, rather than a one-time activity. Each iteration isolated a single variable—such as the model version or prompt—and compared results against a known baseline. This disciplined approach prevented attribution errors and ensured that improvements were genuine rather than coincidental.

Documenting every change, including prompt versions, datasets, and configurations, was also key. This allowed the team to track progress, identify regressions, and maintain reproducibility. For example, they noted that seemingly minor prompt tweaks could significantly alter model behavior, underscoring the importance of controlled, incremental experimentation.

Production-Like Testing is Essential

Another highlight was the emphasis on keeping offline evaluations as close to production workflows as possible. In GitHub’s case, this meant simulating real-world secret-scanning scenarios where models had to assess candidate strings within the context of surrounding code. Simplified evaluation sets, stripped of ambiguity or distractions, would have failed to reveal critical failure modes that could emerge in production.

This aligns with broader industry trends. The National Institute of Standards and Technology (NIST) recently published its TEVV-Athlon Framework draft, emphasizing lifecycle-based evaluation for AI systems, including testing under realistic conditions. Similarly, OpenAI has stressed the importance of independent testing and deployment simulation to identify risks before rollout.

Key Practices for LLM Evaluation

To guide other teams, GitHub shared actionable practices, including:

• Regularly testing model upgrades, as newer models can often simplify prompts and reduce complexity.
• Using synthetic and open datasets to test rare or edge cases, while ensuring these supplement rather than replace real-world data.
• Conducting detailed error analysis to identify recurring failure patterns and address root causes, such as ambiguous inputs or poor prompt framing.
• Deploying LLMs as judges to triage examples for human review, focusing manual efforts on ambiguous or high-stakes cases.

Broader Implications

GitHub’s structured, methodical approach reflects the growing consensus that LLM evaluation must evolve beyond static benchmarks. Tools like OWASP’s LLM Security Verification Standard and NIST’s TEVV-Athlon Framework stress domain-specific testing, adversarial cases, and operational guardrails as critical components of modern AI evaluation.

For developers and organizations, the takeaway is clear: robust evaluation isn’t just about proving a model works—it’s about ensuring it works safely, reliably, and cost-effectively in the real world. As LLM adoption expands, these practices will become indispensable for mitigating risks and maximizing value.

Image source: Shutterstock