![Heather Adkins and Four Flynn presenting talk - Evaluating Threats & Automating Defense at Google at [un]prompted 2026](https://thecyberarchive.com/assets/teasers/ai-vulnerability-discovery-google-big-sleep-code-mender-heather-four.webp)
Imagine a world where an open-source AI agent can autonomously hack into any system within a week — Google says we are not far off. With agentic AI-powered vulnerability discovery already finding deep memory safety bugs that traditional fuzzers miss, and a 35% year-over-year surge in reported CVEs overwhelming the NVD’s 30,000-entry backlog, the vulnerability apocalypse is no longer theoretical.
For security engineers, this talk from Google’s Heather Adkins and DeepMind’s Four Flynn is a blueprint for what comes next. This post breaks down the architecture behind Big Sleep and Code Mender — two research projects designed to find every software vulnerability on Earth and fix them autonomously, with zero false positives and developer-friendly patches.
Key Takeaways
- You'll learn how Google's Big Sleep project uses agentic AI reasoning loops to discover deep memory safety vulnerabilities with zero false positives, far surpassing what traditional fuzzing and manual research can achieve.
- You'll be able to identify the three critical verification stages — fuzzing, formal verification, and differential testing — that Code Mender uses to autonomously generate trusted patches without breaking existing functionality.
- Apply this to anticipate the coming "vulnerability apocalypse" where AI frameworks will find every bug in every system, and understand why your organization must prepare to patch at the speed of AI discovery.
The Vulnerability Apocalypse and Why AI Changes Everything
The Scale Problem Nobody Is Ready For
AI-powered vulnerability discovery is about to break every assumption security teams hold about vulnerability management. Heather Adkins opened her [un]prompted 2026 talk with a blunt assessment: “We are not far off from a world where there will be an open-source full hacking tool where you can type in ‘go hack Google’ and a week later it’ll come back with a root prompt.” That is not speculation from the fringes. That is a VP of Security Engineering at Google telling a room full of practitioners to prepare for a fundamentally different threat landscape.
The evidence is already visible. Between 2024 and 2025, logged vulnerabilities receiving a CVE designation increased by 35% — and that figure undercounts the actual discovery rate, since not every bug found earns a CVE. As of the morning of the talk, the National Vulnerability Database (NVD) carried a 30,000-entry backlog of unanalyzed vulnerabilities. The system designed to catalog and prioritize the world’s software flaws is already overwhelmed, and the flood has barely started.
Why Agentic AI Frameworks Change the Math
What makes this moment different from previous surges in vulnerability disclosure is the convergence of agentic security frameworks with massive capital investment. Adkins cited roughly $1 billion in venture capital flowing into startups focused on vulnerability discovery, penetration testing, and red team simulation. Open-source frameworks for automated pentesting and web application vulnerability scanning already exist. Layer generative AI reasoning on top of those frameworks, and the output scales in ways manual processes never could.
The implication Adkins spelled out directly: “Between all the good guys and all the bad guys, I think it’s fair to say we are at a point where we’ll be able to find every vulnerability in every system with agentic frameworks.” That is not a distant forecast. The tooling exists today in early forms, and each new model generation — arriving roughly every three to four months — brings measurable improvements in code reasoning capability.
Current Vulnerability Management Cannot Survive This
When every system is being probed by AI agents that never sleep, never lose focus, and improve quarterly, the traditional vulnerability management cycle collapses. Adkins argued that even the CVSS scoring system will lose its meaning in this environment. Severity rankings built for a world of scarce, manually discovered bugs do not translate to a world where discovery is cheap, continuous, and comprehensive.
The strategic reality for security engineers is stark. Patching cadences measured in weeks or months become indefensible when AI-driven discovery compresses the window between “unknown flaw” and “documented exploit path” to hours. Organizations that cannot ingest, triage, and remediate at machine speed will accumulate risk faster than they can address it.
This is the problem that drove Google and DeepMind to build Big Sleep and Code Mender — not as academic exercises, but as operational necessities. Adkins framed the mission without hedging: “We simply must eliminate every software vulnerability on Earth.” The sections that follow break down exactly how those two systems attempt to deliver on that ambition.
Actionable Takeaways
- Audit your organization's current vulnerability ingestion and triage pipeline against the assumption that AI-driven discovery will multiply the volume of reported vulnerabilities by an order of magnitude within the next 12-18 months. Identify bottlenecks that depend on manual analysis.
- Re-evaluate your reliance on CVSS scores as a primary prioritization mechanism. Begin developing context-aware prioritization that accounts for exploitability evidence and asset criticality, since raw severity scores will become less meaningful as discovery volume explodes.
- Track the emergence of open-source agentic pentesting frameworks and factor them into your threat model — these tools lower the barrier for both legitimate researchers and adversaries to discover vulnerabilities in your infrastructure.
Common Pitfalls
- Assuming that current patch management cadences will remain adequate as AI-driven vulnerability discovery scales. The 35% year-over-year increase in CVEs is an early signal, not a plateau.
- Treating the NVD backlog as someone else's problem. When the authoritative vulnerability database cannot keep pace with disclosure volume, organizations that depend on NVD analysis for prioritization are flying blind on a growing share of their attack surface.
Big Sleep Architecture for AI-Powered Vulnerability Discovery
The architecture behind Big Sleep is not another wrapper around an LLM prompt that says “find the bug.” It is a deliberate reconstruction of how elite vulnerability researchers — the kind who spend a month tracking a single deep memory safety flaw — actually think and work. Google’s Project Zero team, which has hunted zero-day vulnerabilities for over a decade, provided the behavioral blueprint. Big Sleep’s AI-powered vulnerability discovery pipeline translates that blueprint into three distinct phases, each building on the last, running without human intervention.
Phase 1: Deep Codebase Understanding
Every skilled vulnerability researcher starts the same way — they become an expert in the target codebase before they ever try to break it. As Heather Adkins described, the best researchers “become deeply expert in the codebase they’re working in.” They study the architecture, review historical vulnerabilities, perform variant analysis on past bugs, and develop an intuition for where weaknesses cluster.
Big Sleep’s first phase replicates this. The system ingests the target codebase and builds a contextual understanding of its structure, its history of past vulnerabilities, and the patterns that tend to produce exploitable flaws. This is the foundation that separates Big Sleep from naive approaches — rather than scanning code line-by-line for known patterns, it develops the kind of situated knowledge that a niche researcher accumulates over weeks or months. The agentic AI framework treats this understanding as persistent context that informs every subsequent decision.
For security engineers evaluating similar architectures, this phase is where most tools fall short. Static analysis and even basic LLM prompting skip the “understand the codebase” step entirely. Big Sleep’s approach means the system knows which code paths handle untrusted input, which components have been patched before, and where architectural decisions create implicit trust boundaries that attackers can violate.
Phase 2: Hypothesis-Driven Reasoning Loops
The second phase is where Big Sleep diverges most sharply from traditional automated tools. Instead of pattern matching or fuzzing blindly, the system enters a continuous agentic reasoning loop — generating hypotheses about potential vulnerabilities, then actively testing those hypotheses using real tools.
The system might identify a section of code that resembles a previously vulnerable pattern and form a hypothesis: “This buffer handling in the parsing module may be susceptible to an out-of-bounds write under specific input conditions.” It then tests that hypothesis — not by prompting itself again, but by writing actual code. The agent has access to three critical tools:
- A debugger — to step through execution paths and observe memory states
- A code browser — to navigate the codebase, trace function calls, and understand data flow
- A Python interpreter — to write custom scripts, generate test inputs, and automate hypothesis validation
This tool access is what makes Big Sleep an agentic security framework rather than a glorified code reviewer. The system builds test cases, executes them, observes the results, and feeds those results back into the next iteration of its reasoning loop. If a hypothesis fails, the agent revises it — examining why the crash did not occur, what constraints it missed, what adjacent code paths might be vulnerable instead. Adkins compared this to an expert researcher “standing in the corner staring with their coffee” — cycling through possibilities, unable to let go of a lead until it is either confirmed or definitively ruled out.
This loop runs continuously. Each iteration refines the system’s understanding of the target, narrows the search space, and builds toward higher-confidence vulnerability candidates. The key architectural insight is that the LLM is not doing the verification itself — it is orchestrating deterministic tools that provide ground-truth feedback, then reasoning about that feedback to guide the next step.
Phase 3: Exploit-Verified Reporting with Zero False Positives
Once the reasoning loop produces a successful crash that the agent believes represents a genuine vulnerability, Big Sleep moves into its verification and reporting phase. This is where the system achieves what Adkins called its most significant result: zero false positives on deep memory safety bugs, fully automated, end-to-end, without human involvement.
The verification process works by examining feedback data from the debugger and then constructing an actual proof-of-vulnerability exploit. The system does not simply flag suspicious code and hand it to a developer with a “this might be a problem” note. It proves the vulnerability is real by demonstrating exploitation. Every bug that exits the pipeline is provably a vulnerability — not a candidate, not a heuristic match, but a confirmed flaw with working exploit code.
Gemini[1] generates the final vulnerability report, and the quality bar is deliberately set high. Each report includes:
- A step-by-step walkthrough of every function involved in the vulnerability
- A detailed explanation of the bug’s root cause and exploitation path
- Proof-of-vulnerability code that the developer can run to reproduce the issue
- Enough context that a non-expert developer can understand the flaw without external research
This reporting standard reflects one of Big Sleep’s three design principles: uncompromising quality. The team explicitly rejected the “AI slop” approach — flooding developers with low-confidence findings that waste their time. Everything that reaches a developer’s hands must work out of the box, especially for less experienced engineers who cannot independently assess whether a finding is real.
The results are public. Google publishes Big Sleep’s findings through the Project Zero issue tracker[2], where anyone can review the report quality. As of the presentation, all but five of the discovered vulnerabilities had already been fixed — a testament to both the quality of the reports and the actionability of the exploit proofs.
Why Big Sleep Finds What Fuzzers Miss
A critical benchmark mentioned in the talk is that Big Sleep discovers vulnerabilities that OSS-Fuzz[3] — Google’s large-scale continuous fuzzing platform for open source — misses entirely. This is significant because OSS-Fuzz represents the state of the art in automated vulnerability discovery through fuzzing. The fact that Big Sleep’s hypothesis-driven approach surfaces bugs that brute-force input mutation cannot reach confirms that the system is operating at the depth the team intended.
The distinction matters for security engineers making tooling decisions. Fuzzing excels at finding shallow input-handling bugs through sheer volume of test cases. Big Sleep excels at finding deep memory safety bugs that require understanding code architecture, reasoning about multi-step execution paths, and recognizing subtle interactions between components — the same class of bugs that previously required a human expert and roughly 22 days of dedicated work per the 2017 RAND study on vulnerability research timelines.
Design Principles Driving the Architecture
Three principles shaped every architectural decision in Big Sleep, and they are worth internalizing for anyone building or evaluating automated vulnerability detection systems:
- Uncompromising quality — Zero tolerance for false positives or low-quality reports. Every finding must be exploit-verified and developer-ready.
- Cost-effective at scale — The system must handle billions of lines of code and run continuously. Efficiency is not optional when the target is “every vulnerability on Earth.”
- Real-time integration — Beyond scanning brownfield (legacy) code, the architecture is designed to integrate into the developer journey so new code is validated before it ships.
These principles explain why Big Sleep’s architecture is fundamentally agentic rather than batch-oriented. A batch scanner can find shallow bugs cheaply. An agentic system that reasons, hypothesizes, tests, and iterates can find the deep bugs — and prove they are real — at a cost structure that scales to production codebases.
Big Sleep Discovering Deep Memory Safety Bugs in Open Source Projects
Proof of Concept
-
Codebase Understanding Phase: Big Sleep begins by developing deep familiarity with the target open-source codebase, mirroring how elite vulnerability researchers operate. The system ingests the code architecture, reviews past vulnerabilities and CVE history, and performs variant analysis to identify patterns where similar bugs may exist. This replicates the behavior described in a 2017 RAND study showing expert researchers spend approximately one month understanding code before finding a deeply embedded bug.
-
Hypothesis Generation and Testing Loop: The agentic reasoning system enters a continuous feedback loop where it builds hypotheses about potentially vulnerable code paths — for example, identifying a section of code that handles untrusted input without proper bounds checking. To test each hypothesis, Big Sleep uses three critical tools: a debugger for runtime analysis, a code browser for navigating the codebase, and a Python interpreter for writing custom test scripts. The system writes its own test cases, executes them, evaluates results, and refines its hypotheses iteratively.
-
Crash Discovery and Validation: When the agentic loop produces a successful crash, the system transitions to verification. Rather than flagging potential issues and leaving triage to humans, Big Sleep analyzes the debugger feedback data from the crash to determine whether it represents a genuine exploitable vulnerability, not just a benign fault.
-
Exploit Construction as Proof of Vulnerability: The system then builds an actual exploit — a proof of vulnerability — that demonstrates the bug is real and exploitable. This exploit construction step is the key mechanism that drives the false positive rate to zero. Every bug that exits the pipeline is provably a vulnerability, not a theoretical concern.
-
Automated Report Generation via Gemini: Once the vulnerability is verified with a working exploit, Gemini[1] generates a detailed bug report. The report includes step-by-step walkthroughs of every function involved, a clear explanation of the bug’s root cause, and the proof-of-vulnerability code so developers can reproduce the issue.
-
Public Disclosure and Results: The discovered vulnerabilities are published on the Project Zero issue tracker[2], where the community can review the report quality. As of the presentation, all but five of the reported bugs had been fixed by upstream maintainers. The bugs found are deep memory safety vulnerabilities — not shallow issues like cross-site scripting or simple integer overflows — representing the class of bugs that traditionally require elite human researchers weeks or months to discover.
-
Surpassing Traditional Fuzzing: Big Sleep operates on open-source projects that are also covered by OSS-Fuzz[3]. Critically, Big Sleep is finding vulnerabilities that the fuzzers miss entirely, demonstrating that the agentic reasoning approach reaches depths in the code that coverage-guided fuzzing cannot. Targets include infrastructure components like the V8 JavaScript engine in Chrome and media processing libraries like ffmpeg that routinely handle untrusted input.
Big Sleep Finding Vulnerabilities That OSS-Fuzz Missed
Proof of Concept
-
Baseline fuzzing with OSS-Fuzz: Google runs OSS-Fuzz[3], a continuous fuzzing infrastructure that tests open-source projects around the clock. OSS-Fuzz uses coverage-guided mutation to explore code paths and trigger crashes, representing the state of the art in automated vulnerability discovery through fuzzing.
-
Big Sleep targets the same codebases: The Big Sleep team runs its agentic reasoning pipeline against the same open-source projects already under continuous OSS-Fuzz coverage. This creates a direct comparison — any vulnerabilities Big Sleep finds in these projects are bugs that sustained, large-scale fuzzing campaigns have already missed.
-
Agentic reasoning versus random mutation: Where OSS-Fuzz relies on mutating inputs and observing crashes, Big Sleep takes a fundamentally different approach. It builds deep understanding of the target codebase architecture, studies past vulnerability patterns and performs variant analysis, then generates specific hypotheses about where vulnerabilities might exist — mirroring the workflow of elite Project Zero researchers rather than relying on random input generation.
-
Hypothesis testing with real tools: Big Sleep validates each hypothesis using a debugger, code browser, and Python interpreter in a continuous reasoning loop. It writes its own test scripts, executes them against the target, evaluates the results, and refines its approach — iterating until it either confirms or discards each hypothesis.
-
Discovery of deep memory safety bugs: The vulnerabilities Big Sleep finds in OSS-Fuzz-covered projects are specifically deep memory safety bugs — not shallow integer overflows or surface-level input validation issues. These are bugs embedded deep in the codebase that require understanding code architecture and reasoning about complex state interactions.
-
Zero false positive verification: Each candidate vulnerability goes through an exploit-verification phase where Big Sleep builds a proof-of-vulnerability demonstrating the bug is real and exploitable. The system achieves zero false positives end-to-end without human involvement.
-
Confirmation from Google’s own assessment: Heather Adkins explicitly noted during the talk that “we’re still doing a lot of fuzzing. We develop OSS-Fuzz at Google… but Big Sleep is finding things the fuzzers are missing.” She framed this as validation that Big Sleep is operating at the right depth in the codebase — targeting vulnerability classes that are structurally invisible to coverage-guided fuzzing approaches.
Actionable Takeaways
- Evaluate your vulnerability discovery tooling against the three-phase model — codebase understanding, hypothesis-driven testing, and exploit verification. If your tools skip the first phase (contextual understanding of the target), they are structurally limited to shallow findings regardless of how sophisticated the detection logic is.
- Prioritize zero-false-positive tooling over high-volume scanners for deep memory safety bugs. Developer trust erodes rapidly when automated tools produce findings that waste engineering time, and that erosion makes teams ignore real findings. Big Sleep's architecture demonstrates that exploit-verified reporting is achievable at scale.
- Use Big Sleep's public findings on the Project Zero issue tracker as a benchmark for report quality when evaluating or building automated vulnerability reporting. The step-by-step function walkthroughs and proof-of-vulnerability code represent the standard that makes automated findings actionable for developers of all experience levels.
Common Pitfalls
- Treating LLM-based vulnerability scanning as a replacement for agentic reasoning. Prompting an LLM to "find bugs in this code" produces shallow results — the model lacks the iterative hypothesis testing and tool-assisted verification that Big Sleep uses to reach deep vulnerabilities. Without the reasoning loop and deterministic tool feedback, false positive rates remain high and finding depth remains low.
- Optimizing for volume of findings over quality and verifiability. The Big Sleep team explicitly chose zero false positives over maximum throughput, recognizing that developer trust in the system is the bottleneck for real-world impact. Flooding developers with unverified findings creates alert fatigue and undermines adoption of automated security tooling.
Automated Vulnerability Patching with Code Mender
Finding vulnerabilities at scale is only half the problem. The part that security engineers and developers dread — actually writing the fix — is where Code Mender enters the picture. Google launched Code Mender as a direct response to what the speakers call the vulnerability apocalypse: the recognition that AI-powered vulnerability discovery will soon outpace any human team’s capacity to write patches. If agentic frameworks can find every bug in every system, the automated patching pipeline has to keep up or the entire exercise produces nothing but an ever-growing backlog.
The Three Requirements Every Patch Must Satisfy
Code Mender’s design starts from a deceptively simple observation: a useful patch is not just code that silences a bug report. Four Flynn laid out three properties that every automated vulnerability patch must satisfy before it can be trusted:
-
Fix the security vulnerability. The patch must eliminate the specific flaw — whether that is a buffer overflow, a use-after-free, or an unprotected pointer dereference. A patch that merely changes the crash behavior without closing the attack surface is worthless.
-
Preserve existing functionality. A fix that breaks the application is not a fix. Code Mender must prove that the patched code still behaves identically to the original on all non-malicious inputs. This is the requirement that separates a quick hack from a production-grade repair.
-
Honor the developer’s idioms. This third requirement is the one most automated tools ignore entirely. The generated patch should match the coding style, naming conventions, and structural patterns already present in the codebase. The reasoning is practical: if a patch looks foreign to the developer who owns the code, they will reject it or rewrite it, adding delay and friction. Developer trust is the bottleneck for adoption, and Code Mender is explicitly engineered to clear it.
These three requirements are not aspirational guidelines — they are hard gates. A candidate patch that fails any one of them is discarded or recycled back into the generation loop.
From Verified Vulnerability to Candidate Patches
Code Mender is designed to operate as the second stage of an end-to-end discovery-and-fixing engine. It accepts a verified vulnerability directly from Big Sleep — complete with proof-of-vulnerability code, a detailed report, and debugger output — and uses that context to begin generating candidate patches.
The LLM agent powering Code Mender has access to the same class of tooling that Big Sleep uses: a debugger, a code browser, and a Python interpreter for writing and executing test scripts. This tooling is not decorative. The agent actively tests its own patch hypotheses, building test cases and evaluating whether its candidate fixes survive adversarial inputs. It generates multiple candidate patches per vulnerability, not just one, because the verification stages downstream are designed to rank and filter aggressively.
This is a critical architectural decision. Rather than asking the LLM to produce a single “correct” answer, the system embraces the reality that automated patching is a search problem. Generate many candidates, verify them rigorously, and surface only the ones that pass every check.
The Pluggable Verification Pipeline
The verification pipeline is where Code Mender’s real engineering depth lives. Flynn described it as a pluggable system — meaning new verification strategies can be added without restructuring the pipeline. The current stack includes at least four verification stages:
-
Pre-and-post-patch fuzzing. The system fuzzes the code before and after the patch is applied, comparing behavior across both runs. If the patched version produces different outputs on benign inputs, the patch has altered functionality and is rejected. This directly enforces requirement two — functional preservation.
-
Formal verification. Code Mender applies formal methods to the modified section of the codebase, attempting to mathematically prove that the patched code is functionally equivalent to the original (minus the vulnerability). This is the strongest guarantee available for behavioral preservation, though it is computationally expensive and not always tractable for large code regions.
-
Differential testing with malicious inputs. The system re-fuzzes the patched code specifically with the malicious inputs that triggered the original vulnerability. If the vulnerability is still reachable after patching, the candidate is rejected. This directly enforces requirement one — the patch must actually fix the security flaw.
-
LLM-based patch review. A large language model reviews the candidate patch using a carefully crafted prompt designed to evaluate correctness, style consistency, and completeness. This stage contributes to all three requirements but is particularly aimed at requirement three — idiom matching — since the LLM can assess whether the patch reads like code the original developer would have written.
The pluggable design means this list is not fixed. As new verification techniques become available — whether that is symbolic execution, property-based testing, or domain-specific analyzers — they can be slotted into the pipeline without rearchitecting the system.
The Feedback Loop: Learning from Verification Failures
Not every patch passes. In many cases, none of the initial candidate patches survive the verification pipeline. When that happens, Code Mender does not simply report failure. The learnings from the verification failures — which checks failed, what behavioral divergence was detected, what inputs caused the regression — are fed back into the LLM’s context window. The agent uses this feedback to generate a new round of candidate patches that are informed by the specific reasons the previous round failed.
This iterative refinement loop is what elevates Code Mender beyond a single-shot patch generator. It mirrors the way a skilled developer would respond to a failed code review: read the feedback, understand what went wrong, and produce a better version. The difference is that Code Mender can run this loop at machine speed, testing dozens of candidates per cycle without human intervention.
Only once a patch passes every verification stage is it considered a candidate for submission to the open-source community or to internal Google codebases.
Production Results and the Quality Trade-Off
As of the talk, Code Mender had produced 178 autonomously generated fixes in open-source projects. These span multiple libraries — libwp was called out as one example — and include work on Chrome’s pointer hardening, where automatically generated patches proactively protect buffer pointers before a vulnerability can be triggered.
Flynn acknowledged a deliberate trade-off in Code Mender’s design philosophy: the team is prioritizing verification rigor over speed to market. Every patch that reaches the community has passed the full verification pipeline, which means some valid fixes may take longer to ship. The rationale is that developer trust is fragile. If early automated patches introduce regressions or break builds, adoption will stall across the industry. By shipping only highly verified patches, Google is betting that trust built early will compound into faster adoption later.
Code Mender Generating 178 Autonomous Fixes for Open Source Libraries
Proof of Concept
-
Vulnerability Intake from Big Sleep: Code Mender receives a verified vulnerability report from Big Sleep’s agentic discovery pipeline. The input includes proof-of-vulnerability code and a detailed technical report, giving the patching agent full context on the bug’s location, trigger conditions, and exploitation mechanics.
-
LLM Agent Generates Patch Candidates: The Code Mender agent operates with tool access similar to Big Sleep — a debugger, code browser, and Python interpreter. It reasons about the vulnerable code section and generates multiple candidate patches. Each candidate must satisfy three requirements: (a) fix the security vulnerability, (b) preserve existing functionality so nothing breaks, and (c) honor the coding idioms of the original developer so the patch is easy to review and merge.
-
Pluggable Verification Pipeline — Stage 1, Fuzzing: The system fuzzes the code both before and after applying each candidate patch. Pre-patch fuzzing establishes a behavioral baseline, while post-patch fuzzing validates that the fix has not destroyed existing functionality. This differential fuzzing approach catches regressions that unit tests alone would miss.
-
Pluggable Verification Pipeline — Stage 2, Formal Verification: Code Mender applies formal verification to the patched section of the codebase, attempting to mathematically prove that the modified code remains functionally equivalent to the original for all non-malicious inputs.
-
Pluggable Verification Pipeline — Stage 3, Differential Testing: A dedicated fuzzing round injects the original malicious inputs — the same inputs that triggered the vulnerability — into the patched codebase. This confirms the vulnerability is actually eliminated, not just masked or shifted to a different code path.
-
Pluggable Verification Pipeline — Stage 4, LLM-Based Review: A large language model reviews the patch using a carefully crafted prompt designed to evaluate correctness, completeness, and adherence to coding conventions. This acts as an automated code review layer that catches issues the deterministic verifiers might miss.
-
Feedback Loop on Validation Failure: When none of the candidate patches pass all verification stages, the system does not discard the attempt. Instead, the specific failure signals and learnings from each rejected patch are fed back into the LLM’s context window. The agent uses this feedback to generate an improved set of candidate patches, which then re-enter the verification pipeline.
-
Ranking and Submission: Only patches that clear every verifier stage are considered candidates for submission to the open-source community. Among passing candidates, the system ranks patches based on quality signals — favoring those that best match developer idioms and minimize code churn.
-
Production Results Across Open Source: Code Mender has produced 178 autonomously generated fixes across multiple open-source projects. Lib WP is cited as one target library. Separately, the system has been applied to Chrome to proactively harden pointers in the codebase through automatically generated patches, reducing the attack surface for memory safety vulnerabilities before they are discovered by external researchers.
Proactive Pointer Hardening in Chrome via Automated Patches
Proof of Concept
-
Identify the hardening target: Code Mender’s proactive hardening mode shifts from reactive patching (fixing known vulnerabilities) to offensive-defensive analysis. Rather than waiting for Big Sleep or a fuzzer to flag a specific bug, the system scans the Chrome codebase for pointer operations — buffer accesses, pointer arithmetic, and memory allocations — that lack adequate bounds checking or lifetime protections.
-
Distinguish hardening from patching: The speakers drew an explicit distinction between two Code Mender modes. The first is reactive patching, where a verified vulnerability from Big Sleep feeds into Code Mender for a targeted fix. The second is proactive code hardening, where Code Mender examines code segments for classes of weakness — specifically pointer exceptions — and generates larger patches that protect buffers and constrain pointer behavior before any exploit exists.
-
Generate candidate hardening patches: The LLM agent driving Code Mender uses the same tool access as the patching workflow — a debugger, code browser, and Python interpreter — to reason about pointer usage patterns in Chrome. It produces multiple candidate patches that add bounds checks, null-pointer guards, or lifetime annotations around identified pointer operations.
-
Run the pluggable verification pipeline: Each candidate hardening patch passes through the same multi-stage verification system used for vulnerability patches: fuzzing before and after the patch, formal verification of the modified code section, differential testing with adversarial inputs targeting the hardened pointer paths, and LLM-based review.
-
Validate at Chrome scale: The hardening patches that survive all verification stages are submitted as candidates for the Chrome codebase. The speakers confirmed that “a bunch of work in Chrome” had already been completed “to actually harden pointers in the codebase through automatically generated patches.”
-
Maintain developer trust through quality gates: A key design decision in the Chrome hardening effort is the same quality-first philosophy that governs all Code Mender output. Every hardening patch must honor the idioms and coding style of the original Chrome developers, so that the automatically generated changes are easy for human reviewers to digest and merge.
Actionable Takeaways
- Evaluate your patch management pipeline against the three-requirement framework (security fix, functional preservation, idiom matching) to identify where automated patching tools could integrate without disrupting developer workflows.
- Invest in pluggable verification infrastructure — fuzzing harnesses, differential testing frameworks, and formal verification tooling — so that when AI-generated patches arrive at scale, your organization has the verification capacity to validate them before deployment.
- Measure and optimize your patch deployment velocity now. AI-generated patches will compress the time from discovery to fix, but that gain is meaningless if organizational processes add weeks or months before patches reach production.
Common Pitfalls
- Shipping AI-generated patches without rigorous multi-stage verification. Even if the patch appears to fix the vulnerability, it may introduce functional regressions or behavioral changes that are only caught through differential testing and fuzzing — not code review alone.
- Optimizing for patch generation speed at the expense of developer trust. If early automated patches are unreliable, developers will distrust and manually re-review every AI-generated fix, negating the efficiency gains the system was designed to provide.
Verification Pipelines for Zero False Positive Security Tooling
The single thread connecting every component of Google’s AI-powered vulnerability discovery and patching pipeline is verification. Big Sleep and Code Mender both stake their credibility on a zero false positive rate, and that guarantee does not come from the LLM alone — it comes from layered, deterministic verification stages that sit between raw AI output and anything a developer actually sees. For security engineers building or evaluating automated security tooling, understanding how these verification pipelines work is more valuable than understanding the AI reasoning itself.
Why Verification Is the Hard Problem
Heather Adkins framed the quality bar early in the talk: “We do not want to produce AI slop with the bugs we find. We want everything that lands in a developer’s hands, especially an inexperienced developer, to just work out of the box.” That standard — uncompromising quality at scale — is what forces a multi-stage verification architecture rather than a single confidence threshold. A single LLM judgment call, no matter how capable the model, cannot reliably distinguish a true deep memory safety bug from a plausible-looking false positive. The solution is to never rely on one verification method alone.
On the vulnerability discovery side, Big Sleep achieves zero false positive vulnerability detection by requiring the system to build an actual exploit — a proof of vulnerability — before any bug is reported. The agentic reasoning loop generates a hypothesis, tests it using a debugger and Python interpreter, and only when it produces a successful crash does the pipeline advance to verification. The verification phase then analyzes the debugger feedback and constructs exploit code that proves the vulnerability is real, not just a theoretical code path. This is fundamentally different from static analysis tools that flag potential issues based on pattern matching. Every bug that exits Big Sleep is provably exploitable.
The Four-Layer Verification Framework in Code Mender
Code Mender’s patch verification is where the architecture becomes a reusable pattern. Four Flynn described it as a “pluggable system” — meaning the verification stages are modular and can be composed depending on the codebase and vulnerability type. The four core verification layers are:
1. Pre/Post-Patch Fuzzing for Functional Preservation
The system fuzzes the code both before and after applying a candidate patch. This differential fuzzing validates that the patch has not destroyed existing functionality. If the post-patch fuzzing reveals behavioral changes beyond the targeted vulnerability fix, the patch is rejected.
2. Formal Verification of Functional Equivalence
Beyond fuzzing, Code Mender applies formal verification to the patched section of the codebase to mathematically prove that the modified code remains functionally equivalent to the original — minus the vulnerability. This is a stronger guarantee than testing alone, because formal methods can prove properties hold for all possible inputs rather than just the inputs covered by test cases.
3. Differential Testing with Malicious Inputs
A separate round of fuzzing specifically injects the malicious inputs that triggered the original vulnerability into the patched code. This confirms the vulnerability is actually eliminated, not just masked. Where the first fuzzing stage checks that nothing broke, this stage checks that the specific security issue is resolved.
4. LLM-Based Patch Review
Finally, the system uses an LLM with a “very carefully crafted pre-prompt” to review the patch for correctness. This is explicitly described as complementary to the deterministic methods — it catches semantic issues that fuzzing and formal verification might miss, such as whether the patch honors the coding idioms of the original developer.
The Feedback Loop on Verification Failure
A critical architectural detail is what happens when patches fail verification. Rather than discarding the attempt entirely, the learnings from the validation failures are fed back into the LLM’s context window. The model uses these failure signals to generate a new set of candidate patches that address the specific reasons the previous candidates were rejected. This creates an iterative refinement loop where each failed verification round makes the next generation of patches more likely to succeed.
In some cases, none of the candidate patches pass all verification stages. The system does not force a bad patch through — it loops until it either produces a verified fix or exhausts its attempts. Flynn acknowledged: “We are maybe being too cautious in making sure that we have highly verified outputs for the community before we send something out.” The team’s position is that developer trust in the system’s output is what drives adoption, and a single false positive or broken patch could undermine that trust permanently.
Applying Verification Patterns to Your Own Tooling
The verification pipeline is not specific to Google’s scale. The pattern — layer multiple independent verification methods, require all to pass, and feed failures back as learning signals — applies to any automated vulnerability patching or detection system. The key insight is that each verification method covers different failure modes: fuzzing catches behavioral regressions, formal verification proves equivalence properties, differential testing confirms the fix works, and LLM review catches semantic and stylistic issues. No single method is sufficient, but together they achieve a false positive rate that the team describes as zero across 178 autonomously generated open-source fixes.
Actionable Takeaways
- Design verification pipelines with multiple independent layers rather than relying on a single confidence score or LLM judgment. Combine fuzzing, formal verification, differential testing, and LLM review so each method covers failure modes the others miss.
- Implement feedback loops where verification failures are fed back into the generation process as learning signals, allowing iterative refinement of candidate outputs rather than simple pass/fail gating.
- Prioritize zero false positives over speed when building automated security tooling — developer trust in tool output is the bottleneck for adoption, and a single bad result can permanently damage credibility.
Common Pitfalls
- Relying solely on LLM-based review for patch or vulnerability verification without deterministic validation layers. The LLM review is explicitly described as complementary, not sufficient on its own.
- Shipping automated patches that pass functional tests but have not been verified against the specific malicious inputs that triggered the original vulnerability, leaving the security issue potentially unresolved.
Securing the Software Development Lifecycle with Proactive AI Hardening
Finding and patching vulnerabilities in production code is only half the equation. The broader vision behind Google’s AI-powered vulnerability discovery research is a fundamental shift in how security integrates with the software development lifecycle — moving from reactive patching to proactive AI hardening that prevents entire vulnerability classes before they reach production.
From Reactive Patching to Proactive Code Hardening
Four Flynn outlined a three-tier strategy that reframes how organizations should think about AI-driven security across the SDLC. The first tier addresses legacy brownfield code already running in production — this is where Big Sleep and Code Mender operate today, discovering vulnerabilities and generating verified patches. The second tier takes the same patching architecture and applies it proactively: rather than waiting for a vulnerability to be found, the system scans codebases for dangerous patterns like unprotected pointer operations and generates hardening patches that eliminate the conditions under which vulnerabilities could occur in the first place.
This proactive hardening approach has already been applied to Chrome, where automated patches harden pointers across the codebase — not in response to a specific CVE, but as a preventive measure that closes off entire attack surfaces. For security engineers, this represents a critical shift: instead of triaging individual vulnerabilities after discovery, AI-driven hardening can systematically eliminate categories of risk across an entire codebase.
Securing Greenfield and Vibe-Coded Applications
The third tier targets brand new greenfield code, including applications produced through emerging vibe coding frameworks where developers describe desired behavior and AI generates the implementation. Flynn stressed that this new code must be “secure out of the box” — not merely free of known vulnerabilities, but built on secure frameworks that make certain vulnerability classes structurally impossible.
This is a significant concern for security teams. As AI-assisted code generation accelerates development velocity, the volume of new code entering production will grow dramatically. If that code inherits insecure patterns or relies on unsafe libraries, the vulnerability backlog will compound faster than any patching system can address. The goal is to ensure that AI code generation tools produce outputs that use memory-safe constructs, validated input handling, and secure-by-default configurations from the start.
The Unsolved Problem of Patch Deployment Velocity
Even with fully automated discovery and patching, one critical bottleneck remains that no amount of AI sophistication has solved: getting organizations to actually apply patches in a timely manner. Flynn was candid about this gap, noting that “one of the hardest problems with patching is actually places in the world that struggle to actually apply patches in a timely manner.” He explicitly called out that he does not know how to solve this problem with AI and invited the community to collaborate on it.
This is a sobering reality check for any organization planning its automated vulnerability patching strategy. If AI frameworks can soon find every vulnerability in every system — and the 35% year-over-year increase in CVEs suggests that trajectory is already underway — then the constraint shifts entirely to deployment. Organizations that cannot patch at the speed of AI discovery will face an ever-widening exposure window. The vulnerability apocalypse is not just about the volume of bugs found; it is about the delta between discovery velocity and remediation velocity.
Integrating AI Security into Your SDLC Strategy
For security engineers building their SDLC security programs, the Google research points to a layered integration model:
-
Shift left with proactive hardening: Do not wait for vulnerabilities to be discovered. Use automated analysis to identify and harden dangerous code patterns — unprotected buffers, unsafe pointer arithmetic, missing bounds checks — across existing codebases before attackers find them.
-
Secure the code generation layer: As vibe coding and AI-assisted development become standard, ensure that code generation tools are configured to produce secure-by-default outputs using safe frameworks and validated libraries.
-
Automate the full pipeline: The end-to-end vision is discovery, patching, verification, and deployment operating as a continuous loop with minimal human intervention — but only when verification quality is high enough to maintain developer trust.
-
Address deployment velocity now: The AI side of the equation is advancing rapidly. The organizational and operational side — change management, testing pipelines, deployment automation, and stakeholder buy-in for rapid patching cycles — is where most teams will bottleneck. Start building that capacity before the volume forces your hand.
Actionable Takeaways
- Implement proactive code hardening scans on your existing codebases — identify and remediate dangerous patterns like unprotected pointers and unsafe buffer operations before they become exploitable vulnerabilities, rather than waiting for CVE-driven patching cycles.
- Evaluate your AI-assisted code generation tools and vibe coding workflows for secure-by-default output — ensure generated code uses memory-safe constructs, validated input handling, and secure frameworks so new code does not expand your vulnerability surface.
- Audit your organization's patch deployment velocity and build automation to close the gap between discovery and remediation — with AI-driven discovery accelerating, the deployment pipeline will become the primary constraint on your security posture.
Common Pitfalls
- Focusing exclusively on vulnerability discovery and patching while ignoring deployment velocity — even perfect automated patches are worthless if your organization cannot apply them quickly, and the gap between discovery speed and deployment speed is where attackers will operate.
- Treating AI-generated greenfield code as inherently secure because it was produced by a model — without explicit secure framework requirements and verification, vibe-coded applications can introduce the same vulnerability classes that plague legacy codebases.
Conclusion
Google’s Big Sleep and Code Mender represent one of the most ambitious attempts to fundamentally change how the industry handles software vulnerabilities. The core insight is not that AI can find bugs — that has been demonstrated repeatedly. It is that agentic AI systems, armed with real tools and iterative reasoning loops, can find the deep bugs that fuzzers and manual researchers miss, prove they are exploitable with zero false positives, and then generate production-quality patches that pass rigorous multi-stage verification.
The vulnerability apocalypse Adkins described is not a scare tactic. The 35% annual increase in CVEs, the NVD’s 30,000-entry backlog, and the $1 billion in VC funding flowing into AI-driven vulnerability discovery all point to a near-term reality where every system is probed continuously by agents that never sleep. Organizations that cannot patch at machine speed will fall behind.
For security engineers, the actionable lesson is not to wait for Google to solve this. The verification pipeline patterns — layered independent methods, feedback loops on failure, quality over speed — apply at any scale. Start building that infrastructure now, because the volume is coming whether you are ready or not.
Explore related topics on The Cyber Archive: defensive security strategies, vulnerability research techniques, and security automation frameworks.
References & Tools
- Gemini — Google's large language model powering Big Sleep's automated vulnerability report generation with step-by-step function walkthroughs and proof-of-vulnerability code. ↩
- Project Zero Issue Tracker — Public bug tracker where Big Sleep's discovered vulnerabilities are published with full reports for community review. ↩
- OSS-Fuzz — Google's large-scale continuous fuzzing platform for open-source software, used as a baseline comparison showing Big Sleep finds vulnerabilities fuzzers miss. ↩
Questions from the audience
Related deep dives
Tenderizing the Target | [un]prompted 2026
I SPy - Rethinking Entra ID research for new paths to Global Admin
Vibe Check: Security Failures in AI-Assisted IDEs | [un]prompted 2026
