![Joshua Saxe presenting talk - Joshua Saxe - The Hard Part Isn't Building the Agent: Measuring Effectiveness | [un]prompted 2026 at unprompted 2026](https://thecyberarchive.com/assets/teasers/evaluating-autonomous-ai-cyber-defense-agents-joshua-saxe.webp)
Autonomous AI cyber defense evaluation is quietly the hardest unsolved problem in security AI — not building the agent, but knowing whether it actually works. When a SOC analyst labels an alert and a colleague labels it differently 10–20% of the time, any evaluation built on those labels is measuring noise as much as capability. Add a mere 1–3% label flip rate and classical metrics like precision, recall, and F-score hit a ceiling that hides whether your system is improving at all.
For security engineers building or deploying autonomous agents — for alert triage, access management, code remediation, or cloud posture — this post unpacks why the oracle-based evaluation model borrowed from computer vision breaks down in security, and what a holistic, rubric-driven replacement looks like in practice.
Key Takeaways
- You'll learn why classical ML metrics like precision, recall, and F-score systematically fail for autonomous cyber defense evaluation — and how label noise as low as 1–3% creates a measurement ceiling that makes it impossible to distinguish a good system from a bad one.
- You'll be able to apply a holistic rubric-based evaluation framework that assesses agent reasoning quality, evidence gathering, and decision justification — not just binary outcome accuracy — enabling you to build a defensible deployment bar aligned with leadership.
- Apply the LLM-as-judge approach to automate rubric scoring at scale with as few as 100 calibration samples, giving your team a concrete hill-climbing target and enabling continuous production monitoring of autonomous agent behavior.
Why Classical ML Metrics Fail for Autonomous Cyber Defense Evaluation
The Oracle Assumption: Why Classical ML Metrics Break in Security
Autonomous AI cyber defense evaluation starts with a deceptively simple question: how do you know if the system actually works? The answer most teams reach for — precision, recall, F-score, ROC curves, area under the curve — carries a hidden assumption that is valid in computer vision but systematically false in cybersecurity. That assumption is the existence of a reliable oracle: a ground truth source whose labels are accurate, consistent, and trustworthy.
In classical machine learning, this assumption holds. If you are classifying images of cats and dogs, you can label a dataset with high confidence, compute summary statistics against those labels, and trust the resulting numbers. The label for “cat” is not in dispute.
Cybersecurity is a categorically different domain. As Joshua Saxe explains from his experience building autonomous security systems at Meta and at his current startup, the ground truth in security is fundamentally, structurally noisy — and no amount of effort will eliminate that noise.
Where Ground Truth Breaks Down
Label noise in SOC alert classification is one of the clearest illustrations. When you attempt to build a system that automates alert investigation and triage, you need human-labeled examples: which alerts were true positives worth investigating, and which were false positives. When you collect those labels from SOC analysts, you encounter a double-digit disagreement rate between analysts labeling the same alert. One analyst with a bad lunch decides an alert is not investigation-worthy; a more experienced colleague would have called it critical. These conflicting labels are not outliers — they are the norm.
The same problem appears across every autonomous cyber defense use case:
- Access management automation: When building systems to automatically grant or deny access to sensitive resources (such as a database table containing millions of user messages), access reviewers disagree at double-digit rates about who should have access. The ground truth for “correct access decision” is genuinely contested.
- Autonomous code remediation: When an AI system proposes a patch to a critical program, there is no clean label for “this patch is safe.” You run up against the halting problem and state space explosion — theoretical computer science problems that make it impossible to formally verify whether a patch will crash production or introduce a new vulnerability. The best heuristic available is “it passes the unit tests,” which is far from a reliable oracle label.
- Malware classification: Even the definition of malware is contested. Many binaries are genuinely dual-use — the same tool can be a legitimate penetration testing utility or a weapon depending on context. Labeling these as definitively malicious or benign involves ontological and epistemological judgment calls, not factual determinations.
The Measurement Ceiling Effect
Saxe ran a simulation to quantify exactly what noisy labels do to evaluation reliability, and the results are stark. Consider a hypothetical perfect AI system — one that correctly classifies every SOC alert. Now introduce a modest label flip rate in the evaluation dataset: just 0.5% to 3% of labels are randomly flipped from true positive to false positive or vice versa.
As the flip rate increases from 0.5% to 3%, the perfect system begins to look increasingly inaccurate. The evaluation metrics degrade not because the system got worse, but because the measurement instrument is corrupted. More critically, beyond a certain noise threshold, you hit a measurement ceiling — a point above which you literally cannot detect whether your system is improving. You could ship a significant model improvement and your precision/recall numbers would not move, because the label noise is drowning out the signal.
This is not a theoretical concern. In practice, it means that teams staring at accuracy, precision, recall, and F-score for their autonomous security systems are often not measuring system capability at all — they are measuring the noise floor of their labeling process.
Reducing 100,000 Tokens of Reasoning to a Single Unreliable Bit
There is a second dimension to why classical metrics fail for LLM agent evaluation for security, and it goes beyond label noise. Modern autonomous agents do not produce simple binary outputs. A security agent investigating a SOC alert may generate 100,000 tokens of chain-of-thought reasoning, tool calls, data retrieval, and structured justification before arriving at a decision. That decision represents a rich, complex reasoning process.
Classical evaluation frameworks reduce all of that complexity to a single bit of error information: right or wrong, 1 or 0, compared against a noisy label. Saxe draws an analogy that is hard to dismiss: imagine hiring security engineers based solely on multiple-choice tests, never interrogating their ability to reason under uncertainty. That is precisely what oracle-based evaluation does to autonomous agents.
The field inherited these metrics from computer vision and classical ML without interrogating whether the oracle assumption transfers to security. It does not. And until teams building AI/ML security systems recognize this foundational mismatch, their evaluation programs will produce summary statistics that nobody actually trusts — which is exactly the state of the field today.
Actionable Takeaways
- Before designing any evaluation program for an autonomous security agent, explicitly audit your ground truth labels for inter-annotator agreement. If SOC analysts or access reviewers disagree on a double-digit percentage of cases, treat your labels as noisy inputs, not ground truth — and build that uncertainty directly into how you interpret your metrics.
- When presenting evaluation results to leadership, accompany any precision/recall/F-score numbers with an explicit label noise estimate. If your disagreement rate is 10–15%, make clear that those summary statistics have a built-in noise ceiling that limits their interpretability — otherwise leadership will make deployment decisions on numbers that do not mean what they appear to mean.
- Recognize that autonomous security agents operating on ambiguous tasks (alert triage, access decisions, code remediation) are not operating in a binary classification setting with clean labels. Design your evaluation infrastructure from the start to accommodate noisy, contested ground truth — not as an edge case, but as the expected default.
Common Pitfalls
- Importing classical ML evaluation metrics (precision, recall, F-score, ROC curves) directly into security AI evaluation without questioning the oracle assumption. These metrics were designed for settings where ground truth labels are reliable. In security, they will produce numbers that look authoritative but are substantially measuring label noise rather than system capability — leading to false confidence or false dismissal of systems that are actually working well.
- Treating label disagreement between human reviewers as a data quality problem to be cleaned away rather than as a fundamental property of the security domain. Attempting to resolve disagreements by picking one reviewer's labels as canonical does not eliminate the noise — it just hides it. The underlying ambiguity is real, and evaluation frameworks must be designed to accommodate it rather than paper over it.
The Label Noise Ceiling — How Measurement Breaks Down in Security AI
The Noise Problem Is Not Fixable — It Is Structural
Label noise in security AI is not a data quality problem you can solve by hiring better analysts or labeling more carefully. It is a structural feature of the domain, and understanding that distinction is critical before you can reason clearly about what evaluation should look like.
Joshua Saxe ran a concrete simulation to demonstrate this effect for an alert triage setting. The setup: imagine a hypothetical perfect AI system — one that correctly classifies every SOC alert as true positive or false positive. Now introduce label noise. Instead of transparent, reliable ground truth, you have labels produced by analysts who disagree with each other at double-digit rates. In the simulation, Saxe varied the label flip rate — the percentage of labels where a true positive was recorded as a false positive or vice versa — from 0.5% to 3%.
The result is striking. Even at a 1% flip rate, a system that is objectively performing well begins to look inaccurate when measured against those noisy labels. As the flip rate climbs toward 3%, the measurement degrades to the point where you cannot distinguish improvement from noise. This is the label noise ceiling: a threshold above which your metrics stop tracking actual system quality and start tracking label unreliability.
Label Noise Simulation — Measuring the Evaluation Ceiling in SOC Alert Classification
Proof of Concept
-
Define the ideal system baseline. Model a hypothetical “perfect” AI classifier — one that correctly predicts whether a SOC alert is a true positive or false positive with 100% accuracy on the underlying ground truth.
-
Introduce label noise via flip rate. Rather than corrupting the classifier itself, corrupt the evaluation labels. A “flip rate” is defined as the probability that any given label in the evaluation dataset has been incorrectly assigned — a true positive label flipped to false positive, or vice versa. This directly mirrors the real-world condition where SOC analysts disagree on alert classification at double-digit rates.
-
Sweep the flip rate from 0.5% to 3%. Run the simulation across a range of label noise levels — starting at a 0.5% flip rate and increasing to 3%. At each level, compute standard classification metrics (precision, recall, F-score) for the “perfect” system against the noisy labels.
-
Observe metric degradation. As the flip rate increases from 0.5% to 3%, the perfect system’s measured performance drops substantially. The metrics no longer reflect the system’s true capability — they reflect the noise floor of the label set. The system begins to appear inaccurate even though it is performing perfectly on the actual ground truth.
-
Identify the noise ceiling. Beyond a certain flip rate threshold, the measured performance stabilizes at a low plateau. This is the noise ceiling — the point above which additional improvements to the classifier produce no detectable change in the evaluation metrics. If your system improves from 92% to 96% true accuracy but your label noise floor produces a measured score of, say, 78% regardless, you cannot measure the improvement at all.
-
Map to the real-world SOC context. The simulation’s flip rate directly corresponds to inter-analyst disagreement. Saxe’s practical experience building autonomous SOC automation systems confirmed that analysts disagree at double-digit rates on which alerts are worth investigating versus which are false positives. A 10–20% inter-analyst disagreement rate means the effective flip rate in a crowdsourced label set is far above the 1–3% range that already breaks standard metrics in the simulation.
-
Interpret the result. The takeaway is not that the classifier is broken — it is that the evaluation framework is broken. Classical metrics like precision, recall, and F-score imported from computer vision assume a reliable oracle producing clean labels. When applied to inherently ambiguous cybersecurity classification tasks, they measure label noise as much as classifier capability. This makes it statistically impossible to distinguish a good autonomous cyber defense system from a bad one using standard metrics alone.
Note: This example is rated partial because Saxe describes the simulation conceptually and presents its conclusions but does not provide the full parameter set, code, or statistical model in the talk.
Why This Happens Mathematically
The intuition is straightforward. Classical metrics like precision, recall, F-score, and ROC curves are computed by comparing system outputs against a reference label set. If the label set is noisy, the comparison is corrupted. A system that correctly identifies a true positive gets penalized when the label says “false positive.” A system that correctly dismisses a false positive gets rewarded when the label says “true positive” by accident.
At low noise rates, some of this averages out. But as noise compounds — and in security, 10–20% analyst disagreement rates mean the effective flip rate is far higher than 1% — the metrics become systematically misleading. The signal drowns in noise you cannot see, because you assumed the labels were clean.
Saxe’s framing is precise: “Your ability to measure whether or not your system actually works just plummets as you add noise.” This is not a gradual degradation — it is a ceiling effect. Beyond the noise threshold, more data does not help. Better models do not help. The evaluation framework itself has broken down.
The Double-Digit Disagreement Reality
Saxe grounded this in two domains he has worked in directly:
SOC alert triage: When you try to build a system that automates investigation of SOC alerts and attempt to label which alerts were worth investigating versus false positives, analyst disagreement rates run into the double digits. Two experienced analysts reviewing the same alert will reach different conclusions a meaningful fraction of the time. Those disagreements are not errors — they reflect genuine ambiguity in the underlying problem. Security context is incomplete. Attacker intent is opaque. The same network behavior can be benign or malicious depending on context that no analyst can fully reconstruct.
Access management: When access reviewers assess whether a contractor should have access to a sensitive database table containing user messages, they disagree at double-digit rates on who should actually have access. The “correct” answer is not objectively determinable from the available evidence alone.
In both cases, if you try to train or evaluate a machine learning system using those labels as if they were ground truth, you are building on a foundation that cannot support the weight of your conclusions.
What This Means Before You Can Fix It
Before adopting any replacement evaluation framework, security engineers need to internalize the ceiling effect as a constraint that shapes everything downstream:
- Do not treat a drop in precision or recall as definitive evidence of system degradation. If your label set has 10–20% analyst disagreement embedded in it, a 2% swing in precision is within measurement noise, not a signal.
- Do not hill-climb a single metric optimized against a noisy label set. You will overfit to labeling artifacts, not to actual system improvement.
- When presenting to leadership, explicitly quantify the noise floor. If you cannot measure differences below a 5% performance gap due to label noise, say so. This is not a weakness in your methodology — it is an honest accounting of what the data can and cannot tell you.
The noise ceiling is not a problem you are going to label your way out of. The next step is building an evaluation model that does not depend on a clean oracle in the first place.
Actionable Takeaways
- Before running any precision/recall analysis on an autonomous security system, estimate your label noise floor by having two or more analysts independently label a sample of your test set. If disagreement rates exceed 5–10%, your classical metrics are operating in or near the noise ceiling and cannot reliably distinguish system improvement from label variance. Report this uncertainty explicitly to stakeholders rather than presenting raw accuracy numbers as ground truth.
- When evaluating autonomous SOC alert triage or access management systems, do not hill-climb a single F-score or AUC metric optimized against a noisy label set. Label noise at even 1–3% flip rates creates a measurement ceiling that makes further optimization of those metrics misleading. Treat classical metrics as one noisy signal among several, not as the primary deployment gate.
- If your team is in disagreement about whether a performance drop reflects system regression or label noise, resolve it structurally: sample the disagreement cases, have multiple reviewers re-label them independently, and quantify inter-rater agreement (e.g., Cohen's kappa). This turns a subjective argument into a measurable one and establishes a realistic noise floor for all future evaluation cycles.
Common Pitfalls
- Assuming that analyst-produced labels are ground truth when computing precision, recall, and F-score for autonomous security systems. In domains like SOC alert triage and access management, double-digit analyst disagreement rates mean the label set is fundamentally noisy. Treating those labels as an oracle produces evaluation results that are partly measuring labeling inconsistency, not system quality — and leads to shipping decisions based on misleading statistics.
- Trying to solve label noise through more labeling effort or stricter labeling guidelines. Saxe is explicit that the uncertainty is structural — it stems from theoretical problems like the halting problem, dual-use binaries, and incomplete security context that no amount of analyst effort can resolve. Teams that invest in labeling improvements without also changing their evaluation model will still hit the noise ceiling.
Rubric-Based Evaluation — Treating AI Agents as Decision-Makers Under Uncertainty
The Framing Problem: Why Our Mental Model of AI Evaluation Is Wrong
The root issue with autonomous AI cyber defense evaluation is not a data problem — it is a framing problem. The field has imported an oracle-based mental model from classical computer vision and applied it uncritically to a domain where no oracle exists. When you reduce an agent that has generated 100,000 tokens of tool calls, evidence gathering, and reasoned justification down to a single bit — correct or incorrect against a noisy label — you have discarded almost everything that matters.
Saxe draws an explicit analogy: imagine hiring security engineers exclusively through binary multiple-choice tests and never interrogating their actual ability to reason under uncertainty. That is precisely what teams do when they evaluate autonomous security agents solely on outcome accuracy. A human security engineer is hired to make good, well-validated decisions under the extreme uncertainty characteristic of the field. An AI security agent should be held to the same standard.
The Interview Protocol Model
The practical reframe is to treat agent evaluation the same way you would structure a human interview or performance review. This means defining a rubric — a formal protocol that specifies what “good” looks like not just in terms of outcome but in terms of the process the agent used to reach that outcome.
For an access management agent deciding whether a contractor should be granted access to a sensitive database table containing millions of user messages, the evaluation rubric would ask five distinct questions:
- Evidence gathering — Did the agent retrieve the right data to inform its decision? Did it query access logs, review the contractor’s role definition, check data sensitivity classifications?
- Policy understanding — Did the agent correctly interpret the organization’s access control policies and apply them to the specific case?
- First-principles reasoning — Did the agent reason correctly from the evidence and policy to a logically sound conclusion, rather than pattern-matching to superficial features?
- Auditability — Did the agent produce logs and decision records in a format that security teams and compliance functions can actually review and act on?
- Decision accuracy — Did the agent reach the right outcome?
Note that decision accuracy appears last and is one of five criteria, not the only one. This is deliberate. In a noisy-label environment, outcome accuracy alone is an unreliable signal. Process quality dimensions are more stable — they can be assessed even when the ground-truth label is uncertain, because they measure how the agent reasoned, not just what it concluded.
Access Management Agent Rubric — Holistic Evaluation Across Five Dimensions
Proof of Concept
-
Define the evaluation scenario. The agent under evaluation is tasked with autonomously granting or denying access to a sensitive database table containing millions of user messages. The request comes from a contractor. This is exactly the kind of ambiguous, high-stakes decision where human access investigators themselves disagree at a double-digit rate — making binary outcome accuracy an unreliable sole metric.
- Establish the five rubric dimensions. For each access request the agent processes, score it across all five dimensions:
- Evidence Gathering: Did the agent retrieve the right contextual data before deciding? For an access request, this means pulling the contractor’s role, the sensitivity classification of the target resource, existing access patterns for similar roles, and any policy documents governing that data category. An agent that decides without gathering this evidence fails this dimension regardless of whether its final decision happens to be correct.
- Policy Understanding: Did the agent correctly identify and apply the relevant access control policy? This requires the agent to locate the governing policy (e.g., data minimization principles, least-privilege requirements, contractor access restrictions) and demonstrate that its reasoning is grounded in those policies — not just intuition or pattern-matching against superficially similar past decisions.
- First-Principles Reasoning: Did the agent reason correctly from the evidence and policy to its conclusion? This dimension captures logical validity. An agent may gather the right evidence and cite the right policy but still reach the wrong conclusion through a reasoning error. Scoring this dimension requires a human reviewer (or calibrated LLM judge) to trace the agent’s chain of thought and verify that each inferential step holds.
- Auditability: Did the agent produce logs and decision explanations in a format that supports post-hoc audit? For access management decisions affecting sensitive resources, regulators and internal compliance teams require a clear record of why access was granted or denied. An agent that produces well-reasoned decisions but inadequate audit trails fails this dimension even if its outcome is correct.
- Decision Accuracy: Did the agent reach the correct final decision — grant or deny? This is the classical metric, but it is now one dimension among five rather than the sole signal. Because human labelers disagree at double-digit rates on access decisions, this dimension carries uncertainty that must be acknowledged rather than suppressed.
-
Score agent trajectories using a calibrated LLM judge. Human security engineers manually score a calibration set of approximately 100 access decisions across all five dimensions. This calibration corpus is used to train an LLM judge to replicate human scoring at scale. Once calibrated, the LLM judge can process thousands of agent decision trajectories automatically, producing per-dimension scores for each case without requiring further human review.
-
Define the deployment bar. Establish minimum score thresholds across all five dimensions that the agent must sustain before being approved for autonomous operation. For example: Evidence Gathering ≥ 85%, Policy Understanding ≥ 80%, First-Principles Reasoning ≥ 80%, Auditability ≥ 90%, Decision Accuracy ≥ 75% (with acknowledged label noise). The team then hill-climbs all five dimensions simultaneously — adjusting agent prompts, retrieval strategies, and reasoning scaffolds — until the bar is cleared. This process creates a concrete, leadership-aligned target for deployment readiness rather than a subjective “it feels good enough” judgment.
-
Monitor in production. Once deployed, the same LLM judge continues scoring live agent decisions. Dimensional score drift — for example, a drop in Policy Understanding while Decision Accuracy holds — signals that the agent has encountered a new policy landscape it was not calibrated for, triggering a targeted re-evaluation before the problem compounds into harmful access decisions.
- Interpret multi-dimensional disagreement as a diagnostic signal. When the agent’s Decision Accuracy score is low but its Evidence Gathering and First-Principles Reasoning scores are high, this is a strong signal that the ground truth labels are wrong — not that the agent is wrong. Saxe notes from his own work that agents are frequently correct against human labelers in exactly this pattern. The multi-dimensional rubric makes this diagnostic visible; a single-metric evaluation would have simply recorded a false negative with no way to distinguish agent error from label error.
Why This Multi-Dimensional View Is More Robust
The comparison Saxe makes is between two worlds. In the first — the current state for most teams — you stare at accuracy, precision, recall, and F-score. When those numbers look bad, you cannot tell whether the model is underperforming or the labels are wrong. This creates persistent disagreement inside engineering teams and makes it nearly impossible to build credible arguments for deployment with leadership.
In the second world, you examine many dimensions simultaneously. Each dimension is individually noisy, but together they produce a far more complete picture of whether the system is actually working. If evidence gathering scores are high but decision accuracy is moderate, that is a signal that the agent is reasoning correctly but the labels used to judge outcomes are unreliable — a very different diagnosis from an agent that scores poorly on first-principles reasoning. This dimensional separation is what makes the rubric approach actionable.
Aligning Leadership with the Rubric
One underappreciated benefit of the rubric-based approach is organizational: it gives you a shared language for deployment decisions. Rather than presenting leadership with precision and recall numbers they distrust — because everyone in the room knows the labels are noisy — you can present a multi-dimensional profile of agent behavior grounded in criteria they can reason about directly.
When the rubric is defined up front and leadership has signed off on what “good enough to deploy” looks like across all dimensions, the deployment decision becomes mechanical once the agent clears the bar. This eliminates the subjective judgment calls that typically paralyze shipping decisions and creates a clear hill for the engineering team to climb.
The Broader Principle: Agents as Decision-Makers Under Uncertainty
Saxe’s thesis generalizes beyond access management. The same rubric structure applies to any autonomous security task:
- Autonomous alert triage — Did the agent gather the right corroborating signals? Did it apply the correct escalation policy? Did it explain its false-positive determination in a way an analyst can audit?
- Autonomous code remediation — Did the agent identify the correct root-cause vulnerability class? Did it reason about whether the patch preserves intended behavior? Did it flag uncertainty where the fix might affect production stability?
- Cloud posture management — Did the agent correctly apply the relevant compliance framework? Did it prioritize findings by actual exploitability rather than nominal severity?
In each case, reducing the agent’s behavior to a binary outcome and comparing it against a noisy label is both statistically weak and practically useless for improving the system. The rubric-based model preserves the reasoning signal that makes it possible to diagnose failures and direct improvement effort to the right dimensions.
Actionable Takeaways
- Define a rubric for your agent before you write a single evaluation query. Identify 4–6 dimensions that capture both process quality (evidence gathering, policy understanding, first-principles reasoning, auditability) and outcome accuracy. Each dimension needs a written definition and scoring criteria, not just a name. Involve the same domain experts who would interview a human hire for that role — they understand what correct reasoning looks like in that specific task context.
- Align your deployment bar with leadership against the full rubric, not just accuracy metrics. Present the multi-dimensional profile of agent behavior in early evaluations and anchor the go/no-go decision to explicit thresholds across all criteria. This converts a subjective shipping decision into an objective milestone and eliminates the credibility problem that comes from presenting noisy precision/recall numbers to stakeholders who know the labels are unreliable.
- When an agent's outcome accuracy looks poor but process quality dimensions (evidence gathering, reasoning, auditability) score well, treat that as a signal that the labels — not the agent — are the primary problem. Use this dimensional separation to diagnose the true failure mode before investing in model changes. In Saxe's experience, agents are frequently right against human labelers when reasoning quality is high but outcomes appear off.
Common Pitfalls
- Reducing evaluation to a single binary outcome metric and interpreting it as ground truth. When you collapse an agent's full reasoning trajectory — tool calls, evidence queries, policy interpretation, decision justification — into a single correct/incorrect bit and compare that against a noisy human-generated label, you have discarded the majority of the signal available to you. This not only fails statistically (the label noise ceiling problem) but also fails practically: you cannot diagnose what the agent is doing wrong or direct improvement effort to the right dimension.
- Designing the rubric without domain experts. The criteria in a rubric for an autonomous SOC triage agent are fundamentally different from the criteria for an access management agent or a code remediation agent. If the rubric is designed by generalists without deep knowledge of the investigative workflow, the alert taxonomy, or the access control policies in use, the rubric will fail to distinguish capable agents from lucky ones. The same level of expertise you would require to design a senior hiring process for that role is the minimum bar for rubric design.
LLM Judge Automation and the Deployment Bar — Scaling Rubric Evaluation in Practice
Scaling Rubric Evaluation with an LLM Judge
Once you have a well-defined rubric that evaluates agentic AI systems across dimensions like evidence gathering, policy understanding, first-principles reasoning, auditability, and decision accuracy, the next problem is scale. Human reviewers are expensive. Having security engineers grade hundreds of agent trajectories by hand is not a sustainable process — but it is a necessary one to get started.
Saxe’s approach bridges this gap with a calibrated LLM evaluation judge: a language model trained on a relatively small set of human-graded rubric evaluations that can then score new agent trajectories automatically at scale. The key insight is that you do not need a massive labeled dataset to get this working. In practice, as few as 100 human-graded samples is often enough to calibrate the LLM judge to a useful level of reliability.
The Calibration Process
The calibration model is deliberately Bayesian in character. Human reviewer agreement is used to weight the reliability of each labeled example:
- If a committee of three reviewers all agree strongly on how a rubric dimension should be scored for a given trajectory, those examples carry high weight in the calibration set.
- If reviewers disagree — which happens frequently in cybersecurity because the underlying ground truth is ambiguous — those labels carry lower weight. You want to discount uncertain examples rather than treat them as equally reliable signal.
- This mirrors the earlier problem of noisy labels in classical evaluation: instead of pretending disagreement does not exist, you explicitly model it and reduce its influence on the judge’s calibration.
LLM Judge Calibration — Using Human Disagreement to Weight Reviewer Confidence
Proof of Concept
-
Collect a human-labeled calibration set. Assemble roughly 100 agent trajectories (e.g., SOC alert investigations or access management decisions) and route each to a committee of three human reviewers. Each reviewer independently scores the trajectory on each rubric dimension (evidence gathering, policy understanding, first-principles reasoning, auditability, decision justification). Do not allow reviewers to see each other’s scores before submitting.
-
Compute inter-rater agreement per label. For each trajectory–dimension pair, calculate whether all three reviewers agreed, two agreed (one dissented), or all three disagreed. Record the agreement level as a confidence signal alongside the majority-vote label. In practice, Saxe observes double-digit disagreement rates on tasks like SOC alert classification and access permission decisions — meaning a non-trivial fraction of your calibration labels carry built-in uncertainty.
- Assign label weights using a Bayesian model. Rather than treating every human label as an oracle ground truth, apply differential weighting:
- All three reviewers agree strongly → assign high weight to the label (strong signal, low uncertainty).
- Two of three agree → assign moderate weight; this label is plausible but not certain.
- All three disagree → assign low weight or treat the case as ambiguous; do not use it as a high-confidence calibration point. Saxe explicitly notes this requires “a fairly complicated Bayesian model” that places more uncertainty around labels where disagreement is higher.
-
Train or fine-tune the LLM judge on weighted samples. Use the weighted calibration set to train (or prompt-tune via few-shot examples) an LLM judge to score unseen agent trajectories. High-weight examples (strong reviewer consensus) exert more influence on the judge’s scoring behavior; low-weight examples (high disagreement) contribute less, preventing the judge from confidently replicating human noise.
-
Validate judge calibration against held-out consensus cases. Reserve a subset of the calibration set where all three reviewers agreed (maximum-confidence labels) as a held-out validation set. Check that the judge’s scores on this subset correlate strongly with human consensus. Miscalibration here indicates the judge has overfit to noisy disagreement cases.
-
Deploy the calibrated judge for at-scale trajectory scoring. Once validated, use the LLM judge to automatically score the full population of agent trajectories across all rubric dimensions. This operationalizes the holistic rubric at scale without requiring a human to review every trajectory — enabling continuous production monitoring and providing the team a concrete, multi-dimensional hill-climbing signal.
- Iterate calibration as domain knowledge evolves. As the agent improves and new edge cases emerge, periodically collect fresh human labels on a small batch of trajectories the judge is least confident about (highest variance scores). Reweight and retrain the judge accordingly. This closes the calibration loop and prevents judge drift as the underlying agent architecture changes.
Note: Saxe describes the high-level Bayesian weighting principle and confirms it works in practice, but states he does not have time to go into the full statistical model details in the talk. The exact model specification would require direct follow-up with the speaker or access to the implementation used at Meta and his current startup.
Defining and Hill-Climbing the Deployment Bar
With a calibrated LLM judge in place, the team now has a concrete deployment bar — a threshold that the agent must clear across all rubric dimensions before it is considered safe to ship autonomously. The process works as follows:
- Define the bar upfront, in collaboration with leadership, so everyone agrees on what “good enough to deploy” means before development begins. This is critical: if you align leadership on the rubric dimensions and the numeric threshold before you start building, you remove the ambiguity that normally prevents autonomous systems from shipping.
- Hill-climb all dimensions iteratively. The team improves agent architecture, prompts, retrieval strategies, and tool use — continuously re-evaluating against the rubric until all scores clear the deployment bar.
- Deploy when the bar is cleared, with confidence. The multi-dimensional rubric gives you a defensible answer to “why are you sure this is safe?” that a single accuracy number never could.
This structure transforms the evaluation from a post-hoc audit into an active development target — the team always knows exactly where they stand and what needs to improve.
Evaluation Overhead Is an Investment, Not a Tax
A common objection to rigorous evaluation is that it slows teams down. Saxe’s direct experience contradicts this. In his work — at Meta and at the startup he co-founded — a proper evaluation program typically consumes around 50% of the team’s time, but the result is that the team moves approximately ten times faster overall. The reason is straightforward: without a reliable evaluation signal, engineers are making changes and arguing about whether they helped. With a calibrated LLM judge scoring agent trajectories across multiple dimensions, every change produces a measurable outcome and the team converges on improvements instead of debating them.
Production Monitoring After Deployment
Shipping the agent is not the end of the evaluation work — it is a transition to a different kind of evaluation. Once an autonomous security agent is live, the same rubric and LLM judge infrastructure should be used to monitor agent behavior in production:
- Continuously score live agent trajectories against the rubric dimensions.
- Watch for drift — cases where scores on one or more dimensions begin to degrade over time, indicating the agent has gone off the rails as data distribution shifts.
- Use production monitoring as an early warning system that triggers human review before a failure reaches the level of a dangerous autonomous decision.
This creates a closed loop: rubric defines what good looks like → LLM judge calibrated on human examples scores trajectories at scale → deployment bar cleared → agent ships → production monitoring continues scoring live behavior → alerts trigger when the bar is no longer being met.
From Vibes to Verifiable — Why This Matters for Security AI Teams
Many AI security teams today operate with no formal evaluation at all. Saxe describes this bluntly as a “dystopia” — engineers tweaking agent prompts and saying it works on their machine, with no shared signal for whether the system is actually improving. Without evaluation, it is nearly impossible to ship with rigor because there is nothing objective to point to when leadership asks whether the system is ready.
The LLM judge and deployment bar model solves this directly. It replaces subjective intuition with a scored, multi-dimensional signal that leadership can inspect, that the team can optimize against, and that continues operating after deployment as a safety monitor.
Actionable Takeaways
- Start your LLM judge calibration with a human-graded set of as few as 100 agent trajectory samples. Apply a Bayesian weighting scheme that discounts labels with high inter-reviewer disagreement and weights heavily the examples where reviewers converge — this directly mirrors the noisy-label problem and produces a more reliable judge from limited annotation effort.
- Define your deployment bar in collaboration with leadership before development begins, not after. Align on the specific rubric dimensions (evidence gathering, policy understanding, reasoning quality, auditability, decision accuracy) and the numeric thresholds required for each. This removes downstream ambiguity about whether the agent is ready to ship and turns evaluation into an active hill-climbing target rather than a post-hoc gate.
- After deploying an autonomous agent, repurpose the same LLM judge infrastructure for continuous production monitoring. Score live agent trajectories against the rubric on an ongoing basis and trigger human review when any dimension drifts below the deployment bar threshold — this is your early warning system before a degrading agent makes a dangerous autonomous decision.
Common Pitfalls
- Operating with no formal evaluation — relying purely on engineer intuition and anecdotal "it works on my machine" assessments. Without a scored, multi-dimensional signal, the team cannot distinguish real improvement from noise, cannot align leadership on readiness, and cannot safely ship autonomous systems. Saxe explicitly describes this as a common failure mode across many current AI security teams.
- Treating the deployment bar as a one-time shipping gate rather than an ongoing monitoring threshold. Once an agent is live, data distributions shift and agent behavior drifts. Without continuous rubric scoring in production, a degrading agent can make dangerous autonomous decisions before any human notices the failure.
Automated Agent Optimization Using Evaluation as a Signal
From Manual Prompt Engineering to Systematic Agent Optimization
Once you have a robust security automation evaluation framework in place — a calibrated LLM judge, a well-defined rubric, and a deployment bar the team agrees on — you unlock a capability that most security AI teams are not yet using: automated agent optimization.
Joshua Saxe describes direct experience with this both at Meta and at his startup. The approach uses AI coding tools[1] alongside genetic algorithms for prompt optimization[2] to automatically iterate on agent architecture and prompt design, using rubric scores as the objective function to hill-climb.
How Automated Optimization Works
The feedback loop is straightforward in structure, but powerful in practice:
-
Define the objective function. Your multi-dimensional rubric score — covering evidence gathering, policy understanding, first-principles reasoning, auditability, and decision accuracy — becomes the signal the optimizer is trying to maximize.
-
Use AI coding tools to modify the agent. LLM-based coding assistants can generate and apply prompt variants or architectural changes automatically, without a human hand-writing each iteration.
-
Apply genetic/prompt-optimization algorithms. Genetic algorithms treat prompt configurations as a population, selecting, mutating, and recombining the variants that score highest on your evaluation rubric across generations.
-
Gate against overfitting. Because the optimizer is hill-climbing a scored evaluation set, there is a real risk that the agent learns to score well on that specific set without generalizing. Saxe explicitly flags this as a concern — robust evaluation design (sufficient sample diversity, held-out validation sets) is a prerequisite for automated optimization to be safe.
Why Evaluation Quality Is a Hard Prerequisite
Automated optimization amplifies whatever your evaluation measures. If your rubric is shallow, the optimizer will produce an agent that games shallow metrics. If your evaluation set is too small or too narrow, the optimizer will overfit to it. This is why Saxe treats evaluation as the primary hard problem — not agent architecture, not model selection. The optimization pipeline is only as trustworthy as the evaluation it optimizes against.
Teams without a rigorous evaluation program that attempt automated optimization are likely to produce agents that look better on paper while actually degrading in production behavior.
The Payoff
When the evaluation framework is solid, automated optimization dramatically accelerates the path to deployment. Rather than engineers manually tweaking prompts based on intuition and spot-checking outputs, the system explores the optimization landscape systematically. Saxe notes that this kind of approach has been a meaningful unlocker of progress — and that it represents the next frontier for teams that have already solved the evaluation problem.
Actionable Takeaways
- Before pursuing automated agent optimization, verify your evaluation framework is multi-dimensional and your LLM judge is calibrated against human-graded samples — automated optimization will faithfully maximize whatever you measure, so shallow or noisy metrics will produce agents that game scores rather than improve real-world behavior.
- Use genetic or prompt-optimization algorithms with your rubric score as the objective function, combined with AI coding tools to generate and apply prompt/architecture variants automatically — this converts your deployment bar from a one-time threshold into a continuous improvement target.
- Maintain a held-out evaluation set that the optimizer never trains against, and monitor production behavior post-deployment to detect overfitting — an agent can score well on an optimization set while degrading on out-of-distribution security decisions.
Common Pitfalls
- Attempting automated agent optimization before establishing a robust, calibrated evaluation framework. Without a reliable rubric and LLM judge, the optimization process has no trustworthy signal to hill-climb and will produce agents that appear to improve on paper while potentially getting worse on the dimensions that matter in real deployments.
- Ignoring overfitting risk when using automated optimization. Because the optimizer is explicitly designed to maximize scores on your evaluation set, it will find ways to score well on that specific set if given enough iterations — teams must use held-out validation sets and production monitoring to distinguish genuine capability improvement from evaluation gaming.
Conclusion
The central insight from Joshua Saxe’s talk is that building an autonomous AI cyber defense system is not the hard part — measuring whether it actually works is. Classical ML metrics like precision, recall, and F-score were designed for clean-oracle domains and systematically fail in cybersecurity, where double-digit label disagreement rates are the norm and theoretical problems make formal verification impossible. The label noise ceiling means teams can be shipping real improvements with no measurement signal that anything changed.
The replacement is not a single new metric but a different evaluation paradigm: treat autonomous agents as decision-makers under uncertainty, evaluate the full reasoning trajectory across multiple rubric dimensions, calibrate an LLM judge on a small human-graded set, define a multi-dimensional deployment bar with leadership alignment, and repurpose the same infrastructure for continuous production monitoring. When the evaluation framework is solid, automated optimization becomes tractable and the path to safe autonomous deployment becomes a concrete hill to climb rather than an open-ended risk judgment.
For teams building security operations automation, the practical takeaway is straightforward: invest in evaluation infrastructure first. Fifty percent of your team’s time on evaluation is not overhead — it is what makes the other fifty percent move ten times faster.
Explore related talks on AI/ML security and large language models in security contexts on The Cyber Archive.
References & Tools
- AI Coding Tools — LLM-based coding assistants used to automatically generate and apply prompt variants and architectural changes to agents as part of an automated optimization pipeline. ↩
- Genetic Algorithms for Prompt Optimization — Optimization technique that treats prompt configurations as a population, selecting, mutating, and recombining variants that score highest on rubric-based evaluation metrics — replacing manual prompt engineering with a systematic improvement loop. ↩
Questions from the audience
Related deep dives
Kinetic Risk: Securing and Governing Physical AI in the Wild | [un]prompted 2026
Securing Workspace GenAI at Google Speed | [un]prompted 2026
Glass-Box Security: Operationalizing Mechanistic Interpretability | [un]prompted 2026
