When data becomes the attack vector — not just the target — your existing threat model is incomplete. Threat modeling for AI/ML systems requires a fundamentally different playbook: one where training data pipelines, model supply chains, and user data interactions all represent first-class attack surfaces that traditional STRIDE and DREAD frameworks were never built to evaluate.
This post covers Susanna Cox’s ML SecOps framework from OWASP Global AppSec — three diagnostic questions for evaluating any AI deployment, the shift from static security gates to continuous data validation, and how to use the OWASP AI Exchange to categorize and address AI-specific threats.
Key Takeaways
- You'll learn how to apply structured threat modeling to AI/ML systems using three critical diagnostic questions—Is it secure? Can we operationalize it? Does it scale?—giving you a repeatable framework to assess any AI deployment from the ground up.
- You'll be able to identify data as an active attack vector, not just a target, so you can evaluate data provenance, governance, and user access as first-class security controls in AI/ML environments.
- Apply ML SecOps principles—continuous data validation, staged governance gates, and monitoring pipelines—to defend against AI-specific attacks like data poisoning and model theft before they reach production.
Threat Modeling Fundamentals and the Purple Team Approach for AI Security
Threat modeling for AI/ML systems begins with the same foundation that governs all security analysis: a structured, systematic approach to understanding what you’re protecting, what threatens it, and what you will do when things go wrong. Cox draws directly from the Threat Modeling Manifesto[1], which defines threat modeling as “analyzing representations of a system to highlight concerns about security and privacy characteristics.”
At its core, threat modeling asks four questions:
- What are we working on? — Do we actually understand our systems, their components, and how they interconnect?
- What can go wrong? — What are the realistic threat vectors for this specific system?
- What are we going to do about it? — What mitigations, controls, or responses do we have in place?
- How do we know we did a good enough job? — How do we validate our security posture is adequate?
These questions are deliberately open-ended. Threat modeling is not about achieving a perfect security state — it is about building a structured, repeatable understanding of risk so that teams can make informed decisions and communicate them clearly to stakeholders.
Threat Modeling as a Structured and Stakeholder-Facing Discipline
Cox emphasizes three non-negotiable characteristics of effective threat modeling:
- Clearly articulated: Whatever your threat model produces, it must be understood by the relevant stakeholders — from developers and data scientists to non-technical business decision-makers. A threat model that lives only in a security engineer’s head provides no organizational resilience.
- Consistently applied: Ad hoc threat analysis is not threat modeling. The process must be repeatable and applied uniformly across systems and deployments.
- A documented record of mitigations: Cox identifies this as one of the most overlooked aspects of the practice. Documenting in advance what you will do if a specific threat materializes creates a clear chain of custody and an incident response foundation. It also surfaces gaps before an attack occurs.
This documentation discipline is especially critical in AI/ML environments, where the attack surface is newer, less understood, and evolving faster than traditional application security frameworks can track.
Contextualizing Risk: Moving from Reactive to Proactive Security
The World Economic Forum’s Global Risk Report consistently ranks cybersecurity as a top global threat — and Cox notes this risk is explicitly linked to AI/ML’s growing role in mission-critical infrastructure. The goal threat modeling enables is system resilience: not the prevention of every possible attack, but the systematic reduction of surprise and impact. As Cox frames it: “Don’t make the perfect the enemy of the good.”
Threat Modeling Anti-Patterns to Avoid
The Threat Modeling Manifesto[1] identifies several failure modes that are particularly common in AI/ML security work:
- The “hero threat modeler” fallacy: Threat modeling must be teachable and embedded in team culture — not dependent on a single expert.
- Admiration for the problem: AI/ML security is genuinely novel. Teams can become consumed by the intellectual complexity of the threat landscape without making progress on mitigations.
- Tendency to over-focus: Getting lost in the statistical and architectural minutia of a specific AI system causes teams to miss higher-level structural risks.
- Striving for perfect representation: There is no canonical right way to represent an AI/ML threat model. Different stakeholders need different representations. Chasing a single “correct” diagram at the expense of actionable outputs is a trap.
The Purple Team Imperative for AI Security
Cox identifies threat modeling as “probably the original purple team technique.” In practice, this means getting defenders to actively think like attackers — to understand how a system can be manipulated, exploited, or subverted, not just how it is supposed to function.
For AI/ML systems, this purple team orientation is not optional. AI systems operate on statistical gradients, which means they can be tricked in ways that traditional software cannot. Defenders must develop an AI engineer’s mental model — understanding how MLOps pipelines, data flows, and model inference work from a builder’s perspective. Cox’s core message: you cannot secure AI unless you can think like the person who built it.
Actionable Takeaways
- Embed the four threat modeling questions (What are we working on? What can go wrong? What are we going to do about it? How do we know we did good enough?) into every AI/ML project kickoff and architecture review — not as a one-time exercise but as a recurring, documented process.
- Document mitigations before incidents occur: for each identified threat in your AI/ML system, maintain a written record of the specific controls, response procedures, and escalation paths. This creates the chain of custody needed for post-incident review and regulatory accountability.
- Develop the AI engineer's mental model alongside your security lens — understand how MLOps pipelines, data flows, and model inference work from a builder's perspective so you can identify threat vectors that are invisible to a purely defensive mindset.
Common Pitfalls
- Treating threat modeling as a one-time audit deliverable rather than a living, operationalized discipline. In AI/ML environments where data, models, and infrastructure change continuously, a static threat model becomes dangerously stale within weeks.
- Assuming AI/ML systems are fundamentally different from traditional software and therefore exempt from basic security questions. Cox observed repeatedly in production environments that teams skip foundational controls (access control, credential management, monitoring) specifically because they perceive AI as a separate category.
Data as an Attack Vector: Understanding the AI/ML Security Paradigm Shift
The Paradigm Shift: From Data as Prize to Data as Attack Vector
In 2012, the technology industry declared that data was the new oil. By 2024, the security framing has fundamentally changed: data is the new attack vector. This is not a metaphor — it is a description of a structural vulnerability that is inherent to how AI/ML systems function.
Traditional security frameworks operate under the CIA triad. Data was the prize — something to be stolen, corrupted, or made inaccessible. In AI/ML environments, data occupies a qualitatively different role. The model is the data. The model’s behavior, its outputs, its confidence scores, and its decision boundaries are all direct products of its training data. An adversary who can influence your data — even without ever accessing your infrastructure directly — has indirect access to your model’s behavior at scale.
As Cox states plainly: “Access to your data is access to your model.”
Why AI/ML Systems Are Inherently Susceptible to Data-Layer Attacks
AI/ML systems operate on statistical gradients. Unlike deterministic software where a specific input produces a predictable output, a model produces outputs based on learned probability distributions. This has a critical security implication: any system that works on a gradient can be tricked.
This susceptibility is not a bug in a specific implementation — it is a mathematical property of the architecture. Adversaries who understand this can craft inputs or modify training data to shift model behavior in specific directions without ever touching the underlying code:
- Data poisoning: Introducing malicious samples into training or fine-tuning datasets to alter model behavior. Depending on the system architecture, a bad actor may need relatively few resources — or even just one node on a graph — to execute an effective poisoning attack.
- Concept drift exploitation: Model output degrades when the statistical distribution of input data shifts over time. Malicious actors can accelerate or weaponize this drift by systematically skewing the data the model is exposed to.
- Inference manipulation: At runtime, adversarial inputs can be crafted to push model outputs across decision boundaries — exploiting gradient-based confidence scoring to produce attacker-desired results.
Data Provenance: The New Chain of Custody
Because data is the attack surface, knowing where your data comes from is a security control — not just an operational concern. The key question: who has had access to your data before it reached your model?
Data from brokers: When data is sourced from third-party brokers, outside parties have had direct custody of that data. Security engineers must demand chain-of-custody documentation and validate data quality independently.
Data from users: User-generated data is a significantly higher-risk profile. Cox is direct: when your model is trained or fine-tuned on data that users can directly influence, you are effectively giving the keys to your entire system to any user on your platform. The mitigation is not to avoid user data — it is to implement rigorous, continuous validation pipelines that detect anomalies and statistical shifts before they affect model behavior.
Foundation models and third-party models: The same provenance logic applies to externally developed models. Integrating a foundation model introduces all the data provenance risks of that model’s training pipeline into your system. Cox frames this explicitly: “the model is the data and the data is the model” — they are not separable concerns. This is the AI/ML equivalent of a software supply chain vulnerability.
The Scale Problem: Why Data Vulnerabilities Are Amplified in Production
AI/ML systems require massive data volumes to function at production scale. Scale transforms the impact of any data quality problem — whether introduced by adversarial manipulation or benign drift — from a localized incident into a systemic failure.
Case Study: Pregnancy Prediction Model — Ethical and Scalability Failure at 90% Accuracy
A real-world AI/ML development case where a 90%-accurate pregnancy outcome prediction model trained on 7.5 million patient health records was determined to be fundamentally undeployable — illustrating how scale transforms statistical error rates into mass-harm security and ethical failures, and how overly permissive data access creates data poisoning exposure.
Technical Write-up:
-
Problem definition and data acquisition: A health company tasked their data science team with predicting pregnancy outcomes using patient health data. The initial dataset was extremely large and complex, with millions of rows that no one had successfully loaded prior to this effort.
-
Data loading and feature engineering: After significant engineering effort, the full dataset was loaded. A target vector was engineered from the available health features — characteristic of the messy, real-world data conditions most AI/ML teams operate under.
-
Model training and accuracy assessment: The resulting model achieved 90% prediction accuracy on pregnancy outcomes. By standard data science metrics, this was a success. The team’s instinct was to deploy.
- Scale impact analysis — where the failure surfaces: The dataset contained approximately 7.5 million patient records. Applying 90% accuracy to this population:
- Correct predictions: 6,750,000 patients
- Incorrect predictions: 750,000 patients served wrong results about a critical health outcome
- Consequence analysis: Cox identifies two categories of downstream impact:
- Personal Havoc: Individual patients making medical, emotional, and life decisions based on incorrect AI outputs
- Social Havoc: Systemic erosion of trust in healthcare AI when errors surface at population scale
-
The data access exposure: The data collection methodology gave users excessive access to the data, creating direct exposure to data poisoning — users who can influence what data the model trains on can shift model behavior. Cox notes that data poisoning in this type of system is “incredibly hard to mitigate” once the architecture permits broad user data access.
- Decision outcome: The model was determined to be fundamentally undeployable. The product had to be abandoned or repurposed for a use case where the error rate’s impact was acceptable.
Security lesson: High accuracy does not equal deployability or security. The failure is architectural — the combination of scale, consequence severity, and permissive data access created a system that could not be secured against harm regardless of further optimization.
Without automated monitoring pipelines, there is no reliable way to distinguish between a model performing as expected, a model experiencing benign concept drift, and a model under active data poisoning attack. Cox presents continuous monitoring as a prerequisite, not a best practice:
- Would you know if your model output was degrading? If not, you have a monitoring gap that is also a security gap.
- Would you know if your data quality was degrading? Distinguishing poisoning from drift requires established statistical baselines.
- Who is monitoring the monitors? The monitoring pipeline itself has its own attack surface and integrity requirements.
Actionable Takeaways
- Map every data source feeding your AI/ML systems and classify each by provenance risk tier: first-party controlled data, third-party broker data, and user-generated data. Apply the strictest validation and monitoring requirements to user-generated data, and demand chain-of-custody documentation from data brokers before integrating their datasets.
- Implement continuous data validation pipelines that monitor statistical distributions of incoming data against established baselines — not just periodic audits. Set alerting thresholds for distribution shifts that could indicate concept drift or active data poisoning, and ensure the monitoring pipeline itself is treated as security-critical infrastructure.
- Before integrating any foundation model or third-party model, conduct a data provenance review of that model's training pipeline as part of your standard vendor security assessment. Treat inherited model risk the same way you treat third-party code dependencies in a software supply chain review.
Common Pitfalls
- Treating data access control as an IT operations issue rather than a security control. In AI/ML systems, broad data access — common in data science environments where scientists routinely work with production datasets — creates a direct pathway for insider threat and data poisoning. Access to training data is access to the model's behavior.
- Assuming that high model accuracy means the model is secure. Accuracy metrics say nothing about adversarial robustness, data integrity, or the ethical and security implications of deployment at scale.
ML SecOps: Integrating Security into the AI/ML Operations Pipeline
From DevOps to AI/ML Ops: Understanding the Operational Foundation
To understand ML SecOps, you first need to understand what makes AI/ML operations fundamentally different from traditional software operations. Cox grounds this in the core purpose of any AI/ML system: inference at scale. You cannot have AI security without first having AI operations — because without operational infrastructure, you have no visibility into whether your system is functioning, degrading, or under attack.
AI/ML Ops (MLOps) operationalizes AI/ML development, drawing from the same principles as DevSecOps:
- Communication: Open information sharing among all teams — developers, data scientists, ML engineers, security, and operations.
- Integration of expertise: Bringing together the knowledge of all these disciplines rather than allowing them to operate in isolation.
- Operationalization: Systematizing processes so they can function reliably at production scale — not just in a research environment.
Continuous Deployment in AI/ML Is Not What You Think
In AI/ML environments, continuous deployment means continuous data processing. The pipeline must handle data acquisition, data validation, and pipeline integrity — and that’s before you even reach the model. Beyond data, Cox identifies a risk endemic to AI/ML development environments: the Jupyter notebook problem.
Data scientists do their core development work in Jupyter notebooks[2]. These notebooks contain the organization’s intellectual property: feature engineering logic, model architectures, training configurations, and experimental results. In most organizations, this IP lives in an ungoverned, ad hoc collection of files with no version control, no access tracking, and no review process. The question security engineers must force into the conversation: where does all of your IP live, who has access to it, and what processes ensure it is secure?
ML SecOps: The Security Layer on Top of MLOps
ML SecOps is DevSecOps applied to machine learning — the integration of AI/ML-specific security mitigations directly into the MLOps pipeline. Cox is unambiguous: production-grade AI/ML at scale is literally impossible without ML SecOps.
Research by Google, Deloitte, and other major organizations has consistently shown that the majority of AI/ML projects fail to reach production — and a primary reason is the failure to operationalize. Systems that cannot be operationalized cannot be monitored, and systems that cannot be monitored cannot be secured.
Cox introduces the first published ML SecOps reference architecture[3] (from her paper “Securing AI/ML Systems in the Age of Information Warfare”), which extends the traditional MLOps pipeline with operationalized security reviews at each critical stage:
- Security review gates: Mandatory checkpoints before data, models, or code progress to the next pipeline stage
- Continuous security monitoring: Automated systems tracking access patterns, data anomalies, and output distribution shifts
- Model governance controls: Formal accountability for who releases models, under what conditions, and with what review sign-off
The Actors Problem: Securing a Multi-Discipline Environment
Traditional application security deals primarily with developers. AI/ML security adds two new categories of actors:
- Data scientists: Often operate with minimal security oversight, have direct access to large production datasets, and work in ungoverned notebook environments. Their culture is optimized for rapid experimentation, not security rigor. They are not adversarial — they simply have never been asked to think about security.
- Machine learning engineers: Bridge the gap between data science prototypes and production systems, often extracting code from notebooks and assembling it into pipelines with limited formal review. They inherit the security debt of whatever they pull from.
Cox is candid: “You’ve got a new set of people to wrangle and a new culture to understand.” Understanding MLOps is how security engineers get ahead of these actors — not to police them, but to build governance systems that work with how these teams actually operate.
Governance Gates: The Practical Implementation
Cox’s recommended approach centers on staged repositories and mandatory review gates:
- Dedicated repositories for all IP: Every Jupyter notebook[2], training script, and model artifact must be checked into a monitored repository. “You’d think that should already be happening, but it’s probably not at your organization.”
- Staged repositories with progressive access control: Development work, validated prototypes, and production-ready artifacts live in separate environments with explicitly controlled promotion criteria.
- Security review gates before promotion: No model or dataset moves to a downstream environment without a documented security review. Cox acknowledges this is difficult to fully operationalize due to manpower requirements — but the gate must exist.
- No production releases without governance sign-off: The final gate creates accountability and a documented audit trail for every production AI/ML artifact.
Actionable Takeaways
- Mandate that all AI/ML intellectual property — Jupyter notebooks, training scripts, model configurations, dataset manifests — be stored in version-controlled, access-monitored repositories with documented ownership. Treat ungoverned notebooks as a security finding equivalent to unreviewed code pushed directly to production.
- Implement staged repository environments with explicit security review gates between development, staging, and production. Define the minimum criteria that must be met at each gate — even if initial implementation is lightweight — and ensure no model or dataset can bypass the gate without documented sign-off.
- Audit your continuous monitoring pipeline as security-critical infrastructure: verify that it is itself version-controlled, access-controlled, and that alerting logic and thresholds are documented and regularly reviewed.
Common Pitfalls
- Treating ML SecOps as something to bolt on after the MLOps pipeline is established. Cox is explicit: security must be built in from the start. Organizations that wait until the pipeline is "finished" find that retrofitting security gates into production AI/ML infrastructure is prohibitively expensive in both engineering time and operational disruption.
- Underestimating the cultural gap between data science teams and security teams. Data scientists are optimized for speed and accuracy, not security. Governance systems that don't account for how data scientists actually work — particularly their dependence on notebooks — will be circumvented or ignored.
Mapping the AI Attack Surface: Data Flows, Provenance, and Governance
Before an organization can apply specific controls to an AI/ML system, it must understand where its attack surface actually lies. Cox presents three foundational mapping activities that security engineers can begin immediately, on any AI/ML system, regardless of scale or architecture.
“I can tell you how to break a particular system or how to defend a particular system, but really there are three key items that you can start on right now with any system — and these will give you insight into what you do next.”
Know Your Data Flows
AI attack surface mapping begins with understanding where data moves. Cox recommends data flow diagrams or process flow diagrams as the primary artifact, but emphasizes that the representation format matters less than the outcome: every relevant stakeholder must be able to understand it.
In AI/ML environments, the stakeholder map is broader than in traditional application security. It includes developers, data scientists, ML engineers, and business leadership — each needing a different representation of the same underlying risks. Cox invokes the threat modeling principle of “no perfect representation” explicitly here: a rough diagram that enables action is worth more than a technically complete diagram that only security engineers can interpret.
Specific flow components to capture:
- Input data ingestion points and their sources
- Data transformation and preprocessing stages
- Model training and fine-tuning pipelines
- Inference endpoints and output channels
- Feedback loops where outputs re-enter the system as inputs
Know Your Data Provenance
Because the model’s behavior is a direct function of its training data, the provenance of your data is the provenance of your model’s behavior. The provenance investigation must cover origin, chain of custody, validation history, and the scope of user influence over data inputs.
Cox notes that when security engineers encounter friction when asking about data provenance — when requests for chain-of-custody information are met with resistance or inability to provide answers — that friction is itself a high-severity security finding. An organization that does not know where its data comes from does not know what its AI/ML system has been trained to do.
Know Your Data Governance
Data governance answers the accountability question: who is responsible for the security of the data that powers your AI/ML systems? Cox identifies a persistent and dangerous organizational pattern: in many AI/ML organizations, data governance is treated as a function entirely separate from security. This creates a gap where security-relevant decisions about data access, retention, and validation are made without security oversight.
Security engineers must actively pursue involvement in data governance structures — not treat it as another team’s problem. Data governance has direct implications for regulatory compliance (GDPR, HIPAA, emerging AI regulations) and for the ethical dimensions of AI deployment that carry legal and reputational risk.
Using the OWASP AI Exchange to Bridge Mapping and Controls
Once a data flow diagram is in place, Cox recommends using the OWASP AI Exchange[4] as the structured control library for mapping identified threat vectors to actionable mitigations. The Exchange organizes AI/ML threats into three operational categories:
- Threats through use: Attacks at inference time through the model’s normal user-facing interfaces (prompt injection, adversarial inputs, model extraction)
- Development-time threats: Attacks introduced during model development, training, or fine-tuning (data poisoning, supply chain compromise, insecure development environments)
- Runtime application security threats: Traditional application security vulnerabilities affecting the application layer surrounding the AI/ML model
The practical workflow Cox recommends: build your data flow diagram → map each stage to the relevant OWASP AI Exchange[4] threat category → review associated controls → determine applicability → begin writing policy and implementing controls.
Actionable Takeaways
- Build a data flow diagram for every AI/ML system in your organization — even a rough one. The process of creating it will surface data access patterns, third-party dependencies, and feedback loops that are invisible until mapped. Share it with data scientists and ML engineers to validate accuracy and surface gaps.
- Use the OWASP AI Exchange as your control mapping reference: for each stage in your data flow where data interacts with users, third parties, or external systems, look up the applicable threat category and review associated controls. Prioritize the controls that address your highest-risk data flows first.
- Initiate direct engagement with your organization's data governance function — request access to data provenance documentation, participate in data governance meetings, and identify where security oversight is absent from data-related decisions.
Common Pitfalls
- Attempting to create a comprehensive, exhaustive data flow diagram before taking any action on identified threats. The goal of mapping is to enable action, not to achieve perfect representation. A rough map that enables three security improvements delivers more value than a perfect map that is never finished.
- Siloing data governance from security. When data governance and security operate as separate functions with no formal coordination, security-relevant data decisions — such as granting data scientists access to production training datasets — are made without security input. This gap is where many AI/ML security failures originate.
The Three-Question Security Framework for Scaling AI Systems
One of the most actionable contributions of Cox’s talk is a three-question diagnostic framework that any security engineer can apply to any AI/ML system — regardless of the underlying technology, the maturity of the organization, or the specific use case. These three questions function as an ordered diagnostic loop.
Question 1: Is It Secure?
The first question seems obvious — and that is precisely why Cox leads with it. In practice, the foundational security question is routinely skipped for AI/ML systems, because teams perceive AI as a different category of technology that operates outside normal security requirements.
“Is it secure?” means asking:
- Who has access to every component of this system? Rule-based access control applied to all AI/ML infrastructure: training data, model artifacts, inference endpoints, monitoring systems, development environments, and pipeline tooling.
- Are credentials managed correctly?
- Have we mapped our AI/ML dependencies? Teams frequently assume isolation between AI/ML systems that doesn’t actually exist.
- Are basic security controls applied to the development environment? Getting data scientists and ML engineers to consider “is this secure?” — even as a first-principles exercise — represents significant progress.
Case Study: Biometric Company Breach via Hardcoded AWS Keys in AI/ML Systems
A biometric AI company suffered a severe breach resulting from hardcoded AWS access keys exposed in their codebase — demonstrating that foundational credential hygiene failures in AI/ML systems carry permanently irreversible consequences when the compromised data is biometric in nature.
Technical Write-up:
-
Root cause — credential mismanagement in an AI/ML context: The company hardcoded AWS access keys[5] directly into their codebase. This is a basic credential hygiene failure (CWE-798[6]), but one that is disproportionately common in AI/ML development environments where data scientists and ML engineers — who are not security-trained — often manage infrastructure access directly.
-
Exposure vector: The hardcoded credentials were exposed publicly — most likely via a public code repository or misconfigured artifact. Once AWS keys are exposed, automated credential scanners continuously harvest and exploit leaked cloud credentials within minutes to hours of exposure (ATT&CK T1552.001).
-
Unauthorized access to AI/ML data infrastructure: With valid AWS credentials (ATT&CK T1078), the attacker gained access to the company’s cloud storage and compute infrastructure — specifically, fingerprint templates, facial recognition embeddings, and related biometric feature vectors — the direct outputs of their AI/ML processing pipeline.
-
Data exfiltration: The biometric data was exfiltrated. The breach was characterized as severe.
-
The irreversibility dimension — the AI/ML-specific consequence: Unlike a password breach, where affected users can reset credentials, biometric data cannot be changed. A person whose fingerprint templates or facial recognition embeddings have been compromised is permanently exposed. Any system using those biometrics for authentication is now vulnerable to replay attacks — potentially for the rest of that person’s life.
-
Why this is an AI/ML security failure, not just a DevOps failure: The AI/ML nature of the system amplified the consequences: the data exposed was not just sensitive, it was permanently irreplaceable.
Security lesson : The exploitation path is straightforward — exposed credentials, cloud access, data exfiltration — but the severity is uniquely AI/ML-amplified. Credential management in AI/ML systems must be treated as a zero-tolerance policy.
Question 2: Can We Operationalize It?
“Can we operationalize it?” means asking whether security can be maintained at production scale. Cox makes the stakes explicit: “If you don’t have a plan to operationalize the monitoring of your system, your outputs, and your data, you definitely have security problems.”
The absence of monitoring is not a gap to be addressed later — it is an active vulnerability. The Q&A surfaced an important nuance: operationalizing security reviews takes resources. Implementing security monitoring can itself create scaling friction. Cox’s guidance: this tradeoff should be resolved in favor of security architecture, because unmonitored systems that scale are simply large, undetectable failures waiting to happen.
Question 3: Does It Scale?
“Does it scale?” addresses whether the security posture can hold under production load — where the volume, velocity, and variability of production-scale AI/ML operations stress-tests every control designed in a lower-stakes development environment.
The Diagnostic Loop: When to Go Back
The three questions form a loop, not a linear checklist. If the system fails the scaling question, Cox’s default recommendation is to return to Question 1 — because scaling failures in AI/ML systems are frequently symptoms of missing foundational security controls:
Is it secure? → Can we operationalize it? → Does it scale? → If not, return to the failing layer.
The Cultural Dimension: Why Security Engineers Are Positioned to Drive This
Cox closes with a direct strategic observation: “Coming to AI/ML from security is much easier than going the other way.”
The “move fast and break things” culture that characterizes many AI/ML teams produces exactly the kinds of vulnerabilities the three-question framework is designed to surface: ad hoc access management, ungoverned notebook environments, no monitoring, and no governance gates. Security engineers who invest in understanding AI/ML operations are positioned to be the organizational force that closes these gaps — not through friction and enforcement, but through building governance systems that work with how these teams actually operate.
Actionable Takeaways
- Apply the three-question diagnostic framework at the start of every AI/ML project and at each major milestone: Is it secure? Can we operationalize it? Does it scale? Document the answers and the gaps they reveal. Make this a repeatable practice that the team internalizes, not a one-time security review.
- When you encounter a scaling problem in an AI/ML system, resist the instinct to treat it as a purely operational issue. Use the three-question loop to trace whether the scaling failure originates in missing foundational security controls or in operationalization gaps — because the remediation path differs significantly between the two.
- Invest in learning AI/ML development culture and MLOps workflows. Security engineers who understand how data scientists work, what Jupyter notebooks are used for, and how models move from development to production are significantly more effective at building governance systems that teams will actually follow.
Common Pitfalls
- Treating the three questions as a one-time security review checklist rather than a continuous diagnostic loop. AI/ML systems change continuously — new data sources, model updates, pipeline changes, and new team members all create new security exposure that must be re-evaluated regularly.
- Skipping Question 1 entirely for AI/ML systems because the team perceives AI as categorically different from traditional software. Basic security controls — access management, credential hygiene, dependency mapping — are routinely omitted from AI/ML systems by teams who believe those controls don't apply. They always apply.
Conclusion
Susanna Cox’s talk is a practical orientation guide for security engineers entering the AI/ML security space. The core argument is tight: AI/ML systems have a fundamentally different security profile because data is the attack vector, not just the target — and securing them requires both the structured rigor of threat modeling and the operational discipline of ML SecOps. The three-question framework (Is it secure? Can we operationalize it? Does it scale?) gives practitioners an immediately applicable diagnostic tool that works on any AI/ML system, at any stage of maturity.
The most important takeaway for practitioners: you do not need to be an AI/ML expert to start. You need to ask the questions that AI/ML teams have never been asked, build the governance structures that don’t exist yet, and position yourself at the intersection of security and the AI/ML development culture. That intersection is where the most impactful security work in the next decade will happen.
For further reading on related application security topics, explore our coverage of threat modeling methodologies and DevSecOps pipeline security.
References & Tools
- Threat Modeling Manifesto — Foundational document defining threat modeling's four core questions, essential characteristics, and anti-patterns. ↩
- Jupyter Notebooks — Primary development environment for data scientists; identified as the primary ungoverned IP storage risk in most AI/ML organizations. ↩
- Securing AI/ML Systems in the Age of Information Warfare (Cox, 2022) — First published ML SecOps reference architecture; provides platform-agnostic pipeline designs integrating security review gates and continuous monitoring into MLOps workflows. ↩
- OWASP AI Exchange — Community-driven framework organizing AI/ML threats into three operational categories (threats through use, development-time threats, runtime threats) with associated controls. ↩
- AWS IAM Security Best Practices — AWS guidance on credential management, covering why hardcoded credentials represent a critical exposure. ↩
- CWE-798: Use of Hard-coded Credentials — Common weakness definition for embedding credentials directly in source code or configuration artifacts. ↩
Questions from the audience
Related deep dives
Breaking AI Agents: Exploiting Managed Prompt Templates to Take Over Amazon Bedrock Agents
When Passports Execute: Exploiting AI Driven KYC Pipelines | [un]prompted 2026
Agents Exploiting Auth-by-One Errors | [un]prompted 2026
