When a physical AI system makes the wrong decision, there is no Control-Z. Physical AI security introduces kinetic risk — the point where a model’s error becomes force, motion, and irreversible real-world consequence. A robot arm that grips too hard, an autonomous vehicle that fails to brake, a drone that executes an unauthorized action: these are not data breaches you can patch overnight.
For security engineers, this shift from digital to physical attack surfaces demands an entirely new threat model. This post breaks down the five-layer physical AI stack, explains why latency becomes a safety parameter at sub-millisecond control loops, and examines where current governance frameworks like NIST AI RMF and the EU AI Act fall dangerously short of addressing autonomous physical systems.
Key Takeaways
- You will understand why latency is not a performance metric but a safety parameter in physical AI — and why a correct decision delivered milliseconds late can still cause irreversible physical harm.
- You will be able to identify where liability sits in the physical AI stack and why the control layer — not the model or the sensor — is the last accountable actor before kinetic damage occurs.
- Apply this to design governance that lives inside the execution architecture, not outside it — because external compliance frameworks like NIST AI RMF and the EU AI Act were not built for physical AI and leave critical runtime gaps.
The Physical AI Stack: How Autonomous Systems Perceive, Reason, and Act
AI/ML security begins with understanding the architecture that connects raw sensor data to real-world physical actions. Unlike digital AI systems where errors produce incorrect text or flawed recommendations, physical AI errors move arms, steer vehicles, and operate industrial machinery. To reason about the attack surface, you first need a precise map of the stack.
The Five-Layer Physical AI Stack
Padma Apparao describes physical AI as a closed loop — not a pipeline that terminates, but a continuous cycle from sensing to acting and back again. That loop consists of five distinct layers:
Layer 1 — Sensors The base of the stack. Cameras, LiDAR, microphones, accelerometers, and environmental sensors collect raw data from the physical world. Sensors are passive data collectors — they have minimal inherent intelligence and, critically, minimal accountability. They gather and pass data upward. However, sensors are the first and most accessible attack surface: spoofed GPS signals, blinded LiDAR, adversarial patches in camera fields of view, or simply faulty hardware can corrupt the entire stack from the bottom up.
Layer 2 — Perception The perception layer makes sense of raw sensor data. It resolves conflicts between sensors (which sensor is trustworthy when two disagree?), correlates data across modalities, and produces a coherent environmental representation. This layer must handle spoofed or incorrect sensor inputs and determine which signals are valid. A perception failure here does not just cause one sensor to be wrong — it can cause the entire system’s understanding of reality to diverge from actual physical conditions.
Layer 3 — World Model This is the predictive reasoning core. The world model takes the perception layer’s output — the current state of the environment — and predicts: what will happen next if I take a given action? Apparao illustrates this with a concrete example: a robot tasked with picking up a coffee cup from a table. The world model must determine the cup’s material (ceramic vs. paper), whether it has a handle, the surface type it sits on, and how much grip force is appropriate. It then predicts the outcome of a proposed grasp. If that prediction is wrong — too much force on a fragile cup, wrong grip angle — the coffee spills. The world model’s error becomes a physical consequence.
Layer 4 — Reasoning and Planning (LLMs and VLMs) Large language models (LLMs) and vision-language models (VLMs) occupy this layer. LLMs reason about sequences of actions: given the current world state, what plan achieves the objective? VLMs extend this by grounding reasoning in visual input — given what I see and what I read, what does this mean in the physical context? This layer translates high-level intent into a structured action plan. It operates at latencies (hundreds of milliseconds to seconds) that are fundamentally mismatched with the physical control loop’s requirements.
Layer 5 — VLA Control (Vision-Language-Action Models) The final layer converts reasoning output into precise physical commands. Vision-language-action (VLA) models[1] determine the exact motor instructions: how many degrees to rotate a joint, how much force to apply, at what velocity to move. This layer directly interfaces with actuators — motors, hydraulics, servos. It is the point of no return: once a VLA issues control commands to a physical system, reversal becomes impossible or physically damaging. This is the layer that makes physical AI fundamentally different from any prior class of software system.
How the Stack Maps to Earlier AI Architectures
The five-layer loop is not a new concept — its roots trace back to control theory and robotics from the 1950s through 2000, where world models predicted next system states from current observations. What has changed is the maturity convergence of three independent technology vectors that have simultaneously crossed a deployment threshold:
- Sensor maturation: LiDAR, computer vision, and multi-modal sensing are now precise, cheap, and reliable enough for real-world deployment.
- Foundation model stability: Frontier LLMs and VLMs have reduced hallucination rates and improved reasoning quality to a level suitable for safety-adjacent decisions.
- Edge compute viability: Sufficient computational power is now deployable at the point of action — on the robot, in the vehicle, at the industrial site — without requiring round-trips to cloud infrastructure.
These three vectors converging simultaneously is what has made autonomous physical systems a present deployment reality rather than a research prototype.
Why Layer Boundaries Matter for Security
Each layer boundary is a trust boundary. Sensor data flows upward without cryptographic attestation. World models operate on perception outputs they cannot independently verify. LLMs reason over states they accept as ground truth. VLA models execute plans without re-verifying the assumptions that generated them.
This architecture means that a compromise or failure at Layer 1 (sensors) propagates unchecked through Layers 2, 3, and 4 before manifesting as a physical action at Layer 5. The stack amplifies errors, it does not attenuate them. For a security engineer, this is the foundational threat model: the physical AI stack is a trust amplifier, and every layer that passes corrupted data upward increases the probability of a kinetic outcome at the top.
Coffee Cup Grasping Failure: World Model Error Causing Physical Consequence
Proof of Concept
-
Sensor data ingestion: The robot’s sensors (cameras, lidar, depth sensors) capture the scene — a coffee cup sitting on a table surface.
-
Perception layer processing: The perception layer aggregates and correlates sensor inputs, filtering out noise and resolving any conflicting readings to build a coherent representation of the objects in the environment.
- World model inference (failure point): The world model receives the perception output and must infer multiple physical properties before any grip command is issued:
- Cup material: Is it a paper cup (compressible, lightweight) or a ceramic mug (rigid, heavier)?
- Fragility: How much grip force can the object withstand before deforming or breaking?
- Grip location: Does the cup have a handle? If so, grasping the handle is the correct approach. If not, where on the body should the gripper make contact to avoid tipping?
- Fill state: Is the cup full of hot liquid? An incorrect fill-state estimate changes the required grip force and tilt tolerance.
- Surface type: Is the table a soft surface (more friction, forgiving of minor grip errors) or a hard surface (object slides more easily if nudged)?
-
Incorrect prediction output: Suppose the world model classifies the ceramic mug as a paper cup. It predicts that a light, low-force grip will be sufficient to lift and carry the object. It selects a grip point on the body of the cup rather than the handle, and generates a predicted outcome: cup lifted cleanly, no spill.
-
LLM/VLM reasoning layer: The reasoning layer accepts the world model’s state prediction and plans a sequence of actions — move arm to position, close gripper to specified force, lift, translate, lower. Because the world model prediction is wrong, the planned sequence is built on a false premise, but the reasoning layer has no mechanism to detect this.
-
VLA control command issuance (point of no return): The VLA model translates the plan into precise motor commands: exact arm angles, gripper closure force in Newton-meters, trajectory coordinates. Once these commands are issued to the physical actuator, execution begins. At this point you cannot reverse the action — the physical consequence is already in motion.
-
Kinetic consequence: The gripper closes with too little force (paper-cup assumption applied to ceramic), or contacts the wrong point on the cup body, causing the cup to tip or be knocked over. Hot liquid spills. The physical AI system has caused an irreversible real-world event — not because of a cyberattack or sensor spoofing, but because the world model’s internal prediction of physical reality was wrong.
-
Why this failure cannot be caught late in the stack: There is a window in the pipeline — after world model prediction, before VLA command issuance — where an error could theoretically still be detected and the action aborted. Once the control layer has issued the motor commands, no corrective intervention is possible. Safety checking must occur before the action command crosses into the physical execution domain, not after.
- Security and governance implication: This failure mode is classified as an accidental (non-adversarial) kinetic risk — the world model failed because of an incorrect inference, not because of injected adversarial data. However, the same failure pathway can be weaponized: an attacker who can manipulate sensor data or poison the world model’s training inputs can cause the system to predictably misclassify object properties and trigger specific physical consequences on demand.
Actionable Takeaways
- Map every physical AI system you evaluate against the five-layer stack (sensors, perception, world model, LLM/VLM reasoning, VLA control) before threat modeling. Each layer boundary is a trust boundary — identify what data crosses it, whether it is validated, and what assumptions the receiving layer makes about its integrity.
- Treat sensor spoofing and perception layer manipulation as high-priority attack vectors. Because the world model and reasoning layers accept perception output as ground truth, a low-cost sensor attack (GPS spoofing, LiDAR blinding, adversarial visual patches) can drive physically harmful decisions at Layer 5 without triggering any model-level anomaly detection.
- When assessing VLA control layer risk, determine the system's "point of no return" — the moment after which an issued command cannot be safely reversed. Governance controls and override mechanisms must be positioned before this boundary, not after it.
Common Pitfalls
- Treating physical AI security as equivalent to digital AI security. Security engineers familiar with LLM safety (prompt injection, jailbreaks, output filtering) may focus exclusively on the reasoning layers (LLMs/VLMs) while ignoring the sensor and perception layers, which are the most accessible and hardest-to-detect attack surfaces in a deployed physical system.
- Assuming that model accuracy guarantees system safety. A world model or LLM that is 99% accurate under lab conditions operates on perception inputs that may be degraded, spoofed, or out-of-distribution in real deployments. Stack-level safety cannot be inferred from component-level accuracy metrics.
Kinetic Risk: Why Physical AI Failures Are Not Reversible
Kinetic risk is the term that defines what separates physical AI security from every other domain in cybersecurity. In the digital world, a bad decision — a misconfigured rule, a compromised model output, a hallucinated response — can almost always be undone. You roll back the deployment. You revoke the token. You patch the service. The damage is logical, not physical, and the system state can be restored.
When physical AI systems make the wrong decision, that recovery path disappears. The errors are no longer information artifacts — they are force, motion, and consequence in the real world. A robot arm that misclassifies a fragile cup and grips too hard has already shattered it before any override signal can arrive. An autonomous vehicle that fails to recognize a pedestrian crossing does not produce a log entry you can roll back — it produces a collision. This is what Padma Apparao defines as kinetic risk: the moment AI’s mistakes move from the screen into the physical world.
Digital vs. Physical: The Irreversibility Gap
The contrast is stark and matters deeply for how security engineers design threat models:
- Digital AI failure: Acts on information. Outputs are text, decisions, recommendations. A wrong output can be corrected before it causes downstream harm. You have the privilege of Control-Z.
- Physical AI failure: Acts on matter. Outputs are motor commands, joint torques, navigation vectors, actuator signals. Once issued, those commands execute against physics — and physics does not accept undo requests.
Apparao frames this with a memorable analogy: digital AI is like playing with funny money. You have paper gains, paper losses — nothing is real until you realize the value. Physical AI is where you realize the value, and the losses are irreversible.
Two Distinct Failure Modes: Accidental and Intentional
Physical AI systems face two categories of failure, both of which carry kinetic consequences:
1. Accidental failure (safety failures) These occur when the system behaves incorrectly due to sensor errors, model misprediction, or edge-case conditions the system was not designed to handle. The classic example: a GPS giving a “make a U-turn” instruction when no safe U-turn exists. The GPS is providing advice, not checking whether the action is physically safe. In an autonomous system with no human in the loop, that advice becomes an execution command — and the system may act on it even if the real-world context makes it dangerous.
2. Intentional failure (security attacks) These occur when an adversary deliberately compromises one or more layers of the physical AI stack — spoofing sensors, injecting malicious inputs into the perception or reasoning layers, or subverting the control layer directly. The result is a system that fails not by accident but by design. An attacker who can manipulate what the world model believes about the environment can cause the system to take targeted, harmful physical actions that look indistinguishable from a normal operation until the damage is done.
Both failure modes demand a fail-safe architecture that treats kinetic risk as a first-class design constraint — not as an edge case to be documented in an incident report after the fact.
Why the Point of No Return Matters
The physical AI loop — from sensing through perception, world modeling, reasoning, and finally control and action — contains a critical threshold. Up to a certain point in that pipeline, a decision can still be reversed. The world model can recalculate. The LLM can revise its plan. But once the VLA model and the control layer issue motor commands to the physical system, it is too late. The action has been committed to the physical world.
This point of no return has two implications for security engineers:
- Attack surface prioritization: Attacks that compromise layers earlier in the pipeline (sensors, perception) may be detectable and correctable before action is taken. Attacks that compromise the control layer — or that race to inject commands after the point of no return — are the highest-severity class of physical AI vulnerability.
- Defense-in-depth requirements: Because the last defense before kinetic harm is the control layer, that layer must have the most robust validation, the tightest scope constraints, and the most reliable override mechanisms. Treating it as just another component in the stack is a critical design error.
The Speed Constraint That Disables Human Oversight
One of the most operationally significant aspects of kinetic risk is the speed at which it materializes. Human-in-the-loop reaction time operates in the range of hundreds of milliseconds to seconds — and that is under ideal, alert conditions. Physical AI control loops operate at sub-100ms cycle times. The gap between human reaction speed and machine execution speed means that human-in-the-loop cannot function as a safety control in physical AI execution paths.
This is not a future limitation — it is a current architectural reality. By the time a human perceives an anomaly, interprets the situation, and initiates an override, multiple control cycles have already completed. The kinetic event has already occurred.
GPS U-Turn Command: Advice vs. Permission in Physical AI Decision Chains
Proof of Concept
-
GPS issues advisory command: The navigation system computes a route correction and emits a “make a U-turn” command. This output is generated purely from map data and route logic — the system has no sensor input about the physical state of the road (oncoming vehicles, barriers, pedestrians, road surface).
-
Human-in-the-loop intercept (legacy scenario): In a human-driven vehicle, the driver receives the command as advice. The human applies real-world perception — checks mirrors, assesses traffic, evaluates road conditions — and decides whether the action is safe to execute. The human is the permission layer. The GPS command alone does not actuate anything.
-
Failure mode in physical AI (autonomous scenario): In an autonomous system where the GPS or routing module feeds directly into a VLA model or a control layer, the advisory command can be passed down the stack toward actuation without a human permission checkpoint. If the perception layer or world model does not have sufficient sensor coverage to detect the unsafe condition (e.g., oncoming vehicle in the opposing lane), the control layer may issue the U-turn maneuver command to the physical actuators.
-
The “advice vs. permission” architectural gap: This is a governance and architecture problem: the GPS system was never designed to grant permission — only to provide routing advice. When physical AI pipelines collapse the advice layer and the permission layer into a single flow without an interstitial safety check, the system becomes unsafe by design. As Apparao notes: “It is just saying you need to make a U-turn. It’s up to the human being to get into that loop and figure out whether it is safe to make the action.”
-
Partial hardware mitigation: Some modern systems include hardware sensors that can detect conditions blocking an action and delay execution. This represents a partial architectural safeguard — a sensor-driven interlock that can veto a control command when a physical constraint is detected. However, sensor coverage gaps, spoofed sensor data, or latency in the sensor-to-veto path can all allow unsafe commands to pass through.
-
Kinetic risk outcome: If the U-turn command reaches the actuation layer without being vetoed by a valid perception or world model check, the vehicle executes the maneuver into unsafe conditions. Unlike a digital error (which can be rolled back), the physical action — a vehicle turning into oncoming traffic — is irreversible.
-
Governance implication: Every command in the decision chain must be evaluated not just for correctness (is this the right route?) but for permission (is it safe to execute this action right now, given current real-world state?). Fail-safe architecture must embed this permission check inside the execution stack — not rely on a human who can no longer be in the loop at machine speeds.
Actionable Takeaways
- Redesign your physical AI threat model to treat the control layer as the highest-severity attack surface. Unlike upstream layers (sensors, perception, world model), a compromise or error at the control layer executes directly into the physical world with no remaining opportunity for correction. Prioritize integrity guarantees, scope constraints, and override mechanisms at this layer first.
- Classify your physical AI failure modes explicitly into accidental (safety) and intentional (security) categories during architecture review. Each class demands different mitigations — graceful degradation and fail-safe defaults address accidental failure; input validation, spoofing resistance, and adversarial robustness address intentional attacks. Conflating them leads to architectures that protect against one while leaving the other unaddressed.
- Remove humans from the real-time execution loop and reposition human oversight at the boundary-setting and pre-execution planning level. Design rate limiters, geo-fencing, energy caps, and kill-switch logic to operate autonomously within the architecture — these are the machine-speed equivalents of human judgment, and they must be present and enforced by the system itself, not applied reactively by an operator.
Common Pitfalls
- Assuming that model accuracy guarantees physical safety. A model can be 100% correct in its classification or prediction and still cause kinetic harm if the system executes that correct decision after the physical point of no return — for example, correctly identifying a stop sign but issuing the brake command 50ms too late to prevent a collision. Accuracy and safety are not the same parameter in physical AI; latency is the coupling variable between them.
- Positioning human review as a runtime safety control for physical AI systems. Human reaction time (hundreds of milliseconds to seconds) is orders of magnitude slower than physical AI control loop cycles (sub-100ms). Designing an architecture that assumes a human can intervene to prevent kinetic harm in real time is not a safety feature — it is an untested assumption that will fail in production. Human authority must be exercised before execution, not during it.
Latency as a Safety Parameter in Real-Time AI Control Loops
In traditional AI systems, latency is treated as a performance metric — you optimize time-to-first-token (TTFT) or tokens-per-second to improve user experience. For real-time AI safety, this framing is dangerously wrong. Latency in physical AI is a hard safety boundary: a correct decision delivered too late is functionally indistinguishable from a wrong decision. The physical harm is the same either way.
The Scaling Inversion: VLAs vs. LLMs
Large language models scale with parameter count. The historical trajectory — 7B, 15B, 45B, 175B parameters, context windows expanding from 4K to 1M tokens — reflects a model quality improvement paradigm. More compute, bigger model, better outputs. Latency in this context was secondary.
Vision-Language-Action (VLA) models[1] invert this entirely. VLAs do not scale with model size. They scale with speed of response. The relevant performance dimension is not how many tokens a model can process, but how fast the entire perception-to-action loop completes. Consider the pipeline timing requirements:
- Vision encoder models: must process incoming sensor data at sub-millisecond latency for individual components
- World model inference: must complete within tight timing windows to remain actionable
- Full sense-to-action loop: must complete within 100–200 milliseconds total end-to-end
An LLM operating in the hundreds-of-milliseconds-to-seconds range is entirely appropriate for text generation. The same latency profile applied to a physical control loop creates a safety failure — not a performance degradation.
The Pedestrian Scenario: Correct but Too Late
The talk makes this concrete with an autonomous vehicle example. A vehicle’s physical AI system correctly identifies a pedestrian jaywalking and determines the correct action: brake. The model’s decision is accurate. The plan of action is sound. But if the reaction time through the entire sensing-to-control pipeline is even a few hundred milliseconds too slow, the vehicle strikes the pedestrian — despite the model being 100% correct.
This is the fundamental safety insight: accuracy is necessary but not sufficient. Timing is a co-equal safety property, not an optimization variable.
The Point of No Return in the Control Loop
Within the five-layer physical AI stack, there exists a point of no return where reversibility ends. This boundary sits between the reasoning layers and the action execution layer:
- Before the point: The world model has assessed state, the LLM has produced a plan, and the VLM has interpreted the environment. Actions can still be reconsidered or halted. This region is recoverable.
- At the point: The VLA model and the hardware control layer have issued action commands to the physical system — robot arm movement, vehicle steering, actuator engagement. Once these commands are issued, they cannot be retracted. Physical momentum and mechanical response mean the action is already executing.
Security engineers designing physical AI systems must treat this boundary as a hard architectural constraint. Any monitoring, override, or safety intervention must operate upstream of this point — or it is too late.
Why Human-in-the-Loop Fails at Machine Speed
The standard security and governance instinct is to add a human approval step for high-stakes actions. In physical AI, this instinct is architecturally incompatible with real-time control requirements. Humans operate in the seconds-to-minutes range. Physical AI control loops operate in the sub-100ms range. These timescales are incompatible by roughly three orders of magnitude.
This is not a training or interface problem — it is a physics and cognition constraint. No amount of UI optimization closes a gap of 100x to 1000x in reaction time. The practical consequence for system design:
- Humans cannot be in the execution path of a real-time physical AI control loop
- Human authority must be upstream of execution: designing boundaries, defining intent, configuring constraints, setting operating envelopes
- The autonomous execution stack itself must enforce safety within those pre-set boundaries — without human intervention in the loop
This is not a reduction in human control; it is a structural repositioning of where human authority is exercised. Humans retain strategic authority over the architecture, governance parameters, and operational boundaries. They cede real-time execution authority to the autonomous stack by necessity — because the physics of the problem demand it.
Bounded Latency as a Governance Primitive
From a governance and fail-safe architecture standpoint, latency becomes a configurable safety primitive — not just an engineering constraint. Governance frameworks for physical AI systems must define:
- Maximum acceptable latency for each stage of the control loop (sensing, perception, world model inference, VLA output, actuator response)
- Degradation behavior when latency thresholds are exceeded: does the system halt, slow down, or transition to a more conservative operating mode?
- Deterministic control paths that guarantee bounded response times, separate from probabilistic ML inference paths
The distinction between deterministic and probabilistic components in the architecture is critical. Safety-critical timing guarantees cannot rely on non-deterministic ML inference alone — they require hardware and software layers with provable latency bounds.
Actionable Takeaways
- Treat latency as a safety specification, not a performance target: define maximum acceptable end-to-end control loop latency for your physical AI system (typically sub-100ms for VLA-controlled systems) and enforce it as a hard constraint in system requirements — not an optimization goal.
- Architect human oversight upstream of the execution loop: position human authority at the governance and boundary-setting layer (defining operating envelopes, speed limits, geo-fences, and degradation policies) rather than in real-time approval paths. Design the autonomous stack to enforce those pre-set boundaries without requiring live human intervention.
- Identify and explicitly mark the point of no return in your control loop architecture: document which layer issues irreversible action commands and ensure all safety monitoring, override logic, and anomaly detection operates upstream of that layer, while there is still time to intervene.
Common Pitfalls
- Treating latency as a performance KPI rather than a safety parameter: teams that benchmark physical AI systems the same way they benchmark LLMs — optimizing for throughput and average latency — miss the safety-critical nature of worst-case latency bounds. A system that is fast on average but occasionally exceeds its safety timing window is a system that will eventually cause harm.
- Designing human-in-the-loop approval gates into real-time physical AI execution paths: this is architecturally incompatible with sub-100ms control loops. Human reaction times are in the seconds-to-minutes range — inserting a human approval step into execution does not add safety, it adds latency that guarantees the system cannot meet its timing requirements and will fail to act in time.
Fail-Safe Architecture and Governance for Autonomous Physical Systems
Physical AI security does not permit the same recovery paths as digital systems. In software, a bad state can be rolled back, rebooted, or patched. When an autonomous systems security failure occurs, the consequences — broken components, physical injury, equipment damage — are often irreversible. This is why fail-safe architecture is not a performance concern or a compliance checkbox; it is the foundational design requirement for any system that translates AI inference into real-world motion.
Two categories of failure apply:
- Accidental failure — the system behaves incorrectly due to sensor error, model hallucination, or edge cases in the environment. The classic example is a GPS that commands a U-turn without knowledge of physical road conditions. The system is giving advice, not exercising judgment about whether that action is physically safe.
- Intentional failure (security attack) — an adversary spoofs sensor data, injects malicious inputs, or exploits a vulnerability in the inference pipeline to cause the system to take harmful action deliberately.
Both failure modes demand the same architectural response: a fail-safe layer that constrains what the system is permitted to do, regardless of what its models recommend.
Engineering Primitives of a Fail-Safe Architecture
The following concrete primitives form the foundation of a robust fail-safe design:
1. Physical kill switches and override buttons Hard-wired interrupt mechanisms that bypass the software stack entirely. These exist outside the inference loop and can halt motion regardless of model state. They are the last line of defense when all software-layer controls have failed.
2. Graceful degradation Rather than a hard system failure that causes abrupt, dangerous motion, the system should be designed to degrade gracefully — reducing capability incrementally rather than halting suddenly. This mirrors reliability engineering principles well-established in aerospace and industrial control systems.
3. Deterministic control partitions Critical safety-enforcement logic — boundary checks, maximum force limits, velocity caps — should run in deterministic code paths, isolated from probabilistic model inference. This ensures that even if the AI layer produces an unsafe recommendation, the control partition can reject or constrain it before the physical actuator executes.
4. Bounded latency The entire sense-to-act loop must complete within a defined time window. If the system cannot guarantee that a safety check or override fires within the required latency budget, that latency budget is itself a safety constraint. A deterministic timer that aborts an action if a safety response has not completed within threshold is a standard pattern borrowed from hard real-time systems.
5. Rate limiters, energy caps, and geo-fencing These are runtime governance guardrails that constrain what actions the system is authorized to take. A geo-fence prevents the system from operating outside a defined physical boundary. An energy cap limits the maximum force or power the actuator can apply. Rate limiters prevent rapid, unchecked sequences of actions. All three are examples of human-defined authority expressed as hard constraints in the execution environment — not suggestions, not model outputs, but enforced limits.
The Point of No Return: Why Governance Must Be Embedded
One of the most operationally significant observations is the concept of a point of no return in the physical AI pipeline. Between sensing and action, there is a window in which the system can still reverse course — the world model has predicted a state, the LLM has proposed a plan, but the VLA and control layer have not yet issued actuator commands. Once the VLA model fires and the robot arm or vehicle begins moving, reversal is no longer physically possible within the required time frame.
This architectural reality has a direct implication for governance: governance cannot be an external audit or a post-deployment compliance review. It must be embedded inside the execution stack, running at the same latency scale as the system itself. A governance layer that operates in seconds cannot protect a system whose control loop runs in milliseconds.
The speaker is explicit: governance should be inside the system, setting the operational boundaries, not sitting outside and evaluating paperwork after the fact.
Where NIST AI RMF Falls Short for Physical AI
The NIST AI Risk Management Framework (AI RMF)[2] defines four functions: govern, map, manage, and measure. It is a well-structured framework for reasoning about AI risk at the organizational level. However, it was not designed for real-time physical AI systems, and the gap is significant:
- The NIST AI RMF focuses on model attestation, model accuracy, and data purity — all pre-deployment or batch-evaluation concerns.
- It has no provisions for real-time runtime monitoring of physical AI systems during operation.
- It does not define what runtime controls should be present in an autonomous physical system, nor how they should be enforced.
- There is no guidance on kinetic failure modes — what happens when a high-confidence model output results in physical harm.
For a security engineer designing an autonomous system, the NIST AI Risk Management Framework provides useful organizational scaffolding but leaves a critical gap at the execution layer where physical safety is actually enforced.
Where the EU AI Act Falls Short for Physical AI
The EU AI Act[3] classifies AI systems by risk category: low, medium, high, and ultra-high risk. This classification approach is reasonable for many AI deployments but breaks down for physical AI for a structural reason: the Act explicitly defers governance of physical AI systems to sector-specific regulations.
In practice, this means that if an autonomous robot is deployed in a hospital, medical device regulations govern its behavior. If it operates in a vehicle, automotive safety standards apply. The EU AI Act itself provides no unified framework for physical AI runtime safety — it outsources that responsibility to domains that may not yet have developed the necessary standards for AI-driven autonomous systems.
The technology is outpacing the regulatory frameworks governing it. Physical AI is scaling rapidly, but governance remains fragmented, sector-dependent, and largely silent on the real-time control requirements that matter most.
The Governance Gap: Technology Outpacing Policy
The core challenge is structural: the pace of innovation in physical AI — across sensors, frontier models, and edge compute — is significantly faster than the pace at which governance frameworks evolve. New capabilities are deployed before regulations can define the boundaries under which they should operate.
The historical parallel is instructive. Centralized air traffic control did not exist before the 1950s. It was implemented in response to fatal mid-air collisions. The flight data recorder (black box) became mandatory after crashes where investigators had no way to reconstruct what happened. In both cases, governance infrastructure followed catastrophic failure rather than preceding it.
The implication for physical AI: security and governance engineers should not wait for regulatory mandates. If governance is not designed into the architecture, the system will eventually produce an accident report, a lawsuit, or a headline that forces the issue retroactively — at far higher cost.
Designing Governance as a Runtime Primitive
The prescription is clear: governance must be enforced at runtime, inside the execution architecture. This means:
- Continuous runtime monitoring — not periodic audits or batch model evaluations, but live telemetry that tracks system behavior during operation.
- Embedded authority boundaries — rate limiters, geo-fences, energy caps, and force limits defined as enforced constraints, not advisory limits.
- Escalation paths that are machine-speed — if a governance condition is violated, the system must be able to transition to a safe state autonomously, without waiting for human authorization.
- System-level certification — not just model-level certification from an AI vendor, but end-to-end certification of the deployed physical system: sensors, perception, world model, reasoning layer, control layer, and all governance primitives together.
DHS Drone Friendly-Fire Incident: Human-in-the-Loop Failure at Machine Speed
Proof of Concept
-
Context — autonomous drone operation under DHS authority: A Department of Homeland Security drone was launched as part of an authorized operational mission. The drone represented a physical AI system operating in the real world with kinetic potential — its actions could cause irreversible physical consequences.
-
The human-in-the-loop dependency — mandatory inter-agency notification: The existing governance model required that DHS inform the Pentagon before launching the drone. This is a textbook human-in-the-loop control: a human actor (or human-staffed agency) must explicitly authorize or be notified of an action before or as it occurs.
-
The failure mode — communication breakdown at machine speed: The required human coordination step failed. The notification either did not occur, was not received in time, or was not acted upon within the operational window. As Apparao described it: “that was a human-in-the-loop problem — why did the Department of Homeland Security have to inform the Pentagon that they were launching a drone.”
-
The consequence — friendly-fire outcome: The drone was brought down. The failure of the human notification loop — a process designed for human-timescale coordination — directly caused a friendly-fire incident. The system that should have provided a safety check instead introduced a point of failure.
-
The architectural lesson — human approval in the execution path cannot scale to machine speed: Requiring a human in the execution loop for a time-sensitive autonomous action is fundamentally broken by design. The human coordination step assumes humans can act within the operational tempo of the system. At machine speed — where physical AI control loops operate in sub-100ms ranges — humans cannot. Apparao’s explicit conclusion: “You don’t need a human in the loop to say okay you have the permission to do it. This should have happened automatically.”
-
The correct governance model — humans set boundaries, machines execute: Rather than placing a human in the execution path (where their latency guarantees failure under time pressure), governance should be designed so that humans define the authorization boundaries, scope, rules of engagement, and escalation triggers in advance. The machine then executes autonomously within those pre-defined constraints. The human authority is exercised at design time and policy time — not at execution time.
-
The broader implication — physical AI governance frameworks must reflect operational tempo: This incident illustrates that governance models inherited from slower, human-paced operational environments are directly dangerous when applied to autonomous physical systems. The fix is not to add more human checkpoints in the execution chain — it is to embed governance logic (rate limiters, geo-fencing, scope boundaries, escalation paths) directly into the system architecture so that autonomous execution is bounded, auditable, and self-enforcing without requiring a human to be present in the loop at the moment of action.
Actionable Takeaways
- Embed governance primitives — rate limiters, geo-fencing, energy caps, bounded latency timers — directly inside the execution stack as hard-enforced constraints, not advisory recommendations. These must run at the same latency scale as the control loop they protect.
- Do not treat NIST AI RMF or EU AI Act compliance as a substitute for runtime physical AI safety. Both frameworks have critical gaps for autonomous physical systems: neither addresses real-time monitoring, runtime control enforcement, or kinetic failure modes. Use them as organizational scaffolding, then build the execution-layer controls independently.
- Design deterministic control partitions that are isolated from probabilistic model inference. Safety-enforcement logic — force limits, velocity caps, boundary checks — should execute in deterministic code paths so that an unsafe model output can be intercepted and rejected before it reaches the actuator.
Common Pitfalls
- Treating external compliance frameworks (NIST AI RMF, EU AI Act) as sufficient governance for physical AI systems. Both were designed for digital AI contexts and explicitly lack provisions for real-time runtime monitoring and control enforcement. Relying on them without supplementing with embedded runtime governance leaves the execution layer unprotected.
- Assuming human-in-the-loop oversight is a viable safety mechanism for physical AI. Human reaction time operates in seconds to minutes; physical AI control loops operate in milliseconds. Designing governance that depends on human real-time intervention will fail at machine speed — humans must set boundaries before execution, not intervene during it.
Accountability and Liability in the Physical AI Stack
When a physical AI system causes injury, property damage, or a fatality, the question of accountability is not theoretical — it will be litigated. The challenge is that a physical AI stack is a layered system with multiple contributors: sensor manufacturers, world model developers, LLM/VLM providers, VLA integrators, and the operators who deploy the finished system. Assigning liability in physical AI requires understanding how responsibility flows through that stack at the moment of a kinetic event.
Sensors are largely blameless. They collect and pass data — that is their entire function. They carry minimal responsibility. The world model carries partial responsibility — it can only act on what it perceives, and its predictions are constrained by the quality of sensor input and the fidelity of its internal model of the world.
Liability Flows Up to the Control Layer
The control layer is the ultimate accountable actor. This is the layer that issues the final action commands to the physical system — instructing a robot how far to move an arm, at what angle, with what force. It is the last point in the pipeline where harm could have been prevented. As Apparao puts it: “The last actor that could have prevented that action is liable for any physical damage.”
This has direct implications for anyone integrating or deploying autonomous systems. The organization or individual responsible for the VLA and control layer carries disproportionate liability exposure compared to upstream component providers. If you are an integrator building on top of third-party sensors and foundation models, your liability is concentrated at the layer you own — the execution layer.
Why Human-in-the-Loop Cannot Absorb Liability at Runtime
A common assumption is that placing a human supervisor in the loop transfers liability back to the human operator. This assumption falls apart on technical grounds: humans cannot act in the sub-millisecond range required by physical AI control loops. Human reaction times operate in seconds and minutes; physical AI control loops operate in tens to hundreds of milliseconds.
This means that at execution time, human-in-the-loop is not a meaningful safety or accountability mechanism. The human cannot intervene before kinetic action has already occurred. What humans can do is operate in the strategic authority layer: setting system boundaries, defining operating envelopes, approving architectures, and establishing governance guardrails before deployment. The human role shifts from real-time operator to pre-execution architect of constraints.
Liability for runtime failures attaches to the control layer and the design decisions made before deployment — not to a human supervisor who could not have physically intervened in time.
Continuous Monitoring, Telemetry, and Incident Reconstruction
Establishing accountability after a kinetic AI failure requires evidence. The speaker identifies two paths to accountability: evidence and traceability. Both depend on continuous runtime monitoring — not attestations that models were accurate at training time, but ongoing telemetry capturing what the system perceived, decided, and acted upon at the moment of the incident.
Sully Hudson Landing: Incident Reconstruction and the Black Box Precedent
Proof of Concept
-
The incident: Captain Sullenberger safely ditched Flight 1549 in the Hudson River on January 15, 2009, after a bird strike took out both engines at low altitude shortly after takeoff from LaGuardia. All 155 people on board survived.
-
The reconstruction challenge: The NTSB and investigators ran 17 separate simulation trials attempting to recreate the exact conditions — altitude, airspeed, bird strike timing, engine failure sequence — to determine whether alternative courses of action (returning to LaGuardia, diverting to Teterboro) were feasible. This was not a straightforward replay; it required careful modeling of the aircraft’s energy state, pilot reaction time, and aerodynamic behavior to produce a credible reconstruction.
-
The black box precedent: Aviation did not always have flight data recorders or cockpit voice recorders. Black box regulations emerged between 1950 and 1970 specifically because aircraft were disappearing or crashing with no recoverable data about what occurred. Regulators had no way to reconstruct incidents, so they mandated continuous onboard recording of flight parameters and crew communications. The black box was a governance response to an accountability gap — not a proactive engineering decision.
-
The mapping to physical AI governance: Physical AI systems operating autonomously in the real world face the same accountability gap that aviation faced before black box mandates. If an autonomous robot, vehicle, or drone causes physical harm, investigators need to reconstruct the exact sequence of sensor inputs, model inferences, world model predictions, and control commands that led to the outcome. Without continuous monitoring and telemetry baked into the physical AI stack, that reconstruction is impossible — or relies on 17 simulation trials, as the NTSB required for the Hudson landing.
-
The governance implication: Accountability and traceability are governance primitives for physical AI. “Continuous monitoring” means capturing runtime state across the full five-layer stack (sensor readings, perception outputs, world model state, LLM/VLM reasoning outputs, and control commands issued) so that any incident can be reconstructed after the fact. This is the physical AI equivalent of a black box.
-
The policy gap: Current frameworks — NIST AI RMF[2] and the EU AI Act[3] — focus on model attestation, model accuracy, and data purity. Neither mandates runtime telemetry or incident reconstruction capability for physical AI systems. The black box precedent shows that this gap will likely only be closed reactively, after incidents force regulatory action, mirroring the historical aviation pattern.
Governance Must Be Embedded, Not External
Current frameworks — NIST AI RMF and the EU AI Act — cannot enforce runtime accountability for physical AI systems. NIST AI RMF focuses on model attestation, model accuracy, and data purity. The EU AI Act classifies systems by risk category and defers physical AI governance to sectoral regulators (e.g., medical device law governs hospital robots). Neither framework addresses real-time monitoring requirements, runtime control primitives, or the kind of continuous telemetry needed for incident reconstruction.
The gap is structural: governance written on paper and enforced externally cannot see inside a sub-100ms control loop. To make accountability real, governance must be embedded inside the execution architecture. Monitoring, boundary enforcement, telemetry capture, and escalation logic must be components of the system, not checklists filled out before deployment.
Apparao’s warning is direct: “If you don’t design governance in, you’re going to hear it in accident reports, lawsuits, and making headlines.”
The Inevitability of Failure — and What That Means for Accountability
A final, pragmatic point anchors the accountability framework: physical AI systems will fail. Zero-risk is impossible. The design goal is not to prevent all failures but to control damage, fail gracefully, and generate sufficient telemetry to reconstruct what happened. The relevant question is not whether the system will fail, but how it will fail and whether the failure can be bounded and explained.
For security engineers and system architects, this reframes accountability as an engineering discipline rather than a compliance exercise. Building in deterministic control paths, bounded latency, rate limiters, geo-fencing, and continuous telemetry is not just good systems engineering — it is the foundation of any defensible accountability posture when kinetic harm occurs.
Actionable Takeaways
- Map liability ownership across your physical AI stack before deployment: identify who owns the control layer, because that is where the highest liability exposure sits. If you are the integrator, you own the execution layer regardless of which upstream models or sensors you source.
- Design continuous runtime telemetry and incident reconstruction capability into the system from the start — not as a post-deployment add-on. Capture sensor state, perception outputs, world model predictions, and control commands with sufficient fidelity to reconstruct any incident. Treat this the same way aviation treats black box requirements.
- Shift human authority from runtime supervision (which is physically impossible at machine speeds) to pre-deployment boundary-setting: define operating envelopes, approve architectures, set kill-switch thresholds, and enforce governance guardrails as embedded components of the execution stack — not external compliance documentation.
Common Pitfalls
- Assuming human-in-the-loop supervision transfers liability at runtime. Humans cannot intervene in sub-millisecond control loops, so a human supervisor cannot meaningfully absorb accountability for a kinetic failure that occurs at machine speed. This assumption will not hold up in a post-incident legal or regulatory review.
- Treating model-level certification (accuracy attestations from model providers) as sufficient for system-level accountability. NIST AI RMF and EU AI Act compliance addresses model quality at training time — neither framework provides runtime monitoring, telemetry, or incident reconstruction capability. System-level accountability requires system-level evidence, captured at runtime.
Conclusion
Physical AI security is a fundamentally different domain from digital AI security. The threat model changes when errors produce physical consequences that cannot be undone. The five-layer stack — sensors, perception, world model, LLM/VLM reasoning, and VLA control — is not just an engineering architecture; it is a liability map and an attack surface diagram.
The three central insights from this talk hold regardless of what autonomous system you are building or evaluating:
-
Latency is a safety parameter. A correct model output delivered too late is indistinguishable from a wrong one. Design bounded latency into system requirements before you write a single line of code.
-
Governance must be inside the stack. NIST AI RMF and the EU AI Act are organizational scaffolding — they do not protect a sub-100ms control loop. Rate limiters, geo-fencing, deterministic control partitions, and continuous telemetry must be embedded components, not bolt-on compliance measures.
-
The control layer is where liability concentrates. If you integrate the execution layer, you own its failure modes. Build incident reconstruction capability from day one — not as an afterthought after an accident forces the issue.
The aviation analogy is not alarmist. Black box regulations came after crashes. Air traffic control came after collisions. Physical AI governance is at the same inflection point. The engineers and architects who build accountability and fail-safe architecture proactively will set the standard. Those who wait for the accident report will be writing it.
For further reading on related topics, explore AI/ML security threat modeling approaches, AI governance frameworks, and edge computing security considerations for systems where compute is deployed at the point of physical action.
References & Tools
- Vision-Language-Action Models (VLA) — Model class at the action output layer of the physical AI stack; converts reasoning outputs into precise physical actuator commands at sub-100ms latency. ↩
- NIST AI Risk Management Framework (AI RMF) — US framework for organizational AI risk management with four functions: govern, map, manage, measure. Lacks provisions for real-time physical AI monitoring or kinetic failure modes. ↩
- EU AI Act — European regulatory framework classifying AI systems by risk category (low, medium, high, ultra-high); defers physical AI governance to sector-specific regulations, leaving gaps for autonomous physical system runtime safety. ↩
Questions from the audience
Related deep dives
Securing Workspace GenAI at Google Speed | [un]prompted 2026
The AI Security Larsen Effect - How to Stop the Feedback Loop | [un]prompted 2026
Glass-Box Security: Operationalizing Mechanistic Interpretability | [un]prompted 2026
