A red teamer generates 10,000 subdomain permutations in seconds using a single AI prompt — then a blue team analyst uses the same model to draft a Splunk query that detects the technique before it lands. Generative AI for security operations is already reshaping what’s possible on both sides of the firewall, compressing hours of manual work into minutes.
This post distills Jason Haddix’s OWASP AppSec US 2024 keynote covering a proven 6-component prompt engineering methodology for building security bots, real red team AI use cases from live engagements, and blue team and purple team workflows you can start implementing immediately.
Key Takeaways
- You'll learn a repeatable prompt engineering methodology for building specialized AI security bots — covering persona, instructions, "weird machine tricks," related research terms, and output formatting — that consistently produces 30–35% better results than single-shot prompting.
- You'll be able to identify and apply concrete red team AI use cases including AI-driven subdomain generation, acquisition OSINT, spear phishing pretext automation, XSS mutation testing, and JS code analysis — all grounded in real engagements.
- Apply blue team and purple team AI patterns — from SIEM query bots (Splunk/ELK) and IR playbook generators to threat modeling portals and executive communication bots — to immediately accelerate your SOC and security program operations.
Generative AI Foundations for Security Practitioners
From Machine Learning to Generative AI: What Security Engineers Need to Know
Generative AI for security operations begins with understanding what you’re actually working with. LLMs are not magic — they are sophisticated autocomplete systems trained on massive datasets. As Jason Haddix puts it: “It is the fanciest autocomplete you will ever meet.” Knowing this removes the mysticism and lets you use these tools pragmatically.
The path from classical AI to today’s LLMs runs through several public milestones — IBM Deep Blue defeating chess grandmaster Garry Kasparov (1997), IBM Watson dominating Jeopardy (2011), and OpenAI’s model crushing the world’s best five DOTA 2 players in a live exhibition match. Each milestone reflected a convergence of three things: massive GPU compute (driven by the gaming industry), explosion of internet-scale training data, and increasingly sophisticated neural network architectures. The 2017 Google paper “Attention Is All You Need” introduced the Transformer architecture that underpins today’s models — if you want to understand the core mechanism, that paper (or 3Blue1Brown’s YouTube visualization) is the starting point.
The Current Model Landscape
Security practitioners have several tiers of models to choose from:
- Cloud models (proprietary): OpenAI GPT-4o[4], Anthropic Claude[5] (Sonnet/Opus/Haiku), Google Gemini — highest capability, require sending data to external APIs
- Semi-open source: Meta Llama 3.1 — runs locally or on-prem, increasingly competitive with cloud models, preferred for internal/private deployments
- Local frontends: Fabric[1] (CLI, by Daniel Miessler), LM Studio[2] (desktop GUI), Ollama[3], Open Web UI — let you route queries to local or cloud models with fine-grained control
Model selection tradeoffs matter. Haddix’s empirical guidance from building security bots:
- Claude (Anthropic): Better for coding tasks, hallucinates less, stronger at maintaining context across a large codebase
- GPT-4o (OpenAI): Better for text generation, creativity, and writing-heavy tasks; GPT o1 introduced chain-of-thought reasoning and agent-style execution
- Llama 3.1 (Meta): Go-to for internal, air-gapped, or privacy-constrained deployments; capability gap is narrowing
Knowledge Cutoffs and Hallucination: The Operational Risks
Every cloud model has a knowledge cutoff — a hard date beyond which it has no training data. Without internet search enabled, these models cannot answer questions about recent CVEs, newly released tools, or current threat actor TTPs. Always verify whether your model has live search before trusting any time-sensitive output.
Hallucination is real and domain-specific. In security contexts, a hallucinated CVE number, an incorrect CVSS score, or a fabricated API endpoint is worse than no answer — it wastes time and can mislead investigations. Mistrust but verify everything the model gives you.
Privacy Architecture for Internal AI Deployments
A common constraint in enterprise security teams is that sensitive data — pentest reports, SIEM logs, internal playbooks — cannot be sent to external cloud APIs. Two patterns address this:
Pattern 1: Inline Anonymization Proxy — Deploy a local model (e.g., Llama 3.1 via Ollama[3]) as a preprocessing layer. User queries hit the local model first, which strips PII (IP addresses, hostnames, names), reconstructs a sanitized query, forwards it to the cloud model, and injects any private context back into the response on return.
Pattern 2: Azure OpenAI Service — If your organization already uses Microsoft Azure and has enterprise agreements in place, Azure OpenAI hosts GPT-4o[4] under your existing privacy and data handling contracts — a pragmatic middle ground for teams hesitant about cloud AI.
For fully air-gapped environments, open-source models via LM Studio[2] or Ollama[3] remain the only option — at a capability cost that is shrinking rapidly.
Actionable Takeaways
- Choose your model based on task type, not brand loyalty: use Claude for code-heavy security tooling (less hallucination, stronger context retention) and GPT-4o for written outputs like reports, phishing pretext, or policy documents.
- Before relying on any model output for security decisions, check its knowledge cutoff date and verify claims independently — especially for CVE details, tool syntax, and API behavior that may have changed post-training.
- For internal deployments handling sensitive data, implement an anonymization proxy layer using a local Llama model or route through Azure OpenAI — don't send raw pentest reports or SIEM logs to public cloud APIs without a privacy architecture in place.
Common Pitfalls
- Treating single-shot model responses as authoritative — Haddix explicitly calls out the "first try fallacy": even well-prompted models can produce poor output on the first attempt. Prompt at least five times for any result you plan to act on, and compare variance across responses.
- Using pre-jailbroken or heavily post-trained open-source models (e.g., "unrestricted" Llama variants from Hugging Face) to bypass content filters — Haddix notes these models have degraded embeddings and consistently perform worse than prompt-engineering around filters in the original model.
Security Bot Development with Prompt Engineering
Why System Prompts Outperform Single-Shot Queries
The core insight behind security bot prompt engineering is simple: every token you add to a system prompt narrows the model’s output distribution toward the answer you actually want. When a user sends a bare query to GPT-4o[4], the model draws from its entire training corpus. When that query arrives wrapped in a detailed system prompt defining role, expertise, methodology, and output format, the model’s effective weight space collapses toward expert-level responses in that specific domain.
Jason Haddix quantifies this: his structured prompt methodology produces results 30–35% better than single-shot prompting on the same question. That’s not a marginal gain — it’s the difference between a bot you rely on daily and one you tried once and abandoned.
The Six-Component Bot Methodology
Haddix’s system prompt structure has six components. Components 1 and 6 are optional for internal/API use; components 2–5 are the core:
Component 1: Prompt Protection (GPT Store only) If publishing to the ChatGPT store, add instructions telling the bot to protect its system prompt from extraction attempts. Not needed for API-based bots.
Component 2: Identity and Purpose (Persona) Give the bot a specific role and make it an expert. The persona shapes the model’s entire output distribution. Key principle: be specific about domain and goal. “Security expert” is weak. “Expert OSINT analyst specializing in corporate acquisition research using Crunchbase, PitchBook, and SEC filings” is strong.
Component 3: Instructions — “Weird Machine Tricks” + Flow State This is the most counterintuitive component. Prompt engineers have empirically discovered ~15 phrases that measurably improve LLM output quality — independent of the model’s designers. Haddix includes 2–5 of these in every bot:
- “Take a deep breath, relax, and enter a state of flow as if you’ve just taken Adderall mixed with amphetamine salts.”
- “If you follow all instructions and exceed expectations, you’ll earn a giant bonus.”
- “You have 20 years of experience in [domain].” (Even if the domain hasn’t existed that long — it still works.)
- “Work step by step.”
These phrases co-occur in training data with high-quality outputs — expert advice, peer-reviewed writing, careful technical documentation — nudging the model’s attention toward that class of output.
Component 4: Main Task Instructions The heart of the system prompt. Define exactly what the bot needs to accomplish, broken into explicit steps: what is it trying to accomplish, what are the ordered steps, what correct approaches should it use for predictable scenarios, and what sources should it prioritize.
Example from the Acquisition bot:
“When asked about a company’s acquisitions, reply by formatting each acquisition per line with the date of acquisition and the source you found it from, sorted from recent to oldest. Try to find at least 25 acquisitions.”
Component 5: Related Research Terms
The SEO equivalent for LLM prompts. By embedding domain-specific terminology into the system prompt, you bias the model’s attention mechanism toward training data that uses that vocabulary — which tends to be higher-quality, more technical content. For a web application security bot: OWASP Top 10, XSS, SSRF, DOM clobbering, deserialization, SAST, DAST.
Component 6: Output Format Rules Define exactly how the bot should respond: always use bulleted lists or checklists, always cite sources with links, be deeply thorough, double-check statements before printing.
Building Bots That Build Bots: System Bot (BotBot)
Once Haddix had the methodology, he automated it. System Bot is a meta-bot: you describe the security role and objective in plain language, and it generates a complete, production-ready system prompt using the six-component methodology.
System Bot (BotBot): Automated AI Security Bot Generator via Meta-Prompting
System Bot accepts a plain-language description of a security role and objective, then generates a complete, production-ready system prompt using Haddix’s six-component methodology — persona, flow-state instructions, task steps, related research terms, and output format rules.
Proof of Concept Steps:
-
System Bot is itself built using the six-component methodology. Its system prompt defines its role as a bot architect: given a description of a desired security bot, it must produce a complete, ready-to-deploy system prompt following the full methodology.
- Operator inputs a plain-language bot description. No prompt engineering knowledge required. Example input used in the live demonstration:
“I am a CISO in a modern cloud-first organization. I want to design a security program and need a bot that prioritizes cloud security, AppSec, and security engineering as opposed to corporate/endpoint security, where we have very little in terms of local LAN and endpoints.”
-
System Bot generates the complete system prompt. Output produced (live, unedited):
Identity and Purpose:
“You are a Cloud-First Security Expert with over 20 years of experience designing security programs for modern cloud environments. You specialize in security architecture, application security (AppSec), and cloud security engineering. Your expertise extends to cloud-native security tools, Zero Trust architecture, container security, and secure software development practices. Unlike traditional corporate security, your focus is on security in cloud infrastructure, SaaS applications, APIs, and CI/CD pipelines.”
Note: The operator did not mention Zero Trust, container security, or CI/CD pipelines — the bot inferred these as relevant and added them automatically.
Auto-generated related research terms: Cloud Security Posture Management (CSPM), Infrastructure as Code (IaC) security, Zero Trust architecture, Cloud-native security, Application security (AppSec), Secure Software Development Lifecycle (SSDLC).
-
Operator copies the generated system prompt into the ChatGPT GPT builder. In the live demo, Haddix pasted the output directly into the custom GPT creation interface and named the resulting bot “Cloud Cobot.”
-
AB test the new bot immediately. Audience test query: “Which version of TLS is recommended for secure communication in 2024?” Cloud Cobot responded correctly — TLS 1.3, with nginx configuration examples, reasoning for deprecating TLS 1.2, and zero round-trip time resumption explanation. First-try result: accurate and production-relevant.
- Operational pattern: Describe role → generate prompt via System Bot → lightly edit → deploy via GPT Store or API → AB test 5+ times before production reliance.
Temperature: The One Parameter That Changes Everything
When using the API (not the GPT Store), always set temperature to 0.6 for security bots:
- Temperature 0: Deterministic, highly repetitive — bad for security research where varied approaches matter
- Temperature 0.6: Creative enough to explore different angles, grounded enough to stay technically accurate — Haddix’s empirically validated sweet spot
- Temperature 1.0+: Too creative, drifts toward hallucination and off-topic responses
Note: The ChatGPT GPT Store locks temperature and cannot be controlled — a meaningful limitation for security use cases where output consistency matters.
AB Testing Your Bots
Never deploy a bot after a single test. Prompt the same bot at least five times with varied inputs and compare outputs. Create two variants of your system prompt with a single changed component and compare results across 5–10 queries. This is how you tune a bot from “pretty good” to “something I actually trust in production.”
Actionable Takeaways
- Use the six-component structure for every security bot you build: persona → weird machine tricks → main task steps → related research terms → output format. Skipping any component measurably degrades output quality — treat it as a checklist, not a suggestion.
- Seed your system prompts with 5–10 domain-specific research terms relevant to the security function the bot covers. These terms act like SEO keywords — they bias the model's attention toward higher-quality, more technical training data and produce expert-tier outputs instead of generic responses.
- Set temperature to 0.6 when using the API for security bots. Test every bot at least five times before trusting its output — the "first try fallacy" is real, and output quality variance across runs is often the signal that your system prompt needs refinement.
Common Pitfalls
- Building bots in the ChatGPT GPT Store when temperature control matters — the Store locks temperature and removes a key tuning lever. For any security bot where output consistency and precision are important, use the API directly.
- Skipping the "related research terms" component because it seems redundant — this is actually one of the highest-leverage components. Without it, the model defaults to generalist-level vocabulary and misses the expert-tier training examples that produce the best security analysis.
Red Team AI Automation and OSINT
Mapping AI Across the Red Team Kill Chain
AI red team automation isn’t a single tool — it’s a layer applied across every phase of the offensive lifecycle. Haddix’s approach starts with the Raffi Regman CISO mind map as a structured decomposition of the security function, then reverse-engineers each role’s daily tasks to find AI leverage points. For red teamers, that means five phases: Recon/OSINT, Initial Access, Persistence and Lateral Movement, Privilege Escalation, and Compromise. AI provides the most leverage in the bookend phases — recon and exploitation iteration — where volume and pattern recognition matter most.
Recon Phase: Subdomain Generation with AI
Subdomain Doctor is an LLM-based subdomain generation bot built for bug bounty hunting and red team recon. The core insight: large organizations name subdomains predictably — numerical iterations, thematic naming conventions (Greek gods, anime characters, internal project codenames), and service-type prefixes. A human analyst can spot patterns; an LLM can generate thousands of plausible candidates from those patterns instantly.
Subdomain Doctor: LLM-Based Subdomain Enumeration and Generation
Subdomain Doctor analyzes a known list of subdomains, identifies naming patterns and conventions, and generates statistically plausible candidate subdomains that are likely to exist but haven’t yet been discovered — extending recon coverage beyond what passive DNS, certificate transparency logs, and brute-force wordlists typically surface.
Proof of Concept Steps:
-
Gather a seed subdomain list. Using standard passive recon methods (passive DNS services, certificate transparency via crt.sh, prior bug bounty reports, Amass, subfinder), compile a list of known subdomains. In the T-Mobile example from the talk, the bug bounty team had an existing list of known subdomains under
t-mobile.comfrom prior hunting sessions. - Feed the seed list to Subdomain Doctor. The bot’s system prompt instructs it to analyze for:
- Numeric iteration patterns (e.g.,
mail84176,api-06,03-ptr-3301) - Thematic naming conventions (Greek gods, Naruto characters, internal project codenames, geography-based names)
- Service-type prefixes and suffixes (e.g.,
api-,-dev,-staging,-internal,-v2) - Hyphenation, underscore, and character substitution conventions
- Numeric iteration patterns (e.g.,
-
Bot generates candidate subdomain list. If the seed list contains
mail84176,mail84177,mail84178, the bot infers the numeric sequence and generates additional iterations. If the seed list contains Greek-god-named subdomains, it generates additional names from the same mythology set. -
Automate via CLI pipeline. Generated candidates are fed into DNS resolution tooling (e.g.,
massdns,puredns,dnsx) to identify which candidates resolve to live IP addresses. Non-resolving candidates are discarded. - Result from T-Mobile bug bounty work. The bot surfaced previously unknown subdomains that were live and resolving but absent from passive DNS records, certificate transparency logs, and prior research.
Recon Phase: Corporate Acquisition OSINT
The Acquisition Recon Bot solves a core red team problem: on full-scope engagements, the target isn’t just their primary domain — it’s every company they’ve ever acquired, which may run on aging infrastructure with different security posture, unpatched systems, and separate vulnerability management programs.
Acquisition OSINT Bot: AI-Driven Corporate Acquisition Discovery
The Acquisition OSINT Bot is a ChatGPT-based bot with internet search enabled that systematically queries multiple M&A intelligence sources to build a comprehensive acquisition history for any target organization — surfacing subsidiaries and acquired companies that standard commercial OSINT databases miss.
Proof of Concept Steps:
-
Configure the bot with internet-enabled search and source prioritization. The bot’s system prompt specifies the sources it must check: Crunchbase (primary M&A database), PitchBook, Merger Market, Wikipedia, Google, and SEC EDGAR filings (for publicly traded companies, 8-K and 10-K filings disclose acquisitions).
-
Operator submits the target company name. No URL or domain needed — just the company name. Example from the talk: “Tesla.”
-
Bot queries all configured sources in sequence. Its instructions specify: “Try to find at least 25 acquisitions.”
-
Bot returns structured acquisition list. Each result is formatted per line with: acquired company name, date of acquisition, and source. Results sorted most recent to oldest.
-
Bot surfaces acquisitions missing from primary databases. Key finding from the Tesla demonstration: the bot identified smaller, non-newsworthy acquisitions that Crunchbase did not have — appearing in trade publications and business filings that weren’t indexed by major M&A platforms but were present in the LLM’s training data or discoverable via broad Google search.
-
Red teamer expands scope to acquired domains. Each identified acquisition becomes a new recon target. Acquired companies frequently retain their original domain infrastructure post-acquisition, run on older tech stacks, and haven’t been brought into the parent company’s security program.
-
Related research terms embedded in system prompt:
OSINT,RECON,business analytics,business intelligence platform,company insights,acquired— these terms bias the model toward M&A and business intelligence training data.
Initial Access Phase: AI-Powered Spear Phishing Pretext
This is the most operationally significant red team AI application covered in the talk. The “Fishing with Dynamite” framework demonstrates how a multi-agent AI system can automate the entire spear phishing pretext pipeline.
AI-Powered Spear Phishing Pretext Generation: Fishing with Dynamite Framework
A multi-agent AI system fully automates spear phishing pretext construction — from a bare target name to a personalized, psychologically precise phishing email — by fanning out across open-source intelligence sources in parallel and synthesizing collected data into a contextually accurate lure, as demonstrated live at DEF CON’s Social Engineering Village.
Proof of Concept Steps:
-
Operator inputs target identity. The only required input is the target’s name. In the DEF CON demonstration, the target was Dave Kennedy — a well-known offensive security professional, founder of TrustedSec, based in Cleveland.
- Agent team fans out across OSINT sources in parallel. Multiple specialized AI agents simultaneously query:
- Social networks (LinkedIn, Twitter/X, Facebook) for professional history, interests, public posts
- Data aggregator sites (Spokeo, BeenVerified, Whitepages) for location, family connections, and contact data
- Professional directories and conference speaker databases for role, employer, and community involvement
- Public records and community sites for local affiliations, memberships, and activities
-
Agents return structured profile data. Each agent’s findings are consolidated into a target profile: employer (TrustedSec), location (Cleveland, OH), known personal interests (bodybuilding), professional reputation (former NSA, high-profile offensive security community figure).
-
Synthesis agent constructs the phishing email. A final agent takes the consolidated profile and generates a personalized spear phishing email using social engineering principles — plausible sender identity, contextually relevant pretext, specific personal hooks, and a clear call to action.
Resulting email (verbatim from the demonstration):
Subject: Bodybuilding.com New Cleveland Chapter Needs You “Hi Dave, I hope you’re doing well. Over the past few months we’ve been considering expanding Bodybuilding.com community chapters, and Cleveland has always been on our radar given its active and dedicated members like you. Given your unwavering commitment to the community and your efforts in promoting better health and wellness, we thought it’d be great for you to have a pivotal role in this new chapter. We have a few roles and responsibilities drafted out and would love to get your input. Please find the attached document outlining some initial plans and roles.” — Tiffany Brock, Community Development, Bodybuilding.com
-
Sender identity verification. Tiffany Brock was confirmed to be a real project manager at Bodybuilding.com for Community Development at the time of the demonstration — the system didn’t fabricate the sender, it identified a real person in a plausible organizational role. This dramatically increases the email’s credibility against even security-aware targets.
-
Target assessment. Dave Kennedy — shown the email live while in the audience — stated directly that he would have clicked the attachment. The combination of correct personal interest, correct geography, correct community context, and a real sender identity made the email indistinguishable from a legitimate outreach.
- Variant framework (not publicly released). A second, related framework maps the target’s family and friends graph concentrically and identifies attack paths through trusted third parties (family members, close colleagues) to reach the primary target — increasing success probability against targets with strong phishing awareness.
Key technical factors that made this effective:
- Parallelized agent architecture — simultaneous multi-source OSINT collection compresses hours of manual research into seconds
- Real sender identity discovery — the system identifies actual people in plausible roles rather than fabricating names, bypassing a common red flag that trained users look for
- Personal context precision — the email hooks on a highly specific, verified personal interest (bodybuilding) rather than generic professional context, creating emotional resonance that overrides analytical skepticism
Note: The framework was not publicly released — both due to weaponization risk and because external entities requested it not be made public. Private red team shops are independently building equivalent systems.
Web Application Testing: JavaScript Code Analysis
For web application pentesters, one of the highest-value AI use cases is JavaScript code analysis. Modern web apps lazy-load chunked JavaScript bundles that contain route definitions, API endpoint paths, authentication logic, and sometimes hardcoded credentials or keys.
Haddix’s workflow for web testing with AI:
- Extract the app’s primary JavaScript bundles (app.js, vendor.js, lazy-loaded chunks)
- Paste into the security bot with a standardized set of queries:
- “What frameworks, libraries, and technologies does this app use?”
- “List all API routes and endpoints you can identify”
- “Deobfuscate or unpack any compressed/obfuscated code”
- “Identify all authentication methods — do all routes require tokens?”
- “Generate curl commands for the API endpoints you found”
- “Identify potential sinks for XSS, DOM XSS, or prototype pollution vulnerabilities”
- The bot returns a structured technical brief of the application’s attack surface in seconds
This process compresses what used to be 30–60 minutes of manual JS review into under 5 minutes. Burp Suite / BurpGPT[7] extends this further with GPT-based plugins for automated analysis of intercepted HTTP traffic.
Exploitation Phase: Attack String Mutation for Filter Bypass
When a developer patches a vulnerability, they typically implement a filter around the specific payload that was reported. Attack string mutation bots bypass these filters by generating dozens of payload variants from a single working original.
XSS Mutation Bot: AI-Assisted Filter Bypass and Attack String Variation
The XSS Mutation Bot takes a single working cross-site scripting payload and generates approximately 40 systematic mutations across two categories — encoding/filter-bypass variants and event-handler substitutions — enabling testers to rapidly bypass weak or narrow XSS filters without manually enumerating bypass techniques from memory.
Proof of Concept Steps:
-
Identify the original payload. Start with a confirmed or baseline XSS payload that has been filtered or patched. Example generic payload:
<img src=x onerror=alert(1)>. -
Submit the payload to the XSS Mutation Bot. The bot’s system prompt is trained on XSS filter bypass methodology.
- Bot generates Category 1 mutations — encoding and filter bypass variants:
- URL encoding:
%3Cimg%20src%3Dx%20onerror%3Dalert(1)%3E, double-encoding, triple-encoding - HTML entity encoding:
<img src=x onerror=alert(1)> - Mixed case:
<ImG SrC=x OnErRoR=alert(1)>— bypasses case-sensitive regex filters - Null bytes injected between characters:
<i\x00mg src=x onerror=alert(1)> - Unicode normalization variants using lookalike characters that normalize to ASCII equivalents after sanitization
- Whitespace variations: tabs, newlines, and non-standard whitespace characters between attributes
- URL encoding:
- Bot generates Category 2 mutations — event handler substitutions:
<img src=x onload=alert(1)><div onmouseover=alert(1)><input onfocus=alert(1) autofocus><body onanimationend=alert(1)><svg onpointerover=alert(1)>- SVG-specific vectors:
<svg><script>alert(1)</script></svg> - CSS injection vectors for environments where JavaScript is blocked
-
Tester submits mutations against the patched endpoint. Narrow filters that only block the specific reported payload will fail against encoding variants or uncommon event handlers.
- Equivalent bots exist for SQL injection and SSRF:
- SQLi mutation bot: encoding variants, comment-injection variants, whitespace substitutions, alternative syntax forms (e.g.,
UNION%09SELECT,UN/**/ION SEL/**/ECT) - SSRF mutation bot: protocol variants (
gopher://,file://), IP encoding variants (decimal, octal, hex), redirect-based bypass attempts
- SQLi mutation bot: encoding variants, comment-injection variants, whitespace substitutions, alternative syntax forms (e.g.,
- Operational result. What previously required a pentester to recall dozens of bypass techniques from memory is compressed into a single bot query. Output: approximately 40 mutations covering the majority of common filter bypass vectors for the vulnerability class.
Tooling Automation: Nuclei Template Generation
Nuclei[6] is the de facto standard for automated vulnerability scanning in bug bounty and red team workflows. An AI bot trained on Nuclei’s YAML template syntax can generate a complete, ready-to-run template from a CVE description, a URL with HTTP request/response, or a natural language description of the detection logic — eliminating the template authoring bottleneck when a new CVE drops.
Actionable Takeaways
- Build or adopt a subdomain generation bot seeded with your target's known subdomain patterns — naming conventions, numeric sequences, and thematic clusters — and pipe the output into DNS brute-force tools. This surfaces assets that passive DNS and certificate transparency logs miss, especially for large enterprise targets.
- Use a JavaScript analysis bot with a standardized question set for every web app test. Feed in lazy-loaded JS bundles and ask for routes, auth methods, API endpoints, and XSS sinks in a single pass — this compresses manual JS review from hours to minutes and creates a reusable attack surface map.
- When a patch blocks your payload, use an attack string mutation bot before escalating — generate 40+ encoding, case, null-byte, and event-handler variants from your original payload automatically. Assume the filter is narrow; systematic mutation will find the bypass faster than manual trial-and-error.
Common Pitfalls
- Relying solely on commercial OSINT databases (Crunchbase, Shodan, passive DNS) for recon without augmenting with AI — LLM training data coverage is broader than any single commercial tool and regularly surfaces acquisitions, subsidiaries, and assets that indexed databases miss.
- Treating AI-generated attack payloads as verified exploits without manual testing — the model may hallucinate parameter names, endpoint paths, or payload syntax, especially for targets with unusual tech stacks or proprietary frameworks. Always validate AI-generated candidates against live targets before reporting.
Blue Team and SOC AI Integration
Applying AI Across the SOC Lifecycle
LLM blue team security workflows follow a natural split: wartime (active incident response) and peacetime (proactive detection, threat hunting, playbook development). In wartime, the bottleneck is speed, communication, and cognitive load under stress. In peacetime, it’s throughput — playbook authoring, query writing, and rule creation are high-value but time-consuming tasks that LLMs handle extremely well.
Haddix’s methodology applies here the same way it does for red team bots: use the six-component system prompt structure, use System Bot to generate the initial prompt, and tune from there.
Wartime: Incident Coordination and Communication
One of the first blue team bots built was a SOC Manager Briefer Bot — because incident coordinators spend significant time translating technical incident data into two very different formats simultaneously:
- Executive briefings — non-technical status emails to leadership summarizing what happened, what’s contained, and the business impact
- Technical coordination — precise, actionable instructions to IR engineers, analysts, and on-call responders
The bot accepts a generic incident description and produces both formats on demand — a meaningful cognitive load reduction for coordinators simultaneously managing phones, Slack, escalations, and an active investigation.
An Incident Coordination Checklist Bot addresses a related problem: under stress, experienced responders forget basic steps. The bot generates a comprehensive checklist tailored to the incident type, ensuring nothing critical is overlooked regardless of who is managing the incident.
Wartime: SIEM Query Automation
The highest-friction bottleneck for most SOC analysts during incidents is SIEM query authoring. Both Splunk[8] SPL and ELK’s[9] query DSL have steep learning curves with enough syntax quirks that even experienced analysts spend time debugging queries rather than analyzing data.
Splunk Bot and ELK Security Bot solve this directly:
- Analyst describes what they’re looking for in plain language
- Bot generates a syntactically correct, optimized query
- Bot can also prioritize which queries to run first based on the incident type
Real engagement example: the team used a Splunk bot to generate queries detecting Dropbox and GoToMyPC usage as exfiltration channels during a live engagement. Queries that would have taken a competent analyst 20–30 minutes to write from scratch were generated in seconds. The open-source SIEM ecosystem — including Wazuh[10] — receives equivalent AI-powered query bot coverage.
Peacetime: IR Playbooks and Detection Engineering
IR Playbook Bot generates complete incident response playbooks for any scenario:
- Input: incident type (e.g., “ransomware,” “credential stuffing,” “insider data exfiltration”)
- Output: a structured playbook covering detection indicators, initial triage steps, containment actions, eradication, recovery, and post-incident documentation requirements
The bot can be tuned to your organization’s specific SIEM, EDR, and ticketing stack by including those tools in the system prompt. The resulting playbook is a starting point for human review — it compresses authoring time from days to minutes.
Detection Engineering gets two AI boosts:
- Threat Intelligence Bots — ingest threat intel feeds and transform IOCs, TTPs, and adversary profiles into structured formats (STIX/TAXII, CSV) for import into your SIEM or threat intelligence platform
- YARA Rule Generator[11] — built on the 100 Days of YARA project dataset, this bot generates YARA rules for detecting malware, suspicious binaries, or specific file patterns from natural language descriptions or sample IOCs
Peacetime: Endpoint Telemetry and Network Analysis
osquery Bot[12] addresses a common operational gap: osquery is powerful for endpoint telemetry, but writing osquery SQL queries requires knowledge of both SQL syntax and osquery’s specific table schema. The bot translates natural language detection requirements into valid osquery queries.
Real use case: the team used osquery to hunt for DLL sideloading across an organization’s endpoints. The osquery bot generated queries targeting suspicious DLL load events, unsigned DLL executions, and process trees where known-good processes loaded unexpected DLLs — a detection scenario that requires precise schema knowledge to write correctly.
WShark Bot (Wireshark) generates valid Wireshark display filters from plain-language descriptions:
- “Show me all TLS traffic to external IPs on non-standard ports”
- “Filter for DNS queries with unusually long domain names”
- “Show me all HTTP POST requests containing multipart form data”
Peacetime: Policy and PowerShell Automation
Policy Bot generates modern security policies that are technically accurate and written in language that appeals to developers — a non-trivial balance. Most security policies are written by and for security teams, which means developers either ignore them or can’t parse the technical requirements.
PowerShell Bot handles a reality of both red and blue team work: a significant portion of operational tasks eventually require PowerShell scripting. The bot generates PowerShell scripts for common security operations — log collection, registry queries, credential auditing, service enumeration — from natural language descriptions.
Actionable Takeaways
- Deploy a SIEM query bot (Splunk or ELK) loaded with your organization's field naming conventions and log source schema in the system prompt. This lets any analyst — regardless of SPL/KQL expertise — generate investigation queries during incidents, dramatically reducing the skill ceiling bottleneck in your SOC.
- Use an IR Playbook Bot to generate first-draft playbooks for your top 10 incident scenarios. Include your specific tooling (EDR, SIEM, ticketing system) in the system prompt so the output is directly actionable for your team rather than generic.
- Build a YARA Bot seeded with the 100 Days of YARA dataset and your existing rule library. When new malware samples appear, generate candidate detection rules immediately rather than waiting for community-published signatures.
Common Pitfalls
- Deploying AI-generated SIEM queries to production without review — the bot may produce syntactically valid but semantically incorrect queries that miss events or generate excessive false positives. Always have a senior analyst validate logic before adding queries to active monitoring.
- Using AI-generated IR playbooks as final artifacts without tailoring them to your environment — generic playbooks miss organization-specific constraints like regulatory notification timelines, vendor escalation contacts, and environment-specific containment procedures. Treat bot output as a structured first draft.
Purple Team and Security Leadership AI Applications
AI at the Intersection of Offense, Defense, and Leadership
GenAI offensive security techniques reach their most strategic application in the purple team and security leadership layer — where the objective shifts from executing attacks or responding to incidents to improving the security program itself. Haddix describes purple teaming as “developing and refining attack and defense strategies” — and AI is actively being applied to both sides simultaneously: automating adversary simulation on one side and improving detection coverage mapping on the other.
Purple Team: MITRE ATT&CK Bots
MITRE ATT&CK is the foundational framework for mapping adversary techniques to detection and defensive controls. Building and maintaining ATT&CK coverage maps, identifying gaps, and correlating red team findings to specific technique IDs is a time-consuming manual process. AI bots automate this:
- Coverage mapping: Given a list of red team findings or blue team detections, map them to ATT&CK technique IDs automatically
- Gap analysis: Compare your current detection coverage against an adversary profile or threat actor group and surface uncovered techniques
- Technique lookup: Query the bot with natural language descriptions of observed behavior and get back relevant ATT&CK technique mappings with associated data sources and detection recommendations
At the time of the talk, Haddix and his team were actively building MITRE ATT&CK bots — described as “really cool” and in active development.
Purple Team: Threat Modeling with Stride GPT
Stride GPT[13] is an AI-powered threat modeling portal deployed as a self-service interface for development teams:
- Developer or security engineer inputs their application’s architecture, data flows, and trust boundaries
- Stride GPT (backed by GPT-4 or Claude) applies the STRIDE methodology (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) and generates a structured threat model
- The output identifies potential threats, affected components, and recommended mitigations
- If the threat model meets a defined severity or completeness threshold, it passes automatically; if it requires a full security review, it’s escalated
This converts threat modeling from a gate requiring a security engineer’s time on every feature to a self-service process that only escalates when genuinely needed — dramatically improving both coverage and developer autonomy.
Purple Team: Static Analysis Bots
Two static analysis tools get AI-powered bots:
Semgrep Bot[14] — seeded with Trail of Bits’ Advanced Testing Handbook, it generates Semgrep rules from vulnerability descriptions or code samples. It also helps interpret Semgrep findings and explain false positive patterns.
CodeQL Bot[15] — similarly seeded with Trail of Bits’ Advanced CodeQL materials, it generates CodeQL queries for common vulnerability patterns, explains existing queries, and helps researchers adapt queries to new codebases.
Both bots represent a pattern: take the highest-quality publicly available training material for a tool, embed it as context in a system prompt, and create an expert assistant that’s meaningfully better than asking the base model cold.
Security Leadership: The Boardroom Bot
One of the most distinctive bots in Haddix’s catalog is the Boardroom Bot — built from his experience as CISO at Ubisoft. It argues back.
Use case: a CISO drafts a security program proposal, then runs it through the Boardroom Bot. The bot plays the role of a skeptical executive or board member and challenges the proposal:
- “Why does this cost this much?”
- “What’s the ROI on this control?”
- “What’s the business impact if we defer this?”
- “How does this compare to what our competitors are doing?”
By forcing the CISO to defend their arguments against an adversarial AI, the bot surfaces weak points before they’re exposed in a real board meeting. This is the security equivalent of red-teaming your own defense.
Security Leadership: Budget and Communication Bots
Budget Bot helps security leaders prioritize spending when resources are constrained — the default state for most security programs. Given a list of proposed investments, current risk exposure, and available budget, it stack-ranks initiatives by expected risk reduction per dollar.
Tech Explains Bot solves a persistent communication problem for technical security leaders: translating complex technical findings into language that non-technical executives and board members can understand and act on. The bot takes technical security content and rewrites it at a business decision-maker level — removing jargon, adding business impact framing, and highlighting the “so what.”
Next-Gen CISO Bot is Haddix’s most personal leadership tool — a bot that encodes his own philosophy about how modern security leadership should operate. Used to stress-test decisions, validate program direction, and refine leadership communication.
Actionable Takeaways
- Deploy Stride GPT or an equivalent threat modeling portal as a self-service tool for development teams. Configure it so that low-risk applications pass automatically and only high-risk architectures require security team review — this scales your threat modeling coverage without scaling your headcount.
- Build static analysis bots for Semgrep and CodeQL seeded with the highest-quality public references for those tools (Trail of Bits' handbooks are the gold standard). This dramatically lowers the barrier for engineering teams to write and interpret security rules without dedicated AppSec support on every query.
- Before any major security budget or program proposal, run it through a Boardroom Bot that argues against it. If the bot can find a weak point, a real board member can too — use the AI adversarial review to harden your arguments before the actual meeting.
Common Pitfalls
- Treating AI-generated threat models as complete security reviews — Stride GPT and similar tools apply STRIDE systematically but lack the context of your specific codebase, deployment environment, and threat actor profile. Human review remains necessary for high-risk systems and novel architectures.
- Using executive communication bots to over-simplify risk to the point of distortion — the goal is clarity, not minimization. AI-generated business-language summaries should preserve the severity and urgency of the underlying finding, not soften it for palatability.
Conclusion
Jason Haddix’s OWASP AppSec US 2024 keynote is a practitioner-first roadmap for applying generative AI across every security function — not as a research exercise, but as an operational force multiplier deployable today. The through-line across all use cases is the six-component prompt engineering methodology: a structured, repeatable approach to building AI bots that produce consistently expert-level output across red team, blue team, and leadership functions.
The key insight is that AI doesn’t replace security expertise — it amplifies it. The practitioners who get the most from these tools are those who encode their domain knowledge into well-engineered system prompts, verify outputs rigorously, and continuously refine their bots based on real-world performance.
For deeper context on how AI is reshaping offensive security, explore our coverage of application security automation and red team methodologies. If you’re focused on the defensive side, see our content on blue team AI tooling and how SOC teams are integrating LLMs into detection engineering workflows.
References & Tools
- Fabric — Daniel Miessler's CLI tool for querying cloud and local LLMs from the command line; used for security automation workflows. ↩
- LM Studio — Desktop app for running open-source models (e.g., Llama 3) locally with tunable parameters; used for private/internal AI deployments. ↩
- Ollama — Tool for running local LLMs privately; recommended for organizations that cannot send data to cloud APIs. ↩
- ChatGPT / GPT-4o — Primary model used for text, creativity, and bot-building demonstrations throughout the talk. ↩
- Claude (Anthropic) — Preferred model for coding tasks and bot development; cited as hallucinating less and better at code generation. ↩
- Nuclei — Vulnerability scanner whose template format is automated by an AI bot to instantly generate detection templates for new CVEs. ↩
- Burp Suite / BurpGPT — Web app testing proxy extended with GPT-based plugins for automated analysis of intercepted traffic. ↩
- Splunk — Commercial SIEM for which a dedicated AI bot was built to generate queries and detection logic. ↩
- ELK (Elasticsearch/Logstash/Kibana) — Open-source SIEM stack supported by an AI bot that helps SOC analysts create and prioritize detection queries. ↩
- Wazuh — Open-source security monitoring platform included in the SIEM bot ecosystem alongside ELK and Security Onion. ↩
- YARA — Malware detection rule language; an AI bot based on the 100 Days of YARA project was built to auto-generate YARA rules from descriptions. ↩
- osquery — Endpoint telemetry tool used to detect DLL sideloading; an AI bot was built to generate osquery queries from natural language. ↩
- Stride GPT — AI-powered threat modeling portal (GPT-4/Claude backend) used as a self-service threat modeling interface for development teams. ↩
- Semgrep — Static analysis tool for which an AI bot was built and seeded with Trail of Bits' advanced testing handbook. ↩
- CodeQL — Code analysis engine; AI bot was fed Trail of Bits' advanced CodeQL guide to improve static analysis query generation. ↩
Questions from the audience
Related deep dives
This Wasnt in the Job Description- Building a production-ready AWS environment from scratch
Tenderizing the Target | [un]prompted 2026
I SPy - Rethinking Entra ID research for new paths to Global Admin
