Learn
LoginRegister
← All playbooks

Professional workflow · AI Security

AI application threat modeling

Objective

Threat-model an LLM/RAG/agent feature and produce mitigations mapped to abuse cases.

Authorized scope

Only applications you own or are contracted to assess. No jailbreak testing against third-party public services beyond their ToS and your policy.

Prerequisites

  • AI / LLM Security course recommended
  • Architecture diagram of the AI feature
  • Data classification for prompts and corpora
Course →Roadmap L38 →

Step-by-step workflow

  1. Step 01

    Frame the system

    • Draw trust boundaries (user, model, tools, data stores)
    • List privileges of tools/plugins
    • Identify sensitive data in prompts/RAG

  2. Step 02

    Enumerate threats

    • Prompt injection (direct/indirect)
    • Data exfiltration via model/tools
    • Over-privileged agents / confused deputy

  3. Step 03

    Controls

    • Input/output filtering and allow-listed tools
    • Human approval for sensitive actions
    • Monitoring for policy violations

  4. Step 04

    Test plan

    • Define abuse-case tests in a lab app
    • Regression pack for CI where feasible
    • Owner sign-off on residual risk

Expected evidence

  • Annotated architecture / DFD
  • Threat list with STRIDE or OWASP LLM mapping
  • Control matrix
  • Test cases

Checklist

  • Untrusted retrieval content assumed hostile
  • Tool permissions least-privilege
  • Logging of tool calls enabled

Reporting template

AI threat model — [feature]
Assets: […]
Trust boundaries: […]
Threats: ID | description | impact | mitigations
Residual risk: […]
Test plan: […]

Remediation

  • Separate system instructions from untrusted content
  • Restrict tools; require confirmation for side effects
  • Add detection for exfil patterns and policy breaks

Related tools