inflearn logo

AI Red Teaming 3 - Red Team Work and Threat Modeling

Before execution, define the scope, rules of engagement, test conditions, and grounds for assessment. Link AI assets and trust boundaries with STRIDE scenarios to create a red team plan and threat model, and distinguish what can be stated based on the results of repeated tests.

1 learners are taking this course

Level Basic

Course period Unlimited

Penetration Testing
Penetration Testing
security
security
AI
AI
LLM
LLM
Offensive Security
Offensive Security
Penetration Testing
Penetration Testing
security
security
AI
AI
LLM
LLM
Offensive Security
Offensive Security

What you will gain after the course

  • Distinguishes the roles of the red team, blue team, and purple team, as well as the differences in their diagnostic approaches.

  • We will draft rules of engagement and a plan outlining the scope, prohibited actions, termination criteria, and approval procedures.

  • Under the same conditions, distinguish between completed evaluations, the number of successful instances, errors, and unobserved cases, and record the limitations of repeated results.

  • Connect AI assets, trust boundaries, STRIDE, and scenario cards to complete the Mission 3 threat model document.

Course Materials and Practice Guide

Course Introduction (0-1) Download the integrated student materials ZIP from the class materials download button. It includes the latest study guide, correction practice kit, existing mission templates, and evidence assessment cases and explanations. Also make use of the supplementary notes and cases from the related assessment lessons.

All sessions use AI-synthesized narration based on the instructor’s voice. The exercises use actual Amazon Bedrock models, while the tools and documents are simulated for educational purposes. No actual emails are sent or files deleted.

AWS usage charges are separate from the course fee and vary depending on the model, region, call volume, log retention, maintained resources, and free-tier conditions. The course cost is an estimated example under specific conditions, and budget alerts do not automatically block usage. Check your usage after the hands-on exercise and clean up the resources.

Why this course?

I entered the same prompt five times: twice, the system prompt was revealed, and three times, the request was refused. If you write only “vulnerable” in the report, the recipient won’t know what to do, while writing “safe” would be a lie.

This is where AI assessment diverges from web assessment, and how to write this one line is the question this course addresses.

Before attacking indiscriminately, first learn what professional red teams do. Define the scope and rules of engagement, establish reproducibility and evaluation criteria, and create a threat model for the backend uploaded in Fundamentals 1. You will finish with two documents in hand: a red team plan and a threat model.

What You’ll Be Able to Do After Completing This Course

  • Distinguish red teaming, blue teaming, purple teaming, vulnerability assessments, penetration testing, and red teaming from one another on a single page.

  • Write a plan that protects assessors by specifying the scope, prohibited actions, emergency termination procedures, and approvals in the Rules of Engagement (ROE) (Mission 2)

  • Under the same conditions, distinguish completed evaluations, the number of valid occurrences, and errors, then establish decision rules and a testing scope appropriate to the basis for validity and the limits of observation.

  • Dissect the AI application into six sections and catalog its assets in three categories: models, data, and interfaces.

  • Draw five trust boundaries and apply all six STRIDE categories to AI systems.

  • Write attack scenario cards and complete the threat model document (Mission 3)

Recommended for people who:

  • Learners who have completed Basics 2 and drawn an attack surface map.

  • A security practitioner who needs to write an AI system assessment report but is unsure how to describe that “the same attack works sometimes but not other times”"

  • A security professional who has tried threat modeling for the web but doesn’t know how to apply it to LLMs and agents

  • Job seekers who need a project plan and threat model deliverable to include in their portfolio

This may not be suitable for those who:

  • Those expecting hands-on attack exercises in this course—it involves no console operations and consists of writing two documents.

  • Those who want to learn how to use automated threat modeling tools — the two templates are filled out by hand.

  • Those who want to delve deeply into web threat modeling theory - STRIDE is reviewed only as much as necessary for AI.

  • Those without Mission 1 of Basics 2 (the attack surface map) - Part 2’s threat model continues from that map.

Course Flow - Red Team Practice → Evaluation Criteria → Threat Model → Two Deliverables

A total of 12 sessions. The section order is the learning order.

  • Section 1 · Course Introduction (1 lesson · 1 preview lesson)

  • Section 2 · Part 1 · How Does Red Teaming Work? (5 lessons · 1 preview lesson)

  • Section 3 · Part 2 · AI Threat Modeling (6 lessons · 1 preview)

Things to Check Before Taking the Course

  • This assumes that you have completed Mission 1 of Basics 2.

  • This lecture’s document assignment can be completed without making any new model calls. Please check separately for the ongoing maintenance and usage costs of existing AWS resources.

  • Download and fill out the two templates (redteam-plan.md · threat-model.md) from the resource library.

  • The narration is an AI-generated voice created based on the instructor’s voice.

Detailed course content

Part 1 · How Does a Red Team Work?

In Part 1, we look at how red teams actually work. We summarize on a single page what the red team, blue team, and purple team do, as well as how vulnerability assessments, penetration testing, and red teaming differ in scope and depth, and then cover the rules of engagement—the point where disputes arise. A document that records why the exclusion list is more important than the inclusion list, what signals require an immediate stop, who approves what, and what must be documented protects the assessor. Next comes nondeterminism.

Distinguish completed evaluations, the number of valid occurrences, and errors, and establish the basis for determinations and the scope of testing. Three, 10, and 20 trials are examples of a testing plan for the course; the number of trials alone does not guarantee safety. Also use supplementary cases that distinguish “not observed” from “structurally not applicable.” Part 1 concludes with Mission 2, in which you write the red-team plan yourself.

Part 2 · AI Threat Modeling

Part 2 is threat modeling. We dissect an AI application into the six stages a single request passes through, distinguish what we build from what we borrow, and record assets in three categories: models, data, and interfaces. We then draw five trust boundaries, and these five numbers are used unchanged in the subsequent lectures from Intermediate 1 through Advanced. After relearning the six letters of STRIDE for AI and mastering how to write attack scenario cards, we complete the threat model document in Mission 3.

Structure and Learning Method

The course consists of 12 sessions. Following the course introduction, Part 1 comprises five sessions and Part 2 six sessions, with each section concluding with a mission. Missions 2 and 3 continue with the same subject. In 1-5, you write a plan for a hypothetical in-house AI chatbot, and in 2-6, you use that same subject to complete the threat model document. This way, the two documents do not stand alone but become the preparation materials for a single assessment.

In 2-5, we’ll actually write an attack scenario card using an internal document assistant chatbot as an example, and cover the criteria for deciding which cards to review first once twenty cards have been created. If you’re unsure how to judge any item while filling out the form, post just that item on the Q&A board, and we’ll answer with the rationale.

The next lecture

This concludes the fundamentals section. Starting with the next lecture, Intermediate 1, we will directly work through the screen we saw in the final lesson of Fundamentals 1, recording how a single parameter and a single line of code change recall.

Course Features

  • Practice how to record the number of successful attempts among completed evaluations, along with the basis and limitations of the determination.

  • The numbers for the five trust boundaries continue through the end of the series, so all findings in later lectures are organized using these numbers.

  • Since this involves filling out two types of forms (a plan and a threat model), the deliverables remain intact as-is.

  • It is explained in a form that can be used immediately in practice, by simply adding three columns to the existing information asset register.

  • Topics covered – penetration testing · offensive security · AI security from an artificial intelligence (AI) security perspective, LLM threat modeling

After completing the course

By the end of the course, you will have a red team plan and a threat model document in hand. Both are created by filling out the templates in the resource library, so you can include them directly in your portfolio, and the attack surface map you drew in Fundamentals 2 grows into a threat model here. The documentation assignment can be completed without making new model calls; check separately for the ongoing maintenance and usage costs of existing AWS resources.

Series “AI Red Teaming by Breaking In Yourself” — Lesson 7

  • AI Red Teaming 1 - Introduction to LLM Security and Hands-On Environment

  • AI Red Teaming 2 - How LLMs Work and the Attack Surface

  • AI Red Teaming 3 - Red Team Tasks and Threat Modeling ← This lecture

  • AI Red Teaming 4 - Directly Bypassing Prompt Injection

  • AI Red Teaming 5 - The Attack Opened by One Missing Setting

  • AI Red Teaming 6 - Breaking Through Agents and RAG

  • AI Red Teaming 7 - Control Design and Red Team Report

The mission deliverables from the previous course serve as the material for the next course, so they are released in numerical order, and unreleased courses will be released sequentially.

Recommended for
these people

Who is this course right for?

  • Security personnel and diagnosticians who must determine the scope of AI assessments and the basis for their determinations

  • Developers and architects who design or review trust boundaries for AI services

Need to know before starting?

  • Prepare the attack surface map for Lesson 2, Mission 1. Understand the components of the provided practice environment and use a document editor to write the Mission 2 plan and Mission 3 threat model.

Hello
This is rmsxodxod1289

Career Verified

I am a practitioner who started with penetration testing and builds information security management systems.

I handle annual security assessments of infrastructure, web, and applications for automotive manufacturers and energy companies.

I assessed them layer by layer and handled the same client’s ISMS-P certification through three rounds: initial, follow-up, and renewal.

I responded by independently carrying out the technical safeguards component.

Currently, as the information security officer in a one-person team, I am preparing for initial certification while establishing AI security governance.

I am designing the framework. Since this is an area where standards are still being established, starting with defining the scope of controls themselves,

It is a position where I must establish them myself.

I hold certifications as an Engineer Information Security, CPPG, and ISO/IEC 27001 Auditor, and at Dongguk University’s International Information Security Department,

I am researching leakage paths in AI environments and comparing them against authentication criteria in the Graduate School’s Department of Artificial Intelligence Security.

5 years and 9 months of experience. Client names and vulnerabilities actually discovered are subject to confidentiality, so in lectures

do not cover them in lectures.

More

Reviews

Not enough reviews.
Please write a valuable review that helps everyone!

rmsxodxod1289's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!

Limited time deal

$204,957.00

68%

$24.20