inflearn logo

AI Red Teaming 2 - How LLMs Work and Attack Surfaces

From tokens and context to RAG, agents, and guardrails, we examine how AI services work from a security perspective. We compare the concepts with the provided backend code and organize the areas to inspect across input, retrieval, and tool calls into a single attack-surface map.

14 learners are taking this course

Level Beginner

Course period Unlimited

AI
AI
prompt engineering
prompt engineering
LLM
LLM
Offensive Security
Offensive Security
RAG
RAG
AI
AI
prompt engineering
prompt engineering
LLM
LLM
Offensive Security
Offensive Security
RAG
RAG

What you will gain after the course

  • Explains how tokens, context, and the system prompt make up the model input.

  • Distinguish RAG’s indexing, retrieval, and input assembly from the agent’s tool-calling flow.

  • This summarizes the roles and limitations handled by guardrails and application controls.

  • Compare the provided backend code with the service architecture to create the Mission 1 attack surface map.

Course Materials and Practice Guide

Course Introduction (0-1) Download the integrated course materials ZIP from the Download Materials button in the class. It includes the latest learning guide, the correction practice kit, existing mission templates, and evidence judgment examples and explanations. Also make use of the supplementary notes and examples from the relevant judgment lessons.

All sessions use AI-synthesized narration based on the instructor’s voice. The hands-on exercises use actual Amazon Bedrock models, while the tools and documents are simulated environments for training purposes. No actual emails are sent or files deleted.

AWS usage charges are separate from the course fee and vary depending on the model, region, call volume, log retention, retained resources, and free-tier conditions. The course cost is an estimated example under specific conditions, and budget alerts do not automatically block usage. Check your usage after the hands-on exercises and clean up your resources.

Why This Course?

Even after reading twenty examples of prompt injection, some questions remain unanswered. Why might input filtering alone not be sufficient? Why does a system prompt that says “Do not show this” end up being exposed? Why can a single retrieved document change the model’s behavior?

Memorizing techniques won’t provide the answer. You need to understand how the model interprets input to find the answer.

This course covers that principle only as much as an attacker needs to know. We explain it with a single diagram, apply it directly to the app.py file of your backend from Basics 1 to identify which line corresponds to which part of the diagram, and then map the attack surface of the AI service you actually use.

What you will be able to do after completing this course

  • Understand token processing and input construction, and explain the role and limitations of input filters.

  • Map out how system instructions, role-based conversation messages, and reference materials make up the model’s input, along with the differences in trustworthiness between their sources.

  • It identifies which of the three stages—indexing, retrieval, and injection—in RAG constitutes the attack surface, and whose permissions the agent operates under when calling tools.

  • We distinguish what the five layers of controls used by actual services can and cannot prevent.

  • Check in a mapping table how this structure is incorporated into the single line that calls `converse` in app.py, uploaded in Basics 1.

  • I map the attack surface of the AI service I use with a four-layer map (Mission 1)

Recommended for:

  • Learners who have completed Basics 1 and set up the practice backend.

  • Developers and security professionals who want to understand the conditions under which prompt injection succeeds and the limitations of controls, starting from the underlying principles.

  • Planners and PMs who want to summarize on a single page where risks arise when introducing LLMs, RAG, and agents

This may not be suitable for those who:

  • Those looking to learn tokenization algorithms or Transformer equations - no mathematics is covered.

  • Those expecting hands-on exercises involving writing code—the only activity in 2-6 is reading app.py, not writing it.

  • Those who want to carry out attacks themselves in this course - hands-on exploitation starts in Lesson 4 (Intermediate 1); here, we look at why vulnerabilities can be exploited

  • If you skipped Basic 1’s Mission 0 (deploying the practice backend), sessions 2–6 open that code.

Course flow - Principles → Apply them to my code → Attack surface map

A total of 8 lessons. The section order is the learning order.

  • Section 1 · Course Introduction (1 lesson · 1 preview lesson)

  • Section 2 · How Does an LLM Interpret Input? (5 parts · 1 preview)

  • Section 3 · Apply It to My Backend Code and Map the Attack Surface (2 lessons · 1 preview)

Pre-Course Checklist

  • This assumes you have completed Mission 0 of Fundamentals 1.

  • This course focuses on reading code and creating maps. Calling a new model is not required, but be sure to separately check the usage and maintenance costs of existing AWS resources.

  • This is a browser-based course, and nothing needs to be installed on your PC.

  • The narration is AI-synthesized speech created based on the instructor’s voice.

Detailed Course Content

How does an LLM receive input?

The model processes text in units called tokens, and the API distinguishes system instructions from conversation messages by role. The Bedrock Converse call in the exercise also sends `system` as a separate field and uses `user` and `assistant` messages. Reference materials are added to the body of the user message along with source indicators. We examine separately the fact that roles or source indicators are present and whether the application’s authorization and data-access controls are actually guaranteed.

Next, we will examine in order why it is difficult to guarantee authorization boundaries using only system prompts, which of the three RAG stages—chunking, retrieving, and injecting documents—is the attack surface, and whose permissions an agent operates under when calling a tool. We will also pinpoint what the five layers of controls used by real services can and cannot prevent.

Apply it to my backend code and draw an attack surface map

We don’t stop at diagrams. In sessions 2–6, open the app.py of your backend deployed in Fundamentals 1 and use a mapping table to verify how the system, messages, and toolConfig of the Converse call are each configured. We also distinguish chat_assemble, which creates source-specific snippets, from the call path that builds role-based messages and reference materials. At this point, you can see where the lecture’s diagram appears in your account’s code and which control layers are missing. In the final session, organize everything you have seen into a four-layer map, then use Mission 1 to map the attack surface of the AI service you actually use. This map will serve as the foundation for the trust boundaries you will draw in Fundamentals 3.

There are no math or coding assignments. It is assumed that you have completed Mission 0 in Basics 1; if you deleted the stack, follow the deployment guide to set it up again. The preparation time varies depending on your account, permissions, and model access status, and you should also check the costs of existing resources.

Structure and Learning Method

The course consists of 8 sessions, and the playback time for each session can be found in the current curriculum. By connecting diagrams with code, you can listen to the six conceptual sessions while on the go, and only need to watch 2-6 Code Substitution and 2-7 Mission 1 in front of a screen. The summary slides at the end of each session repeat the diagram and one sentence from that session, so you can refer back to those slides and the code explanations later when you get stuck at the intermediate level.

For Mission 1, choose one AI service you actually use and draw its four-layer map without any formatting. If you’re unsure which service to choose, simply post the service name on the Q&A board, and we’ll help point you in the right direction.

The next lecture

In the next lecture, Fundamentals 3, you’ll learn the sequence of tasks a practical red team performs before attacking indiscriminately: defining the scope, counting assets, and drawing a threat model. You’ll then develop the attack surface map created here into a threat model document.

Course Features

  • We explain the sentence “What is followed 90 percent of the time is different from what must always be followed” as a principle.

  • Figure → my account’s code → attack surface map, moving from the abstract to the tangible in that order.

  • From a practical perspective, we point out where to account for things that are easy to overlook in an asset inventory, such as vector databases.

  • You will learn the 8th session in the order of understanding the concepts, comparing the code, and mapping the attack surface.

  • Areas covered - artificial intelligence (AI) security, how LLMs work, attack surfaces from a prompt engineering perspective, RAG and AI agent architecture, and offensive security

Upon completing the course

Upon completing the course, you’ll be able to explain the principle behind the statement, “Being protected 90 percent of the time is not the same as being protected every time,” and you’ll have an attack surface map for one AI service you use. In Fundamentals 3, this map will grow into a threat model document.

AI Red Teaming Parts 1–7 Now Available

All 7 parts are now available. If you’re just getting started, begin with Parts 1–3 to set up your environment and prepare an attack surface map and threat model, then move on to the hands-on assessment. You’ll reuse the outputs from earlier parts in later ones.

Part 2 assumes you have completed Mission 0 of Part 1. Part 4 requires the environment from Part 1 and the Mission 3 threat model document from Part 3. Part 6 requires the environment from Part 1 and experience completing the worksheet in Part 4; Part 5 is not a required prerequisite. Please check the prerequisites and lab materials for each part first. AWS usage fees are separate from the course fee.

Recommended for
these people

Who is this course right for?

  • A security practitioner looking to organize where to start assessing LLMs, RAG, and agents

  • Developers and PMs who want to understand AI service components and risk areas together

Need to know before starting?

  • You will need the practice environment deployed as Mission 0 in Lecture 1 and experience with your first query. No math or coding is required, and one session involves reading the provided backend code by following the instructions.

Hello
This is rmsxodxod1289

Career Verified

111

Learners

4

Reviews

1

Answers

3.0

Rating

8

Courses

I am a practitioner who started with penetration testing and builds information security management systems.

I handle annual security assessments of infrastructure, web, and applications for automotive manufacturers and energy companies.

I assessed them layer by layer and handled the same client’s ISMS-P certification through three rounds: initial, follow-up, and renewal.

I responded by independently carrying out the technical safeguards component.

Currently, as the information security officer in a one-person team, I am preparing for initial certification while establishing AI security governance.

I am designing the framework. Since this is an area where standards are still being established, starting with defining the scope of controls themselves,

It is a position where I must establish them myself.

I hold certifications as an Engineer Information Security, CPPG, and ISO/IEC 27001 Auditor, and at Dongguk University’s International Information Security Department,

I am researching leakage paths in AI environments and comparing them against authentication criteria in the Graduate School’s Department of Artificial Intelligence Security.

5 years and 9 months of experience. Client names and vulnerabilities actually discovered are subject to confidentiality, so in lectures

do not cover them in lectures.

More
Published: 
Last updated: 

Reviews

Not enough reviews.
Please write a valuable review that helps everyone!

rmsxodxod1289's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!

25% off for new members

$16.50

25%

$16.50