inflearn logo

Early-Morning Alerts, Handed to AI — A Commerce Operations and Incident Response Agent Built with Claude

[Final Goal] Build an operational agent that handles the full cycle: fault injection → alert → initial AI analysis → runbook and remediation plan → human-approved action → postmortem draft. Run the virtual commerce platform 'Loop Market' with Docker and practice using Prometheus, Grafana, Loki, and Claude.

1 learners are taking this course

Level Intermediate

Course period Unlimited

Prometheus
Prometheus
grafana
grafana
claude
claude
sre
sre
Model Context Protocol
Model Context Protocol
Prometheus
Prometheus
grafana
grafana
claude
claude
sre
sre
Model Context Protocol
Model Context Protocol

What you will gain after the course

  • Add `runbook_url` to the alert annotations so the bot can automatically share the runbook, on-call information, and dashboard in the incident channel.

  • With SLI·SLO and burn rate multi-window alerts, wake people only when there’s 'user impact', and filter false-positive alerts with a rule checker.

  • Separate deterministic collectors (metrics, logs, DB sessions) from the AI decision engine, and validate AI responses using evidence IDs and a list of permitted actions.

  • We build in safeguards that make it safe to bring in-house: secret masking, a kill switch (feature flag), a human approval gate, and a read-only account.

  • Build an observability MCP server that reads from Prometheus, Loki, and Alertmanager, and enforces time windows, query scope limits, and audit logging on the server.

  • Daily anomaly and early warning reports, along with postmortem drafts, are automated according to the principle: “Code handles the records; AI handles the narrative.”

Ultimate goal — What will you build and run after completing the course?

You’ll build and run on your own laptop a system that injects a failure, triggers an alert, has an agent check metrics, logs, and the database to perform an initial analysis, posts a runbook link and recommended actions in the incident channel (actions are taken only after human approval), and drafts a postmortem once the incident is resolved.

In Lesson 0 (free preview), we’ll first show you three minutes of the finished system actually running. This is exactly what was recorded.

  • 23:34:02 — Fault injection: keep the session (hog) that locked the orders table running for 200 seconds.

  • 23:34:34 — 🚨 Database connection pool exhaustion alert + the bot adds links to the runbook, on-call, and dashboard

  • 23:34:45 — ↳ The rapid burn of the order error budget (an alert that wakes someone up) is grouped with the same incident.

  • 23:35:07 — 🤖 AI initial analysis: "Hog session holds an exclusive lock on orders · 8 shop-api sessions waiting" (evidence IDs D1·D2·M7·L1)

  • 23:35:21 — ✋ On-call approved “terminate the locked session” → execute only the single allowlisted command

  • 23:37:35 — ✅ Incident resolved (150 seconds from alert to resolution) → 📝 Postmortem draft

In the final capstone, you’ll rebuild this workflow yourselves from a blank slate.

What Makes This Course Different

  • AI makes the decision; code handles collection. For each alert, predefined queries collect metrics, logs, and DB sessions, and AI responses must pass validation against evidence IDs and the list of permitted actions before being posted to the channel.

  • We show you exactly what it looks like when AI gets things wrong. We even use as course material records of cases where it couldn’t identify the cause without supporting DB evidence, and a bug where simultaneous alerts caused the analysis to be posted twice (which the postmortem draft itself caught).

  • Safeguards for bringing this in-house from day one. Secret masking, a kill switch (feature flag), a human approval gate, read-only accounts, time-window and query-scope limits for the observability MCP, and audit logs.

  • Go all the way without an API key. Switch the evaluator to offline rules to follow the same workflow. If you have Claude Code or an API key, it switches to AI evaluation on the spot.

By the end of this lesson

  • Add `runbook_url` to the alert annotation, and have the bot automatically post the runbook, on-call contact, and dashboard to the incident channel.

  • With SLI·SLO and burn rate multi-window alerts, wake people only when there is 'user impact' and filter false-positive alerts with a rule checker.

  • Separate deterministic collectors (metrics, logs, and DB sessions) from the AI decision engine, and validate AI responses using evidence IDs and a list of permitted actions.

  • Add safeguards for internal deployment: secret masking, a kill switch (feature flag), a human approval gate, and read-only accounts.

  • Build an observability MCP server that reads Prometheus, Loki, and Alertmanager, and enforces time-window and query-scope limits and audit logging on the server.

  • Automate daily anomaly and early-warning reports and postmortem drafts based on the principle, 'Code records, AI narrates.'

Recommended for

  • Backend developers on call for commerce and platform services

  • SRE and DevOps engineers who want to reduce alert fatigue and middle-of-the-night pages

  • Team leads who want to bring AI into operations automation but are concerned about security and reliability.

Curriculum (9 sections · 26 lessons · approximately 173 minutes)

  • Section 0. See the final goal first — Lesson 0: See the finished version first — An agent handles an overnight outage in 3 minutes

  • Section 1. Practice Environment — Loop Market — Lesson 1: Launch Loop Market · Lesson 2: Read Metrics and Logs · Lesson 3: Inject Four Types of Incidents

  • Section 2. Alerts and Runbooks — Lesson 4: From Alert Rules to Incident Channels · Lesson 5: Runbook Links and an On-Call Guidance Bot · Lesson 6: Cleaning Up False-Positive Alerts

  • Section 3. SLOs and burn rate — Lesson 7: Defining SLIs and SLOs with numbers · Lesson 8: Multi-window burn rate alerts · Lesson 9: Wake people only when users are affected

  • Section 4. AI First-Pass Analysis — Lesson 10: Separate Roles—Code Collects, AI Judges · Lesson 11: Build a Deterministic Collector · Lesson 12: Judgment Prompts and Output Validation · Lesson 13: Connect Claude and Set Up Fallbacks · Lesson 14: Alert Storms and Incident Grouping

  • Section 5. Safeguards — Lesson 15: Secret Masking · Lesson 16: AI Feature Kill Switch · Lesson 17: Human Approval Gate and Read-Only Mode

  • Section 6. Observability MCP — Lecture 18: Building an Observability MCP Server · Lecture 19: MCP Safety Rules · Lecture 20: Investigating Incidents with Claude Code

  • Section 7. Daily Anomaly and Early Warning Reports — Lesson 21: Daily Anomaly and Early Warning Reports · Lesson 22: General SRE Questions vs. Our Metrics · Lesson 23: Wiki Documents as Execution Prompts

  • Section 8. Postmortem and Capstone — Lecture 24: Writing a Postmortem Draft · Lecture 25: Capstone — Build It from Scratch and Put It into Production

Hands-on Environment

Launch LoopMarket, a virtual sports and outdoor goods superstore, with a single Docker Compose setup: shopping mall API (orders · products) · PostgreSQL · Redis · Prometheus · Alertmanager · Grafana · Loki · incident channel receiver · fault injection scripts (latency · 5xx · connection exhaustion · cache bandwidth surge · gradual slowdown). The agent uses only the standard library included with macOS’s default python3. The course materials (ZIP) include assignments and solutions for each lesson, unit tests, a rules checker, sample runbooks, and daily checklists.

Prerequisites

  • Experience using Docker and docker compose

  • Basic monitoring concepts (metrics, logs, and alerts)

  • A macOS (or Linux machine running Docker) laptop with at least 4GB of free memory.

What This Course Does Not Cover

  • A complete introduction to installing and operating Prometheus and Grafana

  • Kubernetes Operations

  • How to use specific commercial monitoring SaaS platforms

  • LLM model training and fine-tuning

Price

Regular price 99,000 won → 20% launch discount → Sale price: 79,200 won

Notice

The slide narration is synthesized speech. The AI responses shown in the lecture are actual execution records, and the wording may vary from run to run. All data is fictional and for educational purposes (from the fictional company Loop Market); times and counts may vary depending on the execution environment.

Recommended for
these people

Who is this course right for?

  • Backend developer on call for commerce and platform services

  • SRE and DevOps engineers who want to reduce alert fatigue and late-night pages

  • A team lead concerned about security and trust when adopting AI for operations automation

Need to know before starting?

  • Experience using Docker and Docker Compose

  • Basic Monitoring Concepts (Metrics, Logs, and Alerts)

  • A macOS (or Linux running Docker) laptop with at least 4 GB of available memory

Hello
This is dorian05202988

Career Verified

A data craftsman with over 20 years of experience in databases and IT. As a DBA and manager of a DB Platform team, I work with database platforms and specialize in MySQL, PostgreSQL, Oracle, MongoDB, SQL tuning, AWS, Terraform, and Python. Drawing on experience in the gaming, finance, and IT services industries, I explain database and cloud architecture. I have published 『Oracle Practical Tuning: Learning SQL Patterns』 and 『Oracle Pro*C Practical Project Guide』, and have presented on serverless architectures, database selection, and multi-database strategies at AWS Summit 2018 and 2019 and DataYa Nolja 2020. Recently, I have been working on slow-query analysis and tuning recommendations using Bedrock LLM, automated DB specification sizing, and query workflows through a DB metadata MCP server and Slack bot. I explain database automation for developers and AI applications for non-developers through hands-on exercises tailored to the audience and objectives of each course. Rather than applying AI suggestions as-is, I teach a decision-making framework that verifies result equivalence, execution plans, and observability conditions, and defers action when the supporting evidence is insufficient. Company operational data and credentials are not included in the course materials.

More

Curriculum

All

26 lectures ∙ (2hr 52min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

Not enough reviews.
Please write a valuable review that helps everyone!

dorian05202988's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!

Limited time deal

$10,057.00

19%

$77.00