inflearn logo

From Synchronous to Asynchronous — A Practical Guide to Migrating Commerce APIs to Message Queues, Outbox, and Workflow Engines

Replace synchronous coupon issuance and product registration APIs that buckle under sale-launch traffic with a queue, outbox, idempotent consumer, DLQ, and Temporal—and reproduce the before-and-after metrics (latency, failure rate, and DB commits) under the same load on my computer using Docker.

1 learners are taking this course

Level Intermediate

Course period Unlimited

PostgreSQL
PostgreSQL
Kafka
Kafka
MSA
MSA
temporal
temporal
PostgreSQL
PostgreSQL
Kafka
Kafka
MSA
MSA
temporal
temporal

What you will gain after the course

  • Benchmark the synchronous API under a fixed-arrival-rate load to find the point where latency, failure rate, and DB commit count start to deteriorate, and determine whether the cause is CPU or lock/commit waits.

  • Switch to an intake API that queues issuance requests and responds with 202, paired with a batch consumer, and present the before-and-after figures in a table under the same load.

  • We deliberately create dual-write losses, duplicate relays, out-of-order messages, and poison messages, then prevent them with an outbox, idempotent consumers, partition keys, and a DLQ.

  • We’ll convert the registration API, which takes several seconds, into a submit-and-check-status contract, and use measurements to decide in an ADR how to handle scheduled sends and publishing changes.

  • Move failures that quietly disappear in function-and-queue setups to Temporal workflows, so retries and failure detection can be managed in one place.

  • Practice a rollback sequence that switches between synchronous and asynchronous modes via a feature flag without redeploying, while ensuring requests remaining in the queue aren’t lost.

We replace synchronous coupon issuance and product registration APIs that buckle under sale-launch traffic with a queue, outbox, idempotent consumer, DLQ, and workflow engine, then use before-and-after figures to show exactly how DB commits, latency, and failure rates change under the same load.

Final results (measured in the Lesson 0 demo, at the same load of 800 requests per second): synchronous issuance failure rate 77.29%, response p50 9.82 seconds, and 7,944 DB commits → after switching to asynchronous processing, a 0% failure rate, p50 of 10.62 ms, and 820 DB commits. In the final capstone (Lesson 30), students migrate coupon issuance and product registration themselves and produce these before-and-after numbers.

Replace synchronous APIs that buckle under sale-launch loads with message queues, Transactional Outbox, idempotent consumers, DLQ, and Temporal, then reproduce the before-and-after metrics (latency, failure rate, and DB commits) under the same load using Docker on your own computer.

Most message queue courses stop at how to use the tools. This course covers the work of moving an existing synchronous API to asynchronous processing. It includes before-and-after measurements, acceptance contracts, how to deliberately create and prevent failures such as message loss, duplication, ordering issues, and poison messages, decision records, and feature-flag rollouts and rollbacks. We’ll run the API, PostgreSQL, Redpanda (Kafka-compatible), Temporal, k6, Prometheus, and Grafana for a fictional sports and outdoor specialty store, 'Loop Market', growing to a team of 10–20 developers—all in Docker. Every number on screen comes from an actual run.

By the end of this course, you will be able to

  • Measure the synchronous API under a fixed arrival-rate load to find where latency, failure rate, and DB commits start to break down, and determine whether the cause is CPU saturation or lock and commit waits.

  • We replace it with an intake API that queues issuance requests and responds with 202, plus a batch consumer, then present the before-and-after numbers in a table under the same load.

  • We deliberately create dual-write losses, duplicate relays, out-of-order processing, and poison messages, then prevent them with an outbox, idempotent consumers, partition keys, and a DLQ.

  • We change the registration API, which takes several seconds, into an accept → status-query contract, and use measured results to decide on the scheduled-send and change-publication approach in an ADR.

  • Move failures that quietly disappear in a function-and-queue setup to Temporal workflows, so retries and failure detection can be monitored in one place.

  • Practice switching between synchronous and asynchronous processing with feature flags without deploying, and rehearse a rollback sequence that won’t lose requests left in the queue.

Recommended for

  • E-commerce backend developers who have experienced outages due to database write spikes during sales and events

  • For those who have used Kafka but aren’t sure how to safely migrate existing synchronous APIs

  • For those who know about outbox, idempotency, and DLQs in theory but want to reproduce failures firsthand.

Curriculum (8 sections · 32 lessons · approximately 195 minutes)

  • Section 0. Orientation: See the finished result first — Demo of the finished result: same load, different numbers

  • Section 1. When the Synchronous Architecture Breaks Down — Set up the practice environment, read Loop Market’s synchronous coupon issuance API, generate open load with k6, watch synchronous issuance break down, diagnose the bottleneck: row locks and commit count

  • Section 2. Buffering with a Queue — Broker basics: topics, partitions, and consumer groups; switch to an intake API: respond with 202; batch database writes with a batch consumer; compare before and after under the same load; use consumer lag as a metric.

  • Section 3. Make sure messages are neither lost nor duplicated — deliberately create the dual-write problem, transactional outbox, what happens when the relay dies: duplicate publishing, idempotency keys and idempotent consumers, ordering guarantees and partition keys: reward points, retries and DLQs, checklist for duplicate and message-loss scenarios

  • Section 4. Make Long-Running APIs Asynchronous — Measure a product registration API that takes several seconds, define the submission and status-check contract, compare alternatives: keep it synchronous, use threads, queues, or workflows, apply it to another domain: delivery instructions

  • Section 5. Documenting Design Decisions — Scheduled delivery: per-item scheduling vs. batch ADR, publishing changes: full snapshot vs. row-level delta, loading only the consumer’s scope

  • Section 6. Durable Execution: Workflow Engines — The limitations of composing functions and queues yourself, product registration workflows with Temporal, detecting failures at the workflow level

  • Section 7. Migration and Capstone — Switching between synchronous and asynchronous processing with feature flags, rollback practice: what remains after reverting, capstone: completing the Loop Market migration, wrap-up: migration checklist

Lab Environment

Practice materials (ZIP): Start all 9 services with a single compose.yml, then run scenarios/sNN.sh as-is for each lesson to reproduce the behavior shown in the lecture. Includes check scripts, k6 scenarios, a migration checklist, ADRs, and intake contract templates.

Prerequisites

  • Experience building REST APIs (any language; the hands-on code is in Python)

  • RDB and Transaction Fundamentals

  • Experience running containers with Docker (we recommend allocating 4 CPUs and at least 8 GB of memory to Docker)

What This Course Does Not Cover

  • Kafka Cluster Installation and Operations

  • Kubernetes Deployment

  • How to use specific cloud-managed services

Notice

The video narration is AI-generated. The commands, metrics, and results shown on screen are outputs from actual executions in the practice environment. All data is simulated for educational purposes, and the numbers may vary depending on the execution environment.

Recommended for
these people

Who is this course right for?

  • A commerce backend developer who experienced outages due to spikes in database writes during sales and events

  • For those who have used Kafka but aren’t sure how to safely migrate existing synchronous APIs.

  • For anyone who knows about the outbox, idempotency, and DLQ only in theory and wants to reproduce failures firsthand.

Need to know before starting?

  • Experience building a REST API (any language; hands-on coding in Python)

  • RDB and Transaction Basics

  • Experience running containers with Docker (we recommend allocating at least 4 CPUs and 8 GB of memory to Docker)

Hello
This is dorian05202988

Career Verified

A data craftsman with over 20 years of experience in databases and IT. As a DBA and manager of a DB Platform team, I work with database platforms and specialize in MySQL, PostgreSQL, Oracle, MongoDB, SQL tuning, AWS, Terraform, and Python. Drawing on experience in the gaming, finance, and IT services industries, I explain database and cloud architecture. I have published 『Oracle Practical Tuning: Learning SQL Patterns』 and 『Oracle Pro*C Practical Project Guide』, and have presented on serverless architectures, database selection, and multi-database strategies at AWS Summit 2018 and 2019 and DataYa Nolja 2020. Recently, I have been working on slow-query analysis and tuning recommendations using Bedrock LLM, automated DB specification sizing, and query workflows through a DB metadata MCP server and Slack bot. I explain database automation for developers and AI applications for non-developers through hands-on exercises tailored to the audience and objectives of each course. Rather than applying AI suggestions as-is, I teach a decision-making framework that verifies result equivalence, execution plans, and observability conditions, and defers action when the supporting evidence is insufficient. Company operational data and credentials are not included in the course materials.

More

Curriculum

All

32 lectures ∙ (3hr 15min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

Not enough reviews.
Please write a valuable review that helps everyone!

dorian05202988's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!

Limited time deal

$1,101,429.00

19%

$51.70