A CDC Pipeline Built with Debezium, MySQL, and Kafka
This course is designed for those who are new to Debezium-based CDC and unsure where to begin.
You will gain hands-on experience with the entire process of collecting MySQL change data in real time via Kafka and delivering it to PostgreSQL.
By learning everything from the basic principles of CDC to configuration and operational best practices, you will be able to build real-time data pipelines on your own.
Go beyond the limitations of batch,
and build a real-time CDC pipeline yourself
Operational data is constantly changing—are you querying the entire table every time?
This course covers everything step by step, from the core principles of CDC (Change Data Capture), which detects and delivers database changes in near real time, to building an actual CDC pipeline.
Debezium
MySQL
Kafka
Kafka Connect
PostgreSQL
It does not end with simply entering the configuration values.
You will learn why CDC is needed, how each component transmits data,
and what needs to be considered in operations.
You will learn the following.
After understanding the concepts, you will configure it yourself and ultimately connect them to an operational perspective.
01
Why CDC Is Needed and Its Core Principles
Explore the limitations of batch processing and updated_at-based incremental collection, and understand why CDC tracks data changes more quickly and accurately.
Collecting Insert, Update, and Delete Change Events
The Difference Between Batch and CDC
Analysis, search synchronization, audit logs, and other practical use cases
02
How to Read and Interpret MySQL Binlog
Learn the structure of the MySQL Binlog, the starting point of CDC, and how change data is recorded, then study how Debezium reads it with a focus on the overall flow.
How Binlog Works and Its Main Formats
Understanding the contents of Binlog files
MySQL Configuration and Permissions for CDC
03
The Roles of Kafka and Kafka Connect
Learn how Kafka reliably stores and delivers change events, and how Kafka Connect connects Sources and Sinks.
Core Concepts of Topics, Partitions, and Offsets
Source Connector and Sink Connector
Basic design considerations for message ordering, reprocessing, and scalability
04
Debezium CDC Pipeline Hands-on Practice
In a Docker Compose-based local environment, you will set up the entire pipeline yourself, delivering changed data all the way to PostgreSQL.
MySQL→Debezium→Kafka→PostgreSQL
Registering the connector and checking its configuration
Analyzing the Structure of Generated CDC Events
Verify Insert · Update · Delete Data Replication
05
Problems Encountered in Real-World Operations and How to Address Them
Rather than stopping at running the pipeline once, this course covers the Snapshot, Offset, message transformation, and failure recovery scenarios required in production environments.
Debezium Snapshot Optimization
Connector Restart and Offset-Based Reprocessing
SMT, data ordering guarantees, and Schema Evolution strategies
Instructor Introduction
I am a practicing data engineer who has designed and operated data flows for large-scale services.
INSTRUCTOR CAREER
From practical experience and an interviewer’s perspective, clearly explaining the core concepts of CDC.
Former Kakao data engineer and interviewer
Current leader of a global service data engineering team
Rather than listing features, we focus on why this architecture was chosen, where problems are likely to arise, and how data engineers should approach solving them.
Notes Before Taking the Course
Preparing the following environment and basic concepts will make it easier to follow the hands-on exercises.