강의

챌린지

멘토링

로드맵

커뮤니티

Design Patterns for Large-Scale Data Processing Based on Data Workflow Management with Toss Developers

Learn how to build data pipelines with Apache Airflow, from the basics to real-world applications. Understand Airflow’s core concepts and architecture, and practice advanced design patterns frequently used in the field, including dynamic DAGs, parallel processing, distributed processing, and Custom Operators. Set up a hands-on environment with Python and Docker, design and operate real-world workflows, and develop practical skills for production use.

(4.7) 20 reviews

232 learners

Level Basic

Course period Unlimited

Big Data
Big Data
Docker
Docker
docker-compose
docker-compose
airflow
airflow
Big Data
Big Data
Docker
Docker
docker-compose
docker-compose
airflow
airflow

Reviews from Early Learners

4.7

5.0

미래 1인 개발자

93% enrolled

I'm a developer working at Toss who prepared this lecture together on the topic of Airflow, which can be called the flower of batch processing. While the service called Airflow is often quite unfamiliar to many people, as services grow larger, workflow services like this become extremely useful. This is because it's the service that's given top priority when handling large-scale batch processing. Even if you're in an environment where you don't need to learn Airflow yet, the various perspectives and concepts taught in this lecture will definitely help you in your development and study environment. I don't think this is a lecture that simply teaches only Airflow. I ask for your great interest and hope you'll look forward to the next lecture as well. Thank you!!

5.0

dellahong

62% enrolled

I've been using Airflow for 3 years now, but as the scale of data processing has grown, frequent errors have started occurring, so I became curious about how other companies use it and decided to take this course. It's been incredibly helpful! Having both conceptual understanding and hands-on practice of Airflow in a practical context really helps with understanding and applying it :)

5.0

텐버거!

100% enrolled

I think this is a meaningful lecture that introduced me to various concepts from perspectives I had never seen or couldn't see before. The content itself was... how should I put it, it seemed like content that broadened my knowledge. Thank you so much for the great lecture. Tenbagger!

What you will gain after the course

  • Understanding the Concepts and Necessity of Apache Airflow

  • Understanding the Structure of Airflow Core Components

  • How to Design a Dynamic DAG (Dynamic DAG)

  • TaskGroup and Dependency Management Patterns

  • Parallel Processing and Large-Scale Data Reprocessing Strategy

  • Custom Operators and Their Encapsulation and Use of Decorators

  • Setting Up a Python & Docker-Based Practice Environment

Which service would be best to use for a pipeline that processes large volumes of data in batches? 🤔

❗This is an actual conversation.❗

😁 Toss: Hong, do you happen to know about Airflow??

😄 Hong: I know of it, but I haven't used it. Why?

😁 Toss: You know the workflow course I made before? I thought it might be good to cover Airflow as well... I've only used Airflow.

😄 Hong : But I airflowhaven’t used it, so I don’t really know. What am I supposed to do?

😁 Toss: No worries, I’m using it at work right now, so I can take the lead in teaching you. I’ll give it my all for my student.

😄 Hong: Hahaha, you’ve really nailed the concept. Got it. But do we really have to use this?? I honestly don’t see much difference between this and regular batch processing or cron jobs.

😁 Toss: The fact that you're even thinking that way is itself a reason to use airflow. Airflowis somewhat different from batch processing or cron jobs; put simply, it's about why you should use workflows, and there's big data too.

What does the Toss senior developer mean by what they said last in the preceding conversation??🤔

Do I really need Airflow to build a data processing module? Why do I have to use it? I feel like I could just implement it using a standard batch processing module or a cron job.

Have you ever had thoughts like this? If so, studying how to use and adopt Airflow through this course will be a great help to your career.


The answer lies in workflow management. How can this series of processes—from data extraction to transformation and processing—flow reliably and be managed sequentially and dependently like a pipeline? What if all of these processes could be supported through a single platform?


Rather than being a boring course that merely lists theories, this practical guide is designed to help you completely master the key features of workflow-based large-scale data pipeline design by examining how everything works together. 🚀

Features of this course ⚡

📌 A rich course structure with around 30 diagrams and lecture summary files

* Rather than explaining things solely through words, this course provides actual source code, diagrams, sequence diagrams, and additional brief summary files covering the course content.

📌 60% theory, 40% practice, and a perfect testing environment

* This is not a course that simply lists theory. Instead, we provide a lightweight environment where you can see and learn the material firsthand, with the flexibility to test your exercises in that environment.

Expertise demonstrated through previous courses (as of 9.27) 👨‍🏫 

🧑‍🎓 307 ⭐ 5.0

🧑‍🎓 379 ⭐ 4.9

🧑‍🎓 483 ⭐ 4.7

🧑‍🎓 239 ⭐ 4.8

The course covers the following topics. 🧩

* What is Airflow?

*Batch Job & Cron Job Vs Airflow

*Disadvantages of Apache Airfow and anti-patterns for adoption

* Introduction to the Overall Core Components Architecture

* WebServer Components Deep Ei

* Scheduler Components Deep Dive

* Executor Components Deep Dive

* MetaDataDB Components Deep Dive

* Dynamic DAG generation patterns [ Dynamic DAG ]

* Cross-DAG Dependencies and data dependencies

* Designing Complex Workflows Using TaskGroups

* Custom Operators for Reusability and Encapsulation

*Docker, docker-composefor setting up a lightweight environment

* Airlfow's Parallel and Distributed Processing Strategies

* Notifications using Slack

* Distributed data processing using CeleryExecutor

Recommended resources to view together 🚀

The person who created this course 🤭

  • A developer who started without a computer science background and currently works as a platform backend developer in Pangyo

  • My goal is to share practical development methods and philosophies, and I am a knowledge sharer who creates courses together with skilled people around me rather than alone.

  • A knowledge sharer who, thanks to his active efforts, was featured in an interview on Inflearn.


  • A server developer who majored in computer science outside the capital, worked as a developer at Naver, and is currently doing backend development at Toss

  • A developer who is always short on time and gets scolded a lot by Hong...

  • A developer who wants to achieve financial freedom and dreams of becoming a solo developer

Notes

Practice environment

  • python3, pip3

    • Python 3.13.2

    • 25.0 from /opt/homebrew/lib/python3.13/site-packages/pip (python 3.13)

  • docker, docker-compose

    • Docker version 28.0.0, build f9ced58158

    • Docker Compose version 2.33.1

  • OS

    • Apple M3 Air

To provide a greater discount to those who purchase this course early, the discount rate will be adjusted over time. Please keep this in mind.

Recommended for
these people

Who is this course right for?

  • Server/data engineer handling large-scale data in practice

  • A developer who wants to gain experience designing and operating data pipelines

  • Technical personnel at companies looking to adopt or further enhance Airflow

  • An architect interested in distributed processing and workflow automation

  • Team leads/senior developers who want to build a stable data platform in a production environment

Hello
This is Hong

Inflearn Verified

Career Verified

10,237

Learners

609

Reviews

175

Answers

4.8

Rating

30

Courses

I started studying development after becoming interested in it while idling at home, and I am currently responsible for platform server development in Pangyo. I am continuing my activities as a knowledge sharer because I want to provide you with the methods I used to study, as well as the various problems and solutions you may encounter in practice.

 

These lectures are not created solely through my own knowledge. There are others who collaborate on every lecture.

 

[Instructor Career]

[Former] Blockchain developer related to Sandbox IP

[Former] Metaverse Backend Developer

[Current] A server developer becoming a veteran in Pangyo

 

[Interview History]

[Other Inquiries]

[Official Site]

More

Curriculum

All

29 lectures ∙ (4hr 39min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

All

20 reviews

4.7

20 reviews

  • ho6227574978님의 프로필 이미지
    ho6227574978

    Reviews 9

    ∙

    Average Rating 5.0

    5

    90% enrolled

    Thank you for the great lecture. I think this is the first time I've seen you use a language other than Spring and Java in your lectures. Still, I watched it well without any major inconveniences, and I think it would be even better to watch if you know Python, but if you simply want to utilize Airflow, test it out, and get a taste of it, so to speak, I think it's a lecture you don't necessarily need to know it for. Thank you again for such a wonderful lecture this time, and I'm proud that you're gradually improving your teaching skills and I feel like I'm growing along with you 😊😊 I look forward to your continued good work with great topics in the future!

    • jhong
      Instructor

      Hello Jios Ho, thank you for leaving a review. It seems like language constraints are becoming less restrictive these days. So this time, I prepared a lecture with some unique characteristics. Thank you!

  • youngba8935643님의 프로필 이미지
    youngba8935643

    Reviews 10

    ∙

    Average Rating 5.0

    5

    100% enrolled

    I think this is a meaningful lecture that introduced me to various concepts from perspectives I had never seen or couldn't see before. The content itself was... how should I put it, it seemed like content that broadened my knowledge. Thank you so much for the great lecture. Tenbagger!

    • jhong
      Instructor

      Hello Tenburger! Thank you for leaving such a good review!! I will work hard to provide you with even better content in the future!!

  • tttos님의 프로필 이미지
    tttos

    Reviews 8

    ∙

    Average Rating 5.0

    5

    93% enrolled

    I'm a developer working at Toss who prepared this lecture together on the topic of Airflow, which can be called the flower of batch processing. While the service called Airflow is often quite unfamiliar to many people, as services grow larger, workflow services like this become extremely useful. This is because it's the service that's given top priority when handling large-scale batch processing. Even if you're in an environment where you don't need to learn Airflow yet, the various perspectives and concepts taught in this lecture will definitely help you in your development and study environment. I don't think this is a lecture that simply teaches only Airflow. I ask for your great interest and hope you'll look forward to the next lecture as well. Thank you!!

    • jhong
      Instructor

      💜

  • dellahong님의 프로필 이미지
    dellahong

    Reviews 1

    ∙

    Average Rating 5.0

    5

    62% enrolled

    I've been using Airflow for 3 years now, but as the scale of data processing has grown, frequent errors have started occurring, so I became curious about how other companies use it and decided to take this course. It's been incredibly helpful! Having both conceptual understanding and hands-on practice of Airflow in a practical context really helps with understanding and applying it :)

    • jhong
      Instructor

      Thank you for the kind review, dellahong. It's even more meaningful to receive feedback from a practitioner. I'll continue to work hard!

  • miaaade9585868님의 프로필 이미지
    miaaade9585868

    Reviews 9

    ∙

    Average Rating 5.0

    5

    93% enrolled

    I haven't tried it myself, but I think this is a good lecture where you can learn perspectives on large-scale batch processing. I enjoyed the lecture.

    • jhong
      Instructor

      Hello a mi! Thank you for leaving such a good review!! I will work hard to provide you with even better lectures in the future. Have a great day!

Hong's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!