inflearn logo

Design Patterns for Large-Scale Data Processing Based on Data Workflow Management with Toss Developers

Learn how to build data pipelines with Apache Airflow, from the basics to real-world applications. Understand Airflow’s core concepts and architecture, and practice advanced design patterns frequently used in the field, including dynamic DAGs, parallel processing, distributed processing, and Custom Operators. Set up a hands-on environment with Python and Docker, design and operate real-world workflows, and develop practical skills for production use.

(4.7) 22 reviews

245 learners

Level Basic

Course period Unlimited

Big Data
Big Data
Docker
Docker
docker-compose
docker-compose
airflow
airflow
Big Data
Big Data
Docker
Docker
docker-compose
docker-compose
airflow
airflow

Reviews from Early Learners

4.7

5.0

미래 1인 개발자

93% enrolled

I'm a developer working at Toss who prepared this lecture together on the topic of Airflow, which can be called the flower of batch processing. While the service called Airflow is often quite unfamiliar to many people, as services grow larger, workflow services like this become extremely useful. This is because it's the service that's given top priority when handling large-scale batch processing. Even if you're in an environment where you don't need to learn Airflow yet, the various perspectives and concepts taught in this lecture will definitely help you in your development and study environment. I don't think this is a lecture that simply teaches only Airflow. I ask for your great interest and hope you'll look forward to the next lecture as well. Thank you!!

5.0

dellahong

62% enrolled

I've been using Airflow for 3 years now, but as the scale of data processing has grown, frequent errors have started occurring, so I became curious about how other companies use it and decided to take this course. It's been incredibly helpful! Having both conceptual understanding and hands-on practice of Airflow in a practical context really helps with understanding and applying it :)

5.0

텐버거!

100% enrolled

I think this is a meaningful lecture that introduced me to various concepts from perspectives I had never seen or couldn't see before. The content itself was... how should I put it, it seemed like content that broadened my knowledge. Thank you so much for the great lecture. Tenbagger!

What you will gain after the course

  • Understanding the Concepts and Necessity of Apache Airflow

  • Understanding the Structure of Airflow Core Components

  • How to Design a Dynamic DAG (Dynamic DAG)

  • TaskGroup and Dependency Management Patterns

  • Parallel Processing and Large-Scale Data Reprocessing Strategy

  • Custom Operators and Their Encapsulation and Use of Decorators

  • Setting Up a Python & Docker-Based Practice Environment

Recommended for
these people

Who is this course right for?

  • Server/data engineer handling large-scale data in practice

  • A developer who wants to gain experience designing and operating data pipelines

  • Technical personnel at companies looking to adopt or further enhance Airflow

  • An architect interested in distributed processing and workflow automation

  • Team leads/senior developers who want to build a stable data platform in a production environment

Hello
This is Hong

Inflearn Verified

Career Verified

13,933

Learners

714

Reviews

187

Answers

4.8

Rating

34

Courses

I started studying development after becoming interested in it while idling around at home, and I am currently responsible for platform server development in Pangyo. I continue sharing my knowledge because I want to provide you with the methods I used to study, as well as various issues you may encounter in practice and their solutions.

 

The courses are not created solely from my own knowledge. Every course is made possible with the help of others.

 

[Knowledge Sharer Experience]

[Formerly] Blockchain developer working on Sandbox IP

[Formerly a metaverse backend developer]

[Current] A server developer growing stagnant in Pangyo

 

[Interview History]

[Other inquiries]

  • unduck2022@gmail.com

[Official Website and Social Media]

More

Curriculum

All

29 lectures ∙ (4hr 39min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

All

22 reviews

4.7

22 reviews

  • miaaade9585868님의 프로필 이미지
    miaaade9585868

    Reviews 9

    ∙

    Average Rating 5.0

    5

    93% enrolled

    I haven't tried it myself, but I think this is a good lecture where you can learn perspectives on large-scale batch processing. I enjoyed the lecture.

    • jhong
      Instructor

      Hello a mi! Thank you for leaving such a good review!! I will work hard to provide you with even better lectures in the future. Have a great day!

  • youngba8935643님의 프로필 이미지
    youngba8935643

    Reviews 10

    ∙

    Average Rating 5.0

    5

    100% enrolled

    I think this is a meaningful lecture that introduced me to various concepts from perspectives I had never seen or couldn't see before. The content itself was... how should I put it, it seemed like content that broadened my knowledge. Thank you so much for the great lecture. Tenbagger!

    • jhong
      Instructor

      Hello Tenburger! Thank you for leaving such a good review!! I will work hard to provide you with even better content in the future!!

  • ho6227574978님의 프로필 이미지
    ho6227574978

    Reviews 9

    ∙

    Average Rating 5.0

    5

    90% enrolled

    Thank you for the great lecture. I think this is the first time I've seen you use a language other than Spring and Java in your lectures. Still, I watched it well without any major inconveniences, and I think it would be even better to watch if you know Python, but if you simply want to utilize Airflow, test it out, and get a taste of it, so to speak, I think it's a lecture you don't necessarily need to know it for. Thank you again for such a wonderful lecture this time, and I'm proud that you're gradually improving your teaching skills and I feel like I'm growing along with you 😊😊 I look forward to your continued good work with great topics in the future!

    • jhong
      Instructor

      Hello Jios Ho, thank you for leaving a review. It seems like language constraints are becoming less restrictive these days. So this time, I prepared a lecture with some unique characteristics. Thank you!

  • tttos님의 프로필 이미지
    tttos

    Reviews 8

    ∙

    Average Rating 5.0

    5

    93% enrolled

    I'm a developer working at Toss who prepared this lecture together on the topic of Airflow, which can be called the flower of batch processing. While the service called Airflow is often quite unfamiliar to many people, as services grow larger, workflow services like this become extremely useful. This is because it's the service that's given top priority when handling large-scale batch processing. Even if you're in an environment where you don't need to learn Airflow yet, the various perspectives and concepts taught in this lecture will definitely help you in your development and study environment. I don't think this is a lecture that simply teaches only Airflow. I ask for your great interest and hope you'll look forward to the next lecture as well. Thank you!!

    • jhong
      Instructor

      💜

  • dellahong님의 프로필 이미지
    dellahong

    Reviews 1

    ∙

    Average Rating 5.0

    5

    62% enrolled

    I've been using Airflow for 3 years now, but as the scale of data processing has grown, frequent errors have started occurring, so I became curious about how other companies use it and decided to take this course. It's been incredibly helpful! Having both conceptual understanding and hands-on practice of Airflow in a practical context really helps with understanding and applying it :)

    • jhong
      Instructor

      Thank you for the kind review, dellahong. It's even more meaningful to receive feedback from a practitioner. I'll continue to work hard!

Hong's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!

Limited time deal ends in 3 days

$51.70

22%

$67.10