강의

멘토링

커뮤니티

Programming

/

Back-end

Toss Senior Developer's Data Workflow Management-Based Large-Scale Data Processing Design Patterns [ By. Non-CS Major & Toss Developer ]

Learn the process of building data pipelines using Apache Airflow from basics to practical application. Understand Airflow's core concepts and architecture, and master advanced design patterns frequently used in practice such as dynamic DAGs, parallel processing, distributed processing, and Custom Operators through hands-on exercises. Set up a practice environment with Python and Docker, and develop practical skills to design and operate real workflows.

(5.0) 9 reviews

123 learners

  • Hong
실습 중심
자격증
시험
데이터분석
데이터베이스
Big Data
Docker
docker-compose
airflow

Reviews from Early Learners

What you will learn!

  • Understanding the Concept and Necessity of Apache Airflow

  • Understanding the Structure of Airflow Core Components

  • Dynamic DAG Design Methods

  • TaskGroup and Dependency Management Patterns

  • Parallel Processing and Large-Scale Data Reprocessing Strategy

  • Custom Operator and its related encapsulation, decorator utilization

  • Python & Docker-based Practice Environment Setup

What services would be good to use for large-scale data batch processing pipelines? 🤔

❗This content is from actual conversations.❗

😁 Toss : Hong, do you happen to know airflow??

😄 Hong : I know about it, but I haven't tried it. Why?

😁 Toss : You know that workflow lecture I made last time, I was thinking maybe it would be good to cover airflow as well after seeing that.. I've only used airflow

😄 Hong : But I airflowhaven't used it before, I don't really know how to do it

😁 Toss : It's fine, I'm actually using it in real work right now, so I can proactively teach you. I'll burn myself out for my student

😄 Hong : 😆😆😆😆 Nice concept. Got it. But do we really have to use this?? I honestly can't feel much difference from regular batch processing or cron job processing?

😁 Toss : The fact that you're even thinking about it airflowis already a reason why you should use airflow.. There are some differences between airflow and batch processing or cron jobs. Simply put, it's the same as why you should use workflows, and big data exists too.

What did the Toss senior developer's last statement mean in the previous conversation??🤔

Is Airflow really necessary for building data processing modules?? Why must we use it?? From my perspective, it seems like we could just implement it using regular batch processing modules or cron jobs??

Did you perhaps have thoughts like this?? If so, studying the process of utilizing and implementing Airflow through this course will be a great help for your career.


The answer lies in workflow management. How can this series of processes—from data extraction, processing, to handling—flow stably and be managed sequentially and dependently like a pipeline? What if this entire series of processes could be supported through a single platform?


Rather than a boring lecture that just lists theories, I've prepared this as a practical guide that perfectly conquers the core functions of workflow-based large-scale data pipeline design by examining the operational processes together. 🚀

Features of this course

📌 Rich course content with approximately 30 diagrams and lecture summary files

* This is not just a lecture that explains things with words alone, but also provides actual source code, diagrams, sequence diagrams, and additionally simple summary files of the lecture content.

📌 60% theory, 40% practice, complete testing environment provided

* This is not a lecture that simply lists theories, but provides a lightweight environment where you can actually see and learn the content with your own eyes, and allows you to flexibly test and practice in this environment.

Expertise proven through previous lectures (as of 9.27) 👨‍🏫

🧑‍🎓 3075.0

🧑‍🎓 3794.9

🧑‍🎓 4834.7

🧑‍🎓 2394.8

The course covers these topics. 🧩

* What is Airflow?

*Batch Job & Cron Job Vs Airflow

*Apache Airflow's Disadvantages and Anti-Patterns in Implementation

* Introduction to Overall Core Components Architecture

* WebServer Components Deep Dive

* Scheduler Components Deep Dive

* Executor Components Deep Dive

* MetaDataDB Components Deep Dive

* Dynamic DAG Generation Pattern [ Dynamic DAG ]

* Cross-DAG Dependencies and Data Dependencies

* Designing Complex Workflows Using TaskGroup

* Custom Operator for Reusability and Encapsulation

*Docker, docker-composeLightweight Environment Setup

* Airflow's Parallel Processing and Distributed Processing Strategies

* Notification using Slack

* Distributed Data Processing Using CeleryExecutor

What makes this course special

📌Event providing 50 coupons

Course Early Bird Discount PeriodWe will select 50 people who purchase during the early bird discount period and provide each with one 50% coupon

Related Resources 🚀

The person who created this course 🤭

  • I started as a non-major and am currently working as a platform backend developer in Pangyo

  • The goal is to teach realistic development methods and development theories, and I am a knowledge sharer who creates lectures together with capable acquaintances around me, not alone

  • A knowledge sharer who conducted an interview at Inflearn thanks to their diligent activities


  • A server developer who majored in computer engineering in a rural area, worked as a developer at Naver, and is currently doing backend development at Toss

  • A developer who always gets scolded by Hong for not having enough time...

  • A developer who wants to achieve financial freedom and dreams of solo development

Reference Notes

Practice Environment

  • python3, pip3

    • Python 3.13.2

    • 25.0 from /opt/homebrew/lib/python3.13/site-packages/pip (python 3.13)

  • docker, docker-compose

    • Docker version 28.0.0, build f9ced58158

    • Docker Compose version 2.33.1

  • OS

    • Apple M3 Air

This course is planned to have its discount rate adjusted over time to provide higher discount rates to those who purchase early. Please take note of this.

Recommended for
these people

Who is this course right for?

  • Server/Data Engineer handling large-scale data in production environments

  • Developers who want to build experience in data pipeline design and operations

  • Technical personnel at companies looking to introduce or advance Airflow

  • Architects interested in distributed processing and workflow automation

  • Team leads/senior developers who want to build a stable data platform in production environments

Hello
This is

4,186

Learners

303

Reviews

109

Answers

4.6

Rating

18

Courses

자기 소개

집에서 빈둥대다 개발에 흥미를 느껴 개발 공부를 시작하였고 현재는 판교에서 플랫폼 서버 개발을 담당하여 진행하고 있습니다.

 

제가 공부를 했던 방법과 실무에서 접하실 수 있는 여러가지 문제점들과 해결책을 여러분들에게 제공하고 싶어 지식공유자 활동을 이어나가고 있습니다.

 

강의는 오로지 저만의 지식을 통해 만들어지지 않습니다. 모든 강의는 함께하시는 분들이 계십니다.

 

지식공유자 경력

[前] 샌드박스 블록체인 개발자

[前] 넥슨 자회사 백엔드 개발자

[] 판교에서 고여가는 서버 개발자

 

인터뷰 이력

Curriculum

All

29 lectures ∙ (4hr 39min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

All

9 reviews

5.0

9 reviews

  • 미래 1인 개발자님의 프로필 이미지
    미래 1인 개발자

    Reviews 8

    Average Rating 5.0

    5

    93% enrolled

    이번에 배치처리의 꽃이라고 할 수 있는 Airflow에 대한 주제로 강의를 함께 준비한 토스에서 개발하고있는 개발자 입니다. AIrflow라는 서비스가 생각보다 많이 생소한 경우가 많지만, 서비스가 거대해질수록 이런 워크플로우 서비스는 굉장히 유용합니다. 대용량의 배치 처리를 처리하는데에 있어서 가장 우선시하여 고려되는 서비스이기 떄문입니다. 아직 여러분들이 이 Airflow를 학습 할 필요가 없는 환경이라도 이 강의에서 알려주는 다양한 관점과 개념들은 여러분들이 개발하고 공부하는 환경에 반드시 도움이 될 겁니다. 단순히 Airflow만을 학습하는 강의는 아니라고 생각을 해서요. 많은 관심 부탁드리고 다음 강의도 기대해주시기를 바라겠습니다. 감사합니다!!

    • Hong
      Instructor

      💜

  • joon님의 프로필 이미지
    joon

    Reviews 7

    Average Rating 5.0

    5

    31% enrolled

    • dellahong님의 프로필 이미지
      dellahong

      Reviews 1

      Average Rating 5.0

      5

      62% enrolled

      Airflow를 3년째 사용하고 있지만, 점점 데이터 처리 규모가 커질수록 잦은 에러가 발생하기 시작하면서 다른 회사에서는 어떤 방식으로 사용을 할까 싶어서 강의를 듣게되었습니다. 너무 많은 도움이 되고 있어요! 실무적으로 Airflow의 개념과 실습이 있어서 이해하고 적용하는데 도움이 많이됩니다 :)

      • Hong
        Instructor

        dellahong님 좋은 평 감사합니다. 실무자가 해주는 평가라니 더 뜻깊네요 앞으로 노력하겠습니다!

    • 텐버거!님의 프로필 이미지
      텐버거!

      Reviews 6

      Average Rating 5.0

      5

      100% enrolled

      제가 처음보고 볼 수 없었던 관점에서의 다양한 개념을 알려주는 뜻깊은 강의라고 생각합니다. 내용 자체가 음.. 뭐랄까 제 식견을 넓혀주는 내용이였던거 같아요. 좋은 강의 너무 감사합니다. 텐버거!

      • Hong
        Instructor

        안녕하세요 텐버거!님 좋은 평 남겨주셔서 감사합니다!! 앞으로 더 좋은 내용 제공해 드릴 수 있또록 노력하겠습니다!!

    • Jlos Ho님의 프로필 이미지
      Jlos Ho

      Reviews 9

      Average Rating 5.0

      5

      90% enrolled

      좋은 강의 감사합니다. 공유자님 강의에서 Spring, Java 말고 다른 언어를 사용하시는건 처음보는거 같습니다. 그래도 크게 불편한 부분 없이 잘 보았고, python 언어를 물론 알고 있다면 더 보기에는 더 좋다고 생각하지만, 단순하게 Airflow를 활용하고 테스트해보고 속히말해 맛을 보고 싶다면 굳이 몰라도 되는 강의인거 같습니다. 이번에도 정말 너무 좋은 강의 감사드리고 공유자님도 점점 강의 실력도 성장하고 그에따라서 저도 같이 성장하는거 같아 뿌듯하네요 ㅎㅎ 다음에도 좋은 주제로 잘 부탁드리겠습니다!

      • Hong
        Instructor

        안녕하세요 Jios Ho님 평 남겨주셔서 감사합니다. 요즘에는 언어에 대한 제약이 적어지는 추세같아요. 그래서 이번에는 좀 특징있는 강의로 한번 준비해 보았습니다. 감사합니다!

    $61.60

    Hong's other courses

    Check out other courses by the instructor!

    Similar courses

    Explore other courses in the same field!