강의

멘토링

커뮤니티

BEST
Data Science

/

Data Engineering

Airflow Master Class

This is a course to learn about Airflow, an Orchestration tool for efficiently building and managing data pipelines. Welcome to the Airflow Master Class, where even beginners can learn step-by-step!

(4.9) 62 reviews

1,007 learners

이론 실습 모두
데이터-엔지니어링
데이터파이프라인
airflow
Data Engineering
Python
kakao-tech

Reviews from Early Learners

What you will gain after the course

  • Airflow Concepts and Basics

  • Airflow-based Pipeline Development

  • Sending Automated Emails with Airflow

  • Airflow-based Public Data API Calls and Visualization

  • Airflow & Kakao, Slack for Message Alarm

  • Utilizing ChatGPT with Airflow

Data Pipeline, No More Worries with Airflow 📊

Everything you need to know about Airflow for beginners

  • I heard you use Airflow a lot, but what is Airflow?
  • What can you do with Airflow?
  • How do I create a pipeline with Airflow?
  • How do I integrate with other solutions and extract and store data?
  • How do I integrate messengers like KakaoTalk and Slack with Airflow?
  • Is it possible to automatically write blog posts using ChatGPT in conjunction with Airflow?

👉 It covers everything from the basic concepts of Apache Airflow to the architecture configuration that can operate in a large-scale environment.

Update completed (~Airflow 3.0)

  • Introducing UI changes
  • Dag Bundle
  • Dag Versioning
  • From Dataset to Asset
  • Architecture Changes and REST API V2

Update completed (~Airflow 2.10.5)

  • Added Task Setup & Teardown function (ver 2.6)
  • DAG Params feature description & UI Form introduction (ver 2.6)
  • Added Object Storage Path feature (ver 2.8)
  • Dynamic Task Mapping feature description & Index Naming feature added (ver 2.9)
  • Added Task Bash Decorator (ver 2.9)
  • Object Storage as a Xcom Backend (ver 2.9)
  • Multiple Executor (ver 2.10)
  • Dataset Metadata (ver 2.10)
  • Dataset Alias (ver 2.10)

👉 About 80 practice files can be downloaded from Github .

But why Airflow?

Airflow is a core orchestration solution that creates and manages data pipelines that extract, process, store, and analyze data.

Airflow is the most popular pipeline management tool among similar solutions, and its adoption continues to grow.

I recommend this course to these people

Anyone who wants to be a data engineer
Anyone who needs data pipelines or business automation
For those who are using Airflow but want to learn more

What can you learn?

Airflow Basics

You will learn the basics of Airflow, including the concepts and how to create workflows, through hands-on practice. It is organized so that you can learn step by step with about 60 practice files.

Pipeline Configuration

Learn how to develop and run a DAG pipeline using Airflow, including sending emails with scheduling management.

Data collection

Let's configure a pipeline that receives and stores data via API from the Seoul Metropolitan Government Public Data Portal.

Monitoring and Integration

We will practice receiving alarms such as error messages and DAG status by linking with messenger apps such as KakaoTalk and Slack.

Data Visualization

We introduce the concept of R Shiny, which can be used for visualization using the R language. We will proceed with visualization using data received from the Seoul Public Data Portal.

Architecture

Learn about Airflow's different deployment approaches and architectures, and how to operate reliably in high-volume environments.

Automation of business

Introduce the concept of ChatGPT and learn how to connect Python API and ChatGPT. Practice automation by automatically posting to your blog the content introduced by ChatGPT about stocks that are rising rapidly through the method of retrieving stock information with Python.

Before taking the course, it would be good to know the following ✨

1. Basic knowledge of Python

  • Airflow creates pipelines in Python, so you should have some knowledge of Python.
  • However, it does not require too much knowledge. If you are familiar with basic control syntax such as for, if, while, and lists and dictionaries, you can follow along.
  • The latter half will cover class inheritance and other topics, but don't worry too much. I'll explain everything and move on.

2. Docker and Docker Compose

  • The basic training environment is WSL on Windows.
  • And Airflow is installed using Docker and various practical exercises are performed using Docker Compose, so it will be easy if you know how to handle Docker.
  • But don't worry about that either. We'll teach you everything you need to know about Docker and how to use it.

3. SQL

  • It would be helpful to know basic SQL syntax (SELECT ... FROM ... WHERE).
  • We will explain all the SQL grammar that frequently appears during the practical training.

Please check the practice environment ✨

  • We recommend that the PC or laptop you will practice on has at least 8GB of memory .
  • Since Airflow cannot be installed directly on Windows, we will basically install and practice Airflow using Windows' WSL. (I will explain the entire WSL installation process 😊)
  • Those using macOS can take the course without any special preparation.

Q&A 💬

Q. How are the lectures conducted?

In Airflow, workflow is called DAG , and we will practice by creating DAG together. Except for the time explaining the basic concepts, we will basically practice in each chapter.
If the practice file is long, I create a DAG file in advance and proceed by explaining the logic.

Q. Can I download practice files and study materials?

Of course! You can get all the practice files from Github . Not sure how to use Git? We'll teach you how to use Git too.
We also provide all PDF-based learning materials. You can download them from Section 0 - Download Lecture Materials.

Q. How difficult is the practical training?

In the beginning, you can understand it by just knowing the basic grammar of Python, but as you progress to the latter part, the difficulty level can be a little difficult, so it will be helpful to know concepts such as Python classes and inheritance. But don't worry. The practical content will be explained sufficiently and you will proceed.

Q. What can I do if I learn Airflow?

Bash Shell, anything you can do with Python, you can do. If you're wondering if something can be done with Airflow, first find out if it can be done with Bash Shell or Python. If you can do it with Bash Shell or Python, you can do it with Airflow.

Recommended for
these people

Who is this course right for?

  • Those who want to learn about Data Engineers

  • Those curious about Airflow

  • Airflow users not utilizing it well

  • Requiring data pipeline setup and management.

Need to know before starting?

  • Python Fundamentals

  • Docker & Docker Compose Usage

  • SQL Basic Syntax(SELECT, FROM)

Hello
This is

1,247

Learners

81

Reviews

219

Answers

4.9

Rating

2

Courses

안녕하세요.

데이터 & AI 분야에서 일하고 있는 15년차 현직자입니다.

정보관리기술사를 취득한 이후 지금까지 얻은 지식을 많은 사람들에게 공유하고자 컨텐츠 제작하고 있습니다.

반갑습니다. :)

Contact: hjkim_sun@naver.com

Curriculum

All

107 lectures ∙ (24hr 56min)

Course Materials:

Lecture resources
Published: 
Last updated: 

Reviews

All

62 reviews

4.9

62 reviews

  • dj9610249659님의 프로필 이미지
    dj9610249659

    Reviews 1

    Average Rating 5.0

    5

    13% enrolled

    If there is someone around me who is just starting to study data engineering, I would definitely recommend it. (From a non-major perspective) When I started studying data engineering, I was told that I needed to know git, Linux, Python, and Airflow, but I was confused for a long time because I didn't know how much I needed to know about each. While taking this lecture, I learned the basics of git and Linux that are necessary for Airflow, and it was good. Also, I heard that there will be a lot of DAG practice in the future, so I am quite excited. I will diligently take the course and master Airflow as the title suggests! If you release another Data Engineer lecture, I would definitely listen to it!

    • dj9610249659

      And I really like how you kindly explain even the smallest details. This is the most satisfying lecture I've ever taken!

    • hyunjinkim
      Instructor

      Hello dj961024 Thank you for your touching review ^_^ As someone who believes that understanding basic principles is the most important, as with anything, I thought a lot about how to easily understand the concepts. I'm so glad that it was helpful to you. If you have any questions during the course, please feel free to ask and I hope you continue to study hard!

  • everythx님의 프로필 이미지
    everythx

    Reviews 10

    Average Rating 5.0

    5

    78% enrolled

    This was a really necessary lecture and I am satisfied! Airflow has new features when it is updated, so I would appreciate it if you could update those features as well.

    • hyunjinkim
      Instructor

      Thank you for your review, everythx. I have a lecture that I'm currently working on, and once it's done, I'm planning to update airflow with new features. I'm also a working person, so I don't have much time, so it won't be right away, but I'll definitely update it. Thank you for your suggestion^^

  • kimbyl님의 프로필 이미지
    kimbyl

    Reviews 2

    Average Rating 5.0

    5

    100% enrolled

    This is an Airflow beginner course, but it was very helpful as it explained things in depth. Thank you.

    • hyunjinkim
      Instructor

      Thank you for your review, Buing-ryul. I'm glad that it was helpful to you. ^^ I hope you use it well in your field.

  • jihoon8243님의 프로필 이미지
    jihoon8243

    Reviews 1

    Average Rating 5.0

    5

    100% enrolled

    Best lecture

    • hyunjinkim
      Instructor

      Hello, Lee Ji-hoon. Thank you for your short but powerful review. You have taken the class 100%. I hope it was helpful to you. ^^

  • ahnbm8781님의 프로필 이미지
    ahnbm8781

    Reviews 1

    Average Rating 4.0

    4

    100% enrolled

    I was able to easily understand AIRFLOW's functions through various practical exercises.

    • hyunjinkim
      Instructor

      Hello ahn.bm Thank you for registering your course review. Is there anything you are dissatisfied with? Please give me your suggestions and I will try to reflect them. I hope it was helpful.

Limited time deal ends in 7 days

$101,640.00

30%

$112.20

hyunjinkim's other courses

Check out other courses by the instructor!

Similar courses

Explore other courses in the same field!