inflearn logo
inflearn logo

Airflow Master Class

This is a course to learn about Airflow, an Orchestration tool for efficiently building and managing data pipelines. Welcome to the Airflow Master Class, where even beginners can learn step-by-step!

(4.9) 수강평 69개

강의소개.상단개요.수강생.short

난이도 초급

수강기한 무제한

airflow
airflow
Data Engineering
Data Engineering
Python
Python
airflow
airflow
Data Engineering
Data Engineering
Python
Python

먼저 경험한 수강생들의 후기

먼저 경험한 수강생들의 후기

4.9

5.0

Areum Go

58% 수강 후 작성

I suddenly had to take charge of Airflow and felt lost, but thank you so much for explaining even the basics step-by-step from a beginner's perspective. It was very helpful. I was looking at the official documents myself to see what was different about Airflow 3.0, and thank you so much for updating the new course materials just in time 😭😭 I will study the new course well.

5.0

이동준

13% 수강 후 작성

If there is someone around me who is just starting to study data engineering, I would definitely recommend it. (From a non-major perspective) When I started studying data engineering, I was told that I needed to know git, Linux, Python, and Airflow, but I was confused for a long time because I didn't know how much I needed to know about each. While taking this lecture, I learned the basics of git and Linux that are necessary for Airflow, and it was good. Also, I heard that there will be a lot of DAG practice in the future, so I am quite excited. I will diligently take the course and master Airflow as the title suggests! If you release another Data Engineer lecture, I would definitely listen to it!

5.0

everythx

78% 수강 후 작성

This was a really necessary lecture and I am satisfied! Airflow has new features when it is updated, so I would appreciate it if you could update those features as well.

강의상세_배울수있는것_타이틀

  • Airflow Concepts and Basics

  • Airflow-based Pipeline Development

  • Sending Automated Emails with Airflow

  • Airflow-based Public Data API Calls and Visualization

  • Airflow & Kakao, Slack for Message Alarm

  • Utilizing ChatGPT with Airflow

Data Pipeline, No More Worries with Airflow 📊

Everything you need to know about Airflow for beginners

  • I heard you use Airflow a lot, but what is Airflow?
  • What can you do with Airflow?
  • How do I create a pipeline with Airflow?
  • How do I integrate with other solutions and extract and store data?
  • How do I integrate messengers like KakaoTalk and Slack with Airflow?
  • Is it possible to automatically write blog posts using ChatGPT in conjunction with Airflow?

👉 It covers everything from the basic concepts of Apache Airflow to the architecture configuration that can operate in a large-scale environment.

Update completed (~Airflow 3.0)

  • Introducing UI changes
  • Dag Bundle
  • Dag Versioning
  • From Dataset to Asset
  • Architecture Changes and REST API V2

Update completed (~Airflow 2.10.5)

  • Added Task Setup & Teardown function (ver 2.6)
  • DAG Params feature description & UI Form introduction (ver 2.6)
  • Added Object Storage Path feature (ver 2.8)
  • Dynamic Task Mapping feature description & Index Naming feature added (ver 2.9)
  • Added Task Bash Decorator (ver 2.9)
  • Object Storage as a Xcom Backend (ver 2.9)
  • Multiple Executor (ver 2.10)
  • Dataset Metadata (ver 2.10)
  • Dataset Alias (ver 2.10)

👉 About 80 practice files can be downloaded from Github .

But why Airflow?

Airflow is a core orchestration solution that creates and manages data pipelines that extract, process, store, and analyze data.

Airflow is the most popular pipeline management tool among similar solutions, and its adoption continues to grow.

I recommend this course to these people

Anyone who wants to be a data engineer
Anyone who needs data pipelines or business automation
For those who are using Airflow but want to learn more

What can you learn?

Airflow Basics

You will learn the basics of Airflow, including the concepts and how to create workflows, through hands-on practice. It is organized so that you can learn step by step with about 60 practice files.

Pipeline Configuration

Learn how to develop and run a DAG pipeline using Airflow, including sending emails with scheduling management.

Data collection

Let's configure a pipeline that receives and stores data via API from the Seoul Metropolitan Government Public Data Portal.

Monitoring and Integration

We will practice receiving alarms such as error messages and DAG status by linking with messenger apps such as KakaoTalk and Slack.

Data Visualization

We introduce the concept of R Shiny, which can be used for visualization using the R language. We will proceed with visualization using data received from the Seoul Public Data Portal.

Architecture

Learn about Airflow's different deployment approaches and architectures, and how to operate reliably in high-volume environments.

Automation of business

Introduce the concept of ChatGPT and learn how to connect Python API and ChatGPT. Practice automation by automatically posting to your blog the content introduced by ChatGPT about stocks that are rising rapidly through the method of retrieving stock information with Python.

Before taking the course, it would be good to know the following ✨

1. Basic knowledge of Python

  • Airflow creates pipelines in Python, so you should have some knowledge of Python.
  • However, it does not require too much knowledge. If you are familiar with basic control syntax such as for, if, while, and lists and dictionaries, you can follow along.
  • The latter half will cover class inheritance and other topics, but don't worry too much. I'll explain everything and move on.

2. Docker and Docker Compose

  • The basic training environment is WSL on Windows.
  • And Airflow is installed using Docker and various practical exercises are performed using Docker Compose, so it will be easy if you know how to handle Docker.
  • But don't worry about that either. We'll teach you everything you need to know about Docker and how to use it.

3. SQL

  • It would be helpful to know basic SQL syntax (SELECT ... FROM ... WHERE).
  • We will explain all the SQL grammar that frequently appears during the practical training.

Please check the practice environment ✨

  • We recommend that the PC or laptop you will practice on has at least 8GB of memory .
  • Since Airflow cannot be installed directly on Windows, we will basically install and practice Airflow using Windows' WSL. (I will explain the entire WSL installation process 😊)
  • Those using macOS can take the course without any special preparation.

Q&A 💬

Q. How are the lectures conducted?

In Airflow, workflow is called DAG , and we will practice by creating DAG together. Except for the time explaining the basic concepts, we will basically practice in each chapter.
If the practice file is long, I create a DAG file in advance and proceed by explaining the logic.

Q. Can I download practice files and study materials?

Of course! You can get all the practice files from Github . Not sure how to use Git? We'll teach you how to use Git too.
We also provide all PDF-based learning materials. You can download them from Section 0 - Download Lecture Materials.

Q. How difficult is the practical training?

In the beginning, you can understand it by just knowing the basic grammar of Python, but as you progress to the latter part, the difficulty level can be a little difficult, so it will be helpful to know concepts such as Python classes and inheritance. But don't worry. The practical content will be explained sufficiently and you will proceed.

Q. What can I do if I learn Airflow?

Bash Shell, anything you can do with Python, you can do. If you're wondering if something can be done with Airflow, first find out if it can be done with Bash Shell or Python. If you can do it with Bash Shell or Python, you can do it with Airflow.

강의소개.콘텐츠.추천문구

학습 대상은 누구일까요?

  • Those who want to learn about Data Engineers

  • Those curious about Airflow

  • Airflow users not utilizing it well

  • Requiring data pipeline setup and management.

선수 지식, 필요할까요?

  • Python Fundamentals

  • Docker & Docker Compose Usage

  • SQL Basic Syntax(SELECT, FROM)

강의소개.지공자소개

1,327

수강생

88

수강평

227

답변

4.9

강의 평점

2

강의_other

Hello.

I am a professional with 15 years of experience currently working in the Data & AI field.

Since obtaining my Professional Engineer IT Management certification, I have been creating content to share the knowledge I've gained with as many people as possible.

Nice to meet you. :)

Contact: hjkim_sun@naver.com

더보기

공동 지식공유자

커리큘럼

전체

107개 ∙ (강의상세_런타임_시간 강의상세_런타임_분)

해당 강의에서 제공: [object Object]
강의 게시일: 
마지막 업데이트일: 

수강평

전체

69개

4.9

69개의 수강평

  • ahnbm8781님의 프로필 이미지
    ahnbm8781

    수강평 2

    평균 평점 4.0

    4

    100% 수강 후 작성

    I was able to easily understand AIRFLOW's functions through various practical exercises.

    • hyunjinkim
      지식공유자

      Hello ahn.bm Thank you for registering your course review. Is there anything you are dissatisfied with? Please give me your suggestions and I will try to reflect them. I hope it was helpful.

  • dj9610249659님의 프로필 이미지
    dj9610249659

    수강평 1

    평균 평점 5.0

    5

    13% 수강 후 작성

    If there is someone around me who is just starting to study data engineering, I would definitely recommend it. (From a non-major perspective) When I started studying data engineering, I was told that I needed to know git, Linux, Python, and Airflow, but I was confused for a long time because I didn't know how much I needed to know about each. While taking this lecture, I learned the basics of git and Linux that are necessary for Airflow, and it was good. Also, I heard that there will be a lot of DAG practice in the future, so I am quite excited. I will diligently take the course and master Airflow as the title suggests! If you release another Data Engineer lecture, I would definitely listen to it!

    • And I really like how you kindly explain even the smallest details. This is the most satisfying lecture I've ever taken!

    • hyunjinkim
      지식공유자

      Hello dj961024 Thank you for your touching review ^_^ As someone who believes that understanding basic principles is the most important, as with anything, I thought a lot about how to easily understand the concepts. I'm so glad that it was helpful to you. If you have any questions during the course, please feel free to ask and I hope you continue to study hard!

  • jihoon8243님의 프로필 이미지
    jihoon8243

    수강평 1

    평균 평점 5.0

    5

    100% 수강 후 작성

    Best lecture

    • hyunjinkim
      지식공유자

      Hello, Lee Ji-hoon. Thank you for your short but powerful review. You have taken the class 100%. I hope it was helpful to you. ^^

  • kimbyl님의 프로필 이미지
    kimbyl

    수강평 2

    평균 평점 5.0

    5

    100% 수강 후 작성

    This is an Airflow beginner course, but it was very helpful as it explained things in depth. Thank you.

    • hyunjinkim
      지식공유자

      Thank you for your review, Buing-ryul. I'm glad that it was helpful to you. ^^ I hope you use it well in your field.

  • everythx님의 프로필 이미지
    everythx

    수강평 10

    평균 평점 5.0

    5

    78% 수강 후 작성

    This was a really necessary lecture and I am satisfied! Airflow has new features when it is updated, so I would appreciate it if you could update those features as well.

    • hyunjinkim
      지식공유자

      Thank you for your review, everythx. I have a lecture that I'm currently working on, and once it's done, I'm planning to update airflow with new features. I'm also a working person, so I don't have much time, so it won't be right away, but I'll definitely update it. Thank you for your suggestion^^

비슷한 강의

같은 분야의 다른 강의를 만나보세요!

강의상세.할인문구

$101,640.00

30%

$112.20