Realtime Datalake Using Kafka & Spark
hyunjinkim
Beginner's Kafka & Spark Real-time Pipeline Course. All-in-one: Master concepts to architecture.
초급
Kafka, Apache Spark, pyspark
This is a course to learn about Airflow, an Orchestration tool for efficiently building and managing data pipelines. Welcome to the Airflow Master Class, where even beginners can learn step-by-step!
1,007 learners

Airflow Concepts and Basics
Airflow-based Pipeline Development
Sending Automated Emails with Airflow
Airflow-based Public Data API Calls and Visualization
Airflow & Kakao, Slack for Message Alarm
Utilizing ChatGPT with Airflow
Data Pipeline, No More Worries with Airflow 📊
👉 It covers everything from the basic concepts of Apache Airflow to the architecture configuration that can operate in a large-scale environment.
👉 About 80 practice files can be downloaded from Github .
But why Airflow?
Airflow is a core orchestration solution that creates and manages data pipelines that extract, process, store, and analyze data.
Airflow is the most popular pipeline management tool among similar solutions, and its adoption continues to grow.
Airflow Basics
You will learn the basics of Airflow, including the concepts and how to create workflows, through hands-on practice. It is organized so that you can learn step by step with about 60 practice files.
Pipeline Configuration
Learn how to develop and run a DAG pipeline using Airflow, including sending emails with scheduling management.
Data collection
Let's configure a pipeline that receives and stores data via API from the Seoul Metropolitan Government Public Data Portal.
Monitoring and Integration
We will practice receiving alarms such as error messages and DAG status by linking with messenger apps such as KakaoTalk and Slack.
Data Visualization
We introduce the concept of R Shiny, which can be used for visualization using the R language. We will proceed with visualization using data received from the Seoul Public Data Portal.
Architecture
Learn about Airflow's different deployment approaches and architectures, and how to operate reliably in high-volume environments.
Automation of business
Introduce the concept of ChatGPT and learn how to connect Python API and ChatGPT. Practice automation by automatically posting to your blog the content introduced by ChatGPT about stocks that are rising rapidly through the method of retrieving stock information with Python.
1. Basic knowledge of Python
2. Docker and Docker Compose
3. SQL
Q. How are the lectures conducted?
In Airflow, workflow is called DAG , and we will practice by creating DAG together. Except for the time explaining the basic concepts, we will basically practice in each chapter.
If the practice file is long, I create a DAG file in advance and proceed by explaining the logic.
Q. Can I download practice files and study materials?
Of course! You can get all the practice files from Github . Not sure how to use Git? We'll teach you how to use Git too.
We also provide all PDF-based learning materials. You can download them from Section 0 - Download Lecture Materials.
Q. How difficult is the practical training?
In the beginning, you can understand it by just knowing the basic grammar of Python, but as you progress to the latter part, the difficulty level can be a little difficult, so it will be helpful to know concepts such as Python classes and inheritance. But don't worry. The practical content will be explained sufficiently and you will proceed.
Q. What can I do if I learn Airflow?
Bash Shell, anything you can do with Python, you can do. If you're wondering if something can be done with Airflow, first find out if it can be done with Bash Shell or Python. If you can do it with Bash Shell or Python, you can do it with Airflow.
Who is this course right for?
Those who want to learn about Data Engineers
Those curious about Airflow
Airflow users not utilizing it well
Requiring data pipeline setup and management.
Need to know before starting?
Python Fundamentals
Docker & Docker Compose Usage
SQL Basic Syntax(SELECT, FROM)
1,247
Learners
81
Reviews
219
Answers
4.9
Rating
2
Courses
안녕하세요.
데이터 & AI 분야에서 일하고 있는 15년차 현직자입니다.
정보관리기술사를 취득한 이후 지금까지 얻은 지식을 많은 사람들에게 공유하고자 컨텐츠 제작하고 있습니다.
반갑습니다. :)
Contact: hjkim_sun@naver.com
All
107 lectures ∙ (24hr 56min)
Course Materials:
4. WSL installation
16:44
All
62 reviews
4.9
62 reviews
Reviews 1
∙
Average Rating 5.0
5
If there is someone around me who is just starting to study data engineering, I would definitely recommend it. (From a non-major perspective) When I started studying data engineering, I was told that I needed to know git, Linux, Python, and Airflow, but I was confused for a long time because I didn't know how much I needed to know about each. While taking this lecture, I learned the basics of git and Linux that are necessary for Airflow, and it was good. Also, I heard that there will be a lot of DAG practice in the future, so I am quite excited. I will diligently take the course and master Airflow as the title suggests! If you release another Data Engineer lecture, I would definitely listen to it!
And I really like how you kindly explain even the smallest details. This is the most satisfying lecture I've ever taken!
Hello dj961024 Thank you for your touching review ^_^ As someone who believes that understanding basic principles is the most important, as with anything, I thought a lot about how to easily understand the concepts. I'm so glad that it was helpful to you. If you have any questions during the course, please feel free to ask and I hope you continue to study hard!
Reviews 10
∙
Average Rating 5.0
5
This was a really necessary lecture and I am satisfied! Airflow has new features when it is updated, so I would appreciate it if you could update those features as well.
Thank you for your review, everythx. I have a lecture that I'm currently working on, and once it's done, I'm planning to update airflow with new features. I'm also a working person, so I don't have much time, so it won't be right away, but I'll definitely update it. Thank you for your suggestion^^
Reviews 2
∙
Average Rating 5.0
5
This is an Airflow beginner course, but it was very helpful as it explained things in depth. Thank you.
Thank you for your review, Buing-ryul. I'm glad that it was helpful to you. ^^ I hope you use it well in your field.
Reviews 1
∙
Average Rating 5.0
5
Best lecture
Hello, Lee Ji-hoon. Thank you for your short but powerful review. You have taken the class 100%. I hope it was helpful to you. ^^
Reviews 1
∙
Average Rating 4.0
4
I was able to easily understand AIRFLOW's functions through various practical exercises.
Hello ahn.bm Thank you for registering your course review. Is there anything you are dissatisfied with? Please give me your suggestions and I will try to reflect them. I hope it was helpful.
Limited time deal ends in 7 days
$101,640.00
30%
$112.20
Check out other courses by the instructor!
Explore other courses in the same field!