inflearn logo
inflearn logo

Apache Airflow with Silicon Valley Engineers

You will learn Apache Airflow, the most widely used Orchestrator for creating software data pipelines.

(4.6) 수강평 53개

강의소개.상단개요.수강생.short

난이도 초급

수강기한 무제한

airflow
airflow
Big Data
Big Data
Data Engineering
Data Engineering
Python
Python
airflow
airflow
Big Data
Big Data
Data Engineering
Data Engineering
Python
Python

먼저 경험한 수강생들의 후기

먼저 경험한 수강생들의 후기

4.6

5.0

chocheetah

100% 수강 후 작성

I was considering introducing MWAA in order to separate the existing scheduler in the company and manage the execution history, etc., and I took this lecture. When I had no idea how to start airflow, I wrote the code, configured the local docker environment, and created the DAG, and I got a good sense of how to use it. It may not be easy to find the best practice for each platform in the operating environment, but I think I acquired basic knowledge that I can touch on various things by reading the official documentation. Thank you.

5.0

ground

35% 수강 후 작성

It seems like a lecture that lets you know what tools data engineers use and what those tools do. I will keep going. Thank you for always making good lectures.

5.0

소연

100% 수강 후 작성

I signed up for the course because I wanted to use it well in my work. It seemed like a perfect fit for basic study. I liked that it was focused on practical training.

강의상세_배울수있는것_타이틀

  • Build a data pipeline

  • Apache Airflow

  • Introduction to Apache Spark

  • Introducing MinIO

  • TaskFlow API

  • Slack Integration

Airflow, an essential tool for data pipelines
Learn from Silicon Valley developers! 🔥

Modern data workflow management,
With Apache Airflow 📌

The most used workflow management tool in the field: Apache Airflow

Learn the simple setup and usage of Airflow. We'll guide you through creating your first data workflow.

Leave complex concepts behind and dive into the fascinating world of Airflow!

As data analysis and processing tasks become more complex , issues such as job scheduling, dependency management, and error handling become more important. To effectively address these challenges, Airflow is one of the core tools of choice for many organizations.

This lecture is a lecture prepared in advance for those who are new to Airflow. Apache Airflow, which is popular in Silicon Valley. I will help you get started in a simple and easy way.

Why should you learn Apache Airflow?

Automated workflow management

Airflow provides powerful scheduling capabilities to automatically run and manage tasks over time. You can plan and execute data processing tasks more efficiently.

Dependency Management

Complex data workflows require precise management of dependencies between tasks. Airflow provides the ability to clearly define dependencies between tasks and specify the order of tasks.

Powerful monitoring and notifications

You can monitor the progress of your jobs through the Airflow dashboard while your jobs are running. You can also set up alerts to respond quickly if your jobs fail or encounter issues.

Scalability and flexibility

Airflow supports a variety of plugins and libraries. It can integrate with various data stores, job execution environments, and notification mechanisms. Build custom workflows to fit your needs.

Community and Ecosystem

Airflow has a vibrant community and rich ecosystem, so there are a lot of great resources to help you troubleshoot.


Lecture Features ✨

✅ Easy to understand without difficult concepts! Explains what Airflow is and why it is needed through analogies and examples .

A hands-on course that follows the actual Airflow usage process and creates a simple data workflow!

If you have any questions or do not understand anything during the lecture, please feel free to ask questions at any time. Learn with Q&A !


What you'll learn 📚

  • All lecture materials are in English. The lectures themselves are conducted in Korean and are designed to facilitate future overseas employment.
  • We provide PDF lecture materials and Github code.
Cloud Software Architecture Overview
Introduction to Data Pipeline Orchestrator
Introduction to Apache Airflow
Introduction to the key components of Apache Airflow
Detailed introduction of each component
Detailed analysis of the code

We will share with you the know-how of current Silicon Valley engineers!

I am a current software engineer who runs YouTube's " American Engineer " and Brunch's " Silicon Valley News and Life ." I graduated from the University of California, Berkeley EECS and am currently working on big data at the headquarters of a global big tech company in Silicon Valley. I would like to share the know-how I have learned from my actual work with many people. 🙂


If you are this type of person
Get started right now.

💡

Data Engineer

Automate and schedule data workflows to maintain data quality and consistency .

💡

Data Analyst

Try handling tasks like regular data updates or model retraining .

💡

Data Scientist

Efficiently manage your data science process by automating model training, evaluation, batch predictions, and more .

💡

system administrator

It can increase transparency and reliability of work execution.

💡

Data Engineering and Development Team

You can implement various automated tasks such as ETL (Extract, Transform, Load) tasks and API calls .

💡

Project Manager

You can effectively adjust your project schedule by setting task dependencies, priorities, expected execution times, etc.


Expected Questions Q&A 💬

Q. Why should I learn Apache Airflow?

Apache Airflow is a data workflow management tool used to automate, schedule, and monitor data pipelines. It can efficiently manage data workflows for various roles such as data engineers, data scientists, and system administrators.

In the latter half of the lecture, you will learn how to integrate with big data technology (Apache Spark), which will be of great help to data engineers who manage many pipelines.

Q. Is this a lecture that non-majors can also take?

If you are not a major but know the basics of Python and want to improve the efficiency of your data or task scheduling workflow, this will be of great help.

If you are new to Python, learn the basics of Python through YouTube or take the lecture below first! Even if you only watch the basics, you will have no trouble following the entire lecture.

Q. Is there anything I need to prepare before attending the lecture?

Since the code is written in Python, there will be no basic Python lectures. Also, there will be practical training using Docker, so it will be easier to understand if you have basic knowledge of Docker.


Things to note before taking the class 📢

Practice environment

  • Operating System and Version (OS)
    • The course will be taught on MacOS, but you can follow the exercises on any operating system that has Python (Airflow itself is a Python library).
  • Tools to use
    • Python 3.7+
    • Airflow is Apache licensed, so it's free.
  • PC specifications
    • CPU: 2 cores or more
    • Memory: 4GB or more
    • Disk: 10GB or more

Player Knowledge and Notes

  • Basic knowledge of Python and Docker is required, and the environment for this lecture is set up with Docker. If you want to know more about Docker, I recommend you refer to my free Docker lecture . Lecture link: [ https://inf.run/8eFCL ]
  • If you have any questions, please feel free to ask. However, since I am located in the western United States, it may take some time for me to respond.

강의소개.콘텐츠.추천문구

학습 대상은 누구일까요?

  • Data Engineer

  • If you want to become a data engineer

선수 지식, 필요할까요?

  • Python

강의소개.지공자소개

20,804

수강생

1,047

수강평

337

답변

4.8

강의 평점

29

강의_other

Are you going to finish in Korea? Penetrate the global market with English! 🌍🚀

Hello. I majored in Computer Science (EECS) at UC Berkeley 💻, have worked as a software engineer in Silicon Valley for over 15 years, and am currently a Staff Software Engineer working with Big Data and DevOps at a Big Tech headquarters in Silicon Valley.

  • 🧭 I would now like to share the technologies and know-how I learned firsthand at the forefront of innovation in Silicon Valley with all of you through online lectures.

  • 🚀 Join me, having learned and grown at the forefront of technological innovation, and develop the skills to compete on the global stage!

  • 🫡 I may not be the smartest, but I want to emphasize that you can achieve anything if you stay consistent and never give up. I will always be by your side, supporting you with great resources.

 

더보기

커리큘럼

전체

33개 ∙ (강의상세_런타임_시간 강의상세_런타임_분)

해당 강의에서 제공: [object Object]
강의 게시일: 
마지막 업데이트일: 

수강평

전체

53개

4.6

53개의 수강평

  • sanghyunwi8994님의 프로필 이미지
    sanghyunwi8994

    수강평 1

    평균 평점 5.0

    5

    100% 수강 후 작성

    • altoformula
      지식공유자

      Hello Wi Sang-hyun, Thank you so much for taking the time to leave such a wonderful review.

  • jungmob1024님의 프로필 이미지
    jungmob1024

    수강평 6

    평균 평점 5.0

    5

    30% 수강 후 작성

    • altoformula
      지식공유자

      Hello jungmob1024, Thank you for taking the time to leave such a great review.

  • paulmoon008308님의 프로필 이미지
    paulmoon008308

    수강평 111

    평균 평점 4.9

    5

    12% 수강 후 작성

    • altoformula
      지식공유자

      Hello Spring, Thank you for taking the time to leave a great review!

  • jhlee50877003님의 프로필 이미지
    jhlee50877003

    수강평 3

    평균 평점 5.0

    5

    30% 수강 후 작성

    • altoformula
      지식공유자

      Hello jhlee5087, Thank you for taking the time to take both the Spark and Airflow courses and for leaving such positive reviews.

  • hgim967155182님의 프로필 이미지
    hgim967155182

    수강평 5

    평균 평점 4.8

    5

    61% 수강 후 작성

    I love how you explain things so clearly that even non-majors can easily understand. Learning Airflow is so much fun!

    • altoformula
      지식공유자

      Hello Gong16, Thank you for taking the time to leave such a great review. I'm glad to hear that it was easy to understand! I'm happy that you seem satisfied with the lecture.

altoformula님의 다른 강의

지식공유자님의 다른 강의를 만나보세요!

비슷한 강의

같은 분야의 다른 강의를 만나보세요!

강의상세.할인문구

$49,500.00

25%

$51.70