dbt, The New Standard of Data Analytics Engineering

🔥 End the recurring pain of DW operation with dbt.

😥 Do you find yourself spending every day tracking complex dependencies in your data pipeline, hunting down broken data, and sighing at documentation that no one updates?

If you know SQL and Airflow, but still feel like you're reinventing the wheel by spending time on these repetitive tasks, you've come to the right place.

Have you been paying attention to dbt (data build tool), which is changing the landscape of data analytics engineering? Recently, even major Silicon Valley big tech companies are increasingly requesting dbt skills in their job postings (JDs), demonstrating that dbt is no longer an optional skill, but rather a must-have. Indeed, dbt is creating significant innovation by offering powerful solutions to several long-standing challenges in the data warehouse space.

(Even in Big Tech JD, dbt is slowly creeping in...)

However, the reality is that many companies and teams are still failing to fully utilize the true potential of dbt. "DBT is so great, but how do we integrate it into our warehouse?" For those seeking an answer to this question, the core goal of this course is to apply dbt to real-world tasks , eliminate inefficiencies in data warehouse operations, and help data engineers focus on more important, fundamental problems .

This course goes beyond a simple functional explanation. It delves into the background of dbt's creation (" Why ") and the core principles (" How ") that address persistent challenges in the field, including data lineage management, quality assurance, documentation, and backfilling . Furthermore, I will demonstrate, through my own experience and vivid examples, how the practical application of these principles, along with practical applications like effective integration with Airflow, can lead to remarkable results : a minimum fivefold increase in work productivity ("Impact") .

Leave repetitive, simple tasks to dbt. And with the new possibilities dbt opens up, I hope you'll seize this opportunity to explore the core values of data, including data architecting/modeling and the data engineering lifecycle, and grow into a more capable engineer.

The features of this lecture are:

📖 Systematic curriculum based on storytelling

Unlike the official DBT documentation or lectures, which follow a feature-by-feature omnibus format, this book is structured in a storytelling style, starting with the DBT project setup and gradually expanding and deepening its functionality . Each section is organically connected, allowing for a natural understanding of DBT's overall picture.

💻 Theory + Hands-on + Practical Tips

Provides rich hands-on experience building and running dbt projects directly in a local environment (DuckDB) along with explanation of core dbt concepts.

🆚 Airflow Integration: A Clear Comparison with Existing Methods

This course covers how to use Airflow, a data pipeline orchestrator, with dbt. Specifically, it compares the inefficient pipeline construction process when using Airflow alone (without dbt) with the dramatic improvement achieved when dbt is introduced, demonstrating the power of the dbt + Airflow combination.

📈 The secret to increasing productivity by 5x: Experience-based know-how

Rather than simply listing DBT features, this book vividly conveys why DBT is powerful and how it can dramatically increase work productivity, based on the challenges and problem-solving processes experienced while implementing DBT from the ground up in actual fields.

Learn things like this

Hands-on examples of lecture practice

After taking the class,

✅ You will learn the core functions of DBT and gain a level of understanding that can be applied immediately in practice .

✅ Beyond simply 'knowing' DBT, you will gain a clear understanding of 'why' and 'how' to use DBT and be able to use it confidently.

✅ DBT can improve the quality and stability of your data pipeline and enhance collaboration efficiency .

✅ Free yourself from the repetitive and time-consuming tasks of running a data warehouse (lineage tracking, manual documentation, complex backfill, etc.) .

✅ With dbt, you can grow into an engineer who can focus on core values such as designing efficient SQL logic that accurately reflects business requirements and increasing the structural completeness of data models, freeing you from repetitive operational tasks.

I recommend this to these people

Things to note before taking the course

💻 Practice environment

• This lecture is based on the macOS environment.

  • Windows environment users may need to set up Anaconda prompt or WSL2 (as in the previous lectures), but no guide is provided for these.

    
    

• It is assumed that Anaconda and other environments have already been set up. (Creating an Anaconda virtual environment and using pip are not described.)

• Library version used

  
  

   astronomer-cosmos==1.9.2
  dbt-duckdb==1.9.2
  dbt-core==1.9.4

• Other major tools used besides libraries: VSCode (with Cursor), DuckDB, DBeaver

• The learning materials are provided as source code based on the final version of the lecture (dbt repo, airflow repo).

🧑‍💻 Lecture materials

• The lecture materials (source code) provided contain the completed form as of the last lecture.

• Therefore, we recommend using the provided code as a reference when you get stuck, and practicing by writing your own code according to the course curriculum. By manually entering and executing code, you'll gain a deeper understanding of the material.

🚨 Note

• Please be sure to watch the orientation . It is crucial for understanding the course objectives and scope.

• The library versions used in the lecture videos may differ from the latest library versions at the time of course registration. As with previous lectures, this course focuses on principles, providing a deep understanding of dbt's core philosophy and problem-solving approach . Therefore, another learning goal is to cultivate practical skills to navigate and adapt independently, referencing official documentation and community resources, without being overwhelmed by library version differences or grammar changes. (With a solid grasp of the core principles, you will be able to proactively adapt to a changing technological landscape.)

• Rather than following along from the beginning, it may be more effective to go through the entire flow once and then come back and do the hands-on part (there is a high correlation between sections).

• This course does not cover data extraction from source systems, initial loading into a data lake/warehouse, or real-time streaming data processing. This course focuses on data table transformation and modeling at the warehouse layer.

  


Introduction of knowledge sharers

I am a developer who solves various problems in the field of data engineering.

I, too, have encountered numerous challenges while building and operating a data warehouse (DW) in my field, including difficulties tracking data lineage, repetitive backfill tasks, and managing out-of-sync documents. It was during this painful (or perhaps painfully painful) journey that I discovered dbt, and I personally experienced the remarkable productivity gains and positive changes in development culture it brought. Using dbt, I efficiently manage approximately 5,000 data assets (source tables, DW tables, dashboards, etc.) at work. This has dramatically improved productivity and positively impacted our development culture. I also use it as a core component of my personal quant investment system.

It was disheartening to see so many of my fellow engineers still stuck in the inefficient practice of "reinventing the wheel." So, I prepared this lecture to share the innovative experiences and problem-solving know-how I gained through dbt, creating a pathway for you to more easily and effectively immerse yourself in the powerful world of dbt.

I hope this course goes beyond simply teaching you how to use the tool, and becomes an opportunity for you to break free from repetitive tasks and experience the innovation and growth in the way you work that dbt brings .

Other courses on Inflearn:

• Python is something that anyone can learn, whether you're a liberal arts major or a non-major!

  


• Python Web Crawling & Automation to Take Over My Work

• Data-Driven Quantitative Stock Investing with Python Part 1

• Data-Driven Quantitative Stock Investing with Python Part 2