[Apache Kafka Application Programming] From concepts to consumers, producers, connects, and streams!

Core knowledge for Apache Kafka!

Data Engineer Event Driven
Stream Processing Data Pipeline

Basic concepts + various functions for commercial environment application
The essence of Kafka is thoroughly captured in one lecture 😊

Is this perhaps your concern? 🤷‍♂️

• “The concepts used in Apache Kafka are too difficult.”
• “I want to create a repetitive data pipeline using Kafka.”
• “I don't know what makes Kafka better than other platforms.”
• “I wonder what the ecosystem around Kafka is like.”

1️⃣ Master the vocabulary and concepts used in Apache Kafka

Apache Kafka covers a wide range of terms. Terms like rebalancing, message keys, partitioners, accumulators, and tasks can be unfamiliar to developers unfamiliar with Kafka. Without a clear understanding of the concepts behind each term, understanding and developing Kafka can be quite challenging. This lecture will explain the precise concepts behind key terms used in Kafka.

2️⃣ Types and Understanding of Kafka Applications

Apache Kafka, a distributed event streaming platform, not only supports general messaging queuing functionality but also enables real-time stream processing of events. As a data platform, it's ideal for repeatedly creating and operating data pipelines. However, to utilize and leverage these capabilities, a solid understanding of the surrounding ecosystem is essential. This lecture covers producers, consumers, Kafka Streams, and Kafka Connect, providing application developers with insight into stream data processing.

3️⃣ Learn the functions and options used in practice

When developing Kafka applications, you don't need to learn and master every feature from the beginning. While the consumer already provides over 90 options, you don't necessarily use them all. This lecture will cover the options you should configure and apply immediately when developing your application, as well as the option values I used during development. These guidelines will help you reduce trial and error and deploy Kafka applications to production environments more quickly.

4️⃣ Designing a stream data architecture using Apache Kafka

Kafka's features and options alone are limited in handling streaming data. Depending on the technology and method used, Kafka's usability varies significantly. This lecture will not only provide a technical explanation, but also provide guidelines on how to apply Kafka-based architectures in practice through the types and history of Kafka-based architectures and case studies.

Q. Do I have to purchase the book [Apache Kafka Application Programming with Java] to take this course?

The techniques and code covered here are based on "Apache Kafka Application Programming with Java." However, since all the material covered in this course is provided in PDF format, you don't necessarily need to purchase the book. If you'd like to learn and review additional content (MirrorMaker2, AWS MSK, etc.), purchasing this book is a great option.

Q. Can non-majors also take the course?

This course faithfully explains the fundamental concepts of Apache Kafka and delves into the features and options required for real-world development environments. Furthermore, the course is structured so that even those unfamiliar with other software or big data platforms can easily learn the concepts, making it ideal for non-majors seeking a career as a data engineer.

Q. Why learn using Java? Can I still learn without knowing Java?

The official libraries provided and supported by the open-source Apache Kafka are Java. Furthermore, developing Kafka Connect and Kafka Streams requires the official Java libraries as dependencies. Therefore, to truly appreciate the value of open-source Apache Kafka, you must develop Java-based applications (producers, consumers, Streams, and Connect). Third-party libraries (in other languages) don't provide the same functionality and cannot guarantee perfect compatibility, so this course is written in Java.

Don't worry if you're not familiar with Java. The code is explained line by line, so you'll be able to follow along without difficulty.

Q. Do I need a MacBook to do the training?

No, you can practice on operating systems other than MacBooks. While the lecture and practical environment are based on a MacBook, the scripts are designed to run on Windows or Linux, as long as the JVM is running. Therefore, don't worry even if you don't have a MacBook.

Students on Windows may need to set up a WSL development environment to run shell scripts. For detailed instructions, please refer to Setting Up a WSL Development Environment (link) .

Q. What PC specifications are required for the training?

To run ZooKeeper and the Kafka broker locally and conduct the exercises, you need a laptop or desktop with at least 8GB of memory. An i3 (3GHz) or higher CPU is recommended. Finally, you need at least 1GB of free space on an SSD or hard drive for storage.