Kakao Interviewer's Hands-on Spring Batch: Everything About Large-Scale Data Processing
The biggest challenges in operating batch systems that process millions of records are performance degradation and the difficulty of recovery in the event of a failure. These issues cannot be solved with simple loops or poor design. Based on my experience of reliably processing large-scale data in real-world service environments, this course covers the structure and operating principles of Spring Batch and provides design methods to achieve both performance and stability. Moving beyond simple functional descriptions, this course will help you develop fundamental data processing problem-solving skills through batch architectures that can be applied immediately in practice.
I really enjoyed the lecture. This was my first time learning about Spring Batch, and your explanation was so easy to follow that I could understand everything even as a beginner. It was very helpful!
5.0
letes ky
92% enrolled
It was a great lecture that provided a glimpse into the interviewer's insightful perspectives. Even though I feel like I've been working in development for quite some time, this lecture made me realize there is still so much more to learn.
Thank you for the great content.
5.0
warna
88% enrolled
I was able to focus on and understand just the key points perfectly.
What you will gain after the course
Spring Batch design capability to reliably process large-scale data (millions to tens of millions of records)
Practical skills to implement chunk-based processing, transaction management, and retry/skip strategies in actual code
Ability to design scalable and maintainable batch architectures utilizing Job / Step structures
Problem-solving skills to independently design and apply reprocessing and recovery strategies in the event of a failure
Experience in implementing parallel processing and optimizing batch performance using multi-threading, partitioning, and other techniques.
Development · Programming · Backend
You cannot process tens of millions of records using simple loops. A Spring Batch course for handling large-scale data.
This course is a hands-on lecture that treats Spring Batch not just as a "way to run batches," but as a design for reliably processing large-scale data. You will directly implement everything from Job, Step, and Chunk-based transactions to methods for selecting and reprocessing only failed data, as well as techniques to reduce processing time through parallel processing.
Difficulty: Beginner
Practice-oriented
Java 17 · Spring Boot
Unlimited access period
Job → Step → ChunkBuild the structure by hand from scratch
Skip · Retry · ParallelFrom fault recovery to performance
A curriculum co-designed by a current Kakao server developer and interviewer Although the difficulty level is introductory, the fault tolerance and parallel processing in sections 5 and 6 will be new even to those already operating batches The practice environment is macOS (Apple M3), Java 17, and VS Code
Spring Batch · Large-scale Batch FrameworkJob · Step Based ArchitectureJobRepository · JobLauncher · ExecutionContextTasklet Step and Custom ImplementationChunk Oriented Processing and TransactionsReader · Processor · Writer WorkflowSkip · Retry Fault ToleranceMulti Threaded Step Parallel Processingrestart from the Point of InterruptionJava 17 · Spring Boot · spring-framework24 Lectures · 5 Hours 24 MinutesSpring Batch · Large-scale Batch FrameworkJob · Step Based ArchitectureJobRepository · JobLauncher · ExecutionContextTasklet Step and Custom ImplementationChunk Oriented Processing and TransactionsReader · Processor · Writer WorkflowSkip · Retry Fault ToleranceMulti Threaded Step Parallel Processingrestart from the Point of InterruptionJava 17 · Spring Boot · spring-framework24 Lectures · 5 Hours 24 Minutes
Why this lecture was started
It all started with the comment that running batches is terrifying every time.
Below is the actual conversation. It was the day a Kakao interviewer replied to a Toss developer's complaint by saying, "I'll turn this into a lecture."
Toss DeveloperI'm scared it'll crash every time I run a batch lately... The data has grown too much.
Choi (Kakao Interviewer)How many cases are you processing?? Since it's in the finance sector, I imagine there must be a lot.
Toss DeveloperOnce it runs, tens of millions of records are a given... The processing speed is an issue, and if a failure occurs, recovery is just too difficult.
Ande (Naver)Hmm... Spring itself is heavy, but if you apply things like Multi-Threading, it doesn't run that heavily. Unless it's a DB issue.
Choi (Kakao Interviewer)I agree. Recovery isn't that difficult if you implement things like Listeners. If you add Skip and Retry, it even supports individual transactions for them.
Choi (Kakao Interviewer)I'll try to melt this into a lecture. There are more people than I thought who don't know Spring Batch very well.
HongWell, now... you don't even ask about my schedule anymore, you just throw it at me;;
That is how this course was created. From a conversation between someone who finds Batch scary and someone who knows why it isn't.
“
There are more people than I thought who don't know Spring Batch very well.
Current Kakao Server Developer · Interviewer 12th year · Personally created the lecture after this conversation
What you are currently experiencing
Are you still handling large-scale data batch processing using simple loops?
When working with data in practice, there inevitably comes a moment that goes beyond simple CRUD. That is when the following problems all arrive at once.
You need to process millions to tens of millions of data records at once.
If it fails in the middle, there is no way to know from where it should be restarted.
I'm not sure how far to group the transactions or if the memory will hold up.
Batch scheduling and operations are being handled manually by people every time.
Can these problems be solved with a few lines of code, or will it take tens of thousands?
Why Spring Batch
What kind of framework is Spring Batch?
Spring Batch is a Java and Spring-based framework for processing large volumes of data. It goes beyond simple repetitive tasks to enable processing designed for stability and scalability. It can be used directly on top of Spring Boot and leverages the transaction management of the spring-framework ecosystem as is.
Chunk
Chunk는 Step 안에서 한 번에 처리할 데이터 묶음의 단위입니다. Spring Batch는 이 Chunk 단위로 데이터를 나눠 처리해 메모리 사용과 성능을 함께 잡습니다.
Transaction
Spring Batch는 Chunk 단위로 트랜잭션을 관리해, 실패했을 때 그 단위만 롤백하고 다시 처리할 수 있습니다.
Skip · Retry
Spring Batch는 특정 데이터가 실패해도 전체를 멈추지 않고 건너뛰거나 다시 시도하는 복구 전략을 제공합니다.
Job · Step
Spring Batch는 복잡한 배치 로직을 단계로 나눠 설계하게 해, 어디서 무엇이 도는지 명확해집니다.
Reader · Processor · Writer
ItemReader는 데이터를 하나씩 읽고, ItemProcessor는 읽어온 데이터를 가공하거나 걸러내며, ItemWriter는 그 결과를 Chunk 단위로 모아 저장합니다.
ExecutionContext
Spring Batch는 실행 상태를 저장해 중단된 작업을 그 지점부터 다시 실행(restart)할 수 있습니다.
Parallel Processing
Spring Batch는 멀티 스레드와 파티셔닝으로 처리 시간을 줄이는 병렬 처리 패턴을 지원합니다.
The official documentation describes Spring Batch as follows.
A lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems.
It is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. It provides features such as logging and tracing, transaction management, job statistics, restart, skip processing, and resource management in a reusable form.
A sneak peek at the actual course content
We cover scenes like this
Visuals are faster than explanations. I've cut and brought a portion of the actual lecture as it is.
JobRepository · JobLauncher · ExecutionContextCheck what remembers how far the batch has progressed
Spring Batch-managed workflow DDLDirectly check in the DB tables how far it rolls back when a failure occurs
Multi-threaded Step sharing a single ReaderHow much time is reduced when running in parallel
There is a point where batch processing is no longer intimidating.
24 lectures, 5 hours and 24 minutes. It starts from creating your first Job and leads all the way to parallel processing.
We cover the concepts you must know from an interviewer's perspective.
It doesn't just end with functional explanations. It covers how to design Spring Batch, focusing on the key points that are considered important in actual interviews and practical work.
Almost all of the time is dedicated to hands-on practice.
Batch processing is difficult to understand through theory alone. By implementing and executing it yourself, you will see firsthand how data is processed and where bottlenecks occur.
The point that distinguishes juniors from seniors is data processing.
This course covers the transition from a developer who has only built APIs to one who can design data flows.
Design for failure first.
You will master the structure of resuming from the point of failure by covering Skip, Retry, and ExecutionContext-based restarts.
What you will gain after taking this course
What can I do after finishing this course?
The following five items are things you will be able to implement in code or explain with supporting evidence during an interview after completing this course.
Design
처리할 데이터의 성격을 보고 Tasklet과 Chunk 지향 Step 중 무엇을 쓸지 고르고, 그 이유를 설명할 수 있습니다.
Transaction
Chunk 크기를 얼마로 잡을지 판단하고, 실패했을 때 어디까지 롤백되는지를 DB에서 확인할 수 있습니다.
Recovery
Skip과 Retry 중 무엇을 쓸 상황인지 구분하고, 중단된 Job을 그 지점부터 다시 이어 돌릴 수 있습니다.
Performance
멀티 스레드 Step과 파티셔닝을 적용해 같은 데이터를 더 짧은 시간에 처리할 수 있습니다.
Operations
JobRepository가 남긴 메타데이터를 읽어 어제 배치가 왜 실패했는지 추적할 수 있습니다.
I recommend it to these types of people
Who is this Spring Batch course for?
Backend developers whose batch jobs are slow or fail frequently but do not know the exact cause
Developers who have experienced failures due to memory or performance issues while processing large-scale data
Developers who are using Spring Batch but are implementing it inefficiently because they do not understand its structure
Developers who feel lost on how to design reprocessing and recovery logic when a failure occurs
Developers who want to grow beyond simple CRUD and develop data processing-oriented capabilities
Conversely, if you can immediately explain the boundaries of chunk-oriented transactions, the difference between skip and retry, and the conditions under which multi-threaded steps are safe, then you do not need this course.
Curriculum
Spring Batch Course Curriculum · 24 Lectures Total · 5 Hours 24 Minutes
It is structured in an order designed for developers learning Spring Batch for the first time. It starts with why batch processing is necessary and continues through core concepts, workflows, fault tolerance, and parallel processing.
Section 1. Course Introduction
3강 · 5분
Course Introduction05:52
Source CodeResources
Spring Batch DocsResources
Section 2. Why is batch processing necessary in various domains?
3강 · 32분
What do you think batch processing is?11:21
The unique ecosystem architecture provided by Spring Batch12:04
Practice environment setup and build configuration via Spring Initializr09:17
Section 3. Core Basic Concepts of Spring Batch
4강 · 1시간 14분
Writing and running your first Job and Step, and checking the DB status16:36
JobRepository, JobLauncher, and ExecutionContext22:37
In-depth analysis of the Tasklet Step interface and implementing a custom Tasklet15:34
Chunk-oriented Step Pattern and Transaction Correlation19:29
We will cover the flow of reading, processing, and writing data through hands-on practice.1:25:00
Section 5. Optional Patterns for Fault Tolerance and Troubleshooting
6강 · 1시간 29분
Covers recovery strategies including Skip and Retry, as well as troubleshooting.1:29:00
Section 6. The Key to Performance Improvement: 3 Parallel Processing Patterns
3강 · 36분
Covers three parallel processing patterns, including Multi-threaded Steps.36:00
Technologies Covered · Practice Environment
What will be used for the practice?
Spring Batch
Job · Step
Chunk
Tasklet
Skip · Retry
Multi Threaded Step
Java
Spring
Spring Boot
spring-framework
macOS (Apple M3)
We will proceed using Java 17.0.12 LTS on an Apple M3 Air environment, with VS Code as the editor. We will walk through everything together, starting from the step of creating a project with Spring Initializr.
If you have finished looking through the curriculum
The course duration is unlimited. You can revisit specific sections whenever you need them, such as when a system failure occurs.
Created together by active server developers and interviewers
This course was co-created by Hong, a platform server developer in Pangyo, and Choi, a server developer and interviewer with 12 years of experience at Kakao headquarters.
Knowledge Sharer · Pangyo Platform Server Development
Hong
I am in charge of platform server development in Pangyo. I continue my activities as a knowledge sharer to share the methods I studied personally and the problems and solutions I encounter in practice. I don't create these lectures alone; I develop them together with various developers currently working in the field.
[Current] Pangyo Platform Server Developer
Inflearn Knowledge Sharer · Operator of multiple back-end courses
[Former] Blockchain · Metaverse Backend Developer
Kakao · 12-year Backend Developer · Interviewer
Choi
I am a 12-year backend server developer currently working as both a server developer and an interviewer at Kakao. I met Hong at a conference and have been collaborating since the mid-stages of his teaching activities, creating various topics together. While a career at a major corporation doesn't prove one is a good developer, I believe it allows for gaining more traffic and experience than typical platforms, and I incorporate those experiences into the lectures.
[現] Kakao Headquarters Server Developer · Interviewer
[Former] Computer Science major at a 4-year university in Seoul
Frequently Asked Questions
Spring Batch Lecture Frequently Asked Questions
What is Spring Batch?
Spring Batch is a Java/Spring-based batch framework designed for processing large volumes of data. It provides reusable functions such as logging and tracking, transaction management, job statistics, restarts, and skip processing to help stably process millions of records or more.
What are Jobs and Steps?
A Job is a unit that refers to an entire batch process, and a Step is an execution phase that makes up that Job. A single Job is divided into multiple Steps, and each Step is executed sequentially or based on specific conditions. In Section 3, we will create and run our first Job and Step, and even check the database status.
What is a Chunk?
A Chunk is the unit of data processed at once within a Step. For example, if the Chunk size is 1,000, it reads 1,000 items, processes them, saves them all at once, and then commits. Since transactions are grouped by this Chunk unit, if a failure occurs, only that specific Chunk is rolled back. In Section 3, we will cover the relationship between Chunk-oriented Steps and transactions through hands-on practice.
What do the Reader, Processor, and Writer each do?
ItemReader reads data to be processed one by one, ItemProcessor processes the read data or filters out items that do not meet certain conditions, and ItemWriter collects the processed data into chunks and saves them. These three components are executed repeatedly to form a single Chunk-oriented Step. Section 4 covers this entire workflow.
When should Tasklet and Chunk-oriented Steps each be used?
Tasklets are suitable for simple tasks that run once and finish, while Chunk-oriented Steps are used when repeating a flow of reading, processing, and writing in large volumes. Most large-scale processing is Chunk-oriented. In Section 3, we will examine the Tasklet interface and try a custom implementation, followed by a discussion on the relationship between Chunk-oriented processing and transactions.
If a batch fails in the middle, do I have to run it again from the beginning?
No. Since Spring Batch records the execution status in the ExecutionContext, you can resume execution from the point where it stopped. If the issue is limited to specific data, you can skip it using Skip or try again with Retry. We will cover these recovery strategies through hands-on practice in Section 5.
How do you increase batch processing speed?
You start by adjusting the chunk size, and then use Multi-Threaded Steps where multiple threads share a single Reader, or partitioning where data is divided and processed in parallel. Section 6 covers three parallel processing patterns, and you can see one of them in the preview video.
Does Spring Batch also provide scheduling features?
No. Spring Batch itself does not have a scheduling function. Deciding when and how often to execute a Job is the responsibility of Spring's scheduler or an external scheduler, while Spring Batch is responsible for reliably processing the executed Job and recovering in case of failure. The two are not in competition; rather, their roles are clearly divided.
Can beginners to Spring Batch follow along?
Yes. As an introductory-level course, we start from why Batch is necessary and go through the steps of creating a project with Spring Initializr together. Experience with Java and Spring Boot is sufficient, and the sections on fault tolerance and parallel processing towards the end will offer new insights even for those already operating Batch.
How should the practice environment be configured?
It will be conducted using Java 17.0.12 LTS and VS Code on an Apple M3 Air. In Section 2, we will go through the process of creating a project via Spring Initializr and setting up the build configuration together, so you can simply follow along.
Is there a reason to take this course even if I am already using Spring Batch in practice?
Making it work and explaining why it is structured that way are two different things. If you can immediately answer where you set the chunk boundaries, why you chose Skip over Retry (or vice versa), and under what conditions a multi-threaded Step is safe, then you do not need this course. If you struggle to answer, Sections 3, 5, and 6 will fill those gaps.
What is the order of the Spring Batch course curriculum?
It starts with why batch processing is necessary and the Spring Batch architecture, followed by core concepts such as Job, Step, Tasklet, and Chunk, the Reader-Processor-Writer workflow, fault tolerance and troubleshooting, and finally parallel processing patterns. The course consists of 24 lectures totaling 5 hours and 24 minutes, with unlimited access.
As data accumulates, the people who handle it must also evolve.
From creating your first job to recovery and parallel processing, you will verify everything through 24 hands-on practice sessions.
We operate a developer open chat room where we share career advice, real-world work situations, and technical knowledge. Feel free to share any questions you have while taking the course or any practical concerns from your work.
I started studying development after becoming interested in it while idling at home, and I am currently responsible for platform server development in Pangyo. I am continuing my activities as a knowledge sharer because I want to provide you with the methods I used to study, as well as the various problems and solutions you may encounter in practice.
These lectures are not created solely through my own knowledge. There are others who collaborate on every lecture.
I really enjoyed the lecture. This was my first time learning about Spring Batch, and your explanation was so easy to follow that I could understand everything even as a beginner. It was very helpful!
It was a great lecture that provided a glimpse into the interviewer's insightful perspectives. Even though I feel like I've been working in development for quite some time, this lecture made me realize there is still so much more to learn.
Thank you for the great content.