LLM Evaluation Harness Engineering Explained by a Silicon Valley Developer

You may have experienced Claude Code producing different results for the same request each time, violating the rules specified in its instructions, and being overly lenient in approving its own output. While developing platform servers in Pangyo and spending over $1,000 a month on Claude Code, I addressed these issues by building a harness consisting of instructions, Hooks, permissions, and a Sandbox instead of relying on prompts. In this course, you’ll build that harness from scratch, add Subagents, Agent Teams, and headless execution, and leave with a complete setup that you can copy and use as-is in other projects.

For developers blocked by double payments and discrepancies between external payments and databases, I design and implement robust settlement systems based on hands-on experience with Toss’s backend.

This course is for developers who have built an agent but are unsure how to connect it with other services and frameworks, or what formats to use when exchanging long-running tasks, approvals, statuses, and results. Using a LangGraph agent as the foundation, you will implement the A2A protocol yourself and learn step by step about Agent Cards, Tasks, Artifacts, streaming, authentication, webhooks, and gRPC. Finally, you will verify compatibility with the official SDK.

If you’ve found it difficult to explain what’s happening under the hood while using AI and embeddings, this course may be the answer. Drawing on the experience of a practicing AI developer in Silicon Valley, it helps you understand the principles of AI through code by implementing everything from embeddings and Word2Vec to automatic differentiation and transformers yourself.

You know about AI agent frameworks, but understanding how they work and applying them in practice can be difficult, right? Drawing on firsthand experience from Silicon Valley, we’ll guide you from the core principles to practical applications.

The limitations of LLMs? I’ll share practical solutions to real-world problems based on RAG trial and error and operational know-how from working in Silicon Valley.

Learn the process of building essential observability in a microservices architecture (MSA) environment, from the basics to hands-on practice. Through step-by-step exercises, you will learn standardized data collection using OpenTelemetry, trace storage and search with Grafana Tempo, and visualization using TraceQL. You will also learn how to trace service flows through distributed tracing, identify bottlenecks, and analyze the causes of failures.

A Redis course created with a current Kakao interviewer. From hands-on practice with data structures ranging from String to Sorted Set to Sentinel and cluster configuration, you’ll build design rationale that holds up in interviews and real-world work.

Learn the SAGA transaction management technique for maintaining data consistency in distributed environments through a hands-on approach. You will configure three microservices using Spring Boot and Docker, and implement both Orchestration and Choreography patterns using Kafka and MySQL. Through this course, you will master core concepts applicable to real-world scenarios, including the operational principles of distributed transactions, designing compensating transactions, and handling fault recovery.

This course explores microservice architecture and distributed environments not as a simple list of technologies, but from the “design perspective” required in real-world practice and technical interviews. Starting with the differences between monolithic architectures and MSA, it systematically covers gradual migration strategies, modular monoliths, clean architecture, and hexagonal architecture, clearly explaining why each choice is necessary and the trade-offs that come with it. Through hands-on exercises, you can directly verify the intentions behind each structure and learn about the core challenges inevitably encountered in distributed environments—such as data consistency, transactions, inter-service communication, and failure handling—through realistic scenarios. Based on actual questions and perspectives from Kakao interviewers, this course aims to do more than impart knowledge: it seeks to elevate your way of thinking about distributed systems to the next level.

This course goes beyond basic CRUD-level database usage, helping you develop the PostgreSQL skills needed in the AI era from a practical, real-world perspective. Drawing on AI development experience in Silicon Valley, you’ll gradually learn how to design and use JSON, indexes, transactions, and pgvector in actual services, while gaining the skills to solve complex data structure and performance problems.

Learn how to design and process large-scale data architectures capable of handling hundreds of billions of records, using only pure SQL and MySQL’s core features, alongside a Toss developer who handles hundreds of billions of financial transactions and a developer who started without a computer science background and now works on platform development in Pangyo.

As you work in backend development, you eventually find yourself facing a certain concern. As services multiply and traffic increases, there comes a point when simple REST communication and a CRUD-only structure begin to show their limitations. I experienced the same problem in practice and went through many trials and errors trying to balance performance with structural complexity. The solution I chose in that process was NATS. Its simple, single-binary-based setup, ultra-low-latency performance measured in microseconds, and practical feature set—from Pub/Sub and Queue Groups to JetStream—made it a highly realistic choice for actual service environments. This course is not about listing NATS’s features. Based on the problems I encountered in practice, it explains through code and workflows when and why I chose NATS and how I designed the architecture. From those new to messaging systems, to backend developers preparing for employment, and senior developers considering better architectures, this course will help everyone understand distributed systems and grow to the next level.

When Redis alone can no longer guarantee performance, what options do we have?? There is Valkey, which is 100% compatible with Redis and applies IO-based tuning. Through this course, let’s explore how Valkey guarantees 1M++ TPS and learn about the data structures suited to different traffic patterns and feature implementations.

Systematically learn the distributed NoSQL database Cassandra, from basic concepts to practical application. Develop high-volume data processing capabilities through practice-oriented theory and hands-on exercises, including data modeling, query optimization, and cluster configuration. Even without prior database experience, you can grow into a Cassandra expert based on the know-how of a current Silicon Valley developer.

When you find yourself constantly updating Swagger documentation, aligning API specs, and repeatedly communicating with the frontend, you start to wonder, "Why am I doing the same thing twice?" I also felt similar frustrations in my professional experience, which led me to use GraphQL extensively and develop an interest in data-centric design. This course goes beyond simple GraphQL syntax; it explores the real-world problems it was created to solve, along with practical considerations like Resolver structures, the N+1 problem, and Federation. I've structured this course to help you naturally learn how to reduce communication costs between developers and design more flexible API structures.

The biggest challenges when operating a batch system that processes millions of records are performance degradation and the difficulty of recovery when failures occur. Simple loops or poor design cannot solve these problems. In this course, based on experience handling large volumes of data reliably in real-world service environments, you will learn the architecture and operating principles of Spring Batch, along with design methods that achieve both performance and stability. Going beyond a simple explanation of its features, this course helps you develop fundamental problem-solving skills for data processing through practical batch architectures that can be applied directly in the workplace.

This course is for developers who want to find fundamental solutions in environments where response times slow as traffic increases, thread pools are quickly exhausted, and bottlenecks become more severe as the number of external API calls grows. Rather than simply scaling up servers, it helps you look at these problems from the perspective of changing the underlying architecture itself. Through this course, you will first understand the limitations of the traditional Spring MVC thread-per-request model. You will examine at the code level why throughput fails to increase even when CPU resources remain available, and how blocking I/O actually occupies system resources. Next, you will learn the concepts and standard specifications of Reactive Streams, as well as how to handle data streams using Reactor’s Mono and Flux. Rather than focusing solely on basic usage, you will gain hands-on experience with how data flows begin and are consumed, and how throughput is controlled. In particular, you will implement designs that use back pressure to prevent OOM conditions and overload. You will also learn how Spring WebFlux’s event-loop-based processing structure works and conduct comparative exercises to see how non-blocking I/O operates during actual request processing. By examining MVC and WebFlux side by side, you will identify their structural differences and establish criteria for deciding when to choose Reactive. This course goes beyond simply “learning asynchronous programming.” You will learn how to handle more requests with fewer threads, maintain stable flows in environments with heavy reliance on external APIs, and develop the design instincts needed to achieve high throughput in highly concurrent environments.

This course goes beyond teaching Docker as merely a set of tools and approaches it from the perspective of solving the concerns and problems you will actually encounter in the field. It explains, based on firsthand experience operating financial infrastructure in the industry, everything from unpredictable errors caused by differences between development and production environments, and configuration mistakes stemming from vague concepts of images and containers, to the anxiety of wondering, “How much do I need to know for this to be useful in practice?” Rather than simply listing commands, the course focuses on real-world examples to explain why this structure is necessary, how to design images for easier maintenance, and how to manage containers to operate services reliably. In particular, it places greater emphasis on hands-on practice than theory, with the goal of helping students understand Docker visually and master it through practice—not merely reaching the level of “knowing how to use Docker,” but understanding it from an operational perspective. This course provides practical standards and direction for those encountering Docker for the first time, those who have used it but have not yet organized their understanding of the concepts, and developers who want to take their infrastructure skills to the next level.

It was created to solve the problem of “not knowing where or how to start with Kubernetes.” Based on hands-on experience gained from operating financial infrastructure, it is structured to help learners understand Kubernetes by following the actual process of building and operating it, rather than focusing on theory alone. Through a learning progression covering its relationship with Docker, YAML, Pods, Deployments, and Services, learners will internalize Kubernetes not as a “concept,” but as a “usable technology.”