Multi-LLM & Orchestrated Multi-Agent System Implemented with Spring AI

Multi-LLM & Orchestrated Multi-Agent System Implemented with Spring AI

(Enterprise AI System Design based on DAG · Tool · Task · YAML DSL)

The system learned in this course is designed to build a system that handles the following complex tasks and decision-making processes.

is the objective of this course. In other words, we aim to build a Multi-LLM-based Orchestration-centered Multi-Agent system.

This course is a practical-level program designed to architect and implement Multi-LLM architectures and Orchestrated Multi-Agent systems by strategically combining GPT, Gemini, and LLaMA based on Spring AI.

We will build the structure according to the 3-step expansion roadmap below. Each step assumes the execution, model, and collaboration methods of the previous step.

In the Appendix, we will proceed with React·REST API expansion and MCP Integration (integrating Tool·Agent runtime with Model Context Protocol) expansion.

What you will learn

1⃣ Building a Spring AI development environment and Multi-LLM environment in SpringBoot

Before designing the AI architecture, we will complete a professional-level Spring AI development environment.

The class is conducted based on hands-on projects rather than simply adding libraries.

✔ Teaching materials included within the practice project
✔ Check code and theory simultaneously
✔ Provides a ready-to-run structure

Learning Content

1. Explanation of the development environment and overall course structure

2. Creating the practice project and setting up the teaching materials

3. Issuing and integrating OpenAI API Key

4. Issuing and integrating Gemini API Key

5. Ollama environment setup and Llama 3.2 setup

6. Spring AI Environment Configuration and Understanding the Basic Structure

7. Redis integration settings

   
   

2⃣Chapter 1. Multi-LLM Architecture (Designing Multi-Model and AI Architecture)

Design a strategic Multi-LLM architecture, rather than relying on a single model.

As the number of models increases, operational, cost, and security issues grow. Multi-LLM Architecture serves as the foundation for creating long-term replaceable, scalable, and auditable AI services, and becomes a core foundational technology for building subsequent Orchestrated Multi-Agent Systems.

Learning Content

1. Multi LLM

We will implement a Multi-LLM based service architecture that utilizes three LLMs—GPT, Gemini, and LLaMA—simultaneously and selects the appropriate model based on request types and strategies.

2. Multi LLM Stream

Streaming processing based on Reactive Streams (Project Reactor)

3. Fallback (Fault Tolerance Architecture)

In the enterprise, availability is as important as accuracy. By grouping primary and secondary models, failover/timeout switching, and retry policies, we ensure that a single model failure does not stop the entire service. (Applying Resilience4j CircuitBreaker)

4. Router (Model Strategy Classification)

After understanding the request, this is the decision-making layer that routes it to the appropriate model based on criteria such as request type, cost, accuracy, complexity, and latency. It is not a simple conditional branch, but a layer where service policies are manifested.

5. Security (Sensitive Information Protection Architecture)

After analyzing the input and determining the sensitivity of the information, it decides whether to send it to an internal (on-premise, etc.) model or an external API model. This is security-focused routing tailored to data sovereignty and regulatory requirements.

3⃣Chapter 2. Agentic Workflow Patterns (5 Agent Workflow Patterns)

It covers patterns that wrap LLM calls into orchestration with state, branching, concurrency, and feedback loops.

Based on the workflow types presented in Spring AI's Effective Agents and the Building Effective Agents post by Anthropic Engineering, which emphasizes simple abstraction and explicit composability, this is explained by dividing it into five patterns.

Learning Content

1. Chain Workflow

• A structure that connects tasks sequentially, passing the output of the previous step as the input for the next step.

• Reason for use: Ensures step-by-step thinking and procedural execution

2. Parallelization Workflow

• A structure that executes the same input across multiple LLMs in parallel and merges the results.

• Reason for use: Simultaneous utilization of expertise across models

3. Routing Workflow

• A structure that branches to the optimal LLM based on input conditions

• Reason for use: Selecting the optimal model based on the difficulty and type of the request

4. Orchestrator-Workers

• A structure where a central Orchestrator executes multiple LLMs by role and then combines the final results.

• Reason for use: Implementing a role-based separation of responsibilities structure

5. Evaluator-Optimizer

• An iterative structure that evaluates and improves generation results

• Reason for use: Stabilizing quality through iterative improvement

4⃣Chapter 3. Orchestrated Multi-Agent Patterns (Pipeline Implementation)

Implementing Multi-Agent structures into actual service pipelines

By combining five different patterns, we will materialize the Orchestrated Multi-Agent Pattern.
It covers the end-to-end flow, including step-by-step state transitions, exception isolation, and retries. The goal is not just simple response generation, but designing a system that considers scalability (additional steps), quality stability (evaluation and improvement), and reliability (consensus and cross-validation) required in various real-world operational environments.

Learning Content

1. Strategic Analysis Pattern

• Implement an iterative improvement-based analysis service to maximize strategic quality.

  
  

2. Precision Execution Pattern

• Implement an execution-oriented service that transforms ideas into actual deliverables (code/documents).

  
  

3. Voting Analysis Pattern

• Implement a consensus-based analysis service to eliminate model bias and improve reliability.

  
  

4. Auto Pattern Routing

• Analyzes the question and automatically delegates it to the appropriate pipeline among Strategic, Precision, or Voting.

  
  

  
  

5⃣ Chapter 4. Multi-Agent Architecture (Main Agent & SubAgent Separation Strategy)

A structure consisting of one Main (Main agent, Orchestrator agent) + multiple role-specific SubAgents

This covers a Multi-Agent structure that divides the control layer handled by the Main (Main agent, Orchestrator agent) and the execution layer (Sub agent) dedicated to specific roles handled by the Sub (Sub agent, Tool agent, etc.) .

6⃣ Chapter 5. Tool-Orchestrated Multi-Agent (Tool-based Execution Layer Separation)

Execution Layer Separation Architecture Using Tools

If RAG or database integration is required, you can bundle “Agent + Tool + SubAgent + External Data” into a single layer by calling external APIs, DBs, or Vector Stores within that Tool and passing the results to the SubAgent.

7⃣ Chapter 6. Task-Orchestrated Multi-Agent (TaskTool Agent Runtime)

TaskTool-based Agent Runtime Design

“Agent Runtime(TaskTool) + Markdown Sub-Agent + TodoWrite” Implementation

8⃣ Chapter 7. DAG-Orchestrated Multi-Agent (DAG-based AI Workflow Design)

Enterprise-grade DAG-based AI Workflow Design

Build a DAG (Directed Acyclic Graph) engine to change the system into a form that reads the graph and schedules executable nodes.

9⃣ Chapter 8. Declarative Agent Workflow with YAML (YAML-based DAG)
Separation of YAML-based DAG Definition and Execution Engine

By replacing the workflow with a declarative YAML-based workflow DSL definition on top of the DAG engine, complex Agent Workflows can be operated more safely and flexibly. Furthermore, using the DSL on a single DAG engine provides flexibility in changing workflow structures and facilitates easy runtime reconfiguration.

🔟 Chapter 9. Validated Agent Workflow DSL (DAG based on DSL validation)

Reliable YAML Workflow DSL with Schema and Dependency Validation

Before putting the DSL statement into the DAG engine's scheduling, add a validation layer in the loading pipeline that filters out declaration errors such as typos, incorrect content, cycles, or missing required fields.

🅰️Appendix A. React Front-End & REST API Server Integration

Convert the Thymeleaf practice into a React and REST API server.

🅱️ Appendix B. MCP Integration (Integrating Tool/Agent Runtime with MCP)
MCP Client and how it connects to the MCP Server's agent/orchestrator.

Notes before taking the course