Skip to main content
AIDevOps
  • Learn
  • Learning Paths
  • Practice
  • Open Source
  • Books
  • Engineering

    AI DevOpsThe full map of building and operating AI servicesLLMOpsLLM deployment Β· evaluation Β· observabilityHands-on ProjectsBuild AI Agent projects

    Knowledge

    DocsTechnical documentationBlogEngineering articlesPloggerDevelopment log feed

    Validate

    Certification3-level skills certification Β· coming soon
AI Models
LlamaTutorial|MistralTutorial|GemmaTutorial|DeepSeekTutorial|QwenTutorial
🌱 Spring Cloud
Spring Intro & RoadmapSpring Cloud GatewaySpring BootJava|Spring AISpring SecuritySpring BatchSpring JPA
πŸ€– AI Applied Development
Applied AI Intro & RoadmapHugging FaceLangChainLlamaIndexLLMOps|LangGraphMCPMulti-AgentAgent Evaluation
🧠 AI Core
AI Intro & RoadmapML FundamentalsLLM Fundamentals|Python AIC++|PyTorchTensorFlowJAX
🧠 AI Agent Development
Finance AI AgentLLM API ServerStock Investing AgentAIOps AI AgentEducation AI AgentCoding AI Agent
🐳 DevOps
DevOps Intro & RoadmapLinuxDockerCI/CD|Kubernetes BasicsK8s AdvancedPrometheusGrafana
🧱 Infrastructure
Infrastructure Intro & RoadmapNginxRedis
☁️ Cloud
Cloud Intro & RoadmapAWSGCPAzureNCPCloudflare
🎨 Frontend
Frontend Intro & RoadmapJavaScriptTypeScript|ReactNext.js|VueNuxt|Electron
πŸ“± Mobile
Mobile Intro & RoadmapKotlinAndroidFlutter
βš™οΈ Backend
Backend Intro & RoadmapPython BasicsFastAPIDjangoFlask|CGoGinNode.js
πŸ’Ύ Database
DB Intro & RoadmapCore SQLOracleMySQLPostgreSQL|MongoDBVector DB
πŸ§ͺ Testing
k6JMeternGrinder
AIDevOps

Engineering AI. From Code to Production.
An engineering learning platform for building and operating AI and AI Agents

Learn

  • All Guides
  • Learning Paths
  • AI Tutorial
  • Practice
  • Books

Resources

  • AI DevOps
  • LLMOps
  • Hands-on Projects
  • Docs
  • Blog
  • Plogger
  • Open Source
  • Certification (coming soon)

Start Here

  • AI Core Roadmap
  • AI Applied Development Roadmap
  • Spring Cloud Roadmap
  • DevOps Roadmap
  • Infrastructure Roadmap

Β 

  • Cloud Roadmap
  • Frontend Roadmap
  • Mobile Roadmap
  • Backend Roadmap
  • Database Roadmap
Β© 2026 AIDevOps. All rights reserved.
Terms of ServicePrivacy PolicySitemaptestforge.kr
  1. Home
  2. Learn
  3. Spring Cloud
  4. Spring AI
Guide to AI applications with Spring Boot (Spring AI 2.0)

AI Spring AI Complete Guide

Visitors

How to add LLM features to a service the Spring Boot way with Spring AI 2.0: ChatClient, prompt templates, structured output, chat memory, Tool Calling, RAG (vector stores), and MCP servers.

  • Intermediate
  • Updated Oct 1, 2026
  • ~12 min read
  • 11 sections
  • 22 code examples

Learning Paths that include this guide

This guide is one step in the paths below. Study it together with the steps before and after.

  • Spring Cloud Engineer β†’
LLM chat APIsStructured output (POJO mapping)Tool CallingRAG document searchMCP servers

Related frameworks & environments

πŸƒSpring Bootβ†’πŸ”Spring Securityβ†’GWSpring Cloud Gatewayβ†’πŸ”ŒMCPβ†’πŸ§ Vector DBβ†’LCLangChainβ†’

Contents

0 / 14
  1. How to use this guide
  2. Architecture diagrams
  3. What is Spring AI?
  4. Project setup
  5. ChatClient basics
  6. Prompt templates
  7. Structured output
  8. Chat memory
  9. Tool Calling
  10. RAG & vector stores
  11. MCP servers
  12. Observability & operations
  13. Testing strategy
  14. Frequently asked questions
Contents 14 sections
  1. How to use this guide
  2. Architecture diagrams
  3. What is Spring AI?
  4. Project setup
  5. ChatClient basics
  6. Prompt templates
  7. Structured output
  8. Chat memory
  9. Tool Calling
  10. RAG & vector stores
  11. MCP servers
  12. Observability & operations
  13. Testing strategy
  14. Frequently asked questions

How to use this guide

How to read it

Understanding Spring AI as a real-world workflow

How to add LLM features to a service the Spring Boot way with Spring AI 2.0: ChatClient, prompt templates, structured output, chat memory, Tool Calling, RAG (vector stores), and MCP servers. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.

Key perspective

Backend / systems development

Before syntax, pin down the boundaries a request crosses: validation, processing, storage, and response.

LLM chat APIsStructured output (POJO mapping)Tool CallingRAG document searchMCP servers

Architecture diagrams

To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Spring AI.

Learning flow

Rendering diagram…

Architecture view

Rendering diagram…

What is Spring AI?

When you first open Spring AI, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.

Spring AI is a project that wraps AI models such as OpenAI, Anthropic, Google, Amazon Bedrock, and Ollama, and vector databases such as PGVector, Redis, Chroma, and Qdrant, in Spring Boot auto-configuration and portable APIs. Even when you switch model providers, the business code depends only on abstractions such as ChatClient and VectorStore, so it hides the LLM vendor the way JDBC hid the database vendor. This guide is based on Spring AI 2.0.x, the official GA release (built on Spring Boot 4.0/4.1).
ComponentRoleTypical API
ChatClientA fluent API that handles prompt assembly, the call, and response conversionprompt().user().call().content()
AdvisorA chain that sits between the request and response (memory, RAG, logging)MessageChatMemoryAdvisor, QuestionAnswerAdvisor
Structured OutputMaps the LLM response to a Java record/POJO.entity(MyRecord.class)
Tool CallingThe model requests execution of an application method@Tool, .tools(...)
VectorStoreStores embeddings and runs similarity search (the retrieval step of RAG)add(), similaritySearch()
MCPBoot starters for Model Context Protocol servers/clientsspring-ai-starter-mcp-server-webmvc

Tip

Copying Spring AI 1.x examples over as-is produces compile errors in many places. 2.0 is built on Spring Boot 4 / Spring Framework 7, the conversation id for chat memory became mandatory, and the `.options` level was dropped from the model option properties. Check the version first when following blog examples.

Project setup

Here you look at Project setup alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

On start.spring.io, choose Spring Web and the model starter you want (for example OpenAI). Always manage Spring AI module versions in one place with spring-ai-bom β€” writing a version on each starter separately lets the core and model module versions drift, causing a NoSuchMethodError at runtime. Starter names follow the rule spring-ai-starter-model-{provider}, spring-ai-starter-vector-store-{store}.
build.gradle.ktsKOTLIN
plugins {
    id("java")
    id("org.springframework.boot") version "4.1.0"
    id("io.spring.dependency-management") version "1.1.7"
}

java { toolchain { languageVersion.set(JavaLanguageVersion.of(21)) } }

extra["springAiVersion"] = "2.0.1"

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("org.springframework.boot:spring-boot-starter-actuator")

    // Model provider starters (picking just one is fine)
    implementation("org.springframework.ai:spring-ai-starter-model-openai")
    // implementation("org.springframework.ai:spring-ai-starter-model-anthropic")
    // implementation("org.springframework.ai:spring-ai-starter-model-ollama")

    // When using RAG
    implementation("org.springframework.ai:spring-ai-starter-vector-store-pgvector")
    implementation("org.springframework.ai:spring-ai-vector-store-advisor")

    testImplementation("org.springframework.boot:spring-boot-starter-test")
}

dependencyManagement {
    imports {
        mavenBom("org.springframework.ai:spring-ai-bom:${property("springAiVersion")}")
    }
}
application.ymlYAML
spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        model: gpt-4o-mini
        temperature: 0.2
      embedding:
        model: text-embedding-3-small
    # When using Anthropic
    # anthropic:
    #   api-key: ${ANTHROPIC_API_KEY}
    #   chat:
    #     model: claude-sonnet-4-20250514
    #     max-tokens: 1024

Tip

  • Do not write API keys directly in application.yml; inject them through environment variables or a Secret Manager. If a key reaches Git even once, it must be rotated immediately.
  • A common setup is to start local development at no cost with the Ollama starter and enable the commercial model starter configuration only in the production profile. If the code depends only on ChatClient, switching profiles is all it takes to change the model.

ChatClient basics

Here you look at ChatClient basics alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

A ChatClient is built from the ChatClient.Builder that the starter registers automatically. Settings repeated on every request, such as the system prompt or common Advisors, go into the builder once through defaultSystem and defaultAdvisors, and only the user input that differs per request is filled in at prompt(). Use call() for a synchronous response and stream() for token-by-token streaming.
ChatConfig.javaJAVA
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
class ChatConfig {
    @Bean
    ChatClient chatClient(ChatClient.Builder builder) {
        return builder
            .defaultSystem("""
                You are an assistant that guides people through the internal developer docs.
                Do not guess at things you do not know; say that you do not know.
                Write answers concisely in English.
                """)
            .build();
    }
}
ChatController.javaJAVA
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.*;
import reactor.core.publisher.Flux;

@RestController
@RequestMapping("/api/chat")
class ChatController {
    private final ChatClient chatClient;

    ChatController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    // Synchronous call β€” returns the whole response at once
    @PostMapping
    String chat(@RequestBody String message) {
        return chatClient.prompt()
            .user(message)
            .call()
            .content();
    }

    // Streaming β€” sent over SSE as tokens are generated
    @PostMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    Flux<String> stream(@RequestBody String message) {
        return chatClient.prompt()
            .user(message)
            .stream()
            .content();
    }
}

Tip

  • Perceived speed is governed by the time to first token (TTFT). For screens that generate long answers, just switching to stream() cuts the "it is slow" feedback substantially.
  • stream() returns a Reactor Flux, so it works in Spring MVC apps too, but with many concurrent streaming connections a WebFlux-based setup is better in terms of thread resources.

Prompt templates

Here you look at Prompt templates alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Splicing user input into a prompt by string concatenation blurs the boundary between template and data and leaves you open to prompt injection. Separate the template from the variables with user(u -> u.text(...).param(...)), and move long prompts into classpath resource files so they come under code review and version control.
src/main/resources/prompts/review.stTEXT
Review the following {language} code.
- Point out the parts that may contain bugs first.
- Attach a severity (HIGH/MEDIUM/LOW) to each item.

Code:
{code}
CodeReviewService.javaJAVA
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

@Service
class CodeReviewService {
    private final ChatClient chatClient;
    private final Resource reviewPrompt;

    CodeReviewService(ChatClient chatClient,
                      @Value("classpath:prompts/review.st") Resource reviewPrompt) {
        this.chatClient = chatClient;
        this.reviewPrompt = reviewPrompt;
    }

    String review(String language, String code) {
        return chatClient.prompt()
            .user(u -> u.text(reviewPrompt)
                .param("language", language)
                .param("code", code))
            .call()
            .content();
    }
}

Tip

The template variable delimiter is {curly braces} by default. Writing text with many braces, such as code or JSON, directly in the template body gets it mistaken for variables, so always pass such content in through param.

Structured output (POJO mapping)

Here you look at Structured output (POJO mapping) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Instead of receiving the LLM response as a string and parsing it yourself, pass a Java record type to .entity() and Spring AI appends JSON schema instructions to the prompt and converts the response into an object. Where type erasure occurs, as with a generic list, use ParameterizedTypeReference.
TicketClassifier.javaJAVA
import java.util.List;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.stereotype.Service;

record TicketTriage(String category, String priority, String summary, List<String> tags) {}

@Service
class TicketClassifier {
    private final ChatClient chatClient;

    TicketClassifier(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    TicketTriage classify(String ticketBody) {
        return chatClient.prompt()
            .system("Classify customer inquiries. For priority, use only one of P1, P2, P3.")
            .user(ticketBody)
            .call()
            .entity(TicketTriage.class);
    }

    List<TicketTriage> classifyAll(String bulkText) {
        return chatClient.prompt()
            .user(bulkText)
            .call()
            .entity(new ParameterizedTypeReference<List<TicketTriage>>() {});
    }
}

Tip

  • There is no guarantee the model follows the schema 100% of the time. Validate the converted result once more with Bean Validation or similar, and prepare a path that retries on failure or sends the item to a human review queue.
  • Declaring a field with a fixed range of values, such as priority, as an enum makes the conversion step fail right away when a disallowed value comes in, catching the error early.

Chat memory

Here you look at Chat memory alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

An LLM is stateless, so to make it remember the earlier conversation the previous messages have to be sent along with every request. MessageChatMemoryAdvisor does this for you, and the store can be swapped through ChatMemoryRepository (in-memory by default). From Spring AI 2.0 on there is no default conversation id, so an IllegalArgumentException is thrown unless the ChatMemory.CONVERSATION_ID parameter is passed on each request.
MemoryChatConfig.javaJAVA
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor;
import org.springframework.ai.chat.memory.ChatMemory;
import org.springframework.ai.chat.memory.ChatMemoryRepository;
import org.springframework.ai.chat.memory.MessageWindowChatMemory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
class MemoryChatConfig {
    @Bean
    ChatMemory chatMemory(ChatMemoryRepository repository) {
        // Keep only the latest 20 messages β€” limits token cost and context length
        return MessageWindowChatMemory.builder()
            .chatMemoryRepository(repository)
            .maxMessages(20)
            .build();
    }

    @Bean
    ChatClient memoryChatClient(ChatClient.Builder builder, ChatMemory chatMemory) {
        return builder
            .defaultAdvisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
            .build();
    }
}
ConversationService.javaJAVA
String reply(String userId, String sessionId, String message) {
    // Build the id from the user + session so conversations of different users do not mix
    String conversationId = userId + ":" + sessionId;

    return memoryChatClient.prompt()
        .user(message)
        .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
        .call()
        .content();
}
application.yml (JDBC store)YAML
# implementation("org.springframework.ai:spring-ai-starter-model-chat-memory-repository-jdbc")
spring:
  ai:
    chat:
      memory:
        repository:
          jdbc:
            initialize-schema: always   # in production, never + Flyway migrations is recommended

Tip

  • Using a conversation id exactly as the client sent it makes it possible to read the chat history of someone else. Always combine it with the authenticated user id, or use only values issued by the server.
  • Projects that used the 1.x JDBC memory table need a schema migration, because 2.0 added a sequence_id column for message ordering.

Tool Calling

Here you look at Tool Calling alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

In Tool Calling, the model requests "run this function with these arguments", and the application runs the actual method and returns the result to the model. The model does not execute code itself; it only decides which tool to use, and the authority to execute remains with our server. The description of @Tool is the only basis the model has for choosing a tool, so write it concretely, as if explaining to a person.
OrderTools.javaJAVA
import org.springframework.ai.tool.annotation.Tool;
import org.springframework.ai.tool.annotation.ToolParam;
import org.springframework.stereotype.Component;

@Component
class OrderTools {
    private final OrderRepository orders;

    OrderTools(OrderRepository orders) {
        this.orders = orders;
    }

    @Tool(description = "Look up the order status and estimated delivery date by order number")
    OrderStatus getOrderStatus(@ToolParam(description = "Order number, e.g. ORD-2026-0001") String orderId) {
        return orders.findStatus(orderId)
            .orElseThrow(() -> new IllegalArgumentException("Order does not exist: " + orderId));
    }
}
SupportService.javaJAVA
String answer(String question) {
    return chatClient.prompt()
        .user(question)            // "When will ORD-2026-0001 arrive?"
        .tools(orderTools)         // explicitly attach the tools to use for this request only
        .call()
        .content();
}
application.ymlYAML
spring:
  ai:
    tools:
      limits:
        max-calls-per-tool-default: 5    # default 40 β€” lower it to suit the service
        max-total-tool-calls: 15         # default 150
        on-limit-exceeded: THROW

Tip

  • Always check permissions inside the tool. The model can produce any argument depending on user input, so the server code has to block the case where "an order number belonging to someone else" comes in.
  • For irreversible operations such as payment and deletion, a two-step structure is safe: the tool does not execute right away but only creates a "pending confirmation" state, and the user approves it in the UI.
  • From 2.0 on, tools must be attached to the request explicitly (automatic resolution by bean name is off by default), and limits on the number of tool calls apply by default. They are a safeguard against runaway cost from an endless tool call loop, so do not turn them off.

RAG & vector stores

Here you look at RAG & vector stores alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

RAG (Retrieval-Augmented Generation) is a pattern that finds document chunks semantically close to the question in a vector store and puts them in the prompt, so the model answers on the basis of internal documents it was never trained on. Design the ingestion phase (read documents β†’ split into chunks β†’ embed β†’ store) and the query phase (retrieve β†’ augment the prompt β†’ generate) separately. For simple cases use QuestionAnswerAdvisor; if you need query rewriting or multi-retrieval, use RetrievalAugmentationAdvisor from spring-ai-rag.
application.ymlYAML
spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/ragdb
    username: rag
    password: ${DB_PASSWORD}
  ai:
    vectorstore:
      pgvector:
        initialize-schema: true
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: 1536   # must match the embedding model dimensions
DocumentIngestor.javaJAVA
import java.util.List;
import org.springframework.ai.document.Document;
import org.springframework.ai.reader.TextReader;
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

@Service
class DocumentIngestor {
    private final VectorStore vectorStore;

    DocumentIngestor(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    void ingest(Resource file, String team) {
        List<Document> raw = new TextReader(file).get();
        List<Document> chunks = new TokenTextSplitter().apply(raw);
        // Attaching metadata lets you filter by team/permission at search time
        chunks.forEach(doc -> doc.getMetadata().put("team", team));
        vectorStore.add(chunks);   // compute the embeddings + store
    }
}
DocsQaService.javaJAVA
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.vectorstore.SearchRequest;
import org.springframework.ai.vectorstore.VectorStore;
import org.springframework.stereotype.Service;

@Service
class DocsQaService {
    private final ChatClient chatClient;
    private final VectorStore vectorStore;

    DocsQaService(ChatClient chatClient, VectorStore vectorStore) {
        this.chatClient = chatClient;
        this.vectorStore = vectorStore;
    }

    String ask(String question, String team) {
        var qaAdvisor = QuestionAnswerAdvisor.builder(vectorStore)
            .searchRequest(SearchRequest.builder()
                .topK(5)
                .similarityThreshold(0.7)
                .filterExpression("team == '" + team + "'")
                .build())
            .build();

        return chatClient.prompt()
            .advisors(qaAdvisor)
            .user(question)
            .call()
            .content();
    }
}
Tuning pointIf too smallIf too large
Chunk sizeContext is cut off, leaving too little grounding for the answerIrrelevant content mixes in, lowering accuracy and raising token cost
topKMisses the document with the answer (lower recall)More noise, prompt length balloons
similarityThresholdIncludes even unrelated documents0 search results β†’ the model answers with no grounding

Tip

  • Take the values that go into filterExpression (team and so on) from the authentication information, not from user input. The document permission filter is the heart of RAG security.
  • Changing the embedding model makes the dimensions and space differ from the existing vectors, rendering search meaningless. Plan a full reindex when you change models.
  • When using a provider with no embedding API, such as Anthropic, as the chat model, configure just the embeddings with a separate starter such as OpenAI or Ollama.

Building an MCP server

Here you look at Building an MCP server alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

MCP (Model Context Protocol) is a protocol that lets AI clients such as Claude Desktop and IDE agents call external tools in a standard way. With the Spring AI MCP server starter, you just put @McpTool on existing Spring service methods to expose internal systems to AI agents safely. Use the WebMVC/WebFlux starter (Streamable HTTP) for remote servers and the STDIO starter for local processes.
build.gradle.ktsKOTLIN
dependencies {
    implementation("org.springframework.ai:spring-ai-starter-mcp-server-webmvc")
}
application.ymlYAML
spring:
  ai:
    mcp:
      server:
        name: order-mcp-server
        version: 1.0.0
        type: SYNC
        protocol: STREAMABLE
OrderMcpTools.javaJAVA
@Component
class OrderMcpTools {
    private final OrderRepository orders;

    OrderMcpTools(OrderRepository orders) {
        this.orders = orders;
    }

    @McpTool(name = "get_order_status", description = "Look up the order status by order number")
    OrderStatus getOrderStatus(
            @McpToolParam(description = "Order number", required = true) String orderId) {
        return orders.findStatus(orderId).orElseThrow();
    }
}

Tip

  • Add authentication with Spring Security when exposing an MCP server on the network. An MCP endpoint is in effect "an internal API that AI can call", so it needs the same level of authentication and audit logging as an ordinary REST API.
  • MCP concepts and client-side configuration are covered in more detail in the MCP guide.

Observability & operations

Here you look at Observability & operations alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

Spring AI has Micrometer Observation built in, so just turning on Actuator records ChatClient calls, model calls, vector searches, and tool executions as metrics and traces. LLM features vary widely in response time and cost, so token usage and latency need to be watched more closely than for an ordinary API.
application.ymlYAML
management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  tracing:
    sampling:
      probability: 0.1

spring:
  ai:
    chat:
      observations:
        log-prompt: false       # prompts may contain personal data, so false in production
        log-completion: false
MetricWhy watch itExample alert
gen_ai_client_token_usage_totalInput/output tokens = costDaily tokens more than double the usual
gen_ai_chat_client_operation_secondsModel response latencyp95 above 10 seconds for 5 minutes
Model call error rateDetects a rise in 429 (rate limit) and 5xxError rate above 5%
db_vector_client_operation_secondsVector search latencyp95 above 500ms

Tip

  • To prepare for 429/5xx from the model provider, set the retry count and backoff with spring.ai.retry.*, and put a circuit breaker on the Gateway side so a failure does not spread to the whole service.
  • Recording token usage per user gives you data you can use directly for plan limits, abuse detection, and cost accounting per feature.

Testing strategy

Here you look at Testing strategy alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.

LLM responses differ every time, so tests based on "does the string match exactly?" break quickly. Split it up: test business logic deterministically by replacing the ChatModel with a mock, and judge real model quality in a small number of evaluation tests that decide "is it relevant?" and "is it grounded in the evidence?". Spring AI provides RelevancyEvaluator and FactCheckingEvaluator for these judgments.
TicketClassifierTest.javaJAVA
@SpringBootTest
class TicketClassifierTest {
    @MockitoBean
    ChatModel chatModel;

    @Autowired
    TicketClassifier classifier;

    @Test
    void convertsModelResponseToRecord() {
        var json = """
            {"category":"BILLING","priority":"P2","summary":"Refund inquiry","tags":["refund"]}
            """;
        given(chatModel.call(any(Prompt.class)))
            .willReturn(new ChatResponse(List.of(new Generation(new AssistantMessage(json)))));

        TicketTriage result = classifier.classify("I was charged twice");

        assertThat(result.priority()).isEqualTo("P2");
        assertThat(result.tags()).contains("refund");
    }
}
DocsQaEvaluationIT.javaJAVA
@SpringBootTest
@Tag("llm-eval")   // excluded from the default CI run; run only nightly/manually
class DocsQaEvaluationIT {
    @Autowired ChatClient.Builder builder;
    @Autowired DocsQaService qa;

    @Test
    void answerIsRelevantToQuestion() {
        String question = "What is the deployment approval procedure?";
        String answer = qa.ask(question, "platform");

        var evaluator = new RelevancyEvaluator(builder);
        var response = evaluator.evaluate(new EvaluationRequest(question, List.of(), answer));

        assertThat(response.isPass()).isTrue();
    }
}

Tip

  • Tests that call a real model cost money and time, and their results fluctuate. Separate them with a tag and run them not on every PR but only in a nightly batch or when prompts change.
  • A rule that evaluation test results be attached to the PR whenever a prompt file (.st) is modified catches, at review time, the regression where "changing one line of the prompt broke other answers".

Spring AI: frequently asked questions

What does Spring AI provide?

It provides a common interface for calling LLMs from Java and Spring applications. You call different model providers the same way, and use embeddings and vector store integration, tool calling, and conversion of output into objects through Spring-style configuration.

Do I have to build AI features separately in Python for a Java project?

Calling model APIs and building RAG can be done well in Java. The advantage is that your existing authentication, transactions, and monitoring keep working. If you need to train models or use libraries that exist only in Python, split just that part into a separate service.

Can I receive an LLM response as a Java object?

Yes. Use the feature that tells the model the structure of your class and converts the response to that type. Models sometimes break the format, so handle conversion failures and add code that validates value ranges and required fields.

← Previous guideJavaNext guide β†’Spring Security