Guide to AI applications with Spring Boot (Spring AI 2.0)
AI Spring AI Complete Guide
How to add LLM features to a service the Spring Boot way with Spring AI 2.0: ChatClient, prompt templates, structured output, chat memory, Tool Calling, RAG (vector stores), and MCP servers.
Intermediate
Updated Oct 1, 2026
~12 min read
11 sections
22 code examples
Learning Paths that include this guide
This guide is one step in the paths below. Study it together with the steps before and after.
How to add LLM features to a service the Spring Boot way with Spring AI 2.0: ChatClient, prompt templates, structured output, chat memory, Tool Calling, RAG (vector stores), and MCP servers. Rather than listing concepts, this guide is organized so you can follow the content in the order you have to make decisions on a real project.
Key perspective
Backend / systems development
Before syntax, pin down the boundaries a request crosses: validation, processing, storage, and response.
To make what you read stick, it helps to capture the flow as a picture first. The two diagrams below are reference maps you can keep coming back to while learning Spring AI.
Learning flow
Rendering diagramβ¦
Architecture view
Rendering diagramβ¦
What is Spring AI?
When you first open Spring AI, the big picture matters more than individual commands. This section starts with the problems the concepts ahead were created to solve.
Spring AI is a project that wraps AI models such as OpenAI, Anthropic, Google, Amazon Bedrock, and Ollama, and vector databases such as PGVector, Redis, Chroma, and Qdrant, in Spring Boot auto-configuration and portable APIs. Even when you switch model providers, the business code depends only on abstractions such as ChatClient and VectorStore, so it hides the LLM vendor the way JDBC hid the database vendor. This guide is based on Spring AI 2.0.x, the official GA release (built on Spring Boot 4.0/4.1).
Component
Role
Typical API
ChatClient
A fluent API that handles prompt assembly, the call, and response conversion
prompt().user().call().content()
Advisor
A chain that sits between the request and response (memory, RAG, logging)
MessageChatMemoryAdvisor, QuestionAnswerAdvisor
Structured Output
Maps the LLM response to a Java record/POJO
.entity(MyRecord.class)
Tool Calling
The model requests execution of an application method
@Tool, .tools(...)
VectorStore
Stores embeddings and runs similarity search (the retrieval step of RAG)
add(), similaritySearch()
MCP
Boot starters for Model Context Protocol servers/clients
spring-ai-starter-mcp-server-webmvc
Project setup
Here you look at Project setup alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
On start.spring.io, choose Spring Web and the model starter you want (for example OpenAI). Always manage Spring AI module versions in one place with spring-ai-bom β writing a version on each starter separately lets the core and model module versions drift, causing a NoSuchMethodError at runtime. Starter names follow the rule spring-ai-starter-model-{provider}, spring-ai-starter-vector-store-{store}.
build.gradle.ktsKOTLIN
plugins { id("java") id("org.springframework.boot") version "4.1.0" id("io.spring.dependency-management") version "1.1.7"}java { toolchain { languageVersion.set(JavaLanguageVersion.of(21)) } }extra["springAiVersion"] = "2.0.1"dependencies { implementation("org.springframework.boot:spring-boot-starter-web") implementation("org.springframework.boot:spring-boot-starter-actuator") // Model provider starters (picking just one is fine) implementation("org.springframework.ai:spring-ai-starter-model-openai") // implementation("org.springframework.ai:spring-ai-starter-model-anthropic") // implementation("org.springframework.ai:spring-ai-starter-model-ollama") // When using RAG implementation("org.springframework.ai:spring-ai-starter-vector-store-pgvector") implementation("org.springframework.ai:spring-ai-vector-store-advisor") testImplementation("org.springframework.boot:spring-boot-starter-test")}dependencyManagement { imports { mavenBom("org.springframework.ai:spring-ai-bom:${property("springAiVersion")}") }}
Here you look at ChatClient basics alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
A ChatClient is built from the ChatClient.Builder that the starter registers automatically. Settings repeated on every request, such as the system prompt or common Advisors, go into the builder once through defaultSystem and defaultAdvisors, and only the user input that differs per request is filled in at prompt(). Use call() for a synchronous response and stream() for token-by-token streaming.
ChatConfig.javaJAVA
import org.springframework.ai.chat.client.ChatClient;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;@Configurationclass ChatConfig { @Bean ChatClient chatClient(ChatClient.Builder builder) { return builder .defaultSystem(""" You are an assistant that guides people through the internal developer docs. Do not guess at things you do not know; say that you do not know. Write answers concisely in English. """) .build(); }}
ChatController.javaJAVA
import org.springframework.ai.chat.client.ChatClient;import org.springframework.http.MediaType;import org.springframework.web.bind.annotation.*;import reactor.core.publisher.Flux;@RestController@RequestMapping("/api/chat")class ChatController { private final ChatClient chatClient; ChatController(ChatClient chatClient) { this.chatClient = chatClient; } // Synchronous call β returns the whole response at once @PostMapping String chat(@RequestBody String message) { return chatClient.prompt() .user(message) .call() .content(); } // Streaming β sent over SSE as tokens are generated @PostMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) Flux<String> stream(@RequestBody String message) { return chatClient.prompt() .user(message) .stream() .content(); }}
Prompt templates
Here you look at Prompt templates alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Splicing user input into a prompt by string concatenation blurs the boundary between template and data and leaves you open to prompt injection. Separate the template from the variables with user(u -> u.text(...).param(...)), and move long prompts into classpath resource files so they come under code review and version control.
src/main/resources/prompts/review.stTEXT
Review the following {language} code.
- Point out the parts that may contain bugs first.
- Attach a severity (HIGH/MEDIUM/LOW) to each item.
Code:
{code}
Here you look at Structured output (POJO mapping) alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Instead of receiving the LLM response as a string and parsing it yourself, pass a Java record type to .entity() and Spring AI appends JSON schema instructions to the prompt and converts the response into an object. Where type erasure occurs, as with a generic list, use ParameterizedTypeReference.
Here you look at Chat memory alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
An LLM is stateless, so to make it remember the earlier conversation the previous messages have to be sent along with every request. MessageChatMemoryAdvisor does this for you, and the store can be swapped through ChatMemoryRepository (in-memory by default). From Spring AI 2.0 on there is no default conversation id, so an IllegalArgumentException is thrown unless the ChatMemory.CONVERSATION_ID parameter is passed on each request.
String reply(String userId, String sessionId, String message) { // Build the id from the user + session so conversations of different users do not mix String conversationId = userId + ":" + sessionId; return memoryChatClient.prompt() .user(message) .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId)) .call() .content();}
application.yml (JDBC store)YAML
# implementation("org.springframework.ai:spring-ai-starter-model-chat-memory-repository-jdbc")spring: ai: chat: memory: repository: jdbc: initialize-schema: always # in production, never + Flyway migrations is recommended
Tool Calling
Here you look at Tool Calling alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
In Tool Calling, the model requests "run this function with these arguments", and the application runs the actual method and returns the result to the model. The model does not execute code itself; it only decides which tool to use, and the authority to execute remains with our server. The description of @Tool is the only basis the model has for choosing a tool, so write it concretely, as if explaining to a person.
OrderTools.javaJAVA
import org.springframework.ai.tool.annotation.Tool;import org.springframework.ai.tool.annotation.ToolParam;import org.springframework.stereotype.Component;@Componentclass OrderTools { private final OrderRepository orders; OrderTools(OrderRepository orders) { this.orders = orders; } @Tool(description = "Look up the order status and estimated delivery date by order number") OrderStatus getOrderStatus(@ToolParam(description = "Order number, e.g. ORD-2026-0001") String orderId) { return orders.findStatus(orderId) .orElseThrow(() -> new IllegalArgumentException("Order does not exist: " + orderId)); }}
SupportService.javaJAVA
String answer(String question) { return chatClient.prompt() .user(question) // "When will ORD-2026-0001 arrive?" .tools(orderTools) // explicitly attach the tools to use for this request only .call() .content();}
application.ymlYAML
spring: ai: tools: limits: max-calls-per-tool-default: 5 # default 40 β lower it to suit the service max-total-tool-calls: 15 # default 150 on-limit-exceeded: THROW
RAG & vector stores
Here you look at RAG & vector stores alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
RAG (Retrieval-Augmented Generation) is a pattern that finds document chunks semantically close to the question in a vector store and puts them in the prompt, so the model answers on the basis of internal documents it was never trained on. Design the ingestion phase (read documents β split into chunks β embed β store) and the query phase (retrieve β augment the prompt β generate) separately. For simple cases use QuestionAnswerAdvisor; if you need query rewriting or multi-retrieval, use RetrievalAugmentationAdvisor from spring-ai-rag.
application.ymlYAML
spring: datasource: url: jdbc:postgresql://localhost:5432/ragdb username: rag password: ${DB_PASSWORD} ai: vectorstore: pgvector: initialize-schema: true index-type: HNSW distance-type: COSINE_DISTANCE dimensions: 1536 # must match the embedding model dimensions
DocumentIngestor.javaJAVA
import java.util.List;import org.springframework.ai.document.Document;import org.springframework.ai.reader.TextReader;import org.springframework.ai.transformer.splitter.TokenTextSplitter;import org.springframework.ai.vectorstore.VectorStore;import org.springframework.core.io.Resource;import org.springframework.stereotype.Service;@Serviceclass DocumentIngestor { private final VectorStore vectorStore; DocumentIngestor(VectorStore vectorStore) { this.vectorStore = vectorStore; } void ingest(Resource file, String team) { List<Document> raw = new TextReader(file).get(); List<Document> chunks = new TokenTextSplitter().apply(raw); // Attaching metadata lets you filter by team/permission at search time chunks.forEach(doc -> doc.getMetadata().put("team", team)); vectorStore.add(chunks); // compute the embeddings + store }}
Context is cut off, leaving too little grounding for the answer
Irrelevant content mixes in, lowering accuracy and raising token cost
topK
Misses the document with the answer (lower recall)
More noise, prompt length balloons
similarityThreshold
Includes even unrelated documents
0 search results β the model answers with no grounding
Building an MCP server
Here you look at Building an MCP server alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
MCP (Model Context Protocol) is a protocol that lets AI clients such as Claude Desktop and IDE agents call external tools in a standard way. With the Spring AI MCP server starter, you just put @McpTool on existing Spring service methods to expose internal systems to AI agents safely. Use the WebMVC/WebFlux starter (Streamable HTTP) for remote servers and the STDIO starter for local processes.
@Componentclass OrderMcpTools { private final OrderRepository orders; OrderMcpTools(OrderRepository orders) { this.orders = orders; } @McpTool(name = "get_order_status", description = "Look up the order status by order number") OrderStatus getOrderStatus( @McpToolParam(description = "Order number", required = true) String orderId) { return orders.findStatus(orderId).orElseThrow(); }}
Observability & operations
Here you look at Observability & operations alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
Spring AI has Micrometer Observation built in, so just turning on Actuator records ChatClient calls, model calls, vector searches, and tool executions as metrics and traces. LLM features vary widely in response time and cost, so token usage and latency need to be watched more closely than for an ordinary API.
application.ymlYAML
management: endpoints: web: exposure: include: health,info,prometheus tracing: sampling: probability: 0.1spring: ai: chat: observations: log-prompt: false # prompts may contain personal data, so false in production log-completion: false
Metric
Why watch it
Example alert
gen_ai_client_token_usage_total
Input/output tokens = cost
Daily tokens more than double the usual
gen_ai_chat_client_operation_seconds
Model response latency
p95 above 10 seconds for 5 minutes
Model call error rate
Detects a rise in 429 (rate limit) and 5xx
Error rate above 5%
db_vector_client_operation_seconds
Vector search latency
p95 above 500ms
Testing strategy
Here you look at Testing strategy alongside real code. Rather than copying the example as-is, read it with an eye on the inputs, the outputs, and the parts most likely to change.
LLM responses differ every time, so tests based on "does the string match exactly?" break quickly. Split it up: test business logic deterministically by replacing the ChatModel with a mock, and judge real model quality in a small number of evaluation tests that decide "is it relevant?" and "is it grounded in the evidence?". Spring AI provides RelevancyEvaluator and FactCheckingEvaluator for these judgments.
@SpringBootTest@Tag("llm-eval") // excluded from the default CI run; run only nightly/manuallyclass DocsQaEvaluationIT { @Autowired ChatClient.Builder builder; @Autowired DocsQaService qa; @Test void answerIsRelevantToQuestion() { String question = "What is the deployment approval procedure?"; String answer = qa.ask(question, "platform"); var evaluator = new RelevancyEvaluator(builder); var response = evaluator.evaluate(new EvaluationRequest(question, List.of(), answer)); assertThat(response.isPass()).isTrue(); }}
Spring AI: frequently asked questions
What does Spring AI provide?
It provides a common interface for calling LLMs from Java and Spring applications. You call different model providers the same way, and use embeddings and vector store integration, tool calling, and conversion of output into objects through Spring-style configuration.
Do I have to build AI features separately in Python for a Java project?
Calling model APIs and building RAG can be done well in Java. The advantage is that your existing authentication, transactions, and monitoring keep working. If you need to train models or use libraries that exist only in Python, split just that part into a separate service.
Can I receive an LLM response as a Java object?
Yes. Use the feature that tells the model the structure of your class and converts the response to that type. Models sometimes break the format, so handle conversion failures and add code that validates value ranges and required fields.