In the previous parts, we connected Spring AI with cloud-based AI models.
But there is one important question: What if you don't want to send your data to an external AI provider?
What if you want to: Run an LLM on your own machine Develop AI applications without API costs Work without an internet connection Keep sensitive company data private Experiment with different open-source models Build AI features locally before moving them to production This is where Ollama becomes very useful.
In this article, we will learn how to run a local LLM using Ollama and connect it with Spring AI.
We will build a simple real-world AI Customer Support Assistant using Java, Spring Boot, Spring AI, and Ollama.
What We Are Building Our application will look like this: The important part is that the LLM is running locally.
There is no need to send every prompt to OpenAI, Anthropic, or another cloud provider.
1.
What Is Ollama?
Ollama makes it easy to run open-source LLMs locally.
Instead of calling a remote API like: we can run: Ollama can run models such as: Llama Qwen Gemma Mistral DeepSeek and many other compatible models The exact models available change over time, so always check the Ollama model library before choosing one.
2.
Why Run an LLM Locally?
Imagine you are building an internal HR application.
Employees may send questions such as: or: You may not want internal company information leaving your infrastructure.
A local LLM can help: This can provide a useful privacy boundary.
However, remember: Running an LLM locally does not automatically make your application secure.
You still need proper authentication, authorization, logging, data protection, network security, and prompt/data controls.
3.
Install Ollama First, install Ollama on your operating system.
After installation, verify it: If the command works, Ollama is installed.
Now we need an LLM.
For example: Then run it: You can now talk to the model directly from your terminal.
For example: The model will generate a response locally.
4.
How Ollama Works At a high level, the architecture is: Ollama exposes an API that applications can communicate with.
Spring AI can communicate with this API for us.
That means we don't have to manually build HTTP requests to the Ollama API.
5.
Create a Spring Boot Project Let's create a Spring Boot application.
You can use Spring Initializr or your preferred IDE.
Basic project: We need: Spring Web Spring AI Ollama
6.
Maven Dependency Add the Spring AI Ollama starter to your .
You should use the Spring AI version compatible with your Spring Boot version.
For production projects, avoid randomly mixing Spring Boot and Spring AI versions.
7.
Configure Ollama Now configure the Ollama model.
The important part is: This is the default Ollama API endpoint.
Your architecture now becomes:
8.
Create the ChatClient Spring AI provides , which gives us a clean API for interacting with chat models.
Create a configuration class: That's it.
Spring AI will use the configured Ollama chat model.
9.
Create Our First AI Endpoint Now let's create a simple controller.
Start your Spring Boot application.
Then call: The request flows like this: And the response comes back to the client.
10.
The Real-World Example Let's make our application more useful.
Imagine we are building an AI Customer Support Assistant.
A customer sends: Instead of simply passing the question to the model, we can provide a system instruction.
Now the model has some context about its role.
11.
System Prompt vs User Prompt Spring AI allows us to separate instructions from user input.
For example: Think about it like this: This separation becomes extremely useful when building production AI applications.
12.
Create a Service Layer Putting everything inside the controller is not a good architecture.
Instead: Create: Then the controller becomes: This structure is much easier to extend later.
13.
Building a Better Support Assistant Let's make the prompt more useful.
Now a question like: could produce something like: This is already a useful AI feature.
14.
Add Conversation Memory A basic AI call is stateless.
For example: The model doesn't automatically know the previous conversation unless we provide that context.
In a real application, we need conversation memory.
The architecture becomes: Depending on your application, the conversation history can be stored in databases such as PostgreSQL or Redis.
For example: Then the relevant history can be included when making the next model call.
This is an important step from a simple AI demo toward a production AI application.
15.
Local LLM + RAG This is where local models become especially interesting.
Suppose your company has: We can build a RAG system: Now the LLM doesn't need to know your company information beforehand.
We retrieve the relevant information and provide it as context.
16.
Example: Internal HR Assistant Imagine an employee asks: The system can search the company's HR documents.
Suppose the vector database retrieves: Spring AI can then construct a prompt: The local LLM generates: The complete architecture becomes: This is much closer to a real enterprise AI architecture.
17.
Why This Architecture Is Powerful Imagine an organization has sensitive documents.
With a cloud-only architecture: With a local architecture: This can be attractive for: Internal knowledge assistants Developer tools Private document analysis Customer support prototypes Offline applications Sensitive enterprise workloads But again, local inference is not a complete security strategy by itself.
18.
Streaming AI Responses For chat applications, waiting for the entire response can feel slow.
A better experience is: Spring AI supports streaming responses through .
For example: The client can receive pieces of the response as they are generated.
This is useful for: AI chat applications Coding assistants Customer support AI search Writing assistants
19.
Model Selection Matters Not every local model is good for every task.
For example: Your choice depends on: RAM GPU/VRAM CPU Model size Context length Response speed Task complexity Accuracy requirements For a simple local experiment, start with a relatively small model.
Then benchmark larger models if your hardware allows it.
20.
Ollama vs Cloud LLMs A simple comparison: Feature Local Ollama Cloud LLM Internet required Usually no Yes API cost No per-token cloud fee Usually usage-based Data leaves machine Can stay local Sent to provider Hardware required Yes Mostly no Scaling Your responsibility Provider handles infrastructure Model choice Open/local models Provider-specific models Setup More infrastructure Usually easier Latency Depends on hardware Depends on network/provider There is no universal winner.
A practical architecture may even use both.
21.
Hybrid AI Architecture For example: You could use: and: The routing decision can be implemented inside your application.
This gives you more flexibility.
22.
Production Considerations Running Ollama on your laptop is great for development.
Production is different.
You need to think about: Infrastructure Observability Track: Request latency Model latency Token usage where available Error rate Timeout rate Model failures Concurrent requests Security Protect: Ollama endpoints Internal APIs User prompts Retrieved documents Conversation history Do not expose an unauthenticated Ollama service directly to the public internet.
23.
Handling Errors AI services can fail.
Your application should not assume every model call succeeds.
For example: In a production application, use a proper exception hierarchy and global exception handling rather than exposing raw exceptions.
You may also add:
24.
A Better Production Architecture A more realistic architecture could look like this: This architecture allows you to evolve from a simple local experiment into a production-grade AI platform.
25.
Complete Minimal Example Here is the complete service: Controller: Configuration: Architecture: That's enough to build your first local AI