Running a large language model on your own machine is no longer a research project. Ollama downloads open models, runs them behind a small local server, and the official ollama Python library talks to that server in a few lines of code. Your prompts never leave the computer, there is no per-token bill, no API key and no rate limit, and the same code works offline on a laptop, a desktop or a small server. In this tutorial we go beyond a one-line chat: we stream tokens, force the model to return JSON that matches a Pydantic model, let it call real Python functions as tools, build a tiny semantic search with embeddings, and send concurrent requests with AsyncClient. Everything was run on Python 3.14.10 with ollama 0.6.3 (the Python library) and the Ollama 0.40.16 server, using the small qwen3:1.7b chat model and the embeddinggemma:300m embedding model. Every output block below is the real output of the script above it.
Why use the ollama library instead of raw HTTP or a framework
Ollama exposes a REST API on http://localhost:11434, so you could call it with requests. The official library is a thin, typed wrapper around that API built on httpx and pydantic: responses are Pydantic objects with attributes like response.message.content, streaming is a normal Python generator, errors become ollama.ResponseError, and Python functions can be passed directly as tools. Frameworks such as LangChain can sit on top of Ollama too, but for scripts, APIs and small agents the official client is the shortest path with the fewest moving parts. Recent Ollama releases added the features this guide relies on: JSON-schema structured outputs, tool calling, the batch embed endpoint and a think switch for reasoning models such as Qwen3.
Setup: install Ollama, pull two models, install the library
Install the Ollama app from ollama.com (Windows and macOS installers, or the install script on Linux), then pull a chat model and an embedding model. Small models are perfect for learning because they load in seconds and run without a big GPU; swap in a larger tag such as qwen3:8b or gemma4:12b later without changing any Python code.
ollama pull qwen3:1.7b # 1.4 GB chat model with tool support
ollama pull embeddinggemma:300m # 621 MB embedding model
python -m venv .venv # or: uv venv
pip install ollama # installs ollama, httpx and pydantic
ollama --version
ollama list
ollama version is 0.40.16
NAME ID SIZE MODIFIED
embeddinggemma:300m 85462619ee72 621 MB 54 seconds ago
qwen3:1.7b 8f68893c685c 1.4 GB About a minute ago
The desktop app starts the server automatically. On a headless Linux box run ollama serve in another terminal. If the server lives on another machine, set the OLLAMA_HOST environment variable or pass host= to Client.
Step 1: your first chat request
ollama.chat() takes a model name and a list of messages, exactly like the chat format used by hosted APIs. A system message sets the behaviour. We set think=False because Qwen3 is a reasoning model and would otherwise spend tokens on hidden reasoning before a short answer, and we fix temperature and seed so runs are repeatable.
import ollama
MODEL = "qwen3:1.7b"
for m in ollama.list().models:
print(f"installed: {m.model:22} {m.size / 1e9:.2f} GB")
response = ollama.chat(
model=MODEL,
messages=[
{"role": "system", "content": "You are a concise Python tutor. Answer in at most two sentences."},
{"role": "user", "content": "What is a Python list comprehension?"},
],
think=False, # skip qwen3's hidden reasoning for short answers
options={"temperature": 0, "seed": 42},
)
print("answer:", response.message.content.strip())
tps = response.eval_count / (response.eval_duration / 1e9)
print(f"tokens in/out: {response.prompt_eval_count}/{response.eval_count}, speed: {tps:.1f} tokens/s")
Output:
installed: embeddinggemma:300m 0.62 GB
installed: qwen3:1.7b 1.36 GB
answer: A Python list comprehension is a concise way to create lists using a simple expression. It uses the syntax [expression for item in iterable], allowing you to transform or filter elements efficiently.
tokens in/out: 42/37, speed: 24.2 tokens/s
The response object also carries token counts and timings in nanoseconds, so measuring speed takes one line. The very first call is slower because Ollama loads the model into memory; it then stays loaded for five minutes by default (change it with the keep_alive argument).
Step 2: stream tokens as they are generated
For chat interfaces you want text to appear immediately. Pass stream=True and the call returns a generator of partial responses:
import time
import ollama
start = time.perf_counter()
first = None
parts = []
stream = ollama.chat(
model="qwen3:1.7b",
messages=[{"role": "user", "content": "Give three short tips for writing clean Python functions, one line each."}],
stream=True,
think=False,
options={"temperature": 0, "seed": 42},
)
for chunk in stream:
if first is None:
first = time.perf_counter() - start
print(chunk.message.content, end="", flush=True)
parts.append(chunk.message.content)
print(f"\nfirst token after {first:.2f}s, {len(parts)} chunks, total {time.perf_counter() - start:.2f}s")
Output:
1. **Use descriptive variable names** – make your code readable and maintainable.
2. **Avoid unnecessary indentation** – use 4 spaces or tabs consistently.
3. **Keep functions small and focused** – each function should do one thing.
first token after 0.05s, 53 chunks, total 2.26s
The first token arrived almost instantly because the model was already loaded. Each chunk has the same shape as a normal response, so the code that prints a full answer also prints a streamed one. In a FastAPI app you can forward these chunks with a StreamingResponse.
Step 3: structured outputs with Pydantic
Free text is hard to use in code. Passing a JSON schema in the format argument makes the Ollama server constrain generation so the reply is valid JSON for that schema. Pydantic writes the schema for us, and a Literal type turns into an enum, so the model can only answer low, medium or high:
from typing import Literal
from pydantic import BaseModel
import ollama
class Ticket(BaseModel):
customer: str
product: str
priority: Literal["low", "medium", "high"] # becomes an enum in the JSON schema
refund_requested: bool
email = ("Hi, this is Sara Khan. My new X200 headphones stopped charging after two days. "
"I need them for work tomorrow, please fix this urgently or give me my money back.")
response = ollama.chat(
model="qwen3:1.7b",
messages=[{"role": "user", "content": f"Extract a support ticket from this email. Use high priority when the customer needs it urgently.\n\n{email}"}],
format=Ticket.model_json_schema(), # the server constrains output to this JSON schema
think=False,
options={"temperature": 0, "seed": 42},
)
print("raw JSON:", response.message.content)
ticket = Ticket.model_validate_json(response.message.content)
print("validated:", ticket)
print("route to refunds team?", ticket.refund_requested and ticket.priority == "high")
Output:
raw JSON: {
"customer": "Sara Khan",
"product": "X200 Headphones",
"priority": "high",
"refund_requested": true
}
validated: customer='Sara Khan' product='X200 Headphones' priority='high' refund_requested=True
route to refunds team? True
The model read an unstructured email and returned a typed object we can route on. Always validate with model_validate_json(): the schema guarantees the shape, but your business rules (here: urgent means high) still depend on a clear prompt. If you want automatic retries on validation errors, the Instructor guide for typed LLM outputs shows that pattern, and it works with local Ollama models too.
Step 4: tool calling with plain Python functions
Tool calling lets the model ask your code to run a function and then answer with the result. The ollama library can take Python functions directly in tools=[...]; it builds the tool schema from the type hints and the Google-style docstring. Your code stays in control: the model only returns the function name and arguments, and you decide whether to run it.
import ollama
def get_order_status(order_id: str) -> str:
"""Look up the shipping status of an order.
Args:
order_id: The order number, for example A-1001
Returns:
The current status of the order
"""
orders = {"A-1001": "shipped, arriving Friday", "A-1002": "processing"}
return orders.get(order_id, "not found")
def add(a: int, b: int) -> int:
"""Add two integers.
Args:
a: The first integer
b: The second integer
Returns:
The sum
"""
return a + b
TOOLS = {"get_order_status": get_order_status, "add": add}
messages = [{"role": "user", "content": "Where is my order A-1001?"}]
response = ollama.chat(model="qwen3:1.7b", messages=messages, tools=[get_order_status, add],
think=False, options={"temperature": 0, "seed": 42})
messages.append(response.message)
for call in response.message.tool_calls or []:
result = TOOLS[call.function.name](**call.function.arguments)
print(f"model called {call.function.name}({call.function.arguments}) -> {result!r}")
messages.append({"role": "tool", "tool_name": call.function.name, "content": str(result)})
final = ollama.chat(model="qwen3:1.7b", messages=messages, think=False, options={"temperature": 0, "seed": 42})
print("final answer:", final.message.content.strip())
Output:
model called get_order_status({'order_id': 'A-1001'}) -> 'shipped, arriving Friday'
final answer: Your order A-1001 has been shipped and should arrive by Friday. Let me know if you need any further assistance!
The model picked get_order_status (not add) and extracted the order number on its own. We appended the assistant message and a tool message with the result, and the second request turned that into a friendly answer. This loop is the core of every agent; libraries like Pydantic AI typed agents automate it with validation and dependency injection when your project grows.
Step 5: embeddings and semantic search
Embeddings turn text into vectors where similar meanings sit close together. ollama.embed() accepts one string or a list, so a whole batch of documents needs one request. Here we rank four help-centre sentences against a question using cosine similarity, with no extra libraries:
import math
import ollama
docs = [
"Reset your password from the account settings page.",
"Refunds are issued to the original card within 5 business days.",
"Our headphones come with a two year warranty.",
"You can track a parcel with the order number in your email.",
]
query = "How long until I get my money back?"
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
doc_vecs = ollama.embed(model="embeddinggemma:300m", input=docs).embeddings # one call, many texts
q_vec = ollama.embed(model="embeddinggemma:300m", input=query).embeddings[0]
print("vector size:", len(q_vec))
for score, doc in sorted(((cosine(q_vec, v), d) for v, d in zip(doc_vecs, docs)), reverse=True):
print(f"{score:.3f} {doc}")
Output:
vector size: 768
0.658 Refunds are issued to the original card within 5 business days.
0.336 Our headphones come with a two year warranty.
0.295 You can track a parcel with the order number in your email.
0.238 Reset your password from the account settings page.
The refund sentence scored clearly highest even though it never uses the words "money back". That is the retrieval half of RAG. Store the vectors in a database such as LanceDB or Chroma and pass the best matches to the chat model, as explained in our production RAG pipelines guide. Use the same embedding model for documents and queries, because vectors from different models cannot be compared.
Step 6: async requests and error handling
AsyncClient has the same methods as the module-level functions but returns coroutines, so it fits FastAPI endpoints and other asyncio code. Errors from the server, such as a model that is not installed, raise ollama.ResponseError with a status code:
import asyncio
import time
import ollama
from ollama import AsyncClient
client = AsyncClient(host="http://localhost:11434")
async def ask(word: str) -> str:
r = await client.chat(model="qwen3:1.7b", think=False, options={"temperature": 0, "seed": 42},
messages=[{"role": "user", "content": f"Translate '{word}' to French. Reply with one word."}])
return f"{word} -> {r.message.content.strip()}"
async def main():
start = time.perf_counter()
for line in await asyncio.gather(*(ask(w) for w in ["cat", "house", "water"])):
print(line)
print(f"3 requests in {time.perf_counter() - start:.2f}s")
asyncio.run(main())
try:
ollama.chat(model="llama9:404b", messages=[{"role": "user", "content": "hi"}])
except ollama.ResponseError as e:
print(f"ResponseError {e.status_code}: {e.error}")
Output:
cat -> chat
house -> maison
water -> eau
3 requests in 0.39s
ResponseError 404: model 'llama9:404b' not found
A 404 usually means a typo in the model tag or a model you have not pulled yet; you can call ollama.pull(name) in the except block. By default the server handles one request at a time per model and queues the rest (OLLAMA_NUM_PARALLEL=1), so the gain here is that your program stays responsive while the three short translations finish; raise OLLAMA_NUM_PARALLEL if you need truly parallel generation and have the memory for it. Note that a 1.7B model also makes small mistakes, such as returning "chat" for "cat": that word means cat in French, but bigger models give more reliable answers.
Quick reference
| Task | Code |
|---|---|
| Chat | ollama.chat(model=m, messages=msgs) |
| Read the answer | response.message.content |
| Stream | for chunk in ollama.chat(..., stream=True) |
| JSON that matches a model | format=MyModel.model_json_schema() |
| Tool calling | tools=[my_function], then read response.message.tool_calls |
| Embeddings | ollama.embed(model=m, input=[...]).embeddings |
| Skip reasoning tokens | think=False |
| Remote server | Client(host="http://server:11434") or OLLAMA_HOST |
| Async | await AsyncClient().chat(...) |
Common errors and fixes
ConnectionError: Failed to connect to Ollama: the server is not running. Start the app or runollama serve, and checkOLLAMA_HOST.ResponseError 404: model not found: runollama pullwith the exact tag shown byollama list.- Empty or strange answers from Qwen3 or other reasoning models: the model may still be thinking; set
think=Falseor readresponse.message.thinking. - Tool is never called: use a model that supports tools, give the function a clear docstring with an
Args:section, and make the user request specific. - Slow first response: that is model loading. Keep it warm with
keep_aliveor pick a smaller quantized tag.
When to use a local model and when not to
Local models are ideal for private documents, offline tools, prototypes, classification and extraction jobs, test suites that should not hit paid APIs, and RAG over company data. Small models like the 1.7B one used here are fast and cheap but less accurate on complex reasoning, so evaluate on your own data and move up to 8B or larger models when quality matters. Because the code is identical for every model tag, switching is a one-word change.
Summary
With Ollama and the official ollama Python library you can run open LLMs entirely on your own hardware. Use chat() for answers, stream=True for live output, format= with a Pydantic schema for reliable JSON, plain Python functions as tools, embed() for semantic search, and AsyncClient for concurrent requests. That covers the building blocks of chatbots, extractors, agents and RAG systems, without an API key.