Ollama Python Tutorial: Run Local LLMs with Chat, Streaming, Structured Outputs, Tools and Embeddings (2026)

Llm And Generative Ai Nov 19, 2026

Watch on YouTube

Running a large language model on your own machine is no longer a research project. Ollama downloads open models, runs them behind a small local server, and the official ollama Python library talks to that server in a few lines of code. Your prompts never leave the computer, there is no per-token bill, no API key and no rate limit, and the same code works offline on a laptop, a desktop or a small server. In this tutorial we go beyond a one-line chat: we stream tokens, force the model to return JSON that matches a Pydantic model, let it call real Python functions as tools, build a tiny semantic search with embeddings, and send concurrent requests with AsyncClient. Everything was run on Python 3.14.10 with ollama 0.6.3 (the Python library) and the Ollama 0.40.16 server, using the small qwen3:1.7b chat model and the embeddinggemma:300m embedding model. Every output block below is the real output of the script above it.

Why use the ollama library instead of raw HTTP or a framework

Ollama exposes a REST API on http://localhost:11434, so you could call it with requests. The official library is a thin, typed wrapper around that API built on httpx and pydantic: responses are Pydantic objects with attributes like response.message.content, streaming is a normal Python generator, errors become ollama.ResponseError, and Python functions can be passed directly as tools. Frameworks such as LangChain can sit on top of Ollama too, but for scripts, APIs and small agents the official client is the shortest path with the fewest moving parts. Recent Ollama releases added the features this guide relies on: JSON-schema structured outputs, tool calling, the batch embed endpoint and a think switch for reasoning models such as Qwen3.

Setup: install Ollama, pull two models, install the library

Install the Ollama app from ollama.com (Windows and macOS installers, or the install script on Linux), then pull a chat model and an embedding model. Small models are perfect for learning because they load in seconds and run without a big GPU; swap in a larger tag such as qwen3:8b or gemma4:12b later without changing any Python code.

ollama pull qwen3:1.7b            # 1.4 GB chat model with tool support
ollama pull embeddinggemma:300m   # 621 MB embedding model
python -m venv .venv               # or: uv venv
pip install ollama                 # installs ollama, httpx and pydantic
ollama --version
ollama list
ollama version is 0.40.16
NAME                   ID              SIZE      MODIFIED           
embeddinggemma:300m    85462619ee72    621 MB    54 seconds ago        
qwen3:1.7b             8f68893c685c    1.4 GB    About a minute ago

The desktop app starts the server automatically. On a headless Linux box run ollama serve in another terminal. If the server lives on another machine, set the OLLAMA_HOST environment variable or pass host= to Client.

Step 1: your first chat request

ollama.chat() takes a model name and a list of messages, exactly like the chat format used by hosted APIs. A system message sets the behaviour. We set think=False because Qwen3 is a reasoning model and would otherwise spend tokens on hidden reasoning before a short answer, and we fix temperature and seed so runs are repeatable.

import ollama

MODEL = "qwen3:1.7b"

for m in ollama.list().models:
    print(f"installed: {m.model:22} {m.size / 1e9:.2f} GB")

response = ollama.chat(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a concise Python tutor. Answer in at most two sentences."},
        {"role": "user", "content": "What is a Python list comprehension?"},
    ],
    think=False,                       # skip qwen3's hidden reasoning for short answers
    options={"temperature": 0, "seed": 42},
)
print("answer:", response.message.content.strip())
tps = response.eval_count / (response.eval_duration / 1e9)
print(f"tokens in/out: {response.prompt_eval_count}/{response.eval_count}, speed: {tps:.1f} tokens/s")

Output:

installed: embeddinggemma:300m    0.62 GB
installed: qwen3:1.7b             1.36 GB
answer: A Python list comprehension is a concise way to create lists using a simple expression. It uses the syntax [expression for item in iterable], allowing you to transform or filter elements efficiently.
tokens in/out: 42/37, speed: 24.2 tokens/s

The response object also carries token counts and timings in nanoseconds, so measuring speed takes one line. The very first call is slower because Ollama loads the model into memory; it then stays loaded for five minutes by default (change it with the keep_alive argument).

Step 2: stream tokens as they are generated

For chat interfaces you want text to appear immediately. Pass stream=True and the call returns a generator of partial responses:

import time
import ollama

start = time.perf_counter()
first = None
parts = []
stream = ollama.chat(
    model="qwen3:1.7b",
    messages=[{"role": "user", "content": "Give three short tips for writing clean Python functions, one line each."}],
    stream=True,
    think=False,
    options={"temperature": 0, "seed": 42},
)
for chunk in stream:
    if first is None:
        first = time.perf_counter() - start
    print(chunk.message.content, end="", flush=True)
    parts.append(chunk.message.content)
print(f"\nfirst token after {first:.2f}s, {len(parts)} chunks, total {time.perf_counter() - start:.2f}s")

Output:

1. **Use descriptive variable names** – make your code readable and maintainable.  
2. **Avoid unnecessary indentation** – use 4 spaces or tabs consistently.  
3. **Keep functions small and focused** – each function should do one thing.
first token after 0.05s, 53 chunks, total 2.26s

The first token arrived almost instantly because the model was already loaded. Each chunk has the same shape as a normal response, so the code that prints a full answer also prints a streamed one. In a FastAPI app you can forward these chunks with a StreamingResponse.

Step 3: structured outputs with Pydantic

Free text is hard to use in code. Passing a JSON schema in the format argument makes the Ollama server constrain generation so the reply is valid JSON for that schema. Pydantic writes the schema for us, and a Literal type turns into an enum, so the model can only answer low, medium or high:

from typing import Literal
from pydantic import BaseModel
import ollama

class Ticket(BaseModel):
    customer: str
    product: str
    priority: Literal["low", "medium", "high"]   # becomes an enum in the JSON schema
    refund_requested: bool

email = ("Hi, this is Sara Khan. My new X200 headphones stopped charging after two days. "
         "I need them for work tomorrow, please fix this urgently or give me my money back.")

response = ollama.chat(
    model="qwen3:1.7b",
    messages=[{"role": "user", "content": f"Extract a support ticket from this email. Use high priority when the customer needs it urgently.\n\n{email}"}],
    format=Ticket.model_json_schema(),  # the server constrains output to this JSON schema
    think=False,
    options={"temperature": 0, "seed": 42},
)
print("raw JSON:", response.message.content)
ticket = Ticket.model_validate_json(response.message.content)
print("validated:", ticket)
print("route to refunds team?", ticket.refund_requested and ticket.priority == "high")

Output:

raw JSON: {
  "customer": "Sara Khan",
  "product": "X200 Headphones",
  "priority": "high",
  "refund_requested": true
}
validated: customer='Sara Khan' product='X200 Headphones' priority='high' refund_requested=True
route to refunds team? True

The model read an unstructured email and returned a typed object we can route on. Always validate with model_validate_json(): the schema guarantees the shape, but your business rules (here: urgent means high) still depend on a clear prompt. If you want automatic retries on validation errors, the Instructor guide for typed LLM outputs shows that pattern, and it works with local Ollama models too.

Step 4: tool calling with plain Python functions

Tool calling lets the model ask your code to run a function and then answer with the result. The ollama library can take Python functions directly in tools=[...]; it builds the tool schema from the type hints and the Google-style docstring. Your code stays in control: the model only returns the function name and arguments, and you decide whether to run it.

import ollama

def get_order_status(order_id: str) -> str:
    """Look up the shipping status of an order.

    Args:
        order_id: The order number, for example A-1001

    Returns:
        The current status of the order
    """
    orders = {"A-1001": "shipped, arriving Friday", "A-1002": "processing"}
    return orders.get(order_id, "not found")

def add(a: int, b: int) -> int:
    """Add two integers.

    Args:
        a: The first integer
        b: The second integer

    Returns:
        The sum
    """
    return a + b

TOOLS = {"get_order_status": get_order_status, "add": add}
messages = [{"role": "user", "content": "Where is my order A-1001?"}]

response = ollama.chat(model="qwen3:1.7b", messages=messages, tools=[get_order_status, add],
                       think=False, options={"temperature": 0, "seed": 42})
messages.append(response.message)
for call in response.message.tool_calls or []:
    result = TOOLS[call.function.name](**call.function.arguments)
    print(f"model called {call.function.name}({call.function.arguments}) -> {result!r}")
    messages.append({"role": "tool", "tool_name": call.function.name, "content": str(result)})

final = ollama.chat(model="qwen3:1.7b", messages=messages, think=False, options={"temperature": 0, "seed": 42})
print("final answer:", final.message.content.strip())

Output:

model called get_order_status({'order_id': 'A-1001'}) -> 'shipped, arriving Friday'
final answer: Your order A-1001 has been shipped and should arrive by Friday. Let me know if you need any further assistance!

The model picked get_order_status (not add) and extracted the order number on its own. We appended the assistant message and a tool message with the result, and the second request turned that into a friendly answer. This loop is the core of every agent; libraries like Pydantic AI typed agents automate it with validation and dependency injection when your project grows.

Step 5: embeddings and semantic search

Embeddings turn text into vectors where similar meanings sit close together. ollama.embed() accepts one string or a list, so a whole batch of documents needs one request. Here we rank four help-centre sentences against a question using cosine similarity, with no extra libraries:

import math
import ollama

docs = [
    "Reset your password from the account settings page.",
    "Refunds are issued to the original card within 5 business days.",
    "Our headphones come with a two year warranty.",
    "You can track a parcel with the order number in your email.",
]
query = "How long until I get my money back?"

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

doc_vecs = ollama.embed(model="embeddinggemma:300m", input=docs).embeddings   # one call, many texts
q_vec = ollama.embed(model="embeddinggemma:300m", input=query).embeddings[0]
print("vector size:", len(q_vec))
for score, doc in sorted(((cosine(q_vec, v), d) for v, d in zip(doc_vecs, docs)), reverse=True):
    print(f"{score:.3f}  {doc}")

Output:

vector size: 768
0.658  Refunds are issued to the original card within 5 business days.
0.336  Our headphones come with a two year warranty.
0.295  You can track a parcel with the order number in your email.
0.238  Reset your password from the account settings page.

The refund sentence scored clearly highest even though it never uses the words "money back". That is the retrieval half of RAG. Store the vectors in a database such as LanceDB or Chroma and pass the best matches to the chat model, as explained in our production RAG pipelines guide. Use the same embedding model for documents and queries, because vectors from different models cannot be compared.

Step 6: async requests and error handling

AsyncClient has the same methods as the module-level functions but returns coroutines, so it fits FastAPI endpoints and other asyncio code. Errors from the server, such as a model that is not installed, raise ollama.ResponseError with a status code:

import asyncio
import time
import ollama
from ollama import AsyncClient

client = AsyncClient(host="http://localhost:11434")

async def ask(word: str) -> str:
    r = await client.chat(model="qwen3:1.7b", think=False, options={"temperature": 0, "seed": 42},
                          messages=[{"role": "user", "content": f"Translate '{word}' to French. Reply with one word."}])
    return f"{word} -> {r.message.content.strip()}"

async def main():
    start = time.perf_counter()
    for line in await asyncio.gather(*(ask(w) for w in ["cat", "house", "water"])):
        print(line)
    print(f"3 requests in {time.perf_counter() - start:.2f}s")

asyncio.run(main())

try:
    ollama.chat(model="llama9:404b", messages=[{"role": "user", "content": "hi"}])
except ollama.ResponseError as e:
    print(f"ResponseError {e.status_code}: {e.error}")

Output:

cat -> chat
house -> maison
water -> eau
3 requests in 0.39s
ResponseError 404: model 'llama9:404b' not found

A 404 usually means a typo in the model tag or a model you have not pulled yet; you can call ollama.pull(name) in the except block. By default the server handles one request at a time per model and queues the rest (OLLAMA_NUM_PARALLEL=1), so the gain here is that your program stays responsive while the three short translations finish; raise OLLAMA_NUM_PARALLEL if you need truly parallel generation and have the memory for it. Note that a 1.7B model also makes small mistakes, such as returning "chat" for "cat": that word means cat in French, but bigger models give more reliable answers.

Quick reference

TaskCode
Chatollama.chat(model=m, messages=msgs)
Read the answerresponse.message.content
Streamfor chunk in ollama.chat(..., stream=True)
JSON that matches a modelformat=MyModel.model_json_schema()
Tool callingtools=[my_function], then read response.message.tool_calls
Embeddingsollama.embed(model=m, input=[...]).embeddings
Skip reasoning tokensthink=False
Remote serverClient(host="http://server:11434") or OLLAMA_HOST
Asyncawait AsyncClient().chat(...)

Common errors and fixes

  • ConnectionError: Failed to connect to Ollama: the server is not running. Start the app or run ollama serve, and check OLLAMA_HOST.
  • ResponseError 404: model not found: run ollama pull with the exact tag shown by ollama list.
  • Empty or strange answers from Qwen3 or other reasoning models: the model may still be thinking; set think=False or read response.message.thinking.
  • Tool is never called: use a model that supports tools, give the function a clear docstring with an Args: section, and make the user request specific.
  • Slow first response: that is model loading. Keep it warm with keep_alive or pick a smaller quantized tag.

When to use a local model and when not to

Local models are ideal for private documents, offline tools, prototypes, classification and extraction jobs, test suites that should not hit paid APIs, and RAG over company data. Small models like the 1.7B one used here are fast and cheap but less accurate on complex reasoning, so evaluate on your own data and move up to 8B or larger models when quality matters. Because the code is identical for every model tag, switching is a one-word change.

Summary

With Ollama and the official ollama Python library you can run open LLMs entirely on your own hardware. Use chat() for answers, stream=True for live output, format= with a Pydantic schema for reliable JSON, plain Python functions as tools, embed() for semantic search, and AsyncClient for concurrent requests. That covers the building blocks of chatbots, extractors, agents and RAG systems, without an API key.

Share:
Last updated: November 2026
Built-in Functions in Python 2026 str() in Python 2026: String Conversion + Modern Formatting & Best Practices memoryview with JAX in Python 2026: Zero-Copy NumPy → JAX Array Interop + Efficient ML Examples memoryview with TensorFlow in Python 2026: Zero-Copy NumPy → Tensor Interop + GPU Pinning & ML Examples memoryview with TensorFlow in Python 2026: Zero-Copy NumPy → Tensor Interop + ML Examples memoryview with NumPy & PyTorch in Python 2026: Zero-Copy Views, Efficient Slicing & ML Interop Examples memoryview with NumPy in Python 2026: Zero-Copy Views, Efficient Slicing & Real ML Examples memoryview() in Python 2026: Zero-Copy Magic for Large Binary Data + Real Examples import() in Python 2026: How It Works, Security Risks & Modern Alternatives (importlib) zip() in Python 2026: How to Use It, strict=True Behavior & Real-World Examples vars() in Python 2026: Accessing Object Namespace + Modern Introspection Patterns type() in Python 2026: Dynamic Type Inspection & Object Creation + Modern Patterns tuple() in Python 2026: Immutable Sequences + Modern Patterns & Best Practices super() in Python 2026: Method Resolution & Modern Inheritance Patterns sum() in Python 2026: Summing Iterables + Modern Numeric Patterns & Best Practices staticmethod() in Python 2026: Static Methods, Utility Functions & Modern Best Practices sorted() in Python 2026: Sorting Iterables + Modern Patterns & Best Practices slice() in Python 2026: Creating Slice Objects + Modern Patterns & Best Practices setattr() in Python 2026: Dynamic Attribute Setting + Modern Patterns & Safety set() in Python 2026: Mutable Sets Creation + Modern Patterns & Best Practices