Ollama Engine¶

The Ollama engine provides inference using the Ollama local LLM server.

Status¶

Production - Fully implemented and tested.

Features¶

Text generation (chat and completion)
Embedding generation
Vision/multimodal models (LLaVA, etc.)
Streaming responses
Automatic model management (pull, list, delete)

Requirements¶

Ollama server running locally (default: http://localhost:11434)
Models pre-pulled or available for on-demand pulling

Configuration¶

engine:
  available:
    - ollama

  ollama:
    base_url: ${OLLAMA_BASE_URL:-http://localhost:11434}
    timeout: 600  # 10 minutes for large generations

Environment Variables¶

Variable	Default	Description
`OLLAMA_BASE_URL`	`http://localhost:11434`	Ollama server URL

Supported Model Types¶

Chat models: llama3, mistral, qwen, etc.
Embedding models: nomic-embed-text, mxbai-embed-large, etc.
Vision models: llava, bakllava, qwen2.5-vl, etc.

Usage¶

Starting Ollama¶

# Start Ollama server
ollama serve

# Pull models
ollama pull llama3.1:8b
ollama pull nomic-embed-text

Job Examples¶

Text Generation:

{
  "model_id": "llama3.1:8b",
  "platform": "ollama",
  "job_type": "llm",
  "input_data": "Explain quantum computing in simple terms."
}

Embedding:

{
  "model_id": "nomic-embed-text",
  "platform": "ollama",
  "job_type": "embed",
  "input_data": {"texts": ["Hello world", "How are you?"]}
}

Vision:

{
  "model_id": "llava:7b",
  "platform": "ollama",
  "job_type": "llm",
  "input_data": "Describe this image",
  "attached_files": [{"download_url": "...", "file_type": "image/png"}]
}

Parameter Mapping¶

Generic parameters are automatically mapped to Ollama-specific parameters:

Generic	Ollama	Description
`max_tokens`	`num_predict`	Maximum tokens to generate
`temperature`	`temperature`	Sampling temperature
`top_p`	`top_p`	Nucleus sampling
`top_k`	`top_k`	Top-k sampling
`repetition_penalty`	`repeat_penalty`	Repetition penalty
`presence_penalty`	`presence_penalty`	Presence penalty
`frequency_penalty`	`frequency_penalty`	Frequency penalty
`seed`	`seed`	Deterministic sampling seed
`stop_sequences`	`stop`	Stop sequences

Thinking / reasoning models¶

Some Ollama models (e.g. reasoning variants) emit a chain-of-thought separately from the answer. The worker handles these natively:

Capability detection — get_model_info reads the model's capabilities from Ollama's /api/show; ModelInfo.supports_thinking is true when "thinking" is present. The worker only sends Ollama's think parameter to models that advertise it (sending it to a non-thinking model errors).
Per-job control — the server includes a thinking flag in the job payload (default on). thinking: false disables reasoning so the entire max_tokens budget goes to the answer. This matters because thinking tokens count against num_predict — a small budget can otherwise be consumed by reasoning before any answer is produced.
Reasoning output — when a model thinks, the worker captures the trace and returns it as a separate reasoning_content field on the result (DeepSeek-R1 / o1 convention), keeping the answer text clean. Non-thinking responses omit the field.

Implementation Files¶

src/engines/ollama.py - OllamaEngine class
src/engines/base.py - InferenceEngine base class

Troubleshooting¶

Connection refused¶

Ensure Ollama server is running: ollama serve

Model not found¶

Pull the model first: ollama pull <model-name>

Timeout errors¶

Increase timeout in config for large models/generations.