OpenAI-Compatible Engine

The OpenAI-compatible engine provides inference using any server that implements the OpenAI API standard. This includes Ubuntu inference snaps, vLLM in OpenAI mode, LocalAI, LiteLLM, and more.

Status

Production - Fully implemented and tested.

Features

• Chat completions (streaming and non-streaming)
• Embedding generation
• Vision/multimodal models (Qwen-VL, LLaVA, etc.)
• Automatic model type detection
• Multiple named instances (run several servers simultaneously)
• Works with any OpenAI-compatible endpoint

Supported Servers

Server	Description	Example base_url
Ubuntu Inference Snaps	Snap-packaged LLM servers	http://192.168.2.50:8326/v3
vLLM (OpenAI mode)	High-performance inference	http://localhost:8000/v1
LocalAI	Local OpenAI-compatible API	http://localhost:8080/v1
LiteLLM	Proxy for 100+ LLM providers	http://localhost:4000/v1
Ollama (OpenAI mode)	Ollama's OpenAI-compatible endpoint	http://localhost:11434/v1
OpenAI	OpenAI cloud API	https://api.openai.com/v1

Requirements

• An OpenAI-compatible server reachable from the worker
• The openai Python package (pip install openai>=1.0.0)

Configuration

Single Instance

engine:
  available:
    - openai_compat

  openai_compat:
    base_url: ${OPENAI_COMPAT_BASE_URL:-http://localhost:8326/v3}
    api_key: ${OPENAI_COMPAT_API_KEY:-}   # optional for local servers
    timeout: 600

Multiple Named Instances

You can run multiple OpenAI-compatible servers simultaneously using the openai_compat:<name> format. Each named instance gets its own config section under engine.<name>:

engine:
  available:
    - ollama
    - openai_compat:qwen-snap       # Ubuntu snap running Qwen-VL
    - openai_compat:llama-vllm      # vLLM serving Llama
    - openai_compat:claude-proxy    # LiteLLM proxy for Anthropic Claude

  qwen-snap:
    base_url: http://192.168.2.50:8326/v3

  llama-vllm:
    base_url: http://192.168.2.51:8000/v1

  claude-proxy:
    base_url: http://192.168.2.52:4000/v1
    api_key: ${ANTHROPIC_API_KEY:-}

Each instance:

• Loads independently and has its own connection
• Reports its own model list in heartbeats
• Routes jobs via its platform name (e.g., "platform": "openai_compat:qwen-snap")

Environment Variables

Variable	Default	Description
OPENAI_COMPAT_BASE_URL	http://localhost:8326/v3	Server URL (single instance)
OPENAI_COMPAT_API_KEY	(empty)	API key (optional for local servers)

For named instances, configure via YAML or use custom env vars in the config.

Supported Model Types

Model type is auto-detected from the model ID:

Pattern	Detected Type
*embed*, *embedding*	Embedding
*vl*, *vision*, *llava*, *multimodal*	Multimodal
*dall-e*, *sdxl*, *stable-diffusion*	Image
*whisper*, *tts*	Audio
Everything else	Chat

Usage

Ubuntu Inference Snaps

Ubuntu inference snaps provide pre-packaged LLM servers with OpenAI-compatible APIs. After installing a snap, the server runs on a dedicated port:

# Install a snap (example: Qwen-VL)
sudo snap install qwen-vl

# Check the server status to find the port
snap info qwen-vl
# API endpoint: http://localhost:8326/v3

Configure the worker:

engine:
  available:
    - openai_compat:qwen-snap

  qwen-snap:
    base_url: http://localhost:8326/v3

Job Examples

Text Generation:

{
  "model_id": "qwen2.5:7b",
  "platform": "openai_compat:qwen-snap",
  "job_type": "llm",
  "input_data": "Explain quantum computing in simple terms."
}

Chat Completion:

{
  "model_id": "qwen2.5:7b",
  "platform": "openai_compat:qwen-snap",
  "job_type": "llm",
  "llm_interaction_type": "chat",
  "input_data": {
    "system_prompt": "You are a helpful assistant.",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }
}

Vision/Multimodal (Qwen-VL, LLaVA, etc.):

{
  "model_id": "qwen-vl:7b",
  "platform": "openai_compat:qwen-snap",
  "job_type": "llm",
  "input_data": "Describe this image in detail.",
  "attached_files": [
    {"download_url": "/api/v1/files/abc123", "file_type": "image/png"}
  ]
}

Images are automatically base64-encoded and sent as OpenAI-format image content parts.

Embedding:

{
  "model_id": "text-embedding-model",
  "platform": "openai_compat:llama-vllm",
  "job_type": "embed",
  "input_data": {"texts": ["Hello world", "How are you?"]}
}

Streaming:

{
  "model_id": "qwen2.5:7b",
  "platform": "openai_compat:qwen-snap",
  "job_type": "llm",
  "stream": true,
  "input_data": "Write a short story about a robot."
}

Using with LiteLLM for Anthropic Claude

LiteLLM provides an OpenAI-compatible proxy for 100+ LLM providers including Anthropic Claude:

# Start LiteLLM proxy
litellm --model claude-sonnet-4-20250514 --port 4000

engine:
  available:
    - openai_compat:claude-proxy

  claude-proxy:
    base_url: http://localhost:4000/v1
    api_key: ${ANTHROPIC_API_KEY:-}

{
  "model_id": "claude-sonnet-4-20250514",
  "platform": "openai_compat:claude-proxy",
  "job_type": "llm",
  "input_data": "Explain the theory of relativity."
}

Model Aliases

Some providers use different model names than what you use internally. For example, Groq prefixes models with openai/, or appends context length suffixes. The model_aliases config maps your internal names to the provider's names:

engine:
  available:
    - openai_compat:groq

  groq:
    base_url: https://api.groq.com/openai/v1
    api_key: ${GROQ_API_KEY:-}
    model_aliases:
      llama-3.3-70b: llama-3.3-70b-versatile
      gpt-oss-20b: openai/gpt-oss-20b
      mixtral-8x7b: mixtral-8x7b-32768

With this config:

• Jobs use the internal name: "model_id": "gpt-oss-20b"
• The engine translates to openai/gpt-oss-20b when calling the Groq API
• Model listings translate back: Groq's openai/gpt-oss-20b appears as gpt-oss-20b in heartbeats

The mapping applies to all API calls (chat completions, embeddings, model info). Models without an alias pass through unchanged.

Example: Groq

groq:
  base_url: https://api.groq.com/openai/v1
  api_key: ${GROQ_API_KEY:-}
  model_aliases:
    llama-3.3-70b: llama-3.3-70b-versatile
    gemma2-9b: gemma2-9b-it

Example: Azure OpenAI

azure:
  base_url: https://my-resource.openai.azure.com/openai/deployments
  api_key: ${AZURE_OPENAI_KEY:-}
  model_aliases:
    gpt-4: my-gpt4-deployment
    gpt-4o: my-gpt4o-deployment

Parameter Mapping

Most parameters pass through directly to the OpenAI API:

Generic	OpenAI	Description
temperature	temperature	Sampling temperature
max_tokens	max_tokens	Maximum tokens to generate
top_p	top_p	Nucleus sampling
frequency_penalty	frequency_penalty	Frequency penalty
presence_penalty	presence_penalty	Presence penalty
seed	seed	Random seed
stop_sequences	stop	Stop sequences
top_k	extra_body.top_k	Top-k sampling (via extra_body for non-standard support)

API Endpoints Used

The engine uses these OpenAI API endpoints:

Method	Endpoint	Purpose
GET	/models	List available models, health check
POST	/chat/completions	Chat completions (streaming and non-streaming)
POST	/embeddings	Embedding generation

The base URL can use any API version prefix (e.g., /v1, /v3).

Model Management

Since models are managed server-side:

• pull_model() - Logs a warning (manage models on the server directly)
• delete_model() - Logs a warning (manage models on the server directly)
• load_model() - Verifies the model exists on the server
• unload_model() - Removes from local tracking only

Cost limiting (spending caps)

Because an openai_compat instance can point at a paid provider (Groq, Azure OpenAI, Bedrock via LiteLLM, etc.), each instance can enforce a dollar budget so a worker never spends past a real credit:

engine:
  config:
    openai_compat:
      groq:
        base_url: https://api.groq.com/openai/v1
        api_key: ${GROQ_API_KEY}
        cost_limit_usd: 50.0          # stop claiming once cumulative spend hits this
        pricing:                       # USD per 1M tokens
          llama-3.3-70b-versatile:
            input: 0.59
            output: 0.79

How it works:

• After each job the worker records the actual cost (from the provider's reported token usage; an estimate when usage is omitted, e.g. streaming) into a persistent ledger at ~/.microdc/spend.json — cumulative across restarts.
• Before claiming a new job the worker estimates its cost and releases it back to the pool if running it would push cumulative spend over cost_limit_usd (reusing the same admission path as the VRAM/RAM check).
• Models with no pricing entry are treated as free. Delete an engine's entry in spend.json to start a new credit period.

See engine.<name>.cost_limit_usd / pricing in config/default.yaml, and src/jobs/cost_ledger.py.

Implementation Files

• src/engines/openai_compat.py - OpenAICompatEngine class
• src/engines/base.py - InferenceEngine base class
• tests/test_openai_compat.py - Unit tests (44 tests)

Troubleshooting

Connection refused

Ensure the server is running and the base_url is correct. For Ubuntu snaps, check the status with snap info <snap-name>.

No API key needed

Most local servers don't require an API key. The engine uses a placeholder (no-key) automatically. Only set api_key when connecting to servers that require authentication.

Model not found

The server must already have the model loaded/available. Use the server's management interface to add models.

Timeout errors

Increase timeout in config for large models or long generations:

openai_compat:
  timeout: 1200  # 20 minutes

Wrong instance used

Ensure jobs specify the correct platform with the instance name:

{"platform": "openai_compat:qwen-snap"}

Not just:

{"platform": "openai_compat"}