Multi-Engine Architecture¶

The MicroDC Worker supports running multiple inference engines simultaneously, loading them on-demand based on job requirements.

Overview¶

Instead of a single static engine, the worker can be configured with multiple available engines. Each engine is loaded only when a job requests it, optimizing resource usage.

Configuration¶

Available Engines¶

Configure which engines this worker supports. Supported types: ollama, openai_compat (incl. named instances like openai_compat:groq), vllm (native, worker-launched; also supports named instances vllm:<name>), transformers, plus container (docker/Podman) and document-processing (docling/surya) engines.

engine:
  available:
    - ollama
    - transformers

Or via environment variable:

export MICRODC_ENGINES="[ollama, transformers]"

Engine-Specific Configuration¶

Each engine has its own configuration section:

engine:
  available:
    - ollama
    - transformers

  ollama:
    base_url: http://localhost:11434
    timeout: 600

  transformers:
    model_path: ./models
    device: auto
    max_memory_mb: 0
    auto_unload: true

Named Instances¶

The OpenAI-compatible engine supports named instances, allowing you to connect multiple servers of the same type simultaneously. Use the type:name format:

engine:
  available:
    - ollama
    - openai_compat:qwen-snap       # Ubuntu snap running Qwen
    - openai_compat:llama-vllm      # vLLM serving Llama
    - openai_compat:claude-proxy    # LiteLLM proxy for Anthropic Claude

  qwen-snap:
    base_url: http://192.168.2.50:8326/v3

  llama-vllm:
    base_url: http://192.168.2.51:8000/v1

  claude-proxy:
    base_url: http://192.168.2.52:4000/v1
    api_key: ${ANTHROPIC_API_KEY:-}

Each named instance:

Is loaded and initialized independently
Reports its own model list in heartbeats (platform: openai_compat:qwen-snap)
Routes jobs via "platform": "openai_compat:qwen-snap"
Has its own config section under engine.<name>

How It Works¶

1. Engine Discovery¶

At startup the worker reads engine.available and initializes every listed engine so their local models can be discovered and advertised to the server in heartbeats (_get_or_load_engine provides a lazy fallback if an engine is needed but not yet loaded).

2. Job Routing¶

When a job arrives, the worker checks the platform field:

{
  "model_id": "llama3.1:8b",
  "platform": "ollama",
  "job_type": "llm",
  "input_data": "Hello!"
}

3. Lazy fallback loading¶

If a job needs an engine that isn't initialized yet (and it's in engine.available):

Worker creates the engine instance
Engine initializes with its config
Engine is cached for future jobs
Job is executed

4. Default Platform¶

If no platform is specified, the worker uses the first available engine.

Job Examples¶

Ollama Job¶

{
  "model_id": "llama3.1:8b",
  "platform": "ollama",
  "job_type": "llm",
  "input_data": "Explain quantum computing."
}

OpenAI-Compatible Job (Named Instance)¶

{
  "model_id": "qwen2.5:7b",
  "platform": "openai_compat:qwen-snap",
  "job_type": "llm",
  "input_data": "Explain quantum computing."
}

Claude via LiteLLM Proxy¶

{
  "model_id": "claude-sonnet-4-20250514",
  "platform": "openai_compat:claude-proxy",
  "job_type": "llm",
  "input_data": "Write a haiku about coding."
}

Transformers Job¶

{
  "model_id": "meta-llama/Llama-2-7b-chat-hf",
  "platform": "transformers",
  "job_type": "llm",
  "input_data": "Write a haiku about coding."
}

Embedding Job (Auto-Platform)¶

{
  "model_id": "nomic-embed-text",
  "job_type": "embed",
  "input_data": {"texts": ["Hello world"]}
}

Without platform, uses first available engine that has the model.

Heartbeat Reporting¶

The worker reports all available engines and their loaded models in heartbeats:

{
  "engines": ["ollama", "openai_compat:qwen-snap", "transformers"],
  "models": [
    {"id": "llama3.1:8b", "platform": "ollama"},
    {"id": "qwen2.5:7b", "platform": "openai_compat:qwen-snap"},
    {"id": "nomic-embed-text", "platform": "transformers"}
  ]
}

Memory Management¶

Each engine manages its own memory independently:

Ollama: Managed by the Ollama server
OpenAI-Compatible: Managed by the remote server (no local GPU usage)
vLLM (native): Managed by the worker-launched vLLM process (a fixed model per process; KV-cache-bound concurrency)
Transformers: LRU eviction with VRAM tracking

When multiple engines are loaded, be aware of total GPU memory usage. OpenAI-compatible engines that connect to remote servers don't use local GPU memory.

VRAM/RAM-aware admission and model gating¶

Across all engines, a ResourceAdmissionController (src/jobs/vram_admission.py) prevents the worker from taking on work it can't fit in memory:

Advertising — at model discovery, models that can't fit this host's total VRAM (or system RAM on CPU-only hosts) with a safety margin are not registered or reported, so the scheduler never routes them here.
Admission — before a claimed inference job runs, the worker checks the model fits in currently free memory given already-loaded models (fit margin + headroom). On GPU hosts VRAM is aggregated across all GPUs; on CPU-only hosts it gates on system RAM. A no-fit job is released back to the pool.

Cost-capped openai_compat engines additionally gate on a dollar budget (see OpenAI-Compatible → Cost limiting). Tunables live under resources.* in config/default.yaml (vram_model_margin, vram_headroom_percent).

Adding Custom Engines¶

To add a new engine:

Create engine class inheriting from InferenceEngine
Implement all abstract methods (see src/engines/base.py)
Add configuration section to config/default.yaml
Register in src/core/client.py:_create_engine()
Add documentation in docs/engines/

Troubleshooting¶

Engine not loading¶

Check engine is listed in engine.available
Verify engine dependencies are installed
Check engine-specific config is valid

Wrong engine used¶

Explicitly set platform field in job
Check model exists in expected engine

Out of memory¶

Limit engines to what you need
Use quantization for Transformers models
Enable auto_unload for dynamic memory management