Inference Engines¶

This directory contains documentation for the inference engines supported by the MicroDC Worker.

Available Engines¶

Engine	Status	Description
Ollama	Production	Local LLM inference via Ollama
OpenAI-Compatible	Production	Any OpenAI API-compatible server (Ubuntu snaps, vLLM, LocalAI, LiteLLM, etc.)
Transformers	Production	HuggingFace Transformers for local model inference
vLLM	Production	Native high-performance LLM server, launched and supervised by the worker

Multi-Engine Architecture¶

The worker supports running multiple engines simultaneously. See MULTI_ENGINE.md for details on:

Configuring multiple engines
On-demand engine loading
Job routing based on platform
Memory management across engines

Quick Start¶

# config/default.yaml
engine:
  available:
    - ollama
    - openai_compat:qwen-snap
    - transformers

  qwen-snap:
    base_url: http://192.168.2.50:8326/v3

Engine Selection for Jobs¶

Jobs specify which engine to use via the platform field:

{
  "model_id": "llama3.1:8b",
  "platform": "ollama",
  "input_data": "Hello, world!"
}

For named instances, include the instance name:

{
  "model_id": "qwen2.5:7b",
  "platform": "openai_compat:qwen-snap",
  "input_data": "Hello, world!"
}

If no platform is specified, the worker uses the first available engine.

Adding a New Engine¶

To add a new inference engine:

Create engine class inheriting from InferenceEngine (see src/engines/base.py)
Implement all required abstract methods
Add configuration section to config/default.yaml
Register engine in src/core/client.py:_create_engine()
Add documentation in docs/engines/
Add tests in tests/

See openai_compat.md or transformers.md for complete examples.