Skip to content

Worker Architecture

This page describes how the MicroDC worker client is actually built. It reflects the shipped, claim-based implementation. For per-engine details see Engines; for the wire protocol see API Reference.

Overview

The worker is a long-running async service that:

  1. authenticates to the MicroDC server (saved credentials, else bootstrap registration — see Authentication),
  2. discovers local inference models and advertises the ones that fit this host,
  3. polls for available jobs, claims them, runs them on the right engine, and reports results,
  4. heartbeats system/GPU metrics and honors server commands (e.g. cancellation).

It is engine-agnostic: Ollama, OpenAI-compatible endpoints, a worker-managed vLLM server, local Transformers, container jobs (Podman/Docker), and document processing (Docling/Surya) all sit behind a common interface.

Source layout

src/
├── core/
│   ├── client.py          # WorkerClient — the main loop (claim/execute/heartbeat)
│   ├── cli.py             # Typer CLI (start, stop, status, register, ...)
│   ├── config.py          # WorkerConfig (YAML + env overrides)
│   ├── config_wizard.py   # hardware-aware `worker.yaml` generator
│   ├── auto_updater.py    # optional in-place updater (disabled by default)
│   └── exceptions.py
├── api/
│   ├── server_client.py   # all worker→server HTTP calls (v1 claim API)
│   └── models.py          # Job / JobResult / heartbeat pydantic models
├── auth/credentials.py    # local credential storage (~/.microdc, 0600/0700)
├── engines/               # one module per backend (see Engines docs)
│   ├── base.py            # InferenceEngine ABC (+ ModelInfo)
│   ├── ollama.py  openai_compat.py  vllm_engine.py  transformers_engine.py
│   ├── docker_engine.py  container_stream_engine.py
│   └── docling_engine.py  surya_engine.py
├── jobs/
│   ├── executor.py        # routing, per-engine concurrency, result handling
│   ├── vram_admission.py  # ResourceAdmissionController (VRAM/RAM fit checks)
│   ├── cost_ledger.py     # persistent spend tracking for cost-capped engines
│   ├── queue.py  monitor.py
├── models/                # registry, loader, capability mapping
├── processors/            # document_processor.py, surya_processor.py
├── downloads/manager.py   # model download manager
└── utils/                 # system (GPU/NVML), encryption, health, logging, docker

Job lifecycle (claim-based / "V2")

The worker never has work pushed to it; it pulls and claims:

  1. Discover — GET /api/v1/workers/jobs/available?limit=N (server_client.get_available_jobs).
  2. Pre-claim gate — for each candidate, the admission controller checks the model fits this host's VRAM/RAM before claiming, so the worker never claims work it can't run.
  3. Claim — POST /api/v1/workers/jobs/{id}/claim. Another worker may have taken it (already_claimed) or we may be at capacity (worker_busy).
  4. Admit — a post-claim VRAM check (safety net) and, for cost-capped engines, a budget check; a no-fit/over-budget job is released back.
  5. Queue & execute — the job is queued and run by JobExecutor, which routes it to the engine for its platform/model and enforces the per-job timeout. Batch and streaming paths are supported.
  6. Report — success → POST .../{id}/complete with the output (and reasoning_content for thinking models); failure (including a 0-token generation or timeout) → POST .../{id}/fail. Empty output from a text-generation job is reported FAILED, never COMPLETED.

Concurrency is bounded per engine (e.g. Ollama = 1) and by a global cap; the executor only claims up to remaining capacity.

Encrypted jobs

Workers approved for encryption (admin-granted capability and "encryption" reported in heartbeat capabilities) can claim encrypted jobs (is_encrypted: true). The claim response carries the AES-256-GCM payload key and IV alongside the ciphertext; utils/encryption.py::decrypt_payload decrypts in memory and the plaintext runs through the normal engine path — the engines are encryption-unaware. The result goes back under hybrid encryption (encrypt_result): a fresh one-time AES-256-GCM key encrypts the result, and that key is wrapped with the customer's RSA public key (RSA-OAEP/SHA-256) and embedded in the returned blob, so only the customer's private key can open it — the platform cannot.

The protection is layered: TLS (enforced at the platform edge in production) covers everything in transit; payloads exist on the platform only as AES-256-GCM ciphertext, with the key held in platform custody and released to the approved worker at claim time; results are sealed to the customer's RSA key, so only the customer can read them — not even the platform. Workers must never log or persist key material, decrypted payloads, or plaintext results — logs record sizes and metadata only. Wire formats, blob layout, and the full security model: WORKER_SERVER_API.md § Encrypted Job Processing.

Engine layer

All engines implement InferenceEngine (engines/base.py): initialize, generate / generate_stream, generate_embeddings, list_models, get_model_info, estimate_model_memory, load_model/unload_model, health_check. generate carries an optional think flag and an out_meta dict so reasoning models can return a reasoning_content trace out-of-band.

Engines listed in engine.available are initialized at startup (so their models can be advertised in heartbeats); _get_or_load_engine is a lazy fallback. See Multi-Engine for routing and named instances (e.g. openai_compat:groq, vllm:local).

Admission control

ResourceAdmissionController (jobs/vram_admission.py) gates inference jobs on memory, not just concurrency:

  • Advertise — at discovery, fits_total_capacity drops models that can't fit this host's total VRAM (or RAM on CPU hosts) with a safety margin, so the scheduler never targets impossible work.
  • Admit — at claim time, can_admit checks the model fits in currently free memory given already-loaded models (two-condition rule: fit margin + headroom). On GPU hosts this aggregates VRAM across all GPUs; on CPU-only hosts it gates on system RAM.

Cost-capped openai_compat engines additionally gate on a dollar budget via cost_ledger.py (persistent across restarts).

Background loops

WorkerClient.start() launches concurrent tasks:

  • job polling — discover/claim/queue (above)
  • job processing — drains the queue, runs jobs as async tasks
  • heartbeat — periodic system + GPU metrics (utils/system.py via NVML)
  • health check — engine/connectivity health
  • model refresh — re-discovers local models
  • server-availability monitor — triggers reconnect if the server is down too long
  • auto-update check — only if auto-update is explicitly enabled (off by default)

CLI

microdc-worker (Typer) exposes: start, stop, status, register, deregister, health, logs, metrics. See src/core/cli.py.

Configuration

Config is layered YAML + environment overrides (core/config.py); generate a host-tuned worker.yaml with the wizard (scripts/generate_config.py). The full reference lives in config/default.yaml. Notable defaults: engines are opt-in via engine.available; Docling/Surya processing and auto-update are disabled by default.