MicroDC.ai Worker Client - Development TODO

Document the encrypted-job flow exactly as implemented (2026-07-06) ✅

• [x] Goal: after the E2E review, make the docs state precisely what the code does, framed as the layered security model it is (TLS at the edge + encrypted payloads with platform key custody + results sealed to the customer key) rather than as a list of caveats.
• [x] WORKER_SERVER_API.md § Encrypted Job Processing: added "How it works, end to end" (key-lifecycle table: payload AES key is client-generated, held in platform key custody, delivered to the approved worker over TLS; result AES key is worker-generated fresh per result; RSA private key never leaves the client), the 6-step flow submit→claim→decrypt→hybrid-encrypt→submit→client-decrypt, a "Result blob format" spec (base64(<4-byte BE len><RSA-OAEP-wrapped AES key><AES-GCM ciphertext+tag>) + separate result_iv), why hybrid is required (RSA-OAEP/SHA-256 at 4096 bits caps at 446 bytes), and a "Security model" section with per-layer trust boundaries for implementers. Replaced the old one-line security note.
• [x] WORKER.md: new "Encrypted jobs" architecture section (capability gates, in-memory decrypt, encryption-unaware engines, hybrid result path, layered protection summary, no-logging obligation) linking to the API doc.
• [x] Framing (user direction): docs must not read as a vulnerability list — the platform is marketed on this feature. Describe each layer positively and precisely; the platform's role in payload-key custody is stated as a trust boundary, not a weakness.

Remove plaintext/key-material leaks from worker logs (2026-07-06) ✅

• [x] Context: E2E-encryption review of the worker found debug-era logging that undermined the "decrypted in memory" guarantee — plaintext-equivalent job content and credentials were being written to persistent log files (utils/logging.py attaches a FileHandler).
• [x] Removed (4 leak points): (1) client.py claim path dumped the complete raw job JSON at INFO — for encrypted jobs that's the AES encryption_key + ciphertext together, i.e. the plaintext; replaced with a one-line summary (job id, type, model, encrypted flag). (2) executor.py logged the first 500 chars of the (decrypted) input prompt; now logs input size only. (3) executor.py logged a 200-char plaintext output preview on completion; removed (output length was already logged). (4) server_client.py logged the login request body at DEBUG, which contained the worker secret_key; now logs URL + node_id only.
• [x] Gate: black/isort/ruff clean; bandit clean; mypy unchanged (10 pre-existing Optional union-attr errors in untouched engine/cost code); full suite 449 passing.
• Still open from the same review (needs platform/client coordination, not fixed here): the payload AES key arrives in plaintext from the server (job_data["encryption_key"]) so payload encryption is not E2E; customer_public_key is server-supplied and unauthenticated (key-substitution risk); no RSA key-size enforcement in encrypt_result; encrypted result submission falls back to plaintext if encryption_metadata is lost (should fail closed).

Documentation audit: archive abandoned docs, bring the rest current (2026-06-29) ✅

• [x] Goal: review all docs against current code, move abandoned docs to an archive folder (for eventual removal), and fix staleness everywhere else. Audited ~20 docs via three parallel code-cross-referenced passes.
• [x] Archived to docs/archive/ (with an archive/README.md explaining why): WORKER.md (pre-implementation design doc — wrong component tree, obsolete pre-claim flow, fictional CLI/roadmap), DOCUMENT_PROCESSING_INTEGRATION.md (cross-repo plan; missing Surya; Docling shown default-on), mdcworkers_updates.txt (personal deploy cheat-sheet, never in the published nav).
• [x] New concise, accurate docs/WORKER.md architecture page (claim-based loop, source layout, engine layer, VRAM/RAM admission, background loops, CLI). Removed the Document Processing entry from mkdocs.yml nav; fixed dangling links to the archived docs in README.md and both CONTRIBUTING.md copies.
• [x] Engine docs: engines/README.md vLLM Planned→Production; ollama.md added thinking/reasoning section + completed the param-mapping table; openai_compat.md added cost-cap (cost_limit_usd/pricing) section; MULTI_ENGINE.md fixed the "no engines loaded at startup" premise (engines init at startup; lazy fallback), added vLLM + a VRAM-admission/model-gating section; vllm.md reconciled the concurrency default (example/table 5→32 to match config/default.yaml).
• [x] Setup/auth/versioning: UBUNTU_SETUP.md rewritten for the uv install, --with-vllm/--with-podman, updated dir tree, and the auto-update-now-OFF-by-default reversal (env template + "Updating" section); WINDOWS_SETUP.md + setup/README.md fixed the auto-update default and dev-setup (uv + config wizard); AUTHENTICATION.md corrected the credential perms (file 0600 in a 0700 dir); VERSIONING.md stopped hard-coding 0.1.0 (points at src/version.py; history de-stub'd).
• [x] API/dev: WORKER_SERVER_API.md added output.reasoning_content, corrected the job-heartbeat/cancellation + release-404 notes, and the heartbeat cadence (60→30s default); CODE_QUALITY.md and both CONTRIBUTING.md fixed line-length 100→88 (and "Black's default" is 88, not 100).
• [x] Code drift fixed: setup.py/pyproject.toml hard-coded version = "0.1.0" while src/version.py is 0.1.190. Synced both, and extended tools/bump_version.py to keep all three in sync on every bump (so they never drift again).
• [x] Verification: 0 broken relative links across all docs; new bump-sync verified by a dry bump+revert. Untracked build artifacts (site/, coverage.json, models/) left alone (not in git).
• Why this matters: the docs had drifted badly behind a fast-moving codebase (vLLM, VRAM gating, thinking models, uv, auto-update default flip). Archiving the genuinely-dead docs keeps the published set trustworthy, and the version-sync stops a recurring packaging drift.

Consume the per-job thinking flag; return reasoning as reasoning_content (2026-06-26) ✅

• [x] Symptom: after the VRAM fixes let gemma4:e4b run, jobs failed with LLM produced no output (0 tokens). Root cause: gemma4:e4b is a thinking model. Ollama defaults thinking-capable models to think on, and thinking tokens count against num_predict (our max_tokens). With max_tokens=256, the ~250-token reasoning block consumed the whole budget before any answer reached the response field, so the worker collected 0 answer tokens. The 0-token→FAILED guard (correctly) reported it FAILED.
• [x] Contract (coordinated with server): the job payload now carries a thinking bool (default true; the server also 400s a think-on job whose max_tokens < THINKING_MIN_OUTPUT_TOKENS). Reasoning is returned as reasoning_content (DeepSeek-R1 / o1 convention); the server's extract_reasoning_from_result reads reasoning_content/reasoning/thinking.
• [x] Worker consumption (5 touch-points): (1) OllamaEngine.get_model_info now captures the model's capabilities from /api/show; ModelInfo.supports_thinking reads the thinking cap. (2) Job.thinking added and threaded from payload.thinking in client.py. (3) InferenceEngine.generate/_generate_impl gained think and out_meta params (other engines accept-and-ignore). (4) Ollama sends think only to thinking-capable models (None→on, False→off; sending it to a non-thinking model errors), captures chunk["thinking"] into out_meta["reasoning_content"], and keeps the answer string clean. (5) executor passes think=job.thinking + out_meta=result.metadata; submit_result_v2 emits output.reasoning_content when present.
• [x] Effect: thinking=False jobs now spend the whole budget on the answer (fixes the 0-token failure); thinking=True jobs return the answer plus a separate reasoning_content trace. Non-thinking models are untouched.
• [x] Tests/gate: new tests/test_ollama_thinking.py (3 cases: default-on + reasoning capture, think=False disables, non-thinking never receives think); updated mock engines for the new signature. Full suite 449 passing; ruff/black/bandit clean; mypy unchanged (4 pre-existing _current_job_engine Optional errors, no regression).

Don't claim jobs we can't run; gate advertised models by VRAM; auto-update off by default (2026-06-26) ✅

• [x] Symptom (from mdcw05, a 16GB-GPU host): the server kept routing gpt-oss:20b jobs (needs ~26GB, ~31.5GB with margin) to a worker that physically can't load them. The worker claimed each job, then released it on the VRAM admission check. The release endpoint POST /api/v1/workers/jobs/{job_id}/release returned 404, so the server still believed the worker held the job — every subsequent claim came back worker_busy "at capacity (1/1 jobs)" with worker_status: idle (a phantom assignment). The worker spun: claim → fail-release → wedged until the server-side must_complete_by expired, repeat.
• [x] Root causes: (1) VRAM admission ran after claiming (client.py::_admit_job_resources), turning every too-big job into a claim/release round-trip; (2) the worker advertised every Ollama-discovered model (discover_and_register_models) regardless of whether it could fit, so the scheduler kept targeting impossible work here; (3) the release call's swallowed 404 left the worker phantom-busy.
• [x] Fix — pre-claim gate: added WorkerClient._resolve_admission(model_id, job_type) (shared by the gate and the post-claim safety net). _job_polling_loop now runs the VRAM/RAM admission check on the available-job model before claim_job, and skips (⏭️ Skipping job … (won't run here)) instead of claiming-then-releasing. Sidesteps the broken release path entirely for the common case.
• [x] Fix — gate advertised models: new ResourceAdmissionController.fits_total_capacity(model_info, engine) measures a model against the host's total VRAM (or RAM on CPU hosts) with the same safety margin, ignoring current occupancy. discover_and_register_models skips models that can't fit and logs Discovered N model(s); registered M that fit this host, so the worker stops advertising (and the scheduler stops routing) models it can never serve. Fails open when size is unknown.
• [x] release_job 404 — deferred (per user): the corrected release contract (likely needs assignment_id or a changed route) wasn't confirmed this round, so release_job is unchanged. The pre-claim gate means release rarely fires now. Tracked for follow-up.
• [x] Auto-update disabled by default: code default was already False (config.py), but the fallbacks (client.py init, config.py summary, auto_updater.py) defaulted True and the installer printed "Auto-update is enabled by default". Flipped all fallbacks to False and rewrote the ubuntu_setup.sh final message to "DISABLED by default … to enable, set AUTO_UPDATE_ENABLED=true". With the updater unset, the forced-update_required path (which ignored the flag and was failing on a 404 download URL) no longer runs unless explicitly opted in.
• [x] Follow-up — model-size overestimate excluded fitting models: gemma4:e4b (9.6GB on a 16GB GPU) was wrongly skipped as "exceeds this host's memory capacity". Two compounding defects: (A) discovery's OllamaEngine._parse_model_info set quantization from the name only (parsed.get("quantization")), so a tag like e4b (no qN_N) looked un-quantized — unlike get_model_info, which also reads Ollama's reported quantization_level; (B) InferenceEngine.estimate_model_memory then hit its unknown-quant fallback and doubled the on-disk size assuming FP16 (9.6→19.2GB → ×1.2 margin = 23GB > 16.3GB → excluded), even though an Ollama GGUF's on-disk size already is the quantized weight footprint. Fixed both: _parse_model_info now falls back to quantization_level (so Q4_K_M → q4 branch → vram ≈ size); and the estimator treats gguf/ggml with unknown quant as size × 1.2 (KV/activation headroom) instead of × 2, reserving the FP16 × 2 assumption for genuinely non-quantized formats. Also widened quant matching (q2/q3/q5/q6 in addition to q4/q8). gemma now fits under either path (verified: 11.5–14.2GB ≤ 16.3GB). Full suite 446 green; ruff/black/mypy clean.
• [x] Tests/gate: full suite 446 passing; black/isort/ruff clean; mypy unchanged (6 pre-existing Optional errors in the untouched _admit_job_cost, no regression — guarded the new release call to avoid adding one); bandit clean.
• Why this matters: a worker that can't run a model should never claim its jobs or advertise it. The pre-claim gate stops the claim/release churn and the phantom-busy lockup at the source, and capacity-gating the advertised set stops the scheduler from sending impossible work in the first place — independent of the still-broken release endpoint.

Config generator wizard with hardware-based recommendations (2026-06-25) ✅

• [x] Goal: help users author a worker config instead of hand-editing the 17K config/default.yaml. The tool detects the host's hardware and recommends opinionated defaults — most importantly worker.max_context_tier — then writes the simplest working config (only the keys that matter; everything else falls back to built-in defaults). Default output is worker.yaml in the CWD (not config/, which the loader searches first — avoids silently shadowing a deployed config).
• [x] Design (user choices): logic lives in src/core/config_wizard.py; scripts/generate_config.py is a thin standalone entry point (not a microdc subcommand). Plain yes/no wizard (no TUI); auto-detect VRAM/RAM/CPU and recommend; cover all engines. Flow is "summary + accept/customize": detect → show the full recommended config with reasoning → [A]ccept all / [C]ustomize / [Q]uit.
• [x] Structure (OO refactor): replaced the original procedural draft (triplicated per-engine if name == ... branching in build/recommend/customize) with a strategy pattern — an EnginePlan ABC with OllamaPlan/VllmPlan/OpenAICompatPlan subclasses, each owning its concurrency recommendation, engine.<key> YAML block, interactive prompts, and rationale; a make_engine_plan() registry factory. Adding an engine = one subclass + one registry line, no edits to the builder or wizard. A Recommendation dataclass assembles the plan and a Wizard class drives I/O through injectable reader/writer callables (so the A/C/Q flow is unit-testable without a TTY).
• [x] Reuses WorkerConfig (per review feedback — don't duplicate the config model): Recommendation.to_worker_config() builds a real WorkerConfig via a new WorkerConfig.from_dict(); serialization/validation go through the existing WorkerConfig.validate() and a new WorkerConfig.to_yaml(header=...) / save(header=...) (header support added so the generated file keeps its explanatory comment block). No hand-rolled YAML or __new__ validation hack.
• [x] Opinions: max_context_tier from VRAM on GPU hosts (≥24→4, 12–23→3, 6–11→2, <6→1) and from RAM on CPU hosts (capped at tier 2 — long contexts are slow without a GPU). Per-engine concurrency scales with VRAM (vLLM: KV-cache bound, up to 64; ollama GPU: 1–3; openai_compat: 5 provider-limited); global resources.max_concurrent_jobs = max per-engine concurrency. Each recommendation is shown with a one-line rationale.
• [x] Detection: nvidia-smi --query-gpu=memory.total,name for GPU VRAM (largest single GPU), psutil→os.sysconf→sysctl hw.memsize fallback chain for RAM, os.cpu_count() for CPUs. All probes degrade gracefully to None. --vram/--ram override detection; --engine/--max-tier/--server-url/--organization override recommendations; --yes accepts all (non-interactive/CI); --stdout prints instead of writing; --force overwrites.
• [x] Tests: tests/test_config_wizard.py (39 cases) — tier opinion tables, per-engine EnginePlan concurrency/blocks, make_engine_plan factory (+ unknown-engine rejection), nvidia-smi parse (mocked: multi-GPU takes largest, error/non-zero handling), Recommendation.to_config shape per engine + processing + max-jobs, flag overrides, summary reasoning, a parametrized check that every generated config passes WorkerConfig.validate(), the Wizard accept/customize/quit flow via a scripted reader, and the new from_dict/to_yaml helpers. Imported normally (from src.core.config_wizard import …) since the logic now lives in the package. ruff-clean and mypy-clean; full suite 446 passing.
• Why this matters: most of the config surface is irrelevant to any single host. A hardware-aware generator turns "read a 17K reference and guess your tier" into "confirm three recommended values." The strategy split keeps the engine-specific opinions cohesive and extensible, and reusing WorkerConfig means the generator and the running worker share one schema/validator/serializer instead of drifting.

Report 0-token / timed-out generations as FAILED, not COMPLETED (2026-06-25) ✅

• [x] Symptom: a generation job that hit the timeout (or whose stream ended) with 0 tokens was reported to the server as COMPLETED with empty output. Claiming completion on a non-result makes the server treat a failed generation as a finished job — it never gets retried/surfaced. If we got nothing back from the LLM, the job did not complete.
• [x] Root cause (src/jobs/executor.py): after _execute_streaming/_execute_batch returned, result.status was set to JobStatus.COMPLETED unconditionally — empty output was indistinguishable from a real answer. Separately, the per-job job.timeout_seconds (models.py, alias timeout) was never enforced, so the existing except TimeoutError handler was effectively dead code; a stalled generation could hang on the engine's own read-timeout and then fall through as "success".
• [x] Fix: (1) wrap the generation in asyncio.wait_for(..., timeout=job.timeout_seconds) when a timeout is set, re-raising as TimeoutError so the existing handler marks the job FAILED (timeout); (2) after a successful return, treat empty/whitespace-only output from a text-generation/chat job as FAILED (error_type="EmptyGenerationError") instead of COMPLETED. JSON-producing job types (embeddings, container, document/OCR) are exempt. Both paths still flow through the finally block, so the FAILED result is reported to the server's /fail endpoint.
• [x] Tests: tests/test_executor_routing.py::TestEmptyAndTimeoutReportedAsFailed (6 cases) — empty string → FAILED, whitespace-only → FAILED, real output → COMPLETED (regression guard), timeout exceeded → FAILED (timeout), empty stream → FAILED, and the result is still submitted to the server. Full executor suite green (31), broader job suites green (45 total). No new mypy errors.
• Why this matters: mislabeling a 0-token timeout as COMPLETED silently drops real work — the server thinks the job succeeded and returns an empty answer to the user. Failing the job (with a timeout reason) lets the platform retry or surface it.

Native vLLM engine — worker-managed vllm serve (2026-06-24) ✅

• [x] Goal: a first-class vllm engine that the worker launches and supervises itself, rather than only connecting to a vLLM server started out-of-band (openai_compat:vllmlocal). Reproduces, from config, an invocation like VLLM_USE_FLASHINFER_SAMPLER=0 CUDA_HOME=... uv run vllm serve openai/gpt-oss-20b --async-scheduling --max-model-len 100000 --port 8000, with a gpt-oss-20b -> openai/gpt-oss-20b alias.
• [x] Design (user choices): worker manages the process (not attach-only); one fixed model per process (no on-demand switching); structured config fields (launcher/model/host/port/extra_args/env) over a raw command string; auto-restart with backoff on crash; wait indefinitely for load but health-gate readiness via /health; vLLM stdout/stderr to a dedicated log file; subprocess strategy (not in-process AsyncLLMEngine, which couples torch/CUDA into the worker, sacrifices crash isolation, and still spawns vLLM's own child processes).
• [x] Implementation: src/engines/vllm_engine.py → VLLMEngine(OpenAICompatEngine). vllm serve is OpenAI-compatible, so the subclass inherits all inference/streaming/embeddings/cost/alias logic and adds only lifecycle: initialize() launches the process (own session/process group via start_new_session=True) and returns without blocking; health_check() is True only when the process is alive and GET /health → 200; a background monitor relaunches on unexpected exit (_relaunch_with_backoff, capped at restart.max_backoff); shutdown() SIGTERMs the process group, escalates to SIGKILL after a grace period, and closes the OpenAI client. list_models() returns [] until healthy so heartbeats stay quiet during load. base_url is derived from the managed host/port. Importing the module needs only httpx — not vllm/torch/CUDA (those live in whatever env launcher resolves to).
• [x] Refactor: extracted OpenAICompatEngine._apply_common_config() + _connect() from initialize() so the subclass reuses base_url/timeout/api_key/alias/cost setup without the parent's connectivity health-gate. openai_compat behavior unchanged (existing suite green).
• [x] Wiring: client.py::_create_engine gains a vllm branch (supports vllm:<instance> named instances); config.py already listed vllm as a valid engine type; engines/__init__.py exports VLLMEngine; per-engine concurrency is read from engine.vllm.concurrency like other engines.
• [x] Config: documented engine.vllm block in config/default.yaml (commented in available) matching the user's launch command, with restart, health_path, log_file (~/.microdc/vllm.log).
• [x] Concurrency tuning: bumped the default engine.vllm.concurrency to 32 (was 5). Real-world telemetry on an RTX 5090 (32GB) serving gpt-oss-20b showed GPU KV cache usage at ~0.5–1.3% with Running: 2–4 reqs, Waiting: 0 — i.e. vLLM was starved, not saturated; the worker's concurrency: 5 was the bottleneck, not the GPU. Added a "Tuning concurrency on GPU hosts" section to docs/engines/vllm.md explaining the Running/Waiting/GPU KV cache usage signals, the worst-case --max-model-len interaction, and — importantly — that the global resources.max_concurrent_jobs (MAX_JOBS, default 10) caps the sum across engines and will clamp the per-engine value unless raised too. Left the global default and per-engine env override (VLLM_CONCURRENCY) intact so non-GPU workers are unaffected. Follow-up: added an efficient tuning procedure (use GPU KV cache usage as a linear predictor — target ≈ current × (0.80 / current_KV) — to jump to ~80% KV instead of incrementing, then confirm the compute/queue knee) plus a worked example table from mdcw06 (RTX 5090, gpt-oss-20b: 5→~540 tok/s starved, 32→~1500 tok/s still cap-bound at 40% KV, ~64 predicted next) to docs/engines/vllm.md.
• [x] Tests: tests/test_vllm_engine.py (6 cases) — argv/env construction (+ inherited env, own process group), missing-model raises, health gated on both process liveness and /health probe (+ list_models empty until ready), crash triggers relaunch, restart-disabled stays down, shutdown SIGTERMs the group and reaps. Fakes the subprocess + /health so no real vLLM is needed. All 6 pass; test_openai_compat*, test_config, test_executor_routing still green (106 total). mypy src/engines/vllm_engine.py clean (pre-existing mypy noise in openai_compat.py/client.py unchanged).
• [x] Docs: rewrote docs/engines/vllm.md (Planned → Implemented: how-it-works, full config reference, troubleshooting); README Features + Recent Changes updated.
• [x] Install (uv migration): ubuntu_setup.sh now installs the worker with uv. uv is installed system-wide to /usr/local/bin via the official installer with UV_UNMANAGED_INSTALL (fixed dir, self-update disabled) so the service user and the vLLM launcher can use it. The venv is built with uv venv --seed (the --seed is load-bearing: the in-place auto-updater runs <venv>/bin/python -m pip install -r requirements.txt, so pip must exist inside the venv); deps install via uv pip install --python <venv> .... The systemd ExecStart is unchanged (venv/bin/python) since uv venv produces a standard venv.
• [x] Migration safety (re-run on old installs): re-running the script on an existing pip/venv install migrates it to uv — it detects a running service (systemctl is-active) and stops it before recreating the venv (rm -rf venv then uv venv), then restarts it afterward if it had been active. Idempotent; safe to re-run.
• [x] vLLM runtime env (--with-vllm): opt-in flag + interactive prompt (mirrors the podman pattern) installs a dedicated uv env with vllm at /srv/microdcworker/vllm-env. Gated because vLLM pulls multi-GB CUDA wheels and only suits GPU hosts — the worker itself stays free of vllm/torch/CUDA. Final notes print the exact engine.vllm config pointing launcher at vllm-env/bin/vllm (calling the env binary directly avoids needing uv run under the hardened unit). The vLLM env is pinned to Python 3.12 (VLLM_PYTHON, uv auto-downloads it) regardless of the system/worker Python — vLLM wheels lag on 3.13, and proven hosts run 3.12; uv-managed interpreters go under $INSTALL_DIR/uv-python (via UV_PYTHON_INSTALL_DIR) so the hardened service user can execute them. Install uses --torch-backend=auto to match torch's CUDA wheels to the driver (covers Blackwell/CUDA 13); VLLM_PIP_ARGS passes extra flags. docs/engines/vllm.md gained a "Sizing a model to VRAM" section (dense vs MoE, bf16/FP8/4-bit, tensor-parallel) with gpt-oss-20b@32GB and Qwen3-32B@96GB worked examples.
• [x] Systemd hardening for vLLM: added Environment=HOME=/srv/microdcworker and UV_CACHE_DIR=/srv/microdcworker/.cache/uv so uv's cache and vLLM's caches (~/.microdc/vllm.log, ~/.cache, ~/.triton) land inside the existing ReadWritePaths=/srv/microdcworker (the home is /srv/..., unaffected by ProtectHome=true). bash -n clean.
• Why this matters: a single engine.available: [vllm] now brings up, supervises, and tears down a high-throughput vLLM server as part of the worker lifecycle — health-gated so the platform never routes a job into a model that's still loading, and crash-resilient so a vLLM OOM doesn't take the worker down or require manual restart. Keeping it a subprocess (not in-process) preserves crash isolation and lets vLLM keep its own CUDA venv. The installer migrates existing pip/venv deployments to uv without manual steps and, on GPU hosts, can provision the vLLM env turnkey.

Exit (don't retry-loop) on fatal registration failures (2026-06-22) ✅

• [x] Symptom: when the server returned 400 {"detail":"Token was already used and re-authentication is not enabled..."}, the worker logged the error and the run() retry loop re-tried registration every 5s forever (systemd Restart=always would also keep relaunching). The failure needs an operator action — enable re-auth or issue a new token — so retrying is pointless and hides the real problem.
• [x] Root cause: register_worker raised this 400 as a generic ServerCommunicationError, which run()'s except Exception treated as transient → backoff + retry. Additionally, register_worker's trailing except Exception flattened all our typed errors into ServerCommunicationError, so even a fatal type raised earlier would have been downgraded before reaching the loop.
• [x] Fix: new FatalRegistrationError(RegistrationError) in src/core/exceptions.py. server_client.register_worker now raises it for the non-retryable 400s (token already used / re-auth disabled, and expired/invalid token) with an actionable log block, and a except WorkerError: raise guard before the generic handler preserves typed errors. client.register() and client.start() re-raise FatalRegistrationError/FatalCredentialError instead of wrapping them, and client.run() catches them, stops background tasks, and re-raises so the process exits rather than looping.
• [x] Exit codes + systemd: cli.py handles FatalRegistrationError with a tailored recovery message and sys.exit(3) (credential errors stay sys.exit(2)). ubuntu_setup.sh adds RestartPreventExitStatus=2 3 to the unit so systemd does not restart-loop on these operator-actionable exits.
• [x] Tests: tests/test_fatal_registration.py (6 cases) — token-already-used → fatal, expired-token → fatal, generic 400 stays non-fatal ServerCommunicationError, run() exits on FatalRegistrationError and on FatalCredentialError (cleans up tasks, skips the post-loop stop), and run() still retries a transient ServerCommunicationError. Full suite: 394 passing.
• Why this matters: a spent token with re-auth disabled is a dead end until someone acts. Silently retrying forever buries the one log line that tells the operator what to do; exiting with a distinct, restart-prevented code surfaces the problem immediately.

Per-engine spending cap for openai_compat ($ budget) (2026-06-22) ✅

• [x] Goal: let a worker use a finite provider credit (e.g. a $50 AWS Bedrock balance) through the microDC platform and stop once that credit is spent. Cap is per openai_compat instance — openai_compat:bedrock can have its own $50 limit independent of openai_compat:groq.
• [x] Cost model (user choice): per-model pricing in config — input_per_1m / output_per_1m (USD per 1M tokens), keyed by internal model id. Cost = prompt_tokens/1e6 * input + completion_tokens/1e6 * output. Models with no pricing entry count as $0. Bedrock/Groq OpenAI-compat endpoints return token counts in response.usage but no dollar amount, so the worker prices tokens itself.
• [x] Persistence (user choice): cumulative spend per engine in ~/.microdc/spend.json (src/jobs/cost_ledger.py → CostLedger, process-wide singleton via get_cost_ledger()). Atomic temp-file+rename writes under a lock; tolerates a missing/corrupt file by starting fresh. Survives restarts so a real credit isn't exceeded across runs. reset(engine) clears one engine's tally for a new credit period.
• [x] Recording: OpenAICompatEngine._record_usage captures response.usage after each non-streaming completion (_generate_impl) and adds the computed cost to the ledger. Streaming (generate_stream) accumulates output and records from the final usage chunk if present, otherwise an estimate (prompt chars + accumulated output at ~4 chars/token).
• [x] Admission (user choice — "stop claiming, release jobs"): WorkerClient._admit_job_cost(job) runs in the claim loop right after the VRAM/RAM check. Engines with no cost_limit_usd always pass. For cost-limited engines it estimates the next job's cost (prompt length + max_tokens) and, if spend has reached the cap or the estimate would exceed it, releases the job back to the pool via release_job(reason=...) so another worker can take it. is_over_budget treats at-the-cap as exhausted.
• [x] Config: engine.<name>.cost_limit_usd + engine.<name>.pricing documented in config/default.yaml (concrete example on the commented bedrock block; doc block in the openai_compat section). Opt-in — absent/zero limit disables tracking entirely.
• [x] Tests: tests/test_cost_ledger.py (5 cases — add/get, non-positive ignored, persist+reload, per-engine reset, corrupt-file fresh start) and tests/test_openai_compat_cost.py (8 cases — compute_cost, unpriced-is-free, estimate from prompt+max_tokens, ledger accumulation, over-budget boundary, no-limit no-op, config parsing, disabled-when-unset). All 13 passing; existing executor/vram-admission suites still green; no new mypy errors.
• Why this matters: without a cap, pointing a worker at a metered endpoint risks burning the whole credit (or real money) the moment the platform routes enough jobs its way. The gate is best-effort and fail-open — if an engine can't be resolved or an estimate errors, the job is admitted rather than the worker starved — but actual spend is always recorded from real usage, so the persisted total is accurate even if a pre-flight estimate was off.

Fix over-claiming: has_any_capacity() ignored pending jobs (2026-06-19) ✅

• [x] Symptom: with max_concurrent_jobs: 1, every poll cycle the worker claimed one job, then immediately attempted to claim the other jobs in the same batch — each rejected server-side with worker_busy (1/1). 4 wasted claim calls + log spam per cycle.
• [x] Root cause: the polling loop (client.py:_job_polling_loop) breaks on not has_any_capacity() after queuing each job, but JobExecutor.has_any_capacity() (executor.py) only subtracted _pending_job_count on the global semaphore branch. The actual limit is the per-engine semaphore (ollama: 1), and its branch returned any(sem._value > 0) — a just-queued job hasn't acquired its engine semaphore yet (notify_job_queued only bumps _pending_job_count), so the engine still looked free and the break never fired.
• [x] Fix: per-engine branch now returns (sum(sem._value) - _pending_job_count) > 0, so a queued-but-not-started job correctly marks a 1-slot engine as full. The server's worker_busy guard already prevented real over-claiming; this stops the redundant claim attempts at the source.
• [x] Tests: tests/test_executor_routing.py::TestHasAnyCapacity (3 cases) — pending job fills a single-slot engine, started job keeps it full, single pending job leaves room on a multi-slot engine. Full file 21 passing.

VRAM/RAM-aware job admission (2026-06-15) ✅

• [x] Symptom: job admission was concurrency-only. The polling loop (client.py:_job_polling_loop) gated claims on JobExecutor.has_any_capacity() (semaphore slots) with no notion of GPU memory. With qwen3:32b already resident in VRAM, a worker would still claim a gpt-oss:20b job and then try to load a second large model into a GPU that couldn't hold it.
• [x] Fix: new src/jobs/vram_admission.py → ResourceAdmissionController. For a claimed job whose model is not already loaded, it estimates the model's footprint (via the engine's estimate_model_memory) and admits only if both conditions hold: free >= estimate * vram_model_margin (fit-with-margin, default 1.2) and (free - estimate) >= vram_headroom_percent% of total (post-load headroom, default 20). Already-loaded models short-circuit to admit. Occupancy is summed from every loaded engine's _loaded_models so the estimate is consistent with how the incoming model is sized.
• [x] GPU vs CPU: on hosts with GPUs the check uses aggregate VRAM across all GPUs (Ollama can spread a model over multiple devices, so it need not fit on one). On CPU-only hosts the same two-condition rule is applied to system RAM (using live available_mb for occupancy). Models whose size can't be resolved fail open (admit + warn) so the worker isn't starved by missing metadata.
• [x] Wiring: WorkerClient._admit_job_resources(job) runs after the job is built and before job_queue.put. Container/docker jobs and jobs without a model_id bypass the check. On no-fit the job is released via server_client.release_job(reason=...) so another worker can take it (the user-chosen "skip / don't claim" behavior). The controller is constructed in initialize_components with headroom_percent/model_margin read from config.
• [x] Config: resources.vram_headroom_percent (env VRAM_HEADROOM_PERCENT, default 20) and resources.vram_model_margin (env VRAM_MODEL_MARGIN, default 1.2) added to config/default.yaml.
• [x] Tests: tests/test_vram_admission.py (9 cases) — already-loaded short-circuit, fits-on-empty-GPU, the qwen/gpt-oss block, headroom-boundary block, multi-GPU aggregate admit, occupancy summed across engines, CPU/RAM admit + block, and unknown-size fail-open. All passing.
• Why this matters: this is the difference between a worker that politely declines work it can't run and one that claims a job, fails to load the model (OOM or thrashing), and either errors the job or evicts a model another in-flight job needs. The gate is best-effort and fail-open, so it shrinks the OOM window without ever deadlocking the worker when sizing data is missing.

CI: docker-build resilience to Docker Hub stream resets (2026-05-05) ✅

• [x] Symptom: GitLab docker-build job intermittently fails partway through unpacking the nvidia/cuda:12.2.2-runtime-ubuntu22.04 base image with error building image: error building stage: failed to get filesystem from image: stream error: stream ID 13; INTERNAL_ERROR; received from peer. Manifest fetch succeeds (kaniko even logs Returning cached image manifest), then the HTTP/2 stream carrying the rootfs layer is reset by Docker Hub mid-transfer. Pure registry flake — same SHA builds fine on retry.
• [x] Fix: .gitlab-ci.yml docker-build job gains retry: { max: 2, when: runner_system_failure } and the kaniko invocation now passes --image-fs-extract-retry=3 --push-retry=3. Two layers of defense: kaniko re-pulls a single failed layer in-process before the runner gives up on the job, and the GitLab runner re-launches the whole job up to twice if kaniko itself dies on a system-level fault.
• Why this matters: the failure mode is non-deterministic and external — it isn't fixable in the Dockerfile or our code. Without job-level retry every Docker Hub hiccup is a red pipeline that needs a human to click "Retry"; with these two flags, transient registry resets self-heal.

Auto-update: make it actually work end-to-end (2026-05-04) ✅

• [x] Symptom: every auto_update.* setting from YAML/env was silently ignored, and install_update was a no-op # TODO block — perform_update would graceful-shutdown → download → verify → backup → do nothing → call restart_worker, which then tried systemctl restart microdc-worker (wrong service name) from an unprivileged process. Net effect: the worker bounced for nothing and came back at the same version.
• [x] Root cause #1 (config plumbing): WorkerAutoUpdater._get_config_value walked self.config.__dict__, but the real WorkerConfig keeps its data in self._config. Every YAML/env value fell through to the hardcoded default. Tests didn't catch it because the fixture used Mock(spec=WorkerConfig).__dict__["auto_update"] — a shape that never occurs in production.
• [x] Root cause #2 (install no-op): the entire tarball/zip extraction + version-file rewrite was commented out behind # TODO we need a better way to do auto updates.
• [x] Root cause #3 (restart strategy): restart_worker called systemctl restart microdc-worker. Under ubuntu_setup.sh the unit is microdcworker, and the worker runs as the unprivileged microdcworker user — it can't run privileged systemctl anyway.
• [x] Root cause #4 (deferred updates lost): save_worker_state persisted pending_update but load_worker_state was never called on startup, so out-of-window deferrals evaporated on every restart.
• [x] Root cause #5 (env vars not wired): ubuntu_setup.sh's environment file template documents AUTO_UPDATE_ENABLED / UPDATE_CHECK_INTERVAL, but WorkerConfig._apply_env_overrides had no mapping for them.
• [x] Root cause #6 (manual scripts targeted wrong layout): scripts/update/update_worker.sh defaulted to /opt/microdc/worker + service microdc-worker. ubuntu_setup.sh installs to /srv/microdcworker + service microdcworker.
• [x] Fix: _get_config_value now delegates to self.config.get(key, default) (with a duck-typed fallback for tests). install_update is implemented for real — safe tarball/zip extraction (data filter on 3.12+, manual traversal validation otherwise), atomic rename swap of <worker_dir>/src with rollback on copy failure, regex-based version.py rewrite, best-effort pip install -r requirements.txt against the deployed venv. restart_worker exits cleanly with os._exit(0); systemd Restart=always brings the new code back. _hydrate_state_from_disk runs in __init__ and restores pending_update. handle_update_check re-applies a previously deferred update on the first tick that lands inside the maintenance window, and persists deferrals so they survive restarts. WorkerConfig._apply_env_overrides now wires AUTO_UPDATE_ENABLED, UPDATE_CHECK_INTERVAL, ALLOW_MAJOR_UPDATES, UPDATE_RESTART_DELAY, UPDATE_BACKUP_RETENTION with bool/int coercion. update_worker.sh defaults switched to /srv/microdcworker + service microdcworker, with WORKER_DIR / BACKUP_DIR / SERVICE_NAME env-overrides for non-default installs.
• [x] Tests: replaced Mock(spec=WorkerConfig) fixture with a real WorkerConfig constructed via __new__ and pre-populated _config, then sandboxed worker_dir / state_file to tmp_path so save_worker_state / _hydrate_state_from_disk don't touch the repo root. Added coverage: _get_config_value against real WorkerConfig, install_update happy path with synthetic tarball, path-traversal rejection, exit-based restart, deferral persistence, hydrate-pending-update from state file, AUTO_UPDATE_ENABLED env override end-to-end. Full suite: 358 tests passing.
• [x] Docs: README "Auto-Update Feature" section now says explicitly that the feature is geared toward ubuntu_setup.sh-installed (systemd-managed) systems, defaults to disabled, and that the worker exits to let systemd relaunch instead of calling systemctl. default*.yaml already had enabled: false baked in; the in-code default in WorkerConfig._get_defaults is now also False for consistency.
• Why this matters: auto-update was the kind of feature where the tests were green, the config keys looked right, and the logs even said "Update completed successfully" — but no update actually happened. This was a six-defect compound failure that any one fix would have masked rather than resolved. The combined fix means auto-update either works (on ubuntu_setup.sh-installed systems with AUTO_UPDATE_ENABLED=true) or stays cleanly off (everywhere else); there's no longer a silent in-between.

libpod mount translation: split SELinux options into a list (2026-05-02) ✅

• [x] Symptom: first real container_stream job after the podman-py refactor failed on mdcw06 with 500 Server Error: Internal Server Error (unknown mount option "ro,Z": invalid mount option). Job: linuxserver/blender, two bind mounts (/input ro,Z and /output rw,Z).
• [x] Root cause: the engines (docker_engine.py, container_stream_engine.py) build volumes={path: {"bind": ..., "mode": "ro,Z"}} and append the SELinux relabel suffix (Z) when talking to podman. The docker-compat API used to split that comma-separated mode into individual options server-side. The libpod API does not — it treats the whole mode string as one option name and rejects "ro,Z" because there's no option literally called that. podman-py passes the mode through verbatim.
• [x] Fix: new _volumes_dict_to_libpod_mounts() helper in src/utils/docker.py. _PodmanBackend.run_container() now translates volumes={...} into mounts=[{type, source, target, read_only, options}] with the comma-separated mode pre-split: "ro,Z" → read_only=True, options=["Z"]. SELinux relabel and any future multi-flag modes (Z, z, U, nosuid, …) work through libpod. _DockerBackend is unchanged — docker-compat still parses mode correctly.
• [x] Tests: added TestVolumesDictToLibpodMounts (8 cases) covering rw/ro/Z/multiple-extra-options/whitespace-stripping, plus test_run_container_translates_volumes_to_mounts_with_split_options to assert the podman backend passes mounts= (not volumes=) when given a docker-py volumes dict. Total suite: 350 tests passing.
• Why this matters: this was the first real-job regression after the docker-py → podman-py refactor. The libpod API is stricter than docker-compat in subtle ways; this fix addresses one such case and the helper is now the place to handle any future libpod-vs-docker-compat translation issues for mounts.

src/utils/docker.py: backend split — DockerBackend + PodmanBackend (2026-05-02) ✅

• [x] Root cause: podman 4.9.3's docker-compat API silently ignores HostConfig.DeviceRequests[{Driver:"cdi", DeviceIDs:[...]}] (the docker-py CDI form). On mdcw06, every other layer of GPU passthrough verified clean — host driver, CDI spec, socket access, podman version 4.9.3 ≥ 4.1, native podman --remote --device nvidia.com/gpu=all worked end-to-end (Layer 3a) — but docker-py through compat (Layer 3b) silently started containers without a GPU. Both device_requests (Driver:"cdi") and devices=["nvidia.com/gpu=all"] failed: the former gets dropped silently, the latter gets stat'd as a host path. The libpod API on the same socket honors CDI cleanly; only docker-py + the compat endpoint can't see it.
• [x] Fix: refactored src/utils/docker.py to split daemon I/O behind a _ContainerBackend ABC with two implementations:
• _DockerBackend — wraps docker-py, used for real Docker daemons. GPU = DeviceRequest(capabilities=[["gpu"]]) (nvidia OCI hook). Existing logic, factored out byte-for-byte.
• _PodmanBackend — wraps podman-py (PodmanClient), used for Podman daemons. Talks to the libpod endpoint on the same socket file (different URL path, same Unix socket). GPU = devices=["nvidia.com/gpu=all"] — the form the libpod API actually translates to CDI. Container lifecycle is create() + start() (vs docker-py's containers.run()) for predictable shape across podman-py versions.
• [x] Backend selection (_detect_backend()):
• DOCKER_HOST contains "podman" → _PodmanBackend immediately (skips probe entirely).
• Else docker-py probe — if version().Components mentions "podman" → _PodmanBackend.
• Else _DockerBackend.
• [x] Facade (DockerContainerManager) is unchanged from the engines' POV — same method signatures, same semantics. Internal _is_podman boolean replaced with is_podman property; engines updated to use the property (docker_engine.py, container_stream_engine.py). Container objects returned by run_container() stay opaque (.short_id, .start, .stop, .wait, .remove, .logs, .put_archive all exist on both clients).
• [x] Normalized return shapes: wait_container() returns {"StatusCode": int} regardless of backend (podman-py's container.wait() returns int; docker-py returns dict). get_container_output() decodes podman-py's chunk-iterator output to match docker-py's bytes-blob interface.
• [x] Tests: new tests/test_container_backends.py — 45 tests covering both backends + the facade + _detect_backend + _wrap_command_for_s6. Critical asserts: Docker GPU path emits DeviceRequest(capabilities=[["gpu"]]) and no devices=; Podman GPU path emits devices=["nvidia.com/gpu=all"] and no device_requests=. Full suite (341 tests) passes.
• [x] Dependencies: setup.py [container] extra now includes both docker>=7.0.0 and podman>=5.0.0. setup_podman.sh installs both into the worker venv as $SERVICE_USER (with -H so pip cache lands in the user's home, not /root).
• Why this matters: this closes the loop on a multi-week debugging odyssey. Every previous attempt (registries config, group membership, /run/podman traversal, podman version gating, --device vs --gpus, device_requests vs devices) addressed real issues but never the underlying truth — the docker-compat API is a Docker-shaped pipe and CDI is not Docker-shaped. The right answer was always to use the right pipe (libpod) for the right protocol. The verifier we built proves the cure works; the refactor makes the cure permanent.

/run/podman directory traversal + true idempotency (2026-05-02) ✅

• [x] Root cause: systemd creates /run/podman/ with default mode 0700 root:root when binding the socket. Even though the socket file itself is srw-rw---- root:podman and the worker user is in the podman group, any connect(2) from the worker fails with PermissionError(13, 'Permission denied') because access(2) fails at the directory-traversal step before it ever reaches the socket. Symptom on mdcw06: worker logs Failed to initialize Docker engine: ... PermissionError(13, 'Permission denied') after a fresh ubuntu_setup.sh --with-podman. Fully reproducible — sudo -u microdcworker ls /run/podman/ returns "Permission denied" while srw-rw---- root podman on the socket itself looks fine. Misled me into chasing AppArmor for an hour.
• [x] Fix in setup_podman.sh:
• Added ExecStartPre=/usr/bin/install -d -m 0750 -o root -g podman /run/podman to the /etc/systemd/system/podman.socket.d/group.conf drop-in. Re-applied on every socket start, so the directory keeps its perms across reboots and socket restarts.
• Added a one-shot chgrp podman /run/podman && chmod 0750 /run/podman immediately after writing the drop-in, so the live system is fixed without waiting for a socket restart.
• [x] True idempotency: setup_podman.sh now ends with a self-verification step that runs client.ping() as $SERVICE_USER against the socket using docker-py from the worker's venv — the same code path the worker exercises at runtime. If this fails, the script dumps diagnostics (groups in /etc/group vs sudo'd, socket perms, AppArmor profile count, dmesg denials, NSS-cache hints) and exits non-zero. Going forward, setup_podman.sh returning successfully is a contract: the worker user can talk to the daemon. No more "the script said it succeeded but the worker still can't connect."
• Why this matters: this was the third class of failure where individual checks (id microdcworker shows the group, socket perms look right, drop-in is loaded) all passed but the system didn't work. The one-line connection test catches all such cases — including ones we don't anticipate yet — turning silent install-time bugs into loud install-time errors instead of runtime errors in a service journal three hours later.

podman version gating for CDI (2026-04-30) ✅

• [x] Both setup_podman.sh (post-apt-install) and verify_podman_gpu.sh (Layer 2c, before any container test) now check podman --version / daemon Server.Version and surface a loud, actionable error when it's < 4.1.
• [x] Why: Ubuntu 22.04 ships podman 3.4.4. CDI device support landed in podman 4.1. On 3.4.x the daemon silently ignores DeviceRequest(driver="cdi", ...) over the docker-compat socket and --gpus / --device nvidia.com/gpu=* on the CLI — containers start without a GPU and fail with the misleading "nvidia-smi: executable not found" (because the CDI hook never fires to bind-mount the driver libs from the host). Caught on aisrv02 during verification.
• [x] Failure messages point at the upgrade paths: Ubuntu 24.04 LTS (podman 4.9.x) or Kubic backports for 22.04. verify_podman_gpu.sh exits non-zero before Layer 3a so the operator doesn't waste time interpreting "stat ... no such file or directory" or runc errors.
• Why this matters: this was the first failure mode where Layer 1 (host driver), Layer 2 (CDI registration via nvidia-ctk), and the socket access checks all passed but the daemon couldn't honor the requests. A single version check turns a confusing runtime error into a one-line upgrade instruction.

verify_podman_gpu.sh: non-root run mode (2026-04-30) ✅

• [x] Auto-detects EUID. As root, runs the full check (switching to $SERVICE_USER for Layer 3a, using the worker venv for Layer 3b — the authoritative test). As a non-root user already in the podman group, runs Layers 1-3a as the current user (Layer 3b skipped with a note pointing at sudo).
• [x] Layer 3a uses --device nvidia.com/gpu=all (podman's native CDI device-request syntax, added in 4.1). Do not use --gpus all: that's a Docker-compat flag that requires the legacy nvidia-container-runtime OCI hook configured in containers.conf. setup_podman.sh produces a CDI-only setup, so --gpus silently no-ops and the container starts without a GPU (failure mode: exec: "nvidia-smi": executable file not found — the CDI hook never fires to bind-mount the binary). Confirmed on aisrv07 (podman 4.9.3): --gpus all failed, --device nvidia.com/gpu=all succeeded. The Layer 2c version gate ensures --device <CDI-ref> isn't sent to a pre-4.1 daemon (which would stat it as a host path and fail with "no such file or directory").
• Why this matters: operators want to sanity-check GPU passthrough from their own shell without sudo. The split keeps the worker-runtime-path validation (Layer 3b) gated on root while still letting non-root operators verify the daemon end-to-end.

setup_podman.sh: standalone podman/CDI setup script (2026-04-30) ✅

• [x] Extracted the setup_podman() function from ubuntu_setup.sh into a standalone setup_podman.sh at the repo root. Same behavior, now invokable independently of the worker install — useful for adding Podman to an already-deployed worker, re-running after a driver upgrade, or adding non-service users to the podman group.
• [x] Flags (env var fallbacks): --service-user= (SERVICE_USER), --install-dir= (INSTALL_DIR), --service-name= (SERVICE_NAME), --jobs-dir= (JOBS_DIR), --podman-users=u1,u2 (PODMAN_USERS), --no-cdi (NO_CDI=1), -h/--help. Sensible defaults match the worker's installed paths so a no-arg invocation Just Works.
• [x] --podman-users=<list>: new feature. Comma-separated list of extra users to add to the podman group on top of the service user. Lets operator accounts (e.g. jeffreyr) share the worker's rootful daemon via podman --remote --url unix:///run/podman/podman.sock so they see the same image cache and CDI runtime as the worker. The script warns that group membership doesn't apply to current shells — operators must newgrp podman or re-login.
• [x] ubuntu_setup.sh --with-podman now delegates to setup_podman.sh (resolved via $SCRIPT_DIR/setup_podman.sh), forwarding --service-user/--install-dir/--service-name/--jobs-dir. The original ergonomics survive — sudo ./ubuntu_setup.sh --with-podman still works as a one-shot install — but the podman logic no longer lives in two places.
• [x] Post-install summary in ubuntu_setup.sh now points at sudo ./setup_podman.sh as the faster path for adding Podman later (no need to re-run the full worker installer).
• [x] README quick-start gained a snippet showing the standalone script + --podman-users= usage.
• Why this matters: the original setup_podman() was buried inside the installer, so adding a new user to the podman group meant either re-running the whole installer (rebuilds venv, copies source) or knowing the four usermod / drop-in commands by heart. The standalone script makes incremental ops trivial.

ubuntu_setup.sh: registries config via drop-in (2026-04-30) ✅

• [x] Switched the unqualified-search-registries configuration in setup_podman() from >> /etc/containers/registries.conf to a drop-in at /etc/containers/registries.conf.d/00-microdcworker.conf.
• [x] Why: appending to the main registries.conf is TOML-fragile — if the file already contains a [section] (e.g. [aliases] shipped with Ubuntu's podman package), the appended line lands inside that section as a sub-key and podman silently ignores it. Operator symptom: Error: short-name "nvidia/cuda:..." did not resolve to an alias and no unqualified-search registries are defined, even though the script appeared to write the line. Drop-in files under registries.conf.d/ are merged independently of the main file's structure, take precedence in lex order, and are safe to overwrite on re-run.
• For operators hit by this on an existing install: re-run sudo ./ubuntu_setup.sh --with-podman, or apply the fix manually with sudo mkdir -p /etc/containers/registries.conf.d && printf 'unqualified-search-registries = ["docker.io"]\n' | sudo tee /etc/containers/registries.conf.d/00-microdcworker.conf.

verify_podman_gpu.sh: end-to-end NVIDIA passthrough verifier (2026-04-30) ✅

• [x] New verify_podman_gpu.sh at the repo root, runs sudo and exits non-zero on the first failed layer with an actionable hint:
• Layer 1 — nvidia-smi -L lists ≥ 1 GPU (host driver healthy).
• Layer 2 — /etc/cdi/nvidia.yaml exists and nvidia-ctk cdi list registers nvidia.com/gpu=* devices.
• Layer 2b — microdcworker user can read+write /run/podman/podman.sock (catches missing podman group / drop-in regressions).
• Layer 3a — sudo -u microdcworker podman --remote --url unix:///run/podman/podman.sock run --rm --device nvidia.com/gpu=all <image> nvidia-smi -L. Must hit the rootful socket via --remote, not rootless podman: rootless podman run as the worker user fails with no subuid ranges found for user "microdcworker" and is not the path the worker takes at runtime — the worker user's docker-py talks to the rootful daemon's docker-compat socket and the daemon runs containers as root. Image is pulled in a separate step first so a registry/network failure (e.g. missing unqualified-search-registries) is reported distinctly from a CDI runtime failure.
• Layer 3b — boots $INSTALL_DIR/venv/bin/python as microdcworker, imports docker, points it at unix:///run/podman/podman.sock, and runs containers.run(..., device_requests=[DeviceRequest(driver="cdi", device_ids=["nvidia.com/gpu=all"])]) — the exact call shape used by src/utils/docker.py:226-230. This is the only layer that actually validates what the worker does at job time.
• [x] Flags: --skip-pull (layers 1-2 only, no network), --image=<ref> (override default nvidia/cuda:12.4.0-base-ubuntu22.04), -h/--help.
• [x] README Quick Start section now points operators at verify_podman_gpu.sh immediately after ubuntu_setup.sh --with-podman.
• Bug surfaced + fixed: docker (docker-py) was imported by src/utils/docker.py but not declared anywhere — fresh --with-podman installs would ImportError at the first container job. Added a [container] extra ("docker>=7.0.0") to setup.py alongside the existing [gpu]/[ollama]/etc. pattern (container support is opt-in, so doesn't belong in core requirements.txt), and setup_podman() now runs pip install 'docker>=7.0.0' into the worker venv. Operators on existing installs can fix in place: sudo /srv/microdcworker/venv/bin/pip install 'docker>=7.0.0'. Layer 3b of the verifier catches the regression if it ever recurs.
• Why this matters: setup script ran cleanly was a poor proxy for "GPU jobs will work" — three independent things have to align (host driver, CDI registration, docker-compat API translating device_requests[driver=cdi] correctly), and only the third is what the worker actually exercises. Operators now have a one-shot verifier that mirrors the runtime path.

ubuntu_setup.sh: opt-in Podman + CDI configuration (2026-04-29) ✅

• [x] Added --with-podman / --no-podman CLI flags (and WITH_PODMAN=1 env var) to ubuntu_setup.sh. When neither flag is set, the script prompts interactively in a TTY and defaults to "no" non-interactively.
• [x] New setup_podman() function installs podman + uidmap, creates the podman group, adds the service user to it, drops in a /etc/systemd/system/podman.socket.d/group.conf (SocketGroup=podman, SocketMode=0660), and enables the rootful socket at /run/podman/podman.sock.
• [x] If nvidia-smi -L succeeds, also installs nvidia-container-toolkit from the official repo and writes the CDI spec to /etc/cdi/nvidia.yaml. Skips silently with a warning otherwise (CPU-only path still works).
• [x] Writes a worker drop-in at /etc/systemd/system/microdcworker.service.d/podman.conf:
• StateDirectory=microdcworker → grants /var/lib/microdcworker write access under the existing ProtectSystem=strict
• SupplementaryGroups=podman → socket access for the service user
• Environment=DOCKER_HOST=unix:///run/podman/podman.sock
• Environment=DOCKER_ENGINE_ENABLED=true
• Environment=DOCKER_OUTPUT_DIR=/var/lib/microdcworker/jobs (avoids the PrivateTmp=true isolation that broke /tmp/microdc/... previously)
• [x] Idempotent: re-running with --with-podman is safe (group/usermod/dropin/CDI all overwrite cleanly). The drop-in approach keeps the base unit unchanged.
• [x] README updated: Quick Start shows the three invocation modes.
• Why this matters: setting up Podman + CDI + permissions + the worker drop-in by hand has 6+ steps and is the exact path that bit us throughout debugging this conversation. One flag now wires it all up correctly.

Auto-restore env vars for s6-overlay container images (2026-04-29) ✅

• [x] Symptom: jobs running on linuxserver/* (and other s6-overlay) images saw KeyError on env vars passed via the job's environment field, even though the values were correctly set on the container. s6's init supervisor strips them from the CMD process and stashes them in /var/run/s6/container_environment/ — customer scripts had to know to use with-contenv to recover them.
• [x] Worker now auto-wraps command in DockerContainerManager.run_container() (src/utils/docker.py) with a tiny /bin/sh -c prefix that re-exports anything in /var/run/s6/container_environment/ before exec-ing the original command. On non-s6 images the directory doesn't exist, the if check is false, and the wrapper is a runtime no-op.
• [x] Both string-form ("python main.py") and list-form (["python","-c","..."]) commands handled. shlex.quote used for string-form to keep the original semantics intact.
• [x] No image inspection or per-image policy needed — the wrapper is universally safe and adds one fork+exec (microseconds).
• Why this matters: customers shouldn't have to know that an image uses s6-overlay or that they need to write ["/usr/bin/with-contenv", "bash", "/input/setup_run.sh"] instead of ["bash", "/input/setup_run.sh"]. The worker normalizes this so jobs behave the same way regardless of the image's init system.

Container entrypoint override (2026-04-29) ✅

• [x] Added entrypoint parameter to DockerContainerManager.run_container() (src/utils/docker.py). Passing None keeps the image default; passing "" or [] clears the entrypoint entirely (matches docker run --entrypoint ""); a string or list sets a specific entrypoint.
• [x] DockerEngine.process() and ContainerStreamEngine.process() now read input_data.get("entrypoint") and forward it to run_container. Both container and container_stream job types support it.
• [x] Documented the new entrypoint field in the README's container input-field table, with a callout that it's the right knob for desktop/init-style images (e.g. linuxserver/blender) where the default ENTRYPOINT (s6-overlay) doesn't run user scripts.
• Server-side recommendation (not yet done): surface entrypoint in the job-submission API alongside command / environment / gpu so submitters can opt out of the image's ENTRYPOINT. The current payload schema accepts arbitrary keys (it's stored verbatim and forwarded to the worker), so entrypoint already flows through end-to-end if the submitter sets it — the recommendation is to make it a first-class, validated field in the API spec.
• Client/SDK recommendation (not yet done): add an entrypoint field to whatever job-builder helpers exist on the submitter side. Convention: omit (image default), ""/[] (clear), or a string/list (override). This unblocks images whose default ENTRYPOINT is a desktop/init system, while leaving inference-image jobs untouched.

Podman support: GPU passthrough via CDI (2026-04-29) ✅

• [x] Root cause: when DOCKER_HOST points at Podman's docker-compat socket, the worker connects fine but GPU jobs silently start without a GPU. Podman's docker-compat API does not honor docker-py's standard DeviceRequest(capabilities=[["gpu"]]) (the nvidia driver plugin path); it expects CDI device IDs like nvidia.com/gpu=all.
• [x] DockerContainerManager.initialize() in src/utils/docker.py now inspects client.version()["Components"] and sets self._is_podman when "Podman" appears — single detection at startup, no per-call cost.
• [x] run_container() branches on _is_podman: emits DeviceRequest(driver="cdi", device_ids=["nvidia.com/gpu=all"]) (or per-index nvidia.com/gpu=<id> when gpu_device_ids is set) for Podman; preserves the existing nvidia-toolkit DeviceRequest for real Docker.
• [x] Updated README.md Docker section: prerequisites now list Podman ≥ 4.4 + CDI (nvidia-ctk cdi generate) and added a "Using Podman instead of Docker" subsection covering rootful/rootless socket setup and DOCKER_HOST configuration.
• Operational notes:
• Requires Podman ≥ 4.4 (when Driver: "cdi" was added to the docker-compat handler).
• CDI spec must exist at /etc/cdi/nvidia.yaml — regenerate after driver upgrades.
• For rootless Podman as a dedicated service user, loginctl enable-linger is required so the user socket survives logout.

ContainerStreamEngine — streaming Docker for training & long-running jobs (2026-04-20) ✅

• [x] New ContainerStreamEngine (src/engines/container_stream_engine.py) — subclass of DockerEngine with always-on log streaming, batched heartbeats, and cancellation support
• [x] job_type="container_stream" routes to the new engine; existing job_type="container" unchanged
• [x] _LogBatcher class batches log lines (20 lines or 2s interval) into single heartbeat messages (~0.5 RPC/sec max)
• [x] Server-driven cancellation: if heartbeat response signals cancel_requested=true, container is stopped gracefully and result includes "cancelled": true
• [x] Added JobHeartbeatResponse model to src/api/models.py for structured heartbeat responses
• [x] Enhanced send_job_heartbeat in src/api/server_client.py to parse structured response — returns JobHeartbeatResponse
• [x] Added stop_container method to DockerContainerManager in src/utils/docker.py
• [x] Added CONTAINER_STREAM capability to EngineCapability enum
• [x] Extracted _prepare_container_job() helper in executor — shared by both container job types
• [x] Updated _get_processing_engine_for_model() to disambiguate by job_type when multiple engines register the same model_id
• [x] Registered ContainerStreamEngine in src/core/client.py alongside DockerEngine
• [x] Updated README.md with container_stream documentation
• Server-side recommendations (not yet implemented on hub):
• Heartbeat response should return {"acknowledged": true, "cancel_requested": false}
• New POST /api/v1/jobs/{job_id}/cancel endpoint for user-initiated cancellation
• Store batched log lines and expose GET /api/v1/jobs/{job_id}/logs?follow=true (SSE) for real-time tailing

• Tolerate ~0.5 heartbeat RPCs/second/job without rate limiting

Fix worker registration crash on context_length must be positive (2026-04-23) ✅

• [x] Root cause: TransformersEngine.list_models() appends placeholder ModelInfo entries for Hub-allowed models that have not been downloaded yet with context_length=0. WorkerClient.discover_and_register_models() calls model_info.to_capability() on every listed model, and ModelCapability validates context_length > 0 — so the unknown placeholder failed Pydantic validation, aborting the entire discover_models call and preventing the worker from registering with the server.
• [x] Fixed at src/engines/transformers_engine.py:424 by using context_length=4096 as the placeholder for undownloaded Hub-allowed models, matching the "unknown" default used elsewhere (ollama.py:146, ollama.py:630, openai_compat.py:181). The real value is written back on load via _config_to_model_info.
• [x] Verified with tests/test_transformers_engine.py (48 passed) and tests/test_api_models.py + tests/test_models.py (38 passed).

Fix string-"false" truthiness bug in env-var-driven .enabled flags (2026-04-22) ✅

• [x] Root cause: WorkerConfig._resolve_env_vars returns env-var values (and their :-default fallbacks) as strings. So ${DOCKER_ENGINE_ENABLED:-false} resolves to the literal string "false", which Python considers truthy — causing Docker/Docling/Surya engines to load even when the intent was "disabled". Verified with a direct call to _resolve_env_vars.
• [x] Added WorkerConfig.get_bool(key, default) in src/core/config.py that coerces the common string forms (true/false/1/0/yes/no/on/off plus empty string) to Python bool. Real bool/int values pass through unchanged.
• [x] Updated the five .enabled call-sites in src/core/client.py to use get_bool: docling, surya, docker (processing engines) plus model_downloads.enabled and auto_update.enabled.
• [x] Policy change: docling and surya are now opt-in. YAML defaults in config/default.yaml and config/default_mini.yaml flipped from :-true to :-false. Owner must set DOCLING_ENABLED=true / SURYA_ENABLED=true (or override in config) to run them.

Fix retry-after-partial-init crash 'NoneType' object has no attribute '_models' (2026-04-22) ✅

• [x] Root cause: WorkerClient.start() decided whether to re-run initialize_components() by checking self.server_client is None. server_client is assigned very early in initialize_components() (before engines, model_registry, etc.), so if any later step raised, server_client was set but self.model_registry stayed None. The retry loop in run_forever then re-entered start(), skipped initialization, and register() → discover_and_register_models() crashed on self.model_registry._models.clear().
• [x] Added self._components_initialized flag in src/core/client.py; start() now gates on that flag instead of server_client. Flag is set to True only after initialize_components() completes successfully.
• [x] On init failure, partial state (server_client, engine, _loaded_engines, model_registry, model_loader, job/monitoring components) is reset so the next retry re-initializes cleanly.
• [x] Trigger identified: the worker was installed without the optional ollama-python extra, but src/core/client.py did an eager top-level from ..engines.ollama import OllamaEngine — module import failed before any engine fallback could kick in. Moved the Ollama import to be lazy inside _create_engine (matching the existing pattern for openai_compat and transformers). The surrounding try/except ImportError now catches the missing package and returns None cleanly, so init proceeds with zero engines.
• [x] Changed default for engine.available from ["ollama"] to [] in src/core/client.py and src/core/config.py (both the validator fallback and get_config_summary). Workers now come up with no inference engines unless the owner opts in via config or MICRODC_ENGINES env var.

openai_compat logs include engine/instance (2026-04-15) ✅

• [x] Prefixed log lines in src/engines/openai_compat.py with [openai_compat:<instance>] (e.g., [openai_compat:groq]) so init, model verify/untrack, generate response, warnings, and param debug lines show which named instance is active
• [x] Uses existing get_platform_name() — falls back to openai_compat for the default (unnamed) instance
• [x] Added per-call req=<8-hex> correlation id + start/done timing in generate() so concurrent requests (e.g., concurrency: 25) can be distinguished in logs
• [x] Moved the correlation-id/timing wrapper into InferenceEngine.generate() (base class) via template-method pattern; each engine now implements _generate_impl(). All inference engines (ollama, transformers, openai_compat) inherit consistent request logging
• [x] Updated mock engine in tests/test_models.py to implement _generate_impl
• [x] Overrode get_platform_name() in OllamaEngine and TransformersEngine so log tags read [ollama req=…] / [transformers req=…] instead of the generic [inference req=…]

Capacity-aware routing across API-compatible engines (2026-04-15) ✅

• [x] Added WorkerClient.get_compatible_engines(platform, model_id) — returns ordered list of loaded engines that can serve the pair. openai_compat:* engines whose _model_aliases contain the model_id are compatible with platform: ollama
• [x] JobExecutor now takes a compat_resolver and picks the first compatible engine whose semaphore is not saturated at dispatch time (_choose_engine_with_capacity). Falls back to the native platform if all are busy
• [x] Per-job engine assignment stored in _job_engine_assignment so get_engine_for_job returns the engine chosen at dispatch (not re-resolving from platform)
• [x] Result: when local ollama (concurrency 1) is busy, jobs for models aliased in openai_compat:groq (concurrency 25) overflow to Groq automatically

max_context_tier default removed (2026-04-15) ✅

• [x] max_context_tier no longer defaults to 1 when MAX_CONTEXT_TIER env is unset
• [x] Field default changed to None in src/api/models.py; heartbeat uses exclude_none=True so it is omitted from the payload when unset
• [x] Updated YAML configs to ${MAX_CONTEXT_TIER:-} and client to coerce empty/None properly
• [x] Tests updated in tests/test_api_models.py and tests/test_config.py

Config Documentation (2026-04-14)

MAX_CONTEXT_TIER documented in YAML examples ✅

• [x] Added inline tier-definition comments above max_context_tier in config/default.yaml, config/default_example.yaml, config/default_mini.yaml
• [x] Tiers: 1=Small (0–1,500 chars), 2=Medium (1,501–6,000), 3=Large (6,001–24,000), 4=Extra-large (24,001+)
• [x] Noted routing behavior: server only routes a job to workers whose tier >= the job's tier

Test Warning Cleanup (2026-04-14) ✅

• [x] Migrated src/api/models.py from Pydantic V1 (@validator, class Config) to V2 (@field_validator, @model_validator, ConfigDict)
• [x] Replaced .dict() with .model_dump() in src/jobs/executor.py
• [x] Replaced module-level pytestmark = pytest.mark.asyncio with per-test decorators in tests/test_openai_compat.py
• [x] Replaced all datetime.utcnow() with datetime.now(timezone.utc) across src/ and tests/ (Python 3.12+ deprecation)
• [x] Result: 270 passed, 0 warnings (previously 138 warnings)

Encrypted Job Support (2026-04-02)

Capabilities Config ✅

• [x] Added worker.capabilities key to YAML config (default.yaml, default_mini.yaml, default_example.yaml)
• [x] Added capabilities field to WorkerHeartbeat model in src/api/models.py
• [x] Wired capabilities from config into heartbeat loop in src/core/client.py
• [x] Added WORKER_CAPABILITIES environment variable support
• [x] Added default in _get_defaults() in src/core/config.py

Encrypted Job Processing ✅

• [x] Implement decrypt_payload() in src/utils/encryption.py
• [x] Implement encrypt_result() with hybrid RSA-OAEP + AES-256-GCM in src/utils/encryption.py
• [x] Add cryptography>=41.0.0 to requirements.txt
• [x] Handle is_encrypted flag in job claim response (src/core/client.py)
• [x] Decrypt payload before building Job object — extracts prompt/params from decrypted data
• [x] Added is_encrypted and encryption_metadata fields to Job model (src/api/models.py)
• [x] Submit encrypted results with encrypted_output + result_iv (src/api/server_client.py)
• [x] Report input_tokens/output_tokens for encrypted jobs
• [x] Improved error message in executor when encrypted job has no input_data
• [ ] Add unit tests for encryption/decryption round-trip
• [ ] Security audit: ensure decrypted payloads and keys are never logged or persisted

CI/CD & Infrastructure ✅ (2026-02-13)

Dockerfile ✅

• [x] Created multi-stage Dockerfile based on nvidia/cuda:12.2.2-runtime-ubuntu22.04
• [x] Builder stage installs Python 3.11 and pip dependencies in a venv
• [x] Runtime stage copies only the venv and source code for smaller image
• [x] Configuration via environment variables (MICRODC_API_KEY, MICRODC_SERVER_URL, etc.)
• [x] Entrypoint set to python -m src.core.cli
• [x] Added .dockerignore to exclude .git, venv, tests, docs, caches, models

GitLab CI Pipeline ✅

• [x] Created .gitlab-ci.yml with 4 stages: test, lint, build, deploy
• [x] unit-tests job: runs pytest on Python 3.11
• [x] lint job: runs black --check, isort --check, ruff check
• [x] docker-build job: builds and pushes image to GitLab Container Registry (main only)
• [x] pages job: builds MkDocs site and deploys to GitLab Pages (main only)
• [x] All jobs tagged with aisrv05-docker
• [x] Pip cache configured for faster CI runs

MkDocs Documentation Site ✅

• [x] Created mkdocs.yml with Material theme (dark/light toggle)
• [x] Created requirements-docs.txt (mkdocs + mkdocs-material)
• [x] Created docs/index.md landing page adapted from README
• [x] Nav structure covers all existing docs (setup, architecture, engines, API, etc.)
• [x] GitLab Pages deployment configured in CI pipeline

New Features ✅ (2025-11-21)

Multimodal File Support ✅

• [x] Added attached_files field to Job model (models.py:284-287)
• [x] Implemented file download method in ServerClient (server_client.py:978-1010)
• [x] Updated client.py to pass attached_files to Job objects (client.py:1012)
• [x] Added file download and base64 encoding in JobExecutor (executor.py:333-377)
• [x] Updated OllamaEngineV2.generate() to accept images parameter (ollama_v2.py:329, 365-366, 383, 404)
• [x] Added logging for attached files in job details (client.py:1050-1053)
• [x] Support for vision models (qwen2.5vl:7b, llava, bakllava, etc.)
• [x] Support for document processing models (docling)
• [x] Automatic base64 encoding for image files
• [x] Document file handling for processing engines (executor.py:404-432)
• [x] Job routing based on job_type first, not model_id (executor.py:384-396)
• [x] Case-insensitive job_type handling (executor.py:164, 380)
• [x] Relaxed input_data validation for document jobs with attached files (executor.py:173-178)
• [x] Graceful handling of unsupported file types

Bug Fixes ✅ (2025-11-21)

Configuration Type Casting Fixes ✅

• [x] Fixed multiple "'<' not supported between instances of 'int' and 'str'" errors
• [x] Root cause: Environment variable substitution in YAML returns strings, not ints
• [x] Fixed job queue max_size comparison (client.py:287)
• [x] Fixed docling max_file_size_mb comparison (docling_engine.py:51)
• [x] Fixed docling timeout_seconds (docling_engine.py:52)
• [x] Enhanced priority validator to handle None values (models.py:289)
• [x] Added explicit int() cast for all numeric priority values (models.py:299)
• [x] Changed priority default from string "normal" to integer 5 (client.py:971)

Ollama API Parameter Handling Fix ✅

• [x] Fixed TypeError when passing None values for optional parameters
• [x] Changed from passing images=None to conditionally including images parameter
• [x] Build parameter dictionaries dynamically (ollama_v2.py:378-415)
• [x] Only include options/images when they have actual values

Embed Job Payload Compatibility Fix ✅

• [x] Fixed embed job execution to accept both texts and input field formats
• [x] Server sends input field in payload, worker expected texts
• [x] Updated executor.py:367 to try both fields with fallback
• [x] Maintains compatibility with both payload formats

Test Suite CI/CD Compatibility Fix ✅

• [x] Fixed test failures in automation/CI environments (test_config.py)
• [x] Tests were failing due to MICRODC_API_KEY environment variable overriding config
• [x] Added @patch.dict decorator to isolate test environment (test_config.py:30, 58)
• [x] Clear MICRODC_* environment variables at start of affected tests
• [x] Both failing tests now pass consistently in automation environments
• [x] All 115 tests now pass reliably in CI/CD pipelines

Heartbeat Model Reporting Enhancement ✅

• [x] Added processing engine models (e.g., docling) to heartbeat current_models field
• [x] Worker now reports both inference engine models (Ollama) and processing engine models
• [x] Added timeout protection (1.0s) for processing engine model listing
• [x] Enhanced debug logging to distinguish between inference and processing engines
• [x] Server now receives complete model inventory including docling
• [x] Added get_platform_name() method to Engine base class
• [x] DoclingEngine reports platform as "docling" (not "ollama")
• [x] Each engine reports its own platform name for accurate server tracking
• [x] Updated run.sh and run_prod.sh to display processing engines in configuration output

Document Processing Integration ✅ (2025-11-21)

Docling Integration ✅

• [x] Added Docling dependency to requirements.txt
• [x] Created DocumentProcessor class in src/processors/
• [x] Implemented support for PDF, DOCX, PPTX, XLSX, HTML, images, audio
• [x] Added URL, base64, and path input handling
• [x] Implemented multiple output formats (markdown, HTML, JSON, doctags)
• [x] Added OCR and table extraction capabilities
• [x] Implemented file size limits and timeout handling
• [x] Added comprehensive error handling and validation

Surya OCR Integration ✅ (2025-12-09)

• [x] Created SuryaProcessor class in src/processors/surya_processor.py
• [x] Created SuryaEngine class in src/engines/surya_engine.py
• [x] Implemented support for PDF and image files (PNG, JPEG, TIFF, BMP, WEBP)
• [x] Added OCR text recognition with 90+ language support
• [x] Implemented layout detection feature
• [x] Implemented table recognition feature
• [x] Added URL, base64, and path input handling
• [x] Implemented multiple output formats (markdown, JSON, text)
• [x] Added file size limits and timeout handling
• [x] Added comprehensive error handling and validation
• [x] Added configuration to default.yaml
• [x] Registered engine in client.py
• [x] Updated job executor to route by model_id (not just job_type)
• [x] Auto-detect device (cuda/mps/cpu) for optimal performance
• [x] Updated to new Surya API (Predictor classes)
• [x] Updated documentation (README.md, FUTURE_ENGINES.md, TODO.md)

Engine Architecture Refactoring ✅

• [x] Created Engine base class and EngineType/EngineCapability enums
• [x] Created ProcessingEngine abstract class for non-inference engines
• [x] Refactored InferenceEngine to inherit from Engine base
• [x] Implemented DoclingEngine as ProcessingEngine
• [x] Added model registration system for processing engines
• [x] Implemented model-based routing (jobs route by model_id, not job_type)
• [x] Added lazy model lookup for processing engines
• [x] Updated JobExecutor to support List[Engine] for processing engines
• [x] Implemented _get_processing_engine_for_model() helper method
• [x] Fixed event loop handling for async model registration
• [x] Updated all tests for model-based routing (17/17 passing)
• [x] Updated WorkerClient to pass processing engines as list
• [x] Fixed capability registration to iterate over list

Job Routing Changes ✅

• [x] Changed from job_type-based routing to model_id-based routing
• [x] Jobs now specify model_id="docling" to route to DoclingEngine
• [x] Supports multiple engines of same type (e.g., model_id="tesseract" vs "easyocr")
• [x] All jobs now require model_id for routing
• [x] Processing engine models skip model loading (always available)
• [x] Inference engine models follow normal load/unload lifecycle

Configuration ✅

• [x] Added engine.processing.docling section to config/default.yaml
• [x] Moved configuration under engine section for consistency
• [x] Implemented environment variable support for all engine settings
• [x] Added configurable max file size, timeout
• [x] Implemented temp directory configuration
• [x] Updated WorkerClient to read from engine.processing.docling path
• [x] Cleaned up default.yaml to remove unimplemented engines (vLLM, Transformers)
• [x] Created docs/FUTURE_ENGINES.md for planned engine configurations
• [x] Updated README.md to reference FUTURE_ENGINES.md
• [x] Removed unused config sections (performance, security, development)
• [x] Removed unused config values (metrics_interval, model loading strategy)
• [x] Simplified config to only show what's actually implemented

Testing ✅

• [x] Created comprehensive test suite (tests/test_document_processor.py)
• [x] Unit tests for DocumentProcessor class
• [x] Integration tests for JobExecutor document handling
• [x] Tests for error conditions and edge cases
• [x] Tests for JobParameters document field validation

Documentation ✅

• [x] Created DOCUMENT_PROCESSING_INTEGRATION.md guide
• [x] Documented server-side changes needed
• [x] Documented Python client integration requirements
• [x] Updated README.md with document processing section
• [x] Added document processing to feature list
• [x] Updated project structure diagram

Production Readiness ✅ (2025-11-14)

Long-Running Job Heartbeat Fix ✅

• [x] Fixed heartbeat blocking during long-running jobs
• [x] Added 5s timeout to engine.list_models() in heartbeat loop
• [x] Added 3s timeout to system metrics collection with asyncio.to_thread()
• [x] Added 15s timeout to heartbeat send operation
• [x] Implemented fallback to cached model registry on timeout
• [x] Heartbeats now guaranteed to send every 30s even during 10min+ jobs
• [x] Job progress included in heartbeat to reset server-side timeout

Code Quality and Standards ✅

• [x] Fixed all ruff linting errors (14 errors resolved)
• [x] Updated pyproject.toml to modern ruff configuration format
• [x] Replaced all bare except clauses with specific Exception handling
• [x] Fixed unused variable warnings
• [x] All 115 tests passing (updated from 98 after multimodal/document features)
• [x] Set up Black for code formatting (v24.10.0) - available in dev extras
• [x] Set up Ruff for linting (v0.9.1) - configured in pyproject.toml
• [x] Set up MyPy for type checking (v1.14.0) - available in dev extras
• [x] Configure pre-commit hooks (2025-11-23)
• [x] Created .pre-commit-config.yaml with 10 hooks
• [x] Configured Black (line-length=100)
• [x] Configured isort (profile=black)
• [x] Configured Ruff (auto-fix enabled, excludes notebooks)
• [x] Configured mypy (excludes tests/examples/notebooks)
• [x] Configured bandit for security checks
• [x] Configured markdownlint
• [x] Configured YAML formatter (pretty-format-yaml)
• [x] Added python-safety-dependencies-check
• [x] Configured general file checks (trailing-whitespace, end-of-file-fixer, etc.)
• [x] Install pre-commit: pip install pre-commit && pre-commit install
• [x] Run hooks: pre-commit run --all-files
• [x] Fixed all Bandit security issues (2025-11-24)
• [x] Fixed 2 high-severity tarfile extraction vulnerabilities (B202)
• [x] Replaced unsafe os.system() calls with subprocess.run()
• [x] Replaced 13 empty except:pass blocks with proper error handling
• [x] Replaced 4 assert statements with RuntimeError exceptions
• [x] Added #nosec comments for legitimate subprocess usage
• [x] All security checks now pass with 0 issues
• [ ] Add type hints to all functions (in progress - core modules have hints)
• [ ] Achieve >90% test coverage (currently 37% overall - 4547 statements, 2880 uncovered)
• High coverage modules: api/models.py (98%), core/exceptions.py (100%), core/config.py (78%)
• Medium coverage: engines/base.py (72%), jobs/monitor.py (87%), models/registry.py (72%)
• Low coverage: cli.py (0%), client.py (0%), server_client.py (10%) - mostly integration code
• Focus areas: Increase coverage for job executor (47%), ollama engines, document processor (74%)

Documentation and File Management ✅

• [x] Added docs/AUTHENTICATION.md to version control
• [x] Added docs/CODE_QUALITY.md for development tools and standards (2025-11-23)
• [x] Created CONTRIBUTING.md with contributor guidelines and workflow (2025-11-23)
• [x] Removed deprecated documentation files (SETUP.md, WORKER_CHANGE_REQUEST.md)
• [x] Removed unused example_usage.py file
• [x] Moved test_max_tokens.py to tests/ directory with proper pytest decorator
• [x] Updated README.md with code quality tools section and contributor guide

TODO Comments Resolution ✅

• [x] Replaced all TODO comments with explanatory notes
• [x] Documented that log rotation is handled by systemd
• [x] Noted signature verification is future enhancement (checksums provide integrity)
• [x] Clarified credential validation handled by ServerClient
• [x] Updated daemon/log viewing commands to reference systemd

Project Hygiene ✅

• [x] Enhanced .gitignore with comprehensive Python patterns
• [x] Cleaned all build artifacts (pycache, *.pyc, .DS_Store)
• [x] All changes staged and ready for commit

Phase 1: Core Implementation 🚀 ✅

Project Setup ✅

• [x] Set up project structure and core directories
• [x] Create requirements.txt and setup.py
• [x] Create default configuration file (config/default.yaml)

Core Components ✅

• [x] Implement core configuration management (config.py)
• [x] Create Pydantic models for API communication
• [x] Implement abstract InferenceEngine base class
• [x] Build Ollama engine integration

Server Communication ✅

• [x] Create server API client for communication
• [x] Implement worker registration flow
• [x] Build model discovery and capability reporting
• [x] Implement heartbeat mechanism

Job Processing ✅

• [x] Create job executor for processing inference jobs
• [x] Build job queue management system
• [x] Implement system resource monitoring utilities

Client Orchestration ✅

• [x] Create main client orchestrator
• [x] Add CLI interface with commands
• [x] Set up logging and health checks

Phase 2: Testing & Robustness 🛡️

Testing

• [x] Write unit tests for core components
• [x] Config tests (tests/test_config.py)
• [x] Model management tests (tests/test_models.py)
• [x] Job queue tests (tests/test_jobs.py)
• [x] System monitor tests (tests/test_system.py)
• [x] API model validation tests (tests/test_api_models.py)
• [ ] Test end-to-end worker registration and job execution
• [ ] Integration tests with mock server

Error Handling ✅

• [x] Add error handling and retry logic
• [x] Implement graceful shutdown
• [x] Add resource limits enforcement

GPU Support ✅

• [x] NVIDIA GPU detection via nvidia-ml-py
• [x] Apple Metal Performance Shaders (MPS) detection (requires PyTorch)
• [x] Unified memory tracking for Apple Silicon
• [x] GPU capability reporting in system info
• [x] PyTorch dependency added to GPU extras for MPS support

Development Tools ✅

• [x] Development run script (tools/run.sh)
• [x] Test script with interactive menu (tools/test_worker.sh)
• [x] Makefile for common tasks
• [x] Watch mode for auto-restart
• [x] Interactive shell mode for debugging

Phase 3: Additional Features (Future) 🔮

Engine Support

• [x] OpenAI-compatible engine (2026-03-05)
• [x] Created OpenAICompatEngine class in src/engines/openai_compat.py
• [x] Works with any OpenAI API-compatible server (Ubuntu inference snaps, vLLM, LocalAI, LiteLLM)
• [x] Configurable base_url for any endpoint (e.g., http://192.168.2.50:8326/v3)
• [x] Chat completions, streaming, and embeddings support
• [x] Multimodal/vision support via base64 image content parts
• [x] Parameter mapping (temperature, max_tokens, top_p, stop, top_k via extra_body)
• [x] Model type auto-detection (chat, embedding, image, multimodal)
• [x] No API key required for local servers (placeholder used)
• [x] Registered in engine factory (src/core/client.py)
• [x] Configuration in config/default.yaml
• [x] Added openai>=1.0.0 dependency
• [x] Model aliases for provider-specific naming (e.g., Groq's openai/gpt-oss-20b)
• [x] Per-engine concurrency control (configurable via concurrency: per engine)
• [x] Immediate job pickup after completion via asyncio.Event
• [x] Graceful shutdown job cleanup — releases queued jobs and fails active jobs on SIGTERM/SIGINT
• [x] Comprehensive test suite (tests/test_openai_compat.py - 66 tests)
• [x] Updated README.md, docs/engines/openai_compat.md, and docs/TODO.md
• [ ] vLLM integration
• [x] HuggingFace Transformers integration (2025-12-15)
• [x] Created TransformersEngine class inheriting from InferenceEngine
• [x] Support for text generation, embeddings, and multimodal models
• [x] Dynamic VRAM management with LRU model eviction
• [x] bitsandbytes 4-bit/8-bit quantization support
• [x] HuggingFace Hub download with configurable allowlist/blocklist
• [x] Case-insensitive allowlist matching
• [x] On-demand model download for allowlisted Hub models
• [x] Streaming text generation via TextIteratorStreamer
• [x] Auto device selection (CUDA, MPS, CPU)
• [x] Job platform routing via platform field
• [x] 47 unit tests with 66% coverage
• [ ] Custom engine plugin system

Docker Container Execution Engine (2026-04-02)

• [x] Added CONTAINER EngineType and CONTAINER_EXECUTION EngineCapability to engine_base.py
• [x] Created DockerContainerManager utility class (src/utils/docker.py)
• Async wrapper around Docker SDK with asyncio.to_thread() for all sync calls
• Image allowlist enforcement using fnmatch patterns
• Network mode allowlist enforcement
• Container lifecycle: pull, run, wait, stream logs, collect output, remove
• GPU passthrough via nvidia-container-toolkit (--gpus flag)
• Container labeling for crash recovery cleanup
• [x] Created DockerEngine as ProcessingEngine subclass (src/engines/docker_engine.py)
• MODEL_ID = "docker", job_type = "container"
• DEFAULT_CONCURRENCY = 1 for exclusive resource model
• Supports both short-lived tasks and long-running containers
• Short-lived: wait for exit, capture stdout/stderr
• Long-running: stream logs via job progress heartbeat while waiting
• Input data: image, command, environment, volumes, GPU, network mode, resource limits
• Output: exit_code, stdout, stderr, output_files from /output mount, runtime_seconds
• Attached files downloaded and bind-mounted as /input
• Image pull with timeout protection
• Security: image allowlist (deny-all default), network allowlist (default: none), no --privileged
• Boolean config values properly parsed from string env vars ("true"/"false")
• [x] Added "container" job type routing in executor.py
• Downloads attached files to configurable DOCKER_OUTPUT_DIR/{job-id}/input/
• Container output goes to DOCKER_OUTPUT_DIR/{job-id}/output/
• Skips generic multimodal file handler for container jobs (prevents double download)
• Injects heartbeat callback for log streaming
• Injects job metadata for container tracking
• [x] Added container job parsing in polling loop (client.py)
• Container jobs pass entire payload as input_data (not just prompt/input field)
• Server's model field (image name) mapped to model_id="docker" for engine routing
• [x] Wired DockerEngine into WorkerClient initialization (client.py)
• [x] Added capabilities field to WorkerHeartbeat model (models.py)
• Auto-detects and reports: gpu, docker, encryption
• Server uses capabilities to route container jobs to Docker-capable workers
• [x] Added Docker configuration defaults to config.py
• engine.processing.docker.enabled (default: false)
• allowed_images, allowed_network_modes, default_network_mode
• output_dir, max_runtime_seconds, gpu_enabled, pull_timeout_seconds
• DOCKER_ENGINE_ENABLED env var mapping
• [x] Added docker>=7.0.0 as optional dependency in pyproject.toml
• [x] Added Docker config to all YAML config files (default, default_mini, default_example)
• [x] Tested end-to-end: PyTorch CPU inference in container with attached script file

Advanced Features

• [x] Auto-Update System (2025-09-23)
• [x] Version checking against server requirements
• [x] Automatic download and installation of updates
• [x] Graceful shutdown before updates
• [x] Rollback capability on update failure
• [x] Maintenance window support
• [x] Platform-specific update scripts (Linux/macOS/Windows)
• [x] Model auto-pulling on demand (Server-Initiated Model Downloads)
• [x] Added PendingDownloadRequest, DownloadResponseType Pydantic models
• [x] Added model_downloads config section with enable toggle, allowlists, hardware thresholds
• [x] Created ModelDownloadManager for download orchestration with hardware validation
• [x] Integrated with heartbeat response pending_download_requests field
• [x] API endpoints: respond, progress, complete (GET/POST /api/v1/workers/download-requests/)
• [x] Hardware compatibility checking (estimated_size_gb, required_vram_gb, required_ram_gb)
• [x] Platform-specific allowlist/blocklist support with wildcard patterns
• [x] Automatic model existence check before downloading
• [x] Resume detection via GET /api/v1/workers/active-downloads on startup
• [ ] Multi-GPU support
• [ ] Job prioritization (basic priority support already implemented)
• [ ] Result caching
• [ ] Distributed inference support

Code Cleanup & Refactoring ✅

• [x] Resolve duplicate class definitions (GPUInfo, CPUInfo classes are correctly separated - internal dataclasses with to_api_model converters)
• [x] Remove unused exception classes (removed ConfigurationError)
• [x] Fix placeholder code (removed "not yet implemented" message in logging.py)
• [x] Fix linting issues (resolved unused variables and bare except clauses)
• [x] Enhanced error messages for missing API key configuration
• [x] Add parameter tracking for max_tokens → num_predict conversion
• [x] Create test utilities for parameter verification
• [ ] Review and remove/document unused API models (kept as they define server API contract)
• [ ] Clean up unused system utility functions (many are CLI entry points or public APIs)
• [ ] Add tests for public APIs to prevent accidental removal
• [ ] Document which "unused" functions are actually public APIs or future hooks

Documentation

• [ ] API documentation
• [x] Deployment guide
• [x] Created docs/setup/UBUNTU_SETUP.md - comprehensive Ubuntu/systemd installation guide
• [x] Created docs/setup/WINDOWS_SETUP.md - Windows installation guide with service options
• [ ] Configuration reference
• [ ] Troubleshooting guide

Operations

• [ ] Prometheus metrics export
• [ ] OpenTelemetry tracing
• [ ] Performance profiling
• [ ] Auto-scaling support

Current Status 📊

Completed ✅

• All core infrastructure and main components
• Ollama engine integration with full API support
• Complete server communication layer with retry logic
• Comprehensive resource monitoring (CPU, GPU, Memory, Storage)
• Model lifecycle management with loading strategies
• Job processing pipeline with queue management
• Rich CLI interface with all essential commands
• Error handling with custom exceptions
• Unit test suite covering core components
• Claim-based job assignment system implementation
• Bearer token authentication with credential persistence
• Automatic credential reuse on worker restart
• Support for both "input" and "prompt" fields in job payload
• Proper handling of required tokens_used field in job completion
• Race condition mitigation with completion delay
• Detailed server response logging for debugging
• Updated to match new server API (removed capabilities field, uses normalized tables)
• Fixed heartbeat format to include status object and supported_models list
• Enhanced error messaging for missing API key configuration
• Parameter tracking and logging for max_tokens → num_predict conversion
• Code cleanup: removed unused code, fixed linting issues
• Test utilities for parameter verification
• Multimodal support: Job model and client support for llm_interaction_type and modalities
• Model platform tracking: Reports inference platform (ollama, vLLM, etc.) for each model in heartbeat (registration simplified to not include models)
• Resilient server communication: Worker survives temporary server outages with automatic retry and graceful recovery

In Progress 🚧

• Integration testing with real MicroHub server
• Performance optimization for high-throughput scenarios

Recently Completed ✅

• HuggingFace Transformers Engine (2025-12-15)
• Created TransformersEngine class in src/engines/transformers_engine.py (~1000 lines)
• Support for text generation models (CausalLM, Seq2SeqLM)
• Support for embedding models (BERT, RoBERTa, sentence-transformers)
• Support for multimodal models (LLaVA, Qwen-VL)
• Dynamic VRAM management with LRU eviction for multiple loaded models
• bitsandbytes 4-bit/8-bit quantization support
• HuggingFace Hub download with configurable allowlist/blocklist
• Case-insensitive allowlist matching for model names
• On-demand model download for allowlisted Hub models
• Streaming text generation via TextIteratorStreamer
• Auto device selection (CUDA, MPS, CPU)
• Job routing via platform field in job data
• 47 unit tests with 66% coverage
• Configuration in config/default.yaml
• Integration in src/core/client.py
• Documentation in docs/engines/transformers.md
• Silenced noisy third-party loggers (urllib3, filelock, etc.)

• Removed deprecated engine.type config (use engine.available list)

• Fixed ReadTimeout Error for Long-Running Generations (2025-11-07)

• Fixed timeout handling in OllamaEngineV2 to prevent ReadTimeout errors
• ollama_v2.py:10: Added httpx import for granular timeout configuration
• ollama_v2.py:56-61: Configured httpx.Timeout with 120s read timeout between chunks
• ollama_v2.py:318-417: Refactored generate() to use internal streaming for batch jobs
• Batch jobs now internally stream from Ollama while returning complete results
• Set reasonable read timeout (120s between chunks) that detects actual stalls
• No need for artificial unlimited timeouts - as long as tokens flow, job continues
• Better error detection - quickly identifies when Ollama actually stalls
• Maintained connection timeout (30s), write timeout (30s), pool timeout (10s)
• Fixes "Generation failed: ReadTimeout:" errors on large models (e.g., deepseek-r1:70b)
• Jobs no longer fail prematurely during extended generation tasks

• Worker can now handle models that require unlimited generation time

• Automatic Version Management System (2025-11-04)

• Implemented fully automated version bumping on every git commit
• tools/bump_version.py: Python utility for version incrementing
• tools/git-hooks/pre-commit: Git hook for automatic PATCH version bump
• tools/install-git-hooks.sh: One-time setup script for git hooks
• version.py:15: Updated to version 0.1.0 (initial development)
• docs/VERSIONING.md: Comprehensive guide with automatic versioning instructions
• README.md:568-599: Added automatic version management section
• Every commit now auto-increments PATCH version (0.1.0 → 0.1.1)
• Manual bumps available for MINOR/MAJOR versions
• Zero-configuration after one-time hook installation

• Established 0.x.x for development, 1.0.0 for first production release

• Worker Version Tracking (2025-11-04)

• Added worker_version field to WorkerHeartbeat model
• models.py:368: Added worker_version as optional string field
• client.py:684: Heartbeat now includes version from version.py
• README.md:271: Updated heartbeat format example to include worker_version
• Worker version now reported in every heartbeat for tracking and debugging
• Server can track which worker versions are deployed in the fleet

• Enables version-specific debugging and compatibility checks

• Embed Job Execution Fix (2025-11-04)

• Fixed embed job execution to properly handle payload with "texts" field
• client.py:795: Extract job_type early to use in payload parsing logic
• client.py:805-807: Added special handling for job_type="embed" to preserve entire payload
• executor.py:89-91: Updated validation to accept dict input_data for embed/chat jobs
• executor.py:103-105: Store job_type in result metadata for proper output formatting
• executor.py:240-253: Extract "texts" field from payload dict for embedding generation
• server_client.py:1012-1026: Format output based on job_type (embeddings vs text)
• Fixes error: "Job has no input_data/prompt" when executing embed jobs
• Fixes output format: embeddings now properly structured as {"embeddings": [...], "finish_reason": "stop"}

• Worker now fully supports embedding job execution with proper payload parsing and result formatting

• Embedding Model Support Fix (2025-10-26)

• Fixed load_model() method in both Ollama engines to handle embedding models
• ollama.py:249: Added embedding model detection and proper test method
• ollama_v2.py:261: Added embedding model detection and proper test method
• Embedding models (containing "embed" in name) now tested with generate_embeddings()
• Non-embedding models continue to use generate() test
• Fixes error: "does not support generate" when loading embedding models

• Worker now fully supports embedding models like qwen3-embedding:8b, nomic-embed-text, etc.

• Test Suite Fixes (2025-10-14)

• Fixed all 6 failing tests after resilient server communication implementation
• test_worker_registration: Updated for new WorkerRegistration schema without supported_models
• test_required_fields: Changed to test ModelCapability instead of Job model
• test_download_update_success: Fixed async context manager mock for streaming
• test_periodic_update_check: Fixed async task cancellation pattern
• test_default_config_loading: Fixed config path and expected values
• test_nested_config_access: Added handling for string vs int config values

• All 97 tests now passing successfully

• Resilient Server Communication (2025-10-14)

• Worker now handles server unavailability gracefully without crashing
• Registration retry: Added automatic retry with exponential backoff for network errors
• Continuous operation: Heartbeat and job polling loops continue even when server is down
• Server availability tracking: New _server_unavailable_since and _last_server_success tracking
• Helper methods: Added _mark_server_success(), _mark_server_failure(), _check_server_unavailability()
• Graceful shutdown: Worker exits gracefully after configurable max unavailability (default: 5 minutes)
• Server availability monitor: New background task monitors server status periodically
• Smart recovery: Automatically resumes when server comes back online with logging
• Configuration: Added server.retry.* and server.unavailable.* config options

• Workers now survive temporary server outages and network issues

• Model Platform Tracking (2025-10-13)

• Worker now reports which inference platform hosts each model
• Registration simplified: Removed supported_models field (server gets updates via heartbeat only)
• Heartbeat includes platform info in current_models field with model+platform objects
• Added ModelWithPlatform Pydantic model for structured platform data
• Platform auto-detection based on engine type configuration
• Support for 7 standard platforms plus custom platform type
• Helper method _models_to_platform_format() for converting model lists
• Backward compatible Union type supports both old and new formats
• Enables server-side platform-specific job routing and performance tracking

• Model updates are now fully dynamic via heartbeat system

• Runtime Token Refresh (2025-10-11)

• Fixed authentication failures during worker runtime (heartbeat, job polling, etc.)
• Enhanced _make_request() with automatic token refresh on 401 errors
• Added transparent request retry after successful token refresh
• Prevents infinite retry loops with retry_auth flag
• Enhanced error logging in heartbeat and job polling loops for authentication failures
• Workers now seamlessly handle token expiration during long-running sessions

• No service interruption when tokens expire - automatic recovery

• Configuration Loading Fix (2025-10-11)

• Fixed "Config file not found" warnings on production workers
• WorkerConfig now respects MICRODC_CONFIG environment variable
• Added config/worker.yaml to search paths (used by ubuntu_setup.sh)
• Proper configuration loading for systemd service installations

• Production workers now correctly load configuration from /srv/microdcworker/config/worker.yaml

• Automatic Token Refresh (2025-10-09)

• Fixed token expiration handling for long-running workers
• Added proper calculation of token expiration time from expires_in field
• Implemented automatic credential refresh using saved secret_key
• Added _refresh_credentials() method in ServerClient to handle token renewal
• Workers now save expires_at, refresh_token, and secret_key with credentials
• Enhanced error messages for expired bootstrap tokens with clear renewal instructions

• Workers can now run indefinitely without "Invalid or expired token" errors

• Multimodal Support (2025-10-04)

• Added llm_interaction_type field to Job model (generation vs chat)
• Added input_modalities and output_modalities fields to Job model
• Added job_type field to Job model
• Enhanced job executor logging with multimodal information
• Updated client job claim parsing to extract multimodal fields
• Enhanced job claim logging with modality and interaction type details
• Implemented routing in job executor based on interaction type
• Added _prepare_chat_messages() helper for chat format conversion
• Implemented embedding job support with JSON result formatting
• Handle chat messages in payload for proper message extraction

• Full compatibility with server WORKER_CHANGE_REQUEST.md specifications

• Auto-Update System (2025-09-23)

• Created WorkerAutoUpdater class with comprehensive update management
• Implemented version checking with server endpoints (/api/v1/version/)
• Added automatic download with progress tracking
• Created backup and rollback functionality
• Integrated graceful shutdown for running jobs
• Added maintenance window support for controlled updates
• Created platform-specific update scripts for Linux/macOS/Windows
• Implemented update configuration in default.yaml
• Added comprehensive test coverage for update functionality

• Integrated auto-updater with main worker client lifecycle

• Enhanced Heartbeat System (2025-09-22)

• Added SystemMetricsCollector class for comprehensive metrics collection
• Enhanced WorkerHeartbeat model with SystemMetrics field
• Implemented collection of: load average, CPU count, memory metrics, disk metrics, GPU metrics, network I/O, uptime
• Added temperature monitoring (CPU and GPU when available)
• Dynamic model reporting in current_models field
• Backward compatible with legacy heartbeat format

• Tested on macOS with Apple Silicon (MPS GPU support)

• Dynamic Model Reporting (2025-09-22)

• Heartbeat now fetches fresh model list from engine on each cycle
• Added refresh_models() method for silent registry updates
• Implemented model_refresh_loop() for periodic registry synchronization
• Model registry cleared and repopulated on refresh to prevent stale entries
• Fallback to cached registry if engine query fails
• Configurable refresh interval (default: 5 minutes)
• Successfully tested with 31 Ollama models

Next Steps 📋

1. Monitor and optimize job completion success rate
2. Add metrics collection for job processing performance
3. Implement adaptive retry strategies based on error types
4. Add vLLM inference engine support
5. Add Prometheus metrics export for monitoring
6. Implement distributed inference support for large models

Notes

Core Implementation ✅

• All Phase 1 core implementation completed successfully
• Ollama is fully integrated as the primary inference engine
• Async/await patterns used throughout for optimal concurrency
• Comprehensive error handling with retry logic implemented
• Structured logging with configurable outputs and detailed job logging
• Pluggable architecture ready for additional engines

API Compatibility Updates ✅

• Server Schema Changes: Worker updated to match new server API without capabilities field
• Normalized Tables: Hardware specs and supported models sent for server's normalized database storage
• Heartbeat Format: Correctly wraps status object and includes supported_models list
• Parameter Mapping: Properly converts generic parameters to engine-specific ones (e.g., max_tokens → num_predict)

Job Processing ✅

• Claim-based system: Atomic job claiming prevents race conditions
• Flexible payload handling: Supports both "input" and "prompt" fields from server
• Token usage: Always includes required tokens_used field in completions
• Race condition fix: 0.5s delay before result submission prevents "Assignment not active" errors
• Error reporting: All job failures properly reported to server with detailed error messages

Authentication & Security ✅

• Credential persistence: Saves credentials after registration for reuse
• Automatic token refresh: Transparently refreshes expired tokens using saved secret_key
• Token expiration handling: Properly calculates and tracks token expiration times
• Clean error handling: Worker exits cleanly on authentication failures
• Bearer token auth: Proper implementation of bearer token authentication
• Bootstrap token: Initial registration uses one-time bootstrap token
• Enhanced error messages: Clear guidance when bootstrap tokens expire

Testing & Quality

• Unit tests provide good coverage for individual components
• Ready for integration testing with actual MicroDC server
• Comprehensive logging for debugging and monitoring