Skip to content

Worker Authentication Flow

Overview

The MicroDC worker uses a multi-step authentication process to securely connect to the server. The authentication flow supports both initial registration with a bootstrap token and subsequent reconnections using saved credentials.

Authentication Steps

1. Initial Registration (First Time Setup)

When a worker starts for the first time, it needs to register with the server using a bootstrap token:

# Set the bootstrap token provided by the server admin
export MICRODC_API_KEY="mdc_wrk_xxxxx"

The registration process follows these steps:

  1. Check Saved Credentials: Worker first checks if valid credentials exist in ~/.microdc/worker_credentials.json
  2. Bootstrap Registration: If no saved credentials, worker registers using the bootstrap token
  3. Receive Credentials: Server returns node_id and secret_key
  4. Login: Worker uses node_id and secret_key to get a worker_token
  5. Save Credentials: Worker saves the worker_token for future use

2. Subsequent Connections (Using Saved Credentials)

After initial registration, the worker can reconnect without the bootstrap token:

  1. Load Saved Credentials: Worker loads credentials from ~/.microdc/worker_credentials.json
  2. Verify Credentials: Worker sends a test heartbeat to verify the token is still valid
  3. Use Existing Token: If valid, worker uses the saved worker_token for all API calls
  4. Re-register if Needed: If token is invalid/expired, worker falls back to bootstrap token

Authentication Flow Diagram

┌─────────────┐
│   Worker    │
│   Starts    │
└──────┬──────┘
       │
       ▼
┌──────────────────┐     ┌─────────────────┐
│ Check for Saved  │────►│ Credentials     │
│   Credentials    │  No │ Not Found       │
└────────┬─────────┘     └────────┬────────┘
         │                        │
         │ Yes, Found             │
         ▼                        ▼
┌──────────────────┐     ┌─────────────────┐
│ Test Heartbeat   │     │ Use Bootstrap   │
│ (Verify Token)   │     │     Token       │
└────────┬─────────┘     └────────┬────────┘
         │                        │
         │ Valid                  ▼
         │               ┌─────────────────┐
         │               │ Register with   │
         │               │     Server      │
         │               └────────┬────────┘
         │                        │
         │                        ▼
         │               ┌─────────────────┐
         │               │ Get node_id &   │
         │               │   secret_key    │
         │               └────────┬────────┘
         │                        │
         │                        ▼
         │               ┌─────────────────┐
         │               │  Login to Get   │
         │               │  worker_token   │
         │               └────────┬────────┘
         │                        │
         │                        ▼
         │               ┌─────────────────┐
         │               │ Save Credentials│
         │               │    to Disk      │
         │               └────────┬────────┘
         │                        │
         ▼                        ▼
┌─────────────────────────────────────────┐
│        Worker Authenticated              │
│   (Uses Bearer token for all API calls)  │
└──────────────────────────────────────────┘

Code Implementation

Registration Process (src/core/client.py)

async def register(self) -> None:
    # 1. Try saved credentials first
    if await self.server_client.try_load_credentials():
        self.worker_id = self.server_client.worker_id
        self.worker_token = self.server_client.auth_token
        self.registered = True
        return

    # 2. Get bootstrap token from config
    token = self.config.get("server.api_key")
    if not token:
        raise RegistrationError("No API key/token configured")

    # 3. Register with bootstrap token
    registration = WorkerRegistration(
        token=token,
        node_name=f"{hostname}-{worker_id}",
        hardware_specs=hardware_specs,
        supported_models=model_ids,
        # ... other fields
    )
    response = await self.server_client.register_worker(registration)

    # 4. Login with credentials from registration
    node_id = response.node_id
    secret_key = response.credentials["secret_key"]
    login_response = await self.server_client.login_worker(node_id, secret_key)

    # 5. Save worker token
    self.worker_token = login_response.worker_token
    self.server_client.credential_store.save_credentials(
        worker_id=node_id,
        auth_token=self.worker_token,
        server_url=server_url
    )

Server Communication (src/api/server_client.py)

class ServerClient:
    async def register_worker(self, registration: WorkerRegistration):
        # POST /api/v1/workers/register
        # Bootstrap token in request body
        response = await client.post(url, json=registration.model_dump())
        return WorkerRegistrationResponse.model_validate(response.json())

    async def login_worker(self, node_id: str, secret_key: str):
        # POST /api/v1/workers/auth/login
        # Returns worker_token for future API calls
        response = await client.post(url, json={
            "node_id": node_id,
            "secret_key": secret_key
        })
        return WorkerLoginResponse.model_validate(response.json())

    async def try_load_credentials(self) -> bool:
        # Load from ~/.microdc/worker_credentials.json
        credentials = self.credential_store.load_credentials(self.base_url)
        if credentials:
            self.auth_token = credentials["auth_token"]
            # Test with heartbeat
            if await self.send_simple_heartbeat(...):
                return True
        return False

Credential Storage

Credentials are stored in ~/.microdc/worker_credentials.json:

{
  "worker_id": "worker_afl3BfE1Xzok0fSGn7TsnQ",
  "auth_token": "wt_xxxxxxxxxxxxx",
  "server_url": "http://localhost:8000",
  "saved_at": "2025-09-23T10:30:00Z",
  "status": "valid",
  "secret_key": "ws_xxxxxxxxxxxxx",
  "bootstrap_token": "mdc_wrk_xxxxx"
}

Credential Status Tracking

The credentials file includes a status field that tracks validity:

  • "valid": Credentials are working and can be used
  • "invalid": Credentials failed authentication and worker has shut down

When credentials become invalid, they are preserved (not deleted) to allow:

  1. Remote credential updates via microDC console
  2. Manual credential repair without physical server access
  3. Credential inspection for troubleshooting

Security Features

  1. Token Types:
  2. Bootstrap Token (mdc_wrk_): One-time use for initial registration
  3. Secret Key (ws_): Worker-specific secret for login

  4. Worker Token (wt_): Bearer token for API authentication

  5. Credential Protection:

  6. Stored with 0700 permissions (owner read/write only)
  7. Located in user's home directory

  8. Never logged in plaintext

  9. Token Validation:

  10. Worker verifies saved tokens with heartbeat before use
  11. Automatic re-registration if token is invalid
  12. Graceful fallback to bootstrap token

API Endpoints Used

Endpoint Method Purpose Authentication
/api/v1/workers/register POST Initial registration Bootstrap token in body
/api/v1/workers/auth/login POST Get worker token node_id + secret_key
/api/v1/workers/heartbeat POST Send status updates Bearer token
/api/v1/jobs/claim POST Claim jobs Bearer token
/api/v1/jobs/{id}/complete POST Submit results Bearer token

Common Authentication Scenarios

Scenario 1: Fresh Installation

  1. Admin provides bootstrap token
  2. Worker registers and saves credentials
  3. Worker operates normally

Scenario 2: Worker Restart

  1. Worker loads saved credentials
  2. Verifies token is still valid
  3. Continues without re-registration

Scenario 3: Token Expiration (Automatic Refresh)

  1. Worker loads saved credentials
  2. Token expires during operation
  3. Worker automatically refreshes token using secret_key
  4. Updates saved credentials with new token
  5. Continues operating without interruption

Scenario 4: Failed Credential Refresh (Worker Shutdown)

  1. Worker loads saved credentials
  2. Token refresh fails (no secret_key or refresh error)
  3. Worker marks credentials as "invalid" in the file
  4. Worker shuts down gracefully to prevent repeated failed auth attempts
  5. Credentials remain in file for console/manual updates
  6. Recovery requires console update, new bootstrap token, or manual credential fix

Scenario 5: Server Restart

  1. Worker's saved credentials remain valid
  2. Worker reconnects automatically
  3. No re-registration needed

Troubleshooting

"No API key/token configured"

  • Set MICRODC_API_KEY environment variable
  • Or add api_key: your-token to config file under server section

"Invalid API key or authentication failed"

  • Bootstrap token may be expired or wrong
  • Check with server admin for new token

"Saved credentials invalid" / "FATAL CREDENTIAL ERROR"

  • Credentials marked as invalid and worker has shut down
  • Credentials are preserved in ~/.microdc/worker_credentials.json with "status": "invalid"
  • Worker will not repeatedly attempt authentication to avoid server load

Recovery Options:

  1. Console Update (Recommended): Update credentials via microDC console/dashboard
  2. New Bootstrap Token: Set MICRODC_BOOTSTRAP_TOKEN environment variable and restart
  3. Manual Repair: Edit ~/.microdc/worker_credentials.json and set "status": "valid" with correct credentials

Credential File Location

  • Linux/Mac: ~/.microdc/worker_credentials.json
  • Windows: %USERPROFILE%\.microdc\worker_credentials.json

Best Practices

  1. Keep Bootstrap Token Secure: Only use in initial setup
  2. Don't Share Credentials: Each worker has unique credentials
  3. Monitor Token Expiry: Check logs for authentication failures
  4. Backup Credentials: Save credential file when migrating workers
  5. Rotate Tokens Periodically: Re-register workers for security
  6. Console Management: Use microDC console to update worker credentials remotely when possible
  7. Credential Preservation: Never manually delete credential files - they enable remote recovery