> ## Documentation Index
> Fetch the complete documentation index at: https://internal.september.wtf/llms.txt
> Use this file to discover all available pages before exploring further.

# Backends

> How the orchestrator actually launches engine processes — Docker (production) and subprocess (dev).

The orchestrator's lifecycle layer doesn't know how to start an
engine. It delegates to a backend that implements `EngineBackend`.
Two backends exist today: Docker (production) and subprocess (dev /
testing).

You pick which to run via `ORCH_ENGINE_BACKEND`.

## EngineBackend interface

Every backend implements:

```python theme={null}
class EngineBackend(abc.ABC):
    async def create(self, engine_id: UUID, port: int,
                     env: dict, image: str) -> EngineEndpoint: ...
    async def start(self, engine_id: UUID) -> None: ...
    async def stop(self, engine_id: UUID, timeout: int = 30) -> None: ...
    async def destroy(self, engine_id: UUID) -> None: ...
    async def is_running(self, engine_id: UUID) -> bool: ...
```

Where `EngineEndpoint` is `{host: str, port: int, container_id: str | None}`.

Lifecycle calls these in order: `create` (during provision) →
`start`/`stop`/`destroy` (during the corresponding HTTP endpoint).

## Docker backend

**File:** `orchestrator/backends/docker_backend.py`

**Use when:** running in production. The default for `ORCH_ENGINE_BACKEND=docker`.

### What it does

Each engine becomes a Docker container. On `create()`:

1. Pulls the image if missing (`ORCH_ENGINE_IMAGE`).
2. Creates a named volume for the brain:
   `engine-data-{engine_id[:12]}`.
3. Starts a container with:
   * The named volume mounted at `/data` (rw).
   * `ORCH_CATALOG_MOUNT_PATH` (host) mounted at `/catalog` (ro).
   * The Docker network `ORCH_ENGINE_NETWORK` (default `engine_net`).
   * Environment: `ENGINE_KEY_HASH`, `SQLITE_DB_PATH=/data/brain.sqlite`,
     `CATALOG_DIR=/catalog`, plus any vars in
     `ORCH_ENGINE_ENV_PASSTHROUGH`.
   * Restart policy: `unless-stopped`.
   * Container port mapped to host port (the one allocated by the
     orchestrator), bound to `127.0.0.1`.

### Why Docker

* **Isolation.** One user's Engine can't see another's filesystem.
* **Resource limits.** CPU and memory ceilings per container; one
  rogue user can't starve the rest.
* **Image pinning.** `ORCH_ENGINE_IMAGE` is a specific tag; upgrades
  are deliberate.
* **Standard tooling.** Logs, exec-into, restart, ps — all the usual
  Docker affordances.

### Requirements

The orchestrator container must have:

* `/var/run/docker.sock` mounted (or a remote Docker daemon configured
  via `DOCKER_HOST`).
* The `docker` Python SDK installed (in the orchestrator image
  already).
* Permission to pull images from the configured registry.

Your host needs:

* Docker Engine running.
* Enough disk for the `engine-data-*` named volumes.
* Enough RAM to hold N concurrent engine containers.

### What you lose

* **Cold-start cost.** Each engine takes a few seconds to boot
  (migrations + catalog load). The orchestrator's auto-wake helps
  amortize this.
* **Per-engine memory overhead.** \~200 MB per container at idle. For
  thousands of users, this adds up; consider `ORCH_IDLE_SLEEP_THRESHOLD_S`
  to auto-stop sleeping engines if RAM is tight.

## Subprocess backend

**File:** `orchestrator/backends/subprocess_backend.py`

**Use when:** developing or running tests on a single machine
without Docker. Set `ORCH_ENGINE_BACKEND=subprocess`.

### What it does

Each engine becomes a local Python subprocess running:

```
python -m uvicorn src.server:app --host 0.0.0.0 --port {port} --workers 1
```

with environment variables passed in directly.

### Why exists

* **Faster iteration.** No image build, no container start. Useful
  during engine development.
* **CI without Docker-in-Docker.** Tests can run subprocess engines
  inside a single CI runner.
* **Debugger access.** `pdb` works; container barriers don't get in
  the way.

### Limitations

* **Not isolated.** Every subprocess shares the host filesystem and
  network. Don't run multi-tenant.
* **No restart support.** `start()` is a no-op (warning logged); to
  restart, destroy and re-create.
* **No resource limits.** A runaway subprocess can take down the host.
* **Different sandbox availability.** The engine's bash sandbox needs
  `bubblewrap`, which is harder to get right inline. Some engine
  features may not work outside Docker.

Use subprocess for unit tests and quick orchestrator-side
development. Use Docker for everything else.

## Switching

`ORCH_ENGINE_BACKEND` is read at orchestrator startup. To switch:

1. Stop the orchestrator.
2. Stop and destroy any in-flight engines.
3. Update `ORCH_ENGINE_BACKEND` in the env.
4. Restart.

You can't mix backends in one orchestrator instance.

## Future backends

The interface is stable, so additional backends are possible. Likely
candidates if we need them:

* **Kubernetes Pod** — for cluster-native deployments. Each engine
  becomes a Pod managed via the K8s API.
* **Firecracker microVM** — for stronger isolation than containers.
* **Remote Docker** — running engines on a different host than the
  orchestrator.

None are implemented today.

## See also

* [Architecture](./architecture) — where backends sit in the system.
* [Lifecycle](./lifecycle) — what backends are called for at each
  state transition.
* [Environment variables](./config/env-vars#orchestrator-configuration)
  — `ORCH_ENGINE_BACKEND`, `ORCH_ENGINE_IMAGE`, etc.