> ## Documentation Index
> Fetch the complete documentation index at: https://internal.september.wtf/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture

> What's inside bap-engine — the orchestrator, registry, lifecycle, health loop, and backends.

bap-engine is a single FastAPI process backed by PostgreSQL. It speaks
HTTP to products on one side and the Docker API to Engine containers
on the other. This page maps what's inside.

## The picture

```mermaid theme={null}
flowchart TB
    subgraph Process["bap-engine process — uvicorn orchestrator.server:app:8000"]
        HTTP["HTTP API (FastAPI)<br/>/engines /products /status /metrics /health"]
        Auth["auth.py<br/>key generation + Fernet encryption"]
        Reg["registry.py<br/>Product + Engine CRUD"]
        Disc["discovery.py<br/>user_id → engine + admission"]
        Pol["policy.py<br/>quota + rate limiting"]
        Life["lifecycle.py<br/>state machine<br/>(provision / start / stop / wake / destroy)"]
        Health["health.py<br/>background health loop"]
        Met["metrics.py<br/>fleet status"]
        Aud["audit.py<br/>persistent audit log"]
        Backends["backends/<br/>EngineBackend (ABC)"]
        Docker["docker_backend.py"]
        Sub["subprocess_backend.py"]
    end

    PG[("PostgreSQL<br/>products, engines,<br/>port_allocations,<br/>audit_log")]
    DockerAPI["Docker API<br/>(/var/run/docker.sock)"]
    Engines["Engine containers<br/>(per-user)"]

    HTTP --> Auth
    HTTP --> Disc
    Disc --> Reg
    Disc --> Pol
    Disc --> Life
    Life --> Backends
    Backends --> Docker
    Backends --> Sub
    Docker -->|create / start / stop / destroy| DockerAPI
    DockerAPI --> Engines
    Health --> Engines
    Health --> Reg
    Reg --> PG
    Aud --> PG
    Met --> PG

    classDef boundary fill:#dbeafe,stroke:#1d4ed8
    classDef store fill:#fef3c7,stroke:#a16207
    classDef external fill:#f4f4f5,stroke:#71717a
    class HTTP,Auth,Reg,Disc,Pol,Life,Health,Met,Aud,Backends,Docker,Sub boundary
    class PG store
    class DockerAPI,Engines external
```

## The components

### HTTP API (`server.py`)

A FastAPI app with five route groups:

* **Engine lifecycle** — `POST /engines/provision`, `DELETE
  /engines/{user_id}`, `POST /engines/{user_id}/{start,stop,
  rotate-key}`.
* **Discovery + admission** — `GET /engines/{user_id}`,
  `POST /engines/{user_id}/admit`.
* **Fleet** — `GET /engines`, `GET /status`, `GET /metrics`.
* **Admin** — `POST /products/register`, `PUT
  /products/{product_id}/policy`. Requires `X-Admin-Key`.
* **Health** — `GET /health` (no auth).

Auth is enforced at the route level by `_get_product()`, which hashes
`X-Platform-Key`, looks up the matching product, and rejects unknown
keys.

### Registry (`registry.py`)

The CRUD layer over Postgres. Two main types:

* `Product` — a tenant on the orchestrator. Has a slug, a hashed
  platform API key, and a JSON policy.
* `Engine` — one user's Engine instance. Has a status, a port, a
  hashed engine API key, and a Fernet-encrypted plaintext key.

Registry methods are pure — no side effects on Docker, no audit
writes. Higher layers compose them.

### Lifecycle (`lifecycle.py`)

The state machine for an Engine. Valid states:

```
provisioning → running → sleeping → running → stopped → destroying
                ↓                              ↑
              failed ──── (auto-restart) ─────┘
```

Each transition does the right Docker call (via the chosen backend),
updates the registry, and writes an audit entry. Lifecycle is the
only layer allowed to call into the backend — everyone else asks
lifecycle.

### Discovery (`discovery.py`)

The product's main entry point goes through here:
`POST /engines/{user_id}/admit` calls `discovery.admit()`, which:

1. Asks `policy.check_admit()` — rate-limit + quota.
2. Looks up the engine in the registry.
3. If none and `auto_provision=true` and policy allows, provisions.
4. If sleeping and `auto_wake=true`, wakes.
5. Returns `{admitted: true, engine: {url, api_key, ...}}` or
   `{admitted: false, ...}`.

Discovery is the only layer that's allowed to auto-provision. Direct
calls to `/engines/provision` are explicit and skip the auto-policy.

### Policy (`policy.py`)

Two checks:

* **Quota.** A product's policy carries `max_engines: int`. Provision
  fails if the count would exceed it.
* **Rate limit.** A product's policy carries `rate_limit_rpm: int`.
  Admit fails if the per-product per-minute rate would exceed it.

Policy is a simple `PolicyEngine` that reads the product's policy
JSON. No external store; no Redis. The state lives in the registry
plus an in-memory rate counter.

### Health (`health.py`)

A background loop that runs every `ORCH_HEALTH_CHECK_INTERVAL_S`
(default 30 s). For each running engine, in parallel up to a
semaphore of 50:

1. `GET /health` against the engine's URL with `ORCH_HEALTH_CHECK_TIMEOUT_S`
   (default 10 s).
2. On 200, reset `health_failures`, update `last_health_at`.
3. On failure, increment `health_failures`. After
   `ORCH_HEALTH_MAX_FAILURES` (default 3), mark the engine `failed`.
4. Schedule auto-restart with exponential backoff:
   `min(ORCH_RESTART_BACKOFF_BASE_S * 2^attempt, ORCH_RESTART_BACKOFF_MAX_S)`.
5. After `ORCH_RESTART_MAX_ATTEMPTS` (default 8), give up. The engine
   stays `failed` until ops intervenes.

Health is what makes the fleet self-healing.

### Backends (`backends/`)

Two implementations of `EngineBackend` (an abstract base):

* **`docker_backend.py`** — production. Creates Docker containers,
  mounts the data volume, attaches to the Docker network, runs the
  configured engine image.
* **`subprocess_backend.py`** — dev/test. Spawns `python -m uvicorn
  src.server:app` as a local subprocess. No Docker.

The orchestrator picks via `ORCH_ENGINE_BACKEND`. See
[Backends](./backends).

### Audit (`audit.py`)

Every lifecycle action writes a row to `audit_log`: action, actor,
product/user/engine IDs, metadata, duration. Used for incident
investigation and compliance reporting.

### Metrics (`metrics.py`)

Aggregates over the registry and audit log:

* `GET /status` — count by state, unhealthy count, overall health.
* `GET /metrics` — provisions, crashes, restarts in the last hour;
  average boot time; lifetime totals.

Stateless reads; no metric store of its own.

## What state lives where

| State                       | Where                     | Lifetime                           |
| --------------------------- | ------------------------- | ---------------------------------- |
| Products                    | `products` table          | Forever                            |
| Engines                     | `engines` table           | Until destroyed                    |
| Port allocations            | `port_allocations` table  | Until engine destroyed             |
| Audit                       | `audit_log` table         | Forever (until pruned)             |
| Rate-limit counters         | In-memory                 | Process lifetime                   |
| Engine API keys (plaintext) | Never stored              | Returned to caller, then encrypted |
| Engine API keys (encrypted) | `engines.engine_key_enc`  | Until destroyed                    |
| Engine API keys (hashed)    | `engines.engine_key_hash` | Until destroyed                    |

The plaintext platform API key never lives in the orchestrator. The
plaintext engine API key only exists at provision time (returned to
the caller) and at start time (returned to the caller after
decryption with `ORCH_MASTER_KEY`).

## What runs where

| Container           | Image                        | Purpose                                               |
| ------------------- | ---------------------------- | ----------------------------------------------------- |
| `postgres`          | `postgres:16-alpine`         | Orchestrator state.                                   |
| `orchestrator`      | bap-engine prod image        | The FastAPI process.                                  |
| `engine` containers | `september-engine:<version>` | Per-user Engine instances. Started/stopped on demand. |

The orchestrator and engines share a Docker network
(`engine_net`) so the orchestrator can reach engine `/health`
endpoints by container hostname.

## Where it doesn't go

Some choices the orchestrator deliberately doesn't make:

* **Multi-region.** All engines run on one host. For multi-region, you
  run multiple bap-engine deployments, one per region.
* **Migration.** No live migration of an engine from one host to
  another. To move a user, destroy the source engine and provision a
  fresh one (their brain is in the volume; mount it on the new host).
* **Cross-product user reuse.** A user\_id is per-product. The
  orchestrator does not know that user `u-123` in product A is the
  same human as user `u-123` in product B.

## See also

* [Engine contract](./engine-contract) — the API surface between
  orchestrator and engines.
* [Lifecycle](./lifecycle) — the state machine in detail.
* [Backends](./backends) — Docker vs subprocess.
* [Health](./health) — auto-restart and the loop.