> ## Documentation Index
> Fetch the complete documentation index at: https://internal.september.wtf/llms.txt
> Use this file to discover all available pages before exploring further.

# Security

> Keys, encryption, rotation, and the trust boundaries inside bap-engine.

bap-engine handles two kinds of secrets: platform API keys (used by
products to authenticate to the orchestrator) and engine API keys
(generated for each provisioned engine, used by the product to
authenticate to that engine). Plus the master key that encrypts engine
keys at rest.

This page covers what each secret is, where it lives, and how to
rotate it.

## The keys

### Master key (`ORCH_MASTER_KEY`)

The most critical secret in the whole system. A Fernet key
(base64-encoded 32 bytes) used to encrypt engine API keys at rest.
Without it, no engine API key can be decrypted; products lose access
to every engine they admit.

* **Format:** `Fernet.generate_key().decode()` (43 chars, URL-safe).
* **Where it lives:** `ORCH_MASTER_KEY` env var on the orchestrator.
  Should be injected at runtime from your secret manager.
* **Where it doesn't:** Postgres, logs, the orchestrator's filesystem.
* **Rotation:** Heavy operation. Requires re-encrypting every
  engine's `engine_key_enc`. There's no built-in rotation script
  today; doing it requires a custom migration.

If `ORCH_MASTER_KEY` ever leaks, every plaintext engine API key the
orchestrator could decrypt is now exposed. Treat it like a database
master password — store in a secret manager, rotate on schedule,
audit access.

### Admin key (`ORCH_ADMIN_KEY`)

Used by `POST /products/register` and
`PUT /products/{product_id}/policy`. Without it, you can't onboard
new products.

* **Format:** any opaque string (we use `secrets.token_urlsafe(48)`).
* **Where it lives:** `ORCH_ADMIN_KEY` env var on the orchestrator.
* **How it's checked:** plain string comparison against the
  `X-Admin-Key` header.
* **Rotation:** trivial. Generate a new value, swap the env var,
  restart the orchestrator. Tooling that uses the old value gets
  `INVALID_KEY`; update them.

### Platform API key (per product)

Generated by the orchestrator at `/products/register`. The product
uses it on every request as `X-Platform-Key`.

* **Format:** `pk-sept-<URL-safe random>` (42+ chars).
* **Where it lives:** `products.api_key_hash` (SHA-256). The plaintext
  is returned **once** at registration and never stored.
* **Rotation:** today, no built-in rotation endpoint. To rotate,
  generate a new key (re-register the product) and update the
  consumer. Future versions will add `POST /products/{id}/rotate-key`.

### Engine API key (per engine)

Generated by the orchestrator at provision time. The product uses it
to authenticate to the specific engine via `X-Engine-Key`.

* **Format:** `sk-sept-<URL-safe random>` (42+ chars).
* **Where it lives:**
  * `engines.engine_key_hash` — SHA-256, stored in Postgres.
  * `engines.engine_key_enc` — Fernet-encrypted plaintext, stored in
    Postgres. The orchestrator decrypts when admitting the same
    user later.
  * The engine's `ENGINE_KEY_HASH` env var — the engine validates
    incoming `X-Engine-Key` against this.
* **Rotation:** `POST /engines/{user_id}/rotate-key`. Generates a
  fresh key, updates both forms in Postgres, swaps the engine's
  hash, returns the new plaintext.

## What's never stored

* Plaintext platform API keys.
* Plaintext engine API keys (the encrypted form is stored, but
  plaintext only exists in flight: at provision/admit response, and
  at the engine's `ENGINE_KEY_HASH` validation step).
* `ORCH_MASTER_KEY` itself — read from the env at startup.

## The auth flow end-to-end

```mermaid theme={null}
sequenceDiagram
    participant Admin
    participant Product
    participant Orch as bap-engine
    participant Engine

    Note over Admin,Orch: One-time product onboarding
    Admin->>Orch: POST /products/register (X-Admin-Key)
    Orch->>Orch: generate platform_api_key
    Orch-->>Admin: {platform_api_key: pk-sept-...}
    Note over Admin: Hand platform_api_key to product

    Note over Product,Engine: Every user request
    Product->>Orch: POST /admit (X-Platform-Key)
    Orch->>Orch: hash, look up product
    Orch->>Orch: generate engine_api_key (sk-sept-...)
    Orch->>Orch: hash + Fernet-encrypt
    Orch->>Engine: launch with ENGINE_KEY_HASH
    Orch-->>Product: {engine: {url, api_key}}
    Product->>Engine: POST /execute (X-Engine-Key)
    Engine->>Engine: hash, compare to ENGINE_KEY_HASH
    Engine-->>Product: SSE stream
```

## What the orchestrator does NOT enforce

* **Per-user authentication beyond the platform key.** The
  orchestrator trusts that the product authenticated the user
  upstream. From the orchestrator's perspective, `user_id` is just an
  identifier.
* **Encryption of in-flight network traffic.** TLS belongs at the
  reverse proxy upstream of the orchestrator. The orchestrator
  itself serves HTTP.
* **DDoS protection.** Rate limiting at the policy layer is
  per-product, not per-IP. Layer 7 protection should be upstream.
* **MCP credential security.** The engine handles per-user OAuth and
  Fernet-encrypts MCP credentials at the engine layer using a
  separate `AD_ENCRYPTION_KEY`. The orchestrator doesn't see them.

## Trust boundaries

```mermaid theme={null}
flowchart TB
    Public["Public internet"]
    LB["TLS terminator<br/>(reverse proxy)"]
    Product["Product (BAP)"]
    Orch["bap-engine<br/>(plaintext platform key on incoming)"]
    PG[("PostgreSQL<br/>hashed + encrypted secrets")]
    Engine["Engine container<br/>(only sees its own ENGINE_KEY_HASH)"]

    Public -->|HTTPS| LB
    LB -->|HTTP, internal| Product
    Product -->|HTTP, X-Platform-Key| Orch
    Orch --> PG
    Orch -->|Docker API + ENGINE_KEY_HASH env| Engine

    classDef public fill:#fee2e2,stroke:#991b1b
    classDef internal fill:#dbeafe,stroke:#1d4ed8
    classDef trusted fill:#dcfce7,stroke:#15803d
    class Public public
    class LB,Product internal
    class Orch,PG,Engine trusted
```

The boundaries:

* **Public ↔ TLS terminator.** TLS handshake. Standard.
* **Product ↔ orchestrator.** Internal network. Plaintext platform key
  on the wire (HTTP), but only inside the trusted network. Run on a
  VPC or internal subnet.
* **Orchestrator ↔ Postgres.** Internal. Use Postgres TLS if you
  cross VPCs.
* **Orchestrator ↔ engine container.** Internal Docker network.
  Plaintext engine key on the wire. Never expose engine ports
  beyond the host.

## Rotation playbook

### Rotate `ORCH_MASTER_KEY`

**This requires downtime and is heavy. Don't rotate unless you must.**

1. Schedule maintenance window.
2. Take a Postgres snapshot.
3. Stop the orchestrator.
4. Run a one-off script that:
   * Reads every `engines.engine_key_enc` row.
   * Decrypts with old `ORCH_MASTER_KEY`.
   * Re-encrypts with new key.
   * Writes back.
5. Update `ORCH_MASTER_KEY` env on the orchestrator.
6. Start the orchestrator.
7. Verify by admitting a known user; confirm engine key still works.

If you rotate without re-encrypting, every product that admits an
existing user gets back an undecryptable key, and the engine's
`ENGINE_KEY_HASH` won't match — every existing engine is effectively
locked. Provision new ones; the old ones are dead weight.

### Rotate `ORCH_ADMIN_KEY`

1. Generate new value.
2. Update env on the orchestrator.
3. Restart.
4. Update tooling that calls `/products/register` or `/policy`.

Trivial.

### Rotate a platform API key

Today, register the product again with a new slug or update the
plaintext secret out-of-band. Future: a dedicated rotation endpoint.

### Rotate an engine API key

```bash theme={null}
curl -X POST "$ORCH_URL/engines/{user_id}/rotate-key" \
  -H "X-Platform-Key: $PLATFORM_KEY"
```

Returns the new plaintext. Old key invalid immediately. The product
must update its cached key.

## What goes wrong

| Symptom                                                           | Likely cause                                                                                                               |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `INVALID_PLATFORM_KEY` for known products                         | Postgres restored from a backup older than the registration. Re-register.                                                  |
| All engine keys reject after restart                              | `ORCH_MASTER_KEY` changed without re-encrypt.                                                                              |
| Product gets a fresh key on every admit                           | Ignoring the cached key returned by previous admit. Cache and reuse.                                                       |
| Engine returns `INVALID_KEY` despite using the orchestrator's key | The engine's `ENGINE_KEY_HASH` env var is stale (engine restarted before the orchestrator pushed the new hash). Re-rotate. |

## See also

* [Architecture](./architecture#auth-authpy) — implementation.
* [API reference: auth](./api-reference/auth) — the `X-Platform-Key`
  flow.
* [Engine contract](./engine-contract#auth-flow) — the engine side
  of the same flow.