> ## Documentation Index
> Fetch the complete documentation index at: https://internal.september.wtf/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets

> Where secrets live, how they get to the Engine, and how to rotate them.

The Engine handles several kinds of secrets. Each has different
sensitivity and rotation cadence. This page covers what they are, how
they should flow, and how to rotate.

## What's secret

| Secret                                  | Lives in                     | Sensitivity                                             |
| --------------------------------------- | ---------------------------- | ------------------------------------------------------- |
| `LLM_API_KEY`                           | Engine env                   | High — full access to LLM provider account.             |
| `OPENAI_API_KEY`                        | Engine env                   | High — used for embeddings.                             |
| `GEMINI_API_KEY`                        | Engine env                   | High — provider account.                                |
| `BRAVE_API_KEY` / `TAVILY_API_KEY`      | Engine env                   | Medium — search quotas.                                 |
| `ENGINE_API_KEY` (or `ENGINE_KEY_HASH`) | Engine env / upstream router | High — controls API access to this Engine.              |
| `AD_ENCRYPTION_KEY`                     | Engine env                   | Critical — losing it breaks all stored MCP credentials. |
| MCP credentials per connection          | Brain DB, encrypted          | High — full access to user's external services.         |

## How secrets should flow

The right flow:

1. Secret lives in your **secret manager** (Vault, AWS Secrets Manager,
   GCP Secret Manager, k8s Secret).
2. At Engine startup, the secret manager **injects the secret into the
   container's environment**.
3. The Engine reads from `os.environ` and never logs the value.

The wrong flows:

* Secret committed to git. Even in `.env.example`. Never.
* Secret in the docker image (baked at build time). Image can leak.
* Secret passed as a CLI argument. Visible in process listings.
* Secret in a JSON config file mounted into the container. Just use env.

For local dev, `.env` is fine — the file is gitignored, the dev
machine isn't a target. Don't use `.env` files in production.

## Secret-manager integration

### AWS Secrets Manager

Use the `secrets` field on your task definition (ECS) or set up CSI
driver (EKS) to inject secrets as env vars at boot.

### GCP Secret Manager

Use Workload Identity to authenticate, then pull at boot. The
[gcp-secrets-csi-driver](https://github.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp)
mounts secrets as env vars.

### Kubernetes Secrets

```yaml theme={null}
env:
  - name: LLM_API_KEY
    valueFrom:
      secretKeyRef:
        name: engine-secrets
        key: llm-api-key
```

For sensitive secrets, prefer external-secrets-operator over native k8s
Secrets (which are base64, not encrypted).

### HashiCorp Vault

Use the Vault Agent Injector or the Vault CSI provider to mount as
env vars at startup.

### Doppler / 1Password / Infisical

These tools inject secrets at process start time:

```bash theme={null}
doppler run -- python -m src.server
```

## Don't put secrets in the prompt

The Engine sanitizes outbound content with the secret scanner
(`src/security/secret_scanner.py`). It's defense-in-depth, not the
first line. Don't pass secrets through the user message or through
context fields.

If the agent legitimately needs to use a secret (calling an external
API), that secret lives in the MCP connection's encrypted credentials,
not in any prompt.

## Rotation

### LLM provider keys

**Cadence:** at least every 90 days; immediately if compromised.

**Process:**

1. Generate a new key in the provider's console.
2. Update the secret manager with the new value.
3. Restart the Engine (or roll the deployment).
4. Confirm `/health` ok and a test request succeeds.
5. Revoke the old key in the provider's console.

The Engine doesn't support runtime key rotation for provider keys —
they're read at boot.

### `ENGINE_API_KEY` rotation with overlap

The Engine supports zero-downtime rotation of its own API key via the
overlap mechanism:

```bash theme={null}
BAP_KEY_ROTATION_MIN_OVERLAP_SECONDS=0
BAP_KEY_ROTATION_MAX_OVERLAP_SECONDS=300
```

**Process:**

1. Generate a new key.
2. Compute its hash: `sha256sum`.
3. Update the secret manager with the new key (for callers) and the
   new hash (for the Engine).
4. Configure the Engine to accept both old and new keys for the
   overlap window.
5. Roll callers to use the new key.
6. After the overlap window, the Engine rejects the old key
   automatically.

For details, see [Authentication](../../platform/api-reference/auth).

### `AD_ENCRYPTION_KEY` rotation

**Critical.** Rotating this key without a re-encrypt step breaks every
stored MCP credential.

**Process:**

1. Generate a new Fernet key:
   ```bash theme={null}
   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
   ```
2. Run a re-encrypt migration: decrypt every credential with the old
   key, re-encrypt with the new. (No script ships today; this is a
   manual operation requiring downtime.)
3. Replace `AD_ENCRYPTION_KEY` with the new value.
4. Restart the Engine.
5. Verify a known MCP connection still works.

**Don't rotate `AD_ENCRYPTION_KEY` without a re-encrypt plan.** If you
must rotate without re-encrypting (e.g. you suspect the old key
leaked), expect every user to need to reconnect every MCP server.

### MCP credentials

These rotate themselves through OAuth refresh. The Engine's
`refresh_scheduler` refreshes credentials before they expire. If
refresh fails (the user revoked access), the connection moves to
`expired` and the user has to reconnect.

You don't manually rotate user OAuth credentials. You can force a
reconnection by deleting the connection (`DELETE /assets/connections/{ref_id}`)
and prompting the user to reconnect.

## Secret-handling rules

* **Never log a secret.** Even at DEBUG level. The Engine's logging is
  designed to redact, but if you write your own logs, don't log
  request bodies that may include keys.
* **Never echo a secret to the user.** Tools that need to display
  configuration should mask: show `sk-...***...`, not the full value.
* **Never commit a secret.** Use pre-commit hooks (`detect-secrets`,
  `gitleaks`) to catch them before they land.
* **Audit access.** Production secrets should be readable only by the
  service that needs them and the on-call humans. Audit periodically.

## When a secret leaks

If you discover a secret in logs, in git history, in a screenshot:

1. **Treat it as compromised** even if you think nobody saw it.
2. **Rotate immediately.** Don't wait for the next planned rotation.
3. **Audit access logs** at the provider for the period the secret
   was exposed. Look for unusual usage.
4. **File an incident.** Even "no impact" leaks deserve a postmortem
   so you fix the process that caused them.

## See also

* [Environment variables](./env-vars) — where each secret lives.
* [Threat model](../security/threat-model) — what we defend against.
* [Authentication](../../platform/api-reference/auth) — `ENGINE_API_KEY`
  rotation specifics.