> ## Documentation Index > Fetch the complete documentation index at: https://internal.september.wtf/llms.txt > Use this file to discover all available pages before exploring further. # Database > Operating the brain SQLite database. The Engine's brain is a single SQLite file. SQLite gets a lot of grief from people who haven't run it in production at scale; in practice it handles single-user agentic workloads beautifully. This page covers operating the brain — backup, restore, vacuum, integrity checks, sizing. ## What's in the brain 20+ tables. The full schema lives in `migrations/*.sql`. The major ones: | Table | What it holds | Growth rate | | --------------------------------------------------- | ------------------------------- | ----------------- | | `episodes`, `episodes_vec` | Episodic memory. | Slow | | `knowledge_store`, `knowledge_store_vec` | Knowledge facts. | Slow | | `social_graph_nodes`, `_edges`, `_vec` | People and relationships. | Slow | | `working_memory_log` | Within-task context. TTL-bound. | Fast (but pruned) | | `trajectories` | Per-task execution traces. | Fast | | `conversations` | Conversation threading. | Slow | | `channel_state_snapshots` | SSE checkpoints. TTL-bound. | Fast (but pruned) | | `connections`, `server_registry`, `action_registry` | Asset Directory. | Slow | | `learning_signals` | User feedback. | Slow | | `observability_events` | Telemetry. | Fast | | `ingestion_chunks` | Large media chunks. | Variable | `*_vec` tables hold 1536-dim float embeddings; they grow with episodes, knowledge facts, and social-graph nodes. ## Sizing A working brain for an actively-used agent typically grows at: * A few KB to a few MB per turn (working memory + observability events). * A few KB per episode (long-term, after Learning Centre processes trajectories). * A few hundred bytes per knowledge fact. * A few KB per MCP connection (encrypted credentials are small). Order-of-magnitude after sustained use: | Time | Brain size (typical user) | | ----------- | ------------------------- | | First week | 5–20 MB | | First month | 50–200 MB | | First year | 500 MB – 2 GB | Multi-GB brains run fine. The bottleneck is your backup strategy, not SQLite. ## Backup Three options, in increasing rigor: ### `/memory/export` The Engine exposes an API endpoint that returns the entire memory as a portable JSON bundle: ```bash theme={null} curl -fsS "$ENGINE_URL/memory/export" \ -H "X-Engine-Key: $ENGINE_KEY" \ > brain-export-$(date +%s).json ``` Restore via `POST /memory/import`. Useful for migrating between Engines or for application-level backups. Caveat: the export covers memory and connections. It does NOT cover: * Channel state (transient). * Observability events. * Trajectories not yet consolidated. For a true full backup, use one of the next two. ### SQLite `.backup` SQLite's online backup API takes a consistent snapshot without requiring downtime: ```bash theme={null} docker compose exec engine sqlite3 /data/brain.sqlite \ ".backup /data/brain-$(date +%s).db" ``` The backup file is itself a SQLite file you can copy out and store. To run on a schedule (cron / k8s CronJob): ```bash theme={null} #!/bin/bash TS=$(date +%Y%m%d-%H%M%S) docker compose exec engine sqlite3 /data/brain.sqlite \ ".backup /data/backups/brain-${TS}.db" docker compose exec engine sh -c \ 'find /data/backups -mtime +7 -delete' # 7-day retention ``` ### Volume snapshot If your infrastructure provides volume snapshots (EBS, GCP persistent disks, k8s VolumeSnapshot): ``` schedule: daily at 02:00 retain: 7 daily, 4 weekly, 12 monthly ``` Volume snapshots are the cheapest, fastest, and most complete backup. They include the entire data directory, not just the SQLite file. Recommended for production. ## Restore ### From `/memory/export` ```bash theme={null} curl -X POST "$ENGINE_URL/memory/import" \ -H "X-Engine-Key: $ENGINE_KEY" \ -H "Content-Type: application/json" \ -d @brain-export.json ``` The import is additive — existing memory is preserved, new memory is merged in. To restore to a clean state, wipe the brain volume first. ### From `.backup` ```bash theme={null} # 1. stop the engine docker compose stop engine # 2. restore docker compose run --rm --entrypoint sh engine -c \ 'sqlite3 /data/brain.sqlite ".restore /data/backups/brain-20260420.db"' # 3. start the engine docker compose up -d engine ``` ### From volume snapshot Detach the data volume; attach the snapshot; start the Engine. Your infrastructure's documentation has the specific commands. ## Integrity check SQLite has a built-in integrity check: ```bash theme={null} docker compose exec engine sqlite3 /data/brain.sqlite \ 'PRAGMA integrity_check;' ``` Should print `ok`. Run after any restore, after a hard kill, or on a schedule (weekly). If it returns errors, the brain is damaged. Restore from backup. ## Vacuum SQLite doesn't reclaim disk space automatically. As rows are deleted (working memory entries expiring, old trajectories cleaned up), the file size doesn't shrink — the deleted space sits as free pages. To reclaim: ```bash theme={null} docker compose exec engine sqlite3 /data/brain.sqlite 'VACUUM;' ``` This requires temporary disk space equal to the brain's current size (SQLite copies into a temp file). Run during low-traffic windows; the database is locked during the vacuum. For high-write workloads, consider auto-vacuum mode (set at table creation; the Engine's migrations don't enable it today). A reasonable schedule: vacuum monthly. The file size growth between vacuums isn't dramatic in normal use. ## Migrations Migrations run automatically on Engine startup. Each migration file is a SQL script: ``` migrations/ 001_memory_schema.sql 002_utility_directory.sql ... 014_token_issued_at.sql ``` The Engine tracks which migrations have been applied in a `_migrations_applied` table. On startup, it runs any unapplied migrations in order. **Don't edit applied migrations.** Once a migration is in production, treat it as immutable. To change a schema, add a new migration with the change. **Don't run migrations manually.** The Engine is the only thing that should write to `_migrations_applied`. ## Cleanup The Engine doesn't auto-clean old data today. A reasonable retention policy: ```bash theme={null} # trajectories older than 30 days sqlite3 /data/brain.sqlite \ 'DELETE FROM trajectories WHERE created_at < datetime("now","-30 days");' # observability events older than 90 days sqlite3 /data/brain.sqlite \ 'DELETE FROM observability_events WHERE created_at < datetime("now","-90 days");' # channel state snapshots are TTL-pruned automatically ``` Run weekly via cron / CronJob, then `VACUUM` monthly. ## Performance tuning For deployments with high write rates: * **WAL mode.** The Engine enables WAL by default. Verify: ```sql theme={null} PRAGMA journal_mode; ``` Should return `wal`. * **Synchronous mode.** Default is `NORMAL`. For higher write throughput at the cost of slightly more durability risk on crash, use `synchronous = NORMAL` (already the default). * **Cache size.** The Engine sets a sensible default. Override only if profiling shows cache misses. ## When the brain grows too large A few GB is fine. Tens of GB is fine. Hundreds of GB starts to feel slow. If you're approaching that scale on a single brain: 1. **Audit retention.** Are you keeping data you don't need? 2. **Move ingestion\_chunks out.** Large media chunks are the most common bloat. Consider a separate object store (S3, GCS) and store only references in the brain. 3. **Consider sharding.** Run multiple Engines per user with topic- specific brains. Outside the design today, but possible. ## See also * [Database in components](../../internal/architecture/components#infra) — the implementation. * [Production deploy](../deploy/prod-deploy) — backup setup at deploy time. * [Memory endpoints](../../platform/api-reference/endpoints/memory) — application-level backup via `/memory/export`.