Skip to main content
Plan rollback before you need it. The middle of an incident is not the time to figure out which command reverts which database, or whether your snapshots are recent enough.

​
When to roll back

Roll back when:
  • /health returns non-ok and you can’t fix it within minutes.
  • Error rate or latency is sharply worse than baseline and not improving.
  • A regression eval that passed pre-upgrade is now failing.
  • Customer reports confirm the new version is broken.
Don’t roll back for:
  • One outlier metric that hasn’t actually impacted users.
  • Behavior changes that were expected per the changelog.
  • Issues that are clearly local to one bad request rather than the service as a whole.
If you’re unsure, roll back. The cost of an unnecessary rollback is low; the cost of leaving a bad version live is high.

​
What rollback looks like

Rollback has two layers:
  1. Image rollback — switch the Engine container back to the previous image tag.
  2. Data rollback — restore the brain database to a pre-upgrade snapshot, if migrations made the new version’s schema incompatible with the old.
Most rollbacks need only #1. Some need #2.

​
Image rollback

If the new version’s migrations didn’t change the brain schema in a way the old version doesn’t understand:
If the previous version’s image was deleted from the registry, you may need to rebuild from the previous git tag. Plan for this — keep at least the last 5 versions of each image.

​
Brain rollback

When a forward-only migration shipped in the new version (most migrations), the old Engine version can’t read the upgraded brain. You have to restore the brain to the pre-upgrade state.

​
Restoring from snapshot

You will lose any data written between the snapshot and the rollback — typically minutes to an hour, depending on how long the bad version was live.

​
What gets lost

Specifically:
  • Any episodes, knowledge, or social-graph entries written during the bad window.
  • Any conversation history from the bad window.
  • Any feedback submitted during the bad window.
Depending on your application, you may need to inform affected users or replay important events.

​
What’s preserved

  • The pre-upgrade brain state, exactly.
  • All long-term memory from before the upgrade.
  • All migrations applied before the upgrade.

​
Volume snapshot rollback

If your infrastructure provides volume snapshots (EBS, GCP persistent disks, etc.):
Volume snapshots are usually faster to restore than file-level restores for large brains. Test the restore path before you need it.

​
Drain before stopping

The Engine has graceful-shutdown logic. Stopping with a drain timeout lets in-flight turns complete:
60 seconds is usually enough; longer turns get cut off but the brain state is consistent because the agent loop persists at boundaries (after every model call, every tool result).

​
Communicating

While rolling back:
  • Post in the incident channel: “Rolling back to v2.3.1 due to $symptom.”
  • Update the status page if customer impact was real.
  • After rollback completes: confirm /health ok, errors recovered, post the all-clear.

​
Post-rollback

Once the rollback is in and stable:
  1. Capture the diagnostic. Logs, traces, metrics from the bad window. You’ll need them to fix forward.
  2. Don’t redeploy in panic. Whatever broke needs root cause first.
  3. File a postmortem ticket. See Postmortems.
  4. Add a regression eval. The exact failure mode should be a permanent test case. See Regression.

​
When rollback isn’t possible

Some failure modes can’t be cleanly rolled back:
  • The brain is corrupted and snapshots are unusable. You’re now in a data-recovery scenario; involve whoever owns the brain.
  • Migrations dropped data that the old version expects. Same scenario.
  • The new version sent destructive MCP calls (deleted external resources, sent emails). The Engine’s state is recoverable; the external state isn’t.
For these, the rollback is partial — restore what you can, accept the loss for what you can’t, and design future migrations to be reversible.

​
Practice rollback

Run a rollback drill in staging quarterly. The exercise:
  1. Bring staging up on the current production version.
  2. Take a snapshot.
  3. Upgrade staging to a deliberately-broken version (e.g. an internal build with a known bug).
  4. Roll back using the procedure above.
  5. Time how long it took. Identify what slowed you down.
A rollback you’ve never practiced is a rollback that won’t go to plan.

​
See also

  • Upgrade — how to do the upgrade so rollback isn’t needed.
  • Postmortems — what comes after.
  • Database — backup and restore details.