kami/Maven
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

update: roll back what the restart actually deploys

Browse Source 
On the deployment deploy/README.md documents, source_dir and install_dir are
the same tree and the restart command rebuilds the image from it. The
Dockerfile builds from cmd/ and internal/ and .dockerignore keeps the host
binaries out, so restoring the snapshotted binaries restored bytes nothing
reads. A bad commit therefore cost two health timeouts and two image builds
and ended in ErrRollbackFailed with an instruction to copy files back by hand,
which would not have helped either.

A deployment that rebuilds from source now has to say how the source is put
back. source_rollback "git" records the commit before the update and checks it
back out before the rollback restart. It refuses a dirty tree, because the
recorded commit does not describe one and a forced checkout would delete his
work. A build-from-source config that says nothing is refused by Validate, at
startup, rather than at the one rollback that mattered.

Also in this change, all from the same review:

  - MethodPing, the one method a locked daemon answers. Preflight passed on an
    unlocked daemon and the post-restart Presence read failed on a locked one,
    so a good update read as SHE IS PROBABLY DOWN once the env key is gone.
  - A dial failure is reported apart from a read failure. The documented
    socket is under /var/lib/docker, which a non-root operator cannot
    traverse, and "she is not answering" was the wrong diagnosis.
  - Verify refuses to run as root over a tree owned by someone else. It runs
    make build and make test in place, and root-owned artifacts break his next
    ordinary make.
  - A rollback no longer reverts config_files. That undid every config edit
    since the last apply, phraser.model_path among them.
  - The verify-failure path no longer reports rolled_back for a compile error.
  - waitHealthy caps each attempt at the remaining budget, so a 90s timeout
    cannot run to 99s.
  - tail cuts on a rune boundary. Russian test names showed the seam.
  - The claim that mavend does not import internal/update is replaced with
    what is enforced: mavend constructs no Updater and nothing can call Apply.
  - snapshot_dir inside source_dir is refused. It landed in the build context.

Found in review of #69.
This commit is contained in:
kami
2026-08-01 14:06:00 +04:00
parent 7f42cc73be
commit 810076451f
18 changed files with 683 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/update/update.go
+115 -12
View File
@@ -10,12 +10,16 @@
// - It is never automatic and never on a timer. There is no checker, no
// channel, no "check for updates" call and nothing that fires from the tick
// loop. Apply runs exactly when a human runs cmd/mavupdate on the box.
// - The daemon cannot update itself. mavend does not import this package and
// there is no IPC method and no web route that reaches it, so no act, no
// intent, no tool and no LLM output can start an update. The trigger needs
// shell access to the host, which is a strictly higher bar than the step-up
// passkey gate that guards /tools — an update is not a thing to expose to
// anything reachable over the network.
// - The daemon cannot update itself. mavend never constructs an Updater and
// nothing in the daemon can call Apply: there is no IPC method and no web
// route that reaches this package, so no act, no intent, no tool and no LLM
// output can start an update. (The package IS linked into mavend, via
// internal/config, which calls Config.Validate so a bad update block is
// caught at daemon startup rather than on the night it is needed. Linked is
// not reachable — the guarantee is the absent caller, not an absent
// import.) The trigger needs shell access to the host, which is a strictly
// higher bar than the step-up passkey gate that guards /tools — an update
// is not a thing to expose to anything reachable over the network.
// - It does not fetch code. Nothing here talks to a release server, a
// registry, or GitHub. The new version is whatever is in the working tree
// the operator points it at, which he pulled himself. Downloading and
@@ -32,13 +36,18 @@
//
// # The order of operations, and why
//
// Apply is: health-check the CURRENT daemon → build → test → snapshot → install
// → restart → health-check → rollback on any failure.
// Apply is: health-check the CURRENT daemon → snapshot → build → test →
// install → restart → health-check → rollback on any failure.
//
// The first health check is not ceremony. If she is already not answering, a
// failed update and a broken box are indistinguishable afterwards, and the
// rollback has nothing to prove itself against — so Apply refuses to start.
//
// The snapshot comes before the build, not after, and the comment in apply.go
// spells out why: `make build` writes into the working tree, which on the
// docker deployment IS the install dir, so a snapshot taken after it would
// snapshot the new artifacts.
//
// Build and test run BEFORE anything is written to the install dir, so a broken
// tree costs nothing but time. Install is per-file write-temp-then-rename, so a
// crash mid-install leaves whole files, not half ones.
@@ -49,6 +58,24 @@
// a migration, and does not need the update to have gotten far enough to leave
// a working anything behind.
//
// # What the snapshot has to cover
//
// A rollback is only real if it puts back the thing the restart command
// deploys. For a bare-metal layout that is the binaries in InstallDir. For the
// docker layout it is not: the image is built from the source tree, and the
// host binaries never enter it. Restoring binaries there rebuilds the same bad
// image and burns a second health timeout proving it. So a deployment that
// rebuilds from source must say how the source is put back
// (Config.SourceRollback), and one that cannot say is refused by Validate
// rather than discovering it during the one rollback that mattered.
//
// # What an update is not
//
// It is not turn-safe. Nothing quiesces the daemon first: the restart kills the
// process mid-utterance if one is in flight. The model swap in
// internal/phraser drains, because a swap is a routine operation on a running
// box; an update is a deliberate restart and the operator picked the moment.
//
// # What is out of scope on purpose
//
// The database is not snapshotted or rolled back. It is encrypted, live, and
@@ -84,6 +111,26 @@ var (
// so the previous snapshot was restored. Wraps the underlying failure.
ErrRolledBack = errors.New("update: rolled back")
// ErrSourceRollback — the deployment rebuilds from source, and nothing in
// the config says how to put the source back. Refused at Validate: see
// Config.SourceRollback.
ErrSourceRollback = errors.New("update: this deployment rebuilds from source and has no way to roll the source back")
// ErrDirtyTree — source_rollback is "git" and the working tree has
// uncommitted changes, so the recorded commit does not describe what is
// deployed and a checkout would throw work away. Refused before anything is
// built.
ErrDirtyTree = errors.New("update: the source tree has uncommitted changes — commit or stash them first")
// ErrRootOnHisTree — running as root over a tree owned by somebody else.
// Refused: Verify runs `make build` and `make test` in SourceDir, and as
// root that leaves root-owned binaries, object files and a build cache in
// his working tree. His next ordinary `make` then fails, so one root apply
// breaks the normal build. This fires easily, because the documented health
// socket lives under a root-only directory and sudo is the obvious way past
// that.
ErrRootOnHisTree = errors.New("update: refusing to build someone else's tree as root — it would leave root-owned artifacts and break his next make")
// ErrRollbackFailed — the worst case: the new build failed AND the restore
// did not bring her back. The operator has to fix the box by hand; the
// snapshot directory is named in the result so he knows what to copy.
@@ -105,18 +152,47 @@ type Config struct {
InstallDir string `json:"install_dir"`
// SnapshotDir — where the pre-install copies live. Must not be inside
// InstallDir: a restore reading from a directory the install is writing to
// is not a restore.
// InstallDir or SourceDir: a restore reading from a directory the install is
// writing to is not a restore, and a snapshot dir inside the source tree
// lands in the docker build context and in whatever make and git do there.
SnapshotDir string `json:"snapshot_dir"`
// SourceRollback — how the SOURCE is put back when the deployment rebuilds
// from it. "" means it is not, which is only valid when the built binaries
// are what gets deployed.
//
// This exists because of what a rollback has to undo, which is not always
// the binaries. When RestartCmd is `docker compose up -d --build`, the image
// is built by the Dockerfile from cmd/ and internal/, and the host binaries
// are excluded by .dockerignore. Restoring them then restores bytes nothing
// reads: the restart rebuilds the same bad image from the same bad source,
// and the box stays down through two health timeouts for no reason.
//
// "git" makes the source part of the snapshot: the commit is recorded before
// the update and a rollback checks it back out before restarting. It
// requires a clean tree, because a recorded commit does not describe a dirty
// one and a forced checkout would throw uncommitted work away.
//
// Validate refuses a build-from-source deployment (SourceDir == InstallDir)
// that leaves this empty, rather than letting the operator find out during
// the one rollback he needed.
SourceRollback string `json:"source_rollback,omitempty"`
// Binaries — the artifact names to snapshot and install, relative to
// SourceDir (built) and InstallDir (deployed). Listed explicitly rather than
// globbed so a stray file in the tree never gets deployed.
Binaries []string `json:"binaries"`
// ConfigFiles — extra files to snapshot alongside the binaries, relative to
// InstallDir. Snapshotted, never overwritten by an install: the operator's
// config is not something an update gets to replace.
// InstallDir. Snapshotted and never written back: not by an install, and
// not by a rollback either. The operator's config is not something an update
// gets to replace, and a rollback that reverted it would silently undo every
// edit since the last apply. The copies are in the snapshot dir if he wants
// one back.
//
// The exception is source_rollback "git": a checkout moves every tracked
// file, config included. That is the same rollback the deployment needs to
// work at all, so on that shape a config edit belongs in a commit.
ConfigFiles []string `json:"config_files,omitempty"`
// RestartCmd — how this deployment restarts mavend, e.g.
@@ -156,6 +232,17 @@ func (c Config) Validate() error {
if within(c.SnapshotDir, c.InstallDir) {
return fmt.Errorf("update: snapshot_dir %q is inside install_dir %q — a restore must not read from what the install writes", c.SnapshotDir, c.InstallDir)
}
if within(c.SnapshotDir, c.SourceDir) {
return fmt.Errorf("update: snapshot_dir %q is inside source_dir %q — snapshots would land in the build context, and in whatever make and git do to that tree", c.SnapshotDir, c.SourceDir)
}
switch c.SourceRollback {
case "", "git":
default:
return fmt.Errorf("update: source_rollback %q is not a thing — use \"git\" or leave it out", c.SourceRollback)
}
if c.buildsFromSource() && c.SourceRollback == "" {
return fmt.Errorf("%w: source_dir and install_dir are both %q, so the restart deploys the tree and a restore of the binaries would undo nothing. Set \"source_rollback\": \"git\", or split the layout so install_dir holds what actually runs", ErrSourceRollback, c.SourceDir)
}
if len(c.Binaries) == 0 {
return errors.New("update: binaries is empty — nothing to install")
}
@@ -186,6 +273,13 @@ func (c Config) withDefaults() Config {
return c
}
// buildsFromSource — the deployment whose restart command rebuilds from the
// tree, which is what SourceDir == InstallDir means in practice (install is a
// no-op copy and the artifacts that matter are produced inside the image).
func (c Config) buildsFromSource() bool {
return filepath.Clean(c.SourceDir) == filepath.Clean(c.InstallDir)
}
func (c Config) healthTimeout() time.Duration {
return time.Duration(c.HealthTimeoutSec) * time.Second
}
@@ -234,6 +328,9 @@ type Updater struct {
health HealthCheck
log Logger
now func() time.Time
// ids reports the running euid and the owner of a directory. Injected so
// the root-build refusal is testable without a second account.
ids func(dir string) (int, uint32, error)
}
// New builds an Updater. Every seam has a real default; the tests replace them.
@@ -248,6 +345,7 @@ func New(cfg Config, opts ...Option) (*Updater, error) {
health: DialHealth,
log: func(string, ...any) {},
now: time.Now,
ids: realIDs,
}
for _, o := range opts {
o(u)
@@ -266,5 +364,10 @@ func WithClock(f func() time.Time) Option {
return func(u *Updater) { u.now = f }
}
// WithIDs replaces the euid/owner lookup behind the root-build refusal.
func WithIDs(f func(dir string) (int, uint32, error)) Option {
return func(u *Updater) { u.ids = f }
}
// Snapshots lists what is available to roll back to, newest first.
func (u *Updater) Snapshots() ([]Snapshot, error) { return u.store.List() }