Files
Maven/internal/update/update.go
kami 810076451f update: roll back what the restart actually deploys
On the deployment deploy/README.md documents, source_dir and install_dir are
the same tree and the restart command rebuilds the image from it. The
Dockerfile builds from cmd/ and internal/ and .dockerignore keeps the host
binaries out, so restoring the snapshotted binaries restored bytes nothing
reads. A bad commit therefore cost two health timeouts and two image builds
and ended in ErrRollbackFailed with an instruction to copy files back by hand,
which would not have helped either.

A deployment that rebuilds from source now has to say how the source is put
back. source_rollback "git" records the commit before the update and checks it
back out before the rollback restart. It refuses a dirty tree, because the
recorded commit does not describe one and a forced checkout would delete his
work. A build-from-source config that says nothing is refused by Validate, at
startup, rather than at the one rollback that mattered.

Also in this change, all from the same review:

  - MethodPing, the one method a locked daemon answers. Preflight passed on an
    unlocked daemon and the post-restart Presence read failed on a locked one,
    so a good update read as SHE IS PROBABLY DOWN once the env key is gone.
  - A dial failure is reported apart from a read failure. The documented
    socket is under /var/lib/docker, which a non-root operator cannot
    traverse, and "she is not answering" was the wrong diagnosis.
  - Verify refuses to run as root over a tree owned by someone else. It runs
    make build and make test in place, and root-owned artifacts break his next
    ordinary make.
  - A rollback no longer reverts config_files. That undid every config edit
    since the last apply, phraser.model_path among them.
  - The verify-failure path no longer reports rolled_back for a compile error.
  - waitHealthy caps each attempt at the remaining budget, so a 90s timeout
    cannot run to 99s.
  - tail cuts on a rune boundary. Russian test names showed the seam.
  - The claim that mavend does not import internal/update is replaced with
    what is enforced: mavend constructs no Updater and nothing can call Apply.
  - snapshot_dir inside source_dir is refused. It landed in the build context.

Found in review of #69.
2026-08-01 14:06:00 +04:00

374 lines
17 KiB
Go

// Package update applies a new build of Maven to the box she runs on, with a
// verified-before-committed install and an automatic rollback (Vikunja #249).
//
// # What this package refuses to be
//
// This is the highest-risk capability in the backlog — code that changes the
// running system — so the refusals are as much of the design as the features,
// and they are enforced here rather than described in a doc:
//
// - It is never automatic and never on a timer. There is no checker, no
// channel, no "check for updates" call and nothing that fires from the tick
// loop. Apply runs exactly when a human runs cmd/mavupdate on the box.
// - The daemon cannot update itself. mavend never constructs an Updater and
// nothing in the daemon can call Apply: there is no IPC method and no web
// route that reaches this package, so no act, no intent, no tool and no LLM
// output can start an update. (The package IS linked into mavend, via
// internal/config, which calls Config.Validate so a bad update block is
// caught at daemon startup rather than on the night it is needed. Linked is
// not reachable — the guarantee is the absent caller, not an absent
// import.) The trigger needs shell access to the host, which is a strictly
// higher bar than the step-up passkey gate that guards /tools — an update
// is not a thing to expose to anything reachable over the network.
// - It does not fetch code. Nothing here talks to a release server, a
// registry, or GitHub. The new version is whatever is in the working tree
// the operator points it at, which he pulled himself. Downloading and
// running code on the strength of a checksum in the same download is not a
// property we can verify on one box.
// - It does not supervise its own death. The plan asked for an in-process
// crash-loop detector; a process cannot reliably notice that it keeps
// dying, and one that thinks it can is worse than nothing. Restart-on-crash
// belongs to whatever starts mavend (compose `restart: unless-stopped`,
// systemd `Restart=`). What this package guarantees instead is narrower and
// real: within one Apply, the new build is proven to answer before the old
// one is considered replaced, and if it does not answer the old bytes go
// back and are proven to answer again.
//
// # The order of operations, and why
//
// Apply is: health-check the CURRENT daemon → snapshot → build → test →
// install → restart → health-check → rollback on any failure.
//
// The first health check is not ceremony. If she is already not answering, a
// failed update and a broken box are indistinguishable afterwards, and the
// rollback has nothing to prove itself against — so Apply refuses to start.
//
// The snapshot comes before the build, not after, and the comment in apply.go
// spells out why: `make build` writes into the working tree, which on the
// docker deployment IS the install dir, so a snapshot taken after it would
// snapshot the new artifacts.
//
// Build and test run BEFORE anything is written to the install dir, so a broken
// tree costs nothing but time. Install is per-file write-temp-then-rename, so a
// crash mid-install leaves whole files, not half ones.
//
// The rollback path deliberately depends on nothing that just changed: it copies
// byte-for-byte from a snapshot taken before the install and re-runs the same
// restart command. It does not ask the new binary to do anything, does not run
// a migration, and does not need the update to have gotten far enough to leave
// a working anything behind.
//
// # What the snapshot has to cover
//
// A rollback is only real if it puts back the thing the restart command
// deploys. For a bare-metal layout that is the binaries in InstallDir. For the
// docker layout it is not: the image is built from the source tree, and the
// host binaries never enter it. Restoring binaries there rebuilds the same bad
// image and burns a second health timeout proving it. So a deployment that
// rebuilds from source must say how the source is put back
// (Config.SourceRollback), and one that cannot say is refused by Validate
// rather than discovering it during the one rollback that mattered.
//
// # What an update is not
//
// It is not turn-safe. Nothing quiesces the daemon first: the restart kills the
// process mid-utterance if one is in flight. The model swap in
// internal/phraser drains, because a swap is a routine operation on a running
// box; an update is a deliberate restart and the operator picked the moment.
//
// # What is out of scope on purpose
//
// The database is not snapshotted or rolled back. It is encrypted, live, and
// often larger than the disk headroom; a store rolled back under a schema that
// already migrated forward loses writes silently, which is worse than a failed
// update. Schema compatibility is store.Migrate's job. A snapshot here is the
// deployable artifacts only: binaries and config.
package update
import (
"context"
"errors"
"fmt"
"os/exec"
"path/filepath"
"strings"
"time"
)
var (
// ErrNotConfigured — no update block in the config. The capability does not
// exist unless the operator described his own deployment.
ErrNotConfigured = errors.New("update: not configured")
// ErrUnhealthyBefore — the daemon was already not answering when Apply
// started. Refused: see the package comment.
ErrUnhealthyBefore = errors.New("update: the running daemon is not healthy — refusing to update on top of a broken box")
// ErrVerifyFailed — build or test failed. Nothing was installed.
ErrVerifyFailed = errors.New("update: verification failed")
// ErrRolledBack — the new build was installed and did not come up healthy,
// so the previous snapshot was restored. Wraps the underlying failure.
ErrRolledBack = errors.New("update: rolled back")
// ErrSourceRollback — the deployment rebuilds from source, and nothing in
// the config says how to put the source back. Refused at Validate: see
// Config.SourceRollback.
ErrSourceRollback = errors.New("update: this deployment rebuilds from source and has no way to roll the source back")
// ErrDirtyTree — source_rollback is "git" and the working tree has
// uncommitted changes, so the recorded commit does not describe what is
// deployed and a checkout would throw work away. Refused before anything is
// built.
ErrDirtyTree = errors.New("update: the source tree has uncommitted changes — commit or stash them first")
// ErrRootOnHisTree — running as root over a tree owned by somebody else.
// Refused: Verify runs `make build` and `make test` in SourceDir, and as
// root that leaves root-owned binaries, object files and a build cache in
// his working tree. His next ordinary `make` then fails, so one root apply
// breaks the normal build. This fires easily, because the documented health
// socket lives under a root-only directory and sudo is the obvious way past
// that.
ErrRootOnHisTree = errors.New("update: refusing to build someone else's tree as root — it would leave root-owned artifacts and break his next make")
// ErrRollbackFailed — the worst case: the new build failed AND the restore
// did not bring her back. The operator has to fix the box by hand; the
// snapshot directory is named in the result so he knows what to copy.
ErrRollbackFailed = errors.New("update: ROLLBACK FAILED — manual recovery required")
)
// Config — the operator's description of his own deployment. Every path is
// absolute and validated; nothing is guessed, because guessing wrong here means
// overwriting the wrong file.
type Config struct {
// SourceDir — the git working tree to build. The operator pulls it himself;
// this package never fetches.
SourceDir string `json:"source_dir"`
// InstallDir — where the built binaries are copied to. On the docker
// deployment this is the tree the image is built from, so it is usually the
// same as SourceDir and Install is a no-op copy; on a bare-metal deployment
// it is /opt/maven/bin.
InstallDir string `json:"install_dir"`
// SnapshotDir — where the pre-install copies live. Must not be inside
// InstallDir or SourceDir: a restore reading from a directory the install is
// writing to is not a restore, and a snapshot dir inside the source tree
// lands in the docker build context and in whatever make and git do there.
SnapshotDir string `json:"snapshot_dir"`
// SourceRollback — how the SOURCE is put back when the deployment rebuilds
// from it. "" means it is not, which is only valid when the built binaries
// are what gets deployed.
//
// This exists because of what a rollback has to undo, which is not always
// the binaries. When RestartCmd is `docker compose up -d --build`, the image
// is built by the Dockerfile from cmd/ and internal/, and the host binaries
// are excluded by .dockerignore. Restoring them then restores bytes nothing
// reads: the restart rebuilds the same bad image from the same bad source,
// and the box stays down through two health timeouts for no reason.
//
// "git" makes the source part of the snapshot: the commit is recorded before
// the update and a rollback checks it back out before restarting. It
// requires a clean tree, because a recorded commit does not describe a dirty
// one and a forced checkout would throw uncommitted work away.
//
// Validate refuses a build-from-source deployment (SourceDir == InstallDir)
// that leaves this empty, rather than letting the operator find out during
// the one rollback he needed.
SourceRollback string `json:"source_rollback,omitempty"`
// Binaries — the artifact names to snapshot and install, relative to
// SourceDir (built) and InstallDir (deployed). Listed explicitly rather than
// globbed so a stray file in the tree never gets deployed.
Binaries []string `json:"binaries"`
// ConfigFiles — extra files to snapshot alongside the binaries, relative to
// InstallDir. Snapshotted and never written back: not by an install, and
// not by a rollback either. The operator's config is not something an update
// gets to replace, and a rollback that reverted it would silently undo every
// edit since the last apply. The copies are in the snapshot dir if he wants
// one back.
//
// The exception is source_rollback "git": a checkout moves every tracked
// file, config included. That is the same rollback the deployment needs to
// work at all, so on that shape a config edit belongs in a commit.
ConfigFiles []string `json:"config_files,omitempty"`
// RestartCmd — how this deployment restarts mavend, e.g.
// ["docker","compose","up","-d","--build","mavend"] or
// ["systemctl","restart","mavend"]. Run in SourceDir. Required: there is no
// portable default and picking one would mean restarting the wrong thing.
RestartCmd []string `json:"restart_cmd"`
// HealthSocket — mavend's IPC socket, used to prove she answers after a
// restart. Required: without a health check there is no signal to roll back
// on, and an update that cannot detect its own failure is not what this
// package is for.
HealthSocket string `json:"health_socket"`
// HealthTimeoutSec — how long to wait for the restarted daemon to answer.
// Default 90s; she loads a 1.7B on boot, so this is not a couple of seconds.
HealthTimeoutSec int `json:"health_timeout_sec,omitempty"`
// VerifyTimeoutMin — cap on `make build` + `make test`. Default 20m.
VerifyTimeoutMin int `json:"verify_timeout_min,omitempty"`
// KeepSnapshots — how many snapshots to retain. Default 5, minimum 1: the
// most recent one is the rollback target and is never pruned.
KeepSnapshots int `json:"keep_snapshots,omitempty"`
}
// Validate — fail at startup, not halfway through an install.
func (c Config) Validate() error {
if c.SourceDir == "" || c.InstallDir == "" || c.SnapshotDir == "" {
return errors.New("update: source_dir, install_dir and snapshot_dir are all required")
}
for _, p := range []string{c.SourceDir, c.InstallDir, c.SnapshotDir} {
if !filepath.IsAbs(p) {
return fmt.Errorf("update: %q must be an absolute path", p)
}
}
if within(c.SnapshotDir, c.InstallDir) {
return fmt.Errorf("update: snapshot_dir %q is inside install_dir %q — a restore must not read from what the install writes", c.SnapshotDir, c.InstallDir)
}
if within(c.SnapshotDir, c.SourceDir) {
return fmt.Errorf("update: snapshot_dir %q is inside source_dir %q — snapshots would land in the build context, and in whatever make and git do to that tree", c.SnapshotDir, c.SourceDir)
}
switch c.SourceRollback {
case "", "git":
default:
return fmt.Errorf("update: source_rollback %q is not a thing — use \"git\" or leave it out", c.SourceRollback)
}
if c.buildsFromSource() && c.SourceRollback == "" {
return fmt.Errorf("%w: source_dir and install_dir are both %q, so the restart deploys the tree and a restore of the binaries would undo nothing. Set \"source_rollback\": \"git\", or split the layout so install_dir holds what actually runs", ErrSourceRollback, c.SourceDir)
}
if len(c.Binaries) == 0 {
return errors.New("update: binaries is empty — nothing to install")
}
for _, b := range append(append([]string{}, c.Binaries...), c.ConfigFiles...) {
if filepath.IsAbs(b) || strings.Contains(b, "..") {
return fmt.Errorf("update: %q must be a plain relative name", b)
}
}
if len(c.RestartCmd) == 0 {
return errors.New("update: restart_cmd is required — there is no safe default for restarting someone else's deployment")
}
if c.HealthSocket == "" {
return errors.New("update: health_socket is required — an update that cannot check its own result cannot roll back on failure")
}
return nil
}
func (c Config) withDefaults() Config {
if c.HealthTimeoutSec <= 0 {
c.HealthTimeoutSec = 90
}
if c.VerifyTimeoutMin <= 0 {
c.VerifyTimeoutMin = 20
}
if c.KeepSnapshots < 1 {
c.KeepSnapshots = 5
}
return c
}
// buildsFromSource — the deployment whose restart command rebuilds from the
// tree, which is what SourceDir == InstallDir means in practice (install is a
// no-op copy and the artifacts that matter are produced inside the image).
func (c Config) buildsFromSource() bool {
return filepath.Clean(c.SourceDir) == filepath.Clean(c.InstallDir)
}
func (c Config) healthTimeout() time.Duration {
return time.Duration(c.HealthTimeoutSec) * time.Second
}
func (c Config) verifyTimeout() time.Duration {
return time.Duration(c.VerifyTimeoutMin) * time.Minute
}
// within reports whether p is dir or lives under it.
func within(p, dir string) bool {
p, dir = filepath.Clean(p), filepath.Clean(dir)
if p == dir {
return true
}
rel, err := filepath.Rel(dir, p)
return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
}
// Runner runs one command and returns its combined output. Injected so the
// tests can drive build/test/restart failures without a toolchain, a container
// or a real daemon to break.
type Runner func(ctx context.Context, dir string, argv []string) (string, error)
// ExecRunner is the real one.
func ExecRunner(ctx context.Context, dir string, argv []string) (string, error) {
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
return string(out), err
}
// HealthCheck proves the daemon at socket answers. Injected for the same reason
// as Runner.
type HealthCheck func(ctx context.Context, socket string) error
// Logger receives one line per step. The CLI prints these as they happen: an
// update that goes quiet for four minutes during `make test` reads as a hang.
type Logger func(format string, args ...any)
// Updater is the whole capability. Construct with New and call Apply or
// Rollback; there is no background goroutine and nothing starts on its own.
type Updater struct {
cfg Config
store *Store
run Runner
health HealthCheck
log Logger
now func() time.Time
// ids reports the running euid and the owner of a directory. Injected so
// the root-build refusal is testable without a second account.
ids func(dir string) (int, uint32, error)
}
// New builds an Updater. Every seam has a real default; the tests replace them.
func New(cfg Config, opts ...Option) (*Updater, error) {
if err := cfg.Validate(); err != nil {
return nil, err
}
u := &Updater{
cfg: cfg.withDefaults(),
store: &Store{Dir: cfg.SnapshotDir},
run: ExecRunner,
health: DialHealth,
log: func(string, ...any) {},
now: time.Now,
ids: realIDs,
}
for _, o := range opts {
o(u)
}
u.store.now = u.now
return u, nil
}
// Option — a constructor seam.
type Option func(*Updater)
func WithRunner(r Runner) Option { return func(u *Updater) { u.run = r } }
func WithHealth(h HealthCheck) Option { return func(u *Updater) { u.health = h } }
func WithLogger(l Logger) Option { return func(u *Updater) { u.log = l } }
func WithClock(f func() time.Time) Option {
return func(u *Updater) { u.now = f }
}
// WithIDs replaces the euid/owner lookup behind the root-build refusal.
func WithIDs(f func(dir string) (int, uint32, error)) Option {
return func(u *Updater) { u.ids = f }
}
// Snapshots lists what is available to roll back to, newest first.
func (u *Updater) Snapshots() ([]Snapshot, error) { return u.store.List() }