Files
Maven/internal/update/update.go
T
kami be066a4b04 Deploy a new build with verification and automatic rollback (#249)
internal/update applies a new build of Maven to the box she runs on and
undoes it when the new build does not come up. cmd/mavupdate is the only
trigger: a CLI the owner runs on the host.

Apply is health-check the running daemon, snapshot the deployed artifacts,
make build, make test, install, restart, health-check — and restore the
snapshot on any failure. The order is load-bearing:

  - The preflight health check refuses to update a daemon that is already
    not answering. Without a working baseline, a failed update and a box
    that was already broken are indistinguishable, and the rollback has
    nothing to prove itself against.
  - The snapshot is taken BEFORE the build, because make build writes its
    binaries into the working tree and on the docker deployment the tree
    is the install dir — snapshotting afterwards would snapshot the new
    artifacts and leave nothing to roll back to.
  - Verification is make build plus make test, before anything is
    deployed, so a broken tree costs time and nothing else. A failed
    verify also puts the tree's artifacts back, so a later restart by
    hand cannot deploy code that failed its own tests.
  - The rollback depends on nothing that just changed: byte-for-byte
    copies out of the snapshot dir, sha256-verified on the way in, and
    the same restart command. No build, no migration, no cooperation from
    the code being replaced. It also runs on an uncancellable context —
    a rollback interrupted halfway is worse than the failure that caused
    it. When the restore itself fails it says so and names the directory
    to copy back by hand rather than reporting a tidy rollback.

Off unless configured, and the refusals are code, not documentation. The
daemon does not import this package: there is no IPC method, no web route,
no timer and no act that can start an update, so nothing Maven says or
routes reaches it. Nothing fetches code — the new version is whatever the
owner pulled into the tree. The plan's release checker, auto-update
channel and in-process crash-loop supervisor are deliberately absent; a
process cannot reliably notice that it keeps dying, and restart-on-crash
belongs to compose or systemd. The database is never snapshotted or rolled
back; schema compatibility stays store.Migrate's job.

The config is refused at load without a health socket, since an update
that cannot check its own result cannot roll back, and refused when the
snapshot dir is inside the install dir, since a restore must not read from
what the install writes.

Vikunja #249
2026-08-01 04:09:30 +04:00

271 lines
11 KiB
Go

// Package update applies a new build of Maven to the box she runs on, with a
// verified-before-committed install and an automatic rollback (Vikunja #249).
//
// # What this package refuses to be
//
// This is the highest-risk capability in the backlog — code that changes the
// running system — so the refusals are as much of the design as the features,
// and they are enforced here rather than described in a doc:
//
// - It is never automatic and never on a timer. There is no checker, no
// channel, no "check for updates" call and nothing that fires from the tick
// loop. Apply runs exactly when a human runs cmd/mavupdate on the box.
// - The daemon cannot update itself. mavend does not import this package and
// there is no IPC method and no web route that reaches it, so no act, no
// intent, no tool and no LLM output can start an update. The trigger needs
// shell access to the host, which is a strictly higher bar than the step-up
// passkey gate that guards /tools — an update is not a thing to expose to
// anything reachable over the network.
// - It does not fetch code. Nothing here talks to a release server, a
// registry, or GitHub. The new version is whatever is in the working tree
// the operator points it at, which he pulled himself. Downloading and
// running code on the strength of a checksum in the same download is not a
// property we can verify on one box.
// - It does not supervise its own death. The plan asked for an in-process
// crash-loop detector; a process cannot reliably notice that it keeps
// dying, and one that thinks it can is worse than nothing. Restart-on-crash
// belongs to whatever starts mavend (compose `restart: unless-stopped`,
// systemd `Restart=`). What this package guarantees instead is narrower and
// real: within one Apply, the new build is proven to answer before the old
// one is considered replaced, and if it does not answer the old bytes go
// back and are proven to answer again.
//
// # The order of operations, and why
//
// Apply is: health-check the CURRENT daemon → build → test → snapshot → install
// → restart → health-check → rollback on any failure.
//
// The first health check is not ceremony. If she is already not answering, a
// failed update and a broken box are indistinguishable afterwards, and the
// rollback has nothing to prove itself against — so Apply refuses to start.
//
// Build and test run BEFORE anything is written to the install dir, so a broken
// tree costs nothing but time. Install is per-file write-temp-then-rename, so a
// crash mid-install leaves whole files, not half ones.
//
// The rollback path deliberately depends on nothing that just changed: it copies
// byte-for-byte from a snapshot taken before the install and re-runs the same
// restart command. It does not ask the new binary to do anything, does not run
// a migration, and does not need the update to have gotten far enough to leave
// a working anything behind.
//
// # What is out of scope on purpose
//
// The database is not snapshotted or rolled back. It is encrypted, live, and
// often larger than the disk headroom; a store rolled back under a schema that
// already migrated forward loses writes silently, which is worse than a failed
// update. Schema compatibility is store.Migrate's job. A snapshot here is the
// deployable artifacts only: binaries and config.
package update
import (
"context"
"errors"
"fmt"
"os/exec"
"path/filepath"
"strings"
"time"
)
var (
// ErrNotConfigured — no update block in the config. The capability does not
// exist unless the operator described his own deployment.
ErrNotConfigured = errors.New("update: not configured")
// ErrUnhealthyBefore — the daemon was already not answering when Apply
// started. Refused: see the package comment.
ErrUnhealthyBefore = errors.New("update: the running daemon is not healthy — refusing to update on top of a broken box")
// ErrVerifyFailed — build or test failed. Nothing was installed.
ErrVerifyFailed = errors.New("update: verification failed")
// ErrRolledBack — the new build was installed and did not come up healthy,
// so the previous snapshot was restored. Wraps the underlying failure.
ErrRolledBack = errors.New("update: rolled back")
// ErrRollbackFailed — the worst case: the new build failed AND the restore
// did not bring her back. The operator has to fix the box by hand; the
// snapshot directory is named in the result so he knows what to copy.
ErrRollbackFailed = errors.New("update: ROLLBACK FAILED — manual recovery required")
)
// Config — the operator's description of his own deployment. Every path is
// absolute and validated; nothing is guessed, because guessing wrong here means
// overwriting the wrong file.
type Config struct {
// SourceDir — the git working tree to build. The operator pulls it himself;
// this package never fetches.
SourceDir string `json:"source_dir"`
// InstallDir — where the built binaries are copied to. On the docker
// deployment this is the tree the image is built from, so it is usually the
// same as SourceDir and Install is a no-op copy; on a bare-metal deployment
// it is /opt/maven/bin.
InstallDir string `json:"install_dir"`
// SnapshotDir — where the pre-install copies live. Must not be inside
// InstallDir: a restore reading from a directory the install is writing to
// is not a restore.
SnapshotDir string `json:"snapshot_dir"`
// Binaries — the artifact names to snapshot and install, relative to
// SourceDir (built) and InstallDir (deployed). Listed explicitly rather than
// globbed so a stray file in the tree never gets deployed.
Binaries []string `json:"binaries"`
// ConfigFiles — extra files to snapshot alongside the binaries, relative to
// InstallDir. Snapshotted, never overwritten by an install: the operator's
// config is not something an update gets to replace.
ConfigFiles []string `json:"config_files,omitempty"`
// RestartCmd — how this deployment restarts mavend, e.g.
// ["docker","compose","up","-d","--build","mavend"] or
// ["systemctl","restart","mavend"]. Run in SourceDir. Required: there is no
// portable default and picking one would mean restarting the wrong thing.
RestartCmd []string `json:"restart_cmd"`
// HealthSocket — mavend's IPC socket, used to prove she answers after a
// restart. Required: without a health check there is no signal to roll back
// on, and an update that cannot detect its own failure is not what this
// package is for.
HealthSocket string `json:"health_socket"`
// HealthTimeoutSec — how long to wait for the restarted daemon to answer.
// Default 90s; she loads a 1.7B on boot, so this is not a couple of seconds.
HealthTimeoutSec int `json:"health_timeout_sec,omitempty"`
// VerifyTimeoutMin — cap on `make build` + `make test`. Default 20m.
VerifyTimeoutMin int `json:"verify_timeout_min,omitempty"`
// KeepSnapshots — how many snapshots to retain. Default 5, minimum 1: the
// most recent one is the rollback target and is never pruned.
KeepSnapshots int `json:"keep_snapshots,omitempty"`
}
// Validate — fail at startup, not halfway through an install.
func (c Config) Validate() error {
if c.SourceDir == "" || c.InstallDir == "" || c.SnapshotDir == "" {
return errors.New("update: source_dir, install_dir and snapshot_dir are all required")
}
for _, p := range []string{c.SourceDir, c.InstallDir, c.SnapshotDir} {
if !filepath.IsAbs(p) {
return fmt.Errorf("update: %q must be an absolute path", p)
}
}
if within(c.SnapshotDir, c.InstallDir) {
return fmt.Errorf("update: snapshot_dir %q is inside install_dir %q — a restore must not read from what the install writes", c.SnapshotDir, c.InstallDir)
}
if len(c.Binaries) == 0 {
return errors.New("update: binaries is empty — nothing to install")
}
for _, b := range append(append([]string{}, c.Binaries...), c.ConfigFiles...) {
if filepath.IsAbs(b) || strings.Contains(b, "..") {
return fmt.Errorf("update: %q must be a plain relative name", b)
}
}
if len(c.RestartCmd) == 0 {
return errors.New("update: restart_cmd is required — there is no safe default for restarting someone else's deployment")
}
if c.HealthSocket == "" {
return errors.New("update: health_socket is required — an update that cannot check its own result cannot roll back on failure")
}
return nil
}
func (c Config) withDefaults() Config {
if c.HealthTimeoutSec <= 0 {
c.HealthTimeoutSec = 90
}
if c.VerifyTimeoutMin <= 0 {
c.VerifyTimeoutMin = 20
}
if c.KeepSnapshots < 1 {
c.KeepSnapshots = 5
}
return c
}
func (c Config) healthTimeout() time.Duration {
return time.Duration(c.HealthTimeoutSec) * time.Second
}
func (c Config) verifyTimeout() time.Duration {
return time.Duration(c.VerifyTimeoutMin) * time.Minute
}
// within reports whether p is dir or lives under it.
func within(p, dir string) bool {
p, dir = filepath.Clean(p), filepath.Clean(dir)
if p == dir {
return true
}
rel, err := filepath.Rel(dir, p)
return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
}
// Runner runs one command and returns its combined output. Injected so the
// tests can drive build/test/restart failures without a toolchain, a container
// or a real daemon to break.
type Runner func(ctx context.Context, dir string, argv []string) (string, error)
// ExecRunner is the real one.
func ExecRunner(ctx context.Context, dir string, argv []string) (string, error) {
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
cmd.Dir = dir
out, err := cmd.CombinedOutput()
return string(out), err
}
// HealthCheck proves the daemon at socket answers. Injected for the same reason
// as Runner.
type HealthCheck func(ctx context.Context, socket string) error
// Logger receives one line per step. The CLI prints these as they happen: an
// update that goes quiet for four minutes during `make test` reads as a hang.
type Logger func(format string, args ...any)
// Updater is the whole capability. Construct with New and call Apply or
// Rollback; there is no background goroutine and nothing starts on its own.
type Updater struct {
cfg Config
store *Store
run Runner
health HealthCheck
log Logger
now func() time.Time
}
// New builds an Updater. Every seam has a real default; the tests replace them.
func New(cfg Config, opts ...Option) (*Updater, error) {
if err := cfg.Validate(); err != nil {
return nil, err
}
u := &Updater{
cfg: cfg.withDefaults(),
store: &Store{Dir: cfg.SnapshotDir},
run: ExecRunner,
health: DialHealth,
log: func(string, ...any) {},
now: time.Now,
}
for _, o := range opts {
o(u)
}
u.store.now = u.now
return u, nil
}
// Option — a constructor seam.
type Option func(*Updater)
func WithRunner(r Runner) Option { return func(u *Updater) { u.run = r } }
func WithHealth(h HealthCheck) Option { return func(u *Updater) { u.health = h } }
func WithLogger(l Logger) Option { return func(u *Updater) { u.log = l } }
func WithClock(f func() time.Time) Option {
return func(u *Updater) { u.now = f }
}
// Snapshots lists what is available to roll back to, newest first.
func (u *Updater) Snapshots() ([]Snapshot, error) { return u.store.List() }