810076451f
On the deployment deploy/README.md documents, source_dir and install_dir are
the same tree and the restart command rebuilds the image from it. The
Dockerfile builds from cmd/ and internal/ and .dockerignore keeps the host
binaries out, so restoring the snapshotted binaries restored bytes nothing
reads. A bad commit therefore cost two health timeouts and two image builds
and ended in ErrRollbackFailed with an instruction to copy files back by hand,
which would not have helped either.
A deployment that rebuilds from source now has to say how the source is put
back. source_rollback "git" records the commit before the update and checks it
back out before the rollback restart. It refuses a dirty tree, because the
recorded commit does not describe one and a forced checkout would delete his
work. A build-from-source config that says nothing is refused by Validate, at
startup, rather than at the one rollback that mattered.
Also in this change, all from the same review:
- MethodPing, the one method a locked daemon answers. Preflight passed on an
unlocked daemon and the post-restart Presence read failed on a locked one,
so a good update read as SHE IS PROBABLY DOWN once the env key is gone.
- A dial failure is reported apart from a read failure. The documented
socket is under /var/lib/docker, which a non-root operator cannot
traverse, and "she is not answering" was the wrong diagnosis.
- Verify refuses to run as root over a tree owned by someone else. It runs
make build and make test in place, and root-owned artifacts break his next
ordinary make.
- A rollback no longer reverts config_files. That undid every config edit
since the last apply, phraser.model_path among them.
- The verify-failure path no longer reports rolled_back for a compile error.
- waitHealthy caps each attempt at the remaining budget, so a 90s timeout
cannot run to 99s.
- tail cuts on a rune boundary. Russian test names showed the seam.
- The claim that mavend does not import internal/update is replaced with
what is enforced: mavend constructs no Updater and nothing can call Apply.
- snapshot_dir inside source_dir is refused. It landed in the build context.
Found in review of #69.
374 lines
17 KiB
Go
374 lines
17 KiB
Go
// Package update applies a new build of Maven to the box she runs on, with a
|
|
// verified-before-committed install and an automatic rollback (Vikunja #249).
|
|
//
|
|
// # What this package refuses to be
|
|
//
|
|
// This is the highest-risk capability in the backlog — code that changes the
|
|
// running system — so the refusals are as much of the design as the features,
|
|
// and they are enforced here rather than described in a doc:
|
|
//
|
|
// - It is never automatic and never on a timer. There is no checker, no
|
|
// channel, no "check for updates" call and nothing that fires from the tick
|
|
// loop. Apply runs exactly when a human runs cmd/mavupdate on the box.
|
|
// - The daemon cannot update itself. mavend never constructs an Updater and
|
|
// nothing in the daemon can call Apply: there is no IPC method and no web
|
|
// route that reaches this package, so no act, no intent, no tool and no LLM
|
|
// output can start an update. (The package IS linked into mavend, via
|
|
// internal/config, which calls Config.Validate so a bad update block is
|
|
// caught at daemon startup rather than on the night it is needed. Linked is
|
|
// not reachable — the guarantee is the absent caller, not an absent
|
|
// import.) The trigger needs shell access to the host, which is a strictly
|
|
// higher bar than the step-up passkey gate that guards /tools — an update
|
|
// is not a thing to expose to anything reachable over the network.
|
|
// - It does not fetch code. Nothing here talks to a release server, a
|
|
// registry, or GitHub. The new version is whatever is in the working tree
|
|
// the operator points it at, which he pulled himself. Downloading and
|
|
// running code on the strength of a checksum in the same download is not a
|
|
// property we can verify on one box.
|
|
// - It does not supervise its own death. The plan asked for an in-process
|
|
// crash-loop detector; a process cannot reliably notice that it keeps
|
|
// dying, and one that thinks it can is worse than nothing. Restart-on-crash
|
|
// belongs to whatever starts mavend (compose `restart: unless-stopped`,
|
|
// systemd `Restart=`). What this package guarantees instead is narrower and
|
|
// real: within one Apply, the new build is proven to answer before the old
|
|
// one is considered replaced, and if it does not answer the old bytes go
|
|
// back and are proven to answer again.
|
|
//
|
|
// # The order of operations, and why
|
|
//
|
|
// Apply is: health-check the CURRENT daemon → snapshot → build → test →
|
|
// install → restart → health-check → rollback on any failure.
|
|
//
|
|
// The first health check is not ceremony. If she is already not answering, a
|
|
// failed update and a broken box are indistinguishable afterwards, and the
|
|
// rollback has nothing to prove itself against — so Apply refuses to start.
|
|
//
|
|
// The snapshot comes before the build, not after, and the comment in apply.go
|
|
// spells out why: `make build` writes into the working tree, which on the
|
|
// docker deployment IS the install dir, so a snapshot taken after it would
|
|
// snapshot the new artifacts.
|
|
//
|
|
// Build and test run BEFORE anything is written to the install dir, so a broken
|
|
// tree costs nothing but time. Install is per-file write-temp-then-rename, so a
|
|
// crash mid-install leaves whole files, not half ones.
|
|
//
|
|
// The rollback path deliberately depends on nothing that just changed: it copies
|
|
// byte-for-byte from a snapshot taken before the install and re-runs the same
|
|
// restart command. It does not ask the new binary to do anything, does not run
|
|
// a migration, and does not need the update to have gotten far enough to leave
|
|
// a working anything behind.
|
|
//
|
|
// # What the snapshot has to cover
|
|
//
|
|
// A rollback is only real if it puts back the thing the restart command
|
|
// deploys. For a bare-metal layout that is the binaries in InstallDir. For the
|
|
// docker layout it is not: the image is built from the source tree, and the
|
|
// host binaries never enter it. Restoring binaries there rebuilds the same bad
|
|
// image and burns a second health timeout proving it. So a deployment that
|
|
// rebuilds from source must say how the source is put back
|
|
// (Config.SourceRollback), and one that cannot say is refused by Validate
|
|
// rather than discovering it during the one rollback that mattered.
|
|
//
|
|
// # What an update is not
|
|
//
|
|
// It is not turn-safe. Nothing quiesces the daemon first: the restart kills the
|
|
// process mid-utterance if one is in flight. The model swap in
|
|
// internal/phraser drains, because a swap is a routine operation on a running
|
|
// box; an update is a deliberate restart and the operator picked the moment.
|
|
//
|
|
// # What is out of scope on purpose
|
|
//
|
|
// The database is not snapshotted or rolled back. It is encrypted, live, and
|
|
// often larger than the disk headroom; a store rolled back under a schema that
|
|
// already migrated forward loses writes silently, which is worse than a failed
|
|
// update. Schema compatibility is store.Migrate's job. A snapshot here is the
|
|
// deployable artifacts only: binaries and config.
|
|
package update
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strings"
|
|
"time"
|
|
)
|
|
|
|
var (
|
|
// ErrNotConfigured — no update block in the config. The capability does not
|
|
// exist unless the operator described his own deployment.
|
|
ErrNotConfigured = errors.New("update: not configured")
|
|
|
|
// ErrUnhealthyBefore — the daemon was already not answering when Apply
|
|
// started. Refused: see the package comment.
|
|
ErrUnhealthyBefore = errors.New("update: the running daemon is not healthy — refusing to update on top of a broken box")
|
|
|
|
// ErrVerifyFailed — build or test failed. Nothing was installed.
|
|
ErrVerifyFailed = errors.New("update: verification failed")
|
|
|
|
// ErrRolledBack — the new build was installed and did not come up healthy,
|
|
// so the previous snapshot was restored. Wraps the underlying failure.
|
|
ErrRolledBack = errors.New("update: rolled back")
|
|
|
|
// ErrSourceRollback — the deployment rebuilds from source, and nothing in
|
|
// the config says how to put the source back. Refused at Validate: see
|
|
// Config.SourceRollback.
|
|
ErrSourceRollback = errors.New("update: this deployment rebuilds from source and has no way to roll the source back")
|
|
|
|
// ErrDirtyTree — source_rollback is "git" and the working tree has
|
|
// uncommitted changes, so the recorded commit does not describe what is
|
|
// deployed and a checkout would throw work away. Refused before anything is
|
|
// built.
|
|
ErrDirtyTree = errors.New("update: the source tree has uncommitted changes — commit or stash them first")
|
|
|
|
// ErrRootOnHisTree — running as root over a tree owned by somebody else.
|
|
// Refused: Verify runs `make build` and `make test` in SourceDir, and as
|
|
// root that leaves root-owned binaries, object files and a build cache in
|
|
// his working tree. His next ordinary `make` then fails, so one root apply
|
|
// breaks the normal build. This fires easily, because the documented health
|
|
// socket lives under a root-only directory and sudo is the obvious way past
|
|
// that.
|
|
ErrRootOnHisTree = errors.New("update: refusing to build someone else's tree as root — it would leave root-owned artifacts and break his next make")
|
|
|
|
// ErrRollbackFailed — the worst case: the new build failed AND the restore
|
|
// did not bring her back. The operator has to fix the box by hand; the
|
|
// snapshot directory is named in the result so he knows what to copy.
|
|
ErrRollbackFailed = errors.New("update: ROLLBACK FAILED — manual recovery required")
|
|
)
|
|
|
|
// Config — the operator's description of his own deployment. Every path is
|
|
// absolute and validated; nothing is guessed, because guessing wrong here means
|
|
// overwriting the wrong file.
|
|
type Config struct {
|
|
// SourceDir — the git working tree to build. The operator pulls it himself;
|
|
// this package never fetches.
|
|
SourceDir string `json:"source_dir"`
|
|
|
|
// InstallDir — where the built binaries are copied to. On the docker
|
|
// deployment this is the tree the image is built from, so it is usually the
|
|
// same as SourceDir and Install is a no-op copy; on a bare-metal deployment
|
|
// it is /opt/maven/bin.
|
|
InstallDir string `json:"install_dir"`
|
|
|
|
// SnapshotDir — where the pre-install copies live. Must not be inside
|
|
// InstallDir or SourceDir: a restore reading from a directory the install is
|
|
// writing to is not a restore, and a snapshot dir inside the source tree
|
|
// lands in the docker build context and in whatever make and git do there.
|
|
SnapshotDir string `json:"snapshot_dir"`
|
|
|
|
// SourceRollback — how the SOURCE is put back when the deployment rebuilds
|
|
// from it. "" means it is not, which is only valid when the built binaries
|
|
// are what gets deployed.
|
|
//
|
|
// This exists because of what a rollback has to undo, which is not always
|
|
// the binaries. When RestartCmd is `docker compose up -d --build`, the image
|
|
// is built by the Dockerfile from cmd/ and internal/, and the host binaries
|
|
// are excluded by .dockerignore. Restoring them then restores bytes nothing
|
|
// reads: the restart rebuilds the same bad image from the same bad source,
|
|
// and the box stays down through two health timeouts for no reason.
|
|
//
|
|
// "git" makes the source part of the snapshot: the commit is recorded before
|
|
// the update and a rollback checks it back out before restarting. It
|
|
// requires a clean tree, because a recorded commit does not describe a dirty
|
|
// one and a forced checkout would throw uncommitted work away.
|
|
//
|
|
// Validate refuses a build-from-source deployment (SourceDir == InstallDir)
|
|
// that leaves this empty, rather than letting the operator find out during
|
|
// the one rollback he needed.
|
|
SourceRollback string `json:"source_rollback,omitempty"`
|
|
|
|
// Binaries — the artifact names to snapshot and install, relative to
|
|
// SourceDir (built) and InstallDir (deployed). Listed explicitly rather than
|
|
// globbed so a stray file in the tree never gets deployed.
|
|
Binaries []string `json:"binaries"`
|
|
|
|
// ConfigFiles — extra files to snapshot alongside the binaries, relative to
|
|
// InstallDir. Snapshotted and never written back: not by an install, and
|
|
// not by a rollback either. The operator's config is not something an update
|
|
// gets to replace, and a rollback that reverted it would silently undo every
|
|
// edit since the last apply. The copies are in the snapshot dir if he wants
|
|
// one back.
|
|
//
|
|
// The exception is source_rollback "git": a checkout moves every tracked
|
|
// file, config included. That is the same rollback the deployment needs to
|
|
// work at all, so on that shape a config edit belongs in a commit.
|
|
ConfigFiles []string `json:"config_files,omitempty"`
|
|
|
|
// RestartCmd — how this deployment restarts mavend, e.g.
|
|
// ["docker","compose","up","-d","--build","mavend"] or
|
|
// ["systemctl","restart","mavend"]. Run in SourceDir. Required: there is no
|
|
// portable default and picking one would mean restarting the wrong thing.
|
|
RestartCmd []string `json:"restart_cmd"`
|
|
|
|
// HealthSocket — mavend's IPC socket, used to prove she answers after a
|
|
// restart. Required: without a health check there is no signal to roll back
|
|
// on, and an update that cannot detect its own failure is not what this
|
|
// package is for.
|
|
HealthSocket string `json:"health_socket"`
|
|
|
|
// HealthTimeoutSec — how long to wait for the restarted daemon to answer.
|
|
// Default 90s; she loads a 1.7B on boot, so this is not a couple of seconds.
|
|
HealthTimeoutSec int `json:"health_timeout_sec,omitempty"`
|
|
|
|
// VerifyTimeoutMin — cap on `make build` + `make test`. Default 20m.
|
|
VerifyTimeoutMin int `json:"verify_timeout_min,omitempty"`
|
|
|
|
// KeepSnapshots — how many snapshots to retain. Default 5, minimum 1: the
|
|
// most recent one is the rollback target and is never pruned.
|
|
KeepSnapshots int `json:"keep_snapshots,omitempty"`
|
|
}
|
|
|
|
// Validate — fail at startup, not halfway through an install.
|
|
func (c Config) Validate() error {
|
|
if c.SourceDir == "" || c.InstallDir == "" || c.SnapshotDir == "" {
|
|
return errors.New("update: source_dir, install_dir and snapshot_dir are all required")
|
|
}
|
|
for _, p := range []string{c.SourceDir, c.InstallDir, c.SnapshotDir} {
|
|
if !filepath.IsAbs(p) {
|
|
return fmt.Errorf("update: %q must be an absolute path", p)
|
|
}
|
|
}
|
|
if within(c.SnapshotDir, c.InstallDir) {
|
|
return fmt.Errorf("update: snapshot_dir %q is inside install_dir %q — a restore must not read from what the install writes", c.SnapshotDir, c.InstallDir)
|
|
}
|
|
if within(c.SnapshotDir, c.SourceDir) {
|
|
return fmt.Errorf("update: snapshot_dir %q is inside source_dir %q — snapshots would land in the build context, and in whatever make and git do to that tree", c.SnapshotDir, c.SourceDir)
|
|
}
|
|
switch c.SourceRollback {
|
|
case "", "git":
|
|
default:
|
|
return fmt.Errorf("update: source_rollback %q is not a thing — use \"git\" or leave it out", c.SourceRollback)
|
|
}
|
|
if c.buildsFromSource() && c.SourceRollback == "" {
|
|
return fmt.Errorf("%w: source_dir and install_dir are both %q, so the restart deploys the tree and a restore of the binaries would undo nothing. Set \"source_rollback\": \"git\", or split the layout so install_dir holds what actually runs", ErrSourceRollback, c.SourceDir)
|
|
}
|
|
if len(c.Binaries) == 0 {
|
|
return errors.New("update: binaries is empty — nothing to install")
|
|
}
|
|
for _, b := range append(append([]string{}, c.Binaries...), c.ConfigFiles...) {
|
|
if filepath.IsAbs(b) || strings.Contains(b, "..") {
|
|
return fmt.Errorf("update: %q must be a plain relative name", b)
|
|
}
|
|
}
|
|
if len(c.RestartCmd) == 0 {
|
|
return errors.New("update: restart_cmd is required — there is no safe default for restarting someone else's deployment")
|
|
}
|
|
if c.HealthSocket == "" {
|
|
return errors.New("update: health_socket is required — an update that cannot check its own result cannot roll back on failure")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (c Config) withDefaults() Config {
|
|
if c.HealthTimeoutSec <= 0 {
|
|
c.HealthTimeoutSec = 90
|
|
}
|
|
if c.VerifyTimeoutMin <= 0 {
|
|
c.VerifyTimeoutMin = 20
|
|
}
|
|
if c.KeepSnapshots < 1 {
|
|
c.KeepSnapshots = 5
|
|
}
|
|
return c
|
|
}
|
|
|
|
// buildsFromSource — the deployment whose restart command rebuilds from the
|
|
// tree, which is what SourceDir == InstallDir means in practice (install is a
|
|
// no-op copy and the artifacts that matter are produced inside the image).
|
|
func (c Config) buildsFromSource() bool {
|
|
return filepath.Clean(c.SourceDir) == filepath.Clean(c.InstallDir)
|
|
}
|
|
|
|
func (c Config) healthTimeout() time.Duration {
|
|
return time.Duration(c.HealthTimeoutSec) * time.Second
|
|
}
|
|
|
|
func (c Config) verifyTimeout() time.Duration {
|
|
return time.Duration(c.VerifyTimeoutMin) * time.Minute
|
|
}
|
|
|
|
// within reports whether p is dir or lives under it.
|
|
func within(p, dir string) bool {
|
|
p, dir = filepath.Clean(p), filepath.Clean(dir)
|
|
if p == dir {
|
|
return true
|
|
}
|
|
rel, err := filepath.Rel(dir, p)
|
|
return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
|
|
}
|
|
|
|
// Runner runs one command and returns its combined output. Injected so the
|
|
// tests can drive build/test/restart failures without a toolchain, a container
|
|
// or a real daemon to break.
|
|
type Runner func(ctx context.Context, dir string, argv []string) (string, error)
|
|
|
|
// ExecRunner is the real one.
|
|
func ExecRunner(ctx context.Context, dir string, argv []string) (string, error) {
|
|
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
|
|
cmd.Dir = dir
|
|
out, err := cmd.CombinedOutput()
|
|
return string(out), err
|
|
}
|
|
|
|
// HealthCheck proves the daemon at socket answers. Injected for the same reason
|
|
// as Runner.
|
|
type HealthCheck func(ctx context.Context, socket string) error
|
|
|
|
// Logger receives one line per step. The CLI prints these as they happen: an
|
|
// update that goes quiet for four minutes during `make test` reads as a hang.
|
|
type Logger func(format string, args ...any)
|
|
|
|
// Updater is the whole capability. Construct with New and call Apply or
|
|
// Rollback; there is no background goroutine and nothing starts on its own.
|
|
type Updater struct {
|
|
cfg Config
|
|
store *Store
|
|
run Runner
|
|
health HealthCheck
|
|
log Logger
|
|
now func() time.Time
|
|
// ids reports the running euid and the owner of a directory. Injected so
|
|
// the root-build refusal is testable without a second account.
|
|
ids func(dir string) (int, uint32, error)
|
|
}
|
|
|
|
// New builds an Updater. Every seam has a real default; the tests replace them.
|
|
func New(cfg Config, opts ...Option) (*Updater, error) {
|
|
if err := cfg.Validate(); err != nil {
|
|
return nil, err
|
|
}
|
|
u := &Updater{
|
|
cfg: cfg.withDefaults(),
|
|
store: &Store{Dir: cfg.SnapshotDir},
|
|
run: ExecRunner,
|
|
health: DialHealth,
|
|
log: func(string, ...any) {},
|
|
now: time.Now,
|
|
ids: realIDs,
|
|
}
|
|
for _, o := range opts {
|
|
o(u)
|
|
}
|
|
u.store.now = u.now
|
|
return u, nil
|
|
}
|
|
|
|
// Option — a constructor seam.
|
|
type Option func(*Updater)
|
|
|
|
func WithRunner(r Runner) Option { return func(u *Updater) { u.run = r } }
|
|
func WithHealth(h HealthCheck) Option { return func(u *Updater) { u.health = h } }
|
|
func WithLogger(l Logger) Option { return func(u *Updater) { u.log = l } }
|
|
func WithClock(f func() time.Time) Option {
|
|
return func(u *Updater) { u.now = f }
|
|
}
|
|
|
|
// WithIDs replaces the euid/owner lookup behind the root-build refusal.
|
|
func WithIDs(f func(dir string) (int, uint32, error)) Option {
|
|
return func(u *Updater) { u.ids = f }
|
|
}
|
|
|
|
// Snapshots lists what is available to roll back to, newest first.
|
|
func (u *Updater) Snapshots() ([]Snapshot, error) { return u.store.List() }
|