Deploy a new build with verification and automatic rollback #69
Closed
claude
wants to merge 1 commits from
overnight/self-update into overnight/model-swap
pull from: overnight/self-update
merge into: kami:overnight/model-swap
kami:master
kami:task/725-capability-ledger-and-empirical-baseline
kami:task/692-heads-path-may-equal-model-path-and-noth
kami:task/694-staticcheck-and-deadcode-are-still-not-i
kami:task/682-go-1-25-5-and-x-text-0-14-0-carry-20-rea
kami:task/674-caveats
kami:task/673-mavgpud-serves-the-model-to-the-whole-la
kami:task/487-capture-device-doc
kami:task/487-capture-device
kami:task/487-wake-word-deploy
kami:task/487-wake-word-threshold
kami:task/487-wake-word-stage-two
kami:task/671-mavwaked-registers-as-a-voice-consumer-i
kami:task/670-cut-claude-md-to-200-lines
kami:task/515-deploy-mavwaked-workpc
kami:task/669-prune-claude-md
kami:task/668-e4b-phrasing
kami:task/668-title-capital
kami:task/668-kiwix-answers-a-question-it-cannot-answe
kami:task/666-only-a-stage-0-grammar-may-take-the-pers
kami:task/487-mavwaked-has-no-wake-word-only-an-energy
kami:task/486-deploy-the-workstation-transcriber
kami:task/486-move-stt-and-tts-to-the-workstation-wher
kami:task/665-crisperwhisper-2-russian
kami:task/664-routing-heads-in-go
kami:task/662-usage-harness-source-badge
kami:task/661-post-merge-usage-rerun
kami:task/661-routing-heads-step-3-train-the-multi-hea
kami:task/660-router-prompt-destination
kami:task/659-destination-fixture
kami:task/655-query-source-is-a-routing-decision-made
kami:task/654-a-pending-clarify-has-no-way-out-neither
kami:task/654-week-of-usage-eval-docs
kami:task/649-needs-kami-telegram-is-the-only-reach-an
kami:task/643-memorystore-search-decodes-and-unmarshal
kami:task/641-two-maps-grow-for-the-process-lifetime-w
kami:task/644-mavcaldav-is-built-documented-as-running
kami:task/642-the-store-caps-sqlite-at-one-connection
kami:task/647-factenrichmentworker-walks-the-pending-q
kami:task/646-v-637-follow-up-telegram-intake-has-no-d
kami:task/638-no-deadline-survives-the-turn-path-from
kami:task/637-inbound-telegram-turns-and-corrections-f
kami:task/636-correcting-a-turn-from-telegram-and-from
kami:task/634-an-act-alias-resolves-the-verb-but-not-t
kami:task/630-one-gesture-correction-on-chat-v-628
kami:task/629-persist-the-routing-trace-and-record-it
kami:task/631-mode-inventory-written-from-the-handlers
kami:task/586-defaultfactparser-uses-hand-written-russ
kami:task/633-reconcile-the-seed-labels-with-the-handl
kami:task/627-reminder-verbs-has-no-alarm-verb-so-an-a
kami:task/626-the-classifier-seeds-teach-an-older-inte
kami:task/546-route-with-a-fine-tuned-e5-small-instead
kami:task/586-measure-the-fact-parser
kami:fix/gofmt-ecosystem-acts
kami:task/584-media-store-a-failed-write-leaks-its-bud
kami:task/518-no-write-path-for-a-backdated-event-so-t
kami:task/287-qa-voice-session-quality-polish
kami:task/492-qa-plan-reconcile
kami:task/530-sweep-tail-four-files-the-russian-sweep
kami:task/405-score-how-often-a-real-utterance-reaches
kami:task/529-money-and-list-pick-a-mechanism
kami:task/528-sweep-tail-the-three-files-on-467
kami:task/527-embedder-open-set-phrasings-stop-being-r
kami:task/526-morphology-a-dictionary-answers-the-gram
kami:task/525-lexicons-the-finite-russian-sets-move-to
kami:task/524-entity-reference-ask-nexus-about-every-l
kami:task/523-risk-tiers-take-hexis-s-tier-for-a-hexis
kami:task/521-review-pr-111-query-strings-declension-h
kami:task/491-llama-server-core-dumps-on-every-sigterm
kami:task/479-bug-an-unconfigured-capability-does-not
kami:task/467-bug-spoken-task-capture-is-dead-the-rout
kami:task/463-deploy-mavwaked-and-mavenclient-run-nowh
kami:task/480-hearing-no-shipped-client-can-start-a-re
kami:task/432-ambient-calendar-intake-is-fragile-and-p
kami:task/431-board-surface-maven-holds-the-work-board
kami:task/433-reactivehandler-has-30-fields-and-is-pas
kami:task/371-swap-the-embedder-for-an-asymmetric-retr
kami:task/408-review-31-07-split-the-30-method-coreapi
kami:task/410-review-31-07-hand-rolled-string-enums-st
kami:task/423-review-pr50-split-internal-ipc-server-go
kami:task/422-review-pr50-split-cmd-mavend-tick-go-860
kami:task/409-review-31-07-finish-moving-mavweb-markup
kami:task/482-ambient-ingest-reads-a-notification-s-ti
kami:task/444-kuma-a-fact-per-monitor-so-she-can-name
kami:task/452-capability-model-homelab-docker-restart
kami:task/449-destructive-confirm-policy-risk-tiers-no
kami:task/453-grocery-list-items-table-fourth-append-o
kami:task/399-run-the-persona-checks-inside-the-daemon
kami:task/448-bounded-follow-up-state-pending-candidat
kami:task/455-conversation-repair-name-the-misroute-co
kami:task/454-go-mod-tidy
kami:task/458-pronunciation-dictionary-for-piper
kami:task/456-command-history-read-only-query-over-exi
kami:task/457-clarification-templates-for-the-router-s
kami:task/474-query-source-ordering-feeds-and-calendar
kami:task/469-reminders-spelled-out-times-fail-the-bod
kami:task/475-bug-the-praxis-attention-capability-is-u
kami:task/481-bug-a-transient-complaint-is-stored-as-a
kami:task/476-bug-the-router-transliterates-latin-enti
kami:task/385-decide-whether-a-parked-clarify-question
kami:task/377-backfill-routines
kami:task/421-weather-geocoder
kami:task/390-no-read-path-for-delivery-attempts
kami:task/386-recall-fixture-filler-note-ids
kami:task/473-bug-morning-item-has-no-required-flag
kami:task/465-bug-make-simulate-routes-with-an-empty
kami:task/467-bug-spoken-task-capture-is-dead
kami:task/466-bug-a-pending-clarify-is-global-so-one-u
kami:task/468-bug-pattern-detect-has-no-minimum-interv
kami:task/462-bug-checkfeminine-flags-second-person-ma
kami:task/443-safekey-drops-cyrillic-so-russian-calend
kami:task/471-bug-agendaquerygrammars-covers-today-but
kami:task/383-slottext-in-clarify-answer-would-clobber
kami:task/323-qa-phraser-coverage-is-65-3-but-the-llam
kami:task/498-bug-and-x-reach-the-model-with-no-determ
kami:task/506-strings-family-6-summaries-and-reports-i
kami:task/504-strings-family-4-act-and-smart-home-repl
kami:task/503-strings-family-3-query-answers-and-gaps
kami:task/502-strings-family-2-capture-acknowledgement
kami:task/501-strings-family-1-phrasing-fallbacks-into
kami:task/397-phrasechat-and-phrasequery-hide-model-fa
kami:task/396-the-reply-path-can-t-be-tested-llmreplie
kami:task/496-recall-a-cross-language-question-loses-i
kami:task/495-bug-x-escapes-the-personal-boundary-and
kami:task/499-llama-server-holds-7-9gb-rss-for-a-1-1gb
kami:task/470-bug-a-question-writes-invented-knowledge
kami:task/493-bug-the-memory-index-stores-the-raw-utte
kami:task/490-name-the-gap-world-questions-through-the
kami:task/485-run-the-big-model-on-the-workstation-wit
kami:task/489-workstation-deploy-mavgpud-on-workpc-and
kami:task/488-workstation-a-supervisor-that-keeps-llam
kami:task/483-docs-offload-design
kami:task/483-design-offload-ml-to-the-workstation-kee
kami:task/459-docs-refresh-the-qa-plan-against-the-liv
kami:task/446-doc-reorg-tier-the-tree-retire-the-three
kami:fix/367-voice-parks-routine-accept
kami:task/365-dialogue-slots-and-router-slots-are-hand
kami:task/364-snooze-does-nothing-at-runtime-the-gate
kami:task/447-retire-progress-md-the-backlog-and-the-f
kami:task/445-session-workflow
kami:overnight/eco-versioned-traces
kami:overnight/eco-entity-refs
kami:overnight/eco-degraded-suite
kami:overnight/netscan
kami:overnight/smarthome
kami:overnight/replay-simulator
kami:overnight/event-envelope
kami:overnight/coldstart-unlock
kami:overnight/voice-barge-in
kami:overnight/stt-golden-audio
kami:overnight/senses-speaker
kami:overnight/senses-hearing
kami:overnight/senses-media-vision
kami:overnight/mcp-tools
kami:overnight/mcp-client
kami:overnight/model-swap
kami:overnight/web-crawler
kami:overnight/rss-feeds
kami:overnight/email-poller
kami:overnight/email-extract
kami:overnight/email-imap
kami:overnight/money-zenmoney
kami:overnight/task-priority
kami:overnight/task-capture
kami:overnight/behavior-profile
kami:overnight/day-plan
kami:overnight/ambient-calendar
kami:overnight/local-calendar
kami:overnight/memory-eval
kami:overnight/proactive-proposals
kami:overnight/split-voice-quiet
kami:overnight/nginx-maven-block
kami:overnight/stepup-chat-surface
kami:integration/small-batch
kami:docs/fix-drift
kami:fix/ru-wording
kami:integration/jul31
kami:overnight/resident-1.7b
kami:overnight/nudge-templates
kami:overnight/kiwix-rewrite
kami:overnight/eval-writeup
kami:overnight/fix-truncation
kami:overnight/kiwix-client
kami:overnight/ru-prompts
kami:overnight/external-data
kami:overnight/phrasing-grammar
kami:overnight/talk-eval
kami:overnight/prompt-context
kami:overnight/prompt-address
kami:overnight/eval-label-kill
kami:overnight/delivery-boundary
kami:overnight/address-check
kami:overnight/system-replies-pr
kami:overnight/clock-intent-pr
kami:overnight/embedder-backfill-pr
kami:overnight/embedder-marker-pr
kami:overnight/note-recall-pr
kami:overnight/thinking-off-pr
kami:overnight/dialogue-persist-pr
kami:overnight/persona-2p-pr
kami:overnight/clarify-expiry-pr
kami:overnight/clarify-rework
kami:overnight/phrasing
kami:overnight/bakeoff
kami:overnight/recall-margin
kami:overnight/router-on
kami:overnight/slot-extract
kami:overnight/embedder-e5
kami:overnight/router-refusal
kami:overnight/eval-rerun
kami:overnight/eval-harnesses
kami:overnight/eval-rerun-base
kami:overnight/fmt-gate
kami:overnight/routines-fire
kami:overnight/router-prompt
kami:overnight/away-leak
kami:overnight/recall-eval
kami:overnight/snooze-works
kami:overnight/clarify-wiring
kami:overnight/delivery-tests
kami:overnight/routine-accept
kami:overnight/llm-router-flag
kami:overnight/clarify-data-layer
kami:overnight/loop-rule-tests
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "overnight/self-update"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
internal/update+cmd/mavupdate: deploy a new build of Maven to the box sheruns on, verified before it is committed and rolled back automatically when it
does not come up. Off unless an
updateblock is inmavend.json.Applyis: health-check the running daemon → snapshot the deployed artifacts →make build→make test→ install → restart → health-check → restore thesnapshot on any failure.
Why the order is what it is
update and an already-broken box are indistinguishable afterwards and the
rollback has no baseline to prove itself against.
Applyrefuses to start.make buildwrites its binaries into theworking tree, and on the docker deployment the tree is the install dir — so
snapshotting after the build would snapshot the new artifacts and leave nothing
to roll back to. This was the one non-obvious ordering bug available here.
make build+make test(notgo build— the CGO daemons need the Makefile's toolchain and lib paths). A failedverify also restores the tree's artifacts, so a later restart by hand cannot
deploy code that failed its own tests.
the snapshot dir, sha256-verified against the manifest on the way in, plus the
same restart command. No build, no toolchain, no migration, no cooperation from
the code being replaced. It runs on
context.WithoutCancel— a rollbackinterrupted halfway is worse than the failure that caused it. A corrupt
snapshot is refused rather than restored. When the restore or its restart fails
anyway, it returns
ErrRollbackFailed, says she is probably down, and names thedirectory to copy back by hand instead of reporting a tidy rollback.
install dir gets clobbered by the very install it exists to undo, and a
git-based undo needs a clean tree and a rebuild — two things a failed update is
likely to have taken away.
What I refused to build, and why
The task said to ship the safe subset and say plainly what was left out. All of
this is enforced by the code's shape, not by a note:
channel, no timer, no tick-loop hook.
Applyruns when a human runs the CLI.MethodApplyUpdateplus a button on
/toolsbehind the step-up passkey gate — was considered andrefused. Step-up protects against the wrong person clicking; it does not change
the fact that anything reachable over the network becomes, given one mavweb
bug, a remote code path with a build system attached. The trigger requires
shell access on the host, a strictly higher bar than the gate guarding the tool
allowlist. This is the one place I did not follow the "put it behind step-up
like /tools" instruction, and it is because no-remote-trigger is strictly
stronger than step-up, not weaker.
mavenddoes not importinternal/update, so there is no act, intent, tool or LLM output that leadshere. She cannot update herself. She can be updated, by him.
version is whatever the owner pulled into the working tree. Downloading code
and trusting a checksum that arrived in the same download is not a property
verifiable on one box, and it is the shape most supply-chain compromises take.
cannot reliably notice that it keeps dying, and one that believes it can is
worse than nothing. Restart-on-crash belongs to whatever starts mavend
(compose
restart: unless-stopped). What is guaranteed instead is narrower andreal: within one
Apply, the new build must answer before it is considereddeployed, and if it does not the old bytes go back and must answer again.
often bigger than the disk headroom, and a store rolled back under a schema
that already migrated forward loses writes silently — worse than a failed
update. Schema compatibility stays
store.Migrate's job. A snapshot here isthe deployable artifacts only: binaries and config.
Also: config files are snapshotted but never overwritten by an install. An
update does not get to replace the operator's config.
Health check
Not "the process is up" — mavend can be running with a dead store or a socket it
never bound. It dials the real IPC socket and performs a real
Presenceread,which exercises the socket, the dispatch table and the store in one call.
Presenceis read-only, so a health check never leaves a trace in her memory.The config is refused at load without
health_socket: an update that cannot checkits own result cannot roll back.
Verified
make build— 10 binaries including the newmavupdate, exit 0.make test— full-racesuite, 48 packages ok, exit 0.internal/updateat 76.6% coverage, 13 cases against a fake box (a temp dirfor the install tree, an injected
Runnerfor make/git/docker, an injectedHealthCheckfor mavend) so the paths nobody exercises by hand are the onesunder test: refusal on an already-down daemon, build failure short-circuiting
the tests, test failure deploying nothing and restoring the tree, unhealthy
after restart rolling back to the old bytes with the restore proven to have
happened before the second restart, restart failure reported as manual
recovery, rollback working with the source tree deleted and the toolchain
failing, config never overwritten, corrupt snapshot refused, manifest-less
snapshot never offered as a target, prune never dropping the newest.
that is the QA step on homesrv, and it is written to be run in the order
verify→apply→ deliberately-brokenapply→rollback.deploy/README.mdgains the config block and the exact commands. Nothing wasadded to
deploy/mavend.json: the capability stays off until the owner writes it.Vikunja #249
internal/update applies a new build of Maven to the box she runs on and undoes it when the new build does not come up. cmd/mavupdate is the only trigger: a CLI the owner runs on the host. Apply is health-check the running daemon, snapshot the deployed artifacts, make build, make test, install, restart, health-check — and restore the snapshot on any failure. The order is load-bearing: - The preflight health check refuses to update a daemon that is already not answering. Without a working baseline, a failed update and a box that was already broken are indistinguishable, and the rollback has nothing to prove itself against. - The snapshot is taken BEFORE the build, because make build writes its binaries into the working tree and on the docker deployment the tree is the install dir — snapshotting afterwards would snapshot the new artifacts and leave nothing to roll back to. - Verification is make build plus make test, before anything is deployed, so a broken tree costs time and nothing else. A failed verify also puts the tree's artifacts back, so a later restart by hand cannot deploy code that failed its own tests. - The rollback depends on nothing that just changed: byte-for-byte copies out of the snapshot dir, sha256-verified on the way in, and the same restart command. No build, no migration, no cooperation from the code being replaced. It also runs on an uncancellable context — a rollback interrupted halfway is worse than the failure that caused it. When the restore itself fails it says so and names the directory to copy back by hand rather than reporting a tidy rollback. Off unless configured, and the refusals are code, not documentation. The daemon does not import this package: there is no IPC method, no web route, no timer and no act that can start an update, so nothing Maven says or routes reaches it. Nothing fetches code — the new version is whatever the owner pulled into the tree. The plan's release checker, auto-update channel and in-process crash-loop supervisor are deliberately absent; a process cannot reliably notice that it keeps dying, and restart-on-crash belongs to compose or systemd. The database is never snapshotted or rolled back; schema compatibility stays store.Migrate's job. The config is refused at load without a health socket, since an update that cannot check its own result cannot roll back, and refused when the snapshot dir is inside the install dir, since a restore must not read from what the install writes. Vikunja #249The refusals in the package comment are the best part of this PR, and they are enforced rather than described. No timer, no IPC method, no web route, no fetch of code from anywhere. Snapshotting before the build, with the reason spelled out, is the non-obvious ordering and it is the correct one. The preflight health check is right, and for the reason given. Without a baseline, a failed update and an already broken box are the same picture.
context.WithoutCancelaround the rollback is the detail that makes a Ctrl-C during the health wait safe. Copies rather than hardlinks or a git stash, re-hashed on the way back out. A restore is checked, not hoped for.Four things.
1. On the documented deployment the rollback restores bytes that nothing reads. The README config sets
source_dirandinstall_dirto the repo andrestart_cmdtodocker compose up -d --build. Follow it through.installis a no-op because the two dirs match. The restart rebuilds the image, and this repo'sDockerfilecopiescmd/andinternal/and runsgo buildinside the builder stage. It never copies a host binary..dockerignoreexcludes the built binaries by name, with the comment "rebuilt inside the image". So what gets deployed is the source tree, whichApplynever touches andRestorenever reverts.The failure case is the one this package exists for. He pulls a bad commit and runs
apply -yes. Build and test pass, the restart builds an image from the bad source, she does not answer,waitHealthyburns 120s. The rollback copies the old binaries into the tree and runs the same restart command. That rebuilds the same image from the same bad source. She does not answer again. It returnsErrRollbackFailedwith "SHE IS PROBABLY DOWN" and an instruction to copy files back by hand. Copying them back by hand would not have helped either. The box stays down for two health timeouts plus two image builds. The only recovery is agit checkoutthe operator has to work out himself.The one part of the restore that does reach the running system is
deploy/mavend.json, because compose bind-mounts it read-only from the tree. That is the file theConfigcomment says an update never replaces.The snapshot needs to cover whatever the restart command deploys. For an image built from source that means the commit, and
gitHeadis already recorded. So either record and restore the tree state for that deployment shape, or refuse that config outright. A README that sends him to a two-stage build-then-copy layout would be the honest alternative.2. The health socket in the README cannot be opened by the account the README tells him to use. The documented path is
/var/lib/docker/volumes/maven_sockets/_data/mavend.sock. On this box/var/lib/dockerisdrwx--x--- root root, sokamigets EACCES before reaching the socket. The socket itself is 0600 owned by uid 10001, from the Dockerfile'suseradd -r -u 10001 maven. So everyapplystops at the preflight withErrUnhealthyBefore. The message blames the daemon for not answering, when the cause is a permission error on the dial.Running it as root does work, and that is the worse outcome.
Verifyrunsmake buildandmake testinSourceDiras root, which leaves root-owned binaries, object files and a root-owned build cache in his working tree. The next non-rootmakefails, so a single rootapplybreaks the ordinary build. Bind-mount the socket to a host path he owns and document that, or drop privileges for the verify step. At minimum, separate the dial error from the read error at preflight and say "cannot open the socket".3. A cold-start locked daemon makes a good update look like the manual-recovery case.
main.godocuments the locked mode: with a passkey enrolled and no env key, mavend starts locked and every CoreAPI method returnserrLockeduntil an assertion arrives.DialHealthcallsPresence, which is a CoreAPI method. Preflight passes because the running daemon is already unlocked. After the restart she comes up locked,waitHealthyfails for 90s, the rollback restores, restarts, and she comes up locked again. The result isErrRollbackFailedand "SHE IS PROBABLY DOWN" for an update that was fine. He now has a daemon waiting for a passkey, and a tool telling him to copy files by hand.The deployed config uses
db_key.envtoday, so this is latent. It fires on the day he removes the env key, which themain.gocomment describes as the intended end state. The health check needs a liveness signal a locked daemon can answer. OtherwiseApplyhas to refuse a deployment that boots locked.4.
mavend does not import internal/updateis no longer true. It is stated three times: in the package comment, in thecmd/mavupdatecomment, and beside theUpdatefield. This PR adds"github.com/kami/maven/internal/update"to the import block ofinternal/config/config.goand callsc.Update.Validate()fromvalidate(). mavend importsinternal/config, so every mavend build links the package. The property that matters still holds, since there is no method, route, timer or caller. The proof offered for it does not. Validating the block from config is worth keeping. Reword the claim to what is enforced: mavend never constructs anUpdater, and nothing in the daemon can callApply.Smaller notes:
res.RolledBack = truewhen nothing was installed and nothing was restarted.summarizethen printsrolled_back=truefor a plain compile error, andcmdApplyfalls to thedefaultbranch, so he sees a rollback flag with no rollback message. Leave the flag false and let the log line carry it.Applysnapshots beforeVerify, and the comment inapply.goexplains at length why it must. The two comments contradict each other, and the one that is wrong is the one someone reads first.RollbackrestoresConfigFilesoverInstallDir. TheConfigdoc says config is "Snapshotted, never overwritten by an install", which is true ofinstalland not of the restore. A rollback silently reverts any config edit made since the last apply. That includes aphraser.model_pathchange, which is how the resident model gets swapped.mavupdateis now built bymake buildand is absent from the README'sbinarieslist. The updater is the one artifact never snapshotted and never installed. A rollback leaves the newmavupdatein place against restored binaries.ValidatechecksSnapshotDiragainstInstallDirbut not againstSourceDir. With the split layout, a snapshot dir under the source tree lands inside the docker build context. It also lands inside whatevermakeand git do there.waitHealthytests the deadline only after an attempt, and each attempt gets its own 10s budget. With a 90s timeout the last attempt can start at 89s and run to 99s. Cap the attempt at the remaining time.cmdRollbackdies with the bare error onErrRollbackFailed.cmdApplyprints the loud "SHE IS PROBABLY DOWN" paragraph for the same condition. The standalone rollback is the path he reaches for when something is already wrong, so it needs that text more.tailslices bytes, so a truncatedmake testlog can start with half a rune. Russian test names and fixture strings will show it.Configcomment showsrestart_cmdending in"mavend"and the README example omits it, so the documented command restarts every service in the compose file.Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.
Pull request closed