update: roll back what the restart actually deploys
On the deployment deploy/README.md documents, source_dir and install_dir are
the same tree and the restart command rebuilds the image from it. The
Dockerfile builds from cmd/ and internal/ and .dockerignore keeps the host
binaries out, so restoring the snapshotted binaries restored bytes nothing
reads. A bad commit therefore cost two health timeouts and two image builds
and ended in ErrRollbackFailed with an instruction to copy files back by hand,
which would not have helped either.
A deployment that rebuilds from source now has to say how the source is put
back. source_rollback "git" records the commit before the update and checks it
back out before the rollback restart. It refuses a dirty tree, because the
recorded commit does not describe one and a forced checkout would delete his
work. A build-from-source config that says nothing is refused by Validate, at
startup, rather than at the one rollback that mattered.
Also in this change, all from the same review:
- MethodPing, the one method a locked daemon answers. Preflight passed on an
unlocked daemon and the post-restart Presence read failed on a locked one,
so a good update read as SHE IS PROBABLY DOWN once the env key is gone.
- A dial failure is reported apart from a read failure. The documented
socket is under /var/lib/docker, which a non-root operator cannot
traverse, and "she is not answering" was the wrong diagnosis.
- Verify refuses to run as root over a tree owned by someone else. It runs
make build and make test in place, and root-owned artifacts break his next
ordinary make.
- A rollback no longer reverts config_files. That undid every config edit
since the last apply, phraser.model_path among them.
- The verify-failure path no longer reports rolled_back for a compile error.
- waitHealthy caps each attempt at the remaining budget, so a 90s timeout
cannot run to 99s.
- tail cuts on a rune boundary. Russian test names showed the seam.
- The claim that mavend does not import internal/update is replaced with
what is enforced: mavend constructs no Updater and nothing can call Apply.
- snapshot_dir inside source_dir is refused. It landed in the build context.
Found in review of #69.
This commit is contained in:
+49
-6
@@ -132,18 +132,61 @@ it), with paths as they exist **on the host**, not inside a container:
|
||||
"source_dir": "/home/kami/apps/Maven",
|
||||
"install_dir": "/home/kami/apps/Maven",
|
||||
"snapshot_dir": "/var/lib/maven-snapshots",
|
||||
"source_rollback": "git",
|
||||
"binaries": ["mavend", "mavweb", "mavsttd", "mavttsd", "mavwaked",
|
||||
"mavenclient", "mavpoll", "mavcaldav", "mavmaild"],
|
||||
"mavenclient", "mavpoll", "mavcaldav", "mavmaild", "mavupdate"],
|
||||
"config_files": ["deploy/mavend.json"],
|
||||
"restart_cmd": ["docker", "compose", "up", "-d", "--build"],
|
||||
"health_socket": "/var/lib/docker/volumes/maven_sockets/_data/mavend.sock",
|
||||
"restart_cmd": ["docker", "compose", "up", "-d", "--build", "mavend"],
|
||||
"health_socket": "/run/maven-host/mavend.sock",
|
||||
"health_timeout_sec": 120
|
||||
}
|
||||
```
|
||||
|
||||
`snapshot_dir` must be outside `install_dir` (a restore must not read from what
|
||||
the install writes) and `health_socket` is required: an update that cannot check
|
||||
its own result cannot roll itself back, so the config is refused without one.
|
||||
`snapshot_dir` must be outside both `install_dir` and `source_dir` (a restore
|
||||
must not read from what the install writes, and a snapshot dir inside the tree
|
||||
lands in the docker build context). `health_socket` is required: an update that
|
||||
cannot check its own result cannot roll itself back, so the config is refused
|
||||
without one.
|
||||
|
||||
**`source_rollback` is what makes a rollback real on this deployment.** Compose
|
||||
builds the image from the tree — the Dockerfile copies `cmd/` and `internal/`
|
||||
and runs the build in the builder stage, and `.dockerignore` keeps the host
|
||||
binaries out — so `install_dir` is the tree, `install` is a no-op, and putting
|
||||
the old binaries back puts back bytes nothing reads. A rollback that only did
|
||||
that would rebuild the same bad image and burn a second health timeout proving
|
||||
it. With `"source_rollback": "git"` the commit is recorded before the update and
|
||||
checked back out before the restart, so the restore is of the thing that
|
||||
actually gets deployed. It requires a clean tree: `apply` refuses to start with
|
||||
uncommitted changes, because the recorded commit would not describe what is
|
||||
deployed and the forced checkout on the way back would delete the work. It also
|
||||
means a rollback moves every tracked file, `deploy/mavend.json` included, so on
|
||||
this deployment a config edit belongs in a commit.
|
||||
|
||||
Leaving `source_rollback` out is only valid when `install_dir` holds what
|
||||
actually runs. `Validate` refuses the combination of "same dir" and "no way to
|
||||
put the source back" at startup rather than at the one rollback that mattered.
|
||||
|
||||
**The socket has to be one the account running `mavupdate` can open.** The
|
||||
compose stack keeps IPC in a named volume, whose host path
|
||||
(`/var/lib/docker/volumes/maven_sockets/_data`) is under a `drwx--x--- root
|
||||
root` directory, and the socket itself is 0600 owned by the container's uid
|
||||
10001. A non-root `mavupdate` gets EACCES on the dial, which reports as
|
||||
`update: cannot open the health socket` rather than as a daemon that will not
|
||||
answer. Bind-mount the socket dir to a host path he owns and run the daemon
|
||||
under his uid instead:
|
||||
|
||||
```yaml
|
||||
mavend:
|
||||
user: "1000:1000"
|
||||
volumes:
|
||||
- /run/maven-host:/run/maven
|
||||
```
|
||||
|
||||
Do **not** work around it with `sudo mavupdate apply`. `verify` runs `make
|
||||
build` and `make test` in `source_dir`, and as root that leaves root-owned
|
||||
binaries, object files and a build cache in the working tree, so the next
|
||||
ordinary `make` fails. `Verify` refuses to run as root over a tree owned by
|
||||
someone else for exactly that reason.
|
||||
|
||||
Then:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user