ipc: client redials on a dropped core connection

mavweb (and every ipc.Client) held one net.Conn from Dial and reused it for the
life of the process. When mavend restarted, the socket got a new inode, the
cached conn went dead, and every call failed forever with "broken pipe" — the
dash and page-heartbeat 502'd until mavweb was manually restarted.

Fix in the one place all 25 methods route through (call): on a lost connection
— write failure OR read EOF, since a peer restart can surface on either phase
depending on socket-buffer timing — drop the conn, re-dial the remembered path,
and retry once. Safe for the case that happens (core restarted, request never
processed); the rare committed-then-died window can double-apply a write, but
the store is append-only so a duplicate is a superseding row, not corruption.
ponytail: retry-once, not request-ids — revisit if double-apply ever bites.

Test reproduces the exact incident: server restart on the same socket path, and
asserts the next call transparently reconnects.

Note (not fixed here): Server.Close waits on its handler goroutines, which park
reading live client conns — so a graceful core shutdown with a client attached
blocks until the client disconnects. Minor; surfaces as a slow SIGTERM.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
kami
2026-07-03 23:33:56 +04:00
parent 4c4b129789
commit 9e2a9690bf
2 changed files with 118 additions and 7 deletions
+56
View File
@@ -57,6 +57,62 @@ func newServerWithStore(t *testing.T) (CoreAPI, *Server, *Client, *store.Store)
return api, srv, cli, s
}
// TestClient_ReconnectsAfterServerRestart — a long-lived module (e.g. mavweb)
// must survive a core restart. The first server is closed and a new one is
// brought up on the SAME socket path (as a daemon restart does); the client's
// cached conn is now dead. The next call must transparently re-dial and succeed
// instead of failing forever with "broken pipe".
func TestClient_ReconnectsAfterServerRestart(t *testing.T) {
dir := t.TempDir()
sock := filepath.Join(dir, "maven.sock")
serve := func() *Server {
s, err := store.Open(context.Background(), filepath.Join(dir, "maven.db"))
if err != nil {
t.Fatalf("open store: %v", err)
}
t.Cleanup(func() { _ = s.Close() })
srv, err := Listen(sock, NewStoreAPI(s))
if err != nil {
t.Fatalf("listen: %v", err)
}
go func() { _ = srv.Serve() }()
return srv
}
srv1 := serve()
cli, err := Dial(sock)
if err != nil {
t.Fatalf("dial: %v", err)
}
t.Cleanup(func() { _ = cli.Close() })
// works against the first server
if _, err := cli.Presence(context.Background()); err != nil {
t.Fatalf("call before restart: %v", err)
}
// Simulate a core restart: the client's conn dies (as it would when the
// daemon process exits), then a fresh server binds the SAME path. Closing
// the client side first also lets srv1's handler goroutine see EOF and
// exit, so srv1.Close()'s wg.Wait() returns instead of blocking on a
// parked reader.
cli.conn.Close()
_ = srv1.Close()
srv2 := serve()
// The cached conn is dead — the call must transparently re-dial and succeed.
if _, err := cli.Presence(context.Background()); err != nil {
t.Fatalf("call after restart should have re-dialed, got: %v", err)
}
// Teardown order matters: Server.Close waits for its handler goroutine,
// which is parked reading the (now live, re-dialed) client conn. Close the
// client first so the handler sees EOF and Close returns instead of hanging.
_ = cli.Close()
_ = srv2.Close()
}
// TestFrame_Roundtrip — JSON over a length prefix survives the loop, and the
// prefix itself encodes the length exactly. The framing is the only thing
// keeping a module's request paired with core's reply; it's worth a direct test.