ipc: client redials on a dropped core connection
mavweb (and every ipc.Client) held one net.Conn from Dial and reused it for the life of the process. When mavend restarted, the socket got a new inode, the cached conn went dead, and every call failed forever with "broken pipe" — the dash and page-heartbeat 502'd until mavweb was manually restarted. Fix in the one place all 25 methods route through (call): on a lost connection — write failure OR read EOF, since a peer restart can surface on either phase depending on socket-buffer timing — drop the conn, re-dial the remembered path, and retry once. Safe for the case that happens (core restarted, request never processed); the rare committed-then-died window can double-apply a write, but the store is append-only so a duplicate is a superseding row, not corruption. ponytail: retry-once, not request-ids — revisit if double-apply ever bites. Test reproduces the exact incident: server restart on the same socket path, and asserts the next call transparently reconnects. Note (not fixed here): Server.Close waits on its handler goroutines, which park reading live client conns — so a graceful core shutdown with a client attached blocks until the client disconnects. Minor; surfaces as a slow SIGTERM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -57,6 +57,62 @@ func newServerWithStore(t *testing.T) (CoreAPI, *Server, *Client, *store.Store)
|
||||
return api, srv, cli, s
|
||||
}
|
||||
|
||||
// TestClient_ReconnectsAfterServerRestart — a long-lived module (e.g. mavweb)
|
||||
// must survive a core restart. The first server is closed and a new one is
|
||||
// brought up on the SAME socket path (as a daemon restart does); the client's
|
||||
// cached conn is now dead. The next call must transparently re-dial and succeed
|
||||
// instead of failing forever with "broken pipe".
|
||||
func TestClient_ReconnectsAfterServerRestart(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
sock := filepath.Join(dir, "maven.sock")
|
||||
|
||||
serve := func() *Server {
|
||||
s, err := store.Open(context.Background(), filepath.Join(dir, "maven.db"))
|
||||
if err != nil {
|
||||
t.Fatalf("open store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = s.Close() })
|
||||
srv, err := Listen(sock, NewStoreAPI(s))
|
||||
if err != nil {
|
||||
t.Fatalf("listen: %v", err)
|
||||
}
|
||||
go func() { _ = srv.Serve() }()
|
||||
return srv
|
||||
}
|
||||
|
||||
srv1 := serve()
|
||||
cli, err := Dial(sock)
|
||||
if err != nil {
|
||||
t.Fatalf("dial: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = cli.Close() })
|
||||
|
||||
// works against the first server
|
||||
if _, err := cli.Presence(context.Background()); err != nil {
|
||||
t.Fatalf("call before restart: %v", err)
|
||||
}
|
||||
|
||||
// Simulate a core restart: the client's conn dies (as it would when the
|
||||
// daemon process exits), then a fresh server binds the SAME path. Closing
|
||||
// the client side first also lets srv1's handler goroutine see EOF and
|
||||
// exit, so srv1.Close()'s wg.Wait() returns instead of blocking on a
|
||||
// parked reader.
|
||||
cli.conn.Close()
|
||||
_ = srv1.Close()
|
||||
srv2 := serve()
|
||||
|
||||
// The cached conn is dead — the call must transparently re-dial and succeed.
|
||||
if _, err := cli.Presence(context.Background()); err != nil {
|
||||
t.Fatalf("call after restart should have re-dialed, got: %v", err)
|
||||
}
|
||||
|
||||
// Teardown order matters: Server.Close waits for its handler goroutine,
|
||||
// which is parked reading the (now live, re-dialed) client conn. Close the
|
||||
// client first so the handler sees EOF and Close returns instead of hanging.
|
||||
_ = cli.Close()
|
||||
_ = srv2.Close()
|
||||
}
|
||||
|
||||
// TestFrame_Roundtrip — JSON over a length prefix survives the loop, and the
|
||||
// prefix itself encodes the length exactly. The framing is the only thing
|
||||
// keeping a module's request paired with core's reply; it's worth a direct test.
|
||||
|
||||
Reference in New Issue
Block a user