A WebSocket deployment succeeds when clients can reconnect and recover useful application state. Replacing the server process is only one part of that outcome. If every client retries immediately, the new instance receives a burst of handshakes just as it is warming up. If the client resumes without knowing which events it processed, a successful connection can still leave the application inconsistent.
Two recent r/node discussions exposed these separate failure modes. One described deployments dropping thousands of sockets. Another asked how to deduplicate events and test reconnects when the feed provides no replay mechanism. These are useful engineering questions even for a small service.
The approach below divides the work into three contracts: how a server retires, how a client retries, and how application state recovers. The code demonstrates controlled shutdown and bounded deduplication. Durable replay requires additional application infrastructure, which we will describe explicitly rather than hide behind a reconnect helper.
Download the complete runnable examples, including setup instructions and verification commands.
Define what recovery means for your application
Start with the consequence of a missing event. A live activity indicator can usually refresh from the current state. A financial ledger or an inventory change requires a durable record and an accountable processing path. The same transport can support both applications, but their recovery policies must differ.
Write down your event identity, ordering scope, retention window, and source of truth. An event identifier answers whether two deliveries represent the same event. A sequence number answers which event follows another within a defined stream. A timestamp may help with monitoring or retention; it is usually a poor substitute for either identity or ordering.
For a dashboard, a reasonable recovery promise might be: reconnect, fetch a current snapshot, and continue from a snapshot boundary. For an event processor, it might be: resume after a stored cursor, apply each event transactionally, and tolerate repeated delivery. Neither promise comes automatically from opening another WebSocket.
Treat retirement as a state transition
A server should move from accepting work to draining work before it exits. During that transition, it must refuse new upgrades while existing connections have time to finish their closing handshake. New instances should be ready before the old instance starts retiring.
A practical sequence is to mark readiness unavailable, allow the routing change to propagate, stop accepting new TCP connections, spread close frames across existing clients, and wait for completion. Enforce a hard deadline for clients that never finish. The routing propagation allowance and close window must fit inside the process supervisor's termination budget.
Readiness and liveness have different purposes. An instance can be alive and intentionally unavailable for new connections. Returning failure from readiness during a deploy should remove it from new traffic; it should not create a cycle in which another watchdog repeatedly restarts it before cleanup finishes. Kubernetes documents its termination sequence and grace period in the Pod lifecycle reference. Your reverse proxy or hosting platform may provide different hooks, so verify the actual behavior in that environment.
Readiness changes routing for future connections. Existing upgraded connections still need their own retirement path.
Keep upgrade admission under application control
The ws library supports an external HTTP server and explicit upgrade handling. That arrangement lets the application check its retirement state before accepting a connection. In the full example, the WebSocket server uses noServer: true, a small payload limit, and this admission check:
server.on('upgrade', (req, socket, head) => {
socket.on('error', () => {});
if (draining || req.url !== '/events') {
socket.end(
'HTTP/1.1 503 Service Unavailable\r\n' +
'Connection: close\r\n\r\n'
);
return;
}
wss.handleUpgrade(req, socket, head, ws => {
wss.emit('connection', ws, req);
});
});This local demonstration deliberately omits authentication. A public service needs authentication and authorization before upgrade, an appropriate origin policy for browser clients, TLS at the deployment boundary, and limits appropriate to its messages. Keep an upgrade rejection cheap: a retiring instance should not perform expensive session recovery before deciding that it cannot accept the socket.
The important race is a request arriving after the readiness change but before the router stops forwarding traffic. The local draining flag closes that gap. The same check also helps when a proxy keeps an upstream connection alive beyond the moment you expected routing to change. The library's external-server and authentication examples show the underlying integration points.
Close WebSockets explicitly and bound the wait
Calling the HTTP server's close method is insufficient for a complete WebSocket retirement policy. Node distinguishes ordinary HTTP connections from sockets that have upgraded to another protocol. Its HTTP server documentation explains that closing all HTTP connections does not destroy upgraded WebSocket sockets.
The example tracks connections through wss.clients. It schedules a close frame for each open connection at a random point inside a short spread window. Code 1001, meaning that an endpoint is going away, is appropriate for this demonstration. The application should document how its clients interpret planned retirement and abnormal failures.
const hardStop = setTimeout(() => {
for (const ws of wss.clients) ws.terminate();
server.closeAllConnections();
}, deadlineMs);
await delay(propagationMs);
const httpClosed = new Promise((done, fail) => {
server.close(error => error ? fail(error) : done());
});
const wsClosed = new Promise(done => wss.close(done));
for (const ws of wss.clients) {
setTimeout(() => {
if (ws.readyState === WebSocket.OPEN) {
ws.close(1001, 'deploy');
}
}, Math.floor(Math.random() * spreadMs));
}
await Promise.all([httpClosed, wsClosed]);
clearTimeout(hardStop);The complete runnable version also retains and clears scheduled timers, makes repeated drain calls return the same promise, and registers signal handlers. Once the application installs a signal listener, it owns the shutdown path; it should release its resources so the process can finish. Node documents that behavior in its signal event reference.
Set the supervisor grace period longer than the application's deadline, with enough margin for cleanup and scheduling delays. A deadline protects the rollout from a permanently stuck client. It does not guarantee delivery of a final application message, and forcefully terminating a socket can discard buffered data.
Spread retries on the client as well
Server-side spreading helps planned deployments. It cannot coordinate a power failure, a network partition, or a proxy restart. Client-side backoff remains necessary because those events may disconnect the whole population at once.
Full jitter chooses a random delay between zero and an exponentially increasing ceiling. A cap prevents the delay from growing without limit. Injecting the random function makes the policy deterministic in tests:
export function retryDelay(attempt, random = Math.random) {
const ceiling = Math.min(
30_000,
500 * 2 ** Math.min(attempt, 16)
);
return Math.floor(random() * ceiling);
}The values are example policy choices, not performance recommendations for every service. Choose them around your user experience, connection admission capacity, and expected recovery time. Where appropriate, distinguish retryable network failures from authentication or policy failures that require another action.
Use one path to schedule reconnects. With ws, a failed handshake can emit an error and then a close event. Scheduling a retry in both handlers creates parallel connection attempts. In the supplied client, the error handler records the problem and the close handler schedules the next attempt. Intentional application shutdown cancels that timer.
Avoid resetting the attempt counter immediately on every open event. A server that accepts a handshake and fails a moment later would otherwise keep clients in their shortest retry window. The example resets after a stable connection interval. Production applications may instead use an acknowledged subscription or another meaningful recovery milestone.
Deduplicate by event identity
A bounded identifier cache is useful when duplicates are harmless to detect locally and replay overlaps are short. The following implementation preserves insertion order and evicts the oldest identifier when it reaches its configured capacity:
export function createDeduper({ maxIds = 10_000 } = {}) {
const seen = new Map();
return {
accept(id) {
if (typeof id !== 'string' || id.length === 0) {
throw new TypeError('event id required');
}
if (seen.has(id)) return false;
seen.set(id, true);
if (seen.size > maxIds) {
seen.delete(seen.keys().next().value);
}
return true;
},
get size() { return seen.size; }
};
}Use it after validating the message shape and before updating a disposable display. The example client logs accepted events, so marking an identifier before logging is acceptable for the demonstration. A durable processor needs a stronger boundary: recording completion and changing business state should succeed atomically, or a crash can leave them inconsistent.
The cache provides a limited memory of identifiers within one running process. Evicted events can be accepted again. Restarting the process clears it. Separate workers have separate caches. The correct cache size depends on event rate and the expected duplicate window; a count limit alone does not establish a time guarantee.
Timestamps introduce different mistakes. Two events can share a timestamp, events can arrive out of order, and clocks can disagree. A last-seen timestamp may accidentally discard a legitimate delayed event. Preserve identifiers for deduplication, and use a separately defined cursor when the source supports replay.
Make the recovery boundary visible
Reconnecting restores the transport. Replay or reconciliation restores application knowledge.
With a durable event log, a client can reconnect with the last committed cursor. The server replays retained events after that cursor and then joins the live stream. Each event still needs an identity because the boundary may deliberately overlap, especially when an acknowledgment was lost.
The replay-to-live handoff must avoid a second gap. A simple implementation might read history, subscribe to live delivery, and accidentally miss an event committed between those operations. Use a log that provides a continuous ordered subscription, or capture a boundary and buffer live events while applying history. Specify whether ordering applies globally, per account, per document, or per partition.
A snapshot also needs a boundary. Returning current state without a version leaves the client unable to distinguish already represented updates from newer ones. Prefer a snapshot paired with a cursor or version, then apply events after that point. If the cursor has expired from retention, tell the client to obtain another snapshot rather than silently skipping history.
When the upstream feed has no replay, neither a timestamp nor a local file can reconstruct events that never arrived. The honest options are to reconcile against another authoritative API, rebuild from a current snapshot, show a data-gap indicator, or accept loss because the feature is explicitly best effort. Document the choice where operators and downstream consumers can see it.
Test the failure path on demand
A useful test suite should trigger retirement directly instead of waiting for a random outage. The supplied integration test starts the server on an ephemeral loopback port, opens a real WebSocket client, and confirms that readiness initially succeeds. It begins draining, verifies readiness returns 503, attempts another upgrade, and verifies that the existing client receives close code 1001.
The test also checks idempotency: a second drain request must return the same completion promise. Separate unit cases verify that distinct out-of-order identifiers are accepted, repeated identifiers are rejected, and an evicted identifier can be accepted again. That final assertion is intentional documentation of the cache's limit.
npm install
npm test
# In two separate terminals:
npm start
npm run client
# Send SIGTERM to the server process and watch the client retry.The example was checked with Node 22.12.0 and an available ws 8.17.1 installation; all four tests passed. The package manifest requests a maintained 8.x release for a new installation. Treat the test output as behavioral evidence for these examples, not a throughput benchmark or a security endorsement of the older validation dependency.
Additional deployment tests should cover a failed close handshake, a client with an expired token, an interrupted replay, a stale cursor, a restart during event application, and a router that continues forwarding briefly after readiness fails. The fourth integration case uses a raw peer that never answers the close handshake and verifies that the hard deadline terminates it. Repeat this behavior through your actual proxy in staging.
Operate the contracts you designed
Track open connections per instance, connection admission failures, close codes, reconnect attempts, time to restore subscriptions, and the number of forced terminations. For durable consumers, add cursor lag, replay failures, and duplicate application attempts. For best-effort feeds, measure and expose known gaps instead of representing them as zero loss.
Alert on symptoms that change a user's outcome. A few planned closes during a rollout are expected. A growing population that repeatedly connects and immediately disconnects is a recovery problem even if the health endpoint stays green. Logs should connect the deployment identifier, instance identifier, and recovery session without recording credentials or sensitive message bodies.
Before calling the rollout complete, demonstrate three outcomes: retiring instances stop accepting upgrades, retry traffic remains within the new instance's admission capacity, and clients recover the state promised by the application contract. Those outcomes are testable. Keeping them explicit makes long-lived connections a manageable part of ordinary Node.js operations.
