fix(realtime): keep the file-doc store reconnecting instead of dying quietly

[icecrasher321]

A relay that lost Redis for longer than its retry budget did not degrade — it went silently split-brain and stayed that way until the process restarted.

What was happening

Leaving a tab open long enough for the Upstash connection to drop produced this, twice a second, forever:

[Realtime] [WARN] [FileDocStore] FileDocStore reader error; retrying { "error": "The client is closed" }
[Realtime] [WARN] [FileDocStore] FileDocStore reader error; retrying { "error": "The client is closed" }
…

Two bugs stacked:

The client gave up permanently. The reconnect strategy returned an Error after ten attempts:

reconnectStrategy: (retries) => {
  if (retries > 10) return new Error('FileDocStore Redis reconnection failed')
  return Math.min(retries * 100, 3000)
}

Returning an Error there tells node-redis to stop reconnecting and close the client, and a closed client rejects every command with "The client is closed" for the rest of the process's life. So an outage longer than roughly a minute did not degrade the task — it took it out, quietly. The task kept serving clients while:

• its rooms stopped receiving other tasks' updates (the split-brain the store exists to prevent),
• its own edits stopped reaching the shared stream, which is also the crash buffer between persists,
• seeds, the seed/merge/compaction locks, and the persist If-Match token all failed.

The tail loop spun on it. runReader's catch only breaks on !this.running, which is true solely during shutdown — so a permanently-closed client was retried every 500ms forever, one warning per attempt. A tab left open overnight yields thousands of identical lines, which is how the actual failure stayed invisible.

Changes

• Never stop reconnecting. This process holds live documents whose only convergence path is that connection, so a connection it can rebuild is always worth rebuilding. Same capped backoff, now through the shared backoffWithJitter, and no error return.
• The reader backs off after a failed read (500ms → 10s) instead of retrying at the read cadence, re-opens a client that was closed — node-redis reconnects a dropped client, never a closed one — and logs the first failure of a streak then one in twenty, carrying the streak length, so an outage stays visible without burying itself.

Verification

A test models a closed connection (xRead throws The client is closed and flips isOpen) and asserts the loop slowed down and tried to recover:

	read attempts in 3s
before	6 — test fails
after	~3, plus a connect()

278 realtime tests, tsc and biome clean.

This is independent of the collaborative-files work it was found alongside: any long-lived Redis drop hits it.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 13, 2026 5:34pm

[cursor]

PR Summary

High Risk
Changes core realtime multi-replica convergence (Redis stream tailer and reconnect behavior); prolonged outages previously caused permanent silent divergence until process restart.

Overview
Fixes silent split-brain when Redis stays down past the old reconnect cap: node-redis was told to stop reconnecting (returning an Error from reconnectStrategy), which closes the client and leaves every command failing with "The client is closed" for the rest of the process.

Redis clients now use capped backoffWithJitter reconnect delays and never return an error from the strategy, so the connection keeps trying to come back.

The multiplexed reader tail loop no longer retries failed reads every 500ms with a warning each time. It backs off (500ms up to 10s), resets the failure streak when any read succeeds—including idle blocking timeouts with no new messages—so a later blip does not start at max backoff, re-opens the read client when isOpen is false, and throttles logs (first failure of a streak, then every 20).

Tests extend the in-memory Redis fake for closed-client behavior and add coverage for backoff/reconnect and streak reset on idle reads.

^{Reviewed by Cursor Bugbot for commit 064b4aa. Configure here.}

[greptile-apps]

Greptile Summary

The PR keeps the Redis-backed file-document store recoverable during extended connection outages.

• Replaces the finite Redis reconnection budget with indefinite capped jittered backoff.
• Adds reader-level retry backoff, rate-limited failure logging, and recovery for closed clients.
• Adds tests covering closed-client recovery and failure-streak reset after an idle successful read.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/realtime/src/handlers/file-doc-store.ts	Introduces indefinite Redis reconnection, backed-off reader recovery, closed-client reopening, and correct failure-streak reset after any successful read.
apps/realtime/src/handlers/file-doc-store.test.ts	Adds closed-reader recovery and idle-read streak-reset coverage while consistently using the shared sleep helper.

Sequence Diagram

sequenceDiagram
  participant Reader as FileDocStore reader
  participant Redis as Redis client
  Reader->>Redis: xRead()
  Redis-->>Reader: read failure
  Reader->>Reader: Increment failure streak
  Reader->>Reader: Sleep with capped jittered backoff
  alt Client is closed
    Reader->>Redis: connect()
  end
  Reader->>Redis: xRead()
  Redis-->>Reader: messages or idle timeout
  Reader->>Reader: Reset failure streak

Loading

_{Reviews (3): Last reviewed commit: "test(realtime): assert the retry delay, ..." | Re-trigger Greptile}

[icecrasher321]

@greptile

[icecrasher321]

@cursor review

[icecrasher321]

@greptile

[icecrasher321]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 064b4aa. Configure here.}