feat(speculate): hold speculating until the batch can be sent to merge

[behinddwalls]

Summary

Why?

A request's trail read batched → speculating → speculated → speculating → speculated → landing → landed, and the repeats looked like the pipeline regressing. They were not a reporting glitch: RequestStatusSpeculated meant "a build passed on a path still consistent with how its dependencies are resolving", so it was published while the batch was still blocked, and reportSpeculation republished speculating whenever a dependency later resolved against that path's guess. Each extra pair was one speculative guess that passed and was then invalidated.

That made speculated a per-path, provisional fact wearing a status — the exact shape RequestEvent exists for. A batch is not done speculating until it can be sent to merge; waiting on dependencies is still speculating.

What?

Two events join the vocabulary. waiting records that a path passed and the batch has nothing of its own left to run; invalidated records that a dependency resolved against the guess that path made. Both are occurrence-keyed on the path ID, so a passed path re-observed across runs collapses to one entry.

waiting is gated on outcomeWait rather than on merely holding a live passed path. A merge is decided on that same predicate — mergeablePath implies livePassedPath — so an ungated report would claim a wait on every request that merges straight through. reportSpeculation moves below decide to see the outcome; both it and decide only read, so the reorder observes nothing different.

speculated stays a status but now means speculation finished, published from dispatchMerge once the batch is cleared to merge. It goes ahead of the dispatch because the merge stage publishes landing as its first act on receiving one, and both statuses are non-terminal — so a speculated sent afterwards could carry the later timestamp and beat landing in the summary.

The hadPassed && !hasPassed republish of speculating is gone. The status never leaves, so there is nothing to republish, and the oscillation goes with it.

One trade-off worth naming: speculated is now near-instantaneous, so "is this batch blocked on dependencies?" is answerable from the latest event rather than from the status.

The second commit adds the end-to-end coverage this had been missing. Nothing in test/integration/ touches the speculate pipeline — the orchestrator integration suite is TestPingAPI and nothing else — and the e2e happy path has no dependencies, so it never speculates across one. e2e-respeculate-queue is registered in the gateway's queue list and deliberately takes no profile of its own: falling through to the baseline is what gives it the all analyzer, which serializes the queue so a second request becomes a batch depending on the first.

The new test forces the wait rather than racing it. Batch IDs come from a per-queue counter as <queue>/batch/<n>, so on a fresh queue the leader is batch/1, and the build topic partitions by batch — closing the consumer gate on that partition before anything is published holds the leader's build and nothing else. The follower reaches a passed path while its dependency is still outstanding, reports waiting, and is asserted to still be speculating. Releasing the gate fails the leader, and the follower re-plans and lands with speculating and speculated recorded exactly once each.

invalidated is deliberately not asserted end to end. A passed path stops occupying build budget, so by the time the leader fails the follower has usually funded the other side of the guess as well; it never loses its last live passed path, which is the state invalidated reports. Forcing that end to end would mean starving the queue's budget, which cannot be done without also starving the follower's first build. A unit test covers the case e2e cannot reach deterministically: a dependency turning terminal in the same run that walks the head resting on it, so the break is seen by a later generation of the finalize loop rather than by the read.

A third commit adapts TestDependentBatch_IsWokenByTheMergeAhead, which #576 added to main after this branch was cut. It parks the lead batch's merge behind a closed gate, waits for the dependent to reach speculated, and only then opens the gate — an ordering that encodes the old meaning of speculated, which a batch could reach while its dependency was still outstanding. Under this change the dependent rests at speculating instead, so the test waited for a status that could not arrive until it opened the gate, and did not open the gate until it arrived. It now waits for the waiting event: the same fact it was reaching for, expressed as the signal that carries it, and reachable while the lead is still parked. The gate still opens next and both requests are still asserted to land, so the wake-up remains attributable to the fan-out alone.

Test Plan

✅ bazel test //submitqueue/... //platform/... //service/... — 73 tests pass
✅ bazel test //test/e2e/... — 3/3 pass, including the new scenario
✅ make lint, make check-gazelle, make check-tidy, make check-mocks

Re-verified after rebasing onto main at 42d1cb72: bazel test //submitqueue/... //platform/... — 69 tests pass; bazel test //test/e2e/submitqueue — passes in 120s, against the 300s timeout it hit before the third commit.

The two reportSpeculation tests now assert events. New unit coverage: a merging head reports speculated and no wait (the gate's regression test); speculated is published before the TopicKeyMerge dispatch; and the same-run cascade reports invalidated.

TestLand_HappyPath_ReachesLanded needs no change: speculating → speculated → landing → landed still holds as an ordered subsequence, now for a different reason and at a different point in time.

Issue

Closes https://linear.app/uber/issue/CODEM-443

Stack

1. fix(speculate): write the merge state before dispatching the batch #585
2. @ feat(speculate): hold speculating until the batch can be sent to merge #586