Skip to content

feat(example): create, enqueue, and watch demo pull requests - #569

Open
behinddwalls wants to merge 5 commits into
sq/provider-examplefrom
sq/demo-pr
Open

feat(example): create, enqueue, and watch demo pull requests#569
behinddwalls wants to merge 5 commits into
sq/provider-examplefrom
sq/demo-pr

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Why?

Exercising the demo meant opening pull requests by hand and pasting their URLs into a land command — slow enough to discourage running it, and it made the interesting case impossible to reach in practice. A queue that only ever holds one request in flight never batches, never analyzes a conflict against another batch, and never speculates.

What?

make demo-pr creates pull requests in the scratch repository, enqueues them, and watches them settle.

Each pull request is enqueued the moment it is created, so the queue is already working on the first while the last is still being opened. Nothing is awaited until everything is in; the wait happens once, against all of them. That overlap is what puts requests in flight against each other, which is the whole point of running more than one.

STACKED=true is the exception, and necessarily so: one request carries the whole chain, so it can only be submitted once every change in it exists. That is the atomic-stack path, where the set reaches the target in a single push.

Progress is a table keyed on the gateway's status API, redrawn as requests move, so it is visible which stage each one is in rather than only whether it finished. On a terminal it redraws in place; piped to a file it prints a fresh table whenever something changes, so logs stay readable.

It lives beside the configuration it drives, under service/submitqueue/example/, and reaches GitHub through the REST API rather than a clone — so it needs no checkout and no git binary, only GITHUB_TOKEN, the credential the stack already uses.

Test Plan

✅ bazel build //service/submitqueue/example/pr; make help lists demo-pr.

Running it end to end needs a scratch repository and a token, so it is exercised the same way the rest of the provider path is — manually. See doc/howto/PROVIDER-E2E.md.

Also in this PR

  • feat(demo): live table with the full state trail per request
  • feat(messagequeue): level the queue's own logs separately
  • docs(demo): correct the sample trail to states the pipeline emits
  • feat(demo): spread each change over a sharded file tree

Each commit above carries its own rationale and test plan in its message.

Issues

@behinddwalls
behinddwalls marked this pull request as ready for review August 11, 2026 21:08
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 11, 2026 21:08
@behinddwalls
behinddwalls force-pushed the sq/demo-pr branch 2 times, most recently from 08c97bd to 69e03e6 Compare August 12, 2026 05:47
@behinddwalls
behinddwalls force-pushed the sq/demo-pr branch 2 times, most recently from fdd3ea5 to 1159b1b Compare August 12, 2026 21:41
@behinddwalls
feat(example): create, enqueue, and watch demo pull requests
528556f 
## Summary

### Why?

Exercising the demo meant opening pull requests by hand and pasting their URLs into a land command — slow enough to discourage running it, and it made the interesting case impossible to reach in practice. A queue that only ever holds one request in flight never batches, never analyzes a conflict against another batch, and never speculates.

### What?

`make demo-pr` creates pull requests in the scratch repository, enqueues them, and watches them settle.

Each pull request is enqueued the moment it is created, so the queue is already working on the first while the last is still being opened. Nothing is awaited until everything is in; the wait happens once, against all of them. That overlap is what puts requests in flight against each other, which is the whole point of running more than one.

`STACKED=true` is the exception, and necessarily so: one request carries the whole chain, so it can only be submitted once every change in it exists. That is the atomic-stack path, where the set reaches the target in a single push.

Progress is a table keyed on the gateway's status API, redrawn as requests move, so it is visible which stage each one is in rather than only whether it finished. On a terminal it redraws in place; piped to a file it prints a fresh table whenever something changes, so logs stay readable.

It lives beside the configuration it drives, under `service/submitqueue/example/`, and reaches GitHub through the REST API rather than a clone — so it needs no checkout and no git binary, only `GITHUB_TOKEN`, the credential the stack already uses.

## Test Plan

✅ `bazel build //service/submitqueue/example/pr`; `make help` lists `demo-pr`.

Running it end to end needs a scratch repository and a token, so it is exercised the same way the rest of the provider path is — manually. See `doc/howto/PROVIDER-E2E.md`.
@behinddwalls
feat(demo): live table with the full state trail per request
74a8768 
## Summary

### Why?

The table only appeared once every pull request existed, and then showed a single word per row. Both are the wrong way round for the thing the demo is meant to make visible.

Creation scrolled past as loose `created` / `enqueued` lines while the table waited, so the most interesting stretch — early requests moving through the queue while later ones are still being opened — had nothing to look at. At `COUNT=20` that is most of the run.

The single word came from polling `GetRequestSummaryByID` every two seconds, which does not just fail to show where a request has been: it misses transitions outright. A request that batches and speculates between two ticks shows neither. That a request went `accepted → started → batched → speculating → landed` is the story worth telling, and sampling cannot tell it.

### What?

The rows are allocated before anything is created — one per land request, or a single row for a stack — so the table is complete from the first draw and only ever fills in. Whatever is happening right now is one line underneath it rather than output that pushes the table away.

Statuses come from `GetRequestHistoryByID`, which returns every recorded transition, so the trail is complete rather than sampled. It replaces the summary call rather than adding to it, and current status and last error fall out of the same response.

A `tracker` owns the rows behind a mutex and polls on its own clock, concurrently with creation. Polling only after the last pull request existed is what left the trail empty for the whole creation phase.

`CHANGES` links to the pull request: an OSC 8 hyperlink on a terminal, the full URL when piped. `ELAPSED` starts when the gateway accepts the request and freezes when it settles, so a finished row keeps the time it took.

An error is wrapped onto continuation lines indented under its row rather than truncated into a cell. The tail of a pipeline error is usually the part that says what went wrong.

Two latent bugs in the renderer are fixed along the way. The cursor-up distance was fixed at construction and would have drifted once the table gained a status line; it now counts the lines it actually emitted. And OSC 8 escapes carry many bytes at zero display width, so padding and the truncation budget both had to stop measuring them — measuring the rendered prefix cut a 51-character trail down to 24 with 85 columns free.

## Test Plan

✅ `bazel test //service/submitqueue/demo/pr:go_default_test`, covering trail derivation, the frozen clock, visible-width padding, wrapping, and the line accounting the redraw depends on.

✅ Same suite under `--@rules_go//go/config:race`, which is what the concurrent poll-and-update test is for.

✅ `make test` — 98 targets.

The alignment test was checked against the bug it guards: reinstating the rune-count budget fails `TestRowLineAlignment`.

Running it end to end needs a scratch repository and a token, so it is exercised the same way the rest of the provider path is — manually. See `doc/howto/PROVIDER-E2E.md`.
@behinddwalls
feat(messagequeue): level the queue's own logs separately
e5e50dd 
## Summary

### Why?

The services run at debug, and the queue logs a line per message published, fetched, leased and acked. Reading a service's logs means scrolling past hundreds of `fetched messages {"count": 0}` to find the one line that matters — following a request through the pipeline is impractical at the level everything else is useful at.

Turning the whole service down to info loses the output that is actually being read. The queue's chatter is the part worth silencing, not the service's.

### What?

`Params.LogLevel` sets the minimum level for the queue's own logs, applied with `zap.IncreaseLevel` so the rest of the service keeps the level it was built with. Empty selects info, which is what stops the chatter by default.

The four servers pass `QUEUE_LOG_LEVEL` through, and the compose files forward it, so the queue can be turned back up for a run that is chasing a message that never arrived:

```
QUEUE_LOG_LEVEL=debug make local-submitqueue-start
```

The level is a raw string on `Params`, parsed in the package, which keeps the environment lookup in the wiring layer where the other knobs live. `IncreaseLevel` can only raise the level, never lower it, so this cannot be used to make a quiet service verbose — an unparseable value is rejected at construction rather than silently ignored.

## Test Plan

✅ `bazel test //platform/extension/messagequeue/mysql:go_default_test` — the level is accepted, and a value that is not a level fails construction.

✅ `make test` — 98 targets.

✅ Passthrough verified rather than assumed, since compose forwards only what it declares: `QUEUE_LOG_LEVEL=debug docker compose -f service/submitqueue/docker-compose.yml config` shows the variable set on all three services.
@behinddwalls
docs(demo): correct the sample trail to states the pipeline emits
45b237a 
## Summary

### Why?

The sample table showed `accepted → started → batched → speculating → landed`, which cannot happen. `RequestStatusSpeculating` is defined in the entity package and never published by any non-test code — along with `validating`, `batching`, `speculated`, `building`, `built`, `waitingpath`, `landing` and `processing`. Nine of the eighteen statuses are dead vocabulary today.

Getting this wrong in the one place that shows what the tool looks like is worse than saying nothing, because it invites exactly the wrong conclusion: a request sitting on `batched` for minutes looks stuck, when its batch is in fact speculating and building normally with nothing to report it.

### What?

The sample is the real thing the renderer produces for a trail the pipeline can actually emit, and the text says plainly what the vocabulary is limited to and how to read a request that rests on `batched`.

The gap itself is CODEM-426.

## Test Plan

✅ Sample generated from the renderer rather than written by hand, so the column widths and rule lengths are the ones the tool produces.
@behinddwalls
feat(demo): spread each change over a sharded file tree
8ed373b 
## Summary

### Why?

Every demo change wrote exactly one file, and every one of those files went into a single flat `demo/` directory. Two things follow from that, and neither is what the demo wants to show. The directory grows without bound as runs accumulate, until it is the kind of directory nobody wants in a repository. And a pull request that adds one line to one file is not much of a pull request — it exercises none of the handling a real change needs, and reads as a toy in a walkthrough whose whole purpose is to look like the real thing.

### What?

A change now writes several files, each at a path sharded into two levels of hex buckets under `demo/` — `demo/c2/91/<tag>-<change>-<file>.txt`. The buckets are the leading bytes of the leaf name's SHA-256, so the spread is uniform without any coordination and stable across runs, and no directory becomes a dumping ground however many times the demo is run.

Uniqueness is preserved, and it is load-bearing rather than incidental: the leaf carries the run tag, the change index and the file index, so no two files in a run and no two runs against one repository can name the same path. Independent changes that collided on content would make the run measure conflict handling instead of the throughput it exists to show. Two unrelated changes landing in the same bucket is expected and harmless — the bucket is only a directory, and it is the leaf that has to be distinct.

How many files a change touches varies rather than being fixed, so a run does not produce a row of identically shaped pull requests. `-files` sets the floor (default 3, exposed as `FILES` on `make demo-pr`) and the actual count runs a little above it. The variation is derived from the run tag and the change index rather than from a clock, so replaying a tag reproduces the same run — a demo that cannot be reproduced is hard to talk about once something in it goes wrong.

Each file is committed separately, so a change arrives as a multi-file, multi-commit pull request. That is closer to a real change, and it means the demo exercises replaying a range of commits rather than always handing the merger a single one.

## Test Plan

✅ `bazel test //service/submitqueue/demo/pr:go_default_test` — five new cases over the two pure functions: paths are unique across every change and file of two runs, the layout is the demo root plus hex buckets plus the leaf, 80 files spread over more than 50 distinct buckets (a layout that put everything in one directory would pass the uniqueness test alone), the file count respects its floor and clamps a non-positive one, and the count both varies across changes and reproduces for a given tag.

✅ Cross-checked the generated layout against an independent recomputation of the same hashes in shell, rather than only against the implementation's own output.

Not yet exercised against GitHub: the path and count logic is covered above, but the multi-commit loop against the live API — several sequential commits on one branch, and the last SHA as the head the change URI pins — will first run on the next `make demo-pr`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mnoah1 mnoah1 mnoah1 approved these changes
@sbalabanov sbalabanov Awaiting requested review from sbalabanov sbalabanov is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@behinddwalls @mnoah1