Skip to content

feat(orchestrator): build extension profiles from configuration - #545

Merged
behinddwalls merged 1 commit into
mainfrom
sq/orchestrator-profiles
Aug 13, 2026
Merged

feat(orchestrator): build extension profiles from configuration#545
behinddwalls merged 1 commit into
mainfrom
sq/orchestrator-profiles

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

Which implementation of each extension a queue resolved to was hardcoded: the build runner was the fake one for every queue regardless of what was available, and the change provider was a global all-or-nothing environment gate. A deployment could not run one queue against a real provider next to one running entirely on fakes, which is what a stack serving both a test environment and a live repository needs.

What?

PROFILES_CONFIG_PATH names a YAML file selecting the change provider, build runner, and conflict analyzer per queue. Each extension is independently optional, so a queue that differs only in its analyzer says only that. kind is an open string rather than a closed schema, so supporting a new provider is a new value and an implementation behind it, not a change to the file's shape.

The file holds no secret: each integration names the environment variable carrying its credential.

The pathoverlap analyzer takes a by granularity — file, or directory to coarsen a queue's conflicts to a shared parent. It defaults to file, the narrower of the two: a default that widened what conflicts would serialize a queue more than its configuration asked for.

With no config file the built-in example topology applies, reproducing the previous behavior exactly — including the per-queue analyzers the E2E suite depends on, and a routing change provider that still falls back to the fake when no token is set. That is what keeps the existing suite meaningful as a regression gate.

Extensions are reused across queues configured alike. This is load-bearing for the build runner: the build and buildsignal controllers look it up separately and the fake holds a build's outcome in memory, so two instances would lose the result between triggering a build and polling it.

Test Plan

bazel test //service/submitqueue/orchestrator/server:go_default_test — pins the built-in topology against what the E2E suite expects, covers per-extension inheritance, provider defaults, every validation rejection, that queues configured alike share one build runner, that each queue's analyzer behaves as configured, and that a missing token fails at startup rather than mid-merge.

✅ Added with the by granularity: it defaults to file, directory is accepted, and an unrecognized value is rejected at load rather than at the first queue that resolves it.

@behinddwalls
behinddwalls marked this pull request as ready for review August 7, 2026 18:24
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 7, 2026 18:24
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from 0c95872 to 2390a96 Compare August 7, 2026 18:36
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from 2390a96 to 1a26c63 Compare August 7, 2026 21:21
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from 1a26c63 to a2b03cd Compare August 7, 2026 22:19
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch 2 times, most recently from 5a99ea1 to 71c6078 Compare August 11, 2026 21:06
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from 71c6078 to 33143de Compare August 11, 2026 22:39
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from 33143de to edac343 Compare August 11, 2026 22:52
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from edac343 to d72a5b4 Compare August 12, 2026 01:57
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch 2 times, most recently from b1d2195 to f8e4063 Compare August 12, 2026 19:04
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from f8e4063 to 2c7736e Compare August 12, 2026 20:24
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from 2c7736e to f904b2f Compare August 12, 2026 20:47
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch 2 times, most recently from 420022e to f904b2f Compare August 12, 2026 21:37
Base automatically changed from sq/runway-merge-targets to main August 12, 2026 21:41
## Summary

### Why?

Which implementation of each extension a queue resolved to was hardcoded: the build runner was the fake one for every queue regardless of what was available, and the change provider was a global all-or-nothing environment gate. A deployment could not run one queue against a real provider next to one running entirely on fakes, which is what a stack serving both a test environment and a live repository needs.

### What?

`PROFILES_CONFIG_PATH` names a YAML file selecting the change provider, build runner, and conflict analyzer per queue. Each extension is independently optional, so a queue that differs only in its analyzer says only that. `kind` is an open string rather than a closed schema, so supporting a new provider is a new value and an implementation behind it, not a change to the file's shape.

The file holds no secret: each integration names the environment variable carrying its credential.

The `pathoverlap` analyzer takes a `by` granularity — `file`, or `directory` to coarsen a queue's conflicts to a shared parent. It defaults to `file`, the narrower of the two: a default that widened what conflicts would serialize a queue more than its configuration asked for.

With no config file the built-in example topology applies, reproducing the previous behavior exactly — including the per-queue analyzers the E2E suite depends on, and a routing change provider that still falls back to the fake when no token is set. That is what keeps the existing suite meaningful as a regression gate.

Extensions are reused across queues configured alike. This is load-bearing for the build runner: the build and buildsignal controllers look it up separately and the fake holds a build's outcome in memory, so two instances would lose the result between triggering a build and polling it.

## Test Plan

✅ `bazel test //service/submitqueue/orchestrator/server:go_default_test` — pins the built-in topology against what the E2E suite expects, covers per-extension inheritance, provider defaults, every validation rejection, that queues configured alike share one build runner, that each queue's analyzer behaves as configured, and that a missing token fails at startup rather than mid-merge.

✅ Added with the `by` granularity: it defaults to `file`, `directory` is accepted, and an unrecognized value is rejected at load rather than at the first queue that resolves it.
@behinddwalls
behinddwalls force-pushed the sq/orchestrator-profiles branch from f904b2f to 782e786 Compare August 12, 2026 21:41
@behinddwalls
behinddwalls added this pull request to the merge queue Aug 13, 2026
Merged via the queue into main with commit a16dcd8 Aug 13, 2026
15 checks passed
@behinddwalls
behinddwalls deployed to stack-rebase August 13, 2026 16:31 — with GitHub Actions Active
@behinddwalls
behinddwalls deleted the sq/orchestrator-profiles branch August 13, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants