[draft] "sampler" build runner to sample between multiple build runners

AGENTS.md

@@ -333,6 +333,7 @@ CI runs on every PR and enforces all checks via a `required-checks` gate. **Befo
 2. **Interfaces for behavior, structs for data** — use interfaces for behavioral contracts (Consumer, Controller, Storage). Use structs for data containers, configs, and registries (TopicRegistry, SubscriptionConfig).
 3. **Value types over pointers** — prefer value types for structs, configs, and return values. Use `(T, bool)` to signal absence instead of `*T`. Pointers only when mutation or shared ownership is needed.
 4. **Errors for failures, not control flow** — reserve `error` returns for unexpected or infrastructure failures. Use result types (structs, bools) for expected outcomes like `(Result, error)` or `(T, bool)`. Avoid sentinel errors that represent non-failure states.
+5. **Prefix unexported globals with `_`** — package-level `const` and `var` names that are unexported take a leading underscore (`_defaultSlowBuildDuration`, `_tokenFail`), per the [Uber Go style guide](https://github.com/uber-go/guide/blob/master/style.md#prefix-unexported-globals-with-_). The prefix makes a global unmistakable at its use site and makes accidental shadowing by a local obvious. Exported identifiers keep their plain names, and function-local constants are not globals — neither takes the prefix. `var _ Iface = (*impl)(nil)` interface assertions and generated code (`protopb/`, `mock/`) are exempt. Most of the existing tree predates this rule; new and edited code follows it, and older packages are migrated as they are touched rather than in one sweep.
 
 ### Error Classification (`platform/errs`)
 

service/stovepipe/README.md

@@ -16,6 +16,7 @@ Stovepipe therefore needs two MySQL databases: a **storage** database (the `requ
 
 - **`inMemoryCounter`** — a process-local `counter.Counter` for sequence numbers; not durable. A real deployment uses a persistent implementation (e.g. `platform/extension/counter/mysql`).
 - **`fakeSourceControlFactory`** — seeds each queue with a deterministic single-commit history so ingest resolves a stable head URI (and re-ingesting the same queue exercises the dedup path). A real deployment supplies a VCS-backed `sourcecontrol.Factory`.
+- **`newBuildRunnerFactory`** — builds the `buildrunner.Factory` from the environment. Every runner it hands out is bound to the resolving queue's `Config` and shares the profile the `BUILD_RUNNER_*` knobs describe (failure rate, build duration, and its spread). `BUILD_RUNNER=fake` (the default) gives each queue one fake runner; `BUILD_RUNNER=sampler` gives it two and splits builds between them by percentage, which is how a backend rollout or comparison is exercised without a real CI system. A real deployment keeps this shape and swaps the constructed runners for Buildkite or GitHub Actions ones (see [`stovepipe/extension/buildrunner`](../../stovepipe/extension/buildrunner)); a deployment that gives queues *different* backends routes on `Config.QueueName` in `For`, as the SubmitQueue orchestrator does in its `profiles.go`.
 
 ## Layout
 
@@ -40,6 +41,36 @@ The Stovepipe controllers live under [`stovepipe/controller/`](../../stovepipe/c
 | `PORT`              | no       | gRPC listen address                      | `:8083`              |
 | `HOSTNAME`          | no       | Subscriber name for the process consumer | `stovepipe-<unix_ts>` |
 
+### Build runner
+
+`BUILD_RUNNER` selects the implementation; the bare `BUILD_RUNNER_*` knobs configure the runner that takes unsampled builds, and the `_CANDIDATE_` ones configure the runner that `sampler` splits traffic with. A malformed value fails startup rather than silently falling back to the default.
+
+| Variable                                          | Required | Description                                                              | Default |
+|---------------------------------------------------|----------|--------------------------------------------------------------------------|---------|
+| `BUILD_RUNNER`                                    | no       | `fake` or `sampler`                                                      | `fake`  |
+| `BUILD_RUNNER_FAILURE_PERCENT`                    | no       | Share of builds (0-100) the runner reports as failed                     | `0`     |
+| `BUILD_RUNNER_DURATION_MS`                        | no       | How long each build reports running before turning terminal              | `0`     |
+| `BUILD_RUNNER_DURATION_JITTER_PERCENT`            | no       | Spread (0-100) applied to that duration per build                        | `0`     |
+| `BUILD_RUNNER_SAMPLE_PERCENT`                     | no       | Share of builds (0-100) routed to the candidate runner                   | `0`     |
+| `BUILD_RUNNER_CANDIDATE_FAILURE_PERCENT`          | no       | Failure share for the candidate runner                                   | `0`     |
+| `BUILD_RUNNER_CANDIDATE_DURATION_MS`              | no       | Build duration for the candidate runner                                  | `0`     |
+| `BUILD_RUNNER_CANDIDATE_DURATION_JITTER_PERCENT`  | no       | Duration spread for the candidate runner                                 | `0`     |
+
+The defaults reproduce the original behavior: one fake runner that succeeds immediately unless a request's head URI carries a `buildrunner-fake=<token>` marker. Markers keep working under a configured rate — they pin the outcome for the build that carries them.
+
+Durations are configured as a typical value plus a spread, so the bounds stay easy to state: `BUILD_RUNNER_DURATION_MS=60000` with `BUILD_RUNNER_DURATION_JITTER_PERCENT=25` means every build takes between 45s and 75s, drawn uniformly. A jitter of `0` pins every build to exactly the configured duration; `100` is the widest setting, spanning "terminal immediately" to twice the duration.
+
+Under Compose these are set through `SQ_`-prefixed variables, so a sampled stack that sends a tenth of its builds to a slow, flaky runner starts with:
+
+```bash
+SQ_BUILD_RUNNER=sampler \
+SQ_BUILD_RUNNER_SAMPLE_PERCENT=10 \
+SQ_BUILD_RUNNER_CANDIDATE_FAILURE_PERCENT=50 \
+SQ_BUILD_RUNNER_CANDIDATE_DURATION_MS=5000 \
+SQ_BUILD_RUNNER_CANDIDATE_DURATION_JITTER_PERCENT=40 \
+make local-stovepipe-start
+```
+
 ## Running
 
 ### Docker Compose (recommended)

service/stovepipe/docker-compose.yml

@@ -68,6 +68,18 @@ services:
       - STORAGE_MYSQL_DSN=root:root@tcp(mysql-app:3306)/submitqueue?parseTime=true
       - QUEUE_MYSQL_DSN=root:root@tcp(mysql-queue:3306)/submitqueue?parseTime=true
       - HOSTNAME=stovepipe-dev
+      # Build runner selection and behavior. Unset means the default: a single
+      # fake runner that succeeds immediately unless a request's head URI
+      # carries a buildrunner-fake marker. Set SQ_BUILD_RUNNER=sampler to split
+      # builds between two fakes with different profiles — see the README.
+      - BUILD_RUNNER=${SQ_BUILD_RUNNER:-}
+      - BUILD_RUNNER_FAILURE_PERCENT=${SQ_BUILD_RUNNER_FAILURE_PERCENT:-}
+      - BUILD_RUNNER_DURATION_MS=${SQ_BUILD_RUNNER_DURATION_MS:-}
+      - BUILD_RUNNER_DURATION_JITTER_PERCENT=${SQ_BUILD_RUNNER_DURATION_JITTER_PERCENT:-}
+      - BUILD_RUNNER_SAMPLE_PERCENT=${SQ_BUILD_RUNNER_SAMPLE_PERCENT:-}
+      - BUILD_RUNNER_CANDIDATE_FAILURE_PERCENT=${SQ_BUILD_RUNNER_CANDIDATE_FAILURE_PERCENT:-}
+      - BUILD_RUNNER_CANDIDATE_DURATION_MS=${SQ_BUILD_RUNNER_CANDIDATE_DURATION_MS:-}
+      - BUILD_RUNNER_CANDIDATE_DURATION_JITTER_PERCENT=${SQ_BUILD_RUNNER_CANDIDATE_DURATION_JITTER_PERCENT:-}
     depends_on:
       mysql-app:
         condition: service_healthy

service/stovepipe/server/BUILD.bazel

@@ -25,6 +25,7 @@ go_library(
         "//stovepipe/core/messagequeue:go_default_library",
         "//stovepipe/extension/buildrunner:go_default_library",
         "//stovepipe/extension/buildrunner/fake:go_default_library",
+        "//stovepipe/extension/buildrunner/sampler:go_default_library",
         "//stovepipe/extension/queueconfig/default:go_default_library",
         "//stovepipe/extension/sourcecontrol:go_default_library",
         "//stovepipe/extension/sourcecontrol/fake:go_default_library",

service/stovepipe/server/main.go

@@ -22,6 +22,8 @@ import (
 	"net"
 	"os"
 	"os/signal"
+	"strconv"
+	"strings"
 	"sync"
 	"syscall"
 	"time"
@@ -47,6 +49,7 @@ import (
 	stovepipemq "github.com/uber/submitqueue/stovepipe/core/messagequeue"
 	"github.com/uber/submitqueue/stovepipe/extension/buildrunner"
 	buildrunnerfake "github.com/uber/submitqueue/stovepipe/extension/buildrunner/fake"
+	buildrunnersampler "github.com/uber/submitqueue/stovepipe/extension/buildrunner/sampler"
 	queueconfigdefault "github.com/uber/submitqueue/stovepipe/extension/queueconfig/default"
 	"github.com/uber/submitqueue/stovepipe/extension/sourcecontrol"
 	sourcecontrolfake "github.com/uber/submitqueue/stovepipe/extension/sourcecontrol/fake"
@@ -136,13 +139,172 @@ func (fakeSourceControlFactory) For(cfg sourcecontrol.Config) (sourcecontrol.Sou
 	return sourcecontrolfake.New(cfg, []string{fmt.Sprintf("git://%s/HEAD", cfg.QueueName)}), nil
 }
 
+// Environment variables selecting and configuring the BuildRunner. The bare
+// BUILD_RUNNER_* knobs configure the runner that takes unsampled builds; the
+// _CANDIDATE_ knobs configure the one BUILD_RUNNER=sampler splits traffic with.
+const (
+	_envBuildRunner   = "BUILD_RUNNER"
+	_envSamplePercent = "BUILD_RUNNER_SAMPLE_PERCENT"
+)
+
+// fakeEnv names the environment variables configuring one fake build runner.
+// Two sets exist so the sampler's two runners can be given different profiles
+// from the same code path.
+type fakeEnv struct {
+	failurePercent        string
+	buildDurationMs       string
+	durationJitterPercent string
+}
+
+var (
+	_baselineFakeEnv = fakeEnv{
+		failurePercent:        "BUILD_RUNNER_FAILURE_PERCENT",
+		buildDurationMs:       "BUILD_RUNNER_DURATION_MS",
+		durationJitterPercent: "BUILD_RUNNER_DURATION_JITTER_PERCENT",
+	}
+	_candidateFakeEnv = fakeEnv{
+		failurePercent:        "BUILD_RUNNER_CANDIDATE_FAILURE_PERCENT",
+		buildDurationMs:       "BUILD_RUNNER_CANDIDATE_DURATION_MS",
+		durationJitterPercent: "BUILD_RUNNER_CANDIDATE_DURATION_JITTER_PERCENT",
+	}
+)
+
 // fakeBuildRunnerFactory is the example BuildRunner factory: every queue gets a stateless fake
-// runner bound to its own config, which succeeds unless a caller embeds a failure marker in the
-// head URI. A real deployment supplies a backend-specific factory (e.g. Buildkite, per queue).
-type fakeBuildRunnerFactory struct{}
+// runner bound to its own config, carrying the process-wide profile read from the environment.
+// A real deployment supplies a backend-specific factory (e.g. Buildkite, per queue).
+type fakeBuildRunnerFactory struct {
+	params buildrunnerfake.Params
+}
+
+func (f fakeBuildRunnerFactory) For(cfg buildrunner.Config) (buildrunner.BuildRunner, error) {
+	params := f.params
+	params.Config = cfg
+	return buildrunnerfake.New(params)
+}
 
-func (fakeBuildRunnerFactory) For(cfg buildrunner.Config) (buildrunner.BuildRunner, error) {
-	return buildrunnerfake.New(cfg), nil
+// samplerBuildRunnerFactory gives each queue a sampler over two fake runners
+// bound to that queue, so the split applies per queue rather than through one
+// process-wide instance.
+type samplerBuildRunnerFactory struct {
+	baseline         buildrunnerfake.Params
+	candidate        buildrunnerfake.Params
+	candidatePercent int
+	logger           *zap.SugaredLogger
+}
+
+func (f samplerBuildRunnerFactory) For(cfg buildrunner.Config) (buildrunner.BuildRunner, error) {
+	baseline, err := fakeBuildRunnerFactory{params: f.baseline}.For(cfg)
+	if err != nil {
+		return nil, err
+	}
+	candidate, err := fakeBuildRunnerFactory{params: f.candidate}.For(cfg)
+	if err != nil {
+		return nil, err
+	}
+	return buildrunnersampler.New(buildrunnersampler.Params{
+		Config:           cfg,
+		Baseline:         baseline,
+		Candidate:        candidate,
+		CandidatePercent: f.candidatePercent,
+		Logger:           f.logger,
+	})
+}
+
+// newBuildRunnerFactory builds the BuildRunner factory from the environment.
+// BUILD_RUNNER selects the implementation: "fake" (the default) gives every
+// queue one fake runner, and "sampler" gives it two and splits builds between
+// them by percentage, which is how a rollout or a backend comparison is
+// exercised locally. Both are demo-only; a real deployment keeps this shape and
+// swaps the constructed runners for Buildkite or GitHub Actions ones.
+func newBuildRunnerFactory(logger *zap.SugaredLogger) (buildrunner.Factory, error) {
+	baseline, err := fakeBuildRunnerParams(_baselineFakeEnv)
+	if err != nil {
+		return nil, err
+	}
+
+	var factory buildrunner.Factory
+	switch impl := strings.ToLower(strings.TrimSpace(os.Getenv(_envBuildRunner))); impl {
+	case "", "fake":
+		factory = fakeBuildRunnerFactory{params: baseline}
+		logger.Infow("build runner configured", "impl", "fake")
+
+	case "sampler":
+		candidate, err := fakeBuildRunnerParams(_candidateFakeEnv)
+		if err != nil {
+			return nil, err
+		}
+		candidatePercent, err := envInt(_envSamplePercent)
+		if err != nil {
+			return nil, err
+		}
+		factory = samplerBuildRunnerFactory{
+			baseline:         baseline,
+			candidate:        candidate,
+			candidatePercent: candidatePercent,
+			logger:           logger,
+		}
+		logger.Infow("build runner configured", "impl", "sampler", "candidate_percent", candidatePercent)
+
+	default:
+		return nil, fmt.Errorf("invalid %s %q", _envBuildRunner, impl)
+	}
+
+	// Runners are built per resolution, so a profile the extensions reject
+	// would not surface until the first build. Resolve one here and discard it
+	// to keep that failure at startup.
+	if _, err := factory.For(buildrunner.Config{}); err != nil {
+		return nil, err
+	}
+	return factory, nil
+}
+
+// fakeBuildRunnerParams reads one fake runner's profile from the named
+// environment variables, leaving Config for the factory to fill in per queue.
+// The caller varies the names to give several runners different profiles.
+func fakeBuildRunnerParams(env fakeEnv) (buildrunnerfake.Params, error) {
+	failurePercent, err := envInt(env.failurePercent)
+	if err != nil {
+		return buildrunnerfake.Params{}, err
+	}
+	buildDuration, err := envDurationMs(env.buildDurationMs)
+	if err != nil {
+		return buildrunnerfake.Params{}, err
+	}
+	durationJitterPercent, err := envInt(env.durationJitterPercent)
+	if err != nil {
+		return buildrunnerfake.Params{}, err
+	}
+	return buildrunnerfake.Params{
+		FailurePercent:        failurePercent,
+		BuildDuration:         buildDuration,
+		DurationJitterPercent: durationJitterPercent,
+	}, nil
+}
+
+// envInt reads an integer from the environment, treating unset and empty as 0. A
+// malformed value fails startup rather than silently falling back to the default,
+// which would leave the stack running a profile nobody asked for. Range checks
+// belong to the extension constructors that own the valid range.
+func envInt(name string) (int, error) {
+	raw := strings.TrimSpace(os.Getenv(name))
+	if raw == "" {
+		return 0, nil
+	}
+	value, err := strconv.Atoi(raw)
+	if err != nil {
+		return 0, fmt.Errorf("invalid %s %q: %w", name, raw, err)
+	}
+	return value, nil
+}
+
+// envDurationMs reads a millisecond count from the environment as a duration,
+// treating unset and empty as zero.
+func envDurationMs(name string) (time.Duration, error) {
+	ms, err := envInt(name)
+	if err != nil {
+		return 0, err
+	}
+	return time.Duration(ms) * time.Millisecond, nil
 }
 
 func main() {
@@ -277,7 +439,10 @@ func run() error {
 	// it, so a real (stateful) backend introduced later is shared rather than
 	// silently duplicated across controllers.
 	scf := fakeSourceControlFactory{}
-	brf := fakeBuildRunnerFactory{}
+	brf, err := newBuildRunnerFactory(logger.Sugar())
+	if err != nil {
+		return err
+	}
 
 	storageFty := storageFactory{backend: store}
 	primaryCount, err := registerPrimaryControllers(primaryConsumer, logger.Sugar(), scope, storageFty, registry, scf, brf)

stovepipe/extension/buildrunner/README.md

@@ -10,4 +10,8 @@ Implementations return plain, unclassified errors — the calling controller dec
 
 Real backends (`buildkite`, `githubactions`) are thin adapters over a shared platform client (`platform/extension/buildrunner/{backend}`) — see that package's README for the HTTP client and vendor-specific details.
 
-See [doc/rfc/stovepipe/steps/build.md](../../../doc/rfc/stovepipe/steps/build.md#why-separate-contracts) for why this is a separate contract from SubmitQueue's own `buildrunner` rather than a shared one. To add a backend, create `buildrunner/{backend}/`, implement `BuildRunner`, and return it from a `New(...)` constructor.
+`fake` is the stub for local stacks and tests. By default every build succeeds immediately; its `Params` set a failure rate and a build duration — optionally spread by a percentage so durations vary within bounds the configuration states outright — for every build, and a `buildrunner-fake=<token>` marker in a request's head URI pins the outcome or the running window for one build. It keeps no per-build state — the outcome and the instant the build turns terminal are encoded in the build id — so `Status` can be answered by any instance in any process. Never production.
+
+`sampler` is a composite rather than a backend: it wraps two other `BuildRunner`s and sends a configured percentage of builds to the second one, which is how a new backend is rolled out gradually or compared against the incumbent on live traffic. The sample is drawn per `Trigger`, so the percentage is a rate over many builds. Because `Status` and `Cancel` have to reach whichever runner minted a build's opaque id, the sampler tags each id with the runner behind it and strips the tag before delegating; untagged ids route to the baseline. That tagging is what keeps it stateless across redeliveries and replicas, and it lets samplers nest.
+
+See [doc/rfc/stovepipe/steps/build.md](../../../doc/rfc/stovepipe/steps/build.md#why-separate-contracts) for why this is a separate contract from SubmitQueue's own `buildrunner` rather than a shared one. To add a backend, create `buildrunner/{backend}/`, implement `BuildRunner`, and return it from a `New(...)` constructor. Which backend serves which queue — and whether a queue gets a sampler at all — is decided in the wiring layer ([`service/stovepipe/server`](../../../service/stovepipe/server)), not here.