Pin each conformance leg to a spec-revision wire

[maxisbey]

Follow-up to #3282. Each conformance job now runs three --suite all legs — one forced to the 2025-11-25 wire, one forced to 2026-07-28, and the existing bare leg at the harness's default wire — with one baseline file per leg. The server job's active and draft legs go away.

Motivation and Context

alpha.11 froze a per-revision requirement set (conformance#447) that conformance tier-check --sdk python-sdk scores at each revision's own wire (stateful handshake for 2025-11-25, stateless per-request _meta for 2026-07-28). Our legs selected by suite (active, draft, bare all) and inherited whichever wire the harness defaults each scenario to, which is due to move when 2026-07-28 stops being the harness's draft version (conformance#426). So what CI measured at the 2025-11-25 wire was implicit and would silently change on a future pin bump.

--suite all --spec-version <rev> selects every scenario the harness marks applicable at <rev> and forces that wire. That selection is a superset of the frozen requirement set for <rev> (npx @modelcontextprotocol/conformance@0.2.0-alpha.11 list --requirements <rev> prints it), so the pinned legs give the tier signal directly: an expected-failures.<rev>.yml entry for a scenario in that set means tier-check is below 100% for that revision while CI is green. Both files are empty today (30/30 + 18/18 at 2025-11-25, 37/37 + 32/32 at 2026-07-28).

I went with pinned --suite all legs rather than --requirements <rev> legs because a requirements leg only ever reports the scenarios outside its frozen list (extensions, pending, added-after-release, and anything the harness adds later) — it never judges them — so it would still need an all leg beside it at the same wire, and would be a strict subset of that leg. The pinned all leg is the one leg that does both.

Nothing that was judged before stops being judged: at this pin active ⊂ bare all and draft ⊂ all --spec-version 2026-07-28, at identical wires. The new 2025-11-25 legs additionally hold server-session-lifecycle, server-sse-polling, json-schema-2020-12 and json-schema-2020-12-preservation to that wire explicitly rather than via the harness default. The bare leg stays because it is the only one that reaches extension-tagged scenarios (tasks-*, auth/dpop*, …), which never match a --spec-version filter; the tasks-* and per-check DPoP/WIF entries therefore stay in expected-failures.yml only. Where the bare leg overlaps a pinned leg, a failure needs an entry in both files — the headers say so.

While here, three small CI-evidence changes: both jobs get timeout-minutes: 15 (the harness's server runner has no per-scenario timeout, conformance#427); results upload on every run with if: always() so a timed-out job still uploads what finished; and since an artifact on a green run no longer means "flake rescued", run-client.sh now emits a workflow ::warning when it rescues one.

How Has This Been Tested?

All six legs run locally through run-server.sh / run-client.sh against conformance c321dd3 (== 0.2.0-alpha.11):

Leg	Result
server all --spec-version 2025-11-25 (33 scenarios)	84 passed / 0 failed
server all --spec-version 2026-07-28 (40)	151 / 0
server bare all (62)	204 / 25 — the nine tasks-*, all expected
client all --spec-version 2025-11-25 (19)	224 / 0
client all --spec-version 2026-07-28 (33)	381 / 0
client bare all (44)	458 / 9 — the nine per-check DPoP/WIF entries, all expected

Plus the rescue path on the new client 2025-11-25 leg with a wrapper client that fails sse-retry once: suite reports it unexpected → solo re-run 3/3 → exit 0, FLAKE_RESCUED written, warning annotation emitted. A wrapper that fails tools_call every time still exits 1 ("real failure, not suite contention").

Breaking Changes

None. CI-only.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Documentation
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have added or updated documentation as needed

Additional context

Step names and output directories are now <mode>-2025-11-25, <mode>-2026-07-28, <mode>-all; the server-active / server-draft artifact directories no longer exist.

_{AI Disclaimer}

[claude]

LGTM — CI-only restructuring of the conformance workflow, with the legs themselves validating the change on this PR's own run.

What was reviewed:

• The leg swap in conformance.yml: removed active/draft server legs vs. the new pinned legs, and that each leg pairs with the right baseline file and output dir.
• The new ::warning line in run-client.sh for annotation injection — scenario names are validated against ^[A-Za-z0-9/_-]+$ before reaching it, and output_dir is workflow-controlled.
• run-server.sh argument pass-through for the new --spec-version flags — verbatim "$@", no parsing to break.
• The if: always() upload and timeout-minutes: 15 — consistent with the stated goal of preserving partial results on a timeout.

Extended reasoning...

Overview

The PR touches only CI assets: .github/workflows/conformance.yml (replaces the server active/draft legs and the bare client leg layout with three --suite all legs per job — pinned to 2025-11-25, pinned to 2026-07-28, and bare at the harness default wire), a new empty baseline expected-failures.2025-11-25.yml, header rewrites in the two existing baseline files, and a small addition to run-client.sh that emits a ::warning annotation when a flake is rescued. No src/ or test code changes.

Security risks

None of substance. The workflow keeps permissions: contents: read and persist-credentials: false; all actions remain SHA-pinned and unchanged. The only new string interpolation is the ::warning echo in run-client.sh; the ${scenarios[*]} values it embeds are extracted from harness output but rejected earlier in the script unless they match ^[A-Za-z0-9/_-]+$, which cannot carry annotation-command syntax, and output_dir comes from workflow-authored arguments.

Level of scrutiny

Low-to-moderate. This is CI-only with no effect on the published package or runtime behavior. The main thing that could go wrong is a coverage regression in what conformance CI measures — the PR's own description addresses this (active ⊂ bare all, draft ⊂ the pinned 2026-07-28 leg at identical wires) and backs it with local run counts for all six legs. Crucially, the change is self-validating: the reworked workflow executes on this PR, so a mispaired baseline file, a bad flag, or a stale entry fails the run visibly rather than silently. The failure mode of an empty baseline file being passed is also handled — the runners key off the harness's own baseline output.

Other factors

The baselines follow the established pattern from #3282 (one file per leg, stale-entry enforcement, reconcile-on-pin-bump headers now referencing all three files consistently). run-server.sh forwards arguments verbatim, so the new --spec-version flags need no script changes there. The bug hunting system found no issues, and there are no outstanding reviewer comments on the timeline. The if: always() upload and timeout-minutes additions are standard CI hygiene with clear inline justification.

[cubic-dev-ai]

No issues found across 5 files

_{Re-trigger cubic}