Skip to content

perf(ci): accelerate Windows MSVC validation - #2738

Open
pimlock wants to merge 27 commits into
mainfrom
fix/windows-msvc-cache/pimlock
Open

perf(ci): accelerate Windows MSVC validation#2738
pimlock wants to merge 27 commits into
mainfrom
fix/windows-msvc-cache/pimlock

Conversation

@pimlock

@pimlock pimlock commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add Windows MSVC CI jobs for pull requests, merges to main, and cache warming. The jobs use prebuilt Z3 plus Cargo target and sccache layers to make native Windows validation practical.

Related Issue

No issue required: localized CI infrastructure and performance work following #2496.

Changes

  • add a Windows MSVC pull-request job that lints Windows-supported crates and runs the supported Rust tests
  • add a main cache-seeding job that runs the same lint/test workload to keep PR caches warm
  • add a dependent main job that builds release binaries without publishing them
  • use official prebuilt Z3 archives instead of compiling Z3 from source
  • layer the Cargo target cache with the GitHub Actions sccache backend
  • document the Windows CI, supported crate scope, and cache contract
  • make the OpenTelemetry in-memory exporter an explicit test dependency

Testing

  • mise run pre-commit passes
  • cargo test -p openshell-otel passes
  • target-cache and sccache experiments completed on windows-2025
  • fully warm sccache achieved 62/62 compiler-cache hits with no errors
  • hosted Windows MSVC PR lint and tests pass
  • Branch Checks, E2E, Helm, and required aggregate gates pass
  • E2E tests added/updated (not applicable; CI-only change)

Detailed timing and cache measurements are recorded in the PR comments.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated

pimlock added 11 commits August 12, 2026 11:04
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@pimlock
pimlock marked this pull request as ready for review August 13, 2026 23:31
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Windows MSVC cache experiments

I ran the hosted windows-2025 workflow repeatedly while changing one major cache/build variable at a time. Times below are end-to-end workflow/job times and can vary slightly with runner setup and GitHub cache transfer latency.

Configuration Cache state Runtime Notes
Original workflow / bundled Z3 effectively cold ~75m48s–78m22s Baseline; Z3 compiled from source and the Rust target tree was not providing a useful warm reuse path. run, repeat
Cargo target cache / bundled Z3 cold ~75m16s Populated the target archive. run
Cargo target cache / bundled Z3 warm ~18m39s Large improvement; target artifacts were the most effective cache layer. run
sccache only / bundled Z3 cold ~80m47s Target cache disabled to isolate sccache. Remote writes made the cold run slower. run
sccache only / bundled Z3 warm ~67m55s 1,157 hits / 122 misses (90.46% hit rate), but linking, non-cacheable crate types, Z3 work, and remote reads left total time high. run
Cargo target cache + prebuilt Z3 cold ~37m28s z3-sys downloaded the pinned official prebuilt Z3 archive instead of compiling Z3 via CMake/MSBuild. run
Cargo target cache + prebuilt Z3 warm ~16m24s Check 1m14s, release build 6m36s, tests 4m46s, focused test 2m12s; target restore ~58s. run
Target cache + prebuilt Z3 + sccache target warm, sccache cold ~17m09s 62 misses, 62 successful writes, 0 errors; cache writes took 32.6s. run
Target cache + prebuilt Z3 + sccache both warm ~15m45s 62/62 sccache hits, 0 misses/errors; check 1m07s, release build 5m19s, tests 5m06s, focused test 2m25s. run
Warm hybrid cache, redundant focused test removed both warm ~12m33s Check 1m01s, release build 5m07s, full tests 4m58s. The full suite already includes the Windows unsupported-driver contract test. run

Conclusions

  • Cargo target caching was the largest win: roughly 75 minutes cold to roughly 19 minutes warm with bundled Z3.
  • Prebuilt Z3 materially improved cold builds: the cold target-cache run dropped from roughly 75 minutes to roughly 37 minutes.
  • sccache alone was not competitive, despite a high hit rate. It does not cache final linking or several Rust crate types, and GitHub remote reads still cost time.
  • sccache is useful as a secondary layer on top of the target cache: it reduced the warm release build from roughly 6m36s to 5m19s and produced a clean 100% hit rate for the 62 cacheable compiler requests in the measured run.
  • The best measured stack is Cargo target cache + prebuilt Z3 + sccache, with the redundant focused test removed.
  • The PR/main split is intended to keep PR validation focused on Windows Clippy + Rust tests, run that same workload on main to warm the exact PR cache namespaces, and build (but not upload/publish) release binaries after the seed succeeds.

Current validation note

The earlier single-job hybrid workflow is proven by the successful runs above. The new multi-job split currently receives an opaque GitHub startup_failure before any job is created, on both manual dispatch and the PR mirror. actionlint reports the workflow as valid, and a temporary ref at the last known-good commit starts normally, so I am continuing to isolate the workflow-graph difference before treating the split as ready.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Validation update: the opaque pre-job startup_failure was isolated to jdx/mise-action v4.2.4 on windows-2025. Restoring the previously measured v4.2.0 pin fixes workflow startup. The PR mirror is now running PR lint and test (x64), and a manual dispatch is independently running Seed cache (x64) followed by Build binaries (x64). The newer Swatinem/rust-cache v2.9.2 pin remains in place.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant