test(gem): hermetic hosted capstone + CHECKSUMS/transitive/scan --vendor coverage; gems.rb + CRLF redirect fixes

[Mikola Lysenko (mikolalysenko)]

STACKED on PR #175 (fix/gem-bundler-audit) — base branch is fix/gem-bundler-audit, not main.

Closes the gem audit's top coverage gaps with hardened tests, plus the two small verified redirect fixes the new tests exercise.

Product fixes (both pinned by tests that fail on the base commit)

1. gems.rb/gems.locked hosted support — REDIRECT_CANDIDATE_FILES (cli hosted.rs) now reads the modern pair, and rewrite_gem keys on whichever pair bundler actually reads: gems.rb wins when both spellings exist (verified empirically on bundler 4.0.15, matching setup::gem::discover_bundler_project); byte-identical twins proceed on gems.rb; spellings that diverge beyond the redirect's own footprint fail closed with the new redirect_gem_gemfile_spellings_diverge warning. Divergence is judged on a residue (gem_spelling_residue) with Socket's managed source blocks (rotating grant token/uuid wildcarded) and the redirected deps' own gem lines erased — a raw-byte comparison would trap every re-run behind the divergence run 1 itself creates on identical twins, permanently blocking the rotated-grant URL refresh and hosted record refresh (review finding, fixed in the polish commit). Ledger edit paths carry the real filenames. Before: scan --mode hosted was a silent no-op on gems.rb projects.
   • Pinned by: gems_rb_pair_is_rewritten_with_modern_paths, gems_rb_beats_identical_gemfile, gems_rb_and_gemfile_diverging_fail_closed, gems_rb_identical_twins_rerun_is_a_no_op_not_a_diverge_trap, gems_rb_identical_twins_rerun_refreshes_rotated_grant_url, gems_rb_identical_twins_rerun_after_appended_block_is_no_op, gems_rb_divergence_only_in_redirected_dep_line_proceeds (unit, RED on base / RED on a raw-byte guard) + gem_hosted_gems_rb_spelling_redirects_and_installs (e2e, RED without the candidate-list entry — verified).
2. CRLF Gemfile.lock tolerance — the CHECKSUMS matchers accept \r-terminated lines and both the pin-in-place and the insert-after-header edits preserve the file's CRLF endings byte-for-byte (real bundler accepts CRLF locks — verified via bundle check + frozen install on 4.0.15). Before: a CRLF lock was misdiagnosed as bundler <2.6 (redirect_gem_no_checksums_section) and never pinned.
   • Pinned by: gem_crlf_lock_checksum_pinned_preserving_crlf, gem_crlf_lock_checksums_header_gains_crlf_entry (byte-exact, incl. CRLF rerun no-op; RED on base).

New coverage

3. tests/e2e_redirect_gem_build.rs — hermetic hosted gem capstone (4 tests, #[ignore]-gated like e2e_vendor_gem_build). Fully hermetic: fixture gems authored in-test and built with the real gem build; ONE wiremock plays the upstream compact index, the Socket patch-registry compact index (production's /patch-registry/gem/<token>/<uuid>/ base, real per-info md5 digests in /versions, checksum: sha256s in /info), and the patches API. Chain: scan --mode hosted --vex → fresh checkout → REAL bundle install → installed bytes byte-match afterHash, runtime dep installs, require probe, post-install hash-verified VEX.
   • Pins the compact-index dependency contract production currently violates: the fixture gem has a runtime dep and /info declares it; the red-arm test (…without_deps_breaks_install_like_production) proves a deps-less /info (today's prod shape: not_built index / zero-byte deps API) breaks the install with bundler's APIResponseMismatchError … dependencies not in the API — the exact live-CI signature.
   • NEW FINDING pinned as a canary (gem_hosted_checksums_lock_pins_patched_sha_but_bundler_refuses_mixed_state): on a CHECKSUMS lock (bundler ≥ 4 writes one by default), the current rewrite (Gemfile source block + CHECKSUMS pin, GEM section left on upstream) makes the prescribed unfrozen install fail with "Bundler found mismatched checksums" (exit 37) — bundler still attributes the gem to the upstream source and refuses the lockfile-vs-upstream-API disagreement. Empirically verified on 4.0.15; also verified the fix target: the fully converged lock (patch-registry GEM section + name (= ver)! DEPENDENCIES pin + patched CHECKSUMS sha) passes even a FROZEN install. That fix must land in the TS twin + shared golden fixtures together, so it is out of scope here; the canary makes the breakage visible and self-documents the flip when fixed.
4. docker_e2e_vendor_gem.rs — lockfile_checksums twin (replaces the module-doc TODO): bundle lock --add-checksums on the image's bundler 2.7, vendor swaps the registry sha256= line for bundler's bare path-gem CHECKSUMS form, frozen --network none fresh-checkout install accepts the rewritten lock byte-stably (the exit-16 hazard), revert restores the registry sha256= line VERBATIM, re-vendor re-bares it. Host-side oracle re-asserts from the mounted files. The existing no-CHECKSUMS test is unchanged (both flavors covered).
5. e2e_vendor_gem_build.rs — transitive-dep capstone: rack vendored as a TRANSITIVE dep of rack-test; the managed block (# >>> socket-patch vendor (managed) >>> …) is byte-asserted with hand-pinned marker constants, the DEPENDENCIES pin inserts at bundler's sorted position, and — for the first time — that pair is fed to a REAL frozen bundle install (byte-stable lock, probe through the rack-test require chain), then revert round-trips byte-identically. Lock shape pinned to no-CHECKSUMS on every host (lockfile_checksums false) so bundler 2.5 (CI pin) and 4.x agree; the CHECKSUMS flavor is the docker twin's job.
6. in_process_vendor.rs — gem through scan --vendor (hermetic bundler layout, no ruby needed): JSON-arm end-to-end (manifest written, pair edit + patched artifact + stub gemspec from specifications/, pristine installed tree, already_vendored rerun) + manifest-drop reconcile byte-restore + --detached variant (no manifest, embedded record, vendor --revert exit path).

Empirical validation (before baking into fixtures/code)

Against real bundler 4.0.15 (host) and 2.7.2 (docker image): gems.rb-beats-Gemfile + gems.locked naming; compact-index /versions md5 + /info deps/checksum grammar (a hand-built index accepted by a real bundle install); checksum enforcement (lock pin vs served bytes); CRLF lock acceptance; bundler-4 CHECKSUMS-by-default vs 2.7 opt-in; bundle lock --add-checksums availability; the deps-less /info failure signature; and the converged-lock frozen-install fix shape.

Test results

• e2e_redirect_gem_build (new): 4 passed (hermetic, ~6s)
• docker_e2e_vendor_gem: 2 passed (both flavors, real bundler 2.7 in docker)
• e2e_vendor_gem_build --ignored: 2 passed (incl. new transitive capstone, real bundler 4.0.15)
• in_process_vendor: 29 passed (2 new gem tests)
• socket-patch-core full suite: all green (incl. redirect_golden — classic-spelling output byte-unchanged), patch::redirect 73 passed
• redirect-family CLI suites (in_process_redirect, in_process_redirect_pnpm, e2e_vex_redirect, scan_vendor_e2e, cli_parse_scan, e2e_redirect_rush_sim --ignored, cli --lib): all green
• cargo fmt + cargo clippy --all-targets (both crates, with features): clean

Follow-ups / out of scope (not in my owned files)

• TS-twin port of this PR's two redirect fixes + the shared golden fixtures (depscan registry-rewrite gem rewriter + golden.test.ts): gems.rb/gems.locked pair routing, the residue-based redirect_gem_gemfile_spellings_diverge guard (warning-code parity included), and CRLF Gemfile.lock CHECKSUMS tolerance are CLI-side only for now — the backend twin still silently no-ops on a gems.rb project and misdiagnoses a CRLF lock as bundler <2.6. The shared golden cases (gems.rb pair, diverge fail-closed, CRLF lock) must land TOGETHER with the TS port: depscan's golden.test.ts runs every fixture case with no skip list (verified), so fixtures added here first would break depscan CI at the next submodule bump. depscan has GitHub issues disabled — needs a depscan PR/internal ticket.

• ci.yml e2e matrix entry for the new e2e_redirect_gem_build suite (needs setup-ruby with bundler ≥ 2.6 pinned; the existing gem leg pins 2.5 for e2e_vendor_gem_build). Until added, the suite runs only via --ignored locally.

• CHECKSUMS-lock hosted redirect fix (the canary finding): needs the converged-lock rewrite in the gem rewriter + the TS twin + shared golden fixtures moving together. Alternative shapes verified: removing the pinned line also unblocks (bundler re-records the patched sha from the registry API) but loses first-fetch enforcement.

• The hosted-production e2e gem leg (e2e_hosted_production.rs) can adopt the APIResponseMismatchError red-arm once the prod compact index is rebuilt.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes gem hosted redirect lockfile rewriting (rewrite_gem), which affects how scan --mode hosted edits Gemfiles and locks; risk is mitigated by broad new e2e coverage rather than large unrelated refactors.

Overview
Hosted gem redirect now reads gems.rb/gems.locked (candidate list + rewriter). The rewriter edits the pair bundler actually uses (gems.rb when both spellings exist and match); diverging Gemfile vs gems.rb content fails closed with redirect_gem_gemfile_spellings_diverge. Ledger edit paths use the real filenames. CRLF Gemfile.lock is handled for CHECKSUMS pinning without mis-firing redirect_gem_no_checksums_section.

Tests close major gem gaps: hermetic e2e_redirect_gem_build (wiremock compact index + scan --mode hosted → fresh bundle install → VEX), including gems.rb spelling, a deps-less /info red-arm (APIResponseMismatchError), and a CHECKSUMS-lock canary documenting today’s mixed-state install failure until a converged-lock rewrite lands. Docker lockfile_checksums twin for vendor CHECKSUMS rewrite/revert; host transitive rack vendoring + frozen install; in-process scan --vendor for gem (reconcile + detached).

^{Reviewed by Cursor Bugbot for commit b17172b. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Polish pass for the adversarial review findings (commit 8479fb0):

Major — gems.rb divergence one-way trap: FIXED. The guard now judges divergence on a redirect-footprint residue (gem_spelling_residue) instead of raw bytes: Socket's managed source … do … end blocks (grant token/uuid wildcarded via the shared gem_index_url_pattern, which also replaced the inline wildcarding in the idempotency guard) and the redirected deps' own gem declaration lines are erased from both spellings before comparing. Run 1 on identical twins no longer traps every later run — the rotated-grant URL refresh and hosted record confirmation work again — while genuine divergence outside the footprint still fails closed. Empirically re-verified on bundler 4.0.15 that the post-run-1 diverging pair still resolves gems.rb (bundler warns and ignores the Gemfile). Pinned by 4 new unit tests that FAIL on the raw-byte guard (verified by temporarily restoring it): identical-twins re-run no-op, rotated-grant re-run URL refresh, appended-block (transitive) re-run no-op, and dep-own-line divergence proceeding. The existing fail-closed test now diverges in an unrelated line — by design the redirected dep's own declaration is part of the footprint.

Minor — TS-twin / golden-fixture coupling: documented, fixtures deliberately NOT landed. depscan's golden.test.ts runs every fixture case with no skip list (verified in the depscan source), so gems.rb/CRLF/diverge golden cases added here would break depscan CI at the next submodule bump until the TS port lands — and the TS port is outside this repo. Added a follow-ups entry to the PR body covering the TS port of both fixes + the shared golden cases landing together, including redirect_gem_gemfile_spellings_diverge warning-code parity. depscan has GitHub issues disabled, so this needs a depscan PR/internal ticket rather than an issue.

Minor — docs/ecosystems.md gem row: FIXED. Hosted cell now notes gems.rb/gems.locked editing (bundler's preference order) and the fail-closed divergence warning; vendored cell notes the Gemfile-spelling-only asymmetry (a gems.rb project cannot vendor yet).

Tests: patch::redirect 73 passed (69 + 4 new); full socket-patch-core suite green incl. redirect_golden (classic-spelling bytes unchanged); e2e_redirect_gem_build --ignored 4/4 on host bundler 4.0.15; in_process_redirect / in_process_redirect_pnpm / in_process_vendor / cli --lib green (--features docker-e2e,setup-e2e); cargo fmt + clippy --all-targets clean.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Rebased onto the updated base fix/gem-bundler-audit tip d063cb1 (was branched from 10c0079). Old tip 8479fb0 → new tip 7195b64, force-pushed with --force-with-lease.

Conflict resolution (one textual conflict, in crates/socket-patch-core/src/patch/redirect/mod.rs, inside rewrite_gem's block recognizer):

• Kept this PR's gem_index_url_pattern helper for the rotating-segment wildcarding, and the base's \r?\n CRLF tolerance in block_re plus its explanatory comment. The base's gemfile_rerun_on_crlf_checkout_never_nests test carries over unchanged and passes.
• Semantic unification: the base's recognizer now accepts CRLF source blocks, so this PR's residue-based divergence guard (gem_spelling_residue) had to erase them too — its block regex was LF-only, so after a core.autocrlf checkout of redirected gems.rb/Gemfile twins, the Socket block survived into gems.rb's residue and every re-run (rotated-grant URL refresh included) was trapped behind redirect_gem_gemfile_spellings_diverge. The residue eraser now matches do\r?\n … end\r?, mirroring the recognizer. Pinned by the new test gems_rb_crlf_twins_rerun_is_no_op_and_rotated_grant_refreshes, verified red (exact diverge-trap failure) with an LF-only eraser before the fix.

Verification on the rebased tip (union of both sides' behavior):

• cargo test -p socket-patch-core: all green (2098 lib tests), including the base's gemfile_rerun_on_crlf_checkout_never_nests and all of this PR's gems.rb/CRLF tests.
• cargo test -p socket-patch-cli --features docker-e2e,setup-e2e --test in_process_scan --test in_process_vendor: 26 + 29 passed, including the base's four new dry-run VEX tests (scan_vendor_dry_run_with_vex_interactive_*, scan_apply_json_dry_run_with_vex_*).
• Hosted gem capstone --test e2e_redirect_gem_build -- --ignored against real host ruby 3.4.10 / bundler 4.0.15: all 4 passed (fresh-checkout install, gems.rb spelling, deps red-arm, CHECKSUMS canary).
• cargo fmt --check and cargo clippy --all-targets clean on both crates.

🤖 Generated with Claude Code

[Mikola Lysenko (mikolalysenko)]

Rebased test/gem-coverage-hardening onto the freshly-rebased base fix/gem-bundler-audit (new base tip 99f9cd0, which now carries main's 11 new commits — --vendor-source, hosted JSON envelope changes, maven/nuget GA, etc.). Transplanted only this PR's own 2 commits via git rebase --onto origin/fix/gem-bundler-audit d063cb1; new tip is 707bbf8.

Conflicts: none — the rebase applied cleanly with zero textual conflicts and zero content drift (interdiff of old tip vs new tip is exactly the base-side change). The two files both sides touch were checked semantically: scan/hosted.rs (this PR adds gems.rb/gems.locked to the redirect candidate list, on top of main's takeover-warning changes) and docs/ecosystems.md (gem row edit rebased onto main's table update) — both coherent.

Verification (all green):

• --test in_process_scan (27 passed) + --test in_process_vendor (29 passed) with --features docker-e2e,setup-e2e
• cargo test -p socket-patch-core redirect: 83 unit tests + redirect_golden_fixtures_match
• e2e_redirect_gem_build -- --ignored vs real host ruby 3.4.10 / bundler 4.0.15: 4 passed (fresh-checkout install, gems.rb twin, deps red-arm, CHECKSUMS canary)
• --test docker_e2e_vendor_gem: 2 passed (fresh socket-patch-test-gem:latest)
• clippy: CI's exact gate (--workspace --all-features -- -D warnings) passes clean

Pre-existing note (not from this PR): cargo fmt --check on the touched crates reports rustfmt-1.93.1 diffs, but every dirty hunk is base/main-owned code (e.g. scan/mod.rs/hosted.rs takeover lines from #164, update_notifier.rs, #169's e2e_vendored_production.rs) that this PR never modified — the PR's own hunks are fmt-clean, and CI has no fmt gate. Likewise clippy --all-targets shows 3 pre-existing doc_lazy_continuation warnings in main's e2e_vendored_production.rs (outside CI's gate). Left untouched to avoid smuggling base-side churn into this stacked PR.

🤖 Generated with Claude Code