fix(inference): prepend publisher prefix for Vertex non-Anthropic models - #2735
fix(inference): prepend publisher prefix for Vertex non-Anthropic models#2735politerealism wants to merge 1 commit into
Conversation
|
All contributors have signed the DCO ✍️ ✅ |
9fe55d5 to
1f14173
Compare
|
recheck |
1f14173 to
ae9c221
Compare
|
recheck |
BlockedGator is blocked because the current-head DCO check is failing: commit Head SHA: Next action: @politerealism, add |
Vertex AI's OpenAI-compatible endpoint requires the request body's model field to carry a publisher prefix (e.g. google/gemini-2.5-flash), but validate_vertex_model_id rejects slash as a path-traversal guard. This created a deadlock: bare model IDs pass validation but are rejected by Vertex with HTTP 400 "Malformed publisher model"; prefixed IDs are rejected at configuration time. Fix: in resolve_vertex_ai_route, compute body_model_id for non-Anthropic routes by prepending the publisher from infer_vertex_publisher() or the explicit VERTEX_AI_PUBLISHER config value. The bare model_id still goes through the path-traversal validator unchanged. Anthropic rawPredict routes encode the model in the URL path, not the body, and are unaffected. Both the project/region path and the base-URL-override path apply the prefix. For unrecognised models with no explicit publisher the bare ID is forwarded unchanged; Vertex's 400 is the correct observable signal in that case. Add an integration test in openshell-router that spins up a mock Vertex endpoint accepting only the publisher-prefixed form and rejecting the bare model name, verifying the body rewrite produces the required format. Closes NVIDIA#2351 Signed-off-by: politerealism <burdcat17@gmail.com>
ae9c221 to
5402076
Compare
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
Validation: Project-valid localized fix for confirmed Vertex AI routing bug #2351.
Head SHA: 54020761b65bb32dc423828f8fb7d6ac8f5ae131
Base SHA: c5498239e66581d5842b50773eff07ace65726f1
Merge base SHA: 0f8fad23c4712afc1d4a7b07a06d635b030e9521
Patch ID: d7629ffbecc6e86cc445e539e5dccd096341a601
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Blocking findings:
GATOR-54020761-01: the publisher-prefixed model is lost at the inference-bundle boundary, so live sandbox requests still use the bare model.
Carried findings:
- None
Non-blocking suggestions:
- None
Docs: No docs update is needed for this fix because it restores the already documented Vertex non-Anthropic routing path without changing configuration or user workflow.
Next state: gator:in-review
| route_name, | ||
| base_url, | ||
| model_id, | ||
| &body_model_id, |
There was a problem hiding this comment.
gator-agent
Warning — GATOR-54020761-01
Invariant: The publisher-prefixed Vertex model identifier must survive route resolution into the inference bundle consumed by the supervisor router.
Prerequisite: An operator configures a supported non-Anthropic google-vertex-ai route whose publisher is inferred or supplied with VERTEX_AI_PUBLISHER.
Entry point → sink: openshell inference set --provider <vertex-provider> --model gemini-2.5-flash, followed by an OpenAI-compatible request through inference.local → the supervisor converts GetInferenceBundleResponse.model_id into ResolvedRoute.model, which the router writes into the outbound Vertex request body.
Base → head: Base carried the bare model through resolution and runtime. Head creates the prefixed value here, so endpoint verification can use it, but resolve_route_by_name_with_credentials still exports model_id: config.model_id.clone(); the supervisor therefore reconstructs the live route with the bare model.
Impact: Live Gemini, Llama, Mistral, AI21, DeepSeek, or explicitly published Vertex requests still fail with a malformed publisher model even when endpoint verification succeeds.
Reproducer: Store a Vertex provider and inference.local route for gemini-2.5-flash, resolve the inference bundle, and assert its route model_id is google/gemini-2.5-flash. At this head it remains gemini-2.5-flash; converting the bundle and preparing a request emits the bare model.
PR ownership: This PR introduces body_model_id specifically to repair runtime body shaping, but the value is not propagated across the production server-to-supervisor bundle boundary. The added router test manually constructs a prefixed route and bypasses that boundary.
Requested change: Populate the resolved bundle's model_id from resolved.route.model while retaining the configured bare ID only in persisted/user-facing configuration, and add a bundle-path test proving the supervisor receives the prefixed model. This applies to both system/user routes and both Vertex endpoint construction branches.
Summary
google-vertex-aiprovider type cannot route to Gemini models — publisher-prefix requirement conflicts with model ID validation #2351 wheregemini-2.5-flashpasses validation but Vertex rejects it with HTTP 400 "Malformed publisher model", andgoogle/gemini-2.5-flashwould be accepted by Vertex but is rejected by CLI validation (slash guard)resolve_vertex_ai_route, computes abody_model_idfor non-Anthropic routes by prepending the publisher frominfer_vertex_publisher()(e.g.googlefor Gemini,metafor Llama) or the explicitVERTEX_AI_PUBLISHERconfig value — the baremodel_idstill goes through the path-traversal validator unchangedbackend.rsintegration test with a mock Vertex server that accepts onlygoogle/gemini-2.5-flashand explicitly expects zero hits on the bare model nameRelated Issue
Closes #2351
Changes
crates/openshell-server/src/inference.rsbody_model_idwith publisher prefix in non-Anthropic branch; update existing tests; add 4 new unit testscrates/openshell-router/src/backend.rsTesting
cargo check -p openshell-server— cleancargo clippy -p openshell-server -p openshell-router— cleancargo test -p openshell-router— 18 passedexpect(0)on bare model ensures pre-fix path is never hitChecklist
Signed-off-bypresent