Skip to content

fix(inference): prepend publisher prefix for Vertex non-Anthropic models - #2735

Open
politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:fix/vertex-gemini-publisher-prefix
Open

fix(inference): prepend publisher prefix for Vertex non-Anthropic models#2735
politerealism wants to merge 1 commit into
NVIDIA:mainfrom
politerealism:fix/vertex-gemini-publisher-prefix

Conversation

@politerealism

Copy link
Copy Markdown
Contributor

Summary

  • Fixes the deadlock in google-vertex-ai provider type cannot route to Gemini models — publisher-prefix requirement conflicts with model ID validation #2351 where gemini-2.5-flash passes validation but Vertex rejects it with HTTP 400 "Malformed publisher model", and google/gemini-2.5-flash would be accepted by Vertex but is rejected by CLI validation (slash guard)
  • In resolve_vertex_ai_route, computes a body_model_id for non-Anthropic routes by prepending the publisher from infer_vertex_publisher() (e.g. google for Gemini, meta for Llama) or the explicit VERTEX_AI_PUBLISHER config value — the bare model_id still goes through the path-traversal validator unchanged
  • Applies to both the project/region path and the base-URL-override path; Anthropic rawPredict routes are unaffected (model encoded in URL, not body)
  • Adds a backend.rs integration test with a mock Vertex server that accepts only google/gemini-2.5-flash and explicitly expects zero hits on the bare model name

Related Issue

Closes #2351

Changes

File Change
crates/openshell-server/src/inference.rs Compute body_model_id with publisher prefix in non-Anthropic branch; update existing tests; add 4 new unit tests
crates/openshell-router/src/backend.rs Add integration test with mock Vertex endpoint verifying publisher-prefixed body model field

Testing

  • cargo check -p openshell-server — clean
  • cargo clippy -p openshell-server -p openshell-router — clean
  • cargo test -p openshell-router — 18 passed
  • Integration test: mock Vertex server accepts prefixed form, expect(0) on bare model ensures pre-fix path is never hit

Checklist

  • Conventional Commits format
  • DCO Signed-off-by present
  • No unrelated changes
  • Existing tests updated to assert new behaviour; no tests deleted

@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@politerealism
politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from 9fe55d5 to 1f14173 Compare August 13, 2026 16:40
@politerealism

Copy link
Copy Markdown
Contributor Author

recheck

@politerealism
politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from 1f14173 to ae9c221 Compare August 13, 2026 20:11
@politerealism

Copy link
Copy Markdown
Contributor Author

recheck

@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Blocked

Gator is blocked because the current-head DCO check is failing: commit ae9c2211b10f591283d6a1e1f800c54b2f2cc8fb is authored as Quinn Burdine with sburdine@redhat.com, but GitHub does not associate that commit author identity with a GitHub account.

Head SHA: ae9c2211b10f591283d6a1e1f800c54b2f2cc8fb
Gator payload: 4

Next action: @politerealism, add sburdine@redhat.com to the GitHub account that owns this commit (or replace the commit with one using an email already associated with that account), then comment recheck so DCO Assistant reruns.

@johntmyers johntmyers added the gator:blocked Gator is blocked by process or repository gates label Aug 13, 2026
Vertex AI's OpenAI-compatible endpoint requires the request body's model
field to carry a publisher prefix (e.g. google/gemini-2.5-flash), but
validate_vertex_model_id rejects slash as a path-traversal guard. This
created a deadlock: bare model IDs pass validation but are rejected by
Vertex with HTTP 400 "Malformed publisher model"; prefixed IDs are
rejected at configuration time.

Fix: in resolve_vertex_ai_route, compute body_model_id for non-Anthropic
routes by prepending the publisher from infer_vertex_publisher() or the
explicit VERTEX_AI_PUBLISHER config value. The bare model_id still goes
through the path-traversal validator unchanged. Anthropic rawPredict routes
encode the model in the URL path, not the body, and are unaffected. Both
the project/region path and the base-URL-override path apply the prefix.
For unrecognised models with no explicit publisher the bare ID is forwarded
unchanged; Vertex's 400 is the correct observable signal in that case.

Add an integration test in openshell-router that spins up a mock Vertex
endpoint accepting only the publisher-prefixed form and rejecting the bare
model name, verifying the body rewrite produces the required format.

Closes NVIDIA#2351

Signed-off-by: politerealism <burdcat17@gmail.com>
@politerealism
politerealism force-pushed the fix/vertex-gemini-publisher-prefix branch from ae9c221 to 5402076 Compare August 13, 2026 23:54

@johntmyers johntmyers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

PR Review Status

Validation: Project-valid localized fix for confirmed Vertex AI routing bug #2351.
Head SHA: 54020761b65bb32dc423828f8fb7d6ac8f5ae131
Base SHA: c5498239e66581d5842b50773eff07ace65726f1
Merge base SHA: 0f8fad23c4712afc1d4a7b07a06d635b030e9521
Patch ID: d7629ffbecc6e86cc445e539e5dccd096341a601
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no

Blocking findings:

  • GATOR-54020761-01: the publisher-prefixed model is lost at the inference-bundle boundary, so live sandbox requests still use the bare model.

Carried findings:

  • None

Non-blocking suggestions:

  • None

Docs: No docs update is needed for this fix because it restores the already documented Vertex non-Anthropic routing path without changing configuration or user workflow.

Next state: gator:in-review

route_name,
base_url,
model_id,
&body_model_id,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gator-agent

Warning — GATOR-54020761-01

Invariant: The publisher-prefixed Vertex model identifier must survive route resolution into the inference bundle consumed by the supervisor router.

Prerequisite: An operator configures a supported non-Anthropic google-vertex-ai route whose publisher is inferred or supplied with VERTEX_AI_PUBLISHER.

Entry point → sink: openshell inference set --provider <vertex-provider> --model gemini-2.5-flash, followed by an OpenAI-compatible request through inference.local → the supervisor converts GetInferenceBundleResponse.model_id into ResolvedRoute.model, which the router writes into the outbound Vertex request body.

Base → head: Base carried the bare model through resolution and runtime. Head creates the prefixed value here, so endpoint verification can use it, but resolve_route_by_name_with_credentials still exports model_id: config.model_id.clone(); the supervisor therefore reconstructs the live route with the bare model.

Impact: Live Gemini, Llama, Mistral, AI21, DeepSeek, or explicitly published Vertex requests still fail with a malformed publisher model even when endpoint verification succeeds.

Reproducer: Store a Vertex provider and inference.local route for gemini-2.5-flash, resolve the inference bundle, and assert its route model_id is google/gemini-2.5-flash. At this head it remains gemini-2.5-flash; converting the bundle and preparing a request emits the bare model.

PR ownership: This PR introduces body_model_id specifically to repair runtime body shaping, but the value is not propagated across the production server-to-supervisor bundle boundary. The added router test manually constructs a prefixed route and bypasses that boundary.

Requested change: Populate the resolved bundle's model_id from resolved.route.model while retaining the configured bare ID only in persisted/user-facing configuration, and add a bundle-path test proving the supervisor receives the prefixed model. This applies to both system/user routes and both Vertex endpoint construction branches.

@johntmyers johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:blocked Gator is blocked by process or repository gates labels Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:in-review Gator is reviewing or awaiting PR review feedback

Projects

None yet

Development

Successfully merging this pull request may close these issues.

google-vertex-ai provider type cannot route to Gemini models — publisher-prefix requirement conflicts with model ID validation

2 participants