Skip to content

feat(evaluation): add tool parameter matching - #6726

Open
gioboa wants to merge 1 commit into
google:mainfrom
gioboa:fix/5643
Open

feat(evaluation): add tool parameter matching#6726
gioboa wants to merge 1 commit into
google:mainfrom
gioboa:fix/5643

Conversation

@gioboa

@gioboa gioboa commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue:

Problem:

The existing trajectory metric scores a tool call as entirely correct or incorrect. It cannot show partial correctness when only some expected tool arguments match.

Solution:

Add an exact tool_parameter_match metric that averages matches across expected argument keys. Register the metric and cover matching, alignment, empty-input, and partial-score cases with unit tests. This is intentionally a small MVP;advanced matching strategies proposed in #5643 are outside this PR.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All relevant unit tests pass locally.

Results:

  • Evaluation unit suite: 843 passed
  • Final focused suite: 40 passed
  • Ruff, isort, Pyink, mypy, compliance, and codespell checks passed.

Manual End-to-End (E2E) Tests:

Compared an expected search(city="Rome", rooms=1) call with an actual
search(city="Rome", rooms=2) call.

Before:

tool_trajectory_avg_score=0.0

After:

tool_parameter_match=0.5, status=PASSED

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing relevant unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • No dependent downstream changes are required for this code change.

Additional context

This PR implements exact argument matching only. Match modes, configurable per-argument strategies, and numeric tolerances can be added separately after maintainer feedback. User-facing documentation will be proposed in the adk-docs repository once the API is approved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants