Skip to content

fix: describe custom actions in HighLevelActionSet.to_tool_description - #422

Open
VitorCoelho wants to merge 1 commit into
ServiceNow:mainfrom
VitorCoelho:fix/tool-description-custom-actions
Open

VitorCoelho wants to merge 1 commit into
ServiceNow:mainfrom
VitorCoelho:fix/tool-description-custom-actions

Conversation

@VitorCoelho

Copy link
Copy Markdown

What

HighLevelActionSet.to_tool_description() now reads each action's signature from the function registered in the action set, instead of globals()[tool_name] in highlevel.py.

Why

globals() of the highlevel module only contains the built-in actions, so:

  • an action set built with custom_actions raised KeyError on the first custom action, so tool-use agents could not describe custom actions at all;
  • a custom action that reuses a built-in name (e.g. its own click(bid, note)) was silently described with the built-in click(bid, button, modifiers) signature.
def my_action(bid: str, times: int = 2):
    """
    Clicks an element several times.

    Examples:
        my_action('a51', 3)
    """

HighLevelActionSet(subsets=["custom"], custom_actions=[my_action]).to_tool_description()
# main: KeyError: 'my_action'

The functions are kept in a private _action_functions dict filled in __init__, next to action_set. to_python_code() and the rest of the public API are unchanged.

How it was tested

  • New test_tool_description_custom_actions in tests/core/test_actions_highlevel.py (no browser needed): checks the schema of a custom action and of a custom action named click, for both the openai and anthropic formats. It fails on main with KeyError and passes with the fix.
  • pytest -n 5 -m 'not pricy' tests/core (headless Chromium, Playwright 1.56 instead of the pinned 1.44): everything passes except test_task.py::test_mock_image_goal_task (needs internet access, which my environment does not have) and test_observation.py::test_axtree_to_text; both fail the same way on main here, so they are unrelated to this change.
  • black 24.2.0 (repo config) is clean.

to_tool_description() looked up each action function with
globals()[tool_name], i.e. in the highlevel module namespace, which only
contains the built-in actions. As a result:

- an action set with custom_actions raised KeyError for the first custom
  action, so tool-use agents could not use custom actions at all;
- a custom action reusing a built-in name (e.g. its own click()) was
  described with the built-in signature instead of its own.

Keep the functions registered in the action set and read the signature
from them. Add a test covering both cases.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant