Skip to content

cli_called: a criterion that silently never matches (typo'd verb, tool, or flag) scores 1.0 under max_count: 0 #122

Description

@alexandrujircan

Raised by @uipreliga in review of #103, filed there as a follow-up rather than fixed in that PR.

The hole

Every existing validator on cli_called guards one failure class — config that passes vacuously: blank verb, empty positional, min_count: 0 with no ceiling, predicates on ignored flags, a missing log, an unparseable log. There is still a gap through the middle:

- type: "cli_called"
  verb: "ixp labellings confrim"   # typo
  min_count: 0
  max_count: 0

Zero records match, the criterion scores 1.0, and the detail reads:

0 invocation(s) matched (verb='ixp labellings confrim'); satisfies min_count=0, max_count=0

That is byte-identical to the detail for a guard the agent genuinely respected. The guard is dead forever and nothing says so.

Same shape for: a wrong tool, a misspelled flag name, or an undeclared value-bearing flag shifting the positionals so the intended record never matches.

Why verb_any_of (#103) widens it

Alternation means N independently typo-able verb strings per criterion. A typo in one alternative silently narrows the match set with no signal. This is not an argument against the feature — the feature's answer to under-specification is "enumerate more spellings", which is more strings that each have to be exactly right.

Why it is not statically decidable here

cli_called has no CLI grammar and no catalog, so it cannot know that confrim is not a verb. The information needed lives in the consuming repo (e.g. assets/uip-catalog-snapshot.json in UiPath/skills, which lists every real verb).

Directions worth considering

  1. Consumer-side linting. The repo that owns the catalog can check verb/verb_any_of against it, and additionally flag a positive criterion whose verb is a non-leaf (has child subcommands) — that specific rule already found a real fail-open, see test(ixp): grade CLI calls structurally via cli_called skills#2565. This is where the grammar lives, so it is the highest-value option.
  2. Report zero-match negative guards as unproven rather than satisfied. A max_count: 0 guard that matched nothing could carry a distinct detail string, or a flag on the result, so a reader can tell "never happened" from "could never have happened". Does not change scoring; just stops the two cases being indistinguishable.
  3. An opt-in liveness assertion — e.g. a companion criterion asserting the log is non-empty, or requiring negative guards in a suite to be paired with at least one positive over the same log. The IXP suite does this by hand today (list_model_options asserts the mock still writes to the graded sink) precisely because a correct run there makes no CLI call at all.

(2) is self-contained and cheap. (1) belongs downstream. (3) is a suite-design convention more than a code change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions