You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Raised by @uipreliga in review of #103, filed there as a follow-up rather than fixed in that PR.
The hole
Every existing validator on cli_called guards one failure class — config that passes vacuously: blank verb, empty positional, min_count: 0 with no ceiling, predicates on ignored flags, a missing log, an unparseable log. There is still a gap through the middle:
That is byte-identical to the detail for a guard the agent genuinely respected. The guard is dead forever and nothing says so.
Same shape for: a wrong tool, a misspelled flag name, or an undeclared value-bearing flag shifting the positionals so the intended record never matches.
Alternation means N independently typo-able verb strings per criterion. A typo in one alternative silently narrows the match set with no signal. This is not an argument against the feature — the feature's answer to under-specification is "enumerate more spellings", which is more strings that each have to be exactly right.
Why it is not statically decidable here
cli_called has no CLI grammar and no catalog, so it cannot know that confrim is not a verb. The information needed lives in the consuming repo (e.g. assets/uip-catalog-snapshot.json in UiPath/skills, which lists every real verb).
Directions worth considering
Consumer-side linting. The repo that owns the catalog can check verb/verb_any_of against it, and additionally flag a positive criterion whose verb is a non-leaf (has child subcommands) — that specific rule already found a real fail-open, see test(ixp): grade CLI calls structurally via cli_called skills#2565. This is where the grammar lives, so it is the highest-value option.
Report zero-match negative guards as unproven rather than satisfied. A max_count: 0 guard that matched nothing could carry a distinct detail string, or a flag on the result, so a reader can tell "never happened" from "could never have happened". Does not change scoring; just stops the two cases being indistinguishable.
An opt-in liveness assertion — e.g. a companion criterion asserting the log is non-empty, or requiring negative guards in a suite to be paired with at least one positive over the same log. The IXP suite does this by hand today (list_model_options asserts the mock still writes to the graded sink) precisely because a correct run there makes no CLI call at all.
(2) is self-contained and cheap. (1) belongs downstream. (3) is a suite-design convention more than a code change.
Raised by @uipreliga in review of #103, filed there as a follow-up rather than fixed in that PR.
The hole
Every existing validator on
cli_calledguards one failure class — config that passes vacuously: blank verb, emptypositional,min_count: 0with no ceiling, predicates on ignored flags, a missing log, an unparseable log. There is still a gap through the middle:Zero records match, the criterion scores 1.0, and the detail reads:
That is byte-identical to the detail for a guard the agent genuinely respected. The guard is dead forever and nothing says so.
Same shape for: a wrong
tool, a misspelled flag name, or an undeclared value-bearing flag shifting the positionals so the intended record never matches.Why
verb_any_of(#103) widens itAlternation means N independently typo-able verb strings per criterion. A typo in one alternative silently narrows the match set with no signal. This is not an argument against the feature — the feature's answer to under-specification is "enumerate more spellings", which is more strings that each have to be exactly right.
Why it is not statically decidable here
cli_calledhas no CLI grammar and no catalog, so it cannot know thatconfrimis not a verb. The information needed lives in the consuming repo (e.g.assets/uip-catalog-snapshot.jsonin UiPath/skills, which lists every real verb).Directions worth considering
verb/verb_any_ofagainst it, and additionally flag a positive criterion whose verb is a non-leaf (has child subcommands) — that specific rule already found a real fail-open, see test(ixp): grade CLI calls structurally via cli_called skills#2565. This is where the grammar lives, so it is the highest-value option.max_count: 0guard that matched nothing could carry a distinct detail string, or a flag on the result, so a reader can tell "never happened" from "could never have happened". Does not change scoring; just stops the two cases being indistinguishable.list_model_optionsasserts the mock still writes to the graded sink) precisely because a correct run there makes no CLI call at all.(2) is self-contained and cheap. (1) belongs downstream. (3) is a suite-design convention more than a code change.