feat(cli-called): accept alternative verbs via verb_any_of

CLAUDE.md

@@ -159,7 +159,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 | `file_matches_regex` | Binary | Regex match on file |
 | `reference_comparison` | Continuous | AST/token/complexity similarity |
 | `command_executed` | Fractional | Agent tool usage verification |
-| `cli_called` | Binary | Structured match over a JSON Lines invocation log: verb / positional / per-flag predicates, with min_count/max_count bounds |
+| `cli_called` | Binary | Structured match over a JSON Lines invocation log: verb (or `verb_any_of` alternation) / positional / per-flag predicates, with min_count/max_count bounds |
 | `commands_efficiency` | Continuous | Agent tool-call efficiency relative to expected budget |
 | `uipath_eval` | Fractional | UiPath agent evaluation results |
 | `classification_match` | Binary | File-based label match (observed vs expected) with `(none)`/`(other)` sentinels; emits `ClassificationCriterionResult` for suite-level P/R/F1 |

docs/TASK_DEFINITION_GUIDE.md

@@ -934,6 +934,20 @@ Use this instead of `command_executed` or `file_matches_regex` when a test shado
   ignore_flags: ["output"]             # Flags dropped before matching (default: ["output"])
 ```
 
+**One operation, several verbs.** Use `verb_any_of` instead of `verb` (mutually exclusive); it matches if any entry does. Each entry is a *complete* verb in the form `verb` takes — not one token of a chain:
+
+```yaml
+- type: "cli_called"
+  description: "Read the project through the CLI"
+  verb_any_of: ["ixp projects list", "ixp projects get"]
+```
+
+Do **not** shorten the verb instead. `verb: "ixp projects"` matches all of its subcommands, so a positive assertion that the agent *read* a project is equally satisfied by `ixp projects delete`. Two entries are rejected when one prefixes the other, since the shorter already accepts everything the longer does.
+
+**The argument tail stays open.** `positional` is a prefix too, so `verb: "ixp projects list"` with `positional: ["proj-1"]` also matches `ixp projects list proj-1 dummy`. To require a specific tail, name every argument in it. `positional: []` is rejected — it would assert nothing.
+
+**Declare value-bearing flags when you use `positional`.** An undeclared flag is treated as a switch, so its value stays among the non-flag arguments and shifts the ones you named. `get proj-1 --folder Finance` matches `positional: ["proj-1"]`, but `get --folder Finance proj-1` does **not** — `Finance` takes the first slot. Add `folder` to `value_flags` (or name it in `flags`) to fix it. Resolving the ambiguity this way is deliberate: guessing that an unknown flag consumes the next token let `--yes proj-1` bind `yes=proj-1` and swallow the project name, which made a `max_count: 0` delete guard pass on the delete it forbade.
+
 `log` defaults to `cli_mocks/calls.jsonl`, where [`sandbox.record_cli`](#recording-cli-invocations) writes — so a task using generated recorders never sets it. Point it elsewhere only when supplying your own mock.
 
 **Log format.** One JSON object per line. Only `argv` is required; `tool` lets one log serve several shadowed executables, and `exit`/`ts` are recorded for reporting rather than matched. Unknown keys are ignored, so a mock may record more.
@@ -1021,7 +1035,7 @@ flags:
 
 **Negative guards.** Set `min_count: 0` and `max_count: 0` to assert a call did **not** happen. A missing log file *fails* rather than counting as zero matches — otherwise a mock writing to the wrong path would make every negative guard pass vacuously.
 
-**Why not a regex over a flattened log line.** A flat `cmd arg arg` string cannot express "verb X was called AND flag Y had value Z" without stacked lookaheads; cannot distinguish a quoted argument containing spaces from two arguments; and cannot stop a match from running across shell operators. Matching `argv` element-wise removes all three problems. `verb` is an **ordered prefix**, so `ixp labellings confirm` is never satisfied by `ixp labellings unconfirm`.
+**Why not a regex over a flattened log line.** A flat `cmd arg arg` string cannot express "verb X was called AND flag Y had value Z" without stacked lookaheads; cannot distinguish a quoted argument containing spaces from two arguments; and cannot stop a match from running across shell operators. Matching `argv` element-wise removes all three problems. `verb` is an **ordered prefix compared token by token**, so `ixp labellings confirm` is never satisfied by `ixp labellings unconfirm`, nor `ixp projects list` by `ixp projects lists`. What a prefix leaves open is the *tail*: `positional` constrains the arguments you name, and anything past them is unconstrained.
 
 ### `commands_efficiency`
 

plugins/coder-eval/reference/criteria.md

@@ -87,9 +87,10 @@ Optional:
 | Field | What it is |
 | --- | --- |
 | `log` | Path to the JSON Lines invocation log, relative to the sandbox working directory. Defaults to 'cli_mocks/calls.jsonl', where SandboxConfig.record_cli writes, so a task using generated recorders never repeats it |
-| `verb` | Whitespace-separated subcommand chain that must be an ORDERED PREFIX of the invocation's non-flag arguments. Order matters, so 'labellings confirm' never matches 'labellings unconfirm' |
+| `verb` | Whitespace-separated subcommand chain that must be an ORDERED PREFIX of the invocation's non-flag arguments, compared token by token (so 'projects list' never matches 'projects lists'). Order matters, so 'labellings confirm' never matches 'labellings unconfirm'. Prefer the full verb over a short one: the tokens after it are unconstrained, which is safe for a max_count 0 guard (it fires on more) but NOT for a positive assertion, where 'projects' credits 'projects delete' as readily as 'projects get'. When one operation has several spellings, use verb_any_of |
+| `verb_any_of` | Alternative whole verbs; matches if ANY of them does, e.g. ['projects list', 'projects get']. Each entry is a complete verb in the same form `verb` takes, NOT one token of a chain — a chain belongs in `verb` as a single string. Mutually exclusive with `verb` |
 | `tool` | Match only records whose 'tool' equals this (e.g. 'uip'). None matches any tool |
-| `positional` | Non-flag arguments that must follow the verb, in order |
+| `positional` | Non-flag arguments that must follow the verb, in order. A PREFIX of what followed, so anything past them is unconstrained: ['proj-1'] also matches 'get proj-1 dummy'. To require a specific tail, name every argument in it. Depends on value_flags being complete — an undeclared flag's value stays non-flag and shifts these slots |
 | `flags` | Flag name (without leading dashes) to predicate. A bare scalar means 'equals'. Flags not listed here are ignored, so an extra --output json never breaks a match |
 | `value_flags` | Flag names (no leading dashes) that consume a following token as their value. Keys of `flags` are value-bearing already; everything else is a switch whose following token stays positional. Declare a flag here when its value would otherwise be read as a positional, e.g. [folder] for `--folder F proj-1`. Defaults to [output] |
 | `min_count` | Minimum matching invocations. Combine min_count: 0 with max_count: 0 for must-NOT-match. Scoring is BINARY (in vs out of bounds), unlike command_executed's fractional field of the same name |

src/coder_eval/criteria/cli_called.py

@@ -150,14 +150,17 @@ def _record_matches(criterion: CliCalledCriterion, argv: list[str], record: dict
     )
 
     offset = 0
-    if criterion.verb is not None:
-        verb_tokens = criterion.verb.split()
-        # ORDERED prefix, not a token subset: `labellings confirm` must never be
-        # satisfied by `labellings unconfirm`, and a project name that happens to
-        # equal a subcommand must not stand in for the subcommand.
-        if positional[: len(verb_tokens)] != verb_tokens:
+    spellings = criterion.verb_spellings
+    if spellings:
+        # Token-wise, not a subset and not a string startswith: `labellings confirm`
+        # must never be satisfied by `labellings unconfirm`. Taking the first match is
+        # safe because validation rejects one spelling prefixing another, so no argv
+        # can match two.
+        matched = next((tokens for tokens in spellings if positional[: len(tokens)] == tokens), None)
+        if matched is None:
             return False
-        offset = len(verb_tokens)
+        # Measured from the spelling that matched, since spellings can differ in length.
+        offset = len(matched)
 
     if criterion.positional is not None:
         expected = criterion.positional
@@ -278,8 +281,10 @@ def _check_impl(
         facets = []
         if criterion.tool is not None:
             facets.append(f"tool={criterion.tool!r}")
-        if criterion.verb is not None:
-            facets.append(f"verb={criterion.verb!r}")
+        # Reading `criterion.verb` here would print no verb at all for a `verb_any_of`
+        # criterion, hiding the constraint that caused the failure.
+        if spellings := criterion.verb_spellings:
+            facets.append(f"verb={' | '.join(' '.join(t) for t in spellings)!r}")
         if criterion.positional is not None:
             facets.append(f"positional={criterion.positional!r}")
         if criterion.flags:

src/coder_eval/models/criteria.py

@@ -8,6 +8,7 @@
 
 from __future__ import annotations
 
+import itertools
 from abc import ABC, abstractmethod
 from typing import Annotated, Any, ClassVar, Literal, Self
 
@@ -578,11 +579,23 @@ class CliCalledCriterion(BaseSuccessCriterion):
     )
     verb: str | None = Field(
         default=None,
-        min_length=1,
         description=(
             "Whitespace-separated subcommand chain that must be an ORDERED PREFIX of the invocation's "
-            "non-flag arguments. Order matters, so 'labellings confirm' never matches "
-            "'labellings unconfirm'"
+            "non-flag arguments, compared token by token (so 'projects list' never matches "
+            "'projects lists'). Order matters, so 'labellings confirm' never matches "
+            "'labellings unconfirm'. Prefer the full verb over a short one: the tokens after it are "
+            "unconstrained, which is safe for a max_count 0 guard (it fires on more) but NOT for a "
+            "positive assertion, where 'projects' credits 'projects delete' as readily as "
+            "'projects get'. When one operation has several spellings, use verb_any_of"
+        ),
+    )
+    verb_any_of: list[str] | None = Field(
+        default=None,
+        description=(
+            "Alternative whole verbs; matches if ANY of them does, e.g. ['projects list', "
+            "'projects get']. Each entry is a complete verb in the same form `verb` takes, NOT one "
+            "token of a chain — a chain belongs in `verb` as a single string. Mutually exclusive "
+            "with `verb`"
         ),
     )
     tool: str | None = Field(
@@ -591,7 +604,12 @@ class CliCalledCriterion(BaseSuccessCriterion):
     )
     positional: list[str] | None = Field(
         default=None,
-        description="Non-flag arguments that must follow the verb, in order",
+        description=(
+            "Non-flag arguments that must follow the verb, in order. A PREFIX of what followed, so "
+            "anything past them is unconstrained: ['proj-1'] also matches 'get proj-1 dummy'. To "
+            "require a specific tail, name every argument in it. Depends on value_flags being "
+            "complete — an undeclared flag's value stays non-flag and shifts these slots"
+        ),
     )
     flags: dict[str, FlagMatch] | None = Field(
         default=None,
@@ -632,6 +650,49 @@ class CliCalledCriterion(BaseSuccessCriterion):
         ),
     )
 
+    @property
+    def verb_spellings(self) -> list[list[str]]:
+        """Each accepted verb as its token list; empty when there is no verb constraint.
+
+        The only place either verb field is split, so the validators, the matcher and
+        the failure detail cannot disagree.
+        """
+        if self.verb is not None:
+            return [self.verb.split()]
+        if self.verb_any_of is not None:
+            return [spelling.split() for spelling in self.verb_any_of]
+        return []
+
+    @model_validator(mode="after")
+    def _validate_verb(self) -> CliCalledCriterion:
+        """Verb rules, kept off _validate_bounds so neither grows unreadable."""
+        if self.verb is not None and self.verb_any_of is not None:
+            msg = "cli_called accepts verb or verb_any_of, not both"
+            raise ValueError(msg)
+        # Falsy, so the at-least-one-facet check below would read it as "no verb".
+        if self.verb_any_of is not None and not self.verb_any_of:
+            msg = "cli_called verb_any_of must not be empty: drop the field to match any verb"
+            raise ValueError(msg)
+        # A character count would pass "   ", whose split() is an empty prefix.
+        if any(not tokens for tokens in self.verb_spellings):
+            msg = "cli_called verb must not be blank: a blank verb is an empty prefix and matches every record"
+            raise ValueError(msg)
+        for first, second in itertools.combinations(self.verb_spellings, 2):
+            if first == second:
+                msg = f"cli_called verb_any_of lists {' '.join(first)!r} twice"
+                raise ValueError(msg)
+            # Sorting by length is total here: two DISTINCT entries of equal length
+            # cannot prefix each other, since an equal-length prefix is the same list.
+            shorter, longer = sorted((first, second), key=len)
+            if longer[: len(shorter)] == shorter:
+                msg = (
+                    f"cli_called verb_any_of entry {' '.join(shorter)!r} is a prefix of "
+                    f"{' '.join(longer)!r}; the shorter one already accepts every invocation the "
+                    "longer one does, so drop the longer entry or list only the verbs you mean."
+                )
+                raise ValueError(msg)
+        return self
+
     @model_validator(mode="after")
     def _validate_bounds(self) -> CliCalledCriterion:
         # min_count 0 with no upper bound is satisfied by every possible log, so
@@ -645,15 +706,18 @@ def _validate_bounds(self) -> CliCalledCriterion:
         if self.max_count is not None and self.max_count < self.min_count:
             msg = f"max_count ({self.max_count}) must be >= min_count ({self.min_count})"
             raise ValueError(msg)
-        # min_length=1 counts characters, so "   " passes it — and `"   ".split()`
-        # is `[]`, an empty prefix that matches every record.
-        if self.verb is not None and not self.verb.strip():
-            msg = "cli_called verb must not be blank: a blank verb is an empty prefix and matches every record"
+        # Matching slices an empty expectation and compares it to itself, so this reads
+        # as "took no arguments" while asserting nothing.
+        if self.positional is not None and not self.positional:
+            msg = (
+                "cli_called positional must not be empty: an empty list asserts nothing. List the "
+                "arguments you expect, or drop the field."
+            )
             raise ValueError(msg)
-        # Falsiness-symmetric on purpose: `verb: ""` used to slip past an `is None`
-        # check here and then match EVERY record (empty prefix), silently scoring 1.0.
-        if not self.verb and not self.positional and not self.flags and not self.tool:
-            msg = "cli_called requires at least one of verb / positional / flags / tool to match on"
+        # Falsiness, not `is None`: `verb: ""` slipped past an `is None` check here and
+        # then matched every record, scoring 1.0.
+        if not self.verb and not self.verb_any_of and not self.positional and not self.flags and not self.tool:
+            msg = "cli_called requires at least one of verb / verb_any_of / positional / flags / tool to match on"
             raise ValueError(msg)
         # A predicate on an ignored flag can never be evaluated: ignore_flags drops
         # the flag before any predicate runs, so `absent` would pass vacuously and