fix: make the zendesk triage call via claude work over API - #37
Open
Bilb wants to merge 16 commits into
Open
Conversation
--effort was parsed and then only ever reached the API path, so a `--backend claude-cli` run silently ignored it and used the CLI's own session default. The flag now maps onto `claude --effort`.
The CLI path used to describe the shape in prose and parse a JSON object back out of the model's reply, which is how platform, app_version and reported_session_id silently came back empty after being added to the schema. `claude --json-schema` takes the same SCHEMA the API path uses and returns it validated in the envelope's `structured_output`, so the hand-maintained field list (and the tests guarding it) can go. An envelope with no `structured_output` means the CLI ignored the flag (pre-v2.1.205) — that exits with the version to check rather than rendering an empty digest.
The CLI resolves an alias against whatever the authenticated plan allows, so a new Opus release needs no edit here and a plan without Opus access degrades instead of 404-ing on an id it can't serve. Documents the trade-off and when to pin a full id instead. Note: `--backend api` needs an explicit `--model claude-opus-4-8` until the follow-up commit removes that path — the Anthropic API takes ids, not Claude Code aliases.
Installs Claude Code on the runner (stable channel, version echoed into the log) and authenticates with a CLAUDE_CODE_OAUTH_TOKEN secret from `claude setup-token`, so the job runs on a Claude subscription rather than API billing. ANTHROPIC_API_KEY is dropped from the step rather than left alongside: an API key outranks the OAuth token in Claude Code's credential precedence, and in -p mode a present key is always used, so keeping it would silently bill the API. DISABLE_AUTOUPDATER pins the run to the version the log reports. Documents the token's one-year life, that runs draw on one person's subscription limits, and why --bare must not be added (it reads ANTHROPIC_API_KEY only and never OAuth credentials).
Nothing uses it now that the scheduled run goes through Claude Code, and keeping it meant maintaining two classification paths plus a dependency the job never loads. Removes analyze(), the anthropic import and pin, and the `api` backend choice — `claude-cli` becomes the default, leaving `file` for rendering findings classified elsewhere. The one behaviour that went with it was the explicit max-tokens message on a truncated batch, since only the API path could read stop_reason. A truncated reply now closes no JSON and yields no structured_output, so that error names both possible causes and the --batch-size to lower.
Brings back analyze(), the anthropic dependency and the `api` backend, and teaches the workflow to choose: `api` when an ANTHROPIC_API_KEY secret exists, `claude-cli` otherwise, overridable per dispatch. Adding the secret therefore switches the scheduled job over with no workflow edit, and the job keeps running on the subscription token until then. Because both credentials are in scope in the run step, it unsets the one the chosen backend doesn't use — an API key outranks the OAuth token inside `claude`, so leaving it set would make a claude-cli run bill the API instead. The Claude Code install is skipped entirely on the API path. DEFAULT_MODEL stays the `opus` alias, so the API path maps aliases to ids via API_MODEL_ALIASES (newest model per family, matching what the CLI's own alias resolution lands on).
Ticket text is written by strangers and the runner has a checkout of this repo, so the CLI ran with more attached to it than the job needs: the instructions rode on stdin next to the untrusted payload, and the session loaded the runner's and the repo's hooks, plugins, skills and CLAUDE.md along with the full tool surface. Now: SYSTEM_PROMPT travels as --system-prompt (stdin carries only the ticket JSON), --setting-sources "" loads no config from either machine, --strict-mcp-config with no config means no MCP servers, and the tool surface is denied by name. A session ends up with one tool, StructuredOutput, and no MCP servers. Dropping the agent preamble and the tool definitions also took a two-ticket fixture from ~$0.29 to ~$0.015, so the README's cost comparison is corrected too. Measured on v2.1.218, and the reason the deny list is by name rather than `*`: a wildcard also denies StructuredOutput, which is how --json-schema is implemented, so the run comes back as prose. `--permission-mode dontAsk` is not a substitute either — a session with no allow rules still executed Bash(echo …). Tests guard the parts that fail silently.
Drops the backend selection step, the Claude Code install and the CLAUDE_CODE_OAUTH_TOKEN secret from the workflow: CI now always passes --backend api. That keeps the runner off a ~270MB CLI download and keeps CI on an org-owned credential rather than an individual's subscription. The claude-cli backend and its lockdown stay in the script as the local default, so a local run still needs no Claude credential. Its notes move to the local-testing section, including the one that bites there instead of in CI: an exported ANTHROPIC_API_KEY outranks your Claude Code login, so a claude-cli run with one in the shell bills the API instead.
Replaces the `opus` alias with the id. An alias resolves to whatever the credential's newest Opus is, which would move severity calibration and cluster labels on someone else's release schedule — not what an unattended daily digest wants, since no one is watching a run to notice the shift. ZENDESK_TRIAGE_MODEL and --model stay as overrides and still accept aliases (sonnet for a large backfill), so API_MODEL_ALIASES stays too. The default lives in the script rather than in a repo variable so there is one place to change it; the workflow's vars.ZENDESK_TRIAGE_MODEL can stay unset. Documents why Opus and why pinned, with the cost figures that make the tier choice a non-question at this volume. Verified the CLI accepts the full id on a subscription login (modelUsage reports claude-opus-5), so the local default works too, and the test now asserts the invariant that matters: DEFAULT_MODEL must resolve to an API id, whether it's a pin or an alias.
One way to reach Claude instead of two. The CLI path existed because there was no API key; now that there is one, keeping it meant maintaining a second classification path, its lockdown flags, and a deny list that needs re-checking on every CLI upgrade — for a path nothing runs. Gone with it: analyze_via_claude_cli, findings_from_cli_envelope, the CLI_ISOLATION_ARGS/CLI_DENIED_TOOLS pair, extract_json_object (only the CLI path parsed JSON out of prose), the subprocess import, and 15 tests. --backend keeps `api` (now the default, so the workflow stops passing it) and `file` for rendering findings classified elsewhere. The model shorthands stay: ZENDESK_TRIAGE_MODEL=sonnet is documented for backfills and would otherwise 404 as a bare alias. Verified with a local dry run against the API on real tickets, and 158 tests pass.
--backend only had two values left, and one of them was inseparable from --findings: `--backend file --findings X` had to be passed together, with a validation branch to catch half of it. `--findings X` now says the whole thing on its own, and the flag it replaced is gone along with that check. Verified both branches: --findings renders a hand-written file with no Zendesk or Anthropic credentials in the environment, and a live dry run still fetches and classifies.
The digest is read by skimming, and an embed gave every field its own
labelled box — sixteen of those is a wall, and Discord's 10-embeds-per-
message cap fanned a busy day out over several posts for no reason.
Each ticket is now a line:
🔥 | 🐞 | #27605 · Notifications only appear after opening the app |
Likely cause: Background push service not waking client
leading with a severity marker (🚨 for the urgent categories, which the
model rates not_applicable because they aren't bugs), then the category
emoji, a masked link on the id, the summary, and the root-cause guess. An
abuse report keeps carrying its reported Session ID, which is the one
field on the old embed worth the characters.
The header keeps the coverage accounting that stops a truncated run from
reading as a quiet day, now with a compact category tally and a duplicate-
cluster line. Chunking moves from Discord's 6,000-char embed budget to the
2,000-char content limit, counting the newlines that join lines; per-message
ticket-id coverage is unchanged, so partial-failure recovery still works.
Category colours went with the embeds: CATEGORY_SPECS now carries an
`urgent` flag where the colour was, and the emoji is derived from the
label so the table stays the one place a category is described.
Two changes to the same line. The lines now travel inside a single embed description rather than message content, purely for the character budget: content caps at 2,000, a description at 4,096. A typical day is ~16 lines at ~180 characters — 55 of which is the masked link on the id — so that is the difference between two messages and one. No `fields` are used; the embed is a bigger text box with a coloured border, orange when something is worth looking into and green when nothing is. Chunking now counts against 4,096, and the first embed carries the title so the rest read as continuations. Each line also gets a platform icon between the category and the id, so "is this mine?" is answerable without reading the summary: 🤖 Android, 🍎 iOS, 🖥️ desktop, 🌐 multiple, ❔ unknown. The three desktop platforms share an icon because the distinction rarely changes who picks a ticket up, and a test asserts every PLATFORMS value has an entry.
actions/cache/{restore,save}@v4 target Node 20, which the runners now
force onto Node 24 with a deprecation warning on every run. v5 was the
node24 migration and v6 an internal ESM change; neither alters the inputs
this workflow passes, and GitHub-hosted runners are well past the 2.327.1
minimum v5+ requires.
These were the last Node 20 actions in the repo — everything else is
already on a current major (checkout@v7, setup-python@v6, and so on).
The lines carry their own structure, so the embed was contributing a border and a title. Back to message content, with the 🗂️ header line leading the first message instead of an embed title, and no colour. The trade-off is the character budget, and it is real: content caps at 2,000 where an embed description got 4,096, and a masked link on a ticket id spends 54 characters the reader never sees. Rebuilding last night's run — 16 analyzed, 9 worth looking into — comes to ~2,400 characters, so it arrives as two messages (1,781 + 641) where the embed held it in one. Chunking, clipping and per-message ticket coverage are otherwise unchanged. Platform icons stay.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.