diff --git a/.github/workflows/zendesk_triage.yml b/.github/workflows/zendesk_triage.yml index eccd46b..4c831e9 100644 --- a/.github/workflows/zendesk_triage.yml +++ b/.github/workflows/zendesk_triage.yml @@ -65,7 +65,7 @@ jobs: # partially-delivered tickets even when a later Discord POST fails, and the # combined action would discard that on a failed job. - name: Restore triage state - uses: actions/cache/restore@v4 + uses: actions/cache/restore@v6 with: path: .triage-state key: zendesk-triage-state-${{ github.run_id }}-${{ github.run_attempt }} @@ -117,7 +117,7 @@ jobs: # later message fails, and that must survive the job's non-zero exit. - name: Save triage state if: always() - uses: actions/cache/save@v4 + uses: actions/cache/save@v6 with: path: .triage-state key: zendesk-triage-state-${{ github.run_id }}-${{ github.run_attempt }} diff --git a/README.md b/README.md index 3570f70..f9efbae 100644 --- a/README.md +++ b/README.md @@ -63,11 +63,11 @@ Runs automatically every Monday at 00:00 UTC. ## Zendesk Ticket Triage -Claude reviews recently-created unsolved Zendesk tickets via the API and posts a summary to Discord that links back to each original ticket and highlights the ones worth looking into. For each ticket it assigns a category, infers severity, guesses a likely root cause, identifies platform and app version, groups likely duplicates into clusters, and ranks by priority. +Claude reviews recently-created unsolved Zendesk tickets fetched from the Zendesk API and posts a summary to Discord that links back to each original ticket and highlights the ones worth looking into. For each ticket it assigns a category, infers severity, guesses a likely root cause, identifies platform and app version, groups likely duplicates into clusters, and ranks by priority. ### Categories -`CATEGORY_SPECS` in [triage.py](zendesk_triage/triage.py) is the single source of truth — the schema enum, the Discord labels, the urgency colours, and the prompt guidance are all derived from it, so adding a category is one edit. +`CATEGORY_SPECS` in [triage.py](zendesk_triage/triage.py) is the single source of truth — the schema enum, the Discord labels and emoji, which categories count as urgent, and the prompt guidance are all derived from it, so adding a category is one edit. | Category | Notes | | --- | --- | @@ -81,7 +81,7 @@ Claude reviews recently-created unsolved Zendesk tickets via the API and posts a | `positive_review` | 4-5★ review, no actionable content | | `feature_request`, `question`, `spam_or_solicitation`, `other` | | -The first three are **urgent categories**: they are not bugs, so the model rates their severity `not_applicable`. Colouring by severity alone painted them the calmest blue and sorted them last, so category urgency wins — they render dark red, sort ahead of everything else, and cannot be pushed out of the digest by the display cap. +The first three are **urgent categories**: they are not bugs, so the model rates their severity `not_applicable`. Marking by severity alone gave them the calmest marker and sorted them last, so category urgency wins — they lead their line with 🚨, sort ahead of everything else, and cannot be pushed out of the digest by the display cap. ### App-store review filtering @@ -93,17 +93,31 @@ Detection uses the Zendesk `via.channel`, which identified reviews with no false Twitter DM tickets arrive with `description` identical to `subject` — both just `"Conversation with "` — which is 15% of non-review tickets and unclassifiable as fetched. For those only, `hydrate_descriptions` fetches a page of up to 10 comments and joins every body that differs from the subject into the description; later replies often carry the actual detail. Hydration is an enrichment, so an HTTP error or an unreachable endpoint leaves the ticket as-is rather than failing the run (`--no-hydrate` to skip it entirely). -The script (`zendesk_triage/triage.py`) fetches the tickets in a rolling time window, sends the whole batch to Claude in one structured-output request, and posts Discord embeds: a summary embed plus one embed per highlighted ticket (linking to the ticket in Zendesk). +The script (`zendesk_triage/triage.py`) fetches the tickets in a rolling time window, classifies the whole batch in one schema-enforced request to the Anthropic API, and posts a Discord digest: a short header, then one line per ticket worth looking into. -The summary embed accounts for the batch in full, so nothing is dropped silently: +Each line leads with a severity marker, a category emoji and a platform icon, links the ticket id, and carries the model's one-line summary plus its root-cause guess: ``` -Analyzed **2** of **47** tickets in the window (created in the past 2 days). Skipped **45** already reported and unchanged. -Backlog: **5,609** unsolved tickets in total (not triaged). -**1** worth looking into. 🔄 **1** changed since last reported. +🗂️ **Zendesk triage** — analyzed **16** of **46** tickets in the window (created in the past 2 days). Skipped **30** positive app-store review(s). +Backlog: **5,680** unsolved tickets in total (not triaged). +**9** worth looking into. +⭐ **6** · 🐛 **3** · ❓ **2** · 🔑 **1** · ⚖️ **1** · 🔒 **1** +Likely duplicates: **push-notifications-not-delivered** ×5 (#27637, #27610, #27606, #27605) +🚨 | ⚖️ | ❔ | #27632 · Police summons demanding user details for a Session ID +🚨 | 🔒 | 🤖 | #27603 · Exported component lets another app obtain internal SharedPreferences | Likely cause: Improperly exported provider allowing external apps to trigger file sharing +🟠 | ⭐ | 🍎 | #27610 · Messages not delivered for days; nothing shows even after opening | Likely cause: Push notification delivery / message retrieval failure +🟠 | 🐛 | 🤖 | 🔄 #27605 · Message and call notifications only appear when the app is opened | Likely cause: Push notification service failure on Android ``` -> **Scope:** the window covers tickets *created* recently, so the long tail of older unsolved tickets is counted in the backlog line but not triaged. That is deliberate — the job is a new-ticket digest, not a backlog sweep. +| Column | Values | +| --- | --- | +| Severity | 🔥 crash · 💥 data loss · 🟠 major · 🟡 minor · ⚪ cosmetic · ▫️ not applicable — replaced by 🚨 on the urgent categories | +| Category | The emoji from `CATEGORY_SPECS`, so it matches the tally line | +| Platform | 🤖 Android · 🍎 iOS · 🖥️ desktop (all three) · 🌐 multiple · ❔ unknown | + +The header accounts for the batch in full, so nothing is dropped silently. An abuse report also carries the reported Session ID on its line, since that is the actionable part and it saves opening the ticket. + +**Plain message content, no embeds.** The lines carry their own structure, so an embed added a border and nothing else. The cost is the character budget: Discord caps message content at 2,000 against an embed description's 4,096, and a masked link on the id spends 54 characters that the reader never sees. A real 9-highlight day comes to ~2,400 characters, so it arrives as two messages. Lines are clipped (`SUMMARY_CHARS`, `ROOT_CAUSE_CHARS`) and chunked against 2,000, counting the newlines that join them; each message records which ticket ids it accounts for, which is what makes a partial post failure recoverable. ### Deduplication @@ -113,7 +127,7 @@ The daily window is 48h, so consecutive runs overlap. A state file (`--state`) r | ------ | ------- | | Not seen before | Analyzed and reported | | Seen, `updated_at` unchanged | **Skipped before the model call** — costs no tokens | -| Seen, `updated_at` moved | Re-analyzed, reported, and flagged 🔄 in the embed title | +| Seen, `updated_at` moved | Re-analyzed, reported, and flagged 🔄 on its line | State is written only on a real run, and only for tickets covered by messages Discord **accepted**. Each message carries the ticket ids it accounts for, so a partial failure records exactly what landed: already-posted messages aren't repeated next run, and undelivered tickets stay eligible. The run then exits non-zero. `--dry-run` never writes state. @@ -131,9 +145,15 @@ Two caveats worth knowing: | `ZENDESK_SUBDOMAIN` | Zendesk subdomain (`mycompany` → `mycompany.zendesk.com`) | | `ZENDESK_EMAIL` | Agent email used for Zendesk API-token auth | | `ZENDESK_API_TOKEN` | Zendesk API token | -| `ANTHROPIC_API_KEY` | Claude API key | +| `ANTHROPIC_API_KEY` | Claude API key — see [Claude authentication](#claude-authentication) | | `DISCORD_WEBHOOK_URL` | Discord webhook (reused from the failure-notification setup) | +### Claude Authentication + +Classification goes through the Anthropic API with an `ANTHROPIC_API_KEY`, in CI and locally alike — an organization-owned credential that doesn't draw on any individual's subscription quota. + +If you go looking for that key and can't find one: an API key only exists inside a **Claude Console organization** (`platform.claude.com`), which is a separate organization from a claude.ai Pro/Max/Team/Enterprise subscription, with its own membership and billing. A claude.ai admin console has no API keys in it at all, so the usual answer is that no Console organization exists yet rather than that you're missing a permission. + ### Optional Configuration | Setting | Where | Default | Description | @@ -142,7 +162,8 @@ Two caveats worth knowing: | `--state` | flag | *(unset)* | Dedup state file. The workflow points this at the cached `.triage-state/seen.json` | | `--state-retention-days` | flag | `30` | Forget state entries older than N days | | `ZENDESK_QUERY` | env / `--query` | *(unset)* | Explicit Zendesk search query. Overrides `--window-hours` entirely | -| `ZENDESK_TRIAGE_MODEL` | repo variable / `--model` | `claude-opus-4-8` | Set to a cheaper model (e.g. `claude-haiku-4-5`) to reduce cost on large batches | +| `ZENDESK_TRIAGE_MODEL` | repo variable / `--model` | `claude-opus-5` | Overrides the model. Takes a full id, or a shorthand (`opus`, `sonnet`, `haiku`) mapped to an id via `API_MODEL_ALIASES`. **Leave it unset for normal operation** — the default lives in the script so there's one place to change it | +| `--findings` | flag | *(unset)* | Render a findings JSON classified elsewhere, skipping Zendesk and Claude entirely. Pairs with `--dump-batch` | | `--max-tickets` | workflow input / flag | `1000` (workflow) / `100` (flag) | Runaway guard on tickets analyzed per run, **not** a batch size. The workflow passes `1000`; a bare `python triage.py` uses the script's own `DEFAULT_MAX_TICKETS` of `100`. Zendesk's search API caps a query at 1000 results, so higher values don't fetch more | | `--batch-size` | flag | `400` | Split batches larger than this across multiple requests | | `--review-star-floor` | flag | `3` | Classify app-store reviews at or below N stars; count the rest | @@ -150,6 +171,17 @@ Two caveats worth knowing: | `--no-hydrate` | flag | off | Skip fetching comments for content-free tickets | | `--effort` | flag | `medium` | Claude reasoning effort (`low`–`max`) | +#### Why this model, and why pinned + +**Opus**, because the hard part of this job isn't per-ticket classification — enum-constrained categories with prompt guidance is squarely mid-tier work. It's the two batch-wide fields: `cluster` has to spot that a German app-store review and an English bug report describe one root cause, and `priority_rank` has to stay consistent across the whole batch. Those need the model to hold ~45 heterogeneous tickets in mind at once. The exact-transcription requirement (a 66-character Session ID copied verbatim) points the same way. And the entire job costs **single-digit dollars a month** on any current model — roughly $10 on Opus 5 against $6 on Sonnet 5 and $2 on Haiku 4.5 — so trading classification quality for a few dollars would be optimising the wrong thing when the cost of a miss is an unseen abuse report. + +**Pinned to an id rather than the `opus` alias**, because this is an unattended digest. An alias resolves to the newest Opus the credential allows, so severity calibration and cluster labels would shift on someone else's release schedule, with no run in between to notice it. Bumping the pin is a deliberate one-line change in [triage.py](zendesk_triage/triage.py) (`DEFAULT_MODEL`). + +Two cases for overriding it: + +- **Large backfills.** A `reset_state` run at `--max-tickets 1000` chunks into 400-ticket requests, where Opus latency and spend actually show up and cross-chunk cluster fidelity is already reduced by design. `ZENDESK_TRIAGE_MODEL=sonnet` for those. +- **Never Fable 5.** It prices above Opus tier, targets long-horizon agentic reasoning, and requires 30-day data retention — all wrong for batch classification of support tickets. + #### Batch size vs. ticket cap These do different jobs, and conflating them is how you get a silently truncated digest: @@ -157,7 +189,7 @@ These do different jobs, and conflating them is how you get a silently truncated - **`--max-tickets`** bounds how much of the Zendesk result set is fetched. At the workflow's 1000 it never binds on a 48h window (~45 tickets); it exists so a spam flood or a wide `reset_state` backfill can't run away. 1000 is also [Zendesk's own search result limit](https://developer.zendesk.com/api-reference/ticketing/ticket-management/search/#results-limit) — the API returns `422` for any page past it, so the fetch stops at 1000 regardless of what you pass, and reports the matched-vs-analyzed gap rather than failing. - **`--batch-size`** bounds how many tickets go into a *single* model request. Anything larger is split across requests and the findings are concatenated. -The split is necessary because output tokens, not context, are the binding constraint. Measured on real tickets: **~118 input tokens and ~102 output tokens per ticket**, with adaptive thinking drawing from the same `max_tokens` budget. +The split is necessary because output tokens, not context, are the binding constraint. Measured on real tickets: **~118 input tokens and ~102 output tokens per ticket**, with adaptive thinking drawing from the same output budget. | Batch | Input | Output needed | Fits in one request? | | ----- | ----- | ------------- | -------------------- | @@ -165,7 +197,7 @@ The split is necessary because output tokens, not context, are the binding const | 400 (`--batch-size`) | ~47K | ~41K | Yes, with room for thinking | | 1000 (`--max-tickets`) | ~118K | ~102K | **No** — leaves only ~26K of the 128K output ceiling for thinking | -If a single request ever does hit the ceiling, the script exits with that explicit reason rather than failing on an incomplete-JSON parse error. +If a single request ever does hit the ceiling, the JSON never closes and no `structured_output` comes back — the script exits naming that and the `--batch-size` to lower, rather than rendering a digest that is silently short. > Chunking is per-request, so `cluster` labels and `priority_rank` are only meaningful within a chunk. Batches large enough to split are ones where completing at all matters more than cross-chunk cluster fidelity. @@ -205,31 +237,28 @@ If you outgrow the cache's guarantees, the next step up is a private store (a pr python -m unittest discover -s zendesk_triage -v ``` -Offline tests covering the window arithmetic, dedup partitioning, state round-trip and pruning, corrupt-state degradation, Discord embed rendering and chunking, defensive JSON parsing, and the retry/pagination behaviour with a stub session. No secrets or network access needed. They run in CI on any push or PR touching `zendesk_triage/`. +Offline tests covering the window arithmetic, dedup partitioning, state round-trip and pruning, corrupt-state degradation, Discord line rendering and message chunking, defensive JSON parsing, and the retry/pagination behaviour with a stub session. No secrets or network access needed. They run in CI on any push or PR touching `zendesk_triage/`. ### Local Testing +Local runs use the same Anthropic API path as CI, so they need an `ANTHROPIC_API_KEY` alongside the Zendesk credentials. `--dry-run` prints the Discord payload instead of posting, so no webhook is needed: + ``` pip install -r zendesk_triage/requirements.txt export ZENDESK_SUBDOMAIN=... ZENDESK_EMAIL=... ZENDESK_API_TOKEN=... ANTHROPIC_API_KEY=... -# fetch + analyze, print the Discord payload, post nothing +# what CI runs, minus the Discord post and the state file python zendesk_triage/triage.py --window-hours 48 --dry-run -``` - -No `ANTHROPIC_API_KEY`? Two debug backends skip the Anthropic API entirely: -``` -# classify via the local `claude` CLI (authenticates as Claude Code) -python zendesk_triage/triage.py --backend claude-cli --window-hours 48 --dry-run +# keep it cheap while iterating on the rendering +python zendesk_triage/triage.py --window-hours 12 --max-tickets 5 --dry-run -# or dump the batch, classify it by hand, and feed the findings back +# or take the model out of the loop: dump the batch, classify it by hand, +# and feed the findings back in to render python zendesk_triage/triage.py --dump-batch /tmp/batch.json --window-hours 48 -python zendesk_triage/triage.py --backend file --findings /tmp/findings.json --dry-run +python zendesk_triage/triage.py --findings /tmp/findings.json --dry-run ``` -The `claude-cli` backend has no structured-output enforcement, so its field values are looser than the API path's (e.g. `"en"` where the schema asks for `"English"`), and each invocation carries ~25K tokens of Claude Code system-prompt overhead. Use it for debugging, not for scheduled runs. - ## Workflow Failure Notificaiton If a workflow fails and is in the list of workflows monitored by the failure notificaiton workflow, the failure notificaiton workflow will send a message to a discord webhook. diff --git a/zendesk_triage/test_triage.py b/zendesk_triage/test_triage.py index e2f880e..b50aee1 100644 --- a/zendesk_triage/test_triage.py +++ b/zendesk_triage/test_triage.py @@ -64,6 +64,11 @@ def build_messages(*args, **kwargs): return triage.build_messages(*args, **kwargs)[0] +def digest_text(messages): + """The digest as one string: every message's content, in order.""" + return "\n".join(m["content"] for m in messages) + + class FakeResponse: # retry-after: 0 keeps the retry tests instant instead of sleeping through # the real backoff, and exercises the header-honoring path while it's at it. @@ -319,108 +324,141 @@ def test_state_written_by_save_state_round_trips_the_version(self): # ---- Discord rendering ----------------------------------------------------- -class TestSummaryEmbed(unittest.TestCase): - def description(self, findings, stats): +class TestHeader(unittest.TestCase): + def header(self, findings, stats): highlights = [f for f in findings if f.get("worth_looking_into")] - return triage.build_summary_embed(findings, highlights, "acme", stats)["description"] + return triage.build_header(findings, highlights, stats) def test_reports_analyzed_against_matched(self): - text = self.description([finding(1)], {"matched": 47}) - self.assertIn("Analyzed **1** of **47** tickets in the window", text) + text = self.header([finding(1)], {"matched": 47}) + self.assertIn("analyzed **1** of **47** tickets in the window", text) def test_names_the_window(self): - text = self.description([finding(1)], {"matched": 5, "scope": "created in the past 2 days"}) + text = self.header([finding(1)], {"matched": 5, "scope": "created in the past 2 days"}) self.assertIn("(created in the past 2 days)", text) def test_reports_skipped_unchanged_tickets(self): - text = self.description([finding(1)], {"matched": 47, "skipped_unchanged": 45}) + text = self.header([finding(1)], {"matched": 47, "skipped_unchanged": 45}) self.assertIn("Skipped **45** already reported and unchanged", text) def test_omits_the_skip_line_when_nothing_was_skipped(self): - self.assertNotIn("Skipped", self.description([finding(1)], {"skipped_unchanged": 0})) + self.assertNotIn("Skipped", self.header([finding(1)], {"skipped_unchanged": 0})) def test_reports_the_untriaged_backlog_with_thousands_separators(self): - text = self.description([finding(1)], {"total_unsolved": 5609}) + text = self.header([finding(1)], {"total_unsolved": 5609}) self.assertIn("Backlog: **5,609** unsolved tickets in total", text) def test_omits_the_backlog_line_when_the_count_is_unavailable(self): - self.assertNotIn("Backlog", self.description([finding(1)], {"total_unsolved": None})) + self.assertNotIn("Backlog", self.header([finding(1)], {"total_unsolved": None})) def test_flags_how_many_were_re_reports(self): - text = self.description([finding(1)], {"updated_count": 3}) + text = self.header([finding(1)], {"updated_count": 3}) self.assertIn("🔄 **3** changed since last reported", text) def test_counts_crash_and_data_loss_as_serious(self): findings = [finding(1, severity="crash"), finding(2, severity="data_loss")] - self.assertIn("**2** crash/data-loss", self.description(findings, {})) + self.assertIn("**2** crash/data-loss", self.header(findings, {})) def test_works_with_no_stats_at_all(self): - text = self.description([finding(1)], None) - self.assertIn("Analyzed **1**", text) + text = self.header([finding(1)], None) + self.assertIn("analyzed **1**", text) self.assertNotIn("of **", text) + def test_tallies_categories_by_emoji(self): + findings = [finding(1), finding(2), finding(3, category="question")] + text = self.header(findings, {}) + self.assertIn(f"{triage.CATEGORY_EMOJI['bug_report']} **2**", text) + self.assertIn(f"{triage.CATEGORY_EMOJI['question']} **1**", text) + def test_groups_repeated_clusters(self): findings = [finding(1, cluster="push"), finding(2, cluster="push"), finding(3, cluster="solo")] - embed = triage.build_summary_embed(findings, findings, "acme", {}) - names = [f["name"] for f in embed["fields"]] - self.assertIn("Likely duplicate clusters", names) - clusters = next(f for f in embed["fields"] if f["name"] == "Likely duplicate clusters") - self.assertIn("push", clusters["value"]) - self.assertNotIn("solo", clusters["value"]) # a single ticket is not a cluster + text = triage.build_header(findings, findings, {}) + self.assertIn("Likely duplicates:", text) + self.assertIn("push", text) + self.assertNotIn("solo", text) # a single ticket is not a cluster -class TestHighlightEmbed(unittest.TestCase): +class TestTicketLine(unittest.TestCase): + def test_reads_as_markers_then_id_then_summary(self): + line = triage.build_ticket_line( + finding(27605, severity="crash", summary="Notifications only appear after opening"), + "acme", + ) + self.assertTrue(line.startswith(f"🔥 | {triage.CATEGORY_EMOJI['bug_report']} | ")) + self.assertIn("[#27605](https://acme.zendesk.com/agent/tickets/27605)", line) + self.assertIn("· Notifications only appear after opening", line) + self.assertIn("| Likely cause: cause", line) + def test_update_marker_only_appears_for_re_reports(self): - fresh = triage.build_highlight_embed(finding(1), "acme", is_update=False) - repeat = triage.build_highlight_embed(finding(1), "acme", is_update=True) - self.assertFalse(fresh["title"].startswith("🔄")) - self.assertTrue(repeat["title"].startswith("🔄")) + fresh = triage.build_ticket_line(finding(1), "acme", is_update=False) + repeat = triage.build_ticket_line(finding(1), "acme", is_update=True) + self.assertNotIn("🔄", fresh) + self.assertIn("🔄 [#1]", repeat) + + def test_omits_the_cause_segment_when_there_is_none(self): + line = triage.build_ticket_line(finding(1, likely_root_cause=""), "acme") + self.assertNotIn("Likely cause", line) + + def test_carries_the_reported_account_for_abuse_reports(self): + """The one field worth the characters: it is what an abuse report is for.""" + line = triage.build_ticket_line( + finding(1, category="abuse_report", reported_session_id="05" + "a" * 64), "acme" + ) + self.assertIn("Reported: `05" + "a" * 64 + "`", line) - def test_links_back_to_the_ticket(self): - embed = triage.build_highlight_embed(finding(42), "acme") - self.assertEqual(embed["url"], "https://acme.zendesk.com/agent/tickets/42") + def test_an_urgent_category_outranks_a_benign_severity(self): + line = triage.build_ticket_line( + finding(1, category="legal_or_data_request", severity="not_applicable"), "acme" + ) + self.assertTrue(line.startswith(triage.URGENT_MARKER)) - def test_title_stays_within_the_discord_limit(self): - embed = triage.build_highlight_embed(finding(1, summary="x" * 500), "acme", is_update=True) - self.assertLessEqual(len(embed["title"]), 256) + def test_a_long_summary_is_clipped(self): + line = triage.build_ticket_line(finding(1, summary="x" * 500), "acme") + self.assertIn("…", line) + self.assertLess(len(line), 500) class TestBuildMessages(unittest.TestCase): - def test_only_tickets_worth_looking_into_get_their_own_embed(self): + def test_only_tickets_worth_looking_into_get_a_line(self): findings = [finding(1), finding(2, worth_looking_into=False)] - messages = build_messages(findings, "acme") - self.assertEqual(len(messages[0]["embeds"]), 2) # summary + one highlight + text = digest_text(build_messages(findings, "acme")) + self.assertEqual(sum(1 for line in text.splitlines() if "[#" in line), 1) + self.assertIn("[#1]", text) def test_highlights_are_ordered_by_priority_rank(self): findings = [finding(1, priority_rank=3), finding(2, priority_rank=1)] - embeds = build_messages(findings, "acme")[0]["embeds"] - self.assertIn("#2", embeds[1]["title"]) - self.assertIn("#1", embeds[2]["title"]) + lines = [l for l in digest_text(build_messages(findings, "acme")).splitlines() if "[#" in l] + self.assertIn("[#2]", lines[0]) + self.assertIn("[#1]", lines[1]) - def test_updated_ids_reach_the_right_embed(self): + def test_updated_ids_mark_the_right_line(self): findings = [finding(1), finding(2)] - embeds = build_messages(findings, "acme", {}, updated_ids={2})[0]["embeds"] - titles = {e["title"].lstrip("🔄 ").split(" ")[0]: e["title"] for e in embeds[1:]} - self.assertFalse(titles["#1"].startswith("🔄")) - self.assertTrue(titles["#2"].startswith("🔄")) + text = digest_text(build_messages(findings, "acme", {}, updated_ids={2})) + self.assertIn("🔄 [#2]", text) + self.assertNotIn("🔄 [#1]", text) + + def test_the_header_leads_the_first_message(self): + messages = build_messages([finding(1)], "acme", {"matched": 3}) + self.assertTrue(messages[0]["content"].startswith("🗂️ **Zendesk triage**")) - def test_embeds_are_chunked_to_the_discord_per_message_limit(self): + def test_every_highlight_reaches_a_message(self): findings = [finding(i, priority_rank=i) for i in range(triage.MAX_HIGHLIGHTS)] - messages = build_messages(findings, "acme") - for message in messages: - self.assertLessEqual(len(message["embeds"]), triage.MAX_EMBEDS_PER_MESSAGE) - total = sum(len(m["embeds"]) for m in messages) - self.assertEqual(total, triage.MAX_HIGHLIGHTS + 1) # + the summary + text = digest_text(build_messages(findings, "acme")) + for i in range(triage.MAX_HIGHLIGHTS): + self.assertIn(f"[#{i}]", text) def test_highlights_beyond_the_cap_are_dropped_but_announced(self): over = triage.MAX_HIGHLIGHTS + 5 findings = [finding(i, priority_rank=i) for i in range(over)] - messages = build_messages(findings, "acme") - self.assertIn(f"top {triage.MAX_HIGHLIGHTS} of {over}", messages[0]["content"]) + text = digest_text(build_messages(findings, "acme")) + self.assertIn(f"top **{triage.MAX_HIGHLIGHTS}** of **{over}**", text) + + def test_no_truncation_notice_when_nothing_was_dropped(self): + self.assertNotIn("Showing the top", digest_text(build_messages([finding(1)], "acme"))) - def test_no_content_line_when_nothing_was_dropped(self): - messages = build_messages([finding(1)], "acme") - self.assertNotIn("content", messages[0]) + def test_messages_are_plain_content(self): + for message in build_messages([finding(1)], "acme"): + self.assertEqual(set(message), {"content"}) # ---- Parsing helpers ------------------------------------------------------- @@ -439,23 +477,6 @@ def test_every_category_is_explained_in_the_system_prompt(self): missing = [c for c in triage.CATEGORIES if c not in triage.SYSTEM_PROMPT] self.assertEqual(missing, []) - def test_every_category_is_listed_in_the_cli_instructions(self): - missing = [c for c in triage.CATEGORIES if c not in triage.CLI_JSON_INSTRUCTIONS] - self.assertEqual(missing, []) - - def test_every_schema_field_is_listed_in_the_cli_instructions(self): - """The CLI backend has no structured-output enforcement, so a field absent - from these instructions comes back empty — which is how platform, app_version - and reported_session_id silently went unpopulated.""" - fields = triage.SCHEMA["properties"]["tickets"]["items"]["properties"] - missing = [f for f in fields if f not in triage.CLI_JSON_INSTRUCTIONS] - self.assertEqual(missing, []) - - def test_every_enum_value_is_listed_in_the_cli_instructions(self): - for values in (triage.CATEGORIES, triage.SEVERITIES, triage.PLATFORMS): - for value in values: - self.assertIn(value, triage.CLI_JSON_INSTRUCTIONS) - def test_schema_enum_matches_the_category_list(self): item = triage.SCHEMA["properties"]["tickets"]["items"] self.assertEqual(item["properties"]["category"]["enum"], triage.CATEGORIES) @@ -478,37 +499,24 @@ def test_platform_enum_is_wired_into_the_schema(self): class TestUrgency(unittest.TestCase): def test_urgent_category_beats_a_benign_severity(self): - """An abuse report is not a bug, so severity is not_applicable — which used to - paint the most serious ticket in the digest the calmest colour.""" + """An abuse report is not a bug, so the model rates it not_applicable — the + calmest marker on the most serious ticket in the digest is backwards.""" abuse = finding(1, category="abuse_report", severity="not_applicable") - self.assertEqual(triage.embed_color(abuse), triage.CATEGORY_COLOR["abuse_report"]) - self.assertNotEqual(triage.embed_color(abuse), - triage.SEVERITY_COLOR["not_applicable"]) + self.assertEqual(triage.severity_marker(abuse), triage.URGENT_MARKER) + self.assertNotEqual(triage.severity_marker(abuse), + triage.SEVERITY_EMOJI["not_applicable"]) def test_non_urgent_category_still_uses_severity(self): - self.assertEqual(triage.embed_color(finding(1, category="bug_report", severity="crash")), - triage.SEVERITY_COLOR["crash"]) - - def test_unknown_severity_falls_back_to_grey(self): - self.assertEqual(triage.embed_color({"category": "other", "severity": "???"}), 0x95A5A6) - - def test_urgent_tickets_are_highlighted_even_if_not_flagged(self): - abuse = finding(1, category="abuse_report", worth_looking_into=False) - shown, _ = triage.select_highlights([abuse]) - self.assertEqual([f["id"] for f in shown], [1]) + self.assertEqual( + triage.severity_marker(finding(1, category="bug_report", severity="crash")), + triage.SEVERITY_EMOJI["crash"]) - def test_urgent_tickets_sort_ahead_of_better_ranked_ordinary_ones(self): - ordinary = finding(1, category="bug_report", priority_rank=1) - abuse = finding(2, category="abuse_report", priority_rank=99) - shown, _ = triage.select_highlights([ordinary, abuse]) - self.assertEqual([f["id"] for f in shown], [2, 1]) + def test_unknown_severity_falls_back_to_a_neutral_marker(self): + self.assertEqual(triage.severity_marker({"category": "other", "severity": "???"}), "▫️") - def test_urgent_tickets_cannot_be_pushed_out_by_the_display_cap(self): - ordinary = [finding(i, priority_rank=i) for i in range(triage.MAX_HIGHLIGHTS + 5)] - abuse = finding(9999, category="abuse_report", priority_rank=9999) - shown, omitted = triage.select_highlights(ordinary + [abuse]) - self.assertIn(9999, [f["id"] for f in shown]) - self.assertNotIn(9999, [f["id"] for f in omitted]) + def test_every_severity_has_a_marker(self): + missing = [sev for sev in triage.SEVERITIES if sev not in triage.SEVERITY_EMOJI] + self.assertEqual(missing, []) class TestReviewFiltering(unittest.TestCase): @@ -654,46 +662,39 @@ def test_hydration_leaves_the_ticket_alone_when_no_comment_adds_anything(self): self.assertEqual(triage.hydrate_descriptions(session, "acme", [row]), 0) -class TestEmbedCharLimit(unittest.TestCase): - """Discord caps a message at 10 embeds *and* 6,000 chars across them; chunking on - count alone can build a payload Discord rejects.""" +class TestMessageCharLimit(unittest.TestCase): + """Discord caps one message's content at 2,000 characters. Every line is + pre-clipped, and chunking has to account for the newlines that join them.""" def fat(self, ticket_id): - # ~1,300 chars of field text: 10 of these would be ~13,000, over the limit. - return finding(ticket_id, summary="s" * 200, likely_root_cause="r" * 300, - affected_component="c" * 100, language="l" * 40) + return finding(ticket_id, summary="s" * 400, likely_root_cause="r" * 400) - def test_every_message_respects_both_limits(self): + def test_every_message_stays_within_the_limit(self): findings = [self.fat(i) for i in range(triage.MAX_HIGHLIGHTS)] - for message in build_messages(findings, "acme"): - self.assertLessEqual(len(message["embeds"]), triage.MAX_EMBEDS_PER_MESSAGE) - total = sum(triage.embed_char_count(e) for e in message["embeds"]) - self.assertLessEqual(total, triage.MAX_EMBED_CHARS_PER_MESSAGE) - - def test_char_limit_splits_where_the_count_limit_would_not(self): - """9 fat highlights + summary = 10 embeds: within the count limit, over 6,000 chars.""" - messages = build_messages([self.fat(i) for i in range(9)], "acme") - embeds = sum(len(m["embeds"]) for m in messages) - self.assertLessEqual(embeds, triage.MAX_EMBEDS_PER_MESSAGE) # count alone: 1 message - self.assertGreater(len(messages), 1) # chars forced the split - - def test_lean_embeds_are_not_split_early(self): - """The char limit must not fragment ordinary digests.""" - messages = build_messages([finding(i, priority_rank=i) for i in range(9)], "acme") - self.assertEqual(len(messages), 1) - - def test_no_embed_is_dropped_while_chunking(self): - findings = [self.fat(i) for i in range(15)] messages = build_messages(findings, "acme") - self.assertEqual(sum(len(m["embeds"]) for m in messages), 16) # 15 + summary + for message in messages: + self.assertLessEqual(len(message["content"]), triage.MAX_MESSAGE_CHARS) + self.assertGreater(len(messages), 1) # fat lines must actually split + + def test_no_line_is_dropped_while_chunking(self): + findings = [self.fat(i) for i in range(triage.MAX_HIGHLIGHTS)] + text = digest_text(build_messages(findings, "acme")) + for i in range(triage.MAX_HIGHLIGHTS): + self.assertIn(f"[#{i}]", text) - def test_char_count_covers_titles_descriptions_and_fields(self): - embed = {"title": "abc", "description": "de", - "fields": [{"name": "fg", "value": "hij"}]} - self.assertEqual(triage.embed_char_count(embed), 3 + 2 + 2 + 3) + def test_lean_lines_are_not_split_early(self): + findings = [finding(i, summary="s", likely_root_cause="") for i in range(5)] + self.assertEqual(len(build_messages(findings, "acme")), 1) - def test_char_count_tolerates_missing_keys(self): - self.assertEqual(triage.embed_char_count({}), 0) + def test_chunking_counts_the_joining_newlines(self): + """Two 1,000-char lines are 2,001 joined — over the cap only if the newline + counts, which is the off-by-one this guards.""" + entries = [("x" * 1000, {1}), ("y" * 1000, {2})] + self.assertEqual(len(triage.chunk_entries(entries)), 2) + + def test_an_oversized_entry_still_gets_a_message(self): + chunks = triage.chunk_entries([("x" * (triage.MAX_MESSAGE_CHARS + 50), {1})]) + self.assertEqual(len(chunks), 1) class TestCoverage(unittest.TestCase): @@ -796,8 +797,8 @@ def test_an_empty_list_is_valid(self): self.assertEqual(triage.tickets_from_payload({"tickets": []}, "x"), []) def test_a_finding_missing_renderer_keys_exits(self): - """The claude-cli backend has no structured-output enforcement, so an entry - without category/severity would otherwise KeyError inside build_summary_embed.""" + """A hand-edited --findings list has nothing enforcing its shape, + so an entry without category/severity would KeyError in build_summary_embed.""" for entry in ({"id": 1}, {"id": 1, "category": "bug_report"}, {"category": "bug_report", "severity": "major"}): with self.assertRaises(SystemExit): @@ -808,31 +809,27 @@ def test_a_non_object_entry_exits(self): triage.tickets_from_payload({"tickets": [["not", "an", "object"]]}, "x") -class TestExtractJsonObject(unittest.TestCase): - def test_bare_object(self): - self.assertEqual(triage.extract_json_object('{"a": 1}'), {"a": 1}) - - def test_object_inside_a_markdown_fence(self): - self.assertEqual(triage.extract_json_object('```json\n{"a": 1}\n```'), {"a": 1}) +class TestResolveApiModel(unittest.TestCase): + """The CLI resolves aliases itself; the API takes ids, so only that path maps.""" - def test_object_surrounded_by_prose(self): - self.assertEqual( - triage.extract_json_object('Sure! Here you go:\n{"a": 1}\nHope that helps.'), {"a": 1} - ) + def test_every_alias_maps_to_an_id(self): + for alias, model_id in triage.API_MODEL_ALIASES.items(): + self.assertEqual(triage.resolve_api_model(alias), model_id) + self.assertTrue(model_id.startswith("claude-"), model_id) - def test_nested_braces_survive(self): - self.assertEqual( - triage.extract_json_object('{"t": [{"id": 1}, {"id": 2}]}'), - {"t": [{"id": 1}, {"id": 2}]}, - ) + def test_the_default_model_resolves_to_an_api_id(self): + """The API 404s on a bare shorthand, so whatever DEFAULT_MODEL is — + a pinned id today, an alias if that ever changes — it has to resolve to one.""" + resolved = triage.resolve_api_model(triage.DEFAULT_MODEL) + self.assertNotIn(resolved, triage.API_MODEL_ALIASES) + self.assertTrue(resolved.startswith("claude-"), resolved) - def test_no_object_exits(self): - with self.assertRaises(SystemExit): - triage.extract_json_object("no json here") + def test_a_full_id_passes_through(self): + self.assertEqual(triage.resolve_api_model("claude-opus-4-8"), "claude-opus-4-8") - def test_malformed_object_exits(self): - with self.assertRaises(SystemExit): - triage.extract_json_object('{"a": }') + def test_an_unknown_value_passes_through(self): + """A model newer than this table should reach the API rather than be rewritten.""" + self.assertEqual(triage.resolve_api_model("claude-future-9"), "claude-future-9") class TestAnalyzeInChunks(unittest.TestCase): diff --git a/zendesk_triage/triage.py b/zendesk_triage/triage.py index d9d423b..f2c74d9 100644 --- a/zendesk_triage/triage.py +++ b/zendesk_triage/triage.py @@ -2,10 +2,10 @@ """ Daily Zendesk ticket triage with Claude, delivered to Discord. -Fetches open Zendesk tickets (broad query by default, not just bugs), sends the -whole batch to Claude in a single structured-output request, and posts a Discord -summary that links back to each original ticket and highlights the ones worth -looking into (crashes, data loss, legal requests, security/legislation, etc.). +Fetches open Zendesk tickets (broad query by default, not just bugs), classifies the +whole batch in one schema-enforced request, and posts a Discord summary that links +back to each original ticket and highlights the ones worth looking into (crashes, +data loss, legal requests, security/legislation, etc.). Because this repo is public, ticket content is never written to the job summary or anywhere public: in a normal run the only place ticket detail goes is the Discord @@ -20,14 +20,19 @@ bug_report | low_star_review | legal_request | security_or_legislation | question | feature_request | other -Config (env vars, or CLI flags for local runs): +Classification goes through the Anthropic API, with structured outputs enforcing +SCHEMA. --findings skips it entirely and renders findings produced elsewhere. + +Config (env vars, or flags for local runs): ZENDESK_SUBDOMAIN e.g. "mycompany" -> https://mycompany.zendesk.com ZENDESK_EMAIL agent email for API token auth ZENDESK_API_TOKEN Zendesk API token - ANTHROPIC_API_KEY Claude API key (read by the SDK automatically) - DISCORD_WEBHOOK_URL Discord incoming webhook + ANTHROPIC_API_KEY Claude API key (read by the SDK itself) + DISCORD_WEBHOOK_URL Discord incoming webhook (not needed with --dry-run) ZENDESK_QUERY (optional) Zendesk search query; see DEFAULT_QUERY - ZENDESK_TRIAGE_MODEL (optional) Claude model id; defaults to claude-opus-4-8 + ZENDESK_TRIAGE_MODEL (optional) Claude model id or alias; defaults to + claude-opus-5. Set it to override, e.g. `sonnet` for a + large backfill. Usage: # real run (CI): reads everything from the environment @@ -42,19 +47,16 @@ # or an explicit query, which overrides --window-hours python triage.py --query "type:ticket status:open tags:bug" --max-tickets 50 - # local debugging without an ANTHROPIC_API_KEY: classify via the `claude` CLI - python triage.py --backend claude-cli --dry-run --max-tickets 20 - - # or split it in two: dump the batch, classify it by hand, feed it back + # split classification out entirely: dump the batch, classify it by hand, + # feed the findings back in to render python triage.py --dump-batch /tmp/batch.json --max-tickets 20 - python triage.py --backend file --findings /tmp/findings.json --dry-run + python triage.py --findings /tmp/findings.json --dry-run """ import argparse import json import math import os import re -import subprocess import sys import textwrap import time @@ -97,11 +99,27 @@ def window_label(hours): days = hours // 24 return f"created in the past {days} day{'s' if days > 1 else ''}" return f"created in the past {hours}h" -DEFAULT_MODEL = "claude-opus-4-8" +# A pinned id rather than the `opus` alias, deliberately. This is an unattended +# digest a human skims: the batch-wide fields (`cluster`, `priority_rank`) and the +# severity calibration shift when the model underneath changes, and an alias would +# move them on someone else's release schedule. Opus rather than a cheaper tier +# because clustering asks the model to recognise one root cause across 45 tickets in +# several languages, and the whole job costs single-digit dollars a month either way. +# Bumping this is a one-line, deliberate change. +DEFAULT_MODEL = "claude-opus-5" +# Shorthands for the override, so ZENDESK_TRIAGE_MODEL=sonnet works for a big +# backfill without anyone looking up an id. The API takes ids only, so they are +# mapped here; each is the newest model in its family, and a full id passes through +# untouched. +API_MODEL_ALIASES = { + "opus": "claude-opus-5", + "sonnet": "claude-sonnet-5", + "haiku": "claude-haiku-4-5", +} DEFAULT_MAX_TICKETS = 100 DESCRIPTION_CHARS = 1500 # per-ticket description sent to Claude (triage only) # One classification runs ~100 output tokens per ticket, and adaptive thinking draws -# from the same max_tokens budget. 400 keeps a chunk far under the 128K output +# from the same output budget. 400 keeps a chunk far under the model's 128K output # ceiling; batches larger than this are split rather than truncated. DEFAULT_BATCH_SIZE = 400 MAX_OUTPUT_TOKENS = 128000 @@ -114,47 +132,49 @@ def window_label(hours): # prompt never explains. # # Percentages come from a 3,662-ticket sample of the 13 months to 2026-08. -# Columns: (name, Discord label, urgency colour or None, guidance for the model) +# Columns: (name, Discord label, urgent, guidance for the model) CATEGORY_SPECS = ( - ("abuse_report", "🚨 Abuse report", 0xC0392B, + ("abuse_report", "🚨 Abuse report", True, "One user reporting another account for illegal or abusive content (CSAM, " "harassment, drugs, impersonation). Usually quotes the offending Session ID. " "~11% of non-review tickets. Always set worth_looking_into."), - ("security_report", "🔒 Security report", 0xC0392B, + ("security_report", "🔒 Security report", True, "A vulnerability, exploit, or account-compromise disclosure. Not the same as a " "policy question. Always set worth_looking_into."), - ("legal_or_data_request", "⚖️ Legal / data request", 0xC0392B, + ("legal_or_data_request", "⚖️ Legal / data request", True, "GDPR or data-deletion request, subpoena, law-enforcement or court order. " "Always set worth_looking_into."), - ("bug_report", "🐞 Bug report", None, + ("bug_report", "🐛 Bug report", False, "Something in the app is broken or misbehaving."), - ("account_access", "🔑 Account access", None, + ("account_access", "🔑 Account access", False, "Lost recovery phrase, locked out, or asking to restore an account. Usually " "irreversible by design, but track the volume."), - ("policy_question", "📜 Policy question", None, + ("policy_question", "📜 Policy question", False, "Questions about law, regulation, or policy — 'Chat Control', encryption " "backdoors, whether Session complies with something."), - ("low_star_review", "⭐ Low-star review", None, + ("low_star_review", "⭐ Low-star review", False, "An app-store review of 3 stars or fewer. These often hide a real bug — put " "the underlying problem in `summary`."), - ("positive_review", "👍 Positive review", None, + ("positive_review", "👍 Positive review", False, "An app-store review of 4-5 stars with no actionable content."), - ("feature_request", "💡 Feature request", None, + ("feature_request", "💡 Feature request", False, "Asking for something the app does not do yet."), - ("question", "❓ Question", None, + ("question", "❓ Question", False, "A how-do-I or usage question that is not a bug."), - ("spam_or_solicitation", "🗑️ Spam / solicitation", None, + ("spam_or_solicitation", "🗑️ Spam / solicitation", False, "Marketing, token or OTC investment offers, partnership pitches, listing spam."), - ("other", "• Other", None, + ("other", "• Other", False, "Genuinely none of the above. Prefer a specific category wherever one fits."), ) CATEGORIES = [name for name, _, _, _ in CATEGORY_SPECS] CATEGORY_LABEL = {name: label for name, label, _, _ in CATEGORY_SPECS} # Categories whose urgency `severity` cannot express. They are not bugs, so the model -# rates them not_applicable — which would otherwise paint the most serious ticket in -# the batch the calmest colour and sort it last. -CATEGORY_COLOR = {name: color for name, _, color, _ in CATEGORY_SPECS if color} -URGENT_CATEGORIES = frozenset(CATEGORY_COLOR) +# rates them not_applicable — which would otherwise give the most serious ticket in +# the batch the calmest marker and sort it last. +# The emoji on its own, for the per-ticket lines. Derived from the label so the +# table stays the single place a category is described. +CATEGORY_EMOJI = {name: label.split(" ", 1)[0] for name, label, _, _ in CATEGORY_SPECS} +URGENT_CATEGORIES = frozenset(name for name, _, urgent, _ in CATEGORY_SPECS if urgent) CATEGORY_GUIDANCE = "\n".join(f"- {name}: {desc}" for name, _, _, desc in CATEGORY_SPECS) SEVERITIES = ["crash", "data_loss", "major", "minor", "cosmetic", "not_applicable"] @@ -609,6 +629,15 @@ def compact_ticket(ticket): } +def resolve_api_model(model): + """Map a shorthand model name onto the id the Anthropic API expects. + + Anything that isn't a known shorthand passes through untouched, so a pinned id + (`claude-opus-4-8`) or a model newer than this table still works. + """ + return API_MODEL_ALIASES.get(model, model) + + def build_analysis_prompt(compact_tickets): return ( "Classify every ticket in this batch and return one object per ticket.\n\n" @@ -617,60 +646,14 @@ def build_analysis_prompt(compact_tickets): ) -def _cli_field_lines(): - """Describe every schema field for the CLI path, derived from TICKET_PROPERTIES. - - The API path has structured outputs to enforce the shape; the CLI path only has - this text. Hardcoding the field list here is how three fields (platform, - app_version, reported_session_id) silently came back empty on the CLI backend - after being added to the schema. - """ - lines = [] - for name, spec in TICKET_PROPERTIES.items(): - shape = spec.get("type", "string") - if "enum" in spec: - shape += "; one of: " + ", ".join(spec["enum"]) - description = spec.get("description", "") - lines.append(f"- {name} ({shape}){': ' + description if description else ''}") - return "\n".join(lines) - - -# The API path gets the shape enforced by structured outputs. The CLI path has no -# such enforcement, so the shape is spelled out here — from the same schema. -CLI_JSON_INSTRUCTIONS = textwrap.dedent( - """ - Return ONLY a single JSON object — no prose, no explanation, no markdown code - fence. The object has exactly one key, "tickets", whose value is an array with - one object per input ticket, each with exactly these keys: - __FIELDS__ - - Every key is required on every object. Use an empty string for text fields you - cannot fill, and the enum's catch-all value ('unknown', 'not_applicable', 'other') - rather than inventing a new one. - """ -).strip().replace("__FIELDS__", _cli_field_lines()) - - -def extract_json_object(text): - """Pull the outermost JSON object out of model prose (tolerates code fences).""" - start = text.find("{") - end = text.rfind("}") - if start == -1 or end <= start: - sys.exit(f"No JSON object found in the model output:\n{text[:500]}") - try: - return json.loads(text[start : end + 1]) - except json.JSONDecodeError as exc: - sys.exit(f"Model output was not valid JSON ({exc}):\n{text[start : start + 500]}") - - REQUIRED_FINDING_KEYS = ("id", "category", "severity") def validate_findings(findings, label): """Exit unless every entry is an object carrying the keys the renderer indexes. - build_summary_embed does f["category"] / f["severity"] and build_highlight_embed - does f["id"], so a missing key surfaces as a KeyError halfway through building a + build_header does f["category"] / f["severity"] and build_ticket_line does + f["id"], so a missing key surfaces as a KeyError halfway through building a Discord payload. Failing here names the offending entry instead. """ for position, entry in enumerate(findings): @@ -686,9 +669,9 @@ def validate_findings(findings, label): def tickets_from_payload(payload, source): """Pull the `tickets` list out of a classification payload, or exit clearly. - Structured outputs guarantee the key and the item shape on the API path, but the - CLI path has neither — a bare KeyError mid-render is a confusing way to learn - that, so both paths are validated here before anything renders them. + Structured outputs guarantee the key and the item shape, so on a normal run this + never fires; it is the guard for --findings, whose contents nothing validates, + and a bare KeyError mid-render is a confusing way to learn a key is missing. """ found = payload.get("tickets") if isinstance(payload, dict) else None if not isinstance(found, list): @@ -697,47 +680,11 @@ def tickets_from_payload(payload, source): return validate_findings(found, source) -def analyze_via_claude_cli(model, compact_tickets, timeout=1800): - """Classify the batch with the local `claude` CLI instead of the Anthropic API. - - Local debugging path: it authenticates as Claude Code, so no ANTHROPIC_API_KEY is - needed. There is no structured-output enforcement here, so the response is parsed - defensively and the schema is described in the prompt. - """ - prompt = "\n\n".join( - [SYSTEM_PROMPT, CLI_JSON_INSTRUCTIONS, build_analysis_prompt(compact_tickets)] - ) - cmd = ["claude", "-p", "--output-format", "json"] - if model: - cmd += ["--model", model] - try: - # Prompt goes over stdin: a full batch can exceed the argv size limit. - proc = subprocess.run( - cmd, input=prompt, capture_output=True, text=True, timeout=timeout - ) - except FileNotFoundError: - sys.exit("`claude` not found on PATH. Install Claude Code, or use --backend api.") - except subprocess.TimeoutExpired: - sys.exit(f"`claude` timed out after {timeout}s. Try a smaller --max-tickets.") - if proc.returncode != 0: - sys.exit(f"`claude` failed ({proc.returncode}): {proc.stderr[:500]}") - - envelope = extract_json_object(proc.stdout) - if envelope.get("is_error") or envelope.get("subtype") != "success": - sys.exit(f"`claude` reported an error: {envelope.get('result') or envelope}") - cost = envelope.get("total_cost_usd") - if cost is not None: - print(f"claude CLI reported ${cost:.4f} for this batch.") - result = extract_json_object(envelope.get("result") or "") - return tickets_from_payload(result, "`claude` CLI") - - def dump_batch(path, compact_tickets, model): """Write the batch to disk so it can be classified by hand. Contains ticket text.""" payload = { "model": model, "system_prompt": SYSTEM_PROMPT, - "instructions": CLI_JSON_INSTRUCTIONS, "schema": SCHEMA, "tickets": compact_tickets, } @@ -811,18 +758,42 @@ def analyze(client, model, effort, compact_tickets): # ---- Discord rendering ----------------------------------------------------- - -SEVERITY_COLOR = { - "crash": 0xE74C3C, # red - "data_loss": 0xC0392B, # dark red - "major": 0xE67E22, # orange - "minor": 0xF1C40F, # yellow - "cosmetic": 0x95A5A6, # grey - "not_applicable": 0x3498DB, # blue +# +# A short header, then one line per ticket — the digest is read by skimming, so the +# lines are the layout. Plain message content, no embeds: the lines carry their own +# structure, so the box added nothing but a border. The cost is the character budget +# (2,000 for content against 4,096 for an embed description), which a busy day can +# spill into a second message — see chunk_entries. + +# Leads each line so severity is scannable straight down the left edge. Urgent +# categories get URGENT_MARKER instead: they are not bugs, so the model rates them +# not_applicable, and the calmest marker on the most serious ticket is backwards. +SEVERITY_EMOJI = { + "crash": "🔥", + "data_loss": "💥", + "major": "🟠", + "minor": "🟡", + "cosmetic": "⚪", + "not_applicable": "▫️", +} +URGENT_MARKER = "🚨" +# Second marker column, so "is this mine?" is answerable without reading the summary. +# Every PLATFORMS value needs an entry; the three desktops share one icon because the +# distinction rarely changes who picks the ticket up. +PLATFORM_EMOJI = { + "android": "🤖", + "ios": "🍎", + "desktop_windows": "🖥️", + "desktop_macos": "🖥️", + "desktop_linux": "🖥️", + "multiple": "🌐", + "unknown": "❔", } -MAX_EMBEDS_PER_MESSAGE = 10 -MAX_EMBED_CHARS_PER_MESSAGE = 6000 # Discord's aggregate limit across one message -MAX_HIGHLIGHTS = 27 # 3 messages of ~9 highlights + a summary embed +# Discord's cap on one message's content. Lines are clipped and chunked against it. +MAX_MESSAGE_CHARS = 2000 +SUMMARY_CHARS = 160 +ROOT_CAUSE_CHARS = 140 +MAX_HIGHLIGHTS = 27 def ticket_url(subdomain, ticket_id): @@ -834,7 +805,54 @@ def clip(text, limit): return text if len(text) <= limit else text[: limit - 1] + "…" -def build_summary_embed(findings, highlights, subdomain, stats=None): +def is_urgent(finding): + return finding.get("category") in URGENT_CATEGORIES + + +def severity_marker(finding): + """The leading emoji: category urgency first, then severity.""" + if is_urgent(finding): + return URGENT_MARKER + return SEVERITY_EMOJI.get(finding.get("severity", "not_applicable"), "▫️") + + +def build_ticket_line(finding, subdomain, is_update=False): + """One skimmable line per ticket: + + 🔥 | 🐛 | 🤖 | #27605 · Notifications only arrive… | Likely cause: push service… + + The id is a masked link, so the ticket stays one click away without spending the + character budget on a visible URL. + """ + tid = finding["id"] + # 🔄 marks a ticket already shown that has since changed, so the reader knows it + # is a follow-up rather than a duplicate post. + marker = "🔄 " if is_update else "" + parts = [ + severity_marker(finding), + CATEGORY_EMOJI.get(finding.get("category"), "•"), + PLATFORM_EMOJI.get(finding.get("platform"), PLATFORM_EMOJI["unknown"]), + f"{marker}[#{tid}]({ticket_url(subdomain, tid)}) · " + f"{clip(finding.get('summary'), SUMMARY_CHARS) or '(no summary)'}", + ] + root = clip(finding.get("likely_root_cause"), ROOT_CAUSE_CHARS) + if root: + parts.append(f"Likely cause: {root}") + # The reported account is the actionable part of an abuse report — carrying it on + # the line saves opening the ticket to copy it. + reported = clip(finding.get("reported_session_id"), 70) + if reported: + parts.append(f"Reported: `{reported}`") + return " | ".join(parts) + + +def build_header(findings, highlights, stats=None): + """The lead lines: what was looked at, the category tally, duplicate clusters. + + Accounts for the batch honestly — how much of the window was analyzed, what was + skipped and why, how big the untriaged backlog behind it is — so a short digest + never reads as a quiet day when it was really a truncated one. + """ by_category = {} by_severity = {} clusters = {} @@ -845,32 +863,16 @@ def build_summary_embed(findings, highlights, subdomain, stats=None): if label: clusters.setdefault(label, []).append(f["id"]) - cat_lines = "\n".join( - f"{CATEGORY_LABEL.get(cat, cat)}: **{count}**" - for cat, count in sorted(by_category.items(), key=lambda kv: -kv[1]) - ) - serious = by_severity.get("crash", 0) + by_severity.get("data_loss", 0) - dup_clusters = {k: v for k, v in clusters.items() if len(v) > 1} - fields = [{"name": "By category", "value": cat_lines or "—", "inline": False}] - if dup_clusters: - cluster_lines = "\n".join( - f"**{clip(label, 40)}** — {len(ids)} tickets (#{', #'.join(str(i) for i in ids[:6])})" - for label, ids in sorted(dup_clusters.items(), key=lambda kv: -len(kv[1]))[:6] - ) - fields.append({"name": "Likely duplicate clusters", "value": clip(cluster_lines, 1024), "inline": False}) - - # Account for the batch honestly: how many of the window we looked at, how many - # we skipped as unchanged, and how big the untriaged backlog is behind it. stats = stats or {} matched = stats.get("matched") skipped = stats.get("skipped_unchanged") or 0 updated = stats.get("updated_count") or 0 backlog = stats.get("total_unsolved") - window = f"Analyzed **{len(findings)}**" + window = f"🗂️ **Zendesk triage** — analyzed **{len(findings)}**" if matched is not None: window += f" of **{matched}**" - window += f" tickets in the window" + window += " tickets in the window" if stats.get("scope"): window += f" ({stats['scope']})" window += "." @@ -884,99 +886,46 @@ def build_summary_embed(findings, highlights, subdomain, stats=None): if backlog is not None: lines.append(f"Backlog: **{backlog:,}** unsolved tickets in total (not triaged).") + serious = by_severity.get("crash", 0) + by_severity.get("data_loss", 0) tail = f"**{len(highlights)}** worth looking into" tail += f", including **{serious}** crash/data-loss." if serious else "." if updated: tail += f" 🔄 **{updated}** changed since last reported." lines.append(tail) - return { - "title": "🗂️ Zendesk triage", - "description": "\n".join(lines), - "color": 0xE67E22 if highlights else 0x2ECC71, - "fields": fields, - } - - -def is_urgent(finding): - return finding.get("category") in URGENT_CATEGORIES - - -def embed_color(finding): - """Colour by category urgency first, then severity. - - An abuse or legal report is not a bug, so the model rates it not_applicable — - which maps to the calmest blue. Category has to win, or the most serious ticket - in the digest looks the most benign. - """ - urgent = CATEGORY_COLOR.get(finding.get("category")) - if urgent: - return urgent - return SEVERITY_COLOR.get(finding.get("severity", "not_applicable"), 0x95A5A6) - - -def build_highlight_embed(finding, subdomain, is_update=False): - tid = finding["id"] - sev = finding.get("severity", "not_applicable") - cat = CATEGORY_LABEL.get(finding.get("category"), finding.get("category", "")) - # 🔄 marks a ticket we already showed that has since changed, so the reader - # knows it is a follow-up rather than a duplicate post. - marker = "🔄 " if is_update else "" - title = f"{marker}#{tid} · {clip(finding.get('summary'), 200) or '(no summary)'}" - fields = [ - {"name": "Category", "value": clip(cat, 60) or "—", "inline": True}, - {"name": "Severity", "value": sev, "inline": True}, - {"name": "Language", "value": clip(finding.get("language"), 40) or "—", "inline": True}, - ] - platform = finding.get("platform") - if platform and platform != "unknown": - fields.append({"name": "Platform", "value": clip(platform, 40), "inline": True}) - component = clip(finding.get("affected_component"), 100) - if component: - fields.append({"name": "Component", "value": component, "inline": True}) - version = clip(finding.get("app_version"), 40) - if version: - fields.append({"name": "Version", "value": version, "inline": True}) - # The reported account is the actionable part of an abuse report — surfacing it - # here saves opening the ticket to copy it. - reported = clip(finding.get("reported_session_id"), 100) - if reported: - fields.append({"name": "Reported account", "value": f"`{reported}`", "inline": False}) - root = clip(finding.get("likely_root_cause"), 300) - description = f"Likely cause: {root}" if root else "" - return { - "title": clip(title, 256), - "url": ticket_url(subdomain, tid), - "description": description, - "color": embed_color(finding), - "fields": fields, - } + if by_category: + lines.append(clip(" · ".join( + f"{CATEGORY_EMOJI.get(cat, '•')} **{count}**" + for cat, count in sorted(by_category.items(), key=lambda kv: -kv[1]) + ), 300)) + dup_clusters = {k: v for k, v in clusters.items() if len(v) > 1} + if dup_clusters: + cluster_lines = " · ".join( + f"**{clip(label, 40)}** ×{len(ids)} (#{', #'.join(str(i) for i in ids[:4])})" + for label, ids in sorted(dup_clusters.items(), key=lambda kv: -len(kv[1]))[:4] + ) + lines.append(f"Likely duplicates: {clip(cluster_lines, 400)}") -def embed_char_count(embed): - """Characters Discord counts against the per-message embed budget.""" - total = len(embed.get("title") or "") + len(embed.get("description") or "") - for field in embed.get("fields") or []: - total += len(field.get("name") or "") + len(field.get("value") or "") - return total + return "\n".join(lines) def chunk_entries(entries): - """Group (embed, ticket_ids) pairs into messages within both Discord limits. + """Group (line, ticket_ids) pairs into messages within MAX_MESSAGE_CHARS. - Discord caps a message at 10 embeds *and* 6,000 characters summed across them; - chunking on count alone can produce a payload that is rejected as too large. + Lines are joined with a newline, so each one after the first costs a character + more than its own length. An entry longer than the cap still gets its own message + rather than being dropped; the pieces are pre-clipped so that shouldn't arise. """ chunks, current, current_chars = [], [], 0 - for embed, ids in entries: - size = embed_char_count(embed) - too_many = len(current) >= MAX_EMBEDS_PER_MESSAGE - too_long = current_chars + size > MAX_EMBED_CHARS_PER_MESSAGE - if current and (too_many or too_long): + for text, ids in entries: + projected = current_chars + len(text) + (1 if current else 0) + if current and projected > MAX_MESSAGE_CHARS: chunks.append(current) current, current_chars = [], 0 - current.append((embed, ids)) - current_chars += size + projected = len(text) + current.append((text, ids)) + current_chars = projected if current: chunks.append(current) return chunks @@ -1006,27 +955,23 @@ def build_messages(findings, subdomain, stats=None, updated_ids=None): shown_ids = {f.get("id") for f in shown} omitted_ids = {f.get("id") for f in omitted} - # The summary embed accounts for every classified ticket except the highlights - # that didn't fit; those are covered by no message and stay eligible. - summary_ids = {f.get("id") for f in findings} - shown_ids - omitted_ids - entries = [(build_summary_embed(findings, shown + omitted, subdomain, stats), summary_ids)] + # The header accounts for every classified ticket except the highlights that + # didn't fit; those are covered by no message and stay eligible next run. + header_ids = {f.get("id") for f in findings} - shown_ids - omitted_ids + header = build_header(findings, shown + omitted, stats) + if omitted: + header += (f"\nShowing the top **{len(shown)}** of " + f"**{len(shown) + len(omitted)}** worth looking into.") + entries = [(header, header_ids)] entries += [ - (build_highlight_embed(f, subdomain, is_update=f.get("id") in updated_ids), + (build_ticket_line(f, subdomain, is_update=f.get("id") in updated_ids), {f.get("id")}) for f in shown ] - content = None - if omitted: - content = (f"Showing the top {len(shown)} of {len(shown) + len(omitted)} " - f"tickets worth looking into.") - messages, coverage = [], [] - for index, chunk in enumerate(chunk_entries(entries)): - payload = {"embeds": [embed for embed, _ in chunk]} - if index == 0 and content: - payload["content"] = content - messages.append(payload) + for chunk in chunk_entries(entries): + messages.append({"content": "\n".join(text for text, _ in chunk)}) covered = set() for _, ids in chunk: covered |= ids @@ -1061,7 +1006,9 @@ def main(): parser.add_argument("--window-hours", type=int, metavar="N", help="Only analyze unsolved tickets created in the last N hours. " "The scheduled daily run uses 48.") - parser.add_argument("--model", help="Claude model id (else ZENDESK_TRIAGE_MODEL, else claude-opus-4-8).") + parser.add_argument("--model", help=f"Claude model id, or an alias (opus, sonnet, " + f"haiku) mapped to an id " + f"(else ZENDESK_TRIAGE_MODEL, else {DEFAULT_MODEL}).") parser.add_argument("--effort", default="medium", choices=["low", "medium", "high", "xhigh", "max"], help="Claude reasoning effort (default: medium).") parser.add_argument("--max-tickets", type=int, default=DEFAULT_MAX_TICKETS, @@ -1072,11 +1019,9 @@ def main(): f"(default: {DEFAULT_BATCH_SIZE}).") parser.add_argument("--dry-run", action="store_true", help="Fetch and analyze, then print the Discord payload instead of posting.") - parser.add_argument("--backend", default="api", choices=["api", "claude-cli", "file"], - help="Where classification happens: the Anthropic API (default), the " - "local `claude` CLI (no API key needed), or a findings file.") - parser.add_argument("--findings", - help="Findings JSON to render instead of classifying (--backend file).") + parser.add_argument("--findings", metavar="PATH", + help="Render findings classified elsewhere, skipping Zendesk and " + "Claude entirely. Pairs with --dump-batch.") parser.add_argument("--review-star-floor", type=int, default=DEFAULT_REVIEW_STAR_FLOOR, metavar="N", help=f"Classify app-store reviews of N stars or fewer; count the rest " @@ -1098,9 +1043,6 @@ def main(): "hand-classification. WARNING: writes ticket content to disk.") args = parser.parse_args() - if args.backend == "file" and not args.findings: - sys.exit("--backend file requires --findings PATH.") - # Subdomain is always needed: it builds the ticket links in the Discord payload. subdomain = get_env("ZENDESK_SUBDOMAIN", args.subdomain) # A dump exits before rendering anything, so it never needs the webhook either. @@ -1113,7 +1055,7 @@ def main(): classified = [] updated_ids = set() - if args.backend == "file": + if args.findings: # Findings already exist, so neither Zendesk nor a model is involved. findings = load_findings(args.findings) print(f"Loaded {len(findings)} findings from {args.findings}.") @@ -1190,14 +1132,11 @@ def main(): dump_batch(args.dump_batch, compact, model) print(f"Wrote {len(compact)} tickets to {args.dump_batch} — this file contains " f"ticket content, so keep it out of the repo.") - print("Classify it, then: --backend file --findings --dry-run") + print("Classify it, then: --findings --dry-run") return - if args.backend == "claude-cli": - analyzer = partial(analyze_via_claude_cli, model) - else: - client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY - analyzer = partial(analyze, client, model, args.effort) + client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY + analyzer = partial(analyze, client, resolve_api_model(model), args.effort) findings = analyze_in_chunks(analyzer, compact, args.batch_size) # Keep only findings whose id maps to a fetched ticket, in case of drift. @@ -1212,7 +1151,7 @@ def main(): print(f"Note: {len(analyzed) - len(classified)} ticket(s) came back without a " f"classification; they stay eligible for the next run.") - # Same selection the embeds use, so the console count can't disagree with the + # Same selection the digest uses, so the console count can't disagree with the # digest — worth_looking_into alone would miss urgent categories the model # failed to flag. shown, omitted = select_highlights(findings)