Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .claude/skills/epic-codegen/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,20 @@ python3 scripts/check_dependencies.py ${EPIC_ID}
```

Exit 0 means proceed; exit 1 names the dependencies that are not done, and
this run stops there.
this run stops there. Record the stop as `codegen_outcome=blocked`, never
`failed`:

```bash
python3 scripts/frontmatter.py merge-run-metadata \
artifacts/codegen-runs/${EPIC_ID}/run-metadata.yaml \
epic_id=${EPIC_ID} codegen_outcome=blocked versions=0
```

Declining to start is not failing. The pipeline reads this to decide whether
the epic keeps its turn: `failed` writes the terminal CI state `Failed` and
the epic is skipped on every future run until someone edits the data repo by
hand, which is what happened to RHAI-761. `blocked` sends it back to `Blocked`
to be retried once the dependency lands.

Check `jira_status`, never the dependency's `status`. Every epic-task file is
regenerated from Jira on each run with `status: Pending` hardcoded
Expand Down
94 changes: 94 additions & 0 deletions docs/bugs/fixed/bug-declined-codegen-marked-terminally-failed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
---
id: bug-declined-codegen-marked-terminally-failed
title: Codegen declining to start was recorded as a terminal failure
type: bug
status: fixed
commits: ["a24e3e5"]
repos: [epic-code-gen]
decisions: [ADR-0025]
---

# Bug: Codegen declining to start was recorded as a terminal failure

## Summary

`/epic-codegen` runs a dependency gate before it generates anything. When the
gate says no, the skill stops — correctly. But it recorded that stop as
`codegen_outcome: failed`, and `_ci_handle_ready` had one branch for a codegen
that did not produce artifacts:

```python
state["status"] = "Failed"
state["failure_reason"] = "codegen failed"
```

`Failed` is in `CI_TERMINAL_STATES`. A terminal state is never revisited, so
the epic is skipped on every subsequent run, forever, until someone edits the
data repo by hand.

Refusing to start and trying and breaking are not the same event. The first is
a statement about the *world* — a dependency isn't done yet — and the world
changes between runs. The second is a statement about the epic. Only the second
justifies giving up on it.

## Reproduction

Run an epic whose dependency the gate reports as not done. In the data repo:

```yaml
status: Failed
codegen_outcome: failed
failure_reason: codegen failed
```

All three are false. Nothing failed; nothing was even attempted.

## Expected

`status: Blocked`, `codegen_outcome: blocked`, `blocked_by:` naming the
dependency — a state the next run re-examines.

## Actual

RHAI-761 was marked terminally `Failed` at 19:07:29 by a gate misfire (see
[[bug-dependency-gate-read-stale-snapshot]]) and stayed out of the pipeline
across every later invocation. The stale snapshot cost one cycle; this defect
turned that into all of them.

## Impact

High. It converts any transient, self-correcting condition into permanent
removal from the pipeline, and it does so silently — the dashboard reads
`Failed` and shows a broken epic, so the operator looks for a bug in the epic
rather than in the pipeline's bookkeeping.

## Fix

Two halves, because there are two writers.

The skill (`.claude/skills/epic-codegen/SKILL.md`) now records a gate stop as
`codegen_outcome=blocked`. `blocked` was added to `CODEGEN_OUTCOMES` in
`artifact_utils.py`, which is the single definition the `codegen-run` schema
enum and `merge_run_metadata`'s validation both derive from — so no other list
needed touching.

The pipeline (`run_pipeline.py`) reads that outcome before deciding the CI
state. `blocked` sends the epic back to `Blocked` with its `blocked_by` intact
and no `failure_reason`; anything else still yields `Failed`. This works
because `_merge_run_metadata_into_state` folds the skill's fields into the live
state dict before the branch runs.

The lowercase outcome `blocked` and the capitalised CI state `Blocked` are
deliberately distinct vocabularies over the same word, so
`read_codegen_outcome({"status": "Blocked"})` must keep returning `None` —
pinned by `test_blocked_ci_state_is_not_the_blocked_outcome` in
`tests/test_artifact_utils.py`. Behaviour is covered by
`TestCodegenDeclinedIsNotFailed` in `tests/test_ci_mode.py`.

## Related

- [[bug-dependency-gate-read-stale-snapshot]] — the misfire that exposed this.
- [[bug-clone-fault-marks-epic-failed]] — still open, and the same mistake one
layer up: a clone or credential fault also writes terminal `Failed`. This fix
covers the skill declining, not the environment breaking.
- ADR-0025 "unrunnable is not failed" — the principle both violate.
91 changes: 91 additions & 0 deletions docs/bugs/fixed/bug-dependency-gate-read-stale-snapshot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
---
id: bug-dependency-gate-read-stale-snapshot
title: Dependency gate read a snapshot the same run had already invalidated
type: bug
status: fixed
commits: ["a24e3e5"]
repos: [epic-code-gen]
decisions: [ADR-0025]
---

# Bug: Dependency gate read a snapshot the same run had already invalidated

## Summary

`check_dependencies.py` decides whether an epic may start by reading each
dependency's `jira_status` out of the on-disk epic-task file:

```python
jira_status = read_frontmatter(dep_path)[0].get("jira_status")
done = jira_status in DONE_STATUSES # Closed, Done, Resolved
```

Those files are written **once**, at the top of a run, by
`fetch_jira_epics.py`. Everything after that point reads a photograph of Jira
taken before the run started — including the parts of the run that change Jira.

A single pipeline invocation therefore both wrote and invalidated its own
source of truth:

```
19:06:49 RHAI-760 → Done, transitioned and closed in Jira by run_pipeline.py
19:07:29 RHAI-761 starts; gate reads artifacts/epic-tasks/RHAI-760.md
→ jira_status: In Progress (40 seconds stale)
→ exit 1, "dependencies not done"
```

Nothing was wrong with the dependency. It had been satisfied by the same
process, 40 seconds earlier, in memory the gate could not see.

## Reproduction

```python
transition_issue("s", "u", "t", "RHAI-760", "Done", tasks_dir)
check_dependencies("RHAI-761", tasks_dir)["all_done"] # False, before the fix
```

## Expected

An epic whose only blocker was closed earlier in the same run is eligible.

## Actual

The gate refuses. The skill then records the refusal as
`codegen_outcome: failed`, which the pipeline maps to the terminal CI state
`Failed` — so RHAI-761 was not merely delayed by one cycle, it was removed from
every future cycle until its state was edited by hand in the data repo. That
second half is its own defect; see
[[bug-declined-codegen-marked-terminally-failed]].

## Impact

High, and it fires precisely where the DAG is doing its job: a chain of
dependent epics under one strategy is the case the dependency graph exists to
handle, and it is the only case where a run closes something another epic is
waiting on. Independent epics never hit it.

## Fix

Keep the snapshot in step with the transitions the run performs.
`sync_epic_task_jira_status()` in `run_pipeline.py` writes the new status back
to `artifacts/epic-tasks/<KEY>.md`, and `transition_issue()` calls it on every
successful transition. All nine epic-level call sites pass the directory; the
two strategy-level sites do not, because strategy keys have no epic-task file
(the helper no-ops on a missing file rather than treating it as an error).

The narrower alternative — have `check_dependencies.py` query Jira live —
was rejected: it puts a network call in a gate that runs once per epic, and it
leaves every *other* reader of the snapshot still stale.

Regression coverage in `tests/test_run_pipeline.py`,
`TestSyncEpicTaskJiraStatus`, including the end-to-end case
`test_dependent_gate_passes_after_transition`, which closes a dependency
through `transition_issue` and then asserts the real
`check_dependencies` gate opens.

## Related

- [[bug-declined-codegen-marked-terminally-failed]] — what turned this
one-cycle delay into a permanent stall.
- [[bug-clone-fault-marks-epic-failed]] — same shape: an environment or timing
fault recorded as an epic-level failure, against ADR-0025.
8 changes: 8 additions & 0 deletions docs/bugs/open/bug-clone-fault-marks-epic-failed.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,16 @@ Classify the clone failure the way preflight already classifies a missing tool:
Worth extracting the retryable-vs-terminal judgement into one helper shared
with the preflight gate, rather than a second ad-hoc copy of the rule.

That helper now has a third caller waiting for it:
[[bug-declined-codegen-marked-terminally-failed]] fixed the same
retryable-written-as-terminal mistake for the codegen branch of
`_ci_handle_ready`, with its own inline rule. Two ad-hoc copies exist; the
clone branch would be the third. Fixing this one should fold all three
together.

## Related

- [[bug-declined-codegen-marked-terminally-failed]] — the sibling case, fixed.
- [[bug-slug-extractor-truncated-repo-names]]
- [[task-toolchain-preflight]]
- [[task-per-repo-github-identity]]
6 changes: 5 additions & 1 deletion scripts/artifact_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,11 @@
"PRCreated", "PRChangesRequested", "Done", "Blocked", "Failed",
)

CODEGEN_OUTCOMES = ("completed", "exhausted", "failed", "error")
# `blocked` is a refusal, not a failure: the skill declined to generate because
# a dependency is not done. It exists so the pipeline can tell "this epic has
# nothing to do yet" from "this epic tried and broke", which decides whether
# the epic keeps its retry (RHAI-761 was marked terminally Failed for it).
CODEGEN_OUTCOMES = ("completed", "exhausted", "failed", "error", "blocked")

# Fields in run-metadata.yaml that belong to the pipeline's state machine. No
# other producer may set them, and a producer's own metadata must never
Expand Down
Loading
Loading