Rebase consolidation improvements and code review fixes - #726
Rebase consolidation improvements and code review fixes#726majamassarini wants to merge 67 commits into
Conversation
PR Summary by QodoConsolidate rebase MRs across sibling Jira issues + version-aware triage fixes
AI Description
Diagram
High-Level Assessment
Files changed (10)
|
Code Review by Qodo
1.
|
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit ec90fb4 |
59f6de4 to
a2b67a7
Compare
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit a2b67a7 |
a2b67a7 to
634f48e
Compare
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 634f48e |
634f48e to
e19b4a0
Compare
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit e19b4a0 |
ce969cb to
cb92d04
Compare
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 6a219db |
6a219db to
7753470
Compare
|
/agentic_review |
|
Code review by qodo was updated up to the latest commit 7753470 |
7753470 to
6d54d9e
Compare
|
/agentic_review |
Increment queued_count immediately after the Redis push (the critical operation), not after comment_in_jira. This prevents miscount when Jira label/comment operations fail but the sibling is already queued. If comment_in_jira raised, we would skip the count increment, leading consolidate_rebase_siblings to believe no siblings were queued and continue without setting waiting_for_siblings=True. Now label/comment are best-effort with individual try-except blocks, and the count reflects actual Redis queue state. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
If an issue has ymir_rebase_sibling label, it's already part of another primary's sibling group. Skip searching for more siblings to avoid: - Re-posting duplicate "Queued for triage as potential sibling" comments on the same siblings - Treating siblings as new primary issues - Creating confusing nested sibling relationships Siblings should just proceed to rebase without consolidation logic. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
get_jira_details returns {key, id, fields}, not top-level labels/comments.
Labels are at fields.labels, comments at fields.comment.comments.
Fixed three locations:
- check_and_queue_primary_if_ready: sibling comment scan (line 468)
- check_and_queue_primary_if_ready: primary label check (line 492)
- find_triaged_rebase_siblings: sibling comment verification (line 634)
Without this fix, siblings never find their primary issue in comments,
and primaries are never detected as waiting for siblings.
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Keep ymir_waiting_for_siblings label on the primary when re-queuing to triage after siblings finish. This allows the primary to bypass the dedup check (non-terminal label exception) and be processed. Triage will automatically remove ymir_waiting_for_siblings along with other ymir_* labels during normal processing (line 1285-1297). Removing the label before re-queuing created a race where triage would see only ymir_triaged_rebase (terminal), fail the dedup check, and skip processing in automated runs. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
When a sibling is queued, there's a race between: 1. Sibling gets queued to triage 2. Sibling label gets added to Jira 3. Sibling starts triaging If (3) happens before (2), the sibling won't see its own label and will treat itself as a primary, queueing more siblings. Now check both: - ymir_rebase_sibling label (fast path) - "Queued for triage as potential sibling" comment (fallback) This prevents siblings from acting as primaries when label hasn't propagated yet. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Before queueing a sibling, check if it has ymir_rebase_sibling or ymir_triaged_rebase labels. This prevents: - Multiple primaries from queuing the same sibling - Duplicate comments on siblings - Siblings being treated as part of multiple groups Labels are more robust than comments for detecting whether a sibling was already processed by another primary. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Queue sibling to triage AFTER setting its label, not before. This ensures the sibling sees ymir_rebase_sibling when it starts processing and won't treat itself as a primary. Previous order: 1. Push to Redis queue 2. Add label (async) 3. Sibling starts triaging (label might not be visible yet) 4. Sibling treats itself as primary New order: 1. Add label (blocks until confirmed) 2. Push to Redis queue 3. Sibling starts triaging (label is visible) 4. Sibling skips consolidation If labeling fails, skip queueing the sibling to avoid it being treated as a primary. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Order of operations for queueing a sibling: 1. Set ymir_rebase_sibling label (blocks until Jira confirms) 2. Post "Queued for triage as potential sibling" comment 3. Push to Redis triage queue This maximizes the chance that when the sibling starts processing, it will see EITHER the label OR the comment (or both) and skip consolidation. Even if Jira has eventual consistency issues, both markers are written before the sibling can start processing. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Siblings should not create their own MR. Only the primary issue consolidates all siblings into a single MR. Add check_if_sibling step at the start of rebase workflow: - Check comments for "Queued for triage as potential sibling" marker - If found, exit immediately without processing - Log that the primary will handle consolidation Cannot check ymir_rebase_sibling label because triage cleanup removes all ymir_* labels before queueing to rebase. Siblings already called check_and_queue_primary_if_ready() in triage (line 1476), so no further action needed in rebase agent. Without this check, siblings would create duplicate MRs for the same rebase. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Siblings inherit the user_triggered flag from the primary issue to preserve priority and signal intent. Changes: - Pass user_triggered to Task.from_issue(candidate_key, user_triggered) - Queue to TRIAGE_QUEUE_TODO when user_triggered=True (priority queue) - Queue to TRIAGE_QUEUE when user_triggered=False (normal queue) Without this fix, siblings of user-triggered primaries would be treated as automatic runs, losing priority and bypassing user-triggered-only logic (like posting acknowledgement comments). Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
check_and_queue_primary_if_ready was only checking for siblings with ymir_rebase_sibling label, but triage removes this label when it starts processing (line 1308 in triage_agent). This caused primaries to be queued too early when some siblings were still in triage. Changes: - Search for siblings with EITHER ymir_rebase_sibling OR ymir_triage_in_progress - For in-progress siblings, verify they are siblings of THIS primary by checking comments - Only queue primary when ALL actual siblings are done (no label OR completed) Without this fix, RHEL-212117 was stuck with ymir_waiting_for_siblings because it was queued before all siblings finished (siblings lost the label when triage started but were still processing). Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
When rebase agent runs consolidation, it should include siblings that were already triaged (have ymir_triaged_rebase label). This allows re-running the primary to consolidate with siblings even after they finish triage. Changes: - Add exclude_triaged parameter to build_rebase_siblings_jql (default True) - find_rebase_siblings sets exclude_triaged=False to include all siblings - Fetch comment field and verify siblings were queued by THIS primary - Filter candidates to only those with sibling comment mentioning this primary - queue_siblings_for_triage uses default (exclude_triaged=True) to only queue new siblings Without this fix, re-running RHEL-212117 would find 0 siblings because all siblings already have ymir_triaged_rebase label, causing the primary to create a separate MR instead of consolidating. The comment verification prevents consolidating siblings that belong to a different primary with the same component/fix_version. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
When a sibling finishes triage, check_and_queue_primary_if_ready is only called if is_sibling is True. The check was using current_labels which was fetched at the start (line 1217), but ymir_rebase_sibling was already removed from Jira by line 1308 cleanup. This caused siblings to NOT call check_and_queue_primary_if_ready, leaving primaries stuck forever. Changes: - Check for sibling comment instead of ymir_rebase_sibling label - Comment "Queued for triage as potential sibling of" is never removed - This ensures siblings always call check_and_queue_primary_if_ready Without this fix, RHEL-211884 finished triage but never checked if RHEL-212117 (primary) was ready, leaving it stuck with ymir_waiting_for_siblings. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
get_jira_details tool only accepts issue_key parameter, not fields. The fields parameter was causing ToolInputValidationError. Fixed in: - triage_agent.py: 2 occurrences (consolidate_rebase_siblings, is_sibling check) - rebase_agent.py: 1 occurrence (check_if_sibling) Error was: Extra inputs are not permitted [type=extra_forbidden, input_value=['comment']] Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical bug: siblings were being queued for their own rebase after triage completion, instead of only triggering check_and_queue_primary_if_ready(). Root cause: - is_sibling check was performed (line 1469-1480) to call check_and_queue_primary_if_ready() - BUT the downstream queueing logic (line 1515-1533) only checked if the issue is WAITING for siblings, not if it IS a sibling - So siblings with Resolution.REBASE were queued normally Result: Multiple rebases triggered (one per sibling), defeating consolidation. Fix: - Added is_sibling check BEFORE waiting_for_siblings check in rebase queueing - Siblings now skip queueing entirely with log message: "Issue X is a sibling, skipping rebase queue (will be consolidated with primary)" This ensures only the primary issue gets queued for rebase after all siblings finish triaging. Discovered via Phoenix/logs analysis: - RHEL-234827 (sibling) completed triage with Resolution.REBASE - Log showed: "Pushed RHEL-234827 to rebase_queue_c10s" - is_sibling check ran but didn't prevent queueing Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Two related bugs:
1. Siblings with different resolutions block primary from queueing
- Primary triages to REBASE
- Sibling triages to BACKPORT/REBUILD/etc (different fix needed)
- Primary waits forever for sibling to "finish"
- But sibling IS finished, just with different resolution
2. Terminal label failures prevent consolidation from proceeding
- Network error prevents setting ymir_triaged_* label
- Sibling stays with ymir_triage_in_progress label
- Primary blocked waiting for sibling (looks in-progress)
Fixes:
1. Exclude all terminal labels from pending sibling check:
- ymir_triaged_rebase, ymir_triaged_backport, ymir_triaged_rebuild,
ymir_triaged_not_affected, ymir_triaged_postponed
- These siblings are done, don't block primary
2. Mark terminal label update as critical (enables retry on failure):
- critical=True triggers 3 retry attempts with backoff
- Raises exception if all retries fail
- Prevents proceeding without terminal label set
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Bug: current_labels.remove(JiraLabels.REBASE_SIBLING.value) can raise
ValueError and crash the triage worker AFTER terminal labels have been
written to Jira.
Root cause:
- current_labels is fetched at line 1199 BEFORE workflow runs
- is_sibling is determined by comment check (line 1373-1381)
- ymir_rebase_sibling label was already removed at line 1308 (cleanup)
- current_labels snapshot is stale and doesn't have the label
- list.remove() raises ValueError if item not present
Scenario:
1. Sibling starts triage
2. Line 1308: ymir_rebase_sibling label removed from Jira
3. Line 1373-1381: is_sibling=True (found via comment)
4. Line 1394-1400: Terminal label written (critical=True, succeeds)
5. Line 1405: current_labels.remove(REBASE_SIBLING) → ValueError!
6. Worker crashes after successful label write
Fix:
Guard the removal with existence check:
if is_sibling and JiraLabels.REBASE_SIBLING.value in current_labels:
current_labels.remove(...)
This prevents ValueError while maintaining the intent to sync
current_labels with Jira state (used at line 1531 for waiting check).
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical bug: Siblings queued without marker comment cannot be detected as siblings, breaking consolidation workflow. Root cause: queue_siblings_for_triage() has two safety writes before queueing: 1. Line 408-417: Add ymir_rebase_sibling label → has 'continue' on failure ✅ 2. Line 421-431: Post marker comment → logs warning but continues ❌ 3. Line 440: Sibling queued to Redis anyway The marker comment is CRITICAL because downstream code depends on it: - is_sibling check (line 1373-1381 in triage_agent.py) scans for this comment - check_and_queue_primary_if_ready() (line 515-527) extracts primary issue from it Without the comment: - Sibling doesn't detect is_sibling=True - Sibling triggers its own rebase (defeats consolidation!) - check_and_queue_primary_if_ready() cannot find primary issue key - Primary never gets queued for rebase Fix: If comment_in_jira fails: 1. Roll back ymir_rebase_sibling label (best effort) 2. Do NOT queue sibling task to Redis (continue to next sibling) 3. Do NOT increment queued_count This ensures siblings are only queued when BOTH safety writes succeed, maintaining the invariant that queued siblings can always be detected and correlated back to their primary. Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
The MCP get_jira_details tool returns comment bodies in Jira's ADF
(Atlassian Document Format), not as plain text. The is_sibling check was
looking for comment.get("body", "") as a string, but "body" is actually a
nested JSON structure:
{
"type": "doc",
"content": [
{"type": "paragraph", "content": [{"type": "text", "text": "..."}]}
]
}
This caused is_sibling to always return False, so siblings were being queued
for rebase instead of being skipped.
Added _extract_text_from_adf() helper to recursively extract text from ADF
nodes, and updated the is_sibling check to use it.
Related issue: RHEL-234827 (sibling) triggered rebase workflow after being
queued as sibling of RHEL-234905.
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
The MCP get_jira_details tool returns comment bodies in Jira's ADF
(Atlassian Document Format), not as plain text. Both the is_sibling check
in triage_agent.py and the primary issue extraction in
check_and_queue_primary_if_ready were looking for comment.get("body", "")
as a string, but "body" is actually a nested JSON structure.
This caused:
1. is_sibling to always return False → siblings queued for rebase
2. check_and_queue_primary_if_ready to fail with "No primary issue found
in comments" → primary never queued even when all siblings done
Added extract_text_from_adf() to ymir.common.utils to recursively extract
text from ADF nodes, and updated both code paths to use it.
Fixes: RHEL-234827 (sibling) triggered rebase workflow
Fixes: RHEL-234905 (primary) not queued when all siblings finished
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
The sibling marker comment contains an inlineCard node with the primary
issue key in its URL:
{"type": "text", "text": "Queued for triage as potential sibling of "},
{"type": "inlineCard", "attrs": {"url": "https://.../RHEL-234905#..."}}
The previous extract_text_from_adf only extracted from "text" nodes and
"content" arrays, so it returned:
"Queued for triage as potential sibling of "
with no issue key.
This caused check_and_queue_primary_if_ready to fail with "No primary
issue found in comments" because the regex couldn't find RHEL-\d+ in the
extracted text.
Updated extract_text_from_adf to also extract URLs from inlineCard nodes,
so the regex can find the issue key in the URL.
Fixes: Primary issue not queued when all siblings finished triaging
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
The pending siblings check at line 624 was still using the old
comment.get("body", "") without extract_text_from_adf().
This caused the check to fail when verifying if a ymir_triage_in_progress
issue is actually a sibling of THIS primary (vs a sibling of some other
primary). The comment body is ADF JSON, not plain text, so the string
checks for "Queued for triage as potential sibling of" and primary_issue
always failed.
Result: pending siblings count was wrong, so check_and_queue_primary_if_ready
thought there were no pending siblings even when RHEL-223787 was still
in progress with ymir_rebase_sibling label.
Updated to use extract_text_from_adf() like the other comment checks.
Fixes: Primary not queued when actual pending siblings exist
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
The fetcher was skipping ANY issue with ANY ymir_* label (except ymir_retry_needed and ymir_todo). This meant that siblings queued by rebase consolidation with ymir_rebase_sibling label would be skipped by the fetcher if they got lost from Redis (e.g. Redis restart, or never successfully pushed). The consolidation code queues siblings and adds the ymir_rebase_sibling label as a marker, but if the Redis queue entry gets lost before the triage agent processes it, the fetcher would never re-queue it because it sees the ymir_* label and marks it as "existing". Result: RHEL-223787 was queued on Aug 10 with ymir_rebase_sibling label but never triaged, blocking the primary RHEL-234905 indefinitely. Added ymir_rebase_sibling to the list of labels that don't prevent fetcher from queueing (like ymir_retry_needed and ymir_todo). Fixes: Siblings queued with ymir_rebase_sibling never get triaged Fixes: Primary blocked indefinitely waiting for lost sibling Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
When all siblings finish triaging, check_and_queue_primary_if_ready re-queues the primary to triage, but the ymir_waiting_for_siblings label was never removed. This causes triage to skip queueing for rebase (line 1540: queue = None) because it sees the label and thinks siblings are still pending. Result: If rebase fails or is interrupted, retriggering the primary will skip queueing because the label is still there, even though all siblings finished long ago. The issue is stuck forever. Fixed by removing ymir_waiting_for_siblings in check_and_queue_primary_if_ready BEFORE re-queueing the primary. This is the right place because we KNOW for certain that all siblings are done at this point. Marked as critical=True so it retries 3 times with backoff - if this fails, the primary is stuck forever anyway. Fixes: Primary stuck with ymir_waiting_for_siblings after rebase interrupted Fixes: RHEL-234905 will be stuck if current rebase fails Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
When triage finds siblings for a rebase, it sets state.waiting_for_siblings=True and adds ymir_waiting_for_siblings label. But it was also adding the terminal label ymir_triaged_rebase at the same time. When all siblings finish, check_and_queue_primary_if_ready re-queues the primary to triage. But triage's dedup check sees ymir_triaged_rebase (terminal label) and skips: "Skipping duplicate triage for RHEL-234905 — already has labels: ['ymir_triaged_rebase']" Result: Primary never queued for rebase, stuck forever. Fixed by skipping terminal label when state.waiting_for_siblings=True or ymir_waiting_for_siblings label is present. The terminal label will be added when the primary is re-triaged after all siblings finish. Fixes: Primary not queued for rebase after all siblings finish Fixes: RHEL-234905 stuck with both ymir_triaged_rebase + ymir_waiting_for_siblings Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
…ncies
Issue 1: Already-triaged siblings queued again
RHEL-223919 had ymir_backported (already finished backport with MR), but
was queued as sibling of RHEL-234905 because the check only looked for
ymir_rebase_sibling and ymir_triaged_rebase, missing other terminal labels.
Fixed by checking ALL terminal labels before queueing siblings:
- ymir_triaged_* (all resolutions)
- ymir_backported, ymir_rebased, ymir_rebuilt
Issue 2: Circular sibling dependencies
RHEL-234375 was queued as sibling of RHEL-234905, then triaged and found
its own siblings, becoming a primary waiting for siblings. Then it was
queued AGAIN as sibling of RHEL-234905, creating circular dependency.
Root cause: The sibling check at line 970 used comment.get("body", "")
which doesn't work with ADF, so it never detected that RHEL-234375 was
a sibling and allowed it to queue siblings.
Fixed by using extract_text_from_adf() in the sibling check so siblings
correctly skip consolidation and don't search for their own siblings.
Fixes: Already-completed issues re-triaged as siblings
Fixes: Circular sibling dependencies (sibling becomes primary)
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Three locations were still using comment.get("body", "") without ADF extraction:
1. rebase_agent.py line 290: check_if_sibling
- Siblings not detected → create duplicate MRs instead of consolidating
2. rebase_consolidation.py line 178: find_rebase_siblings verified_candidates filter
- All candidates filtered out as "not queued by this primary" → empty consolidation
3. rebase_consolidation.py line 759: find_triaged_rebase_siblings
- Verified siblings excluded from consolidation → incomplete MRs
All now use extract_text_from_adf() to handle ADF-formatted comment bodies from MCP.
Fixes: Code review findings packit#1, packit#2, packit#7
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
If primary issue is closed/resolved but rebase workflow runs from stale Redis message, find_triaged_rebase_siblings would still consolidate siblings with the closed primary → MR description references invalid/closed issue → broken links and confusion. Added validation at start of find_triaged_rebase_siblings to check if primary is Closed/Done/Resolved. If so, skip consolidation and log warning. Example scenario prevented: 1. RHEL-500 (primary) + RHEL-400 (sibling) triaged 2. RHEL-500 manually closed 3. Old rebase workflow message processes 4. Would create MR linking RHEL-400 to closed RHEL-500 ✗ 5. Now skips consolidation ✓ Fixes: Code review finding packit#8 Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
When primary issue is waiting for siblings, the comment now lists which
siblings it's waiting for instead of just the count:
Before: "Waiting for 3 sibling(s) to finish triaging before starting rebase"
After: "Waiting for 3 sibling(s) to finish triaging before starting rebase:
RHEL-234827, RHEL-234375, RHEL-224663"
Jira automatically converts issue keys to clickable links, making it easy
to navigate to siblings and track consolidation progress.
Assisted-by: Claude Sonnet 4.5 <noreply@anthropic.com>
70a95c7 to
41b9bc7
Compare
Summary
This PR implements rebase consolidation (grouping multiple CVE issues into a single rebase MR) and addresses code review findings for performance and maintainability.
Key Features
Rebase Consolidation
Code Review Fixes
rpmdev-vercmpinstead of string equality for sibling version matchingasyncio.gather()for sibling analysis and Jira API callsChanges
Consolidation Implementation
Performance & Maintainability
build_siblings_jql()for JQL construction,update_labels_for_all_issues()for label updatescompare_versions()inversion_utils.pywrapsrpmdev-vercmpTesting
Tested against dotnet8.0 CVE issues (RHEL-211859 and 12 siblings):
Related Issues
Addresses feedback from rebase consolidation implementation review and Slack discussion about RHEL-211859 "already at version" error.
🤖 Generated with Claude Code