Repository navigation
Conversation
GitLab marks a job that exceeds its timeout with the failure reason job_execution_timeout. The retry list had only stuck_or_timeout_failure, so a stalled build failed without a retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 2 Pipeline jobs failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 13e758d | Docs | View more details | Give us feedback! |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the update enabling retries of timed-out Bottlecap builds.
No open issues 🎉
What changed in this PR
Updates the shared GitLab build template to retry Bottlecap builds that exceed their job timeout, reducing avoidable failures as the repository adopts Datadog MergeQueue.
Changes:
- Adds
job_execution_timeoutto the existing retry conditions, retaining the maximum of two retries. - Adds a comment distinguishing job timeout failures from
stuck_or_timeout_failure.
| File | Description |
|---|---|
.gitlab/templates/pipeline.yaml.tpl |
Enables timeout retries for all eight extension build variants. |
Review effort: Balanced
Problem
Merging PRs takes too much manual work, and the work grows with the number of PRs. The branch protection on
mainrequires a branch to be up to date before it merges. So each merge puts every other open PR behindmain. Its author must then mergemaininto the branch, wait for CI again, and merge before someone else does. When one person merges a stack of PRs, they repeat this for each PR.Plan
Move merges to the Datadog MergeQueue, with the
/mergecomment. For each PR, the queue makes a temporary branch,mq-working-branch-*, with the latestmainplus the PR. It runs CI on that branch and merges the PR only if CI passes. So the queue tests each PR against the latestmain, and the author does not update the branch.main. It waits with the blocker "branch behind base". A/mergeon #1492 showed this./mergemain, so authors do not update branches.This PR
Makes GitLab retry a build job that stalls and hits its time limit.
How often builds stall: GitLab has 8 build jobs for the extension binary. There is one for each combination of CPU type, FIPS (Federal Information Processing Standards) and Alpine Linux. From 2026-10-06 to 2026-11-04, these jobs timed out in 223 of 3964 attempts (5.6%). About a third of pipelines had at least one timeout. In a stalled job, the log stops in the middle of the compile and shows nothing until GitLab stops the job. A build that passes takes 19 minutes at most.
Why GitLab did not retry them: the job template retries a failed job up to 2 times. The comment above that setting says that the retry is there because the job "sometimes times out". But the retry applies only to the failure reasons in its list. GitLab gives these timeouts the reason
job_execution_timeout, which was not in the list. On the branch of #1492, three pipelines had 4 timeouts, and GitLab retried none of them.The change: adds
job_execution_timeoutto the retry list. A retry adds about 20 minutes to a pipeline that stalls. GitLab 19.1 deprecates this value and replaces it withserver_timeout_running. Our GitLab server is version 18.5.7.Testing
Steps
not_a_reasonin place ofjob_execution_timeout, as a control.Result
job_execution_timeoutnot_a_reasonThe pipeline of this PR rendered the changed template, and all 8 builds passed on the first try. So a retry after a real timeout is not tested, because I cannot make a build stall on demand.
🤖 Generated with Claude Code