Implement reproducer agent

[vmihalko]

TODO:

• Unit tests for Testing Farm / SSH tools (710 lines)
• Unit tests for reproducer input/output models
• Updated existing test_push_to_remote_repository for new communicate() API
• E2E tests (deferred to follow-up PR)
• Doc-strings on all new public classes and helpers
• Skill definition (agents_as_skills/reproducer/SKILL.md)
• Deployment config (compose service, Makefile targets, env template)
• Jinja2 prompt template following existing agent patterns
• Validated on 2 different Jira issues / bug types (RHEL-46618 ksh, RHEL-170657 file)
• 8 end-to-end test runs against real Testing Farm infrastructure

Background

I've been using and improving a custom reproducer skill over the last few months. After a meeting with @TomasKorbar, I took my reproducer skill and @mkyral's cve-test-generator (kudos to him) as references, and let Claude Opus 4.6 read the existing ai-workflows agents and codebase to prepare an implementation plan: basic functionality + tests, monofunctional commits. Once it was working, I started an iterative implement - review - fix - test run loop with Claude. This PR is the result after 16 runs.

What

Adds a reproducer agent that takes a Jira issue, provisions a Testing Farm machine, designs and iterates on a BeakerLib reproducer test against real RHEL, and opens a merge request with the working test.

New components

• Testing Farm / SSH tools (testing_farm.py): ReserveTestingFarmMachine, GetTestingFarmReservationDetails, CancelTestingFarmRequest, RunRemoteCommand, CopyFilesToRemote - full reservation lifecycle and remote execution via the MCP gateway
• TF cleanup middleware (tf_cleanup_middleware.py): event-based tracker that cancels leaked Testing Farm reservations on agent crash, preventing orphaned machines from burning quota
• Reproducer agent (reproducer_agent.py): orchestration with dedup guard (prevents duplicate runs across concurrent deployments), structured output, retry logic, and Jira lifecycle labels
• Prompt template (prompts/reproducer/prompt.j2): 717-line Jinja2 template guiding the LLM through issue analysis, test design, TF provisioning, iterative verification, and MR submission
• Skill definition (agents_as_skills/reproducer/SKILL.md): registers the agent as a Claude Code slash-command skill
• Deployment config: compose service, Makefile targets, env template

Bug fixes in existing GitLab tools

• push_to_remote_repository: was silently failing with a generic error and leaking credentials via git stderr. Now captures stderr via subprocess.PIPE + communicate(), filters auth-related lines through _sanitize_git_stderr, and includes sanitized output in the error message.
• clone_repository: failed on retry when a previous clone left a partial directory behind. Now removes the target directory before cloning (guarded by an allowlist of safe parent paths).
• ToolError double-wrapping: 5 existing tools had except Exception handlers that caught ToolError and re-wrapped it, losing the specific error message. Added except ToolError: raise before each except Exception.

Test coverage

• Unit tests for all Testing Farm/SSH tools (710 lines)
• Unit tests for reproducer input/output models
• Updated test_push_to_remote_repository for the new communicate() API

Note: E2E tests for the reproducer agent are intentionally deferred to a follow-up PR to allow the team to decide on the e2e testing approach independently.

Test runs and results

The agent was validated over 8 end-to-end runs against real Jira issues.

Issue	Package	Result	Cost	Duration	Tool calls
RHEL-46618	ksh	SUCCESS	$11.21 (3.6M tokens)	~22 min	49
RHEL-170657	file	SUCCESS	$6.43	~10 min	36

The cost difference is driven by bug complexity: ksh required 22 remote commands (valgrind, pmap, multiple reproducer variants, RSS threshold tuning) vs file's 6 (create a 32-byte binary, run file, check output). Both runs demonstrate the agent generalizes across different bug types.

Merge requests created by the agent

• https://gitlab.com/redhat/rhel/tests/ksh/-/merge_requests/5 (RHEL-46618, ksh memory leak reproducer - CI failed due to non-executable test files, fixed in the prompt template)
• https://gitlab.com/redhat/rhel/tests/file/-/merge_requests/2 (RHEL-170657, file magic database reproducer - CI failed due to the target repo's TMT plan configuration, not an agent bug)

Related to
#553

[gemini-code-assist]

Code Review

This pull request introduces a new "reproducer" agent skill to Ymir, designed to automatically create, verify, and publish minimal reproducers for RHEL bugs and CVEs. It adds the reproducer workflow, prompts, BeakerLib templates, Testing Farm integration tools (for machine reservation, remote command execution, and file copying), and cleanup middleware. The feedback focuses on critical improvements: wrapping synchronous blocking calls (such as API requests and directory removals) in asyncio.to_thread to prevent event loop starvation, fixing a scoping bug in the BeakerLib test template where variables are lost across subshells, robustly parsing LLM JSON responses to handle markdown code blocks, and using Path.is_relative_to for safer path validation.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

I am having trouble creating individual review comments. Click here to see my feedback.

ymir/tools/privileged/testing_farm.py (346)

The _testing_farm_api_post call is synchronous and blocking. Since this tool is executed within an asynchronous context (async def _run), calling it directly blocks the event loop and can starve other concurrent tasks. Wrap the call in asyncio.to_thread to run it in a separate thread.

            response = await asyncio.to_thread(_testing_farm_api_post, "requests", json=body)

ymir/tools/privileged/testing_farm.py (397-409)

There are two issues here:

1. The _testing_farm_api_get call is synchronous and blocking. It should be run in a separate thread using asyncio.to_thread to avoid blocking the event loop during polling.
2. Catching only requests.HTTPError means other requests-related exceptions (like ConnectionError or Timeout) will propagate to the generic except Exception block and immediately abort the polling loop. Catching requests.RequestException and treating connection/timeout errors as transient makes the polling loop much more resilient to temporary network issues.

                response = await asyncio.to_thread(_testing_farm_api_get, f"requests/{tool_input.request_id}")
            except requests.RequestException as e:
                is_transient = False
                if isinstance(e, requests.HTTPError) and e.response is not None:
                    if e.response.status_code in _TRANSIENT_HTTP_CODES:
                        is_transient = True
                elif isinstance(e, (requests.ConnectionError, requests.Timeout)):
                    is_transient = True

                if is_transient:
                    logger.warning(
                        "Transient error %s polling TF %s (attempt %d/%d)",
                        e, tool_input.request_id, attempt, max_attempts,
                    )
                    if attempt < max_attempts:
                        await asyncio.sleep(poll_interval)
                    continue
                raise ToolError(
                    f"Failed to get Testing Farm reservation details {tool_input.request_id}: {e}"
                ) from e

ymir/tools/privileged/testing_farm.py (424)

The requests.get call to fetch pipeline.log is synchronous and blocking. Wrap it in asyncio.to_thread to prevent blocking the event loop.

                        log_resp = await asyncio.to_thread(requests.get, f"{artifacts_url}/pipeline.log", timeout=30)

ymir/tools/privileged/testing_farm.py (487)

The _testing_farm_api_delete call is synchronous and blocking. Wrap it in asyncio.to_thread to avoid blocking the event loop.

            await asyncio.to_thread(_testing_farm_api_delete, f"requests/{request_id}")

ymir/tools/privileged/gitlab.py (505)

The shutil.rmtree call is synchronous and blocking. Removing a cloned git repository can take several seconds, which will block the event loop. Wrap it in asyncio.to_thread to run it asynchronously.

                await asyncio.to_thread(shutil.rmtree, clone_path)

ymir/agents/prompts/reproducer/prompt.j2 (300-302)

In BeakerLib, each rlRun invocation executes in a separate subshell. Assigning TmpDir=\$(mktemp -d) inside rlRun means the variable is lost immediately after that command exits. Subsequent commands like rlRun "pushd \$TmpDir" will fail because TmpDir is empty. Assign TmpDir in the parent shell context instead, and let it expand in the parent shell when passing to rlRun.

        TmpDir=\$(mktemp -d)
        ORIG_DIR=\$(pwd)
        rlRun "pushd \$TmpDir"

ymir/agents/prompts/reproducer/prompt.j2 (331)

Since TmpDir is now defined in the parent shell context, do not escape the $ in the rlRun call so that it expands in the parent shell before being executed.

        rlRun "rm -rf \$TmpDir" 0 "Removing tmp directory"

agents_as_skills/reproducer/SKILL.md (300-302)

In BeakerLib, each rlRun invocation executes in a separate subshell. Assigning TmpDir=\$(mktemp -d) inside rlRun means the variable is lost immediately after that command exits. Subsequent commands like rlRun "pushd \$TmpDir" will fail because TmpDir is empty. Assign TmpDir in the parent shell context instead, and let it expand in the parent shell when passing to rlRun.

        TmpDir=\$(mktemp -d)
        ORIG_DIR=\$(pwd)
        rlRun "pushd \$TmpDir"

agents_as_skills/reproducer/SKILL.md (331)

Since TmpDir is now defined in the parent shell context, do not escape the $ in the rlRun call so that it expands in the parent shell before being executed.

        rlRun "rm -rf \$TmpDir" 0 "Removing tmp directory"

ymir/agents/reproducer_agent.py (238)

Directly calling OutputSchema.model_validate_json on the raw LLM response text is fragile. If the LLM wraps the JSON in markdown code blocks (e.g., ```json ... ```), validation will fail. Clean the response text to robustly extract the JSON payload before validation.

            raw_text = response.last_message.text.strip()
            if raw_text.startswith("```"):
                lines = raw_text.splitlines()
                if lines[0].startswith("```"):
                    lines = lines[1:]
                if lines and lines[-1].startswith("```"):
                    lines = lines[:-1]
                raw_text = "\n".join(lines).strip()
            state.result = OutputSchema.model_validate_json(raw_text)

ymir/tools/privileged/testing_farm.py (571)

Define _ALLOWED_COPY_BASES as Path objects to enable more robust and idiomatic path validation using is_relative_to instead of string prefix matching.

_ALLOWED_COPY_BASES = (Path("/git-repos"), Path("/tmp"))

ymir/tools/privileged/testing_farm.py (580-587)

Use is_relative_to for path validation to be consistent with the rest of the codebase (e.g., gitlab.py) and to avoid fragile string-based prefix matching.

    @field_validator("local_paths")
    @classmethod
    def validate_local_paths(cls, v: list[str]) -> list[str]:
        for p in v:
            resolved = Path(p).resolve()
            if not any(resolved.is_relative_to(base) for base in _ALLOWED_COPY_BASES):
                raise ValueError(f"Path {p} is not under an allowed directory")
        return v

[vmihalko]

ad gemini findings:

1. _testing_farm_api_post blocking (line 346): Fixed, wrapped with asyncio.to_thread.
2. _testing_farm_api_get blocking + broader exception handling (lines 397-409): Fixed. Wrapped with asyncio.to_thread and broadened to requests.RequestException, treating ConnectionError and Timeout as transient alongside HTTP.
3. requests.get for pipeline.log blocking (line 424): Fixed, wrapped with asyncio.to_thread.
4. _testing_farm_api_delete blocking (line 487): Fixed, wrapped with asyncio.to_thread.
5. shutil.rmtree blocking in gitlab.py (line 505): Fixed, wrapped with asyncio.to_thread.
6. BeakerLib rlRun subshell in prompt.j2 (lines 300-302): False positive. rlRun uses eval in the current shell context, not a subshell. rlRun "TmpDir=$(mktemp -d)" is a standard BeakerLib pattern - the variable persists and is
   available to subsequent commands. All existing ksh tests use exactly rlRun "TmpDir=$(mktemp -d)"...
7. BeakerLib rlRun cleanup in prompt.j2 (line 331): Same as above, not a bug.
8. BeakerLib rlRun subshell in SKILL.md (lines 300-302): Same as 6, false positive.
9. BeakerLib rlRun cleanup in SKILL.md (line 331): Same as 7, not a bug.
10. JSON markdown code block stripping (reproducer_agent.py line 238): Not needed. The BeeAI framework's expected_output parameter already provides explicit JSON formatting instructions and schema. Adding defensive stripping would mask issues rather than fix them.
11. _ALLOWED_COPY_BASES as Path objects (line 571): Fixed.
12. is_relative_to for path validation (lines 580-587): Fixed.

[vmihalko]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new 'reproducer' agent to automate the creation, verification, and publication of minimal reproducers for RHEL bugs and CVEs. It adds the core reproducer agent workflow, prompt templates, Testing Farm reservation cleanup middleware, and several new Testing Farm tools. The review feedback highlights several key areas for robustness and performance improvements, including wrapping blocking synchronous calls in asyncio.to_thread, resolving allowed parent paths to absolute paths to prevent validation failures, defensively parsing LLM JSON responses that may contain markdown wrapping, and safely accessing event attributes in the cleanup middleware.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[TomasKorbar]

Alright team, this is as far i got before PTO.
Agent is now wired and is triggered after triage. It opens MR for the internal tests repository. Workflow is still pretty raw and there are clear spaces for optimization that can take off i think 20% of consumed tokens without too much of an effort, which means refining the tools to provide less output, instructing prompt about it and pulling testing farm reservation and tear down out of the agent run into the deterministic steps. Other (but perhaps much higher) savings could be achieved by dynamic context management eg. implementing possibility for the agent to forget things that it realizes it no longer needs. This could be particularly useful in this workflow as agent very often finds dead ends and changes its approach.

The rollout plan and merge blockers i propose are:

• Reproducer agent passes E2E test
• Reproducer agent passes dry run - verified on RHEL-213761
• Reproducer agent passes local run and opens MR - verified with https://gitlab.com/redhat/rhel/tests/bind/-/merge_requests/125
• [] Reproducer agent manages to create reproducer in prod
• [] Reproducer agent correctly handles multiple CVEs for single component in prod correctly handling conflicts
• [] Reproducer agent manages to verify and adapt test for different stream of RHEL.
• [] Full scale deployment

Now things that are not completely polished and stand out in my mind are:

1. I applied commit to workaround the MCP max message crash that started to occur recently and i think Nikola already did fix for that
2. I am not sure about correctness of the redis queues/race condition prevention (2 agents working on same test in different streams) and this particular section needs careful review by different team member.
3. I did not create openshift deployment configuration.

[jpodivin]

Please factor out memory management tool into a separate PR.

[TomasTomecek]

This is completely unreviewable 😅

Thank you for the design doc, that definitely helped me understand the overall architecture.

At this point we should just deploy it and try it on a few CVEs :)

[TomasTomecek]

really curious about this one for more complex components and networking services

but when the agent explores existing test cases, it should reuse the same patterns, let's see

[TomasTomecek]

I'm assuming the reservation is happening before this so we don't actively busy-wait for the reservation to complete right?

[TomasKorbar]

Agent first meditates on the issue description to find out which image from testing farm it should pick, then submits the reservation requests, starts writing the script and then waits for the VM ssh to come up. Fortunately even if there will be glitch and agent decided to wait directly after submitting reservation, thanks to optimizations from mvadkert it takes like 4 minutes for the machine to boot up. So not a lot of time wasted there, considering we spent 2 minutes in consolidation with sleeping because of the nfs issues :D

[TomasTomecek]

is there some timeout or expiration for the reservation? I can easily imagine for this long process to crash somewhere and the agent would never call cancel.

[TomasKorbar]

There is custom middleware that handles canceling of reservation if agent does not do it. So that is covered.

[TomasKorbar]

I was afraid of reservation hoarding too.

[TomasKorbar]

I'll try to carve out things that make standalone sense, but will keep here stuff that is only used and related to the reproducer agent, as they are only testable with it and it would be worse to review the separately.

[lbarcziova]

this is huge! Thanks a lot for being able to put this together.

First round of some question/notes, but agreed with Tomas, we will probably need to iterate based on real issues run in production.

[lbarcziova]

we need to check before deploying this, how much spare room we have, also after deploying #749

[lbarcziova]

this would be a good place to plug in potential env var controlling enabling/disabling for testing

[lbarcziova]

curious to see this behaviour in prod

[lbarcziova]

the path here is unique right? Asking if we can bump into some race condition in here

[TomasKorbar]

Good catch. Agents working on 2 reproducers for different CVEs would clash here. Fixed with simmilar pattern that we use in other agents.

[lbarcziova]

have you tested this with the 2 scenarios you mentioned? Have you bumped into any context issues? It seems quite low, so I'm wondering if we should go with a bit higher number for safety

[TomasKorbar]

Yep this is tested. I did not see any context issues and i have not seen agent to use the default in any run. It always kept last 3-5 rounds.