Skip to content

Use contentctl-ng test plan/run in unit-testing workflow - #4346

Draft
pyth0n1c wants to merge 3 commits into
developfrom
update-unit-testing-test-plan-run
Draft

pyth0n1c wants to merge 3 commits into
developfrom
update-unit-testing-test-plan-run

Conversation

@pyth0n1c

@pyth0n1c pyth0n1c commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

DRAFT / DO NOT MERGE AS-IS. Two commits are TEMPORARY, exist only to test this workflow, and must both be reverted before taking this PR out of draft:

  1. 22449d9a38 ("TEMPORARY: touch two 7zip detections...") adds a dummy reference URL to 7zip_commandline_to_smb_share_path.yml and anomalous_usage_of_7zip.yml so the CHANGED-mode test plan selects detections.
  2. 363e9630ad ("TEMPORARY: hardcode contentctl-ng version...") hardcodes pip install contentctl-ng==1.2.0b1 in the "Install contentctl-ng" step of unit-testing.yml. That step runs on the base-branch (develop) checkout, whose requirements.txt still pins 1.1.1, so this PR's requirements.txt bump is not read. Restore pip install -r requirements.txt (the requirements.txt pin itself stays).

Summary

  • Split the single contentctl-ng test step in unit-testing.yml into two commands:
    • contentctl-ng test plan --test-type detection.unit --mode CHANGED --git-ref <base_ref>
    • contentctl-ng test run --post-test-behavior NEVER_PAUSE
  • test plan does not connect to Splunk, so it now runs before the slow Splunk provisioning step to fail fast on planning errors. (--post-test-behavior is a test run option; test plan rejects it.)
  • Start Splunk with contentctl-ng install --provision instead of the hand-rolled docker run + sleep 180. It waits until the container is healthy.
  • Removed --verbose. The DEBUG log (contentctl_ng_*.log) is always written to the working directory.
  • Upload everything produced as the test_summary_results artifact: testing_artifacts/ (plan, results YAML, JUnit XML), contentctl_ng_*.log, and dist/*.tar.gz. Upload still runs when tests fail.
  • Rewrote format_test_results.py to read the newest testing_artifacts/test_results_*.yml (the old test_results/summary.yml no longer exists).
  • Added testing_artifacts/ to .gitignore.
  • Pinned contentctl-ng==1.2.0b1 in requirements.txt, the first version with test plan/test run.

Notes

  • contentctl-ng 1.2.0b1 must be published for pip install -r requirements.txt to succeed in CI.
  • --provision uses Docker Compose, so the runner needs docker compose. Worth confirming on large-ubuntu-22.04-32core with this run.
  • The old "cat summary.yml" step was removed; the formatter's summary table and the uploaded artifact replace it.

Test plan

  • Workflow runs through plan, install --provision, and run on this PR
  • The two touched 7zip detections appear in the test plan and are tested
  • test_summary_results artifact contains testing_artifacts/, contentctl_ng_*.log files, and the dist tarball
  • Job summary renders the results table from format_test_results.py
  • Revert both TEMPORARY commits (22449d9a38, 363e9630ad), then mark ready for review

🤖 Generated with Claude Code

pyth0n1c and others added 2 commits October 15, 2026 15:51
Split the single `contentctl-ng test` step into `test plan` (run before
Splunk is provisioned so planning errors fail fast) and `test run`.
Provision Splunk via `install --provision` instead of a hand-rolled
`docker run` + sleep, and upload testing_artifacts/, contentctl-ng logs,
and the built package as job artifacts.

Rewrite format_test_results.py to read the new
testing_artifacts/test_results_*.yml format, ignore testing_artifacts/,
and pin contentctl-ng to 1.2.0b1, the first version with test plan/run.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Adds a dummy reference URL to two detections so the CHANGED-mode test
plan selects them. For testing the updated workflow only; revert this
commit before taking the PR out of draft.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The install step runs against the base-branch checkout, whose
requirements.txt still pins 1.1.1, so the PR's pin is never read.
Hardcode 1.2.0b1 so this PR's run uses the new test plan/run commands.
Revert this commit before merging.

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant