diff --git a/.github/workflows/manual-smoke.yml b/.github/workflows/manual-smoke.yml new file mode 100644 index 0000000..2f39ebd --- /dev/null +++ b/.github/workflows/manual-smoke.yml @@ -0,0 +1,61 @@ +name: Manual Compose smoke + +on: + workflow_dispatch: + inputs: + candidate: + description: Tracked repository-relative release candidate JSON + required: true + type: string + profile: + description: Supported Compose profile + required: true + type: choice + options: + - aml-online-shop-compose-v1 + inspection_only: + description: Explicitly inspect legacy-ineligible candidates + required: false + default: false + type: boolean + +permissions: + contents: read + +jobs: + smoke: + name: Manual Compose runtime smoke + runs-on: ubuntu-24.04 + timeout-minutes: 45 + env: + SMOKE_CANDIDATE: ${{ inputs.candidate }} + SMOKE_PROFILE: ${{ inputs.profile }} + SMOKE_INSPECTION_ONLY: ${{ toJSON(inputs.inspection_only) }} + SMOKE_COMMIT: ${{ github.sha }} + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + with: + ref: ${{ github.sha }} + fetch-depth: 0 + persist-credentials: false + - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6 + with: + python-version: '3.13' + - name: Build and install selected commit + run: | + python -m pip install build + python -m build --wheel + python -m pip install dist/*.whl + - name: Execute bounded smoke + run: python scripts/manual_smoke.py execute + - name: Finalize evidence and best-effort project cleanup + if: ${{ always() }} + run: python scripts/manual_smoke.py finalize + - name: Upload sanitized evidence + if: ${{ always() }} + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: compose-smoke-evidence + path: ${{ runner.temp }}/container-platform-smoke-evidence/report.json + retention-days: 7 + if-no-files-found: error diff --git a/README.md b/README.md index 750a0fc..00d032b 100644 --- a/README.md +++ b/README.md @@ -124,3 +124,5 @@ Add `--registry` to inspect immutable manifest/config bytes and declared platfor ## Optional runtime smoke [Bounded runtime smoke](docs/runtime-smoke.md) requires explicit `--execute`, consumes the rendered candidate snapshot and cleans up its unique Compose project. It checks service process state and a synthetic catalog/cart/checkout flow. Inspection-only baseline execution preserves legacy ineligibility. See the recorded live outcome and limitations; runtime success is separate from release approval and deployment authorization. + +[Manual hosted smoke](docs/manual-smoke-workflow.md) adds a dispatch-only workflow with sanitized evidence and best-effort scoped recovery. Its first hosted baseline run remains pending default-branch workflow registration and owner dispatch; it is not a mandatory PR check. diff --git a/docs/manual-smoke-workflow.md b/docs/manual-smoke-workflow.md new file mode 100644 index 0000000..9fcc3bc --- /dev/null +++ b/docs/manual-smoke-workflow.md @@ -0,0 +1,49 @@ +# Manual hosted Compose smoke + +`.github/workflows/manual-smoke.yml` supplies opt-in runtime evidence on an ephemeral `ubuntu-24.04` Linux/amd64 hosted runner. It has only `workflow_dispatch`, read-only repository permissions, pinned Actions, a 45-minute timeout and no cancellation concurrency policy. Existing PR checks are unchanged. No registry login, secrets, application image builds, publication or deployment are requested. + +## Inputs and tested revision + +- `candidate`: required repository-relative tracked regular file, for example `releases/baseline-2026-10-18.json`. Absolute/traversal paths, symlink files/ancestors, untracked files and content differing from the selected commit are rejected. +- `profile`: `aml-online-shop-compose-v1`, the only supported choice. +- `inspection_only`: boolean, default `false`. Helper parsing accepts only the exact strings `true` and `false` supplied through environment values. + +The workflow's native branch/tag selection chooses the workflow revision. There is no separate code-ref input: checkout pins `github.sha`, includes full Git history and does not persist credentials. The wheel, candidate snapshot, profile and schema come from this commit. The helper checks installed module/resource bytes against Git objects, verifies local inventory objects, then calls the installed [bounded runner](runtime-smoke.md) with explicit execution opt-in. Runtime orchestration and its budgets are not duplicated or weakened. Inputs are never interpolated into shell code. + +Ordinary mode requires successful offline validation. For the legacy baseline, explicit inspection-only is required. A successful runtime result then coexists with validation exit 1 and 12 `legacy_ineligible` failures; it does not grant approval or deployment authorization. + +## Evidence and failure behavior + +The sole uploaded artifact is `compose-smoke-evidence`, containing `report.json` at the predictable `container-platform-smoke-evidence/report.json` location under the runner temporary root. Retention is seven days. It includes tested full commit SHA, candidate path/byte SHA-256, profile/inspection mode, the sanitized runner result, original primary exit code and workflow cleanup verification. The job summary adds release ID, runtime/cleanup outcomes and offline eligibility. Candidate-controlled values are JSON-quoted and HTML-escaped; no raw values are printed as workflow commands. + +Only this report is uploaded, never the workspace, cleanup state, raw Docker logs, cookies or environment. Missing/malformed reports create explicit execution/report failure evidence and cannot yield success. Preparation failures report stable diagnostics; identity fields that could not be established remain unavailable. Runtime nonzero exit codes remain primary even when report processing or cleanup also fails. Upload failures fail the job; the report's primary code describes runtime, not artifact-service availability. Consult the job conclusion as well as the evidence. + +## Cancellation and cleanup + +The runner retains its normal interruption cleanup. A minimal pre-mutation hook saves the exact generated project identity and Compose bytes after successful prerequisites/name-collision checks. An `always()` workflow step checks only that identity and performs bounded project-scoped cleanup if resources remain, then verifies absence of its containers, networks and volumes. Identity, commit and Compose-byte hash must agree before cleanup. It does not search projects by prefix, prune globally or delete cached images. + +Best-effort recovery and upload cannot be guaranteed after forced cancellation, repeated interruption, job timeout, Engine failure or runner loss. Cancellation recovery has offline adapter coverage but has not been verified by a live cancellation experiment. Ephemeral-host disposal is separate from verified application cleanup. A successful ordinary run must contain successful runner cleanup plus verified workflow resource absence. + +## Hosted acceptance: pending registration + +As of 2026-10-23, this workflow is absent from the default branch's registered workflows. GitHub [requires the dispatch workflow to exist on the default branch](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_dispatch). No hosted runtime dispatch has been performed; prior local runtime evidence does not satisfy this criterion. No temporary PR/push runtime trigger or substitute workflow is used. + +After owner review and merge register the workflow on `main`: + +1. Dispatch the baseline with explicit inspection-only: + + ```text + gh workflow run manual-smoke.yml --repo DimitryZH/container-platform --ref main -f candidate=releases/baseline-2026-10-18.json -f profile=aml-online-shop-compose-v1 -f inspection_only=true + ``` + +2. Find and inspect the actual dispatch run: + + ```text + gh run list --repo DimitryZH/container-platform --workflow manual-smoke.yml --event workflow_dispatch --limit 5 + gh run view RUN_ID --repo DimitryZH/container-platform + gh run download RUN_ID --repo DimitryZH/container-platform --name compose-smoke-evidence + ``` + +3. Confirm the intended tested SHA, candidate hash and exact images, all 12 service records, semantic assertions, runtime exit 0, successful runner cleanup, verified workflow resource absence and retained validation exit 1 with 12 legacy failures. Confirm artifact and summary availability. Record a concise hosted observation tied to the run URL; keep issue #15 open until this evidence is reviewed. + +No hosted success, cancellation recovery or milestone completion is claimed before those checks occur. diff --git a/docs/roadmap.md b/docs/roadmap.md index 192824a..3f7732b 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -89,6 +89,8 @@ Invalid, incomplete, mutable, or unavailable releases cannot be presented as val [Pull request checks](pull-request-validation.md) provide tests/package and new-candidate validation. [Offline Compose rendering](compose-rendering.md) provides deterministic configuration from digest-pinned candidates. An [opt-in bounded runtime runner](runtime-smoke.md) is available; its live result and limitations are recorded separately. Required-check settings are owner-configured; runner availability does not complete this milestone or prove baseline runtime compatibility. +A [manual hosted workflow](manual-smoke-workflow.md) is prepared; hosted baseline acceptance remains pending workflow registration on the default branch and an actual owner dispatch. It adds no mandatory runtime PR check and does not close this milestone. + ### Scope - add GitHub Actions checks for release-manifest pull requests; diff --git a/scripts/manual_smoke.py b/scripts/manual_smoke.py new file mode 100644 index 0000000..7e66830 --- /dev/null +++ b/scripts/manual_smoke.py @@ -0,0 +1,330 @@ +"""Manual workflow input binding, evidence and exact-project recovery. + +Runtime orchestration is supplied exclusively by the installed smoke runner. +Only the final sanitized report is uploaded; state remains private to the job. +""" + +import hashlib +import html +from importlib.resources import files +import json +import math +import os +from pathlib import Path, PurePosixPath +import re +import signal +import stat +import sys +import time + + +PROFILE = "aml-online-shop-compose-v1" +PROJECT = re.compile(r"cp-smoke-[0-9a-f]{32}", re.ASCII) + + +class InputError(Exception): + pass + + +def boolean(value): + if value not in ("true", "false"): + raise InputError("inspection_boolean_invalid") + return value == "true" + + +def json_read(path): + def pairs(items): + result = {} + for key, value in items: + if key in result: + raise InputError("report_invalid") + result[key] = value + return result + def constant(_): + raise InputError("report_invalid") + def finite(value): + result = float(value) + if not math.isfinite(result): + raise InputError("report_invalid") + return result + try: + with path.open("rb") as stream: + data = stream.read(2 * 1024 * 1024 + 1) + if len(data) > 2 * 1024 * 1024: + raise InputError("report_invalid") + return json.loads(data, object_pairs_hook=pairs, + parse_constant=constant, parse_float=finite) + except (OSError, ValueError, UnicodeError, RecursionError): + raise InputError("report_missing_or_invalid") from None + + +def write_json(path, value): + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(".tmp") + temporary.write_text(json.dumps(value, indent=2, sort_keys=True, ensure_ascii=True, allow_nan=False) + "\n", encoding="utf-8") + temporary.replace(path) + + +def git(repo, *args): + from container_platform.validation import Failure, git_read + try: + result = git_read(repo, *args) + except Failure: + raise InputError("selected_git_objects_unavailable") from None + if result.returncode: + raise InputError("selected_git_objects_unavailable") + return result.stdout + + +def selected_file(repo, commit, path): + records = git(repo, "ls-tree", "-z", commit, "--", path).split(b"\0") + if len(records) != 2 or not records[0]: + raise InputError("candidate_not_tracked_regular_file") + meta, name = records[0].split(b"\t", 1) + mode, kind, oid = meta.split() + if mode not in (b"100644", b"100755") or kind != b"blob" or name != path.encode("utf-8"): + raise InputError("candidate_not_tracked_regular_file") + return git(repo, "cat-file", "blob", oid.decode("ascii")) + + +def validate_path(repo, value, commit): + if (not isinstance(value, str) or not value or "\\" in value or ":" in value + or any(ord(c) < 32 or ord(c) == 127 for c in value)): + raise InputError("candidate_path_invalid") + relative = PurePosixPath(value) + if relative.is_absolute() or ".." in relative.parts or relative.as_posix() != value or value.startswith("-"): + raise InputError("candidate_path_invalid") + current = repo + for part in relative.parts: + current = current / part + if current.is_symlink(): + raise InputError("candidate_symlink_disallowed") + if not current.is_file() or not stat.S_ISREG(current.stat().st_mode) or not current.resolve().is_relative_to(repo.resolve()): + raise InputError("candidate_path_invalid") + with current.open("rb") as stream: + data = stream.read(1024 * 1024 + 1) + if len(data) > 1024 * 1024 or data != selected_file(repo, commit, value): + raise InputError("candidate_differs_from_selected_commit") + return current, data + + +def verify_install(repo, commit): + # Check every shipped module and both resources, not only the CLI wrapper. + for source in (repo / "src/container_platform").rglob("*.py"): + expected = selected_file(repo, commit, source.relative_to(repo).as_posix()) + installed = files("container_platform").joinpath(*source.relative_to(repo / "src/container_platform").parts) + if installed.read_bytes().replace(b"\r\n", b"\n") != expected: + raise InputError("installed_code_mismatch") + for package, name, path in [ + ("container_platform.schemas", "release-v1.schema.json", "schemas/release-v1.schema.json"), + ("container_platform.profiles", PROFILE + ".json", "src/container_platform/profiles/" + PROFILE + ".json"), + ]: + if files(package).joinpath(name).read_bytes().replace(b"\r\n", b"\n") != selected_file(repo, commit, path): + raise InputError("installed_resource_mismatch") + + +def prepare(repo, env): + commit = env.get("SMOKE_COMMIT", "") + if not re.fullmatch(r"[0-9a-f]{40}", commit) or git(repo, "rev-parse", "HEAD").decode().strip() != commit: + raise InputError("workflow_commit_mismatch") + if env.get("SMOKE_PROFILE") != PROFILE: + raise InputError("profile_unsupported") + inspection = boolean(env.get("SMOKE_INSPECTION_ONLY")) + path, data = validate_path(repo, env.get("SMOKE_CANDIDATE"), commit) + verify_install(repo, commit) + from container_platform.validation import strict_json, resolve_inventory, Failure + try: + candidate = strict_json(data) + # Full schema/policy/eligibility stays in the runner; establish local objects now. + if not isinstance(candidate, dict) or not isinstance(candidate.get("inventory"), dict): + raise InputError("candidate_inventory_invalid") + resolve_inventory(repo, candidate["inventory"]) + except (Failure, KeyError, TypeError): + raise InputError("candidate_inventory_unavailable") from None + return path, {"testedCommit": commit, "candidatePath": path.relative_to(repo).as_posix(), + "candidateSha256": hashlib.sha256(data).hexdigest(), "profile": PROFILE, + "inspectionOnly": inspection} + + +def save_project(state, metadata, project, directory): + if not PROJECT.fullmatch(project): + raise OSError("Invalid generated project identity") + data = (directory / "compose.json").read_bytes() + (state / "compose.json").write_bytes(data) + (state / "empty.env").write_bytes(b"") + write_json(state / "project.json", {"project": project, "testedCommit": metadata["testedCommit"], + "composeSha256": hashlib.sha256(data).hexdigest()}) + + +def execute(repo, state, env, runner=None): + state.mkdir(parents=True, exist_ok=True) + metadata = {} + code = 2 + primary = None + try: + path, metadata = prepare(repo, env) + write_json(state / "metadata.json", metadata) + snapshot = state / "candidate.json" + snapshot.write_bytes(selected_file(repo, metadata["testedCommit"], metadata["candidatePath"])) + if runner is None: + from container_platform.smoke import run + runner = run + result = runner(snapshot, repo, PROFILE, execute=True, inspection_only=metadata["inspectionOnly"], + project_ready=lambda project, directory: save_project(state, metadata, project, directory)) + code = result["exitCode"] + if type(code) is not int or code not in (0, 1, 2, 3, 130): + code = 2 + raise InputError("runtime_exit_invalid") + primary = code + write_json(state / "runtime.json", result) + except InputError as error: + write_json(state / "error.json", {"reason": str(error)}) + except KeyboardInterrupt: + code = 130 + write_json(state / "error.json", {"reason": "workflow_interrupted"}) + except (OSError, ValueError, KeyError, ImportError): + code = code if code in (1, 2, 3, 130) else 2 + write_json(state / "error.json", {"reason": "workflow_execution_failed"}) + finally: + write_json(state / "exit.json", {"primaryExitCode": primary if primary is not None else code}) + return code + + +def cleanup(state, metadata): + if not (state / "project.json").exists(): + return {"outcome": "not_required"} + from container_platform.smoke import checked, resources + from container_platform.smoke_transport import Docker + identity = json_read(state / "project.json") + project = identity.get("project", "") + if (not isinstance(project, str) or not PROJECT.fullmatch(project) + or identity.get("testedCommit") != metadata.get("testedCommit") + or hashlib.sha256((state / "compose.json").read_bytes()).hexdigest() != identity.get("composeSha256")): + raise InputError("cleanup_identity_invalid") + deadline = time.monotonic() + 120 + docker = Docker(project, state) + if any(resources(docker, deadline).values()): + checked(docker.compose(["down", "--volumes", "--remove-orphans", "--timeout", "10"], deadline), "workflow_cleanup_failed") + if any(resources(docker, deadline).values()): + raise InputError("workflow_resources_remaining") + return {"outcome": "verified_absent", "project": project} + + +def runtime_report(state, metadata, primary): + result = json_read(state / "runtime.json") + if (not isinstance(result, dict) or result.get("smokeReportVersion") != 1 + or type(result.get("exitCode")) is not int or result["exitCode"] != primary + or result.get("inspectionOnly") is not metadata.get("inspectionOnly") + or result.get("profile") != metadata.get("profile") + or not isinstance(result.get("cleanup"), dict) + or not isinstance(result.get("checks"), list) + or not isinstance(result.get("diagnostics"), list) + or not isinstance(result.get("runtimeOutcome"), str)): + raise InputError("report_contract_invalid") + candidate = result.get("candidate") + if candidate is not None and (not isinstance(candidate, dict) or candidate.get("sha256") != metadata.get("candidateSha256")): + raise InputError("report_candidate_mismatch") + if primary == 0: + validation = result.get("validation", {}) + from container_platform.compose import IDS + services = result.get("services", []) + assertions = result.get("smokeAssertions", []) + if (candidate is None or not isinstance(validation, dict) + or result.get("runtimeOutcome") != "success" or result.get("cleanup", {}).get("outcome") != "success" + or not isinstance(services, list) or len(services) != len(IDS) + or {s.get("service") for s in services} != IDS or any(s.get("state") != "running" for s in services) + or assertions != ["frontend_catalog_pass", "product_browse_pass", "cart_add_pass", "cart_remove_pass", "synthetic_checkout_pass"]): + raise InputError("report_success_invalid") + if validation.get("exitCode") != 0: + checks = validation.get("checks", []) + if (not metadata["inspectionOnly"] or validation.get("exitCode") != 1 or not checks + or any(c.get("status") != "pass" for c in checks[:-1]) + or checks[-1].get("id") != "candidate.traceability" or checks[-1].get("status") != "fail" + or not checks[-1].get("diagnostics") + or any(d.get("code") != "legacy_ineligible" for d in checks[-1]["diagnostics"])): + raise InputError("report_eligibility_inconsistent") + # The runner already sanitizes diagnostic values. Drop manual commands and any + # unknown top-level payload before artifact upload; never upload private state. + allowed = ("smokeReportVersion", "runtimeOutcome", "exitCode", "inspectionOnly", "profile", "candidate", + "validation", "checks", "services", "smokeAssertions", "cleanup", "diagnostics", "elapsedSeconds", "scope") + return {key: result[key] for key in allowed if key in result} + + +def summary(evidence, path): + runtime = evidence.get("runtime", {}) + candidate = runtime.get("candidate") or {} + validation = runtime.get("validation") or {} + fields = {"Tested commit": evidence.get("testedCommit"), "Candidate": evidence.get("candidatePath"), + "Candidate byte SHA-256": evidence.get("candidateSha256"), "Release ID": candidate.get("releaseId"), + "Profile": evidence.get("profile"), "Inspection only": evidence.get("inspectionOnly"), + "Runtime": runtime.get("runtimeOutcome", "execution/report failure"), + "Runner cleanup": runtime.get("cleanup", {}).get("outcome", "unknown"), + "Workflow cleanup": evidence["workflowCleanup"]["outcome"], + "Offline eligibility": validation.get("result", "not evaluated"), + "Offline validation exit": validation.get("exitCode"), "Evidence": evidence["reportStatus"]} + rows = ["" + key + "" + html.escape(json.dumps(value, ensure_ascii=True)) + "" for key, value in fields.items()] + path.write_text("### Manual Compose smoke\n\n" + "".join(rows) + "
\n\nNo release approval or deployment authorization.\n", encoding="utf-8") + + +def finalize(state, output, summary_path): + evidence = {"workflowEvidenceVersion": 1, "reportStatus": "execution/report failure", "runtime": {}, + "primaryExitCode": None, "workflowExitCode": 2, "workflowCleanup": {"outcome": "not_required"}} + primary = 2 + try: + metadata = json_read(state / "metadata.json") + if not isinstance(metadata, dict): + raise InputError("metadata_invalid") + metadata = {key: metadata[key] for key in ("testedCommit", "candidatePath", "candidateSha256", "profile", "inspectionOnly") if key in metadata} + evidence.update(metadata) + except InputError: + commit = os.environ.get("SMOKE_COMMIT", "") + metadata = {"testedCommit": commit if re.fullmatch(r"[0-9a-f]{40}", commit) else None} + evidence.update(metadata) + try: + primary = json_read(state / "exit.json")["primaryExitCode"] + if type(primary) is not int or primary not in (0, 1, 2, 3, 130): + raise InputError("runtime_exit_invalid") + evidence["primaryExitCode"] = primary + evidence["runtime"] = runtime_report(state, metadata, primary) + evidence.update(reportStatus="valid", workflowExitCode=primary) + except (InputError, KeyError, TypeError, AttributeError): + evidence["workflowExitCode"] = primary if primary in (1, 2, 3, 130) else 2 + try: + reason = json_read(state / "error.json").get("reason", "") + if isinstance(reason, str) and re.fullmatch(r"[a-z_]+", reason): + evidence["workflowDiagnostic"] = reason + evidence["reportStatus"] = "execution/report failure" + if evidence["workflowExitCode"] == 0: + evidence["workflowExitCode"] = 2 + except (InputError, AttributeError): + pass + try: + evidence["workflowCleanup"] = cleanup(state, metadata) + except Exception: + evidence["workflowCleanup"] = {"outcome": "failed", "reason": "workflow_cleanup_unverified"} + if evidence["workflowExitCode"] == 0: + evidence["workflowExitCode"] = 2 + write_json(output, evidence) + summary(evidence, summary_path) + return evidence["workflowExitCode"] + + +def main(): + root = Path(os.environ["RUNNER_TEMP"]) + state = root / "container-platform-smoke-state" + previous = signal.getsignal(signal.SIGTERM) + def interrupted(*_): + raise KeyboardInterrupt() + try: + signal.signal(signal.SIGTERM, interrupted) + if sys.argv[1:] == ["execute"]: + return execute(Path.cwd(), state, os.environ) + if sys.argv[1:] == ["finalize"]: + return finalize(state, root / "container-platform-smoke-evidence/report.json", Path(os.environ["GITHUB_STEP_SUMMARY"])) + return 2 + finally: + signal.signal(signal.SIGTERM, previous) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/container_platform/smoke.py b/src/container_platform/smoke.py index 93949c4..30fe558 100644 --- a/src/container_platform/smoke.py +++ b/src/container_platform/smoke.py @@ -163,7 +163,8 @@ def get(path, form=None): def run(candidate_path, repository, profile, *, execute=False, inspection_only=False, - docker_factory=Docker, session_factory=Session, clock=time.monotonic, sleep=time.sleep): + docker_factory=Docker, session_factory=Session, clock=time.monotonic, sleep=time.sleep, + project_ready=None): start = clock() total_deadline = start + LIMITS["total"] work_deadline = total_deadline - LIMITS["cleanup"] - LIMITS["diagnostics"] @@ -225,6 +226,10 @@ def stage(name, operation): (directory / "empty.env").write_bytes(b"") docker = docker_factory(project, directory, clock) stage("prerequisites", lambda deadline: prerequisites(docker, images, deadline)) + if project_ready is not None: + # Persist exact cleanup identity only after confirming no name collision, + # and before any image pull or startup can be interrupted. + project_ready(project, directory) stage("pull", lambda deadline: checked(docker.compose(["pull", "--policy", "always"], deadline), "image_pull_failed")) mutated = True # A failed/interrupted up can already have created resources. stage("startup", lambda deadline: checked(docker.compose(["up", "--detach", "--no-build", "--pull", "never"], deadline), "compose_startup_failed")) diff --git a/tests/test_manual_smoke.py b/tests/test_manual_smoke.py new file mode 100644 index 0000000..5b4e7b0 --- /dev/null +++ b/tests/test_manual_smoke.py @@ -0,0 +1,201 @@ +"""Manual workflow helper tests use local Git and fake runtime/cleanup adapters.""" + +import copy +import hashlib +import importlib.util +import json +import os +from pathlib import Path +import unittest +from unittest.mock import patch + +import test_compose as fixtures +from container_platform.compose import IDS + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location("manual_smoke", ROOT / "scripts/manual_smoke.py") +helper = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(helper) + + +class ManualSmokeTests(unittest.TestCase): + def setUp(self): + self.fixture = fixtures.ComposeTests() + self.fixture.setUp() + self.addCleanup(self.fixture.doCleanups) + self.repo = self.fixture.repo + self.path = "releases/candidate.json" + self.candidate = self.fixture.candidate + self.commit_candidate() + self.state = Path(self.fixture.fixture.temp.name) / "state" + self.state.mkdir() + self.output = self.state.parent / "evidence/report.json" + self.summary = self.state.parent / "summary.md" + self.install = patch.object(helper, "verify_install") + self.install.start() + self.addCleanup(self.install.stop) + + def commit_candidate(self): + path = self.repo / self.path + path.parent.mkdir(exist_ok=True) + path.write_bytes(json.dumps(self.candidate).encode()) + git = self.fixture.fixture.git + git("add", "-A") + git("commit", "-qm", "test: add candidate") + self.commit = git("rev-parse", "HEAD").decode().strip() + self.env = {"SMOKE_COMMIT": self.commit, "SMOKE_PROFILE": helper.PROFILE, + "SMOKE_CANDIDATE": self.path, "SMOKE_INSPECTION_ONLY": "true"} + + def report(self, code=0): + validation = self.fixture.render()["validation"] + return {"smokeReportVersion": 1, "exitCode": code, "inspectionOnly": True, "profile": helper.PROFILE, + "candidate": {"releaseId": self.candidate["releaseId"], "sha256": hashlib.sha256((self.repo / self.path).read_bytes()).hexdigest()}, + "validation": validation, "cleanup": {"outcome": "success"}, + "runtimeOutcome": "success" if code == 0 else "application_failure", + "checks": [], "diagnostics": [], "services": [{"service": name, "state": "running"} for name in sorted(IDS)], + "smokeAssertions": ["frontend_catalog_pass", "product_browse_pass", "cart_add_pass", "cart_remove_pass", "synthetic_checkout_pass"], + "unknown_private_payload": "must not be uploaded"} + + def write_state(self, report, primary=None): + _, metadata = helper.prepare(self.repo, self.env) + helper.write_json(self.state / "metadata.json", metadata) + helper.write_json(self.state / "runtime.json", report) + helper.write_json(self.state / "exit.json", {"primaryExitCode": report["exitCode"] if primary is None else primary}) + + def test_strict_boolean_profile_and_commit(self): + self.assertFalse(helper.boolean("false")) + self.assertTrue(helper.boolean("true")) + for value in [None, True, False, "True", "FALSE", "1", " false", "true\n"]: + with self.assertRaises(helper.InputError): + helper.boolean(value) + for field, value in [("SMOKE_PROFILE", "other"), ("SMOKE_COMMIT", "main"), ("SMOKE_COMMIT", "f" * 40)]: + env = {**self.env, field: value} + with self.assertRaises(helper.InputError): + helper.prepare(self.repo, env) + + def test_paths_are_tracked_regular_contained_and_unchanged(self): + for value in ["/absolute.json", "C:/absolute.json", "../outside.json", "releases/../../outside.json", "releases\\candidate.json", "releases/./candidate.json", "", "-x.json", "releases/candidate.json\n"]: + with self.subTest(value=value), self.assertRaises(helper.InputError): + helper.validate_path(self.repo, value, self.commit) + untracked = self.repo / "untracked.json" + untracked.write_bytes(b"{}") + with self.assertRaises(helper.InputError): + helper.validate_path(self.repo, "untracked.json", self.commit) + (self.repo / self.path).write_bytes(b"changed") + with self.assertRaises(helper.InputError): + helper.validate_path(self.repo, self.path, self.commit) + + def test_symlink_input_is_rejected(self): + link = self.repo / "link.json" + try: + link.symlink_to(self.repo / self.path) + except OSError: + self.skipTest("Symlink creation is unavailable") + with self.assertRaises(helper.InputError): + helper.validate_path(self.repo, "link.json", self.commit) + + def test_missing_inventory_objects_fail_before_runtime(self): + self.candidate["inventory"]["revision"] = "f" * 40 + self.commit_candidate() + called = [] + code = helper.execute(self.repo, self.state, self.env, runner=lambda *a, **k: called.append(a)) + self.assertEqual(code, 2) + self.assertEqual(called, []) + + def test_filename_is_data_and_runner_receives_committed_snapshot(self): + self.path = "releases/space $(touch forbidden);.json" + self.commit_candidate() + result = self.report() + calls = [] + def runner(path, repo, profile, **kwargs): + calls.append((path, kwargs)) + self.assertEqual(path.read_bytes(), helper.selected_file(self.repo, self.commit, self.path)) + self.assertNotEqual(path, self.repo / self.path) + return result + self.assertEqual(helper.execute(self.repo, self.state, self.env, runner), 0) + self.assertTrue(calls[0][1]["execute"]) + self.assertTrue(calls[0][1]["inspection_only"]) + self.assertFalse((self.repo / "forbidden").exists()) + + def test_primary_nonzero_codes_survive_report_and_cleanup_failures(self): + for code in (1, 2, 3, 130): + with self.subTest(code=code): + result = self.report(code) + self.assertEqual(helper.execute(self.repo, self.state, self.env, lambda *a, **k: result), code) + with patch.object(helper, "cleanup", side_effect=helper.InputError("cleanup_failed")): + self.assertEqual(helper.finalize(self.state, self.output, self.summary), code) + self.assertEqual(json.loads(self.output.read_bytes())["primaryExitCode"], code) + + def test_report_write_error_preserves_original_runtime_code(self): + original = helper.write_json + def write(path, value): + if path.name == "runtime.json": + raise OSError("Simulated report write failure") + return original(path, value) + for primary in (0, 3): + result = self.report(primary) + with patch.object(helper, "write_json", side_effect=write): + code = helper.execute(self.repo, self.state, self.env, lambda *a, **k: result) + self.assertEqual(code, primary if primary else 2) + self.assertEqual(helper.json_read(self.state / "exit.json")["primaryExitCode"], primary) + self.assertEqual(helper.finalize(self.state, self.output, self.summary), primary if primary else 2) + + def test_missing_malformed_and_inconsistent_report_fail_closed(self): + helper.write_json(self.state / "exit.json", {"primaryExitCode": 0}) + for data in [None, b"{", b'{"exitCode":0,"exitCode":1}', b'{"elapsedSeconds":1e999}', b'{}']: + with self.subTest(data=data): + if data is not None: + (self.state / "runtime.json").write_bytes(data) + self.assertEqual(helper.finalize(self.state, self.output, self.summary), 2) + self.assertIn("execution/report failure", self.summary.read_text()) + result = self.report() + self.write_state(result) + result["candidate"]["sha256"] = "f" * 64 + helper.write_json(self.state / "runtime.json", result) + self.assertEqual(helper.finalize(self.state, self.output, self.summary), 2) + + def test_success_cannot_hide_eligibility_or_cleanup_failure(self): + result = self.report() + self.write_state(result) + self.assertEqual(helper.finalize(self.state, self.output, self.summary), 0) + for mutate in [lambda r: r.update(inspectionOnly=False), + lambda r: r["cleanup"].update(outcome="failed"), + lambda r: r["services"].pop(), + lambda r: r["validation"]["checks"][-1]["diagnostics"][0].update(code="other_failure")]: + changed = copy.deepcopy(result) + mutate(changed) + self.write_state(changed) + self.assertEqual(helper.finalize(self.state, self.output, self.summary), 2) + + def test_summary_escapes_candidate_markup_and_artifact_drops_unknown_payload(self): + result = self.report() + result["candidate"]["releaseId"] = ' | `text`\n::warning::bad' + self.write_state(result) + self.assertEqual(helper.finalize(self.state, self.output, self.summary), 0) + text = self.summary.read_text() + self.assertNotIn("