Skip to content

Portable multi-step actions: run a step program in the engine, not through cmake -P or a shell #955

Open

Description

@Sunrisepeak

Motivation

An action runs one program with one argument vector; no shell stands between the engine and the program (ninja starts it with CreateProcess on Windows, and mcpp __action when the action sets env, cwd or a stamp). Packaging steps routinely need more than one command, or a file set that exists only when the step runs:

  • a compiler whose outputs no plan can list (javac, kotlinc), followed by a stamp that stands for them;
  • a tool that takes no directory (d8), so the inputs are listed when the step runs, after stale outputs are removed;
  • copy an archive, then add files to it whose names are known only after an earlier step (classes*.dex);
  • empty a staging directory, copy a bundle into it, create a symbolic link.

Plugins work around the gap in three ways today, none of them general:

  • #!/bin/sh scripts as the action's program (mcpp-plugins dist-apple, and dist-apk before 0.24.0), which a Windows host cannot start;
  • a helper on a tool the plugin already needs (dist-apk 0.24.0 ran a Java single-file program on the JDK's java);
  • cmake -P over a script the plugin writes (deps-cmake, deps-archive), which costs every such build an xim:cmake payload even when the project uses no CMake.

mcpp-plugins 0.25.0 (mcpp-community/mcpp-plugins#59) collects these into one member, tools-steps: a plugin describes a step as a list of portable operations, and a backend runs it. The only backend there is cmake -P, which is the workaround this issue asks the engine to make unnecessary (mcpp-plugins rule 8: engine gaps are reported, not patched).

Proposal

Let an action carry a step program that the engine runs itself, in the process that already wraps actions (mcpp __action), with no external tool:

operation meaning
run(argv, cwd?, append_list?) run one program; a non-zero status fails the action; its output is the action's
touch(file) create or update a file, creating its directory
mkdir(dir), remove(path) create a directory; remove a file or a tree (absent is not an error)
remove_glob(pattern) remove the files a pattern matches
copy_file(from, to), copy_tree(from, to), copy_glob(pattern, dir), rename(from, to), symlink(target, link) file operations with the host's semantics
glob(list, pattern, recursive?, relative_to?) append the matching files, sorted, to a named list
require(list, message) fail the action with message when the list is empty

append_list passes a list's values as further arguments of run. Every value is taken verbatim (no quoting, no splitting), so a path keeps its spaces and semicolons on every host.

The step program is part of the action's command for the purposes of its fingerprint, so a changed program runs the action again, as a changed argument vector does today.

What changes for plugins

mcpp.tools.steps keeps its interface and switches its backend from cmake -P to the engine when the engine's protocol offers this, so no member changes and xim:cmake is no longer asked for by members that run steps. Members whose tool is CMake itself (deps-cmake) are unaffected.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions