Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
843 changes: 785 additions & 58 deletions .github/workflows/build.yaml

Large diffs are not rendered by default.

16 changes: 15 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ GitHub Actions builds every image independently and runs the image tests and str
defined for it before pushing it to Docker Hub. The CI also uses `ghcr.io/webdevops/`, but only
internally; pull the images from Docker Hub.

The build steps pass the token of the run (`GITHUB_TOKEN`) to Docker as the build secret `COMPOSER_AUTH`, for PIE (see [PHP extension builds](#php-extension-builds)).
The token may only read the repository; only the jobs that push images to ghcr.io may also write packages there.

## Building and testing locally

Build a single image from its directory. Its parent image (the `FROM` line) and `webdevops/toolbox` are taken from your local images or pulled from Docker Hub:
Expand Down Expand Up @@ -116,11 +119,22 @@ In the CI, the job `image-test-runner` builds it once per architecture into the

## PHP extension builds

PHP images require Docker BuildKit. The official `ghcr.io/php/pie:bin` image
PHP images require Docker BuildKit 0.16 or later. The official `ghcr.io/php/pie:bin` image
supplies PIE through a build-only mount; the existing INI configuration still
controls extension loading. PIE requests remain unpinned, and compilation uses
`nproc` parallel jobs.

PIE resolves and downloads the extensions through the GitHub API. The `RUN` instructions with PIE mount the build
secret `COMPOSER_AUTH` as an environment variable, which Composer, and with it PIE, reads. Without the secret, PIE
sends its requests without a token, and GitHub limits them per IP address. To build with your token:

```
export COMPOSER_AUTH='{"github-oauth":{"github.com":"<token>"}}'
docker build --secret id=COMPOSER_AUTH,env=COMPOSER_AUTH -t webdevops/php:8.5 docker/php/8.5
```

`./build-local.sh` passes the secret on when `COMPOSER_AUTH` is set.

Protobuf, Vips and XML-RPC are built directly from their official upstream sources.
The existing XML-RPC **1.0.0RC3** pin is preserved on all supported PHP versions.
Protobuf on PHP 8.2–8.5 uses the latest upstream release. If a future release raises its minimum PHP
Expand Down
30 changes: 18 additions & 12 deletions build-local.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,30 @@ if [ "$#" -gt 0 ]; then
fi
fi

# PIE takes a GitHub token from COMPOSER_AUTH, see "PHP extension builds" in CONTRIBUTING.md
SECRET=""
if [[ -n "${COMPOSER_AUTH:-}" ]]; then
SECRET="--secret id=COMPOSER_AUTH,env=COMPOSER_AUTH"
fi

# alpine
docker build -t webdevops/php:$PHP_VERSION-alpine docker/php/$PHP_VERSION-alpine
docker build -t webdevops/php-dev:$PHP_VERSION-alpine docker/php-dev/$PHP_VERSION-alpine
docker build $SECRET -t webdevops/php:$PHP_VERSION-alpine docker/php/$PHP_VERSION-alpine
docker build $SECRET -t webdevops/php-dev:$PHP_VERSION-alpine docker/php-dev/$PHP_VERSION-alpine

docker build -t webdevops/php-nginx:$PHP_VERSION-alpine docker/php-nginx/$PHP_VERSION-alpine
docker build -t webdevops/php-nginx-dev:$PHP_VERSION-alpine docker/php-nginx-dev/$PHP_VERSION-alpine
docker build $SECRET -t webdevops/php-nginx:$PHP_VERSION-alpine docker/php-nginx/$PHP_VERSION-alpine
docker build $SECRET -t webdevops/php-nginx-dev:$PHP_VERSION-alpine docker/php-nginx-dev/$PHP_VERSION-alpine

docker build -t webdevops/php-apache:$PHP_VERSION-alpine docker/php-apache/$PHP_VERSION-alpine
docker build -t webdevops/php-apache-dev:$PHP_VERSION-alpine docker/php-apache-dev/$PHP_VERSION-alpine
docker build $SECRET -t webdevops/php-apache:$PHP_VERSION-alpine docker/php-apache/$PHP_VERSION-alpine
docker build $SECRET -t webdevops/php-apache-dev:$PHP_VERSION-alpine docker/php-apache-dev/$PHP_VERSION-alpine

# debian
docker build -t webdevops/php:$PHP_VERSION docker/php/$PHP_VERSION
docker build -t webdevops/php-dev:$PHP_VERSION docker/php-dev/$PHP_VERSION
docker build $SECRET -t webdevops/php:$PHP_VERSION docker/php/$PHP_VERSION
docker build $SECRET -t webdevops/php-dev:$PHP_VERSION docker/php-dev/$PHP_VERSION

docker build -t webdevops/php-nginx:$PHP_VERSION docker/php-nginx/$PHP_VERSION
docker build -t webdevops/php-nginx-dev:$PHP_VERSION docker/php-nginx-dev/$PHP_VERSION
docker build $SECRET -t webdevops/php-nginx:$PHP_VERSION docker/php-nginx/$PHP_VERSION
docker build $SECRET -t webdevops/php-nginx-dev:$PHP_VERSION docker/php-nginx-dev/$PHP_VERSION

docker build -t webdevops/php-apache:$PHP_VERSION docker/php-apache/$PHP_VERSION
docker build -t webdevops/php-apache-dev:$PHP_VERSION docker/php-apache-dev/$PHP_VERSION
docker build $SECRET -t webdevops/php-apache:$PHP_VERSION docker/php-apache/$PHP_VERSION
docker build $SECRET -t webdevops/php-apache-dev:$PHP_VERSION docker/php-apache-dev/$PHP_VERSION

docker images | grep webdevops | grep $PHP_VERSION
5 changes: 5 additions & 0 deletions ci/src/Commands/GenerateGithubCiCommand.php
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,11 @@ protected function execute(InputInterface $input, OutputInterface $output): int
],
'workflow_dispatch' => null,
],
// The token may only read the repository; the jobs that push images to ghcr.io get packages: write
// themselves (GithubJobBuilder::PUSH_PERMISSIONS).
'permissions' => [
'contents' => 'read',
],
'jobs' => $this->jobs,
];
$yamlString = Yaml::dump(
Expand Down
102 changes: 79 additions & 23 deletions ci/src/GithubJobBuilder.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,34 @@ class GithubJobBuilder

public const string IMAGE_TEST_RUNNER_JOB = 'image-test-runner';

/**
* PIE resolves and downloads the PHP extensions through the GitHub API. Without a token, its requests count against
* the limit for anonymous requests of the runner's IP address, and PIE fails with "Could not authenticate against
* github.com". The build steps pass the token of the run as the secret COMPOSER_AUTH, which the RUN instructions
* with PIE mount as an environment variable for Composer, on which PIE builds. Pull requests from forks get a
* read-only token too.
*
* Every script of those RUN instructions can read the token, not only PIE: the installers of Composer and Blackfire
* and the build systems of the extensions. In the image jobs it may push to ghcr.io (PUSH_PERMISSIONS), and a token
* cannot have fewer permissions in one step. The token ends with the job.
*
* With the token, the PIE requests of all jobs and runs share one limit, that of GITHUB_TOKEN per repository,
* instead of the anonymous limit of each runner's IP address: 7 extensions in 16 PHP jobs and Xdebug in 48
* development jobs per run.
*/
private const array COMPOSER_AUTH_ENV = [
'COMPOSER_AUTH' => '{"github-oauth":{"github.com":"${{ secrets.GITHUB_TOKEN }}"}}',
];

private const string COMPOSER_AUTH_SECRET = 'COMPOSER_AUTH=COMPOSER_AUTH';

/**
* The workflow gives the token only read access to the repository; the image jobs may also push their images to
* ghcr.io. The token gets no other permission, because the build steps hand it to every script of the RUN
* instructions with PIE (COMPOSER_AUTH_ENV).
*/
private const array PUSH_PERMISSIONS = ['contents' => 'read', 'packages' => 'write'];

/**
* Every image is built for these platforms. The README banner lists them too.
*/
Expand Down Expand Up @@ -79,6 +107,13 @@ public function getJobsDescription(array $node): array
$pushTags[] = '-t "ghcr.io/' . $alias . '"';
}

// The images of every architecture that the image job pushed ("Push arch image")
$archImages = array_map(
fn (array $platform): string => '"' . $this->archImage($node['image'], $node['tag'], $platform['arch']) . '"',
self::PLATFORMS,
);
$archImages[array_key_last($archImages)] .= ' && exit 0';

return [
$jobId => [
'strategy' => [
Expand All @@ -92,6 +127,8 @@ public function getJobsDescription(array $node): array
// even run if previous job skipped
'if' => '${{ !failure() && !cancelled() }}',
'runs-on' => '${{ matrix.runner }}',
// pushes its image to ghcr.io on master and takes the parent images from there
'permissions' => self::PUSH_PERMISSIONS,
'container' => 'webdevops/dockerfile-build-env',
'steps' => array_values(
array_filter(
Expand All @@ -106,12 +143,14 @@ public function getJobsDescription(array $node): array
'name' => 'Build (load locally)',
'if' => $hasImageDependencies ? '${{ github.ref == \'refs/heads/master\' }}' : null,
'uses' => 'docker/build-push-action@v7',
'env' => self::COMPOSER_AUTH_ENV,
'with' => $this->buildPushWith($node),
], fn (array|string|null $value): bool => $value !== null),
$hasImageDependencies ? [
'name' => 'Build (load locally, from parent artifact)',
'if' => '${{ github.ref != \'refs/heads/master\' }}',
'uses' => 'docker/build-push-action@v7',
'env' => self::COMPOSER_AUTH_ENV,
'with' => array_merge(
$this->buildPushWith($node),
['build-contexts' => $this->buildContexts($imageDependencies)],
Expand All @@ -138,23 +177,19 @@ public function getJobsDescription(array $node): array
] : null,
[
'if' => '${{github.ref == \'refs/heads/master\'}}',
'name' => 'Login to ghcr.io',
'uses' => 'docker/login-action@v4',
'with' => [
'registry' => 'ghcr.io',
'username' => '${{ github.actor }}',
'password' => '${{ secrets.GITHUB_TOKEN }}',
],
...$this->loginGhcrStep(),
],
[
'name' => 'Push arch image',
'if' => '${{github.ref == \'refs/heads/master\'}}',
'run' => 'docker push "ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-${{ matrix.arch }}"-' . $node['tag'],
'run' => 'docker push "' . $this->archImage($node['image'], $node['tag']) . '"',
],
$hasChildren ? [
'name' => 'Export image (OCI layout)',
'if' => '${{ github.ref != \'refs/heads/master\' }}',
'uses' => 'docker/build-push-action@v7',
// Only needed if a layer is not in the cache after all.
'env' => self::COMPOSER_AUTH_ENV,
// Reuse the tested local build, including on no-cache reruns.
'with' => array_merge(
[
Expand All @@ -164,6 +199,7 @@ public function getJobsDescription(array $node): array
'build-args' => implode("\n", [
'TARGETARCH=${{ matrix.arch }}',
]),
'secret-envs' => self::COMPOSER_AUTH_SECRET,
],
// Must use the exact same build-contexts override as the
// tested build above, otherwise this second Buildx invocation
Expand Down Expand Up @@ -195,19 +231,12 @@ public function getJobsDescription(array $node): array
$jobId . '_publish' => [
'name' => $node['name'] . ' - Publish',
'runs-on' => 'ubuntu-latest',
'permissions' => ['packages' => 'write'],
'needs' => $jobId,
'if' => '${{github.ref == \'refs/heads/master\'}}',
'steps' => [
['uses' => 'docker/setup-buildx-action@v4'],
[
'name' => 'Login to ghcr.io',
'uses' => 'docker/login-action@v4',
'with' => [
'registry' => 'ghcr.io',
'username' => '${{ github.actor }}',
'password' => '${{ secrets.GITHUB_TOKEN }}',
],
],
$this->loginGhcrStep(),
[
'name' => 'Login to hub.docker.com',
'uses' => 'docker/login-action@v4',
Expand All @@ -226,8 +255,7 @@ public function getJobsDescription(array $node): array
' ' . implode(" \\\n ", [
'docker buildx imagetools create',
...$pushTags,
'"ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-amd64-' . $node['tag'] . '"',
'"ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-arm64-' . $node['tag'] . '" && exit 0',
...$archImages,
]),
' sleep $((i*i))',
'done',
Expand Down Expand Up @@ -270,13 +298,41 @@ private function buildPushWith(array $node): array
'context' => $this->context($node),
'platforms' => '${{ matrix.platform }}',
'load' => true,
'tags' => 'ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-${{ matrix.arch }}-' . $node['tag'],
'tags' => $this->archImage($node['image'], $node['tag']),
// Re-run all jobs to test a fresh build of the full image chain.
'no-cache' => '${{ github.event_name == \'schedule\' || github.run_attempt > 1 }}',
...$this->buildCacheWith($node),
'build-args' => implode("\n", [
'TARGETARCH=${{ matrix.arch }}',
]),
'secret-envs' => self::COMPOSER_AUTH_SECRET,
];
}

/**
* The image of one architecture that the job of webdevops/$image:$tag builds and, on master, pushes to ghcr.io,
* e.g. "ghcr.io/webdevops/php:sha-<commit>-amd64-8.4"; by default the architecture of the matrix job.
*/
private function archImage(string $image, string $tag, string $arch = '${{ matrix.arch }}'): string
{
return 'ghcr.io/webdevops/' . $image . ':sha-${{ github.sha }}-' . $arch . '-' . $tag;
}

/**
* Logs in to ghcr.io with the token of the run, to push the images there.
*
* @return array<string, mixed>
*/
private function loginGhcrStep(): array
{
return [
'name' => 'Login to ghcr.io',
'uses' => 'docker/login-action@v4',
'with' => [
'registry' => 'ghcr.io',
'username' => '${{ github.actor }}',
'password' => '${{ secrets.GITHUB_TOKEN }}',
],
];
}

Expand Down Expand Up @@ -411,7 +467,7 @@ private function imageTests(array $node): ?string
}

return 'make test-image-run IMAGE=' . $node['id']
. ' REFERENCE=ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-${{ matrix.arch }}-' . $node['tag'];
. ' REFERENCE=' . $this->archImage($node['image'], $node['tag']);
}

/**
Expand All @@ -426,9 +482,9 @@ private function structuredTests(array $node): array
if (file_exists($this->rootPath . '/tests/structure-test/' . $node['image'] . '/test.yaml')) {
$script[] = 'cd tests/structure-test';
if (file_exists($this->rootPath . '/tests/structure-test/' . $node['image'] . '/' . $node['tag'] . '/test.yaml')) {
$script[] = '/usr/local/bin/container-structure-test test --image ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-${{ matrix.arch }}-' . $node['tag'] . ' --config ' . $node['image'] . '/test.yaml --config ' . $node['image'] . '/' . $node['tag'] . '/test.yaml';
$script[] = '/usr/local/bin/container-structure-test test --image ' . $this->archImage($node['image'], $node['tag']) . ' --config ' . $node['image'] . '/test.yaml --config ' . $node['image'] . '/' . $node['tag'] . '/test.yaml';
} else {
$script[] = '/usr/local/bin/container-structure-test test --image ghcr.io/webdevops/' . $node['image'] . ':sha-${{ github.sha }}-${{ matrix.arch }}-' . $node['tag'] . ' --config ' . $node['image'] . '/test.yaml';
$script[] = '/usr/local/bin/container-structure-test test --image ' . $this->archImage($node['image'], $node['tag']) . ' --config ' . $node['image'] . '/test.yaml';
}
}

Expand Down
3 changes: 3 additions & 0 deletions ci/tests/Commands/GenerateGithubCiCommandTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ public function testWritesTheWorkflowWithTheValidationFirstAndTheImageJobsSorted
'pull_request' => ['branches' => ['master']],
'workflow_dispatch' => null,
], $workflow['on']);
self::assertSame(['contents' => 'read'], $workflow['permissions'], 'only the jobs that push may write packages');
self::assertArrayNotHasKey('permissions', $workflow['jobs']['validate-automation']);
self::assertArrayNotHasKey('permissions', $workflow['jobs']['image-test-runner']);
self::assertSame(self::ALL_JOBS, array_keys($workflow['jobs']));
self::assertSame('Validate Automation', $workflow['jobs']['validate-automation']['name']);
}
Expand Down
20 changes: 19 additions & 1 deletion ci/tests/GithubJobBuilderTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ public function testBuildAndPublishJob(): void
self::assertSame('${{ matrix.runner }}', $jobs['base_alpine']['runs-on']);
self::assertSame('webdevops/dockerfile-build-env', $jobs['base_alpine']['container']);
self::assertSame('base_alpine', $jobs['base_alpine_publish']['needs']);
self::assertSame(['contents' => 'read', 'packages' => 'write'], $jobs['base_alpine']['permissions'], 'pushes to ghcr.io');
self::assertSame(['packages' => 'write'], $jobs['base_alpine_publish']['permissions'], 'pushes to ghcr.io, needs no checkout');
self::assertSame('${{github.ref == \'refs/heads/master\'}}', $jobs['base_alpine_publish']['if']);
}

Expand Down Expand Up @@ -71,6 +73,22 @@ public function testImageTestRunnerIsBuiltOncePerArchitectureIntoTheCache(): voi
self::assertTrue($job['steps'][2]['continue-on-error'], 'a failure must not skip the image jobs that need this job');
}

public function testEveryBuildGetsTheTokenOfTheRunForPie(): void
{
$steps = $this->steps($this->jobs([
'imageParent' => 'webdevops/base:ubuntu-22.04',
'imageParentRef' => 'webdevops/base:latest',
'hasChildren' => true,
]));

foreach (['Build (load locally)', 'Build (load locally, from parent artifact)', 'Export image (OCI layout)'] as $name) {
self::assertSame(['COMPOSER_AUTH' => '{"github-oauth":{"github.com":"${{ secrets.GITHUB_TOKEN }}"}}'], $steps[$name]['env'], $name);
self::assertSame('COMPOSER_AUTH=COMPOSER_AUTH', $steps[$name]['with']['secret-envs'], $name);
}

self::assertArrayNotHasKey('env', $steps['Build the image test runner'], 'the runner does not use PIE');
}

public function testBuildsTheDirectoryOfTheDockerfile(): void
{
$with = $this->step($this->jobs(), 'Build (load locally)')['with'];
Expand Down Expand Up @@ -100,7 +118,7 @@ public function testImageWithoutDependenciesIsBuiltOnce(): void
], array_keys($steps));
self::assertArrayNotHasKey('if', $steps['Build (load locally)']);
self::assertSame(
'docker push "ghcr.io/webdevops/base:sha-${{ github.sha }}-${{ matrix.arch }}"-alpine',
'docker push "ghcr.io/webdevops/base:sha-${{ github.sha }}-${{ matrix.arch }}-alpine"',
$steps['Push arch image']['run'],
);
}
Expand Down
Loading
Loading