Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- 'stubs/**'
- 'composer.lock'
- 'tests/bench/**'
- 'tally.php'
- '.github/workflows/benchmarks.yml'
push:
branches:
Expand All @@ -18,6 +19,7 @@
- 'stubs/**'
- 'composer.lock'
- 'tests/bench/**'
- 'tally.php'
- '.github/workflows/benchmarks.yml'

concurrency:
Expand All @@ -31,21 +33,21 @@
CODSPEED_VERSION: "5.6.3"
# of codspeed-runner-x86_64-unknown-linux-musl.tar.gz in that release
CODSPEED_SHA256: "f29532a5cbcb3462bc5d6bcf5570ddb8c8cbfffed579cb48c35d3910ed5f7f00"
# tests/bench/measure.php calls it over FFI to measure only the analysis of the benchmark
# Tally calls it over FFI to count only the benchmark, not the warm-up before it
INSTRUMENT_HOOKS_COMMIT: "8fd5b878499f819561b33219cdafaa6f8c28b4e4"

jobs:
run:
name: "Benchmarks shard ${{ matrix.shard }}"
# Pinned because the instruction counts depend on the PHP build and the system libraries
runs-on: ubuntu-24.04
# The slowest shard takes about 20 minutes, and run.php stops a benchmark after 25
# The slowest shard takes about 20 minutes, and tally.php stops a benchmark after 25
timeout-minutes: 40

strategy:
fail-fast: false
matrix:
# Each shard is one runner, see tests/bench/shard.php
# Each shard is one runner, tally run --shard balances them by the expected instructions
shard: [1, 2, 3, 4]

steps:
Expand All @@ -64,7 +66,7 @@
with:
coverage: "none"
php-version: "8.5"
extensions: ds,ffi,mbstring
extensions: ds,ffi,mbstring,posix
ini-file: production
ini-values: memory_limit=-1

Expand All @@ -91,23 +93,19 @@
- name: "Run benchmarks"
env:
INSTRUMENT_HOOKS_LIBRARY: ${{ runner.temp }}/instrument-hooks/libinstrument_hooks.so
SHARD: ${{ matrix.shard }}
SHARDS: ${{ strategy.job-total }}
# Valgrind runs each benchmark on one CPU, so the shard runs as many at once as the runner has
run: |
mkdir -p tmp/bench-results
php tests/bench/shard.php "$SHARDS" "$SHARD" | xargs -P "$(nproc)" -I {} php tests/bench/run.php "tests/bench/data/{}" "tmp/bench-results/{}.neon"
run: "php vendor/bin/tally run --shard=${{ matrix.shard }}/${{ strategy.job-total }} --output=tmp/tally-results"

- name: "Upload results"
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: benchmark-results-${{ matrix.shard }}
path: tmp/bench-results/*.neon
name: tally-results-${{ matrix.shard }}
path: tmp/tally-results
if-no-files-found: ignore

compare:
name: "Compare with expected.neon"
name: "Compare all shards"
needs: run
if: ${{ !cancelled() }}
runs-on: ubuntu-latest
Expand Down Expand Up @@ -135,16 +133,18 @@
- name: "Download results"
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: benchmark-results-*
pattern: tally-results-*
merge-multiple: true
path: tmp/bench-results
path: tmp/tally-results

- name: "Compare"
run: "php tests/bench/compare.php tmp/bench-results tmp/expected.neon"
run: "php vendor/bin/tally compare --table tmp/tally-results"

- name: "Upload measured metrics"
# vendor/bin/tally accept <this artifact> writes the numbers into the files in tests/bench/data
- name: "Upload results of all shards"
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: benchmark-measured
path: tmp/expected.neon
name: tally-results
path: tmp/tally-results
if-no-files-found: ignore
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,7 @@ New introduced classes in these categories should be shadowed with a native C++
- **Always verify the test fails before the fix.** Stating "tests pass" is not enough — that only proves the fix doesn't break things. `git stash` the source change, run the test, see it fail for the right reason; `git stash pop`, run again, see it pass. Only then is the fix confirmed. The `regression-test` skill bakes this in for bug-issue workflows.
- **Reproducers come from the OP's playground link** in the issue body, not from later comments. Copy the snippet as close to verbatim as practical.
- **Integration tests**: `AnalyserIntegrationTest` runs full analysis on test files and checks error output.
- **Benchmarks**: every file in `tests/bench/data` is analysed in the Benchmarks workflow under CodSpeed's CPU simulation, which counts instructions instead of measuring time, so the numbers are the same on every runner. `tests/bench/measure.php` boots PHPStan and analyses a warm-up file first, so only the analysis of the benchmark file itself is counted. `tests/bench/compare.php` compares them against `tests/bench/expected.neon`; when a change moves them on purpose or adds a benchmark, replace that file with the workflow's `benchmark-measured` artifact.
- **Benchmarks**: every file in `tests/bench/data` is analysed by [Tally](https://github.com/phpstan/tally) in the Benchmarks workflow, under CodSpeed's CPU simulation, which counts instructions instead of measuring time, so the numbers are the same on every runner. `tests/bench/AnalysisBench.php` boots PHPStan and analyses `tests/bench/Warmup.php` uncounted in a `#[Before]` method, so only the analysis of the benchmark file itself is counted. The expected instructions, cycles and memory are in the comment at the top of each data file (`tests/bench/AnalysedFileHeader.php`); a new file without the comment passes and gets its numbers from CI. When a change moves them on purpose or adds a benchmark, download the workflow's `tally-results` artifact and run `vendor/bin/tally accept <directory>`. The configuration is `tally.php`.

### Adding support for new PHP versions

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ lint:
--exclude tests/PHPStan/Rules/Variables/data/bug-14352.php \
--exclude tests/PHPStan/Rules/Variables/data/bug-14351.php \
--exclude tests/PHPStan/Rules/Properties/data/bug-14457.php \
src tests
src tests tally.php

install-paratest:
composer install --working-dir tests
Expand Down
2 changes: 1 addition & 1 deletion build/ignore-by-php-version.neon.php
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
$includes[] = __DIR__ . '/enums.neon';
$includes[] = __DIR__ . '/readonly-property.neon';
$includes[] = __DIR__ . '/turbo-declaration-generator-pre-8.1.neon';
$includes[] = __DIR__ . '/tally-pre-8.1.neon';
}

// PHP 8.1's serialize() does not trim its buffer, so the rule's cache keys cost 4 KB each
Expand All @@ -27,7 +28,6 @@

if (PHP_VERSION_ID < 80200) {
$includes[] = __DIR__ . '/randomizer.neon';
$includes[] = __DIR__ . '/memory-reset-peak-usage.neon';
}

if (PHP_VERSION_ID >= 80000) {
Expand Down
8 changes: 0 additions & 8 deletions build/memory-reset-peak-usage.neon

This file was deleted.

7 changes: 6 additions & 1 deletion build/phpstan.neon
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ parameters:
- PHPStan
- ../src
- ../tests
- ../tally.php
bootstrapFiles:
- ../tests/phpstan-bootstrap.php
scanFiles:
Expand Down Expand Up @@ -90,7 +91,6 @@ parameters:
paths:
- ../src/File/FsEventsFileMonitor.php
- ../src/File/InotifyFileMonitor.php
- ../tests/bench/measure.php
-
identifier: shipmonk.deadMethod
message: '#^Unused .*?::__construct#' # likely used in DIC
Expand All @@ -109,6 +109,11 @@ parameters:
- ../tests/PHPStan/Tests
- ../tests/e2e
reportUnmatched: false
-
# Tally calls the benchmark methods, their data providers and #[Before] methods
identifier: shipmonk.deadMethod
path: ../tests/bench/AnalysisBench.php
reportUnmatched: false
-
identifier: shipmonk.deadConstant
paths:
Expand Down
7 changes: 7 additions & 0 deletions build/tally-pre-8.1.neon
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
parameters:
excludePaths:
# Tally's sources use enums and readonly properties, which PHPStan does not parse for PHP < 8.1
- ../tally.php
- ../tests/bench/AnalysedFileExpectations.php
- ../tests/bench/AnalysedFileHeader.php
- ../tests/bench/AnalysisBench.php
1 change: 1 addition & 0 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@
"phpstan/phpstan-nette": "^2.0",
"phpstan/phpstan-phpunit": "^2.1",
"phpstan/phpstan-strict-rules": "^2.1",
"phpstan/tally": "dev-main",
"phpunit/phpunit": "^11.5.23",
"shipmonk/composer-dependency-analyser": "^1.5",
"shipmonk/dead-code-detector": "^1.4.2",
Expand Down
62 changes: 60 additions & 2 deletions composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions phpcs.xml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
<file>build</file>
<file>src</file>
<file>tests</file>
<file>tally.php</file>
<file>compiler/src</file>
<file>compiler/tests</file>
<file>apigen/src</file>
Expand Down
15 changes: 15 additions & 0 deletions tally.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<?php declare(strict_types = 1);

namespace PHPStan\Benchmark;

use PHPStan\Process\Restart\OpcacheSettings;
use Tally\Configuration;

return Configuration::create()
->withPaths([__DIR__ . '/tests/bench/AnalysisBench.php'])
// bin/phpstan restarts PHP with these OPcache settings, the benchmarks run without the restart
->withIniEntries([...OpcacheSettings::getIniEntries(), 'memory_limit=-1'])
// The slowest benchmark takes a few seconds natively and about 13 minutes under the simulation.
// One stuck in a loop fails in the native run, one that regresses far beyond that fails as not
// measured instead of holding up its shard.
->withTimeLimits(30, 25 * 60);
Loading
Loading