Skip to content

Run the benchmarks with Tally - #7059

Merged
ondrejmirtes merged 2 commits into
2.3.xfrom
tally-benchmarks
Nov 13, 2026
Merged

ondrejmirtes merged 2 commits into
2.3.xfrom
tally-benchmarks

Conversation

@ondrejmirtes

Copy link
Copy Markdown
Member

Replaces the scripts in tests/bench with phpstan/tally.

  • tests/bench/AnalysisBench.php analyses each file in tests/bench/data. PHPStan boots and analyses Warmup.php in an uncounted #[Before] method, so only the analysis of the file is counted, same as measure.php did.
  • The expected numbers moved from expected.neon into a comment at the top of each data file. They are the old CI numbers, so this run compares instructions and cycles with them.
  • Memory is now the peak above the start of the analysis, not including the container, so every case reports memory as faster. The numbers get accepted from this PR's tally-results artifact with vendor/bin/tally accept.
  • The workflow runs tally run --shard=N/4 on four runners and tally compare --table over all shards.
  • phpstan/tally is in the root require-dev, the configuration is tally.php.

The benchmarks in tests/bench/data are now cases of AnalysisBench, run by
phpstan/tally. The expected numbers moved from expected.neon into the
comment at the top of each file, and the workflow runs tally run --shard
and tally compare instead of the scripts in tests/bench.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d525-9c4c-73cf-8edc-000aa8e78378
run: |
mkdir -p tmp/bench-results
php tests/bench/shard.php "$SHARDS" "$SHARD" | xargs -P "$(nproc)" -I {} php tests/bench/run.php "tests/bench/data/{}" "tmp/bench-results/{}.neon"
run: "php vendor/bin/tally run --shard=${{ matrix.shard }}/${{ strategy.job-total }} --output=tmp/tally-results"
run: |
mkdir -p tmp/bench-results
php tests/bench/shard.php "$SHARDS" "$SHARD" | xargs -P "$(nproc)" -I {} php tests/bench/run.php "tests/bench/data/{}" "tmp/bench-results/{}.neon"
run: "php vendor/bin/tally run --shard=${{ matrix.shard }}/${{ strategy.job-total }} --output=tmp/tally-results"
run: |
mkdir -p tmp/bench-results
php tests/bench/shard.php "$SHARDS" "$SHARD" | xargs -P "$(nproc)" -I {} php tests/bench/run.php "tests/bench/data/{}" "tmp/bench-results/{}.neon"
run: "php vendor/bin/tally run --shard=${{ matrix.shard }}/${{ strategy.job-total }} --output=tmp/tally-results"
PHP 7.4 reads #[ as a comment until the end of the line, so the
attribute of AnalysisBench is on one line. Tally's sources use enums and
readonly properties, which PHPStan does not parse for PHP < 8.1, so the
benchmarks are not analysed there.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1d525-9c4c-73cf-8edc-000aa8e78378
@ondrejmirtes
ondrejmirtes merged commit fef62b1 into 2.3.x Nov 13, 2026
900 of 905 checks passed
@ondrejmirtes
ondrejmirtes deleted the tally-benchmarks branch November 13, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants