From 01b45a1292fae8ec5efc190f00a8b2b4108262da Mon Sep 17 00:00:00 2001 From: Martin Kourim Date: Thu, 12 Nov 2026 19:05:40 +0100 Subject: [PATCH] docs: update 11.2.1 tag testing report Distinguish fresh 11.2.1 runs from retained 11.2.0 coverage for the performance-fix release. Update regression counts, timings, and Dijkstra failures from the linked reports. Summarize Mainnet sync improvements and comparison limits, remove stale copied claims, and register the report in the tag testing index. --- .../source/test_results/node/tag_11_2_1.rst | 260 ++++++++++++++++++ src_docs/source/test_results/tag_tests.rst | 1 + 2 files changed, 261 insertions(+) create mode 100644 src_docs/source/test_results/node/tag_11_2_1.rst diff --git a/src_docs/source/test_results/node/tag_11_2_1.rst b/src_docs/source/test_results/node/tag_11_2_1.rst new file mode 100644 index 000000000..1927d7dcd --- /dev/null +++ b/src_docs/source/test_results/node/tag_11_2_1.rst @@ -0,0 +1,260 @@ +11.2.1 +====== + +* Release notes - + + +What Was Tested +--------------- + +``cardano-node`` 11.2.1 is a minor release focused on fixing several performance +regressions. Most of the :doc:`11.2.0 testing ` remains valid, so only the +activities needed to validate the changes were repeated rather than the full release +testing checklist. + +Fresh 11.2.1 runs covered the Conway regression suite at protocol version 11 with the +default UTxO backend and with the LSM disk backend combined with Genesis consensus mode, +the Dijkstra suite at protocol version 12, and a full Mainnet sync from genesis. +The remaining checklist results are carried forward from 11.2.0, including Preview, +the 11.1.3 to 11.2.0 binary upgrade, and Byron to Conway hard-fork testing. Rows labeled +11.2.0 identify that retained coverage, not new runs against 11.2.1. + +A |:heavy_check_mark:| means the activity finished with no unexpected failures. Tests that +hit an already known and tracked issue are reported as *broken* rather than *failed*, and +tests that do not apply to a given configuration are reported as *skipped*. + + +Release Testing Checklists +-------------------------- + +.. list-table:: Regression Testsuite + :widths: 64 7 + :header-rows: 0 + + * - `Default UTxO backend `__ + - |:heavy_check_mark:| + * - `LSM disk UTxO backend `__ + - |:heavy_check_mark:| + * - `Genesis consensus mode `__ + - |:heavy_check_mark:| + * - `Dijkstra PV12 `__ + - |:x:| [1]_ + * - Testing on Preview (11.2.0) + - |:heavy_check_mark:| + +.. list-table:: Other Testing + :widths: 64 7 + :header-rows: 0 + + * - Upgrade testing (11.1.3 to 11.2.0) + - |:heavy_check_mark:| + * - Rollback testing (11.2.0) + - |:heavy_check_mark:| + * - Reconnection testing (11.2.0) + - |:heavy_check_mark:| + * - Block-production testing (11.2.0) + - |:heavy_check_mark:| + * - Testing on Ubuntu, Debian, Mint (11.2.0) + - |:heavy_check_mark:| + * - Shutdown testing (IPC, block synced, slot synced, Ctrl+C; 11.2.0) + - |:heavy_check_mark:| + * - :doc:`Sync testing on Mainnet (Linux) <../sync_reports/mainnet_11_2_1>` + - |:heavy_check_mark:| + * - Byron to Conway 11 hard-fork testing (11.2.0) + - |:heavy_check_mark:| + * - Check build instructions changes (11.2.0) + - |:heavy_check_mark:| + +.. [1] The 11.2.1 Dijkstra run had two failed tests: + + * Missing local-submission support for intentionally phase-2-invalid transactions, + see `node issue #6748 `__. + * Guard-signature rejection closes the connection, likely fixed by + `ledger PR #6189 `__. + + +Regression Testsuite Results +---------------------------- + +Both 11.2.1 Conway regression configurations collected the same 2,414 tests with the same +outcome: no failures, and nine tests reported as *broken* because they hit known issues. +The Dijkstra run collected 2,414 tests too, but had two failures and 25 *broken* results. + +.. list-table:: + :widths: 30 14 12 12 12 12 + :header-rows: 1 + + * - Configuration + - Total + - Passed + - Broken + - Skipped + - Failed + * - `Default UTxO backend `__ + - 2,414 + - 1,936 + - 9 + - 469 + - 0 + * - `LSM disk UTxO backend, Genesis consensus mode `__ + - 2,414 + - 1,936 + - 9 + - 469 + - 0 + * - `Dijkstra PV12, default UTxO backend `__ + - 2,414 + - 2,091 + - 25 + - 296 + - 2 + +The Conway default-backend run took approximately 1 h 16 min of wall-clock time, and the +disk-backend run took 1 h 22 min. Both used local testnet clusters with three stake pools +each. The disk-backend run limited concurrent tests per cluster to five, compared with +eight for the default backend. The Dijkstra run took approximately 1 h 10 min. + +The suite covers transactions and fees, native tokens and metadata, the mempool, Plutus V1, +V2 and V3, staking, rewards and stake pool lifecycle, KES key rotation, Conway governance, +``cardano-cli`` behavior, node metrics, node configuration, and socket handling. + +Known issues hit by both Conway runs (reported as *broken*): + +.. list-table:: + :widths: 62 10 + :header-rows: 1 + + * - Issue + - Tests + * - `cardano-ledger#4566 `__ - repeated certificates stripped from a Conway transaction + - 6 + * - `cardano-ledger#6102 `__ - genesis staking injection leaves ``psVRFKeyHashes`` empty + - 1 + * - `cardano-node#4002 `__ - ``PastHorizon`` error in ``query leadership-schedule`` + - 1 + * - `cardano-node#4488 `__ - Plutus debug output does not return the evaluation error + - 1 + +These tests are tracked as expected failures. The issue counts above describe the linked +Conway runs. The Dijkstra run's 25 *broken* results comprise 19 tests affected by missing +local-submission support for intentionally phase-2-invalid transactions +(`node issue #6748 `__), two +awaiting the parent/child datum-allowance ledger fix ``846a63f764``, two affected by +ledger issue #6102, and one each affected by node issues #4002 and #4488. Its two failed +tests are described in footnote [1]_. + + +Mainnet Sync Results +-------------------- + +The :doc:`11.2.1 Mainnet sync report <../sync_reports/mainnet_11_2_1>` shows substantial +improvements over 11.2.0: + +* Peak resident memory fell from 22.08 to 16.98 GiB (-23.1%). +* Peak garbage-collector live data fell from 18.03 to 11.00 GiB (-39.0%). +* Sync time fell from 132,111 to 113,689 seconds (-13.9%), about five hours sooner, + despite syncing 1,090,011 more slots. Babbage accounted for most of the saving. +* Total CPU time fell 9.6%, while mean CPU utilization rose from 112% to 118% of one core. + +These measurements support the performance fixes, but do not establish that every +historical regression is resolved. See the comparison limits below and the full report. + + +What the Checklist Items Cover +------------------------------ + +**Default UTxO backend** - the whole end-to-end test suite against a local testnet in the +Conway era, with the node keeping the UTxO set in memory, which is the node default. + +**LSM disk UTxO backend** - the same full suite, but the node keeps the UTxO set on disk. +The on-disk backend changes how ledger state is accessed, so it can expose problems such +as query stalls, performance cliffs, and corruption that the in-memory backend hides. + +**Genesis consensus mode** - the same full suite with Ouroboros Genesis instead of the +default Praos chain-selection mode. For this tag it was combined with the disk backend in a +single run, which is why the two rows link to the same report. + +**Testing in Dijkstra PV12** - testing in the Dijkstra era at protocol version 12, +separately from the Conway protocol version 11 regression runs. This activity exposed +the local-submission and guard-signature issues described in footnote [1]_. + +**Testing on Preview** - a subset of the suite run against the real, long-running Preview +network instead of a local testnet, exercising the release against real peers, real traffic, +and a chain that cannot be reset. + +**Upgrade testing** - a simulation of a real network upgrade. A cluster is started on the +base release, then nodes are upgraded in stages so that the network temporarily runs mixed +versions, and the sanity suite is rerun after each step. It catches problems that appear +only across versions: configuration and genesis incompatibilities, block diffusion between +old and new nodes, Plutus cost model updates, and unexpected log errors. + +**Rollback testing** - the network is deliberately split in two, both halves accept +conflicting transactions, and the halves are then reconnected. Two scenarios are checked: +reconnecting after enough blocks that the fork must become permanent, and reconnecting +early enough that global consensus must be restored and the losing branch rolled back. + +**Reconnection testing** - a block-producing node is stopped and restarted repeatedly; it +must rejoin the network, resynchronize, and return its peer-connection metrics to their +expected values without manual intervention. + +**Block-production testing** - a long run over many epochs in which pools with equal stake, +half of them on the in-memory and half on the disk backend, must produce roughly the same +number of blocks. A configuration that silently slows down forging shows up here as a +skewed block distribution rather than as a failing assertion elsewhere. + +**Testing on Ubuntu, Debian, Mint** - a regression run inside a container based on a +mainstream Linux distribution. The node binaries themselves are still built with Nix and are +the same in every distribution; what varies is the surrounding environment: the system +configuration, certificates, locale, shell utilities, and filesystem behavior the node and +the test framework run on top of. + +**Shutdown testing** - the node is shut down through every supported mechanism (slot-based, +block-based, inter-process communication, Ctrl+C) and must exit promptly, log a clean +shutdown, and start again afterwards without replaying or corrupting its database. + +**Sync testing on Mainnet** - a full Mainnet sync from genesis, measuring sync time, +per-era and per-epoch speed, and CPU and memory usage against previous releases. This is +the activity that measures resource consumption at Mainnet scale. + +**Byron to Conway hard-fork testing** - a local testnet started in the Byron era and +hard-forked through every intermediate era up to the target Conway protocol version, +exercising era translations and hard-fork paths that a testnet started directly in Conway +never touches. + +**Check build instructions changes** - building the node from source in a clean container +by following the published build instructions, to confirm those instructions are still +correct for the tag. + + +Scope and Limits +---------------- + +* **Preview runs only a subset of the suite.** A Preview epoch lasts two hours, so tests + that wait for an epoch boundary or span several reward cycles are excluded. Preview + also runs against a single node that joins the network, and the cluster is never + restarted during the run. + +* **Retained coverage is not a fresh 11.2.1 result.** Preview, upgrade, rollback, + reconnection, block-production, distribution, shutdown, hard-fork, and build-instruction + results come from 11.2.0 and remain applicable for this limited performance-fix release. + Their checkmarks do not mean those activities were repeated against 11.2.1. + +* **The retained upgrade test did not include a hard fork.** The upgrade from 11.1.3 + to 11.2.0 stayed at protocol version 11, exercising the binary upgrade and mixed-version + period. No separate upgrade to 11.2.1 is reported here. The retained Byron to Conway + activity covered hard-fork handling through protocol version 11, not a hard fork to + Dijkstra protocol version 12. + +* **The 469 skipped tests in each Conway run do not apply to those configurations.** Examples + include tests for other eras, other UTxO backends, or components not enabled in the run. + Their count should not be read as successful coverage of those other configurations. + +* **The passing Conway runs do not imply a passing Dijkstra result.** The 11.2.1 Dijkstra + checklist item remains unsuccessful because of the two failures in footnote [1]_. + +* **Mainnet sync improved, but the comparison is not fully controlled.** The 11.2.0 and + 11.2.1 runs differ in maximum snapshot delay, kernel, CLI version, and chain endpoint. + Each build was measured once. Peak resident memory remains 10.0% above the historical + 11.0.1 baseline, early-chain memory is not uniformly lower, and delays remain around + Babbage epochs 461-475. The :doc:`Mainnet sync report <../sync_reports/mainnet_11_2_1>` + details these limits and the supporting experimental fix-verification run. diff --git a/src_docs/source/test_results/tag_tests.rst b/src_docs/source/test_results/tag_tests.rst index 107e81370..95b243ef8 100644 --- a/src_docs/source/test_results/tag_tests.rst +++ b/src_docs/source/test_results/tag_tests.rst @@ -4,6 +4,7 @@ Tag Testing .. toctree:: :maxdepth: 4 + node/tag_11_2_1.rst node/tag_11_2_0.rst node/tag_11_1_1.rst node/tag_11_1_0.rst