Improve mempool retry resilience and SECP256k1 test setup

README.md

@@ -192,6 +192,7 @@ You can fine-tune test runs using these environment variables:
 | `TX_FIREHOSE_REV`   | `tx-firehose` version (default: `leios-prototype`).      |
 | `PYTEST_ARGS`       | Extra options passed to pytest.                          |
 | `TEST_THREADS`      | Number of pytest workers (default: `20`).                |
+| `MONITOR_INTERVAL`  | Resource snapshot interval in seconds (default: `60`).   |
 | `TIME_DRIFT_POOL`   | Pool (1-3) for libfaketime clock skew; unset = off.      |
 | `TIME_DRIFT_SPEC`   | libfaketime `FAKETIME` string, e.g. `+0 x1.008`.         |
 | `LIBFAKETIME_PATH`  | Override path to `libfaketimeMT.so.1`.                   |

agent_docs/failure_analysis_prompt.md

@@ -12,7 +12,9 @@ Inputs available under `{RUN_DIR}/` (use only what exists):
 - `{RUN_DIR}/errors_all.log` — output of `runner/grep_errors.sh`: case-insensitive grep for `:error:|failed|failure` in `*.stdout`/`*.stderr` files under `testing_artifacts/`, with paths relative to that dir. Rotated logs (`*.stdout.<n>`, `*.stderr.<n>`) are **not** grepped, so a thin or empty `errors_all.log` doesn't prove there were no errors - grep the rotated logs directly when an early crash is suspected
 - `{RUN_DIR}/scheduling.log` — cluster instance manager log
 - `{RUN_DIR}/testrun-report.xml` — junit XML
-- `{RUN_DIR}/monitor.log` — system resource snapshots every 10 min
+- `{RUN_DIR}/monitor.log` - resource snapshots at the configured interval (60 seconds by default
+  in current regression runs; older runs used 10 minutes), including host usage, node thread
+  scheduling counters, visible cgroup CPU throttling, and available node GC/CPU metrics
 - `{RUN_DIR}/testing_artifacts/pytest-*/cm-status.db` — cluster-management SQLite status database ("test running", resource and flag records as they were at the end of the run); query with `sqlite3 -readonly -header <db_file> 'SELECT * FROM overview ORDER BY instance_num, kind'`
 
 Counting tests (IMPORTANT):

agent_docs/mempool_timeouts.md

@@ -0,0 +1,44 @@
+# Investigating mempool latency
+
+`MempoolTxTooSlow` measures wall-clock validation latency, including GC and scheduling
+delays. Local submissions use this rejection for both soft and hard timeouts. An empty
+mempool does not rule out CPU starvation.
+
+## Resource diagnostics
+
+`runner/regression.sh` samples `monitor.log` every 60 seconds by default. Override with
+`MONITOR_INTERVAL=10` for short incidents. Snapshots include:
+
+- Host CPU, memory, disk and CPU pressure.
+- Node PIDs, lifetime-average CPU usage, thread counts and main-thread context switches.
+- `/proc/<pid>/task/*/schedstat`: CPU runtime, runnable queue wait, and timeslices,
+  summed over live threads, including RTS workers. Compare snapshots for the same PID;
+  thread exits/restarts can reset counters, so sums are not strictly monotonic.
+- Cgroup v2 CPU quotas, throttling counters and pressure at visible node/ancestor groups.
+  Rising `nr_throttled`/`throttled_usec` points at quota contention. Namespace-hidden
+  groups and cgroup v1 cannot be fully diagnosed with these snapshots.
+- GC, allocation, heap, pause and CPU metrics exposed by each configured local
+  Prometheus endpoint. Names and availability depend on the node/tracing configuration;
+  absent metrics are not evidence that there were no GC pauses.
+  Both legacy `hasPrometheus` and newer `PrometheusSimple` backends are discovered.
+
+Scrapes are best-effort, capped at two seconds per endpoint. Sampling interval is the
+sleep between snapshots; collecting a snapshot adds some time. These are diagnostic
+counters, not an automatic overload detector or a reason to relax node protection.
+
+## Controlled reruns
+
+Reduce concurrency explicitly, without changing repository defaults:
+
+```sh
+TEST_THREADS=2 MAX_TESTS_PER_CLUSTER=2 MONITOR_INTERVAL=10 ./runner/regression.sh
+```
+
+To rerun selected tests on an existing dev cluster, source its environment and use
+`MAX_TESTS_PER_CLUSTER=2 pytest -n 2 ...`. Keep the original workload and lower-load
+results separate; a passing lower-load run does not prove the contention was fixed.
+
+Read-only query timeouts are separate from submission rejections. Do not retry an
+entire transaction-building operation or extend CLI timeouts indiscriminately. A
+transport timeout can leave submission status unknown; existing confirmation logic
+must resolve it before any resubmission.

cardano_node_tests/tests/tests_plutus_v2/test_mint_secp256k1_build.py

@@ -2,7 +2,6 @@
 
 import logging
 import pathlib as pl
-import typing as tp
 
 import allure
 import pytest
@@ -32,52 +31,62 @@ def payment_addrs(
     cluster_manager: cluster_management.ClusterManager,
     cluster: clusterlib.ClusterLib,
 ) -> list[clusterlib.AddressRecord]:
-    """Create new payment address."""
+    """Cache payment and issuer addresses per worker/cluster, refilling the source as needed."""
     addrs = addrs_common.get_payment_addrs(
         name_template=common.get_test_id(cluster),
         cluster_manager=cluster_manager,
         cluster_obj=cluster,
         num=2,
         fund_idx=[0],
         amount=3_000_000_000,
+        min_amount=300_000_000,
+        caching_key=helpers.get_current_line_str(),
     )
     return addrs
 
 
+@pytest.fixture
+def issuer_funding(
+    cluster: clusterlib.ClusterLib,
+    payment_addrs: list[clusterlib.AddressRecord],
+) -> tuple[list[clusterlib.UTXOData], list[clusterlib.UTXOData]]:
+    """Fund fresh minting inputs and collateral before testing script rejection."""
+    minting_cost = plutus_common.compute_cost(
+        execution_cost=plutus_common.MINTING_V2_REF_COST,
+        protocol_params=cluster.g_query.get_protocol_params(),
+    )
+    mint_utxos, collateral_utxos, *_ = mint_build._fund_issuer(
+        cluster_obj=cluster,
+        temp_template=common.get_test_id(cluster),
+        payment_addr=payment_addrs[0],
+        issuer_addr=payment_addrs[1],
+        minting_cost=minting_cost,
+        amount=200_000_000,
+    )
+    return mint_utxos, collateral_utxos
+
+
 class TestSECP256k1:
-    def _fund_issuer_mint_token(
+    """Test SECP256k1 minting with valid and invalid signature vectors."""
+
+    def _mint_token(
         self,
         cluster_obj: clusterlib.ClusterLib,
         temp_template: str,
         payment_addrs: list[clusterlib.AddressRecord],
+        issuer_funding: tuple[list[clusterlib.UTXOData], list[clusterlib.UTXOData]],
         script_file: pl.Path,
         redeemer_file: pl.Path,
-    ):
-        """Fund the token issuer and mint a token."""
-        __: tp.Any  # mypy workaround
+    ) -> None:
+        """Mint a token using previously funded inputs and collateral."""
         payment_addr = payment_addrs[0]
         issuer_addr = payment_addrs[1]
 
         lovelace_amount = 2_000_000
         token_amount = 5
-        script_fund = 200_000_000
-
-        minting_cost = plutus_common.compute_cost(
-            execution_cost=plutus_common.MINTING_V2_REF_COST,
-            protocol_params=cluster_obj.g_query.get_protocol_params(),
-        )
-
-        mint_utxos, collateral_utxos, __, __ = mint_build._fund_issuer(
-            cluster_obj=cluster_obj,
-            temp_template=temp_template,
-            payment_addr=payment_addr,
-            issuer_addr=issuer_addr,
-            minting_cost=minting_cost,
-            amount=script_fund,
-        )
-
-        # Step 2: mint the "qacoin"
+        mint_utxos, collateral_utxos = issuer_funding
 
+        # Build the script transaction using only this test's funding outputs.
         policyid = cluster_obj.g_transaction.get_policyid(script_file)
         asset_name = f"qacoin{clusterlib.get_rand_str(4)}".encode().hex()
         token = f"{policyid}.{asset_name}"
@@ -139,6 +148,7 @@ def test_use_secp_builtin_functions(
         self,
         cluster: clusterlib.ClusterLib,
         payment_addrs: list[clusterlib.AddressRecord],
+        issuer_funding: tuple[list[clusterlib.UTXOData], list[clusterlib.UTXOData]],
         algorithm: str,
         plutus_version: str,
     ):
@@ -165,10 +175,11 @@ def test_use_secp_builtin_functions(
         redeemer_file = redeemer_dir / "positive.redeemer"
 
         try:
-            self._fund_issuer_mint_token(
+            self._mint_token(
                 cluster_obj=cluster,
                 temp_template=temp_template,
                 payment_addrs=payment_addrs,
+                issuer_funding=issuer_funding,
                 script_file=script_file,
                 redeemer_file=redeemer_file,
             )
@@ -215,13 +226,14 @@ def test_negative_secp_builtin_functions(
         self,
         cluster: clusterlib.ClusterLib,
         payment_addrs: list[clusterlib.AddressRecord],
+        issuer_funding: tuple[list[clusterlib.UTXOData], list[clusterlib.UTXOData]],
         test_vector: str,
         algorithm: str,
         plutus_version: str,
     ):
-        """Try to mint a token with invalid test vectors.
+        """Reject invalid signature vectors after successfully funding the issuer.
 
-        Expect failure.
+        Funding failures must surface as setup errors, not expected script rejections.
         """
         temp_template = common.get_test_id(cluster)
 
@@ -241,16 +253,19 @@ def test_negative_secp_builtin_functions(
 
         before_pv8 = clusterlib_utils.get_protocol_version(cluster_obj=cluster) < 8
 
+        # Capture only the script operation, never the successful funding setup.
         with pytest.raises(clusterlib.CLIError) as excinfo:
-            self._fund_issuer_mint_token(
+            self._mint_token(
                 cluster_obj=cluster,
                 temp_template=temp_template,
                 payment_addrs=payment_addrs,
+                issuer_funding=issuer_funding,
                 script_file=script_file,
                 redeemer_file=redeemer_file,
             )
 
         exc_value = str(excinfo.value)
+        assert "MempoolTxTooSlow" not in exc_value, exc_value
 
         # Before protocol version 8 the SECP256k1 is blocked.
         # After that the usage is limited by high cost model.
@@ -269,7 +284,8 @@ def test_negative_secp_builtin_functions(
         )
 
         if before_pv8:
-            assert is_forbidden or is_overspending, exc_value
+            with common.allow_unstable_error_messages():
+                assert is_forbidden or is_overspending, exc_value
         # From protocol version 8 the SECP256k1 functions are allowed and
         # when we provide wrong data meaningful error messages are expected.
         elif plutus_version == "v2":
@@ -282,18 +298,24 @@ def test_negative_secp_builtin_functions(
                 "no_pubkey": "Invalid verification key",
                 "no_sig": "Invalid signature",
             }
-            assert expected_error_messages[test_vector] in exc_value, exc_value
+            with common.allow_unstable_error_messages():
+                assert expected_error_messages[test_vector] in exc_value, exc_value
         elif plutus_version == "v3":
-            # PT5: PlutusTx.Prelude.check: input is 'False'. The untyped script we use for
-            # PlutusV3 is different from the PlutusV2 script, and doesn't produce the same error
-            # messages.
+            # These untyped V3 scripts fail in the CEK machine, rather than emitting
+            # PlutusTx.Prelude.check's PT5 trace. Malformed inputs also report builtin logs.
+            validation_error = (
+                "The machine terminated because of an error, "
+                "either from a built-in function or from an explicit use of 'error'."
+            )
             expected_error_messages = {
-                "invalid_sig": "PT5",
-                "invalid_pubkey": "PT5",
-                "no_msg": ("Invalid message hash" if algorithm == "ecdsa" else "PT5"),
-                "no_pubkey": "PT5",
-                "no_sig": "PT5",
+                "invalid_sig": validation_error,
+                "invalid_pubkey": validation_error,
+                "no_msg": ("Invalid message hash" if algorithm == "ecdsa" else validation_error),
+                "no_pubkey": "Invalid verification key",
+                "no_sig": "Invalid signature",
             }
+            with common.allow_unstable_error_messages():
+                assert expected_error_messages[test_vector] in exc_value, exc_value
         else:
             _msg = f"Unsupported plutus version: {plutus_version}"
             raise ValueError(_msg)