This is the main repository of the benchmark accompanying the 'Asymmetric Grid Quorum Systems for Heterogeneous Processes' paper, to be published at CCS 2026. For instructions on how to run the benchmark yourself, see the section below.
For organizational reasons, the benchmark artifact consists of multiple repositories. These are:
| Repository | Tag(s) | Live URL | Purpose |
|---|---|---|---|
tenderload |
ccs26 |
cryptobern/tenderload | A benchmarking pipeline, containing tools to provision and configure infrastructure, apply workloads to a CometBFT deployment, collect performance metrics, and analyze the results. |
tendergrid |
ccs26-baseline, ccs26-grid |
cryptobern/tendergrid | A fork of a recent stable version of CometBFT, changed to optionally support asymmetric quorum systems |
tendercounter |
ccs26 |
cryptobern/tendercounter | A replicated application running atop the CometBFT platform ('ABCI app'), which implements a basic repliacted counter. |
asymmetric-quorums |
ccs26 |
cryptobern/asymmetric-quorums | A Go library, providing methods to analyze and interact with asymmetric quorum systems. |
Each of these repositories has one or multiple tags, which identify the
versions used to produce the results of the paper, respectively those to be
used for artifact evaluation. For most repositories there is only a single
ccs26 tag. For tendergrid there are two tags - ccs26-baseline and
ccs26-grid - denoting the unmodified version of CometBFT serving as a
baseline, respectively the modified version with support for asymmetric quorum
systems.
A bundle of all these repositories is also available as a single artifact on
Zenodo. That artifact consist of five top-level directories
(respectively zip archives): One for each of tenderload, tendercounter and
asymmetric-quorums, and one for each of the two versions of tendergrid.
This repository consists of multiple components, each in its own top-level
directory. Each directory contains its own README.md with more detailed
instructions. These components are:
infrastructure/: A set of scripts to automatically create, and delete, a set of VMs on our own Proxmox-based virtualization platform. Unlikely to be useful for anyone else.ansible/: A set of Ansible playbooks, to automatically configure (install software, deploy configuration files etc) a set of benchmark hosts, as well as deploy and run the benchmarking software.client/: A small Go application using thecometbft-load-testframework. Functions as a driver for the benchmarks.benchmark/: Raw benchmark results used for the publication.plotting/: A Python script for analyzing and plotting benchmark results.
In this section we aim to document how to run the benchmark yourself, to replicate our results. We start by summarizing required software and hardware, before walking you through the process, step by step.
We assume that you have read at least the abstract and chapters 6 and 7 of the paper, so that you are familiar with our goal, as well as the high-level architecture.
To run the benchmark, you will need:
- A service on which you can host Linux VMs. This can be a cloud provider such as AWS or DigitalOcean, or a local hypervisor running e.g. Proxmox or Hyper-V.
- To configure the VMs and run the benchmarks, an installation of a recent
version of Ansible. Ansible can be installed via
pipif you've got an existing Python development environment, via your system's package manager on many Linux distributions, or even in a Docker container as a so-called 'Ansible Execution Environment'. For more details, check the official installation instructions. We used version14.0.0of Ansible, with version2.21.0of Ansible-core. - Optional: To build the containers containing the benchmark application, a working
Docker development environment. This is best installed via your OS' official
package manager. Otherwise, check the official
documentation on how to install it.
- This requirement is OPTIONAL: We provide pre-built container images on our Dockerhub.
- Optional: To compile the benchmarking client and server, a working Go development
environment. This can be installed via your system's package manager, an
official binary installer, or as an official
Docker image.
- This requirement is OPTIONAL: We provide pre-built binaries on our Github.
- To analyze data and plot it, a working development environment of Python 3. Python 3 is provided as a package on most Linux distributions, as binary installers maintained by the community, or as official Docker containers. It is also advisable to install a utility to manage so-called Python 'virtualenvs', to prevent unwanted interactions with potential OS-wide Python installations. Common tools to do this are the built-in venv, or separate tools such as uv, virtualenv or pew.
Replicating the full results of the paper is rather time-consuming, due to both the setup effort, the time spent waiting for benchmark results, and the (in)stability of the system in some configurations. We suggest to attempt replication of the following, representative, subset of results:
- Consider only the deployment with
n = 28nodes, not the one withn = 49 - Consider only the CometBFT baseline, not the modification with support for asymmetric quorum systems
- Consider only a subset of message invocation rates, e.g.
200, 500, 1000, 1500, 2000.
Doing this should demonstrate the workings of the benchmarking pipeline, as well as confirm some of the results of the paper.
There is a rare liveness bug in our modification of CometBFT, which can cause the system to become unresponsive, which we have not managed to fix in time. If you choose to benchmark the version of CometBFT with support for asymmetric trust there is, especially for high invocation rates, a chance you encounter this bug. This will be evident by throughput dropping to zero, all of a sudden.
If this does happen, you will have to reset the system, and restart the benchmark from the last-known good point. Later sections of this document will provide more information on how to do so.
To start with, you require a set of VMs on which to run the benchmark. We heavily recommend to use VMs running Debian 12, as this is the OS on which we performed our benchmark. We recommend to dedicate one of these VMs to running the benchmark client, while the others will be running CometBFT. This will minimize unwanted interactions between the two systems.
Important: In our automated deployment, CometBFT nodes will use the hostname of the VM they run on to infer which node they are. This imposes certain requirements which your VMs' hostnames must fulfill:
- The hostname of each VM must consist of an all-lowercase letter-only part, followed by a hyphen, followed by a number
- The number part must be an incrementing sequence, starting at
0. Numbers can optionally be zero-padded to the left.
If possible, we recommend to set the hostnames of your VMs to tenderload-00,
tenderload-01, and so on, though any name is permitted as long as it follows
the requirements above. If you cannot set the hostname of your VMs in such a
way, we will later show you how to define the mapping of VMs to CometBFT nodes:
Be sure to read the Manually mapping VMs to CometBFT nodes section further
down.
If you stick to this naming scheme, you might e.g. have tenderload-00 through
tenderload-27 hosting a CometBFT deployment with 28 nodes, and
tenderload-28 hosting the benchmarking client.
In terms of hardware, we recommend specs similar to the ones used in the paper:
- 4 vCPU
- 4 GiB of RAM
- 20 GiB of fast, SSD-backed, storage
In terms of numbers, smaller deployments are much easier to run the benchmark on. Similarly, we advise to run the VMs in a LAN-like setting, e.g. have them all running in the same data center, rather than spread throughout the world. This also matches what was done for the paper.
VMs are required to have IPv4 connectivity, as some of the CometBFT-internal tools used for the benchmark do not yet support IPv6 (in 2026, we know...).
You can further simplify the setup by ensuring that the VMs used to host CometBFT have sequential IPs. This is easy to do in an internal network you control, less so on a cloud provider. If you cannot ensure that they have sequential IPs, there will be instructions later on on how to adapt the generated configuration files.
The VMs should be able to communicate with each other via reliable TCP links. If network firewalls are involved, the following ports must be opened:
- CometBFT VMs must be able to talk with each other on port 26656 TCP for P2P traffic
- The benchmarking VM must be able to talk with CometBFT VMs on port 26657 TCP for RPC traffic
At this point, prepare a list with the IPv4 addresses (not DNS names) of all VMs which will be running CometBFT (i.e. excluding the VM dedicated to running the benchmark client).
With VMs provisioned, ensure you can connect to each of them via SSH without needing to enter your password manually. This is easiest done by deploying your SSH public key onto those VMs. This is likely supported out of the box by whatever software or service you use to provision the VMs.
It is further recommended to configure your SSH client such that you need not
manually specify the user to use to connect to the VMs. This can be done by
e.g. adding suitable entries to your .ssh/config on UNIX systems. If you
cannot do so, you will have to later configure Ansible to use the appropriate
SSH user.
Lastly it is convenient to configure your SSH client to accept the VM's public
key on first use, rather than asking you whether it should be trusted, as this
interactivity would not be supported by Ansible. For OpenSSH, this can be done
by setting StrictHostKeyChecking=accept-new. If you cannot do so, connect to
each VM manually at least once, and accept its public key.
For the rest we assume you have provisioned 29 VMs with hostnames
tenderload-00 through tenderload-28. Of these, tenderload-28 is dedicated
to running the benchmarking client, while tenderload-00 through
tenderload-27 will run CometBFT. Public-key-based SSH access to the VMs is
possible as the debian user, as which one can change to the root user by
means of sudo. For such a situation you might e.g. add the following entries
to your .ssh/config:
Host tenderload-00
HostName 10.42.8.167 # CHANGEME
User debian
# Connect via an SSH jumphost if required
# ProxyJump CHANGEME
# Accept public key on first use
StrictHostKeyChecking=accept-new
# ...
Host tenderload-28
HostName 10.42.8.195 # CHANGEME
User debian
# Connect via an SSH jumphost if required
# ProxyJump CHANGEME
# Accept public key on first use
StrictHostKeyChecking=accept-new
As a next step, you must perform basic configuration of the VMs. This will
involve installing tools and dependencies required for the benchmark. For this,
use the provided Ansible playbooks in the ansible/ subdirectory. All commands
in this section should be executed from within that directory, unless otherwise
specified.
As a first step, you must tell Ansible about which VMs it should provision. To
do this, open the inventory.yaml file. In there, you will see two groups of
hosts: tendergrid, which designates the VMs which will be running the
replicated state machine, and client, which designates the VM(s) which will
be running the benchmarking client. You probably only want a single VM to act
as benchmarking client.
The provided inventory.yaml file matches the example use case of using 29
VMs, with tenderload-28 being for the client, and tenderload-00 through
tenderload-27 running the CometBFT state machine. Adjust this to match your
deployment, if required. See the official documentation
on Ansible inventories for further information.
Take note of the ansible_become: true and ansible_become_user: root
directives. They instruct Ansible that, after having connected to these hosts,
it should (ansible_become: true) use sudo to change to
(ansible_become_user: root) root before executing its commands. You may
have to change this, depending on your VMs. This is also where you could
specify ansible_user: CHANGEME to instruct Ansible which user to use to
connect via SSH, if not done so via e.g. your .ssh/config file.
Once done, test connectivity to all your hosts by having Ansible ping them:
ansible -m ping all -i inventory.yaml
This should produce a lot of green SUCCESS outputs. If any of them fails,
check the output, and figure out why, before proceeding.
Before we prepare the VMs for the benchmark, let's take a look at how to
configure benchmark parameters. Open the group_vars/all.yml file.
First, take a look at the benchmark.testnet section. Set the n parameter to
the number of CometBFT VMs (not including the dedicated benchmark client
VM, if you have one!) you are running. Then, set starting_ip to the IP of the
VM with the lowest IP among those. If you do not have sequential IPs, then
comment out the starting_ip directive, and uncomment the cometbft_ips
one. Then, specify the IPs of your CometBFT VMs (again, excluding the one for
the benchmarking client) as a YAML list under that configuration directive.
Next, take a look at benchmark.version and benchmark.image. They specify
the Docker image and version containing CometBFT which will be benchmarked. The
provided benchmark.image points to our public Dockerhub repository, and the
benchmark.version defaults to the baseline version of CometBFT. If you want
to benchmark both the baseline version as well as the version with support for
asymmetric quorums, you will have to change this setting at a later point.
Next, take a look at benchmark.request_rates. This is the list of
invocation rates which the benchmarking client will use. The provided value is
a subset of the ones used in the paper, to allow replicating some of the
results with less of a time commitment.
Lastly, take a look at benchmark.benchmark_time and
benchmark.cooldown_time. They define the time (per request rate) for which to
submit requests, respectively the time to wait in between.
Other settings should not be something you plausibly have to modify.
Note: This is only relevant if your VMs' hostnames do not follow the
requirements outlined earlier, i.e. if they do not consist of part1-part2,
where part1 is a single lowercase word, and part2 a monotonically
incrementing sequence starting at 0.
If you were not able to provision VMs whose hostnames match the requirements
outlined earlier, take a look at cometbft.node_map. Change it, replacing the
automatic string with a manual mapping of VM hostnames to CometBFT
identifiers, as in the example. Recall that CometBFT node identifiers must be a
simple monotonic sequence, starting at 0.
We now apply the base configuration to VMs by executing the configure.yml
playbook. This playbook is idempotent, meaning that you can execute it as many
times as required. If execution fails due to a transient error, it is thus safe
to re-execute it. If execution keeps failing, consult the output to figure out
the cause.
Depending on the specs of the machine from which you execute Ansible, you may want
to parallelize operation using the -f parameter. In the example below, we run
it on 10 VMs in parallel. On a modern notebook, you should be able to easily
configure 20 VMs in parallel. Going higher can risk running into timeouts, as
Ansible cannot keep up.
ansible-playbook configure.yml -i inventory.yaml -f 10
Ensure this playbook terminates without errors before moving on.
Next, we create configuration files as well as keypairs for our CometBFT deployment. This will take place on the VM dedicated to running the benchmarking client. After generation, Ansible will download the created configuration archive to the host on which you are running Ansible from.
First, open create-configuration.yml. Near the top you will find a vars
section. Within, set the download_location to the path (including filename)
to where you want Ansible to download the generated configuration archive. The
default will store the archive in /var/tmp/tendermint_testnet.tar.gz,
suitable for Linux hosts. If on Windows, you may want to change this to e.g.
your download directory.
Then, run this playbook. As you should only have a single benchmarking client,
there is no need for parallel execution via -f this time. If you happen to
have more than one VM dedicated as benchmarking client, be sure to limit
execution to only a single one using -l, to prevent bugs due to race
conditions.
Important: This will overwrite your past configuration, if present. Be sure to back it up, if you still require access to it.
ansible-playbook create-configuration.yml -i inventory.yaml
If you are benchmarking the CometBFT baseline, you can skip this section.
If you are benchmarking CometBFT with support for asymmetric grid quorum systems, you will need to provide each node with the quorum system to use. You could create those by hand, but we do provide a utility to generate those.
To do so, you must first find out the unique identifier which has been assigned
to each CometBFT node, during config generation. These are derived from their
public key, and can be found in the generated configuration files. Thus, first
open the .tar.gz archive generated in the previous step. You may want to do
this in a dedicated directory, as it will contain one subdirectory for each
node of the deployment. If on Linux, this may e.g. look something like this:
cd /var/tmp
mkdir tendermint_configuration/
# Extract configuration archive into dedicated directory
tar -x -f tendermint_testnet.tar.gz -z -v -C tendermint_configuration/
Change to the directory where you extracted the archive. Note that there will
be one directory for each node. For what we are about to do, pick any of them,
e.g. node0. Within, there is a config directory with the node's keys, chain
genesis information, and the node's config file, config.toml. Open this file,
and look for the persistent_peers directive.
This directive is a single string, consisting of a list of comma-separated
peers the node knows about. Observe that each entry in that list consists of a
hexadecimal identifier, followed by an @ symbol, followed by an IP where that
node can be found. These hexadecimal identifiers are the ones we care about,
extract them, and write them, one per line, to a dedicated file.
For convenience, we provide a Python script to extract these identifiers from a
configuration archive in utilities/extract-node-identifiers.py in the root of
this repository. Pass it the path to the .tar.gz archive as a parameter, and
it will print the list of extracted identifiers to STDOUT:
# From the root of this repository; not the `ansible` subdirectory:
python utilities/extract-node-identifiers.py /var/tmp/tendermint_testnet.tar.gz > /var/tmp/node_identifiers.csv
cat /var/tmp/node_identifiers.csv
With the identifiers at hand, change to the asymmetric-quorums repository.
First, ensure that the generator binary can be built. For this you require a
working Go development environment:
go build ./cmd/generator
go run ./cmd/generator/ grid --help
Now you have to come up with (fictional) attributes in which your processes
differ. Recall from the paper that we aim to arrange our nodes in a
k-dimensional grid of attributes, with one node at each position. You thus have
to come up with a set of attributes such that the product of their
cardinalities is equal to n, the number of nodes.
If you use n = 28 and want to replicate the results of the paper, you can e.g. use the two attributes:
- OS: Windows, Debian, FreeBSD, Mac OS
- Location: CH, IT, DE, AT, FR, UK, NL
Using this example, and assuming your file with the extracted node identifiers
is at /var/tmp/node_identifiers.csv, you would use the generator as follows:
go run ./cmd/generator/ grid --identities=/var/tmp/node_identifiers.csv "OS=Windows,Debian,FreeBSD,Mac OS" "Location=CH,IT,DE,AT,FR,UK,NL" > /var/tmp/grid_quorum.json
The generator will spit out a JSON-encoded quorum specification to STDOUT. Redirect it to a file of your choice, as above.
Now it is time to push the generated configuration onto the CometBFT nodes. Crucially, this will reset the deployed state machine, meaning all state is lost. You can repeat this process to reset the deployed system if it ever seems to get stuck.
First, open deploy-configuration.yml. Again near the top, locate the vars
section. If you changed the path to where you downloaded the configuration file
archive in the previous section, you will have to adjust this accordingly,
under vars.testnet_configuration.
If you are benchmarking CometBFT with an asymmetric grid quorum system, you will have to configure the CometBFT nodes to use that system. To do so, change two variables in this playbook:
- Set
inject_grid_quorum_systemtotrue - Set
grid_quorum_specificationto the path where the previously-generated quorum specification has been stored. If you are instead benchmarking the CometBFT baseline, be sure thatinject_grid_quorum_systemis set tofalse.
Now, run the playbook to deploy the configuration:
ansible-playbook deploy-configuration.yml -i inventory.yaml -f 10
Note: Deploying the configuration will also reset each node's state. This is an easy way to reset the network, should it end up stuck.
Now it is time to start the CometBFT network. To do so, execute the
start-network.yml playbook:
ansible-playbook start-network.yml -i inventory.yaml -f 10
To verify that it has been started successfully, connected to one or two of the
CometBFT hosts via SSH, and become root. First, verify that both the
cometbft as well as abci-app containers are running, and healthy:
root@tenderload-02:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
de7542dd6dad cryptobern/tendergrid:ccs26-baseline "/usr/bin/cometbft n…" 26 seconds ago Up 24 seconds 0.0.0.0:26656-26657->26656-26657/tcp, :::26656-26657->26656-26657/tcp cometbft
2c82e5f08ed9 cryptobern/tendercounter:ccs26 "/usr/bin/tendercoun…" 29 seconds ago Up 27 seconds abci-app
Then, take a look at the logs of the cometbft container. You should see that
it is actively producing blocks, at a rate of roughly one per second. The
example below shows production of the block at height 71:
I[2026-06-19|08:33:29.591] received proposal module=consensus proposal="Proposal{71/0 (D0833D6AB58581C296D0D4E605047979514D5B938433987DD4E86A84B5642B51:1:A32B791A6492, -1) F4FA9907BCA2 @ 2026-06-19T08:33:29.488053057Z}" proposer=9C460B7B74263FB4CFA77672F08C8B52E14DAD41
I[2026-06-19|08:33:29.591] received complete proposal block module=consensus height=71 hash=D0833D6AB58581C296D0D4E605047979514D5B938433987DD4E86A84B5642B51
I[2026-06-19|08:33:29.810] finalizing commit of block module=consensus height=71 hash=D0833D6AB58581C296D0D4E605047979514D5B938433987DD4E86A84B5642B51 root= num_txs=0
I[2026-06-19|08:33:29.818] finalized block module=state height=71 num_txs_res=0 num_val_updates=0 block_app_hash=
I[2026-06-19|08:33:29.818] executed block module=state height=71 app_hash=
I[2026-06-19|08:33:29.822] committed state module=state height=71 block_app_hash=
I[2026-06-19|08:33:29.826] indexed block events module=txindex height=71
If you check the logs of the abci-app container, you should see that the
replicated counter is stuck at value 0. This is expected, as there are
currently no commands being sent to the replicated applications - the produced
blocks are empty:
2026/06/19 08:34:48 Commit: Persisted value of counter. Is now 0
2026/06/19 08:34:49 Commit: Persisted value of counter. Is now 0
As a last preparation, you must compile and deploy the benchmarking client. To
do so, change to the client/ subdirectory in this repository's root, and
build the contained Go application. This should lead to creation of a
tenderload binary in the same directory:
cd client/
go build .
./tenderload --help
For convenience, we provide a built version of this binary on Github.
Now, place this binary in /root/tenderload of the VM dedicated to running the
benchmarking client. If you followed our naming scheme, this will be
tenderload-28.
scp tenderload tenderload-28:/tmp/
ssh tenderload-28 sudo mv /tmp/tenderload /root/tenderload
To start the benchmarking client, connect to the VM dedicated to run it. If you
followed our naming example, this will be tenderload-28. There, change to
root. Verify that there is a benchmark-results/ directory in /root, as
well as a benchmark-runner.py script.
root@tenderload-28:~# ls -l /root/
total 20
drwx------ 2 root root 4096 Jun 18 12:46 benchmark-results
-rwx------ 1 root root 4294 Jun 19 08:21 benchmark-runner.py
-rwx------ 1 root root 1171 Jun 18 14:25 confgen.sh
drwx------ 3 root root 4096 Jun 18 12:27 snap
During execution, the benchmark client will submit requests at different invocation rates to the replicated application. For each invocation rate, it will write separate output files. The names of these output files will be made out of several components:
- The version of the benchmark container which is deployed, e.g.
ccs26-baseline - The number of peers in the system, e.g.
n_28 - The invocation rate in requests per second, e.g.
300 - An additional numerical identifier you have to choose when you start the benchmark
This last identifier allows you to e.g. have multiple independent measurements
for the same set of parameters. To configure it, you have to set the
BENCHMARK_SERIES environment variable when executing the runner script.
During execution, the benchmark client will go through the configured
invocation rates one-by-one. For each rate, it will send requests to the
CometBFT deployment at that rate for five minutes. In between such runs it will
sleep for 30 seconds, to let the system cool off. This means that if you
configured it to e.g. benchmark four different invocation rates, a full
benchmark run will take slightly north of 20 minutes. For this reason it may be
advisable to start the benchmark runner within e.g. screen or tmux, to
prevent interrupting it should your connection to the server fail.
With that said, time to execute the benchmark runner:
BENCHMARK_SERIES=1 python3 benchmark-runner.py
While it is running, observe the logs of the cometbft and abci-app
containers. You should see:
- That CometBFT is continuing to produce blocks, at a potentially faster rate.
- That the value of the replicated counter is increasing.
It may also be interesting to observe load on the CometBFT VMs. Standard
utilities such as htop come pre-installed.
After the benchmark run has finished, consult the contents of the
/root/benchmark-results/ directory. For each invocation rate, you should see:
- A logfile, containing the full output of the CometBFT benchmarking tool
- A CSV file, containing the metrics recorded by the CometBFT benchmarking tool
Further there should be a single JSON file, containing among others the deduced execution rate (as opposed to invocation rate) for all executions. This is the file containing the bulk of our benchmark results:
root@tenderload-28:~# ls -l benchmark-results/
total 948
-rw-r--r-- 1 root root 189 Jun 19 09:08 ccs26-baseline_n_28_1000_1.csv
-rw-r--r-- 1 root root 235301 Jun 19 09:08 ccs26-baseline_n_28_1000_1.log
-rw-r--r-- 1 root root 187 Jun 19 09:04 ccs26-baseline_n_28_250_1.csv
-rw-r--r-- 1 root root 233648 Jun 19 09:04 ccs26-baseline_n_28_250_1.log
-rw-r--r-- 1 root root 188 Jun 19 09:06 ccs26-baseline_n_28_500_1.csv
-rw-r--r-- 1 root root 235300 Jun 19 09:06 ccs26-baseline_n_28_500_1.log
-rw-r--r-- 1 root root 189 Jun 19 09:07 ccs26-baseline_n_28_750_1.csv
-rw-r--r-- 1 root root 235300 Jun 19 09:07 ccs26-baseline_n_28_750_1.log
-rw-r--r-- 1 root root 848 Jun 19 09:08 ccs26-baseline_n_28_metrics_1.json
You may want to inspect the file to get an intuition on its structure:
jq < benchmark-results/ccs26-baseline_n_28_metrics_1.json
It can happen that the benchmark results are only partially useful, because the system e.g. crashed halfway through. The easiest fix in this case is to simply delete the offending output files, and restart the benchmark run.
However, this does discard some results which are perfectly usable. It may thus instead be desirable to salvage those parts of the measurement which worked fine, and only repeat those which did not. Especially for large benchmark runs this can save a lot of time.
To do so, first figure out at which point the system broke. Let's assume in the
case above that the results for invocation rates 250 and 500 were fine,
whereas those for invocation rates 750 and 1000 are faulty because some VMs
crashed. We first remove the output files of the offending results:
root@tenderload-28:~# rm benchmark-results/ccs26-baseline_n_28_500_1.{csv,log}
root@tenderload-28:~# rm benchmark-results/ccs26-baseline_n_28_1000_1.{csv,log}
We also have to remove the relevant results from the JSON file written by the benchmark runner. To do so, it's easiest to first nicely format the JSON file:
jq < benchmark-results/ccs26-baseline_n_28_metrics_1.json | sponge benchmark-results/ccs26-baseline_n_28_metrics_1.json
Now, it's easy to edit the file, and remove those JSON objects with a
target_rate of 750 respectively 1000. Don't forget to remove the trailing
comma after the last array element!
# Edit the file
vim benchmark-results/ccs26-baseline_n_28_metrics_1.json
# Verify it's still a valid JSON file
jq < benchmark-results/ccs26-baseline_n_28_metrics_1.json
With faulty data purged, we now configure the benchmark runner to only re-run
those invocation rates which failed. You could do so by editing
group_vars/all.yml and re-running the configure.yml playbook.
Alternatively, simply open benchmark-runner.py in your favorite text editor,
and look for the TARGET_RATES list near the top. Change it to only contain
those invocation rates you wish to benchmark - in our case, that'd be
TARGET_RATES = [750, 1000].
Now, re-run the benchmark with a new BENCHMARK_SERIES environment
variable, so as not to overwrite the good data:
BENCHMARK_SERIES=2 python3 benchmark-runner.py
After this, you will have an additional set of output files in the
benchmark-results/ directory, with which you can proceed.
Don't forget to reset the benchmark runner script to a clean state before the
next run. An easy way to achieve this is to simply re-run the configure.yml
playbook.
If the network ends up in a stuck state, you may want to reset it to a clean state. To do this:
- Execute the
stop-network.ymlplaybook to kill all containers - Execute the
deploy-configuration.ymlplaybook to cleanly re-deploy configuration files, and reset nodes to the genesis state - Execute the
start-network.ymlplaybook to start containers anew.
If you want to instead change benchmark parameters, such as invocation rates or even the version of CometBFT or number of peers you are benchmarking:
- Execute the
stop-network.ymlplaybook to kill all containers - Make your changes to
group_vars/all.yml - If you changed the number of peers, you have to fully regenerate the
configuration as described above, using the
create-configuration.ymlplaybook. This also includes regenerating the asymmetric quorum system! - Deploy the (potentially new) configuration and benchmark containers using
deploy-configuration.yml - Once more start the network using
start-network.yml
Once you have collected the data you want, download the full
/root/benchmark-results directory to your local machine, where analysis of
the data will happen. There's many ways to do it, the example below shows how
to do so by creating an archive on tenderload-28, then downloading and
extracting it to your local machine.
In whatever way you choose to download the files, ensure that they all end up
in the benchmark/ top-level directory of this repository.
# Assuming we execute this from the root of this repository
# Create archive and download
ssh tenderload-28 sudo tar cfvz /tmp/benchmark_results.tar.gz /root/benchmark-results
scp tenderload-28:/tmp/benchmark_results.tar.gz .
# Open archive and move to appropriate place in this repository
tar xfvz benchmark_results.tar.gz
mv root/benchmark-results/* benchmark/
# Cleanup
rm -r root/
rm benchmark_results.tar.gz
To make it more clear which data is yours, and which data is the one used for
the paper's results, you may want to remove the existing contents of the
benchmark/ directory, or move them to a separate directory.
Finally it's time to plot the results. For this, change into the plotting/
top-level directory of this repository. Plotting is done in Python, and uses a
handful of dependencies which are not shipped as part of the Python standard
library. Create a Python virtualenv, and install the dependencies listed in
requirements.txt.
Note: If installation of these dependencies fails, consult the documentation of Pandas 2.3, Matplotlib 3.10 and Numpy 2.4 for further information.
Next you need to configure some details of the plotting process. Open
plotting/plot.py.
- Ensure that
DATA_ROOTis the directory, relative toplot.py, containing your benchmark data - Ensure that
DATA_SERIES_FILENAME_PATTERNmatches the JSON files containing your benchmark data. If you used our pre-made benchmark images, this will be fine as-is. - Adjust
FIGURE_ROOTto point to the directory, relative toplot.py, into which to store the rendered figures - Configure
INVOCATION_RATE_RANGESto specify, for the deployment sizes you evaluated, the range of invocation ranges for which you ran the benchmark. The setting in the providedplot.pymirrors what was used in the paper, you will likely have to adapt it if you only reproduced parts of the results.
Finally, run the plotting script:
python plot.py
If all went well, you can now look at the figures in the plotting/renders
directory, which will be named n_${number_of_nodes_in_deployment}.png.