API evolution benchmark

Measures how accurately Java breaking-change detection tools identify source- and binary-incompatible API changes, using a Java 25 compiler/linker-derived ground truth.

Evaluated tools

• Roseau 0.7.0
• japicmp 0.26.2
• Revapi 0.28.4

Datasets

• Jezek (310 cases): presented in API Evolution and Compatibility: A Data Corpus and Tool Evaluation by Jezek and Dietrich. Manually fixed some buggy cases and significantly strengthened the clients to address false negatives.
• Roseau (423 cases): the cases are automatically extracted from Roseau's test suite

Results

Dataset	Category	Metric	Roseau	japicmp	Revapi
Jezek (310 cases)	Breaking	Precision	0.98	0.89	0.82
		Recall	1.00	0.83	0.97
		F1	0.99	0.86	0.89
	Source	Precision	0.90	0.78	0.74
		Recall	1.00	0.81	0.96
		F1	0.95	0.80	0.84
	Binary	Precision	0.95	0.91	0.92
		Recall	1.00	1.00	0.97
		F1	0.98	0.95	0.94
Roseau (423 cases)	Breaking	Precision	0.99	0.70	0.75
		Recall	0.99	0.85	0.91
		F1	0.99	0.77	0.82
	Source	Precision	0.99	0.65	0.71
		Recall	0.99	0.84	0.89
		F1	0.99	0.73	0.79
	Binary	Precision	0.84	0.75	0.70
		Recall	1.00	0.98	0.95
		F1	0.91	0.85	0.80

How it works

Each case consists of a baseline API (v1), an updated version of that API with a single change introduced (v2), and a client with a main() method that uses baseline symbols.

The ground truth is derived automatically:

1. v1 and v2 are each compiled and packaged into a JAR.
2. The client is compiled and packaged against v1.
3. The client sources are recompiled against v2 — a compiler error marks the case source-incompatible.
4. The client JAR is executed against v2 — a linkage error marks the case binary-incompatible.

Each tool is then given only the two API JARs (never the client) and must report whether the change is source- and/or binary-breaking. Verdicts are compared per case against the ground truth to produce precision, recall, and F1.

Running it

Requires JDK 25 and Maven 3.9+.

cd harness && mvn -DskipTests package && mvn exec:java

This benchmarks every tool on every dataset. To run a single one, name it:

mvn exec:java -Dexec.args=jezek

Each dataset writes two CSVs to results/<dataset>/:

• results-by-case.csv — one row per case: the ground truth, each tool's verdict, and whether it was correct.
• results-by-tool.csv — precision/recall/F1 per tool, for the breaking, source, and binary scopes.

Caveats

• Only syntactic (source/binary) compatibility is evaluated; behavioral and semantic changes are out of scope.
• A case flagged as breaking is definitely breaking. The converse is weaker: a case may be non-breaking only because the corpus lacks a client that would have exposed the break.
• The library and the client live in different packages, so the benchmark treats package-private symbols as outside the API.
• The benchmark evaluates whether the tools identify some breaking changes with the right compatibility level (source or binary). However, it does not evaluate whether the breaking change kind reported by the tools (e.g., CLASS_NOW_FINAL) indeed corresponds to the case.