Optimize DataSetWithoutTimeGenerator with primitive long heap/set by hongzhi-gao · Pull Request #906 · apache/tsfile

hongzhi-gao · 2026-08-17T03:17:51Z

Summary

Optimize the multi-way merge hot path in DataSetWithoutTimeGenerator by replacing PriorityQueue<Long> / HashSet<Long> with trimmed, fastutil 8.5.8–derived primitive containers, avoiding autoboxing of timestamps and the related CPU/GC cost.

CPU profiles showed the hotspot on HashSet.contains → HashMap.getNode via timeHeapPut, which is hit frequently in workloads such as Pipe extraction.

img_v3_0214g_8b329c57-96ab-4599-9fd7-6e86c133b31g

Changes

Add LongHeapPriorityQueue and LongOpenHashSet (forked from fastutil, APIs trimmed to what TsFile needs; key methods annotated with upstream origins)
Switch DataSetWithoutTimeGenerator to these containers, with capacity pre-sized by series count
Update root NOTICE for the fastutil-derived code and copyrights
Add regression tests:
- DataSetWithoutTimeGeneratorTest: sparse multi-way merge and cross-batch continuation
- LongOpenHashSetAndHeapPriorityQueueTest: heap/set correctness (including edge cases), parity vs boxed JDK collections, combined dedup-merge pattern, and a micro-benchmark

Performance results

Micro-benchmark of the same heap+set dedup-merge pattern used by DataSetWithoutTimeGenerator (50 series × 100,000 rows), comparing JDK PriorityQueue<Long>+HashSet<Long> with the primitive implementation:

Implementation	Time	Speedup
boxed (`PriorityQueue` + `HashSet`)	~40–55 ms	1.0x
primitive (`LongHeapPriorityQueue` + `LongOpenHashSet`)	~9 ms	~4.5x–6.0x

Notes: micro-benchmarks vary with machine load; local runs typically land around ~2x–6x. Absolute gain depends on series count and total points. Correctness tests vs boxed behavior all passed.

Test plan

mvn -pl tsfile test -Dtest=LongOpenHashSetAndHeapPriorityQueueTest,DataSetWithoutTimeGeneratorTest
Review fastutil fork scope and LICENSE compliance
(Optional) Re-profile IoTDB Pipe extraction and confirm the HashSet.contains hotspot shrinks

Replace PriorityQueue<Long>/HashSet<Long> with fastutil-derived LongHeapPriorityQueue/LongOpenHashSet to avoid boxing on the multi-way merge hot path. Update NOTICE and add regression tests.

Copilot

Pull request overview

This PR optimizes the multi-way merge hot path in DataSetWithoutTimeGenerator by replacing boxed timestamp containers (PriorityQueue<Long> / HashSet<Long>) with fastutil-derived primitive long heap/set implementations to reduce autoboxing overhead and GC pressure in query workloads.

Changes:

Introduces fastutil 8.5.8–derived primitive containers: LongHeapPriorityQueue and LongOpenHashSet.
Switches DataSetWithoutTimeGenerator to use the primitive heap/set, pre-sized by series count.
Adds NOTICE attribution and regression tests covering merge correctness, data-structure correctness, and a (currently unguarded) micro-benchmark.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
NOTICE	Adds required attribution for fastutil-derived code.
java/tsfile/src/main/java/org/apache/tsfile/utils/LongOpenHashSet.java	New primitive open-addressing `long` hash set (trimmed from fastutil).
java/tsfile/src/main/java/org/apache/tsfile/utils/LongHeapPriorityQueue.java	New primitive `long` min-heap priority queue (trimmed from fastutil).
java/tsfile/src/main/java/org/apache/tsfile/read/query/dataset/DataSetWithoutTimeGenerator.java	Replaces boxed heap/set with primitive implementations and pre-sizes them.
java/tsfile/src/test/java/org/apache/tsfile/read/query/dataset/DataSetWithoutTimeGeneratorTest.java	Adds regression tests for sparse multi-way merge and cross-batch continuation.
java/tsfile/src/test/java/org/apache/tsfile/utils/LongOpenHashSetAndHeapPriorityQueueTest.java	Adds correctness tests plus a micro-benchmark/perf smoke test for the new containers.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

hongzhi-gao · 2026-08-17T03:41:57Z

  private void timeHeapPut(long time) {
    if (!timeSet.contains(time)) {
      timeSet.add(time);
-      timeHeap.add(time);
+      timeHeap.enqueue(time);
    }


Agreed, switched to if (timeSet.add(time)) to avoid the double probe.

hongzhi-gao · 2026-08-17T03:42:29Z

+  public void testPrimitiveStructuresPerformance() {
+    final int series = 50;
+    final int rows = 100_000;
+    final long baseTime = 1_700_000_000_000L;
+
+    runBoxed(series, 10_000, baseTime);
+    runPrimitive(series, 10_000, baseTime);
+


Agreed, gated with Assume.assumeTrue behind -Dtsfile.runPerformanceTests=true, same as other TsFile perf tests.

codecov-commenter · 2026-08-17T04:06:30Z

Codecov Report

❌ Patch coverage is 95.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.71%. Comparing base (5681978) to head (17dcc8d).

Files with missing lines	Patch %	Lines
.../java/org/apache/tsfile/utils/LongOpenHashSet.java	93.91%	7 Missing ⚠️
...org/apache/tsfile/utils/LongHeapPriorityQueue.java	98.18%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #906      +/-   ##
===========================================
+ Coverage    60.60%   60.71%   +0.11%     
===========================================
  Files          745      747       +2     
  Lines        49508    49678     +170     
  Branches      7945     7979      +34     
===========================================
+ Hits         30002    30163     +161     
- Misses       18046    18055       +9     
  Partials      1460     1460

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

JackieTien97 · 2026-08-17T06:04:30Z

+
+============================================================================
+
+This product includes code derived from fastutil (http://fastutil.di.unimi.it/):
+  Copyright (C) 2002-2022 Sebastiano Vigna
+  Copyright (C) 2003-2022 Paolo Boldi and Sebastiano Vigna
+  Licensed under the Apache License, Version 2.0


no need to add this

ColinLeeo

LGTM

hongzhi-gao added 2 commits August 17, 2026 11:12

Optimize DataSetWithoutTimeGenerator with primitive long heap/set

3b9cc9f

Replace PriorityQueue<Long>/HashSet<Long> with fastutil-derived LongHeapPriorityQueue/LongOpenHashSet to avoid boxing on the multi-way merge hot path. Update NOTICE and add regression tests.

Fix spotless formatting for LongOpenHashSet and LongHeapPriorityQueue

9137ad8

hongzhi-gao requested review from ColinLeeo and HTHou August 17, 2026 03:22

ColinLeeo requested a lite review from Copilot August 17, 2026 03:26

Copilot started reviewing on behalf of ColinLeeo August 17, 2026 03:26 View session

Copilot AI reviewed Aug 17, 2026

View reviewed changes

Address review: use Set.add return value and gate perf test

fd2bd72

Add fastutil license attribution for derived heap and hash set

ac902c7

JackieTien97 requested changes Aug 17, 2026

View reviewed changes

Remove fastutil attribution from NOTICE

17dcc8d

hongzhi-gao requested a review from JackieTien97 August 17, 2026 06:50

ColinLeeo approved these changes Aug 17, 2026

View reviewed changes

hongzhi-gao merged commit 7798b10 into develop Aug 17, 2026
38 checks passed

Navigation Menu

Uh oh!

Optimize DataSetWithoutTimeGenerator with primitive long heap/set - #906

Optimize DataSetWithoutTimeGenerator with primitive long heap/set#906
hongzhi-gao merged 5 commits into
developfrom
optimize/dataset-without-time-generator-primitive-heap

hongzhi-gao commented Aug 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

hongzhi-gao Aug 17, 2026

Uh oh!

hongzhi-gao Aug 17, 2026

Uh oh!

codecov-commenter commented Aug 17, 2026 •

edited

Loading

Uh oh!

JackieTien97 Aug 17, 2026

Uh oh!

hongzhi-gao Aug 17, 2026

Uh oh!

ColinLeeo left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

hongzhi-gao commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Performance results

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

hongzhi-gao Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

hongzhi-gao Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JackieTien97 Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

hongzhi-gao Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

ColinLeeo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hongzhi-gao commented Aug 17, 2026 •

edited

Loading

codecov-commenter commented Aug 17, 2026 •

edited

Loading