Skip to content

feat: add opt-in vector search timing diagnostics - #711

Draft
jerry-024 wants to merge 3 commits into
apache:mainfrom
jerry-024:feat/vector-search-timing
Draft

feat: add opt-in vector search timing diagnostics#711
jerry-024 wants to merge 3 commits into
apache:mainfrom
jerry-024:feat/vector-search-timing

Conversation

@jerry-024

@jerry-024 jerry-024 commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add opt-in diagnostics for identifying latency and I/O amplification in batch vector search. Logging is disabled by default and enabled only when PAIMON_LOG_VECTOR_SEARCH_TIMING=1.

Brief change log

  • Report API and evaluation timings for snapshot/manifest loading, global-index search, refinement, raw fallback, result merging, and finalization.
  • Report vindex reader lifecycle timings, native batch chunk sizes, scalar fallbacks, concurrency, and memory budget.
  • Track shared logical and physical range-read statistics across cloned vindex readers.
  • Add ignored, controlled Parquet diagnostics for clustered, scattered, and missing-offset-index refinement reads.

Tests

  • cargo fmt --all -- --check
  • git diff --check
  • cargo check -p paimon --lib (blocked on macOS ARM by paimon-vindex-core 0.3.0 using the unstable stdarch_neon_f16 feature)
  • Run the ignored Parquet I/O diagnostics once the dependency build is available.

API and Format

No public API or storage-format changes. The new stderr diagnostics require the environment variable above.

Documentation

No documentation changes.

@jerry-024
jerry-024 marked this pull request as draft August 14, 2026 06:08
@jerry-024 jerry-024 changed the title feat: add vector search timing logs feat: add opt-in vector search timing diagnostics Aug 14, 2026
…timing

* upstream/main:
  fix(vindex): avoid full index reads during refine (apache#708)
  feat(vindex): build indexes from streamed data splits (apache#707)
  feat: support deletion vector merge-on-read (apache#706)

# Conflicts:
#	crates/paimon/src/table/vector_search_builder.rs
#	crates/paimon/src/vindex/mod.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant