Skip to content

Report CSR freshness in Lance index terms and add an optimize step #75

Closed

Description

@prrao87

Problem

refresh() applies changes as a delta over the base CSR and rebuilds once pending changes pass a fixed 25% of the base (CSR_REBUILD_FRACTION). csr_stats() reports version, delta_edges, tombstones, update and build_ms, but not how much of the current dataset the base covers, and users can't choose when the delta is folded in.

Idea

Follow Lance's index freshness contract, as proposed in lance-format/lance-graph#159: report the version the base covers against the current version, and the number of edges outside the base; and add an explicit optimize step that merges the delta into a new base, alongside or instead of the fixed threshold.

The pieces exist: fragment-level diffs between versions (edge_changes), the base plus delta in Adjacency, and csr_stats().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions