VLE: cache edge/vertex classification and prune PATHS_BETWEEN via reverse BFS - #2515
Open
lkozeev wants to merge 9 commits into
Open
VLE: cache edge/vertex classification and prune PATHS_BETWEEN via reverse BFS#2515lkozeev wants to merge 9 commits into
lkozeev wants to merge 9 commits into
Conversation
… NULL Bypasses dead code that always forced NULL, allowing VLE execution to extract the ID and optimize path finding.
The VLE DFS previously called get_edge_entry_with_hash() and get_next_vertex() for every edge it accepted. The edge's start and end vertices are stable, so this lookup is unnecessary once the edge has been examined. Add start_vertex_id and end_vertex_id to edge_state_entry and populate them when an edge is classified for the first time. Add get_next_vertex_from_state() as a cache-based counterpart to get_next_vertex(), and use it in dfs_find_a_path_between() and dfs_find_a_path_from(). For CYPHER_REL_DIR_NONE the next vertex depends on the endpoint currently on the DFS stack. Preserve the existing vertex-stack logic, but read the endpoints from edge_state_entry instead of fetching the edge_entry again. Zero the new fields on a fresh edge_state_entry and add assertions for the undirected traversal stack invariant. No path-order or result change is intended.
add_valid_vertex_edges() previously re-walked a vertex's adjacency arrays and re-looked up edge entries on every visit. The result of is_an_edge_match() is static for a VLE_local_context, while only used_in_path and is_edge_in_path depend on the current DFS path. Introduce a vertex_edge_cache hash table mapping a vertex id to the list of adjacent edges that pass the static edge match. Build the list on the first visit to the vertex and reuse it on later visits. The dynamic checks are still performed for every visit, but only for edges that are already known to be statically valid. Allocate the cached edge arrays in a dedicated child memory context and free that context in free_VLE_local_context(). Keep the batched lookup pipeline and the original out/in/self processing order so the DFS stack order and generated paths are unchanged. This can reduce repeated adjacency scans and edge-table lookups, especially when a VLE_local_context is reused across many SRF calls.
PATHS_BETWEEN is the only VLE mode with a fixed target vertex, so DFS branches can be pruned when they cannot reach that target within the remaining hop budget. Add a lazy reverse-BFS from the target vertex. It computes shortest distances to the target over matched edges, ignoring edges already used by the current DFS path. These distances are therefore safe lower bounds on the remaining path length. In add_valid_vertex_edges(), resolve the candidate next vertex from the cached edge endpoints and look up its reverse distance. Skip the candidate if it is unreachable or if the current depth plus one plus the reverse distance exceeds the upper bound. The reverse-distance table and BFS queue are target-specific and are kept in a dedicated memory context. The queue is a power-of-two circular buffer that grows on demand and shrinks when later targets need a much smaller frontier. No user-visible behavior change is intended other than performance.
has_been_matched stored, matched in a single uint8 field. These replace three separate booleans to save space in the hash table.
The VLE edge-state, vertex-edge, and reverse-distance caches grew unboundedly with the edges and vertices touched during traversal, which could exhaust backend memory on dense, hub-heavy graphs with low-selectivity predicates. Add age.vle_* GUCs capping each cache and reclaiming entries with a clock sweep once the cap is exceeded. edge_state_hashtable now holds only matched edges and tracks a per-entry pin_count (live occurrences on the DFS edge stack) so an in-use entry is never evicted; evicted entries are rebuilt transparently on next access. The vertex-edge cache is also bounded by allocated bytes to cover high-degree hubs. The reverse-BFS distance table, when capped, degrades to fail-open (skip pruning) rather than claiming unreachability, so a memory limit can never discard valid paths.
and argument types in age_match_vle_edge_to_id_qual() error messages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
VLE: cache edge/vertex classification and prune PATHS_BETWEEN via reverse BFS
Problem
dfs_find_a_path_between()(PATHS_BETWEEN/shortestPath-style[:REL*min..max],both endpoints known) did this on every visit to a vertex, regardless of prior visits:
— the graph context carries no label-based pre-filtering. For each edge in that array,
two hashtable probes (
edge_table,edge_state_hashtable) ran on every visit, eventhough
is_an_edge_match()'s verdict was already memoized per edge. Cost:O(k · deg_total(v))instead ofO(deg_total(v) + k · deg_match(v)).start_vertex_id/end_vertex_idare fixed at edge-creation time.the hop budget was walked in full (
O(d^r)) before backtracking, instead of rejected inO(1).No memory bound on any of the above — a pathological query could grow them unbounded for
the duration of the call.
Changes
NULL) — requiredfor
PATHS_BETWEENclassification to ever trigger.edge_state_entry— resolved once, not per traversal step.vertex_edge_cache) — built once per vertex, skipsnon-matching-label edges on every later visit.
PATHS_BETWEEN— reject an unreachablebranch in
O(1).5–6.
edge_state_entry: remove struct padding; bitpack 3bools into oneuint8.age.vle_*); fails open under pressure (neverwrong, only less pruning).
Assert, wrong function names in error messages.No change in visible behavior is expected.
Results
Env: AMD Ryzen 5 5500U (6c/12t), 16 GiB RAM, boost off, governor=
performance. PG18736d880vs. vanilla PG18+AGE. SNB graph viagenerate_graph.sql(gsmall=SF1,gmid=SF10).ShortestPath_hard—MATCH path=(p1)-[:KNOWS*1..5]-(p2) ... min(length(path))Path— same pattern,RETURN path LIMIT 1(first match, no min aggregation)Reproduce
generate_graph.sql
shortest_path.sql
path.sql
New GUCs
age.vle_edge_state_htab_initial_sizevle_edge_statecache, default16384age.vle_vertex_edge_htab_initial_sizevertex_edge_cache, default1024age.vle_edge_state_max_entriesvle_edge_statecache before background eviction of unreferenced entries kicks in, default2000000age.vle_vertex_edge_cache_max_entriesvertex_edge_cachebefore background eviction of unreferenced entries kicks in, default200000age.vle_vertex_edge_cache_max_kbvertex_edge_cacheadjacency arrays, default65536age.vle_reverse_dist_max_entries500000age.vle_max_cached_contexts5age.vle_edge_state_eviction_enabledvle_edge_statecache, defaulttrueAI assistance
Used throughout; no clean split of responsibility is possible to state. All code comments
were AI-generated.