Release 0.0.18: hub-node scale remediation

CHANGELOG.md

@@ -7,6 +7,128 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.0.18] - 2026-09-13
+
+### Added
+
+- **Hub-node scale recorder** (`tests/benchmarks/test_hub_node_bench.py`,
+  `tests/benchmarks/hub_bench_report.py`). Seeds a Postgres hub with 1k / 10k /
+  100k edges and records p50/p95 latency, DB round trips and on-disk sizes
+  for `connect()`, `save()`, neighbour listings, the `len(nodes())` count
+  pattern and 32-way concurrent `connect()`. Opt-in via `-m bench`
+  (`bench_slow` for 100k). Baseline and per-change results:
+  `docs/bench/2026-09-hub-node-baseline.md`.
+- **Node adjacency mode API** (`jvspatial/db/database.py`,
+  `jvspatial/core/context.py`): `Database.edge_ids_mode` capability flag,
+  `resolve_edge_ids_mode(db)`, `GraphContext.persists_edge_ids()`,
+  `Node.edges(limit=...)`, `strip_node_edges()` on `PostgresDB` / `MongoDB`,
+  and the `jvspatial migrate strip-node-edges` CLI (dry run by default,
+  `--apply` to write).
+- **Neighbour counts and pages in the database** (`jvspatial/core/entities/node.py`):
+  `Node.count_nodes(...)` (one `COUNT` round trip — use it instead of
+  `len(await n.nodes(...))`), `Node.nodes_page(sort=..., cursor=..., limit=...)`
+  (keyset-paginated neighbours, cursor encoding shared with
+  `GraphContext.find_page` via `jvspatial.core.pager`), and
+  `nodes_bulk(limit_per_source=...)` (one windowed query on Postgres).
+  `count_neighbors` now delegates to `count_nodes`.
+- **`$text` full-text search** (`jvspatial/db/_postgres_translate.py`,
+  `jvspatial/db/query.py`): `{"$text": {"$search": "...", "$fields": [...]}}`
+  becomes `to_tsvector('simple', …) @@ plainto_tsquery('simple', …)` on
+  Postgres, backed by a GIN index declared with `@fulltext_index([...])` or
+  `attribute(fulltext=True)`; other backends evaluate the same semantics in
+  memory. Previously `$text` fell back to a full scan and raised in memory.
+- **`jvspatial.db.escape_regex()`** for building literal `$regex` patterns
+  from user input (`$regex` is never index-backed).
+- **`find_edges_between(limit=...)`** pushes the limit to the database.
+- **`JVSPATIAL_POSTGRES_COMMAND_TIMEOUT`** sets `PostgresDB`'s per-statement
+  timeout (default 60 s); the postgres guide gains a pool-sizing rule of thumb
+  and transaction-pooler notes for tenant scoping.
+- **`gin_index="off"` / `JVSPATIAL_PG_GIN_INDEX=off`** skips the
+  whole-document `GIN (data jsonb_path_ops)` on new Postgres collections.
+- **`expand_node` keyset paging** (`jvspatial/core/graph_expansion.py`):
+  `after=` / `pagination.next_after` (and the `after` query parameter on the
+  graph expand endpoint) page incident edges by edge id in O(page). The
+  integer `cursor` offset keeps working.
+
+### Changed
+
+- **Postgres, MongoDB and SQLite no longer persist node adjacency on node
+  rows** (`edge_ids_mode="derive"`). The indexed edge collection is the only
+  source of truth: `connect()`, `disconnect()` and `save()` never rewrite or
+  row-lock a node, so their cost no longer grows with the node's degree and
+  concurrent connects to one hub no longer serialise. `Node.edges()`,
+  `connection_count()`, cascade delete, `expand_node` and `subgraph_bfs` read
+  the edge collection; `Node.edge_ids` stays an empty in-memory list. Legacy
+  `edges` arrays on existing rows are ignored on read and dropped on the next
+  save — run `jvspatial migrate strip-node-edges --dsn … --apply` (safe
+  online) to reclaim the space at once. JsonDB and DynamoDB keep `"persist"`.
+  Opt out with `JVSPATIAL_NODE_EDGE_IDS=persist` or
+  `db.edge_ids_mode = "persist"`.
+- **`Node.nodes()` pushes every filter shape to the database.** The list
+  form (`edge=[E], node=["Leaf"]`), strings, subclass-inclusive node classes,
+  `{Name: criteria}` dicts and property kwargs all become one
+  `find_connected_nodes` round trip on Postgres, MongoDB and SQLite, with
+  `limit` pushed down; `direction="both"` included. Previously only the class
+  form took the join and everything else loaded every incident edge and
+  hydrated every neighbour before slicing in Python. Backends without the
+  pushdown keep the Python path, now filtering in the database `find`.
+- **Postgres per-class indexes are `entity`-leading** (`PostgresDB.create_index`,
+  `GraphContext.ensure_indexes`). Indexes declared with
+  `attribute(indexed=True)` / `@compound_index` become `(entity, <fields>)`
+  (`<col>_entity_<fields>_idx`, shared by classes declaring the same fields),
+  or `WHERE entity = '<Class>'` with `index_partial_by_entity=True` /
+  `partial_by_entity=True`; the unscoped pre-0.0.18 index of the same fields
+  is dropped once its replacement exists. Descending keys are now
+  `DESC NULLS LAST` so sorted + limited finds walk the index, and `entity` /
+  `id` / `tenant_id` are indexed as the real columns the translator compares
+  (the edge `(source, target, entity)` unique index previously indexed
+  `data->entity`, which no query used). Indexes built by the old rules are
+  rebuilt in place on the next `ensure_indexes`.
+- **`expand_node` / `subgraph_bfs` source incident edges from the edge
+  collection** in every mode (one query per node instead of one `get` per
+  edge id); `total_edge_count` is a count of incident edges, and neighbours
+  are fetched with a single `find_many`.
+
+### Fixed
+
+- **List-form edge filters were dropped by `Node.nodes()`**
+  (`jvspatial/core/entities/node.py`). `nodes(edge=[E])`, `edge=["E"]` and
+  `edge=[{"E": {...}}]` returned neighbours reached through *any* edge type,
+  and `direction="both"` capped at 10 000 edges. Edge types and criteria are
+  now always applied.
+- **Several Postgres paths ignored `PostgresDB.tenant(...)`**
+  (`jvspatial/db/postgres.py`). `traverse`, `find_one_and_update`,
+  `find_one_and_delete` and `bulk_save_detailed` took a raw pool connection
+  without the `app.tenant_id` GUC, so under `enable_rls` they saw nothing
+  (atomic ops returned `None`, `traverse` returned no hops) and the COPY path
+  failed the policy's `WITH CHECK` before falling back to per-record saves.
+  They now run on the tenant-scoped connection like every other operation.
+- **`observe=True` / caching wrappers turned `bulk_save_detailed` into one
+  round trip per record** (`jvspatial/db/_observable.py`,
+  `jvspatial/db/_cache.py`). Both wrappers subclass `Database`, whose default
+  `bulk_save_detailed` is a serial `save` loop; it shadowed `__getattr__`
+  forwarding, so a wrapped Postgres `COPY` (or Mongo `bulk_write`) never ran.
+  Both now forward to the backend (the cache also refreshes saved ids).
+- **`ObservableDatabase` advertised graph pushdowns its backend lacks**
+  (`jvspatial/db/_observable.py`). `find_connected_nodes` / `traverse` were
+  plain methods, so `getattr(db, "find_connected_nodes", None)` looked
+  callable over JsonDB and the call then raised. They now exist on the
+  wrapper only when the wrapped adapter implements them.
+- **`$pull` was ignored by `QueryEngine.apply_update`** (`jvspatial/db/query.py`).
+  Backends that apply updates in Python — Postgres `find_one_and_update`,
+  and the JsonDB / SQLite / DynamoDB defaults — silently skipped it, so on
+  Postgres in persist mode `disconnect()`, edge deletion and cascade delete
+  left stale edge ids on node rows (and `connection_count()` over-reported).
+
+- **`JVSPATIAL_WEBHOOK_API_KEY_REQUIRE_HTTPS` was allowlist-rejected**
+  (`jvspatial/env_adapter.py`). The key was never in `ALLOWED_ENV_KEYS`, so
+  startup warned that it was ignored and
+  `server_config_overrides_from_env()` never mapped it onto
+  `WebhookConfig.webhook_api_key_require_https`. Local HTTP callback tunnels
+  (ngrok → plain `http://127.0.0.1`) could not disable the query-param HTTPS
+  gate via env alone. Also maps `JVSPATIAL_WEBHOOK_HTTPS_REQUIRED` into
+  `WebhookConfig.webhook_https_required` for the same ServerConfig path.
+
 ### Changed
 
 - **`uvicorn` is capped below 1.0** (`pyproject.toml`). It was floor-only
@@ -472,7 +594,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
-- **BREAKING (limited):** `JsonDBTransaction(db).save/get/delete/find()` now raises `NotImplementedError` by default instead of silently no-op'ing. Pass `best_effort=True` to opt into the buffered-commit semantics, or check `Database.supports_transactions` and fall back to non-transactional writes. Audited downstream consumers (`jvagent`, `integral`) — neither uses this surface, so no coordinated change required.
+- **BREAKING (limited):** `JsonDBTransaction(db).save/get/delete/find()` now raises `NotImplementedError` by default instead of silently no-op'ing. Pass `best_effort=True` to opt into the buffered-commit semantics, or check `Database.supports_transactions` and fall back to non-transactional writes. Audited known adopters — none use this surface, so no coordinated change required.
 - **`QueryEngine` optimization cache** is now bounded by an LRU (default 1024 entries, configurable via `QueryEngine(cache_size=...)`). Was unbounded.
 - **MongoDB retries** now go through the shared retry helper (`utils/retry.py`); behavior preserved (one retry on connection-error with reset).
 - CI coverage gate raised from 55 → 60% to reflect the new tested code.

CONTRIBUTING.md

@@ -142,6 +142,10 @@ flake8 jvspatial/ tests/
 
 # performance benchmarks (only run when asked; not part of -q above)
 pytest tests/benchmarks --benchmark-only
+
+# hub-node scale recorder (Postgres; 100k tier opt-in via bench_slow)
+JVSPATIAL_POSTGRES_TEST_DSN=postgresql://... \
+  pytest tests/benchmarks/test_hub_node_bench.py -m bench -s
 ```
 
 For details on the benchmark suite (what's in it, what CI does with

SPEC.md

@@ -111,7 +111,7 @@ AttributeMixin + pydantic.BaseModel
 ### 2.2 Node — graph node
 
 `jvspatial/core/entities/node.py:34+`. Adds:
-- `edge_ids: List[str]` — transient in memory, persisted at the top level as `edges` (`Node._get_top_level_fields` line 55-60)
+- `edge_ids: List[str]` — transient in-memory list. Persisted at the top level as `edges` (`Node._get_top_level_fields`) **only** when the backend's adjacency mode is `persist` (JsonDB, DynamoDB). In `derive` mode (Postgres, MongoDB, SQLite default) node rows carry no `edges`, `connect()` / `disconnect()` / `save()` never write or lock a node row, and adjacency is read from the indexed edge collection (`Node.edges`, `Node.connection_count`, cascade delete, `expand_node`); `edge_ids` stays empty and a legacy stored `edges` array is ignored on read and dropped on the next save (`GraphContext.persists_edge_ids`, `context.py`; §4.2)
 - `_visitor_ref: weakref` — currently visiting walker, transient
 - `_visit_hooks: ClassVar` — populated by `__init_subclass__` from `@on_visit`-decorated methods (line 62+)
 
@@ -251,6 +251,7 @@ cases are documented rather than normalized:
 Adapters declare capabilities as class attributes:
 
 - `supports_transactions: bool` — `True` for MongoDB (replica set); `False` for SQLite (best-effort), JSON, DynamoDB.
+- `edge_ids_mode: str` — where node adjacency lives. `"derive"` for Postgres, MongoDB, SQLite: the edge collection (indexed on `source` / `target` by `Edge.get_indexes`) is the only source of truth. `"persist"` for JSON, DynamoDB: node rows also carry an `edges` array. Resolve the effective value with `resolve_edge_ids_mode(db)` (`db/database.py`): an `edge_ids_mode` set on the adapter instance → `JVSPATIAL_NODE_EDGE_IDS` (`persist` | `derive`) → class default; observability / cache wrappers are unwrapped via `inner`. Rows written in persist mode are converted with `jvspatial migrate strip-node-edges` (`strip_node_edges()` on `PostgresDB` / `MongoDB`).
 
 Callers branching on capabilities should test the flag, not the adapter class.
 
@@ -290,11 +291,13 @@ No built-in migration framework. Adapters do not enforce schemas. Adding optiona
 | `$in`, `$nin` | Membership |
 | `$exists` | Field presence |
 | `$and`, `$or` | Logical combinators |
-| `$regex` | Regex match (string fields) |
+| `$regex` | Regex match (string fields). Never index-backed; build patterns from user input with `jvspatial.db.escape_regex` |
+| `$text` | Top-level `{"$text": {"$search": "words", "$fields": ["context.a", ...]}}`: every search word (case-insensitive, `\w+` tokens, no stemming) occurs in the concatenated fields; without `$fields` every string value is searched |
 
 ### 5.2 Pushdown vs in-memory
 
-- **MongoDB**: native pushdown; queries run server-side.
+- **Postgres**: `translate_query` pushes the whole operator surface into JSONB SQL; `$text` becomes `to_tsvector('simple', …) @@ plainto_tsquery('simple', …)` (requires `$fields`; the GIN from `@fulltext_index` / `attribute(fulltext=True)` serves it when the fields match in order). Per-class indexes are `(entity, <fields>)` (or `WHERE entity = …` with `partial_by_entity`), descending keys `DESC NULLS LAST`, so typed `find(sort=…, limit=…)` walks an index; the whole-document GIN is optional (`JVSPATIAL_PG_GIN_INDEX`). Untranslatable queries fall back to a full scan + `QueryEngine.match`.
+- **MongoDB**: native pushdown; queries run server-side (`$text` uses the collection's text index; `$fields` is stripped).
 - **SQLite**: translated to SQL via `SQLiteTranslator` (subset; complex `$or` chains may fall back).
 - **DynamoDB**: limited pushdown via `Select=COUNT` and key conditions; remainder filtered client-side.
 - **JSON**: full in-memory evaluation after loading matching collection.
@@ -364,6 +367,16 @@ Enabling prefetch may enqueue neighbors before hook-driven `visit()` calls; visi
 
 `await node.neighborhood(depth=k, direction=..., edge=..., node=..., limit=...)` returns hydrated `Node` instances within `k` hops. Postgres uses `Database.traverse` + `get_batch`; other backends use per-hop `nodes()` BFS.
 
+### 6.8 Neighbour queries (`Node.nodes` / `count_nodes` / `nodes_page`)
+
+`nodes(direction, node=, edge=, limit=, **props)` normalises every filter shape to `(edge_entities, edge_query, node_entities, node_query)` (`_neighbor_filter_spec`, `node.py`):
+
+- a name or list of names → exact entity match; an **edge** class → its exact entity; a **node** class → the class and every loaded subclass (`isinstance` semantics; `Node` itself means any type);
+- `{Name: criteria}` items → an `$or` of `{"entity": Name, <criteria>}` branches; property kwargs AND onto the node side;
+- bare criteria keys map to `context.<key>` (the attribute `_matches_property_filter` reads); `id`, `entity` (and `source`, `target`, `bidirectional` on edges) are top-level record paths.
+
+When the backend exposes `find_connected_nodes` (Postgres, MongoDB, SQLite) the spec plus `limit` go down in **one** round trip, `direction="both"` included. Each neighbour appears once. Postgres picks a join for one edge type in one direction (duplicate-free under the `(source, target, entity)` unique index, so `LIMIT` streams) and an `id IN (...)` semi-join otherwise. A criterion that does not translate raises `NotImplementedError` in the adapter and `nodes()` takes the Python path — one edge `find` plus a node `find` that still applies every filter; no filter is ever dropped. `count_nodes(...)` is the matching `COUNT` (`count_connected_nodes`; `count_neighbors` is an alias). `nodes_page(sort=, cursor=, limit=)` pages neighbours by keyset with the cursor encoding of `GraphContext.find_page` (`core/pager.py`); the keyset is expressed as a node-side query, so it follows the same pushdown. `nodes_bulk(..., limit_per_source=N)` caps neighbours per source (`ROW_NUMBER() OVER (PARTITION BY source)` on Postgres).
+
 ---
 
 ## 7. GraphContext and Dependency Injection