absorb sqon-builder into modules/sqon

Author: justincorrigible

.dev/sessions.md

@@ -8,6 +8,28 @@ Newest first.
 
 ## 2026-06-30
 
+Absorbed `@overture-stack/sqon-builder` functionality into `modules/sqon`, making it the single package for SQON construction and validation; `sqon-builder` is now redundant and will be deprecated separately.
+
+- `modules/sqon/src/schema/constants.ts`: added `zod.boolean()` to `SqonScalarValueSchema`; exported `SqonScalar` and `SqonScalarOrArray` types
+- `modules/sqon/src/schema/index.ts`: re-exported the two new types
+- `modules/sqon/src/builder/utils.ts`: new file - `SqonFieldFilter` type, `isGroupNode`, `isFieldFilter`, `asArray`, `emptySqon`, `checkMatchingArrays`, `checkMatchingFilter`; `SqonFieldFilter` defined as `SqonNode & { content: { fieldName: string; value: SqonScalarOrArray } }` so it satisfies the type predicate constraint
+- `modules/sqon/src/builder/reduce.ts`: new file - `reduceSqon`; merges in/not-in/some-not-in/all value arrays, applies boundary semantics to gt/gte/lt/lte, removes empty combinations, unwraps single-item and/or (except pivoted ones), flattens same-op same-pivot combinations; `between` intentionally left non-reducible (v2)
+- `modules/sqon/src/builder/index.ts`: new file - `SQON` factory and `SqonBuilder` type; covers `in`, `not-in`, `some-not-in`, `all`, `gt`, `gte`, `lt`, `lte`, `between`, `fuzzy` (op stays `'filter'` for serialized SQON compatibility), `and`, `or`, `not`, `setFilter`, `removeFilter`, `removeExactFilter`; builder is a pure wrapper over `SqonNode`, not the `& SQON` anti-pattern
+- `modules/sqon/src/index.ts`: exported `SQON`, `SqonBuilder`, `SqonFieldFilter`, `SqonFieldFilterKey`, `checkMatchingArrays`, `checkMatchingFilter`, `emptySqon`
+- `modules/sqon/src/builder/index.test.ts`, `modules/sqon/src/__tests__/schema.test.ts`: new/extended BDD tests (91 + 152 total passing)
+- `modules/graphql-router/src/network/utils/sqon.ts`: swapped `@overture-stack/sqon-builder` import for `@overture-stack/sqon`; removed `@ts-expect-error`; `convertToSqon` now returns `Result<SqonNode, ...>` via `SQON.from(filter).toValue()`
+- `modules/components/src/DataContext/types.ts`: `SQONType` now aliases `SqonNode | null` (from `@overture-stack/sqon`)
+- `modules/components/src/Table/DownloadButton/DownloadButton.tsx`: swapped `SQONBuilder.in(...)` for `SQON.in(...)`
+- `modules/graphql-router/package.json`, `modules/components/package.json`: replaced `@overture-stack/sqon-builder` dependency with `@overture-stack/sqon` (local workspace ref)
+- `modules/sqon/src/index.ts`: updated exports to use combined value+type re-export syntax
+- `modules/sqon/src/builder/index.test.ts`: updated all references to match renamed identifiers
+- `modules/graphql-router/src/network/utils/sqon.ts`, `modules/components/src/Table/DownloadButton/DownloadButton.tsx`: updated to `SqonBuilder` (was `SQON`)
+- `modules/sqon/README.md`: rewrote to document builder API; added usage section covering the recommended `SqonNode`-at-boundaries pattern, `SqonBuilder.from()` for validation, filter and combination examples, and type reference table
+- `modules/sqon/src/__tests__/buildQueryFilter.test.js`, `modules/sqon/src/builder/index.test.ts`, `modules/sqon/src/__tests__/schema.test.ts`, `modules/graphql-router/src/middleware/__tests__/buildQuery/normalizeFilter.test.js`, `modules/components/src/SQONViewer/utils.test.js`: added failing tests specifying the `'filter'` → `'fuzzy'` canonical op rename; all intentionally red pending implementation
+- `modules/sqon/package.json`, `modules/sqon/tsup.config.ts` removed in favour of CLI flags, `modules/sqon/src/version/constants.ts`: switched sqon build from `tsc` to `tsup` with dual ESM+CJS output; added conditional `exports` field so CJS consumers can `require()` the package; fixed `import.meta.url` usage to fall back gracefully in CJS context; resolves `integration-tests-import` failure where components' Babel CJS build could not load the ESM-only sqon package
+- `integration-tests/import/package.json`, `integration-tests/import/test.ts`: added `@overture-stack/sqon` as a direct dep; added test asserting `SqonBuilder`, `SqonSchema`, and `SQON_SCHEMA_VERSION` are importable from the CJS build; removed sqon from the "pure ESM" exclusion comment (graphql-router remains excluded)
+- `package.json`: added `"esbuild": "0.17.19"` to both `overrides` and `devDependencies`; the override alone is insufficient because npm does not apply overrides to peer dep auto-installation - making it an explicit devDep forces the correct version at the root where bundle-require loads it; tsup upgrade to 8.5.1 attempted and reverted (esbuild hoisting conflicts under npm); logged pnpm migration and tsup upgrade as standalone tech-debt
+
 Fact-checked the search engine permissions reference against the authoritative OpenSearch source (`static_action_groups.yml`); corrected two mistakes introduced in the 2026-06-28 session; corrected a further mistake about alias resolution permission level, verified against live cluster behaviour and OpenSearch static plugin config.
 
 - `modules/graphql-router/src/searchClient/index.ts`: fixed wrong permission name in the dual-403 error message: `cluster:monitor/nodes_info` is not a real permission; the correct transport action is `cluster:monitor/nodes/info`
@@ -31,6 +53,8 @@ Fact-checked the search engine permissions reference against the authoritative O
 - `.dev/roadmap.md`: added "Decouple startup health check from application credential" to the Deployment section; describes why the current coupling is wrong and what the correct fix is (`GET /` via `cluster:monitor/main` for liveness; remove `cluster:monitor/health` dependency)
 - `scripts/ping-elasticsearch.sh`: removed mutational engine-label logic (placeholder assignment in a `case`, conditionally overwritten later based on a string-equality check); engine detection is now a single-assignment `engine_label()` function; `if [ $? -ne 0 ]` replaced with `if ! command; then`
 - `.dev/roadmap.md`: rewrote "Decouple startup health check from application credential" with an init-container design: elevated credential (`cluster:monitor/health`) confined to an init container that owns the `wait_for_status=yellow` readiness gate and the cluster-status log output; main container runs with `cluster:monitor/main` only; scope note updated to flag the new Vault role/policy, VSO secret, and Helm `initContainers` work this requires
+- `modules/graphql-router/src/middleware/__tests__/buildQuery/buildQueryFilter.test.js`: added a failing test specifying that `op: 'fuzzy'` must produce the same ES output as `op: 'filter'`; this is the behavioral specification for the upcoming canonical-op flip; the test is intentionally red now and will go green when the flip is complete
+- `~/.claude/CLAUDE.md`, `agentics/CLAUDE.md`, `agentics/AGENTS.md`, `agentics/template/CLAUDE.md`: added "verify purpose alignment before implementing" to Interaction parameters; added to agentics CHANGELOG.md
 
 ---
 

.dev/tech-debt.md

@@ -5,6 +5,16 @@ Issues logged here when found scope-adjacent to other work. Not a priority backl
 
 ---
 
+## build tooling
+
+### Migrate from npm to pnpm
+standalone: yes
+context: npm's flat hoisting causes esbuild binary version conflicts across workspaces when multiple packages use tsup. Adding sqon as a second tsup consumer caused bundle-require's peer esbuild to be hoisted to root at a mismatched version. pnpm's strict per-package isolation would prevent this class of issue; each package sees only what it declares. Migration requires updating the Jenkins pipeline and any Dockerfiles that invoke npm.
+
+### Upgrade tsup from 6.7.0 to 8.5.1
+standalone: no
+context: tsup@6.7.0 is ~2 years old. Upgrading to 8.5.1 is blocked by the npm hoisting problem above: tsup@8.5.1 brings esbuild@^0.27.0, which conflicts with tsx's esbuild@~0.28.0 and bundle-require's peer dep resolution under npm. Revisit after pnpm migration.
+
 ## apps/mcp-server
 
 ### `InMemoryEventStore` is not suitable for production
@@ -313,6 +323,33 @@ When Arranger Server (`apps/search-server`) is updated to use `catalogue`, the M
 **Issue:** `resolveSetsInSqon` has two paths - SQON contains no `set_id:` values (no-op, returns SQON unchanged) and SQON contains `set_id:` values (expands to stored IDs via an ES search). Neither path has a unit test.
 **Standalone:** yes; but note the file also carries the `hackyTemporaryEsSetResolution` tech-debt entry; evaluate for removal during Sets full-feature implementation rather than investing deeply in tests for code that may be replaced
 
+### No unit tests for `convertToSqon` or other `network/utils/` functions
+
+**Files:** `modules/graphql-router/src/network/utils/sqon.ts`, `modules/graphql-router/src/network/utils/gql.ts`, `modules/graphql-router/src/network/utils/promise.ts`
+**Severity:** medium
+**Kind:** missing test coverage
+**Issue:** `convertToSqon` is a pure function at a user-input boundary: it parses an unknown value and returns `Result<SqonNode, { INVALID_SQON: string }>`. Every incoming SQON passes through it, making it security-relevant, yet it has zero test coverage. The other two utils files (`gql.ts`, `promise.ts`) are also untested.
+**Fix:** Unit tests for `convertToSqon` covering: valid SQON returns `success(SqonNode)`; invalid SQON (wrong shape, missing `op`) returns failure with `INVALID_SQON`; null/undefined input returns failure; JSON string input is accepted. Add tests for `gql.ts` and `promise.ts` once their exported surface is confirmed non-trivial.
+**Standalone:** yes; pure functions, no mocking required
+
+### No unit tests for network resolvers
+
+**Files:** `modules/graphql-router/src/network/resolvers/` (aggregations.ts, fetch.ts, networkNode.ts, query.ts, response.ts)
+**Severity:** medium
+**Kind:** missing test coverage
+**Issue:** The entire network resolver layer has no tests. This is the core async multi-node query execution path: aggregation response resolving, remote node data fetching, network node response building, query construction, and response transformation. Bugs here affect all multi-catalogue network searches silently.
+**Fix:** Unit tests with mocked network node responses. The pure transformation files (`response.ts`, `networkNode.ts`, `query.ts`) can be tested directly. `fetch.ts` requires HTTP-level mocking (e.g. `undici MockAgent` or similar). Cover: single-node success; partial node failure (one down, others succeed); empty response; aggregation accumulation across nodes.
+**Standalone:** partial; transformation functions are standalone; `fetch.ts` depends on establishing the HTTP mock pattern first
+
+### No unit tests for `dataToExportFormat`
+
+**File:** `modules/graphql-router/src/utils/dataToExportFormat.js`
+**Severity:** medium
+**Kind:** missing test coverage
+**Issue:** `dataToExportFormat` transforms ES hit data into the export column format, handling `extendedDisplayValues` label substitution, `jsonPath` extraction, column visibility, and hit flattening. No unit tests exist.
+**Fix:** Unit tests covering: basic field mapping; `jsonPath` extraction; `extendedDisplayValues` label substitution; columns with `show: false` excluded; empty hit set returns empty array.
+**Standalone:** yes; pure transformation function
+
 ### `hackyTemporaryEsSetResolution.js`: stale ES 6.2 workaround + convention violation
 
 **File:** `modules/graphql-router/src/mapping/hackyTemporaryEsSetResolution.js`
@@ -326,6 +363,15 @@ When Arranger Server (`apps/search-server`) is updated to use `catalogue`, the M
 
 ## modules/types
 
+### No unit tests for `tools/` utilities or `networkAggregationConfigUtils`
+
+**Files:** `modules/types/src/tools/stringFns.ts`, `modules/types/src/tools/typeFns.ts`, `modules/types/src/configs/networkAggregationConfigUtils.ts`
+**Severity:** low
+**Kind:** missing test coverage
+**Issue:** All three files are exported from the package and used across the monorepo but have zero test coverage. `stringFns.ts` and `typeFns.ts` are utility and type guard functions where a regression would propagate silently to every consumer. `networkAggregationConfigUtils.ts` contains non-trivial domain logic for network aggregation config setup.
+**Fix:** Co-located unit tests (e.g. `stringFns.test.ts` alongside `stringFns.ts`) covering each exported function. Prioritize `networkAggregationConfigUtils` as the highest-complexity target.
+**Standalone:** yes
+
 ### Config constants need reorganization (blocked on architecture work)
 
 **File:** `modules/types/src/configs/constants.ts`
@@ -486,11 +532,11 @@ When Arranger Server (`apps/search-server`) is updated to use `catalogue`, the M
 **Files:** `integration-tests/import/test.ts`, `integration-tests/import/package.json`
 **Severity:** low (gap in regression coverage)
 **Kind:** missing test coverage
-**Issue:** `integration-tests/import` runs under Jest + ts-jest, which handles CJS and TypeScript source but cannot import pure-ESM dist packages (`.js` files with `"type": "module"` and no `"require"` export) without additional configuration. `@overture-stack/arranger-graphql-router` and `@overture-stack/sqon` are pure ESM; both are missing from the import smoke test. `@overture-stack/arranger-types` (CJS + ESM hybrid) and `@overture-stack/arranger-components` (CJS via Babel) are covered. An import regression in `graphql-router` or `sqon` would not be caught by this test.
+**Issue:** `integration-tests/import` runs under Jest + ts-jest, which handles CJS and TypeScript source but cannot import pure-ESM dist packages (`.js` files with `"type": "module"` and no `"require"` export) without additional configuration. `@overture-stack/arranger-graphql-router` is pure ESM and is missing from the import smoke test. `@overture-stack/arranger-types` (CJS + ESM hybrid), `@overture-stack/arranger-components` (CJS via Babel), and `@overture-stack/sqon` (dual ESM+CJS since 2026-06-30) are covered. An import regression in `graphql-router` would not be caught by this test.
 
 Additionally: `integration-tests/import` resolves all deps via npm workspaces symlinks (`file:` paths), so it tests local build output, not the published tarball. Publishing regressions (e.g. stale `file:` refs in `package.json`) are caught by `npm run release:check` (`scripts/verify-pack.mjs`), not by this test.
 
-**Fix:** Either configure Jest to handle pure-ESM packages (update `transformIgnorePatterns`, enable `--experimental-vm-modules`), or add a separate lightweight smoke test using `node --input-type=module` or `tsx` that imports from `arranger-graphql-router` and `arranger-sqon` and checks their key exports.
+**Fix:** Either configure Jest to handle pure-ESM packages (update `transformIgnorePatterns`, enable `--experimental-vm-modules`), or add a separate lightweight smoke test using `node --input-type=module` or `tsx` that imports from `arranger-graphql-router` and checks its key exports.
 
 **Additional TODOs on top of the ESM gap:**
 1. **Verify `exports` subpaths, not just package root.** The smoke test should assert each named subpath in the `exports` field (`./utils`, `./download`, etc.) resolves and exposes the expected named exports. A missing barrel re-export (e.g. `getAllData` was absent from `utils/index.ts`) causes `ERR_PACKAGE_PATH_NOT_EXPORTED` for consumers importing via a subpath, which the current test would not catch.
@@ -525,6 +571,15 @@ Additionally: `integration-tests/import` resolves all deps via npm workspaces sy
 
 ## apps/search-server
 
+### No unit tests for catalog config loading or `catalogId`
+
+**Files:** `apps/search-server/src/configs/index.ts`, `apps/search-server/src/configs/catalogId.ts`, `apps/search-server/src/configs/fromFiles/` (4 files), `apps/search-server/src/configs/fromEnv/` (3 files)
+**Severity:** high
+**Kind:** missing test coverage
+**Issue:** The catalog loading logic - recursing subdirectories, aggregating config files, generating unique IDs - has no tests. This is startup-critical: bugs cause startup failures or silent misconfiguration in multicatalog deployments. `catalogId.ts` tracks ID uniqueness across loads but is also untested. `fromFiles/` and `fromEnv/` parsing has no coverage either.
+**Fix:** Unit tests using a temporary directory fixture for flat (single-catalog) and nested (multicatalog) layouts; error handling on malformed config files; unique ID generation and collision detection in `catalogId.ts`; env var aggregation in `fromEnv/`.
+**Standalone:** yes; no running server required; mock the filesystem with a temp directory
+
 ### No README
 
 **File:** `apps/search-server/`; no `README.md` present

integration-tests/import/package.json

@@ -2,7 +2,8 @@
 	"name": "integration-tests-import",
 	"dependencies": {
 		"@overture-stack/arranger-components": "file:../../modules/components",
-		"@overture-stack/arranger-types": "file:../../modules/types"
+		"@overture-stack/arranger-types": "file:../../modules/types",
+		"@overture-stack/sqon": "file:../../modules/sqon"
 	},
 	"devDependencies": {
 		"jest": "^29.7.0",

integration-tests/import/test.ts

@@ -1,9 +1,10 @@
 import { describe, expect, test } from '@jest/globals';
 import { Query, Aggregations, SQONViewer, Table } from '@overture-stack/arranger-components';
 import { configs, elastic } from '@overture-stack/arranger-types';
+import { SqonBuilder, SqonSchema, SQON_SCHEMA_VERSION } from '@overture-stack/sqon';
 
-// NOTE: @overture-stack/arranger-graphql-router and @overture-stack/sqon are pure ESM
-// and require a separate test runner (see tech-debt: ESM import test gap).
+// NOTE: @overture-stack/arranger-graphql-router is pure ESM and requires a separate
+// test runner (see tech-debt: ESM import test gap).
 //
 // NOTE: These tests use file: workspace deps and therefore exercise local build output,
 // not the published npm tarball. Publishing regressions (e.g. file: paths in package.json)
@@ -23,4 +24,10 @@ describe('integration-tests/import', () => {
 		expect(configs.configOptionalProperties).toBeDefined();
 		expect(elastic.esToAggTypesMap).toBeDefined();
 	});
+
+	test('imports @overture-stack/sqon CJS exports', () => {
+		expect(SqonBuilder).toBeDefined();
+		expect(SqonSchema).toBeDefined();
+		expect(typeof SQON_SCHEMA_VERSION).toBe('string');
+	});
 });

modules/components/package.json

@@ -32,7 +32,7 @@
 		"@overture-stack/arranger-types": "file:../types",
 		"@emotion/react": "^11.0.0",
 		"@emotion/styled": "^11.0.0",
-		"@overture-stack/sqon-builder": "^1.1.0",
+		"@overture-stack/sqon": "file:../sqon",
 		"@reach/component-component": "^0.17.0",
 		"@tanstack/react-table": "^8.7.9",
 		"axios": "^1.16.0",

modules/components/src/DataContext/types.ts

@@ -1,4 +1,4 @@
-import type { SQON } from '@overture-stack/sqon-builder';
+import type { SqonNode } from '@overture-stack/sqon';
 import type { AxiosResponse, Method } from 'axios';
 import type { Dispatch, SetStateAction } from 'react';
 
@@ -80,7 +80,7 @@ export type APIFetcherFn = (options: {
 	url?: string;
 }) => Promise<AxiosResponse<unknown>>;
 
-export type SQONType = SQON | null;
+export type SQONType = SqonNode | null;
 
 export type FetchDataFn = (options?: {
 	config?: Record<string, any>;

modules/components/src/Table/DownloadButton/DownloadButton.tsx

@@ -1,5 +1,5 @@
 import { css } from '@emotion/react';
-import SQONBuilder from '@overture-stack/sqon-builder';
+import { SqonBuilder } from '@overture-stack/sqon';
 import cx from 'classnames';
 import { merge } from 'lodash-es';
 import urlJoin from 'url-join';
@@ -103,7 +103,7 @@ const DownloadButton = ({
 
 	const downloadSqon =
 		!disableRowSelection && hasSelectedRows
-			? SQONBuilder.in(exportSelectedRowsField, selectedRows).toValue()
+			? SqonBuilder.in(exportSelectedRowsField, selectedRows).toValue()
 			: sqon;
 
 	const handleExporterClick = (

modules/graphql-router/package.json

@@ -38,7 +38,7 @@
 		"@graphql-tools/utils": "^10.2.2",
 		"@opensearch-project/opensearch": "^3.6.0",
 		"@overture-stack/arranger-types": "file:../types",
-		"@overture-stack/sqon-builder": "^1.1.0",
+		"@overture-stack/sqon": "file:../sqon",
 		"apollo-server": "^3.10.3",
 		"apollo-server-core": "^3.10.3",
 		"apollo-server-express": "^3.10.3",