improve search engine startup verification script

Author: justincorrigible

.dev/docs/search-engine-integration.md

@@ -94,6 +94,8 @@ The two clients have compatible APIs for the operations Arranger uses (search, m
 
 **Entry point:** `modules/graphql-router/src/graphqlRoutes.ts::arrangerRoutes`
 
+**Note on startup script vs. application permissions:** Before `arrangerRoutes` runs, the container entrypoint (`scripts/ping-elasticsearch.sh`) probes `GET /_cluster/health` to display cluster status. This probe uses the application user's credentials and requires `cluster:monitor/health`. That permission is a startup-script concern only - no application code path calls `/_cluster/health`. See `docs/setup.md § Startup health display` and the roadmap entry "Decouple startup health check from application credential" for the planned fix.
+
 The following happens once per catalogue, in this order, when an `arrangerRoutes` instance starts:
 
 ### 1. Alias resolution
@@ -104,13 +106,13 @@ The following happens once per catalogue, in this order, when an `arrangerRoutes
 GET /_cat/aliases?format=json
 ```
 
-> **Required permission:** `indices:admin/aliases/get*` at the **cluster** level.
+> **Required permission:** `indices:admin/aliases/get` as an **index-level** permission on `*`.
 
-This is a cluster-wide listing - no index filter is applied. OpenSearch evaluates it against cluster-level permissions. The `cluster_composite_ops_ro` built-in action group includes `indices:admin/aliases/get*` and is the idiomatic way to grant this. See [OpenSearch CAT aliases API](https://docs.opensearch.org/latest/api-reference/cat/cat-aliases/).
+Although `cluster_composite_ops_ro` also contains `indices:admin/aliases/get*`, that is a cluster-type action group grant and does not cover direct alias API calls. OpenSearch's static plugin config (`static_action_groups.yml`) classifies `indices:admin/aliases/get` as an index-level permission: the `manage_aliases` built-in action group has `type: "index"` and includes `indices:admin/aliases*`. A direct `GET /_cat/aliases` request resolves to all indices and is evaluated by the index-level privilege evaluator. Granting `indices:admin/aliases/get` only on a specific index pattern (e.g. `analyses-*`) still results in 403 because the request scope is all indices. The permission must be on `*`. See [OpenSearch CAT aliases API](https://docs.opensearch.org/latest/api-reference/cat/cat-aliases/) and [default action groups](https://docs.opensearch.org/latest/security/access-control/default-action-groups/).
 
 The response lists all aliases the user can see. `checkESAlias` scans the list for the configured `esIndex` value. If found, the actual backing index name (e.g. `analyses-1`) is used for all subsequent calls; otherwise `esIndex` is used as-is.
 
-> **Known issue:** `cat.aliases` retrieves all cluster aliases and filters client-side. A targeted `indices.getAlias({ index: esIndex })` call would achieve the same with index-level `indices:admin/aliases/get` only, removing the cluster-level dependency. See tech-debt.
+> **Known issue:** `cat.aliases` retrieves all cluster aliases and filters client-side. A targeted `indices.getAlias({ index: esIndex })` call would achieve the same result with `indices:admin/aliases/get` scoped to the data index pattern only, removing the `*` wildcard requirement. See tech-debt.
 
 ### 2. Mapping fetch
 
@@ -240,9 +242,10 @@ All transport actions Arranger can initiate, grouped by phase:
 
 | Phase                       | API call                     | Transport action                                    | Minimum grant                                        |
 | --------------------------- | ---------------------------- | --------------------------------------------------- | ---------------------------------------------------- |
+| Entrypoint script†          | `GET /_cluster/health`       | `cluster:monitor/health`                            | cluster-level explicit (startup script only - not application code) |
 | Startup: detection          | `GET /`                      | `cluster:monitor/main`                              | cluster-level explicit, or set `SEARCH_ENGINE`       |
 | Startup: detection fallback | `GET /_nodes/_local`         | `cluster:monitor/nodes/info`                        | cluster-level explicit (not needed if `GET /` works) |
-| Startup: alias resolution   | `GET /_cat/aliases`          | `indices:admin/aliases/get*`                        | `cluster_composite_ops_ro` (cluster-level)           |
+| Startup: alias resolution   | `GET /_cat/aliases`          | `indices:admin/aliases/get`                         | index-level on `*` (explicit; `cluster_composite_ops_ro` does not cover direct alias API calls) |
 | Startup: mapping fetch      | `GET /<index>/_mapping`      | `indices:admin/mappings/get`                        | explicit on data index                               |
 | Startup: sets check         | `HEAD /<setsIndex>`          | `indices:admin/exists`                              | `manage` on sets index                               |
 | Startup: sets creation      | `PUT /<setsIndex>`           | `indices:admin/create`, `indices:admin/mapping/put` | `manage` on sets index                               |
@@ -252,15 +255,15 @@ All transport actions Arranger can initiate, grouped by phase:
 | saveSet: collect IDs        | `POST /<dataIndex>/_search`  | `indices:data/read/search`                          | `read` on data index                                 |
 | saveSet: write set          | `PUT /<setsIndex>/_doc/<id>` | `indices:data/write/index`                          | `write` on sets index                                |
 
+† `cluster:monitor/health` is called by `scripts/ping-elasticsearch.sh` before the Node.js process starts. The application itself never calls `/_cluster/health`. This permission can be omitted if the startup display is not needed; startup still succeeds. See roadmap: "Decouple startup health check from application credential".
+
 **Notes:**
 
 - `read` = `indices:data/read*` + `indices:admin/mappings/fields/get*` + `indices:admin/resolve/index`. Source: [OpenSearch default action groups](https://docs.opensearch.org/latest/security/access-control/default-action-groups/).
-- `cluster_composite_ops_ro` = `mget` + `msearch` + `mtv` + `aliases/exists*` + `aliases/get*` + `scroll` + `resolve/index`. Source: same link.
+- `cluster_composite_ops_ro` = `mget` + `msearch` + `mtv` + `aliases/exists*` + `aliases/get*` + `scroll` + `resolve/index`. Source: same link. Although the group contains `indices:admin/aliases/get*`, granting it does not cover `GET /_cat/aliases`: that API is evaluated by the index-level privilege evaluator. The cluster-type grant applies only during cluster-coordination operations such as internal alias routing in mget/msearch.
 - `manage` = `indices:monitor/*` + `indices:admin/*`. Covers all admin operations including `exists` and `create`.
 - Permission names are identical for OpenSearch and Elasticsearch; both security plugins share the same transport action naming (OpenSearch forked the security plugin from the Elasticsearch codebase).
 
-The `cluster_composite_ops_ro` action group is also included in the `readall_and_monitor` built-in role, which OpenSearch uses as the reference pattern for "read everything with cluster monitoring". Arranger's permission needs are a subset of that role.
-
 ---
 
 ## External references

.dev/roadmap.md

@@ -586,6 +586,27 @@ _Needs design before implementation. Treat as a blocking design question for any
 
 ## Deployment
 
+### Decouple startup health check from application credential
+
+`scripts/ping-elasticsearch.sh` calls `GET /_cluster/health` using the application user's credentials (`ES_USER`/`ES_PASS`). This forces `cluster:monitor/health` to be granted to the application role even though no application code path ever calls `/_cluster/health`. The permission exists solely so a shell script can display a status block on startup.
+
+This violates the principle of least privilege: the application user's role carries a permission it never exercises. Any operator following the documented minimum permissions will grant `cluster:monitor/health` to their search engine user indefinitely, with no indication that it is a startup-script concern rather than an application requirement.
+
+**Correct fix:**
+
+Move the readiness gate into a Kubernetes init container that runs with its own, more privileged credential, separate from the application's runtime credential:
+
+- **Init container:** runs `ping-elasticsearch.sh` (or its successor) with an elevated credential granted `cluster:monitor/main` + `cluster:monitor/health`. It retries `GET /_cluster/health?wait_for_status=yellow&...` until the cluster is ready or the container times out, gating the pod's main container start via the normal init-container mechanism. The cluster status block (cluster name, status, node count, shards) stays as this container's log output, visible via `kubectl logs <pod> -c <init-name>`, and never needs to be a concern for the app container.
+- **Main container:** the application runs with a leaner credential, `cluster:monitor/main` only, which is all it needs for `GET /` engine auto-detection. It never holds `cluster:monitor/health`.
+
+This preserves the `wait_for_status=yellow` readiness gate (true cluster-ready signal, not just HTTP connectivity) while still removing the unused permission from the long-running application process: the actual least-privilege violation the rest of this item is about.
+
+A simpler alternative (drop the readiness gate, probe `GET /` only, no init container) was considered and rejected: it trades a real readiness check for a permissions fix, and `GET /` only confirms the process is responding, not that shards are allocated.
+
+**Related:** the script filename (`ping-elasticsearch.sh`) and env var names (`ES_HOST`, `ES_USER`, `ES_PASS`) are also Elasticsearch-first and should be renamed in the same effort. See tech-debt: "Elasticsearch-first naming in startup script and env vars".
+
+_Standalone: yes, once the approach is agreed. Needs a new Vault role/policy for the init container's elevated credential, a second VSO-injected secret, and an `initContainers` stanza added to the Helm chart, in addition to script changes. If the Helm chart lives in a separate repo from this one, that work belongs there. No application logic changes._
+
 ### Helm chart update
 
 _Priority: medium. Maintenance burden for production deployments._

.dev/sessions.md

@@ -8,7 +8,7 @@ Newest first.
 
 ## 2026-06-30
 
-Fact-checked the search engine permissions reference against the authoritative OpenSearch source (`static_action_groups.yml`); corrected two mistakes introduced in the 2026-06-28 session.
+Fact-checked the search engine permissions reference against the authoritative OpenSearch source (`static_action_groups.yml`); corrected two mistakes introduced in the 2026-06-28 session; corrected a further mistake about alias resolution permission level, verified against live cluster behaviour and OpenSearch static plugin config.
 
 - `modules/graphql-router/src/searchClient/index.ts`: fixed wrong permission name in the dual-403 error message: `cluster:monitor/nodes_info` is not a real permission; the correct transport action is `cluster:monitor/nodes/info`
 - `docs/setup.md`: (1) corrected the core search section: `read` does not cover `indices:admin/mappings/get` or `indices:admin/aliases/get`; both must be granted explicitly alongside `read`; (2) fixed `cluster:monitor/nodes_info` → `cluster:monitor/nodes/info` in the auto-detection table; (3) updated the summary table data index column to list all three required permissions
@@ -24,6 +24,13 @@ Fact-checked the search engine permissions reference against the authoritative O
 - `modules/types/src/configs/constants.ts`, `modules/graphql-router/src/config/utils/index.ts`, `modules/graphql-router/src/graphqlRoutes.ts`, `apps/search-server/src/configs/fromEnv/localEnvs.ts`, `apps/search-server/src/configs/fromEnv/aggregator.ts`, `integration-tests/server/test/index.test.ts`, `integration-tests/mcp-server/test/index.test.ts`, `apps/search-server/.env.schema`, `modules/graphql-router/README.md`, `.dev/roadmap.md`, `.dev/tech-debt.md`: renamed `DISABLE_SETS`/`disableSets` to `ENABLE_SETS`/`enableSets`; Sets now defaults to disabled and requires explicit opt-in
 - `modules/graphql-router/src/config/utils/index.ts`, `modules/graphql-router/src/graphqlRoutes.ts`: fixed multicatalog race condition: `resource_already_exists_exception` during `initializeSets` is now treated as success; Sets initialization failure no longer takes down the catalogue's GraphQL endpoint
 - `modules/graphql-router/src/config/utils/index.test.ts`: added tests for the race condition path (exception treated as success) and non-race errors (still propagate)
+- `scripts/ping-elasticsearch.sh`: when `SEARCH_ENGINE` is not set, the script now detects the engine type after the cluster is ready by probing `GET /` and checking for `"distribution":"opensearch"`; the "Ready" banner and all subsequent output use the correct label; mirrors Arranger's own Stage 1 auto-detection logic
+- `.dev/docs/search-engine-integration.md`, `docs/setup.md`: corrected alias resolution permission: `indices:admin/aliases/get` is an index-level permission (not cluster-level); OpenSearch's `manage_aliases` built-in group is `type: "index"` in `static_action_groups.yml`; `GET /_cat/aliases` is evaluated by the index-level privilege evaluator against all indices and requires the permission on `*`; `cluster_composite_ops_ro`'s `indices:admin/aliases/get*` grant does not cover direct alias API calls; removed `cluster_composite_ops_ro` from the summary table cluster column (it was listed there solely as the alias resolution grant, which was incorrect)
+- `scripts/ping-elasticsearch.sh`: fixed two bugs: (1) the retry loop used `> /dev/null` so curl exit code 0 was treated as success even on 403, making the "ready" check a pure TCP check with no HTTP validation; now checks for a 2xx response code using `-w '%{http_code}'` piped to grep; (2) the health display silently showed blank fields and ❓ when `/_cluster/health` returned 403, with no indication of why; now shows a clear message with the HTTP code and the permission name to add; also uses `SEARCH_ENGINE` env var for the label so the output says "OpenSearch" instead of "Elasticsearch" when configured accordingly; requires `cluster:monitor/health` on the reader role for the retry to succeed and for full health display
+- `.dev/docs/search-engine-integration.md`, `docs/setup.md`: added `cluster:monitor/health` to startup sequence, permission reference table, and user-facing setup guide; clearly marked as startup-script only (not application code); added footnote distinguishing it from application-level permissions
+- `.dev/roadmap.md`: added "Decouple startup health check from application credential" to the Deployment section; describes why the current coupling is wrong and what the correct fix is (`GET /` via `cluster:monitor/main` for liveness; remove `cluster:monitor/health` dependency)
+- `scripts/ping-elasticsearch.sh`: removed mutational engine-label logic (placeholder assignment in a `case`, conditionally overwritten later based on a string-equality check); engine detection is now a single-assignment `engine_label()` function; `if [ $? -ne 0 ]` replaced with `if ! command; then`
+- `.dev/roadmap.md`: rewrote "Decouple startup health check from application credential" with an init-container design: elevated credential (`cluster:monitor/health`) confined to an init container that owns the `wait_for_status=yellow` readiness gate and the cluster-status log output; main container runs with `cluster:monitor/main` only; scope note updated to flag the new Vault role/policy, VSO secret, and Helm `initContainers` work this requires
 
 ---
 

.dev/tech-debt.md

@@ -91,6 +91,15 @@ The preferred pattern is **(B)**. Mixing the two makes it harder to find tests,
 **Fix:** Audit the monorepo and move all `__tests__/` test files to be co-located with their source file, following pattern (B). Update any Jest/node:test config glob patterns that rely on `__tests__/` directory discovery.
 **Standalone:** yes; mechanical file moves plus config glob updates, no logic changes
 
+### Elasticsearch-first naming in startup script and env vars
+
+**Files:** `scripts/ping-elasticsearch.sh`; env vars `ES_HOST`, `ES_USER`, `ES_PASS` set by the chart
+**Severity:** low (misleading branding; confusing for operators using OpenSearch)
+**Kind:** terminology / naming
+**Issue:** The startup readiness script is named `ping-elasticsearch.sh` and prints "Elasticsearch Ready" regardless of the configured engine. The env vars exposed by the chart (`ES_HOST`, `ES_USER`, `ES_PASS`) carry the "ES" prefix even when connecting to OpenSearch. The display label in the script has been updated to derive from `SEARCH_ENGINE` (outputs "OpenSearch", "Elasticsearch", or "Search Engine"), but the script filename and chart env var names remain Elasticsearch-first.
+**Fix:** Rename `ping-elasticsearch.sh` to `ping-search-engine.sh` (or `ping-cluster.sh`) and update the reference in the Dockerfile/entrypoint. Coordinate with the chart to rename `ES_HOST`, `ES_USER`, `ES_PASS` to engine-neutral names (`SEARCH_HOST`, `SEARCH_USER`, `SEARCH_PASS` or similar). Both changes require a coordinated release since the chart and image must agree on env var names.
+**Standalone:** no; script rename is trivially standalone, but env var rename requires a matching chart release
+
 ### Inconsistent spelling of `catalogue`
 
 **File:** throughout the monorepo
@@ -274,11 +283,11 @@ When Arranger Server (`apps/search-server`) is updated to use `catalogue`, the M
 ### `fetchMapping` uses `cat.aliases` instead of `indices.getAlias`
 
 **File:** `modules/graphql-router/src/searchClient/fetchMapping.ts` (`getESAliases`)
-**Severity:** low (over-privileged; unnecessary cluster-level dependency)
+**Severity:** low (over-privileged; requires `*` index permission for alias lookup)
 **Kind:** privilege minimization
-**Issue:** `getESAliases` calls `esClient.cat.aliases({ format: 'json' })` with no index filter, retrieving ALL cluster aliases and doing client-side filtering. `GET /_cat/aliases` is a cluster-wide API that requires `indices:admin/aliases/get*` at the cluster level (provided by `cluster_composite_ops_ro`). A targeted `indices.getAlias({ index: esIndex })` call would achieve the same result with only index-level `indices:admin/aliases/get` on the specific index, removing the cluster-level dependency entirely.
-**Fix:** Replace `esClient.cat.aliases()` + `checkESAlias` with `esClient.indices.getAlias({ index: esIndex })`. If the alias exists, the response contains the backing index name; if not, handle the 404. This reduces the required cluster permissions for core search (no `cluster_composite_ops_ro` needed just for alias resolution; `cluster:monitor/main` or `SEARCH_ENGINE` bypass becomes the only cluster-level concern).
-**Standalone:** yes; confined to `fetchMapping.ts`; update the caller in `fetchMapping` and adjust the docs/setup.md permissions reference when done
+**Issue:** `getESAliases` calls `esClient.cat.aliases({ format: 'json' })` with no index filter, retrieving ALL cluster aliases and doing client-side filtering. `GET /_cat/aliases` evaluates `indices:admin/aliases/get` as an index-level permission (OpenSearch `manage_aliases` group is `type: "index"` in the static plugin config) against all indices; the permission must be granted on `*` because the request is unscoped. A targeted `indices.getAlias({ index: esIndex })` call makes a scoped request, so the permission need only be granted on the data index pattern.
+**Fix:** Replace `esClient.cat.aliases()` + `checkESAlias` with `esClient.indices.getAlias({ index: esIndex })`. If the alias exists, the response contains the backing index name; if not, handle the 404. The `indices:admin/aliases/get` permission on `*` can then be removed from the role and scoped down to the data index pattern.
+**Standalone:** yes; confined to `fetchMapping.ts`; update `docs/setup.md` and `.dev/docs/search-engine-integration.md` permission tables when done
 
 ### No unit tests for `getESAliases` alias resolution