feat(persistence): opt-in limit on concurrent broad MongoDB searches

[doctorvk]

Closes #1748.

Summary

Adds an opt-in limit on how many potentially broad standard searches run at the
same time in one server process on the MongoDB backend. It is off unless
HFS_MONGODB_BROAD_SEARCH_CONCURRENCY is set, so existing deployments behave
exactly as before.

When it is set to N, a potentially broad search waits for one of N permits
before its first database operation and holds it until its response is built.
Reads, writes, history, narrow searches and _contained searches never wait. The
limit does not reserve connections; it reduces how much of the driver pool broad
searches can hold at once, so other requests wait less behind them.

• search_admission.rs (new): the classification and the limit check.
  • Narrow: at least one unmodified, unchained _id, reference or uri
    predicate; a date whose values all use eq or ap; identifier with both
    system and code; or a _list or compartment search.
  • A search with no predicates is narrow only as a plain paged read: it is
    potentially broad when it computes a total, sorts on a search parameter, or
    skips rows (_offset > 0). _total=none and _offset=0 count as omitted.
  • A direct search_count always counts, so it is classified as computing a
    total whatever its _total option.
  • _has searches and everything else are potentially broad.
• MongoBackendConfig::broad_search_concurrency: Option<usize>, read by
  apply_search_env from HFS_MONGODB_BROAD_SEARCH_CONCURRENCY. None (the
  default) creates no semaphore. Zero or a non-integer fails startup; a value at
  or above max_connections is accepted with a warning.
• search and search_count take a permit after validating the query.
  search computes its own total through count_standard, so it never asks for
  a second permit (which would deadlock with a limit of 1).
• hfs calls apply_search_env next to apply_reindex_env. The README lists
  the variable.

Behavior

HFS_MONGODB_BROAD_SEARCH_CONCURRENCY	effect
unset	no limit, no semaphore (unchanged behavior)
N (positive integer)	at most N potentially broad searches at once per process
N >= HFS_MONGODB_MAX_CONNECTIONS	accepted; startup warning that broad searches can hold every pooled connection
0, -1, two	startup fails: HFS_MONGODB_BROAD_SEARCH_CONCURRENCY must be a positive integer

A search that cannot get a permit before HFS_REQUEST_TIMEOUT ends with the
existing 408. The limit is per process; with several processes the cluster-wide
limit is that many times the setting.

Before / after

Server built with --no-default-features --features R4,mongodb, started with
--storage-backend mongodb against a single-node replica set, with
HFS_MONGODB_MAX_CONNECTIONS=2. 150 Observations with one code. A failCommand
failpoint blocks getMore for 5 s (twice) on the server's connections; two
Observation?code=http://loinc.org|8867-4&_count=10 searches start, and a
Patient/{id} read is timed while they run. A read by id never sends getMore,
so the failpoint never blocks it directly.

server	blocked getMore	read by id during load	broad searches
main (acab4363)	2	3.41 s	5.0 s, 5.0 s
this PR, setting unset	2	3.42 s	5.0 s, 5.0 s
this PR, HFS_MONGODB_BROAD_SEARCH_CONCURRENCY=1	1	1.3 ms	5.0 s, 10.0 s

Unset behaves as main. With a limit of 1 the second search waits for the
permit instead of taking the second connection, and the read gets it.

Tests

• Unit (backends/mongodb/search_admission.rs): narrowing predicates; other
  predicates broad whatever their count; weakened narrowing predicates (ranges,
  mixed prefixes, |code, system|, :not, :missing, chains, no values);
  parameterless searches with _total, _sort, _offset and their no-op
  values; _has, _list and compartment searches; direct counts independent of
  _total; limit validation.
• Unit (backends/mongodb/backend.rs): env parsing (trim, blank, unset,
  invalid values); semaphore sizing (none when unset); a broad search waits while
  the permit is held and a narrow one does not; without a limit 64 broad searches
  proceed at once; a search that fails on an invalid cursor returns its permit; a
  queued search_count does not open the database client and returns its permit
  when cancelled.
• Unit (hfs/src/main.rs): build_mongodb_config reads the variable and fails
  on 0, -1 and two.
• MongoDB integration (tests/mongodb/broad_search_admission.rs), pool of 2,
  failpoint blocking only getMore:
  • mongodb_broad_search_limit_leaves_a_connection_for_reads: limit 1; waits
    with waitForFailPoint until the first search is blocked, checks the second
    does not reach getMore, and a read by id completes within 1 s.
  • mongodb_broad_searches_at_the_pool_size_block_reads: limit 2; both searches
    blocked, the read does not complete within 1 s (control).
  • mongodb_broad_search_with_total_uses_one_permit: limit 1; a broad search
    with _total=accurate completes and returns its total, then a count runs.
• With the classification forced to "narrow", the limit-1 integration test fails
  at "the second broad search waits for the permit".

Full helios-persistence (with mongodb), helios-rest and helios-hfs suites
against a MongoDB replica set: 4,585 passed, 0 failed, 0 skipped.

Notes for review

• Off by default to keep existing behavior; the issue asked whether you would
  prefer a default (for example half of HFS_MONGODB_MAX_CONNECTIONS). Happy to
  change it.
• The classification is a heuristic and can be wrong in both directions; a
  reference to a large organization or a busy day is treated as narrow.
• _contained searches are not limited. Under _contained=both the top-level
  part goes through search and is limited like any standard search.

Run locally against a MongoDB replica set started with
--setParameter enableTestCommands=1 (HFS_TEST_MONGODB_URL):

cargo fmt --all -- --check
cargo clippy -p helios-persistence -p helios-hfs --all-targets --all-features -- -D warnings \
  -A clippy::items_after_test_module -A clippy::large_enum_variant -A clippy::question_mark \
  -A clippy::collapsible_match -A clippy::collapsible_if -A clippy::field_reassign_with_default \
  -A clippy::doc-overindented-list-items -A clippy::doc-lazy-continuation
cargo test -p helios-persistence --features mongodb --lib -- search_admission broad_search apply_search_env
cargo test -p helios-persistence --features mongodb --test mongodb_tests -- broad_search
cargo test -p helios-hfs --features mongodb -- broad_search_concurrency