streams API: support per-request env overrides for config templating

[g-hurst]

Closes #481

Summary

Adds an optional top-level env field to the streams-mode HTTP API's config payloads, so a caller can supply per-request environment variable overrides instead of pre-rendering ${FOO}-style templates themselves or relying on real OS environment variables.

{
  "env": { "TOPIC": "orders", "BROKER": "localhost:9092" },
  "input": { "kafka": { "addresses": ["${BROKER}"], "topics": ["${TOPIC}"] } },
  "output": { "stdout": {} }
}

Covered endpoints:

• POST/PUT /streams/{id} (HandleStreamCRUD)
• POST /resources/{type}/{id} (HandleResourceCRUD)

Bulk POST /streams is out of scope, as described in the issue — it has no env substitution today, so adding overrides there means building that support for the first time rather than reusing it.

No new templating syntax is introduced. This only changes where the existing envLookupFunc seam gets its answers from.

Implementation

internal/config/reader.go — OptAddEnvLookupOverrides

A new reader option that composes with the reader's existing lookup func rather than replacing it: overrides are checked first, and any name absent from them falls through to whatever the reader already had (by default os.LookupEnv). An override therefore shadows a same-named OS env var, while every other variable continues to resolve from the OS as usual.

Wrapping the lookup func — rather than pre-substituting the document — is what keeps the rest of the interpolation contract intact. ${FOO:default} still falls back to its default when FOO is neither overridden nor set, and a ${{FOO}} escape is still left alone, because ReplaceEnvVariables remains the single thing doing the parsing.

internal/stream/manager/api.go — extractEnvOverrides

docs.FieldSpecs.LintYAML raises an error-level LintUnknown for any unrecognised top-level field, and both handlers treat a non-empty lint list as a hard 400, so env has to be removed from the document before linting and parsing.

The stripping is done by editing the yaml.Node tree in place instead of decoding to generic Go values and re-encoding. A config body is a template, and a decode/re-encode round trip destroys two properties it depends on:

• Implicit scalar re-resolution — 2024-01-02 comes back as a timestamp, 0123456 as octal.
• The caller's own quoting around a ${VAR} — which is precisely what stops an interpolated value containing YAML metacharacters from altering the document's structure.

Node-level editing leaves every untouched scalar byte-for-byte as written.

Two further details:

• A body that fails to parse is passed through untouched rather than erroring, because a body is only guaranteed to parse after substitution has run — the documented ${VAR: default} form puts a : inside an otherwise plain scalar. The existing downstream path reports the error as it does today.
• Value typing is validated from the node tag, so a quoted "5" is accepted and a bare 5 is rejected with a 400 — a decode into map[string]string could not tell those apart.

Everything downstream (lint, ParsedConfigFromAny, stream.FromParsed, chilled/ErrMissingEnvVars handling) is untouched.

Divergences from the sketch in #481

Two deliberate departures from the implementation sketch in the issue:

Stripping env by editing the yaml.Node tree rather than a gabs round trip. The sketch proposed following patchConfig's decode → gabs.Wrap → mutate → re-encode pattern. That pattern is safe where patchConfig uses it, because it runs against a config that has already been through substitution — this stripping has to run against the raw template before substitution, where a round trip re-resolves implicit scalars and drops the caller's quoting around a ${VAR}, as above.

A new OptAddEnvLookupOverrides rather than a local precedence helper passed to the existing OptUseEnvLookupFunc. The sketch put the "check overrides, fall back to os.LookupEnv" helper in manager. This puts the composition in the reader instead, so it falls through to whatever lookup func the reader already holds rather than hardcoding os.LookupEnv. To be clear about what that does and doesn't buy: neither call site exercises the difference today, since both build a fresh reader whose lookup func is the default — it's a forward-looking choice rather than a fix, and what it protects is the case where the streams manager gains a secret lookup func the way internal/cli/common/reader.go already has one, at which point a hardcoded fallback would silently drop secrets. If you'd rather not grow the option set in config for a single consumer, I'm happy to move this back to a local helper in manager — it's a contained change and the tests keep their shape.

Acceptance criteria

• An env map resolves ${VAR} placeholders in the rest of the document
• A request env value takes precedence over a same-named OS environment variable
• Omitting env behaves exactly as today — no behavior change for existing callers
• A missing variable still produces today's ErrMissingEnvVars/chilled behavior
• A non-string env value is rejected with a 400, not coerced
• Test coverage for both /streams/{id} and /resources/{type}/{id}

Tests

Added in internal/stream/manager/api_test.go and internal/config/env_vars_test.go:

• TestEnvLookupOverrides — override precedence, fallthrough to the wrapped lookup, defaults preserved, ${{...}} escapes untouched
• TestTypeAPIStreamEnvOverrides / TestResourceAPIEnvOverrides — override applied, precedence over a real OS env var, fallback to OS env when no override is given, mixed resolution from both sources, missing-var behavior unchanged, non-string rejected with 400, and a no-env regression check
• TestTypeAPIStreamEnvOverridesPreserveDocument / ...PreserveScalars — the stripping round trip does not mutate quoting or re-resolve implicit scalar types

Verification

• make test — full suite passes
• make lint — 0 issues
• make fmt — no changes

The endpoint description strings for /streams/{id} and /resources/{type}/{id} were updated to document the new field; no generated docs are affected.

[CLAassistant]

All committers have signed the CLA.

[g-hurst]

Hi @josephwoodward — no rush at all on this one, just flagging it since the workflows here need a maintainer to kick them off for first-time contributors, and I don't think they've been approved to run yet (only the CLA check has reported so far). Whenever you have a spare moment, would you mind approving a CI run? Happy to fix anything that falls out.

For context on the motivation: I'm a happy Connect user, and this came out of actually running into the limitation rather than from reading the API surface. In a streams-mode deployment the process env is shared by every stream in it, so there's no way to post the same config template twice with different parameters — you either pre-render the template yourself before posting or split things across separate processes. Making the templating per-request removes that, and since Connect builds on Benthos the ergonomic win lands downstream in Connect, which is the outcome I'm really after here.

If you get a chance to look at the approach itself later on, I'd really value your feedback. I've just added a short "Divergences from the sketch in #481" section to the description covering the two places this departs from what I originally proposed in the issue, including one where I'd happily take the smaller-surface option if you prefer it.

Thanks for all the review work you do on this repo — much appreciated.