feat(batch-evaluation): add --output-config, rename --qualifier to --endpoint, require timestamp timezones

[jariy17]

Third of four PRs. Stacked on #2262 (grouped help + examples) — GitHub retargets this to refactor as the stack merges.

Three changes to eval batch-evaluation evaluate and simulate, plus the SDK release that makes the first possible. Two are breaking, listed below.

--output-config (new)

Lets a customer say where evaluation results and metrics are written instead of taking the service-managed default. An API-shaped passthrough: BatchOutputConfig resolves inline JSON, file://<path>, or - (stdin) and hands the parsed object to StartBatchEvaluation with field names and nested values untouched.

The CLI adds no defaults and validates no field combinations — the service is the only thing that knows which are legal (e.g. logGroupName is illegal with SOURCE_LOG_GROUP, and can't sit under the reserved /aws/bedrock-agentcore/evaluations/ namespace). What the CLI does guarantee is that malformed JSON is rejected before any side effect.

Long-form documentation lives in the flag's help: block, rendered in Parameter details:, following the existing harness/parameterHelp.tsx convention rather than being crammed into the one-line description. --data-source-config gets the same treatment — it was previously documented only by a 70-character description.

simulate --qualifier → --endpoint ⚠️ breaking

SessionSource has always called this concept --endpoint; simulate was the one eval command spelling it --qualifier. No alias — the refactor CLI hasn't shipped GA.

Worth a reviewer's attention: harness endpoint * and runtime * still use --qualifier. This moves simulate to match its own command family (eval) rather than those. The InvokeDatasetInput field stays qualifier, since that's the Runtime API's name and not ours.

Timestamps are unchanged

An earlier revision of this PR required an explicit timezone on --start-time / --end-time, rejecting 2026-09-01T00:00:00. That is a breaking change and has been dropped — parsing stays exactly as it is on refactor.

Dependency

@aws-sdk/client-bedrock-agentcore and -control → 3.1129.0, the first release exposing request-side outputConfig on StartBatchEvaluationRequest. The generated type is used directly — no cast papering over an older model.

simulate resolves output config before invoking

The replay bills the customer's Runtime once per dataset example. A malformed --output-config must surface before the first invocation, not after the whole run — so it's resolved up front, and there's a test asserting invokeDataset is never reached.

Live verification — account 725476964917 (EXPLORE_PROFILE), us-west-2

All three input forms submitted real jobs; the service echoed back exactly what was passed.

Inline JSON, custom dedicated group — job pr3_outputconfig_verify_1788996491-5011fdbc56, COMPLETED:

"outputConfig": {
  "cloudWatchConfig": {
    "logGroupName": "/company/pr3-verify-evaluations",
    "logStreamName": "run-pr3_outputconfig_verify_1788996491-5011fdbc56",
    "metricsNamespace": "Company/PR3Verify",
    "resultDestination": "DEDICATED_LOG_GROUP"
  }
}

The service created /company/pr3-verify-evaluations and a stream named after the job. It held 0 bytes because the target Runtime had no in-scope sessions — the destination wiring is what this proves, not result content.

file://, SOURCE_LOG_GROUP — job pr3_sourcegroup_verify_1788996663-8f6486e573, HTTP 202. No logGroupName echoed back, correct for that destination.

stdin (-) — job pr3_stdin_verify_1788996665-07d97ca802, HTTP 202. metricsNamespace honored; the service filled in its own default dedicated group.

Rejected locally, before any AWS call: malformed --output-config JSON, and --qualifier (now an unknown option).

Cleanup: the /company/pr3-verify-evaluations log group was deleted (confirmed 0 remaining). The three batch-evaluation job records remain — there is no delete API for them, the same limitation the existing fixture suite notes.

Tests

• src/core/eval.test.ts (new, 2 tests) — this file exists because the other layers left a real gap. The TestCoreClient suites assert what a handler hands to Core; the fixture suites assert whole recorded requests but need an account to re-record. Deleting Core's outputConfig: input.outputConfig line passed every existing test. It stubs only the data client and asserts the constructed StartBatchEvaluationCommand.
• batch-evaluation.test.tsx (+7) — output config from inline / file / stdin reaching the request unchanged; omitted staying undefined; malformed JSON rejected with an empty Core call log; --endpoint driving both the invocation qualifier and the graded session source; malformed output config aborting before invokeDataset.

Mutation-checked: dropping Core's outputConfig forwarding fails the new Core test.

Verification

• bun test — 3201 pass, 0 fail
• bun run typecheck, bun run lint:check, bun run format:check — clean

Follow-up

Fourth PR covers online-eval: --output-config (control-plane shape, deliberately not sharing this PR's data-plane type), --tags on create, --description on update, and widening the CLI-managed execution role so a custom destination is actually writable.

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Looks good

Nice, well-scoped PR. The three changes are cleanly separated, the tests cover each layer where it can fail (handler, SessionSource timestamp parsing, and — deliberately — the Core→SDK seam that the TestCoreClient and fixture suites can't cover), and the "resolve --output-config before any Runtime invocation" ordering in simulate is exactly the right call for a command that bills per example. The BatchOutputConfig module and long-form parameter help follow existing conventions and are appropriately API-shaped.

No blocking issues. A couple of small notes, take or leave:

• eval ondemand simulate still uses --qualifier (src/handlers/eval/ondemand/simulate/index.tsx:19,69,92). The PR body justifies the rename by aligning simulate with its own command family (eval), but this sibling command in the same family is left inconsistent. If it's intentionally deferred, a follow-up TODO/issue reference would help; otherwise consider renaming it in the same breaking change so users only see one flag flip.
• Timestamp regex accepts HH:MM with no seconds (sessionSource.tsx:171) — that's fine and matches ISO-8601, but the error message example (2026-09-01T00:00:00Z) and all tests use the seconds form; worth confirming 2026-09-01T00:00Z is intentionally allowed (looks like it is, and it's a superset of what customers will type).

Live verification, mutation-testing notes, and the explanation for the new src/core/eval.test.ts file (the gap between handler-level TestCoreClient assertions and re-record-only fixture assertions) are all appreciated.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.06%. Comparing base (c8936e9) to head (290780f).

Additional details and impacted files

@@                     Coverage Diff                      @@
##           feat/eval-cli-router-groups    #2265   +/-   ##
============================================================
  Coverage                        97.06%   97.06%           
============================================================
  Files                              568      569    +1     
  Lines                            39343    39374   +31     
============================================================
+ Hits                             38188    38219   +31     
  Misses                            1155     1155           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jariy17]

Use Golden tests.