Repository navigation
Open
Description
Why
The current probe layer (src/main/probes/) only knows Anthropic. The catalog has 18 entries but probeProvider() returns error: No probe implemented for "<id>" for any provider whose probeKind is aws, gcp, or azure.
Bedrock and Vertex don't expose a cheap "validate this credential" HTTP endpoint the way OpenAI/Anthropic do — there's no /v1/models you can hit. The standard check is to call a tiny STS-shaped API and parse the response. Three axes:
- Bedrock — call
bedrock:ListFoundationModels(orbedrock:ListCustomModels) with the user's AWS credentials (default chain). 403/expired token ≠ working. - Vertex — call
aiplatform.googleapis.com/v1/projects/<project>/locations/global/publishers/google/modelswith the configured ADC + project + region. TheenvKeys[]for Vertex is empty today so we need to surface a "Project + Region required" step in the wizard before this can be wired. - Azure —
GET <endpoint>/openai/deployments?api-version=2024-02-01with the user'sAZURE_OPENAI_API_KEY. Or just probe the deployment the user names.
Scope
- Wire
awsprobe kind: uses SDK's default credential provider chain; minimal IAM sigv4 againstbedrock-runtimeorbedrockcontrol plane; returnsvalid | errorwith the AWS error code surfaced indetail. - Wire
gcpprobe kind: takes aprojectandregionfrom wizard config; uses@google-cloud/aiplatformor just an authenticated RESTGET; requiresgcloud auth application-default loginupstream. - Wire
azureprobe kind (andazure-openai): simpleGETagainst the deployment list with the configured key. - Vault entry for Vertex needs to carry
{project, region}alongside the credential. Either widenSecretWriteOptionsor surface project/region as separate UI fields in the Keys step. - Update
PROVIDER_CATALOGnotes for cloud_creds rows to indicate "wizard will ask for project/region first". - Renderer Keys step: when user picks Vertex, ask for project id + region before "Save"; persist into vault alongside the credential (or as a sibling secret).
- App + CLI both pick up the same probes via the shared
runProbe()dispatcher.
Acceptance criteria
-
runProbe({ providerId: 'bedrock' })returns{ valid: true, status: 'ok' }against a working AWS account with Bedrock access. - Same returns
{ valid: false, status: 'invalid', detail: 'AWS error: ...' }for an unconfigured credential chain. - Vertex probe requires a saved project + region; errors sensibly without one.
- Azure probe works against an actual deployment.
- All three appear in the
Validatebutton in the renderer without code changes beyond the probe files. - Tests (or fixture-mocked HTTP) for at least the error paths.
-
npm run typecheck && npm run lint && npm run buildclean.
References
- ADR-0003 (
docs/adr/0003-key-validity-and-expiry-probing.md) - AWS SigV4 reference: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
- Vertex
aiplatform.googleapis.com: https://cloud.google.com/vertex-ai/generative-ai/docs/learn/rest - Azure OpenAI REST: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
Metadata
Metadata
Assignees
Labels
No labels
Activity
johnnyhuy commentedon Nov 8, 2026
Split for review size: the probe layer (
src/main/probes/aws/gcp/azure kinds, ProbeContext project/region, catalog notes, mocked-HTTP tests) is being worked as #47 on branchagent/issue-47-cloud-probes. The vault storage for Vertex {project, region} and the renderer Keys step fields stay on this issue for a later run. CLI parity incli/src/lib/probes.tswill be a separate PR (independent release flow).johnnyhuy commentedon Nov 8, 2026
Progress from the work loop - the probe layer landed in #48 (closes #47), and the vault and renderer slice for Vertex project and region is up in #50 (closes #49). Still open on this issue - CLI probe parity in cli/src/lib/probes.ts as a separate CLI PR, and live-key verification against real Bedrock and Vertex accounts.
johnnyhuy commentedon Nov 8, 2026
CLI probe parity landed in #52 (the CLI half of this issue, split out as #51). The app probe layer from #48, the vault wiring from #50 and the CLI dispatch now cover Bedrock, Vertex and Azure with mocked HTTP tests on both sides. What remains here needs live provider accounts: a Bedrock probe against a working AWS account, a Vertex probe against a real GCP project, and an Azure probe against a real deployment. Parking the live verification with this note rather than closing, since no agent run can supply those credentials.