Skip to content

Probe Bedrock and Vertex provider keys (ADR-0003 follow-up) #15

Open

Description

@johnnyhuy

Why

The current probe layer (src/main/probes/) only knows Anthropic. The catalog has 18 entries but probeProvider() returns error: No probe implemented for "<id>" for any provider whose probeKind is aws, gcp, or azure.

Bedrock and Vertex don't expose a cheap "validate this credential" HTTP endpoint the way OpenAI/Anthropic do — there's no /v1/models you can hit. The standard check is to call a tiny STS-shaped API and parse the response. Three axes:

  1. Bedrock — call bedrock:ListFoundationModels (or bedrock:ListCustomModels) with the user's AWS credentials (default chain). 403/expired token ≠ working.
  2. Vertex — call aiplatform.googleapis.com/v1/projects/<project>/locations/global/publishers/google/models with the configured ADC + project + region. The envKeys[] for Vertex is empty today so we need to surface a "Project + Region required" step in the wizard before this can be wired.
  3. Azure — GET <endpoint>/openai/deployments?api-version=2024-02-01 with the user's AZURE_OPENAI_API_KEY. Or just probe the deployment the user names.

Scope

  • Wire aws probe kind: uses SDK's default credential provider chain; minimal IAM sigv4 against bedrock-runtime or bedrock control plane; returns valid | error with the AWS error code surfaced in detail.
  • Wire gcp probe kind: takes a project and region from wizard config; uses @google-cloud/aiplatform or just an authenticated REST GET; requires gcloud auth application-default login upstream.
  • Wire azure probe kind (and azure-openai): simple GET against the deployment list with the configured key.
  • Vault entry for Vertex needs to carry {project, region} alongside the credential. Either widen SecretWriteOptions or surface project/region as separate UI fields in the Keys step.
  • Update PROVIDER_CATALOG notes for cloud_creds rows to indicate "wizard will ask for project/region first".
  • Renderer Keys step: when user picks Vertex, ask for project id + region before "Save"; persist into vault alongside the credential (or as a sibling secret).
  • App + CLI both pick up the same probes via the shared runProbe() dispatcher.

Acceptance criteria

  • runProbe({ providerId: 'bedrock' }) returns { valid: true, status: 'ok' } against a working AWS account with Bedrock access.
  • Same returns { valid: false, status: 'invalid', detail: 'AWS error: ...' } for an unconfigured credential chain.
  • Vertex probe requires a saved project + region; errors sensibly without one.
  • Azure probe works against an actual deployment.
  • All three appear in the Validate button in the renderer without code changes beyond the probe files.
  • Tests (or fixture-mocked HTTP) for at least the error paths.
  • npm run typecheck && npm run lint && npm run build clean.

References

Activity

  1. johnnyhuy commented on Nov 8, 2026

    @johnnyhuy
    ContributorAuthor

    Split for review size: the probe layer (src/main/probes/ aws/gcp/azure kinds, ProbeContext project/region, catalog notes, mocked-HTTP tests) is being worked as #47 on branch agent/issue-47-cloud-probes. The vault storage for Vertex {project, region} and the renderer Keys step fields stay on this issue for a later run. CLI parity in cli/src/lib/probes.ts will be a separate PR (independent release flow).

  2. johnnyhuy commented on Nov 8, 2026

    @johnnyhuy
    ContributorAuthor

    Progress from the work loop - the probe layer landed in #48 (closes #47), and the vault and renderer slice for Vertex project and region is up in #50 (closes #49). Still open on this issue - CLI probe parity in cli/src/lib/probes.ts as a separate CLI PR, and live-key verification against real Bedrock and Vertex accounts.

  3. johnnyhuy commented on Nov 8, 2026

    @johnnyhuy
    ContributorAuthor

    CLI probe parity landed in #52 (the CLI half of this issue, split out as #51). The app probe layer from #48, the vault wiring from #50 and the CLI dispatch now cover Bedrock, Vertex and Azure with mocked HTTP tests on both sides. What remains here needs live provider accounts: a Bedrock probe against a working AWS account, a Vertex probe against a real GCP project, and an Azure probe against a real deployment. Parking the live verification with this note rather than closing, since no agent run can supply those credentials.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions