Skip to content

feat: creds --format env points S3 and STS at the data proxy - #99

Open
alukach wants to merge 2 commits into
mainfrom
feat/creds-env-endpoint
Open

alukach wants to merge 2 commits into
mainfrom
feat/creds-env-endpoint

Conversation

@alukach

@alukach alukach commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

source-coop creds --format env (and auth login --no-cache --format env) now also points the AWS SDKs and CLI at the data proxy for the selected --proxy-url. After eval, aws s3 ls s3://ACCOUNT/PRODUCT/ works with no --endpoint-url, --region or profile.

eval $(source-coop creds --format env)
aws s3 ls s3://ACCOUNT/PRODUCT/                           # no --endpoint-url needed

# staging, or any other environment: the endpoints follow --proxy-url
eval $(source-coop creds --format env --proxy-url https://data.staging.source.coop)

How it behaves

After the three credential exports, --format env prints:

export AWS_REGION=us-west-2
export AWS_ENDPOINT_URL_S3=https://data.source.coop
export AWS_ENDPOINT_URL_STS=https://data.source.coop/.sts
  • AWS_ENDPOINT_URL_S3 / _STS, not the global AWS_ENDPOINT_URL. The global one would send IAM, and every other service, to the proxy too.
  • A region, because the Rust and Go SDKs won't run without one, and the proxy accepts any.
  • A trailing / on --proxy-url is dropped, so STS is never …//.sts.
  • These are the same lines service-account usage already printed for API keys; both now come from one helper, output::endpoint_env.
  • The other formats (credential-process, aws-credentials) are unchanged, and so are exit codes.

Not breaking in practice: a script that evals these and then calls real AWS was already failing, since source.coop's keys aren't valid there. Only S3 and STS move.

Who honours AWS_ENDPOINT_URL_S3: AWS CLI v2 (2.13+), boto3 (1.28+), s3fs, obstore / object_store and polars. GDAL and DuckDB don't (see below).

Demos

These run against docs/recorder/stub.py, not a real environment: the recorder draws the stub's URLs as source.coop / data.source.coop and the throwaway HOME as ~. The credentials are made up.

eval $(source-coop creds --format env), then aws s3 ls with no other flags

eval $(source-coop creds --format env), then aws s3 ls with no other flags

The README's aws demo (docs/demos/aws.svg) is re-recorded to end with the same eval and a flagless aws s3 ls.

Docs changes

Not in this PR

  • GDAL and DuckDB don't read AWS_ENDPOINT_URL_S3. Separate PRs add --format gdal-env and --format duckdb for them.
  • No global AWS_ENDPOINT_URL, for the reason above.

Testing

  • cargo fmt, cargo clippy --all-targets -- -D warnings, cargo test and cargo test --features staging pass (180 tests).
  • One unit test added in src/output.rs: the full env output, with and without a trailing / on the proxy URL. service-account usage's existing test still covers its lines.
  • Tried by hand only against the stub (the demos above, with AWS CLI 2.27). Not tried against staging or production, nor with boto3 / s3fs / obstore / polars: their support comes from their docs.
  • Docs site built locally with docs/site/build.py.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@claude

claude Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Claude finished @alukach's task in 9s —— View job


✅ No blocking issues — safe to merge.

The diff has no correctness or security problems.

  • print_env is now called with args.env.proxy_url in both run_login (--no-cache) and run_creds (src/main.rs). That matches the --proxy-url the credentials were fetched for.
  • endpoint_env drops a trailing /, so STS is never …//.sts. The new unit test covers this.
  • service-account usage now takes its lines from the shared helper. It previously did not trim a trailing slash, so it now trims one too. That is harmless, and arguably a fix.
  • The credential values are printed unquoted, as before. This change doesn't alter that.

I read the diff only. I did not build the project or run the tests.

Simplify (ponytail)

  • src/output.rs endpoint_env: it returns [String; 3], and service_account.rs then calls .to_vec() on it. Returning Vec<String> would drop that call. This is a nit.

💰 Estimated review cost: $0.11 · 0m09s · 4 turns

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview: https://cli.source.coop/pr-preview/pr-99/

Built from 350b575. It goes live a minute or two after this run, once GitHub Pages deploys it, and is removed when the PR closes.

github-actions Bot added a commit that referenced this pull request Nov 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions Bot added a commit that referenced this pull request Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant