Skip to content

feat: product view and list give the data's S3 path (s3_uri, s3_endpoint) - #103

Merged
alukach merged 4 commits into
mainfrom
feat/product-s3-uri
Nov 9, 2026
Merged

alukach merged 4 commits into
mainfrom
feat/product-s3-uri

Conversation

@alukach

@alukach alukach commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

product view and product list now say where to read a product's data with S3 tools. Before, an agent running product view ACCOUNT/PRODUCT --json saw the mirrors' backend prefix and the website URL, but not the path through the S3 data proxy, so it copied the backend prefix. Now --json carries s3_uri (s3://ACCOUNT/PRODUCT/) and s3_endpoint (the proxy to send requests to), and the human view shows an S3 line with the endpoint it's read through.

# the S3 line sits under URL
source-coop product view my-org/my-product

# s3_uri and s3_endpoint on the product
source-coop product view my-org/my-product --json | jq -c '{s3_uri, s3_endpoint}'

# on each product of a listing
source-coop product list my-org --json | jq -r '.[].s3_uri'

# read public data with them, without signing in
p=$(source-coop product view my-org/my-product --json)
aws s3 ls "$(jq -r .s3_uri <<<"$p")" --endpoint-url "$(jq -r .s3_endpoint <<<"$p")" --no-sign-request

How it behaves

  • --json adds two derived fields to each product record from product view, product list, product create and product edit:

    • s3_uri: s3://ACCOUNT/PRODUCT/, from the record's account_id and product_id.
    • s3_endpoint: the --proxy-url / SOURCE_PROXY_URL the command ran with, trailing / removed (https://data.source.coop by default, https://data.staging.source.coop on staging). The proxy is path-style: the account is the bucket.

    Everything else is the API's record as before. Adding fields is non-breaking per the JSON contract. product delete --json is unchanged: the product is gone, so there's nothing to read.

  • Human output: product view, the details format of product list and the -i record screen (all product_lines) show S3 s3://ACCOUNT/PRODUCT/ on https://data.source.coop (the --proxy-url in use) under URL. The table format is unchanged, to keep it to NAME VISIBILITY TITLE. The line names the endpoint because a bare s3:// path pasted into an AWS tool goes to AWS itself, to a bucket named after the account, which may be someone else's.

  • Why s3_endpoint is included: the proxy URL was already in hand where product commands run (main.rs has args.site.proxy_url, the same value that picks the login environment, and service-account commands already get it through their Ctx). Product commands now take a small product::Site { web, proxy } instead of the site URL alone. Without the endpoint, an agent on staging (or anyone with --proxy-url set) would pair the right path with the wrong host, and auth status --json is a second call to learn it.

  • No new prompts, flags or exit codes.

Demos

These run against docs/recorder/stub.py, not a real API. record.py draws the stub's URLs as production's (source.coop, data.source.coop), so the records shown are made-up fixtures, not real ones.

product view: the S3 line, s3_uri and s3_endpoint in --json, then aws s3 ls with them

product view: the S3 line, s3_uri and s3_endpoint in --json, then aws s3 ls with them

product list: s3_uri on each product in --json, and the S3 line in the details format

product list: s3_uri on each product in --json, and the S3 line in the details format

product view -i: the product's screen shows the S3 line too

product view -i: the product's screen shows the S3 line too

Docs changes

  • Products → List and view: a paragraph on the S3 line, s3_uri and s3_endpoint, why not to use a mirror's prefix, and an aws s3 ls --no-sign-request example.
  • Scripts and AI agents → JSON: notes that a few derived fields, such as these, are added to the API's.
  • Re-recorded the README's tour (its --json now picks s3_uri; the S3 line shows its endpoint), product-create (picks s3_uri) and product-browse (the record screen has the S3 line) demos: Quick start, Products.
  • --proxy-url's help on the API commands says it is also product --json's s3_endpoint.
  • --help for product list --json and product view --json names the two fields.

Not in this PR

  • s3_uri from the source.coop API. It's derived here from the product's IDs. Kept in the CLI for now; worth moving to the API if other clients (the web UI, SDKs) need it.
  • delete --json prints the deleted record without s3_uri/s3_endpoint: there's no data to read once it's gone.
  • s3_uri in account view or other places that mention products. Only product records carry it. Worth adding if member or service-account --json starts embedding product records.

Testing

  • cargo fmt, cargo clippy --all-targets -- -D warnings, cargo test and cargo test --features staging: 181 passed each, rebased on docs: say in --help where the data is and that public data needs no sign-in #101.
  • One test added (json_adds_where_to_read_the_data_through_the_proxy: the record is kept, s3_uri and s3_endpoint are added, a trailing / on the proxy URL is dropped), and the product_lines test now expects the S3 line. Existing wiremock tests of list, create and edit run the new code with --json; they don't capture stdout, so the fields themselves are checked through with_s3, which takes each product by value so list --json copies nothing.
  • Ran the commands in the demos against the stub, including aws s3 ls … --no-sign-request against its S3 listing. Built the docs site locally and checked the new cross-page link resolves.
  • Not tried against the real or staging API or data proxy.

🤖 Generated with Claude Code

@claude

claude Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Claude finished @alukach's task in 13s —— View job


✅ No blocking issues — safe to merge.

I read the src/ diff. Nothing in it is a correctness or security problem. I didn't build the branch or run the tests, and I didn't open the docs preview or the demo recordings.

  • with_s3 takes each product by value and only adds fields when the record is a JSON object. It uses account_id/product_id through the existing name() helper, so list --json copies nothing.
  • report now takes Value, so create and edit --json get the same fields. delete --json is unchanged, as the PR says.
  • Trailing / is trimmed on both the web URL and the proxy URL, and a test covers the proxy case.
  • The S3 line includes the endpoint, so a bare s3:// path won't be sent to AWS by mistake.

Simplify (ponytail)

  • src/product.rs (Site struct): the Site { web, proxy } struct is only needed because both URLs now travel together. That's reasonable, so keep it.
  • src/product.rs (table tests): Layout::new(ListFormat::Table, …, SITE) now gets a Site it never uses for tables. Passing SITE there is harmless. If Layout only needs site for the details format, you could skip it for tables.

💰 Estimated review cost: $0.14 · 0m12s · 5 turns

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview: https://cli.source.coop/pr-preview/pr-103/

Built from 1aaf704. It goes live a minute or two after this run, once GitHub Pages deploys it, and is removed when the PR closes.

github-actions Bot added a commit that referenced this pull request Nov 9, 2026
alukach and others added 3 commits November 9, 2026 11:41
`product view` and `product list --json` add `s3_uri` (s3://ACCOUNT/PRODUCT/)
and `s3_endpoint` (the --proxy-url), so scripts and agents read the data
through the proxy instead of copying a mirror's backend prefix. The human
view, the details list and the -i screen show an S3 line under the URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review:
- The human S3 row reads `s3://ACCOUNT/PRODUCT/ on <proxy>`: a bare
  s3:// path pasted into an AWS tool goes to AWS itself, to a bucket named
  after the account.
- with_s3 takes each product by value, so list --json doesn't clone every
  record to add two fields.
- --proxy-url's help and the README say it is product --json's s3_endpoint.
- Re-record the README's product-create demo, which selects s3_uri.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alukach
alukach force-pushed the feat/product-s3-uri branch from e1e8e1a to 29ac447 Compare November 9, 2026 18:43
github-actions Bot added a commit that referenced this pull request Nov 9, 2026
…opy nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alukach
alukach merged commit 12a6b16 into main Nov 9, 2026
13 checks passed
@alukach
alukach deleted the feat/product-s3-uri branch November 9, 2026 19:11
github-actions Bot added a commit that referenced this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant