Move the data pipeline to GitHub Actions and Cloudflare R2

[nofurtherinformation]

Summary

Moves the data pipeline from AWS Batch and nectr to GitHub Actions and Cloudflare R2. There are two buckets: a private pesticide-data-raw for inputs, and pesticide-data for what the site serves. The explorer (cpr-explorer-v2) will read static parquet files from R2 with DuckDB-WASM. fe-migration/PLAN.md is the plan for that side.

New flow for adding a year:

1. Upload the CalPIP year zip to pesticide-data-raw/calpip/ with Cyberduck and an uploader key scoped to that bucket. No Cloudflare account role is needed.
2. Run Build data. It checks the cleaning logic, builds, and uploads an immutable v<date>-<time>/ folder. It then points preview.json (the site's ?preview link) at it and writes yearly totals next to the live build's, plus any chemicals missing from PAN's category sheet, to the run summary.
3. Run Publish data with that version name. It points manifest.json at the build; any earlier version also works, which is how you roll back. The FE reads the manifest on load, so data updates need no FE rebuild.

Docs:

• docs/updating-the-data.pdf is the step-by-step guide for PAN and replaces the March 2025 AWS guide. Once this merges, GitHub Pages will serve it at https://open-spatial-lab.github.io/cpr/updating-the-data.pdf.
• RUNBOOK.md covers setup, the FE cutover, AWS teardown and routine work.

Why AWS Batch isn't needed any more

Batch existed for RAM. With the clean step in DuckDB, the whole run takes under a minute. The "after" figures below were measured with DuckDB capped at 2 GB. Uncapped, DuckDB uses up to 80% of RAM: locally build_r2 reached 7 GB, which still fits a standard public-repo runner (4 vCPUs, 16 GiB).

Step	Before	After
clean (7 years)	11.1 GB, 200 s	1.2 GB with DuckDB capped at 2 GB, 7 s
meta	9.3 GB	6.0 GB
join (nectr's tables)	not measured; the largest step	removed
build_r2 (new)		2.7 GB with the cap, 30 s

Verification

All of this was run, not just read, in a fresh Python 3.12 venv with the pinned requirements.txt.

• calpip_full: matches the old pipeline's on all 31,205,967 rows except the township ID, which now comes from COMTRS. 1,405 rows (182,696 lbs) that the old join dropped now get an ID, and 18 malformed IDs are fixed.
• cli/check_clean.py and cli/check_build.py: they run the clean and build steps on tiny synthetic datasets in under a second, and assert every output row and column. Together they caught 22 of 22 mutants (10 in clean, 12 in build). They run in a new Checks workflow on every PR and at the start of Build data.
• FE contract: fe-migration/reference_queries.py reproduces 25 snapshots of the live nectr API against v1, both locally and read straight from R2. Newer builds report the township case as an expected difference.
• R2 v1/: all 26 files match the local build on schema, rows and sums.
• Workflows: actionlint passes on all four.
• Not yet run on GitHub: it needs the setup below.

Reviews

Running /pr-review (5 single-factor reviewers plus a false-positive pass) found no blocking issues and 13 Important ones. All 13 are fixed in 1344cbf and 0b13d32:

• Clean step:
  • an uploaded zip's inner file name could inject SQL;
  • a line break inside a field, which the 2021 export has, crashed the read;
  • there was no test for the zip path.
• Lookups: products and sites newer than the PUR code tables were missing from the dropdowns. Their names now come from the CalPIP exports: 24 products and 3 sites.
• Reference check: NaN slipped through; the township case failed every new build; and start, end and usetype weren't escaped.
• Workflows:
  • there were no permissions: blocks;
  • the publish hints were misleading, including "undo with nothing";
  • Build and Publish shared one concurrency group.
• Dependencies: they weren't pinned.
• Transition:
  • the guide described a path that doesn't update the live site yet;
  • the old path had been removed;
  • PLAN pointed at a local path.

Setup before the first run (see RUNBOOK.md)

• Done on R2: v1/ and the manifests are in pesticide-data, with cache headers set, and a rehearsal of Build data against R2 shows 0.00% for every year.
• R2, still to do:
  • create the private pesticide-data-raw and move the inputs into it;
  • remove the inputs from pesticide-data;
  • set CORS and the custom domain on pesticide-data only.
• GitHub:
  • create a data environment limited to main, holding R2_ACCOUNT_ID, R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY;
  • move the old AWS_*/BATCH_* secrets into it;
  • require pull requests on main;
  • set the variables R2_BUCKET=pesticide-data and R2_RAW_BUCKET=pesticide-data-raw, and SITE_URL after the FE cutover.
• Keys: one CI key covering both buckets, and one uploader key per person for pesticide-data-raw only.

A second review from @timhannifan raised 6 Important items, all fixed in 50e45a0:

• uploader keys could reach the site's code, so the buckets are now split;
• anyone with Write access could read repo secrets, so the jobs now run in a main-only environment;
• a runbook rclone sync would have deleted tracked files;
• three test gaps, closed by the hardened clean check and the new build check.

The replies are inline.

Transition

• Until the FE reads from R2, Publish data doesn't change the live site.
• Actions → Run AWS Batch Job (old pipeline) stays available until then. Its configure-aws-credentials action is bumped from v2, which no longer runs on GitHub, to v4.
• Tearing down Batch, ECR, the S3 buckets and nectr comes after the cutover. docs/*.html still call nectr.

🤖 Generated with Claude Code

[timhannifan]

Claude (automated review):

Verdict: APPROVE WITH COMMENTS (posted as a comment; approving is up to a maintainer)

Overview

This PR replaces AWS Batch and the nectr API with two GitHub Actions workflows (Build data and Publish data) and static parquet files on Cloudflare R2. It also ports the clean step from pandas to DuckDB, and adds a new build_r2 step, a front-end migration spec, a runbook and a guide for PAN. It delivers what the description promises.

I checked the main claims by running them, not just by reading:

• check_clean.py passes.
• reference_queries.py matches all 25 snapshots of the live API against the v1 build on R2.
• A full rebuild from the real R2 inputs, using the workflow's own download filters, takes about 30 seconds. It matches v1 on 22 of 26 files. The rest differ exactly where the PR says they would:
  • township totals, by +182,696 lbs
  • product and site lookups, by +24 and +3 entries
  • chemical yearly averages, by float noise in the last digit
• I also ran the old pandas cleaning and the new DuckDB cleaning on the same tricky inputs and got identical output.

The design is careful. Each build goes into its own folder that never changes. Writing the version's manifest last means a half-finished upload can never be published. Rollback is just publishing an older version. The workflow input handling is solid.

No finding blocks merge. The six Important items fall into three groups:

• Who can change what (two security notes). These are about the key and permission setup in RUNBOOK.md, not the code. They should be settled before PAN gets any keys.
• Test coverage (three notes). The tests don't cover the new aggregation code, and the one check that runs in the workflow misses a few realistic regressions. Each fix is small, and I have working drafts I can share.
• One runbook command that deletes files tracked in git.

I'd fix all six in this branch. None of them require changing the pipeline's design.

Severity summary

Severity	Count
High	0
Important	6
Personal preference	3
Opinion	7

The 6 Important findings are inline comments on this review. The preferences and opinions are below; none of them needs a change.

• RUNBOOK.md L23 OPINION [security] A custom domain makes the whole bucket readable — Once the domain is connected, anyone who knows a path can fetch anything in the bucket, including raw/ and the legacy root folders this public repo names. That's fine if none of it is private. Is that the case? No change needed if so.
• cli/build_r2.py L50 PREF [correctness] The prd_share key is looser than the old use_id — Partitioning by year and use_number pools every null use_number row in a year into one "use", and it splits use numbers that repeat across counties. On real data that's 2 null rows plus 2 collisions in 2021, which comes to 36 lbs out of 2.37 billion, so it's negligible today. Adding county_cd, comtrs, lbs_prd_used to the partition, as the old key did, would close it for future years. Your call.
• cli/build_r2.py L58–60 OPINION [correctness] max(prd) per cell can overcount when prodno cleaned to '0' — A cell with a junk product number can mix different products, so an AI filter returns the whole cell's product lbs. This is rare and probably not worth handling. No action needed.
• cli/build_r2.py L84 OPINION [correctness] The year range comes from every row's 2-digit DATE year — One mistyped date like 01-JAN-02 would make the site offer 2002 onward. Would taking the range from the files' YEAR values be safer? No action needed for the current data, which gives 2017–2023.
• cli/build_r2.py L47 OPINION [correctness] No DuckDB memory cap is set, unlike the PR table's "2.7 GB with the cap" — Locally build_r2 peaked at 7.0 GB. DuckDB's default limit is 80% of RAM, which is fine on a 16 GiB runner. This only affects how the PR description reads.
• fe-migration/reference_queries.py L81, L108 PREF [tests] EXPECTED_AFTER_V1 waives any township difference, even when checking v1 itself — A fake fixture with the wrong row count and made-up values still counted as "expected". You could apply the waiver only when the base isn't v1 and still require the row count to match. Also, make sure the waiver doesn't get carried over into pnpm parity.
• fe-migration/reference_queries.py L42–49 OPINION [correctness] A timeseries with no detail filter is sent to summary/, which errors — Both the usetype series without an ai_type filter and the product series without a product filter group by a column summary/ doesn't have. The FE's value checks probably prevent this case, but since this file is the spec for the TypeScript port, routing on keys too would make it safe.
• fe-migration/PLAN.md L67 OPINION [security] The TS port needs more than quote escaping — The Python version is safe because float() and the allowlists do real work. Spelling that out (numbers go through Number.isFinite; geo, series and usetype are checked against allowlists) would stop a literal port from building SQL from a shared link without validation. The impact is small, since it runs in the browser against public files.
• .github/workflows/build-data.yml L36–37 PREF [tests] check_clean only runs after merge, inside the manual Build — A ~10-line on: pull_request workflow would catch a broken clean step in review, not when PAN runs a build.
• .github/workflows/build-data.yml L19–26 OPINION [security] The R2 secrets are visible to every step — Today that gains an attacker nothing over an uploader key. If you split the buckets, moving the secrets to the Download, Upload and Compare steps keeps pip install and the parsers from seeing a key that now has more rights.

[timhannifan]

Claude (automated review):

IMPORTANT [security] Uploader keys can overwrite the DuckDB-WASM code every site visitor runs

Each PAN uploader gets Object Read & Write on all of pesticide-data. PLAN.md L58 has the site load its DuckDB worker JS and .wasm from duckdb-wasm/ in that same bucket (I confirmed it's there on R2). So one leaked Cyberduck key would let someone run their own JavaScript in every visitor's browser, put fake numbers on PAN's site, or delete old versions so rollback stops working. Build and Publish don't stand in the way. The runbook's own fallback is the cheap fix, and it's easiest before any keys go out: a second bucket for inputs that uploaders can write and CI can only read. At minimum, move duckdb-wasm/ somewhere uploader keys can't reach, and say in step 5 that the bucket holds site code.

[nofurtherinformation]

Claude Code:

Agreed. I went with the two-bucket split in 50e45a0.

• Build data now reads inputs from R2_RAW_BUCKET (pesticide-data-raw, private).
• Uploader tokens are scoped to that bucket only; the CI token covers both.
• pesticide-data will hold only v*/, the manifests and duckdb-wasm/.

RUNBOOK steps 3 and 5 cover creating the bucket and moving the inputs out. Today they're still in pesticide-data, but no uploader keys have been issued yet. I've updated the guide and README to say uploads go to pesticide-data-raw/calpip/.

[timhannifan]

Claude (automated review):

IMPORTANT [security] The Write role lets uploaders read repo secrets, including the old AWS keys

Write is needed to run workflows, but it also lets someone push a branch with a new workflow file. That workflow gets every repo-level secret, including AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. main also appears to have no branch protection. A hijacked PAN GitHub account would get the AWS keys. Fix: put the R2_* and AWS_* secrets in a GitHub Environment limited to main, add environment: to the three jobs, and require PRs on main. Uploaders can still run the workflows from main.

[nofurtherinformation]

Claude Code:

Agreed. All three jobs (Build, Publish and the old Batch trigger) now run in a data environment, in 50e45a0.

RUNBOOK step 6 covers the settings:

• create data with deployment branches limited to main;
• move the R2_* and the old AWS_*/BATCH_* secrets into it, then delete the repo-level copies;
• require pull requests on main.

I haven't changed any settings yet. The AWS secret values can't be read back, so someone who has them needs to re-enter them.

[timhannifan]

Claude (automated review):

IMPORTANT [correctness] rclone sync r2:pesticide-data/raw data deletes tracked repo files

raw/ on R2 has no sections/ or pur/*.json, so sync deletes data/sections/sections.parquet and the four data/pur/*.json files that git tracks. Use rclone copy r2:pesticide-data/raw data, and rclone sync only for raw/calpip into data/calpip if you want old year files cleared.

[nofurtherinformation]

Claude Code:

Good catch. I confirmed that data/pur/*.json and data/sections/sections.parquet are tracked.

• The runbook now says rclone copy and explains why.
• sync is suggested only for calpip/ → data/calpip, when stale years need clearing.
• The workflow's aws s3 sync has no --delete, so it never removes local files.

[timhannifan]

Claude (automated review):

IMPORTANT [tests] The check drops null-month rows and never counts rows, so duplicates and the DATE filter go unchecked

Results are stored in a dict keyed by year, filtered with where monthyear is not null. Mutation runs showed these changes pass the check:

• reading every row twice, which would double every total on the site
• removing the DATE filter in build_full
• breaking the filter's 'nan' handling

The same filter hides the second half of the split record, which has a null month and shifted junk. Fix: assert the full row list (or at least count(*)) without the filter, add a blank-DATE row, and have build_full drop rows whose month doesn't parse (clean_calpip.py L85: and monthyear is not null). That last change also stops a stray null month from printing a "None … removed" line in the Build summary, which the guide tells PAN to treat as a stop sign. Real data has one such row today, and it happens to join no geography. I have a hardened draft of this check that kills all four surviving mutants, and can share it.

[nofurtherinformation]

Claude Code:

Confirmed: all three of those mutants passed the old check.

• check_clean.py now asserts every column of the full output with no filter. It also adds a blank-DATE row and a 'nan'-DATE row.
• build_full drops rows whose month doesn't parse (where monthyear is not null). This replaces the separate DATE filter.
• On real data that removes exactly one row: the shifted half of the split 2021 record (0 AI lbs, 1.5 product lbs). Nothing else in the 31.2M rows changes.

The hardened check now catches all 10 mutants I ran, including yours.

[timhannifan]

Claude (automated review):

IMPORTANT [tests] use_number, comtrs, aerial_ground and prodchem_pct are never asserted

Mutants that set use_number to NULL, take aerial_ground from the wrong column, or null out comtrs all pass. use_number drives the prd_share split, comtrs is the join key for four geographies, and aerial_ground is a filter on the site. Add the four columns to the select and give the fixture values for AERIAL_GROUND_INDICATOR and PRODUCT_CHEMICAL_PERCENT.

[nofurtherinformation]

Claude Code:

Done in 50e45a0. All four columns are asserted now, and the fixture sets AERIAL_GROUND_INDICATOR ('G') and PRODUCT_CHEMICAL_PERCENT (25.5). These mutants now fail the check:

• null use_number;
• aerial_ground taken from the wrong column;
• null comtrs;
• zeroed prodchem_pct.

[timhannifan]

Claude (automated review):

IMPORTANT [tests] The aggregation behind every number on the site has no automated check

The prd_share split, area-ratio weighting, the per-cell prd in use/, and the manifest years are untested. The compare step can't catch problems there either. It reads only summary/county.parquet, so my rebuild showed 0.00% for every year even though the township tables changed by 182,696 lbs. An 85-line synthetic check I drafted that runs in 0.16 s and catches 10 of 12 realistic mutations. Either add a trimmed version, or add a step to Build data that runs python fe-migration/reference_queries.py data/r2. That step covers all six geographies on real data in a few seconds.

[nofurtherinformation]

Claude Code:

I added cli/check_build.py in 50e45a0. It runs build_r2 on a 6-row synthetic dataset in 0.1 s and asserts:

• summary/ for all six geographies;
• use/county and use/tract;
• the geo table's values and types;
• the manifest and the counties lookup.

It caught all 12 mutants I tried:

• product lbs not split across AI rows;
• the year missing from the use key;
• the area ratio not applied in use/ or summary/;
• prodno missing from the cell;
• non-ag leaking into tract or section;
• per-row instead of per-cell prd;
• the county code join;
• bigint demography;
• the square-mile conversion;
• the manifest start year.

It runs in a new Checks workflow on every PR and at the start of Build data.

I didn't make reference_queries.py a Build gate. Its fixtures are 2021–2023 snapshots, so a legitimate re-pull of those years would fail the build.

[nofurtherinformation]

Claude Code:

Thanks for the careful review. The six Important items are answered inline. On the preferences and opinions from the review body:

• Whole bucket readable via the custom domain (RUNBOOK L23): resolved by the bucket split. The public pesticide-data will hold only builds, manifests and duckdb-wasm/. The inputs move to the private pesticide-data-raw.

• prd_share key looser than use_id (build_r2 L50): adopted. The key is now year, use number, county, COMTRS and product lbs. On real data it moves 2021 product totals by 37 lbs out of 333M, which matches your estimate, and still shows 0.00% against v1.

• max(prd) with prodno cleaned to '0': agreed; no change.

• Year range from the DATE year: no change. A stray year would show up as a new year row in the Build summary's yearly table, and the guide tells the reviewer to check that table before publishing.

• No DuckDB memory cap: no change. The PR description now says the 2.7 GB figure was measured with a 2 GB cap and that the runner uses DuckDB's default of 80% of RAM.

• EXPECTED_AFTER_V1: tightened. The waiver now applies only when the build's manifest isn't v1, and only if the total row count still matches. A v1-labelled copy with a township difference now fails. PLAN.md says not to port the waiver into pnpm parity.

• Timeseries without a detail filter: use_by now reads use/ whenever the grouping needs chem_code or prodno.

• TS port needs more than escaping: PLAN.md now lists the checks:

  • numbers through Number.isFinite;
  • allowlists for geo and series;
  • usetype limited to AG/NON-AG/*;
  • YYYY-MM for start and end.

  reference_queries.py now does the usetype and date checks itself, so the spec matches the plan.

• check_clean only after merge: added .github/workflows/checks.yml, which runs both checks on every pull request without secrets.

• Secrets visible to every step: the R2 key is now set only on the Download inputs, Upload and Fetch-the-live-build steps. It isn't visible to pip install or the parsers.

Re-verified at 50e45a0:

• both checks pass;
• actionlint is clean on all 4 workflows;
• a rebuild from the real R2 inputs shows 0.00% for every year against v1;
• the reference check passes 25/25 against v1, both locally and on R2.

Repo settings and R2 changes (the environment, main protection, creating pesticide-data-raw, keys) are in RUNBOOK steps 3, 5 and 6. None of them have been applied yet.