From f3552e8f18744d07b3be094897eaec2f18212089 Mon Sep 17 00:00:00 2001 From: Sushil Muthuvelkumar <123318620+sushilm20@users.noreply.github.com> Date: Wed, 28 Oct 2026 20:17:32 -0400 Subject: [PATCH] Record the 2026-10-28 catalog refresh and the REV cron's production CPU --- CLAUDE.md | 6 ++++-- README.md | 6 ++++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 09c42c8..21a75a8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -266,8 +266,8 @@ REV official feed (2026-10-26, REV gave permission; REV is no longer crawled): - **Source.** `apps/worker/src/rev-feed.ts` reads `https://dashboard.revrobotics.com/feeds/usa/products.json` (`vendorSources.rev.officialFeed`; REV has no `sitemap` now). One fetch on 2026-10-26 showed a JSON array of 976 rows with exactly `sku`, `name`, `availability_status` ("Available" / "Out of Stock"), `price` (USD number), `url` (www.revrobotics.com product page), and `image_url`, plus a `Last-Modified` header; it has no pack, stock count, category, or timestamp field. A trimmed copy is `fixtures/vendors/feeds/rev-products.json`. - **Mapping.** `parseRevFeed`: SKU, name, price, stock, and URL as given; `stockQty` null, category `uncategorized`, `source: 'feed'`, `parser: 'official-feed'`, `updatedAt` = the feed's Last-Modified (fetch time without one). Rows without a SKU, a name, or a REV product page are rejected, never repaired. Pack size comes only from REV's `-PKn` SKU suffix (387 rows); every other row is a single unit (checked across the whole feed: names like "1 Piece Shaft Collar ... 4 Pack" are REV-21-1911-PK4, and "2 Piece" collars are one product), and a bundle gets none. Names are never read for a count. `image_url` is not used: the permission covers listings, prices, and stock. - **Nothing carried over.** A published feed generation holds only feed rows, so a SKU REV does not list (REV-31-1152, the discontinued Control Hub) cannot come back from the crawled catalog. -- **Never crawled.** `runVendor` throws for a vendor with `officialFeed` before any request; the scheduled handler sends REV's cron (`36 * * * *`, still paused) to `runRevFeed`; `catalog-load.ts` stages REV only from the feed and refuses `--crawl` and `--spot` for it. The crawl tests that used REV as their BigCommerce example now use ServoCity; the page parser still reads REV fixture pages, but nothing calls it for REV. -- **Staged publish.** `runRevFeed` does one step per run: publish a staged generation once `propagationMs` has passed (re-validated with `validateStaged`, at least `revMinRecords` = 500, at most 20% below a complete live catalog), or, at most every six hours, fetch with the last ETag and stage a changed feed. A truncated (under 500), malformed, mostly unusable (over 10% rejected rows), or unreachable feed fails the run with a failed snapshot and leaves the live catalog. A staged generation that fails validation, or whose live generation changed since staging (a manual publish meanwhile), is deleted and its validators dropped. Each run first retires replaced generations past their day of rollback (`retireReplaced`, 20 keys). State: `cat:rev:feed` = `{ etag, lastModified, checkedAt, staged: { gen, stagedAt, updatedAt, records, over } }` in VENDOR_CATALOG. A saved feed file is staged only with `--updated` (REV's Last-Modified), never the file time. The cron path parses the feed (about 320 KB) inside the Worker; the free plan's 10 ms CPU was not measured for it, so a manual `catalog-load` run is the tested path until crons resume. +- **Never crawled.** `runVendor` throws for a vendor with `officialFeed` before any request; the scheduled handler sends REV's cron (`*/10 * * * *` since 2026-10-28, below) to `runRevFeed`; `catalog-load.ts` stages REV only from the feed and refuses `--crawl` and `--spot` for it. The crawl tests that used REV as their BigCommerce example now use ServoCity; the page parser still reads REV fixture pages, but nothing calls it for REV. +- **Staged publish.** `runRevFeed` does one step per run: publish a staged generation once `propagationMs` has passed (re-validated with `validateStaged`, at least `revMinRecords` = 500, at most 20% below a complete live catalog), or, at most every six hours, fetch with the last ETag and stage a changed feed. A truncated (under 500), malformed, mostly unusable (over 10% rejected rows), or unreachable feed fails the run with a failed snapshot and leaves the live catalog. A staged generation that fails validation, or whose live generation changed since staging (a manual publish meanwhile), is deleted and its validators dropped. Each run first retires replaced generations past their day of rollback (`retireReplaced`, 20 keys). State: `cat:rev:feed` = `{ etag, lastModified, checkedAt, staged: { gen, stagedAt, updatedAt, records, over } }` in VENDOR_CATALOG. A saved feed file is staged only with `--updated` (REV's Last-Modified), never the file time. The cron path's CPU was measured and split into steps on 2026-10-28 (below). - **REV's display terms.** Every REV listing links to `record.url`, REV's product page. `priceDate` (`app/model.ts`) shows "REV feed updated {date, time}" for official-feed records beside every price and stock: Buy list rows and summary, the part drawer (status strip, an Updated row, product choices), Overview's money card and Buy by vendor, and the Control Point preview caption; crawled records keep "as of {date}". The workbook's Buy list and the CSV gain a "Price and Stock As Of" column after Stock (14 Buy list columns; Line Total formulas unchanged), and Exceptions stock rows carry the same time. The Links sheet shows no prices, so it is unchanged. - **Not done here.** Publishing the feed to production KV is a separate owner-approved step (`catalog-load.ts stage --vendor rev`, `validate`, `publish`). @@ -275,6 +275,8 @@ REV feed schedule and catalog refresh (2026-10-28, owner request "vendor data ju - **REV cron.** `crons = ["*/10 * * * *"]`, mapped to REV in `cronVendors`; the four crawls stay off and a test checks that only REV is scheduled. REV's feed was published by hand on 2026-10-27 and nothing refreshed it. - **CPU.** Staging all 976 rows in one run took about 17 ms in a fresh Node process (parse 10 ms, stage 7 ms; JSON.parse of the 322 KB body alone is 0.5 ms), over the free plan's 10 ms. `runRevFeed` therefore takes one step per run: a download (at most every six hours, with the ETag) checks the body is a list of at least 500 rows and keeps it at `cat:rev:feed:body`; each later run stages `revRowsPerRun` (125) rows as one batch, appending to the index; the last writes the manifest and deletes the body; a later run validates and publishes as before. Measured per step on the real feed, one fresh process each: 3.4–7.4 ms, idle runs 0.5–1.1 ms. `FeedState.building` holds the build (gen, the download's validators, rows done, counts, first rejection, `over`); the top-level ETag is replaced only when staging completes. A SKU listed twice keeps its first row (the one-run `stageCatalog` path keeps the last). A build is dropped, with its batches and body, when it fails, when a catalog is published by hand meanwhile, or three hours after its download (`revMaxBuildHours`), so a step stopped at the CPU limit cannot retry forever. - **KV writes.** An idle run writes nothing; an unchanged check writes the state; a changed feed costs about 35 writes and deletes from download to publish, about 100 minutes. +- **In production.** The first cron cycle downloaded at 16:21 UTC and published at 17:50 UTC: one download, eight row steps, one publish, no retries, so no step was stopped. A second cycle forced at 21:22 UTC (state cleared by hand, one more GET of the feed) was traced with `wrangler tail`: Cloudflare's CPU time per run was 5 ms for the download, 8, 7, 9, 23, 11, 7, and 8 ms for the row steps, 14 ms for the final staging step, and 2 ms for an idle run; the tail dropped before the publish run, which published at 23:00 UTC. Every traced run ended `ok`, including the three above 10 ms, so the free plan did not stop them, but that margin is not guaranteed. Smaller steps barely help (50 rows took 3.0–5.6 ms in Node against 3.4–7.4 ms for 125) because most of a step is fixed per-invocation cost, and 20 steps would outlast the three-hour cutoff. If Cloudflare starts stopping steps, the cutoff ends the build and the live catalog stays. +- **Slow re-crawl.** goBILDA, ServoCity, AndyMark, and Pitsco were re-crawled locally with `runVendor` (an untracked driver like PR 7c's: 400 pages per run, 500 ms between page requests, 1,000 ms between Shopify feed pages) and published with `catalog-load.ts` (`--expect` HORS SKUs, `--spot 20`, all 20/20): goBILDA 1,991 SKUs (16:13 UTC), ServoCity 4,318 (21:33 UTC, with Pack Info read from the page's own product: 4,287 verified packs), AndyMark 5,279 (was 5,313), Pitsco 1,625. Each kept its previous generation for a day of rollback. REV was not crawled. ## Tuning, tests, and release diff --git a/README.md b/README.md index b7a706d..fbc0888 100644 --- a/README.md +++ b/README.md @@ -97,9 +97,9 @@ Failed parts are also shown in the app. If every part fails, no ZIP is produced Vendor crawls are **paused** since 2026-10-25 at the owner's request: they used most of the free plan's daily KV writes. goBILDA, ServoCity, AndyMark, and Pitsco are refreshed instead by a slow local crawl loaded with `scripts/catalog-load.ts` (see one-time loads below). The only scheduled cron is REV's official feed, every ten minutes (`*/10 * * * *`). When crawls are enabled again, each vendor's hourly cron is listed in `wrangler.toml`: goBILDA at :00, ServoCity at :12, AndyMark at :24, and Pitsco at :48. Each run respects `robots.txt` and stays within [Cloudflare's 50-subrequest free-plan limit](https://developers.cloudflare.com/changelog/post/2026-02-11-subrequests-limit/). -- **AndyMark and Pitsco** are Shopify stores. A run reads up to four pages of the store's public `/products.json` feed, 250 products each, with every variant's SKU, price, and availability. A full catalog takes two or three runs (AndyMark: 9 pages, 5,313 SKUs; Pitsco: 6 pages, 1,625 SKUs, as of 2026-10-25). Fetching their product pages one by one drew HTTP 429 responses. +- **AndyMark and Pitsco** are Shopify stores. A run reads up to four pages of the store's public `/products.json` feed, 250 products each, with every variant's SKU, price, and availability. A full catalog takes two or three runs (AndyMark: 9 pages, 5,279 SKUs; Pitsco: 7 pages, 1,625 SKUs, as of 2026-10-28). Fetching their product pages one by one drew HTTP 429 responses. - **goBILDA and ServoCity** are BigCommerce stores with no public feed. A run fetches at most 40 product pages from the product sitemap, at no more than two requests per second. A first full crawl takes about 60 runs for goBILDA (2,387 pages) and 130 for ServoCity (5,152). -- **REV** is never crawled. Since 2026-10-26 REV's catalog comes only from its official feed, used with REV's permission: `https://dashboard.revrobotics.com/feeds/usa/products.json`, refreshed daily at midnight Central Time (`apps/worker/src/rev-feed.ts`; the copy read on 2026-10-28 carried a Last-Modified of 11:00 UTC, so the cron checks every six hours rather than at one fixed time). Its cron runs every ten minutes and takes one small step per run, so each stays within the free plan's 10 ms of CPU (staging all 976 rows in one run took about 17 ms): at most every six hours it downloads the feed with the last ETag; a changed feed is kept in KV and staged 125 rows per run, then published on a later run after validating it again. A changed feed goes live about 100 minutes after its download, and a run with nothing to do writes nothing (a refresh costs about 35 KV writes and deletes). An unreachable, malformed, or truncated feed (under 500 products, or over 10% unusable rows), or a build unfinished three hours after its download, leaves the live catalog as it was. REV's terms are shown in the app and workbook: every REV product links to its REV product page, and every REV price and stock shows the feed's update time. To load it by hand: `npx tsx scripts/catalog-load.ts stage --vendor rev`, then `validate` and `publish` as below (REV refuses `--crawl` and `--spot`, which would fetch its pages). +- **REV** is never crawled. Since 2026-10-26 REV's catalog comes only from its official feed, used with REV's permission: `https://dashboard.revrobotics.com/feeds/usa/products.json`, refreshed daily at midnight Central Time (`apps/worker/src/rev-feed.ts`; the copy read on 2026-10-28 carried a Last-Modified of 11:00 UTC, so the cron checks every six hours rather than at one fixed time). Its cron runs every ten minutes and takes one small step per run, to keep each run near the free plan's 10 ms of CPU (staging all 976 rows in one run took about 17 ms; in production the steps measured 5–14 ms, one 23 ms, all completing): at most every six hours it downloads the feed with the last ETag; a changed feed is kept in KV and staged 125 rows per run, then published on a later run after validating it again. A changed feed goes live about 100 minutes after its download, and a run with nothing to do writes nothing (a refresh costs about 35 KV writes and deletes). An unreachable, malformed, or truncated feed (under 500 products, or over 10% unusable rows), or a build unfinished three hours after its download, leaves the live catalog as it was. REV's terms are shown in the app and workbook: every REV product links to its REV product page, and every REV price and stock shows the feed's update time. To load it by hand: `npx tsx scripts/catalog-load.ts stage --vendor rev`, then `validate` and `publish` as below (REV refuses `--crawl` and `--spot`, which would fetch its pages). A rate-limited request (HTTP 429) is retried twice after the vendor's `Retry-After`, at most 10 seconds, before the run stops and resumes next hour. @@ -129,6 +129,8 @@ npx tsx scripts/catalog-load.ts publish --vendor gobilda --gen --expect 11 - While a published official feed is fresher than `VENDOR_STALE_HOURS`, the hourly crawl leaves that vendor alone. When the feed goes stale, crawling resumes as the fallback. - `--crawl ` stages a complete local crawl the same way. +**Slow re-crawl (how the crawled catalogs are refreshed while crons are paused).** Run `runVendor` locally against an in-memory KV saved to a file after every run, at most 2 requests per second for goBILDA and ServoCity pages and 1 per second for the AndyMark and Pitsco `/products.json` feeds, honouring `robots.txt` and `Retry-After`; a stopped crawl resumes from its file. Then `stage --crawl ` and `publish` with `--expect` and `--spot 20` as above. Last refresh, 2026-10-28: goBILDA 1,991 SKUs (about 25 minutes), ServoCity 4,318 (about 50 minutes), AndyMark 5,279 and Pitsco 1,625 (seconds each); every spot check matched 20/20 and every SKU count was within 1% of the catalog it replaced. They show as stale after `VENDOR_STALE_HOURS` (14 days), around 2026-11-11. + **Library mappings and images.** A parts library's part → vendor SKU mapping (for example the FTC Parts Library or FRCDesign) lives in VENDOR_CATALOG at `library-map:{documentId}` as a list of `LibraryMapping`. A mapped part matches exactly through the same SKU rules, with the library credited in its match reasons. Product images are shown only from a source that granted permission (`LicensedImage` with its attribution), in a thumbnail slot that renders nothing until one exists. No library mapping or image is loaded yet. **Storage and publication.** Each run appends its records as one batch and updates the generation's SKU/name index, so a run costs about five KV writes regardless of catalog size. [Free Workers KV allows 1,000 writes per day](https://developers.cloudflare.com/kv/platform/limits/).