REV feed on a ten-minute cron, staged in steps within the free plan's CPU

CLAUDE.md

@@ -271,6 +271,11 @@ REV official feed (2026-10-26, REV gave permission; REV is no longer crawled):
 - **REV's display terms.** Every REV listing links to `record.url`, REV's product page. `priceDate` (`app/model.ts`) shows "REV feed updated {date, time}" for official-feed records beside every price and stock: Buy list rows and summary, the part drawer (status strip, an Updated row, product choices), Overview's money card and Buy by vendor, and the Control Point preview caption; crawled records keep "as of {date}". The workbook's Buy list and the CSV gain a "Price and Stock As Of" column after Stock (14 Buy list columns; Line Total formulas unchanged), and Exceptions stock rows carry the same time. The Links sheet shows no prices, so it is unchanged.
 - **Not done here.** Publishing the feed to production KV is a separate owner-approved step (`catalog-load.ts stage --vendor rev`, `validate`, `publish`).
 
+REV feed schedule and catalog refresh (2026-10-28, owner request "vendor data just recrawl slowly"):
+- **REV cron.** `crons = ["*/10 * * * *"]`, mapped to REV in `cronVendors`; the four crawls stay off and a test checks that only REV is scheduled. REV's feed was published by hand on 2026-10-27 and nothing refreshed it.
+- **CPU.** Staging all 976 rows in one run took about 17 ms in a fresh Node process (parse 10 ms, stage 7 ms; JSON.parse of the 322 KB body alone is 0.5 ms), over the free plan's 10 ms. `runRevFeed` therefore takes one step per run: a download (at most every six hours, with the ETag) checks the body is a list of at least 500 rows and keeps it at `cat:rev:feed:body`; each later run stages `revRowsPerRun` (125) rows as one batch, appending to the index; the last writes the manifest and deletes the body; a later run validates and publishes as before. Measured per step on the real feed, one fresh process each: 3.4–7.4 ms, idle runs 0.5–1.1 ms. `FeedState.building` holds the build (gen, the download's validators, rows done, counts, first rejection, `over`); the top-level ETag is replaced only when staging completes. A SKU listed twice keeps its first row (the one-run `stageCatalog` path keeps the last). A build is dropped, with its batches and body, when it fails, when a catalog is published by hand meanwhile, or three hours after its download (`revMaxBuildHours`), so a step stopped at the CPU limit cannot retry forever.
+- **KV writes.** An idle run writes nothing; an unchanged check writes the state; a changed feed costs about 35 writes and deletes from download to publish, about 100 minutes.
+
 ## Tuning, tests, and release
 
 Match normalization, thresholds, and alternative ranking are in `packages/shared/src/matching.ts`; an exact SKU still forces Buy absent a manual override. Disposition rules (`decideDisposition`), word lists, and print folders are in `packages/shared/src/classifier.ts`; order quantities are in `packages/shared/src/purchasing.ts`. Vendor stale hours and cron schedule are in `wrangler.toml`; catalog crawl batch limits are in `apps/worker/src/vendor-ingest.ts`. Scan request flow, source cache, and backoff are in `apps/worker/src/scan.ts`; BOM column mapping is in `packages/shared/src/bom.ts`; STEP export job bounds (per-call request budget, 5-minute translation timeout, 20 MB file cap) are in `apps/worker/src/step-export.ts`. Change these with fixture tests and record any changed data shape here.

README.md

@@ -95,11 +95,11 @@ Failed parts are also shown in the app. If every part fails, no ZIP is produced
 
 ## Vendor catalog refresh
 
-Vendor crawls are **paused** since 2026-10-25 at the owner's request: they used most of the free plan's daily KV writes, and the published catalogs stay as they are (staged loads with `scripts/catalog-load.ts` still work). When enabled, five staggered Worker crons run every hour: goBILDA at :00, ServoCity at :12, AndyMark at :24, REV at :36 (its official feed, never a crawl), and Pitsco at :48; the schedule is kept as a comment in `wrangler.toml`. Each run respects `robots.txt` and stays within [Cloudflare's 50-subrequest free-plan limit](https://developers.cloudflare.com/changelog/post/2026-02-11-subrequests-limit/).
+Vendor crawls are **paused** since 2026-10-25 at the owner's request: they used most of the free plan's daily KV writes. goBILDA, ServoCity, AndyMark, and Pitsco are refreshed instead by a slow local crawl loaded with `scripts/catalog-load.ts` (see one-time loads below). The only scheduled cron is REV's official feed, every ten minutes (`*/10 * * * *`). When crawls are enabled again, each vendor's hourly cron is listed in `wrangler.toml`: goBILDA at :00, ServoCity at :12, AndyMark at :24, and Pitsco at :48. Each run respects `robots.txt` and stays within [Cloudflare's 50-subrequest free-plan limit](https://developers.cloudflare.com/changelog/post/2026-02-11-subrequests-limit/).
 
 - **AndyMark and Pitsco** are Shopify stores. A run reads up to four pages of the store's public `/products.json` feed, 250 products each, with every variant's SKU, price, and availability. A full catalog takes two or three runs (AndyMark: 9 pages, 5,313 SKUs; Pitsco: 6 pages, 1,625 SKUs, as of 2026-10-25). Fetching their product pages one by one drew HTTP 429 responses.
 - **goBILDA and ServoCity** are BigCommerce stores with no public feed. A run fetches at most 40 product pages from the product sitemap, at no more than two requests per second. A first full crawl takes about 60 runs for goBILDA (2,387 pages) and 130 for ServoCity (5,152).
-- **REV** is never crawled. Since 2026-10-26 REV's catalog comes only from its official feed, used with REV's permission: `https://dashboard.revrobotics.com/feeds/usa/products.json`, refreshed daily at midnight Central Time (`apps/worker/src/rev-feed.ts`). Its cron checks the feed at most every six hours with the last ETag, stages a changed feed, and publishes it on a later run after validating it again; an unreachable, malformed, or truncated feed (under 500 products, or over 10% unusable rows) leaves the live catalog as it was. REV's terms are shown in the app and workbook: every REV product links to its REV product page, and every REV price and stock shows the feed's update time. To load it by hand: `npx tsx scripts/catalog-load.ts stage --vendor rev`, then `validate` and `publish` as below (REV refuses `--crawl` and `--spot`, which would fetch its pages).
+- **REV** is never crawled. Since 2026-10-26 REV's catalog comes only from its official feed, used with REV's permission: `https://dashboard.revrobotics.com/feeds/usa/products.json`, refreshed daily at midnight Central Time (`apps/worker/src/rev-feed.ts`; the copy read on 2026-10-28 carried a Last-Modified of 11:00 UTC, so the cron checks every six hours rather than at one fixed time). Its cron runs every ten minutes and takes one small step per run, so each stays within the free plan's 10 ms of CPU (staging all 976 rows in one run took about 17 ms): at most every six hours it downloads the feed with the last ETag; a changed feed is kept in KV and staged 125 rows per run, then published on a later run after validating it again. A changed feed goes live about 100 minutes after its download, and a run with nothing to do writes nothing (a refresh costs about 35 KV writes and deletes). An unreachable, malformed, or truncated feed (under 500 products, or over 10% unusable rows), or a build unfinished three hours after its download, leaves the live catalog as it was. REV's terms are shown in the app and workbook: every REV product links to its REV product page, and every REV price and stock shows the feed's update time. To load it by hand: `npx tsx scripts/catalog-load.ts stage --vendor rev`, then `validate` and `publish` as below (REV refuses `--crawl` and `--spot`, which would fetch its pages).
 
 A rate-limited request (HTTP 429) is retried twice after the vendor's `Retry-After`, at most 10 seconds, before the run stops and resumes next hour.
 
@@ -152,7 +152,7 @@ Linking and sending follow the two-sided contract in [docs/control-point-link.md
 
 ## Deploy and verify
 
-The production Cloudflare Pages project, Worker, and two KV namespaces already exist. The namespace IDs and the (paused) crawl schedule are in `wrangler.toml`. In a new account, create equivalent Pages, Worker, and KV resources first and replace the namespace IDs. Put `SESSION_ENCRYPTION_KEY`, `ONSHAPE_OAUTH_CLIENT_ID`, and `ONSHAPE_OAUTH_CLIENT_SECRET` in the Worker as secrets, not in `wrangler.toml`, Pages variables, or Git. The optional AI review needs `GEMINI_API_KEY` and/or `ANTHROPIC_API_KEY` as Worker secrets (`pnpm --filter @control-kit/worker exec wrangler secret put GEMINI_API_KEY --config ../../wrangler.toml`, typed at the prompt, never in chat or Git); without either the review stays off and nothing else changes. Control Point linking needs `CONTROL_KIT_CLIENT_SECRET`, the same value Control Point holds in its Render environment, set the same way; without it linking stays off. The owner created the OAuth app and installed all three encrypted Worker secrets on 2026-10-23; production now returns `{"connected":false}` to an unauthenticated `/api/session` request rather than `setupPending`.
+The production Cloudflare Pages project, Worker, and two KV namespaces already exist. The namespace IDs and the cron schedule (REV's feed only; crawls paused) are in `wrangler.toml`. In a new account, create equivalent Pages, Worker, and KV resources first and replace the namespace IDs. Put `SESSION_ENCRYPTION_KEY`, `ONSHAPE_OAUTH_CLIENT_ID`, and `ONSHAPE_OAUTH_CLIENT_SECRET` in the Worker as secrets, not in `wrangler.toml`, Pages variables, or Git. The optional AI review needs `GEMINI_API_KEY` and/or `ANTHROPIC_API_KEY` as Worker secrets (`pnpm --filter @control-kit/worker exec wrangler secret put GEMINI_API_KEY --config ../../wrangler.toml`, typed at the prompt, never in chat or Git); without either the review stays off and nothing else changes. Control Point linking needs `CONTROL_KIT_CLIENT_SECRET`, the same value Control Point holds in its Render environment, set the same way; without it linking stays off. The owner created the OAuth app and installed all three encrypted Worker secrets on 2026-10-23; production now returns `{"connected":false}` to an unauthenticated `/api/session` request rather than `setupPending`.
 
 After `pnpm install --frozen-lockfile` and the checks above, deploy from the repository root:
 

apps/worker/src/index.test.ts

@@ -61,9 +61,10 @@ describe('match decisions', () => {
 describe('vendor crons', () => {
   it('runs every vendor from a cron that wrangler.toml schedules', () => {
     const crons = JSON.parse(readFileSync(new URL('../../../wrangler.toml', import.meta.url), 'utf8').match(/^crons = (\[.*\])$/m)![1]) as string[];
-    // Paused at the owner's request: nothing is scheduled, and anything scheduled later must map to a vendor.
-    expect(crons).toEqual([]);
+    // Crawls are paused at the owner's request: only REV's official feed runs, and anything scheduled later must map to a vendor.
+    expect(crons).toEqual(['*/10 * * * *']);
     expect(crons.every(cron => cron in cronVendors)).toBe(true);
+    expect(crons.map(cron => cronVendors[cron])).toEqual(['rev']);
     expect(new Set(Object.values(cronVendors)).size).toBe(5);
   });
 });

apps/worker/src/index.ts

@@ -541,11 +541,12 @@ async function route(request: Request, env: Env): Promise<Response> {
   return reply({ error: 'Not found' }, 404);
 }
 /**
- * Each vendor crawls hourly, 12 minutes apart, when its cron is scheduled in wrangler.toml; REV's cron reads REV's official
- * feed instead and never crawls. The crons are paused (an empty list) at the owner's request; every cron that is
- * scheduled must be one of these (a test checks).
+ * Each vendor crawls hourly, 12 minutes apart, when its cron is scheduled in wrangler.toml. The crawls are paused at the
+ * owner's request; only REV's cron is scheduled. It never crawls: every ten minutes it takes one small step of reading
+ * REV's official feed (a download at most every six hours, then a few runs to stage and publish it), and a run with
+ * nothing to do writes nothing. Every cron that is scheduled must be one of these (a test checks).
  */
-export const cronVendors: Record<string, VendorId> = { '0 * * * *': 'gobilda', '12 * * * *': 'servocity', '24 * * * *': 'andymark', '36 * * * *': 'rev', '48 * * * *': 'pitsco' };
+export const cronVendors: Record<string, VendorId> = { '0 * * * *': 'gobilda', '12 * * * *': 'servocity', '24 * * * *': 'andymark', '*/10 * * * *': 'rev', '48 * * * *': 'pitsco' };
 
 export default {
   async fetch(request: Request, env: Env) {
@@ -566,7 +567,7 @@ export default {
   },
   scheduled(event: { cron: string }, env: Env, context: { waitUntil(promise: Promise<unknown>): void }) {
     const vendor = cronVendors[event.cron];
-    // REV is never crawled: its cron reads REV's official feed (at most every six hours) and publishes it once staged.
+    // REV is never crawled: its cron reads REV's official feed (at most every six hours) and stages and publishes it in steps.
     if (vendor === 'rev') context.waitUntil(runRevFeed(env.VENDOR_CATALOG, fetch));
     else if (vendor) context.waitUntil(runVendor(vendor, env.VENDOR_CATALOG, fetch, { feedFreshHours: Number(env.VENDOR_STALE_HOURS ?? 336) }));
   },