Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,11 @@ REV feed schedule and catalog refresh (2026-10-28, owner request "vendor data ju
- **In production.** The first cron cycle downloaded at 16:21 UTC and published at 17:50 UTC: one download, eight row steps, one publish, no retries, so no step was stopped. A second cycle forced at 21:22 UTC (state cleared by hand, one more GET of the feed) was traced with `wrangler tail`: Cloudflare's CPU time per run was 5 ms for the download, 8, 7, 9, 23, 11, 7, and 8 ms for the row steps, 14 ms for the final staging step, and 2 ms for an idle run; the tail dropped before the publish run, which published at 23:00 UTC. Every traced run ended `ok`, including the three above 10 ms, so the free plan did not stop them, but that margin is not guaranteed. Smaller steps barely help (50 rows took 3.0–5.6 ms in Node against 3.4–7.4 ms for 125) because most of a step is fixed per-invocation cost, and 20 steps would outlast the three-hour cutoff. If Cloudflare starts stopping steps, the cutoff ends the build and the live catalog stays.
- **Slow re-crawl.** goBILDA, ServoCity, AndyMark, and Pitsco were re-crawled locally with `runVendor` (an untracked driver like PR 7c's: 400 pages per run, 500 ms between page requests, 1,000 ms between Shopify feed pages) and published with `catalog-load.ts` (`--expect` HORS SKUs, `--spot 20`, all 20/20): goBILDA 1,991 SKUs (16:13 UTC), ServoCity 4,318 (21:33 UTC, with Pack Info read from the page's own product: 4,287 verified packs), AndyMark 5,279 (was 5,313), Pitsco 1,625. Each kept its previous generation for a day of rollback. REV was not crawled.

Matching fixes (2026-10-28, owner reports from the HORS panel: a Control Hub that needed confirming, a REV servo and a goBILDA bearing with no match):
- **Discontinued SKUs.** `successorOf` (`known-products.ts`) lists SKUs a vendor no longer sells with what it sells instead, checked against REV's feed of 2026-10-28: REV-31-1152 (Control Hub) is REV-31-1595; REV-41-1097 (Smart Robot Servo) is the V2, Balanced REV-41-3334 or UltraSpeed REV-41-3336. In `matchCatalog`, a part named or numbered with an old SKU becomes an exact match to a single successor ("REV-31-1152 is discontinued; REV Control Hub is sold as REV-31-1595") or a possible match with the successors as choices. The old SKU itself is never listed or priced.
- **Sizes and bores.** `spellSizes` turns "8x14x5mm" (or "8 × 14 × 5 mm") into "8mm 14mm 5mm" for library names and the tagged search; a configuration's own size field is ignored when the name gives sizes. `boreConflict` drops products with the other bore: "Round Bore" excludes goBILDA's REX/hex bore, and "REX" or "hex" excludes round ones.
- **Tagged exact match.** A vendor-tagged part ("(GB)") whose search leaves one product that carries every number in the name is an exact match ("8x14x5mm Bearing - Round Bore (GB)" is 1611-0514-0008; "6000rpm Motor - Bare REX (GB)" is 5203-2402-0001). Sizes written as one token must appear in the product name in the same order (bore, outside diameter, thickness), so "5x14x8mm" is never matched to the 8×14×5 bearing. Several products, or a name without numbers, stay a choice. A part carrying two discontinued numbers (a part number and a name for different products) gets every replacement as a choice.

## Tuning, tests, and release

Match normalization, thresholds, and alternative ranking are in `packages/shared/src/matching.ts`; an exact SKU still forces Buy absent a manual override. Disposition rules (`decideDisposition`), word lists, and print folders are in `packages/shared/src/classifier.ts`; order quantities are in `packages/shared/src/purchasing.ts`. Vendor stale hours and cron schedule are in `wrangler.toml`; catalog crawl batch limits are in `apps/worker/src/vendor-ingest.ts`. Scan request flow, source cache, and backoff are in `apps/worker/src/scan.ts`; BOM column mapping is in `packages/shared/src/bom.ts`; STEP export job bounds (per-call request budget, 5-minute translation timeout, 20 MB file cap) are in `apps/worker/src/step-export.ts`. Change these with fixture tests and record any changed data shape here.
Expand Down
68 changes: 67 additions & 1 deletion apps/worker/src/catalog.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -355,7 +355,8 @@ describe('parts library vendor tags', () => {
const [hub, motor, revHub] = result.lines;
expect(hub.matchStatus).toBe('possible');
expect(hub.choices?.map(choice => choice.sku).sort()).toEqual(['1908-0024-0032', '1908-0025-0032']);
expect(motor.matchStatus).toBe('possible');
// The tag, "REX", and 6000rpm leave one product: the bare 6000 RPM motor with a REX shaft, matched exactly.
expect(motor.matchStatus).toBe('exact');
expect(motor.vendor?.sku).toBe('5203-2402-0001');
expect(motor.choices?.map(choice => choice.sku)).not.toContain('5203-2402-0139');
expect(revHub.vendor?.sku).toBe('REV-11-1855');
Expand All @@ -371,4 +372,69 @@ describe('parts library vendor tags', () => {
await enrichScan(missing, catalog(shelf.filter(item => item.vendorId !== 'gobilda')));
expect(missing.lines[0]).toMatchObject({ matchStatus: 'unmatched', vendor: null });
});
it('reads sizes written as one token and the bore, and matches the one product with every size exactly', async () => {
clearCatalogMemoryCache();
// goBILDA's bearings with these sizes (2026-10-28): the round and REX bore 1611, and look-alikes one size off.
const bearings = [
record('gobilda', '1611-0514-0008', '1611 Series Flanged Ball Bearing (8mm ID x 14mm OD, 5mm Thickness) - 2 Pack', 4.99),
record('gobilda', '1611-0514-4008', '1611 Series Flanged Ball Bearing (8mm REX® ID x 14mm OD, 5mm Thickness) - 2 Pack', 4.99),
record('gobilda', '1601-0414-0008', '1601 Series Flanged Ball Bearing (8mm ID x 14mm OD, 4mm Thickness) - 2 Pack', 4.99),
record('gobilda', '1613-0514-0006', '1613 Series Thrust Ball Bearing (6mm ID x 14mm OD, 5mm Thickness)', 2.99),
];
const result = lines({ name: '8x14x5mm Bearing - Round Bore (GB)', configuration: 'List_a1=8×14×50;List_b2=Mediumm' }, { name: '8 × 14 × 5 mm Bearing - REX Bore (GB)' }, { name: '8x14mm Bearing (GB)' }, { name: '5x14x8mm Bearing - Round Bore (GB)' });
await enrichScan(result, catalog([...shelf, ...bearings]));
const [round, rex, unsized, reordered] = result.lines;
// The same numbers in another order are another bearing (5mm bore, 8mm thick), so it is never matched exactly.
expect(reordered.matchStatus).not.toBe('exact');
// A decimal size keeps its place: goBILDA's 8mm x 15.5mm x 5mm thrust bearing.
clearCatalogMemoryCache();
const thrust = lines({ name: '8x15.5x5mm Thrust Bearing (GB)' });
await enrichScan(thrust, catalog([...shelf, ...bearings, record('gobilda', '1613-0515-0008', '1613 Series Thrust Ball Bearing (8mm ID x 15.5mm OD, 5mm Thickness)', 2.99)]));
expect(thrust.lines[0]).toMatchObject({ matchStatus: 'exact' });
expect(thrust.lines[0].vendor?.sku).toBe('1613-0515-0008');
expect(round).toMatchObject({ matchStatus: 'exact' });
expect(round.vendor?.sku).toBe('1611-0514-0008');
expect(round.matchReasons[0]).toContain('every size in the name');
expect(rex.vendor?.sku).toBe('1611-0514-4008');
expect(rex.matchStatus).toBe('exact');
// Two sizes fit several bearings, so a person still picks.
expect(unsized.matchStatus).toBe('possible');
});
});

describe('discontinued SKUs', () => {
const record = (sku: string, name: string, priceUsd: number): VendorRecord => ({ vendorId: 'rev', sku, normalizedSku: normalizeSku(sku, 'rev'), name, normalizedName: normalizeName(name), url: 'https://www.revrobotics.com/x/', priceUsd, packSize: 1, packSizeVerified: true, stock: 'in', stockQty: null, category: 'uncategorized', updatedAt: new Date().toISOString(), source: 'feed', parser: 'official-feed' });
// REV's feed of 2026-10-28 lists neither REV-31-1152 nor REV-41-1097, only what replaced them.
const feed = [
record('REV-31-1595', 'Control Hub', 375),
record('REV-41-3334', 'Smart Robot Servos V2 (Smart Robot Servo V2 - Balanced (REV-41-3334))', 32.5),
record('REV-41-3336', 'Smart Robot Servos V2 (Smart Robot Servo V2 - UltraSpeed (REV-41-3336))', 32.5),
];
const lines = (...specs: Array<Partial<BomLine> & { name: string }>) => {
const result = aggregateAssembly(fixture('empty'), context, 'Robot');
const base = aggregateAssembly(fixture('simple'), context, 'Robot').lines[0];
result.lines = specs.map((spec, n) => ({ ...base, partNumber: null, configuration: null, bomLineKey: `line-${n}`, qty: 1, ...spec }));
return result;
};
it('orders a discontinued SKU as its successor, and asks which one when there are two', async () => {
clearCatalogMemoryCache();
const result = lines({ name: 'Control Hub REV-31-1152', partNumber: 'REV-31-1152' }, { name: 'Smart Robot Servo REV-41-1097' });
classifyScan(await enrichScan(result, catalog(feed)), { teamNumber: null }, {});
const [hub, servo] = result.lines;
expect(hub).toMatchObject({ matchStatus: 'exact', lineTotalUsd: 375 });
expect(hub.vendor?.sku).toBe('REV-31-1595');
expect(hub.matchReasons[0]).toBe('REV-31-1152 is discontinued; REV Control Hub is sold as REV-31-1595');
expect(servo.matchStatus).toBe('possible');
expect(servo.choices?.map(choice => choice.sku)).toEqual(['REV-41-3334', 'REV-41-3336']);
expect(servo.lineTotalUsd).toBeNull();
});
it('keeps every replacement a choice when a part names two discontinued products', async () => {
clearCatalogMemoryCache();
const result = lines({ name: 'Smart Robot Servo REV-41-1097', partNumber: 'REV-31-1152' });
await enrichScan(result, catalog(feed));
expect(result.lines[0].matchStatus).toBe('possible');
expect(result.lines[0].choices?.map(choice => choice.sku).sort()).toEqual(['REV-31-1595', 'REV-41-3334', 'REV-41-3336']);
expect(result.lines[0].matchReasons[0]).toContain('REV-31-1152 is discontinued');
expect(result.lines[0].matchReasons[0]).toContain('REV-41-1097 is discontinued');
});
});
8 changes: 5 additions & 3 deletions apps/worker/src/rev-feed.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -341,14 +341,16 @@ describe('a REV buy line from the feed', () => {
const context = { documentId: 'doc1', workspaceOrVersion: 'w' as const, workspaceOrVersionId: 'work1', elementId: 'elem1' };
const result = aggregateAssembly(fixture, context, 'Robot');
const base = result.lines[0];
result.lines = [{ ...base, name: 'Control Hub', partNumber: 'REV-31-1595', bomLineKey: 'hub', qty: 2 }, { ...base, name: 'Old Control Hub', partNumber: 'REV-31-1152', bomLineKey: 'old-hub', partId: 'old-hub', qty: 1 }];
result.lines = [{ ...base, name: 'Control Hub', partNumber: 'REV-31-1595', bomLineKey: 'hub', qty: 1 }, { ...base, name: 'Old Control Hub', partNumber: 'REV-31-1152', bomLineKey: 'old-hub', elementId: 'old-hub-studio', partId: 'old-hub', qty: 1 }];
classifyScan(await enrichScan(result, kv), { teamNumber: null }, {});
const hub = result.lines.find(line => line.bomLineKey === 'hub')!;
// The old hub is ordered as its successor, so the order is two hubs on the lead line.
expect(hub).toMatchObject({ disposition: 'buy', matchStatus: 'exact', packsToOrder: 2, lineTotalUsd: 750 });
expect(hub.vendor).toMatchObject({ sku: 'REV-31-1595', priceUsd: 375, url: 'https://www.revrobotics.com/rev-31-1595/', updatedAt: feedTime, parser: 'official-feed' });
// A part numbered with the discontinued SKU finds nothing: it was not in the feed, so no product or price is made up.
// The discontinued SKU itself never comes back: a part numbered with it is ordered as REV's current Control Hub.
const old = result.lines.find(line => line.bomLineKey === 'old-hub')!;
expect(old.vendor?.sku).not.toBe('REV-31-1152');
expect(old.vendor?.sku).toBe('REV-31-1595');
expect(old.matchReasons[0]).toContain('REV-31-1152 is discontinued');
expect(old.lineTotalUsd).toBeNull();
});
});
32 changes: 25 additions & 7 deletions packages/shared/src/catalog-read.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { configurationValues, findBestMatch, findLibraryMapping, libraryMapKey, nameCoverage, nameSimilarity, vendorTag, normalizeName, normalizeSku, rankVendorAlternatives, skuCandidates, type LibraryMapping, type MatchResult, type VendorId, type VendorRecord } from './matching';
import { boreConflict, configurationValues, coversNumbers, findBestMatch, findLibraryMapping, libraryMapKey, nameCoverage, nameSimilarity, vendorTag, normalizeName, normalizeSku, rankVendorAlternatives, skuCandidates, spellSizes, type LibraryMapping, type MatchResult, type VendorId, type VendorRecord } from './matching';
import { decisionRecord, recalculatePurchasing, type MatchDecision, type ScanResult } from './index';
import { knownProduct, withKnownPack } from './known-products';
import { knownProduct, successorOf, withKnownPack } from './known-products';

/**
* Reading published vendor catalogs. Storage layout (written by the Worker's ingestion and staged publish):
Expand Down Expand Up @@ -190,18 +190,36 @@ export async function matchCatalog(result: ScanResult, kv: CatalogReader, getDec
// Without that vendor's catalog the part stays unmatched rather than falling back to another vendor.
if (tag && !tagged) match = { vendor: null, confidence: 0, status: 'unmatched', reasons: [`no published ${tag.vendor} catalog to match "${tag.name}"`] };
if (tag && tagged) {
const wanted = [tag.name, ...configurationValues(line.configuration)].join(' ');
// Sizes in the name ("8x14x5mm") outrank a configuration's own size field, which may be written another way.
const namedSizes = spellSizes(tag.name) !== tag.name;
const wanted = [tag.name, ...configurationValues(line.configuration).filter(value => !(namedSizes && /\d\s*[x×]\s*\d/i.test(value)))].join(' ');
const positions = new Set<number>();
for (const token of tokensOf(wanted.replace(/(\d)(rpm)\b/gi, '$1 $2'))) for (const position of tagged.byToken.get(token) ?? []) positions.add(position);
const ranked = [...positions].map(position => [position, nameCoverage(wanted, tagged.entries[position][1])] as const)
for (const token of tokensOf(spellSizes(wanted).replace(/(\d)(rpm)\b/gi, '$1 $2'))) for (const position of tagged.byToken.get(token) ?? []) positions.add(position);
const ranked = [...positions].filter(position => !boreConflict(tag.name, tagged.entries[position][1]))
.map(position => [position, nameCoverage(wanted, tagged.entries[position][1])] as const)
.filter(([, score]) => score >= 0.5).sort((a, b) => b[1] - a[1] || a[0] - b[0]);
const close = ranked.filter(([, score]) => ranked[0][1] - score <= 0.25).slice(0, 4);
const records = (await Promise.all(close.map(async ([position]) => (await batch(tagged, tagged.entries[position][2])).get(tagged.entries[position][0]) ?? null)))
.filter((record): record is VendorRecord => Boolean(record));
match = records.length ? { vendor: records[0], confidence: 0.9, status: 'possible', choices: records,
reasons: [`"(${line.name.match(/\(([^)]*)\)\s*$/)![1]})" in the name means ${tag.vendor}; closest products by name and configuration, confirm which one`] }
const named = `"(${line.name.match(/\(([^)]*)\)\s*$/)![1]})" in the name means ${tag.vendor}`;
// One product left that carries every size and number the part names is that product: the tag fixes the vendor
// and the numbers fix the item ("8x14x5mm Bearing - Round Bore (GB)" is goBILDA's 8mm x 14mm x 5mm round-bore bearing).
match = records.length === 1 && /\d/.test(tag.name) && coversNumbers(tag.name, records[0].normalizedName)
? { vendor: records[0], confidence: 1, status: 'exact', reasons: [`${named}, and its only product with every size in the name is ${records[0].sku}`] }
: records.length ? { vendor: records[0], confidence: 0.9, status: 'possible', choices: records, reasons: [`${named}; closest products by name and configuration, confirm which one`] }
: { vendor: null, confidence: 0, status: 'unmatched', reasons: [`no ${tag.vendor} product matches "${tag.name}"`] };
}
// A part numbered with a SKU its vendor has discontinued is ordered as the product sold in its place.
// Every discontinued number the part carries counts: a part number and a name that name different old products
// (a hub and a servo) leave all their replacements as choices rather than pricing one of them.
const retired = !mapped && match.status !== 'exact' ? [...new Map(skus.map(successorOf).filter(item => item !== null).map(item => [item.old, item])).values()] : [];
if (retired.length) {
const wanted = [...new Map(retired.flatMap(item => item.skus).map(([vendor, sku]) => [`${vendor}:${sku}`, [vendor, sku] as const])).values()];
const records = (await Promise.all(wanted.map(([vendor, sku]) => recordAt(vendor, sku)))).filter((record): record is VendorRecord => Boolean(record));
const said = retired.map(item => `${item.old} is discontinued; ${item.product} is sold as ${item.skus.map(([, sku]) => sku).join(' or ')}`).join('. ');
if (records.length === 1 && wanted.length === 1) match = { vendor: records[0], confidence: 1, status: 'exact', reasons: [said] };
else if (records.length) match = { vendor: records[0], confidence: 0.9, status: 'possible', choices: records, reasons: [`${said}; pick one`] };
}
// A part named exactly like a known FTC product ("Pinpoint", "Control Hub") is that product; with several SKUs, a person picks.
const found = !mapped && match.status !== 'exact' ? knownProduct(tag ? tag.name : line.name) : null;
// A vendor tag keeps only that vendor's SKUs: "Control Hub (GB)" is not REV's Control Hub.
Expand Down
2 changes: 1 addition & 1 deletion packages/shared/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export type { ClassificationOverride, ClassificationOverrides, Confidence, Dispo
export type { Attribution, LibraryMapping, LicensedImage, VendorId, VendorRecord, MatchResult } from './matching';
export { buildControlPointPayload, controlPointIdempotencyKey, CONTROL_POINT_CAD_URL, ControlPointError, INLINE_FILE_LIMIT_BYTES, MockControlPointAdapter } from './control-point';
export { recalculatePurchasing, updatePurchasing } from './purchasing';
export { isBundle, knownProduct, withKnownPack } from './known-products';
export { isBundle, knownProduct, successorOf, withKnownPack } from './known-products';
export { AI_REVIEW_MAX_LINES, AI_REVIEW_VERSION, aiCandidateRecord, aiDispositions, aiReviewHash, aiReviewInstructions, aiReviewSchema, applyAiReview, buildAiReviewRequest, lineCandidates, needsAiReview, parseAiReview } from './ai-review';
export type { AiCandidate, AiDisposition, AiProvider, AiReviewItem, AiReviewLine, AiReviewRequest, AiSuggestion } from './ai-review';
import { recalculatePurchasing } from './purchasing';
Expand Down
Loading
Loading