feat(import): return per-source resolution metadata - #40
Conversation
WalkthroughURL imports now collect per-source resolution metadata. The metadata includes the requested, redirected, docs-base, and final URLs, the discovery route, and optional scrape-discard status. URL import results expose this data through the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/import.js`:
- Around line 823-830: Update the discard tracking around the resolution
assembly and orphan-triage flow so a general scrape-discarded boolean is set for
both coverage-gate and thin-scrape re-clustering paths. Keep the existing
coverage-specific flag for warning suppression if needed, and use the general
boolean in the `organized.resolution` construction to emit `scrapeDiscarded` for
either discard path.
- Around line 823-828: Update the route assignment in the organized.resolution
metadata to reflect the final organizer selected around the scraped/llms
handling: prefer scraped when it produced the final tree, and use llms only when
the scrape result was absent or discarded, while preserving the existing
Mintlify, Archbee, sitemap, and none labels. Add regression coverage for llms
combined with Mintlify, Archbee, and generic scrape discovery.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4f5704da-8a93-46f4-b288-aa111c127073
📒 Files selected for processing (1)
src/commands/import.js
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
readmeio/ai(manual)readmeio/gitto(manual)readmeio/markdown(manual)readmeio/readme(manual)
| organized.resolution = { | ||
| requestedUrl, | ||
| redirectedUrl: redirected ? redirected.toString() : null, | ||
| docsBaseUrl: docsBase ? docsBase.url.toString() : null, | ||
| finalUrl: sourceUrl.toString(), | ||
| route: llms ? 'llms' : mintlifyNav ? 'mintlify' : archbeeNav ? 'archbee' : scraped ? 'scrape' : sitemapUrl ? 'sitemap' : 'none', |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Make route match the final selected organizer.
Lines 722-728 use scraped when it is available. This includes Mintlify, Archbee, and generic scrape results. Line 828 checks llms first, so the metadata reports llms even when the final organized tree came from scraped.
Compute route from the final branch. Use llms only when the scrape result was discarded or absent.
Proposed route selection
- route: llms ? 'llms' : mintlifyNav ? 'mintlify' : archbeeNav ? 'archbee' : scraped ? 'scrape' : sitemapUrl ? 'sitemap' : 'none',
+ route: scraped
+ ? mintlifyNav
+ ? 'mintlify'
+ : archbeeNav
+ ? 'archbee'
+ : 'scrape'
+ : llms
+ ? 'llms'
+ : sitemapUrl
+ ? 'sitemap'
+ : 'none',Add regression tests for llms combined with Mintlify, Archbee, and generic scrape discovery.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| organized.resolution = { | |
| requestedUrl, | |
| redirectedUrl: redirected ? redirected.toString() : null, | |
| docsBaseUrl: docsBase ? docsBase.url.toString() : null, | |
| finalUrl: sourceUrl.toString(), | |
| route: llms ? 'llms' : mintlifyNav ? 'mintlify' : archbeeNav ? 'archbee' : scraped ? 'scrape' : sitemapUrl ? 'sitemap' : 'none', | |
| organized.resolution = { | |
| requestedUrl, | |
| redirectedUrl: redirected ? redirected.toString() : null, | |
| docsBaseUrl: docsBase ? docsBase.url.toString() : null, | |
| finalUrl: sourceUrl.toString(), | |
| route: scraped | |
| ? mintlifyNav | |
| ? 'mintlify' | |
| : archbeeNav | |
| ? 'archbee' | |
| : 'scrape' | |
| : llms | |
| ? 'llms' | |
| : sitemapUrl | |
| ? 'sitemap' | |
| : 'none', |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/commands/import.js` around lines 823 - 828, Update the route assignment
in the organized.resolution metadata to reflect the final organizer selected
around the scraped/llms handling: prefer scraped when it produced the final
tree, and use llms only when the scrape result was absent or discarded, while
preserving the existing Mintlify, Archbee, sitemap, and none labels. Add
regression coverage for llms combined with Mintlify, Archbee, and generic scrape
discovery.
| organized.resolution = { | ||
| requestedUrl, | ||
| redirectedUrl: redirected ? redirected.toString() : null, | ||
| docsBaseUrl: docsBase ? docsBase.url.toString() : null, | ||
| finalUrl: sourceUrl.toString(), | ||
| route: llms ? 'llms' : mintlifyNav ? 'mintlify' : archbeeNav ? 'archbee' : scraped ? 'scrape' : sitemapUrl ? 'sitemap' : 'none', | ||
| ...(scrapeDiscardedForCoverage ? { scrapeDiscarded: true } : {}), | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Record the orphan-triage discard in scrapeDiscarded.
When orphan triage re-clusters a thin scrape, Lines 609-620 log that the scrape was discarded. The flag used here is set only by the coverage gate at Lines 553-559. The returned metadata therefore omits scrapeDiscarded for this discard path.
Track a general discard boolean for both paths. Keep the coverage-specific boolean if it is still needed for warning suppression.
Proposed discard tracking
let scrapeDiscardedForCoverage = false
+ let scrapeDiscarded = false
...
scraped = null
scrapeDiscardedForCoverage = true
+ scrapeDiscarded = true
...
if (reclustered) {
+ scrapeDiscarded = true
...
- ...(scrapeDiscardedForCoverage ? { scrapeDiscarded: true } : {}),
+ ...(scrapeDiscarded ? { scrapeDiscarded: true } : {}),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/commands/import.js` around lines 823 - 830, Update the discard tracking
around the resolution assembly and orphan-triage flow so a general
scrape-discarded boolean is set for both coverage-gate and thin-scrape
re-clustering paths. Keep the existing coverage-specific flag for warning
suppression if needed, and use the general boolean in the `organized.resolution`
construction to emit `scrapeDiscarded` for either discard path.
|
Closing — direction changed for RM-17863: the non-docs-site check now runs on the marketing homepage before the redirect (quick classify endpoint on the launcher + warning with Continue Anyway on the form), replacing the in-pipeline warning. New PRs incoming. |
1 participant
🧰 What's changing
importDocs()for website-URL imports now returns asourcesarray — one entry per source URL — recording how the CLI resolved it: the requested URL, any redirect, the adopted docs-route base, the final URL, and which discovery route produced the skeleton (llms/mintlify/archbee/scrape/sitemap/none, plus ascrapeDiscardedflag when the coverage gate threw the scrape away).🧬 Why
The import runner needs this signal to decide whether a submitted URL actually resolved to a documentation site. Strong routes (llms.txt, Mintlify, Archbee) are docs sites by construction; weak routes (scrape/sitemap) trigger an AI docs-site check downstream that can surface a soft "this might not be a docs site" warning (slug
non_docs_site) on the import progress screen. The import itself is never blocked.Pure metadata bolt-on — no behavior change to discovery, staging, or zipping.