Skip to content

feat: search body-text indexing with snippet extraction - #18

Merged
Shewart merged 3 commits into
mainfrom
feat/search-body-indexing
Jul 24, 2026
Merged

feat: search body-text indexing with snippet extraction#18
Shewart merged 3 commits into
mainfrom
feat/search-body-indexing

Conversation

@Shewart

@Shewart Shewart commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Extends the search index from title/description-only to full body-text matching, with a 2-line snippet surfaced when the match wouldn't otherwise be visible. Same client-side substring scoring, no build-step change, no new dependencies.

What's new

MarkdownPlainText.Extract(markdown, maxLength = 8000) in ShellDocs.Core

Strips markdown/HTML noise down to searchable prose:

  • YAML frontmatter (leading ---…---)
  • Fenced code blocks (whole block)
  • HTML tags and Razor component tags (<Callout Title="x">body</Callout>body)
  • Images ![alt](/img)alt
  • Links [text](url)text
  • Inline code (backticks removed, content preserved)
  • Emphasis (**bold**, *italic*bold, italic)
  • Heading # prefixes (heading text kept)
  • List markers (-, *, 1.) and blockquote >
  • Whitespace collapsed, trimmed to maxLength

SearchEntry.Body (nullable) — populated for Page entries during SearchIndex.FromGraph. Heading entries stay body-less (they anchor into the same page).

SearchDialog scoring extended

Body substring match adds +6; per-token body match adds +1. Both weighted lower than title/description/section so page name still wins on ties. Hard-reject still applies — every token must appear somewhere.

Body-only match snippet

When the match comes from body (not title/section), the result surfaces a 2-line snippet centered on the first token hit:

Search: "static site"
Result: Deploying
        ShellDocs
        … output as a **static site** ready for GitHub Pages, Vercel, Netlify …

Snippet is 150 chars, ellipsis prefix/suffix when trimmed, case-preserving. .search-result-snippet CSS: 0.75rem muted, -webkit-line-clamp: 2 for the two-line box.

Out of scope (deliberate)

  • Highlighting inside snippet — no <mark> around the matched substring yet. Polish; its own follow-up.
  • Per-heading body splits — body is per-page today. A match near the bottom of a long page anchors the URL at the top. Splitting body at heading boundaries and emitting per-section entries is a real improvement, but doubles the index size and complicates the anchor URLs.
  • Real inverted index / BM25 — substring scoring is fine for docs-sized corpora (~100 pages, seen in shelldocs-preview). If someone builds a 1000-page site with this, we'll rebuild the ranker.

Test plan

  • dotnet build shelldocs.slnx — clean, 0 warnings, 0 errors
  • dotnet test shelldocs.slnx139 / 139 passing (+10 new: 8 MarkdownPlainTextTests, 2 SearchIndexTests)
  • Preview app: all primitive pages + /docs/introduction render clean — SearchEntry schema change (added optional Body field with default null) didn't regress anything downstream
  • Manual: Cmd/Ctrl+K, search for a term that appears only in a page body (not any title or section) — result surfaces with a 2-line snippet under the title, URL navigates to the correct page

New test coverage

MarkdownPlainTextTests (8):

  • Strips frontmatter
  • Strips fenced code blocks (preserves surrounding prose)
  • Strips Razor component tags (preserves body)
  • Unwraps links and images (keeps text, drops URL)
  • Unwraps emphasis and inline code
  • Strips heading # prefixes (keeps heading text)
  • Truncates at maxLength
  • Empty/null input safe

SearchIndexTests (+2):

  • Page entries carry extracted body text
  • Heading entries have null body (dedup — they anchor into the page's body)

Files touched

  • src/ShellDocs.Core/MarkdownPlainText.cs (new)
  • src/ShellDocs.Core/SearchIndex.csSearchEntry.Body field, ExtractBodyFromFile in Walk
  • src/ShellDocs.Components/Chrome/SearchDialog.razor — body scoring, snippet extraction
  • src/ShellDocs.Components/Chrome/SearchDialog.razor.css.search-result-snippet styling
  • tests/ShellDocs.Tests/MarkdownPlainTextTests.cs (new)
  • tests/ShellDocs.Tests/SearchIndexTests.cs (+2 tests)

@Shewart
Shewart merged commit c303425 into main Jul 24, 2026
1 check passed
@Shewart
Shewart deleted the feat/search-body-indexing branch July 25, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant