Skip to content

WIP.Wisdom.md

Latest commit

 

History

History
51 lines (40 loc) · 4.93 KB

WIP.Wisdom.md

File metadata and controls

51 lines (40 loc) · 4.93 KB
Raw

twinBASIC Documentation --- Wisdom

The Discord knowledge-harvesting tool: a three-phase pipeline (export → process → extract) that mines the twinBASIC Discord for things the documentation does not yet say, and drafts additions for human review. Plans in wisdom/PLAN-{1,2,3}.md; implementation under wisdom/. Uses Node.js built-in APIs, and the repository's npm packages through lib/markdown.mjs, which extract/merger.mjs reads staging.md with.

Wisdom runs occasionally, so its invocation detail lives here and not in WIP.md. WIP.md keeps only the rule that Wisdom is an exception to. Open this file when running the tool.

Why its public page is allowed to name Claude

WIP.md's rule is that nothing under docs/Documentation/ should require Claude, an agent or a skill to follow --- the audience is a contributor with an editor and a terminal.

Wisdom is a deliberate exception and stays public. The tool has no manual mode to document --- Phase 3 is a set of Claude agents, and a page describing it without them would describe nothing. A reviewer applying the rule will read its "tell Claude: …" instructions as a scope violation and propose moving the public page to a maintainer file. That is declined: leave docs/Documentation/Wisdom.md where it is.

Phase 3 --- Extract invocation

Phase 3 runs Claude agents over the processed thread .md files and drafts documentation additions. The default flow is incremental: only threads whose Discord-side content changed since the last successful merge are re-extracted. With the incremental export and process phases, the routine update is three commands with no flags:

node wisdom/wisdom.mjs export
node wisdom/wisdom.mjs process
node wisdom/wisdom.mjs extract

The extract step is a three-stage flow that Claude runs: prep → workflow → merge. The merge step grafts new findings into the long-lived staging.md. It replaces matching sections in place, and emits [REFINED?] markers for findings whose prior version was reviewed and removed, so pending review work is never overwritten.

  1. Prep: node wisdom/wisdom.mjs extract filters threads against wisdom/data/findings/extract-state.json (the per-thread last_message_id + message_count watermark, advanced on each successful merge). Only changed threads survive the filter. Shared reference files (package-summary.txt, page-index.json) are written once; per-batch files contain thread file paths, per-thread file sizes, config, and a mode field (incremental, since, all, or force) consumed by the merge step.
    • If the filter is empty (no threads have changed), prep exits with No new threads since the last successful merge and the merge step is unnecessary.
  2. Run the workflow: check which prep layout was produced:
    • Single-batch (≤200 threads) --- extract-prep.json exists: read it, pass its parsed contents as args to the Workflow tool with scriptPath: "wisdom/extract/workflow.mjs". Write the returned { additions: [...] } array to wisdom/data/findings/extract-results-0.json.
    • Multi-batch (>200 threads) --- extract-manifest.json exists: read it, then for each entry in manifest.batches, read the corresponding extract-batch-{i}.json, pass it as args to the Workflow tool, and write the returned additions array to extract-results-{i}.json. Skip batches whose result file already exists (resumability).
  3. Merge: node wisdom/wisdom.mjs extract --merge reads every extract-results-*.json, grafts the additions into staging.md per the merge semantics in wisdom/PLAN-3.md, and advances extract-state.json. Atomic write via temp+rename; previous staging.md retained as staging.md.bak for one generation.
  4. Review: wisdom/data/findings/staging.md is the long-lived human-review file --- grouped by target page, with [DUPLICATE?] markers for cross-thread overlaps, [REFINED?] markers for previously-processed findings whose source thread has grown, and an Unmapped Findings section for findings with no existing doc page. The reviewer can append [LOCKED] to any section header to prevent auto-replacement on the next merge.

Diagnostic modes (mutually exclusive primary flags):

  • extract --since 2026-06-04 --- filter by thread created date instead of watermark. State is not touched; merge writes to staging-since-2026-06-04.md rather than grafting into the canonical staging file.
  • extract --all --- bootstrap / re-baseline. Processes every thread regardless of state and ignores the channel filter. Merge still grafts (replacing matching sections, inserting new ones) so an existing staging.md is not lost.
  • extract --force --- ignores the watermark filter but respects --channel. Useful when the agent prompt changes and specific channels need re-extraction.
  • extract --dry-run --- writes the prep file but does not invoke the workflow; state is not touched.