diff --git a/README.md b/README.md index 0a0684c..8005fa7 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ A CLI tool that analyzes your Claude Code session history to identify token waste patterns and provide actionable fixes. Runs locally, no LLM needed. -**v2.2** adds human-readable findings with plain-English explanations, collapsible HTML report sections, and evidence-based action items. +**v2.3** adds `--analyze-skill` — analyze any AI agent skill package for token efficiency with cost estimates, letter grades, section-level breakdowns, and actionable shortening tips. Designed for integration with skill registries like [Tank](https://github.com/tankpkg/tank). **The mental model: tokenomics is a coach, not a remote control.** It writes suggestions into your CLAUDE.md where Claude can see them. Claude is smart enough to follow most of them — suggesting `/compact` when context grows, recommending Sonnet for simple tasks, warning when you're overspending. But it cannot switch models, run commands, or force behavior. The user is always in control. @@ -240,6 +240,7 @@ Budget config lives at `~/.claude/tokenomics.json`: | `--inject` | Re-analyze sessions + update CLAUDE.md findings | false | | `--quiet` | Suppress terminal output (used by hooks) | false | | `--verbose` | Show discovery progress and debug info | false | +| `--analyze-skill ` | Analyze skill package for token efficiency | - | | `--help` | Show help message | - | | `--version` | Show version | - | @@ -264,13 +265,170 @@ Budget config lives at `~/.claude/tokenomics.json`: | Subagent Opportunity | Delegation opportunities | Parallel execution | | Smart Router | Historical routing patterns | ~80% on simple tasks | +## Skill Analysis (`--analyze-skill`) + +Analyze an AI agent skill package directory for token efficiency. Produces a grade (A/B/C/D), cost per use in real money, per-section token breakdowns, and actionable suggestions for what to cut and why. + +```bash +# Terminal report (default) +tokenomics --analyze-skill ./my-skill + +# JSON output (for scripts, CI, Tank integration) +tokenomics --analyze-skill ./my-skill --json +``` + +### Terminal output + +``` + SKILL TOKEN ANALYSIS + ────────────────────────────────────────────────── + + "Well-structured skill with no significant waste." + + Grade: A (86/100) + Size: 33,598 tokens — above average (avg is ~20,000 tokens) + + [░░░░░░░░░░░░░█░░░░░░░░░░░░░░░░] + ▲ avg + + Cost per use: ~$0.18 (Sonnet) | ~$0.91 (Opus) + + Findings (3): + ● prompt-size [medium] + Prompt file "SKILL.md" is ~2,388 tokens (9,549 chars). + Consider trimming for faster invocations. + Fix: Consider trimming "SKILL.md" to under 2000 tokens. + Extract verbose examples into separate reference files. + + ● section-analysis [low] + SKILL.md: 2 section(s) overlap each other. + Fix: "web app setup" overlaps with "library setup": + consolidate shared content into one section. + Tip: References 18 specific file paths — consider + replacing with a glob pattern or naming convention rule. +``` + +### JSON output + +With `--json`, the output includes all fields for programmatic consumption: + +```json +{ + "one_liner": "Slightly above average size. Works fine, could be leaner.", + "grade": "B", + "estimated_tokens": 8500, + "comparison": "8,500 tokens — below average (avg is ~20,000 tokens)", + "cost_per_use": { + "sonnet": "~$0.05", + "opus": "~$0.23" + }, + "what_this_means": "Bigger skills cost more per invocation and leave less room for conversation. Smaller skills respond faster and cost less.", + "findings": [ + { + "rule": "section-analysis", + "severity": "low", + "description": "SKILL.md: 2 section(s) overlap each other...", + "remediation": "\"Setup\" overlaps with \"Config\": consolidate...", + "sections": [ + { "heading": "Setup", "tokens": 420, "redundantWith": ["Config"] }, + { "heading": "Config", "tokens": 380, "redundantWith": ["Setup"] } + ] + } + ], + "summary": { + "total_findings": 3, + "estimated_tokens_per_invocation": 8500, + "efficiency_score": 72 + } +} +``` + +### Report fields + +| Field | What it means | +|-------|---------------| +| `one_liner` | Plain English summary of the skill's efficiency | +| `grade` | A (lean) / B (good) / C (bloated) / D (wasteful) | +| `estimated_tokens` | Total token count across all skill files | +| `comparison` | How this skill compares to average (~20k tokens) | +| `cost_per_use` | Real money per invocation on Sonnet and Opus | +| `what_this_means` | Why size matters: cost + speed + context window competition | +| `sections[]` | Per-heading token breakdown with redundancy and shortening tips | + +### Analysis rules + +| Rule | What it checks | Thresholds | +|------|---------------|------------| +| `prompt-size` | SKILL.md and `.atom.md` files exceeding token limits | >2000 tokens = medium, >4000 = high | +| `claude-md-size` | CLAUDE.md injection blocks that bloat system prompts | >1500 tokens = medium, >3000 = high | +| `tool-overhead` | Tool definitions in manifest (tank.json, skills.json) | >8 tools = medium, >15 = high | +| `large-files` | Files with >500 lines that are expensive to read | >500 lines = low, >1000 = medium | +| `redundant-instructions` | Duplicated instruction lines across skill files | >30% duplication = low/medium | +| `section-analysis` | Per-section token breakdown, cross-file redundancy, content tips | See below | + +### Section analysis details + +The `section-analysis` rule parses markdown headings and provides: + +**Token breakdown**: Every section gets a token count, so you can see exactly where the weight is. + +**Cross-file redundancy**: Detects when a section in SKILL.md says the same thing as a section in `references/*.md`. Example: `"The Bridge"` in SKILL.md duplicates `"The Bridge Model"` in `references/orchestration-workflow.md`. + +**Content-level shortening tips** (not just structural): + +| Tip | What it detects | +|-----|----------------| +| Concept repetition | Same key phrase restated 3+ times — state once, reference elsewhere | +| Restatements | "remember that", "note that", "in other words" — AI understood the first time | +| Bad/good example pairs | Shows both wrong and right patterns — use a negative constraint rule instead | +| File path enumeration | 6+ specific paths — replace with glob patterns or naming conventions | +| Verbose justification | "this is important because" — AI follows rules without persuasion | +| Large directory trees | ASCII trees >10 lines — keep top-level, link to reference | +| Repeated lines | Identical lines within a section — dedup or collapse | +| Code block bloat | 2+ code blocks — replace with file references | +| Long lists | 8+ list items — group into categories | +| Table bloat | 15+ table rows — move details to reference files | + +### Cost estimation + +Based on Claude API pricing with an 80/20 input/output token split: + +| Tokens | Sonnet cost | Opus cost | +|--------|-------------|-----------| +| 5,000 | ~$0.03 | ~$0.15 | +| 20,000 (avg) | ~$0.12 | ~$0.59 | +| 50,000 | ~$0.29 | ~$1.47 | +| 100,000 | ~$0.59 | ~$2.94 | + +Every skill is loaded into context on every turn. A 50k-token skill costs ~$1.47 per turn on Opus — trimming it to 20k saves ~$0.88 per turn. + +### Integration with skill registries + +Designed to be called as a subprocess from scanning pipelines: + +```python +import subprocess, json + +result = subprocess.run( + ["tokenomics", "--analyze-skill", skill_dir, "--json"], + capture_output=True, text=True, timeout=10 +) +if result.returncode == 0: + analysis = json.loads(result.stdout) + print(f"Grade: {analysis['grade']} | Cost: {analysis['cost_per_use']['sonnet']}/use") + for finding in analysis["findings"]: + print(f"[{finding['severity']}] {finding['rule']}: {finding['description']}") +``` + +Graceful degradation: if tokenomics is not installed, skip the stage and continue. + ## Output Examples ### Terminal ``` TOKENOMICS — Token Intelligence for Claude Code - 60 sessions // 30 day range // v2.2.2 + 60 sessions // 30 day range // v2.3.0 Sessions: 60 Total: 1.2M tokens diff --git a/package.json b/package.json index 7b66cd2..96ad5ee 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "tokenomics", - "version": "2.2.2", + "version": "2.3.0", "description": "Token intelligence for Claude Code — analyze sessions, find waste, get actionable fixes", "type": "module", "main": "dist/analyze.js", diff --git a/src/analyze.ts b/src/analyze.ts index f9980d2..3cf8674 100644 --- a/src/analyze.ts +++ b/src/analyze.ts @@ -20,6 +20,7 @@ import { writeFileSync, mkdirSync } from 'node:fs'; import { homedir } from 'node:os'; import { join } from 'node:path'; import type { AnalysisOutput, CliOptions, DetectorResult, SessionData } from './types.js'; +import { analyzeSkill, renderSkillReport } from './skill-analyzer.js'; import { discoverFiles, logDiscoverySummary } from './discovery.js'; import { extractHumanReadableBlock, renderTerminalBlock } from './recommendation.js'; import { parseSessionFiles } from './parser.js'; @@ -34,7 +35,7 @@ import { auditPrompt } from './auditor.js'; import { renderPromptOutput } from './prompt-output.js'; import { ensureBudgetConfig, readBudgetConfig } from './budget-config.js'; -const VERSION = '2.2.2'; +const VERSION = '2.3.0'; // ─── CLI ────────────────────────────────────────────────────────────────────── @@ -60,6 +61,7 @@ function parseCliArgs(): CliOptions { budget: { type: 'boolean', default: false }, 'budget-check': { type: 'boolean', default: false }, 'no-alerts': { type: 'boolean', default: false }, + 'analyze-skill': { type: 'string' }, }, strict: true, }); @@ -88,6 +90,7 @@ function parseCliArgs(): CliOptions { budget: values.budget, budgetCheck: values['budget-check'], noAlerts: values['no-alerts'], + analyzeSkill: values['analyze-skill'], }; } @@ -131,6 +134,10 @@ PROMPT ANALYSIS --budget-check Lightweight budget check (for hooks) --no-alerts Suppress budget alerts (no CLAUDE.md injection) +SKILL ANALYSIS + --analyze-skill Analyze skill package for token efficiency + Outputs JSON with findings and efficiency score + OTHER --verbose Show discovery progress and debug info --help Show this message @@ -516,6 +523,17 @@ async function main(): Promise { process.exit(result.ceilingExceeded ? 1 : 0); } + // ── Skill analysis mode ── + if (options.analyzeSkill) { + const result = analyzeSkill(options.analyzeSkill); + if (options.json) { + console.log(JSON.stringify(result, null, options.verbose ? 2 : 0)); + } else { + console.log(renderSkillReport(result)); + } + return; + } + // Discover JSONL files (auto-detects all ~/.claude* installations) const discoveryOpts: import('./types.js').DiscoveryOptions = { days: options.days, diff --git a/src/skill-analyzer.ts b/src/skill-analyzer.ts new file mode 100644 index 0000000..78b5bfb --- /dev/null +++ b/src/skill-analyzer.ts @@ -0,0 +1,340 @@ +/** + * Skill Analyzer Engine + * + * Analyzes an AI agent skill package directory for token efficiency. + * Discovers skill files, runs all skill rules, and returns findings. + */ + +import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs' +import { join, extname, relative } from 'node:path' +import type { SkillAnalysisContext, SkillAnalysisReport, SkillAnalysisSummary, SkillCostEstimate, SkillFinding, SkillGrade, SkillRule } from './types.js' + +// Import all rules +import * as promptSizeRule from './skill-rules/prompt-size.js' +import * as claudeMdSizeRule from './skill-rules/claude-md-size.js' +import * as toolOverheadRule from './skill-rules/tool-overhead.js' +import * as largeFilesRule from './skill-rules/large-files.js' +import * as redundantInstructionsRule from './skill-rules/redundant-instructions.js' +import * as sectionAnalysisRule from './skill-rules/section-analysis.js' + +const CHARS_PER_TOKEN = 4 + +const SKILL_FILES = [ + 'SKILL.md', + 'CLAUDE.md', + 'tank.json', + 'skills.json', +] + +const SKILL_EXTENSIONS = ['.md', '.json'] + +const rules: SkillRule[] = [ + { name: 'prompt-size', analyze: promptSizeRule.analyze }, + { name: 'claude-md-size', analyze: claudeMdSizeRule.analyze }, + { name: 'tool-overhead', analyze: toolOverheadRule.analyze }, + { name: 'large-files', analyze: largeFilesRule.analyze }, + { name: 'redundant-instructions', analyze: redundantInstructionsRule.analyze }, + { name: 'section-analysis', analyze: sectionAnalysisRule.analyze }, +] + +function discoverSkillFiles(dir: string): Map { + const files = new Map() + + function walk(currentDir: string): void { + const entries = readdirSync(currentDir, { withFileTypes: true }) + + for (const entry of entries) { + // Skip symlinks to prevent path traversal + if (entry.isSymbolicLink()) continue + + if (entry.isDirectory()) { + if (entry.name === 'node_modules' || entry.name === '.git' || entry.name === 'dist') continue + walk(join(currentDir, entry.name)) + continue + } + + const fullPath = join(currentDir, entry.name) + const relativePath = relative(dir, fullPath) + + // Include known skill files or files with skill extensions + const isKnownFile = SKILL_FILES.some(f => entry.name.toLowerCase() === f.toLowerCase()) + const isAtomFile = entry.name.toLowerCase().endsWith('.atom.md') + const hasSkillExt = SKILL_EXTENSIONS.includes(extname(entry.name).toLowerCase()) + + if (isKnownFile || isAtomFile || hasSkillExt) { + try { + const content = readFileSync(fullPath, 'utf-8') + files.set(relativePath, content) + } catch { + // Skip unreadable files + } + } + } + } + + walk(dir) + return files +} + +function loadManifest(files: Map): Record | null { + // Try tank.json first, then skills.json + for (const manifestName of ['tank.json', 'skills.json']) { + for (const [filename, content] of files) { + if (filename.toLowerCase().endsWith(manifestName.toLowerCase())) { + try { + return JSON.parse(content) as Record + } catch { + // Invalid JSON, skip + } + } + } + } + return null +} + +function estimateTokens(files: Map): number { + let totalChars = 0 + for (const content of files.values()) { + totalChars += content.length + } + return Math.ceil(totalChars / CHARS_PER_TOKEN) +} + +// ── Cost estimation ────────────────────────────────────────────────────────── + +// Approximate USD per 1M tokens (as of 2026) +const SONNET_INPUT_PER_M = 3.00 +const SONNET_OUTPUT_PER_M = 15.00 +const OPUS_INPUT_PER_M = 15.00 +const OPUS_OUTPUT_PER_M = 75.00 + +// Assume ~80% input / ~20% output split for skill context loading +const INPUT_RATIO = 0.8 +const OUTPUT_RATIO = 0.2 + +const AVG_SKILL_TOKENS = 20000 + +function estimateCost(tokens: number): SkillCostEstimate { + const tokenMillions = tokens / 1_000_000 + const sonnetCost = tokenMillions * (SONNET_INPUT_PER_M * INPUT_RATIO + SONNET_OUTPUT_PER_M * OUTPUT_RATIO) + const opusCost = tokenMillions * (OPUS_INPUT_PER_M * INPUT_RATIO + OPUS_OUTPUT_PER_M * OUTPUT_RATIO) + + return { + sonnet: formatUsd(sonnetCost), + opus: formatUsd(opusCost), + } +} + +function formatUsd(amount: number): string { + if (amount < 0.001) return '<$0.001' + if (amount < 0.01) return `~$${amount.toFixed(3)}` + if (amount < 0.1) return `~$${amount.toFixed(2)}` + if (amount < 1) return `~$${amount.toFixed(2)}` + return `~$${amount.toFixed(2)}` +} + +// ── Grading ────────────────────────────────────────────────────────────────── + +function calculateGrade(score: number): SkillGrade { + if (score >= 85) return 'A' + if (score >= 65) return 'B' + if (score >= 40) return 'C' + return 'D' +} + +function calculateEfficiencyScore(findings: SkillFinding[], totalTokens: number): number { + let score = 100 + + for (const finding of findings) { + const deduction = finding.severity === 'high' ? 15 + : finding.severity === 'medium' ? 8 + : finding.severity === 'low' ? 3 + : 1 // info + score -= deduction + } + + if (totalTokens < 1000) score = Math.min(score + 10, 100) + + return Math.max(0, Math.min(100, score)) +} + +// ── Plain-English summaries ────────────────────────────────────────────────── + +function generateOneLiner(grade: SkillGrade, tokens: number, findingsCount: number): string { + if (grade === 'A' && tokens < 5000) return 'Lean and efficient. No meaningful improvements needed.' + if (grade === 'A') return 'Well-structured skill with no significant waste.' + if (grade === 'B') { + if (findingsCount <= 2) return 'Slightly above average size. Works fine, could be leaner.' + return 'Good shape overall, with a few areas that could be tightened up.' + } + if (grade === 'C') return 'Carries noticeable token overhead. Several sections could be trimmed or consolidated.' + return 'Bloated — significant token waste. Multiple sections duplicate content or over-explain.' +} + +function generateComparison(tokens: number): string { + const ratio = tokens / AVG_SKILL_TOKENS + if (ratio < 0.3) return `${tokens.toLocaleString()} tokens — much smaller than avg (~${AVG_SKILL_TOKENS.toLocaleString()} tokens)` + if (ratio < 0.7) return `${tokens.toLocaleString()} tokens — below average (avg is ~${AVG_SKILL_TOKENS.toLocaleString()} tokens)` + if (ratio < 1.3) return `${tokens.toLocaleString()} tokens — about average for a skill (~${AVG_SKILL_TOKENS.toLocaleString()} tokens)` + if (ratio < 2.0) return `${tokens.toLocaleString()} tokens — above average (avg is ~${AVG_SKILL_TOKENS.toLocaleString()} tokens)` + return `${tokens.toLocaleString()} tokens — much larger than avg (~${AVG_SKILL_TOKENS.toLocaleString()} tokens)` +} + +function generateWhatThisMeans(tokens: number): string { + const ratio = tokens / AVG_SKILL_TOKENS + if (ratio < 0.5) return 'Low overhead — the AI loads this context quickly and cheaply. No action needed.' + if (ratio < 1.2) return 'Bigger skills cost more per invocation and leave less room for conversation. Smaller skills respond faster and cost less.' + return 'This skill is expensive to load on every turn — every byte competes with your actual conversation for the context window. Trimming it saves real money and improves response speed.' +} + +// ── Size bar ───────────────────────────────────────────────────────────────── + +function sizeBar(tokens: number): string { + const maxTokens = 80000 + const barWidth = 30 + const fill = Math.min(Math.round((tokens / maxTokens) * barWidth), barWidth) + const avgPos = Math.min(Math.round((AVG_SKILL_TOKENS / maxTokens) * barWidth), barWidth - 1) + const bar = '░'.repeat(fill) + '█' + '░'.repeat(Math.max(barWidth - fill - 1, 0)) + const marker = ' '.repeat(avgPos) + '▲' + return `[${bar}]\n [${marker} avg]` +} + +// ── Main export ────────────────────────────────────────────────────────────── + +export function analyzeSkill(dir: string): SkillAnalysisReport { + // Validate directory + if (!existsSync(dir)) { + throw new Error(`Directory not found: ${dir}`) + } + + if (!statSync(dir).isDirectory()) { + throw new Error(`Not a directory: ${dir}`) + } + + // Discover and read files + const files = discoverSkillFiles(dir) + const manifest = loadManifest(files) + + // Build analysis context + const context: SkillAnalysisContext = { + skillDir: dir, + files, + skillManifest: manifest, + } + + // Run all rules + const allFindings: SkillFinding[] = [] + for (const rule of rules) { + const ruleFindings = rule.analyze(context) + allFindings.push(...ruleFindings) + } + + // Calculate summary + const estimatedTokens = estimateTokens(files) + const efficiencyScore = calculateEfficiencyScore(allFindings, estimatedTokens) + const grade = calculateGrade(efficiencyScore) + + const summary: SkillAnalysisSummary = { + total_findings: allFindings.length, + estimated_tokens_per_invocation: estimatedTokens, + efficiency_score: efficiencyScore, + } + + return { + one_liner: generateOneLiner(grade, estimatedTokens, allFindings.length), + grade, + estimated_tokens: estimatedTokens, + comparison: generateComparison(estimatedTokens), + cost_per_use: estimateCost(estimatedTokens), + what_this_means: generateWhatThisMeans(estimatedTokens), + findings: allFindings, + summary, + } +} + +// ── Terminal renderer ──────────────────────────────────────────────────────── + +function gradeColor(grade: SkillGrade): string { + switch (grade) { + case 'A': return '\x1b[32m' // green + case 'B': return '\x1b[33m' // yellow + case 'C': return '\x1b[33m\x1b[1m' // bold yellow + case 'D': return '\x1b[31m\x1b[1m' // bold red + } +} + +function severityIcon(sev: string): string { + switch (sev) { + case 'high': return '\x1b[31m●\x1b[0m' + case 'medium': return '\x1b[33m●\x1b[0m' + case 'low': return '\x1b[36m●\x1b[0m' + default: return '\x1b[2m●\x1b[0m' + } +} + +function wrapText(text: string, width: number): string[] { + const words = text.split(' ') + const lines: string[] = [] + let current = '' + for (const word of words) { + if (current.length + word.length + 1 > width && current.length > 0) { + lines.push(current) + current = word + } else { + current = current.length === 0 ? word : `${current} ${word}` + } + } + if (current.length > 0) lines.push(current) + return lines +} + +export function renderSkillReport(report: SkillAnalysisReport): string { + const bold = '\x1b[1m' + const dim = '\x1b[2m' + const reset = '\x1b[0m' + const cyan = '\x1b[36m' + const gc = gradeColor(report.grade) + + const lines: string[] = [] + lines.push('') + lines.push(`${cyan}${bold} SKILL TOKEN ANALYSIS${reset}`) + lines.push(`${dim} ${'─'.repeat(50)}${reset}`) + lines.push('') + lines.push(` ${bold}"${report.one_liner}"${reset}`) + lines.push('') + lines.push(` Grade: ${gc}${bold}${report.grade}${reset} (${report.summary.efficiency_score}/100)`) + lines.push(` Size: ${report.comparison}`) + lines.push('') + lines.push(` ${sizeBar(report.estimated_tokens)}`) + lines.push('') + lines.push(` Cost per use: ${report.cost_per_use.sonnet} (Sonnet) | ${report.cost_per_use.opus} (Opus)`) + lines.push('') + + if (report.findings.length > 0) { + lines.push(` ${bold}Findings (${report.findings.length}):${reset}`) + for (const f of report.findings) { + const icon = severityIcon(f.severity) + lines.push(` ${icon} ${bold}${f.rule}${reset} [${f.severity}]`) + // Wrap description to 76 chars (terminal width minus indentation) + for (const ln of wrapText(f.description, 76)) { + lines.push(` ${dim}${ln}${reset}`) + } + if (f.remediation) { + for (const remediationLine of f.remediation.split('\n')) { + for (const ln of wrapText(`Fix: ${remediationLine}`, 76)) { + lines.push(` ${dim}${ln}${reset}`) + } + } + } + lines.push('') + } + } else { + lines.push(` ${dim}No findings. Structure looks clean.${reset}`) + lines.push('') + } + + lines.push(` ${dim}${report.what_this_means}${reset}`) + lines.push('') + + return lines.join('\n') +} diff --git a/src/skill-rules/claude-md-size.ts b/src/skill-rules/claude-md-size.ts new file mode 100644 index 0000000..7646e6d --- /dev/null +++ b/src/skill-rules/claude-md-size.ts @@ -0,0 +1,91 @@ +/** + * CLAUDE.md Size Rule + * + * Detects CLAUDE.md injection blocks and flags oversized ones. + * Looks for markers or large markdown blocks + * in CLAUDE.md files. + * + * Thresholds: + * >1500 tokens (6000 chars) → medium + * >3000 tokens (12000 chars) → high + */ + +import type { SkillAnalysisContext, SkillFinding } from '../types.js' + +const CHARS_PER_TOKEN = 4 +const MEDIUM_TOKEN_THRESHOLD = 1500 +const HIGH_TOKEN_THRESHOLD = 3000 + +export function analyze(context: SkillAnalysisContext): SkillFinding[] { + const findings: SkillFinding[] = [] + + for (const [filename, content] of context.files) { + if (!filename.toLowerCase().includes('claude.md')) continue + + // Check for TOKENOMICS injection blocks + const tokenomicsBlocks = extractTokenomicsBlocks(content) + if (tokenomicsBlocks.length > 0) { + for (const block of tokenomicsBlocks) { + const tokenEstimate = Math.ceil(block.length / CHARS_PER_TOKEN) + flagBlock(findings, filename, tokenEstimate, 'tokenomics injection block') + } + continue + } + + // No injection markers — evaluate the whole file if it looks like an injection target + const tokenEstimate = Math.ceil(content.length / CHARS_PER_TOKEN) + flagBlock(findings, filename, tokenEstimate, 'CLAUDE.md content') + } + + return findings +} + +function extractTokenomicsBlocks(content: string): string[] { + const blocks: string[] = [] + const startMarker = '\n' + 'A'.repeat(12001) + '\n' + const ctx = makeContext({ 'CLAUDE.md': block }) + const findings = claudeMdSizeRule.analyze(ctx) + + expect(findings.length).toBe(1) + expect(findings[0]!.description).toContain('tokenomics injection block') + }) +}) + +// ─── Tool Overhead Rule ───────────────────────────────────────────────────── + +describe('tool-overhead rule', () => { + it('flags >15 tools as high severity', () => { + const tools: Record = {} + for (let i = 0; i < 16; i++) { + tools[`tool-${i}`] = { command: 'echo' } + } + const manifest = { mcpServers: tools } + const ctx = makeContext({}, manifest) + + const findings = toolOverheadRule.analyze(ctx) + expect(findings.length).toBe(1) + expect(findings[0]!.severity).toBe('high') + expect(findings[0]!.rule).toBe('tool-overhead') + }) + + it('flags >8 tools as medium severity', () => { + const tools: Record = {} + for (let i = 0; i < 10; i++) { + tools[`tool-${i}`] = { command: 'echo' } + } + const manifest = { mcpServers: tools } + const ctx = makeContext({}, manifest) + + const findings = toolOverheadRule.analyze(ctx) + expect(findings.length).toBe(1) + expect(findings[0]!.severity).toBe('medium') + }) + + it('does not flag <=8 tools', () => { + const tools: Record = {} + for (let i = 0; i < 5; i++) { + tools[`tool-${i}`] = { command: 'echo' } + } + const manifest = { mcpServers: tools } + const ctx = makeContext({}, manifest) + + const findings = toolOverheadRule.analyze(ctx) + expect(findings.length).toBe(0) + }) + + it('scans config files for tool definitions', () => { + const tankJson = JSON.stringify({ + mcpServers: Object.fromEntries( + Array.from({ length: 12 }, (_, i) => [`tool-${i}`, { command: 'echo' }]) + ), + }) + const ctx = makeContext({ 'tank.json': tankJson }, null) + + const findings = toolOverheadRule.analyze(ctx) + expect(findings.length).toBe(1) + expect(findings[0]!.severity).toBe('medium') + }) +}) + +// ─── Large Files Rule ──────────────────────────────────────────────────────── + +describe('large-files rule', () => { + it('flags files >500 lines', () => { + const lines = Array(501).fill('line of code').join('\n') + const ctx = makeContext({ 'src/large.ts': lines }) + const findings = largeFilesRule.analyze(ctx) + + expect(findings.length).toBe(1) + expect(findings[0]!.rule).toBe('large-files') + expect(findings[0]!.location).toBe('src/large.ts') + }) + + it('flags files >1000 lines as medium severity', () => { + const lines = Array(1001).fill('line of code').join('\n') + const ctx = makeContext({ 'src/very-large.ts': lines }) + const findings = largeFilesRule.analyze(ctx) + + expect(findings.length).toBe(1) + expect(findings[0]!.severity).toBe('medium') + }) + + it('does not flag files <=500 lines', () => { + const lines = Array(200).fill('line of code').join('\n') + const ctx = makeContext({ 'src/small.ts': lines }) + const findings = largeFilesRule.analyze(ctx) + + expect(findings.length).toBe(0) + }) +}) + +// ─── Redundant Instructions Rule ──────────────────────────────────────────── + +describe('redundant-instructions rule', () => { + it('flags when >30% duplication across skill files', () => { + const sharedInstructions = Array(10).fill('Always use TypeScript strict mode for all new files in this project').join('\n') + const uniqueA = 'Specific to SKILL.md that is unique content' + const uniqueB = 'Specific to CLAUDE.md that is different content' + + const ctx = makeContext({ + 'SKILL.md': sharedInstructions + '\n' + uniqueA, + 'CLAUDE.md': sharedInstructions + '\n' + uniqueB, + }) + const findings = redundantInstructionsRule.analyze(ctx) + + expect(findings.length).toBeGreaterThanOrEqual(1) + expect(findings[0]!.rule).toBe('redundant-instructions') + }) + + it('does not flag when duplication is low', () => { + const ctx = makeContext({ + 'SKILL.md': 'Unique content about the first skill that is quite different from others', + 'CLAUDE.md': 'Different content about project guidelines that does not overlap at all', + }) + const findings = redundantInstructionsRule.analyze(ctx) + + expect(findings.length).toBe(0) + }) + + it('returns nothing with fewer than 2 skill files', () => { + const ctx = makeContext({ + 'SKILL.md': 'A'.repeat(500), + }) + const findings = redundantInstructionsRule.analyze(ctx) + + expect(findings.length).toBe(0) + }) +}) + +// ─── Section Analysis Rule ──────────────────────────────────────────────────── + +describe('section-analysis rule', () => { + it('reports per-section token counts', () => { + const skill = [ + '# My Skill', + '', + 'A brief description of the skill.', + '', + '## Instructions', + '', + ...Array(50).fill('When the user asks for help, follow these steps carefully and provide detailed feedback.'), + '', + '## Examples', + '', + ...Array(30).fill('Here is an example of how to handle a request from the user.'), + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + expect(findings.length).toBe(1) + expect(findings[0]!.rule).toBe('section-analysis') + expect(findings[0]!.sections).toBeDefined() + expect(findings[0]!.sections!.length).toBe(3) // preamble, Instructions, Examples + + // Check that sections have token counts + const instructionsSection = findings[0]!.sections!.find(s => s.heading === 'Instructions') + expect(instructionsSection).toBeDefined() + expect(instructionsSection!.tokens).toBeGreaterThan(0) + }) + + it('flags sections with redundant/overlapping content', () => { + const sharedContent = Array(20).fill( + 'Always use TypeScript strict mode for all new files in this project and follow the established patterns' + ).join(' ') + + const skill = [ + '# My Skill', + '', + '## Code Review', + '', + sharedContent, + 'Also check for security issues in the authentication module.', + '', + '## Security Review', + '', + sharedContent, + 'Also verify that authentication tokens are properly validated.', + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + expect(findings.length).toBe(1) + const finding = findings[0]! + expect(finding.description).toContain('overlap') + + // Check redundantWith is populated + const codeReview = finding.sections!.find(s => s.heading === 'Code Review') + expect(codeReview!.redundantWith).toBeDefined() + expect(codeReview!.redundantWith!.some(r => r.includes('Security Review'))).toBe(true) + }) + + it('suggests shortening tips for sections with repeated lines', () => { + const skill = [ + '# My Skill', + '', + '## Guidelines', + '', + ...Array(5).fill('Always check the types before submitting code changes.'), + '', + 'Some unique content here that is different.', + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + // Should have findings since Guidelines has repeated lines + if (findings.length > 0) { + const guidelines = findings[0]!.sections!.find(s => s.heading === 'Guidelines') + if (guidelines?.shorteningTip) { + expect(guidelines.shorteningTip).toContain('repeated') + } + } + }) + + it('suggests shortening for sections with many list items', () => { + const listItems = Array(12).fill(0).map((_, i) => `- Item number ${i + 1} that describes a specific check to perform`).join('\n') + + const skill = [ + '# My Skill', + '', + '## Checklist', + '', + listItems, + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + if (findings.length > 0) { + const checklist = findings[0]!.sections!.find(s => s.heading === 'Checklist') + if (checklist?.shorteningTip) { + expect(checklist.shorteningTip).toContain('list items') + } + } + }) + + it('suggests shortening for sections with many code blocks', () => { + const codeBlocks = Array(3).fill(0).map((_, i) => `Example ${i}:\n\`\`\`ts\nconst x = ${i};\n\`\`\``).join('\n\n') + + const skill = [ + '# My Skill', + '', + '## Examples', + '', + codeBlocks, + '', + 'Some additional content to make this section large enough to be over the threshold.', + 'More filler content that pushes the token count above the detection minimum.', + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + if (findings.length > 0) { + const examples = findings[0]!.sections!.find(s => s.heading === 'Examples') + if (examples?.shorteningTip) { + expect(examples.shorteningTip).toContain('code blocks') + } + } + }) + + it('does not flag small, clean sections', () => { + const skill = [ + '# My Skill', + '', + 'A brief skill that does one thing.', + '', + '## Usage', + '', + 'Run the command with the file path.', + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + expect(findings.length).toBe(0) + }) + + it('ignores non-prompt files', () => { + const ctx = makeContext({ 'package.json': '{"name": "test"}' }) + const findings = sectionAnalysisRule.analyze(ctx) + + expect(findings.length).toBe(0) + }) + + it('analyzes .atom.md files', () => { + const skill = [ + '## Search', + '', + ...Array(60).fill('Detailed search instructions that explain how to search for files in the project directory.'), + ].join('\n') + + const ctx = makeContext({ 'tools/search.atom.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + expect(findings.length).toBe(1) + expect(findings[0]!.location).toContain('.atom.md') + }) + + it('includes sections array in JSON output', () => { + const skill = [ + '# Skill', + '', + 'Preamble content here.', + '', + '## Section A', + '', + 'Content for section A that is reasonably sized.', + '', + '## Section B', + '', + 'Content for section B that is also reasonably sized.', + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + // Even if no findings (sections are small), verify JSON serialization + if (findings.length > 0) { + const json = JSON.stringify(findings[0]) + const parsed = JSON.parse(json) + expect(parsed.sections).toBeDefined() + expect(Array.isArray(parsed.sections)).toBe(true) + } + }) + + it('high severity for sections >1000 tokens', () => { + const hugeSection = 'Very detailed instruction content. '.repeat(600) // ~16800 chars = ~4200 tokens + + const skill = [ + '# Skill', + '', + '## Deep Dive', + '', + hugeSection, + ].join('\n') + + const ctx = makeContext({ 'SKILL.md': skill }) + const findings = sectionAnalysisRule.analyze(ctx) + + expect(findings.length).toBe(1) + expect(findings[0]!.severity).toBe('high') + }) +}) + +// ─── Integration: Full Skill Analyzer ──────────────────────────────────────── + +describe('skill analyzer integration', () => { + let tempDir: string + + beforeEach(() => { + tempDir = createTempDir() + }) + + afterEach(() => { + rmSync(tempDir, { recursive: true, force: true }) + }) + + it('returns valid SkillAnalysisResult for a clean skill', () => { + createFixture(tempDir, { + 'SKILL.md': 'A simple skill that does one thing well.', + 'tank.json': JSON.stringify({ name: 'test-skill', version: '1.0.0' }), + }) + + const result = analyzeSkill(tempDir) + + expect(result).toHaveProperty('findings') + expect(result).toHaveProperty('summary') + expect(Array.isArray(result.findings)).toBe(true) + expect(result.summary.total_findings).toBe(result.findings.length) + expect(result.summary.efficiency_score).toBeGreaterThanOrEqual(0) + expect(result.summary.efficiency_score).toBeLessThanOrEqual(100) + expect(result.summary.estimated_tokens_per_invocation).toBeGreaterThan(0) + }) + + it('detects all issue types in a bloated skill', () => { + const largeSkill = 'Large prompt content. '.repeat(800) // ~19k chars + const largeClaude = 'Inject all the things. '.repeat(800) // ~19k chars + const duplicated = Array(15).fill('Always use TypeScript strict mode for all files in this project').join('\n') + + const tools: Record = {} + for (let i = 0; i < 16; i++) { + tools[`tool-${i}`] = { command: `echo ${i}` } + } + + createFixture(tempDir, { + 'SKILL.md': largeSkill + '\n' + duplicated, + 'CLAUDE.md': largeClaude + '\n' + duplicated, + 'search.atom.md': largeSkill, + 'tank.json': JSON.stringify({ name: 'bloated-skill', mcpServers: tools }), + }) + + const result = analyzeSkill(tempDir) + + expect(result.findings.length).toBeGreaterThanOrEqual(3) + + const rules = result.findings.map(f => f.rule) + expect(rules).toContain('prompt-size') + expect(rules).toContain('claude-md-size') + expect(rules).toContain('tool-overhead') + expect(rules).toContain('redundant-instructions') + }) + + it('throws for non-existent directory', () => { + expect(() => analyzeSkill('/nonexistent/path')).toThrow('Directory not found') + }) + + it('throws for a file path instead of directory', () => { + const filePath = join(tempDir, 'file.txt') + writeFileSync(filePath, 'test', 'utf-8') + expect(() => analyzeSkill(filePath)).toThrow('Not a directory') + }) + + it('produces valid JSON output', () => { + createFixture(tempDir, { + 'SKILL.md': 'Simple skill content.', + }) + + const result = analyzeSkill(tempDir) + const json = JSON.stringify(result) + + expect(() => JSON.parse(json)).not.toThrow() + + const parsed = JSON.parse(json) + expect(parsed).toHaveProperty('findings') + expect(parsed).toHaveProperty('summary') + expect(parsed.summary).toHaveProperty('total_findings') + expect(parsed.summary).toHaveProperty('estimated_tokens_per_invocation') + expect(parsed.summary).toHaveProperty('efficiency_score') + }) + + it('calculates efficiency score correctly for clean skill', () => { + createFixture(tempDir, { + 'SKILL.md': 'A concise skill with minimal overhead.', + }) + + const result = analyzeSkill(tempDir) + + // Clean small skill should score high + expect(result.summary.efficiency_score).toBeGreaterThanOrEqual(90) + }) + + it('calculates efficiency score correctly for bloated skill', () => { + const largeContent = 'X'.repeat(20000) + createFixture(tempDir, { + 'SKILL.md': largeContent, + 'CLAUDE.md': largeContent, + }) + + const result = analyzeSkill(tempDir) + + // Bloated skill should score lower + expect(result.summary.efficiency_score).toBeLessThan(90) + }) +})