Cap LSP worker thread pool to fix memory scaling with project count

[markwpearce]

Problem

Opening a multi-root VSCode workspace with many BrighterScript projects (e.g. ~15+, as in brighterscript-game-engine's 17 bsconfig.json files) can crash the language server with an out-of-memory error.

Root cause: LanguageServer.enableThreadingDefault defaults to true, and ProjectManager spawns one dedicated OS worker thread (a full separate V8 isolate) per project via WorkerPool.getWorker(), which had no cap — it always created a new worker if none were free, and a worker was only returned to the pool when its project was disposed (which doesn't happen for open, in-use projects during a normal session). Each isolate independently loads the whole brighterscript module graph (roku-types static data, globalCallables, type caches, etc.), so total memory scaled roughly linearly with project count, multiplying fixed interpreter/type-system overhead by N.

Fix

Replaces WorkerPool's exclusive one-worker-per-project model with a capped, load-aware assignment model:

• WorkerPool now has a configurable maxWorkers (default os.cpus().length). Each project still gets its own Project/Program instance and its own dedicated MessagePort (via MessageChannel), but once the cap is reached, additional projects' ports are attached to an existing worker instead of spawning a new one.
• Co-located projects share that worker's JS realm, recovering cross-project sharing of the big fixed-cost singletons that per-project threading had made impossible to share.
• A worker is terminated and dropped from the pool as soon as its last attached project releases it, so memory is reclaimed promptly rather than kept warm indefinitely.
• New languageServer.maxProjectWorkers setting, mirroring the existing projectActivationConcurrencyLimit pattern, so this is tunable per-workspace.
• An unexpected worker exit now surfaces a critical-failure notification (previously: requests on that worker would hang forever with no signal to the user).
• Worker lifecycle edge cases handled: idempotent dispose (a project can be disposed more than once without side effects on sibling projects sharing its worker), a crashed worker is evicted from the pool's bookkeeping (so it can't silently swallow a future project's bootstrap), preload()/assignProject() agree on the cap, and assignProject() defends against a misconfigured maxWorkers <= 0.

Testing

• New/updated unit tests in WorkerPool.spec.ts, WorkerThreadProject.spec.ts (including real-worker-thread integration tests proving multiple projects can share one worker while remaining functionally independent, and that a co-tenant surviving a sibling's dispose/double-dispose keeps working), MessageHandler.spec.ts, and LanguageServer.spec.ts.
• Full suite: 3001 passing, 0 failing. npm run lint and npm run build clean.
• Not automated: real-world confirmation against brighterscript-game-engine's multi-project workspace in VSCode (recommended as a manual follow-up check).

Notes for reviewers

• This targets master (not v1) because the affected code (src/lsp/worker/*) is currently identical between the two branches — landing here first means it flows to v1 on the next master merge, rather than requiring a manual backport.
• Scoped out of this PR: the separate Program.ts cache-leak fixes identified during investigation (fileClusters, getFilePathCache, symbolDependencies, util.ts path caches never pruned) — smaller, unrelated issue, tracked separately.
• Also scoped out: automatic reload of sibling projects when a shared worker crashes — no existing mechanism reloads a project on critical-failure today; this PR keeps that event's existing one-way-notification behavior rather than introducing new reload semantics.

🤖 Generated with Claude Code

[markwpearce]

This change makes a substantial difference - this allows opening the Roku Samples repo (~139 projects) without issues.

Additionally, this same fix in v1 also allows opening that repo in VSCode, with no noticeable lag... Taking about 20 seconds to load and validate all the projects.

[markwpearce]

I don't love that MacOS CI tests take longer, but that's due to Github's test runner hardware, I believe.

On my laptop (m3 Macbook pro) npm run test:

• previous master - 8s
• this branch - 12s

Creating Worker threads is just slow, unfortunately.