Skip to content

πŸ—ΊοΈ Roadmap(modernization): Master-Bot Unified Architecture, Lavalink v4 Audio Engine, Dual DB/Redis Fallbacks & Agent EcosystemΒ #828

Description

@PhantomNimbi

πŸ—ΊοΈ Roadmap(modernization): Master-Bot Unified Architecture, Lavalink v4 Audio Engine, Dual DB/Redis Fallbacks & Agent Ecosystem

Note

  • Primary Open Pull Request: All changes tracked in this issue are consolidated in Pull Request #829.
  • Repository Fork: Developed and maintained at PhantomNimbi/Master-Bot.
  • Lifecycle Policy: This master tracking issue details all architectural, infrastructural, and feature differences between upstream galnir/Master-Bot and the PhantomNimbi/Master-Bot modernization fork.

πŸ“– Summary

This issue serves as the master architectural blueprint and tracking roadmap for modernizing Master-Bot. The project unifies the Discord bot gateway and the Next.js 15 web dashboard into a single, high-performance Node.js service, embeds the Lavalink v4 audio server via @helix-origin/lavalink-server, incorporates a universal testing suite via @helix-origin/vitest-suite, introduces dual external PostgreSQL/Redis support with zero-ops SQLite/ioredis-mock fallbacks, transitions hosting recommendations from unreliable PaaS cloud platforms to high-performance low-cost VPS providers (with Heroku as an optional cloud alternative), and deploys an extensive multi-agent ecosystem (.agents/).


🎯 Motivation & Context

Upstream galnir/Master-Bot faced critical operational hurdles:

  1. Broken Audio Engine: Lavalink v3 experienced YouTube IP rate limits, broken scraping, and lacked plugin extensibility.
  2. Operational Complexity: Required separate PostgreSQL and Redis server daemons to run even a single local development instance.
  3. Multi-Process Friction: Separate Node.js processes for bot gateway (3000) and dashboard (3001) caused port collisions and deployment issues.
  4. Cloud PaaS Limitations: Deploying to ephemeral PaaS containers (Render, Railway, Fly.io) resulted in lost SQLite data upon sleep/restart, high costs for 2GB+ RAM required by Java Lavalink, and blocked subdomains in OAuth flows.
  5. Testing & Tooling Gaps: Incomplete test coverage and loose linting configurations led to hidden regressions.

This modernization fork resolves these challenges by making the stack zero-ops out of the box while establishing clear production paths on low-cost unmetered Linux VPS instances.


πŸ“ Structural Guidance (Mermaid Diagrams)

1. Unified Service Architecture

flowchart TD
    subgraph ClientLayer [Clients & Endpoints]
        DiscordAPI[Discord Gateway API]
        WebUsers[Browser Users / Dashboard]
        VoiceGateway[Discord Voice WebSockets]
    end

    subgraph MasterBotService [Master-Bot Unified Service :3000]
        Router[Internal HTTP / SSR Web Server]
        BotClient[Sapphire Discord Client]
        DashboardApp[Next.js 15 App Router & tRPC v11]
        EmbeddedLavalink[Embedded Lavalink Server / Supervisor]
        SessionMgr[In-Memory SessionManager]
    end

    subgraph StorageLayer [Dual Storage & Fallback Architecture]
        subgraph DBEngine [Database Layer]
            direction TB
            PG[(External PostgreSQL)]
            SQLite[(Local SQLite: db.sqlite)]
            DBEngineSelect{DATABASE_URL starts with postgres?}
            DBEngineSelect -->|Yes| PG
            DBEngineSelect -->|No / Default| SQLite
        end

        subgraph CacheEngine [Cache Layer]
            direction TB
            ExtRedis[(External Redis Server)]
            MockRedis[In-Memory ioredis-mock]
            CacheSelect{REDIS_URL or REDIS_HOST set?}
            CacheSelect -->|Yes| ExtRedis
            CacheSelect -->|No / Fallback| MockRedis
        end
    end

    DiscordAPI <--> BotClient
    WebUsers <--> Router
    Router <--> DashboardApp
    DashboardApp <--> BotClient
    BotClient <--> SessionMgr
    SessionMgr <--> StorageLayer
    BotClient <--> EmbeddedLavalink
    VoiceGateway <--> EmbeddedLavalink
Loading

2. Embedded Lavalink Audio Pipeline

sequenceDiagram
    autonumber
    actor User as Discord User
    participant Bot as Master-Bot (Sapphire Client)
    participant Lava as Embedded Lavalink Server (@helix-origin/lavalink-server)
    participant YT as YouTube API / Remote Cipher
    participant DiscVoice as Discord Voice Channel

    User->>Bot: /play query: "lo-fi beats"
    Bot->>Lava: REST search / loadtracks
    Lava->>YT: Resolve track metadata & stream signatures
    YT-->>Lava: Audio track streams
    Lava-->>Bot: Track load response
    Bot->>DiscVoice: Join voice channel
    Bot->>Lava: WebSocket voice update (session ID & token)
    Lava->>DiscVoice: Stream real-time Opus audio packets
    Bot-->>User: Now Playing rich embed with interactive buttons
Loading

3. Database & Cache Seamless Fallback State Machine

stateDiagram-v2
    [*] --> InspectEnv: Service Startup
    InspectEnv --> PostgreSQL: DATABASE_URL = postgresql://...
    InspectEnv --> SQLiteFallback: Default / file:./db.sqlite
    PostgreSQL --> ConnectPostgres
    ConnectPostgres --> PostgresActive: Success
    ConnectPostgres --> SQLiteFallback: Connection Refused / Fallback
    
    InspectEnv --> ExternalRedis: REDIS_URL or REDIS_HOST configured
    InspectEnv --> MockRedisFallback: Default / In-Memory
    ExternalRedis --> ConnectRedis
    ConnectRedis --> RedisActive: Success
    ConnectRedis --> MockRedisFallback: Timeout / Error
Loading

πŸ“Š High-Level Comparison Matrix

Component / Dimension Upstream (galnir/Master-Bot) Modernization Fork (PhantomNimbi/Master-Bot)
Runtime Architecture Multi-process: Bot (3000) & Dashboard (3001) run separately Single Node.js Process: Bot gateway & Dashboard unified on single PORT (3000)
Database Layer External PostgreSQL server only (postgresql://...) Dual Support with Fallback: External PostgreSQL + embedded packages/db/prisma/db.sqlite
Cache / State Store Standalone Redis server process only (redis://...) Dual Support with Fallback: External Redis + internal in-memory ioredis-mock
Audio Engine Lavalink v3 (deprecated, broken YouTube scraping) Lavalink v4: Embedded server via @helix-origin/lavalink-server
Testing Suite Minimal test setup Universal Testing Suite: @helix-origin/vitest-suite with Discord & storage mocks
Web Dashboard Next.js 13/14 Pages Router Next.js 15 (App Router): React 18, tRPC v11, NextAuth v5 beta, 9 Studios
Hosting & Deployment No clear low-cost deployment guide Low-Cost VPS First: Hetzner (€3.79/mo), OVH ($4.20/mo), DigitalOcean, Linode, Vultr, Contabo (optional Heroku)
Keep-Alive & Uptime None (sleeps on free tiers after 15 mins) Built-in Keep-Alive pinger (/health auto-pinged every 10 mins)
Command Suite Legacy slash commands 74 Slash Commands: Music, Moderation, Tickets, Reminders, Games, GIFs
Agent Ecosystem None Extensive .agents/ ecosystem: 8 agents, 8 skills (gh CLI, issue, wiki), 5 rules, 5 templates
Documentation Minimal README and docs 16-Page Comprehensive Wiki + GitHub-formatted Mermaid diagrams

πŸš€ Complete Modernization Breakdown

1. πŸ—οΈ Consolidated Single-Process Architecture

  • Unified Port & Process: Discord Bot Gateway (Sapphire framework) and Next.js 15 Web Dashboard (/dashboard) co-exist in a single Node.js process listening on PORT (default 3000).
  • Internal Web Server (apps/bot/src/lib/server/webServer.ts):
    • Serves landing root (/) and handles dashboard SSR routing.
    • Exposes /health endpoint for container health probes and uptime monitors.
    • Background keepAlive service auto-pings /health every 10 minutes to prevent ephemeral cloud spin-downs.
  • Unified Scripts: pnpm dev and pnpm start operate out of the box across all operating systems.

2. πŸ—„οΈ Dual Database Architecture (PostgreSQL with SQLite Fallback)

  • Zero-Ops Default: Runs SQLite (packages/db/prisma/db.sqlite) automatically without database installation.
  • Dynamic Schema Preparation (packages/db/scripts/prepare-schema.mjs): Inspects DATABASE_URL at build/push time. When pointed at PostgreSQL, dynamically generates PostgreSQL schema with @db.Text annotations; otherwise configures SQLite.
  • Shared ORM Instance: Bot and Dashboard share @master-bot/db client for zero-latency cross-talk.

3. ⚑ Dual Cache Architecture (External Redis with ioredis-mock Fallback)

  • Configurable External Redis: Reads REDIS_URL or REDIS_HOST & REDIS_PORT.
  • Automatic In-Memory Fallback: When Redis is omitted or unreachable, seamlessly activates in-memory ioredis-mock with graceful warning logs instead of crashing the process.

4. 🎡 Embedded Lavalink v4 Audio Engine (@helix-origin/lavalink-server)

  • Embedded Audio Server: Imports @helix-origin/lavalink-server directly into apps/bot.
  • Turnkey Supervisor: Manages Lavalink v4 Java process in-memory or acts as high-throughput proxy gateway.
  • Remote Audio Node Toggle: LAVA_EXTERNAL=true toggle allows instant connection to external remote audio nodes when deploying on constrained container memory limits.

5. 🌐 Low-Cost VPS Hosting Strategy (Replacing Ephemeral PaaS)

  • Why VPS is Preferred: Eliminates ephemeral filesystem resets that destroy SQLite databases, avoids high RAM pricing on PaaS for Lavalink audio, and prevents OAuth domain blocking.
  • Recommended Providers: Hetzner Cloud (CX22 €3.79/mo), OVHcloud Starter ($4.20/mo), DigitalOcean ($4-$6/mo), Linode ($5/mo), Vultr ($3.50-$5/mo), and Contabo (VPS S ~€5.50/mo for 8GB RAM).
  • Heroku Cloud Alternative: Documented as an optional choice for users requiring managed PaaS, configured with Heroku Postgres and external Lavalink.

6. πŸ§ͺ Universal Modular Vitest Testing Suite (@helix-origin/vitest-suite)

  • Monorepo-aware Vitest configuration via defineMonorepoConfig.
  • Discord.js v14 mocks (createMockClient, createMockInteraction, createMockGuild), HTTP server harnesses, and Redis test doubles.

7. πŸ€– Autonomous Agent Ecosystem (.agents/ & AGENTS.md)

  • 8 Specialized Agents: architect, database-engineer, audio-engineer, test-engineer, github-specialist, dashboard-specialist, bot-specialist, release-engineer.
  • 8 Dedicated Skills:
    • gh-cli-expert: Comprehensive GitHub CLI automation (issues, PRs, runs, releases, tarballs).
    • issue-orchestrator: Human-readable issue creation with emojis, Mermaid diagrams, and sub-issues.
    • wiki-management: Proper GitHub wiki links, sidebar, and footer pages.
    • database-fallback: PostgreSQL/SQLite and Redis/ioredis-mock management.
    • embedded-lavalink: Audio node supervision and proxy routing.
    • vitest-suite-expert: Test authoring and quality gating.
    • monorepo-orchestrator: Turbo and pnpm pipeline execution.
    • release-orchestrator: Release tagging and packaging.
  • Standardized Rules & Templates: Commit message standards with emojis, issue templates, and test specifications.

🧩 Sub-Issues & Tasks (Discrete Milestone Tracking)

  • 🧩 Sub-Task 1: Resolve monorepo-wide typos (INTERNAL_URL) and clean environment schemas
  • 🧩 Sub-Task 2: Modernize TypeScript configs (apps/bot, @master-bot/db) and establish pre-compiled declarations
  • 🧩 Sub-Task 3: Implement dual external PostgreSQL support with zero-ops SQLite fallback in @master-bot/db
  • 🧩 Sub-Task 4: Implement dual external Redis support with internal ioredis-mock fallback in @master-bot/db
  • 🧩 Sub-Task 5: Import and embed Lavalink v4 server via @helix-origin/lavalink-server with external node toggle
  • 🧩 Sub-Task 6: Import @helix-origin/vitest-suite testing toolkit and configure workspace test runners
  • 🧩 Sub-Task 7: Replace cloud hosting instructions with recommended low-cost VPS providers and optional Heroku guide
  • 🧩 Sub-Task 8: Construct .agents/ ecosystem (Agents, Skills, Rules, Templates) with GitHub CLI & Issue Standards
  • 🧩 Sub-Task 9: Author complete 16-page documentation wiki with Mermaid diagrams, sidebar, and footer
  • 🧩 Sub-Task 10: Validate monorepo with pnpm lint, pnpm type-check, and pnpm test (0 errors)

πŸ” Acceptance Criteria & Quality Gates

  • Type Safety: pnpm type-check succeeds with 0 TypeScript compilation errors across all workspace packages.
  • Linting & Code Quality: pnpm lint passes with 0 errors via ESLint and manypkg.
  • Unit Testing: pnpm test executes with 100% pass rate using @helix-origin/vitest-suite.
  • Zero-Ops Default: Repository boots from scratch with pnpm install && pnpm start with zero external dependencies.
  • Production Scaling: Connecting PostgreSQL and Redis via environment variables works without code modifications.
  • VPS Ready: Clear guides provided for Docker Compose and Node.js + PM2 on low-cost VPS hosts (Hetzner, OVH, DigitalOcean, Linode, Vultr, Contabo), with Heroku as an optional cloud alternative.

πŸ› οΈ Verification Matrix

βœ“ pnpm lint           β€” 0 errors across 6 workspaces (FULL TURBO)
βœ“ pnpm type-check     β€” 0 errors across all packages (@master-bot/auth, @master-bot/bot, @master-bot/dashboard, @master-bot/db)
βœ“ pnpm test           β€” 100% test pass rate using @helix-origin/vitest-suite presets
βœ“ Database Pre-check  β€” Prepared Prisma schema for SQLite / PostgreSQL dynamically
βœ“ Deployment Docs     β€” VPS guides (Docker/PM2) + low-cost host comparisons + optional Heroku guide

This master tracking issue remains OPEN until Pull Request #829 is accepted and merged into main.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions