Skip to content
fractalsynapsePublic

Repository files navigation

SAIP — Simple AI Platform

A two-service continual-learning platform that trains QLoRA adapters from agent experience and serves them via vLLM with per-request adapter selection. Designed to run on a single 7 GB VRAM GPU (e.g., gaming laptop) or scale to remote GPU nodes.

Architecture

SAIP consists of two interlocking services connected through a shared remote object store (Cloudflare R2) and a shared YAML configuration file.

┌──────────────────────────────────────────────────────────────┐
│                      SAIP Platform                           │
│                                                               │
│   saip.yaml (shared config)        .env (secrets, gitignored) │
│          │                               │                    │
│     ┌────┴─────┐                   ┌────┴─────┐              │
│     ▼          ▼                   ▼          ▼               │
│ ┌──────────┐ ┌──────────┐   ┌──────────┐ ┌──────────┐       │
│ │Inference │ │ Training │   │Inference │ │ Training │       │
│ │ Gateway  │ │ Gateway  │   │ Gateway  │ │ Gateway  │       │
│ │ (Hono)   │ │ (Hono)   │   │ (Hono)   │ │ (Hono)   │       │
│ │ Data API │ │Learning  │   │ Data API │ │Learning  │       │
│ │          │ │ API      │   │          │ │ API      │       │
│ └────┬─────┘ └────┬─────┘   └────┬─────┘ └────┬─────┘       │
│      │            │              │             │              │
│      ▼            ▼              ▼             ▼              │
│    vLLM        Axolotl         vLLM         Axolotl          │
│  + adapters    + QLoRA       + adapters     + QLoRA          │
│  + doc proc.                  + doc proc.                    │
│      │            │              │             │              │
│      └─────┬──────┘              └──────┬──────┘              │
│            │ writes                reads │                    │
│            ▼                            ▼                     │
│      Cloudflare R2 ◄──────────► Cloudflare R2                 │
│      (handoff boundary: adapters + knowledge + sessions +     │
│       feedback)                                               │
│                                                               │
│   HuggingFace Hub ◄── base model weights                      │
│   GitHub ◄── project config (saip.yaml)                       │
│   Hyperstack ◄── remote GPU compute (optional)                │
└──────────────────────────────────────────────────────────────┘

Inference Gateway

Serves OpenAI-compatible chat completions via vLLM with dynamic per-request LoRA adapter selection. Also owns the Data API (/v1/data/*) for document and feedback ingestion — document processors run here because they may need model inference (OCR, PDF parsing). A Hono API layer manages:

  • vLLM lifecycle — start/stop the inference server, load/unload adapters
  • Adapter registry — sync adapter index from R2, resolve user → adapter mapping
  • Session collection — capture conversation histories for training feedback
  • Data API — accept documents and feedback, process them, write results to R2
  • Document processors — run within inference because they may need model inference
  • SystemOne endpoint — decision modeling on the same vLLM instance as inference
  • User resolution — map caller identity to adapter group via file-based manifests

Training Gateway

Runs Axolotl fine-tuning jobs in the background using three training methods. Exposes only the Learning API (/v1/learning/*); data ingestion is handled by the inference gateway. The training GPU is dedicated 100% to Axolotl.

  • Knowledge embedding — continued pretraining on submitted documents, code, datasets
  • Corrective SFT — supervised fine-tuning with per-turn feedback weighting
  • Weighted KTO — preference learning with scalar reward magnitudes (-3 to +3)

A Hono API layer manages:

  • Dataset construction — read processed documents and feedback from R2, transform into Axolotl training formats
  • Training orchestration — queue and run Axolotl jobs, promote resulting adapters

Shared Infrastructure

  • Cloudflare R2 — adapters, knowledge, sessions, feedback (S3-compatible)
  • HuggingFace Hub — base model weights
  • GitHub — project configuration (saip.yaml)
  • Hyperstack — optional remote GPU nodes for training or inference

Both services share a single saip.yaml configuration file and can run on the same machine or on separate GPU nodes connected through R2.

Monorepo Structure

saip/
├── saip.yaml              # Shared platform configuration (model, adapters, users, security)
├── saip.schema.yaml       # JSON Schema for config validation
├── hosts.yaml             # Host registry (local + cloud targets); template: hosts.example.yaml
├── .env.example           # Secret variable template (prefer shell env)
├── cli/                   # saip CLI tool (Cordis app: commands as plugins; --host targeting, deploy)
├── inference/             # Inference Gateway (vLLM + Hono + Cordis; Data API + bearer auth)
├── training/              # Training Gateway (Axolotl + Hono + Cordis; Learning API + bearer auth)
├── shared/                # Shared Cordis services (config, hosting, auth, monitoring, adapter registry, types)
├── infra/tofu/hyperstack/ # OpenTofu module for the 2-GPU cloud environment (gated provisioning)
└── examples/cordis/       # Cordis framework reference

Hosting, security, and monitoring are pluggable seams (DeploymentProvider, AuthProvider, MonitoringProvider) so adding a host kind, an auth mechanism, or a metrics backend never means editing feature code. Local is always the baseline host; cloud targets are additional named entries in hosts.yaml. See the Hosting Model docs.

Each subproject has its own package.json, tsconfig.json, CHANGELOG.md, and README.md.

Technology Stack

Component Technology Purpose
Plugin framework Cordis v4 Service composition, plugin lifecycle, events — used by all four Node packages (both gateways, the CLI, and shared services)
Inference engine vLLM Multi-LoRA serving, AWQ quantization
Training framework Axolotl QLoRA, SFT, KTO, continued pretraining
API framework Hono Lightweight OpenAI-compatible HTTP API
Base model Qwen3.5-4B (AWQ) 7 GB VRAM target
Object storage Cloudflare R2 Adapters, knowledge, sessions
Containerization Docker Compose v2 Local and remote deployment
Language TypeScript (Node.js ≥ 22) All service and CLI code

Getting Started

Prerequisites

  • Node.js ≥ 22
  • Docker and Docker Compose v2
  • NVIDIA GPU with ≥ 7 GB VRAM (for local operation)
  • Cloudflare R2 bucket
  • HuggingFace account with access to Qwen3.5-4B

Quick Start

# 1. Clone and install
git clone https://github.com/adrianwebb/saip.git
cd saip
pnpm install

# 2. Configure secrets
cp .env.example .env
# Edit .env with your R2 credentials, HF token, etc.

# 3. Review/edit configuration
cat saip.yaml

# 4. Start in inference mode
saip awake

# 5. Check status
saip status

# 6. Switch to training mode
saip sleep

# 7. Shut down
saip down

Modes

SAIP operates in two mutually exclusive modes on a single GPU:

  • Awake — inference gateway running, training gateway stopped. The system serves requests, collects session histories, and ingests documents/feedback through the Data API (processing them with model inference for OCR/parsing) and writes results to R2.
  • Asleep — training gateway running, inference gateway stopped. The system reads accumulated data from R2 and trains new adapters. The GPU is dedicated 100% to Axolotl.

On separate GPU nodes, both services can run simultaneously.

Configuration

All platform configuration lives in saip.yaml (committed). Secrets live in .env (never committed). See saip.schema.yaml for the full schema and inference/README.md and training/README.md for service-specific details.

Remote Storage Layout

The Cloudflare R2 bucket follows this structure:

r2://<bucket>/
├── adapters/
│   ├── <adapter-id>/
│   │   ├── adapter_config.json
│   │   ├── adapter_model.safetensors
│   │   └── manifest.json          # version, parent, metrics, timestamps
│   └── _index.json                # Global adapter registry
├── knowledge/
│   ├── <knowledge-id>/
│   │   ├── content.*              # Raw documents, code, datasets
│   │   └── manifest.json          # source, license, hash, timestamps
│   └── _index.json                # Global knowledge registry
├── sessions/
│   ├── <session-id>/
│   │   ├── conversation.jsonl
│   │   ├── artifacts/
│   │   └── manifest.json
│   └── _index.json                # Global session registry
└── feedback/
    ├── <feedback-id>.json
    └── _index.json

License

Apache 2.0 — see LICENSE.

Releases

Packages

Contributors

Languages