From 5e18e896fde096f4c8fde86e19cbbd354ce5ca08 Mon Sep 17 00:00:00 2001 From: Sabyasachi Date: Fri, 14 Aug 2026 11:33:08 +0000 Subject: [PATCH] docs: correct and cut the markdown MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README named the package @stackbox/fp-plugins in all six install and import examples; it is @stackbox-dev/fp-plugins, so following the README could not work. It also documented 168 lines of the event-bus plugin deleted in c001efc, claimed Node 18+ against engines >=22, described the local default as the system temp dir rather than a stackboxwms subdirectory, and gave no registry config for a package published only to GitHub Packages. Rewritten for consumers: 378 -> 154 lines, with env vars checked line by line against file-store.ts, and the automatic FastifyInstance.FileStore typing and lazy SDK loading documented for the first time. CLAUDE.md claimed its list was every script (prepare and prepublishOnly are also defined) and omitted utils.spec.ts. gen-test skill said cloud SDKs are never containerised and there are no Docker-based tests, contradicting the MinIO integration spec. security-reviewer agent reviewed the deleted event-bus — broker connections, the publish endpoint, NATS and RabbitMQ clients. Rescoped to the file store. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/agents/security-reviewer.md | 25 +- .claude/skills/gen-test/SKILL.md | 10 +- CLAUDE.md | 7 +- CONTRIBUTING.md | 5 +- README.md | 366 ++++++---------------------- 5 files changed, 101 insertions(+), 312 deletions(-) diff --git a/.claude/agents/security-reviewer.md b/.claude/agents/security-reviewer.md index 0fa5650..29fea58 100644 --- a/.claude/agents/security-reviewer.md +++ b/.claude/agents/security-reviewer.md @@ -1,18 +1,25 @@ --- name: security-reviewer -description: Reviews event-bus and file-store code for credential handling, injection risks, and insecure defaults +description: Reviews file-store code for credential handling, path traversal, and insecure defaults tools: ["Read", "Grep", "Glob"] --- -You are a security reviewer for a Fastify plugin library that handles cloud credentials and message broker connections. +You are a security reviewer for `@stackbox-dev/fp-plugins`, a Fastify plugin library +that wraps cloud storage (AWS S3, GCS, Azure Blob, MinIO, local filesystem) behind one +`FileStore` interface (`src/file-store.ts`). Review the changed or specified files for: -1. **Credential exposure**: Hardcoded secrets, credentials logged to stdout, tokens in error messages -2. **Unsafe deserialization**: `JSON.parse` without try/catch or validation, especially on message payloads from external brokers -3. **Injection risks**: Unsanitized input in the `/event-bus/publish/:event` endpoint, template literal injection in queue/topic names -4. **Insecure defaults**: Missing TLS, overly permissive CORS, unauthenticated endpoints exposed by default -5. **Resource leaks**: Unclosed connections, missing cleanup in error paths, containers not stopped in tests -6. **Dependency concerns**: Known vulnerable patterns in AWS SDK, Azure SDK, GCP SDK, NATS, or RabbitMQ client usage +1. **Credential exposure**: hardcoded secrets, credentials or endpoints logged, tokens + leaking into error messages +2. **Path traversal**: `filepath` arguments are joined onto a base directory in + `LocalFileStore` and used as object keys in the cloud providers — check whether new + code lets `..` escape `LOCAL_STORAGE_DIR` or weakens key handling +3. **Insecure defaults**: env-var fallbacks that silently point at the wrong + region/endpoint, TLS disabled, anonymous credentials +4. **Resource leaks**: unclosed streams or clients, missing cleanup in error paths +5. **SDK misuse**: unsafe patterns in AWS/GCP/Azure SDK usage, e.g. disabling + certificate validation -Report findings with severity (critical/high/medium/low), file location, and a suggested fix. +Report findings with severity (critical/high/medium/low), file location, and a +suggested fix. diff --git a/.claude/skills/gen-test/SKILL.md b/.claude/skills/gen-test/SKILL.md index b40fd06..f187bd3 100644 --- a/.claude/skills/gen-test/SKILL.md +++ b/.claude/skills/gen-test/SKILL.md @@ -20,9 +20,13 @@ Generate a Jest test file for a given source module in this project. - Test files live alongside source: `src/foo.ts` → `src/foo.spec.ts`. Provider-specific specs are suffixed: `file-store..spec.ts` - `ts-jest` preset, `testEnvironment: "node"` -- Cloud SDKs are **mocked**, never containerised — `jest.mock("@google-cloud/storage")` - and friends. This package has no Docker-based tests and no `testcontainers` - dependency; do not add one +- Cloud SDKs are **mocked** in unit specs — `jest.mock("@google-cloud/storage")` and + friends. The one exception is `src/file-store.minio.integration.spec.ts`, which talks + to a real MinIO over the S3 wire protocol; it self-skips unless `MINIO_TEST_ENDPOINT` + is set and runs via `pnpm run test:integration`. No `testcontainers` dependency; do + not add one +- Behavioural changes to S3/MinIO paths need cases in the integration spec too — + mocked tests cannot catch SDK behaviour changes (see CLAUDE.md, Testing) - `LocalFileStore` is the exception: test it against the real filesystem using `fs.promises.mkdtemp` under `os.tmpdir()`, and clean up in `afterEach` - Save and restore `process.env` around tests that set provider env vars diff --git a/CLAUDE.md b/CLAUDE.md index b0edab5..14ffd34 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,7 +13,8 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co - **Clean**: `pnpm run clean` — removes `dist` - **Transpile**: `pnpm run transpile` — TypeScript compilation only -These are the only scripts defined in `package.json`. CI (`.github/workflows/`) runs +The only other scripts in `package.json` are `prepare` (husky) and `prepublishOnly` +(build before publish). CI (`.github/workflows/`) runs `pnpm install --frozen-lockfile`, then lint, build and test on Node 24; the package declares `engines: node >=22`. @@ -57,8 +58,8 @@ Registered via `fastify-plugin` (v6) and decorates the Fastify instance with ## Testing Jest + ts-jest. Tests live beside their source in `src/` as `*.spec.ts`, split by -provider: `file-store.local.spec.ts`, `.gcs.`, `.s3.`, `.azure.`, plus the original -`file-store.spec.ts` and `integration.spec.ts`. Coverage is 100% on statements, +provider: `file-store.local.spec.ts`, `.gcs.`, `.s3.`, `.azure.`, plus +`file-store.spec.ts`, `integration.spec.ts` and `utils.spec.ts`. Coverage is 100% on statements, branches, functions and lines — keep it there. Cloud SDKs are mocked with `jest.mock(...)`; `LocalFileStore` runs against the real diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1831b29..bb7898f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,8 +2,9 @@ ## Setup -This repository uses **pnpm** (pinned via `packageManager` in `package.json`). Do not -use npm or yarn — `pnpm-lock.yaml` is the committed lockfile and the only one CI reads. +Node.js >= 22 (CI runs 24). This repository uses **pnpm** (pinned via `packageManager` +in `package.json`). Do not use npm or yarn — `pnpm-lock.yaml` is the committed lockfile +and the only one CI reads. ```bash pnpm install diff --git a/README.md b/README.md index 4ebf7e1..2924e84 100644 --- a/README.md +++ b/README.md @@ -1,264 +1,95 @@ -# @stackbox/fp-plugins +# @stackbox-dev/fp-plugins -Fastify plugins for Stackbox applications. +Fastify plugins for Stackbox applications. Currently one plugin: **FileStore**, a +single interface over five storage backends. ## Installation -```bash -pnpm install @stackbox/fp-plugins -# or -npm install @stackbox/fp-plugins -``` - -## Overview - -This package provides a collection of Fastify plugins designed to enhance Stackbox applications. These plugins extend Fastify's functionality while maintaining its performance and developer experience. - -## Available Plugins - -### Event Bus Plugin - -A Fastify plugin that integrates an event bus system into your application, enabling efficient event-driven communication between different parts of your application. - -#### Supported Message Brokers - -- **RabbitMQ** (`rabbitmq`) -- **Google Cloud Pub/Sub** (`gcp-pubsub`) -- **Azure Service Bus** (`azure-servicebus`) -- **In-process** (`in-process`) - for development and testing +The package is published to GitHub Packages, not npmjs.org. Point the +`@stackbox-dev` scope at the GitHub registry in `.npmrc`: -#### Configuration - -```typescript -app.register(Plugins.EventBus, { - // Required: type of message broker to use - busType: "rabbitmq" | "gcp-pubsub" | "azure-servicebus" | "in-process", - - // Required for gcp-pubsub and azure-servicebus - topic: "your-topic-name", - - // Required for azure-servicebus - namespace: "your-namespace", - - // Required: define event handlers - handlers: [ - { - file: "module-name", - handlers: { - "event-name": async function (msg, req) { - // Handle the event - }, - }, - }, - ], - - // Required: message validation function - validateMsg: (event, payload, req) => { - // Validate the message - }, - - // Required: error processing function - processError: (err, ctx) => { - // Process the error - return { err, status: 500 }; - }, - - // Optional: disable the /event-bus/publish/:event route - disableEventPublishRoute: false, - - // Optional: control concurrency of event handlers - actionConcurrency: 1, - - // Optional: Prometheus registry for metrics - registry: new Registry(), -}); ``` - -#### Environment Variables - -##### RabbitMQ Configuration - -- `RABBITMQ_URL`: RabbitMQ connection URL -- `K_SERVICE`: Service name for queue naming - -##### Google Cloud Pub/Sub Configuration - -- `EVENT_TOPIC`: GCP Pub/Sub topic name -- `EVENT_SUBSCRIPTION`: GCP Pub/Sub subscription name -- `EVENT_SUBSCRIPTION_MAX_MESSAGES`: Maximum concurrent messages (default: 10) - -##### Azure Service Bus Configuration - -- `EVENT_NAMESPACE`: Azure Service Bus namespace -- `EVENT_TOPIC`: Azure Service Bus topic -- `EVENT_SUBSCRIPTION`: Azure Service Bus subscription -- `EVENT_SUBSCRIPTION_MAX_CONCURRENT_CALLS`: Maximum concurrent calls (default: 10) - -##### Retry Configuration - -- `EVENT_RETRY_BASE_DELAY`: Base delay for exponential backoff in ms (default: 5000) -- `EVENT_RETRY_MAX_DELAY`: Maximum delay for retries in ms (default: 60000) - -#### Usage Example - -```typescript -import { - EventBus, - EventBusOptions, - EventMessage, - Plugins, -} from "@stackbox/fp-plugins"; - -const app = fastify(); - -// Register event bus -app.register(Plugins.EventBus, { - busType: "rabbitmq", - validateMsg: (event, payload) => { - // Validate event and payload - console.log(`Validating message: ${event}`); - }, - processError: (err, ctx) => { - console.error(`Error processing message: ${err.message}`); - return { err, status: 500 }; - }, - handlers: [ - { - file: "orderModule", - handlers: { - "order.created": async function (msg, req) { - // Process order created event - console.log(`Processing order: ${msg.data.orderId}`); - }, - "order.cancelled": async function (msg, req) { - // Process order cancelled event - console.log(`Processing cancelled order: ${msg.data.orderId}`); - }, - }, - }, - ], -}); - -// Publishing events -app.post("/create-order", async (req, reply) => { - // Create order logic... - - // Publish event - req.EventBus.publish("order.created", { - orderId: "order-123", - customerId: "customer-456", - }); - - return { success: true }; -}); - -// Delayed events (process after specified milliseconds) -app.post("/schedule-reminder", async (req, reply) => { - req.EventBus.publish( - "reminder.send", - { - userId: "user-123", - message: "Reminder to complete your profile", - }, - 3600000, - ); // Process after 1 hour - - return { scheduled: true }; -}); +@stackbox-dev:registry=https://npm.pkg.github.com ``` -#### Event Consumer - -To consume events from external services, use the `CreateEventConsumer` function: - -```typescript -import { CreateEventConsumer } from "@stackbox/fp-plugins"; - -// Create consumer that matches your EventBus busType -const consumer = await CreateEventConsumer(app, "rabbitmq"); - -// Close consumer when application shuts down -app.addHook("onClose", async () => { - await consumer.close(); -}); +```bash +pnpm add @stackbox-dev/fp-plugins ``` -### File Store Plugin - -A plugin that provides an abstraction for file storage operations across different providers: +Requires Node.js >= 22. Fastify 3, 4 or 5 is a peer dependency. -- Local filesystem -- AWS S3 -- Google Cloud Storage -- Azure Blob Storage -- MinIO +## FileStore -#### Configuration +Register the plugin with a provider `type`; all other configuration comes from +environment variables. ```typescript -app.register(Plugins.FileStore, { - // Required: type of storage provider to use - type: "local" | "s3" | "gcs" | "azureBlob" | "minio", +import { fastify } from "fastify"; +import { Plugins } from "@stackbox-dev/fp-plugins"; + +const app = fastify(); +await app.register(Plugins.FileStore, { + type: "s3", // "local" | "gcs" | "s3" | "minio" | "azureBlob" }); ``` -#### Environment Variables - -##### Local File System - -- `LOCAL_STORAGE_DIR`: Directory for file storage (default: system temp directory) +- The plugin decorates the instance as `app.FileStore` and augments Fastify's + types, so `app.FileStore` is fully typed with no declaration on your side. +- Cloud SDKs load lazily: registering `type: "s3"` loads only the AWS SDK; the + GCS and Azure SDKs stay untouched, and vice versa. +- An unknown `type` throws `Unknown storage type: ...` at registration. -##### AWS S3 +### Providers -- `AWS_S3_REGION`: AWS region. Falls back to the standard `AWS_REGION`, then to - "us-east-1". Set `AWS_S3_REGION` only to point S3 at a different region from the - rest of the process. -- `S3_BUCKET`: S3 bucket name (required) -- Standard AWS authentication environment variables +#### `local` — local filesystem -The plugin also supports other AWS authentication methods including: +| Variable | Required | Default | +| ------------------- | -------- | ------------------------- | +| `LOCAL_STORAGE_DIR` | no | `/stackboxwms` | -- ECS/EC2 instance roles -- AWS IAM roles for service accounts (IRSA) -- Web identity providers -- AWS profiles +For development. `getInfo` always reports `contentType: "application/octet-stream"`. -##### Google Cloud Storage +#### `s3` — AWS S3 -- `STORAGE_BUCKET`: GCS bucket name (required) -- Standard GCP authentication environment variables +| Variable | Required | Default | +| --------------- | -------- | -------------------------------------------- | +| `S3_BUCKET` | yes | — | +| `AWS_S3_REGION` | no | falls back to `AWS_REGION`, then `us-east-1` | -The plugin also supports other GCP authentication methods including: +Set `AWS_S3_REGION` only to point S3 at a different region from the rest of the +process. Credentials come from the standard AWS chain: env vars, ECS/EC2 instance +roles, IRSA, web identity, profiles. -- GKE Workload Identity -- Compute Engine service accounts -- GCP Application Default Credentials (ADC) -- Service account key files (not recommended for production) +#### `gcs` — Google Cloud Storage -##### Azure Blob Storage +| Variable | Required | +| ---------------- | -------- | +| `STORAGE_BUCKET` | yes | -- `AZURE_STORAGE_ACCOUNT_URL`: Azure storage account URL (required) -- `AZURE_STORAGE_CONTAINER`: Azure storage container name (required) -- Standard Azure authentication environment variables +Credentials via Application Default Credentials: GKE Workload Identity, Compute +Engine service accounts, or `GOOGLE_APPLICATION_CREDENTIALS`. -The plugin also supports other Azure authentication methods including: +#### `azureBlob` — Azure Blob Storage -- Managed Identities for Azure resources -- Azure AD workload identity -- Azure service principals -- Azure DefaultAzureCredential chain +| Variable | Required | +| --------------------------- | -------- | +| `AZURE_STORAGE_ACCOUNT_URL` | yes | +| `AZURE_STORAGE_CONTAINER` | yes | -##### MinIO +Credentials via `DefaultAzureCredential`: managed identity, workload identity, +service principal, CLI login. -- `MINIO_ENDPOINT`: MinIO server endpoint (required) -- `MINIO_ACCESS_KEY_ID`: MinIO access key (required) -- `MINIO_SECRET_ACCESS_KEY`: MinIO secret key (required) -- `MINIO_REGION`: MinIO region (default: "us-east-1") -- `MINIO_BUCKET`: MinIO bucket name (required) +#### `minio` — MinIO (S3-compatible, path-style addressing) -#### FileStore Interface +| Variable | Required | Default | +| ------------------------- | -------- | ----------- | +| `MINIO_ENDPOINT` | yes | — | +| `MINIO_ACCESS_KEY_ID` | yes | — | +| `MINIO_SECRET_ACCESS_KEY` | yes | — | +| `MINIO_BUCKET` | yes | — | +| `MINIO_REGION` | no | `us-east-1` | -The plugin provides a `FileStore` interface with the following methods: +### API ```typescript interface FileStore { @@ -290,56 +121,24 @@ interface FileInfo { } ``` -#### Usage Example +For a missing file, `exists` returns `false` and `getInfo` returns `null`; +`getAsBuffer` and `getAsStream` throw. -```typescript -import { FileStore, Plugins } from "@stackbox/fp-plugins"; -import { fastify } from "fastify"; - -const app = fastify(); - -// Register file store plugin -app.register(Plugins.FileStore, { - type: "s3", // Choose the appropriate storage type -}); +### Example -// Using the file store +```typescript app.post("/upload", async (request, reply) => { const { filepath, contentType, data } = request.body; - // Check if file exists - const exists = await request.server.FileStore.exists(filepath); - - // Get file info (returns null if file doesn't exist) - const fileInfo = await request.server.FileStore.getInfo(filepath); - if (fileInfo) { - console.log( - `File size: ${fileInfo.size}, Content type: ${fileInfo.contentType}`, - ); - } - - // Save file await request.server.FileStore.save(filepath, contentType, data); - // Get file as buffer - const fileContent = await request.server.FileStore.getAsBuffer(filepath); - - // Get file as stream - const fileStream = await request.server.FileStore.getAsStream(filepath); - - // Copy from stream - await request.server.FileStore.copyFromStream( - filepath, - contentType, - someReadableStream, - ); + const info = await request.server.FileStore.getInfo(filepath); + if (info) { + console.log(`size=${info.size} contentType=${info.contentType}`); + } - // Copy from local file - await request.server.FileStore.copyFromLocalFile( - filepath, - contentType, - "/path/to/local/file", - ); + const buffer = await request.server.FileStore.getAsBuffer(filepath); + const stream = await request.server.FileStore.getAsStream(filepath); return { success: true }; }); @@ -347,31 +146,8 @@ app.post("/upload", async (request, reply) => { ## Development -### Prerequisites - -- Node.js 18+ -- pnpm - -### Setup - -```bash -pnpm install -``` - -### Common Commands - -- `pnpm test` - Run tests -- `pnpm run test:coverage` - Run tests with coverage -- `pnpm run build` - Build the project -- `pnpm run pretty` - Format code - -### Testing - -Tests are written using Jest and located alongside source files with `.spec.ts` extension. - -## Contributing - -Contributions are welcome! Please feel free to submit a Pull Request. +See [CONTRIBUTING.md](CONTRIBUTING.md) for setup, PR rules, and the release +process. Architecture notes live in [CLAUDE.md](CLAUDE.md). ## License