From 438a33ae69cf0c4a92b0967ac1b091a8bb9437f3 Mon Sep 17 00:00:00 2001 From: amar-python Date: Mon, 26 Oct 2026 15:23:42 +1100 Subject: [PATCH 1/3] docs: add ART framework prompt v2 for Milestone 0 (#53) --- docs/prompts/ART_FRAMEWORK.md | 285 ++++++++++++++++++++++++++++++++++ 1 file changed, 285 insertions(+) create mode 100644 docs/prompts/ART_FRAMEWORK.md diff --git a/docs/prompts/ART_FRAMEWORK.md b/docs/prompts/ART_FRAMEWORK.md new file mode 100644 index 0000000..3f5c074 --- /dev/null +++ b/docs/prompts/ART_FRAMEWORK.md @@ -0,0 +1,285 @@ +# ART Framework v2: Altitude, Range and Technical Depth + +**Target repository:** `PostgreDataMigrationAppWithCSVLoader` +**Path (confirmed 2026-10-25):** `C:\Users\User\OneDrive\Desktop\Migration using ai\PostgreDataMigrationAppWithCSVLoader` +**Product owner:** Amarnath (Amar). Scope, requirement wording and direction are his decisions. + +Use this framework to guide all analysis, design, implementation and validation of the CSV Import, Review, Deduplication and Audit capability. This is an **extension of an existing platform**, not a new build. + +--- + +## Decisions already taken (2026-10-25). Do not re-ask these + +| # | Decision | Consequence | +|---|---|---| +| D1 | The platform becomes **customer-facing with multi-tenant isolation**. | Tenant isolation is a core requirement, not an option. | +| D2 | **Full-dataset cell editing** is in scope. | Edits are stored as versions over the preserved original, never in place. | +| D3 | An **approval workflow** is in scope. | Import requires an explicit approval step. | +| D4 | Imports become **staged, approved and transactional**, with every imported row traceable to its approved version. This means threading the database connection (or an equivalent transactional boundary) through the loaders. | Supersedes the current post-commit upload-audit design. Record as an ADR (see Milestone 0). | +| D5 | Every new requirement gets a **BR ID in `docs/VCRM.md`** and is tracked there from day one. | Status starts as "Not started", never "Verified" without a test run. | +| D6 | A **data-classification and sovereignty constraint** applies (see Range). | No runtime dependency on external services. | +| D7 | The audit trail is **append-only, enforced at the database level**. | Convention alone does not satisfy the requirement. | + +### Decisions that remain in force. Do not reopen +- **ADR-0001, PostgreSQL first.** Do not add DDL for other engines, rewrite BR-02, or re-add Tier X to CI. +- **"Production-grade" README framing.** Risk accepted by the product owner. +- Requirements marked "Deferred by decision" in `docs/VCRM.md` (BR-02, BR-21, BR-22) stay deferred. + +--- + +## A — Altitude: business objective and desired outcomes + +Work at the right level of abstraction before making implementation decisions. + +### Business objective + +Extend the existing CSV-to-PostgreSQL migration platform into a secure, reliable, customer-facing, multi-tenant web application. Customers must be able to: + +- upload arbitrary CSV files +- inspect and review the complete dataset +- identify duplicate files and records +- include or exclude rows +- edit values +- obtain approval +- import the approved data into PostgreSQL + +The platform keeps its existing freeform and strict (T&E schema) import modes. + +The application must preserve the original uploaded data and keep a traceable history of every upload and material change. Its audit reports must establish what was uploaded, what was changed, who approved it, what was imported, and who performed each action. + +### Desired outcomes + +1. Customers can understand and validate their data before it enters the target database. +2. Repeated uploads are supported without silently overwriting previously imported data. +3. Duplicate files and records are identified transparently, with configurable handling rules. +4. Customers can correct or exclude unsuitable data before import. +5. Every material review decision, approval and data modification is traceable. +6. Imports are reliable, recoverable and protected against unintended data loss. +7. One tenant can never see or affect another tenant's data. +8. The system is maintainable, testable, secure and suitable for future enhancements. + +### Success measures + +Define measurable acceptance criteria for each of these areas: + +- completeness of the data preview +- accuracy of duplicate detection +- import reconciliation +- audit coverage +- data integrity +- tenant isolation +- performance +- security +- test results + +Record each criterion against its BR ID in the VCRM. + +Successful execution is not the same as business success. Show that each requirement is met through evidence from automated tests and end-to-end workflows. + +--- + +## R — Range: scope, constraints and boundaries + +Define the scope explicitly before implementation. + +### Included scope + +Each item maps to a provisional requirement ID. **Before assigning IDs, read `docs/VCRM.md`.** Where an item is already covered by an existing BR (for example, SHA-256 file deduplication or import audit history), map it to that BR instead of creating a duplicate. Then number genuinely new items from the next free ID. + +| Provisional ID | Requirement | +|---|---| +| BR-23 | Upload registry: unique ID, original filename, SHA-256, uploader, tenant, timestamp, processing status | +| BR-24 | Original uploaded file and parsed source data preserved immutably, separate from edits and approved data | +| BR-25 | Interactive preview of all rows and columns, with column type inference and validation | +| BR-26 | Row inclusion and exclusion, recorded with actor and reason | +| BR-27 | Cell editing stored as versions over the original, with original value, new value, actor, timestamp and reason | +| BR-28 | Exact-file duplicate detection using SHA-256, within and across uploads, per tenant | +| BR-29 | Configurable duplicate-record detection within and across uploads | +| BR-30 | Safe filename-derived PostgreSQL table naming, with support for repeated uploads of the same filename | +| BR-31 | Schema compatibility checks and import conflict handling, with no destructive schema changes | +| BR-32 | Approval workflow: submit, approve or reject, with recorded approver and decision | +| BR-33 | Staged, transactional, idempotent import with safe retries; every imported row traceable to its source row and approved version | +| BR-34 | Import reconciliation: rows in, excluded, edited and imported all account for each other | +| BR-35 | Append-only audit events enforced at the database level; filterable audit reports; exports protected against spreadsheet formula injection | +| BR-36 | Authentication, role-based authorisation and tenant isolation | +| BR-37 | Data classification and sovereignty (see Constraints) | + +Automated tests, documentation and deployment instructions are part of every requirement, not separate items. + +### Constraints + +**Technology** +- React and TypeScript frontend, FastAPI and Python backend, PostgreSQL storage. +- The existing Lovable frontend (`csv-table-hub-main/`) is the React/TS baseline. + +**Reuse before rebuild** +- Inspect the live repository before changing anything. +- Reuse working components and established conventions. +- Existing capabilities to assess first: + - the `api/routers/` and `api/services/` structure + - SHA-256 duplicate detection + - `api/services/audit.py` + - the freeform and strict import modes + - the VCRM traceability view + - the API key gate + - `api/ratelimit.py` + - the bash loaders in `build/csv/` + +**Security** +- Treat uploaded files as untrusted input. +- Protect against: + - SQL injection + - malformed files + - unsafe database identifiers + - resource exhaustion + - unauthorised and cross-tenant access + - spreadsheet formula injection in exported reports + +**Data preservation** +- Preserve original data and import history. +- Never silently discard columns, truncate values, overwrite earlier imports or make destructive schema changes. + +**Data classification and sovereignty** +- Uploaded data, derived data and audit records must stay inside the deployment boundary. +- Prohibited at runtime: + - external SaaS calls + - telemetry + - CDN assets + - third-party AI or processing services +- The full stack must remain installable and runnable on-premise, including in air-gapped environments. This supports the 2026-11-15 offline-install commitment. +- Classification level is to be confirmed. Design so that data handling can be tightened per deployment without code changes. + +**Append-only audit** +- Enforce immutability in the database. Options include: + - a dedicated audit role with no UPDATE or DELETE grant + - triggers that reject modification + - a hash chain over events + + Propose the mechanism and test it, including a negative test that attempts an update and a delete. + +### Unresolved decisions. Raise these in Milestone 0 with a recommended default for each + +1. Maximum file size and expected concurrency. +2. Repeated uploads of the same filename: append to the same table, or create versioned datasets. +3. Default duplicate-record matching rule and resolution policy (skip, flag, or merge). +4. Whether partial imports are permitted. +5. Approval separation of duties: can an uploader approve their own data? +6. Tenant model: schema-per-tenant, database-per-tenant, or row-level security. +7. Data classification level, and the retention and privacy rules for raw uploads and audit records. +8. Performance targets and deployment environment. +9. **Frontend PRs:** close or keep PR #40 in this repo (the plain-HTML frontend at `/app`, which is outside the React/TS constraint) and PR #1 in `csv-table-hub`. + +Do not ask about anything listed under "Decisions already taken". + +Separate essential requirements from optional enhancements and future work. Do not expand scope beyond this document without explaining the benefit and cost, and getting approval. + +--- + +## T — Technical depth: detail, verification and evidence + +Work at the technical depth needed to implement, test and maintain a production-quality solution. + +### Architecture and design depth + +Document: + +- system architecture and component responsibilities +- database entities, relationships, indexes and constraints +- API endpoints, request and response schemas, and error contracts +- CSV parsing, normalisation and type inference rules +- file-level and row-level duplicate detection algorithms +- safe filename-to-table-name mapping +- staging, approval, transaction and retry strategies +- audit event structure, immutability controls and reporting +- authentication, authorisation and tenant isolation +- logging, monitoring, performance and recovery mechanisms + +The loaders currently own their own transactions, and the API shells out to them. For D4, present the options with trade-offs for product-owner approval. Do not implement before approval. Options include: + +- moving the load into Python using psycopg `COPY` inside an API-owned transaction +- having the loaders write only to staging tables, with the API promoting the data transactionally +- other options you identify + +Explain significant design decisions and their alternatives, especially where they affect data integrity, scalability, tenant isolation or auditability. + +### Data integrity and audit depth + +- Every upload has a unique identifier and keeps its tenant, original filename, content hash, uploader, timestamp and processing status. +- Original source data is held separately from proposed edits and from approved import data. +- Each material change records: + - upload and import identifiers + - row and column references + - original and new values, where appropriate + - actor and timestamp + - reason and outcome +- Every imported row can be traced back to its source row and its approved version. +- Transactions, explicit failure handling and idempotency controls prevent partial or repeated imports from silently corrupting the target dataset. + +### Testing and verification depth + +Create automated unit, integration, API, database and end-to-end tests. Cover ordinary workflows and these edge cases: + +- identical files uploaded repeatedly +- same filename with different contents +- duplicate records within and across uploads +- invalid CSV structures and encoding problems +- conflicting columns and incompatible data types +- edited and excluded rows +- approval rejected, then resubmitted +- failed imports, retries and concurrent requests +- unauthorised access and cross-tenant access attempts +- attempts to update or delete audit rows +- audit completeness and export correctness, including formula-injection payloads +- large-file processing and resource limits + +Follow the repository's no-skip policy: a missing prerequisite fails with remediation text rather than skipping. + +### Evidence rules + +- A requirement is "Verified" only when a test run proves it. Quote the actual command and the actual pass, fail and skip counts. +- Documents such as `VCRM.md`, `FIXES_APPLIED.md` and `API_INTEGRATION.md` are claims, not evidence. `API_INTEGRATION.md` describes a superseded API. +- If something could not be run (for example, no database available), say so plainly. +- When searching the repository, exclude the stale worktree `.claude/worktrees/velvety-chasing-badger/`. It holds an old copy of `api/`. + +### Implementation discipline + +Deliver in small, complete, verifiable milestones. For each milestone: + +1. State the objective and the BR IDs covered. +2. Explain the design and the expected changes. +3. Implement the change on a branch and open a PR. Never commit directly to `main`. +4. Run the relevant tests and inspect the actual results. +5. Update the documentation and the VCRM rows. +6. Identify unresolved defects and remaining work. + +Do not claim that unimplemented or untested functionality is complete. + +Every command given to the product owner states where to run it. Use a PowerShell `cd` with a quoted Windows path in the same block. + +--- + +## Milestone 0: assess and plan (no production code changes) + +1. **Baseline.** Run the existing test suite and lint. Record the commands and the actual counts. +2. **Assess the live code** against BR-23 to BR-37. For each item, record: + - status: Exists, Partial or Missing + - file and line evidence + - the test that proves it, if any +3. **Draft VCRM rows** for the new requirements, mapped to existing BRs where they overlap. All start as "Not started" or "Partial", with the reason. +4. **Draft ADRs** for product-owner approval: + - **ADR-0002, transactional staged import (D4).** This supersedes the post-commit audit design and presents the loader options. + - **ADR-0003, customer-facing multi-tenant scope (D1–D3).** This records the positioning change from an internal T&E migration tool and its consequences. +5. **Propose the design** and a milestone plan, ordered so that each milestone is independently verifiable. +6. **Ask** only the unresolved decisions listed under Range, each with a recommended default and its trade-off. + +Then **stop and wait for approval** before writing production code. + +--- + +## ART decision rule + +- Use **Altitude** to judge whether the solution achieves the intended business outcome. +- Use **Range** to judge whether a proposed feature or design is within the agreed scope and constraints. +- Use **Technical depth** to judge whether the implementation is detailed, secure, testable and evidenced enough. + +When these dimensions conflict, explain the trade-off and recommend the simplest solution that meets the agreed requirements. Get approval before materially changing scope. From b9035ab72ec6694d46331574cfb1ac4526234e05 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 26 Oct 2026 15:37:00 +1100 Subject: [PATCH 2/3] docs(art): Milestone 0 assessment, VCRM draft, ADR-0002 and ADR-0003 (#53) Milestone 0 outputs for product-owner approval. No production code changed. - Baseline with actual counts (local + CI run 40245755345) - BR-23..BR-37 assessed against the live code with file:line evidence - Draft VCRM rows (not yet applied to docs/VCRM.md) - ADR-0002 (transactional staged import) and ADR-0003 (customer-facing multi-tenant scope), both Proposed - Design, milestone plan M1-M7, and the nine open decisions with defaults - Defects found: #81, #82, #83, #84 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019YPoXzBSj89ZHgw7nfTmbB --- docs/adr/0002-transactional-staged-import.md | 71 +++++++ docs/adr/0003-customer-facing-multi-tenant.md | 63 +++++++ docs/art/milestone-0-assessment.md | 173 ++++++++++++++++++ 3 files changed, 307 insertions(+) create mode 100644 docs/adr/0002-transactional-staged-import.md create mode 100644 docs/adr/0003-customer-facing-multi-tenant.md create mode 100644 docs/art/milestone-0-assessment.md diff --git a/docs/adr/0002-transactional-staged-import.md b/docs/adr/0002-transactional-staged-import.md new file mode 100644 index 0000000..380d75e --- /dev/null +++ b/docs/adr/0002-transactional-staged-import.md @@ -0,0 +1,71 @@ +# ADR-0002 — Transactional, staged, approved import + +- **Status:** Proposed (awaiting product-owner approval) +- **Date:** 2026-10-26 +- **Decision maker:** Amarnath Rasineni +- **Affects:** BR-24, BR-32, BR-33, BR-34, BR-35; `api/services/dynamic_loader.py`, `api/services/te_loader.py`, `api/services/audit.py` +- **Supersedes:** the post-commit upload-audit design documented in `api/services/audit.py` (O-04) + +--- + +## Context + +Decision D4 (2026-10-25) requires imports to be staged, approved and transactional, with every imported row traceable to its approved version. + +Today an upload is loaded the moment it arrives: + +- **Dynamic mode** parses, casts and inserts in one psycopg2 transaction, then commits (`dynamic_loader.py:131-356`). +- **T&E mode** inserts with savepoints so failing rows are skipped and the rest commit (`te_loader.py`, commit at line 301). Partial import is by design. +- **Audit row** is written afterwards in a separate transaction (`csv_routes.py:86-93`). A crash between the two leaves an import with no audit row. A failed audit write is logged, never raised (`audit.py:9-22`). + +There is no staging, no approval step and no link from an imported row back to its source row. + +**Correction to the ART prompt.** The prompt assumes the API shells out to the bash loaders. It does not: `api/` contains no `subprocess` calls. The bash loaders in `build/csv/` serve only the batch pipeline and e2e tests. So the choice is about restructuring Python code the API already owns, not about wrapping shell scripts. + +## Options + +### A. API-owned transaction: COPY to staging, then promote (recommended) + +1. **On upload:** store the original bytes and parsed source rows (insert-only tables). No target-table writes. +2. **During review:** record decisions and edits as rows, never updates. +3. **On approved import:** one transaction does all of the following, then commits: + - `COPY` the effective rows (source, minus exclusions, plus the latest edits) into a staging table using psycopg2 `copy_expert`. + - Validate types and keys in SQL. + - `INSERT ... SELECT` into the versioned target table with `_source_row_id` and `_import_id`. + - Write the `imports` record with reconciliation counts. + - Write the audit events. +4. **Idempotency:** a client-supplied `Idempotency-Key`, with a UNIQUE constraint on `imports(upload_id, approval_id)`. A retry after a failure re-runs cleanly; a retry after success returns the original result. + +- **For:** one atomic unit, so data, lineage and audit commit together or not at all. COPY is the fastest load path in PostgreSQL. It reuses `csv_parse.py` and the current identifier safety. There is nothing new to deploy. +- **Against:** the largest change to the loaders. Long imports hold one transaction open, which needs a statement timeout and a row cap (the current 100,000-row cap stays). + +### B. Thread the connection through the existing loaders + +Pass the caller's cursor into `upload_dynamic` and `upload_te`, so the audit INSERT joins the load transaction. + +- **For:** small, contained change. It closes the post-commit audit window. +- **Against:** it covers none of staging, approval, lineage or edits. It would only be a step on the way to A. + +### C. Bash loaders write to staging, API promotes + +- **Against:** it adds a shell-out the API does not have today, splits one transaction across two processes, and puts untrusted CSV through bash. Rejected. + +### D. Background import worker with a job table + +The import runs in a worker process, with the job state in PostgreSQL. Promotion is transactional as in A. + +- **For:** handles large files and concurrency without tying up API requests. +- **Against:** a new process to deploy and monitor, which counts against the air-gapped simplicity goal. Not needed at the current 100,000-row cap. + +## Decision (proposed) + +**Adopt A.** Keep D as the documented upgrade path if decision 1 (file size and concurrency) later rises beyond what one request transaction can handle. + +T&E mode moves to the same flow. Its row-level error isolation becomes a review-time result (rows flagged and excluded before approval) instead of a load-time skip. That honours decision 4's recommended default of no partial imports. + +## Consequences + +- `audit.py`'s "known limitation" goes away. `record_audit` (separate transaction) is retired in favour of in-transaction writes. +- Loader signatures change. The existing `/api/csv/upload` stays as a compatibility path until the frontend moves to the review flow, and is then removed in a later milestone. +- Needs Alembic migrations first (assessment finding F4). +- Tests must include failure injection mid-import (no partial target rows, no audit row), a retry after failure, and a retry after success (no duplicate rows). diff --git a/docs/adr/0003-customer-facing-multi-tenant.md b/docs/adr/0003-customer-facing-multi-tenant.md new file mode 100644 index 0000000..27e2e6b --- /dev/null +++ b/docs/adr/0003-customer-facing-multi-tenant.md @@ -0,0 +1,63 @@ +# ADR-0003 — Customer-facing, multi-tenant scope + +- **Status:** Proposed (awaiting product-owner approval) +- **Date:** 2026-10-26 +- **Decision maker:** Amarnath Rasineni +- **Affects:** BR-23 to BR-37 (new); `api/auth.py`, `api/config.py`, `api/db.py`; the frontend; deployment docs +- **Records:** decisions D1 to D3 taken on 2026-10-25 (see `docs/prompts/ART_FRAMEWORK.md`) + +--- + +## Context + +The platform was built as an **internal T&E migration tool for one operator**: + +- One shared `X-API-Key` (`api/auth.py:21-26`), optional in `dev`. +- One uploads schema for everything (`settings.UPLOADS_SCHEMA`). +- No users, roles or tenants. The audit log records actions without an actor. +- Loads happen immediately on upload. +- The frontend's audit view reads browser `localStorage`. + +On 2026-10-25 the product owner decided: + +- **D1:** the platform becomes customer-facing with multi-tenant isolation. +- **D2:** full-dataset cell editing is in scope, stored as versions. +- **D3:** an approval workflow is in scope. + +This ADR records that positioning change and what follows from it. + +## Decision + +The platform is repositioned from a single-operator internal tool to a **multi-tenant web application used by external customers**. It keeps its two import modes (freeform and strict T&E schema) and its T&E/V&V framing. + +## Consequences + +**Identity and access (BR-36)** +- Per-user authentication replaces the shared key for interactive use. The key gate can remain for service-to-service calls. +- Roles at minimum: *uploader*, *reviewer/approver*, *tenant admin*, *auditor (read-only)*. +- The sovereignty constraint (D6) rules out SaaS identity providers at runtime. The proposed approach is OIDC against an IdP the deployment controls, plus a local-account fallback for air-gapped installs. **Needs your confirmation.** + +**Tenant isolation (BR-36)** +- Proposed model: schema per tenant (Milestone 0, decision 6). Dynamic tables already live in a configurable schema, so this extends an existing pattern. +- Every request resolves to exactly one tenant, and the database session is scoped to it. +- Cross-tenant read and write attempts are standing negative tests from Milestone 2 onwards. + +**Data handling (BR-24, BR-37)** +- Customer data must be preserved: the original file and source rows are kept, edits are versioned, and nothing is overwritten. The current `overwrite` path, which drops the prior table, is retired. +- Classification and retention rules become per-deployment configuration (decision 7, #49). + +**Workflow (BR-26, BR-27, BR-32)** +- Upload no longer means import. Data passes review and approval first (ADR-0002). +- Separation of duties defaults to "the uploader cannot approve" (decision 5). + +**Audit (BR-35)** +- Every event carries tenant and actor. The trail becomes append-only at database level, and reports come from the server, not the browser. + +**What does not change** +- ADR-0001 (PostgreSQL first) and the deferred BR-02, BR-21 and BR-22 stay as they are. +- The README's "production-grade" framing stays (risk accepted, 2026-10-04). +- The React/TypeScript frontend (`csv-table-hub-main/`) remains the baseline. The plain-HTML frontend (PR #40) is outside this scope (decision 9, #50). + +**Risk** +- Customer-facing use raises the cost of a defect. Assessment findings F1 to F4 should be fixed in Milestone 1 before any customer data arrives. +- A security review of the threat model is recommended before the first external tenant. diff --git a/docs/art/milestone-0-assessment.md b/docs/art/milestone-0-assessment.md new file mode 100644 index 0000000..06613d5 --- /dev/null +++ b/docs/art/milestone-0-assessment.md @@ -0,0 +1,173 @@ +# ART Milestone 0 — Assessment and Plan + +**Branch:** `art-milestone-0` · **Baseline commit:** `main` @ 032b522 · **Date:** 2026-10-26 +**Prompt:** [`docs/prompts/ART_FRAMEWORK.md`](../prompts/ART_FRAMEWORK.md) (v2) · **Issue:** #53 +**Status:** For product-owner approval. No production code has been changed. + +This document covers Milestone 0 steps 1, 2, 3, 5 and 6. Step 4 is the two proposed ADRs: +[ADR-0002](../adr/0002-transactional-staged-import.md) and [ADR-0003](../adr/0003-customer-facing-multi-tenant.md). + +--- + +## 1. Baseline + +All counts below are from actual runs. Nothing has been inferred from documentation. + +| Check | Command | Result | Where run | +|---|---|---|---| +| Python lint | `bash scripts/lint.sh` (flake8 + bandit) | Clean (flake8 clean, bandit clean) | Local, Python 3.13 | +| Python unit tests | `pytest -m "not integration and not e2e and not parity"` | 119 passed, 0 failed, 0 skipped, 120 deselected | Local (no PostgreSQL) | +| Full Python suite | `python3 scripts/test_report.py --strict` | 239 collected, 239 passed, 0 failed, 0 skipped | CI run 40245755345 on 032b522, PostgreSQL 16 | +| Evals | `evals/runner.py --tiers p,i,s,e` | 26 total, 26 passed, 0 failed, 0 skipped | CI run 40245755345 | +| Evals, offline tier | `evals/runner.py --tiers p` | 23 passed, 0 failed, 0 skipped | Local | +| Frontend tests | `npx vitest run` | 8 files, 102 passed | Local, Node 22 | +| Frontend types | `npx tsc --noEmit` | Exit 0 | Local | +| Frontend lint | `npx eslint .` | **524 errors, 2 warnings.** 517 are `prettier/prettier` formatting; 7 are code rules (4 `no-explicit-any`, 2 `no-unused-expressions`, 1 `no-unused-vars`) | Local | + +**Reading the baseline:** + +- The integration-dependent counts come from CI. No PostgreSQL was available in this assessment environment. +- Frontend lint is not gated in CI and has never been clean. It is pre-existing debt, not an ART regression. Recommendation: clear it once with `prettier --write` in its own PR before ART frontend work starts, so later diffs stay reviewable. + +--- + +## 2. Live code against BR-23 to BR-37 + +**Correction to the prompt's premise.** The prompt says the API shells out to the bash loaders. It does not. Neither `api/` nor anything it imports uses `subprocess`. Uploads load in-process through `api/services/dynamic_loader.py` and `api/services/te_loader.py`, each inside its own psycopg2 transaction. The `build/csv/loader_*.sh` scripts are used only by the batch pipeline (`build/csv_loader.sh`) and the e2e tests. ADR-0002 reflects this. + +**ID check.** `docs/VCRM.md` runs BR-01 to BR-22. None of the existing BRs covers SHA-256 deduplication, the upload registry or the audit log; those were tracked as gaps (O-02, O-04), not requirements. So the provisional IDs BR-23 to BR-37 stand. Overlaps with existing BRs are noted in step 3. + +| BR | Requirement (short) | Status | Evidence (file:line) | Proving test today | +|---|---|---|---|---| +| BR-23 | Upload registry with ID, filename, SHA-256, uploader, tenant, timestamp, status | **Partial** | `csv_files` has id, file_name, file_hash, table_name, mode, row_count, column_names, created_at (`api/db.py:199-208`). No uploader, tenant or processing status. | `test_api_coverage.py` files-list tests (integration) | +| BR-24 | Original file and parsed source preserved immutably | **Missing** | The original file is never stored. Only cast, typed rows land (`dynamic_loader.py:300-319`). Rows that fail to cast are dropped from the data, with at most 200 reported (`dynamic_loader.py:268-277`). Overwrite runs `DROP TABLE` on the prior import (`dynamic_loader.py:159-168`, `194-203`). | None | +| BR-25 | Interactive preview of all rows and columns, with inference and validation | **Partial** | The preview returns 10 sample rows and infers types from the first 200 (`csv_parse.py:237`, `254-255`). Viewing an imported table is capped at 200 rows (`csv_routes.py:125`). No full-dataset paging. | `preview-contract.test.ts`, `test_api_coverage.py` preview tests | +| BR-26 | Row include/exclude with actor and reason | **Missing** | No code path. | None | +| BR-27 | Cell edits as versions over the original | **Missing** | No code path. | None | +| BR-28 | SHA-256 exact-file duplicate detection, within and across uploads, per tenant | **Partial** | Hash computed (`dynamic_loader.py:58`); checked against the registry (`171-191`). It is global, not per tenant. **Defect F1:** the race guard relies on "the registry's UNIQUE indexes" (`dynamic_loader.py:321-336`), but no unique index exists on `csv_files` anywhere in the repo (`db.py:199-208`; no Alembic migrations exist). | `test_overwrite_false_refuses_duplicate` (integration) | +| BR-29 | Configurable duplicate-record detection within and across uploads | **Partial** | Exact whole-row hash within one file (`dynamic_loader.py:281-289`, `ON CONFLICT (_row_hash)` at `312`). Each file gets its own table, so there is no cross-upload detection, and the rule is not configurable. | `test_in_file_duplicate_rows_are_skipped`, `test_row_hash_*` (integration) | +| BR-30 | Safe filename-derived table names; repeated same-filename uploads supported | **Partial** | Names are safe but content-derived, `csv_` (`dynamic_loader.py:236`), not filename-derived. A repeat of the same filename is refused, or replaced destructively with `overwrite` (`144-168`). | `test_injection_shaped_names_are_not_served` (integration) | +| BR-31 | Schema compatibility checks; no destructive schema changes | **Partial** | T&E mode checks headers against the fixed schema (`te_loader.py:45-82`). Dynamic mode always creates a new table, so there is nothing to check against. Overwrite drops tables (see BR-24). | `test_te_loader_integration.py` | +| BR-32 | Approval workflow with recorded approver and decision | **Missing** | No code path. | None | +| BR-33 | Staged, transactional, idempotent import; rows traceable to source and approved version | **Partial** | A dynamic upload is one transaction (`Conn()` at `dynamic_loader.py:131`, commit at `356`). T&E mode isolates failing rows with savepoints and commits the rest, so partial import is by design (`te_loader.py`, commit at `301`). The audit row is written post-commit in a separate transaction (`csv_routes.py:86-93`; `audit.py:9-22`). No staging, no approval link, no source-row lineage. | Upload integration tests | +| BR-34 | Reconciliation: rows in, excluded, edited, imported account for each other | **Partial** | The response returns totalRows, insertedRows, duplicateRowsSkipped and failedRows (`dynamic_loader.py:359-372`), but they are not persisted; the registry keeps `row_count` only. **Defect F2:** cells beyond the header width are silently ignored in both modes (`dynamic_loader.py:263-264`, `te_loader.py:221`), so they appear in no count. | `test_issue_05_import_summary.py` | +| BR-35 | DB-enforced append-only audit; filterable reports; formula-safe exports | **Partial** | `audit_log` is an ordinary table (`db.py:215-223`). Only `upload` and `delete` are recorded (`audit.py:38`), with no actor. `GET /api/audit/log` pages but does not filter (`audit_routes.py:27-29`). The frontend audit page reads browser `localStorage`, not the server log. **Defect F3:** CSV export escapes quotes and commas but not leading `= + - @` (`csv-table-hub-main/src/lib/export.ts:43-46`). | `test_audit_*` (integration), `test_delete_appears_in_audit_log` | +| BR-36 | Authentication, role-based authorisation, tenant isolation | **Missing** (key gate only) | One shared `X-API-Key` compared in constant time (`api/auth.py:21-26`). The API refuses to start outside `dev` without it (`api/main.py`). No users, roles or tenants. | `test_csv_endpoint_still_requires_api_key_when_set` (integration) | +| BR-37 | Classification and sovereignty: no runtime external services; offline install | **Partial** | No outbound calls found in `api/`, and no CDN, analytics or telemetry references in `csv-table-hub-main/src` (grep, 2026-10-26). There is no automated check for this. The offline install doc does not exist (#60). Classification level is undecided (#49). | None | + +**Defects found during the assessment.** These are real today, independent of ART, and each now has a GitHub issue: + +- **F1 (#81). Registry race guard missing.** Concurrent identical uploads can both register, because the unique indexes the code relies on were never created. +- **F2 (#82). Extra cells silently dropped.** In both import modes, a data row wider than the header loses its extra values without being reported. This contradicts BR-10 and the ART data-preservation constraint. +- **F3 (#83). Spreadsheet formula injection in CSV export.** Error and audit report exports write user-supplied cells starting with `=`, `+`, `-` or `@` unchanged. +- **F4 (#84). No schema migration framework.** `alembic.ini` does not exist, so `bootstrap()` always takes the fallback path (`db.py:165-172`). Every ART schema change needs versioned migrations first. + +--- + +## 3. Draft VCRM rows + +Proposed additions to `docs/VCRM.md` §1 and §2. They are not applied yet, because requirement wording is the product owner's call. Each will be written to the VCRM when its milestone starts (D5). + +| ID | Requirement | Overlaps | Initial status | Reason | Verification (planned) | Milestone | +|---|---|---|---|---|---|---| +| BR-23 | Every upload is registered with a unique ID, original filename, SHA-256, uploader, tenant, timestamp and processing status. | — | ⚠️ Partial | Registry exists without uploader, tenant, status | Integration + DB constraint tests | M1, M3 | +| BR-24 | The original file and its parsed source rows are preserved immutably, separate from edits and approved data. | — | ❌ Not started | Original never stored; overwrite drops data | DB negative tests (update/delete refused) | M3 | +| BR-25 | Users can page through every row and column of an upload, with inferred types and validation results. | BR-10 | ⚠️ Partial | 10-row sample, 200-row cap | API paging tests + e2e | M4 | +| BR-26 | Rows can be included or excluded; each decision records actor and reason. | BR-11 | ❌ Not started | No code | API + e2e | M4 | +| BR-27 | Cell edits are stored as versions over the original, with old and new value, actor, time and reason. | — | ❌ Not started | No code | API + DB tests | M4 | +| BR-28 | Exact-file duplicates are detected by SHA-256 within and across uploads, per tenant. | — | ⚠️ Partial | Global not per tenant; race guard missing (F1) | Concurrency + cross-tenant tests | M1, M5 | +| BR-29 | Duplicate records are detected within and across uploads using a configurable rule. | — | ⚠️ Partial | In-file exact match only | Unit + integration | M5 | +| BR-30 | Table names are safely derived from filenames, and repeated uploads of a filename are supported without overwriting. | — | ⚠️ Partial | Content-derived names; repeats refused or destructive | Unit + integration | M5 | +| BR-31 | Imports check schema compatibility and never make destructive schema changes. | — | ⚠️ Partial | T&E mode only; overwrite drops tables | Integration | M5 | +| BR-32 | Imports require submission and an approve or reject decision with a recorded approver. | — | ❌ Not started | No code | API + e2e | M6 | +| BR-33 | Imports are staged, transactional and idempotent with safe retries; every imported row traces to its source row and approved version. | BR-09 (idempotency, deploy level) | ⚠️ Partial | Single-transaction load, no staging or lineage | Integration incl. failure injection | M6 | +| BR-34 | Rows in, excluded, edited and imported reconcile exactly and are persisted per import. | — | ⚠️ Partial | Counts returned, not persisted; F2 | Integration | M1, M6 | +| BR-35 | Audit events are append-only at database level; reports are filterable; exports neutralise formula injection. | — | ⚠️ Partial | Plain table, no actor, no filters, F3 | DB negative tests + unit | M1, M3, M7 | +| BR-36 | Users authenticate; access is role-based; no tenant can read or affect another tenant's data. | — | ❌ Not started | Shared key only | Cross-tenant negative tests | M2 | +| BR-37 | Data stays inside the deployment boundary; no runtime external services; the stack installs offline. | — | ⚠️ Partial | No external calls found; no automated check; no offline doc | Egress-blocked test run + offline dry run | M7 | + +**Success measures** (Altitude) map onto these rows: preview completeness → BR-25; duplicate accuracy → BR-28, BR-29; reconciliation → BR-34; audit coverage → BR-35; data integrity → BR-24, BR-33; tenant isolation → BR-36; security → BR-35, BR-36; sovereignty → BR-37. Performance targets wait on decision 8. **BR-22 (≥ 1 M rows) stays deferred by decision**; the ART targets apply to the capped path. + +--- + +## 5. Proposed design + +### Architecture + +The current shape stays: React/TS frontend → FastAPI → PostgreSQL. Three additions: + +- **Identity and tenancy layer.** Authenticated users carry a tenant and a role. Every request is scoped to one tenant's schema. +- **Dataset lifecycle.** An upload becomes a dataset version that moves through `uploaded → in_review → submitted → approved | rejected → imported | failed`. +- **Audit store.** Append-only, written in the same transaction as the change it records. + +### Data model (per tenant schema, plus a shared `control` schema) + +- `control.tenants`, `control.users`, `control.memberships(user, tenant, role)` +- `uploads`: registry. Columns: id, tenant, original_filename, sha256, uploader, created_at, status, byte_size, row_count. UNIQUE (sha256). +- `upload_blobs`: original file bytes. Insert only. +- `source_rows`: parsed rows as text, with row_number. Insert only. This is the immutable original. +- `row_decisions`: include/exclude, actor, reason, at. +- `cell_edits`: row, column, old, new, actor, reason, at. Versioned and never updated. +- `approvals`: submission, decision, approver, reason, at. +- `imports`: one row per approved import. Columns: idempotency key, counts (in, excluded, edited, imported), status, target table, lineage pointer. +- `audit_events`: tenant, actor, action, object references, before and after, at, prev_hash. +- Target tables: `_v`, with `_source_row_id` and `_import_id` columns for lineage. + +### API (additions) + +- `POST /api/uploads`: register and store the original. +- `GET /api/uploads/{id}/rows?page=`: full paging with validation flags. +- `PUT /api/uploads/{id}/rows/{n}/decision`, `PUT /api/uploads/{id}/cells`: review actions. +- `POST /api/uploads/{id}/submit`, `POST /api/uploads/{id}/approval`: workflow. +- `POST /api/uploads/{id}/import` with an `Idempotency-Key` header. +- `GET /api/audit/events?actor=&action=&from=&to=`: filtered reports. + +Errors keep the existing structured `{status, reason, message}` contract. The existing `/api/csv/*` endpoints stay until the frontend moves over. + +### Key mechanisms + +- **Parsing.** Keep `csv_parse.py`, which is dependency-free and already mirrored in the frontend. Reject ragged rows instead of truncating them (F2). +- **File deduplication.** SHA-256 per tenant, enforced by a UNIQUE constraint rather than check-then-act (F1). +- **Record deduplication.** Default is an exact match on normalised whole-row content. The rule can be configured per dataset as a set of key columns. Matches across earlier imports of the same dataset are found through an indexed `_row_key` hash. +- **Naming.** Lower-case ASCII slug of the filename, truncated, then `_v`, through `sql.Identifier`. Versioned datasets, never overwrite (decision 2). +- **Staging and import.** Per ADR-0002: the API owns one transaction. COPY into staging, apply decisions and edits, insert into the target, write the import record and audit events, then commit. Retries use the idempotency key. +- **Append-only audit.** A dedicated table owned by a migration role. The application role gets INSERT and SELECT only, with no UPDATE or DELETE grant. A trigger rejects UPDATE and DELETE even for the owner, and a `prev_hash` chain detects tampering. Negative tests attempt both. +- **Authentication.** Has to work offline, so no SaaS IdP. Recommended: OIDC against an IdP the deployment controls, such as Keycloak or Entra ID where the deployment allows, with a local-account fallback for air-gapped installs. This is open for your decision under ADR-0003. +- **Tenant isolation.** Schema per tenant (decision 6), using `SET LOCAL search_path` and per-tenant roles inside each request's transaction. Cross-tenant negative tests are part of every milestone from M2 onwards. +- **Operations.** Structured JSON logs with no row content. The existing `/api/health`. Rate limits stay; they are per instance, which is acceptable until scaling out. Recovery depends on the transaction plus the idempotency key, so a failed import leaves nothing behind. + +--- + +## 5b. Milestone plan + +Each milestone is one PR (or a small set), independently testable, and updates its VCRM rows with quoted run counts. + +| # | Milestone | BRs | Verified by | +|---|---|---|---| +| M1 | **Foundations and defect fixes:** Alembic baseline of the current schema (F4); UNIQUE on registry hash (F1); reject ragged rows (F2); neutralise formula injection in exports (F3); persist import counts | BR-23 (part), BR-28 (part), BR-34 (part), BR-35 (part) | Migration up/down test; concurrent-upload test; ragged-row and formula unit tests | +| M2 | **Identity and tenancy:** users, roles, tenants, auth, schema-per-tenant routing | BR-36 | Cross-tenant read and write negative tests; role matrix tests | +| M3 | **Immutable source and append-only audit:** store blobs and source rows; `audit_events` with grants, trigger and hash chain | BR-23, BR-24, BR-35 | UPDATE and DELETE attempts fail; hash-chain verification test | +| M4 | **Review:** full paging, include/exclude, versioned cell edits (API and UI) | BR-25, BR-26, BR-27 | API tests + e2e | +| M5 | **Deduplication and naming:** per-tenant file dedup, configurable record dedup across uploads, versioned dataset naming, schema compatibility | BR-28, BR-29, BR-30, BR-31 | Repeated-file, same-name-different-content and cross-upload duplicate tests | +| M6 | **Approval and transactional import** (ADR-0002): submit/approve/reject, staged COPY and promote, idempotency, lineage, reconciliation | BR-32, BR-33, BR-34 | Rejected-then-resubmitted, failure injection mid-import, retry and concurrent import tests | +| M7 | **Sovereignty and reports:** offline install (#60), egress-blocked test run, classification settings, filtered audit reports, performance targets | BR-35, BR-37 | Offline dry run; test suite with network egress blocked; perf run | + +M7's offline-install part can run in parallel with M1 to meet the 15 Nov deadline. + +--- + +## 6. Decisions needed (with recommended defaults) + +| # | Decision | Recommended default | Trade-off | +|---|---|---|---| +| 1 | Max file size and concurrency | Keep 50 MB and 100,000 rows (current config); 4 concurrent imports per API instance | Larger files need streaming upload and background jobs, a separate milestone | +| 2 | Repeated uploads of the same filename | **Versioned datasets** (`_v`); never overwrite | Users see several versions; an "append to dataset" option can come later | +| 3 | Duplicate-record rule and policy | Exact whole-row match; policy **flag**, with the user deciding per row | Merge needs per-column rules; skip hides data | +| 4 | Partial imports | **Not permitted.** All approved rows import or none do; exclusions are explicit | A file with bad rows needs review before import | +| 5 | Can an uploader approve their own data? | **No by default**; a per-tenant setting allows it for single-user tenants | Slower for small teams | +| 6 | Tenant model | **Schema per tenant** | Migrations run across N schemas; RLS on dynamically created tables would be harder | +| 7 | Classification and retention | Design target **OFFICIAL: Sensitive**; raw uploads kept until a recorded purge; audit kept 7 years by default (configurable, confirm against the client's records authority) | PROTECTED would add IRAP-assessed hosting and ISM PROTECTED controls (#49) | +| 8 | Performance targets and environment | 100,000-row import under 60 s and first preview page under 2 s on a 2 vCPU / 4 GB instance; Azure Container Apps plus an on-prem docker compose | Targets need a perf harness (M7); BR-22 stays deferred | +| 9 | PR #40 and csv-table-hub PR #1 | **Close both:** outside the React/TS constraint | Loses the plain-HTML fallback (#50) | + +**Next step:** approve or amend ADR-0002, ADR-0003 and the nine decisions. M1 starts only after approval. From bd14477c9cf5576a7e034f5a4771615bc4b176fe Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 26 Oct 2026 16:38:41 +1100 Subject: [PATCH 3/3] docs(art): record product-owner approval of Milestone 0 (#53) Approved 2026-10-26: ADR-0002 and ADR-0003 (now Accepted), the M1-M7 plan, and all nine recommended defaults (DECISIONS/2026-10-26-art-milestone-0.md). BR-23..BR-37 added to docs/VCRM.md with their assessed status (D5). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019YPoXzBSj89ZHgw7nfTmbB --- DECISIONS/2026-10-26-art-milestone-0.md | 30 +++++++++++ docs/VCRM.md | 51 ++++++++++++++++++- docs/adr/0002-transactional-staged-import.md | 4 +- docs/adr/0003-customer-facing-multi-tenant.md | 6 +-- docs/art/milestone-0-assessment.md | 4 +- 5 files changed, 86 insertions(+), 9 deletions(-) create mode 100644 DECISIONS/2026-10-26-art-milestone-0.md diff --git a/DECISIONS/2026-10-26-art-milestone-0.md b/DECISIONS/2026-10-26-art-milestone-0.md new file mode 100644 index 0000000..005aec5 --- /dev/null +++ b/DECISIONS/2026-10-26-art-milestone-0.md @@ -0,0 +1,30 @@ +# Decision: ART Milestone 0 approval + +Status: Decided +Date: 2026-10-26 +Decided by: Amarnath Rasineni (product owner) +Source: `docs/art/milestone-0-assessment.md` §6, PR #85, issue #53 + +## Approved + +- **ADR-0002**, transactional staged import: option A (an API-owned transaction with COPY to staging, then promote). Option D (background worker) is the documented upgrade path. +- **ADR-0003**, customer-facing multi-tenant scope, including authentication by OIDC against a deployment-controlled IdP, with a local-account fallback for air-gapped installs. +- **The M1–M7 milestone plan.** + +## The nine open decisions, all taken at the recommended default + +| # | Decision | Outcome | +|---|---|---| +| 1 | Max file size and concurrency | 50 MB and 100,000 rows; 4 concurrent imports per API instance | +| 2 | Repeated uploads of the same filename | Versioned datasets (`_v`); never overwrite | +| 3 | Duplicate-record rule and policy | Exact whole-row match; policy **flag**, the user decides per row | +| 4 | Partial imports | Not permitted. All approved rows import or none do | +| 5 | Uploader approving own data | Not allowed by default; per-tenant setting for single-user tenants | +| 6 | Tenant model | Schema per tenant | +| 7 | Classification and retention | Design target OFFICIAL: Sensitive. Raw uploads kept until a recorded purge. Audit kept 7 years by default (configurable; confirm against the client's records authority) | +| 8 | Performance targets and environment | 100,000-row import under 60 s; first preview page under 2 s on 2 vCPU / 4 GB. Azure Container Apps plus on-prem docker compose. BR-22 stays deferred | +| 9 | PR #40 and csv-table-hub PR #1 | Close both (outside the React/TS constraint) | + +## Unchanged + +ADR-0001, the "production-grade" README framing, and the deferral of BR-02, BR-21 and BR-22. diff --git a/docs/VCRM.md b/docs/VCRM.md index 66e8c38..76d0264 100644 --- a/docs/VCRM.md +++ b/docs/VCRM.md @@ -63,6 +63,28 @@ Test-layer codes used in column headers below: | **BR-19** | The build, test, and eval layers shall be **physically segregated** so a change to one cannot inadvertently break the others' contract. | `docs/ARCHITECTURE.md` | | **BR-20** | The full SQL test suite shall reach **142 of 142 assertions passing** (100.0 % pass rate) on every release. | README §"142 assertions", `tests/run_all_tests.sql` | +### ART extension requirements (added 2026-10-26) + +Added by ART Milestone 0 (#53) and approved by the product owner on 2026-10-26. Source: [`docs/art/milestone-0-assessment.md`](art/milestone-0-assessment.md), [ADR-0002](adr/0002-transactional-staged-import.md), [ADR-0003](adr/0003-customer-facing-multi-tenant.md), [`DECISIONS/2026-10-26-art-milestone-0.md`](../DECISIONS/2026-10-26-art-milestone-0.md). + +| ID | Requirement | Source | +|----|------------|--------| +| **BR-23** | Every upload shall be registered with a unique ID, original filename, SHA-256, uploader, tenant, timestamp and processing status. | ART v2 | +| **BR-24** | The original file and its parsed source rows shall be preserved immutably, separate from edits and approved data. | ART v2, D2 | +| **BR-25** | Users shall be able to page through every row and column of an upload, with inferred types and validation results. | ART v2 | +| **BR-26** | Rows shall be includable or excludable; each decision records actor and reason. | ART v2 | +| **BR-27** | Cell edits shall be stored as versions over the original, with old and new value, actor, time and reason. | ART v2, D2 | +| **BR-28** | Exact-file duplicates shall be detected by SHA-256 within and across uploads, per tenant. | ART v2 | +| **BR-29** | Duplicate records shall be detected within and across uploads using a configurable rule (default: exact match, flagged). | ART v2, decision 3 | +| **BR-30** | Table names shall be safely derived from filenames, and repeated uploads of a filename create versioned datasets without overwriting. | ART v2, decision 2 | +| **BR-31** | Imports shall check schema compatibility and never make destructive schema changes. | ART v2 | +| **BR-32** | Imports shall require submission and an approve or reject decision with a recorded approver; by default an uploader cannot approve their own data. | ART v2, D3, decision 5 | +| **BR-33** | Imports shall be staged, transactional and idempotent with safe retries; every imported row traces to its source row and approved version; partial imports are not permitted. | ART v2, D4, ADR-0002, decision 4 | +| **BR-34** | Rows in, excluded, edited and imported shall reconcile exactly and be persisted per import. | ART v2 | +| **BR-35** | Audit events shall be append-only at database level; reports shall be filterable; exports shall neutralise spreadsheet formula injection. | ART v2, D7 | +| **BR-36** | Users shall authenticate; access shall be role-based; no tenant shall read or affect another tenant's data. | ART v2, D1, ADR-0003, decision 6 | +| **BR-37** | Data shall stay inside the deployment boundary, with no runtime external services, and the stack shall install offline. | ART v2, D6, decision 7 | + ### Out of scope (recorded for completeness) | ID | Requirement | Status / why | @@ -113,6 +135,28 @@ The `input_data/` loader has its own implicit requirements — verified end-to-e | BR-D4 | Duplicate primary keys in source are reported, not silently dropped | 4 (duplicate detection) | ✅ | | BR-D5 | Loaded data is browsable (sample peek) | 1 (sample rows) | ✅ | +### ART extension traceability (BR-23 to BR-37) + +Status as assessed on 2026-10-26 at `main` 032b522. None of these is "Verified" until a test run proves it (D5). + +| ID | Status | Reason | Planned verification | Milestone | +|----|--------|--------|----------------------|-----------| +| BR-23 | ⚠️ Partial | Registry exists without uploader, tenant or status | Integration + DB constraint tests | M1, M3 | +| BR-24 | ❌ Not started | Original never stored; overwrite drops data | DB negative tests | M3 | +| BR-25 | ⚠️ Partial | 10-row preview sample; 200-row cap on table view | API paging + e2e | M4 | +| BR-26 | ❌ Not started | No code | API + e2e | M4 | +| BR-27 | ❌ Not started | No code | API + DB tests | M4 | +| BR-28 | ⚠️ Partial | Global, not per tenant; race guard missing (#81) | Concurrency + cross-tenant tests | M1, M5 | +| BR-29 | ⚠️ Partial | In-file exact match only | Unit + integration | M5 | +| BR-30 | ⚠️ Partial | Content-derived names; repeats refused or destructive | Unit + integration | M5 | +| BR-31 | ⚠️ Partial | T&E mode only; overwrite drops tables | Integration | M5 | +| BR-32 | ❌ Not started | No code | API + e2e | M6 | +| BR-33 | ⚠️ Partial | Single-transaction load; no staging or lineage | Integration incl. failure injection | M6 | +| BR-34 | ⚠️ Partial | Counts returned, not persisted; extra cells dropped (#82) | Integration | M1, M6 | +| BR-35 | ⚠️ Partial | Plain audit table, no actor, no filters; export formula injection (#83) | DB negative tests + unit | M1, M3, M7 | +| BR-36 | ❌ Not started | Shared API key only | Cross-tenant negative tests | M2 | +| BR-37 | ⚠️ Partial | No external calls found; no automated check; no offline doc (#60) | Egress-blocked run + offline dry run | M7 | + --- ## 3. Coverage summary @@ -133,10 +177,13 @@ The `input_data/` loader has its own implicit requirements — verified end-to-e | Status | Count | Requirements | |--------|-------|-------------| | ✅ Verified | 19 | BR-01, BR-03, BR-04, BR-05, BR-06, BR-07, BR-08, BR-09, BR-10, BR-11, BR-12, BR-13, BR-14, BR-15, BR-16, BR-17, BR-18, BR-19, BR-20 | -| ⚠️ Partial | 0 | — | +| ⚠️ Partial | 10 | BR-23, BR-25, BR-28, BR-29, BR-30, BR-31, BR-33, BR-34, BR-35, BR-37 (ART, added 2026-10-26) | +| ❌ Not started | 5 | BR-24, BR-26, BR-27, BR-32, BR-36 (ART, added 2026-10-26) | | ⏸ Deferred by decision | 3 | BR-02, BR-21, BR-22 | -**Headline:** **19 of 22 (86 %)** business requirements are fully verified by at least one automated test condition, up from 17 of 22 (77 %). +**Headline (2026-10-26):** with the 15 ART requirements added, **19 of 37 (51 %)** are verified, 3 are deferred by decision, and the other 15 are partial or not started. + +**Headline before ART:** **19 of 22 (86 %)** business requirements are fully verified by at least one automated test condition, up from 17 of 22 (77 %). **There are no longer any unverified requirements that are not a recorded decision.** The three remaining are deferred deliberately: BR-02 by ADR-0001 (PostgreSQL first), BR-21 and BR-22 by earlier project decisions. That distinction matters in review — an unverified requirement is a gap, a deferred one is a choice with a revisit trigger. diff --git a/docs/adr/0002-transactional-staged-import.md b/docs/adr/0002-transactional-staged-import.md index 380d75e..966568f 100644 --- a/docs/adr/0002-transactional-staged-import.md +++ b/docs/adr/0002-transactional-staged-import.md @@ -1,6 +1,6 @@ # ADR-0002 — Transactional, staged, approved import -- **Status:** Proposed (awaiting product-owner approval) +- **Status:** Accepted (product owner, 2026-10-26) - **Date:** 2026-10-26 - **Decision maker:** Amarnath Rasineni - **Affects:** BR-24, BR-32, BR-33, BR-34, BR-35; `api/services/dynamic_loader.py`, `api/services/te_loader.py`, `api/services/audit.py` @@ -57,7 +57,7 @@ The import runs in a worker process, with the job state in PostgreSQL. Promotion - **For:** handles large files and concurrency without tying up API requests. - **Against:** a new process to deploy and monitor, which counts against the air-gapped simplicity goal. Not needed at the current 100,000-row cap. -## Decision (proposed) +## Decision **Adopt A.** Keep D as the documented upgrade path if decision 1 (file size and concurrency) later rises beyond what one request transaction can handle. diff --git a/docs/adr/0003-customer-facing-multi-tenant.md b/docs/adr/0003-customer-facing-multi-tenant.md index 27e2e6b..ed6e474 100644 --- a/docs/adr/0003-customer-facing-multi-tenant.md +++ b/docs/adr/0003-customer-facing-multi-tenant.md @@ -1,6 +1,6 @@ # ADR-0003 — Customer-facing, multi-tenant scope -- **Status:** Proposed (awaiting product-owner approval) +- **Status:** Accepted (product owner, 2026-10-26) - **Date:** 2026-10-26 - **Decision maker:** Amarnath Rasineni - **Affects:** BR-23 to BR-37 (new); `api/auth.py`, `api/config.py`, `api/db.py`; the frontend; deployment docs @@ -35,10 +35,10 @@ The platform is repositioned from a single-operator internal tool to a **multi-t **Identity and access (BR-36)** - Per-user authentication replaces the shared key for interactive use. The key gate can remain for service-to-service calls. - Roles at minimum: *uploader*, *reviewer/approver*, *tenant admin*, *auditor (read-only)*. -- The sovereignty constraint (D6) rules out SaaS identity providers at runtime. The proposed approach is OIDC against an IdP the deployment controls, plus a local-account fallback for air-gapped installs. **Needs your confirmation.** +- The sovereignty constraint (D6) rules out SaaS identity providers at runtime. Approved approach (2026-10-26): OIDC against an IdP the deployment controls, plus a local-account fallback for air-gapped installs. **Tenant isolation (BR-36)** -- Proposed model: schema per tenant (Milestone 0, decision 6). Dynamic tables already live in a configurable schema, so this extends an existing pattern. +- Approved model (2026-10-26): schema per tenant (Milestone 0, decision 6). Dynamic tables already live in a configurable schema, so this extends an existing pattern. - Every request resolves to exactly one tenant, and the database session is scoped to it. - Cross-tenant read and write attempts are standing negative tests from Milestone 2 onwards. diff --git a/docs/art/milestone-0-assessment.md b/docs/art/milestone-0-assessment.md index 06613d5..76449ab 100644 --- a/docs/art/milestone-0-assessment.md +++ b/docs/art/milestone-0-assessment.md @@ -2,7 +2,7 @@ **Branch:** `art-milestone-0` · **Baseline commit:** `main` @ 032b522 · **Date:** 2026-10-26 **Prompt:** [`docs/prompts/ART_FRAMEWORK.md`](../prompts/ART_FRAMEWORK.md) (v2) · **Issue:** #53 -**Status:** For product-owner approval. No production code has been changed. +**Status:** Approved by the product owner on 2026-10-26: ADR-0002, ADR-0003 and all nine §6 recommended defaults. No production code was changed in Milestone 0. This document covers Milestone 0 steps 1, 2, 3, 5 and 6. Step 4 is the two proposed ADRs: [ADR-0002](../adr/0002-transactional-staged-import.md) and [ADR-0003](../adr/0003-customer-facing-multi-tenant.md). @@ -170,4 +170,4 @@ M7's offline-install part can run in parallel with M1 to meet the 15 Nov deadlin | 8 | Performance targets and environment | 100,000-row import under 60 s and first preview page under 2 s on a 2 vCPU / 4 GB instance; Azure Container Apps plus an on-prem docker compose | Targets need a perf harness (M7); BR-22 stays deferred | | 9 | PR #40 and csv-table-hub PR #1 | **Close both:** outside the React/TS constraint | Loses the plain-HTML fallback (#50) | -**Next step:** approve or amend ADR-0002, ADR-0003 and the nine decisions. M1 starts only after approval. +**Approved 2026-10-26:** all nine recommended defaults, recorded in [`DECISIONS/2026-10-26-art-milestone-0.md`](../../DECISIONS/2026-10-26-art-milestone-0.md). M1 starts next.