Interactive CSV Plot (React + TypeScript + Vite + uPlot)

This app loads a CSV of x,y pairs, displays a windowed line plot, animates the window across the dataset, and shows exact aggregates for the currently displayed window. Rendering uses envelope downsampling with an error band for performance and correctness.

Built for the task “Software Engineer Task - Frontend (02.2025)” with the implementation approach described in SOP_CHECKLIST.md.

Outcome

See how it works on my machine

Or try it yourself here

Quick start

Prerequisites: Node.js 18+.

• Install dependencies:
  • npm ci (or npm install)
• Start dev server:
  • npm run dev then open the printed local URL
• Build for production:
  • npm run build
• Preview production build:
  • npm run preview

Using the app

1. Launch the app and either drag-and-drop a .csv onto the dropzone or click to pick a file.

2. Controls (top panel): controls appear after a file is loaded.

• S (start): left index of the window (default 0).
• N (window): number of points in the window (default 10,000; no hard cap).
• P (shift): how many indices S advances on each tick (default 10).
• T (ms): interval between ticks in milliseconds (default 500; min 16).
• Bins (downsample): target number of rendered buckets/points (default 400). Presets: 200, 400, 800, 2000.
• Buttons: Orange Stress test (sets N=1,000,000; P=10,000; T=16ms), plus Play/Pause and Reset once a file is loaded.

3. Chart:

• Main line shows the downsampled average per bucket.
• Shaded band shows the min–max per bucket (error margin).

4. Statistics:

• For the current window only: min, max, mean, variance (computed exactly on raw window data).

CSV format

• If the first row is not numeric (auto-detected), it is treated as a header. The parser then looks for columns named x and y (case-sensitive). If not found, it falls back to the first two columns.
• If there is no header, the first two columns are interpreted as x and y.
• Malformed rows are skipped.

Architecture

• src/workers/parser.worker.ts: streaming CSV parsing and window serving in a Web Worker.
  • Builds a coarse row index (byte offsets) with configurable indexStride for fast seeks by row number.
  • Maintains a bounded cache for [S - marginBefore, S + N + marginAfter) and reuses overlap between requests; margins are forward‑biased during animation.
  • Computes bucket‑envelope downsampling and exact window aggregates in one pass via computeEnvelopeAndAggregates (src/workers/window_stats.ts).
  • Posts only the compact envelope (x, yAvg, yMin, yMax) plus aggregates back to the main thread.
• src/App.tsx:
  • Holds app state and windowing (S, N, P, T) and the UI‑selected targetBins.
  • Sends requestWindow with margins and targetBins; receives envelope + aggregates from the worker.
  • Runs the animation loop (interval ≥ 16ms) and FPS tracking.
• src/components/Controls.tsx: inputs for S, N, P, T, and Bins (downsample); includes an orange "Stress test" button. Controls are hidden until a file is loaded.
• src/components/Chart.tsx: uPlot chart with a filled band (min–max) beneath the average series. Resize-safe via ResizeObserver with guards.
• src/workers/window_stats.ts: bucket‑based envelope downsampling and exact aggregates over the visible window.
• src/workers/header_detect.ts, src/workers/csv_fast.ts, src/workers/row_index.ts: header/column detection, fast numeric CSV extraction, and row index building.
• src/store/state.ts: control defaults, clamping, and nextStartIndex helpers.

Performance considerations

• Streaming & workers: Parsing happens in a Web Worker with chunked/batched posting via transferable buffers to avoid blocking the UI and minimize copies.
• Typed arrays & views: Windowing uses subarray views, so aggregates/downsampling operate without cloning the underlying data.
• Downsampling target: The app downscales the displayed window to ~400 points by default (configurable via the Bins control) for smooth rendering and consistent FPS even for large N.
• Animation: With T ≥ 16 and sensible N, the app maintains ~60 FPS. The FPS badge displays actual vs theoretical (1000/T).

Notes and constraints:

• The current implementation accumulates parsed rows in memory (Float64Arrays). For 10–100M rows you may need segmentation/ring-buffering or persisting to IndexedDB; see “Future improvements”.
• Aggregates are computed exactly on the window; downsampling is used only for rendering. This ensures data correctness.

How this maps to the checklist

• File load & windowing: Drag/drop or picker; S, N inputs with defaults; Start/Stop animation that increments S by P every T ms.
• Aggregates: min, max, average, variance — computed from the displayed window only (exact).
• Downsampling & error band: Bucket envelope with band rendered as min–max, line as average.
• Performance: Parsing in worker; UI remains responsive; downsampling keeps rendered point count small; guarded resize.
• Limits: T won’t go below 16ms (guarded). No hard cap on N (downsampling applied for rendering).

Configuration knobs

• Downsampling density: use the UI Bins control (defaults to 400). Presets: 2500, 5000, 10000, 20000.
• Stress testing: press the orange Stress test button (N=1,000,000, P=10,000, T=16ms).
• Worker tuning: adjust indexStride and chunkBytes in the init message from src/App.tsx (defaults: stride 1000, chunk 8MB).
• Column selection and header handling: the worker auto-detects header and x/y. You can force xColumn, yColumn, hasHeader by passing them in the init message and wiring a UI.

Troubleshooting

• “DataCloneError: Failed to execute 'postMessage' on 'Worker'”: Large typed arrays posted between threads can exhaust memory if not transferred. Parsing uses transfer lists (zero‑copy). Envelope + aggregates are computed in the worker and only compact arrays are posted back.
• Performance dips for very large N: Increase T, reduce P, or increase downsampling target moderately. Ensure a modern browser and disable devtools throttling.
• Unexpected columns: If your CSV uses different column names, ensure x and y are present or adapt the worker to your header.

Sample data

• A generator is included: sample-gen.js.
  • Example: node sample-gen.js 300000 data.csv 100
  • Produces id,value with a random-walk style series.
• You can also use your own CSVs; large files are parsed in a streaming fashion.

To generate stress tests sample - 100M rows:

node sample-gen.js 100000000 100M.csv 100

Future improvements

• Offload more main‑thread work; consider a dedicated stats/downsampling worker using SharedArrayBuffer to keep the UI thread nearly idle.
• Segmented storage or ring-buffer + optional IndexedDB spillover for 10M–100M rows to keep memory bounded.
• Incremental aggregates for small P shifts (update sums/sums²) and deque-based sliding min/max for faster min/max on large windows.
• Additional downsampling methods (e.g., LTTB) and adaptive point targets based on container width and device pixel ratio.
• Column selector UI for arbitrary CSV schemas and error reporting UI for malformed rows.

Optimizing large file loading strategies

• Stream parsing in a Web Worker with chunked reads (default 8MB) to avoid blocking the UI.
• Build a coarse row index (indexStride ≈ 1000) to seek near windows quickly.
• Keep a sliding cache with forward‑biased margins and reuse overlap to minimize re‑parsing.
• Cap rows per request to bound memory and reduce GC pressure.
• Use a fast numeric path for simple CSVs; fall back to Papa Parse for complex/quoted rows.
• Downsample early in the worker and transfer only the compact envelope and aggregates.
• Expose Bins/P/T/N controls and a one‑click stress profile to evaluate performance.

Tech stack

• React 19 + TypeScript + Vite
• uPlot for plotting with a custom band fill
• Papa Parse for streaming CSV in a Web Worker
• Vitest for unit testing and coverage

Scripts

• npm run dev — start dev server
• npm run build — typecheck and build
• npm run preview — preview production build
• npm run lint — lint the project
• npm run test — run unit tests (Vitest)
• npm run test:coverage — run tests with coverage report

Testing

Unit tests are implemented with Vitest and cover core logic:

• computeEnvelopeAndAggregates (downsampling + aggregates) in src/workers/window_stats.ts
• fastExtractXY (numeric CSV tokenizer) in src/workers/csv_fast.ts
• clampWindow and nextStartIndex (windowing helpers) in src/store/state.ts

Run tests:

npm run test

Run with coverage:

npm run test:coverage