Skip to content
muhammadghiffariPublic

About

WiFi-based Spatial Intelligence Platform - Real-time human pose estimation, 3D Observatory, and CSI sensing visualization using MediaPipe, WebGL, and Rust.

Resources

Stars

28 stars

Watchers

2 watching

Forks

Latest commit

 

History

11 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WeView: Spatial Intelligence & Multi-Modal Sensing Platform

WeView Logo

Real-Time Human Pose Estimation and CSI Sensing Visualization

A comprehensive framework fusing RF (Radio Frequency) Channel State Information (CSI) with optical inference for robust, non-intrusive spatial telemetry.

Live Demo DeepWiki Research Papers


Abstract

WeView is a multi-modal sensing platform developed to utilize Channel State Information (CSI) from commodity WiFi hardware (ESP32-S3) to perform human pose estimation, vital sign monitoring, and disaster recovery sensing. Built from the ground up, the project features a high-performance, domain-driven Rust ecosystem capable of real-time inference and distributed swarm coordination.

The primary goal of WeView is to "see" through obstacles and in low-light conditions by analyzing how human bodies disturb WiFi signals, mapping these disturbances to the DensePose coordinate system (mapping image pixels to 3D surface coordinates of the human body).

High-Level Architecture & Major Subsystems

The codebase is organized into four primary pillars:

  1. Sensing Infrastructure: ESP32-S3 firmware for high-frequency CSI collection and edge processing.
  2. Core Processing (Rust v2): A workspace of 15+ crates handling signal processing, neural network inference (ONNX/Candle), and spatial-temporal analysis.
  3. Agentic Orchestration (Claude Flow V3): A self-learning multi-agent swarm that manages code evolution, system optimization, and consensus-driven decision making.
  4. Visualization & UX: A Tauri-based desktop application, a Three.js-powered pointcloud viewer, and a Glassmorphism web UI for real-time 3D rendering of pose data.

Intelligence & Orchestration: Claude Flow V3

A unique aspect of this codebase is the integration of Claude Flow V3, an agentic framework that manages the system's lifecycle. It uses a hierarchical-mesh topology to coordinate up to 15 specialized AI agents (e.g., coder, security-architect, performance-engineer).

  • SPARC Specification Phase: Acts as the entry point for domain modeling, using GNN-enhanced search to retrieve similar requirement patterns and ensuring that new features align with existing bounded contexts.
  • Agent-Code Interaction: Manages system evolution autonomously while enforcing strict Domain-Driven Design constraints.

Domain-Driven Design (DDD) Models

The WeView project utilizes DDD to manage the high complexity of multi-modal sensing (WiFi, mmWave, and Quantum) and its translation into human pose estimation and vital signs.

Bounded Context Type Responsibility
Hardware Platform Generic ESP32 CSI ingestion, radio abstraction, and frame parsing.
Signal Processing Core Phase sanitization, BVP extraction, and Fresnel zone analysis.
Sensing Server Core Orchestration of real-time data flows, model management, and SONA.
WeSense / WeVector Core Viewpoint attention, CRV signal lines, and multistatic bridge.
WiFi-Mat Supporting Disaster response, survivor triage (START), and Kalman tracking.
Training Pipeline Supporting Dataset loading, virtual augmentation, and rapid adaptation.
CHCI (Claude Flow) Supporting AI agent orchestration, swarm topology, and self-learning hooks.

94-ADR Index (Architecture Decision Records)

The project utilizes the MADR (Markdown Any Decision Records) 3.0 format to maintain consistency and machine-readability. Decisions are organized into primary domains:

  • ADR-002: Modular DDD Architecture - Established the transition from a monolithic Python structure to a modular Rust workspace.
  • ADR-006: Unified Memory Service - Defines how CSI data and neural embeddings are stored in a hybrid memory backend.
  • ADR-081: Adaptive Controller (Firmware) - Governs how the ESP32 firmware dynamically adjusts the CSI sampling rate based on WiFi congestion.

Engineers can interact with the ADR system using the CLI: npx claude-flow@v3alpha adr search ...

Visualization & UI Modules (Frontend)

The frontend operates in decoupled layers to interact with the Rust backend:

  • Dual-Modal Pose Fusion Engine: Uses MediaPipe WASM for optical keypoint extraction, overlaid with CSI data. It introduces an "Honesty Filter" algorithm to mathematically suppress hallucinated lower-body keypoints when bounding boxes indicate only upper-body visibility.
  • 3D Spatial Observatory: A fully interactive spatial rendering engine built with WebGL and Three.js supporting Realistic, DensePose, and X-Ray mapping modes.
  • NVSim Magnetometer: A Vite + Lit TypeScript dashboard designed to visualize NV-diamond magnetometer telemetry.
  • 2D Observatory Telemetry: Canvas engine optimized for rendering top-down occupancy heatmaps.

Getting Started & Build Guide

Before starting the build process, ensure the following toolchains are installed:

  • Rust: 1.75+ (Stable) (Core workspace and sensing server)
  • Docker: 24.0+ (Containerized pipelines and monitoring)
  • Python: 3.10+ (ML training scripts and simulation servers)
  • ESP-IDF: v5.1+ (ESP32-S3 firmware compilation)
  • QEMU: System-xtensa (Firmware testing without hardware)

1. Docker Quick-Start

The fastest way to deploy the sensing infrastructure is via the provided Docker configuration:

docker-compose up --build

This launches the Rust sensing server, simulation pipelines, and the Prometheus/Grafana monitoring stack.

2. Rust Workspace Build

cargo fetch
cargo build --release
cargo test --workspace

3. Firmware & QEMU Testing

To build for physical ESP32-S3 hardware:

cd firmware/esp32-csi-node
idf.py set-target esp32s3
idf.py build

For headless testing via QEMU:

./scripts/qemu-test-swarm.sh

4. The Verification Pipeline (./verify)

To ensure the integrity of the sensing proofs and the WeVector witness chain:

./verify --pipeline all --proof-path ./proofs/latest.json

License & Attribution

This project is released under the MIT License. Developed for Advanced Spatial Intelligence & Real-Time Computer Vision Research. Built for Economic Survival 2026.

About

WiFi-based Spatial Intelligence Platform - Real-time human pose estimation, 3D Observatory, and CSI sensing visualization using MediaPipe, WebGL, and Rust.

Resources

Stars

28 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages