SmartHome System

SmartHome System is an AI-powered smart-home monitoring and interaction prototype. It connects a browser dashboard, a TypeScript cloud backend, an external MJPEG camera source, and a Python Qwen vision worker so the system can stream camera data, answer visual questions, and generate proactive smart-home observations.

---

Project Context

This system was developed for Hubei Wanlu Zhixin Technology as part of my work as an Embedded Software Engineer. The project focused on building a smart-home AI interaction prototype that connects visual monitoring, dashboard interaction, backend orchestration, and AI-powered camera-frame analysis.

Although my role title was Embedded Software Engineer, this project required work across multiple layers of the system:

• smart-home system architecture and runtime flow design;
• dashboard-to-backend integration;
• camera/MJPEG stream integration;
• backend API and WebSocket contract planning;
• Qwen vision worker integration;
• local development workflow design;
• test planning and validation;
• AI-assisted implementation review.

This public repository is intended to demonstrate the engineering work, architecture, and development workflow behind the prototype. It should be treated as a public reference/demo version, not as an official production release of the company.

---

AI-Assisted Engineering Disclosure

This project was built with an AI-assisted software engineering workflow using ChatGPT and OpenAI Codex as development accelerators.

The system was not generated blindly. The product direction, system requirements, architecture decisions, module boundaries, runtime flow, integration strategy, and final engineering judgment were designed and controlled by Myself. ChatGPT and Codex were used as tools to speed up implementation, validate ideas, generate/refactor code in phases, create tests, and improve documentation.

My Role in the Project

I owned the end-to-end system design and development direction for this WiseOne prototype at Hubei Wanlu Zhixin Technology, including:

• translating the company smart-home prototype goal into a working software architecture;
• defining the smart-home assistant concept and feature scope;
• designing the split architecture between dashboard, server, and qwen-vision-worker;
• deciding how the dashboard, backend, camera source, WebSocket layer, and Qwen worker communicate;
• planning the runtime contracts for visual questions, proactive observations, camera status, and dashboard updates;
• reviewing, testing, and correcting AI-generated implementation work;
• restructuring the repository into a public version with dashboard/ and server/;
• writing and validating the final local run process.

How ChatGPT and Codex Were Used

ChatGPT and Codex were used as engineering assistants for:

• architecture planning and implementation breakdown;
• agent-style phased development planning;
• code generation for selected frontend, backend, and worker modules;
• refactoring and cleanup;
• TypeScript/Python test planning;
• debugging local integration issues;
• README and documentation drafting;
• improving developer workflow and project handoff quality.

What This Demonstrates

This project demonstrates an AI-native development workflow: using modern AI coding tools to accelerate delivery while still applying software engineering fundamentals such as architecture ownership, system decomposition, API contracts, integration testing, environment management, and production-style documentation.

In short: AI assisted the implementation, but the system design, technical direction, and final decisions were mine.

---

Table of Contents

• Project Context
• AI-Assisted Engineering Disclosure
• What This System Does
• Architecture
• Repository Structure
• Tech Stack
• Requirements
• Environment Configuration
• How to Run the System
• Optional Checks and Tests
• Main Runtime Contracts
• Development Notes
• Troubleshooting
• Security Notes

---

What This System Does

WiseOne System is designed as a smart-home AI assistant prototype with the following major capabilities:

• Displays a real-time camera feed in the dashboard.
• Reads camera status from a backend API.
• Sends visual questions from the dashboard to the backend.
• Uses a Python Qwen vision worker to analyze camera frames.
• Returns AI-generated answers to the dashboard conversation UI.
• Supports proactive visual analysis, such as item identification and smart-home observations.
• Sends backend-to-dashboard updates through HTTP and WebSocket integration.
• Provides dashboard sections for camera feed, cloud conversation, voice history, item identification, equipment operations, and control-center style device state.

The system was built as a prototype/foundation for a cloud-connected smart-home assistant, not as a production-ready commercial deployment.

---

Architecture

The current system is split into three main local parts and one external camera source:

External Ubuntu MJPEG Camera Server
              |
              v
        server/ NestJS API
              |
              v
Python Qwen Vision Worker
              |
              v
        server/ NestJS API
              |
              v
      dashboard/ React UI

Runtime Flow

1. The Ubuntu camera server exposes an MJPEG stream.
2. The server backend reads or proxies camera-related data.
3. The dashboard calls backend APIs and connects to the backend WebSocket namespace.
4. When the user asks a visual question, the backend captures/uses a camera frame and sends it to the Qwen vision worker.
5. The Qwen worker analyzes the image and returns either:
   • a reactive response for a user question, or
   • a proactive response for automatic smart-home observations.
6. The backend sends the result back to the dashboard.

---

Repository Structure

wiseone-system/
├── dashboard/
│   ├── public/
│   ├── scripts/
│   ├── src/
│   ├── package.json
│   ├── pnpm-workspace.yaml
│   ├── tailwind.config.ts
│   ├── tsconfig.json
│   └── vite.config.ts
│
├── server/
│   ├── apps/
│   │   └── api/
│   ├── infra/
│   ├── packages/
│   │   └── contracts/
│   ├── scripts/
│   ├── workers/
│   │   └── qwen-vision-worker/
│   ├── .env.example
│   ├── package.json
│   ├── pnpm-workspace.yaml
│   └── tsconfig.base.json
│
└── README.md

Important Folders

Path	Purpose
dashboard/	React + Vite smart-home dashboard UI.
dashboard/src/	Main dashboard source code, UI components, integrations, services, and stores.
dashboard/scripts/	Dashboard environment validation script.
server/	Backend, contracts, worker, runtime documentation, and optional infrastructure.
server/apps/api/	NestJS API server.
server/packages/contracts/	Shared runtime/API contract definitions.
server/workers/qwen-vision-worker/	Python FastAPI worker for Qwen visual analysis.
server/infra/	Optional Docker Compose infrastructure for PostgreSQL and Redis.

---

Tech Stack

Dashboard

• React 19
• TypeScript
• Vite
• Tailwind CSS
• Zustand
• Socket.IO Client
• Lucide React icons
• ESLint
• TSX-based test scripts

Backend Server

• Node.js
• TypeScript
• NestJS
• Socket.IO / NestJS WebSockets
• Swagger / OpenAPI support
• Zod
• Class Validator / Class Transformer
• RxJS
• pnpm workspace
• Prisma and ioredis dependencies are present for development/legacy infrastructure support

AI Vision Worker

• Python 3.11+
• FastAPI
• Uvicorn
• Pydantic
• Pydantic Settings
• HTTPX
• python-dotenv
• Tenacity
• Pillow
• Qwen VL API integration

Optional Infrastructure

• Docker Compose
• PostgreSQL 16 Alpine
• Redis 7 Alpine

---

Requirements

Install these before running the project:

• Node.js 20+ recommended
• pnpm 9+
• Python 3.11+
• pip
• A Qwen API key and compatible Qwen VL model access
• An Ubuntu MJPEG camera server or another compatible MJPEG source
• Docker, only if you want to run the optional PostgreSQL/Redis infrastructure

---

Environment Configuration

Server Environment

From inside server/, copy the example environment file:

cp .env.example .env

Then update the values based on your local setup.

Common server variables:

PORT=4000
DATABASE_URL=postgresql://wiseone:wiseone@localhost:5432/wiseone_cloud?schema=public
REDIS_URL=redis://localhost:6379
CORS_ALLOWED_ORIGINS=http://localhost:3000
UBUNTU_MJPEG_URL=http://172.16.105.33:5000
UBUNTU_CHAT_ENABLED=true
UBUNTU_CHAT_HISTORY_LIMIT=100
UBUNTU_CHAT_RECONNECT_MS=5000
UBUNTU_CHAT_REQUEST_TIMEOUT_MS=5000
DEFAULT_HOME_ID=local-home-dev
AI_WORKER_URL=http://localhost:5001
AI_WORKER_ANALYZE_PATH=/analyze-frame
PROACTIVE_ANALYSIS_INTERVAL_MS=10000
QWEN_API_KEY=
QWEN_BASE_URL=
QWEN_VL_MODEL=

Required values to check carefully:

Variable	Purpose
PORT	Backend API port. Default: 4000.
CORS_ALLOWED_ORIGINS	Dashboard origin allowed to call the backend.
UBUNTU_MJPEG_URL	External Ubuntu camera/MJPEG server base URL.
AI_WORKER_URL	Python Qwen worker URL. Default: http://localhost:5001.
AI_WORKER_ANALYZE_PATH	Worker analysis endpoint. Default: /analyze-frame.
QWEN_API_KEY	Qwen API key. Keep this secret.
QWEN_BASE_URL	Qwen API base URL.
QWEN_VL_MODEL	Qwen visual-language model name.

Dashboard Environment

The dashboard has default local values in code, but for explicit local configuration you can create this file:

cd dashboard
nano .env

Recommended local values:

VITE_WISEONE_CLOUD_API_BASE_URL=http://localhost:4000/api/v1
VITE_DASHBOARD_DEVICE_ID=dashboard-dev-01
VITE_WISEONE_HOME_ID=local-home-dev
VITE_WISEONE_DEFAULT_LANGUAGE=en

Dashboard variables must use the VITE_ prefix so Vite can expose them to the browser.

---

How to Run the System

Run the backend, worker, and dashboard in three separate terminals.

Terminal 1: Start the Backend API

cd server
cp .env.example .env
pnpm install
pnpm --filter @wiseone/api dev

Expected backend URL:

http://localhost:4000/api/v1

Health check:

curl http://localhost:4000/api/v1/health

Terminal 2: Start the Qwen Vision Worker

cd server/workers/qwen-vision-worker
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
pip install -e .
python3 -m qwen_vision_worker.main

Expected worker URL:

http://localhost:5001

Worker health check:

curl http://localhost:5001/health

Terminal 3: Start the Dashboard

cd dashboard
pnpm install
pnpm dev

The dashboard usually runs at:

http://127.0.0.1:5173

If Vite prints a different port, open the URL shown in the terminal.

---

Optional Checks and Tests

Server Checks

Run from server/:

pnpm typecheck
pnpm test
pnpm check:env:backend

Useful additional commands:

pnpm lint
pnpm build
pnpm smoke:backend

Dashboard Checks

Run from dashboard/:

pnpm typecheck
pnpm lint
pnpm test
pnpm check:env:dashboard

Optional Infrastructure

The project includes a Docker Compose file for PostgreSQL and Redis under server/infra/.

Run from server/:

pnpm infra:up

Stop infrastructure:

pnpm infra:down

View logs:

pnpm infra:logs

Use this only if the parts of the system you are testing require PostgreSQL or Redis.

---

Main Runtime Contracts

Backend HTTP API

The backend uses the global prefix:

/api/v1

Main routes:

GET  /api/v1/health
GET  /api/v1/camera/status
GET  /api/v1/camera/latest-frame
GET  /api/v1/camera/live.mjpeg
POST /api/v1/vision/ask

AI Worker API

Worker endpoint:

POST /analyze-frame

Reactive request example:

{
  "mode": "reactive",
  "question": "What do you see?",
  "imageBase64": "...",
  "locale": "en"
}

Reactive response example:

{
  "success": true,
  "data": {
    "answer": "I can see ...",
    "items": [],
    "cameraFrameUsed": true
  }
}

Proactive request example:

{
  "mode": "proactive",
  "imageBase64": "...",
  "locale": "en"
}

Proactive response example:

{
  "success": true,
  "data": {
    "items": [
      {
        "label": "bin",
        "confidence": 0.82,
        "location": "left side"
      }
    ],
    "voiceMessage": "I can see a bin near the left side.",
    "shouldNotify": true
  }
}

WebSocket

Namespace:

/ws/cloud

Important event types:

connection.status
camera.status
conversation.userMessage
conversation.aiResponse
proactive.itemIdentification
proactive.voiceHistory

Example AI response event:

{
  "type": "conversation.aiResponse",
  "payload": {
    "text": "Yes, there is one bin visible.",
    "cameraFrameUsed": true,
    "source": "qwen-vl"
  }
}

---

Development Notes

• The root repository contains two main apps: server/ and dashboard/.
• This system was developed for Hubei Wanlu Zhixin Technology during my role as an Embedded Software Engineer.
• The dashboard is now the English version only; the old cn/ folder is no longer part of the project.
• The backend and dashboard are installed separately because each folder has its own package setup.
• The Qwen worker is a separate Python service and must run at the same time as the backend for visual AI responses.
• The camera source is external. If the Ubuntu MJPEG server is offline or the URL is wrong, the dashboard camera/vision features will not work correctly.
• AI tools were used intentionally as engineering accelerators, but the architecture, integration decisions, validation process, and final system direction were owned by the developer.
• Existing subfolder README files may contain older folder names or older EN/CN instructions. Treat this root README as the updated project-level guide.

---

Troubleshooting

Backend Does Not Start

Check that dependencies are installed:

cd server
pnpm install

Check environment values:

pnpm check:env:backend

Check whether port 4000 is already used:

lsof -nP -iTCP:4000 -sTCP:LISTEN

Dashboard Cannot Connect to Backend

Confirm the backend is running:

curl http://localhost:4000/api/v1/health

Check dashboard environment values:

cd dashboard
pnpm check:env:dashboard

Make sure the dashboard API base URL points to:

http://localhost:4000/api/v1

Qwen Worker Returns Configuration Error

Make sure these values are set in the server/worker environment:

QWEN_API_KEY=
QWEN_BASE_URL=
QWEN_VL_MODEL=

Restart the backend and worker after changing environment variables.

Camera Feed Does Not Load

Check the Ubuntu camera server URL in server/.env:

UBUNTU_MJPEG_URL=http://172.16.105.33:5000

Then test backend camera status:

curl http://localhost:4000/api/v1/camera/status

Also test the live MJPEG backend route:

curl http://localhost:4000/api/v1/camera/live.mjpeg

Python Virtual Environment Issues

Recreate the worker virtual environment:

cd server/workers/qwen-vision-worker
rm -rf .venv
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
pip install -e .

---

Security Notes

Because this is a public repository:

• Do not commit .env files.
• Do not commit Qwen API keys.
• Do not expose private camera URLs or private network details in production.
• Rotate any secret that was accidentally pushed to GitHub.
• Add authentication and authorization before using this system outside a trusted local development environment.

---

Author

Developed by Morshed Alam.

Project context: developed for Hubei Wanlu Zhixin Technology during my role as an Embedded Software Engineer.