Attention Tracker

Attention Tracker is a lightweight, real-time engagement analytics library for detecting focus, distraction, drowsiness, gaze behavior, and emotional states directly in the browser.

Designed for edge computing, it analyzes facial expressions, eye movements, and head pose locally on the user's device. This ensures zero-latency feedback and 100% privacy, as no sensitive video data is ever transmitted to external servers.

Note: GIF playback may appear choppy due to compression and screen recording limitations; the actual library runs smoothly in real-time. Watch full high-quality MP4 versions on Google Drive

Compatible with all modern frontend frameworks including Next.js, Vue.js, Angular, Nuxt, Svelte, and vanilla JavaScript/TypeScript bundlers (Webpack, Vite, Rollup).

---

🎯 Broad Use Cases

While originally designed for adaptive educational interfaces (e-learning) and video conferences, the core engine is highly versatile:

• Safety & Driver Monitoring: Detect drowsiness, microsleeps, and erratic gaze behavior.
• Productivity & Self-Control: Monitor focus levels and manage distractions during deep work.
• UX/UI Research: Understand user frustration or focus zones during product testing.
• Telehealth & Therapy: Monitor patient behavior and emotional states during remote sessions.

---

Key Features & Under the Hood

• Real-time Processing: Optimized to run seamlessly in the browser using WebAssembly and Web Workers, preventing UI blocking.
• Modular Architecture: Enable or disable specific neural networks (e.g., emotion tracking or gaze estimation) to save CPU/GPU resources.
• Custom Proprietary Emotion Model: Uses a unique, custom-built and trained neural network model to classify emotional states (Focus, Thinking, Neutral, Happy, Sad).
• Gaze Estimation: Employs a strategy pattern utilizing either fast geometric calculations (math) or a highly accurate OpenVINO neural network (openvino).
• Graceful Degradation: Automatically downgrades from heavy NN models to mathematical approximations (auto strategy) to maintain FPS on low-end devices.
• Blink & Yawn Detection: Dynamically calculates Eye Aspect Ratio (EAR) and Mouth Aspect Ratio (MAR) with auto-adjusting thresholds to detect microsleeps, drowsiness, and yawning.
• Auto-Calibration: Adapts to the user's posture, camera angle, and distance dynamically.

---

📦 Installation

Install the library via npm:

npm install attention-tracker

Assets & Models Setup

Out of the box, the library fetches required WebAssembly binaries and models from CDNs (e.g., jsdelivr). You don't need to configure anything to get started!

However, if you want an offline setup or want to host the models yourself:

1. Download the models folder from the public/models directory of this repository or CDNs.
2. Place it in your project's public directory.
3. Pass your custom paths to the assets object when initializing the library.

---

Usage (Vanilla JS/TS)

Using the library is straightforward. You only need to provide configuration overrides if you want to change the defaults (the default config runs off-thread, uses the GPU backend, and auto-manages gaze strategies).

1. Initialization

Create an instance of the monitor. You can pass an optional configuration object. To see all default values, check out the DEFAULT_CONFIG and types (CONFIGURATION INTERFACES) in the source code. You can fully customize thresholds, time windows, and penalty weights via the configuration object.

import { AttentionMonitor } from "attention-tracker";

// Creates a monitor with default settings (CDN assets, Web Worker enabled)
const monitor = await AttentionMonitor.create();

// OR with custom configuration:

const monitor = await AttentionMonitor.create({
  worker: true,
  backend: "GPU",
  assets: {
    models: {
      emotion: "/my-local-path/emotion_model.onnx", // Use local model
    },
    settings: {
      gaze: {
        strategy: "openvino",
      },
      engine: {
        timeToConfirm: 500, // Ms required to confirm a status change (debouncing)
        weights: {
          gaze: 0.5,
          perclos: 0.35,
          yawn: 0.15,
        },
        adhdDynamics: {
          enabled: true, // Enable ADHD status calculations
        },
      },
      blink: {
        perclosDrowsyThreshold: 0.15, // % of eye closure to trigger drowsiness
        microsleepLimit: 2, // Seconds of continuous eye closure
      },
    },
  },
});

2. Subscribing to Events & Starting

Listen to the attention event to get real-time data, then start the tracker by passing an HTML Video element.

// Listen to results
monitor.on("attention", (result) => {
  console.log(`Status: ${result.status} | Score: ${result.score}`);

  if (result.status === "MICROSLEEP") {
    console.warn("MICROSLEEP detected, user losing focus!");
  }
});

monitor.on("error", (err) => console.error("Monitor Error:", err));

// Start processing the video stream
const videoElement = document.getElementById("user-video") as HTMLVideoElement;
monitor.start(videoElement);

3. Understanding the Payload (AttentionResult)

Every frame, the attention event emits a comprehensive payload containing everything from the high-level semantic status to raw tracking data:

{
  status: "NORMAL", // "DISTRACTED" | "NORMAL" | "DROWSY" | "MICROSLEEP" | "FATIGUED" | "ADHD" | "NOT_DETECTED"
  score: 0.95,  // 0.0 to 1.0
  details: {
    penalties: { gaze: 0, perclos: 0, yawn: 0, emotionModifier: 1 },
    isADHD: false,
    direction: {
      headAngles: [yaw, pitch, roll],
      gazeVector: [x, y, z]
    }
  },
  signals: {
    emotion: "FOCUSED", // Evaluated by the custom neural network model
    blink: { status: "DROWSY", count: 12, perclos: 0.17, threshold: 0.64 },
    yawn: { status: "YAWNING", count: 3, threshold: 1.5 },
    raw: { ear: 0.2, mar: 0.7 },
    performance: { latency: 33.15, isDowngraded: false, activeModel: "OPENVINO" }
  },
  snapshot: { /* Raw geometric data */ },
  calibration: { /* User's baseline data */ }
}

---

⚛️ Usage in React & Vue (Recommended)

The library provides built-in hooks for modern frameworks to handle the entire lifecycle, model loading, event subscriptions, and Web Worker cleanup automatically.

React Integration

Import the hook directly from attention-tracker/react.

import { useRef } from "react";
import { useAttentionMonitor } from "attention-tracker";

export default function App() {
  const videoRef = useRef<HTMLVideoElement>(null);

  const { result, isReady, error } = useAttentionMonitor(
    videoRef,
    {
      worker: true,
      backend: "GPU",
      settings: {
        // Example: Disable the emotion neural network to save resources
        emotion: { enabled: false },
      },
    },
    {
      autoStart: true,
      throttleStateMs: 250, // Limits React re-renders to every 250ms for performance
      onCalibrated: (state) => console.log("Baseline set:", state),
      onFaceLost: () => console.warn("User left the frame!"),
    },
  );

  return (
    <div>
      <video ref={videoRef} autoPlay muted playsInline />

      {!isReady && <p>Loading models...</p>}
      {error && <p>Error: {error.message}</p>}

      {result && (
        <div className="dashboard">
          <h2>Status: {result.status}</h2>
          <p>Attention Score: {Math.round(result.score * 100)}%</p>
          <p>Emotion: {result.signals.emotion}</p>
        </div>
      )}
    </div>
  );
}

Vue Integration

Import the composable directly from attention-tracker/vue. The returned result is a ShallowRef to prevent Vue from recursively watching deep nested objects (like 478 face landmarks at 60 FPS), guaranteeing massive performance gains.

<script setup lang="ts">
import { ref, onMounted } from 'vue'
import { useAttentionMonitor } from 'attention-tracker/vue'

const videoRef = ref<HTMLVideoElement null |>(null)

const { result, isReady, error } = useAttentionMonitor(
  videoRef,
  { worker: true, backend: "GPU" },
  {
    throttleStateMs: 250,
    onStatusChanged: ({ previous, current }) => console.log(`Changed from ${previous} to ${current}`)
  }
)

</script>

<template>
  <div>
    <video ref="videoRef" autoplay playsinline muted></video>

    <div v-if="error">Error: {{ error.message }}</div>
    <div v-else-if="!isReady">Loading Models...</div>
    <div v-else-if="result">
      <h2>Status: {{ result.status }}</h2>
      <p>Score: {{ result.score.toFixed(3) }}</p>
    </div>
  </div>
</template>

⚡ React Performance Tip

Tracking can run at high framerates (up to 30-60 FPS). Updating React/Vue state on every single frame will cause severe performance issues and UI lag.

To prevent this, use throttleStateMs to slow down state updates to a comfortable rate (e.g., 100-250ms). If you need to perform high-frequency tasks like drawing landmarks on an HTML <canvas>, use the onUpdate callback in the hook options, which executes every frame without triggering a UI re-render. You can find more detailed information and examples directly in the source code for React useAttentionMonitor and Vue useAttentionMonitor.

---

🛠️ Configuration Power

Thanks to its highly flexible configuration, you can adapt the engine to track almost any engagement metric. To see all default values, check out the DEFAULT_CONFIG in the source code.

If you don't need certain metrics (e.g., emotions or gaze), you can disable them entirely to save CPU/GPU resources and significantly improve frame processing time.

Driver Monitoring System (Safety Profile)

Notice how quickly the system adapts to the situation after startup, while keeping the status stable without erratic changes. [Watch full high-quality MP4 on Google Drive]

The default configuration is tuned for office/e-learning scenarios. Here is how you can reconfigure it for Driver Monitoring, where microsleeps are fatal and users constantly check mirrors.

You can understand why and how to change values for your purposes and tasks.

const driverConfig = {
  settings: {
    emotion: {
      // Disable emotion neural network to save GPU resources,
      // as it's irrelevant for driver safety.
      enabled: false,
    },
    blink: {
      // A 2-second microsleep at 100km/h is lethal. Drop the limit to 0.8 seconds.
      microsleepLimit: 0.7,
      // Strict fatigue monitoring: lower the PERCLOS threshold to catch early signs of drowsiness.
      perclosDrowsyThreshold: 0.15,
    },
    engine: {
      // React faster to dangerous states. Reduce debounce time from 500ms to 200ms.
      timeToConfirm: 200,
      gazeDynamics: {
        // Drivers MUST check mirrors. Widen the horizontal deadzone to 35° so looking left/right doesn't immediately penalize.
        yawDeadzone: 35,
        // Penalize vertical drops (looking at phone/lap) much more harshly than horizontal (mirrors).
        pitchScale: 15,
      },
      weights: {
        // Note: These weights govern the "soft" attention score calculation.
        // Critical events (like MICROSLEEP) act as hard overrides and instantly drop the score to 0.
        // Shift the penalty weights. Fatigue is much deadlier than being momentarily distracted.
        perclos: 0.65, // Eyes closing is the most critical fatigue indicator
        yawn: 0.3, // Yawning is a huge red flag
        gaze: 0.15, // Forgive mirror checking (lower weight compared to e-learning)
      },
      adhdDynamics: {
        // Tracking fidgeting is irrelevant for driving safety.
        enabled: false,
      },
    },
  },
};

const dashcamMonitor = await AttentionMonitor.create(driverConfig);

---

Running the Demos Locally

Want to see it in action or test it locally? This repository includes ready-to-use demos for both Vanilla TS and React.

1. Clone the repository:

git clone https://github.com/P3Lin0r/attention-tracker.git

cd attention-tracker

2. Install dependencies:

npm install

3. Run the preferred demo:

There are four distinct demos to help you understand, test, and integrate the library. You can run them locally to see the system in action:

• 📸 The Playground (Vanilla TS)

  Want to see the raw power of the tracker? This demo connects to your webcam and visualizes everything in real-time. Watch your Eye Aspect Ratio (EAR), Mouth Aspect Ratio (MAR), and 3D gaze vectors dynamically change on live canvas graphs. It displays a complete breakdown of penalties and raw signals, making it the perfect sandbox to test the library on yourself and fine-tune configuration thresholds.

  Run: npm run dev or npm run dev:vanilla-ts

• ⚛️ React Minimal

  Showcases how incredibly simple it is to drop the tracker into a modern React application utilizing the useAttentionMonitor hook.

  Run: npm run dev:react

• 💚 Vue Minimal

  A clean and reactive implementation for Vue 3 developers utilizing the built-in composable for seamless integration.

  Run: npm run dev:vue

• 👥 Group Attention Dashboard (Video Grid)

  A production-ready simulation of a video conference (like Zoom or Google Meet). This demo processes multiple video streams simultaneously and aggregates the individual metrics into a high-level Global Group Status (e.g., dynamically alerting you if the "AUDIENCE FALLING ASLEEP ⚠️" or if they are "HIGHLY ENGAGED ✅"). It perfectly demonstrates how to scale the library's business logic for e-learning platforms and analytics dashboards.

  ❗ To use it, place your videos into the public/videos folder and add their relative paths to the VIDEO_SOURCES array in main.ts. You can download some videos from Google Drive

  Run: npm run dev:dashboard

(Demos are configured to resolve internal imports seamlessly without needing a production build first).

---

🧠 Architecture Overview

The tracking pipeline is designed for maximum throughput. It offloads heavy tensor operations to Web Workers while keeping the main thread free for UI rendering.

graph TD
    %% Define Styles
    classDef hardware fill:#2d3436,stroke:#dfe6e9,stroke-width:2px,color:#fff;
    classDef core fill:#0984e3,stroke:#74b9ff,stroke-width:2px,color:#fff;
    classDef worker fill:#6c5ce7,stroke:#a29bfe,stroke-width:2px,color:#fff,stroke-dasharray: 5 5;
    classDef engine fill:#00b894,stroke:#55efc4,stroke-width:2px,color:#fff;
    classDef output fill:#d63031,stroke:#ff7675,stroke-width:2px,color:#fff;

    A[📸 Web Camera]:::hardware --> B[FaceTracker Core]:::core

    subgraph Web Worker Thread
    B --> C{MediaPipe Face Landmarker}:::worker

    C -->|478 Landmarks| D[EAR / MAR Detectors]:::worker
    C -->|52 Blendshapes| E[ONNX Emotion NN]:::worker
    C -->|Landmarks / Eyes Crop & Head Angles| F[Gaze Detector<br/>Math/OpenVINO]:::worker
    end

    D --> G[Calibration Manager]:::engine
    E --> G
    F --> G

    G -->|All data & Calibration states| H[Attention Engine]:::engine
    H --> I((Attention Payload<br/>Score, Status & Details)):::output

Loading

📂 Directory Structure

Here is a quick overview of the library's internal structure for contributors:

src/
├── api/            # Public entry points (AttentionMonitor, EventEmitters)
├── core/           # Face tracking orchestrator and performance monitors
│   └── history/    # Zero-allocation circular buffers for time-series math
├── detectors/      # Isolated modules for specific detections
│   ├── gaze/       # Gaze strategies (Math geometry vs OpenVINO)
│   ├── BlinkDetector.ts    # EAR-based microsleep/drowsiness logic
│   ├── EmotionsDetector.ts # ONNX Model execution
│   └── YawnDetector.ts     # MAR-based yawning logic
├── analytics/      # High-level logic aggregating raw detector signals
│   └── CalibrationManager.ts # Adapts to user's posture changes
│   ├── AttentionEngine.ts    # Aggregates data to a final score & status with debouncing
├── hooks/          # Integrations with frameworks (useAttentionMonitor)
├── workers/        # Web Worker entry points for off-thread processing
└── config/         # Default configurations and type definitions

⚖️ License & Acknowledgments

This project is licensed under the MIT License - see the LICENSE file for details.

This library is built upon several open-source technologies. I would like to thank their creators and maintainers:

• MediaPipe Tasks Vision by Google (Licensed under Apache 2.0). Utilized for highly optimized facial landmark detection.
• ONNX Runtime Web by Microsoft (Licensed under MIT). The core inference engine for our neural networks.
• OpenVINO Model Zoo (OMZ) by Intel (Licensed under Apache 2.0). I use the gaze-estimation-adas-0002 topology for precise eye-tracking.