Mixed Reality (MR) In Industrial Robot Performance Optimization & Monitoring

Table of Contents

• Project Overview
• ✨ Features
• 🛠️ Technologies Used
• 🎥 Demo / Visuals
• 💡 How it Works
• 🚀 Getting Started
  • Prerequisites
  • Installation
  • Usage
• 🌐 Future Work
• 🙏 Acknowledgements

---

Team Introduction

This project was made by team Infinity Spark as part of the Capstone Project 2025 @RMIT University Vietnam.

Our team members:

• Ho Quoc Thai - s3927025
• Huynh Quoc Bao - s3818557
• Tran Duc Minh - s3818557
• Jeon Mina - s3764040
• Tran Nguyen Minh Lam - s4015153

Project Overview

This project provides a Mixed Reality (MR) solution for the real-time control and monitoring of industrial robot arms. Leveraging the power of Microsoft HoloLens 2, Unity, and ABB RobotStudio, we've crafted an intuitive, immersive 3D interface. This approach drastically enhances user interaction with robotic systems in smart manufacturing environments, pushing beyond traditional 2D interfaces like SCADA systems and control panels.

By seamlessly integrating spatial awareness, real-time data feedback, and intuitive control, our solution offers a modern paradigm for human-robot collaboration, aligning perfectly with the vision of Industry 4.0 and supporting ABB's ambitious digitization goals.

✨ Features

• Immersive 3D Interface: Experience robot interaction through a spatial 3D interface on HoloLens 2, overlaying robot status and controls directly onto the real workspace.
• Control & Monitoring: Gain immediate oversight and command over industrial robot arms.
• Intuitive User Interaction: Enjoy a more natural and direct interaction method, far surpassing traditional panel-based systems.
• Seamless ABB RobotStudio Integration: Connects directly to ABB RobotStudio for virtual commissioning and robust robot programming via RAPID.
• Robust RWS Communication: Utilizes a custom C# communication layer with ABB's Robot Web Services (RWS) RESTful API for reliable data exchange and command execution.
• Dynamic Parameter Feedback: Retrieve critical KPIs such as cycle times, joint velocity, and joint angles in real-time.
• Foundation for Advanced Automation: Designed to be extensible for future functionalities like automated path execution, collision detection, and multi-robot coordination.

🛠️ Technologies Used

• Hardware:
  • Microsoft HoloLens 2
  • Other OpenXR compatible devices (Meta Quest 3s, Pico 4)
• Software/Platforms:
  • Unity (for MR application development)
  • ABB RobotStudio (Simulation & Control Environment)
  • ABB Robot Web Services (RWS) - RESTful API
  • C# (for backend communication logic)

🎥 Demo / Visuals

Witness the power of Mixed Reality in industrial automation:

Click to watch a short video demonstration of the Mixed Reality application controlling an ABB robot arm in RobotStudio.

How it Works

The system is engineered through a sophisticated integration of several interconnected components:

1. Mixed Reality Application (Unity & HoloLens 2): The user-facing MR application is developed in Unity, specifically optimized for the HoloLens 2. It renders a rich 3D interface that overlays critical robot information, control options, and environmental context directly onto the operator's physical workspace.
2. Robot Simulation and Control (ABB RobotStudio): At the core of robot management is ABB RobotStudio, which handles virtual commissioning, detailed robot cell creation, and programming using ABB's RAPID language.
3. Communication Layer (C# & RWS): A bespoke C# backend module acts as the crucial bridge. It leverages ABB's Robot Web Services (RWS) API – a powerful RESTful interface – to transmit control commands from the Unity application to the robot controller and simultaneously pull back essential real-time operational data.
4. Real-Time Data Flow: Commands initiated via the HoloLens 2 are processed by Unity, passed through the C# RWS communication layer, and executed by the robot controller. Concurrently, real-time feedback (e.g., joint states, cycle times) flows back through RWS, enriching the immersive display on the HoloLens 2, providing operators with immediate and actionable insights.

This harmonious integration delivers unparalleled spatial awareness, live data, and intuitive control within a singular, immersive system.

Getting Started

To get this project up and running, follow these steps:

Prerequisites

• A Microsoft HoloLens 2 or any OPENXR Compatible device for the Mixed Reality experience.
• Unity Hub and Unity Editor (2022.x LTS).
• ABB RobotStudio with appropriate licenses for creating and simulating robot cells.
• A stable network connection enabling communication between your development machine, HoloLens 2, and RobotStudio/robot controller.

Installation

1. Clone the Repository:

   git clone https://github.com/your-username/your-repo-name.git
   cd mixed-reality-robot-control
   

2. Unity Project Setup:
   • Open the UnityProject folder in Unity Editor.
   • Go to File > Build Settings, select Universal Windows Platform, and ensure Mixed Reality Feature Tool and OpenXR are correctly configured for HoloLens 2 deployment.
   • Build and deploy the application to your HoloLens 2 device.
3. RobotStudio Configuration:
   • Launch ABB RobotStudio and load/create your robot cell.
   • Ensure the virtual controller (or physical controller) is configured to enable RWS communication. Refer to the ABB RobotStudio documentation for RWS setup.
4. C# Communication Layer:
   • Navigate to the CommunicationLayer directory.
   • Open the solution in Visual Studio and build the project.
   • Run the compiled executable to start the RWS communication server.

Usage

1. Power on your ABB robot (or start the virtual controller in RobotStudio).
2. Ensure the C# RWS communication layer application is running.
3. Launch the MR Robot Control application on your HoloLens 2.
4. The HoloLens application should automatically establish a connection to the robot via the communication layer.
5. You can now visualize the robot's real-time status and interact with its controls through the immersive 3D overlay.

Future Work

This project lays a robust foundation for exciting future enhancements:

• Real-time Collision Detection & Visualization: Integrating spatial mapping data to visualize potential collision zones and provide real-time warnings within the MR environment.
• Multi-Robot Coordination: Expanding the system to enable intuitive control and orchestration of multiple robotic systems simultaneously.
• Advanced AI/ML Integration: Exploring possibilities for predictive maintenance or adaptive control based on real-time data analysis.

Acknowledgements

• Microsoft: For their pioneering work on the HoloLens 2 and the robust Unity development platform.

• ABB Robotics: For providing sophisticated robotic solutions, RobotStudio, and the essential Robot Web Services API.

• We would like to pay our sincerest gratitude to our team academic and industry supervisors for their supports and guidance throughout the project:

• Dr. Nguyen Hai Nguyen for providing us with encouragements and advice along our development

• Mr. Dien Duc Uy Man for providing us exclusive insights, approachs and advice from the industry perspective