Beyond Visual Realism: Evaluating Geometric Fidelity in Neural Scene Reconstruction for Industrial Inspection

This repository provides an automated pipeline for COLMAP-based preprocessing and quantitative evaluation of depth maps generated by 3D Gaussian Splatting (3DGS) and Neural Radiance Fields (NeRF) against LiDAR Ground-Truth (GT) data (supporting benchmark datasets like ETH3D and RGBD).

---

Requirements & Installation

Install the required dependencies using pip:

pip install -r requirements.txt

---

Usage & Pipeline (main.py)

The main entry point main.py is controlled via the --module argument:

• colmap_service: Preprocesses frames, handles camera poses, and aligns sparse reconstructions via COLMAP.
• compute_metrics: Computes error metrics (MAE, RMSE, and percentiles) and generates visual/distribution plots.

---

1. COLMAP Preprocessing & Pose Alignment

To run the COLMAP service for frame processing and sparse reconstruction:

python main.py --module colmap_service \
  --dataset_mode ETH3D \
  --images_path ./data/eth3d/images \
  --lidar_depth_path ./data/eth3d/ground_truth_depth \
  --frames_json_path ./data/frames.json \
  --output_path ./output/colmap_results \
  --colmap_camera_model OPENCV

---

2. Frames JSON Format (--frames_json_path)

The --frames_json_path parameter expects a JSON configuration file containing metadata for each image frame (such as frame resource ID, full image path, LiDAR reference flags, and 3D world positions).

Example frames.json Structure:

[
  {
    "resource_id": "DSC_0641",
    "full_path": "data/pipes/images/dslr_images\\DSC_0641.JPG",
    "from_lidar": false,
    "worldPosition": [0.5088199780415393, 1.1742453622789228, 0.18602883779739118],
    "phi": 0.0,
    "theta": 0,
    "image_processing_cubemap_size": null,
    "image_processing_cubemap_fov": null
  },
  {
    "resource_id": "DSC_0644",
    "full_path": "data/pipes/images/dslr_images\\DSC_0644.JPG",
    "from_lidar": true,
    "worldPosition": [-0.013847971068348834, -0.025708707042092573, 0.18889821881016786],
    "phi": 0.0,
    "theta": 0,
    "image_processing_cubemap_size": null,
    "image_processing_cubemap_fov": null
  }
]

Key Fields:

• resource_id: Unique identifier for the frame.
• full_path: Path to the image file.
• from_lidar: Boolean flag indicating if the view originates from a LiDAR reference pose.
• worldPosition: [x, y, z] 3D position coordinates of the camera in world space.
• phi / theta: Additional camera orientation parameters (working with photospheres).

---

3. Metric Evaluation (LiDAR GT vs. Predictions)

To evaluate depth map predictions (.npy or .tiff) from 3DGS or NeRF against LiDAR Ground-Truth and compute error metrics (MAE, RMSE, Percentiles):

python main.py --module compute_metrics \
  --metrics_gt_path ./data/eth3d/ground_truth_depth \
  --metrics_predicted_path ./data/eth3d/predictions \
  --metrics_output_path ./output/metrics_results.csv \
  --metrics_max_dist 5.0 \
  --metrics_plot_mae

---

CLI Options & Arguments

General Parameters

• --module: Execution module (help, colmap_service, compute_metrics). Default: help.
• --data_path: Base directory path for input data. Default: data.
• --workspace_path_override: Custom working directory (if omitted, a temporary workspace is created and discarded).

COLMAP Module (--module colmap_service)

• --dataset_mode: Target dataset format (ETH3D, RGBD, OTHER). Default: eth3d.
• --frames_json_path: Path to the JSON dictionary defining frame metadata and 3D world positions.
• --images_path: Path to the RGB image directory.
• --lidar_depth_path: Path to the LiDAR Ground-Truth depth maps.
• --lidar_rgb_path: Path to the LiDAR RGB images (if applicable).
• --poses_filepath: Path to sparse COLMAP poses file in meters (required if frames_json_path is None).
• --output_path: Directory path to save output odometry/pose results.
• --colmap_camera_model: Camera model for COLMAP processing (e.g., OPENCV).
• --colmap_camera_params: Specific camera intrinsic parameters (if applicable).
• --margem_convex_hull: Convex-Hull boundary margin in meters. Default: 1.0.

Metrics Module (--module compute_metrics)

• --metrics_gt_path: Directory containing Ground-Truth depth maps in RGBD format.
• --metrics_predicted_path: Directory containing inferred depth maps (*.npy or *.tiff).
• --metrics_output_path: File path to save evaluation results CSV.
• --metrics_max_dist: Maximum distance threshold for clipping/masking invalid depth values.
• --metrics_plot_mae: Flag to generate and save MAE error distribution plots.

---

Recommended Directory Structure (ETH3D Example)

data/
├── frames.json               # Frame metadata & camera positions
└── eth3d/
    ├── images/               # RGB camera frames
    ├── ground_truth_depth/   # LiDAR GT depth maps
    └── predictions/          # 3DGS / NeRF depth predictions (.npy or .tiff)

---

License

This project is licensed under the MIT License.