Skip to content
HumanoidsBonnPublic

About

Evidential multi-task learning for joint depth estimation and semantic segmentation

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

EvidMTL

EvidMTL extends SwinMTL with evidential depth estimation and semantic segmentation. This public release corresponds to the pre-depth-completion wonka source snapshot used by the combined Evidential Mapping project.

What is new in EvidMTL

Relative to the upstream SwinMTL architecture, this work adds:

  • evidential heads for joint semantic segmentation and monocular depth;
  • semantic and depth uncertainty estimates alongside the task predictions;
  • an evidential SiLog depth objective with uncertainty regularization and a Normal-Inverse-Gamma prior; and
  • the model outputs used by EvidKimera for uncertainty-aware semantic TSDF fusion.

SwinMTL remains the underlying shared encoder-decoder architecture. The original SwinMTL authors and paper are credited below; the modifications above are described in the EvidMTL paper.

Model checkpoints are hosted separately on Hugging Face:

  • silog_klmuprior_alpha1.01_gamma0.001_betacalc_final_model.ckpt uses a noisy ground-truth mean prior.
  • silog_klmuprior_alpha2_beta0.1_gamma1.0_no_noise_final_model.ckpt uses the ground-truth mean directly as the prior.

The source code remains licensed under GPL-3.0; see LICENSE.

Citation and attribution

If you use this repository, please cite both EvidMTL and the upstream SwinMTL paper:

@inproceedings{menon2025evidmtl,
  author={Menon, Rohit and Dengler, Nils and Pan, Sicong and Chenchani, Gokul Krishna and Bennewitz, Maren},
  title={EvidMTL: Evidential Multi-Task Learning for Uncertainty-Aware Semantic Surface Mapping from Monocular RGB Images},
  booktitle={2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2025},
  doi={10.1109/IROS60139.2025.11247528}
}

@inproceedings{taghavi2024swinmtl,
  author={Taghavi, Pardis and Langari, Reza and Pandey, Gaurav},
  title={SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images},
  booktitle={2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year={2024},
  url={https://arxiv.org/abs/2403.10662}
}

SwinMTL

[IROS24] SwinMTL: Multi-Task Learning with Swin Transformer

A multi-task learning framework designed for simultaneous depth estimation and semantic segmentation using the Swin Transformer architecture.

Project Page Paper Papers with Code

News

  • [30th June] Paper Accepted at the IROS 2024 Conference 🔥🔥🔥
Qualitative Results

Installation

To get started, follow these steps:

  1. Only for ROS installation (otherwise skip this part)

    cd catkin_ws/src
    catkin_create_pkg SwinMTL_ROS std_msgs rospy
    cd ..
    catkin_make
    source devel/setup.bash
    cd src/SwinMTL_ROS/src
    git clone https://github.com/PardisTaghavi/SwinMTL.git
    chmod +x inference_ros.py
    mv ./launch/ ./..  
  2. Clone the repository:

    git clone https://github.com/PardisTaghavi/SwinMTL.git
    cd SwinMTL
  3. Create a conda environment and activate it:

    conda env create --file environment.yml
    conda activate prc

Testing

To run the testing for the project, follow the below steps:

  1. Download the required EvidMTL checkpoint from the Hugging Face model repository.

  2. Move Pretrained Models:

    • Create a weights directory and place the EvidMTL checkpoint there.
    • Third-party Swin backbone checkpoints expected under model_zoo are not redistributed by this repository.
    • Refer to testLive.ipynb for testing.

ROS Launch

roslaunch SwinMTL_ROS swinmtl_launch.launch
Zero-shot Results

3D Mapping

3D Mapping Results

Zero-shot Results on the Kitti Dataset

Zero-shot Results

Contributions

  • Introduction of a multi-task learning approach for joint depth estimation and semantic segmentation.
  • Achievement of state-of-the-art performance on Cityscapes and NYUv2 datasets.
  • Utilization of an efficient shared encoder-decoder architecture coupled with novel techniques to enhance accuracy.

We welcome feedback and contributions to the SwinMTL project. Feel free to contact taghavi.pardis@gmail.com.


Acknowledgments

Special thanks to the authors of the following projects for laying the foundation of this work. Our code relies on:

About

Evidential multi-task learning for joint depth estimation and semantic segmentation

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages