benchmark-HDMapping-Orchestration

This benchmark calculates APE (Absolute Pose Error), ARE (Absolute Relative Pose Error) and draws registered trajectories. It does it in single script.

Switch to other branch e.g. Bunker-DVI-Dataset-reg-1 to see results.

Step 1 Prepare data

Create worskpace folder

mkdir -p ~/hdmapping-benchmark

Go to your workspace folder:

cd ~/hdmapping-benchmark

Clone the orchestration repository:

git clone https://github.com/MapsHD/benchmark-HDMapping-Orchestration.git

Available sequences:

seq00 seq01 seq02 seq04 seq05 seq06 seq07 seq08 seq09 seq10

The dataset is available here: KITTI dataset ROS

Make the script executable (if not done yet):

chmod +x ~/hdmapping-benchmark/benchmark-HDMapping-Orchestration/prepare_data_step1/download_data.sh 

Run the script with a sequence number:

~/hdmapping-benchmark/benchmark-HDMapping-Orchestration/prepare_data_step1/download_data.sh seq00

This will download the following into ~/hdmapping-benchmark/data:

kitti_seq00_ros1.bag

kitti_seq00_ros2.zip – ROS2 bag (automatically unzipped into kitti_seq00_ros2/)

Step 2 Clone repositores

Make the script executable (if not done yet):

chmod +x ~/hdmapping-benchmark/benchmark-HDMapping-Orchestration/clone_github_repositories_step2/clone_github_repositories_step2.sh

Run the script:

~/hdmapping-benchmark/benchmark-HDMapping-Orchestration/clone_github_repositories_step2/clone_github_repositories_step2.sh

After running the script, you will be prompted to enter the branch name you want to clone for the repositories.

Result:

The repositories will be cloned into:

~/hdmapping-benchmark

The Docker images required for the benchmark will be built.

Step 3 run benchmark

Make the script executable (if not done yet):

chmod +x ~/hdmapping-benchmark/benchmark-HDMapping-Orchestration/run_benchmark_step3/run_benchmark_step3.sh

Change directory to the data folder:

cd ~/hdmapping-benchmark/data

Run the benchmark script with your ROS1 bag and ROS2 folder:

~/hdmapping-benchmark/benchmark-HDMapping-Orchestration/run_benchmark_step3/run_benchmark_step3.sh kitti_seq00_ros1.bag kitti_seq00_ros2/2011_10_03_drive_0027_extract_ros2/ .

Result:

After running the script, you will get the following folder:

~/hdmapping-benchmark/data/output_hdmapping-ALGONAME/

You should see following data

lio_initial_poses.reg

poses.reg

scan_lio_*.laz

session.json

trajectory_lio_*.csv

Create tag

git tag YEAR_MM_DD_benchmark git push origin --tags

More info

Our paper about benchmark

• https://www.sciencedirect.com/science/article/pii/S2352711026003146

To cite benchmark suite please use as follows:

@article{BEDKOWSKI2026102822,
title = {MapsHD: A benchmark suite for LiDAR odometry frameworks},
journal = {SoftwareX},
volume = {35},
pages = {102822},
year = {2026},
issn = {2352-7110},
doi = {https://doi.org/10.1016/j.softx.2026.102822},
url = {https://www.sciencedirect.com/science/article/pii/S2352711026003146},
author = {Janusz Bȩdkowski and Michał Pełka and Karol Majek and Marcin Matecki and Adrian Radulescu and Charles Hamesse and Ethan Decleyn and Przemysław Lekston and Tomasz Owerko and Przemysław Kuras and Michał Ciszewski and Jakub Kolecki and Karolina Tomaszkiewicz and Łukasz Ambroziński and Joanna Koszyk and Bartosz Hyla and Karolina Pargieła and Anna Malczewska and Tomasz Lipecki and Artur Adamek and Bartosz Mitka and Klapa Przemysław and Pelagia Gawronek and Martin Mokros and Jozef Výboštok and Juliána Chudá and Michal Skladan and Carlos Cabo and Kim André Anstensen and Craciun Daniel-Marian and Antun Jakopec and Michal Wlasiuk and Kornel Mrozowski and Maksymilian Kulicki and Krzysztof Stereńczak and Oskar Bartosz and Jakub Markiewicz and Sławomir Łapiński and Adam Kostrzewa and Mariana Campos and Machi Zawidzki and Jacek Szklarski and Rami Faraj and Loris Redovniković and Jurica Jagetić and Samer Karam and Răzvan Dumbravă and Milosz Mielcarek and Grzegorz Krok and Michal Laszkowski and Jaroslaw Wajs and Jakub Chudziński},
keywords = {LiDAR odometry, LiDAR-inertial odometry, Benchmarking},
abstract = {This paper describes a software toolbox for LiDAR (Light Detection and Ranging) and LiDAR-Inertial Odometry qualitative and quantitative evaluation. We provide software as https://github.com/MapsHD organization with all necessary information at https://github.com/MapsHD/HDMapping. Our software contributions are a) ground truth data processing tool, b) dockerized state-of-the-art LO and LIO algorithms, c) multi-session data registration to common coordinate system, d) Absolute Pose Error (APE) and Relative Pose Error (RPE) metrics, e) import/export tools for easier 3D data handling and visualizing, e.g., in Cloud Compare software. This software is compatible with ROS1 (Robot Operating System) and ROS2 data formats. We show an example benchmark of LeGO-LOAM, LIO-SAM, FAST-LIO, DLO, VoxelMap, Faster-LIO, KISS-ICP, CT-ICP, SLICT, DLIO, GLIM, iG-LIO, LIO-EKF, I2EKF-LO, GenZ-ICP, RESPLE, odometry_ros_wrapper, Point-LIO, and LOAM-Livox algorithms. For all experiments we provide movies. The contribution of the paper is software-oriented LO/LIO algorithm benchmark suite. The novelty lies in the integration of multiple benchmarking steps into a unified framework, thus overall effort needed for qualitative and quantitative evaluation is reduced.}
}