Abstract: SLAM has long been envisioned as a means to assist robots performing navigation tasks in unknown environments. Yet, benchmarking for SLAM most often uses open-loop data sets due for reproducibility. Given that open-loop and closed-loop task performance measures may not align, establishing a means to use SLAM in the closed-loop for benchmarking is an important next step in algorithm development. A ROS/Gazebo environment for SLAM benchmarking has been created and open sourced. Several stereo visual-inertial implementations are tested and compared. The Good Feature SLAM approach provides competitive and robust closed-loop trajectory tracking in unknown environments and without external absolute position measurements.

Abstract: Systematic study of SLAM might benefit from a design of experiments type of benchmarking approach, as well as means to gauge best possible performance versus nominal performance. One way to better understand SLAM systems is if a good set of benchmarks could be established for quickly gauging general performance. Using decision trees dominant SLAM benchmark and SLAM performance factors can be easily determined. They can then inform algorithm development. On the best possible performance, we advocate for the use of slo-mo processing, which provides practically no per-frame time constraint on the SLAM solver. The slo-mo results should be upper bounds on SLAM performance compared to the same method run with real-time constraints.