ISSN 0021-3454 (print version)
ISSN 2500-0381 (online version)
Menu

2
Issue
vol 67 / February, 2024
Article

DOI 10.17586/0021-3454-2022-65-3-204-217

UDC 004.896 + 007.52 + 004.93’1

METRIC-SEMANTIC MAPPING BASED ON DEEP NEURAL NETWORKS FOR SYSTEMS OF INDOOR AUTONOMOUS NAVIGATION

A. R. Berkaev
ITMO University, Faculty of Control Systems and Robotics, International Laboratory of Biomechatronic and Energy-Efficient Robotics;


M. Mohrat
ITMO University, Faculty of Control Systems and Robotics, International Laboratory of Biomechatronic and Energy-Efficient Robotics;


A. M. Burkov
Sberbank, Robotics Laboratory;


S. A. Kolyubin
ITMO University, Saint Petersburg, 197101, Russian Federation; Associate Professor


Read the full article 

Abstract. Results of a study aimed at developing an intelligent autonomous navigation system for warehouse and office logistics using deep neural networks, are presented. The modern and most versatile tools for depth maps retrieval and semantic data segmentation on images in different environments are analyzed. A comparison of depth maps retrieved hardware from RGB-D camera, neural network algorithms, and a modified Hirschmuller algorithm is carried out. Results of testing performed with a specially prepared dataset shot in an office space, including many complex objects such as glass, mirrors, and multiple light sources demonstrate that the proposed solution outperforms the alternatives in accuracy and uses fewer computational resources in the process.
Keywords: segmentation, depth maps, simultaneous localization and mapping, metric-semantic map, mobile robot, logistics, deep neural network, depth estimation, intelligent systems

References:
  1. Zhao C., Sun Q., Zhang C., Tang Y., and Qian F. Science China Technological Sciences, 2020, no. 9(63), pp. 1612–1627, DOI: 10.1007/s11431-020-1582-8.
  2. Yin W., Liu Y. and Shen C. IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI: 10.1109/TPAMI.2021.3097396.
  3. Ranftl R., Lasinger K., Hafner D., Schindler K., and Koltun V. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, no. 03(44), pp. 1623–1637.
  4. Sinha A., Murez Z., Bartolozzi J., Badrinarayanan V., and Rabinovich A. Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, vol. 12366, pp. 104–121, DOI: 10.1007/978-3-030-58589-1_7.
  5. Düzçeker A., Galliani S., Vogel C., Speciale P., Dusmanu M., and Pollefeys M. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, June 2021, pp. 15324–15333, DOI: 10.1109/CVPR46437.2021.01507.
  6. Wofk D., Ma F., Yang T.-J., Karaman S., and Sze V. Proceedings - IEEE International Conference on Robotics and Automation, 2019, vol. 2019, May, pp. 6101–6108, DOI: 10.1109/ICRA.2019.8794182.
  7. Poggi M., Aleotti F., Tosi F., and Mattoccia S. IEEE International Conference on Intelligent Robots and Systems, 2018, pp. 5848–5854, DOI: 10.1109/IROS.2018.8593814.
  8. Bokovoy A., Muravyev K., and Yakovlev K. 2019 European Conference on Mobile Robots (ECMR), 2019, pp. 1–6, DOI: 10.1109/ECMR.2019.8870936.
  9. Smolyanskiy N., Kamenev A., and Birchfield S. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2018, vol. 2018-June, pp. 1120–1128, doi: 10.1109/CVPRW.2018.00147.
  10. Tonioni A., Tosi F., Poggi M., Mattoccia S., and Stefano L.D. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, vol. 2019-June, pp. 195–204, DOI: 10.1109/CVPR.2019.00028.
  11. Xu H. and Zhang J. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 1956–1965, DOI: 10.1109/CVPR42600.2020.00203.
  12. Xia L., Cui J., Shen R., Xu X., Gao Y., and Li X. Int. J. Adv. Robot. Syst., 2020, vol. 17, p. 172988142091918, DOI: 10.1177/1729881420919185.
  13. Chen L.-C., Zhu Y., Papandreou G., Schroff F., and Adam H. Computer Vision – ECCV, Springer International Publishing, Cham, 2018, vol. 11211, pp. 833–851, DOI: 10.1007/978-3-030-01234-2_49.
  14. Ronneberger O., Fischer P., and Brox T. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Springer International Publishing, Cham, 2015, vol. 9351, pp. 234–241, DOI: 10.1007/978-3-319-24574-4_28.
  15. Wang J., Sun K., Cheng T., Jiang B., Deng C., Zhao Y., Liu D., Mu Y., Tan M., Wang X., Liu W. and Xiao B. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, no. 10(43), pp. 3349–3364, DOI: 10.1109/TPAMI.2020.2983686.
  16. Badrinarayanan V., Kendall A., and Cipolla R. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, no. 12(39), pp. 2481–2495, DOI: 10.1109/TPAMI.2016.2644615.
  17. Seichter D., Köhler M., Lewandowski B., Wengefeld T., and Gross H.-M. 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 13525–13531, DOI: 10.1109/ICRA48506.2021.9561675.
  18. He K., Zhang X., Ren S., and Sun J. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778, DOI: 10.1109/CVPR.2016.90.
  19. Oleynikova H., Taylor Z., Fehr M., Siegwart R., and Nieto J. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 1366–1373, DOI: 10.1109/IROS.2017.8202315.
  20. Rosinol A., Abate M., Chang Y., and Carlone L. 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 1689–1696, DOI: 10.1109/ICRA40945.2020.9196885.
  21. Xiang Y. and Fox D. 2017 Robotics: Science and Systems, 2017, july, vol. 13, DOI: 10.15607/RSS.2017.XIII.013.
  22. Grinvald M. et al. IEEE Robot. Autom. Lett., 2019, no. 3(4), pp. 3037–3044, DOI: 10.1109/LRA.2019.2923960.
  23. Reijgwart V., Millane A., Oleynikova H., Siegwart R., Cadena C., and Nieto J. IEEE Robot. Autom. Lett., 2020, no. 1(5), pp. 227–234, DOI: 10.1109/lra.2019.2953859.
  24. Zendel O., Honauer K., Murschitz M., Steininger D., and Dominguez G.F. Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 402–416.
  25. Lambert J., Liu Z., Sener O., Hays J., and Koltun V. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2876–2885, DOI: 10.1109/CVPR42600.2020.00295.
  26. Kirillov A., Wu Y., He K. and Girshick R. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 9796–9805, DOI: 10.1109/CVPR42600.2020.00982.