priborИзвестия высших учебных заведений. ПриборостроениеJournal of Instrument Engineering0021-34542500-0381Национальный исследовательский университет ИТМО10.17586/0021-3454-2022-65-3-204-217pribor-235Research ArticleМЕТОДЫ ИСКУССТВЕННОГО ИНТЕЛЛЕКТА В СИСТЕМАХ НАВИГАЦИИ МОБИЛЬНЫХ РОБОТОВARTIFICIAL INTELLIGENCE METHODS IN MOBILE ROBOT NAVIGATION SYSTEMSМетрико-семантическое картирование на основе глубоких нейронных сетей для систем автономной навигации в помещенияхMetric-semantic mapping based on deep neural networks for systems of indoor autonomous navigationБеркаевА. Р.BerkaevA. R.

Беркаев Амиран Рустамович — студент; факультет систем управления и робототехники, лаборатория биомехатроники и энергоэффективной робототехники.

Санкт-Петербург

Amiran R. Berkaev — Student; ITMO University, Faculty of Control Systems and Robotics, International Laboratory of Biomechatronic and Energy-Efficient Robotics.

St. Petersburg

berkaevamiran@mail.ruМохратМ.MohratM.

Мохрат Малик — студент; факультет систем управления и робототехники, лаборатория биомехатроники и энергоэффективной робототехники.

Санкт-Петербург

Malik Mohrat — Student; ITMO University, Faculty of Control Systems and Robotics, International Laboratory of Biomechatronic and Energy-Efficient Robotics.

St. Petersburg

malik.mohrat@gmail.comБурковА. М.BurkovA. M.

Бурков Алексей Михайлович — лаборатория робототехники; ведущий инженер-разработчик.

Москва

Aleхey M. Burkov — Sberbank, Robotics Laboratory; Lead Engineer-Designer.

Moscow

amburkoff@gmail.comКолюбинС. A.KolyubinS. A.

Колюбин Сергей Алексеевич — д-р техн. наук, доцент; факультет систем управления и робототехники, лаборатория биомехатроники и энергоэффективной робототехники; вед. научный сотрудник.

Санкт-Петербург

Sergey A. Kolyubin — Dr. Sci., Associate Professor; ITMO University, Faculty of Control Systems and Robotics, International Laboratory of Biomechatronic and Energy-Efficient Robotics; Leading Researcher.

St. Petersburg

s.kolyubin@itmo.ruУниверситет ИТМОITMO UniversityСбербанкSberbank202201122024653204217Copyright © Национальный исследовательский университет ИТМО, 20242024Национальный исследовательский университет ИТМОНациональный исследовательский университет ИТМОhttps://pribor.ifmo.ru/jour/about/submissions#copyrightNoticehttps://pribor.ifmo.ru/jour/article/view/235

Представлены результаты исследования, направленного на разработку интеллектуальной автономной навигационной системы для складской и офисной логистики с использованием глубоких нейронных сетей. Проанализированы современные и наиболее универсальные средства для получения карт глубин и семантической сегментации данных на изображениях в различных средах. Проведен сравнительный анализ карт глубин, формируемых RGB-D-камерой, а также с помощью нейросетевых алгоритмов и модифицированного алгоритма Хиршмюллера. Результаты тестирования, проведенного на специально подготовленном наборе данных, снятых в офисном пространстве, демонстрируют, что предложенное решение превосходит альтернативные по точности и позволяет сократить затраты вычислительных ресурсов.

Results of a study aimed at developing an intelligent autonomous navigation system for warehouse and office logistics using deep neural networks, are presented. The modern and most versatile tools for depth maps retrieval and semantic data segmentation on images in different environments are analyzed. A comparison of depth maps retrieved hardware from RGB-D camera, neural network algorithms, and a modified Hirschmuller algorithm is carried out. Results of testing performed with a specially prepared dataset shot in an office space, including many complex objects such as glass, mirrors, and multiple light sources demonstrate that the proposed solution outperforms the alternatives in accuracy and uses fewer computational resources in the process.

семантическая сегментациякарты глубинодновременная локализация и картированиеметрико-семантическая картамобильный роботлогистикаглубокие нейронные сетиоценка глубиныинтеллектуальные системыsegmentationdepth mapssimultaneous localization and mappingmetric-semantic mapmobile robotlogisticsdeep neural networkdepth estimationintelligent systemsReferencesZhao C., Sun Q., Zhang C., Tang Y., Qian F. Monocular depth estimation based on deep learning: An overview // Science China Technological Sciences. 2020. Vol. 63, N 9. P. 1612—1627. DOI: 10.1007/s11431-020-1582-8.Zhao C., Sun Q., Zhang C., Tang Y., and Qian F. Science China Technological Sciences, 2020, no. 9(63), pp. 1612–1627, DOI: 10.1007/s11431-020-1582-8.Yin W., Liu Y., Shen C. Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction // IEEE Trans. on Pattern Analysis and Machine Intelligence. 2021. DOI: 10.1109/TPAMI.2021.3097396.Yin W., Liu Y. and Shen C. IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI: 10.1109/TPAMI.2021.3097396.Ranftl R., Lasinger K., Hafner D., Schindler K., Koltun V. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer // IEEE Trans. on Pattern Analysis and Machine Intelligence. 2022. Vol. 44, N 03. P. 1623—1637.Ranftl R., Lasinger K., Hafner D., Schindler K., and Koltun V. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, no. 03(44), pp. 1623–1637.Sinha A., Murez Z., Bartolozzi J., Badrinarayanan V., Rabinovich A. DELTAS: Depth Estimation by Learning Triangulation and Densification of Sparse Points // Computer Vision – ECCV. 2020. Vol. 12366. P. 104—121. DOI: 10.1007/978-3-030-58589-1_7.Sinha A., Murez Z., Bartolozzi J., Badrinarayanan V., and Rabinovich A. Computer Vision – ECCV 2020, Springer International Publishing, Cham, 2020, vol. 12366, pp. 104–121, DOI: 10.1007/978-3-030-58589-1_7.Duzceker A., Galliani S., Vogel C., Speciale P., Dusmanu M., Pollefeys M. DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion // Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA. 2021. June. P. 15324—15333. DOI: 10.1109/CVPR46437.2021.01507.Düzçeker A., Galliani S., Vogel C., Speciale P., Dusmanu M., and Pollefeys M. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, June 2021, pp. 15324–15333, DOI: 10.1109/CVPR46437.2021.01507.Wofk D., Ma F., Yang T.-J., Karaman S., Sze V. FastDepth: Fast monocular depth estimation on embedded systems // Proc. IEEE Intern. Conf. on Robotics and Automation. 2019. Vol. 2019, May. P. 6101—6108. DOI: 10.1109/ICRA.2019.8794182.Wofk D., Ma F., Yang T.-J., Karaman S., and Sze V. Proceedings - IEEE International Conference on Robotics and Automation, 2019, vol. 2019, May, pp. 6101–6108, DOI: 10.1109/ICRA.2019.8794182.Poggi M., Aleotti F., Tosi F., Mattoccia S. Towards Real-Time Unsupervised Monocular Depth Estimation on CPU // IEEE Intern. Conf. on Intelligent Robots and Systems. 2018. P. 5848—5854. DOI: 10.1109/IROS.2018.8593814.Poggi M., Aleotti F., Tosi F., and Mattoccia S. IEEE International Conference on Intelligent Robots and Systems, 2018, pp. 5848–5854, DOI: 10.1109/IROS.2018.8593814.Bokovoy A., Muravyev K., Yakovlev K. Real-time Vision-based Depth Reconstruction with NVidia Jetson // European Conf. on Mobile Robots (ECMR). 2019. P. 1—6. DOI: 10.1109/ECMR.2019.8870936.Bokovoy A., Muravyev K., and Yakovlev K. 2019 European Conference on Mobile Robots (ECMR), 2019, pp. 1–6, DOI: 10.1109/ECMR.2019.8870936.Smolyanskiy N., Kamenev A., Birchfield S. On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach // IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops. 2018. Vol. 2018, June. P. 1120—1128. DOI: 10.1109/CVPRW.2018.00147.Smolyanskiy N., Kamenev A., and Birchfield S. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2018, vol. 2018-June, pp. 1120–1128, doi: 10.1109/CVPRW.2018.00147.Tonioni A., Tosi F., Poggi M., Mattoccia S., L. Stefano D. Real-time self-adaptive deep stereo // Proc. of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. 2019. Vol. 2019. June. P. 195—204. DOI: 10.1109/CVPR.2019.00028.Tonioni A., Tosi F., Poggi M., Mattoccia S., and Stefano L.D. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019, vol. 2019-June, pp. 195–204, DOI: 10.1109/CVPR.2019.00028.Xu H., Zhang J. AANET: Adaptive aggregation network for efficient stereo matching // Proc. of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. 2020. P. 1956—1965. DOI: 10.1109/CVPR42600.2020.00203.Xu H. and Zhang J. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020, pp. 1956–1965, DOI: 10.1109/CVPR42600.2020.00203.Xia L., Cui J., Shen R., Xu X., Gao Y., Li X. A survey of image semantics-based visual simultaneous localization and mapping: Application-oriented solutions to autonomous navigation of mobile robots // Intern. Journal of Advanced Robotic Systems. 2020. Vol. 17. P. 1—17. DOI: 10.1177/1729881420919185.Xia L., Cui J., Shen R., Xu X., Gao Y., and Li X. Int. J. Adv. Robot. Syst., 2020, vol. 17, p. 172988142091918, DOI: 10.1177/1729881420919185.Chen L.-C., Zhu Y., Papandreou G., Schroff F., Adam H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation // Computer Vision – ECCV. 2018. Vol. 11211. P. 833—851. DOI: 10.1007/978-3-030-01234-2_49.Chen L.-C., Zhu Y., Papandreou G., Schroff F., and Adam H. Computer Vision – ECCV, Springer International Publishing, Cham, 2018, vol. 11211, pp. 833–851, DOI: 10.1007/978-3-030-01234-2_49.Ronneberger O., Fischer P., Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation // Medical Image Computing and Computer-Assisted Intervention – MICCAI. 2015. Vol. 9351. P. 234—241. DOI: 10.1007/978-3-319-24574-4_28.Ronneberger O., Fischer P., and Brox T. Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Springer International Publishing, Cham, 2015, vol. 9351, pp. 234–241, DOI: 10.1007/978-3-319-24574-4_28.Wang J., Sun K., Cheng T., Jiang B., Deng C., Zhao Y., Liu D., Mu Y., Tan M., Wang X., Liu W., Xiao B. Deep High-Resolution Representation Learning for Visual Recognition // IEEE Trans. on Pattern Analysis and Machine Intelligence. 2021. Vol. 43, N 10. P. 3349—3364. DOI: 10.1109/TPAMI.2020.2983686.Wang J., Sun K., Cheng T., Jiang B., Deng C., Zhao Y., Liu D., Mu Y., Tan M., Wang X., Liu W. and Xiao B. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, no. 10(43), pp. 3349–3364, DOI: 10.1109/TPAMI.2020.2983686.Badrinarayanan V., Kendall A., Cipolla R. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation // IEEE Trans. on Pattern Analysis and Machine Intelligence. 2017. Vol. 39, N 12. P. 2481—2495. DOI: 10.1109/TPAMI.2016.2644615.Badrinarayanan V., Kendall A., and Cipolla R. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, no. 12(39), pp. 2481–2495, DOI: 10.1109/TPAMI.2016.2644615.Seichter D., Köhler M., Lewandowski B., Wengefeld T., Gross H. -M. Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis // IEEE Intern. Conf. on Robotics and Automation (ICRA). 2021. P. 13525—13531. DOI: 10.1109/ICRA48506.2021.9561675.Seichter D., Köhler M., Lewandowski B., Wengefeld T., and Gross H.-M. 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021, pp. 13525–13531, DOI: 10.1109/ICRA48506.2021.9561675.He K., Zhang X., Ren S., Sun J. Deep Residual Learning for Image Recognition // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2016. P. 770—778. DOI: 10.1109/CVPR.2016.90.He K., Zhang X., Ren S., and Sun J. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778, DOI: 10.1109/CVPR.2016.90.Oleynikova H., Taylor Z., Fehr M., Siegwart R., Nieto J. Voxblox: Incremental 3D Euclidean Signed Distance Fields for on-board MAV planning // IEEE/RSJ Intern. Conf. on Intelligent Robots and Systems (IROS). 2017. P. 1366—1373. DOI: 10.1109/IROS.2017.8202315.Oleynikova H., Taylor Z., Fehr M., Siegwart R., and Nieto J. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 1366–1373, DOI: 10.1109/IROS.2017.8202315.Rosinol A., Abate M., Chang Y., Carlone L. Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping // IEEE Intern. Conf. on Robotics and Automation (ICRA). 2020. P. 1689—1696. DOI: 10.1109/ICRA40945.2020.9196885.Rosinol A., Abate M., Chang Y., and Carlone L. 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 1689–1696, DOI: 10.1109/ICRA40945.2020.9196885.Xiang Y., Fox D. DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks // Robotics: Science and Systems. 2017. Vol. 13. DOI: 10.15607/RSS.2017.XIII.013.Xiang Y. and Fox D. 2017 Robotics: Science and Systems, 2017, july, vol. 13, DOI: 10.15607/RSS.2017.XIII.013.Grinvald M. et al. Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery // IEEE Robotics and Automation Letters. 2019. Vol. 4, N 3. P. 3037—3044. DOI: 10.1109/LRA.2019.2923960.Grinvald M. et al. IEEE Robot. Autom. Lett., 2019, no. 3(4), pp. 3037–3044, DOI: 10.1109/LRA.2019.2923960.Reijgwart V., Millane A., Oleynikova H., Siegwart R., Cadena C., Nieto J. Voxgraph: Globally Consistent, Volumetric Mapping Using Signed Distance Function Submaps // IEEE Robotics and Automation Letters. 2020. Vol. 5, N 1. P. 227—234. DOI: 10.1109/LRA.2019.2953859.Reijgwart V., Millane A., Oleynikova H., Siegwart R., Cadena C., and Nieto J. IEEE Robot. Autom. Lett., 2020, no. 1(5), pp. 227–234, DOI: 10.1109/lra.2019.2953859.Zendel O., Honauer K., Murschitz M., Steininger D., Dominguez G. F. WildDash — Creating Hazard-Aware Benchmarks // Proc. of the European Conf. on Computer Vision (ECCV). 2018. Sept. P. 402—416.Zendel O., Honauer K., Murschitz M., Steininger D., and Dominguez G.F. Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 402–416.Lambert J., Liu Z., Sener O., Hays J., Koltun V. MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). 2020. P. 2876—2885. DOI: 10.1109/CVPR42600.2020.00295.Lambert J., Liu Z., Sener O., Hays J., and Koltun V. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 2876–2885, DOI: 10.1109/CVPR42600.2020.00295.Kirillov A., Wu Y., He K., Girshick R. PointRend: Image Segmentation As Rendering // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). 2020. P. 9796—9805. DOI: 10.1109/CVPR42600.2020.00982.Kirillov A., Wu Y., He K. and Girshick R. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 9796–9805, DOI: 10.1109/CVPR42600.2020.00982.

The authors declare that there are no conflicts of interest present.