priborИзвестия высших учебных заведений. ПриборостроениеJournal of Instrument Engineering0021-34542500-0381Национальный исследовательский университет ИТМО10.17586/0021-3454-2023-66-11-926-935pribor-220Research ArticleСИСТЕМНЫЙ АНАЛИЗ, УПРАВЛЕНИЕ И ОБРАБОТКА ИНФОРМАЦИИSYSTEM ANALYSIS, MANAGEMENT AND INFORMATION PROCESSINGОбзор методов оптимизации бинарных нейронных сетейReview on Optimization Techniques of Binary Neural NetworksШаккуфА.ShakkoufA.

Али Шаккуф, аспирант; факультет систем управления и робототехники

Санкт-Петербург

Ali Shakkouf, Post-Graduate student; ITMO University, Faculty of Control Systems and Robotics

St. Petersburg

ashakkuf@itmo.ruУниверситет ИТМОITMO University2023011220246611926935Copyright © Национальный исследовательский университет ИТМО, 20242024Национальный исследовательский университет ИТМОНациональный исследовательский университет ИТМОhttps://pribor.ifmo.ru/jour/about/submissions#copyrightNoticehttps://pribor.ifmo.ru/jour/article/view/220

Развертывание моделей сверточных нейронных сетей (СНС) во встраиваемых системах ос ложнено множеством проблем, связанных с вычислительной мощностью, энергопотреблением и объемом памя ти. Для решения этих проблем в 2016 г. создан многообещающий тип нейронных сетей, использующих 1-битную активацию и веса, — бинарные нейронные сети (БНС). Такие сети потребляют меньше энергии и вычислитель ных мощностей, так как заменяют сложную операцию тяжелой свертки простыми побитовыми операциями. Одна ко квантование с 32-разрядной плавающей запятой до 1 бита приводит к потере точности и снижению производи тельности, особенно при больших наборах данных. Представлен обзор ключевых методов оптимизации, которые повлияли на производительность БНС и привели к повышению репрезентативности их моделей, также представ лены обзор способов применения БНС в задачах обнаружения объектов и сравнительный анализ их производи тельности с реальным значением.

The deployment of Convolutional Neural Networks (CNNs) models on embedded systems faces mul tiple problems regarding computation power, power consumption and memory footprint. To solve these problems, a promising type of neural networks that uses 1-bit activations and weights emerged in 2016 called Binary Neural Net works (BNNs). BNN consumes less energy and computation power mainly because it replaces the complex heavy convo lution operation with simple bitwise operations. However, the quantization from 32-float point to 1-bit leads to accuracy loss and poor performance, especially on large datasets. This article presents a review of the key optimization techniques which influenced the performance of BNNs and led to higher representation capacity of BNN models, as well as an overview of the application methods of BNNs in object detection tasks and compares the performance with the real value CNN.

бинарные нейронные сетиоптимизация БНСобнаружение объектовквантованиебинаризациякомпьютерное зрениеискусственный интеллектbinary neural networksBNNs optimizationobject detectionquantizationbinarizationcomputer visionartificial intelligenceReferencesBasha S. S., Dubey S. R., Pulabaigari V. and Mukherjee S. Impact of fully connected layers on performance of convolutional neural networks for image classification // Neurocomputing. 2020. Vol. 378. P. 112—119.Basha S.S., Dubey S.R., Pulabaigari V., and Mukherjee S. Neurocomputing, 2020, vol. 378, pp. 112–119.Zhou W., Wang H. and Wan Z. Ore image classification based on improved CNN // Computers and Electrical Engineering. 2022. Vol. 99, art. N 107819.Zhou W., Wang H., and Wan Z. Computers and Electrical Engineering, 2022, vol. 99, art. no. 107819.Gao X., Xing G., Roy S., and LiuH. RAMP-CNN: A novel neural network for enhanced automotive radar object recognition // IEEE Sensors J. 2021. Vol. 21, N 4. P. 5119—5132.Gao X., Xing G., Roy S., and Liu H. RIEEE Sensors J., 2021, no. 4(21), pp. 5119–5132.Ashiq F., Asif M., Ahmad M. B., Zafar S., Masood K., Mahmood T., Mahmood M. T. and LeeI. H. CNN-based object recognition and tracking system to assist visually impaired people // IEEE Access. 2022. Vol. 10. P. 14819—14834.Ashiq F., Asif M., Ahmad M. B., Zafar S., Masood K., Mahmood T., Mahmood M.T., and Lee I.H. IEEE Access, 2022, vol. 10, pp. 14819–14834.Abdelhamid A. A., El-Kenawy E.-S.-M., Alotaibi B., Amer G. M., Abdelkader M. Y., Ibrahim A. and Eid M. M. Robust speech emotion recognition using CNN+LSTM based on stochastic fractal search optimization algorithm // IEEE Access. 2022. Vol. 10. P. 49265—49284.Abdelhamid A.A., El-Kenawy E.-S.-M., Alotaibi B., Amer G.M., Abdelkader M.Y., Ibrahim A., and Eid M.M. IEEE Access., 2022, vol. 10, pp. 49265–49284.Kwon S. MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach // Expert Systems with Applications. 2021. Vol. 167, art. N 114177.Kwon S. Expert Systems with Applications, 2021, vol. 167, art. no. 114177.Zhang N., Wei X., Chen H. and Liu W. FPGA implementation for CNN-based optical remote sensing object detection // Electronics. 2021. Vol. 10, N 3. P. 282.Zhang N., Wei X., Chen H., and Liu W. Electronics, 2021, vol. 10, no. 3, p. 282.Lopac N., Hrzic F., Vuksanovic I. P. and Lerga J. Detection of non-stationary GW signals in high noise from Cohen’s class of time–frequency representations using deep learning // IEEE Access. 2022. Vol. 10. P. 2408—2428.Lopac N., Hrzic F., Vuksanovic I.P., and Lerga J. IEEE Access., 2022, vol. 10, pp. 2408–2428.Horowitz M. Computing’s Energy Problem (and what we can do about it) // IEEE Intern. Solid State Circuits Conf. 2014. P. 10—14.Horowitz M. IEEE Intern. Solid State Circuits Conf., 2014, pp. 10–14.Courbariaux M. and BengioY. BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1 // ArXiv Journal. 2016. preprint arXiv: 1602.02830.Courbariaux M. and Bengio Y. ArXiv journal, 2016, preprint arXiv: 1602.02830.Wang E., Davis J. J., Moro D., Zielinski P., Lim J. J., Coelho C., Chatterjee S., Cheung P. Y., Constantinides G. A. Enabling binary neural network training on the edge // 5th Intern. Workshop on Embedded and Mobile Deep Learning. 2021. June. P. 37—38.Wang E., Davis J.J., Moro D., Zielinski P., Lim J.J., Coelho C., Chatterjee S., Cheung P.Y., Constantinides G.A. ACM Transactions on Embedded Computing Systems, 2021, DOI:10.1145/3626100.Sohoni N. S., Aberger C. R., Leszczynski M., Zhang J. and CRé. Low-memory neural network training: A technical report // ArXiv journal. 2019. preprint arXiv:1904.10631.Sohoni N.S., Aberger C.R., Leszczynski M., Zhang J., and Ré C. ArXiv journal, 2019, preprint arXiv:1904.10631.Courbariaux M., Bengio Y. and Jean-Pierre D. Binaryconnect: Training deep neural networks with binary weights during propagations // ArXiv e-prints. 2015. abs/1511.00363.Courbariaux M., Bengio Y., and David J.-P. ArXiv e-prints, 2015, abs/1511.00363.Hinton G. Neural networks for machine learning // Coursera (video lectures). 2012.Hinton G. Neural networks for machine learning, Coursera, video lectures, 2012.Bengio Yo. Estimating or propagating gradients through stochastic neurons // Technical Report arXiv:1305.2982. Universite de Montreal. 2013.Bengio Y. Technical Report, Universite de Montreal, arXiv:1305.2982, 2013.Darabi S., Belbahri M., Courbariaux M. and Nia V. P. BNN+: Improved binary network training. 2019. https://openreview.net/pdf?id=SJfHg2A5tQ.Darabi S., Belbahri M., Courbariaux M., and Nia V.P. BNN+: Improved binary network training, 2019, https://openreview.net/pdf?id=SJfHg2A5tQ.Shen M., Han K., Xu C., Wang Y. Searching for accurate binary neural architectures // IEEE/CVF Intern. Conf. on Computer Vision Workshops. 2019.Shen M., Han K., Xu C., Wang Y. IEEE/CVF Intern. Conf. on Computer Vision Workshops, 2019.Wang Z., Lu J., Tao C., Zhou J., Tian Q. Learning channel-wise interactions for binary convolutional neural networks // IEEE/CVF Conf. on Computer Vision and Pattern Recognition. 2019. P. 568.Wang Z., Lu J., Tao C., Zhou J., Tian Q. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, vol. 568.Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network // NIPS. 2017. P. 344—352.Lin X., Zhao C., and Pan W. NIPS, 2017, pр. 344–352.Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm // arXiv preprint. 2018. arXiv:1808.00278.Liu Z., Wu B., Luo W., Yang X., Liu W., and Cheng K.-T. arXiv preprint, 2018, arXiv:1808.00278.Shen M., Liu X., Gong R., Han K. Balanced binary neural networks with gated residual // IEEE Intern. Conf. on Acoustics, Speech and Signal Processing (ICASSP) 2020. 4197.Shen M., Liu X., Gong R., Han K. IEEE Intern. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020, vol. 4197.Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao and Tao Mei. dabnn: A super fast inference framework for binary neural networks on arm devices // arXiv preprint. 2019 arXiv:1908.05858.Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. arXiv preprint, 2019, arXiv:1908.05858.Cai Z.; He X.; Sun J. and Vasconcelos N. Deep learning with low precision by half-wave gaussian quantization // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2017.Cai Z., He X., Sun J., and Vasconcelos N. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.Ioffe S. and Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift // Proc. of the 32nd Intern. Conf. on Machine Learning, ICML. 2015. P. 448—456.Ioffe S., and Szegedy C. Proc. of the 32nd Intern. Conf. on Machine Learning, ICML, 2015, рр. 448–456.Wang P., He X., Li G., Zhao T., Cheng J. Sparsity-inducing binarized neural networks // AAAI Conf. on Artificial Intelligence. 2020. N 34. P. 12192.Wang P., He X., Li G., Zhao T., Cheng J. AAAI Conf. on Artificial Intelligence, 2020, vol.34, рр. 12192.Yang J., Shen X., Xing J., Tian X., Li H., Deng B., Huang J. and Hua X.-s. Quantization networks // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019.Yang J., Shen X., Xing J., Tian X., Li H., Deng B., Huang J., and Hua X.-S. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.Zhu S., Dong X. and Su H. Binary ensemble neural network: More bits per network or more networks per bit? // IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2019.Zhu S., Dong X., and Su H. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.Anderson A. G. and Berg C. P. The High-Dimensional Geometry of Binary Neural Networks // ArXiv Journal. 2017. abs/1705.07199.Anderson A.G. & Berg C.P. ArXiv journal, 2017, abs/1705.07199.Qin Haotong, Gong Ruihao, Liu Xianglong, Bai Xiao, Song Jingkuan, Sebe Nicu. Binary Neural Networks: A Survey // Pattern Recognition. 2020. N 105. 107281. DOI: 10.1016/ j.patcog.107281.Qin H., Gong R., Liu X., Bai X., Song J., Sebe N. Pattern Recognition, 2020, 105. 107281. 10.1016/j.patcog.107281.Liang T., Glossner J., Wang L., Shi S. and Zhang X. Pruning and quantization for deep neural network acceleration: A survey // Neurocomputing. 2021. Vol. 461. P. 370—403.Liang T., Glossner J., Wang L., Shi S., and Zhang X. Neurocomputing, 2021, vol. 461, pp. 370–403.Xu Sheng, Liu Chang, Zhang Baochang, Lu Jinhu, Guo Guodong, Doermann D. BiRe-ID: Binary Neural Network for Efficient Person Re-ID // ACM Trans. on Multimedia Computing, Communications, and Applications. 2022. N 18. DOI: 10.1145/3473340.Xu Sh., Liu Ch., Zhang B., Lu J., Guo G, Doermann D. ACM Transactions on Multimedia Computing, Communications, and Applications, 2022, 18. 10.1145/3473340.Zhang K., Yin M. and Wang Y. Why Quantization Improves Generalization: NTK of Binary Weight Neural Networks // ArXiv Journal. 2022. abs/2206.05916.Zhang K., Yin M., & Wang Y. ArXiv journal, 2022, abs/2206.05916.Darabi S., Belbahri M., Courbariaux M. and Nia V. P. Regularized binary network training // ArXiv Journal. 2018, 1812.11800.Darabi S., Belbahri M., Courbariaux M., and Nia V.P. ArXiv journal, 2018, 1812.11800.Wang S., Zhang C., Su D., Wang L., Jiang H. High-precision binary object detector based on a bsf-xnor convolutional layer // IEEE Access 9. 2021. P. 106169.Wang S., Zhang C., Su D., Wang L., Jiang H. IEEE Access 9, 2021, рр. 106169.Rastegari M., Ordonez V., Redmon J. and Farhadi A. Xnor-net: Imagenet classification using binary convolutional neural networks // ECCV. 2016. P. 525— 542.Rastegari M, Ordonez V, Redmon J, and Farhadi A. ECCV, 2016, pр. 525– 542.Barry D., Shah M., Keijsers M., Khan H. and Hopman B. XYOLO: A model for real-time object detection in humanoid soccer on low-end hardware // ArXiv Journal: 1910.03159. 2019.Barry D., Shah M., Keijsers M., Khan H., and Hopman B. ArXiv journal: 1910.03159, 2019.Chen Hanlin, Zhuo Li'an, Zhang Baochang, Zheng Xiawu, Liu Jianzhuang, Ji Rongrong, Doermann D., Guo Guodong. Binarized Neural Architecture Search for Efficient Object Recognition // ArXiv Journal. 2020.Chen H., Zhuo L., Zhang B., Zheng X., Liu J., Ji R., Doermann D., Guo G. ArXiv journal, 2020.Kung Jaeha, Zhang David, van der Wal G. Chai, Sek Mukhopadhyay S. Efficient Object Detection Using Embedded Binarized Neural Networks // Journal of Signal Processing Systems. 2018. N 90. P. 1—14. DOI: 10.1007/s11265 017-1255-5.Kung J., Zhang D., van der Wal G., Chai S., Mukhopadhyay S. Journal of Signal Processing Systems, 2018, vol. 90, рр. 1–14, DOI: 10.1007/s11265-017-1255-5.Wang Xingang, Siyang Sun, Yin Yingjie, Xu De, Wu Wenqi, Gu Qingyi. Fast Object Detection Based on Binary Deep Convolution Neural Networks // CAAI Trans. on Intelligence Technology. 2018. N 3. DOI: 10.1049/trit.1026.Wang X., Siyang S., Yin Y., Xu D. & Wu W., Gu Q. CAAI Transactions on Intelligence Technology, 2018, no. 3(4), DOI:10.1049/trit.2018.1026.Kim H., Choi K. The implementation of a power efficient BCNN based object detection acceleration on a Xilinx FPGA-SoC // Intern. Conf. Internet Things (iThings) IEEE Green Comput. Commun. (GreenCom) IEEE Cyber, Phys. Social Comput. (CPSCom) IEEE Smart Data (SmartData). 2019. P. 240—243.Kim H. and Choi K. Intern. Conf. Internet Things (iThings) IEEE Green Comput. Commun. (GreenCom) IEEE Cyber, Phys. Social Comput. (CPSCom) IEEE Smart Data (SmartData), 2019, pp. 240–243.Peng H., Chen S. BDNN: Binary convolution neural networks for fast object detection // Pattern Recognition Lett. 2019. Vol. 125. P. 91—97.Peng H. and Chen S. Pattern Recognit. Lett., 2019, vol. 125, pp. 91–97.Ojeda F. C., Bisulco A., Kepple D., Isler V. and Lee D. D. On-device event filtering with binary neural networks for pedestrian detection using neuromorphic vision sensors // IEEE Intern. Conf. Image Process. (ICIP) 2020. P. 3084—3088.Ojeda F.C., Bisulco A., Kepple D., Isler V., and Lee D.D. Intern. Conf. Image Process. (ICIP), 2020, pp. 3084–3088.Wang Z., Wu Z., Lu J. and Zhou J. BiDet: An efficient binarized object detector // IEEE/CVF Conf. Comput. Vis. Pattern Recognition. 2020. P. 2049—2058.Wang Z., Wu Z., Lu J., and Zhou J. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 2049–2058.Xu S., Zhao J., Lu J., Zhang B., Han S. and Doermann D. Layer-wise searching for 1-bit detectors // IEEE/CVF Conf. Comput. Vis. Pattern Recognition. (CVPR). 2021. P. 5682—5691.Xu S., Zhao J., Lu J., Zhang B., Han S., and Doermann D. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 5682–5691.Zhao J., Xu S., Wang R., Zhang B., Guo G., Doermann D. and Sun D. Data-adaptive binary neural networks for efficient object detection and recognition // Pattern Recognition. Lett. 2022. Vol. 153. P. 239—245.Zhao J., Xu S., Wang R., Zhang B., Guo G., Doermann D., and Sun D. Pattern Recognit. Lett., 2022, vol. 153, pp. 239–245.Mani V. R. S., Saravanaselvan A. and Arumugam N. Performance comparison of CNN, QNN and BNN deep neural networks for real-time object detection using Zynq FPGA node // Microelectron. 2022. Vol. 119, art. N 105319.Mani V.R.S., Saravanaselvan A., and Arumugam N. J. Microelectron., 2022, vol. 119, art. no. 105319.Pérez-Hernández F., Tabik S., Lamas A., Olmos R., Fujita H., Herrera F. Object detection binary classifiers methodology based on deep learning to identify small objects handled similarly: application in video surveillance // Knowl. Base Syst. 2020. N 194 P. 105590.Pérez-Hernández F., Tabik S., Lamas A., Olmos R., Fujita H., Herrera F. Knowl. Base Syst., 2020, art. no.105590, https://doi.org/10.1016/j.knosys.2020.105590.Frickenstein A., Vemparala M.-R., Mayr J., Nagaraja N.-S., Unger C., TombariF. and StecheleW. Binary DAD-Net: Binarized driveable area detection network for autonomous driving // IEEE Intern. Conf. Robot. Autom (ICRA). 2020. P. 2295—2301.Frickenstein A., Vemparala M.-R., Mayr J., Nagaraja N.-S., Unger C., Tombari F., and Stechele W. IEEE Intern. Conf. Robot. Autom. (ICRA), 2020, pp. 2295–2301.Ajay B. S., MRao. Binary neural network based real time emotion detection on an edge computing device to detect passenger anomaly // 34th Intern. Conf. VLSI Design, 20th Intern. Conf. Embedded Syst. (VLSID). 2021. P. 175—180.Ajay B.S. and Rao M. 34th Intern. Conf. VLSI Design; 20th Intern. Conf. Embedded Syst. (VLSID), 2021, pp. 175–180.Zhuang B., Shen C., Tan M., Liu L. and Reid I. Structured binary neural networks for accurate image classification and semantic segmentation // IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). 2019. P. 413—422.Zhuang B., Shen C., Tan M., Liu L., and Reid I. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2019, pр. 413–422.

The authors declare that there are no conflicts of interest present.