Reference for citation: Shakkouf A. Review on Optimization Techniques of Binary Neural Networks. Journal of Instrument Engineering. 2023. Vol. 66, N 11. P. 926—935 (in English). DOI: 10.17586/0021-3454-2023-66-11-926-935.

Abstract. The deployment of Convolutional Neural Networks (CNNs) models on embedded systems faces multiple problems regarding computation power, power consumption and memory footprint. To solve these problems, a promising type of neural networks that uses 1-bit activations and weights emerged in 2016 called Binary Neural Networks (BNNs). BNN consumes less energy and computation power mainly because it replaces the complex heavy convolution operation with simple bitwise operations. However, the quantization from 32-float point to 1-bit leads to accuracy loss and poor performance, especially on large datasets. This article presents a review of the key optimization techniques which influenced the performance of BNNs and led to higher representation capacity of BNN models, as well as an overview of the application methods of BNNs in object detection tasks and compares the performance with the real value CNN.

Keywords: binary neural networks, BNNs optimization, object detection, quantization, binarization, computer vision, artificial intelligence

References:

1. Basha S.S., Dubey S.R., Pulabaigari V., and Mukherjee S. Neurocomputing, 2020, vol. 378, pp. 112–119.
2. Zhou W., Wang H., and Wan Z. Computers and Electrical Engineering, 2022, vol. 99, art. no. 107819.
3. Gao X., Xing G., Roy S., and Liu H. RIEEE Sensors J., 2021, no. 4(21), pp. 5119–5132.
4. Ashiq F., Asif M., Ahmad M. B., Zafar S., Masood K., Mahmood T., Mahmood M.T., and Lee I.H. IEEE Access, 2022, vol. 10, pp. 14819–14834.
5. Abdelhamid A.A., El-Kenawy E.-S.-M., Alotaibi B., Amer G.M., Abdelkader M.Y., Ibrahim A., and Eid M.M. IEEE Access., 2022, vol. 10, pp. 49265–49284.
6. Kwon S. Expert Systems with Applications, 2021, vol. 167, art. no. 114177.
7. Zhang N., Wei X., Chen H., and Liu W. Electronics, 2021, vol. 10, no. 3, p. 282.
8. Lopac N., Hrzic F., Vuksanovic I.P., and Lerga J. IEEE Access., 2022, vol. 10, pp. 2408–2428.
9. Horowitz M. IEEE Intern. Solid State Circuits Conf., 2014, pp. 10–14.
10. Courbariaux M. and Bengio Y. ArXiv journal, 2016, preprint arXiv: 1602.02830.
11. Wang E., Davis J.J., Moro D., Zielinski P., Lim J.J., Coelho C., Chatterjee S., Cheung P.Y., Constantinides G.A. ACM Transactions on Embedded Computing Systems, 2021, DOI:10.1145/3626100.
12. Sohoni N.S., Aberger C.R., Leszczynski M., Zhang J., and Ré C. ArXiv journal, 2019, preprint arXiv:1904.10631.
13. Courbariaux M., Bengio Y., and David J.-P. ArXiv e-prints, 2015, abs/1511.00363.
14. Hinton G. Neural networks for machine learning, Coursera, video lectures, 2012.
15. Bengio Y. Technical Report, Universite de Montreal, arXiv:1305.2982, 2013.
16. Darabi S., Belbahri M., Courbariaux M., and Nia V.P. BNN+: Improved binary network training, 2019, https://openreview.net/pdf?id=SJfHg2A5tQ.
17. Shen M., Han K., Xu C., Wang Y. IEEE/CVF Intern. Conf. on Computer Vision Workshops, 2019.
18. Wang Z., Lu J., Tao C., Zhou J., Tian Q. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, vol. 568.
19. Lin X., Zhao C., and Pan W. NIPS, 2017, pр. 344–352.
20. Liu Z., Wu B., Luo W., Yang X., Liu W., and Cheng K.-T. arXiv preprint, 2018, arXiv:1808.00278.
21. Shen M., Liu X., Gong R., Han K. IEEE Intern. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2020, vol. 4197.
22. Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. arXiv preprint, 2019, arXiv:1908.05858.
23. Cai Z., He X., Sun J., and Vasconcelos N. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
24. Ioffe S., and Szegedy C. Proc. of the 32nd Intern. Conf. on Machine Learning, ICML, 2015, рр. 448–456.
25. Wang P., He X., Li G., Zhao T., Cheng J. AAAI Conf. on Artificial Intelligence, 2020, vol.34, рр. 12192.
26. Yang J., Shen X., Xing J., Tian X., Li H., Deng B., Huang J., and Hua X.-S. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
27. Zhu S., Dong X., and Su H. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
28. Anderson A.G. & Berg C.P. ArXiv journal, 2017, abs/1705.07199.
29. Qin H., Gong R., Liu X., Bai X., Song J., Sebe N. Pattern Recognition, 2020, 105. 107281. 10.1016/ j.patcog.107281.
30. Liang T., Glossner J., Wang L., Shi S., and Zhang X. Neurocomputing, 2021, vol. 461, pp. 370–403.
31. Xu Sh., Liu Ch., Zhang B., Lu J., Guo G, Doermann D. ACM Transactions on Multimedia Computing, Communications, and Applications, 2022, 18. 10.1145/3473340.
32. Zhang K., Yin M., & Wang Y. ArXiv journal, 2022, abs/2206.05916.
33. Darabi S., Belbahri M., Courbariaux M., and Nia V.P. ArXiv journal, 2018, 1812.11800.
34. Wang S., Zhang C., Su D., Wang L., Jiang H. IEEE Access 9, 2021, рр. 106169.
35. Rastegari M, Ordonez V, Redmon J, and Farhadi A. ECCV, 2016, pр. 525– 542.
36. Barry D., Shah M., Keijsers M., Khan H., and Hopman B. ArXiv journal: 1910.03159, 2019.
37. Chen H., Zhuo L., Zhang B., Zheng X., Liu J., Ji R., Doermann D., Guo G. ArXiv journal, 2020.
38. Kung J., Zhang D., van der Wal G., Chai S., Mukhopadhyay S. Journal of Signal Processing Systems, 2018, vol. 90, рр. 1–14, DOI: 10.1007/s11265-017-1255-5.
39. Wang X., Siyang S., Yin Y., Xu D. & Wu W., Gu Q. CAAI Transactions on Intelligence Technology, 2018, no. 3(4), DOI:10.1049/trit.2018.1026.
40. Kim H. and Choi K. Intern. Conf. Internet Things (iThings) IEEE Green Comput. Commun. (GreenCom) IEEE Cyber, Phys. Social Comput. (CPSCom) IEEE Smart Data (SmartData), 2019, pp. 240–243.
41. Peng H. and Chen S. Pattern Recognit. Lett., 2019, vol. 125, pp. 91–97.
42. Ojeda F.C., Bisulco A., Kepple D., Isler V., and Lee D.D. Intern. Conf. Image Process. (ICIP), 2020, pp. 3084–3088.
43. Wang Z., Wu Z., Lu J., and Zhou J. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 2049–2058.
44. Xu S., Zhao J., Lu J., Zhang B., Han S., and Doermann D. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 5682–5691.
45. Zhao J., Xu S., Wang R., Zhang B., Guo G., Doermann D., and Sun D. Pattern Recognit. Lett., 2022, vol. 153, pp. 239–245.
46. Mani V.R.S., Saravanaselvan A., and Arumugam N. J. Microelectron., 2022, vol. 119, art. no. 105319.
47. Pérez-Hernández F., Tabik S., Lamas A., Olmos R., Fujita H., Herrera F. Knowl. Base Syst., 2020, art. no.105590, https://doi.org/10.1016/j.knosys.2020.105590.
48. Frickenstein A., Vemparala M.-R., Mayr J., Nagaraja N.-S., Unger C., Tombari F., and Stechele W. IEEE Intern. Conf. Robot. Autom. (ICRA), 2020, pp. 2295–2301.
49. Ajay B.S. and Rao M. 34th Intern. Conf. VLSI Design; 20th Intern. Conf. Embedded Syst. (VLSID), 2021, pp. 175–180.
50. Zhuang B., Shen C., Tan M., Liu L., and Reid I. IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2019, pр. 413–422.