# A Review of Binarized Neural Networks

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Terminology

## 3. Background

#### 3.1. Network Quantization Techniques

#### 3.2. Early Binarization

## 4. An Introduction to BNNs

#### 4.1. Binarization of Weights

#### 4.2. Binarization of Activations

#### 4.3. Bitwise Operations

#### 4.4. Batch Normalization

#### 4.5. Accuracy

#### 4.6. Robustness to Attacks

## 5. Major BNN Developments

#### 5.1. The Original BNN

#### 5.2. XNOR-Net

#### 5.3. DoReFa-Net

#### 5.4. Tang et al.

#### 5.5. ABC-Net

#### 5.6. BNN+

#### 5.7. Comparison

## 6. Improving BNNs

#### 6.1. Scaling with a Gain Term

#### 6.2. Using Multiple Bases

#### 6.3. Partial Binarization

#### 6.4. Learning Rate

#### 6.5. Padding

#### 6.6. More Binarization

#### 6.7. Batch Normalization and Activations as a Threshold

#### 6.8. Layer Order

## 7. Comparison of Accuracy

#### 7.1. Datasets

#### 7.2. Topologies

#### 7.3. Table of Comparisons

## 8. Hardware Implementations

#### 8.1. FPGA Implementations

#### 8.2. Architectures

#### 8.3. High Level Synthesis

#### 8.4. Comparison of FPGA Implementations

#### 8.5. ASICs

## 9. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Courbariaux, M.; Bengio, Y. BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or −1. arXiv
**2016**, arXiv:1602.02830. [Google Scholar] - Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; Adam, H. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv
**2017**, arXiv:1704.04861. [Google Scholar] - Jaderberg, M.; Vedaldi, A.; Zisserman, A. Speeding up Convolutional Neural Networks with Low Rank Expansions. arXiv
**2014**, arXiv:1405.3866. [Google Scholar][Green Version] - Chen, Y.; Wang, N.; Zhang, Z. DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018. [Google Scholar]
- Iandola, F.N.; Moskewicz, M.W.; Ashraf, K.; Han, S.; Dally, W.J.; Keutzer, K. SqueezeNet: AlexNet-level accuracy with 50× fewer parameters and <1 MB model size. arXiv
**2016**, arXiv:1602.07360. [Google Scholar] - Hanson, S.J.; Pratt, L. Comparing Biases for Minimal Network Construction with Back-propagation. In Advances in Neural Information Processing Systems 1; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1989; pp. 177–185. [Google Scholar]
- Cun, Y.L.; Denker, J.S.; Solla, S.A. Optimal Brain Damage. In Advances in Neural Information Processing Systems 2; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1990; pp. 598–605. [Google Scholar]
- Han, S.; Mao, H.; Dally, W.J. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. arXiv
**2015**, arXiv:1510.00149. [Google Scholar] - Gupta, S.; Agrawal, A.; Gopalakrishnan, K.; Narayanan, P. Deep Learning with Limited Numerical Precision. In Proceedings of the International Conference on Machine Learning, Lille, France, 6–11 July 2015. [Google Scholar]
- Courbariaux, M.; Bengio, Y.; David, J.P. Training deep neural networks with low precision multiplications. arXiv
**2014**, arXiv:1412.7024. [Google Scholar] - Zhou, S.; Ni, Z.; Zhou, X.; Wen, H.; Wu, Y.; Zou, Y. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv
**2016**, arXiv:1606.06160. [Google Scholar] - Seo, J.; Yu, J.; Lee, J.; Choi, K. A new approach to binarizing neural networks. In Proceedings of the 2016 International SoC Design Conference (ISOCC), Jeju, Korea, 23–26 October 2016; pp. 77–78. [Google Scholar]
- Yonekawa, H.; Sato, S.; Nakahara, H. A Ternary Weight Binary Input Convolutional Neural Network: Realization on the Embedded Processor. In Proceedings of the 2018 IEEE 48th International Symposium on Multiple-Valued Logic (ISMVL), Linz, Austria, 16–18 May 2018; pp. 174–179. [Google Scholar] [CrossRef]
- Hwang, K.; Sung, W. Fixed-point feedforward deep neural network design using weights +1, 0, and −1. In Proceedings of the 2014 IEEE Workshop on Signal Processing Systems (SiPS), Belfast, UK, 20–22 October 2014; pp. 1–6. [Google Scholar] [CrossRef]
- Prost-Boucle, A.; Bourge, A.; Pétrot, F. High-Efficiency Convolutional Ternary Neural Networks with Custom Adder Trees and Weight Compression. ACM Trans. Reconfigurable Technol. Syst.
**2018**, 11, 1–24. [Google Scholar] [CrossRef][Green Version] - Saad, D.; Marom, E. Training feed forward nets with binary weights via a modified CHIR algorithm. Complex Syst.
**1990**, 4, 573–586. [Google Scholar] - Baldassi, C.; Braunstein, A.; Brunel, N.; Zecchina, R. Efficient supervised learning in networks with binary synapses. Proc. Natl. Acad. Sci. USA
**2007**, 104, 11079–11084. [Google Scholar] [CrossRef][Green Version] - Soudry, D.; Hubara, I.; Meir, R. Expectation Backpropagation: Parameter-free training of multilayer neural networks with real and discrete weights. Adv. Neural Inf. Process. Syst.
**2014**, 2, 963–971. [Google Scholar] - Courbariaux, M.; Bengio, Y.; David, J.P. BinaryConnect: Training Deep Neural Networks with binary weights during propagations. In Proceedings of the Advances in Neural Information Processing Systems, Montréal, QC, Canada, 7–10 December 2015; pp. 3123–3131. [Google Scholar]
- Wan, L.; Zeiler, M.; Zhang, S.; Le, Y.; Cun, R.F. DropConnect. In Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA, 17–19 June 2013. [Google Scholar]
- Hubara, I.; Courbariaux, M.; Soudry, D.; El-Yaniv, R.; Bengio, Y. Binarized Neural Networks. In Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain, 5–8 December 2016; pp. 1–9. [Google Scholar]
- Ding, R.; Liu, Z.; Shi, R.; Marculescu, D.; Blanton, R.S. LightNN. In Proceedings of the on Great Lakes Symposium on VLSI 2017 (GLSVLSI), Banff, AB, Canada, 10–12 September 2017; ACM Press: New York, NY, USA, 2017; pp. 35–40. [Google Scholar] [CrossRef][Green Version]
- Ding, R.; Liu, Z.; Blanton, R.D.S.; Marculescu, D. Lightening the Load with Highly Accurate Storage- and Energy-Efficient LightNNs. ACM Trans. Reconfigurable Technol. Syst.
**2018**, 11, 1–24. [Google Scholar] [CrossRef] - Bengio, Y.; Léonard, N.; Courville, A. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv
**2013**, arXiv:1308.3432. [Google Scholar] - Lin, X.; Zhao, C.; Pan, W. Towards Accurate Binary Convolutional Neural Network. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–7 December 2017. [Google Scholar]
- Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.J.; Fergus, R. Intriguing properties of neural networks. arXiv
**2013**, arXiv:1312.6199. [Google Scholar] - Moosavi-Dezfooli, S.; Fawzi, A.; Fawzi, O.; Frossard, P. Universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Galloway, A.; Taylor, G.W.; Moussa, M. Attacking Binarized Neural Networks. arXiv
**2017**, arXiv:1711.00449. [Google Scholar] - Khalil, E.B.; Gupta, A.; Dilkina, B. Combinatorial Attacks on Binarized Neural Networks. arXiv
**2018**, arXiv:1810.03538. [Google Scholar] - Hubara, I.; Courbariaux, M.; Soudry, D.; El-Yaniv, R.; Bengio, Y. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. J. Mach. Learn. Res.
**2017**, 18, 6869–6898. [Google Scholar] - Rastegari, M.; Ordonez, V.; Redmon, J.; Farhadi, A. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 525–542. [Google Scholar] [CrossRef]
- Kanemura, A.; Sawada, H.; Wakisaka, T.; Hano, H. Experimental exploration of the performance of binary networks. In Proceedings of the 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), Singapore, 4–6 August 2017; pp. 451–455. [Google Scholar]
- Tang, W.; Hua, G.; Wang, L. How to Train a Compact Binary Neural Network with High Accuracy? In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017. [Google Scholar]
- Darabi, S.; Belbahri, M.; Courbariaux, M.; Nia, V.P. BNN+: Improved Binary Network Training. In Proceedings of the Sixth International Conference on Learning Representations, Vancouver, BC, Canada, 29 April–3 May 2018; pp. 1–10. [Google Scholar]
- Ghasemzadeh, M.; Samragh, M.; Koushanfar, F. ReBNet: Residual Binarized Neural Network. In Proceedings of the 2018 IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), Boulder, CO, USA, 29 April–1 May 2018; pp. 57–64. [Google Scholar] [CrossRef]
- Prabhu, A.; Batchu, V.; Gajawada, R.; Munagala, S.A.; Namboodiri, A. Hybrid Binary Networks: Optimizing for Accuracy, Efficiency and Memory. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 821–829. [Google Scholar] [CrossRef]
- Wang, H.; Xu, Y.; Ni, B.; Zhuang, L.; Xu, H. Flexible Network Binarization with Layer-Wise Priority. In Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 2346–2350. [Google Scholar] [CrossRef]
- Zhao, R.; Song, W.; Zhang, W.; Xing, T.; Lin, J.H.; Srivastava, M.; Gupta, R.; Zhang, Z. Accelerating Binarized Convolutional Neural Networks with Software-Programmable FPGAs. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays—FPGA, Monterey, CA, USA, 22–24 February 2017; ACM Press: New York, NY, USA, 2017; pp. 15–24. [Google Scholar] [CrossRef][Green Version]
- Guo, P.; Ma, H.; Chen, R.; Li, P.; Xie, S.; Wang, D. FBNA: A Fully Binarized Neural Network Accelerator. In Proceedings of the 2018 28th International Conference on Field Programmable Logic and Applications (FPL), Dublin, Ireland, 27–31 August 2018; pp. 51–513. [Google Scholar] [CrossRef]
- Fraser, N.J.; Umuroglu, Y.; Gambardella, G.; Blott, M.; Leong, P.; Jahre, M.; Vissers, K. Scaling Binarized Neural Networks on Reconfigurable Logic. In Proceedings of the 8th Workshop and 6th Workshop on Parallel Programming and Run-Time Management Techniques for Many-core Architectures and Design Tools and Architectures for Multicore Embedded Computing Platforms - PARMA-DITAM, Stockholm, Sweden, 25 January 2017; ACM Press: New York, NY, USA, 2017; pp. 25–30. [Google Scholar] [CrossRef][Green Version]
- Umuroglu, Y.; Fraser, N.J.; Gambardella, G.; Blott, M.; Leong, P.; Jahre, M.; Vissers, K. FINN. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays—FPGA, Monterey, CA, USA, 22–24 February 2017; ACM Press: New York, NY, USA, 2017; pp. 65–74. [Google Scholar] [CrossRef][Green Version]
- Song, D.; Yin, S.; Ouyang, P.; Liu, L.; Wei, S. Low Bits: Binary Neural Network for Vad and Wakeup. In Proceedings of the 2018 5th International Conference on Information Science and Control Engineering (ICISCE), Zhengzhou, China, 20–22 July 2018; pp. 306–311. [Google Scholar] [CrossRef]
- Yin, S.; Ouyang, P.; Zheng, S.; Song, D.; Li, X.; Liu, L.; Wei, S. A 141 UW, 2.46 PJ/Neuron Binarized Convolutional Neural Network Based Self-Learning Speech Recognition Processor in 28NM CMOS. In Proceedings of the 2018 IEEE Symposium on VLSI Circuits, Honolulu, HI, USA, 18–22 June 2018; pp. 139–140. [Google Scholar] [CrossRef]
- Li, Y.; Liu, Z.; Liu, W.; Jiang, Y.; Wang, Y.; Goh, W.L.; Yu, H.; Ren, F. A 34-FPS 698-GOP/s/W Binarized Deep Neural Network-based Natural Scene Text Interpretation Accelerator for Mobile Edge Computing. IEEE Trans. Ind. Electron.
**2018**, 66, 7407–7416. [Google Scholar] [CrossRef] - Bulat, A.; Tzimiropoulos, G. Binarized Convolutional Landmark Localizers for Human Pose Estimation and Face Alignment with Limited Resources. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 3726–3734. [Google Scholar] [CrossRef]
- Ma, C.; Guo, Y.; Lei, Y.; An, W. Binary Volumetric Convolutional Neural Networks for 3-D Object Recognition. IEEE Trans. Instrum. Meas.
**2019**, 68, 38–48. [Google Scholar] [CrossRef] - Kim, M.; Smaragdis, P. Bitwise Neural Networks for Efficient Single-Channel Source Separation. In Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada, 15–20 April 2018; pp. 701–705. [Google Scholar] [CrossRef]
- Eccv, A. Efficient Super Resolution Using Binarized Neural Network. arXiv
**2018**, arXiv:1812.06378. [Google Scholar] - Bulat, A.; Tzimiropoulos, Y. Hierarchical binary CNNs for landmark localization with limited resources. IEEE Trans. Pattern Anal. Mach. Intell.
**2018**. [Google Scholar] [CrossRef] [PubMed] - Say, B.; Sanner, S. Planning in factored state and action spaces with learned binarized neural network transition models. In Proceedings of the IJCAI International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13–19 July 2018; pp. 4815–4821. [Google Scholar]
- Chi, C.C.; Jiang, J.H.R. Logic synthesis of binarized neural networks for efficient circuit implementation. In Proceedings of the International Conference on Computer-Aided Design - ICCAD, San Diego, CA, USA, 5–8 November 2018; ACM Press: New York, NY, USA, 2018; pp. 1–7. [Google Scholar] [CrossRef]
- Narodytska, N.; Ryzhyk, L.; Walsh, T. Verifying Properties of Binarized Deep Neural Networks. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; pp. 6615–6624. [Google Scholar]
- Yang, H.; Fritzsche, M.; Bartz, C.; Meinel, C. BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet. In Proceedings of the 25th ACM international conference on Multimedia, Mountain View, CA, USA, 23–27 October 2017. [Google Scholar]
- Blott, M.; Preußer, T.B.; Fraser, N.J.; Gambardella, G.; O’brien, K.; Umuroglu, Y.; Leeser, M.; Vissers, K. FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks. ACM Trans. Reconfigurable Technol. Syst.
**2018**, 11, 1–23. [Google Scholar] [CrossRef] - McDanel, B.; Teerapittayanon, S.; Kung, H.T. Embedded Binarized Neural Networks. arXiv
**2017**, arXiv:1709.02260. [Google Scholar] - Jokic, P.; Emery, S.; Benini, L. BinaryEye: A 20 kfps Streaming Camera System on FPGA with Real-Time On-Device Image Recognition Using Binary Neural Networks. In Proceedings of the 2018 IEEE 13th International Symposium on Industrial Embedded Systems (SIES), Graz, Austria, 6–8 June 2018; pp. 1–7. [Google Scholar] [CrossRef]
- Valavi, H.; Ramadge, P.J.; Nestler, E.; Verma, N. A Mixed-Signal Binarized Convolutional-Neural-Network Accelerator Integrating Dense Weight Storage and Multiplication for Reduced Data Movement. In Proceedings of the 2018 IEEE Symposium on VLSI Circuits, Honolulu, HI, USA, 18–22 June 2018; pp. 141–142. [Google Scholar] [CrossRef]
- Kim, M.; Smaragdis, P.; Edu, P.I. Bitwise Neural Networks. arXiv
**2016**, arXiv:1601.06071. [Google Scholar] - Sun, X.; Yin, S.; Peng, X.; Liu, R.; Seo, J.S.; Yu, S. XNOR-RRAM: A scalable and parallel resistive synaptic architecture for binary neural networks. In Proceedings of the 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), Dresden, Germany, 19–23 March 2018; pp. 1423–1428. [Google Scholar] [CrossRef]
- Yu, S.; Li, Z.; Chen, P.Y.; Wu, H.; Gao, B.; Wang, D.; Wu, W.; Qian, H. Binary neural network with 16 Mb RRAM macro chip for classification and online training. In Proceedings of the 2016 IEEE International Electron Devices Meeting (IEDM), San Francisco, CA, USA, 3–7 December 2016; pp. 16.2.1–16.2.4. [Google Scholar] [CrossRef]
- Zhou, Y.; Redkar, S.; Huang, X. Deep learning binary neural network on an FPGA. In Proceedings of the 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), Boston, MA, USA, 6–9 August 2017; pp. 281–284. [Google Scholar] [CrossRef]
- Nakahara, H.; Fujii, T.; Sato, S. A fully connected layer elimination for a binarizec convolutional neural network on an FPGA. In Proceedings of the 2017 27th International Conference on Field Programmable Logic and Applications (FPL), Gent, Belgium, 4–6 September 2017; pp. 1–4. [Google Scholar] [CrossRef]
- Yang, L.; He, Z.; Fan, D. A Fully Onchip Binarized Convolutional Neural Network FPGA Impelmentation with Accurate Inference. In Proceedings of the International Symposium on Low Power Electronics and Design, Washington, DC, USA, 23–25 July 2018; pp. 50:1–50:6. [Google Scholar] [CrossRef]
- Bankman, D.; Yang, L.; Moons, B.; Verhelst, M.; Murmann, B. An Always-On 3.8 micro J/86% CIFAR-10 Mixed-Signal Binary CNN Processor With All Memory on Chip in 28-nm CMOS. IEEE J. Solid-State Circuits
**2019**, 54, 158–172. [Google Scholar] [CrossRef] - Rusci, M.; Rossi, D.; Flamand, E.; Gottardi, M.; Farella, E.; Benini, L. Always-ON visual node with a hardware-software event-based binarized neural network inference engine. In Proceedings of the 15th ACM International Conference on Computing Frontiers—CF, Ischia, Italy, 8–10 May 2018; ACM Press: New York, NY, USA, 2018; pp. 314–319. [Google Scholar] [CrossRef]
- Ding, R.; Liu, Z.; Blanton, R.D.S.; Marculescu, D. Quantized Deep Neural Networks for Energy Efficient Hardware-based Inference. In Proceedings of the 23rd Asia and South Pacific Design Automation Conference, Jeju, Korea, 22–25 January 2018; pp. 1–8. [Google Scholar]
- Ling, Y.; Zhong, K.; Wu, Y.; Liu, D.; Ren, J.; Liu, R.; Duan, M.; Liu, W.; Liang, L. TaiJiNet: Towards Partial Binarized Convolutional Neural Network for Embedded Systems. In Proceedings of the 2018 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), Hong Kong, China, 8–11 July 2018; pp. 136–141. [Google Scholar] [CrossRef]
- Yonekawa, H.; Nakahara, H. On-Chip Memory Based Binarized Convolutional Deep Neural Network Applying Batch Normalization Free Technique on an FPGA. In Proceedings of the 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Orlando, FL USA, 29 May–2 June 2017; pp. 98–105. [Google Scholar] [CrossRef]
- Rybalkin, V.; Pappalardo, A.; Ghaffar, M.M.; Gambardella, G.; Wehn, N.; Blott, M. FINN-L: Library Extensions and Design Trade-Off Analysis for Variable Precision LSTM Networks on FPGAs. In Proceedings of the 2018 28th International Conference on Field Programmable Logic and Applications (FPL), Dublin, Ireland, 27–31 August 2018; pp. 89–897. [Google Scholar] [CrossRef]
- Nakahara, H.; Yonekawa, H.; Sasao, T.; Iwamoto, H.; Motomura, M. A memory-based realization of a binarized deep convolutional neural network. In Proceedings of the 2016 International Conference on Field-Programmable Technology (FPT), Xi’an, China, 7–9 December 2016; pp. 277–280. [Google Scholar] [CrossRef]
- Nakahara, H.; Yonekawa, H.; Fujii, T.; Sato, S. A Lightweight YOLOv2. In Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays—FPGA, Monterey, CA, USA, 25–27 February 2018; ACM Press: New York, NY, USA, 2018; pp. 31–40. [Google Scholar] [CrossRef]
- Faraone, J.; Fraser, N.; Blott, M.; Leong, P.H.W. SYQ: Learning Symmetric Quantization For Efficient Deep Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar] [CrossRef]
- Nurvitadhi, E.; Sheffield, D.; Sim, J.; Mishra, A.; Venkatesh, G.; Marr, D. Accelerating Binarized Neural Networks: Comparison of FPGA, CPU, GPU, and ASIC. In Proceedings of the 2016 International Conference on Field-Programmable Technology (FPT), Xi’an, China, 7–9 December 2016; pp. 77–84. [Google Scholar] [CrossRef]
- Jafari, A.; Hosseini, M.; Kulkarni, A.; Patel, C.; Mohsenin, T. BiNMAC. In Proceedings of the 2018 on Great Lakes Symposium on VLSI - GLSVLSI, Chicago, IL, USA, 23–25 May 2018; ACM Press: New York, NY, USA, 2018; pp. 443–446. [Google Scholar] [CrossRef]
- Bahou, A.A.; Karunaratne, G.; Andri, R.; Cavigelli, L.; Benini, L. XNORBIN: A 95 TOp/s/W hardware accelerator for binary convolutional neural networks. In Proceedings of the 2018 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS), Yokohama, Japan, 18–20 April 2018; pp. 1–3. [Google Scholar] [CrossRef]
- Rusci, M.; Cavigelli, L.; Benini, L. Design Automation for Binarized Neural Networks: A Quantum Leap Opportunity? In Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 27–30 May 2018; pp. 1–5. [Google Scholar] [CrossRef]
- Sun, X.; Liu, R.; Peng, X.; Yu, S. Computing-in-Memory with SRAM and RRAM for Binary Neural Networks. In Proceedings of the 2018 14th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT), Qingdao, China, 31 October–3 November 2018; pp. 1–4. [Google Scholar] [CrossRef]
- Choi, W.; Jeong, K.; Choi, K.; Lee, K.; Park, J. Content addressable memory based binarized neural network accelerator using time-domain signal processing. In Proceedings of the 55th Annual Design Automation Conference on - DAC, San Francisco, CA, USA, 24–29 June 2018; ACM Press: New York, NY, USA, 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Angizi, S.; Fan, D. IMC: Energy -Efficient In-Memory Concvolver for Accelerating Binarized Deep Neural Networks. In Proceedings of the Neuromorphic Computing Symposium on - NCS, Knoxville, TN, USA, 17–19 July 2017; ACM Press: New York, NY, USA, 2017; pp. 1–8. [Google Scholar] [CrossRef]
- Liu, R.; Peng, X.; Sun, X.; Khwa, W.S.; Si, X.; Chen, J.J.; Li, J.F.; Chang, M.F.; Yu, S. Parallelizing SRAM Arrays with Customized Bit-Cell for Binary Neural Networks. In Proceedings of the 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), San Francisco, CA, USA, 24–28 June 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Zhou, Z.; Huang, P.; Xiang, Y.C.; Shen, W.S.; Zhao, Y.D.; Feng, Y.L.; Gao, B.; Wu, H.Q.; Qian, H.; Liu, L.F.; et al. A new hardware implementation approach of BNNs based on nonlinear 2T2R synaptic cell. In Proceedings of the 2018 IEEE International Electron Devices Meeting (IEDM), San Francisco, CA, USA, 1–5 December 2018; pp. 20.7.1–20.7.4. [Google Scholar] [CrossRef]
- Tang, T.; Xia, L.; Li, B.; Wang, Y.; Yang, H. Binary convolutional neural network on RRAM. In Proceedings of the 2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC), Jeju Island, Korea, 16–19 January 2017; pp. 782–787. [Google Scholar] [CrossRef]

**Figure 1.**A visualization of the sign layer and Straight-Through Estimator (STE). While the real values of the weights are processed by the sign function in the forward pass, the gradient of the binary weights are simply passed through to the real valued weights.

**Figure 2.**Topology of the original Binarized Neural Networks (BNN). Numbers listed denote the number of output channels for the layer. Input channels are determined by the number of channels in the input, usually 3, and the input size for the FC layers.

**Table 1.**This table shows how the XNOR operation of the endorsing can be equivalent to multiplications of the binary values, in parenthesis.

Encoding (Value) | XNOR (Multiply) | |
---|---|---|

0 (−1) | 0 (−1) | 1 (+1) |

0 (−1) | 1 (+1) | 0 (−1) |

1 (+1) | 0 (−1) | 0 (−1) |

1 (+1) | 1 (+1) | 1 (+1) |

Methodology | Topology | Accuracy (%) |
---|---|---|

Original BNN | BNN | 89.85 |

XNOR-Net | BNN | 89.83 |

BNN+ | AlexNet | 87.16 |

BNN+ | DoReFa-Net | 83.92 |

**Table 3.**Comparison of accuracies on the ImageNet dataset from works presented in this section. Full precision network accuracies are included for comparison as well.

Methodology | Topology | Top-1 Accuracy (%) | Top-5 Accuracy (%) |
---|---|---|---|

Original BNN | AlexNet | 41.8 | 67.1 |

Original BNN | GoogleNet | 47.1 | 69.1 |

XNOR-Net | AlexNet | 44.2 | 69.2 |

XNOR-Net | ResNet18 | 51.2 | 73.2 |

DoReFa-Net | AlexNet | 43.6 | - |

Tang et al. | 51.4 | 75.6 | |

ABC-Net | ResNet18 | 65.0 | 85.9 |

ABC-Net | ResNet34 | 68.4 | 88.2 |

ABC-Net | ResNet50 | 76.1 | 92.8 |

BNN+ | AlexNet | 46.11 | 75.70 |

BNN+ | ResNet18 | 52.64 | 72.98 |

Full Precision | AlexNet | 57.1 | 80.2 |

Full Precision | GoogleNet | 71.3 | 90.0 |

Full Precision | ResNet18 | 69.3 | 89.2 |

Full Precision | ResNet34 | 73.3 | 91.3 |

Full Precision | ResNet50 | 76.1 | 92.8 |

**Table 4.**A table of major details of the methods presented in this section. Activation refers to which kind of activation function is used. Gain describes how gain terms were added to the network. Multiplicity refers to how many binary convolutions were performed in parallel in place of full precision convolution layers. The regularization column indicates which kind of regularization was used, if any.

Methodology | Activation | Gain | Multiplicity | Regularization |
---|---|---|---|---|

Original BNN | Sign Function | None | 1 | None |

XNOR-Net | Sign Function | Statistical | 1 | None |

DoReFa-Net | Sign Function | Learned Param. | 1 | None |

Tang et al. | PReLU | Inside PReLU | 2 | L2 |

ABC-Net | Sign w/Thresh. | Learned Param. | 5 | None |

BNN+ | Sign w/$S{S}_{t}$ for STE | Learned Param. | 1 | L1 and L2 |

**Table 5.**BNN accuracies on the MNIST dataset. The accuracy reported for [51] was not explicitly stated by the authors. This number was inferred from the figure provided.

Source | Accuracy (%) | Topology |
---|---|---|

[52] | 95.7 | FC200-3FC100-FC10 |

[1] | 96.0 | MLP |

[51] | 97 | NK |

[53] | 97.0 | LeNet |

[54] | 97.69 | MLP |

[55] | 97.86 | ConvPool-2 |

[35] | 98.25 | 1/4 MLP |

[41] | 98.4 | MLP |

[56] | 98.40 | MLP |

[57] | 98.6 | NK |

[58] | 98.67 | MLP |

[59] | 98.77 | FC784-3FC512-FC10 |

Source | Accuracy (%) | Topology | Precision |
---|---|---|---|

[14] | 95.15 | NK | Ternary values |

[60] | 96.9 | NK | 8-bit values |

[60] | 97.2 | NK | 12-bit values |

[58] | 98.53 | MLP | 2-bits values |

[19] | 98.71 | BinaryConnect deterministic | 32-bit float activations |

[54] | 98.74 | MLP | 32-bit float |

[19] | 98.82 | BinaryConnect stochastic | 32-bit float activations |

[12] | 99.1 | NK | Ternary values |

Source | Accuracy | Topology |
---|---|---|

[54] | 94.9 | 1/2 BNN |

[41] | 94.9 | 1/2 BNN |

[39] | 96.9 | NK |

[35] | 97.00 | C64-MP-2C128-MP-2C256-2FP512-FP10 |

[11] | 97.1 | DoReFa-Net |

[1] | 97.47 | 1/2 BNN |

Source | Accuracy (%) | Topology | Precision |
---|---|---|---|

[15] | 97.60 | 1/2 BNN | Ternary weights |

[15] | 97.70 | BNN | Ternary weights |

[19] | 97.70 | BinaryConnect—deterministic | 32-bit float activations |

[19] | 97.85 | BinaryConnect—stochastic | 32-bit float activations |

Source | Accuracy (%) | Topology | Disambiguation |
---|---|---|---|

[61] | 66.63 | 2 conv. and 2 FC | |

[40] | 79.1 | 1/4 BNN | |

[41] | 80.1 | 1/2 BNN | |

[54] | 80.1 | 1/2 BNN | |

[13] | 80.4 | VGG16 | |

[62] | 81.8 | VGG11 | |

[57] | 83.27 | NK | |

[40] | 88.3 | BNN | |

[34] | 83.52 | DoReFa-Net | R2 regularizer |

[34] | 83.92 | DoReFa-Net | R1 regularizer |

[55] | 84.3 | NK | |

[40] | 85.2 | 1/2 BNN | |

[63] | 85.9 | 6 conv. | |

[53] | 86.0 | ResNet-18 | |

[64] | 86.05 | 9 256-ch conv. | |

[61] | 86.06 | 5 conv. and 2 FC | |

[65] | 86.78 | NK | |

[35] | 86.98 | C64-MP-2C128-MP-2C256-2FC512-FC10 | |

[51] | 87 | NK | |

[34] | 87.16 | AlexNet | R1 regularizer |

[34] | 87.30 | AlexNet | R2 regularizer |

[38] | 87.73 | BNN | $+1$ padding |

[32] | 88 | BNN | 512 channels for FC |

[38] | 88.42 | BNN | 0 padding |

[59] | 88.47 | 6 conv. | |

[39] | 88.61 | NK | |

[1] | 89.85 | BNN |

Source | Accuracy(%) | Topology | Precision |
---|---|---|---|

[13] | 81.0 | VGG16 | Ternary values |

[13] | 82.9 | VGG16 | Ternary values |

[15] | 86.71 | 1/2 BNN | Ternary values |

[15] | 89.39 | BNN | Ternary values |

[19] | 90.10 | BinaryConnect—deterministic | 32-bit float activations |

[19] | 91.73 | BinaryConnect—stochastic | 32-bit float activations |

Source | Top 1 Acc. (%) | Top 5 Acc. (%) | Topology | Details |
---|---|---|---|---|

[21] | 36.1 | 60.1 | BNN AlexNet | |

[11] | 40.1 | Alexnet | ||

[35] | 41.43 | Details in [35] | ||

[30] | 41.8 | 67.1 | BNN AlexNet | |

[31] | 44.2 | 69.2 | AlexNet | |

[11] | 43.6 | Alexnet | Pre-trained on full precision | |

[34] | 45.62 | 70.13 | AlexNet | R2 reg |

[34] | 46.11 | 75.70 | AlexNet | R1 reg |

[21] | 47.1 | 69.1 | BNN GoogleNet | |

[36] | 48.2 | 71.9 | AlexNet | Partial binarization |

[31] | 51.2 | 73.2 | ResNet18 | |

[34] | 52.64 | 72.98 | ResNet-18 | R1 reg |

[34] | 53.01 | 72.55 | ResNet-18 | R2 reg |

[66] | 54.8 | 77.7 | ResNet-18 | Partial binarization |

[67] | 55.8 | 78.7 | AlexNet | Partial binarization |

[25] | 65.0 | 85.9 | ResNet-18 | 5 bases |

[25] | 68.4 | 88.2 | ResNet-34 | 5 bases |

[25] | 70.1 | 89.7 | ResNet-50 | 5 bases |

[68] | 75 | VGG 16 | ||

[33] | 75.6 | 51.4 | AlexNet | binarized last layer |

**Table 12.**Comparison of FPGA implementations. The accuracies reported from [68,70] were not explicitly stated. These numbers were inferred from figures provided. The accuracy for [68] is assumed to be a top-5 accuracy and the accuracy for [35] is assumed to top-1 accuracy, but this was never stated by their respective authors. Datasets: MNIST = MN, SVHN = SV, CIFAR-10 = CI, ImageNet = IN.

Source | Dataset | Acc. (%) | Topology | FPGA | LUTs | BRAMs | Clk (MHz) | FPS | Power (W) |
---|---|---|---|---|---|---|---|---|---|

[54] | MN | 97.69 | MLP | Zynq7 020 | 25,358 | 220 | 100 | 2.5 | |

[54] | MN | 97.69 | MLP | ZynqUltra 3EG | 38,205 | 417 | 300 | 11.8 | |

[35] | MN | 98.25 | See Table 5 | Spartan7 50 | 32,600 | 120 | 200 | ||

[41] | MN | 98.4 | MLP | Zynq7 045 | 82,988 | 396 | 1,561,000 | 22.6 | |

[56] | MN | 98.40 | MLP | Kintex7 325T | 40,000 | 110 | 100 | 10,000 | 12.22 |

[41] | SV | 94.9 | 1/2 BNN | Zynq7 045 | 46,253 | 186 | 21,900 | 11.7 | |

[39] | SV | 96.9 | 6 Conv/3 FC | Zynq7 020 | 29,600 | 103 | 6451 | 3.2 | |

[35] | SV | 97.00 | See Table 7 | Zynq7 020 | 53,200 | 280 | 200 | ||

[61] | CI | 66.63 | 2 Convs/2 FC | Zynq7 045 | 20,264 | ||||

[70] | CI | 78 | See Table 9 | Vertex7 690T | 20,352 | 372 | 450 | 15.44 | |

[40] | CI | 79.1 | 1/4 BNN | KintexUltra 115 | 35,818 | 144 | 125 | 12,000 | |

[54] | CI | 80.10 | 1/2 BNN | ZynqUltra 3EG | 41,733 | 283 | 300 | 10.7 | |

[54] | CI | 80.10 | 1/2 BNN | Zynq7 020 | 25,700 | 242 | 100 | 2.25 | |

[41] | CI | 80.1 | 1/2 BNN | Zynq7 045 | 46,253 | 186 | 21,900 | 11.7 | |

[62] | CI | 81.8 | 1/2 BNN | Zynq7 020 | 14,509 | 32 | 143 | 420 | 2.3 |

[40] | CI | 85.2 | 1/2 BNN | KintexUltra 115 | 93,755 | 386 | 125 | 12,000 | |

[63] | CI | 85.9 | See Table 9 | Zynq7 020 | 23,426 | 135 | 143 | 930 | 2.4 |

[61] | CI | 86.06 | 5 Convs/2 FC | Vertex7 980T | 556,920 | 340 | 332,158 | ||

[35] | CI | 86.98 | See Table 9 | Zynq7 020 | 53,200 | 280 | 200 | ||

[38] | CI | 87.73 | See Table 9 | Zynq7 020 | 46,900 | 140 | 143 | 4.7 | |

[40] | CI | 88.3 | BNN | KintexUltra 115 | 392,947 | 1814 | 125 | 12,000 | |

[39] | CI | 88.61 | 6 Conv/3 FC | Zynq7 020 | 29,600 | 103 | 520 | 3.3 | |

[35] | IN | 41 | See Table 11 | VirtexUltra 095 | 1,075,200 | 3456 | 200 | ||

[68] | IN | 75 | VGG 116 | ZynqUltra 9EG | 191,784 | 1367 | 150 | 31.48 | 22 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Simons, T.; Lee, D.-J. A Review of Binarized Neural Networks. *Electronics* **2019**, *8*, 661.
https://doi.org/10.3390/electronics8060661

**AMA Style**

Simons T, Lee D-J. A Review of Binarized Neural Networks. *Electronics*. 2019; 8(6):661.
https://doi.org/10.3390/electronics8060661

**Chicago/Turabian Style**

Simons, Taylor, and Dah-Jye Lee. 2019. "A Review of Binarized Neural Networks" *Electronics* 8, no. 6: 661.
https://doi.org/10.3390/electronics8060661