Intellino: Processor for Embedded Artificial Intelligence

1. Introduction

As the development of computing technology enables complex computations and huge data processing, research on artificial intelligence (AI) has been promoted. Furthermore, the interest in AI computing, which includes neural networks, has also increased [1,2,3,4,5,6]. The neural network is the set of transmissions of signals between numerous calculating units and memories through entangled connections, similarly to how the brain works with spikes, neurons, and synapses [7,8,9,10,11,12]. Due to this feature, conventional computing technology with CPUs is inadequate to carry out the computation for AI [13,14]. As every neuron is sequentially computed with CPUs, the system workload is significantly increased. Even though GPUs lead the computation parallelization [15,16,17], the power consumption is also increased.

In order to overcome these limitations, the neuromorphic processors implemented in hardware, including neurons, were proposed [18,19,20]. This hardware implementation can achieve the remarkable reduction in computation time and power consumption [21,22,23,24,25,26], and realizing neuromorphic computing on the chip instead of CPUs and GPUs also helps the AI system to become smaller. These advantages lead the hardware realized computation module for the AI system to be applied to not only in high-performance servers but also edge devices of the internet of things (IoT). By including the AI computing in the edge, the IoT can swiftly perform the AI operation and significantly reduce the workload of the system core on the edge. For the AI edge as an embedded system, it is essential that the computation module should be realized with restricted resources as well as operated with low power [27,28,29,30,31,32,33]. Therefore, analyzing the algorithms and neurons which occupy the resources of the AI processor and applying the processor to embedded AI system according to applications are important.

In this paper, we present the low-power processor with less resource usage, named intellino, for embedded AI systems. Intellino operates in low power with fully excluding multiplications thanks to the optimized AI algorithms. Parallelized neurons which form the intellino’s neural network-based architecture help the system core to reduce the workload. Moreover, the designated protocol for intellino enables the embedded system to work efficiently through the system core. We evaluate intellino’s performance with a simulator based on the optimized algorithms. We also analyze the workload of the AI system and the intellino requirements for integrated systems by implementing intellino on an field programmable gate array (FPGA). Finally, we demonstrate our proposal’s feasibility through prototyping.

The rest of this paper is organized as follows. Section 2 describes the AI algorithms functionally and presents the algorithms optimizing for hardware realization. Section 3 explains the details of the intellino microarchitecture and presents the optimization for an embedded system. Section 4 provides the measurement result of intellino performance with the simulator based on the optimized algorithms. Section 5 describes the workload of the AI system and the requirements with intellino through specific AI applications. Section 6 demonstrates the realization and experimental results of intellino with FPGA prototyping. Section 7 summarizes our proposal and presents the future work.

2. Artificial Intelligence Algorithm

As each neuron in neural network includes an AI algorithm and a number of neurons are built in the AI system, the AI algorithm directly affects the system characteristics. In case of hardware implementation, the considerable neurons occupy most of the resources. Therefore, adopting the proper algorithms and realizing the optimized algorithms are important. For intellino, we adopt two AI algorithms, k-nearest neighbor (k-NN) and radial basis function neural network (RBF-NN). The algorithms are optimized for hardware realization.

2.1. k-NN and RBF-NN

The k-NN is one of machine learning algorithms for classification or regression, based on the distance computation. The k-NN classifies a test dataset by comparing distance between the test dataset and training dataset. For the supervised learning, the training dataset should be learned before classification, and each training dataset has its own category and location information (vector) for the distance calculation. As the test dataset also has its own vector, the algorithm performs classification with calculating the distance with each training dataset’s vector and test dataset’s vector. After taking the test dataset, the algorithm has the number of distance results, which equals to the number of trained datasets. The algorithm takes k variable and classifies the test dataset as the major category among the nearest distance of k trained dataset. When the k value is 1, the test dataset is classified as the nearest trained dataset. When the k value is 3, the test dataset finds the three nearest trained datasets and is classified as the major category among the sorted dataset. The k-NN can prevent the under-fitting or over-fitting issue through the k variable and perform the classification with only distance calculation.

The RBF-NN is a single layer (input, one hidden, output) neural network, which includes Gaussian distribution as an activation function in the hidden layer. When the input layer gets a training dataset, the hidden layer builds (trains) probability distributions depending on the previous training dataset. When the training datasets which have the same category are close, the hidden layer builds a narrow and high probability distribution. Otherwise, with scattered training datasets, the distribution is formed as wide and low. When another category of training dataset is learned and located on the built distribution of different category, the distribution becomes small to minimize the interference with other distributions. When the input layer receives a test dataset, the output layer outputs the probability values according to all categories of distributions. Thanks to the flexible distribution of each category, the RBF-NN can perform a precise classification.

2.2. Optimization for Hardware Realization

Intellino performs AI algorithm-based computation in the numerous neurons. As the computation influences the power consumption and resource usage of intellino, the algorithm optimization for hardware realization is important. In order to realize two algorithms in the hardware, we simplify the arithmetic operation and utilize proper properties of the algorithms. The multiplication is fully excluded by applying Manhattan distance calculation instead of Euclidean distance. The adder unit consumes less power and resources than the multiplication unit. In case of 8-bit calculation, a multiplier uses about 10× more resources than an adder. However, the accuracy only grows by about 1% with the multiplier compared with the adder. The adder is enough to fulfill the algorithms performance with less power consumption. Additionally, the algorithm computation units can be retained by each neuron without sharing and reusing, which enables the parallel computation. In addition, the probability of RBF-NN is substituted to matching/non-matching by setting a threshold on the distribution. Whenever the training dataset is learned, each training dataset has an influence field (IF) which decides the test dataset to be classified as its category or not. Unlike the probability distribution, the IF of matching/non-matching is generated by only distance calculation, and this leads the reduction in calculations.

According to the AI algorithm optimization, the main difference between k-NN and RBF-NN is the categorization method. While the k-NN recognizes any test dataset to only the nearest trained dataset’s category, the RBF-NN recognizes some test datasets to additional categories (uncertainty and unknown). In the case of RBF-NN, each trained dataset has the IF by threshold. A test dataset is generally recognized to the nearest category of trained dataset. When the test dataset is located on the IFs intersection of two or more other categories, the test dataset is recognized as uncertainty, and when the test dataset is not involved in any IFs, the test dataset is classified as unknown. The optimized algorithms are realized on intellino, and this makes intellino operate with low power with less resource usage.

3. Intellino Architecture

Intellino is the AI processor involving optimized features for embedded systems. The AI algorithms described in Section 2 are included in the neurons, and this leads to the low power consumption and lower resource utilization of the system. The single neural network-based architecture with parallelized neurons reduces the workload of the system core by facilitating the simultaneous massive recognition. The dedicated protocol enhances the repetitive AI operation (learning/recognition) through the interface module. The microarchitecture of intellino with operation flow and the details of hardware realization to apply intellino in an embedded system are described below.

3.1. Microarchitecture

Figure 1 shows the block diagram of intellino. Intellino includes an interface for communication with system core and an operator for learning/recognition with neuron cells (N-cells). The system core receives an external signal for learning/recognition and transmits a training/test dataset to the interface. A data transceiver receives the data from the system core, which implies the dataset or command for learning/recognition. A finite state machine (FSM) interprets the consecutive data which are organized by the designated protocol. An instruction encoder encodes the interpreted data and announces to the operator.

The operator performs the operation of learning/recognition with the transmitted data. The instruction decoder receives the data/category/distance of the dataset, the writing/reading signals, and the selection of algorithms. When the signal indicates writing data and category, the operator learns the training dataset. When the signal indicates writing data and reading category or distance, the operator recognizes the test dataset. As the instruction decoder informs the completion signal of the operator to the instruction encoder, the instruction encoder can confirm whether the operator finishes the instruction. Additionally, the instruction encoder transmits the algorithm signal and the operator classifies the test dataset with the delivered algorithm signal.

The decoded data make the neuron core to operate learning/recognition. The neuron core consists of a scheduler and N-cells in parallel connection for storing the training dataset in a vector memory and calculating the distance of test dataset. The instruction decoder informs a current state (cur_state) and received data (rx_data) to the scheduler according to the instruction. In the case of the instructions for the learning, the scheduler sequentially receives the training dataset and increases a component pointer (comp_ptr) which indicates the index of received data at the dataset. The received data are delivered to the N-cell, which is assigned by a N-cell pointer (N-Cell_ptr), and stored in the vector memory at the component pointer pointed. After all the data of training dataset are stored, the category value is delivered with a category write enable (category_w_en) signal. By increasing the N-cell pointer, the next N-cell gets ready for the next learning. In the case of the instruction for the recognition, the scheduler also increases the component pointer whenever the data of test dataset are received. In every trained N-cell, the stored (trained) data, which are pointed by the component pointer in the vector memory, are calculated with the received test data. The calculated value is accumulated for the total distance between the trained dataset and test dataset. When the scheduler notifies the last data of test dataset with a last component write enable (l_comp_w_en) signal, every trained N-cell finishes the calculation and outputs the accumulated distance and trained category with a valid and an IF check signals. The scheduler also informs the algorithm to be classified with to the classifier.

The calculation results of distance and category of each trained N-cell are combined through the framer in the classifier. The classification with the framed N-cells data is performed in k-NN or RBF-NN module according to the selected algorithm. The k-NN and RBF-NN module of classifier find the minimum distance to classify the test dataset as the nearest trained dataset. Thanks to the hardware realization, the searching minimum is parallelly conducted based on the divide and conquer methods. Every distance of trained N-cells is compared in pairs; for instance, the minimum searching only takes 10 clocks against the 1024 trained dataset. Especially in the case of RBF-NN, the test dataset is classified as uncertainty or unknown by the IF check signals. As each of the framed N-cells data have the IF check signals, the uncertainty or unknown is determined by the category identifying as well as the divide and conquer methods. When the compared pair have different categories with IF check signal ‘1’, the IF check signal is maintained, and the result indicates the uncertainty. When the pair have IF check signal ‘0’, the result indicates the unknown. Finally, when the system core requires the recognition results, the instruction encoder transmits the reading category or distance signal to the instruction decoder. The recognized category and distance results are delivered through the interface according to the reading signals.

3.2. Optimization for Embedded System

In order to reduce the workload of system core, the calculation of N-cell operates in parallel. For the parallelization, each N-cell includes the vector memory, which is the assembly of components. The dataset matches the vector, and each datum of the dataset is related to the component. In the process of learning, the input data are sequentially stacked up in the components of the N-cell. The category is also allocated to the N-cell and the next N-cell gets ready for the next dataset learning. In the process of recognition, the input data are calculated with every component of the trained N-cell. Whenever the data are received, every trained N-cell calculates and accumulates the distances, simultaneously. Thanks to this feature, the recognition time is not increased, even if the number of trained N-cell gets increased.

The number of N-cells and the size of the vector not only decide the usage of hardware resources but also affect the system accuracy. The increment of the N-cells enables the system to train more dataset, and it is suitable for the application that has coarse-grained dataset with numerous categories. On the other hand, the increase in vector size enables the system to distinguish in detail, and a large vector size is appropriate for the application that has a fine-grained dataset. Therefore, the N-cell counts and vector size of intellino can be decided depending on the application of embedded system.

Intellino also facilitates the efficient learning/recognition through the dedicated protocol. As intellino targets an embedded system, the data transceiver employs the serial interface to reduce pin mapping with the system core. Also, the FSM enables the repetitive data receiving for successive learning/recognition. As shown in Figure 2, the interface communicates data in 8-bit. The first 8-bit is a setup value, which involves the information of single/sequential mode, write/read mode, and an address for the neuron core operation. When the single mode, the 16-bit data are attached right after the setup value for receiving the read data or transmitting the write data. In the case of sequential mode, the dataset length in 16-bit is notated, and the consecutive data, which include components and category, are delivered. For example, the learning is executed by write category (CAT) command, transmitting the setup value, data length, and dataset with category. The recognition is performed with write last component (LCOMP) command, transmitting the setup value, data length, and dataset. Intellino prints out the result of recognition by read one command, which shows the classified category or the calculated nearest distance depending on the address. The address indicates the command for data/distance/category or algorithm and enables the system to get the desired result with the read one command. As the interface is full duplex, the read one command transmits dummy data for receiving the desired data. Through this method, the protocol enables the system to perform learning/recognition without additional overhead, even though the dataset is bigger. Furthermore, since the protocol is based on the serial interface, the system can generate a learning/recognition library with a built-in interface and operate intellino efficiently.

4. Simulator

In order to evaluate the performance of intellino, we built a simulator including the hardware optimized algorithms. For the same operation as intellino, the simulator has the same structure by including the neuron core and the N-cells. Each N-cell retains the vector and optimized algorithms for learning/recognition and is contained in the neuron core. For the evaluation of intellino in accordance with various datasets, the neuron core can configure the number of N-cells and the size of the vector. Through the constructed simulator, we measure the accuracy of intellino.

Figure 3 shows the relation between the N-cell counts, vector sizes, and accuracy depending on the applications. In Figure 3a, application 1 (App1) includes audio-based dataset [34] with two categories, the application 2 (App2) includes an image-based dataset [35] with five categories. As raw data are inadequate for learning/recognition in terms of accuracy as well as data quantity, the raw data should be pre-processed and organized for extracting the feature data for the efficient AI operation. In the case of App1, the audio-based data are pre-processed by a short time Fourier transform (STFT) [36], which extracts the feature regarding the frequency over time. As App1 is for the classifying words and each word has its own frequency, the extracted frequency dataset is used for the recognition. In the case of App2, which is for classifying hand-written digit images, as the image-based dataset is well-organized and the feature dataset by itself, the resizing with linear interpolation is only performed according to the vector sizes. In order to perform learning/recognition with the neurons of intellino, the 2D featured datasets of audio and image are flattened into a 1D array. As shown in the graph, App1 shows the highest accuracy of 87.4%, with 1024 N-cells and 16 vector sizes, and App2 has an accuracy of 92.8% with 256 N-cells and 64 vector sizes. The experimental results show that each application has the suitable combination of the N-cell counts and the vector sizes.

We also analyze the correlation between the number of N-cell and the intellino accuracy. Figure 3b describes the accuracy with the App1 and App2 datasets when the vector size is fixed at 64 bytes, and the number of N-cell is increasing. App1 and App2 show a similar trend, which indicates the growth of accuracy with the increment of N-cell counts. Especially for App2, when the number of N-cell is increased from 16 to 32, the accuracy grows from 68.6% to 81.6%, which shows the 13% increment of accuracy. Otherwise, when the number of N-cell is increased from 2048 to 4096, the accuracy only grows by 0.4%, from 97.6% to 98%. The accuracy of App2 converges around 97%, even if the count increases. In addition, the accuracy of App1 has a slight increment with the enormous N-cell around 90%. Therefore, the N-cell counts and the vector sizes should be determined in accordance with the target applications and the embedded system environments for the accuracy and resource usage. Thanks to the reconfigurable feature of the FPGA, intellino can be implemented with appropriate neuron specification.

5. Workload Analysis

A recognition application with AI is conducted through the series of processes. In the case of learning, the pre-processing raw data and the storing of training dataset should be carried out. For the recognition with AI system, the pre-processing, calculating for classification, and informing of the recognized result should be carried out. The pre-processing is performed for generating the training and test dataset. As the training dataset is prepared in advance and the test dataset is delivered one by one, parallel computing for prompt pre-processing is not essential. The system core in the embedded system is enough to perform the pre-processing. The storing and the calculating are relevant on account of sharing the dataset memory. The storing is sequentially proceeded, since the training dataset is transmitted in sequence. Otherwise, the calculating for classification is executed as many times as the trained dataset. As the test dataset is compared with every trained dataset, a greater number of trained datasets leads to more calculation. The recurring recognition with the system core can cause a bottleneck issue due to the drastic increment of calculation. The parallelization of calculating in intellino reduces the workload of the system core and enables the embedded AI system to process the test data without delay. After the classification, the result of recognition is informed to the system core, and the result is directly displayed or utilized according to the applications.

In order to analyze the concrete workload of AI system, we implement intellino on the FPGA, as shown in Figure 4. The neuron core with 32 N-cells and 256 vector size is implemented on intel DE2-115 evaluation board with Cyclone-IV FPGA. We adopt a face expression recognition application to demonstrate the real-world system applicability of intellino. We generate the training dataset for classifying the face expression of three categories, smile/angry/snooze, and pre-process the raw data and extract the feature data with a CPU-based system. The raw face image is taken from the camera, and the face region is cropped with Haar-like feature [37] for excluding the background. The cropped image is converted into a grayscale image, and Canny edge detection [38] derives an edge for acquiring the feature of face expression. For the more efficient recognition of expression with the face image, the regions near to the eyes, nose, and mouse are extracted for generating the feature dataset. After the pre-processing, the featured training datasets are stored (trained) in each of vector memories of intellino. With trained intellino, the recognition of face expression is performed. The image for recognition is also taken from the camera and pre-processed with the same process as the training dataset. The pre-processed test dataset is delivered to intellino and calculated with every trained dataset in intellino. Thanks to the parallelized neurons, the multiple calculations for classification are promptly conducted. Even when the trained dataset is increased in intellino, the calculation time for recognition is always the same. Then, the result of recognition is transmitted to the system and displayed. The recognition achieves 90.19% accuracy with 36 datasets by k-fold cross validation, 12 datasets for each category and 10/2 for training/test datasets, and the operation with real-time capturing demonstrates the functionality of intellino. Through the face expression recognition application with the classical camera, we analyze the workload of the AI system step by step. Although the AI system has the most workload at the recognition step, the parallel computation of intellino takes charge of the repetitive recognition and improves the system performance by lessening the workload. As intellino receives the feature data for learning/recognition, the system can substitute other sensors for the classical camera. Finally, as the pre-processing in accordance with the attached sensor generates and delivers the feature data to intellino, we also discover that intellino only requires the interface with the neuron core for the data transmission with the system core, which lightens the peripherals of intellino.

6. Realization

After the workload analysis, we finally design the prototype board of intellino for an embedded AI system. Figure 5a shows the integrated AI system with intellino, which is realized on intel MAX10 FPGA and supports up to 88 MHz operation. The prototype board includes the FPGA and joint test action group (JTAG) connector for intellino realization and the general-purpose input/output (GPIO) header for integration. The header enables the system core to be easily connected to intellino and to get external signals for learning/recognition. As intellino adopts a serial interface while employing fewer pins, which is directly related to the size of edge device, the system core can utilize the other GPIOs for various applications. Intellino achieves the same accuracy as the optimized algorithm-based simulator and performs according to the analyzed workload. Therefore, we can operate various AI applications based on image and audio by building the embedded AI system with prototyped intellino. In order to perform the AI operation with the embedded system, we develop a library for the system core including the learning/recognition functions. As the library is based on the built-in interface of the system core and the designated protocol of intellino, the embedded system can easily perform the repetitive AI operation of learning/recognition with pre-processed datasets.

With the prototype board and the built library, we also analyze the performance of the proposed intellino in terms of operation time. We implement intellino with the neuron core of 16 N-cells and 1024 vector size, which also indicates the applicability of diverse specification, and measure the performance with the face expression recognition application. We increase the vector size of intellino and generate the pre-processed face expression dataset with more featured data than the experiment in Section 5 to analyze the operation time with a larger dataset. The training/test datasets are delivered to intellino for learning/recognition, and the communication between intellino and the system core is conducted with a serial peripheral interface (SPI) protocol. A serial clock (SCK) in the SPI communication determines the transmission speed, and we set the SCK frequency to 8 MHz, which enables the 8 Mbps communication, as shown in Figure 5b. Through this communication, intellino performs the recognition of one test dataset (1024-byte) in about 1ms at 50 MHz operation. Even if the trained dataset is increased, the recognition time is maintained thanks to the simultaneous calculating with parallelized neurons. This means that only the number of test dataset recognition determines the system operation time, not the number of trained datasets. Thus, the integrated embedded system takes 30 ms to recognize 30 test datasets, regardless of trained datasets. In order to process 30 datasets (frames) per second for real-time performance, intellino requires just 1.5 MHz operating frequency, which leads to the reduction in power consumption for the computation, rather than higher operating frequency. Consequentially, the experimental result demonstrates the feasibility of our proposal and real-time operation for embedded AI system with intellino.

7. Conclusions

In this paper, we presented the low-power processor, intellino, for embedded AI system. For intellino, we analyzed and optimized the distance-based AI algorithms for hardware realization. By fully excluding multiplication, the optimized AI algorithms enable intellino to operate under low power and exploit fewer resources. We also implemented the parallelized neurons as well as the optimized algorithms. By simultaneously computing in the neurons, the AI operation is performed shortly, and the system core reduces its workload. Furthermore, the designated protocol for repetitive learning/recognition improves the system performance. Thanks to these hardware designs, intellino can be applied to the embedded AI system, including the edge of IoT, while the multicore with deep neural network (DNN)- or spiking neural network (SNN)-based processors [18,20,26,39] becomes the solution of complex AI system. Intellino can conduct the classification for face expression, as described in Section 5, the recognition of driver condition or speaker with smart navigation, the inspection for surface mount technology (SMT), the notification of parking lot space, and other tiny AI applications. Thanks to the architecture of intellino and the feature of FPGA, intellino also has flexibility in specification. While other AI processors targeting the embedded system, such as Intel Curie with 128 N-cells and 128 vector sizes or General Vision NM500 with 576 N-cells and 256 vector sizes, have a fixed specification, the reconfigurability of intellino enables the system to be set with the optimized specification. Through the simulator based on the optimized algorithms, we evaluated intellino, which achieves over 95% accuracy. In addition, we analyzed the detailed workload of a system with intellino through the AI application. Finally, we verified intellino, which supports up to 88 MHz operation, with FPGA prototyping and demonstrated the feasibility of our proposal.

In future work, we will compare the performance of intellino with various AI algorithms and neural networks in aspect of the resource usage and the accuracy. A support vector machine (SVM) for detail classification, principal component analysis (PCA) for unsupervised learning, a DNN or SNN for complex application, and other algorithms and networks can be adopted for the advanced intellino. As the computation units for the algorithms and neural networks have an effect on the AI system, the computing optimization method and network communication will be considered. We will also integrate the hardware realized pre-processing, such as edge detection, STFT or object movement detection [40], with intellino to optimize for specific applications. Furthermore, we will apply the multiple intellino for a multicore AI system. Since the multicore AI system needs a specific module for controlling the cores [41], we will take the efficient multicore management into account. We expect that the multi-intellino will enhance the performance of embedded AI systems and the flexibility against diverse applications.