IoT-Based Image Recognition System for Smart Home-Delivered Meal Services

Abstract

Population ageing is an important global issue. The Taiwanese government has used various Internet of Things (IoT) applications in the “10-year long-term care program 2.0”. It is expected that the efficiency and effectiveness of long-term care services will be improved through IoT support. Home-delivered meal services for the elderly are important for home-based long-term care services. To ensure that the right meals are delivered to the right recipient at the right time, the runners need to take a picture of the meal recipient when the meal is delivered. This study uses the IoT-based image recognition system to design an integrated service to improve the management of image recognition. The core technology of this IoT-based image recognition system is statistical histogram-based k-means clustering for image segmentation. However, this method is time-consuming. Therefore, we proposed using the statistical histogram to obtain a probability density function of pixels of a figure and segmenting these with weighting for the same intensity. This aims to increase the computational performance and achieve the same results as k-means clustering. We combined histogram and k-means clustering in order to overcome the high computational cost for k-means clustering. The results indicate that the proposed method is significantly faster than k-means clustering by more than 10 times.

1. Introduction

1.1. Background

Population ageing is an important global issue [1]. Countries are forced to develop long-term care-related strategic planning and resource reorganization [2]. The rapidly ageing population has been the biggest concern for the Taiwanese government [3]. Taiwan’s elderly population will reach 14.5% of the total population in 2018 to create an elderly society, while this will be as high as 20.6% in 2026, transforming Taiwan into an extremely elderly society [4]. With the rapid growth in the elderly population, the resulting long-term demands and family care responsibilities will become increasingly heavy. In order to construct a long-term care system that meets the needs of the elderly as well as the physically and mentally handicapped, the Executive Yuan of Taiwan passed the “10-year long-term care program 2.0” on 29 September 2016 [5]. Meanwhile, the Taiwan government strongly applied the use of various Internet of Things (IoT) applications in the “10-year long-term care program 2.0”. It is expected that the long-term care system will be improved through information and communication technology (ICT) support [6,7].

“Ten-year long-term care program 2.0” is divided into A, B, and C levels for service. A-level is institutional care; B-level is community care; and C-level is family care [8]. This will be linked by information and communication technologies (ICTs) to overcome the dilemma of long-term care services having different standards and lacking integration.

Based on the vision of the “10-year long-term care program 2.0”, Taiwan is building an IoT system of long-term care in order to overcome the shortcomings of the existing long-term care services. Managing the nutrition of the elderly in home-based care was difficult when the 1.0 plan was implemented. Therefore, this was prioritized in the application of IoT technical assistance in this newer plan [9]. This study uses the IoT-based image recognition system to design smart home-delivered meal services and to provide an integrated service to provide nutrition to home-based elderly individuals. During the long-term care program 1.0, it was difficult to audit the service. Implementing smart home-delivered meal services will help solve this long-standing problem.

At the time that meals are delivered, we will use this application to confirm that our service can be delivered to the right person in the right place and at the right time. At the same time, information about the elderly individual’s physical status, psychological status, and dining status can also be collected. Following this, we pass the relevant information through the IoT technology for analysis, calculation, and interpretation. Eventually, the results of analysis will be sent back to the long-term care 2.0 care management unit for case assessment, risk forecasting, and care management aspects.

The core principle of this IoT-based image recognition system for smart home-delivered meal services is the statistical histogram-based k-means clustering (HKMC) for image segmentation. Segmentation is a process of decomposing certain interesting objects or some constituting regions that have similar characteristics [10,11,12]. The simplest methods of image segmentation involve thresholding. The thresholding method is a technique of segmenting an image depending on the intensity value of pixels and the intensity nature. We can segment objective or defective areas away from the background through a thresholding method when their gray-scale values are significantly different from the image’s background. This method can also be applied to the medical images of X-ray-computed tomography [13,14]. Due to the different structures of the human body possibly having a similar radiopacity, it is not easy to partition them through adjusting imaging parameters. The solution is the thresholding method of segmentation.

Until now, there have been several well-known thresholding methods, such as Otsu’s method [15], the maximum entropy method [16], k-means clustering (KMC) [17,18], and so on. K-means clustering [8,9] requires the input of every pixel one by one in every iterative process, which slows down the computational speed. When an image is larger than 25 megapixels, the segmentation results of the original KMC cannot be displayed in real time. For this reason, the time-consuming phenomenon of the k-means algorithm is regarded as a fatal shortcoming in our study.

Therefore, this study proposes the k-means clustering method based on the statistical histogram. Without altering the basic sense of the KMC method, we can retain the image segmentation effect of the algorithm in addition to enhancing the time and speed of image calculation. The present study used the image segmentation dataset of University of California Berkeley Electrical Engineering and Computer Sciences (UC Berkeley EECS) to perform an experiment, which compares the image segmentation effect and time rate of the original method with the ones of the k-means clustering method based on the statistical histogram. The results show that the method we propose is much faster than traditional k-means clustering. In the multi-level segmentation of the traditional k-means, more clusters can lead to a clearly slower speed due to the increase in the number of clusters. However, there is a significant increase when HKMC is applied. Although the clustering numbers of segmentation are the control variables, the operation time for HKMC is significant less when compared to that of the original k-means.

1.2. Related Works/State-of-the-Art

1.2.1. Image Recognition Systems

Image segmentation technology is an important research topic in the field of computer vision, which is the necessary basis for the application of image recognition systems. Zhang [19] adopted the gray image histogram alignment algorithm to develop a Matlab-based image recognition system. The Matlab-based image recognition system has a higher recognition rate. However, the system has some limitations related to potential applications. For example, the Matlab-based image recognition system only recognizes images in the standard face database of Orlando. However, the actual face images are often affected by many factors that can influence the recognition rate of the Matlab-based image recognition system. Therefore, the Matlab-based image recognition system is unable to have a wide range of applications. Dai et al. [20] proposed a new feedforward neural networks (FNN) algorithm for image recognition to solve that drawback of the traditional FNN approach. With the traditional FNN approach, important information regarding the correlations of elements within the original matrices is lost. On the other hand, expanding the matrix inputs into vectors usually causes a high dimensionality and increases the complexity. The new FNN algorithm was found to be effective for handwritten digits and face-images. Zhang et al. [21] combined the independent component analysis (ICA) and the extreme learning machine (ELM) algorithms to solve the image recognition speed problem, which is valid for face recognition. Due to the training of the image recognition classifier being time-consuming, Cao et al. [22] used the bag-of-words (BoW) method to recognize images of landmarks, although this can only be used for landmark recognition.

The person delivering meals to the elderly may have time constraints and may pay little attention to recording the results of the service. This distortion of the images will limit IoT-related services. Therefore, this study applies the image segmentation technology to improve the efficiency and image quality to overcome the abovementioned problems, such as the time-consuming nature, image quality, and image content type (e.g., only face or landmark recognition).

With the progress of information technology, the arrival of the Internet of Things era has increased the demand for image processing as well as an increasing number of hardware devices with camera lens for manufacturing, business behavior, and even in daily life. Different applications of information systems gradually rely on the automated image recognition technology to complete a variety of new applications. The IoT-based image recognition applications are shown as follows. Researchers have used the combination of IoT and image recognition systems to provide intelligent services for museum visitors. They designed an indoor location-aware architecture that enhances the museum’s user experience and created a wearable device to provide visitors with an interactive cultural experience, which automatically provides visitors with cultural content related to the observed artwork [23].

One of the most important factors in human survival is agricultural production, with many experts committed to making agricultural production better, more stable, and productive. In the field of agricultural science and technology, Chang et al. [24] integrated the Internet of Things and image recognition systems to design a monitoring system for persimmon cultivation. The study deployed weather boxes and WiFi cameras on farmland to collect local weather data, images, and crop growth data. This system provides an accurate identification of persimmon growth stages.

The advancement of information technology has played an important role in medical care, having become a hot area for industrial and academic research in recent years. At present, some related research has used the IoT and image recognition systems to help solve the problem of population ageing. Mano et al. [25] used cameras and image processing in the Internet of Things to help patients and/or elderly individuals to devise a system to handle portraiture recognition and emotional recognition to assist patients and/or the elderly individuals to recover faster from illness. Therefore, based on the above research and applications, this study is based on the perspective of pension health management, and aims to design a smart home-delivered meal service to improve the meal delivery process and strengthen a friendly service management design. The core idea of this study is to extend the concept of continuous collection of information on the health of the elderly. Caregivers can use the relevant devices to continuously and remotely care for elderly individuals without disturbing their lives.

1.2.2. K-Means Clustering Method Based on the Statistical Histogram

The histogram is a two-dimensional statistical table. The two coordinates are the statistical sample and the measurement of this property [26]. In the field of image processing, the histogram is used to count a gray-scale image, with the horizontal axis being the gray-scale value and the vertical axis being the number of pixels of the gray-scale value [27]. In addition to providing useful statistical information, it is useful for the enhancement of the image as well as to compress and segment image processing [28]. Some related works on HKMC are introduced as follows.

Studies were conducted using k-means clustering and histogram-clustering methods to track tumor objects in magnetic resonance (MR) brain images. The study used the HKMC method to convert the gray-scale MRI (magnetic resonance imaging) images into color images, before effectively separating the tumor from the image to assist the pathologist in distinguishing the lesion area and size of the tumor [29]. Another study aimed to analyze color space characteristics, including hue, saturation, and color. The study extracted pixel features by selecting the hue or intensity based on the pixel saturation value as the primary attribute to identify the objects in the image more precisely. In this study, the histogram maintains a uniform color conversion, allowing the image to undergo window-based smoothing during retrieval [28]. In addition, there also has been research related to the integration of the HKM motion detection with segmentation technology instead of using only a simple and universal segmentation method [28].

2. Materials and Methods

2.1. Materials

For the convenience of evaluating the segmentation results and the computational performance, our research images were captured from UC Berkeley EECS’s specialized segmentation dataset on an online address [30]. The images were copyright-free. Due to the abundance, we selected three categories from the segmentation dataset, which were figures, landscapes, and buildings. There were two images selected for each category, resulting in a total of six images for evaluation. In addition, the image pixel size has two types of specifications, which were 321 × 481 or 481 × 321.

2.2. K-Means Clustering

This is a clustering process used to classify training samples so that samples within a cluster are more similar to one another than samples belonging to different clusters. It is employed for similarity measures to classify samples, which depends on the object characteristics. Many clustering approaches, each with their own special characteristics, have been demonstrated [18]. In this paper, the k-means clustering method is used to classify the pixels into a certain number of clusters, with this classification approach changing a gray-scale image in a multi-level image in the end.

K-means clustering [17,31] is a simple unsupervised learning algorithm, which classifies the given data into a certain number of clusters (k-clusters). First, the procedure of k-means calculates the centroid for each cluster. The best way to position all centroids is to place them as far away from each other as possible. Each point is then associated with the nearest centroid according to some criteria. Following this, we re-calculated and updated these new k-centroids. After updating the position of each centroid, a new binding is repeated for the new point. In the loop, the k-centroids updated their location step by step until there were no more changes. The k-means approach, similar to other clustering techniques, minimizes an objective function to achieve the smallest squared error. The aim of the algorithm is to minimize the objective function:

$$J={\displaystyle \sum _{j=1}^k{\displaystyle \sum _{i=1}^n{\Vert x_i-c_j\Vert }^2}},$$

(1)

where x_i represents each observation; c_j is the mean of points in cluster j; and k and n stand for the number of clusters and observations, respectively. In Equation (1), the cluster centroids can be regarded as prototypes for the clusters. For the purpose of minimizing the objective function, the cluster centroids are chosen so that the highest values are assigned to the samples closest to the corresponding cluster centroids.

2.3. K-Means Based on Histogram Clustering

Histogram-based k-means clustering (HKMC) also aims to minimize the value of Equation (1); thus, it is not different from the iterative method of the original KMC. The difference between these two methods is the pixels’ entrance process. Instead of utilizing the one-by-one process, HKMC conducts pre-processing through calculating the number of the gray-scale values. Following this, HKMC obtains a discrete p.d.f. (probability density function) of the gray-scale values, which can also be represented as a discrete statistical histogram form. After this pre-processing, we entered the histogram into the HKMC with weighting and the gray-scale values. In other words, HKMC combines the iterative method with the statistical histogram.

The Algorithm 1 of the proposed HKMC is interpreted as follows:

In step 1, we separated the random k-centroids according to the discrete p.d.f. In step 2, we computed the distance of the gray-scale values for each initial centroid, with the gray-scale value then assigned to the closest centroid cluster. In step 3, we recalculated the centroids (namely, the new centroids) for each cluster using the gray-scale values and their weighting. Finally, we repeated steps 2 and 3 until converge was achieved.

The HKMC ensures convergence, although this does not guarantee that the optimal solution will be obtained. The factors influencing the convergence are the number of clusters and the value of the initial centroid. HKMC still encounters the local minimum problem, similar to traditional KMC. For this reason, this should be conducted several times in order to obtain more accurate segments.

Algorithm 1. Histogram-based k-means clustering

For (i = 0, i < K, i++) // Initial the K centers

InitialCenter [i] = rand ();

Do loop

Center [i] ← InitialCenter [i]

End

For (Gray_value = 0, Gray_value < 256, Gray_value ++)

If (Float_absolute_value (Gray_value-Center [i]) ≤ Minimal_ distance)

Cluster [i] ←Gray_value

NewCenter [i] ←

(${\sum }_{i=0}^{i=n-1}\mathrm{Cluster}\left[\mathrm{i}\right]\times \mathrm{NumbOfPixels}$)/n

Center [i] ← NewCenter [i]

End

Center [i] = NewCenter [i]

End

2.4. Speed Evaluation

In order to evaluate the computational performance precisely, our study considered the image size and time units. The size of each image is 154,401 (321 × 481 or 481 × 321) pixels, with its processing time taking less than one second. Thus, we adopted ticks as the time unit for assessment, which is equal to one microsecond. We repeated 20 runs for each image. In addition, the experimental environment was the Microsoft Windows 7 Professional Edition Service Pack 1 64-bit through the Intel Core i7 2600 processor with 8 GB memory (DDR3 1333 4GB × 2). The algorithm development tool was Microsoft Visual C++ 6.0.

2.5. Privacy/Trust Viewpoint Design

In the planning and design of smart home-delivered meal services, we considered the use of privacy optimization technology. Incorporating the protection principles for the privacy of personal data into the overall system design is the basis for the concept of privacy design in addition to being a key factor for the success of IoT applications.

In the design of the system, we conducted a privacy impact analysis (PIA), so that potential privacy risks can be controlled. In addition, we also planned to import personal data management systems and verification operations, such as ISO 27001, BS-10012, ISO 29100, or ISO 29191.

In addition, we also considered the obligation to protect personal data and the right of individuals to know. Before we started the service, we informed the individual about the use of their images and our privacy obligations; sought his/her consent and authorization; and obtained personal verbal trust. After all this was conducted, we began to collect the relevant information.

3. Experimental Results and Discussion

For the experiment, we chose the major constituent areas as the images of personages, landscapes, and buildings. Following this, we discussed the experimental segmentation results and operation speeds of binary processing, quad processing, hexad processing, and octad processing images. We chose two images from each category for the experiment, with each image evaluated 10 times. After this, we chose the two most representative images (they are respectively in

Table 1. Segmentation result of a personage.

Original Image		Binary	Quad	Hexad	Octad
	K-Means Clustering (KMC)
	Histogram-based K-Means Clustering (HKMC)

,

Table 2. Segmentation result of landscapes.

Original Image	Method	No.	Binary	Quad	Hexad	Octad
	KMC	1
		2
		3
		4
	HKMC	1
		2
		3
		4

and

Table 3. Segmentation result of buildings.

Original Image		Binary	Quad	Hexad	Octad
	KMC
	HKMC

) for discussion and comparison. Furthermore, we used the tables (as shown in

Table 4. Speed comparison of binary processing.

Binary
	Personage	Landscapes	Buildings
Average Speed of KMC (ms)	135.4	138.15	133.5
Average Speed of HKMC (ms)	12.2	12.5	12.35
Ratio	11.098	11.052	10.809

,

Table 5. Speed comparison of quad processing.

Quad
	Personage	Landscapes	Buildings
Average Speed of KMC (ms)	298.6	304.95	468.2
Average Speed of HKMC (ms)	15.5	15.1	17.05
Ratio	19.264	20.195	27.46

and

Table 6. Speed comparison of hexad processing.

Hexad
	Personage	Landscapes	Buildings
Average Speed of KMC (ms)	838.5	685.85	600.75
Average Speed of HKMC (ms)	20.15	18.95	16.75
Ratio	41.612	36.192	35.865

) of processing speeds for assessment. For our experiments, we used milliseconds (ms) as the unit of assessment data and we used multiples to quantify the increase in time.

3.1. Segmentation Results of Personages

Table 1 shows the segmentation results of a personage. We can clearly see that binary processing can separate personages from the complex backgrounds, while quad processing can allow us to approximately identify the personage in the background (the personage dressed in blue). Furthermore, hexad processing can allow us to clearly identify the clothing colors of the elderly lady with gradations, while octad processing can allow us to clearly see the contours of the elderly lady and the clothing colors with obvious contrasts. It can be seen from the segmentation results that the background and personage can be clearly identified, with an obvious contrast between the cloth colors. In addition, the segmentation performances of these two methods are fairly consistent.

3.2. Segmentation Results of Landscapes

Table 2 depicts the segmentation results of landscapes. It can be seen that the whole picture is mainly composed of the blue water and sky, followed by green trees and white boats. The segmentation result of the blue part is very unstable, which may possibly be due to the different initial values of the algorithm having the problem of a local minimum. Therefore, while the values of blue colors are quite similar, each experiment has different initial values and clustering results that classify the similar values of blue colors as different categories. This leads to inconsistent segmentation results. Running more experiments allows us to produce the best segmentation images under the specified numbers of clusters.

With regard to the segmentation results of binary processing, we obtained segmentation results of two images out of the 10 original images. We can only see the white boat in one of them, which is very different from the blue and green colors. In contrast, we see the white boat as well as the blue sky and green trees on the surface of the water and the island in the other image. Thus, the latter image has a better segmentation result.

In terms of quad processing, the boat, water, sky, and the trees of the island are separated from the background, but the gray-scaled shading values and contrast are different. Intuitively, the second image can be regarded as having the better segmentation result, with the segmentation result of the third image being the worst. This is because the sky and water are classified as the same category and have the same gray-scaled value. In comparison with the first image, there is a reflected image of an unknown object in the first image, which is possibly a cloud, smoke, or aircraft. However, the segmentation result is inconsistent and more experiments need to be performed to obtain more accurate segmentation results for the original images.

In terms of hexad processing, we can see more information, including the boat, sky, water, trees on the island, and bridge. The problem of inconsistency in each segmentation result still exists, but the sky and water of some images begin to have the gray-scaled values that are different from those of neighboring regions.

In terms of octad processing, the boat, sky, water, trees on the island, and viaduct are separated from the background, with an increase in the differences between the gray-scaled values of the sky and water. It seems that the blue value is different, so the reflected image and the other areas of the sky are more clearly separated from the other objects.

3.3. Segmentation Result of Buildings

Table 3 shows the segmentation results of buildings. It can be seen from the binary segmentation that the main building is clearly separated from the long-ranged view. The results of both methods are quite consistent. Furthermore, multiple segmentation results will not appear due to the similarity of colors.

For quad processing, the sky of the lower right corner begins to have gray-scaled values that are different from those of neighboring regions. This is closely followed by octad processing, with the sky gradually beginning to have different segmentation colors.

3.4. Speed Assessment

The speed assessment can be divided into two parts. The first one is as described in Table 4, Table 5, Table 6 and

Table 7. Speed comparison of octad processing.

Octad
	Personage	Landscapes	Buildings
Average Speed of KMC (ms)	1294.75	1227.1	1315.4
Average Speed of HKMC (ms)	22.75	22.9	24.3
Ratio	56.912	53.585	54.131

, which show the experiments of binary processing, quad processing, hexad processing, and octad processing. These also include two clustering methods of these three types of images, the average operation speed, and the multiples of increasing speeds.

3.4.1. Binary Processing Speed

Table 4 depicts the speed comparison of binary processing. HKMC is significantly faster than the original KMC in these three images. The fastest speed is the image of a personage, which is 12 times faster. The speed of the image of a building is also 11 times faster, while the average speed of the three categories of images is 11 times fasters. This is the speed performance of an image with 321 × 481 pixels. In fact, when the image has 2048 × 1365 pixels, the increase in the speed of binary processing can be 20 times faster.

3.4.2. Quad Processing Speed

Table 5 is the speed comparison of quad processing. HKMC is significantly faster than the original KMC in these three images. The fastest speed is the image of a personage, which is 27 times faster. The speed of the images of personages is higher than 19 times, while the average speed of the three categories is 22 times the original one. We found that when the clustering numbers is increased to four, the speed of the k-Means clustering method based on the statistical histogram can become faster and faster with a speed multiple of two.

3.4.3. Hexad Processing Speed

Table 6 shows the speed comparison of hexad processing. HKMC is significantly faster than the original KMC in these three images. The fastest speed is the image of a personage, which is 41 times faster. The images of buildings can be 35 times faster, while the average speed of the three categories is 37 times the original one. We found that when the clustering numbers are increased to six, HKMC also is significantly faster and further enhances the leading multiple to more than 37 times. It is important to note that its leading multiple has already reached a speed that is 33 times the speed of binary processing.

3.4.4. Octad Processing Speed

Table 7 depicts the speed comparison of octad processing. It is not influenced by an increase in clustering numbers. The fastest speed is the image of a personage, which is 56 times faster. The image of a landscape is 53 times faster, which is close to the one of a personage, while the average speed of the three categories is 54 times. When the clustering numbers are increased to eight, the speed of HKMC is faster than that of the original KMC, and the leading multiple is increased to more than 54 times.

3.5. Discussions

As can be seen from Table 1, Table 2 and Table 3, the KMC method and KMC method based on the statistical histogram almost have no significant differences in the effects of binary segmentation and multi-valued segmentations. The result will be the same if the initial values of both are the same. Therefore, more segmentations must be performed to obtain more accurate results, while the segmentation images of better results need to be chosen using different initial values.

As can be seen from Table 4, Table 5, Table 6 and Table 7, when using binary processing, the speed of the HKMC method is 11 times that of the original KMC method, and its quad processing speed increases 22 times. The hexad processing speed increases 37 times and the octad processing speed increases 54 times. We found that the speed can be increased when there is an increase in the number of clusters in HKMC.

Furthermore, with regards to the clustering numbers and computing time, the time range of the original KMC method is increased by 842%. HKMC only increases the computing time by 88%, with the increase in time range being 9.56 times that of HKMC. From the perspective of speed, this obvious increase in time has proved that the HKMC is more applicable to multi-valued segmentation compared to the original KMC. This is because more segmentation values can lead to more inconsistent results, with more experiments needed to conduct better segmentation images.

4. Conclusions

This study proposes a method that can significantly improve the deficits of the original KMC method. Furthermore, the image experiments prove that there is no significant difference between the KMC method and the KMC method based on the statistical histogram when using binary, quad, hexad, and octad processing. At the same time, the obvious enhancement of computing speed is supported in the experiment. In the segmentation by binary processing, the speed of the k-means clustering method based on the statistical histogram is 11 times faster. This speed is 22 times faster in the segmentation by quad processing; 37 times faster in the segmentation by hexad processing; and 54 times faster in the segmentation by octad processing. However, HKMC only increases the time by 88%, while the original KMC increases the computing time by 842%. Therefore, the speed of the k-means clustering method based on the statistical histogram proposed in this article is indeed faster than that of the original KMC method, with no subsequent influences on the segmentation results. Therefore, the HKMC method can be applied more readily to multi-valued segmentations. Finally, this study suggests that future plans can be combined with more IoT-based life applications, such as Unmanned Aerial Vehicle (UAV) home delivery, parcel delivery, and other intelligent application designs to promote image recognition technology.