An Intelligent Microwave Oven with Thermal Imaging and Temperature Recommendation Using Deep Learning
- A closed-loop microwave oven is designed which continuously measures the food temperature using a thermal camera while the food is being heated, and stops the heating automatically when the food temperature reaches the target temperature. Thus, the user does not need to calculate the exact needed time in the head. This is easier for the user and ensures the precise target temperature of the food. On a graphical liquid crystal display (LCD), the thermal image of the food is shown in real-time while the food is being heated.
- The paper also proposes an automatic target temperature recommendation method. After food is put in and the door is shut, it captures an image of the food using a camera and trains a deep learning-based image classifier. When a target temperature is set for that food by the user for the first time, that temperature is assigned to that image class. Later, when the user heats the same class of food again (even if the food is on a different stir or transformation condition), the proposed method automatically classifies the food and recommends the target temperature which was assigned to that food class previously. Thus, the user does not need to recall and re-enter the target temperature. The method uses a Convolutional Neural Network (CNN) to classify the images. Whenever a new food item is inserted or an image is misclassified, the method retrains the deep learning model in real-time. In this way, the proposed microwave oven progressively learns the food items that are consumed in that family and becomes smarter in recommending the target temperature.
- The expected shipment of microwaves is 13.5 million units in 2019  and 96% of Americans use this product . According to the survey in , microwaves are used by the Americans to warm and heat more, rather than cook dishes. The microwave oven is behind compared with other major appliances in terms of smart and Artificial Intelligence (AI) features . The proposed machine learning-based intelligent microwave can fill-up this gap in research.
2. Related Works
3. Materials and Methods
3.1. Deep Learning-based Temperature Recommendation Method
3.1.1. Dataset Generation
3.1.2. The Architecture of the Convolutional Neural Network
- Input Image: The input image is a tensor of size (24, 32, 3) having distinct channels for red, green and blue. The data type of the image pixel is transformed into a floating-point. For normalization, pixels are also divided by 255, so the numbers range from 0 to 1.
- Convolutional Layer: Sliding convolutional filters are applied in a 2-D convolutional layer on the input. The filters are moved along the input vertically and horizontally. It computes the dot product of the weights and the input, and then the bias term is added . One convolutional layer having filter sizes of 3 × 3 is used in the proposed model. The filters are learnable network parameters and they are initialized with random values. Conv2d_1 layer in Figure 3 has 16 filters of size 3 × 3 with padding and they generate 16 output layers having the same input layer’s height and width.
- Activation Layer: The Rectified Linear Unit (ReLU) is a nonlinear activation function . This layer is used after the convolutional layer and the dense layers (except the last dense layer). Any element value less than zero is set to zero by the ReLU layer.
- Max Pooling Layer: Down-sampling by dividing the input into rectangular pooling areas and calculating the maximum of each area  is performed by a max-pooling layer. The sizes of the pooling area are set to 2 × 2 in the proposed model.
- Dropout Layer: Input elements are randomly set to zero with a given probability by a dropout layer. This action changes the original network architecture between iterations and helps avoid the network from overfitting . This layer has no learnable parameter. To prevent overfitting of training data within a few epochs, one dropout layer is used in the proposed model. The probability of the dropout layer is set to 0.10.
- Flatten Layer: The input becomes a single column vector by a flatten layer. It breakdowns the spatial dimensions of the input. The flatten layer in this model changes the (12, 16, 16) tensor to a one-dimensional vector of size 3072.
- Loss Function and Optimizer: A loss function measures the agreement between the predicted scores and the ground truth labels, and an optimizer tries to reach the global minima where the loss function reaches the least possible value for the network parameters. The features to classify the images are combined in the last fully connected layer–dense layer 3. Therefore, the last dense layer’s output size is the total number of classes, C, and it will increase with the addition of new image classes. When a food image is captured by the microwave and the user identifies it as a new class of food, then the total number of class, C, is incremented by one. When C is 1, no CNN model is generated, as two classes are required as a minimum for classification. When C becomes greater than one, the cross-entropy loss or softmax loss [25,26] is calculated and RMSprop  optimizer is used. The CNN is then trained and the model file is generated in the embedded system for the new classes. The proposed CNN architecture has a total of 3,409,857 trainable parameters.
3.2. Food Temperature Calculation from Thermal Image
3.3. Prototype Development
Prototype Development Result
Conflicts of Interest
- Osepchuk, J.M. A History of Microwave Heating Applications. IEEE Trans. Microw. Theory Tech. 1984, 32, 1200–1224. [Google Scholar] [CrossRef]
- Total Unit Shipments of Microwave Ovens in the U.S. from 2005 to 2019. Available online: https://www.statista.com/statistics/220122/unit-shipments-of-microwave-ovens/ (accessed on 10 December 2019).
- Williams, A. Surveys of Microwave Ovens in U.S. Homes; Lawrence Berkeley National Laboratory: Berkeley, CA, USA, 2014; Available online: https://escholarship.org/uc/item/3s29h7wd (accessed on 10 December 2019).
- Americans are Eating at Home More; Microwave Usage Increases but not Cooking; NPD: Toronto, ON, Canada, 2009; Available online: https://www.npd.com/wps/portal/npd/us/news/press-releases/pr_091112/ (accessed on 10 December 2019).
- Microwaves in the US; Euromonitor: London, UK, 2019; Available online: http://www.euromonitor.com/microwaves-in-the-us/report (accessed on 10 December 2019).
- Sun, M.H.; Wickersheim, K.A.; Kim, J.H. Fiber optic temperature sensors in the medical setting. Proc. SPIE 1989, 1067, 15–21. [Google Scholar]
- Belotserkovsky, E.; Shenfeld, O.; Katzir, A. Infrared fiberoptic temperature control of the heating process in a microwave oven. IEEE Trans. Microw. Theory Tech. 1994, 42, 901–903. [Google Scholar] [CrossRef]
- Cuccurullo, G.; Giordano, L.; Albanes, D.; Cinquanta, L.; Matteo, M.D. Infrared thermography assisted control for apples microwave drying. J. Food Eng. 2012, 112, 319–325. [Google Scholar] [CrossRef]
- Bows, J.; Joshi, K. Infrared imaging feels the heat in microwave oven. Phys. World 1992, 5, 21–22. [Google Scholar] [CrossRef]
- Liyan, C.; Min, G.; Qingyu, X.; Junhao, W.; Ning, X. Temperature monitoring based on image processing for intelligent microwave heating. In Proceedings of the 27th Chinese Control and Decision Conference (2015 CCDC), Qingdao, China, 23–25 May 2015; pp. 1397–1401. [Google Scholar]
- Sanchez, I.; Banga, J.R.; Alonso, A.A. Temperature control in microwave combination ovens. J. Food Eng. 2000, 46, 21–29. [Google Scholar] [CrossRef]
- Cabrera, J.M.; Molina, J.L.P.; Toledo, A. Feedback control procedure for energy efficiency optimization of microwave-heating ovens. Measurement 2009, 42, 1257–1262. [Google Scholar] [CrossRef][Green Version]
- Li, Z.; Raghavan, G.S.V.; Orsat, V. Optimal power control strategies in microwave drying. J. Food Eng. 2010, 99, 263–268. [Google Scholar] [CrossRef]
- June Intelligent Oven. 2019. Available online: https://juneoven.com (accessed on 10 December 2019).
- Khan, T.H. Smart microwave oven with image classification and temperature recommendation algorithm. Int. J. Electr. Comput. Eng. (IJECE) 2018, 8, 4239–4252. [Google Scholar] [CrossRef]
- Khan, T.H. Towards an autonomous temperature feedback microwave oven with thermal imaging. In Proceedings of the IEEE International Conference on Electro Information Technology (EIT 2018), Rochester, MI, USA, May 2018; pp. 444–448. [Google Scholar]
- Shorten, C.; Khoshgoftaar, T.M. A survey on Image Data Augmentation for Deep Learning. J. Big Data 2019, 6. [Google Scholar] [CrossRef]
- Keras: The Python Deep Learning library. 2019. Available online: https://keras.io/ (accessed on 11 December 2019).
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Nair, V.; Geoffrey, E.H. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel, 21–24 June 2010; pp. 807–814. [Google Scholar]
- Nagi, J.; Ducatelle, F.; di Caro, G.A.; Ciresan, D.; Meier, U.; Giusti, A.; Nagi, F.; Schmidhuber, J.; Gambardella, L.M. Max-Pooling Convolutional Neural Networks for Vision-based Hand Gesture Recognition. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications (ICSIPA2011), Kuala Lumpur, Malaysia, 16–18 November 2011. [Google Scholar]
- Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. 2014, 15, 1929–1958. [Google Scholar]
- Glorot, X.; Yoshua, B. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, Sardinia, Italy, 13–15 May 2010; pp. 249–256. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1026–1034. [Google Scholar]
- Bishop, C.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006. [Google Scholar]
- Softmax Activation Function. Available online: https://keras.io/activations/#softmax (accessed on 19 December 2019).
- Gandhi, R. A Look at Gradient Descent and RMSprop Optimizers. Available online: https://towardsdatascience.com/a-look-at-gradient-descent-and-rmsprop-optimizers-f77d483ef08b (accessed on 11 December 2019).
- FLIR. Radiometric Lepton Dev Kit. Available online: https://www.sparkfun.com/products/14654 (accessed on 12 December 2019).
- Microwave Oven. Available online: http://hyperphysics.phy-astr.gsu.edu/hbase/Waves/mwoven.html (accessed on 12 December 2019).
- Infrared. Available online: http://hyperphysics.phy-astr.gsu.edu/hbase/ems3.html (accessed on 12 December 2019).
- The Physics of Microwave Ovens. Available online: http://tobyzerner.com/microwaves (accessed on 12 December 2019).
- OpenCV Contours. Available online: https://docs.opencv.org/trunk/d4/d73/tutorial_py_contours_begin.html (accessed on 12 December 2019).
- ColorMaps in OpenCV. Available online: https://docs.opencv.org/2.4/modules/contrib/doc/facerec/colormaps.html (accessed on 12 December 2019).
- Amana RMS10DS Light-Duty Commercial Microwave. Available online: https://www.amazon.com/Amana-Medium-Duty-Microwave-Oven/dp/B07MXLXZFV (accessed on 18 December 2019).
- Raspberry, Pi. Available online: https://www.raspberrypi.org (accessed on 12 December 2019).
- Khan, T.H.; Wahid, K. Universal bridge interface for DVP-compatible image sensors. Microprocess. Microsyst. Elsevier 2011, 35, 547–556. [Google Scholar] [CrossRef]
- Raspberry Pi LCD-7” Touchscreen. Available online: https://www.sparkfun.com/products/13733 (accessed on 12 December 2019).
- Raspberry Pi camera Module. Available online: https://www.raspberrypi.org/products/camera-module-v2 (accessed on 12 December 2019).
- Hamburger Mini Speaker. Available online: https://www.sparkfun.com/products/14023 (accessed on 12 December 2019).
- Daylight White E17 LED Bulb. Available online: https://www.amazon.com/dp/B01KX765TE (accessed on 12 December 2019).
- 5v Relay Module. Available online: https://www.amazon.com/Tolako-Arduino-Indicator-Channel-Official/dp/B00VRUAHLE/ (accessed on 12 December 2019).
- Solid State Relay. Available online: https://www.sparkfun.com/products/13015 (accessed on 12 December 2019).
- DC Power Supply. Available online: https://www.sparkfun.com/products/13831 (accessed on 12 December 2019).
- Qt Designer. Available online: https://www.riverbankcomputing.com/static/Docs/PyQt5/designer.html (accessed on 13 December 2019).
- PyQt5 Tutorial. Available online: https://pythonspot.com/pyqt5/ (accessed on 13 December 2019).
- AWS Deep Learning. Available online: https://aws.amazon.com/machine-learning/amis/ (accessed on 23 December 2019).
- Transparent Microwave Plate Cover. Available online: https://www.amazon.com/Re-usable-flexible-Microwave-Transparent-Splatter/dp/B07PJP6MCS (accessed on 31 January 2020).
- Raspberry Pi 3 vs. Raspberry Pi Zero W. Available online: https://www.arrow.com/en/research-and-events/articles/raspberry-pi-3-vs-raspberry-pi-zero-w (accessed on 3 February 2020).
|Work||Contactless Temperature Sensing||Automatic Heating Turn-off||Automatic Target Temperature Recommendation||Ability to Learn Target Temperature of New Food’s||Real-Time Thermal Imaging||Notification in Smartphone|
|E. Belotserkovsky et al. ||Yes, InfraRed (IR) fiberoptic radiometer||No||No||No||No||No|
|G. Cuccurullo et al. ||Yes, Forward-Looking InfraRed (FLIR) camera||A Personal Computer (PC) Controls power level||No||No, apple slices only||Yes||No|
|J. Bows et al. ||Yes, FLIR camera||No||No||No||Yes||No|
|C. Liyan et al. ||Yes, Charge-Coupled Device (CCD) camera||No||No||No||No||No|
|June Intelligent Oven ||No, Resistance Temperature Detector (RTD) temperature sensor||Yes||Yes, for a set of preprogrammed food||No||No||Yes|
|T. Khan ||Yes, the IR temperature sensor||Yes, an embedded system based control||Yes, the histogram-based image classifier||Yes||No||No|
|T. Khan ||Yes, 8 × 8 IR temperature sensor grid||Yes, an embedded system based control||No||No||Yes, low resolution||No|
|Proposed||Yes, using a FLIR camera||Yes, an embedded system based control||Yes, Convolutional Neural Network (CNN) based deep learning||Yes||Yes, high resolution||No|
|Total Class||Epochs||Training Accuracy||Validation Accuracy||Training Time (sec)|
© 2020 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Khan, T. An Intelligent Microwave Oven with Thermal Imaging and Temperature Recommendation Using Deep Learning. Appl. Syst. Innov. 2020, 3, 13. https://doi.org/10.3390/asi3010013
Khan T. An Intelligent Microwave Oven with Thermal Imaging and Temperature Recommendation Using Deep Learning. Applied System Innovation. 2020; 3(1):13. https://doi.org/10.3390/asi3010013Chicago/Turabian Style
Khan, Tareq. 2020. "An Intelligent Microwave Oven with Thermal Imaging and Temperature Recommendation Using Deep Learning" Applied System Innovation 3, no. 1: 13. https://doi.org/10.3390/asi3010013