Research

Open AccessArticle

LAD: Layer-Wise Adaptive Distillation for BERT Model Compression

by

Ying-Jia Lin

,

Kuan-Yu Chen

and

Hung-Yu Kao

Sensors 2023, 23(3), 1483; https://doi.org/10.3390/s23031483 - 28 Jan 2023

Viewed by 645

Abstract

Recent advances with large-scale pre-trained language models (e.g., BERT) have brought significant potential to natural language processing. However, the large model size hinders their use in IoT and edge devices. Several studies have utilized task-specific knowledge distillation to compress the pre-trained language models. [...] Read more.

Recent advances with large-scale pre-trained language models (e.g., BERT) have brought significant potential to natural language processing. However, the large model size hinders their use in IoT and edge devices. Several studies have utilized task-specific knowledge distillation to compress the pre-trained language models. However, to reduce the number of layers in a large model, a sound strategy for distilling knowledge to a student model with fewer layers than the teacher model is lacking. In this work, we present Layer-wise Adaptive Distillation (LAD), a task-specific distillation framework that can be used to reduce the model size of BERT. We design an iterative aggregation mechanism with multiple gate blocks in LAD to adaptively distill layer-wise internal knowledge from the teacher model to the student model. The proposed method enables an effective knowledge transfer process for a student model, without skipping any teacher layers. The experimental results show that both the six-layer and four-layer LAD student models outperform previous task-specific distillation approaches during GLUE tasks. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Smartwatch Sensors with Deep Learning to Predict the Purchase Intentions of Online Shoppers

by

Ray-I Chang

,

Chih-Yung Tsai

and

Pu Chung

Sensors 2023, 23(1), 430; https://doi.org/10.3390/s23010430 - 30 Dec 2022

Viewed by 566

Abstract

In the past decade, the scale of e-commerce has continued to grow. With the outbreak of the COVID-19 epidemic, brick-and-mortar businesses have been actively developing online channels where precision marketing has become the focus. This study proposed using the electrocardiography (ECG) recorded by [...] Read more.

In the past decade, the scale of e-commerce has continued to grow. With the outbreak of the COVID-19 epidemic, brick-and-mortar businesses have been actively developing online channels where precision marketing has become the focus. This study proposed using the electrocardiography (ECG) recorded by wearable devices (e.g., smartwatches) to judge purchase intentions through deep learning. The method of this study included a long short-term memory (LSTM) model supplemented by collective decisions. The experiment was divided into two stages. The first stage aimed to find the regularity of the ECG and verify the research by repeated measurement of a small number of subjects. A total of 201 ECGs were collected for deep learning, and the results showed that the accuracy rate of predicting purchase intention was 75.5%. Then, incremental learning was adopted to carry out the second stage of the experiment. In addition to adding subjects, it also filtered five different frequency ranges. This study employed the data augmentation method and used 480 ECGs for training, and the final accuracy rate reached 82.1%. This study could encourage online marketers to cooperate with health management companies with cross-domain big data analysis to further improve the accuracy of precision marketing. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Developing an Improved Ensemble Learning Approach for Predictive Maintenance in the Textile Manufacturing Process

by

Yu-Hsin Hung

Sensors 2022, 22(23), 9065; https://doi.org/10.3390/s22239065 - 22 Nov 2022

Viewed by 555

Abstract

With the rapid development of digital transformation, paper forms are digitalized as electronic forms (e-Forms). Existing data can be applied in predictive maintenance (PdM) for the enabling of intelligentization and automation manufacturing. This study aims to enhance the utilization of collected e-Form data [...] Read more.

With the rapid development of digital transformation, paper forms are digitalized as electronic forms (e-Forms). Existing data can be applied in predictive maintenance (PdM) for the enabling of intelligentization and automation manufacturing. This study aims to enhance the utilization of collected e-Form data though machine learning approaches and cloud computing to predict and provide maintenance actions. The ensemble learning approach (ELA) requires less computation time and has a simple hardware requirement; it is suitable for processing e-form data with specific attributes. This study proposed an improved ELA to predict the defective class of product data from a manufacturing site’s work order form. This study proposed the resource dispatching approach to arrange data with the corresponding emailing resource for automatic notification. This study’s novelty is the integration of cloud computing and an improved ELA for PdM to assist the textile product manufacturing process. The data analytics results show that the improved ensemble learning algorithm has over 98% accuracy and precision for defective product prediction. The validation results of the dispatching approach show that data can be correctly transmitted in a timely manner to the corresponding resource, along with a notification being sent to users. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Scale-Mark-Based Gauge Reading for Gauge Sensors in Real Environments with Light and Perspective Distortions

by

,

,

and

Sensors 2022, 22(19), 7490; https://doi.org/10.3390/s22197490 - 02 Oct 2022

Viewed by 691

Abstract

Nowadays, many old analog gauges still require the use of manual gauge reading. It is a time-consuming, expensive, and error-prone process. A cost-effective solution for automatic gauge reading has become a very important research topic. Traditionally, different types of gauges have their own [...] Read more.

Nowadays, many old analog gauges still require the use of manual gauge reading. It is a time-consuming, expensive, and error-prone process. A cost-effective solution for automatic gauge reading has become a very important research topic. Traditionally, different types of gauges have their own specific methods for gauge reading. This paper presents a systematized solution called SGR (Scale-mark-based Gauge Reading) to automatically read gauge values from different types of gauges. Since most gauges have scale marks (circular or in an arc), our SGR algorithm utilizes PCA (principal components analysis) to find the primary eigenvector of each scale mark. The intersection of these eigenvectors is extracted as the gauge center to ascertain the scale marks. Then, the endpoint of the gauge pointer is found to calculate the corresponding angles to the gauge’s center. Using OCR (optical character recognition), the corresponding dial values can be extracted to match with their scale marks. Finally, the gauge reading value is obtained by using the linear interpolation of these angles. Our experiments use four videos in real environments with light and perspective distortions. The gauges in the video are first detected by YOLOv4 and the detected regions are clipped as the input images. The obtained results show that SGR can automatically and successfully read gauge values. The average error of SGR is nearly 0.1% for the normal environment. When the environment becomes abnormal with respect to light and perspective distortions, the average error of SGR is still less than 0.5%. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Cost-Effective Fitting Model for Indoor Positioning Systems Based on Bluetooth Low Energy

by

,

,

,

and

Sensors 2022, 22(16), 6007; https://doi.org/10.3390/s22166007 - 11 Aug 2022

Cited by 1 | Viewed by 677

Abstract

Bluetooth Low Energy (BLE) is a positioning technology that is commonly used in indoor positioning systems (IPS) such as shopping malls or underground parking lots, because of its low power consumption and the low cost of Bluetooth devices. It also maintains high positioning [...] Read more.

Bluetooth Low Energy (BLE) is a positioning technology that is commonly used in indoor positioning systems (IPS) such as shopping malls or underground parking lots, because of its low power consumption and the low cost of Bluetooth devices. It also maintains high positioning accuracy. Since the cost of BLE itself is low, it has now been used in larger environments such as parking lots or shopping malls for a long time. However, it is necessary to configure a large number of devices in the environment to obtain accurate positioning results. The most accurate method of using signal strength for positioning is the signal pattern-matching method. The positioning result is compared through a database with the overheads of time and labor costs, since the amount of data will be proportional to the size of the environment for BLE-IPS. A planar model that conforms to the signal strength in the environment was generated, wherein the database comparison method is replaced by an equation solution, to improve various costs but diminish the positioning accuracy. In this paper, we propose to further replace the planar model with a cost-effective fitting model to both save costs and improve positioning accuracy. The experimental results demonstrate that this model can effectively reduce the average positioning error in distance by 31%. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Multilayer Reversible Information Hiding with Prediction-Error Expansion and Dynamic Threshold Analysis

by

,

,

and

Sensors 2022, 22(13), 4872; https://doi.org/10.3390/s22134872 - 28 Jun 2022

Viewed by 578

Abstract

The rapid development of internet and social media has driven the great requirement for information sharing and intelligent property protection. Therefore, reversible information embedding theory has marked some approaches for information security. Assuming reversibility, the original and embedded data must be completely restored. [...] Read more.

The rapid development of internet and social media has driven the great requirement for information sharing and intelligent property protection. Therefore, reversible information embedding theory has marked some approaches for information security. Assuming reversibility, the original and embedded data must be completely restored. In this paper, a high-capacity and multilayer reversible information hiding technique for digital images was presented. First, the integer Haar wavelet transform scheme converted the cover image from the spatial into the frequency domain that was used. Furthermore, we applied dynamic threshold analysis, the parameters of the predicted model, the location map, and the multilayer embedding method to improve the quality of the stego image and restore the cover image. In comparison with current algorithms, the proposed algorithm often had better embedding capacity versus image quality performance. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Edge Computing of Online Bounded-Error Query for Energy-Efficient IoT Sensors

by

Ray-I Chang

,

Jui-Hua Tsai

and

Chia-Hui Wang

Sensors 2022, 22(13), 4799; https://doi.org/10.3390/s22134799 - 24 Jun 2022

Cited by 1 | Viewed by 799

Abstract

Since the power of transmitting one-bit data is higher than that of computing one thousand lines of code in IoT (Internet of Things) applications, it is very important to reduce communication costs to save battery power and prolong system lifetime. In IoT sensors, [...] Read more.

Since the power of transmitting one-bit data is higher than that of computing one thousand lines of code in IoT (Internet of Things) applications, it is very important to reduce communication costs to save battery power and prolong system lifetime. In IoT sensors, the transformation of physical phenomena to data is usually with distortion (bounded-error tolerance). It introduces bounded-error data in IoT applications according to their required QoS² (quality-of-sensor service) or QoD (quality-of-decision making). In our previous work, we proposed a bounded-error data compression scheme called BESDC (Bounded-Error-pruned Sensor Data Compression) to reduce the point-to-point communication cost of WSNs (wireless sensor networks). Based on BESDC, this paper proposes an online bounded-error query (OBEQ) scheme with edge computing to handle the entire online query process. We propose a query filter scheme to reduce the query commands, which will inform WSN to return unnecessary queried data. It not only satisfies the QoS²/QoD requirements, but also reduces the communication cost to request sensing data. Our experiments use real data of WSN to demonstrate the query performance. Results show that an OBEQ with a query filter can reduce up to 88% of the communication cost when compared with the traditional online query process. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Open AccessArticle

Ensemble Machine Learning Model for Accurate Air Pollution Detection Using Commercial Gas Sensors

by

Wei-In Lai

,

Yung-Yu Chen

and

Jia-Hong Sun

Sensors 2022, 22(12), 4393; https://doi.org/10.3390/s22124393 - 10 Jun 2022

Cited by 5 | Viewed by 1165

Abstract

This paper presents the results on developing an ensemble machine learning model to combine commercial gas sensors for accurate concentration detection. Commercial gas sensors have the low-cost advantage and become key components of IoT devices in atmospheric condition monitoring. However, their native coarse [...] Read more.

This paper presents the results on developing an ensemble machine learning model to combine commercial gas sensors for accurate concentration detection. Commercial gas sensors have the low-cost advantage and become key components of IoT devices in atmospheric condition monitoring. However, their native coarse resolution and poor selectivity limit their performance. Thus, we adopted recurrent neural network (RNN) models to extract the time-series concentration data characteristics and improve the detection accuracy. Firstly, four types of RNN models, LSTM and GRU, Bi-LSTM, and Bi-GRU, were optimized to define the best-performance single weak models for CO, O₃, and NO₂ gases, respectively. Next, ensemble models which integrate multiple single weak models with a dynamic model were defined and trained. The testing results show that the ensemble models perform better than the single weak models. Further, a retraining procedure was proposed to make the ensemble model more flexible to adapt to environmental conditions. The significantly improved determination coefficients show that the retraining helps the ensemble models maintain long-term stable sensing performance in an atmospheric environment. The result can serve as an essential reference for the applications of IoT devices with commercial gas sensors in environment condition monitoring. Full article

(This article belongs to the Special Issue Data Engineering in the Internet of Things)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Special Issue "Data Engineering in the Internet of Things"

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Published Papers (8 papers)

Research

Planned Papers

Further Information

Guidelines

MDPI Initiatives

Follow MDPI