Next Issue
Volume 3, September
Previous Issue
Volume 3, March
 
 

Mach. Learn. Knowl. Extr., Volume 3, Issue 2 (June 2021) – 12 articles

Cover Story (view full-size image): Training medical image analysis models traditionally requires large amounts of expertly annotated imaging data, which are time-consuming and expensive to obtain. In this work, we automatically extract 33 labels at sentence level from head CT reports for stroke patients using BERT with a per-label attention mechanism. We propose template creation for data synthesis, which enables us to inject expert knowledge about unseen entities from medical ontologies and to teach the model rules on how to label difficult cases by producing relevant training examples. Our methodology offers a practical way to combine domain knowledge with machine learning for text classification tasks. View this paper
  • Issues are regarded as officially published after their release is announced to the table of contents alert mailing list.
  • You may sign up for e-mail alerts to receive table of contents of newly released issues.
  • PDF is the official format for papers published in both, html and pdf forms. To view the papers in pdf format, click on the "PDF Full-text" link, and use the free Adobe Readerexternal link to open them.
Order results
Result details
Section
Select all
Export citation of selected articles as:
Article
Going to Extremes: Weakly Supervised Medical Image Segmentation
Mach. Learn. Knowl. Extr. 2021, 3(2), 507-524; https://doi.org/10.3390/make3020026 - 02 Jun 2021
Cited by 4 | Viewed by 2112
Abstract
Medical image annotation is a major hurdle for developing precise and robust machine-learning models. Annotation is expensive, time-consuming, and often requires expert knowledge, particularly in the medical field. Here, we suggest using minimal user interaction in the form of extreme point clicks to [...] Read more.
Medical image annotation is a major hurdle for developing precise and robust machine-learning models. Annotation is expensive, time-consuming, and often requires expert knowledge, particularly in the medical field. Here, we suggest using minimal user interaction in the form of extreme point clicks to train a segmentation model which, in effect, can be used to speed up medical image annotation. An initial segmentation is generated based on the extreme points using the random walker algorithm. This initial segmentation is then used as a noisy supervision signal to train a fully convolutional network that can segment the organ of interest, based on the provided user clicks. Through experimentation on several medical imaging datasets, we show that the predictions of the network can be refined using several rounds of training with the prediction from the same weakly annotated data. Further improvements are shown using the clicked points within a custom-designed loss and attention mechanism. Our approach has the potential to speed up the process of generating new training datasets for the development of new machine-learning and deep-learning-based models for, but not exclusively, medical image analysis. Full article
Show Figures

Figure 1

Article
A Machine Learning Evaluation of the Effects of South Africa’s COVID-19 Lockdown Measures on Population Mobility
Mach. Learn. Knowl. Extr. 2021, 3(2), 481-506; https://doi.org/10.3390/make3020025 - 01 Jun 2021
Viewed by 1583
Abstract
Following the declaration by the World Health Organisation (WHO) on 11 March 2020, that the global COVID-19 outbreak had become a pandemic, South Africa implemented a full lockdown from 27 March 2020 for 21 days. The full lockdown was implemented after the publication [...] Read more.
Following the declaration by the World Health Organisation (WHO) on 11 March 2020, that the global COVID-19 outbreak had become a pandemic, South Africa implemented a full lockdown from 27 March 2020 for 21 days. The full lockdown was implemented after the publication of the National Disaster Regulations (NDR) gazette on 18 March 2020. The regulations included lockdowns, public health measures, movement restrictions, social distancing measures, and social and economic measures. We developed a hybrid model that consists of a long-short term memory auto-encoder (LSTMAE) and the kernel quantile estimator (KQE) algorithm to detect change-points. Thereafter, we utilised the Bayesian structural times series models (BSTSMs) to estimate the causal effect of the lockdown measures. The LSTMAE and KQE, successfully detected the changepoint that resulted from the full lockdown that was imposed on 27 March 2020. Additionally, we quantified the causal effect of the full lockdown measure on population mobility in residential places, workplaces, transit stations, parks, grocery and pharmacy, and retail and recreation. In relative terms, population mobility at grocery and pharmacy places decreased significantly by −17,137.04% (p-value = 0.001 < 0.05). In relative terms, population mobility at transit stations, retail and recreation, workplaces, parks, and residential places decreased significantly by −998.59% (p-value = 0.001 < 0.05), −1277.36% (p-value = 0.001 < 0.05), −2175.86% (p-value = 0.001 < 0.05), −370.00% (p-value = 0.001< 0.05), and −22.73% (p-value = 0.001 < 0.05), respectively. Therefore, the full lockdown Level 5 imposed on March 27, 2020 had a causal effect on population mobility in these categories of places. Full article
Show Figures

Figure 1

Article
Hardness of Learning in Rich Environments and Some Consequences for Financial Markets
Mach. Learn. Knowl. Extr. 2021, 3(2), 467-480; https://doi.org/10.3390/make3020024 - 28 May 2021
Viewed by 988
Abstract
This paper examines the computational feasibility of the standard model of learning in economic theory. It is shown that the information update technique at the heart of this model is impossible to compute in all but the simplest scenarios. Specifically, using tools from [...] Read more.
This paper examines the computational feasibility of the standard model of learning in economic theory. It is shown that the information update technique at the heart of this model is impossible to compute in all but the simplest scenarios. Specifically, using tools from theoretical machine learning, the paper first demonstrates that there is no polynomial implementation of the model unless the independence structure of variables in the data is publicly known. Next, it is shown that there cannot exist a polynomial algorithm to infer the independence structure; consequently, the overall learning problem does not have a polynomial implementation. Using the learning model when it is computationally infeasible carries risks, and some of these are explored in the latter part of the paper in the context of financial markets. Especially in rich, high-frequency environments, it implies discarding a lot of useful information, and this can lead to paradoxical outcomes in interactive game-theoretic situations. This is illustrated in a trading example where market prices can never reflect an informed trader’s information, no matter how many rounds of trade. The paper provides new theoretical motivation for the use of bounded rationality models in the study of financial asset pricing—the bound on rationality arising from the computational hardness in learning. Full article
Article
Single-Core Multiscale Residual Network for the Super Resolution of Liquid Metal Specimen Images
Mach. Learn. Knowl. Extr. 2021, 3(2), 453-466; https://doi.org/10.3390/make3020023 - 27 May 2021
Viewed by 1152
Abstract
In a gravity-free or microgravity environment, liquid metals without crystalline nuclei achieve a deep undercooling state. The resulting melts exhibit unique properties, and the research of this phenomenon is critical for exploring new metastable materials. Owing to the rapid crystallization rates of deeply [...] Read more.
In a gravity-free or microgravity environment, liquid metals without crystalline nuclei achieve a deep undercooling state. The resulting melts exhibit unique properties, and the research of this phenomenon is critical for exploring new metastable materials. Owing to the rapid crystallization rates of deeply undercooled liquid metal droplets, as well as cost concerns, experimental systems meant for the study of liquid metal specimens usually use low-resolution, high-framerate, high-speed cameras, which result in low-resolution photographs. To facilitate subsequent studies by material scientists, it is necessary to use super-resolution techniques to increase the resolution of these photographs. However, existing super-resolution algorithms cannot quickly and accurately restore the details contained in images of deeply undercooled liquid metal specimens. To address this problem, we propose the single-core multiscale residual network (SCMSRN) algorithm for photographic images of liquid metal specimens. In this model, multiple cascaded filters are used to obtain feature information, and the multiscale features are then fused by a residual network. Compared to existing state-of-the-art artificial neural network super-resolution algorithms, such as SRCNN, VDSR and MSRN, our model was able to achieve higher PSNR and SSIM scores and reduce network size and training time. Full article
(This article belongs to the Topic Applied Computer Vision and Pattern Recognition)
Show Figures

Figure 1

Article
Assessing the Robustness of Cluster Solutions in Emotionally-Annotated Pictures Using Monte-Carlo Simulation Stabilized K-Means Algorithm
Mach. Learn. Knowl. Extr. 2021, 3(2), 435-452; https://doi.org/10.3390/make3020022 - 04 May 2021
Cited by 4 | Viewed by 1298
Abstract
Clustering is a very popular machine-learning technique that is often used in data exploration of continuous variables. In general, there are two problems commonly encountered in clustering: (1) the selection of the optimal number of clusters, and (2) the undecidability of the affiliation [...] Read more.
Clustering is a very popular machine-learning technique that is often used in data exploration of continuous variables. In general, there are two problems commonly encountered in clustering: (1) the selection of the optimal number of clusters, and (2) the undecidability of the affiliation of border data points to neighboring clusters. We address both problems and describe how to solve them in application to affective multimedia databases. In the experiment, we used the unsupervised learning algorithm k-means and the Nencki Affective Picture System (NAPS) dataset, which contains 1356 semantically and emotionally annotated pictures. The optimal number of centroids was estimated, using the empirical elbow and silhouette rules, and validated using the Monte-Carlo simulation approach. Clustering with k = 1–50 centroids is reported, along with dominant picture keywords and descriptive statistical parameters. Affective multimedia databases, such as the NAPS, have been specifically designed for emotion and attention experiments. By estimating the optimal cluster solutions, it was possible to gain deeper insight into affective features of visual stimuli. Finally, a custom software application was developed for study in the Python programming language. The tool uses the scikit-learn library for the implementation of machine-learning algorithms, data exploration and visualization. The tool is freely available for scientific and non-commercial purposes. Full article
(This article belongs to the Section Data)
Show Figures

Figure 1

Review
Review of Automatic Microexpression Recognition in the Past Decade
Mach. Learn. Knowl. Extr. 2021, 3(2), 414-434; https://doi.org/10.3390/make3020021 - 02 May 2021
Cited by 3 | Viewed by 1409
Abstract
Facial expressions provide important information concerning one’s emotional state. Unlike regular facial expressions, microexpressions are particular kinds of small quick facial movements, which generally last only 0.05 to 0.2 s. They reflect individuals’ subjective emotions and real psychological states more accurately than regular [...] Read more.
Facial expressions provide important information concerning one’s emotional state. Unlike regular facial expressions, microexpressions are particular kinds of small quick facial movements, which generally last only 0.05 to 0.2 s. They reflect individuals’ subjective emotions and real psychological states more accurately than regular expressions which can be acted. However, the small range and short duration of facial movements when microexpressions happen make them challenging to recognize both by humans and machines alike. In the past decade, automatic microexpression recognition has attracted the attention of researchers in psychology, computer science, and security, amongst others. In addition, a number of specialized microexpression databases have been collected and made publicly available. The purpose of this article is to provide a comprehensive overview of the current state of the art automatic facial microexpression recognition work. To be specific, the features and learning methods used in automatic microexpression recognition, the existing microexpression data sets, the major outstanding challenges, and possible future development directions are all discussed. Full article
(This article belongs to the Section Thematic Reviews)
Show Figures

Figure 1

Article
Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology
Mach. Learn. Knowl. Extr. 2021, 3(2), 392-413; https://doi.org/10.3390/make3020020 - 22 Apr 2021
Cited by 19 | Viewed by 4755
Abstract
Machine learning is an established and frequently used technique in industry and academia, but a standard process model to improve success and efficiency of machine learning applications is still missing. Project organizations and machine learning practitioners face manifold challenges and risks when developing [...] Read more.
Machine learning is an established and frequently used technique in industry and academia, but a standard process model to improve success and efficiency of machine learning applications is still missing. Project organizations and machine learning practitioners face manifold challenges and risks when developing machine learning applications and have a need for guidance to meet business expectations. This paper therefore proposes a process model for the development of machine learning applications, covering six phases from defining the scope to maintaining the deployed machine learning application. Business and data understanding are executed simultaneously in the first phase, as both have considerable impact on the feasibility of the project. The next phases are comprised of data preparation, modeling, evaluation, and deployment. Special focus is applied to the last phase, as a model running in changing real-time environments requires close monitoring and maintenance to reduce the risk of performance degradation over time. With each task of the process, this work proposes quality assurance methodology that is suitable to address challenges in machine learning development that are identified in the form of risks. The methodology is drawn from practical experience and scientific literature, and has proven to be general and stable. The process model expands on CRISP-DM, a data mining process model that enjoys strong industry support, but fails to address machine learning specific tasks. The presented work proposes an industry- and application-neutral process model tailored for machine learning applications with a focus on technical tasks for quality assurance. Full article
Show Figures

Figure 1

Article
On the Scale Invariance in State of the Art CNNs Trained on ImageNet
Mach. Learn. Knowl. Extr. 2021, 3(2), 374-391; https://doi.org/10.3390/make3020019 - 03 Apr 2021
Cited by 3 | Viewed by 1862
Abstract
The diffused practice of pre-training Convolutional Neural Networks (CNNs) on large natural image datasets such as ImageNet causes the automatic learning of invariance to object scale variations. This, however, can be detrimental in medical imaging, where pixel spacing has a known physical correspondence [...] Read more.
The diffused practice of pre-training Convolutional Neural Networks (CNNs) on large natural image datasets such as ImageNet causes the automatic learning of invariance to object scale variations. This, however, can be detrimental in medical imaging, where pixel spacing has a known physical correspondence and size is crucial to the diagnosis, for example, the size of lesions, tumors or cell nuclei. In this paper, we use deep learning interpretability to identify at what intermediate layers such invariance is learned. We train and evaluate different regression models on the PASCAL-VOC (Pattern Analysis, Statistical modeling and ComputAtional Learning-Visual Object Classes) annotated data to (i) separate the effects of the closely related yet different notions of image size and object scale, (ii) quantify the presence of scale information in the CNN in terms of the layer-wise correlation between input scale and feature maps in InceptionV3 and ResNet50, and (iii) develop a pruning strategy that reduces the invariance to object scale of the learned features. Results indicate that scale information peaks at central CNN layers and drops close to the softmax, where the invariance is reached. Our pruning strategy uses this to obtain features that preserve scale information. We show that the pruning significantly improves the performance on medical tasks where scale is a relevant factor, for example for the regression of breast histology image magnification. These results show that the presence of scale information at intermediate layers legitimates transfer learning in applications that require scale covariance rather than invariance and that the performance on these tasks can be improved by pruning off the layers where the invariance is learned. All experiments are performed on publicly available data and the code is available on GitHub. Full article
Show Figures

Figure 1

Article
Estimating Neural Network’s Performance with Bootstrap: A Tutorial
Mach. Learn. Knowl. Extr. 2021, 3(2), 357-373; https://doi.org/10.3390/make3020018 - 29 Mar 2021
Cited by 7 | Viewed by 1511
Abstract
Neural networks present characteristics where the results are strongly dependent on the training data, the weight initialisation, and the hyperparameters chosen. The determination of the distribution of a statistical estimator, as the Mean Squared Error (MSE) or the accuracy, is fundamental to evaluate [...] Read more.
Neural networks present characteristics where the results are strongly dependent on the training data, the weight initialisation, and the hyperparameters chosen. The determination of the distribution of a statistical estimator, as the Mean Squared Error (MSE) or the accuracy, is fundamental to evaluate the performance of a neural network model (NNM). For many machine learning models, as linear regression, it is possible to analytically obtain information as variance or confidence intervals on the results. Neural networks present the difficulty of not being analytically tractable due to their complexity. Therefore, it is impossible to easily estimate distributions of statistical estimators. When estimating the global performance of an NNM by estimating the MSE in a regression problem, for example, it is important to know the variance of the MSE. Bootstrap is one of the most important resampling techniques to estimate averages and variances, between other properties, of statistical estimators. In this tutorial, the application of resampling techniques (including bootstrap) to the evaluation of neural networks’ performance is explained from both a theoretical and practical point of view. The pseudo-code of the algorithms is provided to facilitate their implementation. Computational aspects, as the training time, are discussed, since resampling techniques always require simulations to be run many thousands of times and, therefore, are computationally intensive. A specific version of the bootstrap algorithm is presented that allows the estimation of the distribution of a statistical estimator when dealing with an NNM in a computationally effective way. Finally, algorithms are compared on both synthetically generated and real data to demonstrate their performance. Full article
Show Figures

Figure 1

Article
Privacy and Trust Redefined in Federated Machine Learning
Mach. Learn. Knowl. Extr. 2021, 3(2), 333-356; https://doi.org/10.3390/make3020017 - 29 Mar 2021
Cited by 9 | Viewed by 2966
Abstract
A common privacy issue in traditional machine learning is that data needs to be disclosed for the training procedures. In situations with highly sensitive data such as healthcare records, accessing this information is challenging and often prohibited. Luckily, privacy-preserving technologies have been developed [...] Read more.
A common privacy issue in traditional machine learning is that data needs to be disclosed for the training procedures. In situations with highly sensitive data such as healthcare records, accessing this information is challenging and often prohibited. Luckily, privacy-preserving technologies have been developed to overcome this hurdle by distributing the computation of the training and ensuring the data privacy to their owners. The distribution of the computation to multiple participating entities introduces new privacy complications and risks. In this paper, we present a privacy-preserving decentralised workflow that facilitates trusted federated learning among participants. Our proof-of-concept defines a trust framework instantiated using decentralised identity technologies being developed under Hyperledger projects Aries/Indy/Ursa. Only entities in possession of Verifiable Credentials issued from the appropriate authorities are able to establish secure, authenticated communication channels authorised to participate in a federated learning workflow related to mental health data. Full article
Show Figures

Figure 1

Article
Transfer Learning in Smart Environments
Mach. Learn. Knowl. Extr. 2021, 3(2), 318-332; https://doi.org/10.3390/make3020016 - 29 Mar 2021
Cited by 2 | Viewed by 1442
Abstract
The knowledge embodied in cognitive models of smart environments, such as machine learning models, is commonly associated with time-consuming and costly processes such as large-scale data collection, data labeling, network training, and fine-tuning of models. Sharing and reuse of these elaborated resources between [...] Read more.
The knowledge embodied in cognitive models of smart environments, such as machine learning models, is commonly associated with time-consuming and costly processes such as large-scale data collection, data labeling, network training, and fine-tuning of models. Sharing and reuse of these elaborated resources between intelligent systems of different environments, which is known as transfer learning, would facilitate the adoption of cognitive services for the users and accelerate the uptake of intelligent systems in smart building and smart city applications. Currently, machine learning processes are commonly built for intra-organization purposes and tailored towards specific use cases with the assumption of integrated model repositories and feature pools. Transferring such services and models beyond organization boundaries is a challenging task that requires human intervention to find the matching models and evaluate them. This paper investigates the potential of communication and transfer learning between smart environments in order to empower a decentralized and peer-to-peer ecosystem for seamless and automatic transfer of services and machine learning models. To this end, we explore different knowledge types in the context of smart built environments and propose a collaboration framework based on knowledge graph principles for describing the machine learning models and their corresponding dependencies. Full article
(This article belongs to the Special Issue Selected Papers from CD-MAKE 2020 and ARES 2020)
Show Figures

Figure 1

Article
Templated Text Synthesis for Expert-Guided Multi-Label Extraction from Radiology Reports
Mach. Learn. Knowl. Extr. 2021, 3(2), 299-317; https://doi.org/10.3390/make3020015 - 24 Mar 2021
Viewed by 2103
Abstract
Training medical image analysis models traditionally requires large amounts of expertly annotated imaging data which is time-consuming and expensive to obtain. One solution is to automatically extract scan-level labels from radiology reports. Previously, we showed that, by extending BERT with a per-label attention [...] Read more.
Training medical image analysis models traditionally requires large amounts of expertly annotated imaging data which is time-consuming and expensive to obtain. One solution is to automatically extract scan-level labels from radiology reports. Previously, we showed that, by extending BERT with a per-label attention mechanism, we can train a single model to perform automatic extraction of many labels in parallel. However, if we rely on pure data-driven learning, the model sometimes fails to learn critical features or learns the correct answer via simplistic heuristics (e.g., that “likely” indicates positivity), and thus fails to generalise to rarer cases which have not been learned or where the heuristics break down (e.g., “likely represents prominent VR space or lacunar infarct” which indicates uncertainty over two differential diagnoses). In this work, we propose template creation for data synthesis, which enables us to inject expert knowledge about unseen entities from medical ontologies, and to teach the model rules on how to label difficult cases, by producing relevant training examples. Using this technique alongside domain-specific pre-training for our underlying BERT architecture i.e., PubMedBERT, we improve F1 micro from 0.903 to 0.939 and F1 macro from 0.512 to 0.737 on an independent test set for 33 labels in head CT reports for stroke patients. Our methodology offers a practical way to combine domain knowledge with machine learning for text classification tasks. Full article
Show Figures

Figure 1

Previous Issue
Next Issue
Back to TopTop