J. Imaging, Volume 9, Issue 4 (April 2023) – 15 articles

The COVID-19 pandemic has underscored the need for real-time, collaborative virtual tools to support remote activities across various domains, including education and cultural heritage. Virtual walkthroughs provide a potent means of exploring, learning about, and interacting with historical sites worldwide. Nonetheless, creating realistic and user-friendly applications poses a significant challenge. This study investigates the potential of collaborative virtual walkthroughs as an educational tool for cultural heritage sites, with a focus on the Sassi of Matera, a UNESCO World Heritage Site in Italy. The virtual walkthrough application, developed using RealityCapture and Unreal Engine, leveraged photogrammetric reconstruction and deep learning-based hand gesture recognition to offer an immersive and accessible experience, allowing users to interact with the virtual environment using intuitive gestures. A test with 36 participants resulted in positive feedback regarding the application’s effectiveness, intuitiveness, and user-friendliness. The findings suggest that virtual walkthroughs can provide precise representations of complex historical locations, promoting tangible and intangible aspects of heritage. Future work should focus on expanding the reconstructed site, enhancing the performance, and assessing the impact on learning outcomes. Overall, this study highlights the potential of virtual walkthrough applications as a valuable resource for architecture, cultural heritage, and environmental education. Full article

With continuous improvements in oil production, the environmental problems caused by oil exploitation are becoming increasingly serious. Rapid and accurate estimation of soil petroleum hydrocarbon content is of great significance to the investigation and restoration of environments in oil-producing areas. In this study, the content of petroleum hydrocarbon and the hyperspectral data of soil samples collected from an oil-producing area were measured. For the hyperspectral data, spectral transforms, including continuum removal (CR), first- and second-order differential (CR-FD, CR-SD), and Napierian logarithm (CR-LN), were applied to eliminate background noise. At present, there are some shortcomings in the method of feature band selection, such as large quantity, time of calculation, and unclear importance of each feature band obtained. Meanwhile, redundant bands easily exist in the feature set, which seriously affects the accuracy of the inversion algorithm. In order to solve the above problems, a new method (GARF) for hyperspectral characteristic band selection was proposed. It combined the advantage that the grouping search algorithm can effectively reduce the calculation time with the advantage that the point-by-point search algorithm can determine the importance of each band, which provided a clearer direction for further spectroscopic research. The 17 selected bands were used as the input data of partial least squares regression (PLSR) and K-nearest neighbor (KNN) algorithms to estimate soil petroleum hydrocarbon content, and the leave-one-out method was used for cross-validation. The root mean squared error (RMSE) and coefficient of determination (R²) of the estimation result were 3.52 and 0.90, which implemented a high accuracy with only 8.37% of the entire bands. The results showed that compared with the traditional characteristic band selection methods, GARF can effectively reduce the redundant bands and screen out the optimal characteristic bands in the hyperspectral data of soil petroleum hydrocarbon with the method of importance assessment, which retained the physical meaning. It provided a new idea for the research of other substances in soil. Full article

In this article, multilevel principal components analysis (mPCA) is used to treat dynamical changes in shape. Results of standard (single-level) PCA are also presented here as a comparison. Monte Carlo (MC) simulation is used to create univariate data (i.e., a single “outcome” variable) that contain two distinct classes of trajectory with time. MC simulation is also used to create multivariate data of sixteen 2D points that (broadly) represent an eye; these data also have two distinct classes of trajectory (an eye blinking and an eye widening in surprise). This is followed by an application of mPCA and single-level PCA to “real” data consisting of twelve 3D landmarks outlining the mouth that are tracked over all phases of a smile. By consideration of eigenvalues, results for the MC datasets find correctly that variation due to differences in groups between the two classes of trajectories are larger than variation within each group. In both cases, differences in standardized component scores between the two groups are observed as expected. Modes of variation are shown to model the univariate MC data correctly, and good model fits are found for both the “blinking” and “surprised” trajectories for the MC “eye” data. Results for the “smile” data show that the smile trajectory is modelled correctly; that is, the corners of the mouth are drawn backwards and wider during a smile. Furthermore, the first mode of variation at level 1 of the mPCA model shows only subtle and minor changes in mouth shape due to sex; whereas the first mode of variation at level 2 of the mPCA model governs whether the mouth is upturned or downturned. These results are all an excellent test of mPCA, showing that mPCA presents a viable method of modeling dynamical changes in shape. Full article

In this paper, we propose a privacy-preserving image classification method using block-wise scrambled images and a modified ConvMixer. Conventional block-wise scrambled encryption methods usually need the combined use of an adaptation network and a classifier to reduce the influence of image encryption. However, we point out that it is problematic to utilize large-size images with conventional methods using an adaptation network because of the significant increment in computation cost. Thus, we propose a novel privacy-preserving method that allows us not only to apply block-wise scrambled images to ConvMixer for both training and testing without an adaptation network, but also to provide a high classification accuracy and strong robustness against attack methods. Furthermore, we also evaluate the computation cost of state-of-the-art privacy-preserving DNNs to confirm that our proposed method requires fewer computational resources. In an experiment, we evaluated the classification performance of the proposed method on CIFAR-10 and ImageNet compared with other methods and the robustness against various ciphertext-only-attacks. Full article

Millions of people are affected by retinal abnormalities worldwide. Early detection and treatment of these abnormalities could arrest further progression, saving multitudes from avoidable blindness. Manual disease detection is time-consuming, tedious and lacks repeatability. There have been efforts to automate ocular disease detection, riding on the successes of the application of Deep Convolutional Neural Networks (DCNNs) and vision transformers (ViTs) for Computer-Aided Diagnosis (CAD). These models have performed well, however, there remain challenges owing to the complex nature of retinal lesions. This work reviews the most common retinal pathologies, provides an overview of prevalent imaging modalities and presents a critical evaluation of current deep-learning research for the detection and grading of glaucoma, diabetic retinopathy, Age-Related Macular Degeneration and multiple retinal diseases. The work concluded that CAD, through deep learning, will increasingly be vital as an assistive technology. As future work, there is a need to explore the potential impact of using ensemble CNN architectures in multiclass, multilabel tasks. Efforts should also be expended on the improvement of model explainability to win the trust of clinicians and patients. Full article

The images we commonly use are RGB images that contain three pieces of information: red, green, and blue. On the other hand, hyperspectral (HS) images retain wavelength information. HS images are utilized in various fields due to their rich information content, but acquiring them requires specialized and expensive equipment that is not easily accessible to everyone. Recently, Spectral Super-Resolution (SSR), which generates spectral images from RGB images, has been studied. Conventional SSR methods target Low Dynamic Range (LDR) images. However, some practical applications require High Dynamic Range (HDR) images. In this paper, an SSR method for HDR is proposed. As a practical example, we use the HDR-HS images generated by the proposed method as environment maps and perform spectral image-based lighting. The rendering results by our method are more realistic than conventional renderers and LDR SSR methods, and this is the first attempt to utilize SSR for spectral rendering. Full article

Human action recognition has been actively explored over the past two decades to further advancements in video analytics domain. Numerous research studies have been conducted to investigate the complex sequential patterns of human actions in video streams. In this paper, we propose a knowledge distillation framework, which distills spatio-temporal knowledge from a large teacher model to a lightweight student model using an offline knowledge distillation technique. The proposed offline knowledge distillation framework takes two models: a large pre-trained 3DCNN (three-dimensional convolutional neural network) teacher model and a lightweight 3DCNN student model (i.e., the teacher model is pre-trained on the same dataset on which the student model is to be trained on). During offline knowledge distillation training, the distillation algorithm trains only the student model to help enable the student model to achieve the same level of prediction accuracy as the teacher model. To evaluate the performance of the proposed method, we conduct extensive experiments on four benchmark human action datasets. The obtained quantitative results verify the efficiency and robustness of the proposed method over the state-of-the-art human action recognition methods by obtaining up to 35% improvement in accuracy over existing methods. Furthermore, we evaluate the inference time of the proposed method and compare the obtained results with the inference time of the state-of-the-art methods. Experimental results reveal that the proposed method attains an improvement of up to 50× in terms of frames per seconds (FPS) over the state-of-the-art methods. The short inference time and high accuracy make our proposed framework suitable for human activity recognition in real-time applications. Full article

Deep learning has become a popular tool for medical image analysis, but the limited availability of training data remains a major challenge, particularly in the medical field where data acquisition can be costly and subject to privacy regulations. Data augmentation techniques offer a solution by artificially increasing the number of training samples, but these techniques often produce limited and unconvincing results. To address this issue, a growing number of studies have proposed the use of deep generative models to generate more realistic and diverse data that conform to the true distribution of the data. In this review, we focus on three types of deep generative models for medical image augmentation: variational autoencoders, generative adversarial networks, and diffusion models. We provide an overview of the current state of the art in each of these models and discuss their potential for use in different downstream tasks in medical imaging, including classification, segmentation, and cross-modal translation. We also evaluate the strengths and limitations of each model and suggest directions for future research in this field. Our goal is to provide a comprehensive review about the use of deep generative models for medical image augmentation and to highlight the potential of these models for improving the performance of deep learning algorithms in medical image analysis. Full article

This paper focuses on image and video content analysis of handball scenes and applying deep learning methods for detecting and tracking the players and recognizing their activities. Handball is a team sport of two teams played indoors with the ball with well-defined goals and rules. The game is dynamic, with fourteen players moving quickly throughout the field in different directions, changing positions and roles from defensive to offensive, and performing different techniques and actions. Such dynamic team sports present challenging and demanding scenarios for both the object detector and the tracking algorithms and other computer vision tasks, such as action recognition and localization, with much room for improvement of existing algorithms. The aim of the paper is to explore the computer vision-based solutions for recognizing player actions that can be applied in unconstrained handball scenes with no additional sensors and with modest requirements, allowing a broader adoption of computer vision applications in both professional and amateur settings. This paper presents semi-manual creation of custom handball action dataset based on automatic player detection and tracking, and models for handball action recognition and localization using Inflated 3D Networks (I3D). For the task of player and ball detection, different configurations of You Only Look Once (YOLO) and Mask Region-Based Convolutional Neural Network (Mask R-CNN) models fine-tuned on custom handball datasets are compared to original YOLOv7 model to select the best detector that will be used for tracking-by-detection algorithms. For the player tracking, DeepSORT and Bag of tricks for SORT (BoT SORT) algorithms with Mask R-CNN and YOLO detectors were tested and compared. For the task of action recognition, I3D multi-class model and ensemble of binary I3D models are trained with different input frame lengths and frame selection strategies, and the best solution is proposed for handball action recognition. The obtained action recognition models perform well on the test set with nine handball action classes, with average F1 measures of 0.69 and 0.75 for ensemble and multi-class classifiers, respectively. They can be used to index handball videos to facilitate retrieval automatically. Finally, some open issues, challenges in applying deep learning methods in such a dynamic sports environment, and direction for future development will be discussed. Full article

Recently, signature verification systems have been widely adopted for verifying individuals based on their handwritten signatures, especially in forensic and commercial transactions. Generally, feature extraction and classification tremendously impact the accuracy of system authentication. Feature extraction is challenging for signature verification systems due to the diverse forms of signatures and sample circumstances. Current signature verification techniques demonstrate promising results in identifying genuine and forged signatures. However, the overall performance of skilled forgery detection remains rigid to deliver high contentment. Furthermore, most of the current signature verification techniques demand a large number of learning samples to increase verification accuracy. This is the primary disadvantage of using deep learning, as the figure of signature samples is mainly restricted to the functional application of the signature verification system. In addition, the system inputs are scanned signatures that comprise noisy pixels, a complicated background, blurriness, and contrast decay. The main challenge has been attaining a balance between noise and data loss, since some essential information is lost during preprocessing, probably influencing the subsequent stages of the system. This paper tackles the aforementioned issues by presenting four main steps: preprocessing, multifeature fusion, discriminant feature selection using a genetic algorithm based on one class support vector machine (OCSVM-GA), and a one-class learning strategy to address imbalanced signature data in the practical application of a signature verification system. The suggested method employs three databases of signatures: SID-Arabic handwritten signatures, CEDAR, and UTSIG. Experimental results depict that the proposed approach outperforms current systems in terms of false acceptance rate (FAR), false rejection rate (FRR), and equal error rate (EER). Full article

Histopathology image analysis is considered as a gold standard for the early diagnosis of serious diseases such as cancer. The advancements in the field of computer-aided diagnosis (CAD) have led to the development of several algorithms for accurately segmenting histopathology images. However, the application of swarm intelligence for segmenting histopathology images is less explored. In this study, we introduce a Multilevel Multiobjective Particle Swarm Optimization guided Superpixel algorithm (MMPSO-S) for the effective detection and segmentation of various regions of interest (ROIs) from Hematoxylin and Eosin (H&E)-stained histopathology images. Several experiments are conducted on four different datasets such as TNBC, MoNuSeg, MoNuSAC, and LD to ascertain the performance of the proposed algorithm. For the TNBC dataset, the algorithm achieves a Jaccard coefficient of 0.49, a Dice coefficient of 0.65, and an F-measure of 0.65. For the MoNuSeg dataset, the algorithm achieves a Jaccard coefficient of 0.56, a Dice coefficient of 0.72, and an F-measure of 0.72. Finally, for the LD dataset, the algorithm achieves a precision of 0.96, a recall of 0.99, and an F-measure of 0.98. The comparative results demonstrate the superiority of the proposed method over the simple Particle Swarm Optimization (PSO) algorithm, its variants (Darwinian particle swarm optimization (DPSO), fractional order Darwinian particle swarm optimization (FODPSO)), Multiobjective Evolutionary Algorithm based on Decomposition (MOEA/D), non-dominated sorting genetic algorithm 2 (NSGA2), and other state-of-the-art traditional image processing methods. Full article

The rapid spread of deceptive information on the internet can have severe and irreparable consequences. As a result, it is important to develop technology that can detect fake news. Although significant progress has been made in this area, current methods are limited because they focus only on one language and do not incorporate multilingual information. In this work, we propose Multiverse—a new feature based on multilingual evidence that can be used for fake news detection and improve existing approaches. Our hypothesis that cross-lingual evidence can be used as a feature for fake news detection is supported by manual experiments based on a set of true (legit) and fake news. Furthermore, we compared our fake news classification system based on the proposed feature with several baselines on two multi-domain datasets of general-topic news and one fake COVID-19 news dataset, showing that (in combination with linguistic features) it yields significant improvements over the baseline models, bringing additional useful signals to the classifier. Full article

In recent years, extended reality has increasingly been used to enhance the shopping experience for customers. In particular, some virtual dressing room applications have begun to develop, as they allow customers to try on digital clothes and see how they fit. However, recent studies found that the presence of an AI or a real shopping assistant could improve the virtual dressing room experience. In response to this, we have developed a collaborative synchronous virtual dressing room for image consulting that allows customers to try on realistic digital garments chosen by a remotely connected human image consultant. The application has different features for the image consultant and the customer. The image consultant can connect to the application, define a database of garments, select different outfits with different sizes for the customer to try, and communicate with the customer through a single RGB camera system. The customer-side application can visualize the description of the outfit that the avatar is wearing, as well as the virtual shopping cart. The main purpose of the application is to offer an immersive experience, ensured by the presence of a realistic environment, an avatar that resembles the customer, a real-time physically-based cloth simulation algorithm, and a video-chat system. Full article

(1) The aim of our study is to evaluate the capacity of the Visually AcceSAble Rembrandt Images (VASARI) scoring system in discerning between the different degrees of glioma and Isocitrate Dehydrogenase (IDH) status predictions, with a possible application in machine learning. (2) A retrospective study was conducted on 126 patients with gliomas (M/F = 75/51; mean age: 55.30), from which we obtained their histological grade and molecular status. Each patient was analyzed with all 25 features of VASARI, blinded by two residents and three neuroradiologists. The interobserver agreement was assessed. A statistical analysis was conducted to evaluate the distribution of the observations using a box plot and a bar plot. We then performed univariate and multivariate logistic regressions and a Wald test. We also calculated the odds ratios and confidence intervals for each variable and the evaluation matrices with receiver operating characteristic (ROC) curves in order to identify cut-off values that are predictive of a diagnosis. Finally, we did the Pearson correlation test to see if the variables grade and IDH were correlated. (3) An excellent ICC estimate was obtained. For the grade and IDH status prediction, there were statistically significant results by evaluation of the degree of post-contrast impregnation (F4) and the percentage of impregnated area (F5), not impregnated area (F6), and necrotic (F7) tissue. These models showed good performances according to the area under the curve (AUC) values (>70%). (4) Specific MRI features can be used to predict the grade and IDH status of gliomas, with important prognostic implications. The standardization and improvement of these data (aim: AUC > 80%) can be used for programming machine learning software. Full article

The process of image segmentation is partitioning an image into its constituent parts and is a significant approach for extracting interesting features from images. Over a couple of decades, many efficient image segmentation approaches have been formulated for various applications. Still, it is a challenging and complex issue, especially for color image segmentation. To moderate this difficulty, a novel multilevel thresholding approach is proposed in this paper based on the electromagnetism optimization (EMO) technique with an energy curve, named multilevel thresholding based on EMO and energy curve (MTEMOE). To compute the optimized threshold values, Otsu’s variance and Kapur’s entropy are deployed as fitness functions; both values should be maximized to locate optimal threshold values. In both Kapur’s and Otsu’s methods, the pixels of an image are classified into different classes based on the threshold level selected on the histogram. Optimal threshold levels give higher efficiency of segmentation; the EMO technique is used to find optimal thresholds in this research. The methods based on an image’s histograms do not possess the spatial contextual information for finding the optimal threshold levels. To abolish this deficiency an energy curve is used instead of the histogram and this curve can establish the spatial relationship of pixels with their neighbor pixels. To study the experimental results of the proposed scheme, several color benchmark images are considered at various threshold levels and compared with other meta-heuristic algorithms: multi-verse optimization, whale optimization algorithm, and so on. The investigational results are illustrated in terms of mean square error, peak signal-to-noise ratio, the mean value of fitness reach, feature similarity, structural similarity, variation of information, and probability rand index. The results reveal that the proposed MTEMOE approach overtops other state-of-the-art algorithms to solve engineering problems in various fields. Full article