Search Results (34)

The limited field of view, cumulative inter-frame jitter, and dynamic parallax interference in handheld video stitching often lead to misalignment and distortion. In this paper, we propose an end-to-end, unsupervised deep-learning framework that jointly performs real-time video stabilization and stitching. First, collaborative optimization architecture allows the stabilization and stitching modules to share parameters and propagate errors through a fully differentiable network, ensuring consistent image alignment. Second, a Markov trajectory smoothing strategy in relative coordinates models inter-frame motion as incremental relationships, effectively reducing cumulative errors. Third, a dynamic attention mask generates spatiotemporal weight maps based on foreground motion prediction, suppressing misalignment caused by dynamic objects. Experimental evaluation on diverse handheld sequences shows that our method achieves higher stitching quality, lower geometric distortion rates, and improved video stability compared to state-of-the-art baselines, while maintaining real-time processing capabilities. Ablation studies validate that relative trajectory modeling substantially mitigates long-term jitter and that the dynamic attention mask enhances stitching accuracy in dynamic scenes. These results demonstrate that the proposed framework provides a robust solution for high-quality, real-time handheld video stitching. Full article

As the global demand for surveillance cameras increases, the digital footage data also explicitly increases. Analyzing and extracting meaningful content from footage is a resource-depleting and laborious effort. The traditional video synopsis technique is used for constructing a small video by relocating the object in the time and space domains. However, it is computationally expensive, and the obtained synopsis suffers from jitter artifacts; thus, it cannot be hosted on a resource-constrained device. In this research, we propose a panoramic video synopsis framework to address and solve the problems of the efficient analysis of objects for better governance and storage. The surveillance system has multiple cameras sharing a common homography, which the proposed method leverages. The proposed method constructs a panorama by solving the broad viewpoints with significant deviations, collisions, and overlapping among the images. We embed a synopsis framework on the end device to reduce storage, networking, and computational costs. A neural network-based model stitches multiple camera feeds to obtain a panoramic structure from which only tubes with abnormal behavior were extracted and relocated in the space and time domains to construct a shorter video. Comparatively, the proposed model achieved a superior accuracy matching rate of 98.7% when stitching the images. The feature enhancement model also achieves better peak signal-to-noise ratio values, facilitating smooth synopsis construction. Full article

Background: Corn is the main grain crop grown in China, and the ear shape index of corn is an important parameter for breeding new varieties, including ear length, diameter, row number of ears, row number of grains per ear, and so on. Objective: In order to solve the problem of limited field of view associated with computer detection of the corn ear shape index against a complex background, this paper proposes a panoramic splicing method for corn ears against a complex background, which can splice 10 corn ear panoramic images at the same time, to improve information collection efficiency, display comprehensive information, and support data analysis, so as to realize automatic corn seed examination. Methods: A summary of corn ear panoramic stitching methods under complex backgrounds is presented as follows: 1. a perceptual hash algorithm and histogram equalization were used to extract video frames; 2. the U-Net image segmentation model based on transfer learning was used to predict corn labels; 3. a mask preprocessing algorithm was designed; 4. a corn ear splicing positioning algorithm was designed; 5. an algorithm for irregular surface expansion was designed; 6. an image stitching method based on template matching was adopted to assemble the video frames. Results: The experimental results showed that the proposed corn ear panoramic stitching method could effectively solve the problems of virtual stitching, obvious stitching seams, and too-high similarity between multiple images. The success rate of stitching was as high as 100%, and the speed of single-corn-ear panoramic stitching was about 9.4 s, indicating important reference value for corn breeding and disease and insect detection. Discussions: Although the experimental results demonstrated the significant advantages of the panoramic splicing method for corn ear images proposed in this paper in terms of improving information collection efficiency and automating corn assessment, the method still faces certain challenges. Future research will focus on the following points: 1. addressing the issue of environmental interference caused by diseases, pests, and plant nutritional status on the measurement of corn ear parameters in order to enhance the stability and accuracy of the algorithm; 2. expanding the dataset for the U-Net model to include a wider range of corn ears with complex backgrounds, different growth stages, and various environmental conditions to improve the model’s segmentation recognition rate and precision. Recently, our panoramic splicing algorithm has been deployed in practical applications with satisfactory results. We plan to continue optimizing the algorithm and more broadly promote its use in fields such as corn breeding and pest and disease detection in an effort to advance the development of agricultural automation technology. Full article

The automatic video recognition of depression is becoming increasingly important in clinical applications. However, traditional depression recognition models still face challenges in practical applications, such as high computational costs, the poor application effectiveness of facial movement features, and spatial feature degradation due to model stitching. To overcome these challenges, this work proposes a lightweight Time-Context Enhanced Depression Detection Network (TCEDN). We first use attention-weighted blocks to aggregate and enhance video frame-level features, easing the model’s computational workload. Next, by integrating the temporal and spatial changes of video raw features and facial movement features in a self-learning weight manner, we enhance the precision of depression detection. Finally, a fusion network of 3-Dimensional Convolutional Neural Network (3D-CNN) and Convolutional Long Short-Term Memory Network (ConvLSTM) is constructed to minimize spatial feature loss by avoiding feature flattening and to achieve depression score prediction. Tests on the AVEC2013 and AVEC2014 datasets reveal that our approach yields results on par with state-of-the-art techniques for detecting depression using video analysis. Additionally, our method has significantly lower computational complexity than mainstream methods. Full article

For the synthesis of ultra-large scene and ultra-high resolution videos, in order to obtain high-quality large-scene videos, high-quality video stitching and fusion are achieved through multi-scale unstructured array cameras. This paper proposes a network model image feature point extraction algorithm based on symmetric auto-encoding and scale feature fusion. By using the principle of symmetric auto-encoding, the hierarchical restoration of image feature location information is incorporated into the corresponding scale feature, along with deep separable convolution image feature extraction, which not only improves the performance of feature point detection but also significantly reduces the computational complexity of the network model. Based on the calculated high-precision feature point pairing information, a new image localization method is proposed based on area ratio and homography matrix scaling, which improves the speed and accuracy of the array camera image scale alignment and positioning, realizes high-definition perception of local details in large scenes, and obtains clearer synthesis effects of large scenes and high-quality stitched images. The experimental results show that the feature point extraction algorithm proposed in this paper has been experimentally compared with four typical algorithms using the HPatches dataset. The performance of feature point detection has been improved by an average of 4.9%, the performance of homography estimation has been improved by an average of 2.5%, the amount of computation has been reduced by 18%, the number of network model parameters has been reduced by 47%, and the synthesis of billion-pixel videos has been achieved, demonstrating practicality and robustness. Full article

Many rivers are polluted by trash and garbage that can affect the environment. Riverbank inspection usually relies on workers of the environmental protection office, but sometimes the places are unreachable. This study applies unmanned aerial vehicles (UAVs) to perform the inspection task, which can significantly relieve labor work. Two UAVs are used to cover a wide area of riverside and capture riverbank images. The images from different UAVs are stitched using the scale-invariant feature transform (SIFT) algorithm. Static and dynamic image stitching are tested. Different you only look once (YOLO) algorithms are applied to identify riverbank garbage. Modified YOLO algorithms improve the accuracy of riverine waste identification, while the SIFT algorithm stitches the images obtained from the UAV cameras. Then, the stitching results and garbage data are sent to a video streaming server, allowing government officials to check waste information from the real-time multi-camera stitching images. The UAVs utilize 4G communication to transmit the video stream to the server. The transmission distance is long enough for this study, and the reliability is excellent in the test fields that are covered by the 4G communication network. In the automatic reconnection mechanism, we set the timeout to 1.8 s. The UAVs will automatically reconnect to the video streaming server if the disconnection time exceeds the timeout. Based on the energy provided by the onboard battery, the UAV can be operated for 20 min in a mission. The UAV inspection distance along a preplanned path is about 1 km at a speed of 1 m/s. The proposed UAV system can replace inspection labor, successfully identify riverside garbage, and transmit the related information and location on the map to the ground control center in real time. Full article

In this paper, we extended video stitching to nearshore bathymetry for videos that were captured for the same coastal field simultaneously by two unmanned aerial vehicles (UAVs). In practice, a video captured by a single UAV often shows a limited coastal zone with a lack of a wide field of view. To solve this problem, we proposed a framework in which video stitching and bathymetric mapping were performed in sequence. Specifically, our method listed the video acquisition strategy and took two overlapping videos captured by two UAVs as inputs. Then, we adopted a unified video stitching and stabilization optimization to compute the stitching and stabilization of one of the videos separately. In this way, we can obtain the best stitching result. At the same time, background feature points identification on the shore plays the role of short-time visual odometry. Through the obtained panoramic video in Shuang Yue Bay, China, we used the temporal cross-correlation analysis based on the linear dispersion relationship to estimate the water depth. We selected the region of interest (ROI) area from the panoramic video, performed an orthorectification transformation and extracted time-stack images from it. The wave celerity was then estimated from the correlation of the signal through filtering processes. Finally, the bathymetry results were compared with the cBathy. By applying this method to two UAVs, a wider FOV was created and the surveying area was expanded, which provided effective input data for the bathymetry algorithms. Full article

The automation of inspections in aircraft engines is an ever-increasing growing field of research. In particular, the inspection and quantification of coating damages in confined spaces, usually performed manually with handheld endoscopes, comprise tasks that are challenging to automate. In this study, 2D RGB video data provided by commercial instruments are further analyzed in the form of a segmentation of damage areas. For this purpose, large overview images, which are stitched from the video frames, showing the whole coating area are analyzed with convolutional neural networks (CNNs). However, these overview images need to be divided into smaller image patches to keep the CNN architecture at a functional and fixed size, which leads to a significantly reduced field of view (FOV) and therefore a loss of information and reduced network accuracy. A possible solution is a downsampling of the overview image to decrease the number of patches and increase this FOV for each patch. However, while an increased FOV with downsampling or a small FOV without resampling both exhibit a lack of information, these approaches incorporate partly different information and abstractions to be utilized complementary. Based on this hypothesis, we propose a two-stage segmentation pipeline, which processes image patches with different FOV and downsampling factors to increase the overall segmentation accuracy for large images. This includes a novel method to optimize the position of image patches, which leads to a further improvement in accuracy. After a validation of the described hypothesis, an evaluation and comparison of the proposed pipeline and methods against the single-network application is conducted in order to demonstrate the accuracy improvements. Full article

With the increase in video surveillance data, techniques such as video synopsis are being used to construct small videos for analysis, thereby saving storage resources. The video synopsis framework applies in real-time environments, allowing for the creation of synopsis between multiple and single-view cameras; the same framework encompasses optimization, extraction, and object detection algorithms. Contemporary state-of-the-art synopsis frameworks are suitable only for particular scenarios. This paper aims to review the traditional state-of-the-art video synopsis techniques and understand the different methods incorporated in the methodology. A comprehensive review provides analysis of varying video synopsis frameworks and their components, along with insightful evidence for classifying these techniques. We primarily investigate studies based on single-view and multiview cameras, providing a synopsis and taxonomy based on their characteristics, then identifying and briefly discussing the most commonly used datasets and evaluation metrics. At each stage of the synopsis framework, we present new trends and open challenges based on the obtained insights. Finally, we evaluate the different components such as object detection, tracking, optimization, and stitching techniques on a publicly available dataset and identify the lacuna among the different algorithms based on experimental results. Full article

The automotive synthetic aperture radar (SAR) can obtain two-dimensional (2-D) high-resolution images and has good robustness compared with the other sensors. Generally, the 2-D high-resolution always conflicts with the real-time requirement in conventional SAR imaging. This article suggests an automotive video SAR (ViSAR) imaging technique based on sub-aperture spectrum fusion to address this issue. Firstly, the scene space variation problem caused by close observation distance in automotive SAR is analyzed. Moreover, the sub-aperture implementation method, frame rate and resolution of automotive ViSAR are also introduced. Then, the improved Range Doppler algorithm (RDA) is used to focus the sub-aperture data. Finally, a sub-aperture stitching strategy is proposed to obtain a high-resolution frame image. Compared with the available ViSAR imaging method, the proposed method is more efficient, performs better, and is more appropriate for automotive ViSAR. The simulation results and actual data of the automotive SAR validate the effectiveness of the proposed method. Full article

In recent years, with the development of the internet, video has become more and more widely used in life. Adding harmonious music to a video is gradually becoming an artistic task. However, artificially adding music takes a lot of time and effort, so we propose a method to recommend background music for videos. The emotional message of music is rarely taken into account in current work, but it is crucial for video music retrieval. To achieve this, we design two paths to process content information and emotional information between modals. Based on the characteristics of video and music, we design various feature extraction schemes and common representation spaces. In the content path, the pre-trained network is used as the feature extraction network. As these features contain some redundant information, we use an encoder–decoder structure for dimensionality reduction. Where encoder weights are shared to obtain content sharing features for video and music. In the emotion path, an emotion key frames scheme was used for video and a channel attention mechanism was used for music in order to obtain the emotion information effectively. We also added emotion distinguish loss to guarantee that the network acquires the emotion information effectively. More importantly, we propose a way to combine content information with emotional information. That is, content features are first stitched together with sentiment features and then passed through a fused shared space structured as an MLP to obtain more effective fused shared features. In addition, a polarity penalty factor has been added to the classical metric loss function to make it more suitable for this task. Experiments show that this dual path video music retrieval network can effectively merge information. Compared with existing methods, the retrieval task evaluation index increases Recall@1 by 3.94. Full article

Due to the influence of the shooting environment and inherent image characteristics, there is a large amount of interference in the process of image stitching a geological borehole video. To accurately match the acquired image sequences in the inner part of a borehole, this paper presents a new method of stitching an unfolded borehole image, which uses the image generated from the video to construct a large-scale panorama. Firstly, the speeded-up robust feathers (SURF) algorithm is used to extract the image feature points and complete the rough matching. Then, the M-estimator sample consensus (MSAC) algorithm is introduced to remove the mismatched point pairs and obtain the homography matrix. Subsequently, we propose a local homography matrix offset optimization (LHOO) algorithm to obtain the optimal offset. Finally, the above process is cycled frame by frame, and the image sequence is continuously stitched to complete the construction of a cylindrical borehole panorama. The experimental results show that compared with those of the SIFT, Harris, ORB and SURF algorithms, the matching accuracy of our algorithm has been greatly improved. The final test is carried out on 225 consecutive video frames, and the panorama has a good visual effect, and the average time of each frame is 100 ms, which basically meets the requirements of the project. Full article

Defect detection of petrochemical pipelines is an important task for industrial production safety. At present, pipeline defect detection mainly relies on closed circuit television method (CCTV) to take video of the pipeline inner wall and then detect the defective area manually, so the detection is very time-consuming and has a high rate of false and missed detections. To solve the above issues, we proposed an automatic defect detection system for petrochemical pipeline based on Cycle-GAN and improved YOLO v5. Firstly, in order to create the pipeline defect dataset, the original pipeline videos need pre-processing, which includes frame extraction, unfolding, illumination balancing, and image stitching to create coherent and tiled pipeline inner wall images. Secondly, aiming at the problems of small amount of samples and the imbalance of defect and non-defect classes, a sample enhancement strategy based on Cycle-GAN is proposed to generate defect images and expand the data set. Finally, in order to detect defective areas on the pipeline and improve the detection accuracy, a robust defect detection model based on improved YOLO v5 and Transformer attention mechanism is proposed, with the average precision and recall as 93.10% and 90.96%, and the F1-score as 0.920 on the test set. The proposed system can provide reference for operators in pipeline health inspection, improving the efficiency and accuracy of detection. Full article

360° video camera sensing is an increasingly popular technology. Compared with traditional 2D video systems, it is challenging to ensure the viewing experience in 360° video camera sensing because the massive omnidirectional data introduce adverse effects on start-up delay, event-to-eye delay, and frame rate. Therefore, understanding the time consumption of computing tasks in 360° video camera sensing becomes the prerequisite to improving the system’s delay performance and viewing experience. Despite the prior measurement studies on 360° video systems, none of them delves into the system pipeline and dissects the latency at the task level. In this paper, we perform the first in-depth measurement study of task-level time consumption for 360° video camera sensing. We start with identifying the subtle relationship between the three delay metrics and the time consumption breakdown across the system computing task. Next, we develop an open research prototype Zeus to characterize this relationship in various realistic usage scenarios. Our measurement of task-level time consumption demonstrates the importance of the camera CPU-GPU transfer and the server initialization, as well as the negligible effect of 360° video stitching on the delay metrics. Finally, we compare Zeus with a commercial system to validate that our results are representative and can be used to improve today’s 360° video camera sensing systems. Full article

The aim of this paper is to determine the heat transfer properties of biaxial carbon fabrics of different architectures, including non-crimp stitch bonded fabrics, plain, twill and satin woven fabrics. The specific heat capacity was determined via DSC (differential scanning calorimetry). A novel method of numerical analysis of temperature maps from a video using a high-resolution thermal camera is investigated for the measurement of the in-plane and transverse thermal diffusivity and conductivity. The determined thermal conductivity parallel to the fibers of a non-crimp stitch bonded fabric agrees well with the theoretical value calculated employing the rule of mixtures. The presence of voids due to the yarn crossover regions in woven fabrics leads to a reduced value of transverse thermal conductivity, especially in the single ply measurements of this study. Full article