1. Introduction
Haipai New Year Paintings are a special type of folk art with a long history and great cultural value. As a significant part of the intangible cultural heritage of China, these artworks represent the traditional concepts of aesthetics, as well as social life, folk customs, and systems of belief through various periods of history [
1]. However, owing to the fragility of the materials, environmental degradation, and the gradual loss of their traditional skills, Haipai New Year paintings are currently under severe threat with regard to their preservation, transmission, or recording. A vast number of precious works still exist in their physical state, which could face permanent damage or loss.
In the current digital age, the integration of information technology with the conservation, preservation, and analysis of Haipai New Year Paintings is becoming more significant [
2]. Image segmentation is one such important application of information technology that is widely used in image analysis for conservation purposes. In the context of Haipai New Year paintings, image segmentation would help to identify key areas such as human bodies, design elements, background areas, and representation elements effectively. A detailed understanding of the structure of the image is critical for various applications such as image archiving, image restoration of damaged sections, and analysis of image levels, among other features.
More significantly, accurate segmentation results can provide a technical basis for establishing a structured digital database of Haipai New Year paintings. In real-world applications, such techniques can help museums establish high quality digital databases and enable image restoration and damage analysis software for the restoration of paintings, as well as enable the development of interactive learning platforms for educational purposes. More importantly, accurate segmentation techniques can provide a bridge between computer vision research and humanities studies, as they can enable collaboration between researchers from engineering backgrounds and humanities researchers such as art historians.
Recently, great progress has been achieved by deep learning methods in image segmentation tasks. However, it is difficult to apply existing image segmentation models to Haipai New Year paintings directly since there are various artistic styles, complex textures, complex lines, and complex color distributions in Haipai New Year paintings, which are quite dissimilar to those in crossover image segmentation tasks. It is of great theoretical significance and importance to design image segmentation models for Haipai New Year paintings using deep learning techniques.
Traditional image segmentation techniques were almost entirely dependent on edge detection, region growing, or thresholding. These techniques, although useful for segmented images, tend to perform poorly in complex artworks with high-density textures, as observed in Haipai New Year paintings. Later, machine learning-based approaches with handcrafted features were proposed using techniques such as Support Vector Machines or Random Forest, among others. However, their effectiveness is still limited for dealing with the variability of Haipai New Year paintings. The development of deep learning, in particular with the advent of the Fully Convolutional Network, brought about a significant improvement in image segmentation techniques. Later models like U-Net or SegNet further optimized image segmentation. While some research works have utilized deep learning for the segmentation of murals or paints, it still seems that research work related to the segmentation of Chinese Traditional Haipai New Year paintings is limited. In this research, the aim is to propose a segmentation technique for Chinese traditional Haipai New Year paintings using the PSE-Net framework, with the objective of offering a beneficial resource for digital preservation of heritage. The main research content includes:
Building a large-scale dataset of Haipai New Year paintings, covering samples from different regions, periods, and styles;
Designing a deep learning network structure tailored to the features of Haipai New Year paintings to improve segmentation accuracy and robustness;
Proposing a loss function and training strategy suited to the characteristics of Haipai New Year paintings, addressing issues such as class imbalance;
Conducting extensive experiments to verify the effectiveness of the proposed method and comparing it with existing approaches.
4. Experiments
4.2. Dataset Building and Preprocessing
In this study, we built a custom dataset featuring Haipai New Year painting images. The goal is to accurately segment different visual elements found in these traditional artworks. The dataset construction process includes image selection, annotation, and label format conversion.
Our fieldwork included on-site investigations at several key thematic exhibitions, such as “Huashuo Nianhua” (Storytelling through New Year Paintings) at the Shanghai Baoshan International Folk Art Expo (February 2024), “Yichuan Wanbang—The Sino-Western Culture in New Year Paintings” (December 2024), and “Shanghai’s New Year Flavor—The Inheritance and Development of a Century of Xiaojiaochang New Year Paintings” (January 2025). This meticulous selection process ensures that our dataset comprises samples reflecting the current scholarly and artistic consensus on the significance and value of New Year paintings. Furthermore, by drawing upon the authoritative collection of Haipai New Year paintings from the Shanghai Library and referencing the academic monograph “The Complete Collection of Chinese New Year Paintings: Shanghai Volume”, we further reinforced the classic status and systematic nature of our sample set.
We collected 376 images of Haipai New Year paintings. The Dataset is comprised of works from the late Qing to the early Republican period (c. mid-19th to early 20th century), the apogee of the Haipai New Year paintings. It is important to note that the temporal and geographical scope of the dataset is strictly defined by the historical existence of Haipai New Year paintings. This genre flourished specifically during the late Qing Dynasty and the early Republic of China (circa mid-19th to early 20th century), representing the final flourishing and modern transformation of Chinese woodblock New Year paintings. Consequently, the dataset covers the complete historical span of this art form. Extending the timeline further would result in the inclusion of non-Haipai styles or modern reproductions, which falls outside the scope of this specific heritage preservation study.
Geographically, Haipai paintings are distinct from other regional schools such as Suzhou Taohuawu or Tianjin Yangliuqing due to their unique Western-influenced perspective and urban themes. To preserve the stylistic integrity of the genre, we restricted the dataset to the Shanghai region while ensuring comprehensive thematic coverage by selecting works depicting various subjects, such as figures, opera scenes, and news events characteristic of that period. Thus, the dataset constitutes a representative archive of Haipai art and a valid benchmark for segmentation models tailored to complex artistic imagery.
To ensure the data was manageable and annotation was feasible, we filtered the images based on a specific rule: only images with fewer than five human figures were selected. This helped reduce annotation complexity and improve segmentation accuracy. After screening, 98 images met the criteria and were selected for our target dataset.
To ensure accuracy and cultural consistency in the annotations, the annotation process was carried out by a team of five researchers with backgrounds in art design and computer vision research. Before the formal labeling, four categories were defined: people, plants, animals, and objects. The marking process consists of two stages. In the first stage, the images are assigned to five annotators, and each New Year’s painting image is independently annotated by one annotator using the LabelMe 5.3.1 tool. To resolve ambiguity in complex elements and minimize subjectivity, a cross-validation mechanism was employed in the second phase, where labeled samples were exchanged among team members for review. Any discrepancies found during the validation phase are resolved through discussions or rulings with the authors to ensure a consensus is reached, ensuring that the authentic data of the Shanghai New Year paintings reflects both visual features and is semantically correct.
The images in this dataset retain their original resolutions, which are mostly 502 × 910 and 793 × 526 pixels, to ensure that the intricate details of the linework and textures that are characteristic of Haipai art are maintained. In the course of creating masks for this dataset, blank masks were used to match the original dimensions of the images. In order to accommodate this diverse range of dimensions and increase this dataset, a random cropping technique is used to produce 300 training images. This technique ensures that information is not lost due to the standardization of images to a smaller resolution.
In order to solve the problem of uneven data distribution while maintaining the integrity of the Haipai New Year painting art in the Shanghai schools, this study adopts a targeted data enhancement strategy. Due to the historical rarity of the art form and its creative tradition centered on people, there is a natural sample imbalance in the categories of “animals” and “object” in the raw data. To do this, we do not introduce composite images, but rather take advantage of the rich visual elements contained in high-resolution raw images and process them with standard data enhancement techniques such as Random Cropping and Rotation. This approach fully explores the local features in each painting, significantly increasing the model’s sensitivity to non-subject categories by focusing on smaller, less frequent instances. This ensures that the model can accurately identify the detailed elements in the picture, effectively avoiding misjudging them as backgrounds or directly ignoring them, thereby alleviating the category imbalance while preserving the artistic authenticity of the New Year paintings to the greatest extent.
However, deep learning models require a larger volume of data to generalize effectively. To address the limitation of the small dataset size and the class imbalance shown in
Table 1 (where “Man” dominates at 53.52% and “Object” is only 7.18%), we applied data augmentation techniques. The original 98 images were processed using random rotation, horizontal flipping, and cropping. This strategy not only expanded the total dataset to 300 samples (270 for training, 30 for testing) but also allowed us to generate more cropped samples focusing on under-represented classes like “Animal” and “Object,” thereby mitigating the imbalance issue during training.
We used the LabelMe tool for annotation. Each image was manually labeled at the pixel level to mark different elements in the Haipai New Year paintings. Considering the diversity of elements, we defined four main categories to simplify the annotation: man (human figures), plant, animal, and object.
The dataset statistics are shown in
Table 1. The rationale for segmenting Haipai New Year paintings into four primary categories “man, plant, animal, and object” is grounded in a comprehensive assessment of the art form’s cultural essence, its semantic structure, and the practical demands of the task. Artistically, they are semantically distinct, reflecting their unique roles in the composition. Technically, their distinct visual profiles regarding form, texture, and palette facilitate robust feature discrimination by the model. This approach strategically avoids the prohibitive complexity of overly granular labeling, ensuring both annotation quality and feasibility. Thus, the proposed taxonomy strikes a balance between cultural representativeness, visual discriminability, and practical efficiency.
In LabelMe, each target area was carefully marked by hand to ensure accurate boundaries. Each category was assigned a unique color, making it easier to visualize the segmentation labels. After annotation, each image generated a corresponding JSON file. These files stored the coordinates and category labels of all annotated regions.
Since deep learning models typically require label data in the form of masks, we wrote a Python (
https://www.python.org/) script to convert the LabelMe JSON files into mask images. The conversion process followed these steps:
Parse the JSON file and extract the polygon coordinates of each labeled area;
Create a blank mask image with the same size as the original image;
Fill the annotated regions with different colors based on their category, making the labels visually identifiable;
Save the final mask image in PNG format for model training.
Through this workflow, we successfully created a high-quality dataset of Haipai New Year paintings. It consists of 300 original images along with their corresponding segmentation masks. This dataset provides a solid foundation for training the U-Net model and contributes to the underexplored field of segmenting elements in traditional New Year artwork.
4.4. Comparative Analysis of Segmentation
The class-wise evaluation shows clear performance differences among elements in Haipai New Year paintings. The IoU values of each category are shown in
Figure 5. The background achieves the highest IoU (75.41%) due to its uniform texture and large spatial coverage, while the “man” category also performs well (60.10%), reflecting stable learning of dominant human structures. In contrast, the “animal” (23.64%) and “object” (34.89%) categories yield lower IoU values, as these elements are typically small, highly stylized, and embedded within dense decorative patterns with ambiguous boundaries. Failure cases mainly occur in images with heavy ornamentation or overlapping symbolic motifs, where fine details are missed or partially segmented. Highly abstract artistic forms and rare stylistic variations further limit generalization. Therefore, the segmentation results should be treated as analytical references, and future work should focus on expanding culturally diverse datasets and enhancing scale-sensitive feature learning.
The normalized confusion matrix in
Figure 6 illustrates the class-wise prediction performance of the proposed model. The background and man categories achieve the highest recognition accuracy, with diagonal values of 0.89 and 0.88, respectively, indicating reliable discrimination of dominant classes. The object, animal, and plant categories exhibit relatively lower accuracies (0.70, 0.72, and 0.68), mainly due to confusion with the background and man classes. In particular, object samples are frequently misclassified as background (0.17), while animal samples show notable confusion with both background (0.13) and man (0.14). Overall, most misclassifications occur between visually similar or spatially adjacent categories, whereas clear diagonal dominance across all classes demonstrates the robustness of the proposed method.
In the comparative experiments with other algorithms, as shown in
Figure 5, case (1), ResNet produced incorrect segmentation results, whereas our method achieved more accurate segmentation of the image. In case (2), ResNet’s segmentation results were sparse, losing many details and deviating significantly from the ground truth (GT). In case (3), ResNet incorrectly over-segmented details, while our method closely resembled the GT, accurately restoring the details. Similarly, in case (4), ResNet exhibited erroneous segmentation of details, with considerable differences from the GT.
To make a comprehensive and targeted evaluation, the widely used and basic U-Net structure in the task of semantic segmentation was considered as a comparative baseline. As shown in
Table 2, although the U-Net structure obtained an mIoU of 42.52% and Accuracy of 73.05%, it is still not satisfactory compared to the proposed PSE-Net. The main reason for this is the deficiency of the conventional encoder–decoder structure in effectively capturing the “multi-scale semantic features” and “dense ornamental details” in the images of Haipai New Year paintings. In the proposed framework, the PSE module is designed to improve the fusion of multi-scale features.
The impact of maintaining these original resolutions varies across different semantic categories. For dominant elements like “Man” (60.10% IoU), the high resolution provides stable “structural features” and sufficient global context for accurate recognition. However, for “Animal” (23.64% IoU) and “Object” (34.89% IoU) categories, resolution is even more critical because these elements are typically small and “deeply embedded within dense decorative patterns”. The “pixel-level ambiguity” caused by “overlapping symbolic motifs” makes these regions particularly difficult to segment. PSE-Net addresses these challenges by utilizing its “Pyramid Scale Expansion” strategy to integrate multi-scale features, thereby preserving the integrity of “ornamental boundaries” that might otherwise be lost in complex, high-density textures.
Quantitative error analysis using the confusion matrix in
Figure 6 also points to specific challenges in the segmentation process for the complex artistic elements. Although high recognition accuracies of 0.89 and 0.88 are reported for “Background” and “Man” classes, respectively, significant confusion is observed in the smaller classes. For instance, 17% of the “Object” class instances are confused with “Background”, whereas “Animal” instances show significant confusion not only with “Man” (14%) but also “Background” (13%). All the above errors are intrinsically related to the visual characteristics of Haipai-style artworks. In fact, “Object” and “Animal” elements in the artwork are “typically small-scale” and deeply embedded within dense decorative patterns.” Moreover, the boundaries in the artwork are often defined by fine ornamental strokes rather than clear edges, which in turn leads to “pixel-level ambiguity” in the visually crowded areas where symbolic elements overlap. In order to address the specific failure modes in the segmentation process, future research will aim to utilize stronger edge-detection priors as well as investigate the use of Transformer-based architectures in the context of the segmentation task.
In summary, based on the visual results from these experiments, our proposed method, PSE-Net, more effectively extracts decorative patterns from the traditional Haipai New Year painting images. To further provide an intuitive comparison of the network models, the best-performing model from each network—defined as the one with the lowest loss value—was evaluated on the test set, and the corresponding experimental metrics were calculated. Through quantitative data analysis, the generalization ability of each algorithm was assessed. The evaluation results are presented in
Table 2, and the corresponding qualitative visualization results are shown in
Figure 7.
The quantitative results in
Table 2 show that PSE-Net achieved a Mean Intersection over Union (mIoU) of 47.16%, a 2.52% improvement over the ResNet baseline. A detailed class-wise analysis reveals distinct performance differences:
Background (Highest Performance): Achieved the highest IoU (75.41%) due to its relatively uniform texture and large spatial coverage.
Man (Good Performance): The “Man” category achieved a solid IoU of 60.10%. Human figures are the most dominant subjects, providing ample training data, and possess stable structural features.
In
Figure 8, some examples of the segmentation process anomaly for the New Year painting images are shown. The first row shows the original images, the second row presents the incorrect segmentation results produced by our model, and the third row shows the ground truth segmentation results. In the wrong segmentation results, some typical anomaly examples can be found, such as boundary anomaly, region anomaly, and segmentation anomaly. The reasons for the anomaly are the complex texture, many color patterns, and ambiguous boundaries in the New Year painting images.
Failure Case Analysis: Despite the improvements, failure cases still occur, primarily in images with heavy ornamentation or overlapping symbolic motifs. For instance, when small “Objects” (like fans or weapons) overlap with purely decorative patterns on clothing, the model often struggles to distinguish the semantic boundary, resulting in partial segmentation or confusion with the “Man” class. Future work will focus on incorporating stronger edge-detection priors to resolve these ambiguities.
5. Discussion
To ensure that symbolic meaning and artistic integrity are preserved, an interdisciplinary approach was adopted throughout the research process. For instance, a classification hierarchy consisting of the “Man,” “Plant,” “Animal,” and “Object” categories was developed in consultation with art history experts to accurately reflect the traditional iconographic structures of Haipai art. Furthermore, the validation of segmentation results involved close collaboration between technical researchers and art experts, particularly when addressing complex scenarios characterized by significant “element adhesion” and “dense decoration.” By jointly analyzing whether the model-generated boundaries align with the inherent artistic logic of the paintings, a culturally responsible and semantically accurate outcome was ensured.
PSE-Net shows strong segmentation performance on Haipai New Year paintings. However, its limitations are closely related to the intrinsic characteristics of this art form. Haipai New Year Paintings feature stylized line work and symbolic abstraction. Visual elements often overlap. Object boundaries are formed by fine ornamental strokes rather than clear photographic edges. These factors introduce pixel-level ambiguity and increase segmentation difficulty in densely decorated regions. Limited annotated data further constrains the model’s generalization to rare stylistic variations.
Despite these challenges, PSE-Net provides practical value for cultural heritage practitioners. It enables efficient pixel-level separation of figures, objects, and decorative patterns. This supports digital archiving, restoration planning, and quantitative stylistic analysis in a non-invasive manner.
The architectural design of the PSE-Net has been specifically tailored to address the common visual problems in conventional artworks, such as Dunhuang murals and Miao embroidery, which also share common characteristics such as dense lines, large scale variations, and colorful styles with Haipai New Year paintings. For example, the “complex lines and blurred boundaries” in the murals resulting from historical degradation also share common characteristics with the intricate decorative lines in Haipai artworks, which also demand the precise preservation of the lines and the hierarchical feature extraction in the framework. In addition, the unique channel recalibration in the PSE component also has a special advantage in the case of embroidery artworks, where the distinction between the patterns in the foreground and the backgrounds often relies on the recalibration of the colors in the channels instead of the clear outlines in the photographs.
Although the framework has a high structural scalability in coping with different types of conventional art images, it should be further fine-tuned or transferred to adapt to the unique stylistic characteristics and historical differences in different types of cultural heritage artworks.
Ethical issues must also be considered. Automated segmentation may oversimplify artistic intent or symbolic meaning. Digital results should not be treated as authoritative reconstructions. They should serve as analytical references. Interpretation and validation should involve art historians and heritage practitioners to preserve the authenticity and cultural integrity of Haipai New Year paintings.
The practical utility of the PSE-Net extends beyond the scope of quantitative evaluation and encompasses a variety of cultural heritage applications in the real world. For example, the high-precision segmentation results provide the technical basis for the establishment of a structured digital database, which makes it possible to meet the needs of advanced archiving practices, such as the retrieval of all Haipai New Year paintings that include specific “animal” or “object” elements in the motifs—an essential aspect of modern museum practices and art history studies. In addition, in the context of virtual exhibitions and the development of interactive education tools, the segmentation of the elements of a painting makes it possible to digitally decompose the layers of the artwork. This allows the viewer to better understand the intricate compositional layers and artistic elements of Haipai New Year paintings that are not always easy to comprehend in their physical state.