Automatic Identification and Description of Jewelry Through Computer Vision and Neural Networks for Translators and Interpreters

Abstract

Identifying jewelry pieces presents a significant challenge due to the wide range of styles and designs. Currently, precise descriptions are typically limited to industry experts. However, translators and interpreters often require a comprehensive understanding of these items. In this study, we introduce an innovative approach to automatically identify and describe jewelry using neural networks. This method enables translators and interpreters to quickly access accurate information, aiding in resolving queries and gaining essential knowledge about jewelry. Our model operates at three distinct levels of description, employing computer vision techniques and image captioning to emulate expert analysis of accessories. The key innovation involves generating natural language descriptions of jewelry across three hierarchical levels, capturing nuanced details of each piece. Different image captioning architectures are utilized to detect jewels in images and generate descriptions with varying levels of detail. To demonstrate the effectiveness of our approach in recognizing diverse types of jewelry, we assembled a comprehensive database of accessory images. The evaluation process involved comparing various image captioning architectures, focusing particularly on the encoder–decoder model, crucial for generating descriptive captions. After thorough evaluation, our final model achieved a captioning accuracy exceeding 90%.

1. Introduction

Image captioning presents a challenging task in the realms of computer vision and natural language processing, requiring the generation of textual descriptions for given input images [1]. In the field of linguistics, particularly in translation and interpreting, this approach can provide accurate visual representations, precise technical terms, images, meanings, descriptions, and contexts. Translators and interpreters can rely on it to quickly access data about specific terms.

In recent years, deep learning-based approaches have emerged as a promising solution to the image captioning problem, harnessing the capabilities of deep neural networks to learn intricate mappings between images and text [2]. Typically, an encoder–decoder structure is employed for this task, with neural networks serving as encoders to process the input image and extract high-level feature representations. Furthermore, neural networks are commonly utilized as decoders, generating a sequence of words describing the image based on the features extracted by the encoder. The synergy between these two networks has significantly enhanced the performance of image captioning systems, approaching human-level performance on certain benchmark datasets [3].

An artificial intelligence (AI) tool capable of automatically identifying and describing jewelry can emulate human intuitive behavior, fostering increased connections between individuals from diverse cultures and customs. This includes overcoming language and cultural barriers by providing accurate and detailed descriptions of jewelry in various languages. Such an AI tool can encourage curiosity and cultural exchange by offering information about the history, symbolism, and craftsmanship of jewelry, sparking meaningful conversations and dialogues between individuals of different backgrounds. Additionally, it can promote appreciation and respect for diversity, fostering tolerance, inclusivity, and harmony among individuals and communities with different traditions and customs. Moreover, automatic jewelry recognition has practical applications in cultural commerce and tourism, assisting buyers and collectors in making informed decisions and engaging in fair trade of cultural jewelry, as well as attracting tourists interested in exploring and acquiring traditional jewelry during their travels.

In this study, various image captioning encoder–decoder structures are assessed using a dataset comprising images of jewelry accessories. The dataset was created by collecting images of diverse accessories from two well-known online jewelry stores in the city of Córdoba, Spain. Córdoba is celebrated for its artisanal craftsmanship, particularly in the creation of high-quality silver jewelry. The city hosts notable jewelry fairs and events that exhibit distinctive designs, attracting both local and international buyers.

The utility of AI in linguistics becomes evident in unraveling the intricate linguistic landscape of jewelry terminology. AI-powered linguistic analysis can efficiently process vast datasets of historical documents, dowry letters, and cultural references related to jewelry. It helps in identifying patterns, tracing linguistic evolution, and discerning the cultural contexts from which specific terms and expressions emerged.

Given the limited application of image captioning within this specific field of study, the task is intricate and notably captivating. The ultimate goal of the project is to generate detailed captions for accessories featured in the input images, highlighting characteristics such as their type (earrings, necklace, bracelet, etc.), material, color, and the specific jewelry they showcase. The images encompass individuals wearing the accessories, suggesting the potential application of the system to automatically generate descriptions for accessories worn by individuals in images sourced from fashion magazines, jewelry catalogs, and jewelry trade shows, among other sources. Additionally, the methodology enables the creation of simpler descriptions, indicating only the type of accessory, to facilitate a jewelry multiclass classification task.

In the following sections, we navigate through key aspects of linguistic modeling, automatic jewelry recognition, and AI-based description generation. Section 2 conducts a review of related work, highlighting prior research contributions. Section 3 delves into the linguistic levels and terminology specific to jewelry descriptions and explains the database and architectures supporting our approach. Section 4 presents the experimental design and results, showcasing the effectiveness of our linguistic models, automatic recognition techniques, and AI-generated descriptions. Finally, in Section 5, we draw conclusions and outline future research directions.

2. Related Work

In this section, we conduct a thorough review of previous works that have significantly influenced and provided reference points for the current study. Our examination revealed three main categories of research that are relevant to our investigation: the analysis of the impact of AI on linguistics, the application of artificial vision techniques for jewelry analysis, and neural networks for image captioning. Furthermore, we undertook a dedicated study specifically exploring the use of image captioning techniques in the context of jewelry; however, no prior research was identified that specifically addresses this exact subject matter.

2.1. AI and Applied Linguistics

The confluence of AI and linguistics has cultivated a fertile ground for tackling specific challenges in the identification and description of jewelry in images. The capability of these integrated approaches to comprehend both visual content and natural language offers innovative solutions.

A pivotal theory influencing this domain is the hierarchy of grammatical complexity proposed in [4,5]. Their approach underscores that communication through natural language fundamentally involves mapping signals to meanings. The application of this theory extends beyond natural language and aligns with the goal of understanding and describing jewelry images.

Considering image processing, Xu et al. [6] explore the connection between computer vision and natural language processing. The work emphasizes synergies between both fields for joint applications, such as generating descriptions of images. Chen et al. [7] take this approach a step further by introducing visual semantic role labeling, an advanced technique for image description generation, essential for detailed jewelry identification.

To undertake language modeling, a fundamental step for subsequent AI applications, studies such as those by [8] emphasize the need for robust language models.

Firstly, it must be said that the distinction between translating from natural language into formal language and reasoning about that formal language was identified in [9,10].

However, a longstanding goal of AI has been to build intelligent agents that can function with the linguistic dexterity of people, which involves such diverse capabilities as participating in a fluent conversation, using dialog to support task-oriented collaboration, and engaging in lifelong learning through processing speech and text [11].

These studies highlight the necessity of an interdisciplinary perspective, where linguistics and AI intertwine to advance jewelry image identification. They demonstrate how these converging disciplines can catalyze significant innovations in this domain.

2.2. Computer Vision for Jewelry Analysis

Jewelry has played a pivotal role in human culture for millennia, and the capacity to distinguish various types of accessories holds significance across diverse fields, including security, e-commerce, and the analysis of social preferences. Recent years have witnessed substantial progress in computer vision models, particularly in the domains of object recognition and detection. Numerous models now exist that can proficiently discern a wide array of objects, ranging from animals and vehicles to accessories. An exemplar is DeepBE, introduced in [12], which adeptly categorizes accessories worn above the shoulders, such as necklaces, earrings, and glasses.

Despite these advancements, current methods for accessory recognition, like the aforementioned DeepBE, still exhibit certain limitations compared to human expertise. While these models effectively identify the presence of jewelry and other accessories in an image, they fall short in providing intricate details about the specific type of jewelry or its quality. Consequently, there arises a demand for more specialized models that can precisely and comprehensively recognize and classify jewelry and other accessories based on their unique characteristics.

Conversely, certain studies have been dedicated to crafting specialized models tailored to appraising the quality of specific types of jewelry. Notably, the research outlined in [13] concentrates on the quality assessment of pearls based on images, employing computer vision techniques to identify and scrutinize the distinct visual characteristics of these items. The proposed methodology involves tracing light rays and scrutinizing highlight patterns to estimate the level of specularity within the input image of the pearl. These data are then employed to ascertain the equivalent index of appearance, serving as a metric for appraising the object’s quality. Through the application of these techniques, the approach comprehensively evaluates the visual appearance of the object, offering valuable insights into its quality and potential industrial applications.

2.3. Neural Networks for Image Captioning

Describing the content of an image was a task that just a few years ago could only be accomplished by humans. However, advancements in computer vision have led to the creation of new systems capable of replicating this human behavior with impressive performance.

The pioneering steps in this field are documented in works like [1], where the authors developed a probabilistic framework for generating descriptions from images. To achieve this goal, they devised an encoder–decoder structure employing neural networks.

The primary function of the encoder is to process the input image, extracting meaningful features and encoding them into a format that can be easily understood by the neural network. In simpler terms, the encoder transforms the raw visual information of the image into a more abstract representation, highlighting important features. The decoder takes the encoded information from the encoder and generates a coherent and descriptive sequence. In the case of image captioning, this sequence comprises the natural language description of the image. The decoder essentially translates the encoded features back into human-readable language, allowing the system to generate meaningful and contextually relevant captions for the input images.

Neural networks come in various types, each designed for specific tasks. Two prominent types are Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

CNNs are particularly well suited for image-related tasks. They utilize layers of convolutional operations to automatically extract hierarchical features from input images. CNNs are effective in recognizing spatial patterns, making them commonly used in image classification, object detection, and image segmentation tasks.

On the other hand, RNNs are designed to handle sequential data and are well suited for tasks involving a sequence of inputs, such as time-series data or natural language. RNNs have memory units that allow them to capture dependencies and relationships within sequential data. This makes them suitable for tasks like language modeling, speech recognition, and sequence generation.

Therefore, a good approach to addressing the problem of caption generation is the development of an encoder–decoder structure [14]. In this approach, a CNN is employed as the encoder, and an RNN serves as the decoder. This particular architecture has laid the groundwork for image captioning. In this process, the input image undergoes processing through the CNN to encode and extract features, which are then embedded into a fixed-length vector. This fixed-length vector, also known as an “embedding”, succinctly represents the most important features of the image based on the encoding performed by the CNN. This vector representation is subsequently used in the process of generating descriptions (captions) for the image, where relevant information has been condensed into a structured and manageable format for the neural network. Choosing a fixed-length vector facilitates consistency and efficiency in processing information within the network.

After the image has been processed through the CNN to encode features, the generation of final descriptions involves appending the RNN to the last hidden layer of the CNN. The term “last layer” in this context refers to the final or output layer of the CNN. In neural networks, layers are stacked, and the last layer typically produces the final representation or output. In image captioning, the information encoded by the CNN is transferred to the RNN at this stage to further refine and generate coherent textual descriptions.

The choice of utilizing a Long Short-Term Memory (LSTM) network as the RNN is significant. LSTM is a type of recurrent neural network designed to capture and retain long-term dependencies in sequential data. In the context of image captioning, LSTM plays a key role in understanding the sequential nature of language, allowing the model to generate descriptions that are not only contextually relevant but also exhibit a coherent flow. Its effectiveness lies in its ability to handle dependencies over longer sequences, making it well suited for tasks involving the generation of sentences or captions.

Similarly, subsequently to the mentioned study, several works emerged incorporating innovative mechanisms to improve its outcomes. Particularly noteworthy is a proposal outlined in [6], which introduced the concept of attention to the previously employed encoder–decoder architecture. In this scenario, diverse words generated by the LSTM were directed toward different sections of the input image. The outcomes demonstrated that their model produced highly accurate words that directly corresponded to the specific areas of the image that the LSTM focused on at each moment. As a result, a comprehensive and precise description of the image was achieved in the majority of cases.

Other advancements in image captioning have explored alternative approaches, such as generative adversarial networks (GANs). A representative example is CgT-GAN, which combines adversarial training with CLIP-based semantic rewards to generate captions without relying on paired human annotations [15]. These methods can yield more diverse and creative outputs, especially in open-domain settings. However, GAN-based captioning models often suffer from training instability, require large computational resources, and may lack linguistic control, making them less suitable for tasks involving predefined syntactic structures like those used in our multilevel jewelry description framework [16].

Another line of research explores reinforcement learning (RL) techniques to enhance image captioning models. For instance, Ref. [17] propose a strategy that integrates ground-truth captions into a CLIP-guided RL framework. This approach aims to generate more distinctive and fluent captions by leveraging the similarity between generated captions and corresponding images. Similarly, in [18], the authors introduce an RL approach that combines a policy network, responsible for generating captions, with a value network that estimates the expected reward of a generated sequence. This architecture is optimized using sequence-level feedback, allowing the model to refine caption quality based on global evaluation metrics rather than word-by-word accuracy. While RL-based models have demonstrated improvements in generating diverse and contextually relevant captions, they often require extensive training data, careful reward design, and complex tuning processes [19].

3. Materials and Methods

This section outlines the methodological framework adopted to develop a system capable of generating descriptive captions for jewelry images. The proposed approach integrates insights from linguistics and computer vision to address the complexity of jewelry representation. Throughout the following subsections, we describe the linguistic foundations that support the structuring of jewelry descriptions, the creation and preprocessing of the image dataset, and the experimental setup used to train and evaluate the captioning models.

3.1. Levels and Terminology for the Description of Jewelry

Our primary objective is to create a concise and cohesive summary of jewelry images. However, grasping the high-level informational content of these images poses challenges. Initially, a jewelry image conveys a wealth of information, making it impractical to provide all details textually. Secondly, the choice of a jewelry image as a communication medium allows viewers to extract information at various levels. A cursory glance may convey the intended message and salient features, while a more in-depth examination offers further insights.

To address the task of identifying content for jewelry image summaries, we leverage insights gained from an experiment where online jewelry catalogs were used to generate summaries for various images with a shared high-level intention. We follow a similar approach to that employed for graphic descriptions in [20]. The key finding was that the intended message of a jewelry image was consistently conveyed in all summaries regardless of visual features. Online catalogs, emulating human behavior, enhanced the intended message with salient features, depending on the image’s purpose. Similar summaries for a specific jewelry image led us to hypothesize shared perceptions of salient features. Although the set of salient features may be consistent for images with the same underlying intention, differences in summaries arise from the saliency of these features. The exclusion of exhaustive details in summaries aligns with Grice’s Maxim of Quantity [21], emphasizing informative contributions relevant to the current exchange.

To identify specific content in jewelry images, we focus on a set of propositions capturing meaningful information. This set encompasses a variety of details present in the images, including propositions common to all images and propositions specific to certain message categories. Some of these propositions include the following: labels for all jewelry items, type of jewelry, materials used in the jewelry, relationship between the materials, colors present in the jewelry, style or design of the jewelry, specific details like engraved patterns, size, or relative dimensions of the jewelry, and overall impression or aesthetic, including the addition of descriptive adjectives. These propositions cover a spectrum of information, allowing for a comprehensive and nuanced summary of jewelry images.

From a linguistic perspective, the justification for developing linguistic models based on different theories that propose levels of descriptions [22,23,24,25] for jewelry is rooted in the need to capture and express the semantic diversity and inherent complexity of jewelry pieces [26,27]. Each level of description has its own linguistic characteristics that enable effective communication of specific information about jewelry. Here, we provide the descriptions considered in our approach and specific justifications for each level of description:

• Basic description: noun + noun.

  -

  Communicative efficiency: This simple format focuses on the essence of the jewelry by combining two key nouns. It provides concise and direct information about the type of jewelry present. In contexts where brevity is essential, this format allows for quick identification of the jewelry type.

  -

  Clarity and simplicity: This is ideal for quickly identifying the type of jewelry without adding unnecessary details. The straightforward structure enhances clarity, making it suitable for scenarios where a quick overview is required.

• Normal description: adjective + noun + adjective + noun.

  -

  Semantic expansion: The inclusion of adjectives allows for a more detailed and nuanced description. It enables conveying specific features such as color, material, or design. This format provides a broader semantic scope, accommodating a richer set of details.

  -

  Aesthetic appeal: By adding adjectives, the beauty and visual attributes of the jewelry are highlighted. This can be relevant in contexts where aesthetics plays a significant role, such as art exhibitions or design showcases.

• Complete description: superlative adjectives + noun + complement.

  -

  Comprehensiveness and specificity: The inclusion of superlative adjectives indicates the exceptional quality of the jewelry, offering a complete and detailed description. This format is essential for describing unique or high-value pieces, providing comprehensive insights.

  -

  Differentiation and valuation: This allows highlighting the unique features that make the jewelry stand out among others. It is especially valuable in contexts where exclusivity is sought, such as luxury markets or bespoke jewelry presentations.

The development of linguistic models that encompass these types of descriptions allows for a more comprehensive and accurate representation of jewelry. Such models cater to the variety of information that may be relevant in different situations and contexts, ensuring that linguistic descriptions effectively convey the richness and uniqueness of each jewelry piece.

Using this approach, we propose some examples of descriptions at different complexity levels after recognizing the jewelry pieces (see

Table 1. Description levels of jewelry pieces.

Level
Basic	Earrings in yellow gold.	Solitaire in rose gold.	Yellow gold bracelet.
Normal	Yellow gold and diamond earrings.	Ring in rose gold with central diamond.	Yellow gold bracelet with topazes.
Complete	Earrings in sustainable yellow gold adorned with exquisite, brilliant-cut diamonds and featuring a secure push-back clasp.	Iris solitaire in rose gold with central diamond.	Sustainable yellow gold bracelet adorned with dazzling sky topazes.

). The presence of superlative adjectives is always optional, and a complete description can exist without them. For example, “Earrings in sustainable yellow gold adorned with exquisite, brilliant-cut diamonds and featuring a secure push-back clasp” is equivalent to “Earrings in yellow gold with brilliant-cut diamonds and a push-back clasp”.

In the end, systematizing jewelry terms, determining their role, function, and semantic meaning would help to offer high-quality translations by AI. In the transition to a modern digital system, it is relevant to provide background knowledge that contributes to understanding the meaning of concepts and to shift towards new electronic forms of creating dictionaries, glossaries, and thesauri for high-quality translation [28].

3.2. Fundamentals of Jewelry Terminology

In all languages, a lot of idiosyncrasy and cultural referents can be found, especially when we work with specialized texts and speeches. Each jargon is based on the culture where it was originally used or created. This especially happens when a sector is prominent in a society. For instance, wine vocabulary is extensive and rich in most Mediterranean countries but not as much in regions like Alaska, where the weather and habits are completely different from wine areas. In the case of jewelry, we can find a wide range of specific terminology that, at times, is particularly difficult to translate, for example, from Spanish (Spain being a traditional power in the field) into other languages.

Terminology related to jewelry is very wide and varied. It ranges from numerous nouns to name each piece of jewelry to an infinite number of adjectives describing the shape, cut, weight (carat), color or materials it is made of. It can therefore be said that the descriptions of jewelry items are very specific, exhaustive, and dominated by the use of adjectives. There are also different varieties and terms depending on geographical areas and idiolects, as well as cultural implications that underlie jewelry terminology. Nevertheless, there are not many reliable resources from a linguistic, lexicographical, or translational point of view. Most works focus, on one hand, on chemistry, especially on composition, typology and the materials’ alloy [29,30], and, on the other hand, on commerce and marketing [31,32,33,34,35]. Considering the growing demand for internationalization in this sector, there is a strong need to create resources which would help professionals to standardize and use accurate jewelry terminology.

In the context of jewelry, the terminology associated with descriptive words is rich and diverse. Similar to languages that lack a distinct class of adjectives, the descriptions of jewelry items often involve the use of nouns and verbs to convey specific characteristics. Additionally, the concept of gender or noun class, which is common in language, is mirrored in jewelry terminology. For instance, certain jewelry pieces may be classified into categories based on gender, such as masculine or feminine designs, influencing how they are described and categorized. This linguistic parallel illustrates the intricate and nuanced nature of expressing the features of jewelry items.

In the scope of linguistic descriptions, the clarity of explanations often follows the principle that the better the descriptions, the more evident they become, particularly given that those reading them possess a functional knowledge of language. This characteristic holds true for various scientific fields, where intricate issues are articulated in detailed descriptions.

In the context of this research, AI plays a key role in object recognition. The computer vision system identifies objects, and linguistically, we assign a noun to the object along with a brief description. Essentially, jewelry-specific terminology and adjectives are utilized in this linguistic process [36].

The catalog or vocabulary list employed in our approach could provide a broad array of possibilities, but they generally fall into larger categories: fobs, lockets, necklaces, bracelets, armlets, finger rings, watches, buttons, clips, cuff links, pendants, pins, tie clasps, anklets, toe rings, ear ornaments, hair ornaments, and nose ornaments. Furthermore, each group encompasses various items.

The size and shape of each item should be supplemented with details about the material it is made of, the color, and other specifications such as weight and care. Strictly speaking, there are only seven precious stones: pearl, diamond, ruby, emerald, alexandrite, sapphire, and oriental cats-eye. In contrast, the so-called semi-precious stones include amethyst, topaz, tourmaline, aquamarine, chrysoprase, peridot, opal, zircon, and jade.

Information about the hardness, color, and sources of supply of a specific gemstone can be found in various places, and the availability of information may depend on the type of gem and its supplier. Some common places where you might find this information are as follows: gem and diamond certificates, product descriptions in jewelry stores, websites of jewelry suppliers and designers, and books and online resources on gemology.

In our case, we relied on a set of catalogs provided by the Parque Joyero de Córdoba (https://parquejoyero.es/en/home (accessed on 12 May 2025)). Using these catalogs, we created a relational database encompassing materials, precious and semi-precious stones, commonly used terms and tokens, most common meanings, colors, shapes, and common interrelationships in the creation and presentation of each piece of jewelry.

The database serves as a comprehensive resource, capturing a wide array of information related to the field of jewelry. It includes details about the materials used, the types of precious and semi-precious stones featured, prevalent terminology and expressions, as well as the common meanings associated with various elements. Additionally, the database covers information about colors, shapes, and the typical interconnections found in the crafting and display of each jewelry item. This extensive compilation of data provides a foundation for our linguistic models and aids in the development of a nuanced and contextually rich understanding of jewelry descriptions.

3.3. Creation of the Dataset

The jewelry dataset used in the experiments was specifically crafted for this study by extracting, preparing, and merging images from two online jewelry stores (Baquerizo Joyeros: https://baquerizojoyeros.com (accessed on 12 May 2025) and Doñasol: https://doñasol.com (accessed on 12 May 2025)) in Córdoba (Spain). These catalogs included multiple images of the same jewelry item, captured from different perspectives or highlighting particular details (e.g., side views, clasps, gemstone close-ups). These naturally diverse viewpoints were preserved and included in the dataset, enriching the training data beyond standard augmentation techniques.

Nevertheless, as is common in computer vision challenges, the available quantity of images was not sufficient to train robust models adequately for optimal results. To address this limitation, various data augmentation techniques were implemented to diversify the dataset. The augmentation techniques applied to enhance the jewelry database include the following: 90º rotations, width shift by 30%, length shift by 30%, cuts by 15%, enlargement or reduction by 5%, slight color changes, horizontal and vertical flips, and brightness range by 80%.

Following the data augmentation procedures, the merging process resulted in a comprehensive and unified jewelry database comprising a total of 5374 accessory images. For experimentation, this dataset was partitioned into training (75%), validation (15%), and testing (10%) sets. Figure 1 displays some samples from the dataset along with their corresponding captions.

3.4. Experimental Design

The first step in designing the experimentation was to select models that could be relevant for addressing the task of jewelry description. The presence of pretrained neural networks, enriched with extensive datasets, allows us to leverage their capabilities for enhanced results. Consequently, there is no need to build a neural network from the ground up. This approach of repurposing networks is known as Transfer Learning [37]. In the experimentation phase, various combinations of CNN and RNN architectures were explored to identify the optimal encoding and decoding structure for describing the accessory images within the database. Figure 2 provides an overview of the proposed architecture for generating a basic-level jewelry description.

The encoder–decoder structure is selected due to its proven effectiveness in tasks involving short, structured textual outputs, such as image captioning [1], and considering the nature of the available dataset. In our case, the generated textual outputs follow predefined linguistic templates (basic, normal, and complete levels), which significantly reduces the complexity of the sequence modeling task. Furthermore, this architecture offers a favorable trade-off between interpretability, computational efficiency, and performance, especially in low-resource settings where training data and processing capabilities are limited. Although recent Transformer-based architectures [38] have achieved state-of-the-art results in large-scale image captioning tasks, they typically require significantly higher computational resources and larger datasets. For these reasons, the CNN–RNN model remains a solid and accessible solution for domain-specific applications that prioritize structured linguistic output and lightweight deployment.

Three prominent architectures have been explored for CNNs, each with its unique characteristics: VGG-16 [39], InceptionV3 [40], and MobileNet [41].

On one hand, in the field of sequence modeling and predictions, RNNs have been instrumental. However, a common challenge faced by traditional RNNs is short-term memory loss, which hampers their ability to capture long-term dependencies within the data. To overcome this limitation, advanced models like Long Short-Term Memory (LSTM), see [42], and Gated Recurrent Unit (GRU), see [43], have been introduced. Both LSTM and GRU play pivotal roles in enhancing the linguistic finesse of the proposed architectures, making them proficient in describing various jewelry types in images with linguistic sensitivity.

In the context of experimental design, considerations extended beyond the choice of CNN architecture and RNN type. Several critical parameters were systematically explored:

• Number of neurons in the RNN: This parameter affects the model’s capacity to capture complex linguistic patterns. The values considered were 64, 128, 256, 512, and 1024.
• Batch size: This determines the number of samples processed per training iteration. Tested batch sizes included 4, 8, 32, 128, and 512.
• Learning rate: This regulates the step size during the optimization process and is essential for effective model convergence. The explored values were 0.0001, 0.001, 0.01, and 0.1.
• Optimizer: Optimizers influence parameter updates during training, affecting convergence speed and stability. The following optimizers were evaluated: Adam, Adagrad, Adadelta, and RMSProp.

To reduce the risk of overfitting, an early stopping mechanism that monitors the validation loss is implemented. When no improvement is observed over a certain number of epochs, training is halted to prevent the model from fitting non-generalizable patterns in the training data.

Regarding all these considerations, carefully designed experiments were conducted to enable a robust comparison of different parameter configurations for model generation. Following these experiments, the optimal model configuration for generating captions from the jewelry database was determined.

The evaluation of caption quality poses challenges due to the intricate nature of image captioning. While established metrics like METEOR, BLEU, and ROUGE [44] exist for measuring language and semantic accuracy, our dataset’s unique constraints led us to adopt a straightforward approach. We deemed generated captions correct if they matched the original ones provided in the online jewelry stores. For future implementations, there is an intention to enhance the dataset and incorporate these established metrics for a more comprehensive assessment of our model’s performance.

4. Results and Discussion

In this section, we present the results corresponding to the best-performing configurations obtained after a comprehensive evaluation of all tested model architectures and hyperparameter settings. Initially, a classification task was undertaken, categorizing accessories into four distinct classes: necklaces, rings, earrings, and bracelets. After thorough analysis to determine the optimal configuration for each considered encoder and decoder combination, the most favorable outcomes in terms of test Correct Classification Rate (CCR) were attained using VGG-16 as the CNN and GRU as the RNN, as detailed in

Table 2. Best model for identification of the class: necklace, ring, earring, and bracelet.

CNN	RNN	Neurons	Val. CCR	Val. Loss	Test CCR
InceptionV3	GRU	1024	0.9187	0.1026	0.9161
VGG-16	GRU	512	0.9314	0.0390	0.9464
MobileNet	GRU	64	0.8851	0.2696	0.8802
InceptionV3	LSTM	1024	0.9487	0.0301	0.9193
VGG-16	LSTM	512	0.9597	0.0542	0.9227
MobileNet	LSTM	64	0.8811	0.2208	0.8775

. This table also includes additional metrics, such as validation CCR and Loss.

Following the selection of the best configuration, a more comprehensive analysis was conducted, incorporating additional metrics like precision, recall, and F1-Score to assess the model’s performance in classifying each jewelry class independently. Precision measures the accuracy of positive predictions, recall assesses the model’s ability to capture all positive instances, and F1-Score provides a balance between precision and recall, offering a comprehensive evaluation of the model’s performance.

Table 3. Individual class classification quality.

Accessory	Precision	Recall	F1-Score
Necklaces	0.9452	0.9087	0.9131
Rings	0.9276	0.9173	0.9343
Earrings	0.9452	0.9675	0.9674
Bracelets	0.9420	0.8298	0.8807

illustrates the model’s effectiveness in the classification task, achieving scores exceeding 91% for all metrics and accessory types, with the exception of bracelets, which proved more challenging to detect.

It is worth noting that an optimization of the various parameters under consideration was conducted, yielding the best results with the following configuration: number of neurons = 512; batch size = 8; optimizer = Adam; and learning rate = 0.001.

In the context of the primary task of generating comprehensive captions from images of accessories, optimal results were achieved with the VGG-16 and GRU combination. MobileNet was excluded from this investigation due to its tendency to frequently get stuck in local optima, leading to suboptimal performance in handling the specific challenges posed by image captioning in the context of jewelry.

Table 4. Best model for image captioning.

CNN	RNN	Neurons	Val. CCR	Val. Loss	Test CCR
InceptionV3	LSTM	512	0.9307	0.0615	0.8439
VGG-16	LSTM	512	0.9270	0.0881	0.9036
InceptionV3	GRU	256	0.9483	0.0921	0.8354
VGG-16	GRU	256	0.9633	0.0706	0.9345

presents a comparison of all considered encoder–decoder combinations for this task.

In this experimentation, an optimization of the different parameters considered was carried out, resulting in the best outcomes with the following configuration: number of neurons = 256; batch size = 16; optimizer = Adam; and learning rate = 0.001.

Furthermore, the assessment of our proposed approach revealed that the top-performing model exhibited excellent captioning accuracy, closely aligning with the original labels from online jewelry stores. Notably, the captioning accuracy for rings, necklaces, and earrings is very promising. However, the classification accuracy for bracelets, while still commendable, was slightly lower in the test samples. The evaluation criteria involved a meticulous comparison of whether the generated captions precisely matched the authentic descriptions of the accessories captured in the images.

It is important to acknowledge certain limitations in our model. The observed misclassifications, particularly in the case of bracelets, can be attributed to the inherent challenges posed by accessories with similar shapes and materials. Additionally, instances arose where the same type of accessory could exhibit variations in materials or jewels, further contributing to the nuanced nature of accurate classification. These challenges highlight areas for potential refinement and improvement in future iterations of the model.

Ultimately, a three-tiered interface was developed, leveraging the best-performing image captioning models created. This interface allows users to upload an image of an accessory and generate a corresponding description. Illustrated in Figure 3, the interface offers descriptions of varying complexity (see Section 3).

Let us consider an example image of a gold necklace with a pendant featuring a sapphire gemstone. The image is uploaded to the system, and the generated descriptions at different complexity levels are as follows:

• Basic description: ‘A necklace with a pendant’.
• Normal description: ‘A gold necklace with a sapphire pendant’.
• Complete description (after several iterations): ‘This exquisite gold necklace showcases a beautiful sapphire pendant, adding a touch of elegance and sophistication to the accessory’.

From a linguistic perspective, the descriptions exhibit a progressive level of detail, ranging from a simple identification of the accessory to a more elaborate narrative that captures specific attributes, such as the material and gemstone, and an emotional tone. This iterative process in the complete description phase involves multiple iterations to incorporate the embellishment of the jewelry with superlative adjectives, further enhancing the richness of the generated descriptions. This meticulous approach aims to provide potential customers with a more nuanced and compelling understanding of the featured jewelry.

For the sake of reproducibility, the jewelry identification models, multilevel description generation models, and database of jewelry images, along with the necessary code, are available at Jewelry Linguistics Github https://github.com/jewelryling/jewelry_linguistics/ (accessed on 12 May 2025).

5. Conclusions and Future Work

The encoder–decoder architecture, coupled with linguistic models, has proven to be instrumental in generating coherent and contextually relevant descriptions across different types of jewelry. This multilevel linguistic approach contributes to a more versatile and accurate representation of jewelry items. It allows for a tailored and informative description that aligns with diverse user preferences and contextual requirements. The adaptability and scalability of our model underscore its potential in the future for broader applications in the domain of automated image captioning, with implications for diverse industries beyond jewelry analysis.

In considering future avenues for research, the refinement of our linguistic models stands out as a priority to enhance accuracy and accommodate an even broader spectrum of jewelry variations. Exploring advanced network architectures and expanding the dataset to encompass diverse cultural and stylistic influences can contribute significantly to the development of a more robust and culturally sensitive linguistic model. In particular, incorporating attention mechanisms into the encoder–decoder architecture could improve the model’s ability to dynamically focus on relevant visual features, helping to mitigate repetition and enrich caption diversity. Building upon this, we also plan to explore Transformer-based architectures, which leverage self-attention to model long-range dependencies and enable parallel generation, making them especially suitable for producing the complex and expressive descriptions targeted in this work.

Another important direction for future work is to evaluate the robustness of the system under real-world image acquisition conditions, including varying lighting, reflections, and occlusions, which are common in user-generated content and may affect the accuracy of jewelry recognition and description.

Furthermore, incorporating user feedback and preferences into the training process will be pivotal in refining the models for personalized and context-aware jewelry descriptions. This user-centric approach aims to ensure that the generated descriptions align closely with individual preferences and contribute to a more user-friendly and adaptable system.