DRAM: Dynamic Range Modulation for Multimodal Attribute Value Extraction on E-Commerce Product Data

Abstract

With the prosperity of e-commerce applications, the web data of products are presented by multiple modalities, e.g., vision and language. For mining the product characteristics, multimodal attribute values are crucial, which are extracted from textual descriptions, assisted by helpful image regions. However, most previous works (1) fuse the multimodal information within a newly learned range based on co-occurrence rather than language meanings and (2) predict the outputs within a range of all attributes rather than the product-related ones. These issues yield unsatisfactory results; thus, we propose a novel approach via Dynamic Range Modulation (DRAM): (1) First, we propose an Information Range Calibration (IRC) method to dynamically fuse multimodal features of related meanings as Text-Related Embeddings (TEM) within a language range, which is calibrated from the range to fuse language features by a powerful attention mechanism of a pretrained language model. (2) Moreover, an Attribute Range Minimization (ARM) method is proposed to minimize the output attribute range based on the adaptive selection of product-related attribute prototypes. Experiments on the popular multimodal e-commerce benchmarks show that our DRAM performs well compared with previous methods.

1. Introduction

Understanding the web data of e-commerce products, especially mining potentially valuable information, has become significant for online recommendation. Highlighting the product characteristics, attribute value extraction [1,2] is expertly extracts unstructured texts in Figure 1a into attribute value pairs in Figure 1b.

In detail, as one of the information extraction tasks (e.g., Named Entity Recognition (NER) [3]), attribute value extraction predicts the values from a text sequence based on a symbol set. For example, “B-sleeve_type, I-sleeve_type, O” marks the beginning, inside, and outside parts (not a value) of a value “short sleeve” for an attribute “sleeve_type” from the input text “short sleeve long” in Figure 1.

Furthermore, multimodal attribute value extraction is naturally text-dominant, with only text-related image regions useful for fusion, which is different from typical vision-dominant multimodal fusion tasks [4,5] that fully employ the information of all the regions of multiple vision inputs for fusion. However, most previous methods of multimodal attribute value extraction use the text-guided or cross-modal fusion modules such as M-JAVE [6], EKE-MMRC [7], and SMARTAVE [8], whose parameters are newly learned from the co-occurrence of texts and images during training. Moreover, the generative manner of a recent multimodal model like DEFLATE [9] yields unstable prediction rather than fixed-length sequence tagging, which performs unsatisfactorily in the following Section 4. For example, the hemline of a dress is co-occurred and mismatched with the word “Long” and yields wrong output in the left of Figure 1a.

Inspired by the pretrained language models to fuse contextual language features, we propose to fuse multimodal features of related meanings within a language range. Firstly, based on the fine-tuning by multimodal tasks, a language model is “calibrated” to handle multimodal inputs, as shown in the bottom of Figure 2. Then, multimodal features of related language meanings are fused dynamically via its powerful attention mechanism within a multimodal language range, which is calibrated from a unimodal language range.

In addition, the diversity of products leads to different attribute ranges. For example, “battery_life” more rarely occurs in clothes than “sleeve_length”. However, most previous methods set the candidate attribute range as all attributes in the dataset, as shown in Figure 1b. Quantitatively, every textual position is classified into double the attribute amount $\mathrm{N}$ with an “O”, denoted as the numeric range of the predicted attribute classes $C_o$:

$${\mathrm{C}}_{\mathrm{o}}=(\mathrm{N}\times |\{\mathrm{B},\mathrm{I}\}|+|\left\{\mathrm{O}\right\}\left|\right)\times {\mathrm{S}}_1,$$

(1)

where ${\mathrm{S}}_1$ is the text length, and $|·|$ is the number of predicted tagging symbols, “B”, “I”, and “O”, for each kind of attribute class. Such a maximal attribute range leads to non-essential classification outputs, which leads to wrong predictions.

To address the above issues, we design a novel approach to dynamically minimize the attribute range of the specific product. First, the attribute range is determined via only one multi-label classification. Second, it looks up learnable prototypes of these selected attributes. Finally, only values within the range are predicted via the guidance of the chosen prototypes. Given that ${\mathrm{N}}_{\mathrm{m}}<<\mathrm{N}$ is the number of selected attributes, the range of the predicted class is smaller than Equation (1):

$${\mathrm{C}}_{\mathrm{m}}=\mathrm{N}+\left(\right|\{\mathrm{B},\mathrm{I},\mathrm{O}\}|\times {\mathrm{S}}_1)\times {\mathrm{N}}_{\mathrm{m}}\Rightarrow {\mathrm{C}}_{\mathrm{m}}<<{\mathrm{C}}_{\mathrm{o}}.$$

(2)

In conclusion, we have observed the issues that hinder boosting the performance of multimodal attribute value extraction, including both the calibration of the information range and the minimization of the attribute range. As illustrated in Figure 2, we propose a novel attribute value extraction approach, named Dynamic Range Modulation (DRAM). The main contributions of this paper are as follows:

• First, the Information Range Calibration (IRC) method is proposed to dynamically fuse the multimodal information of related meanings into a language range by fine-tuning an attention-based language model to handle each modality as Text-Related Embeddings (TEM).
• Second, the Attribute Range Minimization (ARM) method is designed to decrease wrong predictions in two steps: (1) Once multi-class classification decides the attributes range, (2) learnable prototypes are selected to dynamically predict for chosen attributes.
• Finally, by the integration of the proposed IRC and ARM, our new approach DRAM achieves superior performances compared to the previous state-of-the-art techniques on popular MEPAVE and MAE benchmarks.

2. Related Works

2.1. Unimodal Attribute Value Extraction

Due to the text-centric task characteristics, early works of attribute value extraction were designed for unimodal texts.

For instance, OpenTag [10] makes multi-class prediction at each textual position as sequence tagging via BiLSTM+CRF. Similarly, slot-filling methods [11,12] use frozen word embeddings and fine-tuned RNN. Slot-Gated [13] adopts a gating mechanism. However, each position should be classified into $2\times$ all attribute numbers, leading to limitations on large-scale datasets with thousands of attributes.

Based on OpenTag, SUOpenTag [14] predicts for given attributes of input texts and thus succeeds on large-scale datasets. AVEQA [15] learns a matching task between text and attribute. AdaTag [16] encodes attributes as the dynamic parameters. However, if the known attributes are unavailable, false positives might occur on all the attributes. Joint-BERT [17] fine-tunes a pretrained model on down-stream tasks. Furthermore, K-PLUG [18] is fine-tuned on 25 M extra data.

Recently, in a generative manner of prediction, Joint-AVE [2] explores more modern pretrained models like T5 [19] and FLAN-T5 [20]. ExtractGPT [21] and New-LLM [1] are proposed based on a Large Language Model (LLM) for textual attribute value extraction.

Differently, our proposed ARM minimizes the attribute range for current products by conditional guidance of their prototypes. In addition, our IRC performs fine-tuning without any extra data, and a more dynamic language range of fusion is achieved by the task-calibrated multimodal language model. The generative manner is more unstable than our DRAM constrained by explicitly tagging the text by B-I-O symbols as sequence tagging.

2.2. Multimodal Attribute Value Extraction

Recently, attribute value extraction models have also benefited from the assistance of more modalities, especially visual images of e-commerce products with rich descriptive clues to distinguish the attribute values.

MAE-model [22] predicts the input image, text, and attribute by generating the attribute value. OCR allows PAM [23] to obtain more visual clues, whereas the stability of generation is unsure without the constraint of texts. M-JAVE [6] is a sequence tagging approach, but it extracts task-agnostic features on frozen models. Newly initialized decoders learn from the co-occurrence of texts and images and thus yield an inaccurate non-language range. They predict without any attribute range minimization, thus struggling in a maximum range.

Differently, EKE-MMRC [7] adopts a frozen pretrained object detector to obtain objects which are more related to language via class labels. Its knowledge-guided generative prediction is less stable than sequence tagging. SMARTAVE [8] fine-tunes both visual and textual encoders and embraces extra OCR inputs. However, it adopts known attribute inputs like various unimodal methods [14,15,16]. With a strong language model, our proposed IRC calibrates a language dynamic range of multimodal fusion. Moreover, our ARM minimizes the attribute range via prototype guidance.

Recently, DEFLATE [9] proposed a novel multimodal T5 Encoder [19] with multimodal self-attention and exploited DALL-E [24] as a visual encoder, which is a popular text-to-image multimodal model with a QA-based generation model. Meanwhile, EIVEN [25] is proposed with a multimodal LLM fine-tuned for down-stream multimodal attribute value extraction, which reports its performance on non-mainstream datasets unlike widely used MEPAVE [6].

However, they implicitly generate output from such a generative model, unlike the output of our DRAM constrained, by explicitly tagging the text by B-I-O symbols in a sequence tagging manner.

2.3. Multimodal Encoding via Language Models

Pretrained language models are widely used to encode multimodal information because they do well in finding language-related clues. In Named Entity Recognition (NER), RpBERT [3] extracts textual named entities based on text and images. BERT is fine-tuned on such multimodal tasks. Similarly, visual-language pretraining methods adopt a language model to encode texts and image patches into latent semantics [26]. Because there are text-irrelevant image regions for product data, our proposed IRC selects the parts of related meanings, encodes them into Text-Related Embeddings (TEM), and then feeds them into a language model.

3. Proposed Method

In Figure 2, our proposed Dynamic Range Modulation (DRAM) involves two components: Information Range Calibration (IRC) and Attribute Range Minimization (ARM). IRC decides the ranges of multimodal information to be fused. Then ARM selects the range of attributes to predict values.

3.1. Information Range Calibration

As shown in Figure 3, Information Range Calibration (IRC) calibrates the pretrained language model to handle multimodal inputs as Text-Related Embeddings (TEM). Related multimodal features are fused dynamically via its attention mechanism within a multimodal language range calibrated from pretrained models.

A merely frozen pretrained language model is not trained to adopt multimodal inputs. Hence, fine-tuned by the multimodal attribute extraction task, a language model to fuse related language meanings is calibrated from a pretrained unimodal one into a multimodal one.

In detail, an attention-based language model [27] is composed of multiple stacked layers, where Multi-Head Self-Attention is one of the key elements. After large-scale language pretraining, MHSA is powerful enough to fuse the features with related language meanings within an accurate range of fusion. However, the self-attention mechanism [28] is not trained with multimodal inputs to fuse only textual-related visual parts with their corresponding parts; thus, we propose to calibrate the information range of the vision input, which modifies the attention map by unimodal text-queried attention with multimodal keys and values.

3.1.1. Task-Calibrated Multimodal Language Model

Given two input sequences, $x_Q\in {\mathbb{R}}^{{\mathrm{S}}_{\mathrm{q}}\times \mathrm{D}}$ and $x_K=x_V\in {\mathbb{R}}^{{\mathrm{S}}_{\mathrm{k}}\times \mathrm{D}}$, they are projected by weight matrices $W_Q$, $W_K$, and $W_V\in {\mathbb{R}}^{D\times D^{\prime }}$, where $S_q$ and $S_k$ are the lengths of the two inputs. D and $D^{\prime }$ are the dimensions before and after projection. Thus, the projected sequences are

$$Q=x_Q{W}_Q,K=x_K{W}_K,V=x_V{W}_V.$$

(3)

$Q\in {\mathbb{R}}^{{\mathrm{S}}_{\mathrm{q}}\times {\mathrm{D}}^{\prime }},$K, and $V\in {\mathbb{R}}^{{\mathrm{S}}_{\mathrm{k}}\times {\mathrm{D}}^{\prime }}$ represent three kinds of features, where Q is a “Query” to retrieve useful parts of V “Value”. However, since Q and V might originate differently, e.g., they are either text or an image, directly computing their matching degree is inappropriate. K is paired with each position in V, which performs the “Key” role of the key value pair to match Q. Thus, we have the formula of attention:

$$Attention(K,Q,V)={\displaystyle \frac{Q{K}^{\top }}{\sqrt{D^{\prime }}}}V,$$

(4)

where the attention mapping $M=Q{K}^{\top }/\sqrt{D^{\prime }}\in {\mathbb{R}}^{{\mathrm{S}}_1\times {\mathrm{S}}_2}$ means the matching degrees across each position of Q and K, normalized by $\sqrt{D^{\prime }}$. All the items of V are multiplied to these degrees in M. The result of Equation (4) means the information inside V matched to Q is extracted via the intermediate K, which can be fused to Q.

As illustrated in Figure 4a, considering the inputs T and V, the attention mapping $M=Q{K}^{\top }/\sqrt{D^{\prime }}$ of the 1st layer of the language model in Vanilla IRC can be divided into 4 parts: T→T, V→V, T→V, V→T. Text-irrelevant parts (e.g., a background region with a red ✗ in Figure 4) of V contribute the V→T and V→V and lead to a wrong range of multimodal fusion. So we propose to solve this issue, which is evaluated by experiments in the following sections.

3.1.2. Text-Related Embeddings for Task-Calibrated Multimodal Language Model

Since the Vanilla IRC might produce a wrong text-irrelevant range of multimodal fusion, we proposed stronger Text-Related Embeddings (TEM) to obtain a text-related range. In detail, text embeddings $x^t$ perform as $x_Q$ for both self-attention inside a text and cross-attention between text and image. For self-attention, $x_K=x_V=x^t$. For cross-attention, $x_K=x_V=x^v$. Two attentions are formulated respectively as

$$SelfAtt\left(x^t\right)={\displaystyle \frac{\left(x^t{W}_Q^t\right){\left(x^t{W}_K^t\right)}^{\mathrm{T}}}{\sqrt{D^{\prime }}}}{x}^t{W}_V^t,$$

(5)

$$CrossAtt(x^t,x^v)={\displaystyle \frac{\left(x^t{W}_Q^v\right){\left(x^v{W}_K^v\right)}^{\mathrm{T}}}{\sqrt{D^{\prime }}}}{x}^v{W}_V^v.$$

(6)

In Figure 3c and Figure 4b, based on these two attentions, text embeddings play the roles of queries to extract text-related information as T→V and T→T, without the visual V→T and V→V. Given text and image embeddings $E=x^t,G=x^v$, the extra embeddings $E_m$ are calculated similarly to the positional or token-type ones in BERT or ViT, which can also be added into input text embeddings E:

$$E_m=SelfAttn(E,E,E)+CrossAttn(E,G,G).$$

(7)

Finally, $E_{TEM}=E+E_m(E,G)$ replaces the $\varphi (E,G)$ and is fed into the task-calibrated multimodal language model of our proposed IRC, which achieves a better range of text-related multimodal fusion, compared with Vanilla IRC.

3.2. Attribute Range Minimization

In e-commerce scenarios, the attribute ranges of products diversify greatly, e.g., “battery_life” appears in electronic products more often than clothes. However, most methods [6,10] predict the existence of all attributes, leading to unnecessary false positives, while others [14,16] rely on the known attributes of the input that might not be available with only image–text pairs of the products as inputs.

Having observed the inconsistency between data distribution and model design above, we propose a novel dynamic Attribute Range Minimization (ARM) method to tackle it. In detail, the “Guidance Mechanism” guides the model by classification loss to dynamically select the attributes related to current inputs and punishes the mistakenly selected attributes by the model. With these supervisions, the model is trained to predict only for the selected attributes, which is considered as kind of “Guidance”.

3.2.1. Different Policies of ARM

First, as shown in Figure 5d, the model predicts the attribute range dynamically for the current inputs, as a multi-label classification task:

$$R=\sigma \left(F_{{\theta }_{IRC}}(E,G)\left[\mathrm{CLS}\right]W_{cls}\right).$$

(8)

R denotes the existence score of all attributes given textual and visual inputs $(E,G)$. Our proposed IRC ${\theta }_{IRC}$ outputs multimodal feature F. The $\left[\mathrm{CLS}\right]$ feature is projected into the digits for $\mathrm{N}$ attributes by $W_{cls}\in {\mathbb{R}}^{\mathrm{D}\times \mathrm{N}}$. A sigmoid function $\sigma$ non-linearly normalizes the digits into scores $R\in [0,1]$.

As shown in Figure 5, three different policies are designed: (1) The Prototype-Guided Policy selects the learnable vector “Prototypes” P corresponding to the attributes looked up by R and guides the model to predict values only for these attributes. (2) The DyNet-Guided Policy applies the parameters W of the selected attributes into a dynamic network. (3) BERT-Guided Policy uses a pretrained BERT model to encode the embeddings $\beta$ of chosen attribute words.

Take policy (1) as an example. The score R is binarized with ${\tau }_R$ for “Proto Lookup” in Figure 5d. Then, a subset $P_{min}$ is looked up from the prototypes P of all attributes, where a decreased count of prototypes $|P_{min}|={\mathrm{N}}_{\mathrm{m}}<<\left|P\right|=\mathrm{N}$ is a minimization of the attribute range:

$$P_{min}=P[R>{\tau }_R].$$

(9)

Denote a prototype vector for an attribute $A_i$ selected within the range as $P_{min}^{A_i}\in {\mathbb{R}}^{1\times D}$ and the feature at textual position j for the current input as $F_j\in {\mathbb{R}}^{1\times D}$; the feature $F_j^{A_i}$ is conditioned for $A_i$ via a “Guidance Mechanism”:

$$F_j^{A_i}=Concate(F_j,{\displaystyle \frac{{\left(P_{min}^{A_i}\right)}^{\top }\left(F_j\right)}{\parallel P_{min}^{A_i}\parallel \parallel F_j\parallel }}{F}_j).$$

(10)

${\omega }_{ij}=\left({\left(P_{min}^{A_i}\right)}^{\top }\left(F_j\right)\right)/(\parallel P_{min}^{A_i}\parallel \parallel F_j\parallel )$ between every $F_j\in F$ and $P_{min}^{A_i}$ indicating the potential existence of $A_i$’s values. The concatenation of $F_j$ and ${\omega }_{ij}·F_j$ allows them to contribute equally in feature channels. Each $F^{A_i}$ is fed into the feature decoder to predict a single-class “B-I-O” sequence $Y^{A_i}$.

3.2.2. The Prediction Pipeline of ARM

Furthermore, to explain the prediction pipeline of our proposed ARM, some detailed examples are provided as following in a step-by-step manner:

(1) Given an input textual description and its corresponding image of the product item, the content of this text and its attribute value annotations as a “B-I-O” sequence are

Input text:

“13 inch bag matched with sliver laptop computers”

Annotation:

“B-$\mathrm{bag}\_\mathrm{size}$ I-$\mathrm{bag}\_\mathrm{size}$ O O O O B-$\mathrm{cont}\_\mathrm{type}$ I-$\mathrm{cont}\_\mathrm{type}$”

(2) Assuming the total attributes of the whole dataset are

$$A=\{\mathrm{bag}\_\mathrm{size},\mathrm{cont}\_\mathrm{type},\mathrm{bag}\_\mathrm{thick},\mathrm{bag}\_\mathrm{color}\}.$$

(11)

Multi-label classification of ARM outputs four scores $R\in [0,1]$ which indicate the existence of these attributes. The ground truth of the classification for the input above is $[1,1,0,0]$.

Meanwhile, because it is difficult for the model to learn the conditioned prediction of the selected attributes with both a handful of scores and high-dimensional feature F, high-dimensional vectors are proposed to replace these scores. In this example, the model should have four vectors:

$$P=\{P^{\mathrm{bag}\_\mathrm{size}},P^{\mathrm{cont}\_\mathrm{type}},P^{\mathrm{bag}\_\mathrm{thick}},P^{\mathrm{bag}\_\mathrm{color}}\}.$$

(12)

They are “Prototypes”, since every $P^{A_i}\in {\mathbb{R}}^D$ represents the existence of an attribute $A_i$ and is learnable like the other network parameters which are updated during training.

(3) If the scores are $R=[0.7,0.4,0.6,0.3]$, they are converted by ${\tau }_R=0.5$ into a list $[1,0,1,0]$, which means that a subset “bag_size”, “bag_thick” is selected as the minimized attribute range. Hence, $\{P^{\mathrm{bag}\_\mathrm{size}},P^{\mathrm{bag}\_\mathrm{thick}}\}$ are selected.

With the Guidance Mechanism in Equation (12), the features conditioned by the selected prototypes are $\{F^{\mathrm{bag}\_\mathrm{size}},F^{\mathrm{bag}\_\mathrm{thick}}\}$, which are fed into the feature decoder to predict.

In practice, for a parallel inference, the conditioned features of each input in a mini-batch are accumulated to replace the 1st batch dimension. If the batch comprises three inputs and the number of selected prototypes in each input is $\{2,3,1\}$, the feature $F\in {\mathbb{R}}^{3\times S_1\times D}$ is conditioned by the selected prototypes into

$$F^{cond}=\{F^{\{a_1^1,a_1^2\}}\in {\mathbb{R}}^{2\times S_1\times D},F^{\{a_2^1,a_2^2,a_2^3\}}\in {\mathbb{R}}^{3\times S_1\times D},F^{\left\{a_3^1\right\}}\in {\mathbb{R}}^{1\times S_1\times D}\},$$

(13)

where $F^{cond}\in {\mathbb{R}}^{(2+3+1)\times S_1\times D}$ rather than three separate features which can only be fed into the feature decoder one by one. For simplicity, the batch size is set to one in the following sections.

(4) Based on $\{F^{\mathrm{bag}\_\mathrm{size}},F^{\mathrm{bag}\_\mathrm{thick}}\}$, the ground truths that correspond to them are generated from the original annotation. If the feature decoder predicts single-class “B-I-O” results as

Prediction based on $F^{\mathrm{bag}\_\mathrm{size}}$:

“B I O O O O O O”

Prediction based on $F^{\mathrm{bag}\_\mathrm{thick}}$:

“B O O O O O O”

Original annotation:

“B-$\mathrm{bag}\_\mathrm{size}$ I-$\mathrm{bag}\_\mathrm{size}$ O O O O B-$\mathrm{cont}\_\mathrm{type}$ I-$\mathrm{cont}\_\mathrm{type}$”

Ground truth for $\mathrm{bag}\_\mathrm{size}$:

“B I O O O O O O”

Ground truth for $\mathrm{bag}\_\mathrm{thick}$:

“O O O O O O O O”

Since $\mathrm{bag}\_\mathrm{thick}$ is mistakenly selected, the ground truth of its prediction is generated into a sequence filled with “O”s, but its prediction is still inconsistent. Therefore, the model will be punished by both the Binary Cross-Entropy loss and the CRF loss for attribute value extraction.

3.3. Loss Functions

Log-Likelihood loss in the CRF layer in Figure 3 supervises the attribute value extraction task. Hence, the overall loss function $\mathcal{L}$ has only ${\mathcal{L}}_{CRF}$, which is denoted as

$$\mathcal{L}={\mathcal{L}}_{CRF}=-log{\displaystyle \frac{p_{\mathrm{GT}}}{p_1+p_2+\cdots +p_{{\mathrm{C}}_{\mathrm{o}}}}},$$

(14)

where ${\mathrm{C}}_{\mathrm{o}}$ is the classification time (denoted in Section 1). The CRF layer outputs probability p for each potential “path” of the output sequence in the CRF. For the proposed ARM, multi-label Binary Cross-Entropy loss ${\mathcal{L}}_{BCE}$ optimizes the classification of predicted R and label $\overline{R}$:

$${\mathcal{L}}_{BCE}=-{\displaystyle \frac{1}{\mathrm{N}}}\sum _{k=1}^{\mathrm{N}}{\overline{R}}_{k}log{R}_k+(1-{\overline{R}}_k)log(1-R_k).$$

(15)

The prediction of the CRF layer in the feature decoder equipped with ARM is multiple single-class sequences $Y^{A_i}$, e.g., “O, B, I, O, O”, for attribute $A_i$ from the total selected ${\mathrm{N}}_{\mathrm{m}}$ attributes. Given ground truth ${\mathrm{GT}}_i$ for each selected $A_i$, ${\mathcal{L}}_{CRF}^{ARM}$ is denoted as

$${\mathcal{L}}_{CRF}^{ARM}=-{\displaystyle \frac{1}{{\mathrm{N}}_{\mathrm{m}}}}\sum _{i=1}^{{\mathrm{N}}_{\mathrm{m}}}log{\displaystyle \frac{p_{{\mathrm{GT}}_i}}{p_1+p_2+\cdots +p_{{\mathrm{C}}_{\mathrm{m}}^{\prime }}}},$$

(16)

where ${\mathrm{C}}_{\mathrm{m}}^{\prime }$ and ${\mathrm{C}}_{\mathrm{m}}$ are mentioned in Section 1. Finally, the overall loss function is

$$\mathcal{L}={\mathcal{L}}_{CRF}^{ARM}+\lambda {\mathcal{L}}_{BCE}.$$

(17)

4. Experiments

4.1. Datasets

MEPAVE [6] is collected from a mainstream Chinese e-commerce web platform JD.com, with 87,194 instances with 26 types of attributes. In total, 71,194/8000/8000 is split as the train/val/test set. The F1 score, i.e., harmonic mean of Precision and Recall, is the main metric of both attribute classification (CLS-F1) and attribute value extraction (TAG-F1). Precision and Recall are unavailable in most previous works [6,7] for comparison. The detailed statistics of MEPAVE is shown in

Table 1. Statistics of the MEPAVE dataset [6].

Category	# Product	# Instance	# Attr	# Value
Clothes	12,240	34,154	14	1210
Shoes	9022	20,525	10	1036
Bags	3376	8307	8	631
Luggage	1291	2227	7	275
Dresses	4567	12,283	13	714
Boots	713	2090	11	322
Pants	2832	7608	13	595
Total	34,041	87,194	26	2129

“#” means the number of data or annotations.

.

MAE [22] is an English e-commerce multimodal attribute extraction dataset, collected from various e-commerce websites via Diffbot Product Web API. In total, 2.2 M records with 2 K types of attributes are split into a train/val/test set by 8:1:1, which is a larger scale than MEPAVE in the range of attribute classes and thus challenging. To avoid the attribute values not in the texts, the MAE-text subset is proposed by M-JAVE [6] and denoted as MAE in this paper. Accuracy is the performance metric of attribute value extraction: num(correctly_predicted_attr_values)/num(total_attr_values). The overall statistics of the MAE dataset is shown in

Table 2. Statistics of the MAE dataset [22], where the detailed categories are not tagged during collection, according to its original paper.

# Product	# Instance	# Attr	# Value
2.2 M	7.6 M	2.1 K	23.6 K

due to the detailed categories not being tagged during collection in the original paper.

The MEPAVE dataset can be accessed by submitting the application form to the authors via https://github.com/jd-aig/JAVE (accessed on 15 Februrary 2025). The MAE dataset is available online at https://rloganiv.github.io/mae/ (accessed on 15 Februrary 2025).

4.2. Implementation Details

Our proposed method is implemented on the basis of the BiLSTM+CRF [10] feature decoder and PyTorch 1.6 framework. RoBERTa-small-Chinese is specialized for the Chinese dataset MEPAVE, and BERT-base-uncased is specialized for the English dataset MAE. ResNet-152 pretrained on ImageNet encodes images. For our proposed IRC, Adam optimizes fine-tuning with a learning rate of $5\times {10}^{-5}$. The loss weight $\lambda =1.0\times {10}^3$. For MEPAVE, our model is trained and tested on one Nvidia V100 GPU. For MAE, a total batch size of 560 on 8 GPUs is used.

4.3. Ablation Study

Table 3. Ablation study between frozen or task-calibrated language models of our proposed IRC.

Scheme	Method	TAG-F1
Frozen	M-JAVE	87.17
Task-calibrated	SMARTAVE	91.52
	M-JAVE ‡	95.40
	IRC (ours)	96.35

“‡” is reproduced results of fine-tuned M-JAVE. If not mentioned, all bold are the best results and best methods.

shows the ablation study between the frozen or task-calibrated language models of our proposed IRC. For the multimodal inputs, our IRC surpasses the multimodal M-JAVE [6] with frozen pretrained models and gating mechanisms. To compare with newly learned text-irrelevant parameters for fusion, we also report the results of a fully fine-tuned M-JAVE [6] (marked with “‡”). Another fully fine-tuned method SMARTAVE [8] is also compared. Our IRC surpasses them under the same settings with the advantage of our task-calibrated multimodal language model.

In

Table 4. Ablation study on our proposed IRC regarding different language models and TEM. Note that Vanilla IRC (V-IRC) is affected by the issue in Figure 4.

Modality	Method	TAG-F1
T	Vanilla-IRC (BERT)	94.88
	Vanilla-IRC (RoBERTa)	95.21
T+V	V-IRC (RoBERTa + ResNet)	95.05
	+1 × self-attention layer	94.89
	+TEM (our final IRC)	96.35

, the BERT version might over-fit with more parameters during training, while RoBERTa generalizes ideally as a basic model. To evaluate the problem of a self-attention language model of Vanilla IRC due to text-irrelevant image regions, we compare our TEM with a naïve self-attention layer. Replacing it with our proposed TEM that encodes text-related parts of each modality into text-related embeddings, TAG-F1 clearly increases to 96.35. Moreover,

Table 5. Sensitive analysis about the weight $\lambda$ of classification loss for our proposed ARM in Equation (17).

$\mathit{\lambda }$ in Equation (17)	CLS-F1	TAG-F1
$\lambda =1.0\times {10}^1$	97.58	96.68
$\lambda =1.0\times {10}^3$	97.65	96.86
$\lambda =1.0\times {10}^5$	96.96	95.39

shows the weight $\lambda$ of classification loss for our proposed ARM in Equation (17) affecting the learning direction of attribute classification (CLS-F1) and attribute value extraction (TAG-F1). Too large $\lambda$ suppresses the learning of attribute value extraction and thus to lower TAG-F1. Hence, $\lambda =1.0\times {10}^3$ is the most suitable.

4.4. Analysis of Precision and Recall

Table 6. Analysis on Precision and Recall of ARM and TEM. “+A+T” is the final version of our DRAM.

Method	Precision	Recall	TAG-F1
V-IRC	94.04	96.07	95.05
+ARM	94.99 (0.95)	96.71 (0.64)	95.84 (0.79)
+TEM	95.21 (1.17)	97.52 (1.45)	96.35 (1.30)
+A+T	95.89 (1.85)	97.86 (1.79)	96.86 (1.81)

analyzes the Precision and Recall of ARM and TEM. Both Vanilla IRC+ARM and +ARM+TEM (=DRAM) obtain a higher Precision gain than Recall, which significantly shows that ARM decreases false positives by Attribute Range Minimization. In detail, Precision = True Positives/(True Positives + False Positives) evaluates the false positives from wrong attributes predicted by the model, and our proposed ARM solves this issue by minimizing the attribute range into a correct one. And the Recall and TAG-F1 gains of both TEM and ARM are also prominent, which shows its advantage of eliminate False Negatives from missing attributes by filtering the text-relevant visual parts to emphasize the existence of attributes by TEM. In conclusion, our proposed DRAM wins higher F1-score to further boost the task of attribute value extraction.

4.5. Comparison of Variants of ARM

We evaluate the effectiveness of different policies in Figure 5 for our proposed Attribute Range Minimization (ARM), based on our Vanilla IRC and TEM.

Table 7. Comparison of the variants of our proposed Attribute Range Minimization (ARM): Prototype (P), DyNet (D), and BERT (B).

Method	D	B	P	CLS-F1	TAG-F1
V-IRC+TEM				-	96.35
+DARM	✓			97.52	96.07
+BARM		✓		97.54	96.62
+PARM			✓	97.65	96.86

“✓” denotes the usage of different variants of our proposed ARM.

shows that the DyNet-Guided Policy leads to a slightly lower performance, probably because of high instability during training dynamic parameters. The BERT-Guided Policy ranks the $2\mathrm{nd}$ with more parameters than the prototype, which might over-fit while training. Finally, Prototype-Guided Policy performs the best in both CLS-F1 and TAG-F1. Such performance shows that the model can learn both correct attribute classification (CLS-F1) and attribute value extraction (TAG-F1) via simple but effective learnable “Prototype” vectors, which is stabler than dynamic networks and lighter than heavier BERT pretraining weights. Thus, the prototype-guided ARM is selected as a component of our DRAM.

4.6. Selection of ${\tau }_R$ for ARM

Considering the diversified attribute distribution of different benchmarks of MEPAVE [6] and MAE [22], multiple values of the hyper-parameter ${\tau }_R$ are shown in Figure 6, which are thresholds to filter the attribute range of our proposed ARM. Intuitively, a more balanced distribution of attributes leads to the equal higher probability of attribute existence in each sample; thus, a too-low threshold might always obtain all maximized attributes, which is meaningless. In contrast, a long-tailed distribution makes some tailed classes have low presence probability, and a too-high threshold might always ignore them.

For the 26 relatively compacted and balanced attributes in MEPAVE, a higher ${\tau }_R$ value encourages the attribute range to converge into a smaller one. Therefore, in Figure 6a, ${\tau }_R=0.5$ achieves the best TAG-F1 score over the other values. For the distracted and long-tailed 2 K attributes in MAE, a lower ${\tau }_R$ value obtains all the potential attributes covered in the attribute range, especially the low-frequent ones. Hence, ${\tau }_R=0.4$ improves the performance of $0.5$ significantly in Figure 6b. Due to the constraint memory budget of our GPUs, other values of ${\tau }_R$ on the large-scale benchmark MAE are not experimented with. Note that it is a priori knowledge that ${\tau }_R$ is selected according to data distribution. Even the default 0.5 achieves state-of-the-art performance on both benchmarks.

4.7. State-of-the-Art Comparisons

For MEPAVE, our DRAM is compared with unimodal and multimodal methods (

Table 8. Comparison of state-of-the-art models on MEPAVE. “*” means extra unimodal or multimodal vision-language data pretraining.

Method	Modality	CLS-F1	TAG-F1
RNN-LSTM [12]	T (slot-filling methods)	85.76	82.92
Attn-BiRNN [11]		86.10	83.28
Slot-Gated [13]		86.70	83.35
Joint-BERT [17]		86.93	83.73
SUOpenTag [14]	T (attribute value extraction)	-	77.12
JAVE [6]		87.98	84.78
AVEQA [15]		-	89.15
AdaTag [16]		-	81.36
MAVEQA [29]		-	88.71
SMARTAVE [8]		-	89.21
K-PLUG * [18]		-	95.97
M-JAVE [6]	T+V (attribute value extraction)	90.69	87.17
PAM [23]		-	89.68
SMARTAVE [8]		-	91.52
EKE-MMRC [7]		-	93.52
DEFLATE * [9]		96.09	87.12
DRAM (ours)		97.65	96.86

): slot-filling methods RNN-LSTM [12], Attn-BiRNN [11], Slot-Gated [13], and Joint-BERT [17]; unimodal attribute value extractors SUOpenTag [14], JAVE [6], AVEQA [15], AdaTag [16], MAVEQA [29], SMARTAVE-text [8], and K-PLUG [18] (pretrained with extra 25 M data, marked with “*”); multimodal attribute value extractors M-JAVE [6], PAM [23], SMARTAVE [8], EKE-MMRC [7], and DEFLATE [9] (pretrained with multimodal foundation model DALL-E [24], marked with “*”). Joint-BERT, K-PLUG, and SMARTAVE fine-tune the pretrained models, while the others only use the frozen ones. PAM and SMARTAVE have extra OCR input. The source code link of EIVEN [25] provided in its paper is currently broken; thus, its implementation on the mainstream dataset MEPAVE is unavailable. Our proposed DRAM out-performs previous methods as the new state of the art, especially on the TAG-F1 metric.

For MAE, state-of-the-art methods MAE-model [22] and M-JAVE [6] are compared. Note that the MAE-model is a generative model based on extra known attribute inputs, while our DRAM and M-JAVE need no extra inputs.

Table 9. Comparison with state-of-the-art models on MAE.

Method	Modality	Accuracy
MAE-model [22]	T+V (attribute value extraction)	72.96
M-JAVE [6]		75.01
DRAM (ours)		79.20

shows that our DRAM performs the best on such a large-scale dataset with 2 K attributes. Our DRAM performs well with both benchmarks.

5. Visualizations

As shown in Figure 7, the attention maps on the input image are visualized under the query of Chinese characters from the MEPAVE [6] dataset. Ignoring key parts to indicate the laced pattern of the dress but affected by the text-irrelevant background, Vanilla IRC misunderstands the $2\mathrm{nd}$ Chinese character “silk” as a value of an attribute “Material”. Instead, DRAM captures more laced patterns on the top and bottom of the dress by our proposed TEM, and thus, successfully extracts two Chinese characters as a value “laced” of an attribute “Pattern” in a correct attribute range by our proposed ARM. Meanwhile, such visualization also shows key roles of images for multimodal attribute value extraction.

Moreover, for the examples correcting the missing prediction of attributes, Figure 8 shows model prediction and corresponding attention visualization of other e-commerce products like bags and jackets. From the baseline model Vanilla IRC, the distracted attention cannot focus on the key text-related parts in the input images, i.e., “black” on the surface of the bag for attribute “Color”, and “round collar” on the top of the jacket for the attribute “Collar”. With text-guided attention of our proposed TEM, our DRAM focuses on both correct visual parts for attributes and global contexts around these visual parts to enrich multimodal semantics.

6. Future Works

Our proposed DRAM is task-specific for multimodal attribute value extraction, which requires that the task should be both text-dominant and multimodal, as well as highly relevant with fine-grained attributes of the input. The range of attributes should be predefined rather than unstable by generation, but the current attribute of input images and texts should be unknown. Hence, our proposed DRAM can be applied to some multimodal tasks like fine-grained VQA (Visual Question Answering) [30,31,32], with object attributes in textual questions and the image of the objects. Meanwhile, stronger baselines will be adopted more than sequence tagging methods [6], especially the generative prediction manner of multimodal language models like BLIP [33] and Flamingo [34].

For failure cases, Section 3.2.2 shows a typical case that “Ground Truth of bag_thick” is all “O” for no presence of attributes, while the model selects “B” at the first token in “Prediction based on F^bag_thick”. It is sensitive to the numeric token “13” in the original textual input “13 inch bag matched with sliver laptop computers”; such quantitative thickness cannot be directly reflected by the input image of a bag like Figure 8; thus, the assistance of multimodal input is ineffective, which should be tackled by future works.

7. Conclusions

In this paper, we proposed a novel multimodal e-commerce attribute value extraction method via Dynamic Range Modulation, which comprises two key components: IRC calibrates the pretrained language model to perform multimodal fusion with a language range to fuse features of related language meanings, which is fine-tuned to handle each modality as Text-Related Embeddings (TEM); ARM dynamically determines the range of attributes and selects the learnable prototypes to guide the prediction of the chosen attributes. Different policies (DyNet-Guided, BERT-Guided, and Prototype-Guided) is investigated for attribute range minimization. With the cooperation of IRC and ARM, the novel approach DRAM performs well on challenging benchmarks MEPAVE and MAE.