VL-Few: Vision Language Alignment for Multimodal Few-Shot Meta Learning

Abstract

Complex tasks in the real world involve different modal models, such as visual question answering (VQA). However, traditional multimodal learning requires a large amount of aligned data, such as image text pairs, and constructing a large amount of training data is a challenge for multimodal learning. Therefore, we propose VL-Few, which is a simple and effective method to solve the multimodal few-shot problem. VL-Few (1) proposes the modal alignment, which aligns visual features into language space through a lightweight model network and improves the multimodal understanding ability of the model; (2) adopts few-shot meta learning in the multimodal problem, which constructs a few-shot meta task pool to improve the generalization ability of the model; (3) proposes semantic alignment to enhance the semantic understanding ability of the model for the task, context, and demonstration; (4) proposes task alignment that constructs training data into the target task form and improves the task understanding ability of the model; (5) proposes generation alignment, which adopts the token-level training and multitask fusion loss to improve the generation ability of the model. Our experimental results show the effectiveness of VL-Few for multimodal few-shot problems.

1. Introduction

With the development of language models [1,2,3,4,5,6], vision models [7], and multimodal models [8,9,10,11,12,13,14], there are many applications that bring people more efficiency and convenience in their lives based on this state-of-the-art technology, such as smartphone assistants [15], chatbots [16], smart healthcare [17], smart transportation [18], smart shopping guides [19], autonomous driving [20], and so on. The success of applying this technology requires large different modal data for training like text, image, voice, video, table, graph, machine recording, and so on to train the models. However, how to make the language models understand other different modal data is a different matter. In the visual question answering (VQA) [21,22,23,24,25,26,27,28,29,30,31,32] task, the main modes of the data are vision and language because vision represents the eyesight and language represents the thinking of the brain, which are the key functions people use to interact with the environment and complete tasks. Making language models understand data from other modalities such as images, videos, speech, point clouds, etc., is a promising ability that will help language models understand real-world knowledge, complete more complex and realistic tasks, assist humans in completing tasks, and improve productivity.

The core of multimodal learning ability is modal alignment. For example, aligning visual features into language space enables language models to understand images or videos. The traditional method is to train a multimodal model from scratch, but this approach requires preparing a large amount of high-quality multimodal alignment data in advance. For example, CLIP [10] constructs a large number of image text pairs, but it cannot understand the modalities that have not been learned, thereby resulting in limited scalability. Inheriting the modal processing ability learned by the pretrained model to complete multimodal tasks is a promising research direction.

This problem is more serious in the few-shot scenario because of scarce training data. So, to alleviate the low-resource problem and improve the generalization ability of the model, meta learning [33,34,35,36,37,38] and few-shot learning [39,40,41,42,43,44,45,46,47,48,49,50,51,52] can be considered. Meta learning can construct a task pool to improve the generalization ability of the model. Few-shot learning can reduce the dependence on the data size of the model.

Therefore, to solve the multimodal few-shot problem, we propose VL-Few, a multimodal few-shot meta learning framework as shown in Figure 1. VL-Few uses a small amount of data, combines meta learning to build a task pool, and constructs a small number of training inputs with small samples. Through multimodal feature mappers, the language model can understand visual features and reduce the number of parameters for model training, thus reducing the dependence of the model on data volume and ultimately completing the VQA task. There are some differences between other research studies and ours. For model architecture, other methods require retraining of the vision model (Frozen) or only consider mapping vision image features and language question features into the same (CLIPCap [53]) or fewer (MML [54]) tokens through mappers to help the model understand vision image features. However, our method realizes that the prominent problem with understanding multimodal data is that the features are not in the same feature space, and the failure to align multimodal features effectively results in the model being unable to understand other modal data. Therefore, we consider both vision image features and the interaction features between vision images and language questions. We ultimately fuse them with other auxiliary features through a fully connected layer to form the final input feature. In addition, our experiment used less data to achieve better performance. The experiment shows that our method can improve the efficiency of extracting and fusing modal features, as well as enhance the vision understanding ability of the model language model. Our method uses less data, with only 4 or 20 pieces of data in the training set. In addition, the model has fewer parameters and only about 8.2 M trainable parameters, thus providing higher accuracy.

The main contributions can be summarized as follows:

• We propose multimodal alignment, which uses multimodal feature mappers to align the vision feature into a language feature space that can make the language model understand images;
• We adopt few-shot meta learning in the multimodal problem in which the meta task pool helps the model learn the generalization ability, and few-shot learning uses few data to train the model;
• We propose semantic alignment, which includes task instruction, data context, and demonstration context that provide the input semantic information for learning the understanding ability of the task, context, and demonstration;
• We propose task alignment, which constructs the task training data form like VQA, including the images, questions, and answers, by question generation and caption augmentation;
• We adopt token-level training in the multimodal few-shot meta problem to make the model pay attention to the answer token and adopt the multitask fusion task to help the model learn the multiple-task processing abilities.

The following sections are shown as follows. Section 2 describes the related work about multimodal learning and multimodal few-shot learning. Section 3 introduces our main methods of VL-Few. Section 4 presents the training dataset, pre-training models, and training strategies in the experiments. Section 5 makes the conclusion and some future work.

2. Related Work

There are some research studies on multimodal learning to perform the VQA task. CLIPCap [53] proposes to use CLIP-ViT [10] to extract image features, interact with images and captions through a trainable mapper network, obtain image features containing text, use them as input prefixes, input them together with the question into GPT-2 [3], and finally generate an answer to the image viewing question. This method can enable language models to understand images by training only a small number of model parameters. However, it requires a large amount of data for training, thus making it difficult to apply to small sample scenarios. BLIP-2 [23] proposes a Q-former (querying transformer) structure to map image features into language feature space. Specifically, this method extracts image features through the frozen image encoder CLIP-ViT [10], and it inputs both image features and learned queries into the Q-former. The Q-former fuses image features into learned queries, outputs learned queries, and obtains new image features through a fully connected layer. The new image features are used as prefix text and input together with the question into the language model. Finally, the language model generates the answers based on the image and question. This method obtains image features through a learnable sequence, which is then fused with the questions to obtain image features related to the problem, thus ultimately resulting in a good representation of the image features. However, this method requires a large number of data for pretraining, which is a total of 129 M, thus making it difficult to apply it to small sample scenarios. NExT-GPT [55] proposes to extract features from the data of different modalities through corresponding encoders and then map the features to the language space through the projections. The core task is understood through a language model, and the language features are mapped to different modality spaces through instruction following. Finally, the target modality data are generated through the diffusion model of the different modalities. This framework proposes a unified approach to convert data from different modalities into data from other modalities. However, this method requires a large amount of aligned data during the training phase, which poses challenges for small sample scenarios.

There are some research studies regarding multimodal few-shot learning to perform the VQA task. Frozen [56] proposes a multimodal few-shot method that freezes the language model, trains only the visual encoder, and directly inputs image features into the language model. Frozen not only reduces the number of training parameters by updating the visual encoder parameters but also enables image features to be understood by language models. In addition, Frozen provides demonstrations of the task that help the model understand the task form and the input and output for the task. However, this method requires a large amount of data to train the visual encoder, thus making it difficult to apply to small sample scenarios. MML [54] proposes the meta mapper to map the image features to language feature space. Specifically, MML extracts image features through a visual encoder and inputs them together with a visual prefix into the meta mapper. The meta mapper maps image features to visual prefixes through the network structure. Subsequently, the visual prefix is inputted into the language model along with the sentence to help the model understand the images and generate the answer. This method freezes the visual and language models and only trains the meta mapper network, which reduces the number of trainable parameters and has good scalability. However, this method still requires a certain amount of data for training.

The above research studies need a lot of data for training, even large-scale data for pretraining. However, our method only needs a few samples, like 20 or even 4, for training to perform the VQA task.

3. VL-Few

In this section, we will present our VL-Few method as follows. Section 3.1 talks about instruction prompts, and Section 3.2 introduces how to align the modal understanding ability. Section 3.3 researches few-shot meta learning in the multimodal problem. Section 3.4 talks about how to align semantic understanding ability. Section 3.5 introduces how to align task understanding ability, and Section 3.6 presents about how to align the generation ability.

3.1. Problem Definition

For multimodal learning, VQA is an answer to a question based on an image. We have a VQA dataset D as shown in Equation (1), where V is a set of Ith images, Q is a set of Ith questions, and A is a set of Ith answers.

$$\begin{array}{cc}\hfill D& \hfill =\{(V_1,Q_1,A_1),\dots ,(V_I,Q_I,A_I)\}\hfill \end{array}$$

(1)

For each VQA data value, there are several tokens corresponding to the image $V_I$, question $Q_I$, and answer $A_I$ as shown in Equations (2)–(4), where there are ith image tokens v in the Ith image V, jth question tokens q in the Ith question Q, and kth answer tokens a in the Ith answer token A.

$$\begin{array}{cc}\hfill V_I& \hfill =\{v_1^I,\dots ,v_i^I\}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill Q_I& \hfill =\{q_1^I,\dots ,q_j^I\}\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill A_I& \hfill =\{a_1^I,\dots ,a_k^I\}\hfill \end{array}$$

(4)

We propose the VL-Few, which could help a language model understand the vision information and answer the question based on the image, as shown in Equation (5):

$$\begin{array}{cc}\hfill \mathrm{VL}-\mathrm{Few}(V_I,Q_I)& \hfill =A_I\hfill \end{array}$$

(5)

For meta learning, we first sample N categories and then randomly sample K examples as the support set which is called N-way K-shot meta learning. Fewer N and more K make the task easier. Correspondingly, we randomly sample M examples as the query set. We construct a meta task pool by creating such a task T times in a set.

To perform meta learning, we first randomly sample N categories from the name list of the training set as shown in Equations (6) and (7), where name_list is all the classification of the training set, and we randomly sample n times to construct the meta category list.

$$\mathrm{meta}\_\mathrm{mecategoryta}\_\mathrm{list}=\mathrm{random}.\mathrm{sample}(\mathrm{name}\_\mathrm{list},n)$$

(6)

$$\begin{array}{cc}\hfill & \hfill =\{N_1,\dots ,N_n\}\hfill \end{array}$$

(7)

To construct the support set, we randomly sample K examples based on the N categories in the training set as shown in Equations (8) and (9), where ${(V,Q,A)}^{K_k,N_n}$ means the kth $(V,Q,A)$ in the support set based on the $N_n$ category, and it can be written as $K_k^{N_n}$.

$$\begin{array}{cc}\hfill \mathrm{support}\_\mathrm{set}=& \hfill \{{(V,Q,A)}^{K_1,N_1},\dots ,{(V,Q,A)}^{K_k,N_1},\hfill \\ \hfill & \hfill \dots ,\hfill \\ \hfill & \hfill {(V,Q,A)}^{K_1,N_n},\dots ,{(V,Q,A)}^{K_k,N_n}\}\hfill \\ \hfill =& \{K_1^{N_1},\dots ,K_k^{N_1},\hfill \\ \hfill & \dots ,\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & \hfill K_1^{N_n},\dots ,K_k^{N_n}\}\hfill \end{array}$$

(9)

Similarly, to construct the query set, we randomly sample M examples based on the N categories in the training set as shown in Equations (10) and (11), where ${(V,Q,A)}^{M_m,N_n}$ means the mth $(V,Q,A)$ in the query set based on the $N_n$ category, and it can be written as $M_m^{N_n}$.

$$\begin{array}{cc}\hfill \mathrm{query}\_\mathrm{set}=& \hfill \{{(V,Q,A)}^{M_1,N_1},\dots ,{(V,Q,A)}^{M_m,N_1},\hfill \\ \hfill & \hfill \dots ,\hfill \\ \hfill & \hfill {(V,Q,A)}^{M_1,N_n},\dots ,{(V,Q,A)}^{M_m,N_n}\}\hfill \\ \hfill =& \{M_1^{N_1},\dots ,M_m^{N_1},\hfill \\ \hfill & \dots ,\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & \hfill M_1^{N_n},\dots ,M_m^{N_n}\}\hfill \end{array}$$

(11)

To construct a meta task, we combine the support set and the query set based on each category as shown in Equations (12) and (13):

$$\begin{array}{cc}\hfill \mathrm{meta}\_\mathrm{task}=& \hfill \{{(V,Q,A)}^{K_1,N_1},\dots ,{(V,Q,A)}^{K_k,N_1},\hfill \\ \hfill & \hfill {(V,Q,A)}^{M_1,N_1},\dots ,{(V,Q,A)}^{M_m,N_1},\hfill \\ \hfill & \hfill \dots ,\hfill \\ \hfill & \hfill {(V,Q,A)}^{K_1,N_n},\dots ,{(V,Q,A)}^{K_k,N_n},\hfill \\ \hfill & \hfill {(V,Q,A)}^{M_1,N_n},\dots ,{(V,Q,A)}^{M_m,N_n}\}\hfill \\ \hfill =& \{K_1^{N_1},\dots ,K_k^{N_1},M_1^{N_1},\dots ,M_m^{N_1},\hfill \\ \hfill & \dots ,\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & \hfill K_1^{N_n},\dots ,K_k^{N_n},M_1^{N_n},\dots ,M_m^{N_n}\}\hfill \end{array}$$

(13)

Above all, we construct a task for the meta task pool using meta learning; then, we just repeat this method to obtain T tasks for the final meta task pool as shown in Equations (14) and (15):

$$\begin{array}{cc}\hfill \mathrm{meta}\_\mathrm{task}\_\mathrm{pool}& \hfill =\{{\mathrm{meta}\_\mathrm{task}}_1,\dots ,{\mathrm{meta}\_\mathrm{task}}_t\}\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & \hfill =\{T_1,\dots ,T_t\}\hfill \end{array}$$

(15)

For few-shot learning, we just use a few examples to train the model and want the performance to be similar to the model trained using more data. In our setting, few-shot learning means the training set size T × N × K is small. The details of the few-shot learning setting can be found in Section 3.3.

To make the model understand images, we propose a double mapper architecture, which comprises the vision-mapper that can map the image to the language feature space and the vision-language-mapper that can map the image with the question to the language feature space as shown in Equations (16) and (19), respectively, where $v{m}_a$ is the ath token of the vision feature $VM$, and $v{q}_b$ is the bth token of the vision–language feature $VQ$. So, the input is changed to $VL-Few(VM,VQ,Q)=A$.

$$\begin{array}{cc}\hfill \mathrm{vision}-\mathrm{mapper}\left(V\right)& \hfill =VM\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & \hfill =\{v{m}_1,\dots ,v{m}_a\}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill \mathrm{vision}-\mathrm{language}-\mathrm{mapper}(V,Q)& \hfill =VQ\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & \hfill =\{v{q}_1,\dots ,v{q}_b\}\hfill \end{array}$$

(19)

To help the model understand the task, we provide the instruction as a task instruction $T_I$ as shown in Equation (20), where $t{i}_o$ is the oth task instruction of $T_I$.

$$T_p=\{t{p}_1,\dots ,t{p}_o\}$$

(20)

To give the model more information about the context, we construct some context prompts to provide the information on the input as shown in Equations (21)–(24), where $v{p}_p$ is the pth token of the image (vision) prompt $V_P$, $vq{p}_q$ is the qth token of the image–question prompt $V{Q}_P$, $q{p}_r$ is the rth token of the question prompt $Q_P$, and $a{p}_s$ is the sth token of the answer prompt $A_P$.

$$\begin{array}{cc}\hfill V_P& \hfill =\{v{p}_1,\dots ,v{p}_p\}\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill V{Q}_P& \hfill =\{vq{p}_1,\dots ,vq{p}_q\}\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill Q_P& \hfill =\{q{p}_1,\dots ,q{p}_r\}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill A_P& \hfill =\{a{p}_1,\dots ,a{p}_s\}\hfill \end{array}$$

(24)

To demonstrate the model, we make a demonstration E of the task for the model using in-context learning as shown in Equations (25)–(27), where $v{e}_a$ is the ath token of the $V_E$, $vq{e}_b$ is the bth token of the $V{Q}_E$, $q{e}_f$ is the fth token of the $Q_E$, $a{e}_g$ is the gth token of the $A_E$, and $E_e$ is the eth token of the example E.

$$\begin{array}{cc}\hfill E=& \hfill \{V_P,V_E,V{Q}_P,V{Q}_E,Q_P,Q_E,A_P,A_E\}\hfill \\ \hfill =& \{v{p}_1,\dots ,v{p}_p,v{e}_1,\dots ,v{e}_a,\hfill \\ \hfill & vq{p}_1,\dots ,vq{p}_q,vq{e}_1,\dots ,vq{e}_b,\hfill \\ \hfill & q{p}_1,\dots ,q{p}_r,q{e}_1,\dots ,q{e}_f,\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & \hfill a{p}_1,\dots ,a{p}_s,a{e}_1,\dots ,a{e}_g\}\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill =& \hfill \{E_1,\dots ,E_e\}\hfill \end{array}$$

(27)

The multimodal fusion mapper can fuse all parts as shown in (28) and (29). Based on the Equations (1)–(29), we construct the input X as shown in Equation (30), where $x_m$ is the mth token of the input X.   

$$\begin{array}{cc}\hfill \mathrm{multimodal}\_\mathrm{fusion}\_\mathrm{mapper}(inputs)=& \hfill \{T_P,E,V{M}_P,VM,V{Q}_P,VQ,Q_P,Q,A_P\}\hfill \\ \hfill =& \{t{p}_1,\dots ,t{p}_o,E_1,\dots ,E_e,\hfill \\ \hfill & vm{p}_1,\dots ,vm{p}_p,v{m}_1,\dots ,v{m}_a,\hfill \\ \hfill & vq{p}_1,\dots ,vq{p}_q,v{q}_1,\dots ,v{q}_b,\hfill \\ \hfill & q{p}_1,\dots ,q{p}_r,q_1,\dots ,q_f,\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & \hfill a{p}_1,\dots ,a{p}_s\}\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill X=& \hfill \{x_1,\dots ,x_m\}\hfill \end{array}$$

(30)

The ground truth Y can be represented as shown in Equation (32), where $a_k$ is the kth token of the ground truth Y.

$$\begin{array}{cc}\hfill Y& \hfill =A_K\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill & \hfill =\{a_1,\dots ,a_g\}\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill & \hfill =\{y_1,\dots ,y_n\}\hfill \end{array}$$

(33)

To represent the data, we adopt the decoder to represent the data in the language feature space using token-level training as shown in Equations (34)–(36), where $W_e$ is the token embedding matrix, $W_p$ is the position embedding matrix, $h_0$ is the embedding of the input, $h_1$ is the feature of the input, and the token-level training just inputs tokens one by one to align to the output, which sets the n from one to the length of the input.

$$\begin{array}{cc}\hfill h_0^{m+n-1}& \hfill =\{x_1,\dots ,x_m,y_n,\dots ,y_{n-1}\}{W}_e+W_p,(1<=n<=k)\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill h_1^{m+n}& \hfill =transformer\_block\left(h_0^{m+n-1}\right)\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill P\left(y_n\right|x_1,\dots ,x_m,y_1,\dots ,y_{n-1})& \hfill =softmax\left(h_1^{m+n}{W}_e^T\right)\hfill \end{array}$$

(36)

We reweight three losses to train the model. First, we use the language modeling (LM) loss ${\mathcal{L}}_{lm}$ to make the model learn the generation ability as shown in Equation (37), where $y_n$ is based on the previous section $x_1,\dots ,x_m,y_1,\dots ,y_{n-1}$ of the input.

$${\mathcal{L}}_{lm}\left(y_n\right)=-\sum _{n=1}^{k}logP\left(y_n\right|x_1,\dots ,x_m,y_1,\dots ,y_{n-1})$$

(37)

Second, we adopt the SEMantic (SEM) matching loss [57], which compares the semantic similarity $s(x,y)$ of the predictive sentences and the answers and brings it closer to one as shown in Equations (38) and (39), where the similarity function is the COSine (COS) value, y means the predictions y, and $y^{\ast }$ means the targets $y^{\ast }$.

$$\begin{array}{cc}\hfill s(x,y)& \hfill ={\displaystyle \frac{x·y}{\parallel x\parallel \parallel y\parallel }},\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{sem}& \hfill =1-s(y,y^{\ast })\hfill \end{array}$$

(39)

Third, we use the Cross Entropy (CE) loss to calculate the prediction error during training as shown in Equation (40). The n is the index of the category from 1 to N. The ${\sum }_{n=1}^N{w}_{y_n}·⊮\{y_n\ne \mathrm{ignore}\_\mathrm{index}\}$ is a normalization term used to ensure that the weights of different samples are taken into account when calculating the average. The denominator of this term is the weight $w_{y_n}$ for all samples multiplied by an indicator function. If $y_n$ is not equal to ignore_index then it is 1; otherwise, it is 0. This ensures that the weights of different samples are taken into account when calculating the average. The $w_{y_n}$ is the weight of category $y_n$, which is used for weighting between different categories. This allows for the emphasis or reduction of or in certain categories of contributions in the training. The $log\frac{exp\left(x_{n,y_n}\right)}{{\sum }_{c=1}^{C}exp\left(x_{n,c}\right)}$ measures the error between the output probability distribution of the model and the actual label. The $\{y_n\ne \mathrm{ignore}\_\mathrm{index}\}$ is a means that specifies a target value like a padding token that is ignored and does not contribute to the input gradient.

$${\mathcal{L}}_{ce}=-\sum _{n=1}^N\frac{w_{y_n}log\frac{exp\left(x_{n,y_n}\right)}{{\sum }_{c=1}^{C}exp\left(x_{n,c}\right)}·⊮\{y_n\ne \mathrm{ignore}\_\mathrm{index}\}}{{\sum }_{n=1}^N{w}_{y_n}·⊮\{y_n\ne \mathrm{ignore}\_\mathrm{index}\}}$$

(40)

Based the Equations (37)–(40), the final loss is the weighted sum of the three losses as shown in Equation (41), where $w_{lm}$ is the weight of the loss ${\mathcal{L}}_{lm}$, $w_{sem}$ is the weight of the loss ${\mathcal{L}}_{sem}$, and $w_{ce}$ is the weight of the loss ${\mathcal{L}}_{ce}$.

$$\mathcal{L}=w_{lm}\ast {\mathcal{L}}_{lm}+w_{sem}\ast {\mathcal{L}}_{sem}+w_{ce}\ast {\mathcal{L}}_{ce}$$

(41)

3.2. Modal Alignment

Training a model, even if not a large model but a pretrained model, requires a large computation power and time. For multimodal tasks like VQA, some works retrain the vision encoder to make the vision feature understood by the language decoder. However, this method requires much data, computation power, and time to achieve training, which is an obstacle for low source learners.

Modal alignment aligns data from different modalities into the model space, thereby enabling the model to understand data from different modalities. Firstly, we use the vision encoder, which is the CLIP-ViT with 87 M frozen parameters to extract the vision image features, and we use the language encoder, which is the GPT-2 with 123.65M frozen parameters to extract the language question features of the model. We input the vision feature into the vision mapper, which is a linear layer with 0.59M parameters, and the vision image feature into the vision representation alignment tokens, which can represent the images. Then, we extract visual features by using the vision encoder that has frozen parameters, which is a CLIP ViT with 87 M frozen parameters, and the language encoder, which is a GPT-2 with 123.65M frozen parameters, to extract the language question features. We adopt the vision language mapper, which is a transformer layer with 7.09M parameters, to map the vision image features and language question features into the vision language representation alignment tokens to represent the vision image features based on the language questions to understand the images based on the questions. Subsequently, we adopt the multimodal fusion mapper, which is a linear layer with 0.59M parameters, to fuse all input parts, including task prompt tokens, into one whole input. Finally, we input all of them into the language decoder to generate the answers based on the images and the questions.

3.3. Multimodal Few-Shot Meta Training

The few-shot training dataset was not large enough to train a model, even a light architecture, because of the not large amount of data. Some methods like data augmentation could increase the number of data, but these just provide a little more information and cannot improve the model to understand the task.

The language models need lots of data to construct the task to help the model learn the ability such as answering the question based on the image, which is called VQA, and bounding boxes in image processing for object detection and tracking. Using a few examples to train a model to learn a new domain data or even a new task is difficult because the model can only learn from the data to learn the data knowledge and task knowledge. So, how to train a model using less data to reach a better performance is important and promising.

Single modal tasks are difficult to solve regarding complex real-world problems, so for multimodal tasks, such as VQA, enabling models to understand data from different modalities is a promising solution. Compared to other methods that require the retraining of multimodal models or a large amount of training data, our method has designed an algorithm framework that only requires a small amount of training data to train three mappers, which constitute a transformer layer with 7.09M parameters and a linear layer with 0.59M parameters. This allows us to directly reuse multiple single modal pretrained models to complete multimodal tasks. Based on our previous research on VL-Meta [58], we constructed a task pool using meta learning to improve diversity and generalization, as well as help the model understand the task. Specifically, we constructed a task that samples N categories, and each category samples K data as the training set, samples N categories, and each category samples M data as the evaluation set. We constructed T tasks for the multimodal few-shot meta task pool for training and evaluating our model. These parameters could determine the training pool size T × N × K, validation pool size T × N × M, and the total data pool size T × N × $(K+M)$.

In our experiment setting, we set the few-shot meta task pool size to $T=2$ based on the few-shot problem. The training set was only T × N × $K=2$ × 2 × $1=4$ or T × N × $K=2$ × 2 × $5=20$, and the evaluation set was T × N × $M=2$ × 2 × $5=20$.

3.4. Semantic Alignment

The model finds it difficult to distinguish the different modality inputs and cannot utilize existing knowledge to apply this information where the multiple modal data are input at once. Therefore, to enable the input to be understood by the model for different modal data parts, we used semantic alignment for different modal information.

3.4.1. Task Instruction

Models usually obtain the outputs by processing the inputs, which only include the data but not any description of the task. It can only use the input information or knowledge but can hardly learn what the task means and what needs to be done.

We added the description of the task into the inputs, which could let the model learn what the task is and how to do it. That is a kind of description we call task instruction or task prompting, which gives the model an instruction about the task to indicate how the model does the task.

We designed a task instruction that describes the task content and objectives, which can help the model understand the task information and complete the task, as shown in Listing 1.

Listing 1. VQA task instruction.

3.4.2. Data Context

Inputting all different modal data into the model is a common practice for performing multimodal tasks like VQA. However, the multimodal inputs include some different modal data, which are difficult for the model to identify. We adopted some different kinds of descriptions of the different modal data like image prompt, image–question prompt, question prompt, and answer prompt. We used the data prompts to provide the data types and separate the different modal data, which could help the model distinguish the multimodal data. To provide the model with more information, the content of the answer prompt can change depending on the different datasets to fit their characteristics, especially the answer token length.

We designed a data context, which is the description of data in the context, also known as data meta information, which is equivalent to the data of the data. The data context includes the vision prompt, vision language prompt, question prompt, and answer prompt as shown in Listing 2.

Listing 2. VQA image question and answer prompt.

3.4.3. Demonstration Context

Training, validating, and testing a model on a multimodal task like VQA requires pairs of images, questions, and answers. However, a large model or even a pretrained model learns an ability that if we give more information about the task or data, the performance could be improved. It could invoke the learned knowledge within the model to assist in completing the task. Based on this, we adopted the demonstration context to provide the model with an example of the task, like a pair of images, questions, and answers, in the VQA task to make the model understand what the task looks like and how the complete task should be.

The demonstration context is a complete task example. This task example can provide the model with specific task forms, inputs, and outputs. Specifically, we randomly constructed a demonstration context for each input so that the demonstration context for each input would be different, thereby avoiding overfitting the model to that part and improving the generalization ability of the model. We used the vision encoder to extract the vision image features. We used the language encoder to extract language question features and language answer features and used the language embedding module to extract the language question embeddings. The specific form of the demonstration context is shown in Listing 3.

Listing 3. VQA demonstration context.

3.5. Task Alignment

There may be a distance between the dataset and the task due to the inconsistency between the task form and the dataset. Therefore, we used task alignment to construct the dataset to task form.

3.5.1. Caption Alignment

To improve the model performance for few-shot learning, we found each image corresponding to five images in the training dataset, which could make more training data. Furthermore, we analyzed the captions of images and found that the captions usually described the image from different perspectives, which could provide more information about each image. So, we constructed the training data with multiple captions to help the model understand each image from different viewpoints. That could also reduce the cost of the task because, in an application scenario like the medical imaging field, it is rare for researchers to obtain a medical image of magnetic resonance imaging (MRI) about a rare disease due to cost and rarity. So, only labeling the text based on the image could reduce the cost to obtain more data, which is a useful method to solve the few-shot problem to achieve the data augment. As shown in Figure 2, there are five captions of each image in the training data. We constructed five pairs of image captions to increase the training dataset size.

The training set we constructed is from COCO2017, which is an image caption task. Other studies have not considered the complex visual information contained in an image and have not understood it from different perspectives. For this situation, we considered that one image can correspond to data from multiple captions. Therefore, we constructed our training set based on the dataset COCO2017, thus constructing multiple captions of an image into multiple image caption data.

3.5.2. Question Alignment

In the VQA task, one image corresponds to one question and one answer. However, the dataset we constructed through the caption alignment method only has the images and captions, without any questions. However, because the training data COCO2017 have no question of the images, it cannot learn the complete task form, which will make the model not do well in the actual task. To construct a VQA task, we used the question generation method, which uses the captions as the answer and generates the questions based on the answers by using ChatGPT [16]. That could help the model understand how to respond to the answers based on images and questions tasks. As shown in Figure 3, we used the caption corresponding to the image to generate multiple questions through ChatGPT.

This method can synthesize data into the form required for our task and help the model understand the image based on the question from multiple aspects, thereby improving the understanding of vision abilities.

3.6. Generation Alignment

Input and output are the learning and generation capabilities of a model, respectively. Starting from the training stage and objective function, we used a generation alignment to improve the generation capability of the model.

3.6.1. Token-Level Training

During training, the model and updating the parameters, and the language modeling (LM) predicts the last token based on the previous input tokens. But when generating the output, the model generates the response token by token, which is not the same as the training method. So, based on our previous research using VL-Meta [58], we adopted token-level training to align the training to generating as follows. The sentence here contains a token $[t1,t2,t3,t4,t5]$. According to token-level training, the content we input is $\left[t1\right]$, $[t1,t2]$, $[t1,t2,t3]$, $[t1,t2,t3,t4]$, and $[t1,t2,t3,t4,t5]$. This method aligns the training process of the model with the generation process, improves the generation ability of the model, enables the model to learn more detailed text content, and has the function of data augmentation as shown in Listing 4.

Listing 4. Token-level training example.

3.6.2. Multitask Fusion Loss

The training objective is a core part of training the model. However, only one training objective could do the specific task, which has a low generalization ability. For this reason, we adopted multitask fusion loss to make the model learn more than one objective at the same time to perform the multimodal task. We adopted LM loss to learn the generation ability to output the answers, adopted SEM loss to learn the semantic similarity ability to close the output to the answer, and adopted CE loss to learn the understanding ability to generate better responses.

3.7. Pseudocode

To make our VL-Few easier to understand, we introduce it using pseudocode as in Algorithm 1.

Algorithm 1 VL-Few pseudocode.

1: $image,caption,category\leftarrow load\_data(coco2017\_dataset)$ 2: $images,captions,categories\leftarrow caption\_alignment(images,captions)$ 3: $images,questions,answers,categories\leftarrow question\_alignment(images,captions,categories)$ 4: $support\_set,query\_set\leftarrow multimodal\_few\_shot\_meta\_task\_pool(images,questions,answers,categories,task\_size,category\_number,support\_size,query\_size)$ 5: for  $support\_set,query\_set\leftarrow task\_list$  do 6:    for $images,questions,answers\leftarrow support\_set$ do 7:      $vision\_image\_features\leftarrow vision\_encoder\left(image\right)$ 8:      $language\_question\_features\leftarrow langauge\_embedding\left(questions\right)$ 9:      $vision\_representation\_alignment\_tokens\leftarrow vision\_mapper(vision\_representation\_alignment\_tokens,vision\_image\_features)$ 10:    $vision\_language\_representation\_alignment\_tokens\leftarrow vision\_language\_mapper(vision\_language\_representation\_alignment\_tokens,vision\_image\_features,language\_question\_features)$ 11:    $language\_question\_embeddings\leftarrow language\_encoder\left(questions\right)$ 12:    $input\_tokens\leftarrow multimodal\_fusion\_mapper(task\_instruction,demonstration\_context,vision\_prompt,vision\_representation\_alignment\_tokens,vision\_language\_prompt,vision\_language\_representation\_alignment\_tokens,question\_prompt,language\_question\_embeddings,answer\_prompt)$ 13:    for $answer\_index\in range\left(len\right(answer\_tokens\left)\right)$ do 14:         $answer\_tokens\leftarrow answer\_tokens[:,:answer\_index+1]$ 15:         $support\_logits\leftarrow language\_decoder\left(token\right)$ 16:         $lm\_loss\leftarrow get\_lm\_loss(support\_logits,support\_lables)$ 17:         $sem\_loss\leftarrow get\_sem\_loss(support\_logits,support\_labels)$ 18:         $ce\_loss\leftarrow get\_ce\_loss(support\_logits,support\_labels)$ 19:         $loss\leftarrow lm\_loss\_weight\ast lm\_loss+sem\_loss\_weight\ast sem\_loss+ce\_loss\_weight\ast ce\_loss$ 20:    end for 21:  end for 22:  for $images,questions,answers\leftarrow query\_set$ do 23:    $query\_logits\leftarrow VL\_Few(images,questions)$ 24:    $query\_tokens\leftarrow get\_pred\_token(query\_logits)$ 25:    $query\_acc\leftarrow torch.eq(query\_tokens,answers)$ 26:  end for 27: end for 28: $guided\_vqa\_list\leftarrow load\_test\_data(dataset=guided\_vqa)$ 29: for $images,questions,answers\leftarrow test\_vqa\_list$ do 30:    $test\_logits\leftarrow VL\_Meta(images,questions)$ 31:    $test\_tokens\leftarrow get\_pred\_token(test\_logits)$ 32:    $test\_acc\leftarrow torch.eq(test\_tokens,answers)$ 33: end for

4. Multimodal Evaluation

4.1. Training Dataset

There were two datasets in our experiments, which are COCO2017 [59] for training and Visual Genome [60] for testing.

4.1.1. COCO2017

The dataset COCO2017 [59] is an upgraded version based on COCO2014 [61]. The total amount of the two datasets is the same, but the partition ratio is different as shown in Listing 5.

Listing 5. COCO2017 dataset example.

It includes images of 91 object types and 2.5 million labeled instances from 328 thousand images, which involve complex daily scenes, including common objects in the natural environment. It aims to understand visual scenes involving many tasks like recognizing what objects are present, localizing the objects, and providing a semantic description of the scene. Each image includes the object classifications that could construct the N-way K-shot meta task pool based on the N object categories present in the images. But, this dataset is mainly for the image caption task, which means it has no questions for images. So, we proposed to construct the VQA task by generating questions for the image captions.

4.1.2. Visual Genome

The model was trained on a dataset; however, it can also be used in an out-of-domain dataset. To test the generalization ability of the model, we adopted an out-of-the-training dataset domain named Visual Genome [60], which is a guided VQA dataset. It consists of 101174 images from MSCOCO [61] with 1.7 million question–answer pairs and an average of 17 questions per image as shown in Listing 6.

Listing 6. Visual Genome dataset example.

We used this dataset as the test set to evaluate the generalization ability.

4.2. Crossdomain Evaluation

The evaluation of the same domain of the training data is a general method used to validate model performance. However, it cannot show the ability of the model on a different domain dataset to show the generalization and how well it is in a new domain. We adopted the crossdomain evaluation, which validates the model on a different dataset from the training data to verify the generalization of the model.

4.3. Implementation Details

We proposed the VL-Few model structure as shown in Figure 1. Specifically, we adopted CLIP-ViT with 87.46M frozen parameters as the vision encoder to obtain the vision features from images and employed GPT-2 with 123.65M frozen parameters as the language encoder to obtain the language question features from questions. Then, vision mapper, which is a linear layer with 0.59M parameters, mapped the vision features to the vision representation alignment tokens. The vision-language mapper, which is a transformer layer with 7.09M parameters, mapped the vision and language features to the vision-language representation alignment tokens. These two different tokens contain their input semantics and could represent the original data. Then, we input all the parts into the multimodal fusion mapper, which is a trainable linear layer with 0.59M parameters, to fuse all the inputs. Finally, we input them into the language decoder, which is also the frozen GPT-2, to obtain the answers. The few-shot meta task pool size was 2, which means we only sampled two tasks for training. The lengths of the vision mapper prompt and vision-language mapper were 10. The learning rate was $3\times {10}^{-5}$.

4.4. Evaluation Metric

To compare with other research, we adopted the accuracy as our evaluation metric, as shown in (42).

$$\begin{array}{c}\hfill Accuracy=\sum _1^n\frac{torch.eq(pred\_toke{n}_n,answer\_toke{n}_n).sum\left(\right).item\left(\right)}{answer\_toke{n}_n.numel\left(\right)}\end{array}$$

(42)

Our evaluation metric is accuracy, which is a common metric used to evaluate the performance of the model.

4.5. Results and Discussion

The core problem is that the language model cannot see the vision information. For research on the multimodal few-shot problem, we compared three related methods by using the crossdomain dataset Real-Fast VQA based on 2-way 1-shot or 5-shots to evaluate the performance of our method, as shown in

Table 1. VL-Few comparison on 2-way crossdomain Real-Fast VQA task in accuracy. The × means the methods without meta learning and the ✓ means it with the meta learning. The bold font accuracies are the best with meta in their results. The underlined font accuracies are the best without meta in their results.

		Accuracy(%)
Methods	Meta	1-Shot	5-Shots
Frozen [56]	×	7.8	10.5
MML [54]	×	6.9	10.7
	✓	8.5	13
VL-Meta [58]	×	7.952	10.864
	✓	10.936	14.456
VL-Few (Ours)	×	14.816	15.328
	✓	17.736	18.664

. Note that there are few research studies on this problem; thus, the compared results are not many. With or without $Meta$ means whether or not to use the meta learning method. The evaluation metric is accuracy (%), which can evaluate the understanding of visual information of the model and complete VQA tasks based on this. Frozen [56] freezes the language decoder and trains the vision encoder, which is also the baseline of this problem. MML [54] is a method that uses a mapper to convert the vision feature into language feature space, which trains by using the meta learning to construct a new training dataset. Our method VL-Few proposes to use the learnable vision prompt and vision-language prompt to carry out the vision information to make the model learn images, which could help the model understand the images that are the different modal data. Our training set sizes were T × N × $K=4$ × 2 × $1=8$ or T × N × $K=4$ × 2 × $5=40$, which are small samples for training. VL-Few reached the best accuracy in the 1-shot and 5-shots, which shows the feasibility and effectiveness of our method. More samples could provide more data and task knowledge, which could improve the model performance. Meta could further improve the accuracy of the model. We observed that our VL-Few reached the best accuracy in the 1-shot and 5-shots, even without multimodal few-shot meta training. This shows that our method could improve the modal understanding ability of the model by using multimodal alignment, and multimodal few-shot meta training could further improve the performance of the model. Specifically, the modal alignment method aligns the vision feature into the language space by mapping the vision feature into the language feature space, which improves the modal understanding, and multimodal few-shot meta training reduces the data dependence of the model by constructing the training set and training input, which improves its generalization ability.

4.6. Ablation Study

To evaluate the contributions of the meta learning and the out-of-domain dataset, we designed an ablation experiment as shown in

Table 2. Ablation experiment of comparison of meta and crossdomain of VL-Few on 2-way Real-Fast VQA task in accuracy. The × means the methods without meta learning and the ✓ means it with the meta learning. The bold font is the best accuracy in the results.

			Accuracy (%)
Method	Multimodal Few-Shot Meta Training	Crossdomain	1-Shot	5-Shots
VL-Few	×	×	6.000	7.100
	×	✓	11.160	15.328
	✓	×	7.300	8.450
	✓	✓	14.824	18.664

. Our method VL-Few had four experiment settings on whether or not to use the meta learning method and whether or not to use the crossdomain dataset, which could evaluate the contributions for each setting based on 2-way 1-shot and 5-shots. Whether or not to use the crossdomain means the test set is from MSCOCO2017 or Guided VQA. The first two experiments show that the crossdomain learned the ability to generalize.

Only using the meta task pool method cannot improve the model performance; we analyzed that, although there were many meta tasks in the pool, we sampled just a few tasks to train the model. This method did not receive significant benefits due to the lack of scale effect.

4.7. Qualitative Analysis

To evaluate the contributions of each part and module of VL-Few, we designed another ablation experiment as shown in

Table 3. Ablation experiment of comparison of VL-Few on 2-way Real-Fast VQA task in accuracy. The bold font is the best accuracy in the results.

No.	Methods	Accuracy (%)
		1-Shot	5-Shots
1	multimodal few-shot meta training (w/o)	11.160	15.328
2	crossdomain (w/o)	7.300	8.450
3	task instruction (w/o)	13.064	16.752
4	data context (w/o)	11.904	15.408
5	demonstration context (w/o)	12.104	16.288
6	caption augmentation (w/o)	12.112	16.032
7	question generation (w/o)	12.096	15.928
8	token-level training (w/o)	11.808	14.088
9	multitask fusion loss (w/o)	13.872	14.176
10	VL-Few	14.824	18.664

. We compared each method in our experiment based on 2-way 1-shot and 5-shots.

The task instruction includes the task description that could provide the context for the input of the model. The data context provides differentiation ability for input sequences, which can help the model distinguish different modalities and parts, thereby helping the model understand the input sequence. The demonstration context provides task examples to enable the model to mimic the form of tasks and output expected content. These two results show that semantic alignment can improve the semantic understanding of the model.

The question generation can provide some general questions for datasets that are without questions, which can help the model improve its understanding of images and improve its testing performance. The caption augmentation can construct more data using different captions for one image, which can help alleviate the small sample problem. These two results show that task alignment can help improve the task understanding ability of the model.

The token-level training can make the model pay attention to each answer token, thus giving more opportunities to the model to learn from the training set. The multitask fusion loss makes the model achieve different objects, which helps the model learn different abilities from different perspectives of the dataset to enhance the overall abilities. These two results show that generation alignment helps the generation ability of the model.

5. Conclusions

The few-shot problem is hard to solve because of limited training data, which becomes even more difficult in multimodal scenarios. Constructing multimodal data requires manual alignment, and increasing the amount of data also requires time and cost. Our research provides an effective solution: V-Few. This method achieves the modeling of multimodal tasks with limited data. The main contributions of this article are as follows. A new learnable framework has been proposed, which includes the vision mapper, vision-language mapper, and fusion module. This mechanism could reduce the trainable parameters number and make the language model understand images. The task instruction improves the task content and objectives understanding ability of the model. The data context enhances the interpretability of data and provides modal differentiation prompts, thereby improving the data understanding ability of the model. The demonstration context improves the specific forms of task understanding ability of the model. The question generation method provides the general questions for the training data, which improves the transfer ability in the target task form of the model. The answer augmentation method provides multiple answers per image, which helps the model understand the content of each image and improves the data and task understanding ability of the model. Token-level training enables the model to focus on each token of the answer, thereby improving the answer generation ability of the model. Multitask fusion loss can learn and solve multiple tasks simultaneously, which can improve the overall performance of the model. Our method can effectively alleviate the multimodal few-shot problem, and the good experimental results demonstrate the effectiveness of the model.

Our method VL-Few could help the blind understand real-world vision information to help them perceive the environment. It can help the blind understand real-world vision information, thereby enabling them to navigate and interact with their surroundings with heightened awareness. This technology bridges the gap between the sighted and the visually impaired, thereby fostering independence and inclusivity. We have considered designing the VL-Few algorithm as glasses that perceive the surrounding environment, and visually impaired individuals can perceive the surrounding environment through voice interaction by wearing glasses equipped with our algorithm. When visually impaired people want to drink water, then they need to know the position of the cup. Our method can take photos of the surrounding environment, recognize it as text with the previously recorded question, and input it into the algorithm together. The algorithm generates answers and converts the text into speech through voice broadcasting to help visually impaired people understand the position of the cup. At present, the accuracy of this method is still difficult to achieve in a practical application. We could try to improve the ability of the model by using more data or designing more efficient algorithms.

More data means more knowledge, which can help the model learn tasks and achieve performance improvement. We consider using more data and involving more efficient algorithms to improve the task capability of the model. In the future, we can further explore some directions, such as using more meta task pool samples to analyze the impact of different sizes on model performance. In addition, we can generate more specific questions to explore the impact of fine-grained questions on model performance, fine-tuning the encoder and decoder to explore the impact of task data on the model, and using large language models to provide stronger language ability.