Medical QA Oriented Multi-Task Learning Model for Question Intent Classification and Named Entity Recognition

Abstract

Intent classification and named entity recognition of medical questions are two key subtasks of the natural language understanding module in the question answering system. Most existing methods usually treat medical queries intent classification and named entity recognition as two separate tasks, ignoring the close relationship between the two tasks. In order to optimize the effect of medical queries intent classification and named entity recognition tasks, a multi-task learning model based on ALBERT-BILSTM is proposed for intent classification and named entity recognition of Chinese online medical questions. The multi-task learning model in this paper makes use of encoder parameter sharing, which enables the model’s underlying network to take into account both named entity recognition and intent classification features. The model learns the shared information between the two tasks while maintaining its unique characteristics during the decoding phase. The ALBERT pre-training language model is used to obtain word vectors containing semantic information and the bidirectional LSTM network is used for training. A comparative experiment of different models was conducted on Chinese medical questions dataset. Experimental results show that the proposed multi-task learning method outperforms the benchmark method in terms of precision, recall and F₁ value. Compared with the single-task model, the generalization ability of the model has been improved.

1. Introduction

Along with the rapid development of online medical service technology, people can ask questions and receive answers online through health service websites. Medical question and answer systems require the ability to build a medical knowledge base and apply natural language understanding techniques to extract structured information from users’ questions, and automatically generate answers to them. For this type of health service, the accuracy of the generated answers depends not only on the quality of the knowledge base, but also on the accuracy of the user’s question understanding.

Intent classification and named entity recognition are two subtasks of natural language understanding. Most existing medical natural language processing studies in text classification and named entity recognition are usually performed independently. The purpose of the intent classification task is to first identify possible intent classes in a given domain and then classify sentences to specific intent classes based on contextual information in the text. Named entity recognition aims at extracting medical entities from the text and predicting the different kinds of entities. Both of these tasks can help a medical question and answer system to correctly provide the services required by the user. For example, suppose a user asks the question “I have kidney stones, what should I do?”. Based on the intent analysis, the user is seeking a treatment and based on named entity recognition, we know that the question contains the disease term “kidney stones”. In this case, we can search our knowledge base and return an answer about the treatment for kidney stones. In some tasks where the amount of data is small, training with multi-task learning allows the network to rely on tasks where data is easily available to learn how to extract the underlying information or learn to extract some features common to both tasks, and to obtain better generalization capabilities. Most current medical questions intent classification and named entity recognition methods use single-task learning strategies that ignore the close relationship between these two tasks. To address this problem, this paper proposes a multi-task learning model combining ALBERT pre-training model and BILSTM, where named entity recognition task and intent classification task share ALBERT word embedding layer and BILSTM layer. They facilitate each other’s learning to obtain rich semantic and associative information at word and sentence level by sharing the underlying parameters, Experiments were conducted on the Chinese healthcare questions dataset. In summary, our main contributions can be summarized as follows:

• A multi-task learning model based on ALBERT-BILSTM is proposed for intent classification and named entity recognition of Chinese online medical questions.
• The experimental results demonstrate that the proposed method in this paper outperforms the benchmark methods and improves the model generalization ability compared to the single-task model.

2. Related Work

2.1. Medical Named Entity Recognition

Medical named entity recognition is used to recognize blocks of words in a text that are related to specific entities, such as symptoms, drugs, and treatments. Rule-based approaches play an important role in named entity recognition. Gerner et al. [1] used a dictionary-based approach to identify species names in biomedical literature. Fukuda et al. [2] proposed a rule-based approach to extract names of substances such as proteins from biological documents. However, these methods require manual rules and, thus, lack generality. Researchers have also attempted to recognize entities from unstructured data using machine learning methods. He et al. [3] proposed a conditional random field (CRF) based approach to recognize drug names in biomedical texts. Machine learning methods rely on manual feature design, which is both time consuming and laborious. In recent years, deep learning methods that can improve the performance of named entity recognition without relying on feature engineering have received increasing attention. For example, Chen et al. [4] used a Bi-directional Long Short-Term Memory (BILSTM) model for named entity recognition of adverse drug event reports in China. The BILSTM-CRF based model was validated by many works as a very effective method to solve named entity recognition [5,6,7]. Current research on named entity recognition of medical text is mainly distributed in medical literature, electronic medical records and less on medical question and answer text. In recent years, some researchers have also started to focus on the research on medical question and answer text. Su et al. [8] used CRF to conduct named entity recognition research on their own constructed dataset, and the extracted entities included diseases, drugs, symptoms, treatments and tests. Qin et al. [9] proposed a BERT-BiGRU-CRF neural network model to recognize named entities in electronic medical records of cerebrovascular diseases in order to address the issues associated with neglecting context information in electronic CVD medical entity recognition. In order to win the CHIP2018 competition, Ji et al. [10] proposed a cooperation approach based on multiple neural network models for Chinese medical named entity recognition.

2.2. Intent Classification

Deep learning methods are widely used for text classification [11]. Ravuri et al. [12] proposed applying LSTM models to the intent classification problem. Zhang et al. [13] used convolutional neural networks (CNN) to analyze online cancer community discussion topics and the CNN outperformed support vector machine (SVM) models and LDA topic models. In addition, researchers have used other knowledge and rule-based functions to improve the classification accuracy of CNNs on clinical texts [14]. Jang et al. [15] proposed an attention-based Bi-LSTM+CNN hybrid model that capitalize on the advantages of LSTM and CNN with an additional attention mechanism. Deep learning methods are still plagued by insufficient training data. Zhang et al. [16] proposed a Capsule network model for electronic medical record classification. Recently, pre-trained models generating representations of words with a priori semantic knowledge in large-scale unlabeled corpora have achieved state-of-the-art results in various natural language processing tasks [17]. Devlin et al. [18] published in 2018 the BERT (Bidirectional Encoder Representations from Transformers) pre-trained language model. This model efficiently classifies TCM records and can obtain the best results by training the corpus with a bidirectional Transformer encoder to obtain text representations [19,20]. Various pre-training models have emerged after BERT. Lan et al. [21] proposed the ALBERT (A Lite BERT) model, which is a lightweight pre-trained language model based on the BERT model and use a bidirectional Transformer to obtain feature representations of the text, but ALBERT greatly reduces the parameters in the model and achieves the best results in several NLP tasks. Zhang et al. [22] presented a short text classification algorithm for Chinese clinical medicine combining ALBERT pre-training model and graph attention network.

2.3. Multi-Task Learning

Multi-task Learning (MTL) refers to learning multiple tasks simultaneously, and neural networks generally allow the underlying network to simultaneously take into account different tasks by sharing weights to extract more representative low-level features, because the purpose of single-task learning often focuses only on local information, limiting the generalizability of the model. In contrast, multi-task learning can exploit the potential information between tasks to extract features that are common across tasks and improve model performance [23]. Two common models of multitask learning exist in deep learning: parameter hard sharing mechanisms and parameter soft sharing mechanisms [24]. The parameter hard sharing mechanism usually works by sharing the hidden layer among all tasks and keeping the task-specific output layer. However, under the parameter soft sharing mechanism, each task has its own model parameters, and the similarity of model parameters is ensured by regularizing the parameter distances. Deep neural network-based multitask learning in natural language processing has been shown to be effective [25,26], especially in the presence of insufficient training data. Multi-task learning strategies have been used to improve the performance of named entity recognition for medical text. Researchers [27] used a multi-task bidirectional long short-term memory network (BILSTM)-based model for the named entity recognition task and lexical tagging task to improve named entity recognition in Chinese electronic medical records. Researchers also proposed a multitask learning framework for named entity recognition (NER) and named entity normalization (NEN) [28], which greatly improved the performance of NER and NEN. These studies used a multi-task learning model to solve two-sequence tagging problems. Peng et al. [29] presented a multi-task learning model with multiple decoders on varieties of biomedical and clinical natural language processing tasks such as named entity recognition, relation extraction, text similarity and text inference.

3. Methodology and Model

The overall structure of the proposed ALBERT-BILSTM-based multi-task learning model is shown in Figure 1, which mainly consists of an ALBERT word embedding layer, a BILSTM network layer and a task-specific decoding layer. The model adopts a hard sharing model of parameters, where the named entity recognition task and the intent classification task share the ALBERT word embedding layer and the BILSTM layer. During the training period, the model alternates between intention classification and named entity recognition tasks. The model directly uses online medical question text as input, and first the word embedding layer converts the input sequence into a corresponding word vector sequence using the ALBERT pre-training model. The word vector sequence is input to the BILSTM layer for further semantic encoding using the forward and backward networks to obtain the final representation of the utterance. Finally, for the intent classification task, the hidden states generated by the BILSTM layer are fed to the fully-connected and softmax classifier to obtain the probabilities of the input text in each intent category, thus achieving the final intent classification. For the named entity recognition task, CRF obtains the best sequence labels for the full sentences by decoding word labels with semantic features. The gradient information from both the intent recognition and named entity recognition tasks is passed backwards to the shared encoder part of ALBERT and BILSTM and the model parameters are updated to obtain a more representative underlying representation.

3.1. ALBERT Pre-Trained Language Model

ALBERT pre-trained language model is based on a large corpus and uses unsupervised learning methods to learn feature representations for words, which can characterize the polysemy of words and enhance the semantic representation of sentences. ALBERT is a lightweight language model based on the BERT model. To reduce the parameters of the BERT pre-trained model and enhance semantic understanding, the ALBERT model effectively reduces the BERT model parameters by two methods: parameter sharing between layers and parameter factorization of the embedded layers, which can reduce the memory overhead during training and improve the training speed of the model. To compensate for the shortcomings of the NSP task in BERT proposed by Yang et al. [30], ALBERT utilizes SOP (Sentence Order Prediction) instead of NSP (Next Sentence Prediction) task, through which the SOP task model can learn more semantic relations between sentences.

ALBERT model uses bidirectional Transformer encoder (Trm) to obtain the feature representation of text E₁, E_2, …E_N, which represents each character in the sequence, and after the training of multi-layer bidirectional Transformer encoder, finally we obtain the text feature vector T₁, T₂, …T_N.

Transformer is an Encoder-Decoder [31] structure based on the Self-Attention mechanism. ALBERT is composed by stacking multiple encoder layers, and the model structure of this part is shown in Figure 2. Each encoder layer includes a Self-Attention layer and a feedforward neural network, and with the help of the Self-Attention mechanism enables the model to allow the current node to not only focus on the current word, but to perform relational computation from the global view to obtain the semantics of the context. In addition, to address the degradation problem in deep learning, the Transformer encoder unit contains an Add&Norm layer for each subnetwork layer, which adds and normalizes the input and output of this layer [32] and uses a residual connection between two subnetwork layers [33].

The most important module of the Transformer encoder is the multi-headed Self-Attention mechanism, which is computed as follows.

$$\begin{array}{l}\mathrm{MultiHead}(\mathit{Q},\mathit{K},\mathit{V})=\\ \hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Concate}(hea{d}_1,hea{d}_2,\dots ,hea{d}_h){\mathit{W}}^O\end{array}$$

(1)

$$hea{d}_i=\mathrm{Attention}(\mathit{Q}{\mathit{W}}_i^{\mathit{Q}},\mathit{K}{\mathit{W}}_i^{\mathit{K}},\mathit{V}{\mathit{W}}_i^{\mathit{V}})$$

(2)

$$\mathrm{Attention}(\mathit{Q},\mathit{K},\mathit{V})=\mathrm{Softmax}\left(\frac{\mathit{Q}{\mathit{K}}^T}{\sqrt{d_k}}\right)\mathit{V}$$

(3)

where W^o is the matrix of additional weights, which can compress the dimension of the spliced matrix to the length of the sequence. Q, K, V denotes the query, key and value vectors of each word in the input sequence and $W_i^Q,W_i^K,W_i^V$ are the weights of Q, K, V respectively matrix. d_k denotes the dimension of query and key vectors of each word.

3.2. BILSTM Module

Long short-term memory network (LSTM) [34] is a special type of recurrent neural network that captures the contextual order information of sequences to solve the long dependency problem. LSTM is a variant of RNN that introduces some gate structures to solve the RNN gradient explosion and gradient disappearance problems. The LSTM cell state structure is shown in Figure 3.

LSTM consists of four main components: the storage cell c_t, the input gate i_t, the output gate o_t, and the forget gate f_t. In the LSTM, a gate is a method of selectively passing information through, using specially designed “gates” to introduce or remove information from the cell state c_t. The LSTM computes an output vector based on the current input and the output of the previous cell, which is then used as the input to the next cell. The calculation formula is as follows.

$$i_t=\sigma (x_t·w_{xh}^i+h_{t-1}·w_{h{h}^{\prime }}^i+b_h^i)$$

(4)

$$f_t=\sigma (x_t·w_{xh}^f+h_{t-1}·w_{h{h}^{\prime }}^f+b_h^f)$$

(5)

$$o_t=\sigma (x_t·w_{xh}^o+h_{t-1}·w_{h{h}^{\prime }}^o+b_h^o)$$

(6)

$${\tilde{c}}_t=\tanh (x_t·w_{xh}^c+h_{t-1}·w_{h{h}^{\prime }}^c+b_h^c)$$

(7)

$$c_t=i_t\otimes {\tilde{c}}_t+f_t\otimes c_{t-1}$$

(8)

$$h_t=o_t\otimes \tanh (c_t)$$

(9)

where σ denotes the sigmoid activation function, x_t is the cell input at time t and w, b denotes the weight matrix and bias vector of the input gate, forget gate, and output gate. ${\tilde{c}}_t$ is the intermediate state obtained from the current input. tanh is the hyperbolic tangent function. c_t represents the state at moment t, and h_t is the output at moment t.

LSTM can only encode historical information and ignore future contextual information. In this paper, a bidirectional LSTM (BILSTM) network consisting of forward LSTM and inverse LSTM is used. BILSTM obtains the final hidden layer representation by splicing two different hidden layer representations obtained by sequential and inverse order computation.

3.3. Decoding Unit

(1) Fully connected and normalized exponential function

The fully connected layer reduces the dimensionality of the output vector of BILSTM to the same dimensionality as the total number of medical entity labels. Assuming that S represents the total number of labels, the output of the fully connected layer at the i th position is shown in Equation (10).

$${\mathit{m}}_i=\sigma ({\mathit{W}}_m{\mathit{h}}_i+{\mathit{b}}_m)$$

(10)

where $W_m\in R^{s\ast 2h},b_m\in R^s$.

In multi-classification problems, a normalized exponential function (Softmax function) is usually used as the activation function of the output layer of the network. Softmax function can perform a normalization operation on the output values, transforming all output values into probability values between (0, 1), and all probability values add up to 1. This probability value represents the probability that a word belongs to a certain label. The prediction result selects the tag with the highest probability value.

$${\widehat{y}}_{i,j}=\frac{e^{c_{i,j}}}{{\displaystyle \sum _{j=1}^s{e}^{c_{i,j}}}}$$

(11)

In the training process, the optimal model parameters are learned by minimizing the cross-entropy loss function strategy. The loss function is shown in Equation (12), so that y_i,j denotes the true probability that the ith word belongs to the jth label, and takes the value of 0 or 1. ${\widehat{y}}_{i,j}$ is the model prediction value derived from Equation (11).

$$loss={\displaystyle \sum _{i=1}^n\{-{\displaystyle \sum _{j=1}^s{y}_{i,j}}}log{\widehat{y}}_{i,j}\}$$

(12)

(2) Conditional random field

There are interdependencies between the labels of named entities of medical texts. For example, the next label of the label “I-disease” will not be “I-drug”. It is a widespread practice to use conditional random field (CRF) optimization to predict the sequence of labels, where the CRF layer takes the sequence $x=(x_1,x_2,\cdots ,x_n)$ as input and predicts the most likely sequence of labels $y=(y_1,y_2,\cdots ,y_n)$. Given a training set D, all CRF layer parameters (denoted as θ) are estimated by maximizing log-likelihood as shown in Equation (13).

$$L(\theta )={\displaystyle \sum _{(s,y)\in D}^{}\log p(y|x,\theta )}$$

(13)

where y is the corresponding label sequence of sentence s and p is the conditional probability of y given s and θ. Assuming that $S_{\theta }(x,y)$ is the score of the sentence label sequence y, the conditional probability p can be calculated using the normalization of $S_{\theta }(x,y)$. To take advantage of the dependencies between adjacent labels, the model combines the transfer probability matrix T and the emission probability matrix E to calculate the score of the label sequence $S_{\theta }(x,y)$, as shown in Equation (14).

$$S_{\theta }(x,y)={\displaystyle \sum _{t=1}^n(E_{y_t,t}+T_{y_{t-1},y_t})}$$

(14)

where $E_{y_t,t}$ is the probability of word x_t with label y_t and $T_{y_{t-1},y_t}$ is the probability of word x_t−1 with label y_t−1 followed by word x_t with label y_t. We can find the best sequence of labels for the input sentences by maximizing the log-likelihood over all training sets D by dynamic programming and maximizing the score by using the Viterbi algorithm.

3.4. Multi-Task Learning Step

In the multi-task learning process, both the intent classification and named entity recognition tasks have their own separate training sets D_C and D_N. In order to allow both tasks to learn simultaneously, the model alternates between the intent classification and named entity recognition tasks during the training period to “approximate” simultaneous learning. The loss and optimization functions for the entity recognition and intent classification tasks are independent during alternate learning. The intent classification task uses the standard cross-entropy loss as the loss function, and multi-task learning is performed by alternating calls to each task optimizer with the two tasks using the Adam optimization function to learn the parameters of the multi-task model. This means that we can continuously transfer some information from each task to the other task, which is achieved through a shared layer. At each iteration, a task is randomly selected and then some random training samples are chosen from this task to compute the gradient and update the parameters. The exact procedure of the alternating training phase in multitask learning is shown in Algorithm 1.

Algorithm 1 Multi-task learning training process

Input: two task datasets DC and DN

Batch size K for each task

Maximum number of iterations T, learning rates α and β

Random initialization parameter θ0

for t = 1 ⋯ T do

/*Prepare the data for both tasks*/

Randomly divide DC and DN into small batch sets

BC = {JC,1, …, JC,n}

BN = {JN,1, …, JN,m}

end

Merge all small batch samples B′ = BC ∪ BN

Random sorting B′

for each J $\in$ B′ do

Calculate the loss L(θ) on the small batch sample

/* calculate only the loss of J on the corresponding task */

Update the parameters: ${\theta }_t\leftarrow {\theta }_{t-1}-\alpha \cdot {\nabla }_{\theta }L(\theta )$

end

end

4. Experiments and Results Analysis

4.1. Dataset

The experimental data used in this paper were derived from the Chinese medical question and answer dataset (cMedQA2) [35] and the Chinese medical question sentence intent classification dataset (CMID) [36]. To ensure that each question includes only a single intention, we eliminate samples with unclear intentions and multiple intentions, and select 10,496 user questions from them for the experiments, using the following seven types of intentions to label these questions: definition, prevention, symptom, cause, treatment method, indication and complication. The statistical information of each label is listed in

Table 1. Statistics of sample number of intent query.

Category	Quantity
Definition	963
Symptom	1215
Causes	2177
Prevention	1205
Treatment	2506
Indications	1572
Complications	879

.

The named entity recognition dataset was also derived from questions used for intent classification labeling, for a total of 5856 medical questions. The named entity recognition task uses three types of entities to label interrogative sentences: disease, symptom, and medicine. Chinese medical questions were not subsumed due to the variability of online health texts, sometimes with incorrect expressions, for which current Chinese subsumption tools are unable to meet the demand. Using the BIO annotation model, the interrogative text is manually annotated with the format B-X, I-X and O. B denotes the beginning part of the entity, I represents the Chinese character inside the entity, and O denotes the non-entity. x represents the category of the named entity, which are Disease, Symptom and Drug, representing disease, symptom and drug, respectively. The task is labeled with a total of 7 tags. After obtaining the intent classification and named entity recognition dataset with annotations, we selected 800 user questions as the test set for the intent classification and the named entity recognition task.

4.2. Evaluation Metrics

To evaluate the effectiveness of the model, Precision P, Recall R and F₁ are used to evaluate the effectiveness of the model. The precision rate P is the proportion of correctly predicted samples among all samples with positive predictions; the recall rate R is the proportion of correctly predicted samples among all samples with true positive predictions; and F₁ is the summed mean of the precision rate and recall rate. The calculation formula is as follows.

$$P=TP/(TP+FP)$$

(15)

$$R=TP/(TP+FN)$$

(16)

$$F_1=2PR/(P+R)$$

(17)

where TP is the true case: positive samples predicted as positive by the model; FP indicates the false positive case: negative samples predicted as positive by the model; TN is the true negative case: negative samples predicted as negative by the model; FN represents the false negative case: positive samples predicted as negative by the model.

4.3. Experimental Environment and Parameter Settings

The experiments in this paper are based on Python 3.7 and implemented using the Pytorch deep learning framework. The CPU is Intel(R) Xeon(R) E5-2678 v3 @ 2.50 GHz. GPU graphics card is GeForce RTX 3090 and the running memory is 24 G.

The parameters of ALBERT-BILSTM multitask model mainly include the parameters in ALBERT and BILSTM, and the values of the variable parameters are changed sequentially to obtain the optimal parameters of the model while fixing the other parameters. In this experiment, the Chinese pre-training model “ALBERT-Base” released by Google is used for ALBERT-Base, which has 12 layers. The hidden layer is 768 dimensions and the 12-head model is used with 110M parameters in total. The number of nodes in the hidden layer is 128 and the number of layers in the BILSTM model is 1. Activation function of the model is ReLU and the ratio of Dropout is set to 0.1 in the training phase. Batch size of the ALBERT-BILSTM multi-task model is set to 32 and the number of iterations is set to 30. The learning rate of the intention classification task is set to 0.001 and the learning rate of the named entity recognition task is set to 0.002. Adam (Adaptive Moment Estimation) optimization algorithm is used to learn the parameters of the multi-task model.

4.4. Benchmark

To verify the effectiveness of the proposed ALBERT-BILSTM-based multitask learning model for the medical questions intent classification and named entity recognition, several models with good results on text classification and named entity recognition were selected as comparisons.

In the comparison experiments, the TextCNN, BILSTM and BILSTM-CRF models are combined with word2vec word granularity word vectors and the word2vec parameters are set as follows: the dimension of the word vector is set to 200, the window size is 5 and the training epoch is 20.

4.5. Experimental Results and Analysis

We trained our models using the named entity recognition and the intent classification training dataset and tested the performance on the named entity recognition and the intent classification task, respectively.

For the named entity recognition task, the model performance was evaluated using the micro-averaged F₁ values, recall and accuracy. We chose BILSTM [4] and BILSTM-CRF [5] as benchmark methods. We also compared with the ALBERT model. In addition, to evaluate whether a multi-task learning strategy can improve the performance, we tested the performance of the model using a single-task learning model ALBERT-BILSTM-CRF. The experimental results are shown in

Table 2. Results of NER on different models.

Method	P	R	F1
BILSTM	0.7515	0.7432	0.7473
BILSTM-CRF	0.7578	0.7641	0.7606
ALBERT-CRF	0.7869	0.7954	0.7911
ALBERT-BILSTM-CRF	0.7926	0.8014	0.7970
MTL-ALBERT-BILSTM	0.8103	0.8036	0.8069

.

From the experimental results in Table 2, we can find that our method outperforms the baseline method in terms of F₁ value, precision and recall metrics. We found that the BILSTM-CRF model has a higher F₁ value compared with the BLS TM model and the BILSTM model has a stronger advantage for the sequence annotation task due to its memory function. The CRF model makes full use of the relationship of adjacent tags based on the BILSTM model to optimize the optimal splicing of the whole sequence. ALBERT-BILSTM-CRF model has a higher F₁ value compared with the BILSTM-CRF model and ALBERT-CRF model F₁ values are 3.63% and 0.59% higher, which can better express the semantic information of words because the word vectors generated by the ALBERT pre-trained language model are contextually relevant. In this paper, the multi-task learning model is also trained on the ALBERT embedding layer and BILSTM encoding layer during the intent classification training and our method can effectively improve the named entity recognition effect compared with the ALBERT-BILSTM-CRF model without multi-task learning strategy.

The enhancement of named entity recognition effect can be explained by the intent classification task providing additional training data and the multi-task learning is equivalent to an implicit data enhancement. By sharing the underlying parameters, the rich semantic and association information in the intent recognition task is learned, and the generalization ability of the model is improved.

From the recognition effect of each entity type, the multi-task learning model in this paper also stands out in the F₁ value as shown in Figure 4. The F₁ values of our model are higher than other models in all three entity types, which validates the effectiveness of our method.

For the intent classification task, we used F₁ values, precision and recall to evaluate our model. We used CNN [13], BILSTM and ALBERT [21] as benchmark methods, and we also compared with the single-task model for intent recognition, ALBERT-BILSTM, and the experimental results are shown in

Table 3. Results of intent classification on different model.

Method	P	R	F1
CNN	0.8315	0.7921	0.8113
BILSTM	0.8377	0.8086	0.8229
ALBERT	0.8582	0.8391	0.8485
ALBERT-BILSTM	0.8634	0.8473	0.8553
MTL-ALBERT-BILSTM	0.8842	0.8654	0.8747

.

Our model also outperforms other benchmark model approaches in terms of F₁ value, precision and recall in intent classification task. In addition, our multi-task learning model shows a significant improvement over the ALBERT-BILSTM single-task model, which means that the multi-task learning approach significantly enhances the intent classification capability. The performance improvement of intent classification is more pronounced than named entity recognition, and the F₁ value of the intent classification task is about 2% higher than that of the ALBERT-BILSTM model using a single-task learning strategy. Intent classification is a less complex task in that it only needs to generate labels for the entire sentence unlike the named entity recognition task which generates labels for each word. The model can learn more semantic information in medical named entity labeling, using the entity recognition ability trained by the named entity recognition task, which explains to some extent the significant improvement in the intention classification task. In multi-task learning, each task can “selectively” use the hidden features learned in other tasks to improve its capabilities.

In addition, it can be seen from Figure 5 that the classification performance of the multi-task model in this paper is better than other methods in all seven intent categories, which verifies the feasibility of the ALBERT-BILSTM-based multi-task learning model in this paper.

5. Conclusions and Future Work

In this paper, an ALBERT-BILSTM-based multi-task learning model is proposed for Chinese medical questions intent classification and named entity recognition tasks. Comparative experiments of different models are conducted on the Chinese medical questions dataset. The experimental results show that pre-trained language models and multi-task learning strategies can work together to improve natural language understanding of medical texts. Compared with the benchmark approach and the single-task model, the generalization ability of the model is well improved. The named entity recognition and intent classification task share the ALBERT word embedding layer and BILSTM layer. The model outputs named entity labels or intent labels using their task-specific layers. During multi-task training, parameters in the word embedding layer and BILSTM layer can be updated simultaneously by these tasks, facilitating each other’s learning to obtain rich semantic and associative information at word level and sentence level. There are also some recent works that investigate learning which parameters can be shared, and these works outperform the hard sharing mechanism in a general sense. In future work we try to investigate task-specific goals and inter-task optimization tradeoffs to make the shared representations learned by the model more accurate and thus further improve the predictive performance of the model.