An Italian Patent Multi-Label Classification System to Support the Innovation Demand and Supply Matching

Abstract

The innovation demand and supply matching requires an accurate and time-consuming analysis of patents and the identification of their technological domains; since these tasks can be particularly challenging, this is why recent studies have evaluated the possibility of adopting Artificial Intelligence based on NLP techniques. Here, we present an automated workflow for patent analysis and classification devoted to the Italian patent scenario. High-quality data from the online platform KnowledgeShare (KS) were investigated: KS is the first patent management platform on the Italian innovation scene. A not secondary aspect consisted in determining which words mostly influenced patent classification, thus characterizing the corresponding research areas. Several models were compared to ensure the workflow’s robustness; Logistic Regression (LR) resulted in the best-performing model, and its performance compared well with the State of the Art. For each technological domain in the KS database, we evaluated and discussed its characteristic words; furthermore, a further analysis was focused on explaining why some domains, such as “Packaging” and “Environment,” were particularly confounding. This last aspect is of paramount importance to identify cross-contamination effects among research areas.

1. Introduction

Knowledge valorization is a priority in the European Research Area (ERA); the value creation and societal and economic impacts are part of the common set of values and principles for R&I in the European Union [1]. In modern knowledge economies, intellectual property assets are both drivers of development and factors of social transition. At the European level, industries that make intensive use of intellectual property rights (IPRs) generate 45 percent of annual GDP (EUR 6.6 trillion) and account for 63 million jobs (29 percent of all jobs) [2]. The development of knowledge-intensive and innovative companies is closely linked to intellectual property (IP) because most of a company’s net worth is contained in these intangible assets [3]; for them, the patent study can be a useful proxy for innovation and technological change and diffusion [4]. Since between 70% and 90% of the information about technologies is not published anywhere except in patent documents [5,6], patents are among the best sources of information [7] for the following:

-

Technological mapping and forecasting [8];

-

Predicting core and emerging technologies [9,10,11,12];

-

Diffusion of technologies [8];

-

Convergence of technologies [13];

-

Identification of technological vacuums and hotspots [14];

-

Portfolio analysis [15];

-

Competitive analysis [16,17];

-

Technology trend analysis [18,19,20];

-

Avoiding infringement [21];

-

Identifying technological competitors [14].

Patent documents contain a large quantity of metadata, and, at a time when data production, processing, and management have become key factors in the business of the Industry 4.0 economy, the analysis of these data can serve as a reflection of industrial evolution [22] and economic development. Data presents value for enabling a competitive data-driven economy, which is the core of the Internet of Things and Industry 4.0 [23,24]. The availability of big data represents a great chance to improve decision-making and strategic development and to introduce the next generation of innovative and disruptive technologies [17,25].

Thus, promoting the effective use and development of intellectual property and ensuring easier access to and sharing of IP-protected assets in times of crisis are key priorities of the Union’s IP Action Plan [26]. Universities are a mine of innovation and play an important role in their development; in fact, they are recognized as a critical element in the global competitiveness of enterprises [27,28,29]. Patent transfers between universities and companies have garnered significant attention recently. Previous studies demonstrated that the patent transfer has significant impacts not only on both academia and industry [30] but also on national economies [31,32,33]. To name a few studies, McDevitt et al. [34] showed that patent transfer provides public benefits and influences economic development in addition to generating revenue, increasing funding opportunities, and promoting a culture of entrepreneurship and innovation for universities; Roessner et al. [35] combined U.S. university licensing data from 1996 to 2010 with input-output economic models and found that the contribution of university licensing to the U.S. economy during the period was at least USD 162.1 billion [30,31].

However, technologies patented by universities and research centers, notoriously, at least at the European level, are poorly exploited; in fact, as reported in the ASTP Survey Report on KT Activities FY2019, only 18% of inventions are licensed or optioned [36]. In order to address this lack of valorization and help universities and research centers promote their research results, specific initiatives such as online patent platforms have emerged over time as convenient channels for patent transfer, with the joint effort of both academic and political partners [31,37]. The platforms enable the integration of isolated patents and provide communication and negotiation services to enhance the patent transfer [30,31]. An active and efficient patent marketplace could help reduce transaction costs and facilitate technology transfer, alleviating the information asymmetry problem [38].

Existing initiatives/platforms that are able to create matchmaking between supply and demand for innovation are sometimes ineffective, mainly for the following reasons:

-

They are paid services, not open access—often open innovation platforms;

-

They report the patent document as such, without a usable “translation” for all that facilitates matching;

-

The classification of the patent in a given technological area is a challenging task: users choose a category based on those proposed, but users often do not know how to choose the best, and it is not true that the proposed choices are necessarily the best.

One of the most prominent steps in managing patents is the classification, a particularly expensive and time-consuming phase, above all due to the increase in the number of filed patents and the complexity of the contents; this task is in fact conventionally performed by domain experts [39,40]. Moreover, patent classification is almost always a multi-label classification task, because an invention may be related to different technological areas or be cross-domain, which makes the problem even more complicated. Therefore, finding ways to automate this costly and labor-intensive task is essential to assist domain experts in managing patent documents, facilitating reliable searching, better matching of innovation demand and supply, especially by companies [39,41,42,43].

The solution proposed in this paper is an artificial intelligence-based system designed to accurately recognize technological areas, thereby classifying patents with high precision and recommending the appropriate direction to users, thus eliminating the subjectivity of choice. In this paper, we use a combined approach of Natural Language Processing (NLP) and machine learning (ML) in order to support the patent platforms by addressing two main aspects: on the one hand, it is of paramount importance to create an automated recommender system that can identify the most suitable and correct technological area(s) a patent must be assigned; this is both useful for users looking for specific technologies or patent owners who want to reach the largest fraction of potentially interested users. On the other hand, the methodology allows for the identification of keywords characterizing a patent in an objective and human-independent way; this aspect is particularly useful to create an initial vocabulary of words extracted from patents, thus eventually leading to redefine the available categories and supporting the portal management and, again, the matchmaking among users and patent-owners.

Though the combined framework of NLP-machine learning, in this paper, an explainable patent classification system is proposed for multi-label classification of patents, improving the user friendliness of a platform and enhancing the selection of suitable patents by a company or, in general, boosting the matchmaking of innovation leading to social and economic impact. The proposed approach has been implemented and tested on a patent Italian transfer platform named KnowledgeShare (KS), the largest Italian matchmaking portal for researchers, companies, investors, and innovators. KS is a joint initiative of Politecnico di Torino, the Italian Patent and Trademark Office at the Ministry of Enterprise and Made in Italy, and Netval (the Italian Network for the Valorization of Public Research), which aims to be a meeting point for companies and Italian Public and Private Research Centers. KS has two main goals: to generate a real social and economic impact at national level, transforming research results in useful and tangible innovation and provide a tangible support to not only Italian businesses to accelerate their innovation processes; to drive economic return for Universities and Public Research Organizations to be reinvested in new technology transfer activities within the public research system. KS is the only publicly available patent repository about Italian technology allowing for a thorough investigation of the Italian patent scenario which has not been carried out. In KS, the labels assigned to patents do not follow the International Patent Classification (IPC) and Cooperative Patent Classification (CPC) hierarchies but represent wide technological domains established by KS experts. Thus, an important question about the significance of this categorization naturally arises. Finally, the KS database is particularly appealing for classification purposes as, on the one hand it reduces the number of available categories and mitigates the curse of dimensionality problem; on the other hand, it does not require any arbitrarily set cut-off about the classification hierarchy (e.g., class, sub-class, group, sub-group), allowing to deal with clear and well established categories.

The rest of the paper is structured as follows: in Section 2 we highlight the main features of the KS database; Section 3 outlines the NLP and Machine Learning methods we use in this work; Section 4 shows the results of our pipeline; in Section 5 we discuss the results also making comparisons with those found in the scientific literature highlighting strengths and weaknesses of our analysis; and in Section 6 we draw conclusions and future developments of this study.

2. Materials

The KS (Knowledge-share.eu) database we use in this work includes patents registered from 28 December 1999 to 16 August 2021, consisting of 1731 patents. These documents are uploaded to the platform by 89 Italian Research Centers, both public and private. The yearly number of patents is highly variable, depending on the filing date and the consequent policy about the protection of intellectual property at the national and international levels. This platform can be easily queried by users aiming at obtaining an overview of the state-of-the-art about particular technologies and ground-breaking startups in Italy. On one hand, this service lowers firms’ and investors’ entry barriers for innovations in fundamental and applied science, letting them overcome R&D&I challenges more easily [44]. On the other hand, this platform helps scientists and startups in achieving visibility, expressing their innovative potential, and gaining interest from private and public investors.

KS carves a distinct niche in the international technology transfer landscape by blending a curated, high-quality pipeline with a proactive, service-oriented approach. Unlike many competitors that function as open, passive marketplaces, KS focuses exclusively on innovations originating from universities and research centers, ensuring a baseline of scientific rigor. The platform’s core strength lies in its active management of content. Every technology is presented using a proprietary, standardized template, which guarantees clarity, completeness, and ease of comparison for industrial partners—a feature largely lacking in competing platforms. Furthermore, KS differentiates itself by moving beyond purely algorithmic matching. It employs a team that actively curates content and provides targeted, human-validated recommendations to companies and investors. This proactive scouting, combined with a comprehensive suite of services including valorization support and training, transforms KS from a simple database into an end-to-end technology transfer facilitator. This unique model, which ensures high-quality, standardized information and adds a layer of human intelligence to the matching process, positions Knowledge-share.eu as a potentially more effective and reliable bridge between academic research and industrial application. Moreover, the collaboration with the Ministry allows Netval to provide KS free of charge for all its stakeholders, thus lowering a main barrier to the adoption and thus further supporting a widespread use and accessibility for SMEs and public research centers.

The KS platform allows the user to search for keywords, home institutions, or technological areas. By focusing on a particular patent, the marketing form appears, i.e., a sheet that shows some salient information about the patent, such as patent owner, inventors, patent status, priority number, priority date, licence, commercial rights, and availability, and the content of the patent through some keywords, an explanatory image, an introduction, the technical features, and the possible applications; Figure 1 shows the platform homepage.

Currently, there are more than 1520 registered users on the KS platform, including companies, investors, banks, stakeholders, and so on, which make KS the largest patent platform in Italy and the most accessed platform for patent investigation and technological transfer in Italy. One or more labels can be assigned to each patent; these labels represent the following ten technological domains, from now onwards referred to as categories:

• Aerospace and aviation;
• Agrifood;
• Architecture and design;
• Chemistry, Physics, New materials and Workflows (Basic Science);
• Energy and Renewables (Green Energy);
• Environment and Constructions (Environment);
• Health and Biomedical (Biomed);
• Informatics, Electronics and Communication Systems (Electronics);
• Manufacturing and Packaging (Packaging);
• Transport.

KS data is not equally distributed among labels, a well-known phenomenon in patent analysis studies. In particular, the most populated area is “Health and Biomedical” which accounts for over 30% of the entire database, while the least one is “Aerospace and aviation” with less than 3%, see Figure 2.

Some technological domains were excluded from the analysis as the number of patents was negligible with respect to other areas: “Aerospace and aviation”, “Architecture and design”, and “Transports”. Moreover, patents associated with more than two labels, accounting for less than 5% percent of the available data, were discarded. The final data consisted of 1527 patents and 7 possible labels. As a final remark, KS patents (written in Italian) were translated into English using the deep_translator package in Python (v 3.12.7), as NLP tools for English texts are more developed and consolidated than those for other languages [45].

3. Methods

3.1. The Overall Flowchart

As stated in the Introduction, in this work, we aim at two main goals: (1) classify patents in the KS database upon entering them into Artificial Intelligence models; (2) the identification of the most important keywords characterizing the labels (i.e., the corresponding technological areas). Accordingly, an approach based on the following 3 steps is designed and implemented: (i) textual analysis, (ii) multi-class classification, and (iii) important features underlying patent categorization. A pictorial overview is presented in the following Figure 3.

In the first phase, we use Natural Language Processing (NLP) tools to transform patents’ textual data into matrix form, which is a necessary step in order to use them as input in an Artificial Intelligence pipeline. In the multiclass classification phase, we use Machine Learning algorithms to assign the corresponding labels (i.e., the classes) to patents. In particular, to carry out these first two phases and assess the performance of the classification algorithms, as a zeroth-order approach, we discard patents’ temporal information and consider a 5-fold cross-validation framework repeated 100 times [46]. A pictorial overview of this framework is depicted in Figure 4.

This approach was adopted to highlight the Italian innovation scene, as it is represented by the KS database, more than assessing the temporal evolution. This approach is justified by an ex-post validation: bad classification performances of all the algorithms would indicate the presence of a significant number of disruptive patented innovations during the considered time period, so that training algorithms on a set of patents (included in the training folds) would not be helpful in correctly classifying the remaining patents (comprised in the test fold). Accordingly, we measure the performance of these algorithms using ranking-based evaluation indexes and find the most effective one in patent classification in the KS database [47].

Finally, in the third phase, we accomplish a feature importance task on the top-performer algorithm: we aim at identifying those words that mainly influence the classification of patents in the corresponding classes. In the following sections, these phases are explained in detail.

3.2. From Patents to Feature Representation

In every cross-validation iteration, the set of patents contained in the training folds is transformed, using the appropriate NLP tools, into a suitable representation which can feed machine learning algorithms. First, patents’ words (usually called tokens) are lower-cased. In fact, it is worth noting that learning algorithms are sensitive to upper/lower-case differences; thus, lower-casing every single word is necessary [48,49]; the result of this first pre-processing phase is threefold: uniforming data, obtaining a substantial dimensionality reduction, and yielding a correct and meaningful tokenization. Then, we removed stop words, as they are useful in setting up phrases but meaningless for semantic purposes [50,51]. Finally, we performed lemmatization, a procedure that reduces inflected words to their root form. For example, words like “fishing” or “fished” are reduced to “fish”. Even in this case, the goal is to ease the following learning phase; in fact, grouping words sharing a common root reduces the sparsity of the data, leading to a more compact representation, less prone to the curse of dimensionality [52,53]. Then, we end up with a set of words representing the vocabulary of the patents contained in the training folds.

All these pre-processing phases are propaedeutic for setting up the TF-IDF matrix (where TF = Term Frequency and IDF = Inverse Document Frequency) [54,55]. We chose to use TF-IDF for the sake of simplicity and interpretability. For example, while TF-IDF works directly on the corpus without any training phase, other techniques, such as Word2Vec, require training on a large corpus. Each TF-IDF matrix element is the product of two factors: TF(i,j), representing the frequency of the word i in patent j, and IDF(i,j), a weighting factor that takes into account the frequency of word i in all the patents. In particular, these terms are defined as follows: TF(i,j) = nij/|dj| and IDF(i,j) = log(D/|d(i)|) with nij being the number of occurrences of the word i in patent dj, D the number of patents in the database and d(i) the number of elements in D containing the word i at least once. |.| denotes the set cardinality, i.e., the number of elements within a set. The rationale of this product is the following: the first factor rewards high frequency of a word within a patent; the more a word is cited, the more its importance; the second term penalizes the high frequency in the whole set of patents because a word used in all patents would yield poor discrimination and emphasizes the role played by terms occurring rarely. The result is a matrix having a number of columns corresponding to that of patents in the training folds and a number of rows equal to the length of the respective vocabulary. If a patent has two assigned categories, we duplicate the corresponding column, associating one category with the original column and the other with its copy. Regarding the patents in the test fold, a corresponding TF-IDF matrix is built using the vocabulary of the training folds. This choice is due to avoiding data leakage between the training and test phases and, correspondingly, obtaining over-optimistic results about the generalization ability of Machine Learning algorithms.

3.3. Patent Categorization as a Multi-Class Classification Problem

The TF−IDF representation can be suitably exploited by any machine learning algorithm, with patents representing the available samples, word occurrences being the discriminating features, and the target variable the labels associated with each patent. Since, according to Section 2, at most two out of seven labels are associated with every patent, we are faced with a non-standard multi-class classification problem. In fact, in a standard multi-class classification problem, only one label (out of more than two labels) is associated with every sample. Nonetheless, since multiclass classification algorithms return a ranking of all the possible labels for every sample, we can safely use classical Machine Learning methods, but we need performance measures different from those usually used in standard multiclass classification. Accordingly, we have to refer to measures usually adopted in Information Retrieval tasks to evaluate Recommendation Systems [56,57]. In particular, we first consider two common metrics in these fields, that are defined for each sample:

Precision-top-k (P@k): represents the rate of correct labels for the i-th sample in the top-k labels as predicted by the classification model. In formula:

$${\left(P@k\right)}_i={\displaystyle \frac{number of correct labels in top k for the \mathit{i-th} sample}{k}}$$

Recall-top-k (R@k) is the ratio between the number of corrected labels for the considered sample in the top-k labels as predicted by the classification model and the number of true labels for the sample. In mathematical terms:

$${\left(R@k\right)}_i={\displaystyle \frac{number of correct labels in top k for the \mathit{i-th} sample}{number of true labels}}$$

Both these local metrics have values ranging from 0 to 1.

It is possible to define two corresponding global metrics from the previous local ones by averaging the local values we obtained for each sample:

$$\overline{P@k}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(P@k\right)}_i$$

$$\overline{R@k}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(R@k\right)}_i$$

where N is the number of samples under consideration.

It seems wise to underline that the information carried by the average Precision-at-k and the average Recall-at-k can be condensed in just one metric: the averaged F1-top-k ($\overline{F1@k}$). This metric is defined as the harmonic mean of $\overline{P@k}$ and $\overline{R@k}$:

$$\overline{F1@k}={\displaystyle \frac{2\times \overline{P@k}\times \overline{R@k}}{\overline{P@k}+\overline{R@k}}}$$

$\overline{F1@k}$ ranges from 0 to 1 and is equal to 0 when at least one of the two metrics is 0, while it is equal to 1 when both of them are equal to 1. Since a precision value could be very high at the cost of a very low recall and vice versa, it seems wise to use F1@k as an unbiased performance measure [58,59]. In particular, following the scientific literature on automatic patent classification [60] we consider F1@k with k = 1 as our main performance metric. Accordingly, we can use this score to benchmark our work with the best F1 scores in other articles.

To evaluate the robustness of our pipeline, we consider and evaluate five different models: a Random Classifier (RC), serving as a performance baseline, Logistic Regression (LR) [61], Random Forest (RF) [62], Support Vector Machine with linear kernel (SVM) [63] and XGBoost (XGB) [64]. We consider these Machine Learning models since they are all able to return a measure of features’ importance, although through different mechanisms [65,66]. In fact, LR assigns a coefficient to every input feature, while tree-based models define the importance of a feature according to its ability to increase the purity of a node in the tree, as measured by the Gini index. Moreover, SVM with linear kernel, being a geometrical method that aims at finding a hyperplane in the feature space that separates classes, gives a weight to every dimension (i.e., to every feature). For all the considered algorithms, the standard configuration was used, without hypertuning their parameters. The rationale was to keep the models as simple as possible to achieve a fair evaluation of the informative content provided by the patents, not concealed by the algorithms’ learning phase.

It seems wise to underline that, even though all the models can be used to accomplish a multi-class classification, not all of them natively support it [67,68]. Actually, LR and SVM are able to carry out a binary classification task, not a multi-class one. Nonetheless, this problem can be circumvented by adopting a “One-versus-Rest” (O-v-R) or a “One-versus-all” (O-v-A) framework. In the first case, for each class, a binary model is trained in order to recognize the considered class against all the others. Accordingly, we end up with a model for each class. In the second case, models are trained to distinguish every class from each other, obtaining a much greater number of models with respect to the O-v-R case. In this work, we adopt an O-v-R approach because we can obtain which features (i.e., words) characterize every category with respect to all the others.

3.4. Mitigating Class Imbalance: SMOTE

In the previous sections, we described how to transform textual data into a matrix form and how to set up a multi-class classification problem, together with presenting the appropriate performance metrics. However, it should be noted that, as stated in Section 2, there is an imbalance among classes that can severely impact algorithms’ performance [69,70]. In order to mitigate this effect, we leverage the mathematical form of textual data and use SMOTE (Synthetic Minority Oversampling TEchnique) in order to obtain a perfect balance among classes [71,72]. In particular, we apply this technique in the training folds of every iteration in our cross-validation framework. Then, for every iteration, we train our models on these class-balanced training folds and validate them on the test fold. We then replicate this workflow for every repetition of the cross-validation framework.

4. Results

In this section, we show the results obtained by applying our pipeline to the KnowledgeShare database. In particular, we will show $\overline{P@k}$, $\overline{R@k}$ and $\overline{F1@k}$ for k = 1, for completeness, but we will use $\overline{F1@1}$ to determine the best model. Once the best classification model is identified, we deepen its performance analysis, highlighting the most important words: those that mainly influence it in classifying patents. Moreover, we show how reliable it is in classifying patents in each of the seven categories of the KnowledgeShare database, and which are the most error-prone.

4.1. Top-k Performance Measures

In

Table 1. Mean $\overline{P@1}$, $\overline{R@1}$ and $\overline{F1@1}$ values for the classification algorithms. The corresponding standard deviations are reported in brackets. $\overline{F1@1}$ values are in bold.

Metric	RC	LR	RF	SVM	XGB
$\overline{P@1}$	0.181 (0.022)	0.801 (0.024)	0.755 (0.026)	0.793 (0.023)	0.764 (0.023)
$\overline{R@1}$	0.139 (0.017)	0.679 (0.022)	0.645 (0.024)	0.674 (0.022)	0.648 (0.021)
$\overline{F1@1}$	0.157 (0.019)	0.735 (0.022)	0.695 (0.024)	0.728 (0.022)	0.701 (0.021)

, we report, for each considered classification algorithm, the mean and standard deviation values (in brackets) of $\overline{P@1}$, $\overline{R@1}$ and $\overline{F1@1}$ as determined by our 5-fold cross-validation approach repeated 100 times.

Observing the $\overline{F1@1}$ values in Table 1, it can be stated that the Random Classifier is the worst model, while all the other models clearly overperform it. This means that these last models effectively use words to classify patents. Moreover, it can be observed that LR and SVM have similar performances. Accordingly, we decide to perform a Welch’s t-test to establish if the mean values of $\overline{F1@1}$ of the LR and SVM are equal or not in a statistically significant way. Considering a 5% significance level, we can reject the null hypothesis that the mean values of $\overline{F1@1}$ of LR and SVM models are identical; but, at a 1% significance level, we cannot reject this null hypothesis. In order to remove this ambiguity, following the scientific literature, we consider the performances of LR and SVM as measured by $\overline{F1@2}$. We obtain the results reported in

Table 2. Mean $\overline{P@2}$, $\overline{R@2}$ and $\overline{F1@2}$ values for the classification algorithms. The corresponding standard deviations are reported in brackets. $\overline{F1@2}$ values are in bold.

Metric	RC	LR	RF	SVM	XGB
P@2	0.184 (0.013))	0.544 (0.013)	0.503 (0.016)	0.518 (0.016)	0.508 (0.016)
R@2	0.286 (0.022)	0.867 (0.016)	0.813 (0.022)	0.830 (0.018)	0.818 (0.021)
F1@2	0.224 (0.019)	0.669 (0.013)	0.622 (0.018)	0.637 (0.016)	0.627 (0.017)

, where, for the sake of completeness, we report the results for all the models.

Performing a Welch’s t-test at both 1% and 5% significance levels, we can reject the null hypothesis and consider the mean performances of LR and SVM different in a statistically significant way. Accordingly, we can consider LR our top-performing model in classifying patents.

4.2. How Words Influence Patent Classification

According to the previous section, LR has proved to be the best model in classifying patents in the KS database. We can go deeper in analyzing LR performances, considering how the words in the vocabulary (i.e., the features of the model) influence the classification task carried out by the LR model. To this aim, since in the O-v-R scheme an LR model is built for every category, we evaluate the corresponding coefficients, which can be seen as a measure of the importance of the input features. Moreover, we also determine the words’ global importance ranking by concatenating the words’ coefficients of every category. We end up with a global ranking of 88,144 words. For the sake of clarity, Figure 5 shows the average importance for the first 1000 words in the global ranking. Applying the elbow method, we were able to quantitatively determine the number of most significant words, obtaining a set of 260. A more complete overview of these most important words, together with the category they belong to, can be found in Appendix A.

Figure 6 shows the partition of the 260 words in the 7 technological areas.

As regards Green Energy, the top words include topics related to major renewable sources, heat, electricity, fuel, and power. In the Agrifood area, the top words included specific products like wine, vegetable oil, and milk. Interestingly, among the top words, particular attention was given to the waste theme. In Electronics, the principal words were: user, network, circuit, software, signal, optical, and device. In the Environment and Constructions area, the focus seems to be on buildings with seismic properties and energy- and water-saving solutions. The only 5 words in the area concerning the Packaging appear not so relevant: component, object, material, packaging, and piece. On the contrary, the 5 words describing the Biomed sector are very meaningful because they seem to characterize the areas of greatest investment and innovation in recent years in the medical field: patient-centeredness, tissue engineering, treatment and diagnosis of cancerous and non-cancerous diseases, and stem cells and cell therapy. The last area, Basic Science, is described by only 2 words among the top 50: particle and solvent.

4.3. Confounding Categories for the Best Classifier

In this section, we explore which categories are the most confounding categories for the best-performing model, namely the LR. To this aim, in

Table 3. Mean frequencies of confusion among categories (in percentage). The frequencies of correct labelling, in bold, are reported in the main diagonal. The corresponding standard deviations are in brackets.

Predicted Label	Agrifood	Environment	Basic Science	Green Energy	Electronics	Packaging	Biomed
True Label
Agrifood	68.1 (9.1)	6.2 (3.3)	7.2 (2.1)	2.1 (1.0)	5.1 (2.2)	1.1 (0.4)	10.2 (4.1)
Environment	4.5 (3.2)	42.1 (8.2)	19.3 (6.2)	13.4 (4.4)	13.2 (4.1)	6.2 (2.2)	3.1 (1.0)
Basic Science	6.2 (2.1)	6.7 (2.2)	51.1 (5.1)	6.2 (3.1)	7.6 (2.1)	3.1 (1.0)	20.4 (4.1)
Green Energy	3.8 (1.1)	11.2 (2.3)	11.2 (3.2)	56.9 (8.2)	13.2 (5.3)	2.1 (0.5)	3.4 (1.1)
Electronics	1.8 (0.2)	4.5 (0.8)	6.5 (1.9)	6.8 (1.8)	64.3 (5.2)	5.6 (1.2)	14.5 (2.2)
Packaging	8.7 (1.3)	8.2 (1.2)	23.8 (3.2)	5.2 (1.4)	23.9 (8.4)	24.8 (2.1)	10.8 (2.1))
Biomed	1.1 (0.1)	1.8 (0.2)	8.8 (1.1)	1.6 (0.1)	9.7 (1.1)	2.8 (0.7)	77.8 (2.3)

we show, for every category, the mean frequencies with which it is confused with all the others in our cross-validation framework. Moreover, in Table 3 we also show the frequencies of correct classification of each category.

Observing these results, “Biomed” is the best-recognized category, meaning that it is a well-established and self-contained research area, with its own keywords and concepts, being rarely confused with other research domains. On the other hand, it can be observed that “Packaging” is the worst-recognized category being mainly confused with “Electronics” and “Basic Science”. This indicates that “Packaging” cannot be considered a standalone research area but, on the contrary, it may be seen as a cross-domain that benefits from innovations in Basic Science and Electronics. For example, in Packaging patents, it is possible to retrieve words such as “sensors”, “device”, “system” which can easily be found in other domains. It should also be kept in mind that the category Packaging also includes items belonging to the broader field of manufacturing, in Appendix B, two patents are reported as an example. In fact, an invention may cover several technological areas or be cross-domain, and this is becoming increasingly true in emerging technologies that are becoming more and more “multi-disciplinary” and struggle to remain “confined” to a single technological area. This highlights the need to create new technological domains that are not strictly tied to a single area.

5. Discussion

5.1. Best Method Performance

Patent Automatic Classification (PAC) algorithms have raised an increasing interest in recent years because of the ever-growing number of patents and the consequent need for high-quality analysis [56]. Moreover, correctly classifying patents means understanding why a patent is labelled with some categories and not with others. Then, we can query the classifier algorithm to know which words mainly influence the classification of a patent considering the model’s feature importance. Since this is our work’s main aim, we have to compare our best classifier’s performance with the state-of-the-art found in the literature. It should be noted that there is no officially acknowledged database on which the pipelines’ performances should be compared. Accordingly, there is neither a common set of patents nor a shared classification system of patents (e.g., CPC, IPC). Then, the significance of the following comparisons should be considered in this aspect. Moreover, it should be underlined that, as far as we know, our work is among the first in using a cross-validation approach, presenting mean and standard deviation values of $\overline{P@k}$, $\overline{R@k}$ and $\overline{F1@k}$.

Beginning from articles having a dataset size similar to ours, both Fall et al. [73] and Hepburn [74] consider SVM together with other classification algorithms (k-NN, Naive Bayes) to classify 75,250 patents in 8 categories (IPC section level). In particular, Hepburn [74] uses a 60/40 train-test split and a particular transfer learning technique, reporting $\overline{F1@1}$ = 0.784, without standard deviation, which improves the result of Fall et al. [73], which indicates $\overline{F1@1}$ = 55%. Hu et al. [58] use Neural Networks (both Convolutional and Recursive) to capture semantic correlations among patents. The training, validation, and test datasets contain, respectively, 72,532, 18,133, and 2679 patents from the CLEF-IP competition dataset with 96 labels (CLEF-IP (http://ifs.tuwien.ac.at/~clef-ip/ accessed on 16 May 2023)). Their best $\overline{F1@1}$ is 63.97%. Regarding the CLEF-IP competition itself, Verberne et al. [75] have their best F1@1 equal to 70.59% considering a series of classification experiments with the Linguistic Classification System (LCS). The training dataset has 905,458 patents, and the testing dataset has 1000 patents. Li et al. [56] built a Deep Learning model to classify more than 2 million patents in the USPTO-2M (https://www.uspto.gov/ accessed on 31 January 2024) dataset in 637 classes (the subclass level in IPC), reporting P@1 = 73.9% without disclosing F1@1. Haghighian Roudsari et al. [39] compare different Deep Learning models on the USPTO-2M dataset, obtaining a maximum $\overline{F1@1}$ = 63.33%, with $\overline{P@1}$ = 82.72% and $\overline{F1@1}$ = 55.89%. Lee and Hsiang [60] improve this result by considering a novel USPTO-3M dataset, comprising more than 3 million patents and working both at the IPC subclass level and at the CPC subclass level (656 classes). It reports a maximum $\overline{F1@1}$ = 66.71% and, correspondingly, $\overline{R@1}$ = 54.92% and $\overline{P@1}$ = 84.95%.

Considering that our best model (LR) has $\overline{F1@1}$ = 73.5% (2.2%), $\overline{P@1}$ = 80.1% (2.2%), $\overline{R@1}$ = 67.9% (2.2%), we can say that our results are in line with the state-of-the-art found in the literature, thus justifying the investigation of LR words’ importance and the most confounding categories. Nonetheless, it should be pointed out how this analysis is limited by the sample size, and further evaluations are needed on a larger dataset; a possible strategy could be the collection of data from other countries which would also allow the comparison of the Italian situation with other countries. The Netval is already exploring this possibility along with the integration of the presented remarks in the platform. Of course, the limited sample size prevented this study from exploring a wide and popular class of algorithms, such as those based on deep neural networks. However, the goal of this work was to capture the Italian patent landscape more than delivering a decision support system; this is why we preferred to adopt simpler but definitely more interpretable strategies.

5.2. Categories’ Keywords and How They Explain the Confusion Frequencies

A detailed examination of the words in each category reveals some trends in the Italian innovation scenario and can even explain the confusion frequencies reported in Section 4.3.

As concerns the “Green Energy” category, the innovations speak about automatic systems for cleaning solar photovoltaic modules, water production from the air with solar energy, systems for concentrating the solar power, energy-efficient systems for the use of solar thermal energy, micro turbo gas systems, thermodynamic solar systems, and so on. Since these innovations deal with electronic devices and the exploitation of physical processes, it is not surprising that the main areas it is confused with are “Electronics” and “Basic Science”. Moreover, since green energy innovations aim at the protection of the environment, this strong relationship is reflected in the non-negligible confusion frequency of “Green Energy” with “Environment”.

With reference to the “Agrifood” area, the main identified words suggest research investments in innovating the quality and production systems of wine, lab-on-chip devices for the improvement of olive oil protection, and novel sustainable processes for milk production. The main innovations in this field involve biological processes and methodologies, and this is reflected in the relatively high frequency of confusion of this area with Biomed.

The “Electronics” research area reveals the presence of innovations about user-centered design methods, novel architectures of artificial neural networks, and, above all, deployments of synthetic biological circuits. This relatively new research branch is increasingly growing and explains the high frequency of confusion between “Electronics” and “Biomed”, which mainly encompasses biological tools and methodologies.

As regards the “Environment” category, the most frequent words concern innovations in: cleantech, automatic irrigation systems, water saving, energy production from water or wave, micro-geophysics, remediation of stone buildings, dehumidification processes, building maintenance services, and seismic isolation methods, and so on. It should be noted that these innovations are strictly related to the use of novel electronic devices and sensors and the pioneering exploitation of thermodynamic processes. These observations explain why “Environment” patents are mainly labelled as “Electronics” and “Basic Science”. Moreover, as noted in the previous discussion about the “Green Energy” category, there is a strong relationship between it and “Environment” innovations, which accounts for the relatively high level of confusion between them.

In the “Packaging” area, a number of health-related innovations characterize the research effort in this field: Hybrid locomotion stair-lift wheelchair, artificial muscle, human-robot cooperation, wearable robot for lower limb, exoskeleton for upper limb rehabilitation, adaptive wearable robot, support frame for upper limb exoskeleton, biomimetic active foot and ankle prosthesis. The remaining innovations include 3D printing, multi-sensor dimensional measurements, conversion of traditional vehicles, and language learning devices. It can be seen that innovations registered in the “Packaging” area are mainly linked to the development of novel electronic devices for relieving heavy lifting and learning methods using ad-hoc Artificial Intelligence algorithms. This can be seen as the main reason behind the high confusion frequencies characterising the “Packaging” area, already noted. In fact, it has the two highest confusion frequencies with “Electronics” and “Basic Science”. This clearly indicates that the “Packaging” category cannot be considered as a stand-alone research branch because it largely benefits from innovations both in electronics and in basic science research.

The innovations in the “Biomed” technological area range from wearable devices, methods for diagnosis and/or monitoring of infection, to new molecules for cancerous and chronic diseases, interventional radiotherapy, tests for early detection of tumors, theragnostic radiopharmaceuticals, cell therapies, and selection of cell populations. It should be noted that these novel techniques and devices mainly rely on Artificial Intelligence methodologies. In fact, the greatest part of patents are about methods in tissue engineering, an interdisciplinary area concerned with the development of functional 3D tissues by mixing cells, and bioactive chemicals. Accordingly, we can see that this explains why the “Biomed” category has non-negligible confusion frequencies with “Basic Science” and “Electronics”. Nonetheless, since the Biomed research is even highly characterized by biological-medical terms, the corresponding confusion frequencies are not as high as in all the other categories. Another interesting aspect concerning Biomed is that despite this being the largest community of patents, accounting for almost 30%, in the list of top words (Appendix A), only 12% of entries are related to the Biomed field.

Research efforts in the “Basic Science” category are multi-faceted: they range from innovative, recyclable, and low-cost solvents to field-effect transistor sensors, 3D super-resolution optical microscopes, and magnetic resonance imaging using metamaterials. Since many patents deal with bio-materials or describe bio-medical innovations, it is not surprising that “Basic Science” has a 20.4% mean frequency of confusion with “Biomed”.

In general, as regards the confusion frequencies, it seems evident that the technological areas so defined in KS are sometimes a bit “narrow” to categorize patents that are related to different technology domains, i.e., the 7 technology areas may not be sufficient to capture the entire landscape of academic patents. It would be interesting to use an unsupervised clustering approach, based on the textual content of patents, to develop an alternative and probably more consistent classification, improving the matching of supply and demand for innovation [76]. Of course, this represents a time-independent picture; the dynamical analysis of the Italian landscape was outside the scope of the present work, but we definitely think this aspect would deserve a deep exploration.

6. Conclusions

In this work, we present a fully automated patent multi-classification system based on NLP techniques and Machine Learning that highlights the most important words characterising the classification categories. Patent platforms such as the Italian one, named KnowledgeShare, were created to connect patent buyers with patent providers and help alleviate the problems of information asymmetry and distrust in traditional patent trading. To facilitate their mission and improve their efficacy, tools like the one proposed here are of fundamental importance. We demonstrated that the accuracy of the proposed framework compared well to state-of-the-art and, by relying on an easily interpretable model such as LR, it also allowed us to outline the keywords of each technological domain. An important feature of our methodology is that patents can be assigned to one or more labels simultaneously, which reflects the possibility for emerging technologies to be cross-domain. The proposed method is open access and supports both the patent managers providing a reliable recommendation about the categories which should be assigned to a patent and its technology and the end-users of technological platforms (e.g., investors, companies, …) providing them a quantitative evaluation of the similarity between the technologies they are looking for or they want to patent and the most suitable technological domains. Hence, the recommender system eases the matchmaking between innovation demand and supply, a fundamental aspect to generate social and economic impact with novel technologies. Although limited to the Italian patent ecosystem, the model is general and could be applied to patents from international research centres. As a future technical improvement, Artificial Neural Network architectures can be exploited, together with suitable explainability tools, in order to enhance both classification performance and, consequently, the insights into the innovation. Another fundamental upgrade would consist of the implementation of the model on the KS platform.