Assessing Technology Platforms for Sustainability with Web Data Mining Techniques

Abstract

Public policies have encouraged the proliferation of technology platforms that support the transition towards sustainable agriculture and the development of innovations in the food system. Provided the difficulty associated with assessing the outputs and outcomes of technology platforms, this work proposes a practical assessment method based on the retrieval and analysis of online documents related to the technology platforms. Concretely, the method consists of applying web scraping techniques to retrieve documents related to a technology platform from the Internet and then applying web data-mining techniques to automatically classify these documents into the functions that the platform should fulfill, which are described from the viewpoint of co-evolution of innovation. Data are automatically processed to obtain a variety of metrics, which are applied to measure the impact of European Technology Platforms (ETPs) on promoting an organic food paradigm. This method provides time-series data that helps to follow the evolution of the different functions of the platform and to describe its lifecycle. It has been applied to one platform taken as a case study, TP Organics, which represents a key platform for stakeholders that promote organic farming and agroecology as core components of an ambitious program for sustainable agriculture. The obtained online-based measures have been proven to assess the global evolution of the platform, its dissemination through the European Union (EU) Member States, and the evolution of the different functions expected to be fulfilled by it regarding the diffusion and promotion of innovations in organic agriculture.

1. Introduction

The agricultural research agenda has been subject to major changes in recent years, resulting in a diverse range of innovations. The present stream not only promotes innovations to obtain more food and agricultural raw materials, but also to alleviate rural poverty, improve diets and health, and allow a transition towards sustainable systems. Such a move also includes sustainable agriculture, which involves the adoption of practices that enhance the environmental quality and the natural base of agriculture. Progress towards a sustainable paradigm requires new ways of organizing research, new ways of establishing priorities and flexible ways of evaluating results and impacts [1].

Research policies, their priorities, and implementation have been developed for supporting innovation in the agricultural systems through technological platforms (TPs), which facilitate industry-based partnerships and networking. Specifically, in 2004, the European Commission invited the industry to establish European Technology Platforms (ETPs), especially to define research agendas that would attract private investment. The ETP became an industry-led stakeholder fora to develop research and innovation agendas and roadmaps for action at the EU and national levels [2,3].

The ETPs came to the agri-food sector to involve all relevant stakeholders in the development of common visions that could meet the needs and benefits of society. For the agro-food-forestry-biotechnology sectors, the ETPs were initiated mainly by industrial organizations led by multinational companies, with the support and proposals of scientific organizations. The concept quickly extended from capital-intensive industries to ecological and social sustainability [4,5]. The technology platforms continue to be a tool within European policies and long-term strategies to promote innovation, extended to social and environmental benefits related to public concerns. The ETP concept has developed in sectoral forms, represented by platforms such as EATIP (European Aquaculture Technology and Innovation Platform) in the aquaculture sector or TP Organics in the agroecological sector; in horizontal forms such as the ETP Nanofutures, a multi-sectorial cross-ETP devoted to connect ETPs that require nanotechnologies in their industrial sectors; and has also been accompanied by other initiatives such as the ERANETs and the European Innovation Partnership for agricultural productivity and sustainability (EIP-Agri), which are instruments of the European Commission designed to support public-public and private-public partnerships in their preparation, coordination, and funding [6].

Previous ETP assessments have proven the success of ETPs in developing joint visions, setting strategic research agendas and contributing to the definition of the research priorities [7]. However, the evaluation exercises frequently result in costly efforts given the lack of indicators available to monitor the impact of ETPs, in particular in emerging areas, such as the innovation for sustainable systems. Since most ETPs have been set up recently, it is difficult to assess, in a quantified way, their results and impacts, which are typically revealed after a certain period of time. We acknowledge that evaluation is a comprehensive exercise that attempts to answer questions which depend on the policy objectives of the platform and that are normally supported by surveys and interviews.

Meanwhile, the transformation and digitization of society have resulted in the emergence of social and economic behavior on the internet as a digital footprint. Certainly, this includes the activity of the ETPs and the actors involved in them. The digital activity of individuals and companies is closely related to their underlying behavior, as previous research showed [8,9,10]. For this reason, it is currently being studied by researchers and official institutions as a source of information to monitor and understand the underlying socio-economic behavior. See, for instance, the efforts by the European statistical system with the ESSnet Big Data project [11] or the academic literature reviewed by Blazquez and Domenech [12].

The high availability of online data, together with the advances in Big Data research allows us to envision novel monitoring and assessment methods with lower operational costs as they are not based on surveys and interviews. Current Big Data technology enables us to design assessments systems that include the automatic retrieval of content generated by some economic agents and then apply document analysis and text mining to classify the documents and generate activity indicators. Some examples include evaluating the export orientation of companies [10], the diffusion of technology [13], or job skills demand in the labor market [14].

The objective of this paper is to measure the activity and impact of an ETP using the information publicly available about the platform on the Internet as the main source. With a view to the transition of food systems, we depart from a conceptual framework about the functions an ETP needs to meet and propose a method to retrieve the documents relevant to the ETP and classify them according to the main function they are related to. This would allow us to provide insights on how the activity of the ETP is being carried out and understand it as part of an ETP lifecycle. Our method also enables a regional analysis by mapping the activities of an ETP to the European countries.

To provide a method validation, we focused the method implementation on a specific ETP that has a wide scope as it faces technological, societal, and environmental challenges. We explored TP Organics as a platform that promotes an ambitious transformation of agricultural systems through organic farming and agroecology. TP Organics, according to its website (http://tporganics.eu/about-us/.) aims at strengthening “research & innovation for organics and other agroecological approaches that contribute to sustainable food and farming systems”. It is broad in scope as it combines large companies, small and medium enterprises, researchers, farmers, consumers, and civil society organizations active in the organic value chain from production, input, and supply to food processing, marketing, and consumption. TP Organic, as a technological platform, promotes holistic and long-term agroecological solutions and the “co-creation of knowledge, combining science with the traditional, practical, and local knowledge of producers”

In summary, taking TP Organics as a pilot case, the main contributions of this paper are: (i) formulate a framework for ETP monitoring based on the functions that they seek to meet, (ii) devise a method for online data collection that provides a comprehensive monitoring of the defined functions; (iii) apply the framework and method to an ETP of a broad scope such as TP Organics; (iv) provide a first evaluation of the lifecycle of TP Organics and its functions.

The remainder of the paper is structured as follows. Section 2 defines a conceptual framework based on the functions of innovation intermediaries. Section 3 describes the case study, TP Organics, which is a functional ETP for developing new concepts and stakeholder networking towards sustainable food systems. Section 4 proposes a practical methodology for the digital monitoring of TP functions that aims to monitor the differential performance of innovation functions in TPs from a co-evolutionary perspective that reflects interactions between societal subsystems [15]. Section 5 describes the results obtained after applying the proposed methodology to TP Organics. Finally, Section 6 draws some concluding remarks.

2. Analytical Background

We consider TP (or ETP in its EU version) as a specific form of innovation intermediary, which is a broad concept that can be applied to TP but also to more flexible configurations such as research and innovation networks. For the transition of food systems, an innovation platform has certain characteristics: (i) it combines multiple actors; (ii) it performs activities around agricultural innovation challenges at different levels in agricultural systems (value chain, sector, country, village, etc.).

The TPs can be observed as intermediaries that connect different actors within the innovation systems in order to encourage co-evolution [16]. Howells [17] defines innovation intermediaries as “organization[s] or bod[ies] that act as agent[s] or broker[s] in any aspect of the innovation process between two or more parties”.

TPs and other innovation platforms could be considered single innovation intermediaries that coordinate the platform, but it may be more realistic to see a TP as a space for several intermediaries to achieve the connection between stakeholders and thus facilitate the transition of innovation systems [18,19,20,21]. A TP plays a role of an innovation intermediary or of an ecosystem of individuals and organizations that perform intermediary functions from a bottom-up perspective [22,23]. TPs form a space of nested systems of intermediaries that fulfill complementary functions that foster co-evolution in innovation systems [16].

Kivimaa et al. [24] point out the lack of literature on intermediate activities and on the type of contribution of the different types of intermediaries to better support the acceleration of innovation processes. TPs could be considered as a type of knowledge-intensive business service (KIBS) in multilevel contexts [25,26,27,28]. Seen as KIBS, TPs provide a space for complex operations in which the generation and dissemination of knowledge take place through an intense interaction between actors in which human capital plays a critical role.

Whereas functions of innovation intermediaries could be embedded into a broader perspective of innovation system functions [29,30], we propose to assess TP functions in a perspective which looks at the ways that innovation intermediaries may support innovation. Indeed, much of the literature on innovation intermediaries has focused on their functions, such as the studies of Bessant et al. [31] and Howells [17] that go beyond the coordination and brokering relations in complex multi-actor configurations in the innovation systems.

Kilelu et al. [16,20] established a classification of intermediary functions applicable to the interpretation of innovation platforms. These functions can be adapted to the case of TPs by applying some modifications to the definitions proposed in those studies. As a result, these functions are as described below:

• Function F1: Demand articulation. It involves activities that identify innovation challenges and opportunities, such as information gathering, needs assessment, and strategic planning.
• Function F2: Institutional support. This function promotes institutional change by facilitating changes in regulations, working on attitudes and practice, and also linking science, policy, and practice.
• Function F3: Capacity building. It promotes incubating and strengthening organizational forms and networks. This includes objectives regarding organization development, as well as training and competence building activities.
• Function F4: Network brokering. This involves establishing key relationships among different actors to form partnerships and promote cooperation.
• Function F5: Innovation process management. This function promotes the interaction among different actors for research and innovation activities by aligning agendas and mediating relationships.
• Function F6: Knowledge brokering. It identifies knowledge needs and facilitates knowledge transfer from different sources by articulating, communicating, and disseminating knowledge and technology.

These functions are perfectly valid to assess ETPs; although, ETPs were originally designed as an arena to guide the research and innovation agenda at the EU level, for example, for H2020 projects. Such functions require monitoring efforts and we intend to provide a tool for this based on online data. Once indicators for each function are defined, a further step will be to use the measured indicators to assess the contribution of a TP to the transition of innovation systems. The monitoring of functions can allow us to determine their relevance in terms of impact but it is not possible to assure that a function with more events has a higher economic or social impact than a function with a lower number of events. However, it is possible to build synthetic indicators that will make it possible to explore the dynamics of functions and check possible patterns of interaction between functions.

Whereas the monitoring of the functions is not precise enough to fully unravel co-evolution processes as in previous studies (e.g., Kilelu et al. [16]), comprehensive data mining can provide the basis for monitoring TPs functions. This opens the door to a further understanding of the dynamics of TPs.

The co-evolutionary nature of innovation covers different dimensions including technological change but also social and institutional changes [32,33]. A basic operationalization of the co-evolutionary approach would attach actors’ practices to hardware (technological innovation), software (culture, thinking, and learning) and orgware (organizations and institutions). In a broad sense, we could attach functions F1 (“Demand articulation”) and F5 (“Innovation process management”) to hardware, function F4 (“Network brokering”) and F6 (“Knowledge brokering”) to software, and F2 (“Institutional support”) and F3 (Capacity building) to orgware. Therefore, we could carry out a first assessment on how the alignment of functions takes place in a co-evolution process of innovation systems.

3. TP Organics as a Technology Platform

Organic agriculture is characterized by promoting the efficient use of nutrients by keeping its cycles short and as closed as possible [34,35]. The organic sector has generated a wide range of innovations and has developed new ideas that have been put into practice in farms and businesses across the EU and the world [36,37]. Many organic farms and agro-food companies have become living creative laboratories for smart and green innovations. The organic sector has already generated a plethora of useful new practices for sustainable agriculture both within and outside the organic sector, with a promising future [38]. Controversy on the role of organic agriculture to feed the world continues [39]. However, our intention is to illustrate how online data collection can be used to monitor TP Organics as an innovation platform.

The structure of TP Organics has been summarized in Figure 1. Its core is formed by the secretariat and the steering committee, which is integrated by Bionext (a national technology platform of the Netherlands), Naturland e.V. (an association of farmers), FiBL (the Research Institute of Organics Agriculture), IFOAM EU (an international not-for-profit organization) and ACT Alliance EU (a network of 14 faith-based development agencies), as relevant representatives working towards a sustainable organic system. The rest of the platform is formed by 30 umbrella organizations and international networks, 36 firms, national technology platforms from seven Member States and 27 supporting members which may also act as financial supporters. Also, TP Organics cooperate with 10 initiatives; six at EU level and four at an international level.

We consider that the TP Organics platform is an interesting case to test our monitoring methodology due to several reasons. In the first place, it is a relatively recent ETP, which responds to an increasing demand for facing research challenges of the transition towards sustainable innovation systems. Secondly, the ETP was the result of a bottom-up process, bringing together farmers, processors, retailers, and scientists in a reflection on the future scenarios of agriculture and food systems. Third, TP Organics integrates not only technologies but also many social aspects of innovation, including viable concepts for the empowerment of rural economies, the sustainability of food systems towards eco-sustainable methods and the production of high-quality food as a basis to improve the quality of life and health. The performance of organic systems versus conventional systems of agriculture has been evaluated by the integration of several dimensions that consider production methods, environment, farmers’ livelihood, consumer health, and food access [40]. Additionally, and connected with the previous points, TP Organics is a good example of a double way of governance of innovation systems from a multilevel perspective. TP Organics offers a space for an ecosystem of intermediaries that facilitate the interconnection between innovative practices at the niche level and more structural changes at the regime level with respect to beliefs, general rules, routines and standardized ways of doing things, policy paradigms, and social expectations and norms [41,42]. Therefore, it is a good example that brings together the six functions of the innovation platforms mentioned in the previous section. Finally, TP Organics has a very active corporate website, which is a sign of its offline and online activity.

Even without the initial recognition of the European Commission, promoters of organic products sought the support of interested stakeholders, including an international association (IFOAM), the relevant commercial actors throughout the agri-food value chain, as well as environmental non-governmental organizations (NGOs). In 2008, they published a Vision for an Organic Food and Agriculture Research Agenda until 2025 [43]. As a result of this process, TP Organics was designed with a focus on sustainable food systems and public goods and officially launched in the fall of 2008. This was followed by a Strategic Research Agenda, which linked the term “innovation” with public goods, efficiency, farmers’ knowledge, learning, and competitive advantage. In 2010, an Implementation Action Plan was published describing what is required for developing key research topics: eco-functional intensification; the economy of low use of external inputs; animal health systems. All this evolves with a focus on resilience and sustainability, from agricultural diversity to the diversity of natural foods and the creation of innovation centers in agricultural communities. In 2013, the promoters of TP Organics and other international actors launched a global TP to establish the Technological Innovation Platform of IFOAM—Organics International (TIPI) as a global, informal network to mobilize different organizations working on organic research topics [44]. The timeline of TP Organics is summarized in Figure 2.

4. Methods

The assessment of the activities and impact of the TP organics platform was conducted by applying a combination of web data mining techniques. In particular, a web scraping system was employed to prepare a corpus of TP organics-related documents published online and content mining was applied to analyze the retrieved texts.

The corpus was created by introducing the term “TP Organics” (and the variant “TPOrganics”) in Google and retrieving all documents in the result list. Since our objective is to understand the evolution of the ETP, only PDF files were kept because the time and date of their creation can be reliably determined, unlike HTML web pages. This way, the web mined content is put in the context of the date in which it was created. A deduplication process to remove multiple copies of the same document was also performed to avoid double counting. The web scraping process resulted in the retrieval of 197 different documents related to TP Organics.

To understand the ETP functions involved in the contents of each document, we compiled a set of keywords describing each of the ETP functions introduced in Section 2. For instance, finding “research agenda” in a document suggests that it is related to the demand articulation (F1), while “workshop” suggests capacity building (F3). The complete list of the keywords we used to describe each function can be found in Appendix A. Similar methods for classifying documents were successfully validated in previous works [13,45,46].

Once the keywords for each function are defined, content mining techniques were applied to the documents in the corpus. Particularly, documents were transformed to compute the frequency of the n-gram terms (i.e., the keywords) describing the ETP functions. Also, in order to allow for regionalizing the activity of the ETP, the frequency of the co-occurrence of each keyword with each EU country’s name was also accounted for. To do so, a script that extracts the text from the documents and computes the frequencies of keyword occurrence and co-occurrence was developed in bash. Its execution resulted in a dataset of about 1 million rows, one for each triplet of document, keyword, and country. The dataset also included the creation date timestamp of each document, as extracted from the PDF metadata.

The activity reflected in a document typically involves more than one ETP function, as they are not mutually exclusive. To deal with this, we classified the documents in the function whose keywords were more intensively found in them.

The combination of the PDF creation timestamp with the keyword count allows us to create a powerful tool to monitor the evolution through time of the contents related to the ETP. This is similar to what Google Trends presents for the user searches in the Google Search Engine.

5. Results and Discussion

This section presents the online-based measures related to the assessment of TP Organics that were obtained after applying the methods described above.

5.1. Expansion of the ETP

The first measure is reflected in Figure 3. It shows the evolution over time of the quantity of online documents that mention “TP Organics”. As can be seen, there is a clear increasing tendency, where the number of created documents generally grow during the platform lifetime. This sustained increase in the number of documents related to TP Organics suggests that the activity of the platform has also increased during these years. This constitutes a first approach to assess the activity of the ETP. The increasing tendency observed is in line with the growth in the production and consumption of organic food in the EU, as Figure 4 shows. From 2009 to 2016 (the last year for which data are available), the area dedicated to organic cultivation increased by 43%, from 8.3 million hectares (ha) to almost 12 million ha. In the same period, the retail sales of organic food increased from 16.9 billion euros to 30.7 billion euros, which represents a growth of 82%.

A more profound analysis of the pattern shown in Figure 3 provides some relevant information. Particularly, one can observe two periods in which the platform activity only increases slightly (2009 to 2013, and 2015 to 2017), while there is a sharp increment between years 2013 and 2015, a period in which the number of new documents related to TP Organics more than doubled. This period in which the expansion of the online activity of TP Organics is more pronounced took place at the same time that this platform was officially recognized as a European Technology Platform by the European Commission [7]. This was also influenced by the launch of its Second Research and Strategic Agenda under a larger and more mature structure than when the first one was launched.

5.2. Evolution in the Functions of the ETP

The documents related to TP Organics were classified into each of the six functions according to their content. To do so, each document was assigned to the predominant function, which is the one with the highest proportion of present keywords among the total number of keywords defining the function. Figure 5 shows the evolution of the number of documents that predominantly reflect each function. This chart reveals interesting insights into the activity of TP Organics. First, the variety of functions carried out by the platform increases gradually over time. As one can see, just two functions are predominant in the documents created in the year 2009. Another function appears as predominant in 2010, and the same happens in 2011 and 2012. This trend continues and it is only since 2015 that the six identified functions of the ETP are found to be predominant in the content of the documents related to the platform.

Second, this pattern can be seen as a reflection of the ETP lifecycle. During the first few years from its conception, the platform is commencing its activity and trying to acquire members and resources, so it is located at the first part of the learning curve. Therefore, it is focused only on few functions, especially on those that are more related to determining and settling the strategic perspectives of the platform, i.e., “demand articulation” and “capacity building”. The first is focused on articulating a research and innovation strategy involving the different actors of the system by means of developing a Strategic Research Agenda, which is a main objective of ETPs, while the second is focused on detecting knowledge and technology needs and promoting learning among its members and related community on organic food and sustainable agri-food systems.

After the initial stage, the scope of the ETP increases together with its knowledge and resources. Thus, the number of functions it can fulfill or support intensively is larger. In this regard, the platform incorporates a third function (“institutional support”) in its second year. This would indicate that the ETP was consolidated as a political tool. The third year incorporates the function “innovation process management”, which consists in coordinating the interaction among different actors of the system regarding research and innovation activities, particularly the proposal of projects under the topic of organic food and sustainable agri-food systems under schemes of European funding (FP7, H2020, and FP9). It is reasonable that this function appears in this moment, as it is not possible to identify the key research and development challenges and coordinate different actors to work towards them at least until a Strategic Research Agenda is settled (“demand articulation”), the most important knowledge and technology needs are detected (“capacity building”) and the political influence/role of the platform is developed (“institutional support”). However, “network brokering,” which is the next function to appear, comes unexpectedly late despite being one of the basic activities for which European Technology Platforms were established [47].

It is reasonable that the “knowledge brokering” function is the last one to appear, as it is only possible to spread results of the ETP or inform about the organization and participation in relevant activities after some years of work. However, we should point out that the frequency of documents in which this function is predominant is not as important as expected. The reasons behind this should be explored in future works. Despite this, the way in which the different functions appeared over time is generally consistent with what was expected.

Finally, the order in which the different functions appear over time reflect that, regarding the co-evolutionary approach to innovation, the hardware (technological innovation) and orgware (organizations and institutions) dimensions are represented since the establishment of the platform, while the software dimension (culture, thinking, and learning) is not represented until the platform has been working for some years, when the “network brokering” function appears.

The different stages that TP Organics went through are graphically represented in Figure 6. These stages are summarized into first, the establishment of the platform; second, its growth; and finally, its maturity.

5.3. Geographical Penetration of the ETP

Figure 7 includes two heat maps that represent the frequency of documents mentioning each country in the European Union. The heat map on the left represents the frequency of documents from the years 2009 to 2013; while on the right, the period considered was from the years 2014 to 2017. The values are standardized to a scale of 0 to 1, where 0 represents no documents available and 1 represents the highest frequency in the sample.

As can be seen, the activity of TP Organics has generally increased in the second time period with respect to the first in all countries. This is consistent with the previous results and in line with the ETP lifecycle: in the initial years, the platform has to grow, acquire members and position itself in the sector, so that its impact grows over time and is expected to be higher, as it is, once it has been working some years. It can also be seen that the leading countries, in which the presence of TP Organics is higher, are France and Germany, followed by Belgium and Italy. In the rest of the countries, the impact is lower. This is in line with the origin of the main members of the platform, whose headquarters are established in those countries (for instance, IFOAM and Act Alliance EU are located in Belgium, while Naturland e.V. is located in Germany and FiBL has some offices in France, Belgium, and Germany).

In general, the impact of TP Organics in the European Union increases over time, although there are slight differences in the evolution when looking at each particular country. The geographical spread of TP Organics tends to follow the extension of the total organic area in the EU Member States (see Figure 8). Although, the platform’s incidence tends to concentrate on Western Europe and with some leading countries in organic areas, such as Italy and Spain, not appearing as leading Member States in the TP. The dominant role of France and Germany is maintained in both periods, while in the case of Sweden and Finland, in which the impact of the ETP is similar in the first period, the evolution over time is different as in the second period the impact in Sweden is higher than in Finland (as the orange color reflects by contrast to the greener tonality).

6. Concluding Remarks

The technique shown in the present paper makes uses of online data collection to monitor and evaluate the performance of technology platforms. The method focused on European Technology Platforms, in particular, onTP Organics, a platform that promotes knowledge related to the future challenges of sustainable agriculture. We have not participated in the controversy on the future role of organic agriculture but we recognize that it offers a technology package, which surely has significant room for development, from the viewpoint of yields and environmental benefits per unit of product. Whatever the benefits organic agriculture show in the present and in the future, a great deal of knowledge and partnership will be required through the contribution of technology platforms.

While standardized methods to evaluate technology platforms are not generally agreed, the framework presented here is based on functions and on a co-evolutionary approach that, combined with web data mining techniques, provide the basis for a timely assessment of technology platforms.

The method exploits a number of possibilities that, in the explorative approximation followed in this research, open the chance of monitoring the TP activities, their results, their geographical dissemination, and their development through time. In the case of TP Organics, a lifecycle has been suggested with three different stages (establishment, growth, and maturity) with a varying relevance of the six innovation functions defined. With the proposed method, we are able to assess the impact of the platform regarding each of its functions, as well as over time and per country. The method makes use of publicly accessible and dated information and allows for a promptly update, very suitable to assess the technological platforms by replacing or complementing other methodologies of evaluation based on direct surveys.

Three areas of research will continue the present effort. The first area is the use of online data processing not only for monitoring but also for forecasting. This direction can facilitate policy analysis of the present trends and can help to foresee future directions of innovation activities and policy-making, whether applied to one technology platform or to a group of them.

A second direction is the use of the proposed monitoring technique to compare the evolution of different TP that could enrich the scope of innovation policies applied to sustainability. We acknowledge the scaling up effort that will imply the definition of a standardized and common methodology to describe functions for various TP. If there are several technological solutions to the problems of sustainability in the agri-food system, we would like to assess the comparative performance of the related TPs.

A third direction is the reformulation of TP functions, not according to those adapted to innovation intermediaries (as was the approach followed in the present study) but to specific topics related to the main subject matter of the selected TP. Thus, it is ensured that TP Organics and other platforms for sustainability fulfill sets of functions related to climate adaptation and mitigation, resilience, vulnerability of agricultural systems, and other functions that will require a definition in terms of keywords and indicators to further evaluate TP performance.

Our work is not without limitations. On the one hand, our analysis only focused on PDF documents, thus leaving a high quantity of information out of reach of the contributed method just because it was published in other formats. On the other hand, the results are subject to certain English bias, as the list of keywords is mainly composed of terms in this language.

Although the results presented here are encouraging, there is still room for improvement. There are alternative ways to solve the problem of documents involving several ETP functions that could be explored in the future. This could be addressed by employing more advanced techniques, such as fuzzy clustering or with a term frequency–inverse document frequency (tf-idf) scheme. A more precise document classification would allow for a deeper understanding of the evolution of the platform.

In summary, this article has introduced a new tool for TP assessment with a wide range of possible applications for policy monitoring and evaluation.