Stacking Machine Learning Model for the Assessment of R&D Product’s Readiness and Method for Its Cost Estimation

Abstract

The modern technology universities have the necessary resource and material base for developing and transferring R&D products. However, the cost estimation process is not formalized. There are many methods of estimating the cost of R&D products’ commercialization processes. However, in some cases, we cannot consider any single technique to be the best one as each of them has advantages and disadvantages. In such conditions, all efforts should be made to use a combination of the estimation techniques to arrive at a better cost and quality estimate. The effectiveness of the valuation of R&D products is of particular importance in today’s economy and due to the need to analyze large data sets prepared for transfer from universities to the business environment. This paper presents the model, two methods, and general information technology for R&D products’ readiness level assessment and R&D products’ cost estimation. The article presents the complex method for determining the cost of R&D products, which will allow: increasing the efficiency of the transfer, commercialization, and market launch of R&D products, and promoting the interaction of all components of the national innovation infrastructure, innovations, etc. The need to consider many different indicators when evaluating R&D products has determined the need to use machine learning algorithms. We have designed a new machine learning-based model for the readiness assessment of R&D products, which is based on the principle of “crowd wisdom” and uses a stacking strategy to integrate machine learning methods. It is experimentally established that the new stacking model based on machine learning algorithms that use random forest as a meta-algorithm provides a minimum of a 1.03 times higher RMSE compared to other ensemble strategies.

1. Introduction

The rapid pace of technological development in the world, due to the influence of the IV Industrial Revolution and the globalization of the world economy, identified the need to produce new approaches to managing the generation, transfer, and commercialization of R&D products and their economic evaluation. The shortening of the innovation life cycle and the spread of market effects from R&D products (spillover, convergence, diffusion, etc.) indicate that the product should be evaluated not only when it is ready but also in the initial stages of readiness. In particular, the economic evaluation of an R&D product at the idea stage can help predict product development and answer questions about the technical practicability of the further development of this product and the economic meaning of its market launch [1]. However, the lack of existing methodological support for the transfer and commercialization of R&D products based on their readiness does not allow market demands for R&D products to be met quickly.

At the same time, the active promotion of the paradigm of open innovation [2,3] by the community of developed countries has contributed to the fact that the crucial role in the processes of the transfer and commercialization of R&D products belongs to universities [4,5]. For the most part, modern technology universities have the necessary resource and material base for developing and transferring R&D products, technology transfer centers, etc. Based on the efficient transfer and commercialization of R&D products, universities can ensure the country’s long-term technical and economic growth. The importance of the above is evidenced by the conclusions set out in many analytical documents of the World Economic Forum (WEForum, 2020–2021).

This highlights the need to create methodological support for the transfer and commercialization of R&D products based on their readiness, from universities to the environment. An essential task of the effective development of evaluation methods and models for R&D products is to consider both the peculiarities of the readiness of technologies and the market considered for their commercialization.

In our opinion, a practical approach to the economic evaluation of product R&D based on its readiness is to create a model that contains a system of interactions between the key indicators of the readiness and the market perception of the product. Such interactions are characterized by a complex level of correlation, which depends on internal factors (the process of product development at the university) and external (state, trends, and market development patterns). The model of such an assessment is designed to answer questions about: the range of possible product prices, market launch scenarios, and the type of market coverage strategy, diffusion, and market behavior of the product, and so on. These and other economic categories should be determined by methods that are organically combined in this model.

From a scientific and practical point of view, R&D products’ readiness, as a basis for estimating their value, is the subject of much debate. Several researchers in the field [6,7] consider readiness from the technological maturity of products. Others [8,9,10] propose substantiating the readiness indicator by modeling the achieved level of satisfaction of R&D products with market needs (marketing, legal, social readiness, etc.). There are methodological approaches to determining the readiness of R&D products as objects of intellectual property rights [11,12]. Each of these approaches is designed to establish the level of readiness of the R&D product in specific market conditions and a certain period of economic evaluation.

However, virtually none of the current developments reflect the relationship between the availability and cost of R&D products. It causes significant difficulties during their transfer from universities to the business environment and further commercialization. Taking into account the level of readiness during the valuation allows you to justify the price of R&D of the product. For example, the readiness of the product at low levels (idea, product concept, etc.) will lead to a lower cost of the R&D of the product compared to its readiness at high levels (prototype, production preparation, etc.). From a market standpoint, the higher the product’s technological readiness level, the lower its transfer and commercialization risk.

An economic evaluation of product R&D, aimed at maximizing the various factors that take place in its development and taking into account the cost–income indicators of product R&D, is relevant in much of the work of scientists and practitioners. There are different points of view and solutions to this problem. For example, an R&D product’s assessment can be performed to achieve a set of objectives. Thus, the program and the evaluation mechanism are considered strategic tools for improving activities’ efficiency [13]. Evaluation objectives may include:

• communication studies between R&D spending and the market price of Thai corporate common share [14];
• to explore the strategic entanglements of financial models for managing R&D and building a firm’s competitiveness [15];
• to investigate the relationship between manufacturing–R&D integration and organizational culture in improving quality and product development performance [16];
• to obtain fitter decisions concerning risk reduction and further assist them in reaching higher performances in R&D partnership risk management [17].

In [18], under the ICAPM framework, the authors have proposed that an R&D factor is a proxy for innovations to a state variable.

However, the existing methods that take into account cost and revenue do not always provide satisfactory results regarding the desires of the dynamic market. In particular, the current developments of scientists and practitioners in economics and related fields have not solved these problems:

• the relationship between the cost of product R&D and such essential elements as the level of its technological readiness (TRL), analytical readiness (ARL), consumer readiness (CRL), and patent readiness (PRL);
• creation of a basis for the development of R&D of the product’s commercialization scenarios under different conditions of its readiness and transfer options;
• development of an intellectualized approach to product R&D evaluation, which can take into account both product features and the specifics of the market environment.

The considered difficulties cannot be solved purely analytically. Such tasks require a thorough formalized, algorithmic, and programmatic rationale, which involves establishing relationships between R&D product indicators and their level and nature. At the same time, a significant difficulty is the economic evaluation and combination of value and cost indicators in one system. It should also be borne in mind that the synergistic nature determines the value of the R&D product: the specific level of readiness is taken into account; each component should affect at this level the total cost of the product with a certain weight. At the same time, the combined effect of these components will have a significantly greater impact than each component alone. The importance of considering these elements in evaluating the R&D of the product necessitates the development of new practical tools that can ensure that evaluators obtain adequate results.

The solution to this problem can be considered based on the application of machine learning algorithms. That is why the possibility of using machine learning methods for R&D products’ evaluation is analyzed.

The authors in [19] use the Bayesian belief network model for the prediction of an R&D project’s success. They built a risk quantification model and used it for the prediction of the failure risk probability of R&D projects.

In paper [20], biopharmaceutical R&Ds only are taken into account. The authors formed 123 key R&D risks and grouped them into five R&D value chain segments and 27 respective process domains.

The Cronbach alpha reliability test is used in the paper [21]. In addition, a multiple regression model is built. The paper [22] presents an approach for a customer-perceived value investigation using the structural equation model and opinion mining.

The authors in [23] focused on software cost estimation. They used an empirical approach for this. However, the mentioned method cannot be used for other R&D evaluations because it considers specific software characteristics.

Thus, the literature analysis provides a review of the current methods to estimate the cost of R&D products, taking into account the singularities of market changes and the growing strategic role of the university in the region’s innovation infrastructure. The limitations of the current methods are:

(1) The models support the analysis of particular types of R&D products [20,23]. That is why they can be used only for a specific domain.

(2) The Bayesian belief network model and risk assessment [19] require numerous probability estimation datasets. It is impossible to use for universities’ R&D products due to a limited number of collected surveys.

(3) Regression [21] is widely used for cost prediction. However, comparing the resulting predictive accuracy with other predictors would be interesting.

(4) Feedback analysis and opinion mining [22,24,25] are mainly used for quality evaluation. Unfortunately, for R&D product costs, initial information about possible users’ feedback is usually absent.

(5) The existing studies conducted surveys with global companies and made an empirical examination [26,27,28,29]. However, there is a lack of investigations into relationships between technology commercialization capabilities, type of business, sustainable competitive advantage, sector, industry, etc.

(6) The econometrics models [30,31] can be used for R&D indicators’ evaluation. They allow finding the relationship between the cost of R&D products and the level of their technological readiness. However, the biggest problem with R&D product cost estimation is that it is necessary to combine different approaches, not only parameters’ estimation.

The paper aims to develop a new machine learning approach for R&D products’ technological readiness estimation to provide high prediction accuracy. The level of analytical readiness, technology’s influence level, developers’ parameters, the direction of technology for the consumer, etc., should be considered. Moreover, the influence of different parameters on the technology readiness level and R&D product cost should be evaluated.

The main contributions of this paper are the following:

• we have collected the dataset of R&D products and their parameters based on the expert survey, which provided the opportunity to apply machine learning methodology to reduce time and resources during the assessment of readiness and cost estimation of R&D products;
• we have designed a new machine learning-based model for the readiness assessment of R&D products, which is based on the principle of “wisdom of the crowd” through the use of a stacking strategy with the ensembling machine learning methods that provides an opportunity to improve the accuracy for significantly solving the stated task;
• we have designed a comprehensive method for R&D products’ cost estimation, which, by taking into account the results of the model for the readiness assessment of R&D products, as well as the availability of analogs on the market, allows us to increase the accuracy and reliability of the evaluation results through combinations of cost, revenue, and competition pricing approaches;
• we have developed intelligent information technology that provides an automatic assessment of the readiness and cost estimation of R&D products through the implementation of the above model and method, which allows for forming effective scenarios for the commercialization of such products.

The research methodology is built as follows:

• dataset collection;
• R&D level assessment model development for readiness level;
• cost estimation method development;
• results evaluation;
• system architecture development;
• system development and testing.

The practical value of the proposed methods and models for universities is given below:

• they provide an opportunity for university structures involved in the transfer of R&D products (technology transfer centers, science parks, startup schools, and other innovation entities) to assess the economic feasibility of the product in the early stages of its readiness, which will help reduce the level of risks in the transfer and commercialization of products;
• they apply the author’s development in the educational processes of various specialties and educational and scientific programs of educational institutions;
• they promote sound pricing of R&D products based on a variety of product impact factors;
• they substantiate the strategy of transferring R&D products from universities to the business environment, strategies for their commercialization, etc.

2. Materials and Methods

2.1. Dataset Collection

The research was conducted using case study methodology, modern theories, and data analysis practices in economics (mainly using information-receptive, reproductive, morphological, and heuristic).

Dataset was collected based on pooling results. The polling place was the research laboratories at Lviv Polytechnic National University and Startup school; the polling time was 2019–2021. The sample consists of 56 respondents. Therefore, dataset instances present results of R&D products and startups.

The research tool was the survey, which consisted of 16 concerns of a substantive nature and four questions of a personal character.

The structure of dataset looks like the following (Table 1):

Table 1. Dataset description.

Attribute Title	Attribute’s Value Type
Readiness	num (target attribute)
The level of analytical readiness	num
The patent level	num
The demand readiness level	num
The society impact level	num
Developer’s age	int (categorical)
Influence level	int (categorical)
Wide usage level	int (categorical)
Technological complexity	int (categorical)
Area of usage	int categorical)
The part of market	int (categorical)
Novelty level	int (categorical)
Education level	int (categorical)
Scientific level	int (categorical)
Level of knowledge usage	int (categorical)
Type of scientific research	int (categorical)
Social group	int (categorical)
Direction of technology for the consumer	int (categorical)
Direction of action	int (categorical)
Value	int (categorical)
Innovative level	int (categorical)

The first four features were evaluated; the rest of the features were categorical. That is why they were transformed using one-hot encoding. In total, 256 features were taken into account.

Descriptive Statistics and Correlation Matrix Are Given in Appendix A and Appendix B.

The methodology for R&D readiness components’ evaluation is presented in our previous work [32]. Dataset was divided by training set and testing set in proportion 75% and 25%, respectively.

TRL, ARL, PRL, and CRL were influenced by expert evaluation. In total, 23 experts estimated the importance of each parameter. The results show the significance of consumer readiness level considering the readiness of the technology. The higher the CRL, the more likely the successful commercialization process. The readiness level of R&D depends, in particular, on the experience of potential consumers and the possible benefits of using this product in real terms.

2.2. Assessment Model Development

Product R&D readiness assessment is performed according to expert evaluation. If an expert is alone, this may add subjectivity to such an assessment, which will further affect assessing the cost of a product [33]. This shortcoming can be remedied by the construction of a product R&D readiness assessment process by several experts [34]. Obviously, more experts provide more different opinions, which can then give a final result using majority voting. However, this approach requires much more financial cost, in particular in the form of a reward for all experts.

In general, the above approach can be considered from the point of view of Condorcet’s jury theorem [35,36]. Here the majority of votes form the initial result. However, an essential condition is the independence of experts. Only in this case is it possible to achieve the desired result. To do this, you can carefully select experts, which requires a lot of time, or weigh up the examination results, with the involvement of a meta-expert, who will set the coefficients of importance for each expert from the group [37], etc.

If there are historical data, or new data as a result of the examination are collected, it can be possible to avoid both of the above shortcomings by using ensembles of machine learning [34]. This strategy assumes that different machine learning methods, or weak predictors, act as each individual expert. In addition, there is a general meta-algorithm that weighs the results obtained by all weak predictors and gives the final decision [38].

The scientific literature considers three main methods of creating ensembles—boosting, bagging, and stacking [39]. To create the most accurate machine learning-based model for solving the problem of the readiness assessment of R&D products, as a regression task, we will create and investigate the effectiveness of each of them.

In the first stage, the weak predictors were selected and trained. Multivalued linear regression, k-nearest neighbor classifier, and support vector machine models were built.

The primary purpose of regression analysis is to determine the relationship between a certain characteristic Y of the object and the values of x1, x2, …, xn, which cause the change in the variable Y. Y is called the dependent variable, and the variable effects x1, x2, …, xn are called factors. Establishing a model, determining the form of regression (comparison), and estimating its parameters is the task of regression analysis.

In the regression analysis, a model of the form Y = φ (X) + ε is investigated, where Y is the resulting feature, X is a factor, ε is a random variable that describes factors x from the regression line (residual variable) [21]. The regression equation is given as: y(x) = φ (x, b0, b1 … bp), where x is the value of X; b0, b1, …, bp are the parameters of the regression function φ. Thus, regression analysis is present in certain functions, parameters, and statistical-level studies.

The k-nearest neighbor (k-NN) method is a metric algorithm for automatically organizing objects. The main principle of the nearest neighbor method is that the object is assigned to the class that is most common among the neighbors of a given element. Mathematically, the classification using k-NN is reduced to the calculation

$$CS{V}_i\left(d_j\right)={\displaystyle \sum }_{d_z\in T{R}_k\left(d_j\right)}RSV\left(d_j,d_z\right)\cdot c{a}_{iZ}$$

(1)

where CSV is the categorization status value of the object $d_j$, Tr_k(d_j) is the set k of objects $d_z$, for which we achieve the maximum of $RSV\left(d_j,d_z\right)$,$RSV\left(d_j,d_z\right)$ (retrieval status value) is the similarity measure between training dataset d_j and object $d_z,$ $c{a}_{iZ}$ is the value of the target attribute, k is the threshold (number of objects) indicating how many similar objects have to be considered to calculate $CS{V}_i\left(d_j\right)$. Any similarity function, either a probabilistic or a vector measure, can be used for these purposes.

The following data prediction method is the support vector machine, SVM [33]. The mathematical formulation of the classification problem is as follows: let X be the space of objects (for example, Rⁿ), Y be our classes (for example, Y = {−1, 1}). Specified training sample: you need to construct a function F:X→Y (classifier) that maps the class y of the object x.

The classification function F takes the form

$$F\left(x\right)=sign\left(\left(w,\varphi \left(x\right)\right)+b\right)$$

(2)

Positive certainty is necessary for the corresponding Lagrange function in the optimization problem to be limited from below, i.e., the optimization problem would be correctly defined [31]. The accuracy of the classifier depends, in particular, on the choice of the kernel.

R-squared Error (Rsquared), MAE (Mean Absolute Error), and RMSE (Root Mean Squared Error) [34] are used for prediction accuracy estimation.

The result obtained on the testing dataset is given in Table 2.

Table 2. Results of weak predictors.

Model	MAE	RMSE
Linear regression	0.1186738	0.1497206
k-nearest neighbor, n = 5	0.2039549	0.2020502
Support vector machine, Radial Basis kernel	0.105906	0.1193939

For R&D readiness level assessment, an ensemble of machine learning methods is used [35]. First, multivalued linear regression with the random forest is organized in boosting ensemble. Each time a base learning algorithm is applied, it generates a result of a new weak prediction. It is an iterative process. After multiple iterations, the boosting algorithm combines these weak results into a single strong prediction result. Random forest was used in boosting ensemble with the following hyperparameters:

• number of variables randomly sampled as candidates at each split mtry = floor(sqrt(ncol(x))) = 16,
• number of trees ntree = 500.

Parameters tuning is not used.

Since random forest based on ensemble learning requires a lot of decision trees to obtain high performance, it is not suitable for implementing the algorithm on limited computation resources. Here, we propose a boosted random forest in which boosting algorithm is introduced into the random forest. From the original random forest fit, we extract the residuals and then fit another random forest to these residuals. We call the sum of these two random forests a one-step boosted forest.

Boosted linear regression (lm) is an iterative method that starts with a base linear model and explains the model’s errors through regression trees.

The results of boosting are given in Figure 1.

Figure 1. The predictive accuracy of boosting.

The next ensemble is bagging. A bagged regression tree (CART algorithm) and bagged random forest are built. Bagging is used with decision trees, where it significantly raises the stability of models in improving accuracy and reducing variance, which eliminates the challenge of overfitting.

Bagged random forest is an averaging method that aims to reduce the variance of individual trees by randomly selecting many trees from the dataset and averaging them.

The results of bagged models for the testing dataset are given in Figure 2. The predictive accuracy is closed to boosted rf.

Figure 2. The predictive accuracy of bagging.

Next, we present a more promising model according to the stated task, the stacking machine learning model. Stacking is an ensemble machine learning algorithm that learns how best to combine the predictions from multiple well-performing machine learning models. A majority vote or weighing can combine basic training inputs. Additional data for retention are required if meta-learning parameters are used. It also increases the complexity of the model.

The new stacking model $s_K$ based on machine learning algorithms that use random forest as a meta-algorithm is proposed.

The mathematical formulation of the proposed stacking is the following. We have K cross-folds randomly generated from initial dataset

$$\left\{z_1^1,\dots z_B^1\right\},\left\{z_1^2,\dots z_B^2\right\}\dots ,\left\{z_1^K,\dots z_B^K\right\}$$

where K is the number of folds, B is the size of fold, $z_b^k$ is the b-th observation of the l-th sample.

The task is to train K independent weak regressors

$$w_1(.),w_2(.),\dots ,w_K(.)$$

(3)

and combine the results of training using meta-model $mw$

$$s_K(.)=mw\left(w_1(.)\times w_2(.),w_1(.)\times w_3(.),\dots ,w_{K-1}(.)\times w_K(.)\right)$$

(4)

where $w_i(.)\times w_j(.)$ is the pairwise multiplication of weak predictors’ results.

The main disadvantage of the stacking model is that the meta-attributes on training and the test are different. The meta-attribute in the training sample is not the answers of a particular regressor; it consists of pieces that are the answers of various regressions (with different coefficients), and the meta-attribute on the control sample, in general, is the answer to a completely different regression, tuned to the full training. In classic stacking, situations can arise when a meta-attribute contains few unique values, but many of these values do not intersect in training and testing.

The developed stacking model also combines linear regression, k-nearest neighbors, support vector machine with radial basis function, support vector machine with a linear function as weak predictors. In addition, the meta-features are deformed based on the results of pairwise multiplication. The meta-features are the results of weak predictors’ training. In the end, contorted features are used together with the training dataset in the meta-model. This combination avoids the correlation of weak predictors’ results and increases the model generalization.

The general schema of the proposed new stacking model is given in Figure 3.

Figure 3. The stacking schema. On ox-axis, we have different colored columns: yellow, green and blue columns indicate the results made by weak predictors, dark blue, orange, violet columns indicate the results of pairwise multiplication.

The realization of the new proposed stacking ensemble is given below.

In the first step, R-squared error for weak predictors was found. R-squared error is given below (Table 3):

Table 3. The main statistical indicators of the results of weak predictors.

Statistical Indicators
Weak Predictor	Min	1st Qu.	Median	Mean	3rd Qu.	Max.
rf	0.2717025 × 10−3	0.5283736	0.8681555	0.7250325	1	1
lm	0.1335612 × 10−4	0.2026962	0.7689927	0.5788250	1	1
k-nn	0.2583209 × 10−4	0.5344081	0.9525670	0.7501362	1	1
svmRadial	0.2112816 × 10−4	0.5117884	0.8886191	0.7295303	1	1
svmLinear	0.1894514× 10−5	0.342166	0.9491874	0.6846886	1	1

The multilayer perceptron (MLP) was used with Grid Search for hyperparameter tuning. It had two parameters to tune, the activation function, and the number of neurons in the hidden layer. Only one hidden layer was chosen due to limited dataset size.

Each example assumes that we are interested in the predictive accuracy as the metric we are optimizing, although this can be changed. Moreover, each example estimates the performance of a given model (size and k parameter combination) using repeated n-fold cross-validation, with 10 folds and 3 repeats.

Multilayer perceptron with sigmoid activation function was created with different number of neurons in hidden layer (Table 4):

Table 4. Errors values for different number of neurons in hidden layer of MLP with sigmoid activation function.

Size	RMSE	Rsquared	MAE
3	0.2223931	0.6517460	0.1977703
5	0.2449432	0.7053856	0.2065954
7	0.2710932	0.7192740	0.2174573
9	0.2611066	0.6925929	0.2084706

RMSE was used to select the optimal model using the smallest value. The final values used for the model were size = 3.

Next, MLP with hyperbolic tangent was investigated (Table 5):

Table 5. Errors values for different number of neurons in hidden layer of MLP with hyperbolic tangent activation function.

Size	RMSE	Rsquared	MAE
3	0.2286407	0.6102281	0.1943607
5	0.2073460	0.6470182	0.1766673
7	0.2074397	0.6457215	0.1767149
9	0.2088174	0.6449099	0.1760879

The final values used for the model were size = 7.

The Rsquared error for both MLPs was less than for other models. That was why MLP was excluded from possible weak predictors.

Next, weak predictors were combined at the last stage using random forest. One hundred trees were built for RF with max depth equal to 8 (Table 6). Cross-validation was also used tenfold and repeated three times. Repeated K-fold cross-validation is technically used for small datasets’ validation. The advantage of this technic is the ability for parallelization.

Table 6. Random Forest result.

Number of Variables in Each Split	RMSE	Rsquared	MAE
2	0.1531979	0.6179473	0.1224977
4	0.1497206	0.5990950	0.1186738
6	0.1510873	0.5803320	0.1182399

RMSE was used to select the optimal model based on the smallest value. The final value used for the model was mtry = 4.

2.3. The Method for Cost Estimation of R&D Product

The developed model is used in the next step, particularly for R&D product cost estimation. Our previous work presents a theoretical background for cost estimation and the proposed triple model [32]. The cost estimation for the separated domains is also shown in [40,41,42,43,44,45,46].

This study is essential to evaluate the R&D product when concluding transfer agreements for R&D product commercialization. In general, all known factors in the traditional world approaches to pricing on R&D products can be divided into cost, revenue, and competition. The choice of valuation method depends on the characteristics of STD and valuation objectives.

Based on our previous work, the method for the cost evaluation of R&D products consists of two steps:

• The choosing of the evaluation method.
• The price estimation based on the chosen method or combination of methods. If more than one method is used, the possible price range is returned.

The research showed that, depending on the factors taken into account during the evaluation, it is reasonable to recommend applying one or another method for cost estimation [33]. That is why, based on the previously calculated level of readiness, the cost estimation process is organized using experts’ surveys for the following parameters (Table 7). All coefficient values are chosen empirically.

Table 7. Parameters for cost estimation for R&D products.

Parameters	Rule
competitive_method.analog_implementation_costs (Ia)	numeric, range [0..∞)
competitive_method.analog_quality_value (Pa)	numeric, range (0..1]
competitive_method.analog_support_cost (Sa)	numeric, range [0.. ∞)
competitive_method.k1 (innovation comparison)	numeric, range {1, 1.1, 1.15, 1.2, 1.25}
competitive_method.k2 (ecological parameter)	numeric, {0.6, 0.8, 1, 1.1, 1.3}
competitive_method.k3 (complexity of implementation)	numeric, {0.6, 0.8, 1, 1.1, 1.3}
competitive_method.k4 (support complexness)	numeric, {0.5, 1}
competitive_method.k5 (attractiveness of market conditions)	numeric, (0.8, 0.9, 1, 1.1, 1.2}
competitive_method.own_implementation_costs (Io)	numeric, range [0..∞)
competitive_method.own_quality_value (Po)	numeric, range (0..1],
competitive_method.own_support_cost (So)	numeric, range [0.. ∞)
competitive_method.parameters_count ${\sum }_{i=1}^n{q}_i=1$	array, max:5, min:1
competitive_method.analog_price (Price_a)	numeric, range [0..∞)
expensive_method.percentage_of_cost (PS)	numeric, gte:0, lte:100
expensive_method.sum.commercial_expenses (a1)	numeric, range [0..∞)
expensive_method.sum.defective_lose (a2)	numeric, range [0..∞)
expensive_method.sum.fuel_and_energy (a3)	numeric, range [0..∞)
expensive_method.sum.general_expenses (a4)	numeric, range [0..∞)
expensive_method.sum.other_production_expenses (a5)	numeric, range [0..∞)
expensive_method.sum.raw_materials (a6)	numeric, range [0..∞)
expensive_method.sum.returnable_waste (a7)	numeric, range [0..∞)
expensive_method.sum.social_events_deductions (a8)	numeric, range [0..∞)
expensive_method.sum.third_parties_production (a9)	numeric, range [0..∞)
expensive_method.sum.total_expenditures (a11)	numeric, range [0..∞)
R&D_readiness_level	numeric, gte:1, lte:11
revenue_method.discount_rate (Q)	numeric, range [0..1]
revenue_method.period.expected_cost (C)	numeric, range [1..5]
revenue_method.period.expected_price (P)	numeric, range [1..5]
revenue_method.period.licensor_percentage (∆)	numeric, range [0..1]
revenue_method.period.sales_volume (t)	numeric, range [0..∞)

The algorithm for R&D products’ cost estimation is presented in Figure 4. In this paper, we proposed to estimate the price depending on readiness level and analog availability. The minimum and maximum costs are proposed if more than one approach is used. An algorithmic implementation of this method is shown in Figure 4.

Figure 4. The proposed algorithmic realization of R&D products’ cost estimation method.

3. Results

3.1. Results of Investigated Ensemble-Based Strategies for the Creation of the Model for the Readiness Assessment of R&D Products

This section presents the results of comparing ensemble methods: boosting, bagging, and stacking, created in Section 2.2, for creating a high-precision model for the readiness assessment of R&D products. The results of the comparison are given in Table 8. The new stacking model is compared with a boosted random forest, boosted linear regression, and bagged random forest.

Table 8. The comparison of the best weak predictors and ensembles.

Model	Rsquared	MAE	RMSE
New Stacking model	0.9366	0.0559359	0.05898147
Boosted rf	0.7553046	0.1640238	0.1916724
Boosted lm	0.7217016	0.2720410	0.3206452
Bagged rtree	0.7043159	0.1870193	0.2257885
Bagged rf	0.7541548	0.1662005	0.1937453

The new stacking model allows the RMSE (Root Mean Squared Error) to be decreased 1.03 times compared to other ensembling strategies.

As can be seen from Table 8, as expected, the best results in terms of accuracy were demonstrated by the designed stacking machine learning ensemble for the readiness assessment of R&D products. Therefore it will be used as a base for developing intelligent information technology.

3.2. Assessment Model Development

Figure 5 shows the component diagram. Component “Project” consists of R&D products. The calculation is used for readiness level assessment. Component “User” means the storage of registered users. The component “Result” contains the numerical results of the evaluation. The component “Query” implements the regression coefficients values.

Figure 5. Component diagram.

Figure 6 shows the deployment diagram. The database server is responsible for data saving and management. The workstation is used for system interactions and data visualization. The web server is used for presentation layer realization and as an interface to the database.

Figure 6. Deployment diagram.

The database schema was developed to assess the readiness level of technologies for the transfer (Figure 7).

Figure 7. Database schema.

Table “Project” is used for project storing. Table “Parameters” consists of parameters for readiness level evaluation. In addition, the estimated coefficient of these parameters is stored. The estimation is built on linear regression.

Table “Project_parameters” is used for expert usage. The categorical variable value helps to estimate the importance of each parameter.

3.3. System Development and Testing

The system is implemented as a web-based interface [4].

Figure 8 presents the main webpage of the developed system. The list of R&D products is given in [1,5].

Figure 8. The main page of the information technology.

The system functionality is the following:

• Create project—create new R&D product (Figure 9),
  

  Figure 9. Webpage for new R&D product adding.
• View project—view the existing R&D product analysis (Figure 10),
  

  Figure 10. Webpage for product editing.
• Delete project—delete the current R&D product.

Next, the model features are given in the system. Figure 11 shows the web page for the parameters’ storing and editing. It is possible to add a new parameter or delete an existing parameter. In addition, we can change the coefficient values based on the results of model retraining.

Figure 11. Web page for model parameters.

Model retraining is implemented in a different place using RStudio. The model parameters are exported in csv-file and, after that, they are uploaded to the web service.

The proposed system combines the cost estimating methods of R&D products. Three external users (developer, customer, market expert) have access to the system. The system’s main tasks are to calculate the price using various approaches and evaluate the value obtained in general. The system architecture is presented in Figure 12.

Figure 12. Web page for R&D product cost estimation.

The proposed approach to assessing the level of readiness of R&D products for commercialization allows:

• determination of an integrated indicator of the readiness level of R&D products for commercialization, calculated based on the indicators’ aggregation for each block of the approach. This approach makes it possible to aggregate interdisciplinary positions in evaluating R&D products;
• assessment of the level of readiness of R&D products for a particular evaluation unit; analyzing the possibilities of the commercialization of R&D results in different variations of the ratio of readiness for the components;
• comparison of the levels of readiness of R&D products for commercialization when selecting projects for investment, as the obtained values of the integrated assessments of the readiness levels of R&D products are based on their feasibility study;
• application of the method when deciding whether to include R&D products in the entity’s assets.

4. Discussion and Conclusions

This paper presents the model, two methods, and general information technology for:

• R&D products’ readiness level assessment;
• R&D products’ cost estimation.

The developed R&D products’ readiness level assessment model is based on the stacking strategy of the combination of machine learning methods. This is due to the peculiarities of this task. First of all, the readiness of R&D products is assessed by independent experts, many of whom eliminate subjectivism and ensure optimal decision making through majority voting. All this corresponds to Condorcet’s jury theorem [36]. To avoid high financial costs for the work of experts, we have proposed a technical solution to this problem, which is to build a stacking ensemble of heterogeneous machine learning methods, the results of which are weighed by the meta-algorithm. In particular, the developed stacking model combines linear regression, k-nearest neighbors, support vector machine with radial basis function, and support vector machine with a linear function as the basic machine learning predictors. In addition, the meta-features deformation is added for problems with classical stacking avoidance. The meta-features are the results of weak predictors’ training. In the end, deformed features are used together with the training dataset in the meta-model. This combination avoids the correlation of weak predictors’ results and increases the model generalization. It allows the RMSE (Root Mean Squared Error) to be decreased a minimum of 1.03 times compared to other ensemble-based approaches.

The paper also presents a complex method for determining the R&D product’s cost, which uses the results of the model for the readiness assessment of R&D products, as well as the availability of analogs in the market, and in results provides:

• an increase in the efficiency of transfer, commercialization, and market launch of R&D products,
• promotion of the interaction of all the components of national innovation infrastructure, innovations, etc.

The developed approach can become the main lever for: when deciding on further R&D; the selection and substantiation of investment decisions on the results of R&D, which prepare for commercialization; the development of a pricing strategy, market launch, and further development of R&D products, etc. The proposed methodological support and information system for the transfer and commercialization of R&D products based on their readiness from universities to the external environment will allow:

• to carry out the operational transfer and commercialization of R&D products;
• to develop the policies of market pricing, giving opportunities to clarify the impact of components on the formation of value and, accordingly, the price of R&D products;
• to promptly respond to the market demands for innovation [2];
• to form the basis for the country’s successful technological and economic development [3].

From the economic point of view, the application of the proposed methods and models to assess the cost and readiness of the R&D of the product allows specifying such essential elements of the evaluation process as:

• determining the moment and nature of the added value of product R&D (based on the justification of the relationship between levels of readiness and market perception of the product);
• taking into account the dynamism and extractive nature of the R&D product;
• separating the elements in the R&D of the product, which will further contribute to its market convergence, multiplicity, synergy, diffusion, etc. Economic forecasting of the possibility of such effects at the evaluation stage will allow adjusting the price of the product;
• the value expression of tangible and intangible value (object of intellectual property rights) of the R&D product;
• establishing the level of economic feasibility of product transfer/commercialization;
• modeling consumer sensitivity to the purchase of R&D products.

From the standpoint of business practice in commercialization, there are numerous cost evaluation methods [47], but we cannot consider any single technique to be the best one. Each method has its benefits and drawbacks. To reach a better cost and quality estimate, efforts should be made to use a compound of the estimation techniques.

After analysis of the processes of the commercialization of R&D products, it is possible to identify at least four interrelated management pricing decisions, namely:

(a) establishing a system of indicators that affect the price of R&D products;

(b) determining the method of aggregation of unit indicators;

(c) determining the strength of the impact and the importance of indicators (groups of indicators) for participants in the pricing process;

(d) agreeing on the criteria and evaluations of the proposed R&D products between the parties to and setting a final price.

That is why future research will be focused on primary price identification determination. The dataset of the predicted price and real sold price of R&D products should be collected. The collected dataset is too small. That is why a specific method based on a hierarchical predictor is used for small dataset analysis. Five-fold cross-validation is used for results’ validation too.

One of the objectives of this study was to develop two software products, the purpose of which is to calculate the level of readiness of the result of R&D products for launch and quickly assess the indicative range of development costs. The use of these software products will be helpful in research incubators, Scientifics Parks, or other structures of the domestic innovation ecosystem. However, this implies the possibility of using the developed software in enterprises or organizations engaged in innovation.

For universities, the application of the author’s methods and models to assess the value of R&D products based on their readiness will contribute to:

• striking a balance between “technology push” and “technology pull” strategies for the activities of developers working in university structures;
• the substantiation and selection of potential commercially attractive R&D products at the idea stage;
• a significant reduction in the risk of transferring R&D products from universities to the business environment and their commercialization;
• elaboration of scenarios for the creation of companies such as “spin” (spin-off, spin-out), which are based on the results of the prospects of R&D products, obtained through the author’s approach to modeling the value and readiness of products;
• filling gaps in the predominantly low level of entrepreneurial knowledge and competencies of university developers (and, consequently, insufficient level of understanding of market needs and features of commercialization);
• the substantiation of business models of the transfer of R&D products in universities, etc.

In the macroeconomic context, the proposed author’s methods and models will increase the level of success of technological entrepreneurship in the country. The obtained methodological and practical results are characterized by duality. On the one hand, the author’s developments are valuable for universities when deciding on the transfer of R&D products to the business environment. On the other, they allow modeling of possible factors influencing the product’s external environment at the development stage. The proposed methods and models can be used to justify regional development strategies to help bridge the gap between universities and the market.

The limitation of the study is based on the insufficient dataset of R&D products. Due to the analysis of a short dataset, the ensembles were developed. However, the additional proving of the proposed models should be organized based on other datasets’ analysis. Future research will also be conducted in the area of applying neural network models [48,49,50] to build ensemble methods.