An Assessment Methodology about the Effectiveness of Mobility IT Solutions: Application to Six Demo Sites

Abstract

Considering the significant growth rate of populations in urban areas, public transport has become vital to urban living. Promoting the culture of Mobility as a Service (MaaS) among travelers has become unavoidable to address climatic challenges, particularly the global warming phenomenon. To encourage the use of public transport, it is important to introduce innovative IT solutions to the ecosystem of TSPs (Transport Service Providers) backed by an in-depth analysis to meet the expectations and the needs of the TSPs and the travelers. Building on the work from a previous paper, which introduced an assessment methodology based on the calculation of the effectiveness of IT solutions to meet traveler’s needs and expectations, this paper presents the results of an intersectional assessment by applying this methodology with data collected in six demo sites (Athens, Barcelona, Liberec, Osijek, Padua, and Warsaw). This paper will help to understand better this quantitative assessment methodology, and especially help IT developers and TSPs understand better how to apply it to their IT solutions. Results of this assessment show which of these IT solutions (functionalities) are worth investment, considering the satisfaction level of both TSPs and travelers. This assessment methodology is scalable to other demo sites and datasets in further developments.

1. Introduction

Nowadays, transport services, especially public transport services, play a vital role in every European society. Considering the drastic increase in air and noise pollution caused by GHGs (Green House Gases) and their side effects on every environment, investigating and studying traveler’s behavior in making decisions and understanding their needs and expectations to reduce pollution are undeniable facts for transport experts [1,2].

It is worth mentioning that the number of passengers using public transportation in 2020 and 2021 decreased by 40% to 70% due to COVID-19, the after-effects of which continue to affect the use of public transportation systems to date. With remote working becoming a norm, daily commuting has become less frequent in many countries [3,4,5,6]. In 2020, the declining percentage number of passengers resulted in an 11% decrease in transportation service supply compared to 2019, thereby causing heavy financial losses [7,8,9]. The drop in fare box revenue was anticipated to be 90%. Railways in the European Union lost 24 billion euros in revenue for passenger services in 2020, a 41% decrease from 2019 [10,11].

This paper aims to validate a methodological framework to quantitatively assess how innovative technologies can respond to the needs of travelers and Transport Service Providers (TSPs) involved in the digital ecosystem for door-to-door travel in Europe, thereby increasing the attractiveness of public transport. The framework was introduced in a previous paper [12]. The objective of this paper is to demonstrate the robustness of the methodological framework previously introduced [12].

This work uses a methodological approach to evaluate the needs of travelers with different socio-demographic profiles [13] and TSPs (Transport Service Providers) based on rail transport. It considers social trends like reducing Greenhouse Gas (GHG) emissions and road congestion [14].

The concept of the attractiveness of rail and public transport depends on complex psychological factors from a scientific and technical standpoint; The methodology used in this study consolidates the concept of “user profile” and the ability of the system to respond to the needs and expectations of users (TSPs and travelers), including socio-demographic-related factors such as aging, reduced mobility, and other specific conditions [15,16,17].

2. Literature Review and State of the Art

The Shift2Rail (S2R) Innovation Program 4 (IP4) is dedicated to creating a digital eco-system for door-to-door travel that enables a seamless and multimodal transport system across Europe based on railways [18,19]. The IP4 program aims to significantly transform the way individuals travel within Europe, making public transport and railways more appealing. It also addresses crucial societal trends like reducing greenhouse gas (GHG) emissions and mitigating road congestion [20,21].

To enable easier travel within Europe, S2R IP4 Call for Members (CFM) projects such as ATTRACkTIVE [22], CO-ACTIVE [23], MaaSive [24], and CONNECTIVE [25], and Open Call (OC) projects ST4RT [26], My-TRAC [27], SPRINT [28], and RIDE2RAIL [29] have developed a technical framework of advanced Information Technology (IT) building blocks that can be flexibly combined in multiple configurations into solutions that cater to diverse scenarios [30].

In this regard, IP4MaaS has taken up the challenge of combining all individual IT solutions developed in IP4 CFM and OC projects, consolidated by COHESIVE [31], into solutions for specific demonstration scenarios across multiple real environments in Europe. The IP4MaaS project aims to enhance the adoption of Mobility as a Service (MaaS) schemes by analyzing and testing the technologies developed under the Innovation Program 4 (IP4) of the Shift2Rail Joint Undertaking in six demonstrations across Europe, namely Barcelona, Athens, Warsaw, Osijek, Liberec, and Padua [32,33].

This paper introduces the methodology used for these assessments, and the results and discussions relevant to all of them [34].

As a first step, a literature review was conducted to evaluate the best assessment methodology. Zarehparast Malekzadeh et al. (2023) [12] present a methodology to assess and evaluate how innovative technologies meet the needs of tourists and transport service providers in Europe, using operational key performance indicators and user satisfaction surveys as the main inputs to feed the methodology. The authors apply the concept of effectiveness, which is calculated by merging both types of data, and use regression and Bayesian network analysis to study the correlations among variables. The methodology is applied to six demo sites in the IP4MaaS project.

Also, the combination of all methods (AHP, Bayesian Network analysis, Regression analysis, and ANOVA test) has shown exceptionally good results in previous studies where factors need to be hierarchized and some predictions need to be made. These previous studies can be consulted in the references [35,36].

On the other hand, Sostaric et al. (2021) [37] introduced “Data-Driven Methodology for Sustainable Urban Mobility Assessment and Optimization”. This study proposes a data-driven methodology for assessing and optimizing the sustainability of urban mobility, based on big data analysis of anonymized data sets from mobile telecommunication networks. The authors used the analytic hierarchy process (AHP analysis) to rank the alternatives of mobility solutions according to multiple criteria, such as accessibility, affordability, safety, and environmental impact. The methodology is applied to a case study of the city of Skopje, Macedonia.

In addition, Storme et al. (2021) [38] presented a critical review of the impact assessments of new mobility services, such as shared mobility, mobility as a service, and Mobihubs, in terms of social, environmental, and economic aspects of sustainable mobility. The authors review more than 100 sources from academic journals and media reports and explain the key elements and challenges of evaluating the impact of new mobility services on urban transportation systems and quality of life.

Furthermore, Doorn et al. (2021) [39] presented the article “The JASP guidelines for conducting and reporting a Bayesian analysis” which provides practical guidelines for applying Bayesian procedures and interpreting the results using the open-source statistical software JASP (0.10.2). The article covers four stages of Bayesian statistical reasoning: planning the analysis, executing the analysis, interpreting the results, and reporting the results. The article illustrates the guidelines with a running example and compares the Bayesian results with the classical results.

Moreover, Bergh et al. (2022) [40], introduced an article entitled “Bayesian Repeated-Measures ANOVA: An Updated Methodology Implemented in JASP” that presents an updated methodology for conducting Bayesian repeated measures ANOVA using JASP. The article explains the difference between the new default method and the previous method and shows how the new method brings the Bayesian results more in line with the frequentist results. The article also provides a case study on the Stroop effect and its interaction with breaks.

And finally, Hassan (2024) [41], presented “ANOVA (Analysis of variance)—Formulas, Types, and Examples”. This article provided an overview of ANOVA, a statistical method for testing the differences among the means of several groups. The article explains the basic concepts, formulas, types, assumptions, and examples of ANOVA.

It is worth mentioning that the methods, modules, and mathematical models used in this study, such as AHP, Bayesian Network analysis, regression analysis, and ANOVA test, are capable of analyzing quantitative data, hierarchizing factors, and making predictions. Therefore, they are considered very useful and valuable, which is why they have been utilized in this study.

This paper introduces a methodology that combines the strengths of previous approaches [12] to assess the acceptability of new IT solutions. By gathering both quantitative (operational KPIs) and qualitative data (User satisfaction surveys), this methodology provides a comprehensive metric on the benefits of IT solutions applied to public transport, with a special focus on railways. The ultimate goal of this study is to build equity and improve the lives of users by tailoring IT solutions to their specific needs and expectations. This new approach allows for real-world assessments that mirror reality and are not just theoretical analyses. It’s a revolutionary way of evaluating solutions that will ultimately lead to a better future for all.

3. Research Questions and Objectives of the Study

The purpose of the document is to show the results of the performance assessment implemented in the IP4MaaS project [34]. Specifically, the results have been collected to assess the positive impact encountered thanks to the development of the six demo sites and the use of IP4 technologies. Therefore, the overall performances against the defined project objectives and related indicators have been quantitatively assessed [34].

By applying five modules of the toolbox (AHP analysis [1], Regression analysis [2], Bayesian Network analysis [3], ANOVA test [4], USI travelers, USI TSPs, and effectiveness [5]) the following research questions will be addressed [42]:

-

What are the top 10 benefits among the IT innovations (functionalities) in the IP4 (Innovation Program 4) ecosystem that users value the most? (MODULE 1 to 3)

-

How do the socio-demographic characteristics and profiles of users influence their preferences and behaviors among tested IT innovations (functionalities) presented in the IP4 ecosystem? (MODULE 4)

-

How do the IP4 functionalities affect the satisfaction and performance of different types of users in terms of effectiveness taking into account the operational KPIs and USI surveys? (MODULE 5)

4. Methodology

The methodology employed in this study uses the concepts of:

• “User journeys”, as a travel solution from an Origin to a Destination in which a traveler may interact with an IT solution “j” offered by one or more Transport Service Providers (TSPs) “k”.
• “Demonstration scenario”, as an IT solution “j” offered to travelers by “k”.
• Sensitive profiles “r”, which were identified through a conversational survey and sentiment analysis.

This assessment methodology was introduced in a previous paper [12], and its innovation is based on a combination of operational key performance indicators (KPIs) and user satisfaction index (USI) surveys [43,44]. While the previous paper introduced the baseline assessment methodology and demonstrated it on one demo site, the second paper focuses on:

-

using the methodology from the first paper and validating it in more complex scenarios and parameters including more functionalities, more TSPs, and more socio-demographic profiles.

-

Automating the steps of the methodology by bringing together the mathematical models, equations, and snippets of scripts to work as a single toolbox and produce results in a user-friendly way.

-

Drawing comparisons among the demo sites to evaluate the most effective functionalities across all demo sites and presenting the results to inspire the use of the toolbox to more demo sites in the future.

In that paper [12], the assessment methodology was applied only to one European demo site (Athens) as a test. However, this paper provides a clear list of KPIs and USI questions that consider the requirements and expectations of TSPs and travelers with several sensitive profiles, such as the elderly, the disabled, and women, and the results of applying this assessment methodology to all six demo sites (Barcelona, Padua, Athens, Liberec, Osijek, and Warsaw) [34,45]. All the modules are prepared in a performance assessment toolbox that works sequentially. As a result, the main innovation added to the second paper is the presentation of the performance assessment toolbox, the application of the methodology to all six demo sites (Barcelona, Padua, Athens, Liberec, Osijek, and Warsaw) instead of only Athens demo site in the first paper [12].

This work assesses some IT functionalities developed within the IP4 (Innovation Program 4) [32], offered to users (travelers and TSPs) by a “Travel Companion” (TC) APP, and explains in detail how the effectiveness is calculated per each of these functionalities and each sensitive profile [33].

Three main data fed to the assessment of the Travel Companion APP:

• USI (User Satisfaction Index) travelers from online surveys.
• USI (User Satisfaction Index) TSPs (Transport Service Providers) from online surveys.
• Operational KPIs (Key Performance Indicators) from the TC APP running on the ground.

The assessment, alongside the general profiles of travelers using the TC APP, also considered four sensitive profiles obtained from a conversational survey and sentiment analysis [46,47]:

• General profiles (r = 1).
• Unemployed people, low-income people, retired people, and students (r = 2).
• Disabled or impaired people, people with physical or mental illnesses, people in wheelchairs, people with reduced mobility, people with visual impairment, and hearing impairment (r = 3).
• Elderly (r = 4).
• Women (r = 5).

The following steps summarize the assessment methodology and it is shown in Figure 1 [33]:

Step 1: Definition of the parameters for mathematical models used in the methodology.

The methodology introduces the definition of:

-

User journeys “i” for each demo site: defining the travel solution from origin to destination.

-

Demonstration scenarios “JK”: defining the scenarios for the demo site’s functionalities “J” and Transport Service providers “K”.

-

Sensitive profiles “r”: defining the sensitive and specific profiles of the study (disabled people, low-income people, the elderly, and women).

Several User Journeys “i” were defined per each demo.

Step 2: Identification of operational KPIs (“KPIs”) and benefits provided by functionalities (“j”) to these sensitive profiles (“Br”).

Only those operational KPIs that could be measured during the execution of these demos were considered, and benefits provided by these functionalities to these sensitive profiles were identified. Several focus groups and workshops were used for the identification of these benefits.

Step 3: Data collection of operational KPIs and satisfaction regarding benefits through USI surveys.

Per each demo site, quantifiable operational KPIs were collected, and USI surveys were filled by TSPs and Travelers to assess the benefits provided by each functionality “j” to each sensitive profile “Br”.

Step 4: Calculation of the effectiveness and comparisons among TSPs (“k”), functionalities (“J”), and profiles (“r”).

Operational KPIs and USIs were inserted as input into the effectiveness calculation, which is a metric on how IT solutions match the needs and expectations of travelers and TSPs, from the perspective of an aggregated analysis, by considering general profiles, and also per specific profiles of travelers (low-income people, people with disabilities, elderly, and women).

Step 5: Further data analysis.

To accurately analyze and assess the performance of the TC APP, an in-depth analysis using AHP, Bayesian network, Regression, and ANOVA test was conducted on the collected data of USI surveys and operational KPIs.

The next picture depicts this assessment methodology.

Figure 1. Overview of the assessment methodology of mobility IT solutions [33].

Figure 1. Overview of the assessment methodology of mobility IT solutions [33].

4.1. The Concept of Effectiveness

In this study, the effectiveness can be defined as a metric regarding how IP4 solutions match the needs and expectations of travelers and TSPs, from the perspective of an aggregated analysis and per each group of travelers in intersectional analysis [33,48].

The definition of each variable that is used in the calculation of the effectiveness is:

The User Satisfaction Index (USI) for travelers belonging to a profile vector “r” with the functionality “j” offered by the TSP “k” is calculated as:

$${\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{r}}_{\mathrm{r}\mathrm{j}\mathrm{k}}}=\frac{\sum _{\mathrm{w}=1}^{{\mathrm{m}}_{\mathrm{r}\mathrm{j}\mathrm{k}}}{\sum }_{\mathrm{v}=1}^{{\mathrm{n}}_{1\mathrm{j}\mathrm{k}}+{{\mathrm{n}}_{2\mathrm{j}\mathrm{k}}}^{\mathrm{r}}}{\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e} \mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{w}\mathrm{v}}}{{\mathrm{m}}_{\mathrm{r}\mathrm{j}\mathrm{k}}·\left({\mathrm{n}}_{1\mathrm{j}\mathrm{k}}+{{\mathrm{n}}_{2\mathrm{j}\mathrm{k}}}^{\mathrm{r}}\right)·5}$$

(1)

where:

${\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e} \mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{w}\mathrm{v}}$ = the score to the question “$\mathrm{v}$” by the respondent “$\mathrm{w}$”.

${\mathrm{n}}_{1\mathrm{j}\mathrm{k}}$= the number of questions applicable to all the profiles measuring the satisfaction with the functionality “j” offered by the TSP “k”.

${{\mathrm{n}}_{2\mathrm{j}\mathrm{k}}}^{\mathrm{r}}$ = the number of questions applicable only to the profile “r” measuring the satisfaction with the functionality “j” offered by the TSP “k”.

${\mathrm{m}}_{\mathrm{r}\mathrm{j}\mathrm{k}}$ = the number of respondents to the USI questionnaire belonging to the profile “r” measuring the satisfaction with the functionality “j” offered by the TSP “k”.

The satisfaction index for a TSP “k” regarding a functionality “j” is calculated as:

$${\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{S}\mathrm{P}}_{\mathrm{j}\mathrm{K}}}=\frac{{\sum }_{\mathrm{v}=1}^{{\mathrm{n}}_{\mathrm{j}}}{\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e} \mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{v}}}{{\mathrm{m}}_{\mathrm{j}\mathrm{k}}.{\mathrm{n}}_{\mathrm{j}}·5}$$

(2)

where ${\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e} \mathrm{q}\mathrm{u}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}_{\mathrm{v}}$ is the score to the question number “v”, “${\mathrm{n}}_{\mathrm{j}}$” is the number of questions in the USI questionnaire belonging to a specific functionality, “j” is offered by the TSP “k”, and $\mathrm{“}{\mathrm{m}}_{\mathrm{j}\mathrm{k}}\mathrm{”}$ is the number of respondents to the USI questionnaire measuring the satisfaction with the functionality “j” offered by the TSP “k”.

The answer to each question has a value between 1 (representing the minimum satisfaction) and 5 (representing the maximum satisfaction), which are normalized dividing by five, to obtain values between 0.2 and 1.

All this quantitative data (operational KPIs and USIs) is managed together within the concept of effectiveness. The effectiveness of functionality “j” offered by a TSP “k” for a specific profile “r” in a demonstration scenario “D” is calculated through the following equation. To avoid producing several equations for effectiveness per each group identified in the aforementioned section, a unique formula (Equation (3)) has been prepared and it is implemented for all the groups in this study:

$${\mathrm{E}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{s}\mathrm{s}}_{\mathrm{r}\mathrm{j}\mathrm{k}}=\frac{{\sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathrm{K}\mathrm{P}\mathrm{I}}_{{\mathrm{n}}_{\mathrm{j}\mathrm{k}}}+{\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{r}}_{\mathrm{r}\mathrm{j}\mathrm{k}}}+{\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{S}\mathrm{P}}_{\mathrm{j}\mathrm{k}}}}{\mathrm{N}+{\delta }_{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{r}}+{\delta }_{\mathrm{T}\mathrm{S}\mathrm{P}}}$$

(3)

The definition of the parameters introduced in the above equation (Equation (3)) are as follows:

$$\{\begin{array}{c}{\delta }_{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{r}}=0 \mathrm{i}\mathrm{f} {\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{r}}_{\mathrm{r}\mathrm{j}\mathrm{k}}}=0\\ {\delta }_{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{l}\mathrm{e}\mathrm{r}}=1 \mathrm{i}\mathrm{f} {\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{r}}_{\mathrm{r}\mathrm{j}\mathrm{k}}}\ne 0\end{array}\hspace{1em}\{\begin{array}{c}{\delta }_{\mathrm{T}\mathrm{S}\mathrm{P}}=0 \mathrm{i}\mathrm{f} {\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{S}\mathrm{P}}_{\mathrm{j}\mathrm{k}}}=0\\ {\delta }_{\mathrm{T}\mathrm{S}\mathrm{P}}=1 \mathrm{i}\mathrm{f} {\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{S}\mathrm{P}}_{\mathrm{j}\mathrm{k}}}\ne 0\end{array}$$

And,

“N” is the Number of operational dimensionless KPIs linked to the functionality “j” offered by the TSP “k” (N can be zero for some functionalities),

${\mathrm{K}\mathrm{P}\mathrm{I}}_{{\mathrm{n}}_{\mathrm{j}\mathrm{k}}}$ is the value of the KPI “n” belonging to the functionality “j” offered by the TSP “k”,

${\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{r}}_{\mathrm{r}\mathrm{j}\mathrm{k}}}$ is the value calculated in Equation (1), and

${\mathrm{U}\mathrm{S}\mathrm{I}}_{{\mathrm{T}\mathrm{S}\mathrm{P}}_{\mathrm{j}\mathrm{k}}}$ is the value is calculated in Equation (2).

Given that the effectiveness is dimensionless with a value between 0 and 1, the higher, the better, and different demonstration scenarios “D” can be compared to analyze how the needs of travelers in other locations or demo sites are matched by the same innovative technology “j” offered by different TSPs.

The effectiveness comparison can only be done after grouping based on what parameters are considered in the effectiveness formula: KPIs, USI Travelers, USI TSPs, or combinations among them [33,34].

The three elements in the numerator are summed in a linear way and with an equal weight because an innovative technology with no good operational KPIs, no good acceptance level by travelers, or no good acceptance level by the TSP would not be implemented in practice or would not remain in use for a long time, as it would therefore not be answering users’ needs.

4.2. Extension of the Methodology

This assessment methodology was extended by applying the next analysis methods:

• AHP analysis: Definition of a two-level hierarchical model (shown in Figure 2 and Figure 3) and a weighted hierarchy of TC APP benefits “Br” in the first level [49,50].

In the following Figure 2 and Figure 3, the hierarchical model for both travelers and TSPs is illustrated:

• Regression analysis: The regression analysis was introduced to define forced connections among “Br” and operational KPIs in the BN analysis.
• BN (Bayesian Network) analysis: Identification of correlations among benefits “Br” and operational KPIs in a Bayesian Network with a high likelihood, and definition of a weighted hierarchy of TC APP benefits “Br” in the second level by applying the Bellman shortest path [33,50,51,52,53].
• Impact assessment: Simulations for assessing the overall impact of an investment made on improving a certain benefit “Br” at a demo site. The methodology used in these predictive simulations is as follows:
  The top three second-level TC APP benefits “Br” were selected, and their original USI score was forced to 5 and the impact on scores of the other top three second-level TC APP benefits “Br” was predicted.
• ANOVA test (Analysis of Variance) for Travelers was applied in this analysis to determine if some socio-demographic profiles (age, gender, income level, residential area, traveling with a dependent person, professional status, disability, familiarity with technology) show significant differences regarding the satisfaction with second level benefits “Br” based on the data gathered through the USI travelers survey.

Figure 2. Hierarchical model for travelers.

Figure 2. Hierarchical model for travelers.

Figure 3. Hierarchical model for TSPs.

Figure 3. Hierarchical model for TSPs.

4.3. Making the Knowledge Actionable through an Assessment Toolbox

The performance assessment methodology of this study was made actionable through a toolbox based on several mathematical data analysis operations executed sequentially: AHP (Module 1), Regression Analysis (Module 2), Bayesian Network and impact assessment (Module 3), ANOVA test (Module 4), and calculation of the effectiveness (Module 5) [34].

This toolbox was programmed in Julia V. 1.7.0 and applied in the six demo sites: Athens, Padua, Warsaw, Liberec, Osijek, and Barcelona between March and June 2023.

The definition of each variable that is used in this assessment toolbox is:

• “r” is the type of profile of respondents in this study (r = 1 general profile vectors, r = 2 low-income people, r = 3 people with disability, r = 4 elderly, and r = 5 women).
• “J” is the name of innovative technology or functionality, the associated functionality of each functionality in each demo site is presented in Section 5 of this paper.
• “K” is the name of TSP (Transport Service Provider) which provides that specific functionality. The list of variable “K” for each demo site is presented in the following table (

  Table 1. Final list of TSPs “K” used in performance assessment [34].

  Associated Code “K”	Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site
  1	Athens	OASA
  2		MIRAKLIO
  3		Taxiway
  4		Brainbox
  5	Padua	Trenitalia
  6		BusItalia
  7	Liberec	KORID
  8	Warsaw	ZTM
  9		MZA
  10		TW
  11	Osijek	GPP PT
  12		GPP sharing mobility
  13	Barcelona	TMB
  14		BUSUP
  15		AMTU

  ):

• “q” is the associated question linked to that specific functionality in USI surveys.

On the other hand, the list of operational KPIs that were used in this assessment is listed in

Table 2. The final list of “Operational KPIs” associated with “J” was used in the performance assessment [34].

No.	Name of the Innovative Technology or Functionality	Unit	Associated Code “J” and “KPI”
1	LBE (Location-Based Experience)	Number of entertainment services offered during the demo	J8KPI0
2	JP (Journey Planning)	The average number of modes involved in the journey	J1KPI1
3	JP (Journey Planning)	The average number of shopped offers	J1KPI2
4	JP (Journey Planning)	Number of TSP integrated	J1KPI3
5	Booking	The average number of booked offers	J2KPI4
6	Issuing	Average Number of issued offers	J3KPI5
7	Mobility Packages	Number of mobility packages offered	J4KPI6
8	Guest user	Number of connections without passwords per day	J12KPI7
9	Asset Manager	Number of services integrated with the pilot	J23KPI8
10	Contractual management marketplace	Number of mobility packages handled	J25KPI9
11	Contractual management marketplace	Number of involved stakeholders	J25KPI10
12	Traveler’s feedback	Number/day	J10KPI11
13	Travel Arrangement	number per pilot	J21KPI12

:

An overview of the number of respondents per profile in each demo site is presented in

Table 3. Statistics of respondents to the USI surveys in each IP4MaaS demo site considering each socio-demographic group [54].

Demo Sites	Total Number of Respondents (r = 1)	Number of Low-income Profiles (r = 2)	Number of Disabled Profiles (r = 3)	Number of Elderly (r = 4)	Number of Women (r = 5)
Athens	38	6	-	-	21
Padua	13	8	-	-	6
Warsaw	208	41	6	8	69
Liberec	121	42	5	10	63
Osijek	40	13	-	-	14
Barcelona	11	-	1	1	4

:

5. Application of the Assessment Methodology to the Six Demo Sites

In this section, the actual results of each demo site are presented. The structure of the steps follow, common to all the demo sites, and only focuses on presenting the results later under the section of each demo site. In the first step, the results of “Module 1—AHP analysis” considering the hierarchical model for travelers and TSPs and the rank of each criterion (first level benefit) for Travelers and TSPs are shown. In the second step, the results of “Module 2—Regression analysis” are presented. This analysis shows the variables that are highly correlated (with a statistical p-value < 0.05). This means that improvement in the performance of the first variable will increase the performance of the second variable. In the third step, the results of “Module 3—Bayesian Network analysis” which is the output of BN analysis from each demo site indicating the most influent second-level benefits are demonstrated. In the fourth step, the results of “Module 4—ANOVA test”, which shows the socio-demographic profiles and the second-level benefits with significant differences regarding satisfaction, are introduced. Finally, in the fifth step, the main results regarding the assessment conducted by “Module 5—USI travelers, USI TSPs and Effectiveness” are presented sequentially [55].

5.1. The “Athens” Demo Site

The testing and execution of the TC APP in the Athens demo site was done from 27th to 31 March 2023. In total, four TSPs—OASA, MIRAKLIO, Taxiway, and Brainbox—were assessed in this demo site, and 38 responses were collected regarding USIs. The final list of “J” Innovative technologies or functionalities considering “K” as TSP, which offers each functionality for the Athens demo site, are presented in the following table (

Table 4. The final list of “J” Innovative technologies or functionalities considers “K” as TSP which offers each functionality for the Athens demo site.

Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site	Name of Functionalities (J) Assessed in Each TSP (K) for Travelers	Name of Functionalities (J) Assessed in Each TSP (K) for TSPs
Athens	OASA (K = 1)	Travel companion Web-Portal (J = 16), Guest user (J = 12), Preferences and Profiles (J = 13), Journey planning (J = 1), Intermodal Fare Optimization (J = 17), Issuing (J = 3), Mobility packages (J = 4), Validation and Inspection (J = 5), Navigation (J = 9), LBE (J = 8), Map Content (J = 19), Traveler’s feedback (J = 10)	Asset manager (J = 23), Travelers Orchestration and supervision (J = 29), Specific messages (J = 33), Distributed Ledger—Transaction Anchoring (J = 30), Distributed Ledger—TSP Inclusion (J = 31) Asset manager (J = 23), LBE editor (J = 24), Travelers Orchestration and supervision (J = 29), Specific messages (J = 33), Distributed Ledger—Transaction Anchoring (J = 30), Distributed Ledger—TSP Inclusion (J = 31)
	MIRAKLIO (K = 2)	Travel companion Web-Portal (J = 16), Guest user (J = 12), Preferences, and Profiles (J = 13), Journey planning (J = 1), Navigation (J = 9), LBE (J = 8), Map Content (J = 19), Traveler’s feedback (J = 10)
	Brainbox (K = 4)	Travel companion Web-Portal (J = 16), Booking (J = 2), Guest user (J = 12), Preferences and Profiles (J = 13), Journey planning (J = 1), Intermodal Fare Optimization (J = 17), Issuing (J = 3), Mobility packages (J = 4), Validation and Inspection (J = 5), Navigation (J = 9), LBE (J = 8), Map Content (J = 19), Traveler’s feedback (J = 10)	Asset manager (J = 23), Contractual management Marketplace (CMMP) (J = 25), LBE editor (J = 24), Travelers Orchestration and supervision (J = 29), Specific messages (J = 33), Distributed Ledger—Transaction Anchoring (J = 30), Distributed Ledger—TSP Inclusion (J = 31)
	Taxiway (K = 3)	Travel companion Web-Portal (J = 16), Guest user (J = 12), Preferences and Profiles (J = 13), Journey planning (J = 1), Intermodal Fare Optimization (J = 17), Booking (J = 2), Issuing (J = 3), Mobility packages (J = 4), Validation and Inspection (J = 5), Navigation (J = 9), LBE (J = 8), Map Content (J = 19), Traveler’s feedback (J = 10)

) [34]:

Data analysis results regarding the Athens demo site are shown below:

Results of “Module 1—AHP analysis” for the case of the Athens pilot are shown in

Table 5. Global weights of travelers in AHP (Athens).

CL1 (Travelers)	Description	Weight	Rank
C1	Timesaving by TC functionalities	0.30	1
C2	Cost saving by TC functionalities	0.24	2
C3	General satisfaction with TC functionalities	0.08	5
C4	Comfort with TC functionalities	0.06	6
C5	Safety and Security with TC functionalities	0.15	3
C6	Reliability with TC functionalities	0.13	4

and

Table 6. Global weights of TSPs in AHP (Athens).

CL1 (TSPs)	Description	Weight	Rank
C1	General Satisfaction with the APP	0.25	2
C2	Increase revenues through the APP	0.25	3
C3	Improve customer relationships through the APP	0.29	1
C4	Increase the level of TSP’s security	0.19	4

:

The results of “Module 2—Regression analysis” for the case of the Athens pilot are shown in

Table 7. Analysis of the correlation level between each pair of variables for the Athens demo site (regression analysis).

Variables (Benefits)	Variables Highly Correlated with (p < 0.05) (Highly Correlated Benefits)
Timesaving with Smart location function for all profiles	General satisfaction with the Smart location function for all profiles
Willing to pay for trip sharing function for all profiles	Increase trip safety with trip sharing function for all profiles
Timesaving with Validation and inspection functions for all profiles	General satisfaction with Validation and inspection function for all profiles

:

Results of “Module 3—Bayesian Network analysis”: The output of BN analysis from the Athens demo site indicated that the most influent second-level benefits were “General satisfaction with intermodal fare optimization function for all profiles” (Normalized weight: 0.0093), “Cost saving with intermodal fare optimization function for low-income profiles” (Normalized weight: 0.0087), and “Making traveler’s trip more convenient and comfortable and providing more accessible route with journey planning function for disabled profiles” (Normalized weight: 0.0083).

The results of “Module 4—ANOVA test” for the case of the Athens pilot are shown in the following table (

Table 8. Significant socio-demographic variables and their associated factors among the top 10 variables in the ANOVA test (Athens).

Significant Socio-Demographic Factors (Profiles)	Significant Factors
Income (low-income profiles)	Providing affordable and cheap offers for travelers with Mobility packages function
Traveling weekly with a dependent person (all profiles)	General satisfaction with the trip sharing function
Income (all profiles)	Providing a safe trip with the trip sharing function
Income (all profiles)	General satisfaction with the Preferences and Profiles function
Income (all profiles)	Increase in the usability and fast handling of the application by travelers with the Preferences and Profiles function
Income (all profiles)	General satisfaction with the Intermodal fare optimization function
Income (all profiles)	Cost saving with Intermodal fare optimization function
Income (low-income profiles)	Cost saving with Intermodal fare optimization function

):

Results of “Module 5—USI travelers, USI TSPs and Effectiveness” for the case of the Athens pilot are shown in the following tables (

Table 9. Values of the top 10 sets (Profiles, Functionality, TSP) regarding the USI Traveler in the Athens demo site.

Name of Variable	Value
USI Traveler_r1J9K2 (Navigation functionality provided by MIRAKLIO for all profiles)	1
USI Traveler_r1J17K3 (Intermodal fare optimization functionality provided by Taxiway for all profiles)	0.9
USI Traveler_r2J17K3(Intermodal fare optimization functionality provided by Taxiway for low-income profiles)	0.9
USI Traveler_r2J4K3 (the Mobility packages functionality provided by Taxiway for low-income profiles)	0.9
USI Traveler_r1J4K3 (the Mobility packages functionality provided by Taxiway for all profiles)	0.85
USI Traveler_r1J16K3 (the Travel Companion web-portal functionality provided by Taxiway for all profiles)	0.85
USI Traveler_r1J17K1 (the Intermodal fare optimization provided by OASA for all profiles)	0.85
USI Traveler_r1J19K3 (the Map content functionality provided by Taxiway for all profiles)	0.84
USI Traveler_r1J3K4 (the Issuing functionality provided by Brainbox for all profiles)	0.84
USI Traveler_r1J18K3 (Smart location functionality provided by Taxiway for all profiles)	0.84

,

Table 10. Values of the top 10 sets (functionality, TSP) regarding USI TSPs in the Athens demo site.

Name of Variable	Value
USI TSP_J29K3 (the Travelers Orchestration and supervision functionality provided to Taxiway)	0.9
USI TSP_J33K3 (the Specific messages functionality provided to Taxiway)	0.86
USI TSP_J24K1) (the Location-Based Experience tool provided to OASA)	0.76
USI TSP_J30K1 (the Distributed Ledger—Transaction Anchoring provided to OASA)	0.73
USI TSP_J23K2) (the Asset Manager tool provided to MIRAKLIO)	0.7
USI TSP_J25K3 (the Contractual Management Marketplace provided to Taxiway)	0.68
USI TSP_J25K1 (the Contractual Management marketplace provided to OASA)	0.68
USI TSP_J31K1 (the Distributed Ledger—TSP Inclusion provided to OASA)	0.66
USI TSP_J24K2 (the Location-Based Experience tool provided to MIRAKLIO)	0.66
USI TSP_J33K2 (the Specific messages provided to MIRAKLIO)	0.66

,

Table 11. Values of the top 10 sets (Profile, Functionality, TSP) in terms of effectiveness in Athens (travelers’ functionalities).

Name of Variable	Value
Effectiveness_r1J9K2 (the Navigation functionality provided by MIRAKLIO for all profiles)	1
Effectiveness_r2J4K3 (the Mobility packages functionality provided by Taxiway for low-income profiles)	0.95
Effectiveness_r1J4K3 (the Mobility packages functionality provided by Taxiway for all profiles)	0.92
Effectiveness_r2J4K1 (the Mobility packages functionality provided by OASA for low-income profiles)	0.90
Effectiveness_r1J12K2 (the Guest user provided by MIRAKLIO for all profiles)	0.9
Effectiveness_r1J12K3 (the Guest user provided by Taxiway for all profiles)	0.9
Effectiveness_r1J17K3 (the Intermodal fare optimization functionality provided by Taxiway for all profiles)	0.9
Effectiveness_r2J17K3 (the Intermodal fare optimization functionality provided by Taxiway for low-income profiles)	0.9
Effectiveness_r1J4K1 (the mobility packages provided by OASA for all profiles)	0.89
Effectiveness_r1J12K1 (the Guest user functionality provided by OASA for all profiles)	0.88

and Table 12):

On the other hand, the top 10 variables, in terms of effectiveness are shown in Table 12:

Table 12. Values of the top 10 variables in terms of effectiveness in Athens (TSPs functionalities).

Name of Variable	Value
Effectiveness_J23K3 (Asset manager tool provided to Taxiway)	1
Effectiveness_J29K3 (Travelers Orchestration and supervision provided to Taxiway)	0.9
Effectiveness_J25K1 (the Travelers Orchestration and supervision provided to OASA)	0.89
Effectiveness_J25K3 (the Travelers Orchestration and supervision provided to Taxiway)	0.89
Effectiveness_J33K3 (the Specific messages provided to Taxiway)	0.86
Effectiveness_J23K2 (the Asset manager tool provided to MIRAKLIO)	0.85
Effectiveness_J23K1 (the Asset manager tool provided to OASA	0.82
Effectiveness_J24K1 the LBE tool provided to OASA)	0.766
Effectiveness_J30K1 (the Distributed Ledger—Transaction Anchoring provided to OASA)	0.73
Effectiveness_J31K1 (the Distributed Ledger—TSP Inclusion provided to OASA)	0.66

Table 12. Values of the top 10 variables in terms of effectiveness in Athens (TSPs functionalities).

Name of Variable	Value
Effectiveness_J23K3 (Asset manager tool provided to Taxiway)	1
Effectiveness_J29K3 (Travelers Orchestration and supervision provided to Taxiway)	0.9
Effectiveness_J25K1 (the Travelers Orchestration and supervision provided to OASA)	0.89
Effectiveness_J25K3 (the Travelers Orchestration and supervision provided to Taxiway)	0.89
Effectiveness_J33K3 (the Specific messages provided to Taxiway)	0.86
Effectiveness_J23K2 (the Asset manager tool provided to MIRAKLIO)	0.85
Effectiveness_J23K1 (the Asset manager tool provided to OASA	0.82
Effectiveness_J24K1 the LBE tool provided to OASA)	0.766
Effectiveness_J30K1 (the Distributed Ledger—Transaction Anchoring provided to OASA)	0.73
Effectiveness_J31K1 (the Distributed Ledger—TSP Inclusion provided to OASA)	0.66

5.2. The “Padua” Demo Site

The testing and execution of the Travel Companion APP in the Padua demo site was done from the 17 to the 21 of April 2023. In total, two TSPs—Trenitalia and BusItalia—were assessed in this demo site and 13 responses were collected regarding USIs [34,54]. The final list of “J” Innovative technologies or functionalities considering “K” as TSP which offers each functionality for the Padua demo site are presented in the following table (

Table 13. The final list of “J” Innovative technologies or functionalities considers “K” as TSP which offers each functionality for the Padua demo site.

Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site	Name of Functionalities (J) Assessed in Each TSP (K) for Travelers	Name of Functionalities (J) Assessed in Each TSP (K) for TSPs
Padua	Trenitalia (K = 5)	Guest user (J = 12), Preferences and Profiles (J = 13), Journey planning (J = 1), Trip sharing (J = 11), Booking (J = 2), Issuing (J = 3), Navigation (J = 9), Traveler’s feedback (J = 10), Collaborative space (J = 15)	Asset manager (J = 23), Collaborative space portal (J = 28), Travelers Orchestration and supervision (J = 29), Specific messages (J = 33)
	BusItalia (K = 6)	Guest user (J = 12), Preferences and Profiles (J = 13), Trip sharing (J = 11), Navigation (J = 9), Traveler’s feedback (J = 10), Collaborative space (J = 15)

):

Data analysis results regarding the Padua demo site are shown below:

Results of “Module 1—AHP analysis” for the case of the Padua pilot are shown in

Table 14. Global weights of travelers in AHP (Padua).

CL1 (Travelers)	Description	Weight	Rank
C1	Timesaving by TC functionalities	0.26	1
C2	Cost saving by TC functionalities	0.19	3
C3	General satisfaction with TC functionalities	0.08	5
C4	Comfort with TC functionalities	0.04	6
C5	Safety and Security with TC functionalities	0.19	4
C6	Reliability with TC functionalities	0.21	2

and

Table 15. Global weights of TSPs in AHP (Padua).

CL1 (TSPs)	Description	Weight	Rank
C1	General Satisfaction with the APP	0.16	3
C2	Increased revenues through the APP	0.26	2
C3	Improve customer relationships through the APP	0.56	1

:

The results of “Module 2—Regression analysis” for the case of the Padua pilot are shown in the

Table 16. Analysis of the correlation level between each pair of variables for the case of the Padua demo site (regression analysis).

Variables (Benefits)	Variables Highly Correlated with (p < 0.05) (Highly Correlated Benefits)
General satisfaction with Guest user function for all profiles	Fast access to the TC APP and basic functionalities with Guest user function for all profiles
Helping travelers to make appropriate travel decisions with Journey planning function for all profiles	Cost saving with the Issuing function for all profiles
Willing to pay for trip sharing function for all profiles	Participation and involvement in transport service offers and submitting feedback through the APP with Traveler’s feedback function for all profiles

:

Results of Module 3-Bayesian Network analysis and Bellman shortest path: The output of BN analysis from the Padua demo site indicates the most influent second-level benefits for the acceptance by users of IP4 functionalities offered by TSPs considered in the demo site. This takes into consideration that the Bayes score and cumulative weights are doing the following: giving instant and fast access to TC APP without registration with guest user function for all profiles (Normalized weight: 0.038); encouraging travelers to participate in public transport services offers and submit comments and feedback through TC APP with traveler’s feedback function for all profiles (Normalized weight: 0.033); and general satisfaction with trip sharing function for all profiles (Normalized weight: 0.032)

The results of “Module 4—ANOVA test” for the case of the Padua pilot are shown in the following table (

Table 17. Significant socio-demographic variables and their associated factors among the top 10 variables in the ANOVA test (Padua).

Significant Socio-Demographic Factors (Profiles)	Significant Factors
Income (all profiles)	General satisfaction with the trip sharing function

):

Results of “Module 5—USI travelers, USI TSPs and Effectiveness” for the case of the Padua pilot are shown in the following tables (

Table 18. Values of the top 10 sets (Profiles, Functionality, TSP) regarding the USI Traveler in the Padua demo site.

No.	Name of Variable	Value
1	USI Traveler_r1J10K6 (the Traveler’s feedback functionality provided by BusItalia for all profiles)	0.9
2	USI Traveler_r1J11K5 (Trip sharing functionality provided by Trenitalia for all profiles)	0.83
3	USI Traveler_r1J12K5 (Guest user functionality provided by Trenitalia for all profiles)	0.8
4	USI Traveler_r1J11K6 (Trip sharing functionality provided by BusItalia for all profiles)	0.8
5	USI Traveler_r1J9K6 (the Navigation functionality provided by BusItalia for all profiles)	0.76
6	USI Traveler_r1J15K5 (the Collaborative space portal for traveller’s functionality provided by Trenitalia for all profiles)	0.76
7	USI Traveler_r1J2K5 (the Booking functionality provided by Trenitalia for all profiles)	0.75
8	USI Traveler_r1J10K5 (the Traveller’s feedback functionality provided by Trenitalia for all profiles)	0.75
9	USI Traveler_r1J13K6 (the Preferences and profiles functionality provided by Trenitalia for all profiles)	0.75
10	USI Traveler_r1J13K5 (the Preferences and profiles functionality provided by BusItalia for all profiles)	0.75

,

Table 19. Values of the top five sets (functionality, TSP) regarding USI TSPs in the Padua demo site.

No.	Name of Variable	Value
1	USI TSP_J28K5 (the Collaborative space portal provided to Trenitalia)	0.8
2	USI TSP_J23K5 (the Asset manager tool provided to Trenitalia)	0.75
3	USI TSP_J33K5 (the Specific messages provided to Trenitalia)	0.73
4	USI TSP_J23K6 (the Asset manager tool provided to BusItalia)	0.7
5	USI TSP_J33K6 (the Specific messages provided to BusItalia)	0.6

,

Table 20. Values of the top 10 sets (Profile, Functionality, TSP) in terms of effectiveness in Padua (traveler’s functionalities).

No.	Name of Variable	Value
1	Effectiveness_r1J10K6 (the Traveler’s feedback functionality provided by BusItalia for all profiles)	0.9
2	Effectiveness_r1J11K5 (the Trip sharing functionality provided by Trenitalia for all profiles)	0.83
3	Effectivenes_r1J11K6 (the Trip sharing functionality provided by BusItalia for all profiles)	0.8
4	Effectiveness_r1J12K5 (the Guest user functionality provided by Trenitalia for all profiles)	0.8
5	Effectiveness _r1J9K6 (the Navigation provided by BusItalia for all profiles)	0.76
6	Effectiveness_r1J15K5 (the Collaborative space portal functionality provided by Trenitalia for all profiles)	0.76
7	Effectiveness_r1J10K5 (the Traveler’s feedback functionality provided by Trenitalia for all profiles)	0.75
8	Effectiveness_r1J13K5 (the Preferences and profiles functionality provided by Trenitalia for all profiles)	0.75
9	Effectiveness_r1J13K6 (the Preferences and profiles functionality provided by BusItalia for all profiles)	0.75
10	Effectiveness_r1J9K5 (Navigation functionality provided by Trenitalia to all profiles)	0.74

and

Table 21. Values of the top five variables in terms of effectiveness in the Padua (TSPs functionalities).

No.	Name of Variable	Value
1	Effectiveness_J23K5 (the Asset manager tool provided to Trenitalia)	0.87
2	Effectiveness_J23K6 (the Asset manager tool provided to BusItalia)	0.85
3	Effectiveness_J28K5 (the collaborative space portal provided to Trenitalia)	0.8
4	Effectiveness_J33K5 (the Specific messages tool provided to Trenitalia)	0.73
5	Effectiveness_J33K6 (the Specific messages tool provided to BusItalia)	0.6

):

5.3. The “Warsaw” Demo Site

The testing and execution of the Travel Companion APP in the Warsaw demo site was done from 15th to 19th May 2023. In total, four TSPs—ZTM, MZA, TW, and SKM—were assessed in this demo site, and 208 responses were collected regarding USIs. The final list of “J” Innovative technologies or functionalities considering “K” as TSP which offers each functionality for the Warsaw demo site are presented in the following table (

Table 22. The final list of “J” Innovative technologies or functionalities considers “K” as TSP which offers each functionality for the Warsaw demo site.

Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site	Name of Functionalities (J) Assessed in Each TSP (K) for Travelers	Name of Functionalities (J) Assessed in Each TSP (K) for TSPs
Warsaw	ZTM (K = 10), MZA (K = 8), TW (K = 9)	Guest user (J = 12), Preferences and profiles (J = 13), digital onboarding (J = 20), Journey planning (J = 1), Trip sharing (J = 11), Travel arrangement (J = 21), Navigation (J = 9), Traveler’s feedback (J = 10), Collaborative space (J = 15)	Asset manager (J = 23)

) [34,54]:

Data analysis results regarding the Warsaw demo site are shown below:

Results of “Module 1—AHP analysis” for the case of the Warsaw pilot are shown in

Table 23. Global weights of travelers in AHP (Warsaw).

CL1 (Travelers)	Description	Weight	Rank
C1	Timesaving by TC functionalities	0.26	1
C2	Cost saving by TC functionalities	0.19	3
C3	General satisfaction with TC functionalities	0.08	5
C4	Comfort with TC functionalities	0.04	6
C5	Safety and Security with TC functionalities	0.19	4
C6	Reliability with TC functionalities	0.21	2

and

Table 24. Global weights of TSPs in AHP (Warsaw).

CL1 (TSPs)	Description	Weight	Rank
C1	General Satisfaction with the APP	0.16	3
C2	Increased revenues through the APP	0.26	2
C3	Improve customer relationships through the APP	0.56	1

:

The results of “Module 2—Regression analysis” for the case of the Warsaw pilot is shown in the

Table 25. Analysis of the correlation level between each pair of variables for the case of the Warsaw demo site (regression analysis).

Variables (Benefits)	Variables Highly Correlated with (p < 0.05) (Highly Correlated Benefits)
General satisfaction with Traveler’s feedback function for all profiles	Encouraging to participate in Public Transport Service’s offers and submit comments and feedback through APP with Traveler’s feedback function for all profiles
Giving easier access to the basic function of TC APP with Guest user function for the Elderly	Helping travelers to make appropriate travel decisions with Journey planning function for all profiles
General satisfaction with the Navigation function for all profiles	Timesaving with the Navigation function for all profiles

:

Results of Module 3-Bayesian Network analysis and Bellman shortest path: The output of BN analysis from the Warsaw demo site indicates the most influent second-level benefits for the acceptance by users of IP4 functionalities offered by TSPs considered in the Warsaw demo site, taking into consideration that the Bayes score and cumulative weights are doing the following: providing safe trips with trip sharing functions for all profiles (Normalized weight: 0.013); providing general satisfaction with trip sharing functions for all profiles (Normalized weight: 0.012); and are willing to pay for trip sharing functions for all profiles (Normalized weight: 0.012).

The results of “Module 4—ANOVA test” for the case of the Warsaw pilot are shown in the following table (

Table 26. Significant socio-demographic variables and their associated factors among the top 10 variables in the ANOVA test (Warsaw).

Significant Socio-Demographic Factors (Profiles)	Significant Factors
Profession status	Increase safety with Journey planning function for disabled profiles
Disability	Increase safety with Journey planning function for disabled profiles
Profession status	Providing a safe trip from a COVID-19 perspective for elderly profiles with Journey planning function
Disability	Providing a safe trip from a COVID-19 perspective for elderly profiles with Journey planning function

):

Results of “Module 5—USI travelers, USI TSPs and Effectiveness” for the case of the Warsaw pilot are shown in the following tables (

Table 27. Values of the top 10 sets (Profiles, Functionality, TSP) regarding the USI Traveler in the Warsaw demo site.

No.	Name of Variable	Value
1	USI Traveler_r3J21K10 (the Travel arrangement functionality provided by SKM for disabled profiles)	1
2	USI Traveler_r3J21K8 (the Travel arrangement functionality provided by MZA for disabled profiles)	1
3	USI Traveler_r3J1K10 (the Journey planning functionality provided by SKM for disabled profiles)	0.83
4	USI Traveler_r1J12K10 (Guest user functionality provided by SKM for all profiles)	0.81
5	USI Traveler_r1J21K10 (the Travel arrangement functionality provided by SKM for all profiles)	0.80
6	USI Traveler_r4J12K8 (the Guest user provided by MZA for elderly profiles)	0.8
7	USI Traveler_r4J12K10 (the Guest user provided by SKM for elderly profiles)	0.8
8	USI Traveler_r1J12K8 (the Guest user functionality provided by MZA for all profiles)	0.79
9	USI Traveler_r1J12K9 (the Guest user functionality provided by TW for all profiles)	0.79
10	USI Traveler_r1J21K9 (the Travel arrangement functionality provided by TW for all profiles)	0.79

and

Table 28. Values of the top 10 sets (Profile, Functionality, TSP) in terms of effectiveness in Warsaw (traveler’s functionalities).

No.	Name of Variable	Value
1	Effectiveness_r3J21K8 (the Travel arrangement functionality provided by MZA for disabled profiles)	1
2	Effectiveness_r3J21K10 (the Travel arrangement functionality provided by SKM for disabled profiles)	1
3	Effectiveness_r1J21K10 (the Travel arrangement functionality provided by SKM for all profiles)	0.90
4	Effectiveness_r1J21K9 (the Travel arrangement functionality provided by TW for all profiles)	0.89
5	Effectiveness_r1J21K8 (the Travel arrangement functionality provided by MZA for all profiles)	0.89
6	Effectiveness_r2J1K9 (the Journey planning functionality provided by TW for low-income people)	0.84
7	Effectiveness_r1J1K9 (the Journey planning provided by TW for all profiles)	0.82
8	Effectiveness_r3J1K9 (the Journey planning functionality provided by TW for disabled profiles)	0.82
9	Effectiveness_r5J1K9 (the Journey planning functionality provided by TW for women profiles)	0.82
10	Effectiveness_r1J12K10 (the Guest user provided by SKM for all profiles)	0.81

):

After applying Equation (2) for the calculation of the variable “USI TSP”, which mainly considers the average satisfaction with relevant benefits shown by TSPs (through the USI TSP survey) regarding the functionalities of the Travel Companion, it can be concluded that the (functionality, TSP) achieving the highest satisfaction belongs to the Asset Manager tool provided to MZA (J23K8) with the value equal to 0.61.

On the other hand, taking into account the values of the USI traveler, USI TSPs, and operational KPIs in the Warsaw demo site, in terms of effectiveness, the TC functionalities that are provided to TSPs belong to the Asset Manager tool provided to MZA with the value equal to 0.80 (J23K8).

5.4. The “Liberec” Demo Site

The testing and execution of the Travel Companion APP in the Liberec demo site was done from 15 to 19 May 2023. In total, one TSP—KORID—was assessed in this demo site and 121 responses were collected regarding USIs [34,54]. The final list of “J” Innovative technologies or functionalities considering “K” as TSP which offers each functionality for the Liberec demo site are presented in the following table (

Table 29. The final list of “J” Innovative technologies or functionalities considers “K” as TSP which offers each functionality for the Liberec demo site.

Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site	Name of Functionalities (J) Assessed in Each TSP (K) for Travelers	Name of Functionalities (J) Assessed in Each TSP (K) for TSPs
Liberec	KORID (K = 7)	Travel companion Web-Portal (J = 16), Guest user (J = 12), Journey Planning (J = 1), Smart Locations (J = 18), Booking (J = 2), Issuing (J = 3), Validation, and Inspection (J = 5), Trip tracking orchestration (J = 6), Alternatives calculation (J = 7), Traveler’s feedback (J = 10), Navigation (J = 9), Trip sharing (J = 11), and Travel Arrangement (J = 21)	Asset manager (J = 23)

):

Data analysis results regarding the Liberec demo site are shown below:

Results of “Module 1—AHP analysis” for the case of the Liberec pilot are shown in table (

Table 30. Global weights of travelers in AHP (Liberec).

CL1 (Travelers)	Description	Weight	Rank
C1	Timesaving by TC functionalities	0.26	1
C2	Cost saving by TC functionalities	0.23	2
C3	General satisfaction with TC functionalities	0.11	5
C4	Comfort with TC functionalities	0.04	6
C5	Safety and Security with TC functionalities	0.15	3
C6	Reliability with TC functionalities	0.13	4

):

The results of “Module 2—Regression analysis” for the case of the Liberec pilot are shown in

Table 31. Analysis of the correlation level between each pair of variables for the case of the Liberec demo site (regression analysis).

Benefits	Highly Correlated Benefits
Providing safe trips and avoiding crowds from the perspective of COVID-19 by Journey planning function for the elderly	Providing comfort and comfortable trips with solutions by trip tracking orchestration function for the elderly
Increase safety by Journey planning for disabled profiles	Providing comfort and comfortable trips with solutions by trip tracking orchestration function for the elderly
Increase safety by Journey planning for disabled profiles	Providing safe trips and avoiding crowds from the perspective of COVID-19 by Journey planning function for the elderly

:

Results of Module 3-Bayesian Network analysis and Bellman shortest path: The output of BN analysis from the Liberec demo site indicates the most influent second-level benefits for the acceptance by users of IP4 functionalities offered by TSPs considered in the Liberec demo site, taking into consideration that the Bayes score and cumulative weights are doing the following: providing a convenient tool by TC APP for the people who take care of dependent people with travel arrangement function for disabled profiles (Normalized weight: 0.027); providing a convenient tool for families, kids, and the elderly to support them while traveling with a travel arrangement function for all profiles (Normalized weight: 0.026); and providing convenient and comfortable trips with trip tracking orchestration function for disabled profiles (Normalized weight: 0.026).

The results of “Module 4—ANOVA test” for the case of the Liberec pilot are shown in the following table (

Table 32. Significant socio-demographic variables and their associated factors among the top 10 variables in the ANOVA test (Liberec).

Significant Socio-Demographic Factors (Profiles)	Significant Factors
Residential area	Helping travelers to find the most cost-efficient route with Journey planning for low-income profiles
Age	Helping travelers to find the most cost-efficient route with Journey planning for low-income profiles
Profession status	Helping travelers to find the most cost-efficient route with Journey planning for low-income profiles
Disability	Helping travelers to find the most cost-efficient route with Journey planning for low-income profiles
Residential area	Making a convenient and comfortable trip by providing a solution without knowledge of the local environment for disabled profiles
Profession status	Making a convenient and comfortable trip by providing a solution without knowledge of the local environment for disabled profiles
Disability	Making a convenient and comfortable trip by providing a solution without knowledge of the local environment for disabled profiles
Familiarity with technology and mobile applications	Making a convenient and comfortable trip by providing a solution without knowledge of the local environment for disabled profiles
Residential area	Providing a convenient tool for people who take care of dependent persons with Travel arrangements for disabled profiles

):

Results of “Module 5—USI travelers, USI TSPs and Effectiveness” for the case of the Liberec pilot are shown in the following tables (

Table 33. Values of the top 10 sets (Profiles, Functionality, TSP) regarding the USI Traveler in the Liberec demo site.

No.	Name of Variable	Value
1	USI Traveler_r1J5K7 (the Validation and inspection functionality provided by KORID for all profiles)	0.85
2	USI Traveler_r4J21K7 (the Travel arrangement functionality provided by KORID for elderly profiles)	0.84
3	USI Traveler_r1J18K7 (the Smart location functionality provided by KORID for all profiles)	0.78
4	USI Traveler_r1J10K7 (the Traveller’s feedback functionality provided by KORID for all profiles)	0.78
5	USI Traveler_r1J16K7 (the Travel Companion Web-portal functionality provided by KORID for all profiles)	0.76
6	USI Traveler_r1J21K7 (the Travel arrangement functionality provided by KORID for all profiles)	0.75
7	USI Traveler_r1J12K7 (the Guest user functionality provided by KORID for all profiles)	0.75
8	USI Traveler_r1J9K7 (the Navigation functionality provided by KORID for all profiles)	0.73
9	USI Traveler_r5J3K7 (the Issuing functionality provided by KORID for women profiles)	0.72
10	USI Traveler_r1J7K7 (the Alternative calculation functionality provided by KORID for all profiles)	0.70

and

Table 34. Values of the top 10 sets (Profile, Functionality, TSP) in terms of effectiveness in Liberec (traveler’s functionalities).

No.	Name of Variable	Value
1	Effectiveness_r3J21K7 (the Travel arrangement functionality provided by KORID for disabled profiles)	1
2	Effectiveness_r4J21K7 (the Travel arrangement functionality provided by KORID for elderly profiles)	0.92
3	Effectiveness_r1J21K7 (the Travel arrangement functionality provided by KORID for general profiles)	0.87
4	Effectiveness_r5J3K7 (the Issuing functionality provided by KORID for women)	0.86
5	Effectiveness_r1J5K7 (the Validation and inspection functionality provided by KORID for all profiles)	0.85
6	Effectiveness_r1J3K7 (Issuing function provided by KORID for all profiles)	0.84
7	Effectiveness_r1J2K7 (the Booking functionality provided by KORID for all profiles)	0.81
8	Effectiveness_r1J18K7 (the Smart location provided by KORID for all profiles)	0.78
9	Effectiveness_r1J16K7 (the Travel Companion web-portal function provided by KORID for all profiles)	0.76
10	Effectiveness_r1J12K7 (the Guest user function provided by KORID for all profiles)	0.75

):

After applying Equation (2) for the calculation of the variable “USI TSP”, which mainly considers the average satisfaction with relevant benefits shown by TSPs (through the USI TSP survey) regarding the functionalities of the Travel Companion, it can be concluded that the (functionality, TSP) achieving the highest satisfaction belongs to the Asset Manager tool provided to KORID (J23K7, Value = 1).

On the other hand, the top variable, in terms of effectiveness, for the TC functionalities that are provided to TSPs belongs to the Asset Manager tool provided to KORID (J23K7, Value = 1).

5.5. The “Osijek” Demo Site

The testing and execution of the Travel Companion APP in the Osijek demo site was done from 29 to 2 May 2023. In total, two TSPs—GPP PT and GPP sharing mobility—were assessed in this demo site, and 40 responses were collected regarding USIs [34,54]. The final list of “J” Innovative technologies or functionalities considering “K” as TSP which offers each functionality for the Osijek demo site are presented in the following table (

Table 35. The final list of “J” Innovative technologies or functionalities considers “K” as TSP which offers each functionality for the Osijek demo site.

Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site	Name of Functionalities (J) Assessed in Each TSP (K) for Travelers	Name of Functionalities (J) Assessed in Each TSP (K) for TSPs
Osijek	GPP PT (K = 11)	Journey planning (J = 1), Navigation (J = 9)	CRM portal (J = 32)
	GPP sharing mobility (K = 12)	Journey planning (J = 1), Navigation (J = 9)

):

Data analysis results regarding the Osijek demo site are shown below:

Results of “Module 1—AHP analysis” for the case of the Osijek pilot are shown in table (

Table 36. Global weights of travelers in AHP (Osijek).

CL1 (Travelers)	Description	Weight	Rank
C1	Timesaving by TC functionalities	0.26	1
C2	Cost saving by TC functionalities	0.23	2
C3	General satisfaction with TC functionalities	0.11	5
C4	Comfort with TC functionalities	0.04	6
C5	Safety and Security with TC functionalities	0.14	4
C6	Reliability with TC functionalities	0.20	3

):

The results of “Module 2—Regression analysis” for the case of the Osijek pilot are shown in

Table 37. Analysis of the correlation level between each pair of variables for the case of the Osijek demo site (regression analysis).

Variables (Benefits)	Variables Highly Correlated with (p < 0.05) (Highly Correlated Benefits)
Timesaving with Journey planning for all profiles	Helping travelers to make appropriate travel decisions with journey planning for all profiles
General satisfaction with Journey planning for all profiles	Helping travelers to make appropriate travel decisions with journey planning for all profiles
Helping travelers to make appropriate travel decisions with journey planning for all profiles	General satisfaction with the Navigation function for all profiles

:

Results of Module 3-Bayesian Network analysis and Bellman shortest path: The output of BN analysis from the Osijek demo site indicates the most influent second-level benefits for the acceptance by users of IP4 functionalities offered by TSPs considered in the Osijek demo site, taking into consideration the Bayes score and cumulative weights are doing the following: helping travelers to find more secure routes in off-peak hours for women’s profiles (Normalized weight: 0.067); cost saving with journey planning for all profiles (Normalized weight: 0.056); and timesaving with journey planning for all profiles (Normalized weight: 0.046).

The results of “Module 4—ANOVA test” for the case of the Osijek pilot are shown in the following table (

Table 38. Significant socio-demographic variables and their associated factors among the top 10 variables in the ANOVA test (Osijek).

Significant Socio-Demographic Factors (Profiles)	Significant Factors
Gender	General satisfaction with the Journey planning function for all profiles
Familiarity with technology and mobile applications	General satisfaction with the Journey planning function for all profiles
Gender	Timesaving with Journey planning function for all profiles
Gender	helping travelers make appropriate travel decisions with Journey planning for all profiles
Familiarity with technology and mobile applications	helping travelers make appropriate travel decisions with Journey planning for all profiles
Gender	Helping women to find more secure routes in off-peak hours with Journey planning for women’s profiles
Familiarity with technology and mobile applications	General satisfaction with the Navigation function for all profiles
Familiarity with technology and mobile applications	Timesaving with a Navigation function for all profiles

):

Results of “Module 5—USI travelers, USI TSPs and Effectiveness” for the case of the Osijek pilot are shown in the following tables (

Table 39. Values of the top five sets (Profiles, Functionality, TSP) regarding the USI Traveler in the Osijek demo site.

No.	Name of Variable	Value
1	USI Traveler_r1J9K11 (the Navigation functionality provided by GPP PT for all profiles)	0.76
2	USI Traveler_r1J9K12 (the Navigation functionality provided by GPP sharing mobility (nextbike) for all profiles)	0.76
3	USI Traveler_r2J1K11 (the Journey planning functionality provided by GPP PT for low-income profiles)	0.76
4	USI Traveler_r2J1K12 (the Journey planning functionality provided by GPP sharing mobility (nextbike) for low-income profiles)	0.76
5	USI Traveler_r1J1K11 (the Journey planning functionality provided by GPP PT for all profiles)	0.74

and

Table 40. Values of the top five sets (Profile, Functionality, TSP) in terms of effectiveness in Osijek (traveler’s functionalities).

No.	Name of Variable	Value
1	Effectiveness_r1J9K11 (the Navigation functionality provided by GPP PT for all profiles)	0.76
2	Effectiveness_r1J9K12 (the Navigation functionality provided by GPP sharing mobility (nextbike) for all profiles)	0.76
3	Effectiveness_r2J1K11 (the Journey planning functionality provided by GPP PT for low-income profiles)	0.44
4	Effectiveness_r2J1K12 (the Journey planning functionality provided by GPP PT for low-income profiles)	0.44
5	Effectiveness_r1J1K11 (the Journey planning functionality provided by GPP PT for all profiles)	0.43

):

After applying Equation (2) for the calculation of the variable “USI TSP”, which mainly considers the average satisfaction with relevant benefits shown by TSPs (through the USI TSP survey) regarding the functionalities of the Travel Companion, it can be concluded that the (functionality, TSP) achieving the highest satisfaction belongs to the CRM portal provided to GPP PT (J32K11, Value = 0.6).

On the other hand, the top 10 variables, in terms of effectiveness, for the TC functionalities that are provided to TSPs belong to the CRM portal tool provided to GPP PT (J32K11, Value = 0.6).

5.6. The “Barcelona” Demo Site

The testing and execution of the Travel Companion APP in the Barcelona demo site was done from 5 to 9 June 2023. In total, three TSPs—TMB, BusUp, and AMTU—were assessed in this demo site and 11 responses were collected regarding USIs [34,46,54].

The final list of “J” Innovative technologies or functionalities considering “K” as TSP which offers each functionality for the Barcelona demo site are presented in the following table (

Table 41. The final list of “J” Innovative technologies or functionalities considers “K” as TSP which offers each functionality for the Barcelona demo site.

Name of the Demo Site	Name of TSP (K) Integrated with Each Demo Site	Name of Functionalities (J) Assessed in Each TSP (K) for Travelers	Name of Functionalities (J) Assessed in Each TSP (K) for TSPs
Barcelona	TMB (K = 13)	Guest user (J = 12), Preferences and profiles (J = 13), digital onboarding (J = 20), Journey planning (J = 1), Trip sharing (J = 11), Navigation (J = 9), Traveler’s feedback (J = 10), Collaborative space (J = 15)	Asset manager (J = 23), Contractual management Marketplace (CMMP) (J = 25)
	BusUp (K = 14)	Travel companion Web-Portal (J = 16), Travel Companion for Kids (J = 22), Guest users (J = 12), Preferences and profiles (J = 13), digital onboarding (J = 20), Journey planning (J = 1), Trip sharing (J = 11), Booking (J = 2), Mobility packages (J = 4), Navigation (J = 9), Traveler’s feedback (J = 10), Collaborative space (J = 15)
	AMTU (K = 15)	Guest user (J = 12), Preferences and profiles (J = 13), digital onboarding (J = 20), Journey planning (J = 1), Trip sharing (J = 11), Mobility packages (J = 4), Traveler’s feedback (J = 10), Collaborative space (J = 15), Booking (J = 2)

):

Data analysis results regarding the Barcelona demo site are shown below:

Results of “Module 1—AHP analysis” for the case of the Barcelona pilot are shown in

Table 42. Global weights of travelers in AHP (Barcelona).

CL1 (Travelers)	Description	Weight	Rank
C1	Timesaving by TC functionalities	0.24	1
C2	Cost saving by TC functionalities	0.19	2
C3	General satisfaction with TC functionalities	0.18	3
C4	Comfort with TC functionalities	0.11	5
C5	Safety and Security with TC functionalities	0.11	6
C6	Reliability with TC functionalities	0.14	4

and

Table 43. Global weights of TSPs in AHP (Barcelona).

CL1 (TSPs)	Description	Weight	Rank
C1	General Satisfaction with the APP	0.29	2
C2	Increase revenues through the APP	0.23	3
C3	Improve customer relationships through the APP	0.47	1

:

The results of “Module 2—Regression analysis” for the case of the Barcelona pilot are shown in

Table 44. Analysis of the correlation level between each pair of variables for the case of the Barcelona demo site (regression analysis).

Variables (Benefits)	Variables Highly Correlated with (p < 0.05) (Highly Correlated Benefits)
Providing optimal route in case of delay with Collaborative space portal for all profiles	Helping travelers to choose an optimal route based on real-time feedback with a Collaborative space portal for all profiles
General satisfaction with Preferences and profiles function for all profiles	Helping travelers choose an optimal route based on real-time feedback with a Collaborative space portal for all profiles
General satisfaction with Preferences and profiles function for all profiles	Willing to pay for the Booking function for all profiles

:

Results of Module 3-Bayesian Network analysis and Bellman shortest path: The output of BN analysis from the Barcelona demo site indicates the most influent second-level benefits for the acceptance by users of IP4 functionalities offered by TSPs considered in the Barcelona demo site, taking into consideration the Bayes score and cumulative weights are doing the following: timesaving with Journey planning function for all profiles (Normalized weight: 0.019); timesaving with the Booking function for all profiles (Normalized weight: 0.016); and providing general satisfaction with Preferences and profiles function for all profiles (Normalized weight: 0.016).

Results of Module 4-ANOVA test: Among the top 10 variables in the Barcelona demo site, no significant socio-demographic profiles or significant variables were found.

Results of “Module 5—USI travelers, USI TSPs and Effectiveness” for the case of the Barcelona pilot are shown in the following tables (

Table 45. Values of the top 10 sets (Profiles, Functionality, TSP) regarding the USI Traveler in the Barcelona demo site.

No.	Name of Variable	Value
1	USI Traveler_r1J20K14 (the Digital onboarding functionality provided by BUSUP for all profiles)	0.9
2	USI Traveler_r1J20K15 (the Digital onboarding functionality provided by AMTU for all profiles)	0.9
3	USI Traveler_r1J20K13 (the Digital onboarding functionality provided by TMB for all profiles)	0.9
4	USI Traveler_r4J1K13 (the Journey planning functionality provided by TMB for elderly profiles)	0.84
5	USI Traveler_r1J16K14 (the Travel Companion web-portal functionality provided by BUSUP for all profiles)	0.76
6	USI Traveler_r4J12K13 (the Guest user functionality provided by TMB for elderly profiles)	0.75
7	USI Traveler_r4J12K15 (the Guest user functionality provided by AMTU for elderly profiles)	0.75
8	USI Traveler_r4J12K14 (the Guest user functionality provided by BUSUP for elderly profiles)	0.75
9	USI Traveler_r1J11K13 (the Trip sharing provided by TMB for all profiles)	0.7
10	USI Traveler_r1J11K14 (the Trip sharing provided by BUSUP for all profiles)	0.7

,

Table 46. Values of the top sets (functionality, TSP) regarding USI TSPs in the Barcelona demo site.

No.	Name of Variable	Value
1	USI TSP_J25K14 (the Contractual Management Marketplace provided to BUSUP)	0.44
2	USI TSP_J23K14 (the Asset Manager tool provided to BUSUP)	0.35

,

Table 47. Values of the top 10 sets (Profile, Functionality, TSP) in terms of effectiveness in Barcelona (traveler’s functionalities).

No.	Name of Variable	Value
1	Effectiveness_r1J20K13 (the Digital onboarding functionality provided by TMB for all profiles)	0.9
2	Effectiveness_r1J20K14 (the Digital onboarding functionality provided by BUSUP for all profiles)	0.9
3	Effectiveness_r1J20K15 (the Digital onboarding functionality provided by AMTU for all profiles)	0.9
4	Effectiveness_r1J16K14 (the Travel Companion Web-portal functionality provided by BUSUP for all profiles)	0.76
5	Effectiveness_r4J12K13 (the Guest user functionality provided by TMB for elderly profiles)	0.75
6	Effectiveness_r4J12K14 (the Guest user functionality provided by BUSUP for elderly profiles)	0.75
7	Effectiveness_r4J12K15 (the Guest user functionality provided by AMTU for elderly profiles)	0.75
8	Effectiveness_r1J4K14 (the Mobility packages provided by BUSUP and AMTU for all profiles)	0.7
9	Effectiveness_r1J4K15 (the Mobility packages provided by BUSUP and AMTU for all profiles)	0.71
10	Effectiveness_r1J11K13 (the trip sharing functionality provided by TMB for all profiles)	0.7

and

Table 48. Value of the variables in terms of effectiveness in the Barcelona (TSPs functionalities).

No.	Name of Variable	Value
1	Effectiveness_J23K14 (the Asset Manager tool provided to BUSUP)	0.67
2	Effectiveness_J25K14 (the Contractual Management Marketplace provided to BUSUP)	0.44

):

6. Discussion and Main Results of the Study

This methodology and the Five-Module Toolbox can be applied to other Software and IT innovations; and can be also applied to the Travel Companion APP/IP4 ecosystem in other demo sites in the future [34].

Regarding the BN analysis (Module 3), the following results have been achieved. As it is shown in

Table 49. Top three factors in each IP4MaaS demo site according to BN analysis.

Factors	Demo Sites
	Athens	Padua	Warsaw	Liberec	Osijek	Barcelona
Main factor	General satisfaction with intermodal fare optimization for all profiles	The average number of shopped offers with the journey planning function	Providing safe trips with trip sharing function for all profiles	Providing a convenient tool by TC APP for the people who take care of dependent people with travel arrangement function for disabled profiles	Number of TSPs integrated with Journey planning	The average number of shopped offers with the journey planning function
Second Main factor	Cost saving with intermodal fare optimization for low-income profiles	Giving instant and fast access to TC APP without registration with guest user function for all profiles	General satisfaction with trip sharing function for all profiles	Providing a convenient tool for families, kids, and the elderly to support them while traveling with a travel arrangement function for all profiles	The average number of shopped offers with Journey planning	Timesaving with Journey planning function for all profiles
Third main factor	Making traveler’s trips more comfortable and providing more accessible routes with journey planning for disabled profiles	The average number of modes involved in the journey with the journey planning function	Willing to pay for trip sharing function for all profiles	Providing convenient and comfortable trips with trip tracking orchestration function for disabled profiles	The average number of modes involved in the journey with Journey planning	Number of mobility packages offered by TSP with Mobility packages function

, the top three factors in each demo site are as follows:

According to the result of effectiveness in all of the IP4MaaS demo sites the top 10 variables and factors are listed in the following table (

Table 50. The average value of effectiveness in all demo sites.

No.	Variables	Linked to	The Average Value of Effectiveness Across All Profiles and All IP4MaaS Demo Sites
1	Mobility packages function for low-income profiles	Traveler	0.91
2	Asset manager tool	TSP	0.90
3	Travel arrangement function for all profiles	Traveler	0.89
4	Digital onboarding function for all profiles	Traveler	0.84
5	Intermodal fare optimization for low-income profiles	Traveler	0.83
6	Intermodal fare optimization for all profiles	Traveler	0.82
7	Mobility packages function for all profiles	Traveler	0.82
8	Map content function for all profiles	Traveler	0.79
9	Guest user function for all profiles	Traveler	0.78
10	Smart location function for all profiles	Traveler	0.77

) [34]:

The performance assessment toolbox has the capability to be used in a future project assessing different IP4 innovations or functionalities in different demo sites. This toolbox has the potential to be implemented in different circumstances considering several socio-demographic profiles and evaluating the satisfaction level of both general profiles and specific (sensitive) profiles.

The “Performance Assessment Toolbox” and its scripts, codes, modules, and macros have been exclusively and specifically developed and prepared for the IP4MaaS project.

7. Limitations of the Study

Having a sufficient number of participants from different socio-demographic profiles is crucial to ensure accurate data analysis in this study. Limited participation can negatively impact the assessment phase, leading to inaccurate results [34]. To ensure accurate and precise results, it is essential to consider the required sample size for each module of the quantitative analysis, which includes regression analysis, Bayesian network analysis, and ANOVA testing. This requirement applies to both general and specific profiles. However, the linear distribution of weights among KPIs and USIs in the effectiveness formula may be a limitation due to the varying amounts of data available for each term. In the future, refinements will be necessary when additional data become available and are handled in the IP4MaaS project (2020–2023).

8. Conclusions and Further Developments

This paper presents a methodical assessment approach to quantify how well specific novel technologies created by the IP4 Shift2Rail program meet traveler and TSP needs.

Two quantitative types of data—operational KPIs and USIs—that enable the calculation of the effectiveness of a particular innovative technology offered by a TSP to a profile group of travelers were introduced with this goal in mind. These data types allowed for the definition of demonstration scenarios on which the assessment is conducted [34].

An innovative technology’s effectiveness is determined by how well it meets the demands and expectations of its users, travelers, and TSPs. Effectiveness is dimensionless and has a value between 0 and 1: the greater the number, the better. Comparisons between demonstration scenarios or TSPs and various traveler profiles are possible for a particular technology.

To verify its advantages, move forward with the necessary improvements, and investigate its potential, this study applies quantitative assessment methodology to six demo sites with varied demonstration scenarios defined by the H2020 Shift2Rail IP4MaaS project.

Furthermore, by using machine learning techniques such as Bayesian Networks, statistical correlations between operational KPIs and USIs might be identified.

An assessment methodology and a Five-Module Toolbox have been presented in this study to assess the Travel Companion APP/IP4 ecosystem more in general.

The methodology has been applied to six case studies and TSPs in which several new functionalities for TSPs and travelers were assessed through their effectiveness and compared. These studies took place in the following locations: Athens (TSPs: OASA, MIRAKLIO, Taxiway, Brainbox), Padua (TSPs: Trenitalia and BusItalia), Warsaw (TSPs: ZTM, MZA, TW, and SKM), Liberec (TSPs: KORID), Osijek (TSPs: GPP PT and GPP sharing mobility), and Barcelona (TSPs: TMB, BusUp, and AMTU).

In this use case of the aforementioned demo sites, five analyses have been evaluated and the following results and conclusions have been achieved:

Regarding “Module 1—AHP analysis”, the most important criteria with the highest weights for the case of travelers were “timesaving and cost saving by TC functionalities”. On the other hand, for the case of TSPs the most important criteria with the highest weights were “general satisfaction with the APP” and “increase revenues through the APP”.

Regarding “Module 2—Regression analysis”, this analysis was done to illustrate which variables are highly correlated (with a statistical p-value < 0.05). This means that improvement in the performance of the first variable will increase the performance of the second variable. According to the results, the variables with the highest correlation and lowest p-value in this study were as follows:

For the case of the Athens demo site: “Timesaving with Smart location function for all profiles “ and “General satisfaction with the Smart location function for all profiles”.

For the case of the Padua demo site: “General satisfaction with Guest user function for all profiles” and “Fast access to the TC APP and basic functionalities with Guest user function for all profiles”.

For the case of the Warsaw demo site: “General satisfaction with Traveler’s feedback function for all profiles“ and “Encouraging to participate in Public Transport Service’s offers and submit comments and feedback through APP with Traveller’s feedback function for all profiles”.

For the case of the Liberec demo site: “Providing safe trips and avoiding crowds from the perspective of COVID-19 by Journey planning function for the elderly” and “Providing comfort and comfortable trips with providing solutions by trip tracking orchestration function for the elderly”.

For the case of the Osijek demo site: “Timesaving with Journey planning for all profiles” and “Helping travelers to make appropriate travel decisions with journey planning for all profiles”.

For the case of the Barcelona demo site: “Providing optimal route in case of delay with Collaborative space portal for all profiles” and “Helping travelers to choose an optimal route based on real-time feedback with a Collaborative space portal for all profiles”.

Regarding “Module 3—Bayesian Network analysis”, the output of BN analysis from the case studies of this research indicates the most influential second-level benefits for the acceptance by users of IP4 functionalities offered by TSPs. Taking into consideration the Bayes score and highest cumulative weights in each demo site, it can be concluded that:

For the case of the Athens demo site, the most influential second-level benefit with the highest cumulative weight is “General satisfaction with intermodal fare optimization function for all profiles” (Normalized weight: 0.0093).

For the case of the Padua demo site, the most influent second-level benefit with the highest cumulative weight is “Giving instant and fast access to TC APP without registration with guest user function for all profiles” (Normalized weight: 0.038).

For the case of the Warsaw demo site, the most influential second-level benefit with the highest cumulative weight is “Providing safe trips with trip sharing function for all profiles” (Normalized weight: 0.013).

For the case of the Liberec demo site, the most influential second-level benefit with the highest cumulative weight is “Providing a convenient tool by TC APP for the people who take care of dependent people with travel arrangement function for disabled profiles” (Normalized weight: 0.027).

For the case of the Osijek demo site, the most influential second-level benefit with the highest cumulative weight is “Helping travelers to find more secure routes in off-peak hours for women’s profiles” (Normalized weight: 0.067).

For the case of the Barcelona demo site, the most influential second-level benefit with the highest cumulative weight is “Timesaving with the Journey planning function for all profiles” (Normalized weight: 0.019).

Regarding “Module 4—ANOVA analysis”, the ANOVA test (Analysis of Variance) for Travelers was applied in this analysis to determine if some socio-demographic profiles (per age, gender, incomes level, residential area, traveling with a dependent person, professional status, disability, and familiarity with technology) show significant differences regarding the satisfaction with second-level benefits based on the data gathered through the USI travelers survey, as shown in the following results:

For the case of the Athens demo site, providing affordable and cheap offers for travelers with mobility packages function for low-income profiles; general satisfaction with the trip sharing function for the people who travel weekly with dependent people; and cost saving with intermodal fare optimization function for low-income profiles showed the most significant differences.

For the Padua demo site, general satisfaction with the trip sharing function for low-income profiles showed the most significant differences.

For the Warsaw demo site, the increase in safety with the journey planning function for disabled profiles and providing a safe trip from a COVID-19 perspective for elderly profiles with the journey planning function for the case of professional status and disability showed the most significant differences.

For the Liberec demo site, helping travelers find the most cost-efficient route with Journey planning low-income profiles; helping travelers to find the most cost-efficient route with journey planning for low-income profiles; and helping travelers to find the cost-efficient route with journey planning for low-income profiles showed the most significance.

Finally, for the Osijek demo site, general satisfaction with the journey planning function for all profiles; general satisfaction with the journey planning function for all profiles; and timesaving with the journey planning function for all profiles showed the most significant.

Regarding “Module 5—USI travelers, USI TSPs, and Effectiveness”, considering the terms of USI travelers (Equation (1)) and USI TSPs (Equation (2)), and taking into account the satisfaction level of travelers and TSPs using the IP4 functionalities, it can be concluded that the highest values of effectiveness (minimum 0.2 and maximum 1) at each demo site in this study are as follows:

Athens demo site: the Navigation functionality provided by MIRAKLIO for all profiles (value of effectiveness = 1) and the Asset Manager tool provided to Taxiway (value of effectiveness = 1).

Padua demo site: the Traveler’s feedback functionality provided by BusItalia for all profiles (value of effectiveness = 0.9) and the Asset Manager tool provided to Trenitalia (value of effectiveness = 0.87).

Warsaw demo site: the Travel arrangement functionality provided by MZA for disabled profiles (value of effectiveness = 1), the Travel arrangement functionality provided by SKM for disabled profiles (value of effectiveness = 1), and the Asset Manager tool provided to MZA (value of effectiveness = 0.80).

Liberec demo site: the Travel arrangement functionality provided by KORID for disabled profiles (value of effectiveness = 1) and the Asset Manager tool provided to KORID (value of effectiveness = 1).

Osijek demo site: the Navigation functionality provided by GPP PT for all profiles (value of effectiveness = 0.76), the Navigation functionality provided by GPP sharing mobility (nextbike) for all profiles (value of effectiveness = 0.76), and the CRM portal tool provided to GPP PT ((value of effectiveness = 0.6).

Barcelona demo site: the Digital onboarding functionality provided by TMB, BusUp, and AMTU for all profiles (value of effectiveness = 0.9), and the Asset Manager tool provided to BUSUP(value of effectiveness = 0.67).

When a particular intervention in a research study is applied to the population of interest under “real-world” conditions, it is essential to note that the process of selecting the methods and modules used in the study was thoroughly analyzed based on their characteristics. This methodological step can be applied to other evaluations of Information, Communication, and Technology (ICT) innovation, to assess their functionalities, value, and acceptability levels. By using a combination of methods such as AHP, BN analysis, regression analysis, and ANOVA testing, we can evaluate the functionalities of future innovations and establish correlations and predictions about their improvement.

This methodology and the Five-Module Toolbox can be applied to other Software and IT innovations; and can be also applied to the Travel Companion APP/IP4 ecosystem in other demo sites in the future. The scripts of this methodology assessment have been designed for easy adoption in other projects with similar end goals. While the scripts operate on the codified representation of the traveler profiles, functionalities, and service providers, there is no restriction on the type of codification used. The only factor that the scripts assume is the order of the variables introduced for codification. That is the traveler profile variable (wherever applicable), followed by the functionality variable, followed by the service provider variable. As long as the order is maintained, the scripts self-analyze the number of variables representing the traveler profiles, functionalities, and service providers, and perform calculations (USI, effectiveness) and analysis (regression and BN), Hence, in a similar framework, the scripts may be used for performing data analysis on a large data set with no restriction on the number of functionalities, service providers or subsets of traveler profiles. More detailed information regarding the transferability and extrapolation of this methodology can be consulted at [34].