Project Management Information Systems (PMISs): A Statistical-Based Analysis for the Evaluation of Software Packages Features

: Project Managers (PMs) working in competitive markets are finding Project Management Information Systems (PMISs) useful for planning, organizing and controlling projects of varying complexity. A wide variety of PMIS software is available, suitable for projects differing in scope and user needs. This paper identifies the most useful features found in PMISs. An extensive literature review and analysis of commercial software is made to identify the main features of PMISs. After-wards, the list is reduced by a panel of project management experts, and a statistical analysis is performed on data acquired by means of two different surveys. The relative importance of listed features is properly computed, and the interactions between the respondent’s profiles and PMIS features are also investigated by cluster and respondents’ analyses. The paper provides information for researchers and practitioners interested in PMISs packages and their applications. Furthermore, the analyses may help practitioners when choosing a PMIS, and also for developers of PMISs software in understanding user needs.


Introduction
Nowadays, Project Managers (PMs) deal with increasingly complex projects from both organizational and technological perspectives [1].This complexity is even higher when a multiplicity of conflicting goals is to be achieved [2] and/or limited resources are available [3,4].PMs often have to make fast decisions-rapidly adapting to changes in the project management field-and to satisfy all specifications with the aim to deliver projects in time and within the planned budget [5,6].In this context, Project Management Information Systems (PMISs) are useful tools and techniques to gather, integrate and disseminate the outputs of project management processes [7], supporting PMs in planning, organizing and controlling projects of different complexity.As a result, a wide range of PMIS software is available on the market.Contrary to the customized versions, commercial PMIS software is ready-to-use mass products aimed at generic users [8][9][10], and their choice is a tricky task.Aiming to obtain an exhaustive list of features commonly owned by PMIS software, literature contributions to the field were reviewed, and the main commercial PMIS software was analyzed.Afterwards, collected features were examined by the authors, on the one hand, and by a group of experts, on the other hand, in order to exclude redundant functionalities and/or to merge the similar ones.This way, a reduced list of features was obtained, and a two-level hierarchical structure-comprising features and sub-features-was developed to offer a synthetic and easier overall view of the problem.To evaluate the relative importance of selected features/sub-features, two surveys were first designed-the first by Design of Experiment (DoE) and conjoint analysis, while the second by the ranking scale method-and distributed to a representative panel of experts in the field.Finally, a statistical-based analysis was performed to weigh the selected features, and the interaction between respondents' profiles and PMIS features was also investigated by cluster and respondents' analyses, for the first and second surveys, respectively.
Despite several literature contributions dealing with the application of PMISs and the evaluation of their usefulness and performance [11][12][13][14][15][16][17][18][19][20][21][22][23][24], few of them are concerned with determining the fundamental features that a PMIS should have and assessing their relative importance.In the authors' opinion, the analysis of both the main commercial PMIS software and the scientific contributions in the field provides a structured framework to have an overall view of the state-of-the-art.In addition, the outcomes of the performed quantitative analysis may be used by researchers and practitioners to make proper decisions when choosing and/or developing PMIS software.Therefore, the purpose of the present work is threefold and may be summarized through the following Research Questions (RQs): RQ1: What are the main features that PMIS software should contain?RQ2.What is the relative importance of selected features?RQ3.Is there any interaction between the relative importance of the identified features and the respondent's profile?
The remainder of the paper is organized as follows.The literature review is presented in Section 2, and the methodological approach is described in Section 3. Results and discussion are synthesized in Section 4. Conclusions close the work in Section 5.

Literature Review
Nowadays, PMISs have been increasingly used by PMs as decision-aiding support tools.With this recognition, several contributions deal with the application of PMISs and the evaluation of their usefulness and performance, also focusing on the determination of basic features required by PMIS users.In [11], the impact of PMISs on PMs and projects performance is assessed.In particular, responses to questionnaires administered to thirtynine PMs are analyzed to highlight the positive influence of PMIS on planning, scheduling, monitoring and controlling.In addition, the authors emphasize how important the reporting feature of PMISs is in providing project information and progress over time.In order to design a PMIS software, a REFerence information MODel for enterprise-wide Project Management (Ref-ModPM) is proposed by [12].The model includes all project management processes (e.g., planning, controlling and coordinating), and it is suitable for both single and multi-project management to find out the basic requirements that a commercial software should have.In [13], the author demonstrates the successful contribution of PMIS use in every phase of the project life cycle by a conceptual model tested by means of data acquired from 170 experienced respondents.Referring to a multi-project environment, Caniels and Bakens [14] report PMIS features that facilitate the decision-making process, and point out relationships between PMIS information quality and PMs' satisfaction with PMISs.Results arising from a survey are used to test the hypotheses using a Partial Least Squares (PLS) regression analysis.Referring to construction projects, Lee and Yu [15] propose a model that considers different aspects such as the system, information and service quality, intention of PMIS use, and user satisfaction to assess the impact of PMISs on project management.In [16], the authors claim that the choice of PMISs depends on both the sector in which the company operates and the type of project to be managed.With reference to the shipbuilding industry, this paper proposes an integrated approach constituted by a PMIS to plan and schedule the project activities, Project Data Management (PDM) to deal with documents and to share know-how, and a Control Tower (CT) to monitor and manage events and to broadcast messages among the different actors.Referring to an Indian construction project, Wale et al. [17] compare results obtained by Microsoft Excel with those arising from the use of Microsoft Project, thus highlighting the higher expense in terms of the times and costs of Excel.By the distribution of questionnaires to contractors, consultants, engineers and academic researchers, Obodoh et al. [18] analyze the impact of PMISs on project failure rates in the Nigerian construction industry.Frequency analysis and the Chi-Squared test are used on collected data to demonstrate the positive contribution of PMISs on to the project success.With relation to ERP projects, Nguyen et al. [19] perform Exploratory Factor Analysis (EFA) and regression analysis to demonstrate that users' satisfaction is strongly influenced by friendliness, functional information, and support-service quality.
On the other hand, few contributions specifically focus on determining the basic features required by PMISs software users.In [20], the definition and the conceptual development of a Smart Project Management Information System (SPMIS) are given.The authors report an overview on principal commercial PMISs software at a worldwide level and describe the main 25 features they should have (e.g., initiation, planning and scheduling, resource management, control, risk and documents management).In the authors' opinion, most of the analyzed software does not meet the requirements of today's PMs.Referring to the construction sector, Liberatore et al. [21] highlight that interviewed professionals are mainly interested in activities/resources scheduling, resources levelling and net present value tools.In particular, the authors synthetize analytical techniques commonly used during the project's planning and/or monitoring and show that the most applied technique is the critical path analysis followed by the resource scheduling/levelling and the Earned Value (EV).In the same field, Nitithamyoung and Skibniedìewski [22] identify a list of features and analyze their impact on the success/failure of web-based construction project management systems with the use of Application Service Providers (ASPs).Liberatore and Pollack-Johnson [23] analyze and evaluate factors that influence the use, the type of use (planning versus planning and control) and the choice of PMIS software.In [24], the main functionalities of PMISs are extrapolated by the analysis of 18 commercial software.The authors identify eight criteria, 28 sub-criteria and 44 sub-subcriteria whose relative importance is assessed by means of the Analytic Hierarchy Process (AHP).The analysis confirms the importance of planning and control, whereas a final ranking among software is obtained by a direct evaluation.

Methodological Approach
The multi-step methodological approach implemented to answer the RQs is represented in Figure 1.Aiming to answer RQ1, both a literature review of the scientific contributions in the field (Section 2) and the analysis of the main commercial PMIS software were carried out (Section 3.1).Afterwards, a panel of experts was involved to obtain the final list of PMIS features to be weighted.In this regard, the methodological approaches implemented to answer RQ2 and RQ3 are reported in Sections 3.2.1 and 3.2.2 and 4.2 and 4.3 respectively, along with the discussion of the obtained results.

PMIS Features and Sub-Features
Besides the literature review of the scientific contributions in the field, the main commercial PMIS software available on the market were selected and analyzed on the basis of the related guidelines and used in order to identify and list the owned features.In this paper, 8 software packages were examined, namely Trello (http://trello.com,23  ).Therefore, the initial list of PMIS features was reduced by the authors by removing redundancies and/or grouping the similar ones.Survived functionalities were hierarchically organized to offer a synthetic and easier overall view of the problem, namely they were divided into features and sub-features.Afterwards, a panel of experts in the field (i.e., a full professor, the PM of an international company and a member of the Project Management Institute Southern Italy Chapter-PMI SIC) was involved to analyze the initial hierarchical structure, aiming to further reduce the number of PMIS features to which we should pay attention later on.

Survey Design and Analysis of Data
In order to evaluate the relative importance of the final selected features and sub/features, two different surveys were designed.To verify their consistency, simplicity and readability, surveys were firstly administered to the panel of involved experts by the webbased app Google form.On the basis of the received feedback, the final surveys were designed.In detail, both surveys included a first section to acquire some general information about respondents (e.g., gender, age, hours of project management experience, and work sector), while they were different with respect to the second section, specifically addressed to the computation of the relative importance (i.e., weight) of features and/or sub-features.The two surveys were distributed by internet to allow faster responses and reduced research costs [11].With regard to the respondents, they were properly chosen by the panel of involved experts with the aim of including different aspects such as organization sector, experience, country and so on.
The following two sections detail how the second section of both surveys was designed and the analyses performed on the collected data.

First Survey: Design and Analysis of Data
In the first survey, the evaluation of features and sub-features was performed by the ranking scale method, preferred to the rating one since it forces the respondent to pay more attention to one item over another before deciding which functionality he/she is willing to give up [25][26][27].Therefore, every respondent was asked to rank features and sub-features from the worst (i.e., to be excluded) to the best ones (i.e., deemed to be absolutely necessary in PMIS software).
In addition, the agglomerative hierarchical clustering algorithm was implemented to group similar respondents according to their responses with regard to every feature/subfeature.At the beginning of the hierarchical clustering algorithm, every respondent was considered a singleton cluster.Afterwards, a new cluster was built step-by-step by merging a pair of nearest clusters, according to a similarity measure.The procedure was iterated until all respondents were merged into a single cluster or the desired number of clusters was reached [28].In the present paper, a similarity measure based on the Euclidean distance among responses was used, and the complete linkage approach was applied.The obtained results can be easily visualized using a dendrogram.For every dendrogram, the number of clusters was identified by cutting it at a particular height, that is, level of similarity [29].In general, the cutting level of a dendrogram may be evaluated observing the heights of the jumps, from the top to the bottom.A high jump means that the dissimilarity of the two clusters split is large, that is, a great gain in similarity is obtained by splitting them.Usually, the highest jump is chosen to obtain a small number of clusters.The obtained number of clusters was hence used as input data of the k-means method subsequently implemented to obtain a better clustering of respondents.Starting from the specified number of clusters and setting a centroid for every cluster, the k-means algorithm was used to associate every respondent with the nearest centroid.When all respondents were associated to a cluster, this step was completed and an initial clustering was performed.New k centroids were calculated and the association procedure was reiterated.The algorithm was stopped when no more changes to centroids were observed [30].
The weight of every feature and sub-feature was calculated as the median value of the respondents' evaluations, the latter being the ranking position of the feature/sub-feature under investigation with relation to the whole dataset and per cluster.Normalizing these values in respect to the sum per column, the relative importance of features/subfeatures was obtained.Finally, the interactions between the respondent profiles and PMIS functionalities weights was verified using the χ 2 metric.

Second Survey: Design and Analysis of Data
In the second survey, the Design of Experiment (DoE) was used to develop some full functionalities packages-called prototypes-to be submitted to respondents.In particular, only features were involved in the design of prototypes, whereas sub-features were disregarded because of their high number.In order to allow respondents to evaluate prototypes, a conjoint analysis [31][32][33] was performed.The conjoint analysis allows the determination of whether the presence of a functionality significantly influences the appreciation level of PMIS software or not, also taking into account the interactions that could arise from the simultaneous presence of several functionalities [34].With regard to DoE, a fractional factorial design of experiments was developed.It provided 2 (k-p) runs (i.e., prototypes), two being the number of levels (i.e., presence or absence of a feature, identified by +1 and −1 respectively), k the number of factors (i.e., the eight features) and p the rate of reduction.A well-known design with eight factors foresees 16 runs, so that the resulting reduction rate p is equal to four.It means that the fractional design only comprised 2 4 = 16 runs rather than 2 8 = 256 of the full design.The chosen fraction design can be selected in many ways, 2 4 ways in this particular case.In this paper, the chosen fractional design was obtained by the Minitab© package (Table 1).It was characterized by the presence of four factors (i.e., level +1) and the absence of the other four (i.e., level −1) for 14 prototypes (from 2 to 15), while the first and the last prototypes (i.e., all features are absent or present) represent the zero and the full scale, respectively.Manag.

Utility Access Permits
The 16 runs of Table 1 were the software prototypes considered to evaluate features.In every prototype, the included features were highlighted in yellow whereas the absent ones were in white.An example of a prototype is reported in Figure 2. Every respondent was asked to rate every prototype on a seven-point scale, also allowing decimal scores.Scores equal to one and seven were given to the first and last prototypes, respectively.
Aiming to single out the features' weights on the basis of the prototypes' ratings, a regression model was performed [35][36][37][38][39]. Input data of this model were the dichotomous variables representing the presence (i.e., level +1) or the absence (i.e., level −1) of a functionality within every prototype.On the other hand, the response variable of the model was the rating of every prototype given by every respondent.The regression coefficients of the estimated model represented the features' weights.The only significant terms were considered after using a stepwise regression, which was able to select the significant factors (i.e., features) with respect to their contribution to the total variability of responses.Finally, on the basis of the methodology proposed by Barone et al. [37,40], the normalized weights of features were calculated from the estimated regression model coefficients.

List of PMIS Features and Sub-Features
One hundred and fifty-two PMIS features were initially identified by the literature review and the analysis of PMIS packages.The hierarchical representation of this initial list is reported in the Appendix A, where the description of every feature/sub-feature is also given along with the main references and the software in which they are implemented.Involving the panel of experts, the number of features/sub-features was reduced to 82 by removing redundancies and/or grouping the similar ones.In this regard, some features and sub-features were discarded (e.g., "Interdependencies Management" was considered to be negligible) or merged and/or renamed (e.g., "Budget Control" and "Time Control" sub-features).The resulting hierarchical structure is reported in Figure 3, whereas detailed and modified descriptions of features/sub-features are included in Appendix B.

First Survey
The first survey was administered to 75 professionals, and a total number of 67 responses was received.As reported in Section 3.2.1, the number of clusters was identified by cutting the dendrogram obtained for every feature/sub-feature at a particular level of similarity.As an example, the dendrogram of "Resource Planning" obtained by Minitab© is shown in Figure 4. Four clusters were chosen, where the cutting level is the one represented by the orange dashed line and the four clusters are differently colored.Table 2 shows the number of obtained clusters and the number of respondents per cluster.The relative importance of every feature-total and per cluster-are shown in Table 3.On the basis of the last column of Table 3, "Activity Planning" and "Access Permits" were found to have the highest and lowest importance, respectively.On the other hand, different opinions were observed when comparing feature weights related to every cluster.Therefore, the interactions between respondents' profiles (see Appendix B) and PMIS functionality weights were verified using the χ 2 metric.In particular, taking into account all 67 responses, the distribution of respondents in clusters 3 and 4 was observed to be significantly different from the others in respect to the software usage and best software (Figure 5a,b).It means that a lot of respondents that do not use software or for which the best software does not exist are present in these clusters.Figure 5c shows that the frequency of foreign respondents in clusters 3 and 4 is switched.In detail, cluster 3 was characterized by a higher presence of foreign respondents in proportion to the total.Weights obtained for "Activity Planning", "Resource Planning", "Control", "Risk Analysis", "Reporting", "Communication Management" and "Utility" are reported in Table 4. Considering the χ 2 metric for the sub-features of "Activity Planning", "Resource Planning", "Control", "Reporting", "Communication Management" and "Utility", no significant interactions between weights and clusters were observed.As a consequence, the corresponding histograms are not reported.On the other hand, a significant interaction between clusters of "Risk Analysis" and respondents' education was observed (Figure 6), because of the higher relative frequency of educated respondents in cluster 2 than the other clusters.The corresponding weights per cluster are given in Table 5.

Second Survey
The second survey was administered to 15 experts to rank the 16 prototypes reported in Appendix B. Twelve responses were received for a total number of 192 evaluations.As detailed in Section 3.2.2, a regression model was performed.After using a stepwise regression to select the significant factors (i.e., features) with respect to their contribution to the total variability of responses, only significant terms of the regression model are reported in Table 6.The stepwise procedure highlighted the significance of all main factors, while the interactions between them were eliminated.In addition, the significance of some interactions between features and respondents was demonstrated.The residual analysis (Figure 7)-performed on 192 ratings-confirmed the acceptability of the assumption of normality, which the regression model was based on.The goodness of fit was quite high (R2 = 83.91%;R2adj = 76%).Figure 8 only shows those scenarios where significant interactions between features and the interviewed were observed.For instance, respondents 4 and 12 gave higher ratings to prototypes when "Activity Planning" was included.In detail, respondents 4 and 12 are both male, use PMISs software and they do not have any certification.With relation to "Control", respondent 5 gave an opposite rating in comparison with the others, providing a high increase with relation to "Resource Planning".Respondent 5 works in an IT company, uses software and he/she is trained in project management.On "Communication Management", respondents 6 and 12-having more than 10,000 h of project management experience-provided an opposite rating in respect to the other respondents.As suggested by Barone et al. [37,40], the normalized weights of features (Table 7) were calculated from the previously estimated regression model coefficients.
The input factors being characterized by the same dispersion (for every column of Table 1, eight +1 and eight-1), the regression coefficients can be combined directly, that is, without any standardization.Therefore, weights were calculated by normalizing them to one.Differently to the first survey, the second one returned the highest importance of "Control", while "Activity Planning" took the fifth position.Both methods assigned low weights to "Utility" and "Access Permits", even if "Reporting" also had a low weight in this second analysis.

Conclusions
Today's markets are highly competitive, dynamic and uncertain.As a consequence, Project Managers (PMs) increasingly need the support of tools specifically developed to assist them during all phases of the project life cycle.In this regard, the market makes available a wide variety of Project Management Information Systems (PMISs) software, suitable for projects differing in scope and user needs.Aiming to obtain an exhaustive list of features commonly shared by PMISs and to answer RQ1, a literature review and an analysis of the main commercial PMIS software were performed.A list of 82 features was identified and then analyzed by a panel of involved experts to further reduce the list.Finally, eight features and 24 sub-features were chosen because they were deemed to be fundamental to every PMIS.To obtain the relative importance of listed features and subfeatures, and to answer RQ2, a statistical analysis was performed on data acquired by way of two different surveys properly designed and administered-by the net-to a selected number of experts.To answer RQ3, the interaction between the respondents' profiles and the relative importance of listed PMIS features was also investigated by cluster and respondents' analyses, for the first and second surveys, respectively.The analysis performed highlights that "Utility" and "Access Permits" have a lower importance than other features.On the other hand, "Activity Planning" and "Control" are the most important features from the first and second surveys, respectively.In the first case, "Reporting" and "Communication Management" have higher and lower weights, respectively, than in the second one.Sub-features were analyzed only from the first survey and the relative importance was properly computed.
In the authors' opinion, the analysis of both the main PMIS packages and scientific contributions to the field may represent a structured framework in order to have an overall view of the state-of-the-art.On the other hand, the outcomes of the performed quantitative analysis may be used by researchers and practitioners to make proper decisions when choosing and/or developing PMIS software.In this regard, a possible future development may concern the evaluation of different PMIS software packages, involving the list of features/sub-features and the related weights arising from the presented analysis.However, the choice among PMIS software could also rely on some general characteristics such as cost, usability, and scalability.As consequence, a further research line may be addressed to compare PMIS software from this perspective.

Table 1 .
Fractional factorial design of experiments for features.

Table 2 .
Obtained clusters and respondents.

Table 3 .
Total features weights and per cluster.

Table 4 .
Weights of sub-features.

Table 5 .
Total and relative weights of "Risk Analysis" sub-features.
This function allows the development of the Critical Chain Project Management (i.e., scheduling of activities by identifying critical chains with feed and project buffers) • Cost Management-This function allows to plan and manage all project costs (i.e., resources, raw materials, etc.) • Calendar-This function includes the design and customization of project and resources calendars • Resources Chart-This function allows the visualization of the workload of each resource CONTROL SUB-FEATURE • Progress Control-This feature allows at comparing the actual cost and work values with the planned ones • Quality Management and Validate-This feature allows to include procedures for checking project results (i.e., view inspection results or check whether all requirements are met) RISK MANAGEMENT SUB-FEATURE • PERT-This feature allows to perform the PERT analysis to get the project time to completion when activities durations are random variables • Simulation-In the presence of random costs and times, this function allows to determine the project time to completion through simulation (i.e., Monte Carlo simulation) • Risk/Issue management-This function supports the management of the project risks/issues (i.e., SWOT analysis, creation of risk/problem registers, etc..) REPORTING SUB-FEATURE • Report-This function permits the creation of standard and/or customized reports of the project • Version Tracking-This function allows to track and record all documents related to the project • Dashboard-This feature allows team members to access all project-related information through a graphical, concise and customizable representation.To do list-This feature allows to visualize the list of things to be done on a specific day • Filters-This feature allows to perform advanced searches on project documents (i.e., filters, sorting, grouping) • Customized fields-This feature allows to customize different fields (i.e., calendars, views, tables, filters, etc.)