Validity and Reliability Testing of Geographical Information System (GIS) Quality and User Satisfaction towards Individual Work Performance ^†

Abstract

This article aims to validate the instrument’s reliability and validity in measuring the influence of geographical information system (GIS) quality and user satisfaction on its impact on individual work performance. Based on an extensive literature review and input from experts, 68 items were constructed focusing on GIS quality, user satisfaction, and individual work performance to ascertain the opinions of GIS users were established to be the findings of the study. The instrument was composed and then reviewed by panelists from subject matter experts. Later, a pilot study was conducted and participated by 30 respondents. Concerning the analysis of the data, statistical analysis was performed, the analysis output then demonstrated that the instrument was dependable and that no abnormalities had occurred in the data.

1. Introduction

Geographical information system (GIS) is a solution that contributes to the understanding of the ability to collect, visualize, and produce analysis [1]. GIS is software that works in the spatial area focused on geographic information. In addition, the ability of GIS covers the capturing, analyzing, and presenting of all types of information [2]. Furthermore, GIS can also translate information to give a more comprehensive understanding through a combination of spatial and business data [3]. However, the quality of the GIS is still in a grey area and the evidence with regards contributing factors in GIS quality is still minimal. Therefore, a standard produced by the International Organization for Standardization (ISO), namely the ISO-25010 Software Quality Model, was used in measuring GIS quality for this study.

The main objective of this study is to validate the instrument’s reliability and validity for the researcher to proceed with the actual study in identifying the determinant of GIS quality that are anticipated to influence user satisfaction, and how user satisfaction may affect individual work performance. Literature analysis revealed eight factors that have been identified, namely functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. Hence, this paper aims to validate the instrument’s reliability and validity in measuring the influence of geographical information system (GIS) quality and user satisfaction on its impact on individual work performance.

2. Literature Review

GIS is a powerful solution in giving ideas to users via the ability in providing information based on location-centric called geospatial information [4]. A simple definition of GIS is a system that focuses on data capture, storing, managing, analyzing, and presenting all forms of geographically referenced information identified according to locations [5]. GIS operates differently from other software. The uniqueness of GIS provides functionalities in terms of maps and comprehensive analysis; in some cases, it was able to give solutions regarding business directions. The solutions that GIS bring come with different ways of doing things, especially when the element of “Where” is added as a variable in making decisions. It is able to boost users to undergo innovation in daily operation through spatial information [6]. Spatial information refers to attributes that provide context and meaning when combined with business data [5]. Some spatial information comes in points of locations or shapes of area. Another interesting area offered in GIS is the ability to perform a query and produce analysis faster through a visualization dashboard for users to have an earlier understanding of when data were collected and combined with spatial information. It gives location intelligent analysis that provides proper insight, especially in planning and understanding the nature of the specific areas of businesses [7].

Many software in the market were produced and the measurement of quality were still in grey area. GIS is also a software that needs to be validated in terms of the quality provided to the industry. Quality in GIS is very subjective and depends on the nature of use. Different organizations or users have different objectives towards the use. Software quality is elaborated as the elements that contribute to pleasing and meeting the needs of users [8]. Many models have been developed to measure software quality [9,10,11]. The evolutions of software quality were started by McCall (1977), Boehm (1978), Dromey (1995), with the FURPS model, the ISO-9126 (2001) Software Quality Model and the most updated standard followed by the industry is the ISO-25010 Software Quality Model. For the context of this study, ISO-25010 was chosen as it was the revised version and used vastly in measuring the quality of software.

User satisfaction is one of the major factors contributing to the acceptance of software. The acceptance of GIS has been a subject of debate in the community [12]. According to [13], user satisfaction refers to the perception of GIS’s usefulness, trust, pleasure, and level of comfort when using it. From a usefulness context, GIS was determined by its ability, features, and functionality in producing output [14]. The perspective of trust focuses on whether the GIS is able to produce trusted information and the ability to improve performance as per the user’s expectation. Any situation where a lack of trust occurred will be a big issue hampering any implementation of GIS [15,16]. Hence, it is important to ensure that trust is in place so that users have strong beliefs that GIS is able to perform as per users’ expectations. Meanwhile, pleasure in GIS are situations where users evaluate the software through actual experience in using it. The experience during use can be affected by the internal or external characteristics of GIS.

One of the critical aspects of measuring software implementation’s success was measuring individual work performance [17,18]. Individual work performance is a concept in measuring the quality produced by individuals. Thus, any organization needs an individual who has the capabilities to exhibit a high work performance in achieving goals to increase organization productivity and competitiveness. Individual work performance is defined as actions or behaviors of individuals made in achieving goals [19,20]. In addition, performance is defined as a high result of work efficiency, effectiveness, and quality of work [20,21,22]. The higher an individual’s performance will result in satisfaction and mastery regarding personal tasks or jobs.

3. Instrument and Data Collection

For this study, 68 items were constructed based on a three-domain questionnaire derived from a comprehensive literature review. It was organized to obtain users’ viewpoints on GIS quality accommodated by the organization for use. In addition, for pre-testing, the instrument was reviewed and given feedback by six subject matter experts in software development, and academicians, including the GIS users, were successfully acquired. Several suggestions were raised, including the use of relevant words and verbs to strengthen question clarity and the inclusion of examples to help responders comprehend the questions. Through feedback and suggestions, amendments were implemented in response to the constructive feedback. Subsequently, this pilot study aimed to ensure that the respondents understood the questionnaire items and that no questions were misleading. This pilot study, which included 30 respondents, was examined using Statistical Package for Social Sciences (SPSS) version 26 by International Business Machines Corporation (IBM, Armonk, NY, USA). Finally, Cronbach’s alpha was utilized to analyze the questionnaire’s scale reliability.

Table 1. Items for each Dimension.

Variables	Dimensions	Item
GIS Quality	Functional Suitability	The functions provided in GIS are suitable for me. The functions provided in GIS meet with my work requirements. The information provided in GIS is accurate. The results of the analysis provided from GIS are precise. Functions in GIS facilitate me in the business process. Functions provided in GIS assisting me in meeting organizational goals.
	Performance Efficiency	GIS is able to provide good response time. GIS is fast at doing data processing. GIS requires minimal close guidance from system administrators. Users are not limited to access GIS via personal computer (PC) only. The capacity of data storage provided in GIS is scalable. GIS can perform even in high numbers of user access.
	Compatibility	GIS is able to work with existing available systems. GIS is able to function without giving any negative impact to other systems. GIS enables me to interact with other systems. The output from GIS can be used by other systems.
	Usability	GIS is software that is appropriate for my use. GIS does not require a lot of time to learn how to use it. The training provided is enough for me. I can easily operate GIS without any guidance. GIS requires minimal technical capabilities. GIS is able to notify users required actions in avoiding errors. The GIS interface is well organized. The interface of GIS is interesting. GIS can be accessed by every member of my organization.
	Reliability	GIS has a low rate of failure.
		GIS is always able to return information needed by me.
		GIS is accessible when it is needed.
		GIS maintains its performance even if there are data faults.
		I am able to recover information in GIS even after failure occurs.
	Security	GIS is able to prevent access from unauthorized users.
		GIS provides proper data encryption.
		GIS is able to prevent any modifications of data.
		GIS is able to trace sources of access so it can control manipulation of actions.
		All activities in GIS were uniquely traceable.
		GIS is able to identify the genuine identity of users.
	Maintainability	GIS can be customized according to my needs.
		Any changes in GIS will have no impact on other systems.
		The concept in GIS can be reusable to other systems.
		I am able to identify errors when they happen.
		GIS is able to notify the cause of error.
		GIS is stable even if any modifications are made.
		I can easily test any new changes made in GIS.
	Portability	GIS can be adapted for the use of other departments in my organization.
		GIS is able to evolve according to needs of the organization.
		Process installing GIS on a user’s devices is easy.
		GIS can be easily removed from other user devices.
		GIS has the potential to replace other systems in my organization.
		GIS has the ability to be the main visualization platform in my organization.
User Satisfaction		GIS empowers me to accomplish tasks well. GIS improves my work performance. GIS increases my work productivity. All information produced from GIS is trusted. GIS has its anticipated effects. The experience of using GIS is enjoyable. GIS provides a pleasant experience. GIS causes no stress when using it.
Individual Work Performance	Task Performance	I am able to plan and organize all my work tasks.
		I have an excellent number of work completions.
		I fulfill all work quality criteria for my tasks.
		I complete my work tasks within the timeframe.
		I solve all work-related problems efficiently.
		I always update my knowledge for my work.
	Adaptive Performance	I always come up with creative ideas that are able to provide a new perspective about my tasks.
		I am able to rearrange my tasks goals according to the current situation.
		I am able to learn new tasks by using a combination of existing software and other technologies.
		I am always flexible and open minded in determining the way of delivering my tasks.
		I am able to quickly analyze potential solutions that could be used in completing my tasks.
		I always come up with creative ideas that are able to provide a new perspective about my tasks.

lists the items for each dimension.

4. Pre-Testing, Validity, and Reliability of Research Instrument

Upon the instrument’s establishment, a content validation process must be executed to evaluate the instrument. Hence, the questionnaire was distributed to six experts consisting of software development company representatives, GIS users, and academicians to examine and identify any concerns with the questions. Eventually, they all responded and suggested suitable feedback. The experts comprise two experienced GIS users, two professionals from GIS companies, and two academicians with Ph.D.

Table 2. Panel of Experts.

Panel of Experts	Organization
IT Manager	Organization A
GIS User	Organization A
Academician	University A
Academician	University B
GIS Project Manager	Organization A
Application Analyst	Organization B

presents the information regarding the experts who contributed and were involved in this study. Several concerns were identified, including the need for more clear and precise alternatives when providing respondents with options, the use of more appropriate words and verbs to increase question clarity, and the inclusion of examples to assist respondents’ understanding of the questions. Their response was valuable as it incorporated all insightful comments and recommendations into the current questionnaire, with suitable revisions and additions.

The pilot study validated the questionnaire, allowing the investigational questions to be answered and appropriately verified [23,24]. The pilot study was conducted to validate that the respondents understood the questionnaire items and that no unclear questions were present. The data from such an initial survey were examined using SPSS version 26 and featured 30 individuals. Finally, Cronbach’s alpha was used to define the scale reliability of the questionnaire, or how closely linked a group of questions is.

The questionnaires were designed into four sections. The first section has five items related to demographic information. The second section of the questionnaire consists of 49 items that focus on GIS quality, which includes functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. The third section focuses on user satisfaction with eight items. The fourth section contains 11 items that focus on individual work performance. As a result, 73 items were subsequently included in the questionnaire before the execution of the actual study. Lastly, the questionnaire comes with two open-ended questions to allow respondents to respond subjectively about the challenges of GIS and suggestions to improve the system.

5. Reliability Analysis Result

Referring to

Table 3. Descriptive Statistic Demographic Result.

Item	Response	Percentage
Gender	Male Female	77% 23%
Age	21–30 years old 41–50 years old 31–40 years old	50% 30% 20%
Education Level	Bachelor’s Degree Diploma Master’s Degree Malaysia Education Certificate (SPM)	63% 23% 10% 4%
Working Experience	1–5 years 11–15 years 6–10 years 16–20 years Above 20 years	40% 23% 20% 13% 4%
Frequency of GIS Use	Few Times a Week Once a Week Once a Month	47% 40% 13%

, the survey collected 30 responses from a total of 45 respondents. Based on the responses captured in the survey, answers regarding demographic background indicated that 77% of respondents were male and 23% were female. In the context of age, the majority of the users were between 21 to 30 years old, the second largest group was between 41 to 50 years, and the third group was 31 to 40 years old. For education level, 63% of respondents had a bachelor’s degree, 23% had a diploma, 10% had a master’s degree, and only 4% had SPM. In terms of working experience, 40% of respondents were in the range of 1–5 years’ experience, 23% had 11 to 15 years, 20% had 6–10 years’ experience, 13% had 16 to 20 years’ experience, and 4% had above 20 years. The findings also indicated the highest frequency of use is a few times a week, with once a week as the second-highest, and once a month as the lowest.

Further analysis then proceeded by the identification of the Cronbach’s alpha. Cronbach’s alpha coefficients were evaluated in the investigation to establish the reliability and internal consistency of the scales used in the study, as shown in

Table 4. Cronbach’s Alpha Analysis Result.

Variables	Dimensions	Number of Items	Cronbach’s Alpha Value
GIS Quality	Functional Suitability Performance Efficiency Compatibility Usability Reliability Security Maintainability Portability	6 6 4 9 5 6 7 6	0.861 0.806 0.855 0.892 0.896 0.952 0.886 0.899
User Satisfaction		8	0.942
Individual Work Performance	Task Performance Adaptive Performance	6 5	0.955 0.915

. There are different arguments on the acceptable values of the Cronbach’s alpha value [23]. Any Cronbach’s alpha values beyond 0.6 to 0.95 for all factors indicated that the degree of dependability was acceptable and normal [24]. Based on the output of the SPSS analysis, the Cronbach’s alpha scores of all 68 items in the instrument for each dimension were between 0.942 and 0.975, according to the SPSS analysis findings. This outcome indicates that, at this point in the assessment, the overall index of the scale’s internal consistency within the instrument is reliable, with no unexpected irregularities discovered in the data.

6. Discussion and Conclusions

Ultimately, all factors have been analyzed, and the findings compiled from questionnaire surveys on the quality of GIS used by the organization, user satisfaction, and individual work performance have elaborate the readiness to proceed with the next steps of this study. The main objective of this paper was to have clarity on the reliability of the instrument. Based on Table 3, any values above 0.6 to 0.95 indicated that the items were reliable and could proceed to actual data collection [23,24,25]. For GIS quality, the lowest Cronbach’s alpha value is 0.806, and the highest value in the GIS quality items is 0.952. Meanwhile, for user satisfaction, the value is 0.942. Individual work performance results also indicated that the Cronbach’s alpha value for task performance was 0.955, while adaptive performance was 0.915. Based on Cronbach’s alpha value from the analysis, all items are above 0.6, and it indicates that there are no items that need to be deleted for this study [25,26]. Furthermore, the overall Cronbach’s alpha value of the study is 0.984, which is reliable and can proceed to the actual study.

Therefore, this paper aims to summarize current research progress on the influence of GIS quality. The research findings are intended to provide substantial evidence and insights into the critical factors in GIS quality, namely functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. This study also will provide an evidence on the influence of user satisfaction towards individual work performance and identifies the qualities needed in the GIS solution. Before the actual study, the instrument was pre-tested to identify potential errors and establish the degree of understandability of its items. In addition, six subject matter experts evaluated the instruments and gave constructive feedback. A pilot study was also undertaken, with 30 users participating. The data analysis results indicate that the overall index of the scale’s internal consistency within the instrument is reliable, with no unexpected irregularities identified in the data.