Conceptualization and Survey Instrument Development for Website Usability

Abstract

The aim of this study is to conceptualize website usability and develop a survey instrument to measure related concepts from the perspective of end users. We designed a three-stage methodology. First, concepts related to website usability were derived using content analysis technique. A total of 16 constructs measuring website usability were defined with their explanations and corresponding open codes. Second, a survey instrument was developed according to the defined open codes and the literature. The instrument was first validated using face validity, pilot testing (n = 30), and content validity (n = 40). Third, the survey instrument was validated using explanatory and confirmatory analyses. In the explanatory analysis, 785 questionnaires were collected from e-commerce website users to validate the factor structure of website usability. For confirmatory factor analysis, a new sample collected from 1086 users of e-commerce websites was used to confirm the measurement model. In addition, nomological validation was conducted by analyzing the effect of website usability concepts on three key factors: “continued intention to use”, “satisfaction”, and “brand loyalty”.

1. Introduction

Websites act as gateways for organizations to connect with the world. Despite the increasing prevalence of mobile devices, website usage is still significant among users. According to the statistics, more than 37 million websites are active around the world [1]. Most of them have a positive effect on users from the perspective of ease of life. However, many website designs with a low level of usability cause loss of time, demoralization, and disappointment for the user while using the website [2,3]. Furthermore, the level of usability plays a significant role in users’ adoption of a website. Therefore, usability is an important issue for organizations that develop and use websites to market their products and services [4]. Companies with big brands lose significant revenue because their websites are too complex for customers to understand [5]. The aim of a human-centered design approach is to develop more interactive systems, resulting in increased usage [6]. Institutions and organizations in various fields such as e-commerce sites, banks, aviation, and education depend on websites to respond to changing customer needs. Websites developed within the framework of human-centered design decrease the learning time of users using the system, accelerate expert performance, and consequently minimize user errors [7,8]. However, a website with low usage may direct users to use another equivalent or competitor website [9,10].

The concept of usability is based on human–computer interaction (HCI) literature. It helps designers to identify and rank user requirements by proposing guidelines, methods, and approaches [11,12,13,14]. As defined by the International Standardization Organization (ISO), “Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [15]. The specified context of use indicates that the relative importance of website usability metrics may change according to the context of the website. However, determining usability factors for a specific context may be cumbersome for both practitioners and researchers. The concepts developed for website usability in this study may be used for different types of websites in the development and evaluation phases.

Although studies [16,17] were proposed with an aim to measure the usability level of websites from the perspectives of customers or potential customers based on guidelines, such as the Microsoft Usability Guidelines [16] and Research-Based Web Design and Usability Guidelines [18], this study differs in certain aspects. First, we comprehensively conceptualized website usability based on the BS EN ISO 9241-151 guidelines [19] and the corresponding literature. Furthermore, we conducted a survey to assess website usability, which was validated based on the concepts’ open codes. Second, in studies that used surveys to evaluate website usability [16,17], the survey respondents were not actual users of the website. Therefore, the websites were first introduced to the respondents. Then, they were instructed to browse and examine the websites before answering the questionnaire. In contrast, the respondents in this study were actual users of the website. This allowed us to examine the relationship between website usability factors and factors, such as continued intention to use, brand loyalty, and customer satisfaction. Therefore, this study contributes to the existing literature on IS/IT adoption by linking website usability factors with the continued intention to use a website, customer loyalty, and customer satisfaction. Third, this study differs from that by [16] in that the website usability instrument was developed with a systematic approach based on the BS EN ISO 9241-151 guidelines [19] and measured using multiple-item measures, whereas Agarwal and Venkatesh [16] used a single item to measure subcategories that lacked statistical validity. Similarly, Hoehle and Venkatesh [20] developed an instrument to measure mobile device application usability based on Apple’s usability guidelines, whereas this study proposes an instrument to measure the usability level of websites. Therefore, the aim of this study is to conceptualize website usability based on the BS EN ISO-9241-151 guidelines [19] and the corresponding literature, develop a generic website usability survey instrument, and validate it using a systematic approach with the participation of experts and end users.

2. Literature Review on Website Usability

A literature review of website usability revealed that it can be examined in several ways, such as user testing with real or potential users [21,22,23,24], expert evaluations through heuristics [25,26,27], and reviews through usability guidelines [16,17,28].

In usability evaluation through user testing, potential users of a system perform several tasks determined by the system provider or analyst to understand the system. Based on the performance of the testers, the usability level of the system is measured using subjective or objective performance indicators. Several user-testing studies have been conducted to measure website usability [21,22,23,24,29]. Groth and Haslwanter [21] compared the usability of responsive mobile tourism websites with that of task success level, page views, task success time, and satisfaction. In addition, A/B testing experiments were performed by 20 participants who completed the given tasks, and their performances were calculated and compared. Shasha [22] assessed the usability of Cape Town hotel reservation systems through usability testing. Testing scenarios and questionnaires were given to 55 participants with prior experience in using the Internet to book online accommodation. The results revealed that most participants emphasized content simplicity and understandable product offerings, whereas a smaller percentage found the websites confusing or frustrating. Kous et al. [23] examined the usability of library websites with user testing applied to different types of user groups, such as pupils, students, seniors, and researchers. A total of 25 participants performed the tasks, and their respective performances were measured in terms of effectiveness, efficiency, and satisfaction. Then, group differences among the users were analyzed. Alexander et al. [24] analyzed the influence of user culture on website usability using unmoderated user testing. To compare the cultural differences, websites from China and Australia were translated into English and Chinese, respectively. Then, the tasks defined in user testing were conducted to measure the usability attributes of effectiveness, number of errors, efficiency, and satisfaction. Furthermore, Koonsanit et al. [29] indicated that 60 participants engaged in task-based evaluation of a travel agency website. To address variations in user behavior, the present study utilized randomly ordered tasks that must be completed by the participants in different sequences. The results demonstrate that considering the actual order of user actions over random ordering significantly influences the prediction of final user satisfaction. Although the studies that conducted user testing for website usability provided insights into the usability of the system, the problems associated with usability determined with user testing were limited to the tasks defined and did not provide a comprehensive perspective for overall website usability. However, this study provides a comprehensive way to conceptualize website usability based on several concepts and developed a survey instrument to measure website usability from the perspective of actual website users.

Some studies have also used heuristics to evaluate and identify website usability problems [25,26,27,30]. Torrento et al. [30] defined usability evaluation conception based on heuristics and named the system “Sirius”. Experts evaluated websites by considering the proposed evaluation criteria, and a usability metric that quantified the level of usability achieved for different types of websites was developed. Fung et al. [25] conducted a heuristic evaluation using Nielsen’s ten heuristics for the mobile websites of three university libraries. These websites were examined and compared in terms of usability issues encountered by experts, and possible solutions to the usability problems were proposed. Verkijika and Wet [26] assessed the usability of 279 e-government websites in 31 countries using a heuristic evaluation method. Five experts conducted evaluations using the heuristics of online services, user help, navigation, information architecture, and accessibility. Silvis et al. [27] proposed a set of heuristics by combining Nielsen’s ten heuristics [8], the golden rules of Shneiderman [31], and the dialogue principles of ISO 9241-110 [6]. Each heuristic was explained with reference to the literature, and recommendations related to each heuristic were proposed. Similarly, in the present study, the open codes defined for each concept helped experts develop and evaluate website usability.

A group of studies has focused on understanding the user perspective regarding the usability of specific systems in various domains, such as tourism websites [32,33,34], government websites [35,36], e-commerce websites [37,38,39], or university-related websites [40,41,42,43]. These studies provide valuable insights into the usability aspects of websites in specific domains and help improve user experience and design in these areas. However, the present study’s conceptualization concerned website usability for different types of websites, thus providing a comprehensive description of website usability.

However, certain other studies conceptualized website usability based on guidelines, such as the Microsoft Usability Guidelines (MUG) [16,28], Research-Based Web Design and Usability Guidelines [17], and the literature [44]. Agarwal and Venkatesh [16] described a method to determine the level of usability of websites using a heuristic evaluation. The set of heuristics was adapted from the MUG, which has five major categories: content, ease of use, promotion, made-for-the-medium, and emotion. Four of the five categories have subcategories, and each subcategory is measured using a single-item measure. The respondents first examined and browsed one of the four websites, namely airlines, bookstores, auto manufacturers, and car rental agencies, and then answered the questionnaire as a customer or investor. A total of 1475 questionnaires were collected, and a comparison was drawn across industries and roles. Venkatesh and Ramesh [28] conducted a follow-up research to the study by Agarwal and Venkatesh [16], who used the same usability model to examine the usability of multiple websites across four industries to demonstrate the generalizability of MUG conceptualization. Within the scope of [28], interviews were conducted, and questionnaires were collected from the actual users as a subset of Agarwal and Venkatesh’s previous study [16] to assess wireless site usability and explore distinctions between web and wireless sites. This model was tested in Finland to apply to the airline industry that one of the sectors studied [16]. First, participants operated as customers and indicated the relative importance of the MUG categories; after browsing each website, they rated the airline industry website with MUG on a 10-point scale. The study analyzed how users prioritize the assessment of usability between web and wireless sites based on weighting MUG metrics. The presented results offer robust endorsement for employing MUG as a means to comprehend site usability and usage in both web and wireless environments and demonstrate the generalizability of the MUG to Finland. Venkatesh et al. [17] developed a usability instrument based on Research-Based Web Design and Usability Guidelines [18] to evaluate the usability level of the healthcare insurance website in the U.S. The instrument consists of 16 dimensions to measure the overall usability level and 205 items representing the dimensions. A total of 374 citizens answered the questionnaires, and only 144 of them had previously used the website. Therefore, at the beginning of the survey application, the website was introduced to respondents through video clips and screenshots. According to the results, six (user experience, content organization, navigation, graphs, lists, and screens) of the sixteen usability dimensions were found to be predictors of citizen satisfaction and intention to use e-government portals. Venkatesh et al. [45] used the same dimensions as Venkatesh et al. [17] and found that five dimensions, namely hardware and software, home page, screen, scrolling and paging, and user experience, were rated low in terms of usability. Then, the usability problems for each dimension were determined, and recommendations for the problems were proposed. Lee and Kozar [44] analyzed the common dimensions of website usability through a comprehensive review of 27 academic journals and conference papers. Ten constructs, namely telepresence, supportability, consistency, content relevance, credibility, readability, learnability, simplicity, navigability, and interactivity, as well as their corresponding survey items were defined based on the literature. The causal relationships among the constructs were defined using the quantitative cognitive mapping method. Additional data were collected from 711 Amazon website users, and the relationships among the constructs and their effects on purchase intention and purchase were revealed to validate the nomological network of website usability.

A comparison of this study with the existing literature reveals that it contributes to the literature in several ways. First, it covers the points of the BS EN ISO-9241-151 guidelines [19] offered by the ISO, which aim to help website developers design websites that are usable. However, current guidelines do not provide important points, nor do they provide a way to evaluate websites in a concrete manner. For example, the widely accepted scale, The System Usability Scale (SUS) [46], commonly used for the rapid measurement of system usability, provides general usability satisfaction but does not contain information related to usability issues. Our detailed instrument addresses this gap, which provides a comprehensive evaluation that measures overall satisfaction and identifies specific usability issues. In this study, website usability was conceptualized based on the BS EN ISO-9241-151 guidelines [19], and additionally, literature was included. Subsequently, a comprehensive and context-specific survey instrument was developed and validated to help developers design and improve websites according to the defined usability criteria. Furthermore, the present study is the first to develop and validate a fine-tuned instrument for measuring website usability. Second, the concepts defined by the open codes may be used as a reference list to check the conformance of a specific website to the defined open codes from the experts’ perspective. Therefore, the proposed website usability evaluation system can be used effectively by end users and experts.

3. Conceptualization and Instrument Development

In this section, we implemented a three-step formal methodology. First, website usability was conceptualized using the axial and open coding content analysis techniques proposed by Corbin and Strauss [47]. Subsequently, based on the concepts and open codes defined in the first step, we developed a website usability survey instrument. Finally, we validated the survey instrument and proposed an instrument specifically for website usability measurements.

3.1. Conceptualization of a Website Usability

In this study, BS EN ISO-9241-151 is considered the base content for the conceptualization of website usability. We applied Corbin and Strauss’s [47] open and axial coding procedures for conceptualization. Open coding is “the analytic process through which concepts are identified, and their properties and dimensions are discovered in data” [47]. In general terms, data are segmented into distinct pieces during the open coding process, carefully analyzed, and assessed for similarities and differences [47]. We examined the BS EN ISO-9241-151 guideline to implement the open coding procedure and coded the content using line-by-line analysis. Furthermore, the literature related to website usability was reviewed to include additional open codes [16,17,18,30,48]. The final list of the open codes was examined by five experts experienced in HCI. Next, the experts grouped the open codes that were conceptually similar or related under subcategories.

In later analytic steps, axial coding was implemented. Axial coding is defined as “the process of relating categories to their subcategories, and it is termed “axial” because coding occurs around the axis of a category, linking categories at the level of properties and dimensions [47]. Categories and subcategories were linked in axial coding to create more thorough and accurate conceptual explanations. For example, we identified two open codes related to error prevention on web pages: (1) The content of error messages should be shown on web pages at related positions with clear reason statements, and (2) the user should be directed to solve basic errors by oneself. Then, these two open codes were conceptualized as a single subcategory labeled “Understandable Error Messages”. Then, through axial coding, the category was identified as “Error Prevention”.

The open codes, their corresponding subcategories, and axial codes are given in Appendix A—

Table A1. Axial codes, subcategories, and their open codes.

Axial Code	Subcategory	Open Code
Brand Identity	Clear and accurate identity about website and its owner	The identity of a website, website owner, or business entity should be presented accurately and unambiguously on all relevant pages of website.
	Consistent identity on each page belonging to the same website	All pages or windows associated with a specific website must be easily recognizable as belonging to that website. Consistent design elements, such as branding icons and slogans across all pages, are employed to ensure clarity and prevent any confusion in users’ cognitive perception.
	Informative homepage	Home pages should provide sufficient information about the general framework of website so that user can review and anticipate the purpose of the site.
	Memorable website URL	The URL used to access the website should be crafted as a memorable tagline, thus making it easy for users to remember.
	Overall identity for several websites used in an organization	If an organization disperses several websites that address the same audience, the overall design should be consistent. Brand identity elements should be placed in each website consistently.
Content Design	Logical conceptual model of website	The conceptual model of website should be designed according to the user’s mental model.
	Relevant content according to website’s purpose and target audience	The content should be relevant for the purpose of the website. The particular tasks and information should be comprised of appropriate substance according to the target audience needs.
	Appropriate details about content level	Content should be given in units for well-organized information flow and appropriate level of granularity.
	Up-to-date content availability	In situations where the validity or relevance of content is time-sensitive, it is important to display current information and emphasize its time dependency to the user. The last update and period of validity if applicable should be available.
Personalization and Customization	Personalized access paths	Different access paths or structures should be designed for the differentiated user groups.
	Evident user individualization	When personalization is used according to user profiles, it should be made evident to the user.
	Adjustable user profile settings	If user-specified profiles are used, users should be able to see, modify, and delete that profile on demand.
	Automatic customization to user preferences	Users should be informed about automatic adaptation of the website.
	User permission and accessibility	Users should have an authorization to the complete content of the website.
Privacy and Business Policies	Personal data usage information	Users should be informed about how personal information is used. It is important to seek users’ approval before using personal information.
	Simple and clear privacy policy	When the user inputs personal information or initiates transactions on website, the user should be aware of privacy policy statements that are simple and clear.
	Availability of business policy statement	If business policy statement is needed for operating, it should be available on the website.
	Information on additional installations required	The policy for storing data rights or executable programs that are activated on user’s local device during usage should be informed.
Page Design	Descriptive page title	Every page should display a descriptive title.
	Consistent layout schemes	Pages should be designed using consistent layout schemes such as consistent location for page titles.
	Attract new content or temporary content	If there is new or significant changing content, the user should pay attention to it.
	Appropriate page lengths, including horizontal and vertical scrolling	The length of a page should be designed to support the primary usage objects of the website’s pages. Vertical scrolling should be minimized, and horizontal scrolling should be avoided wherever possible.
	White space design	The white space of page should be organized optimally without eye-straining and limiting the visual skimming of the page.
	Home page design with all features of a homepage	The home page should show features that summarize entire scope as an introduction page to users.
Text Design	Readable texts	Texts on website pages should be presented with high readability according to expected display characteristics and spatial arrangement. Text can be rescaled-to-fit according to user’s preferences.
	Support fast skimming	Fast skimming should be enhanced with encapsulating sentences and short and clear terms supported with comprehensible headings, links, bullets, and highlighted keywords.
	Text quality	To ensure readability, the textual content should maintain a satisfactory level of spelling and grammar quality.
Link Design	Identification of links	The links should be differentiated from basic content by underlining, coloring, or highlighting. Elements other than links should not look like links. All links in same website should be designed in consistent manner.
	Formatting of the links in the text	When the links are combined with text on website, links should be formatted in short so that the presence of links does not block the text readability.
	Descriptive link with labels	The links should be tagged with appropriate descriptive labels.
	Clarify the target addresses of links	Links that direct a new opening browser or pop-up window may cause uncertainty. Thus, user should be informed about where to divert. Links guidance to special targets (such as video and large-scale downloadable files) should be clearly identified with marks. Links that have been previously clicked should be differentiated with distinct coloring, etc.
Interaction Objects	Logical and easy data-entry options	Select interaction objects that align with the logical properties of user’s expected input/output balance and make the user’s tasks easier.
	Providing keyboard shortcuts	By offering keyboard shortcuts, websites can cater to users who prefer or require keyboard-based navigation.
	Identifiable and understandable interaction objects	Interaction objects should be coherent to reality.
Graphics, Images, and Multimedia	Appropriate media objects for the content and user group	The media objects should be chosen as relevant, engaging, and supporting the overall goals of the website.
	Providing multi equivalent options to reach media objects	To ensure continuous accessibility, alternative access methods should be synchronized, such as audio and subtitles for multimedia elements.
	Control time-dependent media objects	Pause, stop, and go back and forth functions should be provided for multimedia objects when they are time dependent.
	Quick view for media elements	Media elements should be previewed without long loading times.
Navigation	Clear navigation structure	Navigation should be designed to orient users to understand the structure of website and how to navigate within pages. When the navigation structure includes multiple levels, it is important to design the navigation component in a way that allows for the simultaneous display of more than one level of navigation. Instead of a straight level of navigation, if a task requires sequential processes, providing progressive navigation structure and its visibility is more convenient for users. Step-back function between sequence steps should be provided for integrity of navigation structure.
	Making navigation links obvious	The main navigation links should be in sight properly whenever the user needs them.
	Placing navigation components consistently	Navigation components should be shown consistently on each page of website.
	Overview navigation structure	To show the grand scheme of the website structure, a site map should be provided for websites to understand website content overall.
	Avoiding opening unnecessary windows	Number of extra opening windows such as splash screens should be minimized. If they do not provide useful extra content or feedback about system, they should be removed. If they are necessary, an option to skip must be available.
	Home page as a top-level node of navigation	The top-level structure of the navigation should be defined in the home page.
Search	Providing search functions	Search functions should be reachable from every part of website with different searching options according to possible experiences on website, such as advanced search and search with keywords.
	Search entry	User should be directed for input entry with appropriate instructions. The search techniques should be clear for the user to understand how to enter input and obtain results.
	Provide user query and search results accordance	The number of results found with the entered query should be provided to the users. To handle large search result sets, displaying the result technique should be stable.
	Appropriate sort and filter options of search results	Sort and filter option of search results is significant for user to find searches. Search results should be aligned with the user needs. There should be alternative sorting and filtering options.
Help	Providing help pages, FAQ section, or special assistance for user	Website should provide help pages, FAQ section, or special assistance like chatbot to users whenever they have issues or problems.
	Accessible communication with website owner	It is important for user to be able to easily communicate with the website owner or business.
Responsiveness	Optimum independent design for different operating systems, browsers, and screen resolutions	Design for popular operating systems, browsers, screen resolutions and optimize for mobile devices. Web user interfaces should be designed to accommodate various input devices, allowing users to activate controls through methods such as keyboards, mice, touchscreens, or assistive technologies.
	Using web technology standards	Generally accepted web technology standards such as XHTML, CSS, and others should be used.
	Flexible to changing technology	Web user interfaces should be robust to changing technology or browsers. The applets or plug-ins embedded in a webpage should have same usability and accessibility requirements with the website.
	Content adaptable to different contexts	Content should be adapted to different contexts such as internalization, mobile, or specific device.
Error Prevention	Minimizing potential user errors	User errors during transactions or performing tasks needed to be minimized with some warning directives to users.
	Understandable error messages	The content of error messages should be shown on web pages at related positions with clear reason statements. User should be directed to solve basic errors by oneself.
	Acceptable loading times	The download times of the webpage should be in acceptable range according to user expectations.
	Time-out warnings	User should be informed about scheduled time outs before being forced to log off. If user requests, additional time can be provided before time expires.
Collaboration	Online feedback mechanism	A mechanism should be provided to allow users to follow-up progresses by sending comments, questions, or ratings within the system and a two-way interaction between the system and users.
Cultural Diversity and Multilingual Use Design	Inputs and outputs of a website compatibility to international audience	The input/output of information or measurement units such as currency, time zone, temperature, phone numbers, address, or postal codes should be designed obviously for inquiring international audience.
	Information about geographical context	Arranging context according to information about geographical location provide users with the information whether the website is proper for their intended tasks.
	Supporting different languages and customize accordingly	Support website for different languages and, if possible, identify the user’s language. Additionally, customize the website according to the characteristics of different languages.

. A total of 16 axial codes were defined: brand identity, content design, personalization, and customization; privacy and business policies; page design, text design, and link design; interaction objects; graphics, images, and multimedia; navigation; search; help; responsiveness; error prevention and collaboration; and cultural diversity and multilingual use design.

3.1.1. Brand Identity

Users’ perception of a website is formed within the first few seconds of a visit, and this impression is largely based on the website’s visual design and branding elements [49] because websites represent businesses [38,50,51]. A memorable website URL or informative homepage affects the branding identity and initial impression of a website user [52,53]. Furthermore, an identifiable and informative website consistent with its intended purpose and target audience is crucial for establishing users’ trust and credibility [54,55]. As a result, it is easier for users to navigate and use such websites effectively [56].

3.1.2. Content Design

Website content should be designed according to users’ goals, knowledge, and preferences. The comprehensiveness, richness, and completeness of the content are also considered as key characteristics of a website [57]. Effective website content enhances user satisfaction, reduces frustration, and improves business outcomes [58]. According to Agarwal and Venkatesh [16], website content is the most crucial attribute of users’ website preferences. The quality, accuracy, and reliability of information are also critical for usability, which makes a website more credible [59,60]. The content must have an appropriate level of detail and organized information and must be presented in a clear and logical structure that can be understood by typical users [19,61,62]. A clear, logical structure enhances user interest, whereas a mismatch between the conceptual and user’s mental models results in confusion and frustration [18,63]. Furthermore, time-dependent content must be up-to-date, and the users must be aware of the last update or validity period [60].

3.1.3. Personalization and Customization

Personalization involves dynamically adapting the content, recommendations, and user interfaces based on the specific characteristics and behaviors of individual users, whereas customization allows users to modify various aspects of a website’s design and functionality. By combining these approaches, websites can create highly personalized and adaptable experiences for every user [64]. Some authors have highlighted that personalization improves trust perception and usability [65,66]. A personalized website provides an experience based on user-specific data, such as location, browsing history, and search queries [67]. It is the ability of a website to adapt to users’ actions and behavior over time [68]. The dynamic nature of user interests and consumer behavior is influenced by easy access to abundant information and social factors [69]. By utilizing technologies such as analytics and advanced machine learning algorithms, vast quantities of data can be analyzed to forecast user preferences and behaviors. Thus, websites should employ analytical processing to enhance learning, user profiling, and predictive models, ultimately providing users with valuable and personalized recommendations [67,69,70,71]. However, customization empowers users to personalize their own experience by giving them control over certain aspects of the website’s design and functionality. Venkatesh and Ramesh [28] argued that customized websites cater to users’ desires in a short period of time. However, it is important to balance personalization and customization to avoid overwhelming users with too many options or too much information.

3.1.4. Privacy and Business Policies

Website usability and user reliance are closely linked to clear and concise privacy and business policies outlining data collection, usage, and sharing practices [72]. A website’s privacy policy outlines how user data are collected, used, and shared, whereas business policies provide details on the website’s terms and conditions for conducting transactions with users [73]. Transparent privacy policies may increase user trust and engagement during transactions in website [74,75]. When collecting personal information for personalization, website policies should establish user trust and protect user rights [19,76,77,78,79]. Several studies have shown that usability is inversely associated with the lack of privacy and security perspectives in areas such as online learning, e-banking, and e-commerce [80,81,82]. For instance, many consumers are concerned about their privacy and security during purchasing transactions, especially in the e-commerce domain [83]. Confusing or difficult-to-locate policies may negatively impact user experience and lead to frustration and abandonment [84]. Similar to privacy, if required, transparent business policies also can increase user satisfaction and trust.

3.1.5. Page Design

Page design properties are significant in accurately conveying content to the user while ensuring the visual integrity of the content. Presenting content via stable, standard webpage elements familiar to users, such as page structure, list level, and style, is important for usability. Well-designed pages improve user experience, increase engagement, and promote conversion rates [85]. A study on e-commerce remarked that the conceptualization of website aesthetics affects the website context [86]. Well-placed headings and short and readable paragraphs are essential for page design; therefore, it is easy for the user to scan. A clear and intuitive layout can guide users through the page and help them quickly find what they are looking for [87]. Furthermore, the visual hierarchy may direct users’ attention to key elements on a page and facilitate information processing [88,89]. It should be ensured that the desired topic is highlighted on a page. According to the research of Leavitt and Shneiderman [18], 80% of users pre-scan when looking at a webpage and aim to find what they are looking for at a single glance.

3.1.6. Text Design

Consistently readable content purged of technical language increases usability. The text format of website content is one of the basic design elements of a website [90,91]. The use of clear, concise, and easy-to-read text may improve comprehension and task completion [92]. However, poor text design, such as small fonts or poor color contrast, results in user frustration and disengagement. Information on websites should be displayed in an appropriate, de facto font size [93,94,95,96]. Research-Based Web Design and Usability Guidelines [18] and other studies advise using a font size of at least 12 points and consistent font formats in all pages [97,98]. Using simple and clear language, avoiding jargon and technical terms, and breaking down information into manageable chunks may improve website usability [99].

3.1.7. Link Design

The role of links is to ensure that internal and external page connections on the website are directed to the correct address when an article, photograph, or object is clicked. Well-designed links may help users to navigate websites efficiently, whereas poorly designed links can lead to confusion [100]. The most important aspect of link design is that it should be distinguishable from ordinary content [101]. Links can be presented in different ways via text, buttons, etc., as long as they are discriminated from other content on the website pages [19]. Furthermore, the use of descriptive links rather than generic phrases improves users’ understanding of the link’s destination and increases their trust in the website [102,103,104,105]. According to previous studies, users are more likely to click on links that are underlined and highlighted in a color different from the surrounding text [106,107,108].

3.1.8. Interaction Objects

Interaction objects allow users to interact with the websites. They are defined with buttons, check boxes, dropdown lists, and any data-entry fields, such as completing forms and texting into boxes that allow the user to interact with the website conversationally [19]. Minimized data-entry sections and simple mechanisms for user interaction should focus on website usability. To ensure the effective use of interaction objects, designers should ensure that they are easily recognizable and that their functions are clearly labeled [58,109,110]. The use of consistent interaction objects throughout a website also improves usability, as users can learn and recognize them quickly [31]. Furthermore, using realistic interaction objects that are compatible with the tasks and expectations of the user enhances website usability by reducing the cognitive load and contributing to trust [111]. In addition, three-dimensional (3D)-style buttons enhance the recognition of clickable buttons and their accessibility [112]. Interaction objects should also be located in easily accessible and used locations, such as in the main navigation bar or header of the website, which ensures that the interaction objects are placed with the alignment of user expectations and behaviors [113,114]. Moreover, each entry field should be clearly and consistently labeled in the appropriate positions to guide the user interactively [18]. Manual information entered externally by users should be minimized as much as possible.

3.1.9. Graphics, Image, and Multimedia

When attractive content, such as images, animation, video, and audio, is used appropriately, it provides additional value to a website [115]. Research has also shown that the quality of graphics and images affects the perception of the user about a website and the credibility of the information presented [116]. Featuring images and graphical elements within a website transmits messages more accurately to users and improves the website’s overall usability and the perceived attractiveness of websites [117,118]. Moreover, promoting the use of appropriate graphics, images, and multimedia enhances website aesthetics, resulting in user satisfaction [95,119]. However, there must be a balance in the use of multimedia objects. Excessive use of these elements may cause distractions and have a negative impact on website usability [120,121]. Images or videos with large file sizes can significantly affect download times, particularly at slower connection speeds, owing to the substantial number of bytes required. The loading speed of graphics and multimedia should be optimized to prevent slow loading times so that users do not get frustrated and abandon the website [101,122,123]. It is beneficial to introduce a medium element before initiation. By utilizing thumbnail versions of larger images, users can preview images without fully downloading them. This approach not only provides a convenient way to preview images but also helps prevent any slowdown in the download time of web pages [17,18]. HHS design guidelines propose reducing image size so as not to slow down page loading [124].

3.1.10. Navigation

Clear navigation provides users with an understanding of the website’s overall structure. Many researchers emphasize the importance of the ease of navigation for usability because effective navigation structures [125] allow users to find information easily [14], thus reducing the users’ effort and time spent on navigation. The homepage serves as the website’s primary navigation element. It is crucial to ensure that the main navigation links are prominently displayed and easily accessible whenever users seek to accomplish their goals on the site. According to Yu and Roh [126], different menu designs affect user perceptions of easy navigation within a website. The placement and design of navigation menus may affect website usability because users often expect to consistently find navigation menus in certain locations [102]. Moreover, establishing a consistent logical mapping between the structure of a website’s content and selecting a suitable placement of navigation elements is important [127]. Additionally, splash screens are often used to provide an initial visual experience when a website or application is loaded. According to ISO 9241-11-2018 [15], if a splash screen is used, users must be provided with a navigation option to bypass the screen.

3.1.11. Search

Search functionality is an essential aspect of website usability, as it allows users to quickly and easily find the required information [128,129,130]. Adequate research has been conducted to show that search functionality can significantly improve user satisfaction and task-completion rates [131,132,133]. According to Spool et al. [107], searching can be a faster and more efficient way for users to find content than browsing menus and links. However, the usability of a search feature depends on factors, such as the search algorithm, which generates pertinent outcomes based on anticipated keywords commonly used by users [134], presentation of search results, and availability of advanced search options. To optimize the user experience, it is recommended to minimize the search interface design and eliminate any superfluous elements that may cause distraction [135]. According to the Nielsen Norman Group, users who cannot find the desired information in their first search rarely attempt the next search, and almost half of them tend to leave the system [136]. As part of the search mechanism, website designers may use advanced techniques such as auto-suggestion, faceted search, and semantic search to enhance their search experience [137,138]. Furthermore, the search results shown to the user should be ordered according to the relevance of the required information. Mechanisms such as sorting and filtering can enhance the performance of a website by allowing users to arrange search results based on their specific needs.

3.1.12. Help

Users complete tasks successfully and express satisfaction when provided with clear and concise instructions or guidance. The use of interactive tutorials, context-sensitive assistance, and error messages with helpful hints may also improve usability. Furthermore, it is important to ensure that help features are visible and easily accessible to users without overwhelming or distracting them from the primary task. In addition, it is important to identify a contact who can obtain information or solve problems [19]. Helping pages, a section on frequently asked questions (FAQ), or special assistance such as chatbots should be easily accessible. Additionally, incorporating natural language processing and machine learning techniques may improve the accuracy and relevance of help responses [139].

3.1.13. Responsiveness

Responsiveness is an important component of system and service quality [140]. Responsive website design allows a website to adjust its layout and content according to the size and type of the device being used [141]. Research has confirmed that poor responsiveness may force users to stop using websites, especially during critical system requirements [79,140,142]. A website’s user interface can be achieved through various input devices; therefore, a robust design that is technology-independent should be adopted. In addition, it is important to keep up with commonly used browsers and screen sizes [143,144,145], and the site should function efficiently on devices with similar designs [146].

3.1.14. Error Prevention

It is essential to implement techniques such as error prevention, error correction, and clear error messages so that users can navigate websites efficiently. Minimizing the effort spent on recovering errors is essential in website design. Thus, to support recovery errors, potential user errors should be minimized as well. It is advisable to use contextual information and visual cues to guide users through websites via lucid and concise messages [114,147,148,149]. Essentially, error messages or alerts should be designed to be easily comprehensible to an average user rather than exclusively targeting IT specialists [19,150]. However, an efficient system design can prevent such problems [19,151]. In addition, designing error-prevention mechanisms such as automatic saving, undo and redo functions, and confirmation dialogs may significantly reduce the occurrence of errors and increase user productivity [152].

3.1.15. Collaboration

A system with collaborative features provides users with opportunities to interact with other users, share information, and provide feedback to the system, such as through ratings, voting, and comments. Recent technological developments have enabled websites to project a personal environment, offer support from experiences shared by other people, and provide motivation from the power of the community rather than just browsing [153]. User-generated content such as comments and reviews may enhance website credibility and increase user engagement [154]. Adequate research has been conducted to show that customer reviews and ratings may influence purchasing decisions, improve customer loyalty, and increase user trust in websites [155,156,157,158]. However, it is important to ensure that user experience features are easy to use and understand because overly complicated systems can detract from usability [159]. Recommendation systems for website collaboration mechanisms may also encourage users to use the website.

3.1.16. Cultural Diversity and Multilingual Use Design

The use of language, symbols, color, and layout may be considered [160] as crucial for supporting cultural diversity and multilingual use. Usability is closely linked to social and cultural characteristics, and website designers should consider the relevant characteristics of different user groups to improve usability [24,161,162]. Furthermore, providing multilingual options and localized content effectively enhances usability, increases business opportunities in global markets [119,163], as well as increases user engagement and satisfaction. For instance, Tsai [164] revealed that users in the U.S. prefer more personalized pages, whereas Taiwanese users prefer collective pages with more images and icons; therefore, the design differs for the same website. Culture and language affect information processing and interaction between users and interfaces as seen in examples, such as Korean, Chinese, and Arabic web pages [124,165,166,167]. Moreover, Ying and Lee [168] revealed that users expect to browse website content along different paths related to their cultural cognitive style. To accommodate cultural diversity and multilingual use, website designers should use visual cues, offer language selection options, and consider the cultural implications of design choices such as text, interaction objects, and input elements to enhance success and user acceptance [24,169].

Table 1. Conceptualization of Website Usability.

Axial Code	Axial Code Brief Definition	Inspiring Prior Literature
Brand Identity	The user’s perception about easily identifiable website identity with its informative homepage and memorable URL.	[16,17,18,19,30,18,48]
Content Design	The user’s perception about the relevant and up-to-date website content with appropriate level of granularity.	[8,16,17,18,19,48]
Personalization and Customization	The user’s perception about individualized and user adapted websites.	[16,17,18,19,30,48]
Privacy and Business Policies	The user’s perception about clear and easy-to-understand website’s privacy and business policies.	[19,30,48]
Page Design	The user’s perception about consistent page layout scheme and descriptive page design.	[17,19,30,48]
Text Design	The user’s perception about readable and high-quality text design.	[17,19,30,48]
Link Design	The user’s perception about consistent, well-defined, and distinguishable link design.	[17,19,30,48]
Interaction Objects	The user’s perception about interaction objects in terms of layout and easiness of use.	[17,19,18,48]
Graphics, Images, and Multimedia	The user’s perception about the appropriateness, controllability, and quick view of the graphics, images, and multimedia elements.	[16,17,18,19,30,48]
Navigation	The user’s perception about clear navigation structure, links, and components.	[17,19,30,48,57]
Search	The user’s perception about search options and their functionality to offer direct and optimal access to the content.	[17,19,30,18,48]
Help	The user’s perception about accessible help options.	[8,17,19,30,48]
Responsiveness	The user’s perception about website independence according to input device and adaptability of content.	[8,17,19,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57]
Error Prevention	The user’s perception about preventing and minimizing errors with clear error messages or directions.	[8,17,19,30,18,48]
Collaboration	The user’s perception about website’s interactive online feedback mechanism.	[16,19]
Cultural Diversity and Multilingual Use Design	The user’s perception about website’s cultural diversity and multilingual use.	[17,19,30,48]

presents the constructs, their definitions, and the literature cited in these constructs.

3.2. Survey Instrument Development

A survey instrument specific to website usability evaluation was developed based on the concepts developed in the first phase. First, five researchers examined the open codes in detail and developed a list of possible items to explain the 16 constructs. In this phase, a total of 134 items were used to define 16 constructs. The survey instrument was validated using face validity, pilot tests, and content validity checks that were performed simultaneously. The face validity of the survey instrument was assessed with a checklist that included the following questions: “Applicability of the items”, “Is the item redundant and must be deleted”, and “Is this item ambiguous and must be deleted”. Ten participants examined each item according to the checklist and provided relevant comments and suggestions. The participants’ reviews were evaluated by the researchers of the study, possible changes were discussed, and a decision regarding the items was made. Based on the decision made, we checked whether removing an item changed the content of the construct, and it was considered that each construct was represented by at least four items. Consequently, 91 items were available for the pilot test.

In the pilot test, the survey instrument prepared in English was translated into Turkish. A bilingual translation was conducted to ensure equivalency of the Turkish and English versions of the survey instrument. The participants answered questions on the most commonly used e-commerce websites. Accordingly, 30 users with shopping experience on e-commerce websites and representing the main population participated in the pilot test study. Thereafter, the respondents’ demographic profiles were framed as shown in Appendix B,

Table A2. Demographic profile of the respondents of pilot study, content validity, and explanatory and confirmatory study.

		Pilot Study		Content Validity Check		Exploratory Factor Analysis		Confirmatory Factor Analysis
Demographic	Category	n = 30	%	n = 40	%	n = 785	%	n = 1086	%
Gender	Male	11	37	12	30	359	46	511	47
	Female	19	63	28	70	413	53	560	52
Age	<20	0	0	0	0	92	12	119	11
	20–29	5	17	6	15	209	27	281	26
	30–39	21	70	32	80	311	40	426	39
	40–49	3	10	2	5	123	16	205	19
	>50	0	0	0	0	43	5	50	5
Job	Banking and Finance	6	20	3	8	134	17	43	4
	IT	5	17	18	45	126	16	65	6
	Education	3	10	9	23	56	7	36	3
	Clothing and Textile Industry	1	3	0	0	14	2	18	2
	Construction and Engineering	4	13	2	5	47	6	220	20
	Public Services	6	20	4	10	27	3	111	10
	Healthcare	2	7	1	3	33	4	34	3
	Insurance	0	0	0	0	12	2	24	2
	Student	0	0	0	0	114	15	168	15
	Marketing and Advertising	1	3	1	3	8	1	18	2
	Telecommunication		0	1	3	15	2	31	3
	Other	2	7	1	3	150	19	241	22
E-commerce Preference	Trendyol	22	73	20	50	313	40	518	48
	Hepsiburada	5	17	10	25	257	33	305	28
	N11	2	7	10	25	215	27	263	24
E-commerce Experience	<2 years	0	0	0	0	125	16	169	16
	2–5 years	9	30	0	0	289	37	454	42
	6–10 years	14	47	4	10	217	28	299	28
	>10 years	6	20	36	90	146	19	157	14
Browser Preference	Google Chrome	27	90	35	88	527	67	750	69
	Microsoft Edge	0	0	1	3	52	7	62	6
	Mozilla Firefox	0	0	2	5	70	9	88	8
	Other	3	10	0	0	122	16	165	15
E-commerce Website Frequency of Use	>Once a day	3	10	3	8	159	20	227	21
	Once a day	2	7	2	5	92	12	150	14
	Several times a week	14	47	20	50	222	28	294	27
	Once a week	3	10	5	13	102	13	140	13
	Several times a month or less	7	23	10	25	198	25	263	24
Average Time Spent on the E-commerce Website (in days)	<One hour	19	63	20	50	394	50	516	48
	1–3 h	10	33	15	38	290	37	436	40
	4–6 h	0	0	5	13	76	10	103	9
	>6 h	0	0	0	0	16	2	19	2

. The participants were requested to review each evaluation item in the survey and provide comments on any item. Based on participants’ evaluations, complex and incomprehensible items were revised. Additionally, items deemed unsuitable for evaluation by end users that could not be evaluated by the participants were removed from the survey. Of the 91 items in the pool, 11 were marked for correction. Considering that each structure needed to be described by at least four items [20], seven items were eliminated during the pilot test, and 84 items remained for the next content validity phase.

Third, a content validity approach was used to screen the items of the survey instrument. Content validity is the degree to which a developed scale represents the domain of a relevant construct [170]. To verify the items, a content validity check was applied following Andersen and Gerbing’s [171] approach. In this approach, the final items are listed in the rows, and the definitions of the constructs are given in the columns. The respondents were then asked to select the most appropriate item and construct a match as shown in

Table 2. Matrix Example for Content Validity.

Items	Constructs Definitions
The website…	1: The user’s perception about website’s brand identity.	…	16: The user’s perception about website’s cultural diversity and multilingual use.
integrates brand elements such as color and icon effectively into the interface;
can be recognized with its logo;
pages have the same branding elements;
URL name is clear and easy to remember;
…
presents information in a familiar format (currency, temperature unit) for user;
adapts context according to geography;
supports other languages;
recognizes user’s language preference automatically.

. Hunt et al. [172] suggested that the minimum number of participants for assessing scope validity is at least 12. Within the scope of the study, 40 participants who had experience with shopping on e-commerce websites were requested to fill out a matrix containing the evaluation items from the survey and the most appropriate categorical structure matching.

Content validity of the data was assessed using the proportion of substantive agreement (P_sa) and substantive validity coefficients (C_sv) as explained by Anderson and Gerbing [171]. The value of P_sa was calculated as the ratio of the number of respondents who positioned items to the intended construct to the total number of respondents. The value of P_sa ranges between 0 and 1, and higher values for each item indicate a higher level of content validity. C_sv is the ratio at which respondents positioned items as the most chosen construct rather than the intended construct. It was calculated as the ratio of the difference between the number of assignments to the intended construct and second-highest assigned construct to the total number of respondents. The range of C_sv values typically is between −1 and 1. Positive values indicated that items were assigned to their intended constructs more frequently than to any other construct [171]. Negative values indicated the opposite. Higher values for both indices indicated greater substantive validity. Usually, 0.5 is the recommended threshold for sufficient validity [171,173]. However, a threshold value of 0.6 was selected for P_sa and C_sv proportions, as suggested by Hoehle and Venkatesh [20] for more prudent results.

In this study, questionnaire items with P_sa and C_sv values of 0.7 and above were accepted directly, while those with values between 0.5–0.7 were revised and used. Those below 0.5 were excluded from the survey: With a content validation study conducted on 84 survey items, 9 criteria were identified. Overall, 62 items were accepted directly, and 13 items were revised and reevaluated. Consequently, 75 items remained at the end of this stage. Items were calculated above the threshold value and accepted as shown in Appendix C,

Table A3. Content validity of the survey instrument: constructs, items, P_sa, C_sv values, and decisions.

Construct	Code	Item	Psa	Csv	Decision
Brand Identity	BRND1	The website integrates brand elements such as color and icon effectively into the interface.	0.75	0.70	Accepted
	BRND2	The website can be recognized with its logo.	0.95	0.93	Accepted
	BRND3	The website’s pages have the same branding elements.	0.60	0.38	Revised **
	BRND4	The website’s purpose can be understood from the home page.	0.15	0.00	Removed *
	BRND5	The website’s URL name is clear and easy to remember. (e.g., www.abc.com)	0.75	0.50	Revised **
Content Design	CONT1	The website offers a simple and logical content structure.	0.78	0.60	Accepted
	CONT2	The website’s content and services are compatible to users’ goal.	0.70	0.50	Revised **
	CONT3	The website groups related information and functions logically.	0.48	0.25	Removed *
	CONT4	The website provides appropriate details to explain the content of the headings.	0.78	0.70	Accepted
	CONT5	The website provides up-to-date content.	0.88	0.80	Accepted
Personalization and Customization	USER1	The website emphasizes the content users care about.	0.53	0.23	Revised **
	USER2	The website adapts its content according to user’s previous preferences.	0.80	0.70	Accepted
	USER3	The website allows users to view and change profile information easily.	0.78	0.73	Accepted
	USER4	The website offers alternatives that fit user’s preferences.	0.85	0.78	Accepted
Privacy and Business Policies	PRVC1	The website asks for permission before using user’s personal data.	0.90	0.83	Accepted
	PRVC2	The website clearly states why it needs user’s personal information.	0.93	0.90	Accepted
	PRVC3	The website provides a simple and clear privacy policy.	0.93	0.88	Accepted
	PRVC4	The website informs user about cookies and executable programs on user’s local machine.	0.88	0.85	Accepted
Page Design	PAGE1	The website’s headings in pages are simple and descriptive about content.	0.60	0.33	Revised **
	PAGE2	The website provides a consistent page layout design.	0.98	0.95	Accepted
	PAGE3	The website informs user during the task about updates.	0.10	−0.10	Removed *
	PAGE4	The website allows user to view the page without scrolling horizontally.	0.90	0.85	Accepted
	PAGE5	The website allows user to scroll fast throughout the pages.	0.90	0.85	Accepted
	PAGE6	The website provides concise and well-organized content on the pages.	0.49	0.21	Removed *
	PAGE7	The website allows user to scan the pages easily.	0.48	0.05	Removed *
	PAGE8	The website uses the white space on the pages in a balanced way.	0.85	0.75	Accepted
	PAGE9	The website’s home page is professionally designed.	0.70	0.58	Revised **
	PAGE10	The website’s home page creates a positive first impression on users.	0.48	0.30	Removed *
Text Design	TEXT1	The website offers easy-to-read text on the pages.	0.90	0.83	Accepted
	TEXT2	The website allows user to resize text for reading comfort.	0.73	0.63	Accepted
	TEXT3	The website allows user to easily skim the text.	0.73	0.60	Accepted
	TEXT4	The website uses text that complies with the terms of grammar and spelling.	0.98	0.95	Accepted
	TEXT5	The website uses an easily understandable terminology without jargon.	0.80	0.70	Accepted
Link Design	LINK1	The website provides easy-to-distinguish links.	1.00	1.00	Accepted
	LINK2	The website uses standard links’ format.	0.98	0.95	Accepted
	LINK3	The website provides link texts describing the link’s destination accurately.	0.88	0.80	Accepted
	LINK4	The website shows the status of visited or active links with some signs.	0.93	0.88	Accepted
Interaction Objects	INTR1	The website allows user to easily enter data without dealing with different input methods.	0.83	0.75	Accepted
	INTR2	The website uses field labels on data-entry forms that clearly indicate the data formats required to enter.	0.88	0.83	Accepted
	INTR3	The website presents the clickable areas (buttons, menus, etc.) in a way that can be easily distinguished.	0.85	0.78	Accepted
	INTR4	The website has labels that accurately represent the functions of the controls.	0.80	0.73	Accepted
Graphics, Images, and Multimedia	GRPH1	Website uses images for which user can easily understand their messages.	0.90	0.85	Accepted
	GRPH2	Website avoids excessive use of animation.	0.98	0.95	Accepted
	GRPH3	Website uses beautiful and engaging media objects that draw user’s attention.	0.98	0.95	Accepted
	GRPH4	The website allows user to easily pause or stop animations or videos.	0.78	0.58	Revised **
	GRPH5	The website allows user to preview multimedia objects (images, videos, etc.) before viewing them.	0.80	0.68	Accepted
Navigation	NAVG1	Website allows users to go back to previous pages.	0.48	0.18	Removed *
	NAVG2	The website allows the user to easily understand their current location.	0.95	0.93	Accepted
	NAVG3	The website allows the user to see the main navigation menu even if they scroll down the pages.	0.83	0.70	Accepted
	NAVG4	The website provides navigation options consistently across all pages.	0.88	0.78	Accepted
	NAVG5	The website allows the user to skip the pop-up screens.	0.70	0.63	Revised **
	NAVG6	The website shows the number of steps left to complete a task.	0.88	0.83	Accepted
	NAVG7	The website orders navigation options in a logical way.	0.90	0.85	Accepted
	NAVG8	The website allows user to access the homepage from any other page.	0.88	0.83	Accepted
Search	SRCH1	The website allows user to make advanced or simple search.	0.95	0.93	Accepted
	SRCH2	The website allows user to access search option in every page.	0.85	0.73	Accepted
	SRCH3	The website allows user to find search options easily.	0.95	0.90	Accepted
	SRCH4	The website provides helpful hints to support user’s search.	0.85	0.75	Accepted
	SRCH5	The website assists to improve user’s search with helpful hints.	0.80	0.63	Revised **
	SRCH6	The website orders search results according to their relevance.	0.90	0.85	Accepted
Help	HELP1	The website provides help when user has an issue.	1.00	1.00	Accepted
	HELP2	The website provides frequently asked questions to solve users’ problems.	0.98	0.95	Accepted
	HELP3	The website provides easy to understand help content with step-by-step instructions.	0.93	0.88	Accepted
	HELP4	The website provides contact mechanisms such as real time chatbot, customer service, message box, or WhatsApp service.	0.80	0.70	Accepted
	HELP5	The website allows users to communicate with the company responsible for assistance.	0.90	0.83	Accepted
Responsiveness	RSPN1	The website allows users to view the content effectively on their own device.	0.90	0.85	Accepted
	RSPN2	The website adapts interface resolution settings automatically on different devices.	0.98	0.95	Accepted
	RSPN3	The website provides user to view the content effectively with user’s browser.	0.95	0.93	Accepted
	RSPN4	The website provides user to activate the links or buttons through user’s keyboard.	0.35	0.00	Removed*
	RSPN5	The website provides access to the website through mobile devices.	0.93	0.90	Accepted
Error Prevention	ERPR1	The website allows user to cancel or redo actions easily.	0.63	0.55	Revised **
	ERPR2	The website asks for confirmation before important actions.	0.48	0.20	Removed*
	ERPR3	The website’s error messages contain clear instructions on what to do next.	0.88	0.80	Accepted
	ERPR4	The website prompts user before correcting an input.	0.75	0.63	Revised **
	ERPR5	The website keeps user informed about unavoidable delays.	0.78	0.63	Revised **
Collaboration	COLL1	The website provides user to share information with social media accounts.	0.93	0.90	Accepted
	COLL2	The website allows user to connect with other people.	0.98	0.95	Accepted
	COLL3	The website provides a convenient way to user for ratings and reviews.	1.00	1.00	Accepted
	COLL4	The website provides users to read the comments made by other people.	0.98	0.95	Accepted
	COLL5	The website provides users to give online feedback about their experience with the website.	0.95	0.93	Accepted
Cultural Diversity and Multilingual Use Design	DVRS1	The website presents information in a familiar format (currency, temperature unit) for user.	0.73	0.58	Revised **
	DVRS2	The website adapts context according to geography.	0.93	0.90	Accepted
	DVRS3	The website supports other languages.	1.00	1.00	Accepted
	DVRS4	The website recognizes user’s language preference automatically,	0.85	0.70	Accepted

* describes the items removed from survey instrument; ** describes the modified items with re-wording.

.

3.3. Evaluation of Survey Instrument

In the third stage, the measurement properties of the instrument were evaluated and optimized through exploratory and confirmatory analyses. Data were collected in two stages from two independent e-commerce website users. Exploratory factor analysis (EFA) measured the factor structure in the first sample, whereas while confirmatory factor analysis (CFA) confirmed the measurement properties of the developed scale in the second example [170].

The survey instrument was designed to measure all items on a 7-point Likert-agreement scale (1 = strongly disagree to 7 = strongly agree). Participants who had shopping experience on the websites of Trendyol, Hepsiburada, and N11, which are commonly used e-commerce companies in Turkey, participated in the survey.

3.3.1. Exploratory Factor Analysis

EFA was used to determine the underlying dimensions of the items. In the analysis, 785 responses were collected from e-commerce website users. The demographic profiles of the respondents are given in Appendix B, Table A2. Python programming (via Jupyter Notebook IDE) was used to perform the analyses.

First, Bartlett’s and Kaiser–Meyer–Olkin (KMO) tests were conducted to determine whether the data obtained from the survey were suitable for factor analysis and normality testing [174]. The calculated KMO value of 0.93 was above the recommended threshold of 0.6 [175]. Furthermore, the Bartlett’s test chi-square value was 35,099.32 and significant at a 99% confidence interval. Therefore, the collected data were suitable for factor analysis. In this study, exploratory factor analysis with varimax rotation was conducted using the maximum likelihood (ML) method to ensure the maximum variance of factors with fewer variables. In the EFA, eigenvalues greater than 1 were considered, and a total of 16 constructs were obtained, which is the same number as decided in the conceptualization stage. The constructs explained 64.18% of the observed cumulative variance, and item loadings are presented in

Table 3. EFA: Items’ descriptive statistics, standardized item loadings, variance explained, and Cronbach’s alpha values.

Constructs	Variance Explained (%)	Mean	Stdev	Standardized Item Loadings	Cronbach α	Constructs	Variance Explained (%)	Mean	Stdev	Standardized Item Loadings	Cronbach’s α
Brand Identity (BRND1-BRND2-BRND3-BRND5)	3.64%	4.63 4.62 4.67 4.69	1.12 1.12 1.11 1.10	0.76 0.79 0.74 0.73	0.87	Graphics, Images, and Multimedia (GRPH1-GRPH2-GRPH4-GRPH5)	4.01%	4.54 4.37 4.48 4.35	1.11 1.13 1.13 1.15	0.82 0.78 0.78 0.77	0.90
Content Design (CONT1-CONT2-CONT4-CONT5)	4.01%	4.93 5.18 5.16 5.20	1.41 1.14 1.14 1.12	0.74 0.71 0.71 0.71	0.88	Navigation (NAVG2-NAVG3-NAVG4-NAVG5-NAVG6-NAVG7)	5.27%	4.48 3.93 4.67 4.39 4.65 4.09	1.26 1.18 1.04 1.18 1.06 1.19	0.70 0.70 0.69 0.70 0.70 0.75	0.89
Personalization and Customization (USER1-USER2-USER3-USER4)	3.22%	4.58 4.60 4.32 4.30	1.08 1.13 1.19 1.12	0.75 0.71 0.62 0.74	0.84	Search (SRCH1-SRCH2-SRCH3-SRCH4-SRCH5-SRCH6)	5.04%	4.40 3.84 4.35 4.36 4.38 4.41	1.08 1.19 1.18 1.22 1.19 1.26	0.71 0.72 0.71 0.69 0.71 0.74	0.89
Privacy and Business Policies (PRVC1-PRVC2-PRVC3-PRVC4)	3.60%	3.52 3.59 3.63 3.54	1.11 1.11 1.10 1.09	0.74 0.76 0.72 0.74	0.87	Help (HELP1-HELP2-HELP3-HELP4)	4.37%	3.95 4.43 4.35 3.85	1.25 1.30 1.47 1.22	0.66 0.75 0.78 0.71	0.89
Page Design (PAGE2-PAGE4-PAGE5-PAGE8-PAGE9)	4.35%	4.33 4.59 4.57 4.60 4.33	1.19 1.10 1.06 1.07 1.15	0.70 0.75 0.76 0.76 0.68	0.88	Responsiveness (RSPN1-RSPN2- RSPN3-RSPN5)	3.39%	4.57 2.53 4.57 4.58	1.07 0.99 1.09 1.12	0.71 0.65 0.74 0.77	0.85
Text Design (TEXT1-TEXT2-TEXT3-TEXT4-TEXT5)	4.48%	4.62 4.35 4.31 4.56 4.25	1.06 1.20 1.21 1.05 1.17	0.72 0.74 0.73 0.73 0.74	0.89	Error Prevention (ERPR1-ERPR3-ERPR4-ERPR5)	3.72%	3.90 3.95 4.24 3.94	1.16 1.11 1.17 1.14	0.76 0.76 0.68 0.75	0.87
Link Design (LINK1-LINK2-LINK3-LINK4)	3.68%	3.31 3.53 3.92 3.73	1.19 1.15 1.29 1.41	0.70 0.70 0.76 0.75	0.84	Collaboration (COLL1-COLL2-COLL3-COLL4-COLL5)	4.11%	4.68 5.19 5.51 4.94 5.03	1.13 1.13 1.01 1.18 1.09	0.71 0.66 0.67 0.70 0.71	0.86
Interaction Objects (INTR1-INTR2-INTR3-INTR4)	3.75%	3.60 3.67 3.63 3.70	1.12 1.11 1.15 1.14	0.77 0.75 0.76 0.75	0.88	Cultural Diversity and Multilingual Use Design (DVRS1-DVRS2-DVRS3-DVRS4)	3.53%	4.57 4.55 3.71 4.58	1.09 1.06 1.09 1.10	0.77 0.75 0.73 0.72	0.86

. Items with factor loadings greater than 0.6 were assigned to their respective factors. A total of four items, which had (‘GRPH3′ (0.24), ‘HELP5’ (0.55), ‘NAVG8’ (0.40), and ‘PAGE1’ (0.44)) low factor loadings and were unrelated to any construct, were dropped from the survey instrument. A total of 71 items representing 16 constructs remained for the CFA. Based on item loadings, each factor was labeled with construct names. Furthermore, Cronbach’s alpha for the underlying factors ranged from 0.84 to 0.90, thus satisfying the suggested level of construct reliability [176]. The final user-centered items list obtained according to the results is given in Appendix D,

Table A4. Final User-centered Items List for Website Usability Evaluation.

Construct	Code	Item
Brand Identity	BRND1	The website integrates brand elements such as color and icon effectively into the interface.
	BRND2	The website can be recognized with its logo.
	BRND3	The website’s pages have the same branding elements.
	BRND5	The website’s URL name is clear and easy to remember (e.g., www.abc.com).
Content Design	CONT1	The website offers a simple and logical content structure.
	CONT2	The website’s content and services are compatible to users’ goal.
	CONT4	The website provides appropriate details to explain the content of the headings.
	CONT5	The website provides up-to-date content.
Personalization and Customization	USER1	The website emphasizes the content users care about.
	USER2	The website adapts its content according to user’s previous preferences.
	USER3	The website allows users to view and change profile information easily.
	USER4	The website offers alternatives that fit user’s preferences.
Privacy and Business Policies	PRVC1	The website asks for permission before using user’s personal data.
	PRVC2	The website clearly states why it needs user’s personal information.
	PRVC3	The website provides a simple and clear privacy policy.
	PRVC4	The website informs user about cookies and executable programs on user’s local machine.
Page Design	PAGE2	The website provides a consistent page layout design.
	PAGE4	The website allows user to view the page without scrolling horizontally.
	PAGE5	The website allows user to scroll fast throughout the pages.
	PAGE8	The website uses the white space on the pages in a balanced way.
	PAGE9	The website’s home page is professionally designed.
Text Design	TEXT1	The website offers easy-to-read texts on the pages.
	TEXT2	The website allows user to resize text for reading comfort.
	TEXT3	The website allows user to easily skim of the text.
	TEXT4	The website uses text that complies with the terms of grammar and spelling.
	TEXT5	The website uses an easily understandable terminology without jargon.
Link Design	LINK1	The website provides easy-to-distinguish links.
	LINK2	The website uses standard links’ format.
	LINK3	The website provides link texts describing the link’s destination accurately.
	LINK4	The website shows the status of visited or active links with some signs.
Interaction Objects	INTR1	The website allows user to easily enter data without dealing with different input methods.
	INTR2	The website uses field labels on data-entry forms that clearly indicate the data formats required to enter.
	INTR3	The website presents the clickable areas (buttons, menus, etc.) in a way that can be easily distinguished.
	INTR4	The website has labels that accurately represent the functions of the controls.
Graphics, Images, and Multimedia	GRPH1	Website uses images for which user can easily understand their messages.
	GRPH2	Website avoids excessive use of animation.
	GRPH4	The website allows user to easily pause or stop animations or videos.
	GRPH5	The website allows user to preview multimedia objects (images, videos, etc.) before viewing them.
Navigation	NAVG2	The website allows the user to easily understand their current location.
	NAVG3	The website allows the user to see the main navigation menu even if they scroll down the pages.
	NAVG4	The website provides navigation options consistently across all pages.
	NAVG5	The website allows the user to skip the pop-up screens.
	NAVG6	The website shows the number of steps left to complete a task.
	NAVG7	The website orders navigation options in a logical way.
Search	SRCH1	The website allows user to make advanced or simple search.
	SRCH2	The website allows user to access search option in every page.
	SRCH3	The website allows user to find search options easily.
	SRCH4	The website provides helpful hints to support user’s search.
	SRCH5	The website assists to improve user’s search with helpful hints.
	SRCH6	The website orders search results according to their relevance.
Help	HELP1	The website provides help when user has an issue.
	HELP2	The website provides frequently asked questions to solve users’ problems.
	HELP3	The website provides easy to understand help content with step-by-step instructions.
	HELP4	The website provides contact mechanisms such as real time chatbot, customer service, message box, or WhatsApp service.
Responsiveness	RSPN1	The website allows users to view the content effectively on their own device.
	RSPN2	The website adapts interface resolution settings automatically on different devices.
	RSPN3	The website provides user to view the content effectively with user’s browser.
	RSPN5	The website provides access to the website through mobile devices.
Error Prevention	ERPR1	The website allows user to cancel or redo actions easily.
	ERPR3	The website’s error messages contain clear instructions on what to do next.
	ERPR4	The website prompts user before correcting an input.
	ERPR5	The website keeps user informed about unavoidable delays.
Collaboration	COLL1	The website provides user to share information with social media accounts.
	COLL2	The website allows user to connect with other people.
	COLL3	The website provides a convenient way to user for ratings and reviews.
	COLL4	The website provides users to read the comments made by other people.
	COLL5	The website provides users to give online feedback about their experience with the website.
Cultural Diversity and Multilingual Use Design	DVRS1	The website presents information in a familiar format (currency, temperature unit) for user.
	DVRS2	The website adapts context according to geography.
	DVRS3	The website supports other languages.
	DVRS4	The website recognizes user’s language preference automatically.

.

3.3.2. Confirmatory Factor Analysis

Lewis et al. [170] reevaluated the final construct obtained from EFA through confirmatory evaluation. CFA is a valuable analytical approach for evaluating the reliability of measurement criteria and assessing the alignment between theoretical concepts and structural equation models [177]. CFA is conducted by employing an independent random sample that is different from the one used in EFA. Then, the survey items are reevaluated, and a rigorous assessment of the measurement structure is performed. Second, Lewis et al. [170] suggested that the nomological network of the scale should also be evaluated as part of the validation assessment. To assess the nomological validity of conceptualized categorical structures, it is essential to gather data that are theoretically related to these structures [178].

To perform CFA, an independent sample was taken from end users who had experience with e-commerce websites and did not participate in the first study. A total of 1086 people agreed to participate in the research, and the demographic profile of the participants is shown in Appendix B, Table A2. IBM SPSS AMOS 26 program was used to analyze the data. In the measurement model, the constructs represent latent variables and survey items represent observed variables. A total of 71 items representing 16 constructs were entered into the measurement model.

First, the skewness value of the data obtained from the end user and normality of the data were tested. Then, the skewness value of all evaluation items was calculated between ±2, and a proof was obtained that the data were in accordance with the normal distribution [176,179,180]. To determine the adequacy of the model, model fit indicators were calculated as shown in

Table 4. Model Fit Indicators.

Metric	Prediction	Threshold Value	Model Fitting	References
CMIN	2638	--	--
DF	2294	--	--
CMIN/DF	1.15	Between 1 and 3	Conform	[181,182]
GFI	0.94	≥0.95 (good fit) or ≥0.90 (reasonable fit)	Conform	[183]
CFI	0.99	≥0.95 (good fit) or ≥0.90 (reasonable fit)	Conform	[179,184]
IFI	0.99	>0.90	Conform	[184]
TLI	0.99	≥0.95 (good fit) or ≥0.90 (reasonable fit)	Conform	[176,179,185]
RMSEA	0.01	≤0.06 (good fit) or ≤0.08 (reasonable fit)	Conform	[176]
SRMR	0.024	≤0.08 (good fit) or ≤0.10 (reasonable fit)	Conform	[176,186,187]

. Chi-square, degrees of freedom ratio (CMIN/DF), comparative fit indices (CFI), Tucker–Lewis index (TLI), incremental fit index (IFI), root mean square error of approximation (RMSEA), and standardized root mean residual (SRMR) values were used to check the conformality of the data. The fit indices of the model from the goodness-of-fit statistics were as follows GFI = 0.938, CFI = 0.993, IFI = 0.993, and TLI = 0.992; error statistics was evaluated using RMSEA = 0.011 and SRMR = 0.0235. The chi-square and degrees of freedom ratios (CMIN/DF) were 1.150. The findings revealed that the fit of the model to the data was satisfactory [176].

The convergent validity of the confirmatory model was evaluated using standardized factor loadings, t-values, Cronbach’s alpha, composite reliability (CR), and average variance extracted (AVE) values [188]. All items’ standardized factor loadings, which were between 0.7 and 0.85, exceeded the suggested threshold value [176,181], and the t-values of items were significant at a 0.01 confidence level [189].

The internal consistency reliability of the measurement model was assessed using Cronbach’s alpha and CR values.

Table 5. Measurement Model: Items’ loadings and Constructs’ Cronbach’s alpha values.

Construct Name	Mean	Stdev	Loadings	t-Value	Cronbach’s α	Construct Name	Mean	Stdev	Loadings	t-Value	Cronbach α
Brand Identity (BRND1-BRND2-BRND3-BRND5)	5.53 5.51 5.51 5.16	1.02 1.03 0.99 0.99	0.85 0.84 0.80 0.75	31.60 31.60 29.29 27.08	0.88	Graphics, Images, and Multimedia (GRPH1-GRPH2-GRPH4-GRPH5)	4.85 4.78 4.81 4.81	1.02 1.03 1.01 1.01	0.76 0.81 0.82 0.81	26.08 26.08 28.31 28.09	0.87
Content Design (CONT1-CONT2-CONT4-CONT5)	5.21 5.20 5.20 5.21	1.07 1.07 1.07 1.11	0.85 0.83 0.82 0.80	31.81 31.82 31.62 30.50	0.89	Navigation (NAVG2-NAVG3-NAVG4-NAVG5-NAVG6-NAVG7)	5.03 5.40 5.03 4.96 5.01 5.02	1.07 0.96 1.05 1.04 1.04 1.06	0.82 0.75 0.82 0.81 0.82 0.83	27.59 27.59 31.19 30.67 31.23 31.91	0.91
Personalization and Customization (USER1-USER2-USER3-USER4)	4.56 4.54 4.56 4.55	1.02 1.05 1.04 1.02	0.80 0.79 0.83 0.82	27.05 27.05 28.62 28.21	0.88	Search (SRCH1-SRCH2-SRCH3-SRCH4-SRCH5-SRCH6)	5.48 5.45 5.47 5.47 5.49 5.49	0.99 0.99 0.99 1 0.99 0.98	0.77 0.78 0.78 0.78 0.77 0.76	26.81 26.81 27.01 26.99 26.49 26.22	0.90
Privacy and Business Policies (PRVC1-PRVC2-PRVC3-PRVC4)	4.13 4.15 4.14 4.11	1.03 1.04 1.04 1.02	0.82 0.81 0.82 0.80	29.29 29.29 29.46 28.50	0.88	Help (HELP1-HELP2-HELP3-HELP4)	5.38 5.35 5.34 5.40	1.01 0.96 0.98 0.97	0.78 0.77 0.76 0.76	25.09 25.09 24.80 24.60	0.85
Page Design (PAGE2-PAGE4-PAGE5-PAGE8-PAGE9)	5.17 5.13 5.12 5.13 5.11	1.03 1.02 1.04 1.05 1.02	0.82 0.79 0.79 0.82 0.80	28.81 28.81 28.88 30.26 29.59	0.9	Responsiveness (RSPN1-RSPN2-RSPN3-RSPN5)	4.48 3.85 4.45 4.50	1.02 1.02 1.06 1.05	0.82 0.81 0.81 0.83	29.14 29.14 29.26 30.21	0.88
Text Design (TEXT1-TEXT2-TEXT3-TEXT4-TEXT5)	5.14 5.15 5.19 5.18 5.20	1.01 1.03 1.02 1.00 1.02	0.82 0.80 0.82 0.77 0.80	29.55 29.55 30.48 28.20 29.42	0.89	Error Prevention (ERPR1-ERPR3-ERPR4-ERPR5)	6.00 5.89 5.85 6.00	0.82 0.90 0.87 0.82	0.78 0.72 0.70 0.79	22.53 22.53 21.78 24.27	0.83
Link Design (LINK1-LINK2-LINK3-LINK4)	4.48 3.51 4.48 4.52	1.05 1.07 1.08 1.03	0.82 0.79 0.82 0.82	28.41 28.41 29.58 29.62	0.88	Collaboration (COLL1-COLL2-COLL3-COLL4-COLL5)	5.46 5.42 5.42 5.44 5.46	0.96 0.98 0.97 0.96 0.97	0.76 0.78 0.76 0.77 0.74	25.48 25.48 24.68 24.96 24.14	0.87
Interaction Objects (INTR1-INTR2-INTR3-INTR4)	5.66 5.66 5.70 5.68	0.91 0.91 0.91 0.93	0.74 0.74 0.75 0.72	21.51 21.51 21.62 20.96	0.82	Cultural Diversity and Multilingual Use Design (DVRS1-DVRS2-DVRS3-DVRS4)	4.25 4.26 3.89 4.57	1.03 1.07 1.03 1.11	0.82 0.84 0.77 0.79	30.06 30.07 27.08 27.95	0.87

presents the results of CR, AVE, and the correlations between each construct. The Cronbach’s alpha values ranged from 0.79 to 0.91, exceeding the threshold value of 0.70 [190]. The CR values were greater than 0.83, and AVE values were greater than 0.54. The minimum value of the CR values calculated for the “Error Prevention” construct was 0.79; therefore, all values were greater than the recommended value of 0.70 [191]. Additionally, the AVE values of the conceptual constructs exceeded the threshold value of 0.5, indicating convergent validity for all factors, as the minimum acceptable values were encountered [176,192].

The constructs’ discriminant validity was assessed using the approach suggested by Fornell and Larcker [192]. In this approach, the discriminant validity of any construct was achieved if the square root of the AVE of each construct was greater than its correlation with the others [177,193]. The square roots of the AVE values calculated in this study were greater than their correlation values with other structures, thus indicating that the conceptual structures achieved discriminant validity [192], as shown in

Table 6. Confirmatory factor analysis: CR, AVE, and correlations.

	CR	AVE	INTR	BRND	HELP	USER	PRVC	COLL	PAGE	GRPH	NAVG	SRCH	ERPR	DVRS	CONT	TEXT	LINK	RSPN
INTR	0.83	0.54	0.74 *
BRND	0.88	0.65	0.032	0.81 *
HELP	0.85	0.60	0.014	0.166	0.77 *
USER	0.88	0.65	0.053	0.003	0.070	0.81 *
PRVC	0.89	0.66	0.023	0.120	0.032	0.087	0.81 *
COLL	0.88	0.58	0.034	0.044	0.125	0.127	0.114	0.76 *
PAGE	0.90	0.65	0.030	0.163	0.116	0.063	0.022	0.072	0.80 *
GRPH	0.88	0.64	0.012	0.093	0.078	0.045	0.064	0.090	0.086	0.80 *
NAVG	0.92	0.65	0.088	0.227	0.180	0.052	0.084	0.096	0.149	0.119	0.81 *
SRCH	0.90	0.60	0.073	0.182	0.170	0.043	0.097	0.059	0.087	0.096	0.119	0.78 *
ERPR	0.83	0.56	0.073	0.137	0.145	0.070	0.069	0.042	0.113	0.015	0.107	0.097	0.75 *
DVRS	0.88	0.64	0.087	0.120	0.142	0.052	0.083	0.056	0.061	0.064	0.186	0.126	0.073	0.80 *
CONT	0.89	0.68	0.034	0.209	0.204	0.083	0.110	0.064	0.135	0.074	0.164	0.199	0.178	0.185	0.82 *
TEXT	0.90	0.64	0.046	0.039	0.026	0.038	0.071	0.062	0.121	0.090	0.077	0.030	0.057	0.028	−0.018	0.80 *
LINK	0.89	0.66	0.027	0.120	0.112	0.058	0.075	0.034	0.165	0.146	0.187	0.037	−0.018	0.039	0.101	0.078	0.81 *
RSPN	0.89	0.67	−0.054	−0.035	0.131	0.026	0.123	0.088	0.076	0.056	0.021	0.075	0.101	0.098	0.081	0.063	0.028	0.82 *

AVE, average variance explained; CR, composite reliability; *, square root of AVE; INTR, Interaction Objects; BRND, Brand Identity; HELP, Help; USER, Personalization and Customization; PRVC, Privacy and Business Policies; COLL, Collaboration; PAGE, Page Design; GRPH, Graphics, Images, and Multimedia; NAVG, Navigation; SRCH, Search; ERPR, Error Prevention; DVRS, Cultural Diversity and Multilingual Use Design; CONT, Content Design; TEXT, Text Design; LINK, Link Design; RSPN, Responsiveness. Significant at 0.05.

.

In order to evaluate the nomological validity of website usability, the impact of website usability constructs on “continued intention to use”, “satisfaction”, and “brand loyalty” were examined. These three structures were identified and adapted from previous studies in the literature [194,195,196,197,198]. The nomological items listed in Table 7 were included in the website usability survey.

Continued intention to use is a concept of website usability that refers to a user’s intention to continue using a website after its initial use. It is a critical factor in determining the long-term success of a website, as it directly influences user loyalty and retention [60]. Satisfaction is defined as the user’s level of contentment or feeling pleased with a website’s usability, features, and overall experience, such as repeat visits, recommendations, and purchases [54,56]. Furthermore, satisfaction is influenced by many factors, including website design, usability, functionality, and content [199]. Brand loyalty is defined as the commitment of a customer to a particular brand and the willingness to make repeated purchases of that brand over time [200].

Table 7. Measured variables of nomological factors.

Nomological Variable		Items	Scales Adapted from (Studies)
Satisfaction	STF1	Overall, I am satisfied with this website.	[194,195,196]
	STF2	I am pleased with the experience of using the website.
	STF3	The website information content met my expectations.
	STF4	My decision to use the website was a wise one.
Continued intention to use	ITU1	I intend to continue using the website rather than discontinue its use.	[194,195,197,201]
	ITU2	My intentions are to continue using the website rather than using any alternative means.
	ITU3	It is likely that I will use the website in the near future.
	ITU4	I intend to conduct my online shopping transactions via the website in the near future.
Brand Loyalty	BLY 1	I prefer my website to other websites.	[198]
	BLY 2	I recommend my website to others (e.g., friends, family).
	BLY 3	I say positive things about the website to other people.
	BLY 4	I encourage friends and relatives to be users of the website

Table 7. Measured variables of nomological factors.

Nomological Variable		Items	Scales Adapted from (Studies)
Satisfaction	STF1	Overall, I am satisfied with this website.	[194,195,196]
	STF2	I am pleased with the experience of using the website.
	STF3	The website information content met my expectations.
	STF4	My decision to use the website was a wise one.
Continued intention to use	ITU1	I intend to continue using the website rather than discontinue its use.	[194,195,197,201]
	ITU2	My intentions are to continue using the website rather than using any alternative means.
	ITU3	It is likely that I will use the website in the near future.
	ITU4	I intend to conduct my online shopping transactions via the website in the near future.
Brand Loyalty	BLY 1	I prefer my website to other websites.	[198]
	BLY 2	I recommend my website to others (e.g., friends, family).
	BLY 3	I say positive things about the website to other people.
	BLY 4	I encourage friends and relatives to be users of the website

The relationships between website usability factors and the nomological constructs were tested.

Table 8. Estimations of Structural Model.

	Satisfaction	Continued Intention to Use	Brand Loyalty
R2 (%)	61.3	61.7	52.7
Brand Identity	0.084 **	0.202 **	0.192 **
Content Design	0.244	0.202	0.060 *
Personalization and Customization	−0.053	−0.031	−0.027
Privacy and Business Policies	0.116 **	0.143 **	0.177 **
Page Design	0.180 **	0.180 **	0.171 **
Text Design	0.168 **	0.118 **	0.138 **
Link Design	−0.006	0.001	0.168 **
Interaction Objects	0.160 **	0.164 **	0.056
Graphics, Images, and Multimedia	0.134 **	0.047	0.071 *
Navigation	0.254 **	0.251 **	0.173 **
Search	0.224 **	0.266 **	0.308 **
Help	0.195 **	0.142 **	0.024
Responsiveness	0.166 **	0.197 **	0.028
Error Prevention	0.197	0.152	0.265
Collaboration	0.071 *	0.080 *	0.098 **
Cultural Diversity and Multilingual Use Design	0.045	0.032	0.037

*, p < 0.05; **, p < 0.01.

presents the results of the study. It was observed that website usability constructs have a positive effect on the following nomological factors: “continued intention to use”, “satisfaction”, and “brand loyalty.” These significant relationships provide evidence for the applicability of the developed survey instrument [170]:

A total of 11 constructs, including “brand identity”, “privacy and business policies”, “page design”, “text design”, “interaction objects”, “graphics, images and multimedia”, “navigation”, “search”, “help”, “responsiveness”, and “collaboration”, explains the 61.3% total variance of satisfaction.

A total of 10 constructs, including “brand identity”, “privacy and business policies”, “page design”, “text design”, “interaction objects”, “navigation”, “search”, “help”, “responsiveness”, and “collaboration”, explains the 61.7% total variance of continued intention to use.

A total of 10 constructs, including “brand identity”, “content design”, “privacy and business policies”, “page design”, “text design”, “link design”, “graphics, images, and multimedia”, “navigation”, “search”, and “collaboration”, explains the 52.7% total variance of brand loyalty.

4. Discussion

In this study, we conceptualized website usability and using a validated construct and measured it via survey instrument. The concepts and their open codes were derived from BS EN ISO-9241-151 as a base document, and the literature was examined in detail to identify possible additional concepts for website usability. A total of 16 constructs were extracted, namely “brand identity”, “content design”, “personalization and customization”, “privacy and business policies”, “page design”, “text design”, “link design”, “interaction objects”, “graphics, images, and multimedia”, “navigation”, “search”, “help”, “responsiveness”, “error prevention”, “collaboration”, and “cultural diversity and multilingual use design”, with the objective of measuring website usability. Each construct was explained using subconstructs and their corresponding open codes, which were used as comprehensive guidelines to assist developers in enhancing website usability during the design process.

Based on these concepts, a website usability survey was developed and validated using a systematic procedure. First, a pretest, pilot test, and content validity checks were conducted to ensure the validity of the survey instrument. Then, a large-scale survey was conducted to conduct EFA of e-commerce website users. It was used to assess the factorial validity and reliability of the constructs. The results show that website usability can be measured using the 16 constructs defined in the conceptualization step. Finally, CFA was performed using the second sample to validate the results obtained from EFA and examine the generalizability of the survey instrument. It was observed that the effects of website usability factors on continued intention to use, brand loyalty, and customer satisfaction were significant. In addition, open codes defined for each construct were used by experts to evaluate websites in conformance with usability principles. The developed survey instrument was used partially or completely to measure user perceptions of a specific website. Therefore, experts’ and end users’ views should be more examined in the evaluation of website usability.

Website usability satisfaction refers to the users’ overall positive evaluation of their experience while interacting with a website. A high level of satisfaction indicates that the website was usable, enjoyable, and fulfilled their needs, resulting in a positive affective state and favorable perception of the website. The study revealed that a total of 11 constructs, namely “brand identity”, “privacy and business policies”, “page design”, “text design”, “interaction objects”, “graphics, images, and multimedia”, “navigation”, “search”, “help”, “responsiveness”, and “collaboration”, significantly contributed to explaining 61.3% of the total variance in satisfaction.

In terms of website usability, continued intention to use refers to users’ willingness and inclination to continue using a website in the future. It is influenced by users’ satisfaction with a website’s usability and perception of its ongoing value and relevance. When users perceive a website to be highly usable and beneficial, they are more likely to express their intention to use it repeatedly, suggesting a desire for sustained engagement and interaction. The findings of our study demonstrated that a total of 10 constructs related to website usability, namely “brand identity”, “privacy and business policies”, “page design”, “text design”, “interaction objects”, “navigation”, “search”, “help”, “responsiveness”, and “collaboration”, collectively accounted for 61.7% of the total variance in continued intention to use.

Brand loyalty, within the context of website usability, represents users’ commitment to and preference for a particular website over its competitors. It is characterized by users’ consistent and repeated use of the website owing to their positive experiences and high satisfaction with its usability. Brand loyalty implies a strong connection between users and a website, engendering trust, loyalty, and a willingness to recommend the website to others, thereby contributing to its sustained success and market position. Ten constructs, namely “brand identity”, “content design”, “privacy and business policies”, “page design”, “text design”, “link design”, “graphics, images, and multimedia”, “navigation”, “search”, and “collaboration”, collectively accounted for 52.7% of the total variance in brand loyalty.

4.1. Theoretical and Practical Implications

The findings of this study highlight the significance of considering website usability during the design, development, and evaluation processes. Developers can create websites that enhance user satisfaction, encourage continued use, and foster brand loyalty by following developed website usability concepts. This study provides valuable insights and practical implications for developers and businesses seeking to optimize website usability and establish strong connections with their users. The open codes generated in the study are structured to be used as a checklist for professionals working on website development. Additionally, they can be used to understand usability issues on existing websites. The proposed instrument also allows for gathering end-user input to enhance effective usability. After applying the developed scale to end users, factors and items with lower scores that significantly impact satisfaction and brand loyalty are identified. Therefore, this scale can help website designers and developers determine which factors to prioritize, ultimately enhancing customer satisfaction.

The ISO-recommended BS EN ISO-9241-151 usability guide, serving as the primary reference point in the study, does not provide a specific classification for website usability issues. Therefore, based on the fundamental principles in existing usability guides, it is meant to quickly identify usability problems, making it adaptable to many websites. It was observed that the identified factors are common across many websites, such as easy navigation, user-friendly page design, fast loading times, and efficient search options, which are essential for most websites. However, considering that not every website has a similar development structure, the scale developed in the study is adaptable to the characteristics and needs of different websites. As highlighted in the study [30], in some cases, certain sub-scales or items may be more important for specific sites while less critical for others. The factors remain general in such cases, but the weights can vary. In its broad outline, the developed website usability guide addresses website issues and can be further developed when assessing highly specialized sites. In such evaluations, the proposed instrument can serve as a starting point, allowing additional parameters to be added while preserving the main framework. It is inherently capable of continuous improvement through ongoing feedback and updates based on this feedback. Consequently, the scale maintains a dynamic nature and can be adapted to the specific needs of different types of websites.

4.2. Limitations and Future Studies

In addition to the contributions highlighted in this study, there are still some limitations. First, the developed survey instrument was validated by applying it to e-commerce users in Turkey. In future studies, the developed survey instrument could be applied to different types of websites in different cultures for further validation. Secondly, our nomological validation indicates that the proposed usability factors significantly explain user intentions, brand loyalty, and satisfaction. Specifically, the explained variances for continued intention to use, satisfaction, and brand loyalty were 61.7%, 61.3%, and 52.7%, respectively. While these results are promising, other factors might influence satisfaction, loyalty, and intention to use. Therefore, future research should be conducted to explore additional factors that impact these outcomes. While the scale we developed can be used to evaluate the usability of various website types, it is essential to recognize that different website types may have unique requirements. As a result, the scale can be modified by introducing new items or factors specific to various websites’ needs. Future researchers are encouraged to use the developed scale as a foundation and customize it for particular contexts. Furthermore, by combining the findings of the developed usability guide with web analytics methods, the content of the study can be enriched, leading to more concrete website usability criteria.

5. Conclusions

In this study, using the BS EN ISO-9241-151 and an extensive literature review, we conceptualized website usability and subsequently developed and validated a survey instrument. We identified 16 key constructs, including “brand identity”, “navigation”, “page design”, and “text design”, among others. These constructs were found to have significant implications on user outcomes such as brand loyalty, satisfaction, and continued intention to use. Understanding and employing these constructs can lead developers to enhance the user experience on their websites. Our research emphasizes the significance of website usability in the digital age and provides resources and insights for both researchers and developers. Especially, researchers can leverage our conceptualization to further study website usability. Meanwhile, developers can utilize our instrument to enhance the design, development, and evaluation of their websites.