User Experience of Green Building Certification Resources: EarthCraft Multifamily

Abstract

To improve the construction industry’s capacity to deliver sustainable infrastructure, guidance on delivering green building systems needs to be more usable. Green buildings have certifications and ratings in place that ensure that projects are environmentally responsible and meet standards in resource efficiency. EarthCraft Multifamily (ECMF), an evolving green building certification, has been successful in increasing the delivery of energy-efficient affordable housing, and this study leverages user experience (UX) methodologies to understand how to further improve ECMF and replicate its success. This study identifies the impact ECMF tools and resources, such as the program manual, worksheet, and technical guidelines, have on enhancing project delivery for architects. This study conducted data analysis on project specifications, heuristic evaluation data, and stakeholder interview data. As the strengths and weaknesses of ECMF were identified, knowledge on the usability of the green building certification program was unveiled. Heuristic evaluations data show that accessibility and usability issues are present in ECMF resources. Interview data show that architects’ experiences with ECMF resources were affected by some of the usability issues identified in the heuristic evaluation data. Coded interview transcripts show the most prominent participant-identified improvements represented within the data. Resources need appropriate visual representation such as readability and hierarchy to improve their usability. Understanding how ECMF resources are utilized during project delivery allows for the appropriate content and options to be strategically framed to improve accessibility and enhance user decision making. ECMF resources can allow for the inclusion of a broader set of stakeholders by lowering the level of expertise required for sustainable infrastructure delivery.

1. Introduction

Over the last two decades, green building certifications have been developed to promote sustainability by providing intelligence and direction to building designers [1]. There are a variety of considerations for green building design, such as climate change mitigation, indoor environmental quality, resiliency, and durability. Furthermore, regions have different contextual goals, which can create conflicting values for a design team to consider [2]. Users of green building certification resources should be provided with an elegant user experience that guides a designer to gain certifications and ratings. Approximately over 100 green building certifications are in place [3] to outline and encourage construction projects to be environmentally responsible for meeting standards of resource efficiency and performance. While differences across regions should be appreciated, green building certifications should provide a common user experience regardless of the program.

Green building certification programs can be characterized into four categories: international, national, regional, and local. Two of the most common programs are BREEAM and LEED, international and national programs, respectively. BREEAM aims to transform the way the built environment is designed and maintained to improve sustainability [4]. LEED aims to provide stakeholders with information to apply technologies for sustainable design. Other green building certification and energy efficiency programs are identified by the type and level of the programs, as shown in

Table 1. Classification of green building certification and energy efficiency programs.

Name	Type of Program	Level of Program
ENERGY STAR	Energy efficiency	National
EarthCraft Multifamily	Green building	Regional/Local
BREEAM	Green building	International
LEED	Green building	National
Green Globes	Green building	National
Living Building Challenge	Energy efficiency/green building	International/National
Passive House	Energy efficiency	International
Green Built Homes	Green building	Local

. Green buildings influence human health at two critically important scales: (1) directly at the individual level through providing optimized indoor environments, and (2) indirectly on a population level through reductions in energy use and thus reductions in air pollutants that cause premature death, cardiovascular disease, exacerbate asthma conditions, and contribute to global climate change, itself associated with a cascade of adverse human health impacts [5]. The impact on human health calls for green buildings to constantly improve and reach more people. Green building certification resources help building designers manage human health as they construct green buildings.

Green building certification resources are utilized to help project teams (e.g., owners, architects, engineering, and contractors) construct green buildings and earn certifications. Such tools and resources include but are not limited to program manuals, technical guidelines, and Microsoft Excel worksheets. Green building certification resources should give flexibility for the project team to focus on the right green building technologies by developing site-specific and client-specific alternative strategies that achieve equal or greater environmental benefits at lower costs [6]. Decision aids such as rating systems and standards are used to design and evaluate infrastructure projects [7].

The construction of single-family homes and residential buildings are types of affordable housing projects that pursue green building certification. EarthCraft Multifamily (ECMF) is a direct pipeline program from EarthCraft, used to construct energy-efficient, affordable multifamily housing units in the Mid-Atlantic and Southeast regions of the United States (e.g., Climate Zones 3 and 4). ECMF, like other green building certification programs, constructs buildings that focus on minimizing negative impacts largely through energy and water conservation measures and improving human health [8]. A comparison of ECMF and other green building certification programs’ parameters are shown in

Table 2. Assessment parameters of ECMF and other green building certification programs.

Name	Features *	Rating Scale	Criteria
ECMF	Holistic Category, credit and points based Credits implicitly weighted rather than weighting categories	Certified (100–149 points) Gold (150–199 points) Platinum (200+ points)	Site Planning) Construction Waste Management) Resource Efficiency Indoor Air Quality High-Performance Building Envelope Energy-Efficient Systems Water Efficiency Education and Operations Innovation
LEED	Holistic Category, credit and points based Credits implicitly weighted rather than weighting categories	Certified (40–49 points) Silver (50–59 points) Gold (60–79 points) Platinum (80+ points)	Location and Transportation Sustainable Sites Water Efficiency Energy and Atmosphere Materials and Resources Indoor Environmental Quality Innovation Regional Priority
BREEAM	Holistic Category, credit and points based Weighted categories	Pass (30–44 points) Good (45–54 points) Very Good (55–69 points) Excellent (70–84 points) Outstanding (85+ points)	Energy Health and Well-Being Innovation Land use Materials Management Pollution Transport Waste Water
Green Globes	Holistic Category, credit and points based Credits implicitly weighted rather than weighting categories	One Green Globe (350–540 points) Two Green Globes (550–690 points) Three Green Globes (700–840 points) Four Green Globes (850–1000 points)	Project Management Site Energy Water Efficiency Materials Indoor Environment

* Source: [4].

. Ideal green buildings consider both costs and benefits as they attempt to use fewer costs to achieve more benefits [9]. Construction firms across the Southeast of the United States have acknowledged the benefits of ECMF during the construction of green buildings as well as areas of improvement. This brought forth the possibility that the areas of improvement are connected to the resources and how they are utilized.

There is an increasing need to understand what psychological factors could affect how users perceive and utilize green building certification resources. A person’s beliefs of usefulness and ease of use related to a technology influence the actual usage of the technology [10]. Research suggests that future research opportunities exist, adhering to the improvement of ECMF resources and modeling tools [11]. Because of the opportunities to improve the resources, this study explored the realm of ECMF and user experiences of the green building certification program. The overall objective of the research presented in this paper was to better understand the effects of implementing user experience methodologies into the utilization of green building certification resources. The specific questions asked were:

• How do architects utilize resources to make decisions when building energy-efficient affordable housing?
• What enhancements to ECMF tools and resources improve user experiences and lead project teams to make more energy-efficient decisions?

1.1. Origins of EarthCraft Multifamily

ECMF provides guidance on addressing sustainability issues in low-rise and mid-rise multifamily projects [12]. In recent years, project teams have utilized EarthCraft resources to assist in certifying more than 40,000 energy-efficient homes and multifamily housing units [13]. ECMF resources are structured around sustainable considerations such as affordability, building quality, environmental performance, and satisfaction to help multifamily classified projects attain a points-based certification level [14]. Using resources such as program guidelines and points-based worksheets, ECMF can verify that every project meets the standards set by the program.

Studies have shown the complexity of the correlation between building technologies and user behavior [15]. During these studies, the utilization of energy models and measured energy use were linked to show a great deal of variability, as energy consumption data were collected from more than 300 ECMF units. Preliminary data analysis has shown that accessibility issues exist during the utilization of ECMF resources. The usability of green building certification tools and resources showcases inconsistencies that have the potential to introduce risks in construction projects. Previous studies have measured ECMF’s performance, and to scale that performance, user experiences of stakeholders need to be enhanced by aligning user choices with program goals to present an opportunity for exploration.

1.2. Using Choice Architecture to Improve Green Certification Tools and Resources

By strategically framing options and information in ECMF resources, stakeholders can be nudged into making optimal energy efficiency decisions. Potential risks can be mitigated by improving the usability of tools and resources to promote sustainability considerations [16]. Two impactful ways of improving the tools and resources are enhancing user experiences and choice architecture. Choice architecture is not fundamental to user experience; however, the appropriate structuring supports user experiences with green building certification tools and resources. Choice architecture is best described as the multiple ways to present a choice to a decision maker and the decision that is chosen depends on how the choice is presented [17]. Well-crafted choice architecture can help set better reference points that enable decision makers to arrive at a decision in their own or clients’ best interests [17,18]. An example of this is incorporating appropriate hyperlinks into ECMF resources allowing users to quickly reach desired information without having to overcome other obstacles along the way. A better understanding of how choice structures influence engineering decisions can provide insight into designing more thoughtful decision aids and ultimately leads to more sustainable infrastructure outcomes [6].

1.3. Theoretical Framework for Implementing User Experience (UX)

Different tools and resources are strategically structured around design strategies and have some elements in common even though each tool or resource has a different function [19]. Such resources and tools assist in minimizing the impacts that building construction has on the environment while also generating naturally healthy living spaces [20]. For an EarthCraft tool or resource to be utilized successfully, it is vital that the understanding of users’ needs is reflected by the tools or resources. The Technology Acceptance Model (TAM), a widely used information computer interaction theory, explains and predicts users’ acceptance of a technology [10,21,22]. Under TAM, a person’s beliefs of usefulness and ease of use related to technology are assumed to predict their attitude towards that technology, attitude establishes possible usage intentions, and lastly influence the actual usage of the technology [10]. Future research on green building tools and resources should involve user experience [23].

Incorporating user experience (UX) methodologies into Architecture, Engineering, and Construction (AEC) professional design procedures can be beneficial in the delivery of infrastructure projects. The concept of UX is best described as a person’s experience with a product or system on all facets [24]. Often considered human-centered design, UX makes systems usable and useful in the advancement of these interactive systems. Usability is best described as the usage of technology, interface, tool, or resource by a specific user to accomplish goals associated with concepts such as effectiveness and satisfaction. Usability is targeted through word recognition, grouping and color of interactions with related functions, adding displays with an abundance of text and icons, and writing logically structured manuals [25]. While the concept of usability mainly focuses on an objective approach to interaction, UX explores emotional, subjective, and temporal aspects characterizing the experience between humans and technology [24]. Using UX theories, experiences can be differentiated based on psychological needs fulfilled through technology use [26]. UX has revolutionized design procedures, making it easier to create innovative deliverables at a lower cost by more stakeholders [26].

ECMF has been successful in improving the delivery of energy-efficient affordable housing. However, there are clear areas of improvement with their educational tools and resources, specifically on the UX front. To understand how those areas of improvement can affect the delivery of energy-efficient affordable housing, a usability study or analysis of ECMF tools is needed. By understanding how and why the issues present in ECMF tools and resources need improvement, changes can be made to not only increase the efficacy of the program but also to reduce the risk for users expanding utilization [11]. Analysis of ECMF resources shows that there are readability issues and inconsistencies within and between the resources. Before changes are made in ECMF resources, the appropriate contextual inquiry must take place for the resources as completed in previous UX studies. Contextual inquiry is known as a UX lifecycle activity to grasp detailed descriptions of user work practices for understanding work activities and underlying rationale [27]. Contextual inquiry often includes interviews of users and observations of work practice as it occurs in a real-world setting. A thorough understanding of user interactions allows for ECMF tools and resources to be established at a greater capacity for users’ wants and their usability to be considered more consistently.

UX methodologies are ways of evaluating the usability and user experience of informational materials. Such methods include but are not limited to heuristic evaluations, user interviews, and stakeholder interviews. Heuristic evaluations are methods of having individual evaluators analyze the condition of a system or product’s accessibility and usability [28]. User interviews provide a deep understanding of experiences by enabling interviewees with the ability to express their experiences narratively. These UX methods are concrete representations of the UX realm and are transferrable to the field of engineering.

1.4. Integrating Engineering and UX

Usability factors such as navigability, readability, legibility, design layout, and delivery platforms (e.g., Microsoft Excel and Adobe PDF) can affect the utilization of certain resources within the realm of engineering. Due to the steady rise in convoluted social and technological dimensions over time, usability study utilization can be beneficial to the iteration of our design-making processes as AEC professionals [29]. By integrating engineering and UX, the effective and efficient utilization of tools and resources is assured. In a study exploring the trade-offs of designing support tools, the researchers concluded that tools need to be designed to meet the requirements of the users to be effectively utilized [30]. The understanding of users’ needs is crucial and should be reflected by the tools or resources.

This study analyzes design features that will evaluate user experience and improve the usability of ECMF resources. For example, a design feature that is of great significance is link appearance. Many usability designers have stated that users expect hyperlinks to be present during their interaction with a web interface [31]. These designers suggest that the most effective hyperlinks are blue and underlined, indicating to the user that the link has importance and ultimately leads them to a destination. The use of hyperlinks can be transferred into interactive documents such as ECMF resources due to the documents having a similar usage as a web interface or website.

While it is important to recognize the ease of use, joy of use is a topic that should be addressed more often [25]. Previous research has identified the importance of having links that return users to the home page [31]. These links promote more flexibility when navigating, thus providing a sense of comfort and safety to the users. A clickable “home” text link on internal pages within the resources communicates to the user that the link will take them to the home page if they desire it.

Most green building certification tools and resources usually require a good amount of scrolling and searching for information due to the enormous number of content-rich pages. The addition of hyperlinks is just one of the many features that have the potential to improve the navigation and usability of ECMF resources. For this reason, the addition of hyperlinks was analyzed during this study. Error prevention, font colors, and font sizes are other features that were analyzed during this study because of their potential to influence usability when paired with seniority and experience level. Studies on these features have not been conducted on ECMF resources as of right now, which presents an opportunity to uncover new knowledge about the resources and program in general. It is of great importance for us to understand design features (e.g., the addition of hyperlinks and design layout) in this context to make user-centered improvements in ECMF resources.

The scope of this study followed the seven features of user experience: useful, usable, desirable, findable, accessible, credible, and valuable. The researchers viewed the seven features of user experiences to be beneficial in analyzing the usability of ECMF resources. Usability.gov, accessed on 31 May 2021, an academic source of UX best practices and strategies, identified and structured the seven features of user experience [32] as recreated in Figure 1 below. The seven features should be used to analyze the usability of other green certification resources in sustainable construction.

2. Methodology

2.1. Research Design

This research study was approved by the Virginia Tech IRB. A usability study project design was used to investigate ways to increase the usability and user experience of ECMF tools and resources. This study conducted data analysis on project specifications, heuristic evaluation data, and stakeholder interview data. As the strengths and weaknesses of ECMF were identified, knowledge of the green building certification program was unveiled. The broad goal of this research study was enhanced by the focus of the conceptual understanding in this usability study.

The methodology for this usability study is ideal for exploring the experience of utilizing ECMF resources by collecting data on the effectiveness, efficiency, and satisfaction of users [24]. This usability study consisted of (1) qualitative usability analysis of the influence that ECMF resources have on technology utilization and (2) qualitative analysis of interview transcripts describing architects’ experiences using ECMF resources. Following a deductive approach, the usability analysis leveraged design theories by prioritizing usable design standards. Figure 2 below serves to provide a visual representation of the paper’s methodology.

2.2. Framework of the Usability Analysis

For the usability study to undergo the steps of improving the utilization of ECMF resources, a framework for the usability analysis was necessary. Construction management theories and universal design concepts were linked to serve as the footing for the development of the framework necessary for experimental user evaluation in the usability analysis [33]. Construction management theories such as appropriate communication strategies, project delivery methods, and project scale were used in this study [1]. Universal design concepts such as interface design and scope help in the identification of usability factors that affect utilization. One such factor is navigability, which is crucial in utilization because it signifies an eloquent flow through content-rich resources or delivery platforms [31]. Additionally, two more similar usability factors are readability and legibility. Readability refers to how easily a reader can understand words and lines in written text, while legibility refers to how easily a reader can recognize and perceive font types of individual characters. These factors are significant since usability is targeted through word recognition, grouping and color of interactions with related functions, and adding displays with an abundance of text and icons [25]. Design layout or hierarchy is another usability factor that can affect utilization due to the structuring of text and icons.

2.3. Stakeholder Interviews

To better understand the context of this usability study, valuable information about ECMF resources had to be presented. This stage included a literature review of studies and websites involving ECMF and other green certification programs (e.g., LEED and BREEAM), critical self-observations of the utilization of ECMF resources, and identifying usability and accessibility issues within ECMF resources. Additionally, this stage included stakeholder interviews with the program administrator personnel. The personnel consisted of the technical director and the operations and outreach manager. Through email conversations and virtual phone calls (GoToMeeting and Zoom platforms), questions were asked to gain more knowledge about ECMF and understand the processes associated with the program. Personnel was asked about ideas for potential improvements in the resources and their process of making those changes. Additionally, program administrator personnel spoke on their desire to produce a better online user interface and how streamlining the ECMF program and resources is the first real step to accomplishing that goal.

2.4. Heuristic Evaluations

Heuristic evaluations are methods of having individual evaluators analyze the condition of a system or product’s accessibility and usability [28]. To ensure the appropriate protocol was in place for the evaluations, the authors reviewed prior literature and usability websites. The evaluators, generally with backgrounds as experts or specialists, use guidelines that are industry-accepted for usability and previous experiences to complete their evaluations. For this study, three UX experts served as the evaluators. One evaluator was a professional UX researcher/expert; another was an assistant professor with several years of experience in building construction and UX research. The last individual was well versed in Civil Engineering and UX research. The heuristic evaluations performed on ECMF tools and resources focused on a pre-created list of heuristics (provided below) to determine usability [34]:

• Visibility of system status;
• Match between resource and the real world;
• User control and freedom;
• Consistency and standards;
• Error prevention;
• Recognition rather than recall;
• Flexibility and efficiency of use;
• Aesthetic and minimalist design;
• Help users recognize, diagnose, and recover from errors;
• Help and documentation.

The list of heuristics was aligned with ECMF resources before the heuristic evaluations because the original intention of the list was for online interfaces. The ECMF resources analyzed during the heuristic evaluations were the Version 5 program manual, technical guidelines, worksheet, and an EarthCraft details plan from a previous project (Case Study 1). Case Study 1 was a previous ECMF project where several energy-efficient affordable apartments were built. The details plan from the project was used in this study to assist the evaluators in better analyzing the resources. To analyze the documents, a task was created to help each evaluator utilize the resources necessary to complete the heuristic evaluations. Each evaluator focused on ten cells or line items in the High-Performance Building Envelope (BE) section in the worksheet. In this section, the evaluators filled out the ten cells based on the descriptions in the technical guidelines and EarthCraft details plan from Case Study 1. Using the details plan, the evaluators searched for the description that matched the line item they were on and then searched for the description of the line item in the technical guidelines. The last step was to figure out how to assign the points in the worksheet based on the details plan and technical guidelines. Afterwards, the evaluators analyzed the program manual by looking for any issues with locating information of your choice, navigation, font sizes and colors, content quality, and design layout. Navigability, readability, legibility, design layout, and content quality are major usability features of each analyzed resource that was utilized during the heuristic evaluations.

During each evaluation, evaluators spent approximately two hours analyzing the resources for this research project. Two hours allowed the evaluators to evaluate each tool and resource twice; the first time to gain a better understanding of the tools and resources, and the second time to focus on key elements that capture attention (e.g., previously mentioned usability principles and elements). Violations to the heuristics were recorded during the individual evaluations via the tabular tally system. In addition to recording the violated heuristics, evaluators also rated the severity of the violations. Severity in heuristic evaluations is best defined as an expected impact on the user’s experience with a product or system [27].

After the completion of the individual evaluations, the team of evaluators came together to discuss heuristic violations, usability issues, and beneficial features of each tool or resource [35]. Through these group meetings, common ground was found between the individual evaluator findings and organized, to appropriately measure the data. The severity of the heuristic violations and usability issues were rated on a scale of 1 to 5. The rating scale, as shown below in

Table 3. Severity scale ratings.

Level of Severity	Description of Severity
1	Not a problem; can vary based on individual and situation
2	Small issue that may need attention in the future but not a problem in the current state
3	Minor usability issue that needs attention but more so a low priority
4	Major usability issue that needs attention right now and should be a top priority
5	Usability catastrophe; new versions of the resources should not be released without addressing this

, provides the descriptions of the numeric values that assisted the evaluators in rating the severity of the violations. The findings from the heuristic evaluations were used to structure the interview questions presented to the architects and architectural students.

2.5. Piloting Stage

There was a piloting stage before the architect interviews began to determine the best experimental setup for study participants. A mix of 5 architectural and non-architectural students were the participants of the pilot. The primary assumption was that architectural students’ knowledge of ECMF or other green building certifications programs (e.g., LEED and EarthCraft) more accurately represented the knowledge of a novice in the architectural field. The use of non-architectural students allowed for the understanding of the questions outside of the field of architecture. There were two iterations of the interview questions based on the responses of the participants. Feedback on the questions included recommendations for splitting some questions into multiple questions to allow the interviewee to speak on each resource asked about in the question, the addition of Likert scales, rewording confusing questions, and connecting similar questions to avoid repetition. The pilot helped structure the interview questions to ensure that the interview sessions would be as convenient as possible for the architects that agreed to participate. The hardware and software equipment were also checked during the piloting stage for performance to verify the ability to capture participants’ varying responses to the questions.

2.6. User Interviews

Semi-structured interviews were ideal for this research study because of the desired content required from architects who previously utilized ECMF on past projects. The interviewees were asked open-ended questions to provide a deeper understanding of how architects of all backgrounds utilize ECMF tools and resources in their project delivery methods. Open-ended questions allowed the interviewees’ experiences with ECMF resources to be highlighted by enabling them to reply to the questions narratively [36].

Six participants were ideal for the research study because it gathered experiences of architects over a large radius and the data collected aligned with the data of the heuristic evaluations. While saturation has been considered to be the gold standard for purposeful sampling in fields such as health science [37], the consistent interview data collection protocol used [38], the naturalistic context [39], and transparent purpose [40] of this study justify the adequacy of this smaller sample that does not provide saturation. All six architects who participated, previously used ECMF and were willing to share their experiences with the program resources. In addition to their experiences with ECMF, the demographic profiles of the participants were also recorded. McCoy found demographics played a role in how energy-efficient housing units were used [41]. This theory is translated to this research study by suggesting that demographics affect the utilization of ECMF resources. Therefore, demographic profiles provide insights into “who” exactly is using ECMF resources.

Regarding recruitment, 87 participants were invited to take part in this study through email. Several google searches assisted in the recruitment of participants. The demographic profile of the six architect participants was recorded before each interview. The demographic profile of the participants is shown below in

Table 4. Demographic profile of architect participants.

Demographic Profile
Age	31–56 years old with an average of 45 years old
Years of Experience	9–30 years of experience with an average of 18 years of experience
Occupation	A mix of Associates, Project Manager, Vice President, and Owner
Education Level	A mix of Bachelor and Master of Architecture Degrees
Location	Maryland, Virginia, and Washington D.C.

.

The interview questions garnered the experiences of the interviewees and allowed for a better understanding of the usability of ECMF tools and resources. The questions asked during the interviews were inspired by [42] because there was a questionnaire provided solely for EarthCraft contractors. Navigability, readability, legibility, design layout, and content quality are major usability features that the interview questions were structured to address.

Interview sessions were held remotely over Lookback 2020, a screen recording software that allowed the participants to respond to the interview questions in real time without having to leave their environment. Lookback 2020 documented participants’ voices while they answered the questions presented to them and told stories of their experiences with ECMF. The findings from the interviews assisted in structuring the tools and resources in the A/B testing phase of this research study.

Coding Process

The professionally transcribed text from the transcripts allowed a rapid analysis to take place by using themes from the framework such as navigability, readability, legibility, and hierarchy (page design and layout). Rapid coding analysis occurred in Microsoft Word, where the audio recordings had been transcribed. Descriptive coding served as the first cycle coding method used to analyze the transcripts after the completion of rapid coding analysis. After coding the interview transcripts, a descriptive coding codebook was assembled to allow other researchers and audiences to review and understand the codes of the transcripts. To ensure that the codebook was assembled properly, the requirements of a codebook listed in “The Coding Manual for Qualitative Researchers” [43] were followed. Following the completion of the codebook, the data were pulled into NVivo, a qualitative data analysis software, to help deal with issues such as coding one piece of text as multiple codes and the addition of sub-codes. NVivo allows for more complex coding structures to be created and exported.

Focused coding served as the method of second cycle coding method, which was used to analyze the transcripts after the completion of first cycle coding. To perform focused coding on the transcripts, all the descriptive codes were observed to determine if an expansion of the codes was needed to better organize the codes. Some codes were adequate to not code further, but others such as “navigability” and “readability” required more defined codes to entail what the interviewee was saying. An example of this was one instance of the code “navigability” expanding into “navigating between the resources.” A codebook was assembled following the requirements listed in [43].

2.7. Data Analysis

2.7.1. Heuristic Evaluations

Following the traditional grounded theory method, the data of this usability study were collected by using the research questions to inform data collection protocols. Data collection and analysis alternated throughout this research study to provide and analyze the results. While the research questions informed the direction of this study, the framework of the usability study served as the initial starting point of this research. The heuristic evaluation data lead to the structuring of the interview questions and the interview data will lead to the recommendations for ECMF resources and answer the research questions. Following the collection of data for each phase, analysis of the data was completed.

For the heuristic evaluations, data collection took place through Microsoft Excel, which assisted with the appropriate presentation for the audience. To appropriately present the findings from the heuristic evaluations, the average severity of the heuristic violations was plotted on a spider chart as shown later in Section 3. To create the appropriate chart, a method was established for each evaluation. The number of times the heuristic violations occurred in the evaluation were counted and the severities of that heuristic were averaged. This was performed for all three of the individual evaluations. After, the average severity of the heuristics in the individual evaluations was brought together and averaged overall.

2.7.2. User Interviews

The codes were tracked by adding comments throughout the documents that aligned with the usability themes mentioned above. After the tracking of the codes, the codes were recorded using a code application analysis chart to show how many times a code appeared in a transcript. During descriptive coding, labels are assigned in a word or short phrase to data to summarize the main topic of a passage of qualitative data [44]. These labels are beneficial when the time comes to categorize or index the codes. During focused coding, the most frequent or significant codes are searched for to develop categories in data using the most logical codes [43]. The descriptive codes from the first cycle of coding were used to contribute to the development of salient categories and subcategories. NVivo allowed for the creation of more complex coding structures with the ability to export them. Analysis of the codes was completed once the data were organized so that the interview questions and participants’ responses were viewed appropriately and easy to understand in a matrix format [44].

2.8. Reliability and Validity

To solidify the reliability and validity of this research study, the utilization of the Q3 framework is needed [45]. Q3 stands for qualifying qualitative research quality and the framework ensures quality in interpretive engineering research.

2.8.1. Theoretical Validation

• Straying clear of making evaluative and generalizable claims by only discussing the usability and user experience of ECMF resources;
• Sample a variety of different users for their perception of ECMF experiences;
• Plan to test participants in the future with A/B tests;
• Variety of different researchers working on this study;
• Interviews structured to cover 1 h instead of shorter times;
• Connect the findings back to previous work such as user experience and usability.

2.8.2. Procedural Validation

• Analyzing the UX issues that participants are experiencing in their minds,
• Professionally transcribed audio recordings,
• Rounds of iterations to the codebooks following recommendations from member checking/peer review, and
• Use of NVivo to manage and analyze the codes.

3. Results

3.1. Heuristic Evaluations

The severity of the heuristic violations of the experts ranged from 1 (not a problem; can vary based on individual and situation) to 5 (usability catastrophe; new versions of the resources should not be released without addressing this). Samples of the heuristic evaluations are visualized by the layout in

Table 5. Sample of heuristic evaluations from experts.

Heuristic Violation	How Heuristic Was Violated	Severity
User control and freedom	No hyperlinks that take the user to desired resources, extended dialogue may be needed to find desired information	5
Aesthetic and minimalist design; visibility of system status	Point total at the end of a section is difficult to see and very easy to look over because font size and font color is same as basic cells; also, does not have lines separating planned and actual totals	2
Consistency and standards	Some inconsistencies when it comes to aligning with technical guidelines and EarthCraft Details Plan	3
Help users recognize, diagnose, and recover from errors	If you are in the wrong place in the technical guidelines, there is no easy way to return to the beginning	3
Help and documentation	Where does a user go if they need help with the resources or have a question? The website?	4

below.

The average severity of the heuristic violations was calculated for each evaluation and then the collection of the evaluations overall. The number of heuristic violations from the individual evaluations and the severity of the violations were recorded as shown in

Table 6. Number of heuristic violations from individual evaluations.

	# of Times E1	AVG Severity E1	# of Times E2	AVG Severity E2	# of Times E3	AVG Severity E3	AVG Severity Overall
Visibility of system status	2	3	10	3	10	5	3.67
Match between system and the real world	5	3.6	0	0	0	0	1.2
User control and freedom	2	3.5	0	0	0	0	1.17
Consistency and standards	3	3.3	8	4.25	8	5	4.18
Error prevention	1	2	10	5	10	5	4
Recognition rather than recall	1	2	10	4	10	5	3.67
Flexibility and efficiency of use	1	2	10	3	10	5	3.33
Aesthetic and minimalist design	3	2.67	0	0	0	0	0.89
Help users recognize, diagnose, and recover from errors	2	3	10	3	10	5	3.67
Help and documentation	1	5	10	4	10	5	4.67

below. The three violations that had the highest average severity were “help and documentation”, “consistency and standards”, and “error prevention”. The average severity of the heuristic violations was calculated for all evaluations first, and then overall as a collection. These calculations are shown below using Equations (1) and (2).

$$\mathrm{AVG} \mathrm{Severity}=\frac{\mathrm{Total} \mathrm{amount} \mathrm{of} \mathrm{severities} \mathrm{for} \mathrm{heuristic} \mathrm{violation}}{\# \mathrm{of} \mathrm{times} \mathrm{heuristic} \mathrm{violation} \mathrm{occured}}$$

(1)

where AVG Severity stands for the average severity of the heuristic violation for an evaluator; total amount of severities for heuristic violations stands for the total amount of severities for a heuristic violation; and # of times heuristic violation occurred stands for the number of times a heuristic violation occurred in an individual evaluation.

$$\mathrm{AVG} \mathrm{Severity} \mathrm{Overall}=\frac{\mathrm{AVG} \mathrm{Severity} \mathrm{E}1+\mathrm{AVG} \mathrm{Seveirty} \mathrm{E}2+\mathrm{AVG} \mathrm{Severity} \mathrm{E}3}{3}$$

(2)

where AVG Severity Overall stands for the average severity of a heuristic violation of all evaluations; AVG Severity E1 stands for the average severity of a heuristic violation in 1st individual evaluation; AVG Severity E2 stands for the average severity of a heuristic violation in 2nd individual evaluation; and AVG Severity E3 stands for the average severity of a heuristic violation in 3rd individual evaluation.

One of the biggest issues identified by the heuristic violations was the navigability of ECMF resources. The “consistency and standards” heuristic states that there should be consistency within the language, situations, and actions across the resources. There is a possibility that the users get lost within the resources while looking for specific content. Another example of a navigability issue is connected to the “user control and freedom” heuristic. This heuristic states that the user should have the ability to freely navigate the resources without needing further instructions. ECMF does not violate this heuristic severely but there is a possibility that too much navigability freedom in the resources may influence the likelihood of users getting lost. In addition to Table 6, the overall average severity of the heuristic violations was plotted on a radar chart to appropriately present the findings from the heuristic evaluations as shown in Figure 3 below.

3.2. Piloting Stage

The pilot study provided preliminary results that answered the interview questions constructed from the findings of the heuristic evaluations. The interview questions underwent multiple iterations to ensure that the participants interpreted and responded appropriately to the questions. One question regarding suggestions for improving ECMF resources was deemed to be effective from the first iteration and was later separated into suggestions for improving each specific resource such as the technical guidelines. The quotes below are responses to the question of suggestions for improving the technical guidelines.

“Organized bookmarks in side panel for easier navigation”—Student with architectural background

“Some of the pictures in the Technical Guidelines are a bit pixelated”—Student with no architectural background

In the first quote, the student stated that the organized bookmarks in a side panel would result in easier navigation through the technical guidelines and program manual. In the second quote, the student stated that several pictures throughout the Technical Guidelines were pixelated and could be improved in the future. Other students suggested that the pictures could be improved as well.

3.3. User Interviews

3.3.1. Experiences

The interviews provided the participants with the opportunity to respond to the questions in an open-ended manner. This was also an opportunity for the participants to reflect on their past experiences. The following quotes are snippets from the interviews. DJ represents the researcher, one of the authors, and P1 and P2 represent the participants in the order they were interviewed.

P1:

This is my first EarthCraft project and mostly used the workbook which to help identify the points that we thought we could get and when we began our design. And then I’ve used the technical guidelines as a fallback if I had any questions about the points. The manual I have not used very much at all.

P2:

Well, I think maybe there’s not enough details to apply to every sort of construction opportunity that happens… The visuals could be improved.

3.3.2. Program Manual

One recurring observation in the interviews was the participants’ inability to identify the program manual. In some cases, the participants thought the technical guidelines were the program manual before being told differently. The reasoning for not identifying the program manual correlates with the participants not using the resource as much as the others. As shown below, participants mentioned that they rarely used the program manual for their projects.

P4:

I think the program manual could probably easily be included within the technical guidelines… I don’t feel that they would need to be separate…

P5:

The manual I probably interacted with the least… I feel like my experience when I have questions that I want to go to the manual for, I’m better off going to the technical advisory and talking to a human.

3.3.3. Weaknesses and Improvements

The interview data solidified concerns presented in the findings of the heuristic evaluations. As shown below, many of the participants had similar responses when asked about the weaknesses of the resources and ways of improvement. These responses correlate with the pilot study quotes previously mentioned.

DJ:

What is the biggest weakness of the workbook in your opinion?

P1:

I think it could be a little bit better with the graphics…But maybe if the graphics were a little bolder so you could find the particular thing you were looking for more quickly if you’re not really familiar with every category.

DJ:

Do you think that hyperlinks to aid navigation for the resources would be beneficial?

P3:

Yeah, 100%. I think the other thing they can add to their PDFs is bookmarks…it’s like having that table of contents available to you wherever you are in the document.

DJ:

Would you think that improvements to the font colors and font sizes benefit any resources?

P4:

Yeah, certainly. I mean, the Excel world that EarthCraft lives in now, lends itself to the more technical aspects of it. But it definitely could be more friendly. It could be more easy to use. Yeah, I definitely think the color palette is a little dated...that kind of the drawings are a little dated. Yeah. I think it would help.

3.3.4. Coding Process

Since the text from the transcripts was professionally transcribed, it allowed a rapid analysis to take place by using themes from the framework and heuristic evaluation data such as navigability, readability, legibility, and hierarchy (page design and layout). After the completion of the rapid coding analysis, the codes were recorded in a code application analysis chart as shown in

Table 7. Code application analysis chart of rapid coding analysis.

	Navigability	Readability	Legibility	Hierarchy	Content Quality	Illustrations	Building Type
P1	9	10	1	6	4	10	15
P2	15	10	4	3	3	22	7
P3	17	1	3	1	7	33	10
P4	14	5	4	7	8	7	2
P5	13	2	1	4	2	11	0
P6	11	6	2	6	11	3	0
Totals	79	34	15	27	35	86	34

. The code application analysis chart shows how many times a code appears in the interview transcripts.

The descriptive coding cycle brought forth new codes that were not identified during rapid coding analysis. In addition to the themes of the framework being used as codes, one- or two-word descriptions of qualities, properties, phenomena, and concepts were used as codes.

Table 8. Code application analysis chart for descriptive coding.

	Accessibility	Building Type	Content Quality	Illustrations	Navigability	Readability
P1	8	14	4	11	8	10
P2	2	7	3	23	14	9
P3	6	12	10	38	21	1
P4	10	10	8	9	14	5
P5	3	3	12	13	14	2
P6	5	0	11	4	12	10
Totals	34	46	48	98	83	37

shows the code application analysis chart of the codes that appeared the most during the descriptive coding cycle and ties back into the heuristic evaluation data.

The focused coding cycle brought forth new codes that were not identified during rapid coding analysis and descriptive coding. The focused codes were organized in a hierarchical structure of categories and subcategories that were centered around improving the resources. Five themes appeared during the heuristic evaluations that contribute to improving the resources. The five themes were error prevention, navigation, accessibility, improving font sizes and colors, and enhancing illustrations. The total number of focused codes for each theme was recorded from each interview transcript as shown in

Table 9. Code application analysis chart for focused coding themes.

	Error Prevention	Navigation	Accessibility	Improving Font Sizes and Colors	Enhancing Illustrations
Participant 1	6	7	11	12	11
Participant 2	0	10	2	3	17
Participant 3	2	25	11	2	23
Participant 4	0	16	15	3	7
Participant 5	2	18	4	2	10
Participant 6	3	16	16	5	3
Totals	13	92	59	27	71

. Each of the themes was split into at least three categories that were organized by specific aspects of that respective theme as shown in Figure 4.

The spider chart in Figure 5 shows the total number of focused codes for each category identified under the five major themes. The categories were split into subcategories that were labeled after specific aspects or features identified during the interviews. The subcategories were not included in any charts because the figure became unreadable by doing so.

The category with the greatest number of focused codes was navigation between the resources, with 55 total codes. This finding aligns with the issues of navigation identified in the heuristic evaluations because the interviewees discussed the lack of linkage between the resources and ideas that may help improve the resources in that category. The category with the second the greatest number of focused codes was appealing to users, with 39 total codes. Some of the interviewees discussed how the resources appealed to them and their belief that the resources will progress to better appeal to other professionals in the future besides architects (e.g., builders, labor men). The illustrations categories provided a steady number of focused codes alongside navigation and accessibility.

4. Discussion

4.1. Navigation

Based on the results from the interview protocol, navigability appears to be a major issue within ECMF resources. The heuristic evaluations allowed the researchers to see that the issues and violations were closely grouped. The interview transcripts and focused codes brought navigability to the forefront of our analysis. Violated heuristics that focused on the navigability of the resources include

• Visibility of system status;
• Recognition rather than recall;
• Help users recognize, diagnose, and recover from errors.

After the completion of the expert heuristic evaluations, all three heuristic violations finished with the same severity rating. The “visibility of system status” heuristic states that the user should always be able to understand where they are in the resources with little to no confusion [28]. There was confusion expressed by the experts in the data collection phases about a user’s location in the resources, especially for those unfamiliar with the content. The “recognition rather than recall” heuristic states that the user should not have to remember information from one resource to another, and the information should be visible or easily retrievable whenever appropriate. The experts deemed that the resources violated this heuristic and reduced the user experience by not having the appropriate linkage in place to connect the resources.

The architect interviews provided evidence for this violation when the participants gave personal recommendations for improving the navigability of the resources. The “help users recognize, diagnose, and recover from error” heuristic states that the user should be able to use the resources to help indicate a problem and then constructively suggest a solution without contributing to any problems or misunderstandings of the users. A concern of the experts was that there was no easy way to return to the beginning pages in the program manual and technical guidelines, both documents with considerable page lengths. The only way of returning to the beginning pages is to scroll up, which has been proven in prior literature to annoy users [31]. The annoyance with scrolling was further evidenced in the interviews as the participants suggested using bookmarks to return to specific pages or sections and minimizing frustration.

To improve navigability, adding hyperlinks or bookmarks to the resources could help architects make quicker decisions. Before incorporating proven techniques to enhance navigability in documents, it is important to consider the recommendations from the participants since they are the users of the resources. The joy of use associated with tools and resources is highly important to users [25], and not obtaining their opinions on improvements could result in non-helpful changes, unhappiness, or annoyance. During the interviews, participants expressed a desire to have hyperlinks included in specific sections of the worksheet. The participants felt that so much time and effort could be saved if there were hyperlinks for them to access, taking them directly to the sections in the technical guidelines. An alternative recommendation of the participants centered around adding mouseover features allowing them to hover over hyperlinks and provide detailed descriptions of material without leaving the worksheet. The qualitative data also showed that there was a desire by the participants to have hyperlinks navigating to or presenting material outside of ECMF resources. Outside sources of material include ASHRAE requirements, AHRI certificate requirements, ACCA software, manuals, and documentation.

4.2. Enhancing Illustrations

The participants explained how the illustrations in the technical guidelines sometimes did not apply to their project, which pushed them to problem-solving methods and assistance from the program administrator to adjust to their project. The disconnect between the illustrations and the project was determined to be the result of missing illustrations which were classified under the “lacking illustrations” code that appeared across the transcripts 12 times. The “expanding illustrations” code also appeared 20 times across the transcripts because the participants believe that ECMF could improve their resources if they covered a greater variety of building types with their diagrams. The architects revealed that there are many building types certified under ECMF, which should be addressed in the resources. Illustrations in educational tools such as ECMF resources are extremely useful for experts in the professional world.

The participants’ feelings about the illustrations were unpredicted takeaways from the architect interviews because they were not identified as an issue in the heuristic evaluations. In the evaluations, the “match between system and the real world” heuristic specifically covered the content within ECMF resources as well as the illustrations. The heuristic had one of the lowest overall severities after the evaluation phase and was not considered an issue in the resources. While the architects expressed their frustrations with a few of the illustrations, they acknowledged the understanding of how it may be difficult for the program administrator to implement all building types in the resources. In fact, every participant gave praise to some of the illustrations and work that the program administrator had performed. The participants spoke on how the visual references were a more powerful tool than the wording because they communicate information quickly and with a level of clarity. Alongside applause, the participants also discussed the concern of the researchers possibly having a negative impact on the resources while attempting to improve them in the future.

4.3. Accessibility

The participants discussed some frustration that came along with accessing the most recent version of the resources online. The researchers documented these instances and coded them under “accessing the resources,” a focused coding category associated with accessibility. The participants expressed the belief that the resources would be easier to access if there was a singular interface containing all the resources. This belief correlates with the desire of the program administrator to expand to an interface in the future, mentioned previously, and the experts’ recommendation of implementing the resources into an interface to mitigate issues.

“Appealing to users” was one of the most expansive categories in the focused coding stage. The participants of this study felt that appealing to the users of the resources is important to improve future ECMF resources. In a study exploring the trade-offs of designing support tools, the researchers concluded that tools need to be designed to meet the requirements of the users to be effectively utilized [30]. ECMF users apply the resources to their projects and thus require the attention of the designers to make appropriate adjusts to the resources. While the participants of the interviews gave several recommendations for improving ECMF resources, they also recommended not to alter the setup of the resources to a great extent. The participants mentioned that the current setup of the resources is easy to use, and altering them too much could cause their utilization to be a burden, as mentioned in the previous section. This concern supports the “KISS” (Keep It Simple Stupid) and “Don’t fix what isn’t broken” approaches. These thinking approaches could also be tied to status quo bias and loss aversion [6]. Status quo bias and loss aversion are cognitive barriers that cause decision makers not to stray from original systems or methods. The possibility of losing ease of use in ECMF resources raises the effect of status quo bias and loss aversion in the architects.

The heuristics of “user control and freedom” as well as “aesthetic and minimalist design” are directly tied to the accessibility of ECMF resources. Both heuristics were the lowest-rated severities following the completion of the heuristic evaluations with average severities of 1.17 and 0.89, respectively. The “user control and freedom heuristic” states that the user should have the ability to freely navigate the resources without needing further instructions [28]. The rating from the heuristic evaluations shows that even though ECMF resources have navigability issues, users still have the freedom to navigate to their own degree. “The aesthetic and minimalist design” heuristic specifically focuses on the aesthetic appeal of the resources and states that the resources should appeal to all users. The rating from the heuristic evaluations correlates with the participants’ statements because it shows that both the experts and participants believe that the resources appeal to their users and minor aesthetic improvements are needed to enhance the resources.

4.4. Improving Font Sizes

The heuristic evaluations showed that there were issues with font sizes within ECMF resources which encouraged the researchers to ask the participants about their feelings regarding font sizes. The main heuristic that focused on font sizes was the “visibility of system status heuristic” which was one of the second highest severely rated heuristics, as mentioned previously in Table 5. During the evaluations, the experts identified concerns with the point totals at the end of the sections in the worksheet as well as other cells. The cells identified by the experts were difficult to see and easy to overlook because the font size was the same as the other cells listed around them, making the user lose track of where they were in the worksheet. This concern was strengthened when some of the participants expressed previous issues with the font sizes and the color of the fonts. Previous research conducted on the role of font size on web usability suggests that a font size of 12 and 14 are preferred over smaller sizes for legibility purposes and enhanced reading [46]. The preference between a font size 12 and 14 depends on the user’s personal interests; however, it is reported that the average adult at the age of 24 prefers a font size 12. Using choice architecture [17] to restructure ECMF resources, appropriate font sizes can nudge users to be more aware of their positioning in the resources. As mentioned before, the participants of this study experienced issues with font colors in addition to the font sizes.

4.5. Improving Font Colors

Like font sizes, the heuristic evaluations showed that appropriate font colors were needed. The participants provided evidence that lessons from color theory and harmony [47] ensure appropriate colors are used when improving ECMF resources. One interviewee spoke on the difficulty of trying to find a cell in an unfamiliar section due to the cells having the same colors and formats. Some of the sections were not spaced out appropriately, so it appeared as if some sections never ended and continued into others. Other participants shared the belief that the color aesthetic of the resources was “outdated”, and minor improvements could be made to the font colors. The same participants also reiterated their concern that attempting to update the font colors could distract users and ultimately disrupt usability. The participants’ comments are evidence to ensure that color theory is used when improving the colors of ECMF resources. Weingerl and Javorsek provide guidance on what colors can improve a document without affecting usability [48]. Using choice architecture [17] to restructure ECMF resources, appropriate font colors can be added to nudge users to be more aware of critical information.

4.6. Error Prevention

The category of “error prevention” was expected to be one of the main issues pointed out by the interviewees, but instead rendered the least number of focused codes with 10. The researchers expected a different outcome based on the findings of the heuristic evaluations. Following the completion of the evaluations, the findings showed that the most severe violations were:

• Help and documentation,
• Consistency and standards, and
• Error prevention.

All three heuristics had the same severity rating and are connected to the concept of error prevention to some degree [28]. According to the experts that completed the evaluations, there were multiple reasons why these violations were the most severe. One of the biggest concerns highlighted with the “help and documentation” heuristic was where the user would go if they had a problem or question about a resource. This concern was later addressed in the interviews as the participants shared their process of directly contacting their Technical Advisor from the program administrator with any questions or concerns.

Another concern identified by the experts was the inconsistencies found between the technical guidelines and the EarthCraft details plan from Case Study 1 which violated the “consistency and standards” heuristic. The “consistency and standards” heuristic states that there should be consistency within the language, situations, and actions across the resources. While not every ECMF project will have every line item identified in the resources, the line items in the details plan of the project should be found within the resources. For example, a specific window is identified in the details plan but there is no information about that specific window in the technical guidelines, which leads to confusion on the construction project and filling out the line item in the workbook. During the interviews, some participants described how the technical guidelines seemed as if they were geared towards a certain type of building rather than a variety of different ones. One interviewee expressed frustration with a version of the technical guidelines and worksheet used on their project due to the two not being consistent throughout. This frustration ties back to the recommendation of having all the resources on one interface to mitigate issues with accessing the latest versions.

4.7. Final Discussions

ECMF resources are very content rich and structured to deliver that content to users. An improved version of ECMF resources could benefit from shifting more focus to how users are utilizing the content. The improved version would improve the way that the users perceive and navigate these resources. The results of this study indicate that improving the way that users perceive and navigate these resources enhances user experience. The current version of ECMF resources contains some hyperlinks but still lacks others that would vastly improve navigability such as linking the resources together. There is also potential to incorporate more hyperlinks to outside sources of information to assist with certain stages of the certification process. Changes to framing in the future versions of the resources may make a prominent change in project performance as the ECMF program progresses.

5. Conclusions

Appropriate content and options can be strategically framed to improve the usability of ECMF resources and enhance user experience. This research study suggests that improvements to the navigability and user experience of ECMF resources enhance architectural decision making. Before the next version of ECMF resources is released, the researchers suggest the program administrator utilize heuristic evaluations to test ECMF resources for usability issues. The evaluations will identify errors such as navigability and accessibility issues which allow for the appropriate UX techniques to be exercised for improvement. It would be beneficial to bring more UX specialists and green building experts onto the research team to provide diverse perspectives to this field of research. One suggestion for future research regarding user interviews is to explore the experiences of other architects from places outside Maryland, Virginia, and Washington D.C. who have had exposure to ECMF resources. The researchers believe that different building codes in certain states will affect how ECMF resources are utilized. It would also be ideal for future studies to test the recommendations from this usability study with similar participants (e.g., architects). Future studies have the potential to expand to other groups of ECMF users such as contractors and builders. Stakeholder interviews are beneficial to this research because they allow the researchers to grasp the different perspectives and goals of multiple users.

The presented study is limited by the architects individually speaking about their experiences with ECMF resources. Therefore, the researchers do not know whether an improved version of the resources will have similar influences on a team of professionals working together. Another limitation of this research is that creating and appropriately explaining tasks to the experts of the heuristic evaluations were difficult due to ECMF resources having a direct connection with construction projects. The only way to create an effective task for the heuristic evaluations is to use an older project (e.g., Case Study 1) or create an imaginary project. Using an older project to help with the task may not directly correlate with newer versions of the resources and cause usability issues. On the other hand, using a created project is considerably more time-consuming and may be less effective than a real project.