Sustainable Assessment of Public Works through a Multi-Criteria Framework

Enhancing the sustainability of public works has been a key agenda in recent years for many governmental organizations. Public works contribute significantly to a large portion of engineering works and have great potential to impact the sustainability of cities. Thus, evaluating the sustainability of these projects is highly relevant, mainly regarding their impacts on environmental, social, and economic aspects. There are currently assessment systems and methods with different scopes and approaches. Yet, there remains uncertainty when it comes to considering public works’ sustainability and how useful criteria can be incorporated into the proposed assessment tasks to ensure such a goal. This study contributes to filling this gap by developing, through an extensive and detailed bibliographic research, a flexible and comprehensive framework composed of 214 criteria distributed across nine categories that measure the degree of sustainability of public works, with emphasis on economic, social and environmental goals. The proposed framework can act as a practical tool, functioning as a checklist applicable to all types of public construction works, and at any stage of the lifecycle. Evaluation of the framework by professionals indicated its suitability when encompassing sustainability objectives, its viability, and its ease of use.


Introduction
The construction industry is known as a sector that consumes a lot of energy, with high rates of use of natural resources, which negatively impacts the environment [1][2][3]. Some of the reported undesirable effects of the industry include air and water pollution, solid waste, deforestation, toxic waste, greenhouse gas emissions, health risks, global warming and other negative consequences [4][5][6][7].
The impacts of construction on the environment are higher in developing countries [8][9][10][11], whose participation in the world economy is growing at a notable rate [12]. Bibliographic research has shown that there is a scarcity of studies that focus on how public works' evaluation is carried out to ensure the sustainability of construction projects in developing countries. Improving working conditions and labor standards, health, and safety The vagueness of definitions and diversity of interpretations Assisting disadvantaged groups in society The separation between capital budget and operational budget Upskilling your workforce to meet the future needs of your organization Lack of sufficient time to address sustainability issues Saving the long term by considering the whole lifecycle cost Lack of long-term perspective Meeting international obligations (e.g., the Kyoto Protocol) The general perception that is addressing sustainability always leads to incurring a higher capital cost Improving the efficiency and transparency of procurement procedures Resistance to change Stimulate the market for green technologies

Insufficient research and development
The need for such assessment tools on most public works' projects have been firmly established as they offer guidance to the decision-making process when it comes to the development of sustainable policies [29]. Any sustainability assessment tool should ideally act as a reference standard to monitor and measure performance [30,31], besides serving as a source promoting knowledge and teaching of sustainability. Existing tools, however, are currently restrictive in terms of application due to their complexities and high time consumption [26].
The present study contributes to filling this gap by proposing a framework for sustainability assessment of public works, which incorporates and synthesizes logically related concepts, guidelines, and criteria proposed by researchers and institutions working within the field. The objective was to develop a tool that would offer a broader set of criteria by incorporating and making compatible the criteria considered important by researchers and those contained in tools: BREEAM, LEED, Green Star, CASBEE, and SBTool. In practice, this means that the evaluator will have a tool that includes a higher number of indicators for each sustainability concept to be evaluated, among which he will be able to select the most suitable ones to fit the characteristics of the enterprise to be evaluated. For public works, this higher coverage is essential, given its variety of typologies. A framework can be understood as a set of coherently associated stages to perform tasks that promote the achievement of a given objective [32]. To make sure that the tool developed in this work is widely accessible to practitioners and researchers in the field, an Excel spreadsheet is developed that operationalizes and facilitates the use of the framework. The developed framework will, therefore, be of great benefit to administrators and professionals in the area of public works. The emphasis in the developed framework has been on ensuring that it is a simple instrument for everyday use, which enables public administrators, even without being specialists in sustainability, to evaluate the sustainability of a public project from its conception. This way, the proposed method can act in a preventative way and promote the necessary alterations to carry out projects with higher levels of sustainability. The proposed framework meets the following premises: (a) it covers any type of work from the simplest to the most complex; (b) it reduces the range and quantity of assessment criteria to only those with correspondence, having similar specificities and characteristics to work being assessed; (c) it enables the assessor to value or devalue a given criterion in relation to the others.
Considering that public works include all types of activities such as construction, renovation, manufacture, recovery or expansion [13], and all sorts of project types, including edification, transport infrastructure, public space or public service (water, sewage, and energy), the wide depth of coverage of the proposed framework is essential to assist public administrators in acting sustainably at the various stages of the construction lifecycle. As a result, the integration of the proposed framework with viability studies, planning, project, construction, operation/use, and demolition, is possible. In Section 2.7.1, we present the summaries of the phases of the construction lifecycle, what each phase contemplates, and the interconnections of the criteria used in the framework with these phases.
To validate the proposed sustainability assessment tool, it was evaluated by professionals that work on public projects, in order to certify its applicability and practicality. In the next section, an outline of the proposed framework is given. Later, some results will be demonstrated based on the evaluations of specialized personnel that have evaluated the tool. Figure 1 outlines the method that has been developed in this study as a framework to guide the sustainability assessment of public work projects. The first step involves conducting a comprehensive literature review to reveal criteria and tools for sustainability assessment that have been previously proposed. The second step involves the selection of a set of suitable criteria from the ones identified in the literature and the creation of a category structure. The third step builds the Consolidated Structure of Criteria (CSC) by incorporating the selected criteria into each of the categories created in the previous step. In the fourth stage, the criteria contained in the CSC are linked to the different characteristics of public works, resulting in the sustainability assessment spreadsheet (SAS). The fifth step is the development of a computerized instrument that facilitates the use of the framework. Finally, in the sixth step, the framework is evaluated by professionals who work with public works to test out the need for further improvement. The details of each of the steps outlined in the framework will be covered in each of the sections below.  After selecting the publications based on keywords, the next step was to eliminate the 724 duplicated studies. The titles and abstracts of the remaining 1422 papers were then read, and only those presenting clear evidence of providing a relevant contribution to the theme were selected. Studies that were excluded include: those without a named author; those published in periodicals without the peer review system; those whose complete texts were not available; studies which had been developed in a language other than English, Spanish or Portuguese, and those whose title or abstract were not related to the field of concern.

Literature Review
Bibliographic research was conducted to identify sustainability assessment criteria and tools in public works. A wide and detailed bibliographic search was performed on databases such as Web of Science, Scopus, and SciELO.
The search was initially conducted through the following keywords: "sustainable construction", "green construction", "sustainable procurement" and "green procurement" and these were also associated with words like "public", "government", "criteria" "indicator", "assessment" and "legislation".
Bibliographic research initially explored publications from the last 10 years presenting criteria for sustainability assessment in public works. Subsequently, due to the scarcity of results, the research increased the period to 20 years, incorporating environmental studies not related to the public sphere and consulting references from the publications selected initially. Other studies made available by different sources were also chosen, providing a list of 2146 studies.
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) recommendations were adopted to guide the bibliographic search. The aim of using PRISMA is to improve the reporting of systematic reviews and meta-analyses. A systematic review relies on selecting relevant research and analyzing data from what is included. When necessary, meta-analysis is used to incorporate statistical techniques into the analysis process. The four phases of the PRISMA flowchart were followed to summarize the results of the literature search by highlighting: (i) the number of articles identified; (ii) articles included and (iii) excluded articles, and (iv) the reason for the removal of the articles [33].
After selecting the publications based on keywords, the next step was to eliminate the 724 duplicated studies. The titles and abstracts of the remaining 1422 papers were then read, and only those presenting clear evidence of providing a relevant contribution to the theme were selected. Studies that were excluded include: those without a named author; those published in periodicals without the peer review system; those whose complete texts were not available; studies which had been developed in a language other than English, Spanish or Portuguese, and those whose title or abstract were not related to the field of concern.
In the remaining 326 articles, we did a quick reading in all sections to capture the general idea of the text and, thus, to be able to decide on its originality, validity, and clarity of the methodology, which resulted in the exclusion of 198 articles.

Selection of Sustainability Criteria for the Assessment of Public Works
A total of 33 documents were selected from the bibliographic review, from which 873 preliminary criteria were extracted, with no type of filter, as per Table 2. These indicators were sequentially listed, each associated only with a description and reference, without any kind of treatment.
The table created from the indicators, and which was associated with the structure of categories described in item 2.5 of Figure 1, served as a base for categorization, reordering, and unification of similar criteria that have the same objective in the assessment of sustainability.

Identification of the Criteria for the Assessment of Sustainability in Works
Among the 59 studies incorporated, 33 publications [1,8,21,22,[27][28][29] possessed a set of sustainability assessment criteria in their content. These were included in the analysis conducted for the framework, regardless of their focus, objective, or whether they were aimed exclusively at public works or not.
The other 19 publications used in this article, although not containing criteria for assessing sustainability in public works, contained theoretical references on different subjects related to the theme; such themes included sustainable development, sustainable construction, bidding processes, and sustainable contracting. The reviewed publications also collectively encompassed relevant economic and social data, and other information that justified the development of the theme and which give consistency to the content of the work.

Selection of Sustainability Criteria for the Assessment of Public Works
A total of 33 documents were selected from the bibliographic review, from which 873 preliminary criteria were extracted, with no type of filter, as per Table 2. These indicators were sequentially listed, each associated only with a description and reference, without any kind of treatment. The table created from the indicators, and which was associated with the structure of categories described in item 2.5 of Figure 1, served as a base for categorization, reordering, and unification of similar criteria that have the same objective in the assessment of sustainability.

Identification of Sustainability Assessment Tools in Construction Work
According to Wang et al. [61], the most widely used tools for assessing the sustainability of works are LEED, BREEAM, CASBEE, and Green Star. Analyzing the 59 studies incorporated, 34 publications were found to refer to the evaluation tools, according to Table 3, and the most cited tools were also LEED, BREEAM, CASBEE, and Green Star. For the reasons mentioned above, they were selected to serve as a basis for forming the category structure of the proposed framework.  Table 4 below. Subsequently, from the concepts contained in each of the tools, a new structure was proposed incorporating concepts that were amalgamated from analyzing the benefits of all the individual systems identified. Table 5 presents the new proposed structure and the full and partial influence on the new categories from the assessment tool categories.

P1-SOCIAL, LOCATION AND TRANSPORT
1 For symbols, please refer to Table 4.
The following descriptions detail what each of the proposed new categories assess. It establishes the parameters that enable the comparison between the 874 criteria extracted from the 33 publications reviewed. P1-Social, location, and transport aspects: Assesses macro aspects for carrying out a project, its urban context, and the points that vary according to region. It considers the environmental, social, and economic differences of each location and promotes their development. P2-Management: Assesses administration techniques (set of tools conceived to support companies in decision-making) aimed at sustainability of different phases of the work (viability, project, construction, use and maintenance, and dismantling), including legal aspects, transparency, ethics, the involvement of interested parties, training (community and industry), resilience, quality policy, human resources, commissioning and planning practices and construction, operation, and maintenance control. P3-Costs and economic aspects: Assesses issues related to costs and economic aspects, including prosperity and productivity of the region, financial accessibility to habitation, job opportunities, cost of capital, operational cost, lifecycle cost, the economic impact on the urban area and economic viability of commercial activities. P4-Soil use and ecology: This is limited to the area occupied by the work and assesses strategies for sustainable deployment of the project, including accessibility; reuse of constructions; reduction of environmental impact on the community and neighborhood; decrease in air, light and water blocks; heat island impediment; minimization of pollution; recovery of biodiversity; flexibility and endeavor in the use of soil. P5-Indoor environmental quality: Assesses issues related to the internal environment of the construction, including accessibility; lighting, thermal and acoustic comfort; wellbeing, health, and safety; air quality, low emission of VOC (volatile organic compounds); external view; natural, hybrid and mechanical light and ventilation (cold and hot); monitoring of external air quality; functionality and efficiency; flexibility and adaptability of the construction; maintenance of environmental quality. P6-Energy: Assesses the design, sustainable use and efficiency of energy, including the incorporation of efficient systems and equipment; generation of renewable energy and energy from alternative sources; reduction of CO2 emissions, greenhouse gases and cooling gases (non-use of CFCs); energy management of the construction and use of the work; reduction in demand; analysis of incorporated energy and operational energy. P7-Materials and resources: Assesses the measures for reducing the impact of construction materials on the environment, including the responsible acquisition of materials with low incorporated ecological impact (extraction, processing and manufacture and recycling); waste reduction; use of certified timber; waste administration (correct destination, selective collection); reuse of discarded materials; lifecycle analysis of the material; minimization of the use of virgin material.

P8-Water:
Assesses the sustainable use of water in the operation of the building and its land, including a reduction in the consumption of potable water (indoor and outdoor); minimization of losses through leakage; utilization and treatment of rainwater and greywater (reuse for watering, cleaning, cooling facilities and toilets); adoption of wastewater treatment stations; incorporation of efficient water-saving devices and rainwater administration. P9-Innovation: Assesses the benefits to sustainability that go beyond that required by the assessment tools (BREEAM, LEED, etc.), including the deployment of innovative practices, processes, and strategies that promote sustainability in the built environment; incorporation of innovations in the projects and innovative professional participation.

Structuring of the Criteria
To obtain the structure of criteria necessary for the development of the framework, the steps detailed in the flowchart in Figure 3 were followed.  As such, two columns were added to Table 2; one referring to the MAIN CATEGORY, already mentioned in the past studies, and the other relating to SECONDARY CATEGORY, which indicates subcategorization, as shown in Table 6.

Reordering of the Criteria in the Categories
After distribution and ordering of the 873 preliminary criteria into the proposed principal and secondary categories, it was possible to distinguish and group identical or similar criteria and propose new wording for criteria that essentially carried the same meaning.
To carry out the next stage, a new column was inserted in which texts containing the new descriptions of the criteria were elaborated and positioned, resulting in Table 7. Verification of the framing of all the criteria was repeated until there were no inconsistencies. As shown in Figure 3, the first step links the criteria from Table 2 to the proposed categories. Second, the table is ordered by category. A check is then conducted to ensure that the criteria have been linked correctly; this procedure is repeated until there are no inconsistencies. Finally, identical or similar criteria are unified with a new text, consolidating the criteria framework (CSC).

Framing of the Criteria in Categories
Due to a large number of criteria, and the fact that the descriptions for a given criterion vary across studies, it was necessary to unify those with the same representations. Detailed analysis of the articles was also needed to identify what the authors intended to represent in the description of each criterion. This then permits the correct framing of the criteria within categories.
Framing also resulted in a high number of criteria concentrated in specific categories, making the unification process difficult. The creation of a secondary category was deemed necessary to improve the analysis.
As such, two columns were added to Table 2; one referring to the MAIN CATEGORY, already mentioned in the past studies, and the other relating to SECONDARY CATEGORY, which indicates subcategorization, as shown in Table 6.

Reordering of the Criteria in the Categories
After distribution and ordering of the 873 preliminary criteria into the proposed principal and secondary categories, it was possible to distinguish and group identical or similar criteria and propose new wording for criteria that essentially carried the same meaning.
To carry out the next stage, a new column was inserted in which texts containing the new descriptions of the criteria were elaborated and positioned, resulting in Table 7. Verification of the framing of all the criteria was repeated until there were no inconsistencies.

Unification of the Same or Similar Criteria
At this stage, the structure of the criteria was consolidated with the new descriptions, and the original criteria eliminated. As a result of unification, a structure was obtained containing 214 criteria distributed across nine categories as per Table S1, which is the basis of the framework for the sustainability assessment of public works.

Linking of the Criteria to Different Characteristics of the Project
To facilitate the work of the assessor, the criteria of the CSC were linked to certain characteristics that differentiate one project from another, as shown in Table 8: Table 8. Characteristics of projects.

Aspects Characteristic
Phases-Lifecycle planning, project and bidding, construction, use, and deconstruction; Types of intervention construction or renovation; and Types of construction building, urbanization, transport infrastructure, and facilities infrastructure.
The 214 criteria that are part of the CSC cover all types of public works. However, to evaluate a specific project, several criteria need not be assessed. For example, when evaluating the sustainability level of building renovation in its use phase, some land use criteria are not adequate and would be discarded in such instances.
To make the tool easier to use, several columns are inserted into the CSC, representing the characteristics of Table 8 and later linked to each of the 214 criteria. Then, the spreadsheet is automated in such a way that the assessor, when entering the system information on the characteristics of the work, automatically eliminates the inappropriate criteria.
This preliminary step assists public administrators, even those without extensive technical knowledge on aspects of sustainability, in measuring the extent of the sustainability of a given construction.
The following items detail the different characteristics of construction works and how linking the criteria contained in the CSC occurred.

Linking of the Criteria to Phases in the Lifecycle of the Project
There is no universal consensus on the phases that make up the lifecycle of a public project. However, some stages reoccur in various studies, as listed below.   Deconstruction: It occurs at the end of the useful life of the work when the construction loses its usefulness and must be discarded or replaced. In this phase, the following works are elaborated: demolition, waste disposal, site recovery [30,66].  The CSC was modified, with five columns representing the different phases of the construction lifecycle being inserted. Each of the criteria was separately analyzed and linked to one or more of these stages. Table 9 shows the number of criteria that were linked to each phase of the work lifecycle, distributed by category. It can be seen that the Project and Bidding phase has the highest number of associated criteria, followed by the Construction and Use phases. This is the phase in which the Bid Notice is prepared, bidding is processed and, especially, where public agents have the chance to alter guidelines, change projects, specify equipment and materials, and determine new constructive solutions that incorporate more sustainable elements into the project.

Linking of the Criteria to Types of Intervention
Type of intervention is another aspect that differentiates one project from another. According to TCU [13], "Public work considers all construction, renovation, manufacture, recovery or extension of a public asset." Upon linking the sustainability criteria to the five types of intervention mentioned, it was found that only those that represent new constructions were differentiated from the others. Therefore, it was possible to group types of intervention into just two groups:


Renovation-incorporates recovery, extension, and renovation itself. This type of intervention impacts pre-existing constructions.  Construction-refers to new works and incorporates manufacturing and construction itself. This type of intervention is differentiated from the previous by impacting aspects of its location and soil occupation.
Once again, the CSC was expanded, this time with two new columns representing types of intervention. Each criterion was analyzed again and linked to one, or both, of the interventions.  The CSC was modified, with five columns representing the different phases of the construction lifecycle being inserted. Each of the criteria was separately analyzed and linked to one or more of these stages. Table 9 shows the number of criteria that were linked to each phase of the work lifecycle, distributed by category. It can be seen that the Project and Bidding phase has the highest number of associated criteria, followed by the Construction and Use phases. This is the phase in which the Bid Notice is prepared, bidding is processed and, especially, where public agents have the chance to alter guidelines, change projects, specify equipment and materials, and determine new constructive solutions that incorporate more sustainable elements into the project.

Linking of the Criteria to Types of Intervention
Type of intervention is another aspect that differentiates one project from another. According to TCU [13], "Public work considers all construction, renovation, manufacture, recovery or extension of a public asset." Upon linking the sustainability criteria to the five types of intervention mentioned, it was found that only those that represent new constructions were differentiated from the others. Therefore, it was possible to group types of intervention into just two groups: • Renovation-incorporates recovery, extension, and renovation itself. This type of intervention impacts pre-existing constructions.

•
Construction-refers to new works and incorporates manufacturing and construction itself. This type of intervention is differentiated from the previous by impacting aspects of its location and soil occupation.
Once again, the CSC was expanded, this time with two new columns representing types of intervention. Each criterion was analyzed again and linked to one, or both, of the interventions.

Linking of the Criteria to Types of Public Construction
As shown in Table 8, the third aspect considered in the present study that differentiates one construction from another is its type: The infrastructure of electric energy-electric energy generation and distribution work and telecommunications [67,68]; • Industrial infrastructure-power stations, factories, refineries [68].
As per the previous items, the CSC was increased with six new columns representing the types of construction and the linkage to all the criteria.
After the linkages, the Sustainability Assessment Spreadsheet (SAS) was obtained, Table S2, which will serve for assessment activities.

Development of the Logical Structure of the Framework for Sustainability Assessment of Public Works and Its Operationalization
The feasibility of the use of the framework is strongly influenced by the demanded procedures and the mechanisms of operationalization. Figure 5 presents the proposed framework.
Procedures for assessment of the degree of sustainability in public works were identified, and a logical structure of procedural relationships was defined. As a result, a framework formed of six stages that group logically related procedures was obtained. To facilitate the use of the framework, it was decided to develop a computerized instrument through an electronic spreadsheet containing logical filters and formulae.
SAS presented in Table S2 acts as the basis for the sustainability assessment. In stages 1-3, evaluators use spreadsheets that automate the framework and inform project characteristics, automatically reducing the breadth of SAS criteria. Only the criteria that have specificities and characteristics similar to those of the work being evaluated remain.
The result obtained is still a generic list, and the assessor, already equipped with knowledge of all the details of the project, begins stage 4 by selecting the criteria that do not fit the characteristics of the construction, consolidating the Assessment Table (AT) with only the relevant criteria.
Despite the framework automatically establishing weights equal to 1 for all the AT criteria, in stage 5, the assessor may value or devalue a given criterion in relation to the others. For example, to increase the weight of a criterion by 20%, simply substitute the value corresponding to 1 for 1.2 and, to decrease by 10%, alter the weight to 0.9. If the assessor does not possess benchmarks or the experience to change said values, the use of those generated by the system is recommended.
All sustainability assessment tools establish a weighting system for their criteria; however, there is no consensus as to which approach or method to use [64]. The weights may be considered by groups of criteria or individually and may vary according to the client, investor, or impacted communities; they are inherent to the systems themselves, and when not specified, they receive equal values [47,64], as suggested in the present study.
In stage 6, the assessor scores the criteria contained in the AT. The score has a subjective component, in which each assessor shall make considerations taking into account their experience and individual interests. As a suggestion, Table 10 presents parameters based on the authors' experience.

Score Orientation for Scoring 0
The criterion is compatible with the venture but was not considered during phases of project/bidding, execution and use. 0 to 4 It was superficially considered in project/bidding and poorly executed.

to 7
It was planned, projected and implanted, but without much detail or depth to the theme.

Score Orientation for Scoring 0
The criterion is compatible with the venture but was not considered during phases of project/bidding, execution and use.

to 4
It was superficially considered in project/bidding and poorly executed.

to 7
It was planned, projected and implanted, but without much detail or depth to the theme.

to 9
Well planned, projected, and implanted, with detail and depth, but without the use of specialized professionals or companies.

to 10
Very well planned, projected and implanted and fully maintained, using specialized professionals and companies with a referential standard of excellence (Benchmark) The sustainability index of the work is automatically obtained after all criteria have been scored and their respective weights defined. The values are calculated and converted to a scale of 0 to 100, as follows: In Equation (1), Si is the value of the sustainability index of a work obtained by the weighted average of the CS scores (criteria score) attributed to each of the criteria and their respective CW weights (criteria weight). To convert the results to the 0-100 scale, the numerator was multiplied by 100 and the denominator by the maximum Smáx value (maximum score) that can be applied to the criteria. Table 11 shows an example of calculating the Sustainability Index using Equation (1) and the information in AT. Table 11. Example of obtaining the sustainability index.

Criterion Objective Score Weight
Envision sporting, social and leisure facilities in occupied places 10 1 Anticipate adequate infrastructure for facilities in occupied places 8 2 Promote actions to mitigate social risks-health, safety, and disasters, among others 4 1 Establish an appropriate density of habitations for the occupied places 0 1 In comparison, the BREEAM, LEED, Green Star and SBTool sustainability tools frame their final results and analyses into classification levels in a variety of ways: In the present study, the value of Si (sustainability index) determines, on a scale from 0 to 100, the extent to which a project is aligned with the precepts of sustainability. To facilitate the analysis of public managers, in Table 12, we suggest the limits between the evaluation ranges considering the distribution practiced by the LEED, BREEAM, Green Star, and SBTool tools. To facilitate the analysis of public administrators and, following the model used by other assessment tools, the obtained results were framed into four levels, as per Table 12.

Interval
Level Description/Impact

0-39 Low
Unacceptable level for public works. Projects with negative environmental, social, and economic impact. The public agent must incorporate sustainable elements to improve the classification.

40-64 Medium
Acceptable level. Projects with a neutral impact to sustainability. Possesses a wide scope for the implantation of improvements. The public agent should incorporate elements to improve the classification.

65-84 High
Recommended level for public works. Projects with positive environmental, social, and economic impact. Constructions, at this level, should be the minimum goal for the public agent.

85-100 Very high
Advanced level. Projects with positive impacts on all aspects of sustainability, reaching the standard of excellence. Works of this level must have their solutions disclosed and awarded.
To facilitate and streamline the execution of the framework stages, a software was used for the elaboration of electronic spreadsheets (Excel) to create an electronic instrument consisting of four parts. The first part contains instructions, recommendations, and the steps to be followed to use the system. It is where the assessor defines the characteristics of the project (Figure 6). The second spreadsheet contains the AT, which is automatically generated after the definition of the characteristics of the project, and is where the assessment itself is given. On this spreadsheet, the assessor scores the criteria and alters their weights if necessary (Figure 7). The third spreadsheet presents the result of the assessment after all the scores have been given, indicating the sustainability index of the project (Figure 8). The fourth spreadsheet shows the SAS (Figure 9), based on the information in the framework.  To facilitate and streamline the execution of the framework stages, a software was used for the elaboration of electronic spreadsheets (Excel) to create an electronic instrument consisting of four parts. The first part contains instructions, recommendations, and the steps to be followed to use the system. It is where the assessor defines the characteristics of the project (Figure 6). The second spreadsheet contains the AT, which is automatically generated after the definition of the characteristics of the project, and is where the assessment itself is given. On this spreadsheet, the assessor scores the criteria and alters their weights if necessary (Figure 7). The third spreadsheet presents the result of the assessment after all the scores have been given, indicating the sustainability index of the project (Figure 8). The fourth spreadsheet shows the SAS (Figure 9), based on the information in the framework. The assessor defines the characteristics of the project using the first spreadsheet (Figure 6), indicating the phase of the work, the type of intervention, and the type(s) of construction, marking the corresponding cells with an "X".
These definitions are automatically applied in column A of the SAS (Figure 9) with the use of logical formulae from the functions of Microsoft Excel and result in the selection of criteria with the indicated characteristics.  Table S3.   The second spreadsheet (Figure 7) is automatically assembled from the SAS through the use of research and reference formulae listing the selected criteria and other information for assessment. At  : Table S4.   The second spreadsheet (Figure 7) is automatically assembled from the SAS through the use of research and reference formulae listing the selected criteria and other information for assessment. At  Table S5.   The second spreadsheet (Figure 7) is automatically assembled from the SAS through the use of research and reference formulae listing the selected criteria and other information for assessment. At The assessor defines the characteristics of the project using the first spreadsheet (Figure 6), indicating the phase of the work, the type of intervention, and the type(s) of construction, marking the corresponding cells with an "X".
These definitions are automatically applied in column A of the SAS (Figure 9) with the use of logical formulae from the functions of Microsoft Excel and result in the selection of criteria with the indicated characteristics.
The second spreadsheet (Figure 7) is automatically assembled from the SAS through the use of research and reference formulae listing the selected criteria and other information for assessment. At this time, the assessor manually selects the criteria that do not fit in with the characteristics of the project, marking the corresponding cells in column F (SCORE), and applying a filter to eliminate the selected lines, consolidating the AT.
In the next step, the assessor scores all the criteria, inserting values from zero to ten in the cells in column F (SCORE) of the second spreadsheet ( Figure 7). As the scores are being inserted for the criteria, the system automatically establishes, through logical formulae, standard weight equal to 1. At this time, the assessor may make individual alterations, manually modifying the pre-defined values in the corresponding cells in column G (WEIGHT).
The third spreadsheet (Figure 8) presents the final result of the assessment. The Si is obtained from the application of Equation (1) to the data contained in columns F (SCORE) and G (WEIGHT) of the spreadsheet in Figure 8. Framing occurs automatically through logical formulae of the Excel functions, linking possible Si results to the intervals in Table 4.
It should be highlighted that the assessor can only edit the cells in yellow or orange, all the others are protected against modifications, impeding the user from editing the formulae and losing the linkages.

Evaluation of the Framework
For the framework evaluation, we used a real case of a project in the bidding phase, because in addition to having a higher number of associated criteria (Table 9), it is the phase that allows public agents to incorporate more sustainable elements into the work. The project was selected at random, through consultation with the website of the Niterói city hall [69], using case bidding process no. 23/2018, whose objective is to benefit an area located in a low-income neighborhood with the construction of a social square, a playground for children, a football field with artificial grass, a grandstand, an outdoor gym for the elderly, and a building containing changing rooms and an office. The documents that are made available to all companies interested in participating in the bidding process include: (a) the main document with the body of the public notice containing the legal requirements and guidelines for proposers; (b) basic project containing the plans that guide the execution; (c) descriptive memorial detailing how the services are to be performed; (d) contractual draft; (e) budget spreadsheet containing quantities and prices of services; (f) physical-financial schedule; (g) calculation memory showing how the values of the services were obtained; (h) photographic report of the area before the interventions.
To evaluate the framework, professionals with experience in the inspection and execution of public works, including personnel with bidding processes, experience where sustainability assessments need to be carried out, and were invited to participate. Three civil engineers and an architect participated in the evaluation, along with four young professionals with at least one year of experience in this segment of work. The professionals had no contact with each other. The procedure adopted was: Step 1-We provide all the documents of the notice and advise them to read the material in detail and understand all aspects of the construction; Step 2-We briefly explain the concepts of sustainability of works, the operation of the spreadsheets detailed in item 3 (Figures 6-9) of the framework, and where to look for the information to be able to evaluate correctly. We present the content with the concern that none of the participants would suffer any kind of influence and would not harm the evaluation process; Step 3-We provide the framework's electronic spreadsheets to individually assess each of the criteria selected for the work, considering the phase, type of intervention and type of construction, finding the work sustainability index ( Figure 5); Step 4-We collect the spreadsheets containing the evaluations and provide a data collection instrument (Questionnaire S1, Table 13) to assess whether the framework has achieved the objectives of each item in the questionnaire, according to a five-point Likert scale, varying from "Very low" to "Very high". We made available in the questionnaire a field for the participants, if necessary, to make additional comments about the research items; Step 5-We collect the research results and organize them in tables to be analyzed and presented. Table 13. Structure of the framework evaluation form.

General aspects
Importance of using sustainability assessment tools, such as that presented, in public works.
The propensity of the assessor to make modifications in the conception, project, or execution of a given work, after using the framework.
The utility of continual use of the framework, by public administrators, to improve the sustainability of public work.
Usage capacity of the system to enhance the expansion and promotion of knowledge on the sustainability of works.
Specific aspects related to the framework Ease of use.
Ease of understanding and scoring the criteria.
The relevance of the criteria to the measurement of sustainability of works.
The capacity of the system to indicate the correct level of sustainability of a project.
The necessity for previous experience in sustainability assessment.
The necessity for specific knowledge, such as that related to the sustainability of works.
Potential in the use of the obtained result to reflect the level of sustainability of a project.
Specific aspects related to the instructions for the use of the framework Clarity of information.

Completeness of information.
For the definition of the questionnaire items, we interviewed four professionals, of which two also participated in the evaluation of the tool. The other two were selected because we found it important to have a view of those who were not from the area, but who had already participated in the development of assessment tools. Our question was: What are the main items that should be considered to assess the framework's ability to be used in practice? After eliminating redundancies, we developed a version of the questionnaire that was submitted to all four professionals and which was considered appropriate by all. Figures 10-13 synthesizes the results of the specialist's evaluation.
The necessity for specific knowledge, such as that related to the sustainability of works. Potential in the use of the obtained result to reflect the level of sustainability of a project. Specific aspects related to the instructions for the use of the framework Clarity of information.
Completeness of information.

General aspects
Usage capacity of the system to enhance the expansion and promotion of knowledge on sustainability of works Utility of continual use of the framework, by public administrators, to improve the sustainability of public work Propensity of the assessor to make modifications in the conception, project or execution of a given work, after using the framework Importance of using sustainability assessment tools, such as that presented, in public works  We considered, as strengths of the system, the items in which the sum of the percentage of the "Very high" and "High" evaluations of the item was equal to 100% and, as weaknesses, when it was equal to or less than the sum of the other evaluations.
Thus, analyzing the graphs in Figure 10, we can see that among the aspects evaluated in the questionnaire in Table 13, which were considered as strengths, there are two items among the General  We considered, as strengths of the system, the items in which the sum of the percentage of the "Very high" and "High" evaluations of the item was equal to 100% and, as weaknesses, when it was equal to or less than the sum of the other evaluations.
Thus, analyzing the graphs in Figure 10, we can see that among the aspects evaluated in the questionnaire in Table 13, which were considered as strengths, there are two items among the General Two items related to General Aspects deserve attention because they signaled an improvement margin despite not being considered as weaknesses. They are: "Importance of using sustainability assessment tools, such as that presented, in public works" and "Usage capacity of the system to enhance the expansion and promotion of knowledge on the sustainability of works".
Among the aspects considered as weaknesses, all are related to the Specific aspects related to the framework: "Ease of understanding and scoring the criteria", "The necessity for previous experience in sustainability assessment" and "The necessity for specific knowledge, such as that related to the sustainability of works".
It has been found that there is a necessity to improve the system, identifying and adding information to the criteria that make using the framework more difficult. The results also demonstrate alignment to that proposed in the present study, which is the continual use of this tool, which would increase and stimulate the search for knowledge directing public works towards sustainability.

Conclusions
The need for enhancing the sustainability of the built environment is evident in the literature [36]. Focus has mostly been on the sustainable assessment of private works, with few studies directed at the assessment of sustainability in public works. To cover this apparent gap, the present study proposed, through an extensive and detailed bibliographic research, a flexible and comprehensive framework for the assessment of public works. A practical tool that functions as a checklist applicable to all types and sizes of public works, at any stage of the lifecycle, was developed. The tool developed incorporates and synthesizes logically related concepts, guidelines, and criteria proposed by various researchers and institutions working with the theme, to be used in all types of work. The sustainability assessment framework achieves two principal objectives, namely the comprehensiveness of the assessment undertaken, and secondly, maintaining the practicality of the proposed approach. Utility of continual use of the framework, by public administrators, to improve the sustainability of public work Propensity of the assessor to make modifications after using the framework

Strenghts and weaknesses
Strenghts Weaknesses Figure 13. Results of the framework evaluation: Strengths and weakness.
We considered, as strengths of the system, the items in which the sum of the percentage of the "Very high" and "High" evaluations of the item was equal to 100% and, as weaknesses, when it was equal to or less than the sum of the other evaluations.
Thus, analyzing the graphs in Figure 10, we can see that among the aspects evaluated in the questionnaire in Table 13, which were considered as strengths, there are two items among the General Aspects: "The propensity of the assessor to make modifications in the conception, project or execution of a given work, after using the framework" and "The utility of continual use of the framework, by public administrators"; four are among the Specific aspects related to framework: "Ease of use", "The relevance of the criteria to the measurement of sustainability of works", "The capacity of the system to indicate the correct level of sustainability of a project" and "Potential in the use of the obtained result to reflect the level of sustainability of a project" and; all Specific aspects items related to the instructions for the use of the framework: "Clarity of information" and "Completeness of information".
Two items related to General Aspects deserve attention because they signaled an improvement margin despite not being considered as weaknesses. They are: "Importance of using sustainability assessment tools, such as that presented, in public works" and "Usage capacity of the system to enhance the expansion and promotion of knowledge on the sustainability of works".
Among the aspects considered as weaknesses, all are related to the Specific aspects related to the framework: "Ease of understanding and scoring the criteria", "The necessity for previous experience in sustainability assessment" and "The necessity for specific knowledge, such as that related to the sustainability of works".
It has been found that there is a necessity to improve the system, identifying and adding information to the criteria that make using the framework more difficult. The results also demonstrate alignment to that proposed in the present study, which is the continual use of this tool, which would increase and stimulate the search for knowledge directing public works towards sustainability.

Conclusions
The need for enhancing the sustainability of the built environment is evident in the literature [36]. Focus has mostly been on the sustainable assessment of private works, with few studies directed at the assessment of sustainability in public works. To cover this apparent gap, the present study proposed, through an extensive and detailed bibliographic research, a flexible and comprehensive framework for the assessment of public works. A practical tool that functions as a checklist applicable to all types and sizes of public works, at any stage of the lifecycle, was developed. The tool developed incorporates and synthesizes logically related concepts, guidelines, and criteria proposed by various researchers and institutions working with the theme, to be used in all types of work. The sustainability assessment framework achieves two principal objectives, namely the comprehensiveness of the assessment undertaken, and secondly, maintaining the practicality of the proposed approach.
The framework was validated by nine specialists that responded to a questionnaire and validated the system. They considered the framework as a practical tool, indicating its high level of easiness, and that the final score obtained from the assessment represents the true level of sustainability of the project. Historical data containing sustainability levels of public works, collected from the use of the framework, serve as a reference for administrators to direct their resources to achieve more sustainable works.
This research has the typical limitations of research using the literature review to support the results. Even though a comprehensive and detailed bibliographic search has been carried out, there is always the possibility that an important article has not been considered. Moreover, despite demonstrating the final sustainability level of work, the results produced by the proposed framework are as good as the level of detail of information incorporated. For example, if the environmental, social, and economic criteria are being met, what percentage, and which category would be more developed is something that the tool cannot demonstrate at this stage.
To improve the results of this research, we have some suggestions. Although the framework considers some criteria that assess subjective and intangible socioeconomic aspects, the vast majority is formed by measurable criteria for tangible elements/aspects, since in the researched literature, this is the main focus. Thus, studies that made it possible to incorporate new criteria with this characteristic would contribute to increasing the structure's ability to consider socio-cultural implications. In addition, other structures such as WELL and ASHRAE standards could also be analyzed in order to identify new relevant indicators and, with this, expand the scope of the framework. The organization of co-creative workshops with industry, public sector professionals, academics, and engineers, would also collaborate to standardize the scores and weight of the criteria.
An important aspect being considered is the possibility of selecting only the most suitable criteria for the project to be analyzed and of establishing different weights for the criteria, which makes it difficult to compare the scores between projects. Assessments usually have, to a lesser or greater degree, a subjective character, as they depend on the assessor's interpretation of what is being assessed. There is no way to eliminate possible deviations in the assessment. However, for the same public agency, the conditions under which the projects are developed do not vary significantly, which means that the weights attributed to the same work should not undergo great variations. We also believe that the continued use of this tool by the public sector will be able to establish evaluation standards that allow the comparison between different works and that, over time, there will be an evolution capable of incorporating more and more sustainable elements and concepts to the works. Regarding the comparison of assessments carried out by public bodies in different cities, they may vary, which may also mean that the specifics of the project and the conditions under which the projects will be developed are being considered.
We hope that this study can contribute to the improvement of sustainability in the construction sector, in particular those related to public works, and to stimulate the development of new tools that operationalize, objectively and comprehensively, the assessment of sustainability.