Implementation of Multilateral Environmental Agreements: Rationale and Design of the Environmental Conventions Index

Abstract

Global environmental conventions are created to address and resolve global environmental problems. Assessments of the achievement of specific environmental goals, however, indicate that there is room for progress and that improved collective action is required. Given the few existing studies that measure the implementation of international environmental law, it is important to expand existing analytical frameworks about international environmental agreements and their translation into national policies. This article explains the rationale and design of the Environmental Conventions Index, an implementation measurement tool we have developed at the Center for Governance and Sustainability at the University of Massachusetts Boston. The Index assesses the implementation of global environmental conventions in two clusters—conservation and pollution—showcasing the main trends for both countries and conventions. Using a mixed-method approach, we explain the development of this measurement tool as an instrument to inform policy changes at the national and global levels and present aggregate results of the analysis. As global environmental challenges in the conservation of natural resources and the persistence of pollutants become critical, assessment of countries’ performance on international environmental goals is essential. We offer an innovative method that provides academic analysis and policy input to improve implementation, and thus the effectiveness of these governance instruments in addressing global environmental challenges.

1. Introduction

The environment is the foundation of human life. Population growth and the quest for economic growth, however, have drastically transformed it, affecting the functioning of planetary systems in ways that threaten Earth’s resilience and adaptation capacity [1]. Humanity has crossed four of the nine planetary boundaries that scholars define as the “safe operating space” within which humans can function [2]. Pressures on climate, biodiversity, chemicals, and land have stressed the stability of earth systems. These issues are of a global character and significance. Since 1972, when 113 governments convened for the first United Nations Conference on the Human Environment, the Stockholm Conference, the international community has been working collectively on environmental action [3]. The system of global environmental governance comprises a range of laws, institutions, international organizations, international agreements, government agencies, local initiatives, and decision-making processes.

Global environmental conventions are a critical part of this system of governance. Also known as multilateral environmental agreements (MEAs), environmental conventions are agreements among multiple governments “intended as legally binding with a primary stated purpose of preventing or managing human impacts on natural resources” [4,5,6]. These international legal instruments address global environmental problems, raise awareness, gather information, and promote coordinated action towards the resolution of environmental problems [7,8,9,10]. Scholars have studied these agreements from multiple perspectives, including the negotiation and treaty-making processes, their design and structure as international law instruments, the reasons behind countries’ decisions to join the agreements, the characteristics of the environmental problems they address, and the measurement of their effectiveness [11,12,13,14,15,16,17,18]. However, analysis of the extent to which countries fulfill their obligations and why remains largely inadequate. Without an assessment of performance on the core treaty obligations, it is difficult to systematically assess the effectiveness of the conventions, i.e., the extent to which they address the problem they were created to address.

Despite the existence of conventions across a range of environmental concerns, the global environment continues to degrade, and planetary boundaries are being pushed to new limits. The 2020 Red List published by the International Union for Conservation of Nature (IUCN) called attention to the global extinction crisis [19]. Ecosystems and biota across the world are threatened by the adverse effects of economic activities and climate change. For example, an estimated 35% of wetlands have been lost since 1970 [20]. Similarly, in 2016, the second meeting of the UN Environment Assembly (UNEA) called attention to the effects of air pollution on health, estimating that “7 million people across the world die each year due to everyday exposure to poor air quality” and 7% of these deaths are caused directly by chemical pollution [21]. Such trends raise key questions about the institutional performance of the environmental conventions, including how national policies based on international environmental commitments contribute to solving global environmental problems and what factors determine countries’ performance on the achievement of international environmental goals [22,23,24,25,26,27,28,29,30,31,32]. In other words, are the conventions being implemented by states, and are they improving the global environment? Answering these questions makes it possible to determine the role of conventions as instruments of global governance while offering an analytical and empirical framework for analyzing their implementation and developing policy responses. In this article, therefore, we take the ratification of conventions as a given and tackle the implementation puzzle [5,33,34,35].

Implementation and effectiveness of global environmental conventions are at the core of the academic discussions on policy responses to protect the environment. Implementation refers to the process where countries establish national policies that reflect global environmental commitments, and effectiveness describes whether and how these policies have a direct role in the resolution of global environmental problems. International legal scholars tend to view implementation and effectiveness as the norm. As Henkin first suggested in his 1979 book How Nations Behave: Law and Foreign Policy, “almost all nations observe almost all principles of international law and almost all of their obligations almost all of the time” [36]. Different studies have approached the question of implementation through different methods, including qualitative country-based analyses, issue-based analyses of effectiveness, and regime-based behavioral analysis of implementation [23,29,30,32].

However, most of these studies evidence critical gaps in measurement and analysis. Factors such as the structure and clarity of obligations defined by the conventions, as well as lack of agreement about what constitutes compliance and implementation, are some of the variables that make it difficult to measure how MEAs are translated into national policies and to what extent state parties fulfill their obligations. To overcome this challenge, scholars must first understand how environmental conventions are being implemented at the national level. Assessing the national implementation of conventions and explaining why countries perform differently is essential to understanding the dynamics and effectiveness of global conventions in addressing environmental challenges. New empirical tools and new analytical frameworks are therefore critical.

This article presents an assessment of the implementation of global environmental conventions through a new measurement tool for implementation—the Environmental Conventions Index (ECI). The analysis seeks to accomplish three goals: (1) characterize the need for a measurement of implementation of MEAs, (2) present the methodology of the ECI, and (3) present the database constructed and the uses it can have both for scholarship and policy. To this end, the article describes the existing general framework of the role and functions of international environmental agreements. It also presents the methodology of the ECI. The analytical work that undergirds this study offers systematic, comparative, time-series data on the performance of all parties to the set of agreements in the Index and allows for analysis by country and by convention. It presents new insights on the importance of data and means for implementation and creates connections among global governance processes essential for sustainable development. This work is in progress and this article invites offers for its improvement and expansion.

2. Global Environmental Agreements and the Implementation Puzzle

Global environmental conventions are the main legal instruments to protect the environment [37,38]. They are intergovernmental, legally binding, and designed with the purpose of managing environmental resources and preventing their depletion [5]. Even though they have been part of the system of international environmental law since the nineteenth century, the past five decades have seen the creation of most of the multilateral environmental agreements, with more than seventy percent of these agreements having been adopted after 1970 [4]. Their membership has also expanded (see Figure 1) and they have opened up to participation from non-state actors [39,40]. Currently, the number of multilateral environmental agreements addressing transboundary environmental problems such as climate change, biodiversity loss, and chemical pollution has reached more than 500 [4]. Ten of the agreements dealing with issues of conservation and pollution are truly global in both membership (universal) and scope (issues of global reach) (see

Table 1. Global Environmental Conventions.

		Start Year	Parties (No.)
Conservation	UN Convention to Combat Desertification (UNCCD)	1994	197
	Convention on Biological Diversity (CBD)	1992	196
	Convention on International Wetlands (Ramsar Convention)	1971	171
	Convention on International Trade in Endangered Species (CITES)	1973	183
	Convention on the Conservation of Migratory Species (CMS)	1979	130
Pollution	Stockholm Convention on Persistent Organic Pollutants	2001	184
	Basel Convention on Transboundary Movements of Hazardous Wastes	1989	187
	Rotterdam Convention on Prior Informed Consent Procedure	1998	161
	UN Framework Convention on Climate Change (UNFCCC)	1992	197
	Vienna Convention and Montreal Protocol on the Ozone Layer	1987	198

Data to 16 July 2020. Sources: [44,45,46,47,48,49,50,51,52,53].

).

Global environmental agreements are the primary means for collective action. They “are rarely the end product, but instead, create the framework and the process that guide responses to the environmental problem in question” [41]. Environmental treaties express collective intentionality [42], bringing governments and actors together under a series of principles and obligations that enhance their capacity to respond to environmental problems [11,42]. They transform “intergovernmental bargaining into deliberative transnational problem-solving” [37] and are fundamental components of the system of global environmental governance [43]. As international organizations, conventions (or rather, their secretariats) support and manage operations, jointly elaborate and produce norms, and coordinate the efforts of countries that agree to work together to address a common issue [15,26]. By opening spaces for interactions among countries, agreements also promote reciprocity and require countries to establish the national policies necessary to address environmental problems. All these functions, outcomes, and substantive measures ultimately seek to improve the state of the environment.

Beyond their design, governance structures, and the issues they address, existing analyses of international law instruments in general, and global environmental conventions in particular, focus on their institutional performance defined in terms of compliance, implementation, and effectiveness. Compliance refers to conformance to expectations, the adherence of state parties to the agreement’s obligations [31,54,55,56,57,58,59,60,61,62,63]. Implementation refers to the adoption of domestic regulations to fulfill international commitments [28,30,31,64,65,66,67]. Effectiveness means fulfilling the goals of the agreement and resolving the environmental problem in question [29,64,65,68,69,70,71]. Particularly, in the context of increasing environmental challenges, the successful implementation of global environmental conventions through goal setting, metrics development, data collection, and resource mobilization is fundamental to coordinating, integrating, and systematizing efforts to protect the environment and promote sustainability.

Implementation is, however, a complex process. While some scholars limit implementation to the adherence of state parties to the obligations of an agreement [28,54,57,72], it is necessary to consider the adoption and implementation of measures that change states’ behavior. Victor, Raustiala, and Skolnikoff [30] are even more specific, referring to national implementation as the creation of new programs and the promulgation and enforcement of laws and standards. What all these approaches have in common is a strong behavioral component that goes beyond actual conformance with specific legal obligations [8,31]. This study defines implementation as the extent to which governments translate their international commitments into national policies, actions, strategies, and enforcement tools such as legislation, institutional appointments, data collection and analysis, technical and financial measures [22,29,30,73,74].

In the case of environmental law, evaluations of the implementation of global environmental conventions differ [28,54]. Results do not share a common definition for measurement standards and do not offer a systematic empirical assessment that demonstrates results across the complete MEAs membership and that establishes common standards across the various conventions. Even within the same environmental issues, treaties have different conceptions of what is acceptable behavior by state parties [28,75]. For example, within the pollution cluster, while for the Basel Convention obligations are based on changes at the regulatory and legislative level, for the Stockholm Convention parties are expected to work on more technical aspects to control the presence of Persistent Organic Pollutants [76,77]. In the biodiversity cluster, obligations also differ. In the CBD, for example, objectives are directly related to the conservation of biological diversity, through its sustainable use and the fair and equitable sharing of the benefits arising out from the utilization of genetic resources [78], while most of the policy changes expected from Convention on International Trade in Endangered Species (CITES) focus on the relationship between trade and environmental policies [79].

Difficulties with the vagueness of legal obligations and with countries’ reporting reduce the ability to determine the extent to which countries are fulfilling their obligations and translating them into national policies. A common understanding of terminology simply does not exist, and key concepts for assessing the effectiveness of the measures taken and their impact on the state of the environment are missing [80,81,82]. Furthermore, in some cases, existing evidence shows that states do not carry out these changes and behave contrary to expectations [54,64,83,84]. Country-based academic studies such as by Brown-Weiss and Jacobson [24], issue-based studies [43,71,85,86,87] and policy reports and assessments conducted by the conventions, such as the Global Wetlands Outlook [20], evidence that obligations are not being implemented to the extent envisioned by the agreements.

Analysis of the literature on global environmental conventions, therefore, identifies three gaps—conceptual, empirical, and methodological. Conceptually, most studies concentrate on effectiveness, without systematically explaining first how the conventions are translated into national environmental policies. There is thus a gap regarding the causal connection between conventions as instruments of governance and the solution of environmental challenges. In other words, assessing individual countries’ implementation of environmental conventions and explaining why countries perform differently is essential to understanding the effectiveness of international environmental governance in addressing global challenges. Secondly, studies rarely offer empirical standardized metrics that would allow for comparison across conventions and across countries. Finally, there is a fundamental methodological gap. Most analyses only offer information for a limited number of countries, whereas other studies focus on a specific agreement without offering national data that allow for comparison among countries. In this context, studies need to be recalibrated to determine how conventions are being implemented across all member states over a period of time, to understand why countries perform differently. The Environmental Conventions Index addresses all three gaps by offering rigorous data and analysis of how conventions are being implemented across all member states over time. The unique methodology can be applied across different agreements on different environmental issues and inform the understanding of implementation as well as policies and decision-making, the strengthening of compliance, and eventually the analysis of MEA effectiveness.

3. Measuring Implementation: The Environmental Conventions Index

A key recent trend in global governance is the use of codification, assessment, measurement of results, quantification, and governance indicators to promote performance evaluation, and accountability [88,89]. Using indicators to inform decision-making, if designed correctly, can offer consistent, efficient, transparent, and impartial metrics to gather information, summarize complex realities, exercise judgment, and support actors in designing and executing policies. Furthermore, scholarly attempts to apply strategic management to international treaties highlight the need to monitor compliance and implementation and evaluate treaty performance through the use of different methods, including reviews and assessment, evaluations, and the use of metrics and indicators [74]. The central purpose behind these metrics is to establish linkages between treaty objectives and outcomes [90].

In the case of global environmental governance, different measures exist to study the implementation and effectiveness of environmental agreements and regimes [91,92]. Qualitative methods seem to dominate the field and there is a clear need for a scoring-based standardized metric that provides a quantitative basis for description, assessment, and explanation [93]. This type of social science approach will make it possible to evaluate, from a positivist perspective, the existing linkages between environmental conventions and changes in policy behaviors, and the extent to which implementation contributes to the improvement of environmental quality. Empirical assessment can provide an accurate description of these linkages and enhance explanatory power. An empirical approach to global environmental governance would improve the understanding of different international instruments used to protect the environment. Indices such as the Environmental Performance Index developed by Yale University [94], and the Environmental Democracy Index developed by the World Resources Institute [95], address ecosystem health and environmental vitality, and environmental rights, respectively. Other indices such as the Human Development Index (HDI), the Ibrahim Index of African Governance (IIAG), or the Global Multidimensional Poverty Index evidence the pertinence of governance indicators to assess performance and influence policy instruments [96,97,98,99,100].

In the case of global environmental conventions, the extent to which countries have established the laws and regulations to fulfill their obligations has yet to be systematically measured under common standards and parameters. This gap persists despite the fact that these international law instruments are broadly acknowledged to be central to environmental protection, sustainable development, and effective global environmental governance.

3.1. The Methodology of the Environmental Conventions Index

To address the need for an empirical indicator that measures the implementation of global environmental conventions, this study presents the design of the Environmental Conventions Index, which assesses the actions signatory countries have taken to fulfill their international commitments. Using a mixed-methods score-based approach that relies exclusively on implementation data submitted by the state parties in a consistent and comparable format, the ECI provides the basis for a semi-quantitative assessment of implementation that allows for comparability and informs analyses on information requirements in international agreements. The process of index construction is based on an eight-step protocol (see

Table 2. Summary of research protocol for the Environmental Conventions Index (ECI).

A Seven-Step Protocol for the Construction of the Index Ensures Analytical Rigor
(1) Identify units of analysis selecting conventions and characterizing their membership.
(2) Collect the national reports submitted by member states to the conventions as the main formal source of information to evaluate implementation and construct the index.
(3) Use the conventions texts and national reports to identify implementation indicators for each convention.
(4) Use the questions in the national reports to identify variables and the options for reported data and group them under the identified indicators.
(5) Create scoring scales for each variable. To this end, each answer to each question under a specific convention is evaluated using an ordinal scale from 0 to 5, with 5 being the highest level of implementation. A score of 0 is given when no information is provided.
(6) Code data from national reports to build a dataset that includes the reported data submitted by each country to each convention for all selected variables. Two researchers conduct the coding process to ensure inter-coder reliability.
(7) Score reported data and rank countries both on whether they have submitted reports according to their obligations and whether their reports demonstrate progress toward the aims of the conventions.
(8) Construct the index using the scores for each variable to assess the indicators. Neither the indicators nor the variables are weighted.

) designed to allow the identification of patterns and for the construction of a database that collects and provides information to countries and conventions to track progress and ensure accountability (see Figure 2).

3.1.1. Units of Analysis

Six global environmental conventions within two thematic clusters—conservation and pollution—are included in the ECI (see

Table 3. Global environmental conventions included in the ECI.

Global Environmental Conventions Included in the ECI

Ramsar Convention on Wetlands (1971) World Heritage Convention (WHC) (1972) Convention on International Trade in Endangered Species of Wild Fauna and Flora (CITES) (1973) Convention on the Control of Transboundary Movements of Hazardous Wastes and their Disposal (Basel Convention) (1989) Agreement on the Conservation of African-Eurasian Migratory Waterbirds (AEWA) (1995) Convention on Persistent Organic Pollutants (Stockholm Convention) (2001)

). Out of the six agreements that are currently part of the ECI, this article engages with four to facilitate the analysis and description of the Index: two in the conservation cluster (Ramsar Convention and CITES) and two in the pollution cluster (Basel and Stockholm Conventions). These agreements were selected based on three criteria:

• Universal membership: Any country could become a party to the convention upon accession and ratification.
• Global scope: The issues the conventions address are of global character and their impact is at the global level.
• Structured national reports: Availability of reports based on questionnaires that collect national-level data in a systematic manner, which allows for standardization and comparison.

In terms of state parties—the main unit of analysis—membership to the conventions is open to the 193 member states of the UN [101], four other states with a different UN status (the Cook Islands, Niue, Palestine, and the Holy See), and one regional organization: the European Union. The Holy See, Palestine, and the EU have received a standing invitation to participate as observers in the sessions and the work of the UN General Assembly, to which they maintain permanent observer missions. The Cook Islands and Niue are states in free association to New Zealand, but they behave as sovereign states in international law. The total 198 units of analysis are distributed across different categories based on regions and levels of development (see

Table 4. Countries included in this study, by type of country and region.

	Countries	Percentage
UN Membership
UN permanent Members	193	97%
Observers	4	2%
Regional Organization	1	1%
Total	198	100%
Level of development
Developed	50	25%
Developing	148	75%
Total	198	100%
Regions
Africa	54	27%
Americas	35	18%
Asia	48	24%
Europe	44	22%
Oceania	16	8%
Regional Organization	1	1%
Total	198	100%

). Both classifications are based on those established by the UN Statistical Division [102]. Additional classification can be made based on regional groupings established by the conventions, or by other characteristics such as income, geography, development, or membership in international groups.

3.1.2. Data Sources

National reports submitted to each of the convention secretariats by state parties constitute the data sources for the ECI. These reports as based on the requirements established by each agreement, including the type of information to be collected, and the periodicity of the reports’ submission. Usually, national reports contain two types of information. On the one hand, they focus on the legal, administrative, and policy measures that state parties adopt or intend to adopt to implement each agreement. They also collect scientific data on the state of the environmental problem addressed by each convention. The Basel Convention, for example, requires information on the transboundary movement of hazardous wastes, on the types and extent of measures that parties have undertaken to implement their obligations, and on the effects of hazardous waste generation, transport, and disposal. CITES requires two types of reports—an annual report on the trade-in regulated species with statistical information about type and volume, and a biannual report, referred to as the implementation report, on the laws, regulations, policies, and administrative procedures countries are carrying out. The data from the CITES annual report are included in the trade volumes database by the secretariat and non-compliance with the annual report obligation over three years carries trade sanctions [103,104]. For the assessment of the implementation of CITES, we use the biannual, implementation report.

National reports across the conventions include information on policy measures and their effectiveness, using relevant statistical data [105] (see

Table 5. Reporting requirements for the conventions included in this study.

Convention	Reporting Requirements
Basel Convention	According to Article 13 of the convention, reports should include:
	Information on focal points Information on transboundary movement of hazardous wastes and other wastes Measures adopted to implement the convention Statistics on the effects of hazardous waste generation, transportation, and disposal Information on accidents, disposal options, and technologies to manage hazardous wastes Information on other agreements for hazardous waste management
Stockholm Convention	According to Article 15 of the convention, reports should include:
	Measures adopted to implement the convention, and their effectiveness Statistical data on the production, import, and export of the chemicals included in the annexes to the convention
Ramsar Convention	After the convention entered into force, the second meeting of the COP recommended the submission of national reports and requested the Bureau of the Convention (then equivalent to the secretariat) to establish the requirements for this process. Reports are submitted for each COP based on a format established by the standing committee. Questions are based on the convention’s strategic plan and are designed to measure progress on key indicators and considering continuity to permit time-series analyses.
CITES	Article VIII para (7) establishes two types of reports for the convention that should be transmitted to the Secretariat:
	An annual report containing a summary of records of trade in specimens regulated by the convention, including detailed information as indicated in Art. VIII para (6) A biannual report on legislative, regulatory, and administrative measures taken to enforce the provisions of the present convention. This is termed the implementation report.

Source: Art. 13 [76], Art. 15 [77], Art. VIII [79,106,107,108,109].

). Parties submit the data in set reporting periods, or cycles (see

Table 6. Reporting cycles structure and availability.

	Structure of Reporting Cycles	Reporting Cycles Available
Basel Convention	Reporting cycles are annual. An electronic reporting system was implemented in 2012.	2001–2017
Stockholm Convention	Reporting cycles are defined by the COP. Four cycles have been established to date.	2002–2006, 2006–2010, 2010–2014 and 2014–2018
Ramsar Convention	Reports are submitted for each COP, which takes place every three to four years.	2005, 2008, 2012, 2015 and 2018
CITES	Reports are biennial	Seven reporting periods: 2003–2004, 2005–2006, 2007–2008, 2009–2010, 2011–2012, 2013–2014, 2015–2017

). The convention secretariats collect the reports, except in the case of the Ramsar Convention, for which reports are officially submitted to the Conference of the Parties (COP). Only the Basel Convention (since 2012) and the Stockholm Convention have electronic reporting systems and one is under development for CITES. Convention secretariats have made reports submitted since 2001 available online.

3.1.3. Framework

Measuring the extent to which the conventions have been implemented determines the breadth of countries’ progress in putting in place regulations, institutions, and strategies needed to achieve global environmental goals, and in addressing challenges to fulfilling their obligations. It also requires the identification and structuring of a series of complex standardized metrics that are part of a multi-state analytical protocol. Methodologically, the design of the ECI aims at assuring replicability across environmental conventions, as well as other international law instruments. The metrics behind the ECI are therefore grounded in two foundational concepts: (1) the key policy behavioral changes that environmental conventions establish as obligations for their state parties [11], and (2) the national reports that state parties submit to the convention secretariats, a legal obligation under each agreement.

Policy changes include a broad spectrum of activities including putting in place regulations, institutions, and strategies needed to achieve global environmental conventions. In this context, the ECI evaluates countries on five indicators that are present consistently in national reports:

• Information: Policies to conduct scientific assessments, measurement, and evaluations associated with the activities connected to each convention; submission of reports to the conventions’ executive bodies; and the establishment and maintenance of databases and records required for the implementation and operation of each convention.
• Management: Designation or creation of administrative bodies and focal points to manage the implementation and general functioning of each convention, the linkages with the conventions’ executive bodies, and the definition of strategic frameworks for the operation of each convention at the national level.
• Regulation: Legislative and policy measures that each state party has to implement according to the framework of each convention.
• Technical: Technical measures and procedures to address or manage the environmental problems associated with each environmental convention.
• Financial: Payment of dues and assistance, and other financial responsibilities by state parties.

The analysis of national reports is the foundation for the construction of the index variables and indicators. Using the structure of each of the national reports, we used strict criteria to identify the variables that measure implementation and indicate how they are related to the obligations established by the agreements. These variables reflect countries’ commitments to put in place policy changes associated with the implementation of each agreement. Variables are then grouped in the different indicators, matching the different types of policy changes needed at the national level to address global environmental challenges.

Each indicator is made up of several variables associated with a specific policy change. Each variable represents specific measures at the national level to develop the required policy instruments to address global environmental challenges. Figure 3 and

Table 7. Number of questions and indicators by reporting cycle.

			Variables per Indicator (Number and Percentage)
	Questions	Total Number of Variables	Information	Management	Regulation	Technical	Financial
Basel Convention
2001–2011	30	15	-	2 (13%)	2 (13%)	11 (74%)	-
2012–2017	30	15	-	2 (13%)	2 (13%)	11 (74%)	-
Stockholm Convention
2002–2006	67	48	4 (8%)	15 (31%)	14 (29%)	12 (25%)	3 (6%)
2006–2010	65	59	4 (7%)	26 (44%)	8 (14%)	18 (31%)	3 (5%)
2010–2014	72	56	4 (7%)	20 (36%)	9 (16%)	20 (36%)	3 (5%)
2014–2018	71	60	5 (8%)	22 (37%)	8 (13%)	22 (37%)	3 (5%)
Ramsar Convention
2005	581	45	8 (18%)	24 (53%)	1 (2%)	11 (24%)	1 (2%)
2008	69	36	8 (22%)	16 (44%)	1 (3%)	10 (28%)	1 (3%)
2012	83	48	14 (29%)	20 (42%)	2 (4%)	11 (23%)	1 (2%)
2015	67	43	13 (30%)	18 (42%)	2 (5%)	9 (21%)	1 (2%)
2018	126	83	16 (19%)	48 (58%)	2 (2%)	16 (19%)	1 (1%)
CITES
2003–2014	120	46	11 (24%)	14 (30%)	10 (22%)	10 (22%)	1 (2%)

present the number of variables under each indicator for each of the reporting cycles for each convention. Subsequent reporting cycles and variables will be added as we develop the analytical capacity at the Center for Governance and Sustainability to correspond to the requirements of time and resources necessary. Supplementary Materials presents the variables under each indicator for each of the agreements in its latest reporting cycle.

3.1.4. Index Construction

Designing and constructing the Environmental Conventions Index involved a multi-stage process to obtain the required information to assess implementation, taking into account the complexity of this process and assuring replicability across environmental conventions, as well as for other mechanisms of international law.

Once the indicators and variables have been identified, the next step in the index construction is the definition of the scoring scales. Scoring scales are based on the options for reported data that the national reports define for each variable. Each variable is then accompanied by a guidance note that includes the question to which it corresponds in the national report for the current and previous reporting cycles (if available). Logical and justifiable scales are in place for each score. Scales rank reported data options, and give them a score from 0 to 5, with 1 meaning activities not being implemented, and 5 meaning full implementation. A value of zero indicates a missing value or no response. When the option for reported data is not applicable, that specific variable is not scored and is not included in the index score calculation.

Table 8. Summary of variables options for reported data and scores.

Convention	Scores for Options for Reported Data
	5	4	3	2	1	0	Blank
Basel Convention	Implemented Exist Used		In preparation		Not implemented Does not exist Not used No information available	No response	Not applicable
Stockholm Convention	Yes		In progress	Being planned	No No information available
Ramsar Convention	Yes		Partly In some cases In progress In some sites	Being planned	No No information available
CITES	Yes		Partly		No No information available

provides a summary of the main options for reported data under each convention and its correspondence in the scoring scale.

Using scoring scales is essential for the empirical assessment behind the index and allows for comparability across indicators and conventions, which otherwise would not be possible, since variables have different reporting options for data. Based on this coding scheme, the index construction continues with the coding process. National reports are coded to build a database that includes the reported data submitted by each country to each convention through the different reporting cycles for all the selected variables. The coding process is done by two researchers in a rotation set up to ensure intercoder reliability. The information resulting from the coding process shows to what extent countries have fulfilled the obligations defined by each convention, according to their own national policies and objectives. National reports also provide insights about which factors determine the overall success of the convention at the national level.

The coded data are then scored according to the previously defined scales. This results in a score for each variable as reported by each country. Scores are assigned automatically by the database. Once the option for reported data is entered, the database automatically calculates the score corresponding to that variable. From there, a process of computation takes place to find the average of the variables within each indicator to obtain the final value for each of them. The overall index score is the arithmetic average of the scores for each country in each indicator.

This protocol (see Figure 2 and Table 2) results in an empirical measurement that assesses implementation by country and by convention and ensures comparability of results. At this stage of development of the index, all the variables and indicators have the same weight, but further progress in the construction of the index might, however, change this approach. Weights for the different types of indicators will be determined based on consultations with experts within research institutions, governments, and multilateral organizations and on consensus. Specific indicators can also be used separately to evaluate specific policy dimensions in the process of implementation. Each variable is accompanied by a guidance note that includes the question to which it corresponds.

3.1.5. Results and Database

Even though the purpose of this article is to present the construction of the ECI and its policy relevance, an overview of the general results is important to evidence its potential and its connection to both the policy process and the literature about the implementation of global environmental conventions. To create this overview, Figure 4, Figure 5, Figure 6 and Figure 7 illustrate the color-coded scores of each country in the ECI. These are based on the latest available reporting cycle for each country in each convention. The scores are presented on a gradual scale. High performing countries are represented in dark green, while lowest-performing countries are represented by dark red. Countries in white are not members of the conventions, and countries in gray have failed to submit the national reports. Importantly, as explained above, scores for CITES are based on the biannual report. Based on all these scores, and the timeline data for previous reporting cycles, the ECI offers trends analysis, overall and regional rankings, and analysis by type of countries. Rankings can also be provided for specific indicators and variables. These data constitute the main input to explain implementation and contribute to understanding why the same country performs differently across different conventions as well as illustrate how different countries perform differently on the same convention.

In the case of the Basel and the Stockholm conventions, contrary to arguments presented in the literature, and by some analyses developed by policy-makers and non-governmental organizations (NGOs) [110,111,112], which argue that these agreements are not being implemented, evidence from the ECI shows a broad spectrum of results. Implementation analysis of empirical data shows that developing countries and economies in transition are implementing many of the obligations in the chemicals regime. It also reveals the importance of factors such as the availability of data, comprehensive regulations, national capacity, cooperation, and funding [110,111,113,114]. These results call for analysis to determine which factors—besides the level of development—act as the main obstacles to progress so that both convention secretariats and countries can address them. National data on implementation is also required to connect the definition of national policies with the effectiveness of the conventions. According to the 2019 Global Chemicals Outlook II, for example, “national reporting on hazardous waste generation varies by country and is often incomplete” [115]. Furthermore, even though developing countries and economies in transition are decreasing the amount of hazardous waste they import, the amount exported has increased considerably. In the case of Persistent Organic Pollutants (POPs), positive results have been achieved for some compounds, but some new substances still need to be phased out [77]. However, since data are incomplete, additional information is necessary to evaluate the extent to which the conventions are effectively addressing the threat of chemical pollution and its effects on human health and the environment.

In the case of the conservation conventions, implementing global environmental commitments around the protection of wetlands and the regulation of illegal wildlife trade reflects the diverse realities of countries and conventions in their national policies, global strategies, and managerial approaches. The results of the Ramsar Convention exemplify the importance of data availability through national reports. Despite a large number of questions in the reporting template (126 in 2018 as Table 6 illustrates), the average national reporting rate for the Ramsar Convention is 88 percent, the highest among the four conventions in this analysis. In the case of CITES, reporting is particularly challenging with an average national reporting rate of 38% for about the same number of questions (120). Nevertheless, even when the ECI scores show a greater distance to the target, the progress achieved by developing countries deserves special attention.

Developing countries demonstrate how the nature of the biodiversity issues influences the process of implementation. States that are either more concerned or better equipped to address the challenges of wetland degradation and illegal wildlife trade exhibit more consistent implementation patterns. The problems also present a different level of complexity that may affect the process of implementation. The corporality of wetlands and the complexity of the criminal activities behind the non-compliance with CITES are among the variables that could be analyzed when explaining why countries achieve different results in the implementation of the biodiversity conventions.

The ECI database, therefore, constitutes the main input to explain implementation and analyze the causes for countries’ performance and the determinants of implementation. The ECI database is available at www.environmentalconventionsindex.org. As a semi-quantitative standardized metric, the index is comparable by convention, by country, and by types of countries. Figure 8 and Figure 9 present the ECI results by convention. It is possible to use the index to establish connections between environmental conventions and to understand changes in policy behaviors and environmental quality. It provides an aggregate overview of the implementation. It also illustrates the gap between developed and developing countries. While for the conservation cluster, there is practically no difference associated with the level of development, in the pollution cluster, developed countries exhibit significantly higher performance (see Figure 8) reflecting on the role of technology and technical capacity in the management of hazardous wastes and persistent organic pollutants. The data also highlight the differences (and lack thereof) among geographical regions (see Figure 9). Importantly, no continent- and no country-evidences consistent excellence in performance across all conventions.

Three important characteristics distinguish the ECI. First, the Index examines data related to the national implementation of the legal obligations defined by the conventions. Second, it assesses to what extent signatory countries have provided the required information, created the necessary institutions, and possess the technical capacity to comply with their obligations. This feedback is important to the convention secretariats, as it will help them determine how to allocate institutional and financial resources, and how to improve national capacities for implementation. Third, by using a large-n analysis—including the entire convention membership—(see Figure 2 and Figure 3), it allows for a robust assessment of implementation results for every country party to the four conventions analyzed (see Table 3).

By evaluating implementation under the same parameters, using a set of indicators based on the national reports that signatory countries submit to the convention secretariats, the Index allows for multifaceted analysis and comparison. It also identifies trends over time for individual countries, groups of countries, and the different conventions. The ECI database, its methodology, and the national implementation profiles contribute to the understanding of how countries are translating their obligations into national environmental policies, offering policy inputs to improve the performance of countries and conventions.

3.1.6. Limitations

Inevitably, there are certain limitations to the construction, analysis, and validity of extensive empirical research and the Environmental Conventions Index is no exception. First and foremost, the fact that the main source of information comes from national reports raises concerns about self-evaluations as a reliable data source. It is possible that the reports reflect inaccurate information both about implementation and the lack thereof. Yet, the fact that the source of information is the national reports ensures that all countries are measured with the same yardstick and since all the data are self-reported, it eliminates researcher bias. Importantly, the national reports have data gaps in terms of unanswered or incomplete questions, lack of consistency across variables and indicators, and the absence of national reports. There is also the possibility that key areas of implementation are not covered by the questions included in the national reports template. The scope of these data gaps evidences the shortcomings in governance indicators and the need for better reporting, data collection, and monitoring mechanisms for environmental governance [116]. It is essential that both international organizations and national governments strive to define better mechanisms to evaluate their policies that achieve greater data coverage and analysis.

Second, the ECI reflects the score constructed based on the data included in the latest available report for each country for each convention. Therefore, data may be outdated and reflect an assessment of implementation that is no longer valid. In the case of CITES, for example, data for three of the top 20 countries in the ECI corresponds to the 2009–2010 reporting cycle, which makes the assessment of implementation at least seven years old by the time the construction of the ECI started (2017). That is why it is important to use the ECI as a baseline for the construction of national qualitative case studies, similar to the methodology that is implemented by global indexes such as the HDI [100].

Third, the ECI is based on indicators and variables that carry the same weight in the final score. However, there can be variables and indicators that are more essential to the process of implementation. Some conventions don’t even consider the financial and information indicators, as is the case of the Basel Convention. The effect of this could be an incomplete perspective on the different dimensions of the process of implementation. Every analysis should account for the possibility of assigning different weights to the various variables, which can evolve based on developments and trends within the different conventions, and specific priorities at the national level. That is why further analysis of the ECI may need to use a multi-criteria approach to define which variables and indicators are more critical to the process of implementation and assign weights accordingly [117,118].

Fourth, as more national reports become available and more reporting cycles are completed, the ECI may be able to include more relevant indicators. At the same time, more rigorous data standards through the analysis of different reporting cycles—current and future—may result in the replacement or omission of variables used in previous iterations. Moreover, considerable labor and financial investment is required in terms of data collection, the review of scoring scales, and data coding and analysis. The ECI can also be expanded to other conventions to offer a more integral perspective about how these instruments result in policy changes to protect the environment.

Finally, there is a lack of systematic data connection between the implementation of global environmental conventions and the effectiveness of these agreements in the resolution of environmental problems. Assessing the effectiveness of the conventions requires comparative data on the state of the environment over time, monitoring, consistent reporting, analysis, and verification. Stating that problem resolution results from the implementation of the convention obligations requires causality analysis across various sets of data. This is certainly an area of possible research expansion where the results of the Index could be used as an independent variable.

Nonetheless, the approach of the ECI and the utilization of the national reports offer a comprehensive overview of the process of implementation at the national and the global level. While limitations emerge mainly from the availability of data, its self-reporting nature and the gap between implementation and effectiveness, by recognizing them the ECI is on the path of constituting a fundamental indicator within the system of global environmental governance and supporting efforts by countries to create national measures to address global environmental challenges. The ECI is the best existing proxy to assess the fulfillment of the obligations that countries acquire when becoming parties to conventions and highlights the relevance of global governance indicators for environmental policymaking.

4. Conclusions

What is clear from the multiple analyses that measure and explain implementation is that questions about this process continue to be as relevant as they are complex. When determining the role of conventions and their effectiveness in solving global environmental problems, clearly, there is a long list of determinant variables. However, implementation is “the central process to turn commitments into actions” [30] and deserves special attention. Previous scholarship has conceptualized the conflict of implementation and assessed its levels, but “very little empirical research [had tried] to answer these questions in a systematic way” [28]. This lack of systematic empirical evidence constitutes a key gap in both the scholarly literature and in policy assessments. As research on global environmental governance advances, multilateral environmental agreements provide opportunities for learning. Improved understanding of the extent to which countries implement their international obligations is fundamental to the ability to measure progress, adjust the mechanisms for ensuring implementation, and achieve both the goals of the conventions in the protection of the environment and sustainable development.

By explaining the extent of implementation of global environmental conventions and the dynamics behind the observed results, scholars can provide input for policy processes that address planetary challenges and proscribe human activities harmful to the environment. Measuring implementation supports effective and better governance, through the definition of a series of policy recommendations that inform governments and international organizations. The methodology of the ECI can be used across policy areas, and new research projects can be developed to expand the work to other global conventions as well as other international agreements related to environment, development, and sustainability. Among several projects, the ECI has served as the foundation for the creation of the World Heritage Site Index, a tool designed to measure the extent to which member states implement the World Heritage Convention in each of the cultural and natural sites regulated by the agreement [118].

Ultimately, data from the Environmental Conventions Index set a semi-quantitative and interdisciplinary performance baseline and promote learning. The index expands the scale of existing studies about the implementation and effectiveness of global environmental conventions to include more countries and to compare different conventions across the same parameters. This allows for the identification of patterns that reflect on and challenge traditional assumptions about the extent of implementation and the factors that determine it calling for alternative explanations of the observed patterns. The index could also be used as the dependent variable in qualitative and quantitative analyses of multiple factors to test theoretical assumptions about the determinants of the process of implementation, including factors such as national characteristics, the status of the international system, the nature of the policy issues addressed by each agreement, and their governance structures, among others [28].

Comparative, time-series, self-reported information provides salient and legitimate input for policy recommendations, addressing and often challenging conventional beliefs about countries’ capacity, the nature of environmental issues, the design and characteristics of the agreements, the roles of international institutions, and the relationship between implementation and effectiveness. It could also broaden the understanding of interactions between state parties and the convention secretariats in defining national policies to achieve global objectives. Identifying the factors that determine implementation in each country is central to understanding national results, establishing best practices, and identifying challenges that must be addressed.

Fulfilling global environmental goals requires information, assessment, and monitoring. The Environmental Conventions Index addresses all these elements. The index brings empirical evidence to the analysis of implementation and informs countries and conventions about the extent of their progress in fulfilling their international environmental obligations providing a baseline and allowing for comparison of results across countries and conventions. Measuring and understanding implementation is necessary to articulate and analyze best practices that scholars and policy-makers could deploy to assist with improving outcomes.

Ultimately, the Environmental Conventions Index would provide an important public information resource. Since the Index is composed of specific information on issues such as legislation, institutions, information, technical environmental operations, and finances, it offers a wealth of data on areas of implementation that require more progress. It can be used to define targeted capacity-building mechanisms to improve national performance as well as interventions by the convention secretariats to better address environmental problems. Analytical outreach strands such as data analysis, policy recommendations, the creation of knowledge-sharing and policy spaces, leadership development, and the solution of global environmental problems are part of the expected outcomes from the use of the ECI. This innovative measurement tool based on publicly available information provided by member states and presented in a comparable and compelling manner offers the possibility to devise evidence-based global responses to environmental problems, improve the legitimacy of the global environmental governance system, and create a basis for co-creation of solutions to issues central to human well-being and sustainable development.