4.2. Main Differences in Baseline Assessments
The SDSN and OECD reports assess how far countries have progressed towards each SDG at a given point in time (so called baseline or static assessments). We compare average results obtained for the 23 EU member states covered in both reports. At the time when the reports were launched, Brexit was not yet completed, so the United Kingdom was retained. Bulgaria, Croatia, Cyprus, Malta, and Romania are not members of the OECD and are not covered in the OECD SDG report. For the SDSN, scores were obtained by calculating the arithmetic average of all goal scores for all 23 countries. The same approach was used to generate the OECD scores, but in addition, OECD scores were rescaled from 0–100 (originally from 0 to 3).
Figure 1 compares the estimated average performance of EU member states for each goal.
The EU average performance in both reports is rather similar for five out of the seventeen SDGs. In both reports, the scores for SDG2 and SDG5 are below average. In both reports, scores for SDG6 and SDG15 are above average. Scores obtained in both reports for SDG11 are rather similar and close to the average.
However, there are major differences in scores and relative SDG ranks between the two reports. The overall score and SDG ranks show no significant correlation—0.05 and 0.04, respectively. On one hand, the 23 EU member states perform worse according to the SDSN methodology than according to the OECD methodology on most SDGs related to “Planet”, including SDG12, SDG13, and SDG14. In the SDSN methodology, the scores obtained on these three goals are much lower than the average SDG score, whereas the OECD report concludes the reverse. Similarly, the estimated performance against SDG7, SDG9, and SDG17 is also much higher in the OECD report than in the SDSN report. Conversely, countries perform better in the SDSN report on the goals related to “People”, including SDG3, and SDG4. They also perform better on SDG8, SDG10, and SDG16.
These differences in scores lead to differences in the relative performances (ranks) of countries on each SDG.
Figure S1 (Supplementary Materials), reinforces the findings presented above by focusing on relative SDG rankings in both reports. For our comparison, we consider rank differences greater than four as denoting a significant difference in results.
Focusing on countries’ ranks instead of scores, EU countries perform similarly in both the OECD and SDSN reports on SDG1, SDG6, SDG11, and SDG15 (medium performance), as well as on SDG2 and SDG5 (poor performance). By contrast, EU countries obtain a better performance in the OECD report than in the SDSN report on SDG7, SDG9, SDG12, SDG13, SDG14, and SDG17. EU countries perform worse in the OECD report than in the SDSN report on SDG3, SDG4, SDG8, and SDG16.
In addition to the graphical representation of the differences in findings presented above, the narrative sections of both reports also reflect these differences. The OECD mentions in their 2019 SDG report that “OECD countries are, on average, closest to achieving goals on Energy, Cities, and Climate (goals 7, 11, and 13) and goals relating to Planet (Water, 6; Sustainable Production, 12; Climate, 13; Oceans, 14; and Biodiversity, 15).” By contrast, the SDSN concludes that these are among the goals where the EU as a whole and the individual EU member states face the greatest challenges (especially SDGs 12, 13, and 15).
4.3. The Influence of Methodology Versus Indicator Selection in Baseline Assessments
We find that the results of the SDSN and the OECD do not become more comparable after controlling for differences in methodologies under scenarios 1 and 2. The goals that correlated well between the SDSN and OECD reports before doing any adjustments also correlate well under adjustment scenarios 1 and 2.
Table S1 (Supplementary Material) presents correlation coefficients between the SDSN’s goal scores and those of the three other reports under various adjustment scenarios. Goals that did not correlate well, mainly environmental and biodiversity goals, continue to correlate poorly among each other in adjustment scenarios 1 and 2.
When applying the SDSN methodology, the results obtained by the SDSN, Eurostat, and ASviS reports generally tend to be more consistent with each other than with the OECD results.
Table 4;
Table 5 present overall score and rank correlation coefficients between all four reports in scenario 2.
These results are explained by the choice of indicators and data sources. The SDSN, Eurostat, and ASviS rely extensively on data collected and compiled by the European Commission services (Eurostat, Joint Research Center, European Environmental Agency, etc.). In contrast, the OECD uses more OECD data and other data sources. The Eurostat and ASviS reports obtain the highest scores and rank correlations because ASviS considers a subset of the SDG indicators selected by Eurostat.
Performances on SDGs 3–5, 9, and 10 are highly correlated across all four reports because similar indicators are used. Typically, these goals cover a more homogeneous set of policy issues. We computed the median rank difference between the maximum rank and the minimum rank obtained for each goal and each country across the four indicator sets under adjustment scenario 2 (
Figure 2). Overall, countries’ ranks on SDG3 and SDG9 are very consistent. These are the goals where inter-item correlations across indicators tend to be highest. In other words, their measurement is homogeneous across indicators. As a result, the choice of indicators for these goals has a smaller impact on overall results compared with other goals that cover more heterogeneous issues.
Some goals are more heterogeneous and, therefore, are much more sensitive to the selection of indicators. Overall, the median rank difference is highest for biodiversity and other environmental goals (SDG6, SDG12–15) as well as for SDG2. Researchers have demonstrated the high heterogeneity of biodiversity and other environmental goals (especially SDG7, SDG13, and SDG15) [
63]. Earlier analyses conducted by the SDSN using its global dataset and based on Principal Component Analysis (PCA) also demonstrated high heterogeneity for SDG2, which captures three distinct components: undernourishment, malnourishment, and sustainable agriculture [
64]. Assessments of SDG progress in the EU only (SDSN, Eurostat, and ASviS) do not cover undernourishment and food insecurity, whereas the OECD covers these aspects under SDG2. The SDSN, the OECD, and Eurostat track obesity rates, whereas ASviS focuses on agricultural efficiency (income per annual work unit), research and development, and sustainability (e.g., area under organic farming). These differences in indicator coverage under SDG2 have a significant impact on countries’ scores and rankings.
We also find that the sensitivity of the results to the indicator selection is more important for certain EU countries. After calculating goal scores under scenario 2 for all four reports, we generate countries’ ranks on all seventeen goals across all four reports. We then calculate the difference between the maximum rank and the minimum rank obtained for each country and for each goal. We then compute the median rank difference across the seventeen goals to obtain this overview. We use the median instead of the mean in order to reduce the influence of strong outliers. Rank differences for all goals are available in the
Supplementary Material.
Overall, we find that countries’ ranks across the seventeen SDGs and the four reports under scenario 2 are more consistent for Austria, Spain, Sweden, and the United Kingdom (
Figure 3). In turn, the sensitivity to the indicator selection is more pronounced for Greece, Ireland, and Portugal. For Greece, this is explained by a much better ranking on SDG7 in the OECD report compared to the SDSN, Eurostat, and ASviS reports. Greece ranks better in the SDSN report on SDG17 compared with the OECD, Eurostat, and ASviS reports mainly due to the inclusion of financial spillover effects (tax havens, profit shifting, financial secrecy) in the SDSN report, where Greece performs better than many EU countries, including Ireland, Luxembourg, the Netherlands, and the United Kingdom.
Table S2 (Supplementary Material) presents the difference between the maximum and minimum rankings obtained for each goal and every country after controlling for methodologies in all four reports.
4.4. The Impact of the Inclusion of Transboundary Impact Measures in Explaining Differences in Baseline Assessments for Environmental and Biodiversity Goals
The removal of spillover indicators from the SDSN assessment helps narrow the differences between results in the SDSN and the OECD baseline assessments for SDG12, SDG13, and SDG17. It also brings the results closer to Eurostat’s and ASviS’s illustrative goal scores. This is highlighted by the increased correlation coefficient in
Table S1 (Supplementary Material) between scenarios 2 and 3 for these three goals in all three reports. One indicator was removed under SDG6 (imported groundwater depletion), one under SDG8 (fatal work-related accidents embodied in imports (per 100,000 population), two under SDG12 (imported SO
2 emissions and imported reactive nitrogen), two under SDG13 (imported CO
2 emissions and contribution to the international 100bn USD commitment on climate related expending), one under SDG15 (imported biodiversity threats) one under SDG16 (exports of major conventional weapons) and two under SDG17 (shifted profits and corporate tax haven score).
However, even after controlling for methodologies and after removing spillover indicators, the results for SDG14 and SDG15 remain very different between SDSN reports and the three other reports. Under the most adjusted scenario, the results for SDG14 in the SDSN report correlated negatively with the OECD results, and exhibit moderate correlations with the Eurostat and ASviS reports. There are no differences between scenarios 2 and 3 because no spillover measures are included under SDG14. Under scenario 2, SDG14 is the best-performing goal for 23 EU member countries according to the OECD analysis, whereas it is ranked 11th in the SDSN assessment. Scores on a 0–100 scale are very different between the SDSN (63.3%) and OECD (86.2%). Under scenario 2, two indicators out of three were removed from the OECD indicator list: the aggregated indicator for policies and practices against Illegal, unreported and unregulated(IUU) fishing and budgetary transfers to individual fishers. Only protected areas as a share of the Exclusive Economic Zone (EEZ) were retained. The inclusion in the SDSN’s indicator list of two indicators that come from non-official statistics on unsustainable fisheries (fish stocks overexploited or collapsed and trawling) explain, to a large extent, the discrepancy in results for SDG14 compared with the other reports.
SDG 15 is an interesting case. The original average scores obtained between the SDSN and OECD reports are quite comparable, yet the performances obtained by individual countries are not at all similar between the two reports. This explains the poor correlation between scores for this goal between the SDSN and OECD reports (0.07 in the best scenario). In the SDSN report, Estonia, Latvia, and Lithuania rank highest for SDG15, and Germany performs rather poorly. In the OECD report, however, Latvia and Lithuania are two worst performers on SDG15, and Germany tops the ranking. This is explained, to a large extent, by the inclusion of a transboundary impact measure (imported biodiversity threats) in the SDSN report to capture impacts generated by the EU and EU member states on biodiversity threats in other countries through trade and consumption. It is further explained by the inclusion in the SDSN report of metrics on pollution in rivers and groundwater, whereas the OECD report gives greater weight to forests and protected areas. The various adjustments we make to the calculation of SDG15 do not increase correlation across results in the reports. The difference with the Eurostat and ASviS assessments is explained primarily by the use of global indicators of biodiversity loss and protected areas in the SDSN report (Red List Index, Protected Key Biodiversity Areas), whereas Eurostat and ASviS make more extensive use of EU-specific frameworks (e.g., surface of terrestrial sites designated under Natura 2000).
4.5. Interpreting Countries’ Trajectories
All four reports find that historic rates of progress in EU countries are insufficient to achieve some SDGs. The trajectory for SDG15 is rather poor on all three reports.
Figure 4 shows how each organization evaluates progress on each of the seventeen goals. For each report, goals are presented in descending order, i.e., from high progress to low or negative progress. The adjusted SDSN results cover the 28 EU member states (EU28), including the UK. The SDSN report does not compute trends for SDG12. The Eurostat report does not report summary trends for SDG6, SDG12, SDG14, and SDG16. The ASviS report does not present trends for SDG6.
There are important differences between the findings presented by the SDSN on the EU SDG trajectories compared with the other three reports. For example, according to the SDSN and Eurostat, progress towards SDG13 is slow or insufficient, while the OECD estimates that the largest number of countries are moving towards SDG13. SDG13 is also included among the rapidly progressing goals in ASviS. SDG8 is the goal that sees the fastest pace of progress in the SDSN report, whereas the majority of countries are moving away from the target according to the OECD report. SDG8 is among the three goals where progress is fastest according to Eurostat, and it is in the middle of the range in the ASviS report. The SDSN concludes that the EU is moving in the wrong direction on SDG2, primarily due to trends in obesity rate and unsustainable diets and agriculture. The OECD findings are similar. By contrast, Eurostat and ASviS conclude that the EU is making some progress on SDG2. Both the SDSN and Eurostat report positive trends for SDG1, whereas progress is flat according to ASviS and negative in the OECD report. It is not possible to make comparisons for SDGs 6, 12, 14, and 16 due to missing data for one or several of the reports.
The findings in the previous sub-sections related to indicator selection also help explain differences in assessments of countries’ trajectories. For instance, on SDG2, the SDSN’s use of the non-official measure of energy intensity of diets (Human Trophic Level) explains the relatively poor results, as 27 out of 28 EU member states have flat or negative progress towards the SDG on this indicator.
However, there is another key methodological aspect that explains differences in the assessment of trajectories across the four reports: the inclusion or not of pre-defined targets to assess progress over time. Conceptually, methodologies to track countries’ trajectories towards the SDGs can be grouped into two categories: (i) those that aim to assess whether countries are on track or off track to achieve the goals (SDSN, and partly Eurostat), and (ii) those that aim to identify the SDGs on which countries are making the most progress (ASviS, Eurostat partly, and OECD).
Eurostat falls into both categories because it has a dual methodology for estimating trajectories. Where pre-defined EU targets could be identified (16 indicators), Eurostat calculates progress towards these goals. For the 83 indicators where no such targets have been politically agreed upon, Eurostat uses a default threshold (+/− 1% increase or decrease) to estimate whether the EU is making good or low progress towards the corresponding SDG.
The OECD, ASviS, and, to a large extent, Eurostat provide an indication of whether countries are moving in the right or wrong direction, but without an indication of whether the pace of progress will be enough to achieve a pre-defined target by 2030. The OECD defines targets to estimate baseline countries’ performance, but does not estimate whether the pace of progress is sufficient or insufficient to achieve the targets. As noted in the OECD report, “Progress towards the target says nothing about whether the pace recently achieved by a country would be sufficient to meet the target level by 2030.”
In contrast, the SDSN has developed a methodology to assess if countries are “on track” towards achieving the 2030 objectives [
21,
64]. This methodology is based on a linear extrapolation of past trend data (typically four years) into the future, all the way to 2030. This approach is quite similar to the Eurostat method for indicators that have pre-defined targets. If the extrapolated trendline exceeds the pace of progress required to achieve the target value by 2030, an upward green arrow is assigned to denote “on-track” performance. If countries have maintained performance over the pre-defined threshold, they also obtain an upward green arrow. The other intermediate arrows are assigned depending on whether the extrapolated growth rate is equivalent to 50% of the needed growth rate (moderate upward arrow) or lower than 50% (flat arrow). A downward red arrow is obtained when progress is negative, i.e., the country moves away from the goal.
The choice of method for the estimation of trajectories has major implications for the findings and their policy interpretations.
Figure 5 provides a graphical representation of the difference in policy interpretations in the SDSN and ASviS reports for SDG14 and SDG16. Both reports underline progress made by the EU on SDG14 since 2010 and very little progress (SDSN) or slight decline (ASviS) on SDG16. This is also broadly consistent with the OECD’s findings where, on average, 19.5 countries are moving towards the target on SDG14 and zero countries are moving away from the target, while on SDG16, 3.75 countries are moving towards the target and four are moving away.
However, the three reports interpret these results very differently. The SDSN approach suggests that, despite progress on SDG14, the pace of progress is vastly insufficient to achieve the target of SDG14 by 2030. For the EU as a whole, this corresponds to a yellow arrow (moderate progress) for SDG14. Despite its slower progress on SDG16, the EU obtains a green arrow overall, since it is much closer to achieving these goals, so a smaller rate of progress is sufficient. In other words, slower progress on a well-performing goal (SDG16) can achieve a better assessment than faster progress towards a goal, where the achievement gap is greater. When combining the SDSN’s baseline dashboards results and dynamic dashboards results (the EU is red on SDG14), one concludes that further reforms and actions are needed to achieve SDG14 by 2030. The ASviS’s trend lines do not suggest the same conclusion, since they show that progress towards SDG14 is among the fastest across the SDGs.
Similar findings apply to differences in treating SDG13. ASviS presents rapid progress (+4% since 2010) on SDG13. The OECD considers that all OECD countries are moving in the right direction on SDG13. However, the SDSN finds that when extrapolating the annual rate of progress over the past few years to 2030, progress towards the climate goal is too slow.
There are other reasons that explain differences in the assessments of SDG trajectories across the reports. While scores are calculated using the latest year available, comparisons over time require the selection of a base year. This base year varies across reports and considerably affects the results. The SDSN typically uses 2015 as a base year (the year when the SDGs were adopted) and computes trends through to 2018 or later (when data are available). When no data are available for 2018 or thereafter, the trend assessment is based on the last four years available (e.g., 2014–2017, 2013–2016, etc.). In this way, each indicator is assessed over a four-year period. The OECD uses a longer time span, typically covering 2005 to 2017. Eurostat uses a dual approach and presents assessments over the long term (2003 to 2018) and short term (2013 to 2018). The overview results presented in the opening sections of the Eurostat report focus on the short-term trend. Finally, ASviS curves typically cover trends from 2010 through to 2017.
The selection of the base year can considerably affect the summary assessment of progress. The Eurostat report, which includes both long-term and short-term assessments, provides a few good examples. On average, the share of “People at risk of poverty or social exclusion” (notably included under SDG1) in the EU moves away from the EU target over the period from 2003 to 2018, but progresses towards the EU target between 2013 and 2018. In total, depending on whether the long-term or short-term trend is retained, the arrow direction shifts (from progress to movement away, or vice versa) for 12 indicators (12%) included in the Eurostat report, including key indicators covered by all three other reports, such as people killed in road accidents or official development assistance (ODA), for instance.
The use of “arrows” or “clusters” makes the results easily communicable to policymakers, but has the disadvantage of being significantly affected by the baseline year. Trajectories may also not reflect a sudden positive or negative trend towards the end of the period. The ASviS approach has the advantage of covering all years but may require more imputations for missing country data over the years. Presenting results in terms of curves and annual data points can better reflect the impact of sudden shocks (such as Covid-19) over one or two years.