Evaluating the Sustainability in Complex Agri-Food Systems: The SAEMETH Framework

Abstract

During the last few years, the definition of sustainability and the translation of its general principles into practical and operative tasks have come into the foreground of scientific research and political agendas throughout the world. The understanding and the evaluation of the environmental, social and economic performances of complex agricultural food systems is probably the real challenge, and the design of more sustainable alternatives has been recognized as necessary for a correct territorial management. This study’s primary goal is the proposition of an interpretive structure “Sustainable Agri-Food Evaluation Methodology” (SAEMETH), able to guide the evaluation of the sustainability of the various organizational forms of the small-scale agri-food supply chain. As a case study, the methodology was applied to 10 small-scale agri-food systems. The application of SAEMETH, as a monitoring tool based on qualitative indicators that are user-friendly and strongly communicative, demonstrates that it is possible to carry out sustainability evaluations of the small-scale agri-food systems through a long-term approach that is participatory, interdisciplinary and multi-institutional and that integrates a solid theoretical base with an operative framework tested in the field. SAEMETH can, in this way, generate a cyclical process that increases the probability of success in the design of sustainable alternatives and the implementation of projects and initiatives at the local/regional scale.

1. Introduction

The definition of sustainable development [1], known and used throughout the world since its birth, has not engendered a unanimous consensus on its significance [2]. Already, in 1998, Hueting and Reijnders [3] emphasized the difficulty in putting sustainability theories into practice, and over the years a wide-ranging debate has developed in the scientific community, including a discussion of the role of “objectivity” or “subjectivity” expressed in any definition of sustainability [4]. In particular, these interpretive difficulties emerge when speaking of sustainability indicators and their application, often to uncertain and complex contexts [5], with the role of subjectivity inevitably predominant in the evaluation.

Thus, the sustainability monitoring of a system appears to be a long-term exercise in which various types of both quantitative and qualitative indicators can be used [6,7]. Technically, those indicators with a qualitative response can be imperfect. However, the success of their evaluation is not only dependent on the indicators themselves, but also, and most of all, on the process and on the exchange of ideas leading to their conception. The discussion and the exchange of ideas between the various subjects involved in the process of verifying and adapting these indicators [8] are especially important. Thus, the indicators always represent a synthesis that simplifies the phenomenon without trivializing it. Instead, they are indicative of the conceptual framework within which they were conceptualized [9].

The use of indicators inside a sustainability evaluation framework surely carries an important role and represents an efficient instrument for understanding and designing alternatives; for example, for land-uses, agronomic techniques and agricultural system and agri-food supply chain management [10,11,12,13]. The conventional evaluation approaches have not been altogether efficient in interpreting the sustainability of complex systems, because they have often concentrated on just a single dimension of sustainability (for example, the economic, environmental, or social dimension) instead of looking at sustainability in the more holistic sense. More in general, the most used approaches in sustainability studies can be divided, as Masera et al. [14] and Lopez-Ridaura [8] affirm, into three principal groups: approaches that use indicator checklists, approaches that use composite indicators, and finally approaches that apply a framework.

The use of indicators was originally focused on economic sustainability, and used such indicators as net income and gross margin [15]. When the concept of environmental sustainability was approached, they were focused on evaluating Man’s management of natural resources and concentrated on such aspects as the evaluation of soil quality [16], pesticide use, crop rotations and on soil fertility or biodiversity management [17,18,19,20].

Instead, other studies have looked at composite indicators positively, where a specific set of indicators is evaluated by integrating them into a single value, such as the Farmer Sustainability Index [21], the Indicator of Sustainable Agricultural Practice (ISAP) [22], and the Agricultural Sustainability Index (ASI) [23]. The use of an index that is able to unite a specific set of indicators certainly facilitates the integration of these same indicators in the process of sustainability evaluation with the limit, however, of neglecting the uniqueness of each system (indicators that are significant for one system can be irrelevant for another). Furthermore, the numeric value obtained does not allow for a discussion of both the specific strengths and weaknesses of the system.

The Frameworks that have been developed to evaluate sustainability have been numerous; some examples are the international Framework for the Evaluation of Sustainable Land Management [15] and the Pressure-State-Response framework (PSR) [24].

However, these frameworks have almost always not been able to fully help the stakeholders in the process of sustainability evaluation. However, other methodologies have tried to compare agriculture systems at a local and regional level such as those proposed by Andreoli and Tellarini [25], Bosshard [4], Tellarini and Caporalli [26], Cornelissen [27] and Van Cauwenbergh et al. [28], as well as the studies of natural resource management reviewed by Galvan-Miyoshi et al. [29] and Mayer [30].

Not withstanding these numerous studies, and just as many methodological advances, the spread of reference frameworks and of indicators has been extremely slow. An important effort in the diffusion and application of these methodologies was made with the collection of a number of important case studies for the evaluation of sustainability using the MESMIS (Marco para la Evalutación de Sistemas de Manejo de recursos naturales mediante Indicadores de Sustentabilidad) [31,32,33,34] and MOTIFS (Monitoring Tool for Integrated Farm Sustainability) [35,36] methods.

The most interesting aspect of these last two methods regards the particular attention they give to the involvement of the final users, since often failures in the application of a framework are not just methodological but instead come from a lack of interaction between the developers and the users of the model [36].

This misalignment between the perspective and needs of a model’s developers and those of its final users ensures that the model’s results will be inefficient and difficult to apply [37]. This is especially true in small-scale agri-food systems in which this integration is fundamental for a complete and functional evaluation [38].

In fact, the purpose of our study was that of developing a monitoring tool for the Small-scale agri-food system starting from indicators that translate into practice the sustainable vision of the food system proposed by the non-profit, the Slow Food Foundation for Biodiversity’s Presidia [39], but that aspire to be applicable in other local food systems as well, such as PGI (protected geographical indication), PDO (protected designation of origin) and other kinds of regional labels. The conditions that enable the replicability of the tool’s use is given to the presence in such local agri-food systems of a common and core denominator charactherized by the valorization of a single product supply chain where quality is at stake, the specification of the area of production, the existence of a shared protocol of production, the rescricted participation of only those producers operating within the identified area and respecting the protocol.

This kind of small-scale system expresses the specificity of a determined region, and as a consequence also expresses the entirety of its production. Therefore, the product, aside from being chosen for its symbolic-identity value, in turn also becomes a symbol of the values expressed by a region and its community. The monitoring instrument was developed along with a substantial group of stakeholders through a participatory process and might be used as a project management tool, but also as a tool aimed at improving the sustainability performances of the small-scale agri-food system as well as a tool to communicate with consumers. In this article, the methodology applied for the development of this monitoring instrument is described. We have taken five successive phases into consideration:

• the definition of the constitutive elements of the analyzed system (the system limits);
• the translation of the socio-cultural, agri-environmental and economic aspects of sustainability into concrete components that are relevant for the single small-scale agri-food system;
• the design of indicators for monitoring the progress made towards sustainability for each of these components;
• the aggregation of these indicators into a sustainability monitoring tool for the small-scale agri-food systems; and
• the application of a monitoring tool to a number of 10 case studies as an attempt of a final validation.

2. Materials and Methods

2.1. Type of Systems to Assess—The Elements that Make Up the Analyzed Systems

In 1999, Cobb et al. [40] had already recognized that “The food chain as a whole is the ultimate framework for a scrutiny of sustainability”.

In this context, sustainability is seen in broader terms than just a business analyses and, in particular, in the food supply chain, the vertical connections that go beyond the production system are taken into consideration. Furthermore, it is necessary to remember that even in the field of the European Union’s Common Agricultural Policy (CA-EU), the objective is and has been that of widening the rural policies into a vision that does not limit itself to being excessively centered on agriculture but that aims to be more integrated and regional [41].

The Slow Foods Presidia project, from which this framework was inspired, originated in a recognition of the necessity of generating a more conscious relationship between food and its sustainable production [38,39,42].

Each local presidium regards a single product chain at risk of extinction (a domestic species, a wild species or a processed product) that is strongly tied to the traditions, region, culture and history of local agriculture in a certain territory. All of the project’s actions have at their heart the idea of agri-biodiversity conservation that is made operational thank to the development of a system of cooperation that involves farmers, artisans, consumers, vendors, chefs, local restaurants, and other participants [39]. The sharing of the criteria that defines sustainable management practices according to the Slow Food Presidia approach is guaranteed by the adoption of specific production protocols, drafted and followed by all of the farmers that adhere to the project. The protocol should follow the guidelines that exist for every product category and defines precisely the area of production, documents the history of the product and describes in details all phases of cultivation (or breeding) and processing. All cultivation techniques must have a low environmental impact and must be ecologically compatible: they must preserve the fertility of the land and the hydrographic ecosystem; avoid the use of chemicals as much as possible by following methods, timescales and quantitities indicated by current rules for organic, byodinamic and integrated agriculture; and maintain traditional method of cultivation by giving priority to the use of manual or mechanichal methods.

All these elements are integrated into a model describing the region as a “bi-modular system” (society-nature), a system that is composed of two sub-systems: the natural module (environmental component) made up of the agri-ecosystem, and the social module (social-economic component) made up of the community (locals) [43,44]. In the end, the products that are identified as regional are very similar to traditional food products, which implicitly suggest the same two-dimensional structure: “A product frequently consumed or associated with specific celebrations and/or seasons, normally transmitted from one generation to another, made accurately in a specific way according to the gastronomic heritage, with little or no processing/manipulation, distinguished and known because of its sensory properties and associated with a certain local area, region or country” [45].

The activities related to the Presidia, such as the organization of training activities for the producers, the product promotion and communication, the development of new commercial market, enable the distribution of the value added of the project along the whole product chain and promote the involment of new producers [39].

2.2. The Dimensions and the Components of Sustainability

The holistic vision inherent in the definition of sustainability creates many difficulties when applied to systems with an elevated complexity [46]. For this reason, the methods that take into consideration agricultural systems often tend to only concentrate on one or two dimensions such as environmental protection and economic sustainability, and less often on themes regarding social acceptance [46]. The reference framework for evaluating the sustainability of the small-scale supply chain through the proposed methodology (SAEMETH) is therefore an attempt to make the concept of sustainability operative in complex territorial systems, taking into consideration all of the dimensions of sustainability that refer to socio-cultural, agri-environmental and economic factors and starting from them to design the process of sustainability assessment and monitoring (Table 1, first column).

Table 1. Sustainability’s dimensions and relative issues considered in the SAEMETH (Sustainable Agri-Food Evaluation Methodology) framework.

Sustainability’s Dimensions Level 1	Issues Contained in Each Dimension
Socio-cultural	Employment and labour market; standards and rights related to work conditions; social inclusion and protection of disadvantaged group; community power of representation, social role of producers, coordination among producers, communication network, equity and non-discrimination; access to education, health, justice and media; cultural and territorial identity; security; governance and participation, cultural heritage (material and immaterial); ethno diversity; conservation of traditional production techniques; embeddedness; tourism promotion; maintenance of historical buildings
Agro-environmental	Safety and security; nutritional quality; taste; freshness; seasonality; soil quality; water quality; air pollution reduction; biodiversity enhancement; landscape conservation; climate change mitigation; waste production/generation/recycling; energy consumption and efficiency; degree of renewal of natural resources; plant health and animal welfare
Economic	Cost and access to food; consumers and household; income and farmers and food artisans; trade and markets; operating and administrative costs of business; supply chain added values; innovation and research

Authors’ rielaboration from Peano et al. 2014 [38].

Table 1. Sustainability’s dimensions and relative issues considered in the SAEMETH (Sustainable Agri-Food Evaluation Methodology) framework.

Sustainability’s Dimensions Level 1	Issues Contained in Each Dimension
Socio-cultural	Employment and labour market; standards and rights related to work conditions; social inclusion and protection of disadvantaged group; community power of representation, social role of producers, coordination among producers, communication network, equity and non-discrimination; access to education, health, justice and media; cultural and territorial identity; security; governance and participation, cultural heritage (material and immaterial); ethno diversity; conservation of traditional production techniques; embeddedness; tourism promotion; maintenance of historical buildings
Agro-environmental	Safety and security; nutritional quality; taste; freshness; seasonality; soil quality; water quality; air pollution reduction; biodiversity enhancement; landscape conservation; climate change mitigation; waste production/generation/recycling; energy consumption and efficiency; degree of renewal of natural resources; plant health and animal welfare
Economic	Cost and access to food; consumers and household; income and farmers and food artisans; trade and markets; operating and administrative costs of business; supply chain added values; innovation and research

Authors’ rielaboration from Peano et al. 2014 [38].

When a method is developed for evaluating the sustainability of a complex system, the key step is deciding who the group of users will be; such as researchers, farmers, agricultural consultants, political decision-makers, or a combination of these groups [47]. In our case, the target users are farmers and food supply chain actors.

The process of identifying dimensions and components, like that that will be explained in the following paragraph regarding the indicators, is based on an interdisciplinary dialogue between many participants in the small-scale agri-food supply chain. In fact, through a mutual agreement between the concerned parties, reached during four structured focus groups that see the involvement of representatives of farmers, consumers and experts, an attempt was made to translate the general definition of sustainability into concrete content that could be used at a practical level to define the components and indicators most suited to the contex.

About the dimensions the outcome of the participative process has attributed to each of the three dimension an equal importance in the total measure of sustainability (equal-weight = max 100 for each measurement.

Furthermore, while each method requires a single criterion to interpret a particular dimension of sustainability, others use various criteria. This second approach is probably more consistent with the definition of sustainability understood as the capacity to meet a diversified collection of objectives [48]. The primary difficulty in using more criteria to evaluate sustainability is the necessity of determining the components and their relative importance. In our case, the definition of components and the attribution of weights to the components (Level 2) of the various dimensions with the equal weights system, reflecting the trade-offs made between the considered objectives and the priorities emphasized by the target users [47], led to the following outcome:

• For the social-cultural dimensions: Four components were selected (product, internal relationships, external relationships, culture/terroir) with a weight equal to 25.
• For the agri-environmental dimensions: Five components were selected (biodiversity, region, soil and water, crop defense, energy) with a weight equal to 20.
• For the economic dimensions: Two components were selected (development, efficiency) with a weight of 50.

2.3. Criteria, Choice and Design of the Indicators

According to Bossel [49], an indicator quantifies and simplifies phenomena that in real life are complex by translating them into useful information for the decision-making/management process of sustainability. In this study, we sought out indicators that would provide a strong signal for guiding the management of the small-scale agri-food supply chain towards a higher level of sustainability. To choose our indicators, a consideration of the criteria proposed by Meul et al. [35] seemed particularly important; specifically:

-

the evident and well-defined relationship between an indicator and the phenomenon to be monitored (causality);

-

the change in situation reflected in a variation of an indicator’s value (sensitivity);

-

the well-documented calculation method’s dependence on external factors (solidity);

-

the reference parameters available for evaluating the indicator’s value (use of reference parameters);

-

the values of indicators and the ease of interpreting their scores (comprehensibility).

The choice of indicators was made based on scientific basis in accordance with our vision of sustainability; in many other cases we turned to the stakeholders involved to choose or design suitable indicators, while always taking into account the predefined components. This kind of approach was already successfully applied by Van Calker et al. [50] and by Meul et al. [35].

Furthermore, to calculate the indicator values, we have sought to use quantitative data relative to the products, number of meetings, number of training sessions, etc. as much as possible; in other cases it was necessary to refer to qualitative data through questionnaires or checklists.

For each of the chosen indicators, we have defined a minimum threshold (0 = for the worst situations) and a maximum (10 = the best situations); the reference values are, in some cases, derived through the best techniques available, in other cases through the results of a questionnaire and, most of all, through the judgment of experts.

This structural design allows for a reciprocal comparison of indicators for the various components of sustainability.

In the end, it must be taken into account that the weight attributed to each indicator in the aggregation is often dependent on subjective scoring [46]. After conferring with the stakeholders, we weighted the indicators based on the presupposition that all of the selected components of sustainability are equally important. In fact, inside each specific component, we considered all of the indicators as equally important and, as a consequence, we gave them equal weight.

One of the most critical aspects in the definition of evaluation methods of sustainability is that of individuating the best validation processes. Bockstaller et al. [51] propose a validation of the information (output validation) and furthermore a confirmation of the usefulness of an indicator (end-use validation), while Cloquell-Ballester et al. [52] propose a method for environmental and social validation with an auto-validation phase, a scientific validation phase and a social validation phase. In our case, we adopted the methodology proposed by Meul et al. [35], which originates from these two, using an interdisciplinary approach of experts and farmers. The results obtained through the application of SAEMETH to 10 case studies were then presented to this panel of experts as well as to the producers themselves.

2.4. Target Users and Type of Visualization to Compare Options

When a method for evaluating sustainability is being adopted, a key step is that of deciding on the users that will adopt it [47]. In particular, the aggregation of the information and the final graphic representation determines the ease of use of the results. In our case, we have chosen the farmers and food supply chain actors (in particular) technicians of small-scale systems as our final users. As emphasized by Von Wirén-Lehr [46], they prefer the display of indicators that easily diagnose the cause of non-sustainability so that they can put corrective measures into play. Therefore, we chose to put dimensions, components and indicators together in a graphic way, so that they can be analyzed both singularly and as a whole. This multi-level tool allows the users to start with a global vision of sustainability (Level 1), and then to narrow into the level of components and indicators in a more detailed way (Levels 2 and 3).

For the information regarding Level 1 (dimension), the data is visibly grouped together in a bar graph in which the minimum sustainability threshold is shown as equal to 50 for each single dimension (threshold defined by a panel of experts).

For Level 2, three histograms offer a panorama of the components inside of sustainability dimension, the economic dimension, the agri-environmental dimension and the socio-cultural dimension. Furthermore, a radar chart shows all of the components of total sustainability together, independently of their size. This operation is made possible by the equal-weights approach regarding the size pertaining to each one. This tool helps farmers to conceive of their management strategies in a holistic way [53]. The values of the indicators of the analyzed systems are positioned along the axes of a radial diagram with a scale from 0 to 100, from the worst (0) to the best (100); therefore, the external ring of the diagram represents the optimal values measured for each component.

Finally, if the necessity exists to narrow in on specifics in the analysis of sustainability through bar graphs, it is possible to show the impact of individual indicators (Level 3) in the final result.

2.5. Case Studies

In this last section, we will present some results from the application of the monitoring tool to 10 Small-scale agri-food systems as a first attempt at validating the final use of SAEMETH. During the year 2012, a group of four technicians visited the producers of the analyzed agri-business systems, they collected the organizations’ data (semi-structured interviews), and they calculated the indicators and created the graphs that explain the results of each analyzed small-scale agri-food system. These results were then presented to a discussion group in which the method’s strengths and weaknesses were analyzed along with those of the production systems. Possible improvement measures were then put forward. This convalidation step resulted in very useful feedback because it allowed the optimization of the indicators and of the tool in its completeness. The selection of the case studies was based on the quality of the information available. The case studies were evaluated in two moments: T0, or ex-ante, evaluated the most prevalent management systems used at a local level defined as management techniques before the launch of the Presidia Project, which represents the reference phase; T1, or ex-post, evaluated the management strategies of the resources applied after a generally brief period defined by the launch of the Project, ranging from 4–6 years.

3. Results and Discussion

The selection of the systems to be studied and their objective comparison was carried out through a longitudinal approach (over time) in the attempt to capture the dynamic aspects if the small-scale agri-business system while simultaneously seeking to overcome the difficulty in evaluating the data over the long term [33].

The possibility of evaluating agri-food systems with a longitudinal approach deserves, in our opinion, more exploration, since it provides an opportunity of evaluating the real effects of alternative management on indicators and components, thereby significantly improving the understanding of the systems’ dynamics.

3.1. Dimensions and Components

For the socio-cultural dimension, considering the wide debate on the theme, we have referred to Ballet et al. [54] who defines socially sustainable development as that which “guarantees for both present and future generations an improvement of the capabilities of well-being (social, economic or environmental) for all, through the aspiration of equity on the one hand-as intra—Generational distribution of these capabilities—And their transmission across generations on the other hand”.

This social-cultural aspect has been conceptualized into four components:

-

Product: Reference is made to the improvement of the intrinsic characteristics of the product (conservation, transformation, and organoleptic quality) as well as to the knowledge of the same at a regional level, most of all in reference to the concept of a regional product proposed by Guerrero et al. [45].

-

Internal relationships: The reference is to the horizontal dimension of the social capital and thus to the various networks and trust relationships between the involved people in the small-scale agri-food supply chain. These relationships can reinforce the social cohesion and the stability inside a group of people, organizations or a society in general.

-

External relationships: The vertical dimension of the social capital and thus the capacity to take advantage of the resources, ideas and information that come from formal and informal institutions outside of the reference community [55].

-

Culture: Reference is made to two intrinsic dimensions (natural and human factors) of local products, which are perceived in a particular way by the consumers that appreciate the uniqueness and the legitimacy of the local community in the management of the product [56].

The agro-environmental dimension was defined by five components. The first three (biodiversity, region, soil and water) refer to the functions inside of a stable agro-ecosystem, which can be assured by the optimization of the production conditions. In this sense, the quality of the natural resources (air, soil and water) comes into play, as well as the conservation of a large ecological foundation that is achieved through the maintenance and use of biodiversity [35]; the component defended by the crops refers to the reduction of external inputs such as agro-chemical products in favor of methods implemented for a more efficient use of both internal and external resources (energy component).

In the economic dimension, the components take into consideration the development in the production factors and their efficiency/dynamism into consideration, as well as the market approach toward diversification and the quest for new selling channels of the group of producers (Table 2).

Table 2. Structure of the SAEMETH method.

Level 1: Dimension	Socio-cultural	Agro-environmental	Economic
Level 2: Component	Product	Biodiversity	Development
	Internal relationships	Region	Efficiency/Dynamism
	External relationships	Soil and water
	Culture/Terroir	Crop defence
		Energy
Diagnostic area
Level 3: Indicator (number of indicators)	22	20	10

Table 2. Structure of the SAEMETH method.

Level 1: Dimension	Socio-cultural	Agro-environmental	Economic
Level 2: Component	Product	Biodiversity	Development
	Internal relationships	Region	Efficiency/Dynamism
	External relationships	Soil and water
	Culture/Terroir	Crop defence
		Energy
Diagnostic area
Level 3: Indicator (number of indicators)	22	20	10

3.2. The Indicators

Table 3 sums up the indicators that we have designed for each of the components. The indicators at Level 3, specifically, regard the fruit trees sector, even if many of them are also valid for other agricultural sectors.

3.3. Application of SAEMETH to 10 Case Studies

We have grouped the scores of the various sustainability indicators together in SAEMETH. As a result, we have developed a user-friendly instrument that can function as an aid in the management of the supply-chain by farmers. In addition, the particular attention that we have given to the communication and ease-of-use aspects has allowed us to propose a monitoring instrument whose application seems particularly interesting in farmer discussion groups for the comparison and exchange of knowledge and skills.

How the considered case studies’ sustainability was monitored is shown in the bar graph (Figure 1), which shows the various dimensions. The radar graph shows all of the compared components (Figure 2). The most innovative elements regard the socio-cultural dimension (social reinforcement of the producers, visibility of the product, growth of self-esteem). It is possible to observe (Figure 1 and Figure 2) how a substantial increase in socio-cultural sustainability became evident for all of the case studies, due to the considerable increase in the relationships inside the group of producers and between these and the outside world, from the greater consciousness of the producers regarding the value of their own work and their own product (thanks to their participation in numerous events and communication efforts). In some cases, we can also add the formation of an association between producers, which improved their organization as well as their contracting power. These aspects are in line with studies by other authors [57,58] that have demonstrated that social ties and the sharing of norms between participants are key factors for the sustainable management of common resources. A particularly interesting aspect in the socio-cultural area regards the characteristics of the relationships that are created and reinforced over time. These relationships are never one-directional, instead they always assume a circular form developing a virtuous progress thanks to a network of subjects (producers, connoisseurs, vendors, chefs, technicians, teachers, students, journalists, gastronomy enthusiasts, buying-groups, etc.) that, in various ways, collaborate to conserve, improve, and promote the products and to spread the word about them and their producers and regions in contexts that are also far away from there production location.

Table 3. Level 1 (Dimension), Level 2 (Component), and Level 3 (Indicator) for SAEMETH, together with a concise definition and identification of the type of data used.

Level 1 Dimension	Level 2 Component	Level 3 Indicator	Indicator Definition	Data Type *	Indicator Weight **
Socio-cultural	Product use	Conservation	Improvement of conservation quality	b	6.25
		Transformation	Rediscovery or experimentation with transformed products	b	6.25
		Organoleptic quality	Improvement of the organoleptic quality	b	6.25
		Consumption	A greater diffusion of the product/consumption	b	6.25
	Internal relationships	Role of younger generations	% of young people on the total of producers/processors in the Presidium	a	3.57
		Role of women	% of woman on the total number of producers/processors in the Presidium	a	3.57
		Organization of the producers	Presence/ absence of an organization of producers into a recognized Presidium/ Consortium	b	3.57
		Decision-making structure	Transparency and clarity between the producers and the Presidium	b	3.57
		Participation of the producers	Participation of the producers in decision-making processes through meetings and contacts	a	3.57
		Knowledge sharing	Sharing decisions and choices	b	3.57
		Educational opportunities for producers	Further education for producers and processors	b	3.57
	External relationships	Relationships with public and private institutions	Improvement of the relationships with public institutions and private entities and the possibility of influencing public policy	b	4.17
		Relationships with food networks	Relationships and integration into the local, national and international food-network	b	4.17
		Relationships with the media and communication	Greater media attention for products and their regions	b	4.17
		Relationships with the consumer	The possibility for consumers to know products and their regions though labelling	b	4.17
		Events	Participation in events related to the Food Network	a	4.17
		Other events	Participation in events outside the Food network (festivals, fairs, etc.)	a	4.17
	Culture/terroir	Product-Region identity	Reinforcement of the consciousness of the tie between product and region	b	5
		Architectural cultural assets	Promotion and restoration of historical and culturally relevant regional architectural elements	b	5
		Horizontal transmission of knowledge	Improvement of the community’s (not just local) consciousness of the necessity of preserving the product and the landscape	b	5
		Vertical transmission of knowledge	Recognition of the role of older generations	b	5
		Tourism development	Birth and development of tourism activity in the region tied to the product (festivals, congresses, dinners, etc.)	a	5
Agro-environmental	Biodiversity	Variety	Contribution to the conservation of local varieties/breeds	b	4
		Transformation techniques	Preservation of traditional production/processing methods	b	4
		Landscape	Conservation of a particular threatened rural landscape (gardens, historic fruit orchards, thousand-year-old olive orchards)	b	4
		Seeds	In-house and/or local production of propagation material	b	4
		Intercropping	Intercropping with other plant species	a	4
	Region	Diversification of products	Increase in product diversification	a	10
		Restoration and conservation actions	Promotion of the conservation of agricultural land and the continued use of traditional farming methods	b	10
	Soil and water	Rotations	Crop rotations	a	5
		Irrigation	Water conservation and an efficient use of resources	b	5
		Fertilization	Use of synthetic chemical fertilizers	a	5
		Organic fertilization	Use of natural fertilizers	a	5
	Crop defence	Defence products	Use of synthetic chemical fertilizers	a	2.86
		Natural defence products	Use of natural products	a	2.86
		Weed control	Use of synthetic chemical products	a	2.86
		Natural weed control	Use of natural products and/or techniques	a	2.86
		Post-harvest treatments	Use of chemical treatments	a	2.86
		Natural post-harvest treatments	Use of natural products	a	2.86
		Certifications	Organic or bio-dynamic certification	a	2.86
	Energy	Renewable energy	Use of renewable energy sources	a	10
		Packaging material	Use of minimal or recycled/recyclable packaging	a	10
Economic	Development	Area	Increase/decrease of the area	a	12.5
		Number of producers	Increase/decrease of the number of producers	a	12.5
		Quantity produced	Increase/decrease of the quantity produced	a	12.5
		Enlargement of the business	Enlargement of existing structures or the construction of new ones	b	12.5
	Efficiency/Dynamism	Employment	Increase in the number of (salaried) workers	a	8.33
		Market diversification	New kinds of markets	b	8.33
		New commercial channels	Activation of new commercial channels	b	8.33
		Contracting power of the producer	Increase in producer contracting power	b	8.33
		Sale price	Profitability of the product	a	8.33
		Economic alliances	Partnership duration with other businesses and distributers	b	8.33
Indicator design based on expert opinions and scoring methods based on expert judgement or a comparison to a reference stakeholders’ group

* a = quantitative data b = questionnaire; ** the weights sum up to 100 for each Level 1 dimension.

Table 3. Level 1 (Dimension), Level 2 (Component), and Level 3 (Indicator) for SAEMETH, together with a concise definition and identification of the type of data used.

Level 1 Dimension	Level 2 Component	Level 3 Indicator	Indicator Definition	Data Type *	Indicator Weight **
Socio-cultural	Product use	Conservation	Improvement of conservation quality	b	6.25
		Transformation	Rediscovery or experimentation with transformed products	b	6.25
		Organoleptic quality	Improvement of the organoleptic quality	b	6.25
		Consumption	A greater diffusion of the product/consumption	b	6.25
	Internal relationships	Role of younger generations	% of young people on the total of producers/processors in the Presidium	a	3.57
		Role of women	% of woman on the total number of producers/processors in the Presidium	a	3.57
		Organization of the producers	Presence/ absence of an organization of producers into a recognized Presidium/ Consortium	b	3.57
		Decision-making structure	Transparency and clarity between the producers and the Presidium	b	3.57
		Participation of the producers	Participation of the producers in decision-making processes through meetings and contacts	a	3.57
		Knowledge sharing	Sharing decisions and choices	b	3.57
		Educational opportunities for producers	Further education for producers and processors	b	3.57
	External relationships	Relationships with public and private institutions	Improvement of the relationships with public institutions and private entities and the possibility of influencing public policy	b	4.17
		Relationships with food networks	Relationships and integration into the local, national and international food-network	b	4.17
		Relationships with the media and communication	Greater media attention for products and their regions	b	4.17
		Relationships with the consumer	The possibility for consumers to know products and their regions though labelling	b	4.17
		Events	Participation in events related to the Food Network	a	4.17
		Other events	Participation in events outside the Food network (festivals, fairs, etc.)	a	4.17
	Culture/terroir	Product-Region identity	Reinforcement of the consciousness of the tie between product and region	b	5
		Architectural cultural assets	Promotion and restoration of historical and culturally relevant regional architectural elements	b	5
		Horizontal transmission of knowledge	Improvement of the community’s (not just local) consciousness of the necessity of preserving the product and the landscape	b	5
		Vertical transmission of knowledge	Recognition of the role of older generations	b	5
		Tourism development	Birth and development of tourism activity in the region tied to the product (festivals, congresses, dinners, etc.)	a	5
Agro-environmental	Biodiversity	Variety	Contribution to the conservation of local varieties/breeds	b	4
		Transformation techniques	Preservation of traditional production/processing methods	b	4
		Landscape	Conservation of a particular threatened rural landscape (gardens, historic fruit orchards, thousand-year-old olive orchards)	b	4
		Seeds	In-house and/or local production of propagation material	b	4
		Intercropping	Intercropping with other plant species	a	4
	Region	Diversification of products	Increase in product diversification	a	10
		Restoration and conservation actions	Promotion of the conservation of agricultural land and the continued use of traditional farming methods	b	10
	Soil and water	Rotations	Crop rotations	a	5
		Irrigation	Water conservation and an efficient use of resources	b	5
		Fertilization	Use of synthetic chemical fertilizers	a	5
		Organic fertilization	Use of natural fertilizers	a	5
	Crop defence	Defence products	Use of synthetic chemical fertilizers	a	2.86
		Natural defence products	Use of natural products	a	2.86
		Weed control	Use of synthetic chemical products	a	2.86
		Natural weed control	Use of natural products and/or techniques	a	2.86
		Post-harvest treatments	Use of chemical treatments	a	2.86
		Natural post-harvest treatments	Use of natural products	a	2.86
		Certifications	Organic or bio-dynamic certification	a	2.86
	Energy	Renewable energy	Use of renewable energy sources	a	10
		Packaging material	Use of minimal or recycled/recyclable packaging	a	10
Economic	Development	Area	Increase/decrease of the area	a	12.5
		Number of producers	Increase/decrease of the number of producers	a	12.5
		Quantity produced	Increase/decrease of the quantity produced	a	12.5
		Enlargement of the business	Enlargement of existing structures or the construction of new ones	b	12.5
	Efficiency/Dynamism	Employment	Increase in the number of (salaried) workers	a	8.33
		Market diversification	New kinds of markets	b	8.33
		New commercial channels	Activation of new commercial channels	b	8.33
		Contracting power of the producer	Increase in producer contracting power	b	8.33
		Sale price	Profitability of the product	a	8.33
		Economic alliances	Partnership duration with other businesses and distributers	b	8.33
Indicator design based on expert opinions and scoring methods based on expert judgement or a comparison to a reference stakeholders’ group

* a = quantitative data b = questionnaire; ** the weights sum up to 100 for each Level 1 dimension.

Figure 1. Comparison between T0 and T1 considering the dimensions (Socio-cultural Agro-environmental Economic) for each case study.

Figure 1. Comparison between T0 and T1 considering the dimensions (Socio-cultural Agro-environmental Economic) for each case study.

Another significant element is the importance of the presence of a key figure (who can be one of the producers, but also a veterinarian, an agronomist, a cook, etc.) acting as a local leader that will accompany the development of a project.

In some cases, the support guaranteed by local regional entities seems just as important. These entities guarantee technical and economic support, thus giving a substantial boost to some small-scale supply-chains. In this way, what Brunett-Perez et al. [59] already observed is reinforced, that the well-organized community is the one that is most able to attract external support for their projects.

One of the causes, and at the same time one of the consequences—of the reinforcement of the socio-cultural dimension—is the increase in the number of young producers, even in more marginal and difficult situations.

The agro-environmental dimension often already shows positive values for the indicators at the T0 moment, and for this reason show a slower growth. This is inevitable, because the changes in this environment, by themselves, require a lot of time: the new orchards must have time to grow and produce fruit, the resources to restore the landscape or the traditional architecture require substantial funding and public authorizations, the conversion to organic farming requires a few transition years. However, all of the case-studies show that crop diversification and the preservation of biodiversity improve the capacity of systems to cope with and respond to the fluctuations of their own environment even if both the minimum and optimal level of diversity and the number of species needed to maintain the ecosystemic services still remains unknown and difficult to determine [60]. It seems interesting to note the approach of some of the producers to renewable energy production systems, which are also able to improve the sustainability of an agri-food system over the medium-long term period.

Figure 2. Comparison between T0 and T1 considering the components for each case study.

Figure 2. Comparison between T0 and T1 considering the components for each case study.

In any case, the necessity of also reinforcing educational actions that inspire the younger generations to more consciously protect the natural environment by improving their agricultural practices with the principles of agro-ecology [61] sustains the efficiency of methodologies such as SAEMETH, that in this way become multifunctional and also at the service of civil society.

The economic results obtained (profit growth, and most of all a differentiation of the markets and an increase in the bargaining power of the producers) yet again confirm the efficiency/dynamism of the small-scale agri-food system tied to regional products [45]. It is indispensable to restore and promote the small-scale local product, or those that in any case are strongly marginalized by the market, and guarantee a greater profitability to producers thereby justifying the greater socio-cultural and economic effort they are making.

3.4. Discussion

Through the description of the steps that have lead to the definition of SAEMETH, we have explained the process that, thanks to a continuous exchange with the stakeholders, has allowed us to select the indicators that are considered relevant for the small-scale agri-food system. As was already described, these indicators were chosen based on the availability of data instead of on its solidity and scientific relevance, following the lead of insights already established in the MOTIFS [35] and MESMIS [8] methods. SAEMETH is based on the equality of the economic, agro-environmental and socio-cultural dimensions of sustainability and this equality is intrinsically constructed and developed in the system with the explicit end-result of unifying an overall integral synthetic vision. What is more, the verification carried out with the case studies demonstrated the functionality of the proposed approach in small-scale agri-business systems thanks also to the validation of a holistic vision that is easy to understand and has a notable significance. The application of the SAEMETH framework puts the spotlight on certain issues that justify an ulterior examination and successive development in terms of the choice and evaluation of sustainability indicators. In fact, the explicit attention given to qualitative indicators has necessarily excluded the evaluation of some important indicators from the grid, most of all those tied to specific aspects of those agronomic practices that follow modern agro-ecological approaches, which are able to substantially modify the sustainability of a system.

Furthermore, the necessity of increasing the scope of the studio to confirm or modify the sustainability indicators in relationship to the other agri-food sectors appears to be an important requirement for guaranteeing a larger application of SAEMETH with results that can be used objectively.

The participation of all of the individuals involved in the small-scale agri-food system as well as the consultation of experts in various phases of the methodology have played a fundamental role in the development of SAEMETH as, in general, a tool for the development of sustainability indicators [50]. Dialogue between the interested parties was already considered fundamental to the process of translating shared understandings and interpretations into collective actions for sustainable development by Olsson and Folke [62]. The application of SAEMETH makes this approach very shared.

Furthermore, it must be remembered that the analyzed supply-chain is often in an initial phase of the Slow Food Presidia project and therefore the system undergoing evaluation cannot yet be defined as stable. The data collected on systems that are still in a transition phase does not always readily show the effects of an alternative management, in particular when the idea is to also measure trends over a longer period. For example, to improve their agro-environmental performance, farmers must invest in the acquisition of organic substances in the form of manure or compost or hire more labor for manual weeding [32].

In the end, the best possible way to translate theory into practice must be found. While in some cases the single farmer, association of farmers, and/or the representative of the agri-food supply chain are able to develop adequate strategies for the “correction” of sustainability on their own, in other cases it will be necessary to guide farmer’s actions towards a higher level of sustainability, all in relation to the specific supply-chain and to the specific region [46]. This is also necessary in order to support adaptive strategies of co-management of the regions themselves.

4. Conclusions

In this article, we have proposed a methodological framework that has lead to the development of SAEMETH, a monitoring instrument meant to guide the small-scale agri-business system towards a higher level of sustainability. Furthermore, we have illustrated its practical application in a restricted number of case studies. The participation of the interested parties and the consultation of experts have played an important role in each of the methodological steps taken. In our opinion, the final-use validation of this instrument is of critical importance for its optimization and for its continued improvement. For this reason, we encourage its application in other small-scale agri-food systems, even if not all of the indicators have been elaborated in detail in this phase. Today, the described methodological framework is being actively applied in 70 European situations and, through the construction of appropriate (sector-based) indicators, it has been applied to all of the different types of small-scale agri-food systems thanks to the specific interest of the European Union for this kind of methodological framework. The interdisciplinary work group is following-up their research in order to overcome some of the difficulties made evident through a global strategy that involves the development and dissemination of educational resources as well as further research on the evaluation of sustainability; all of this in order to undertake the evaluation of other case-studies, not just in Italy, but of developing a database of the sustainability of the small-scale agri-food supply chain and to allow for the exchange of experiences with other research groups.

This activity will provide precious feedback and greater experience in the application of the method for the organization of a more complete evaluation structure, bringing improvements on both the more critical theoretical as well as operative aspects.

In the end, the hope is that this study will contribute to increasing the sustainability of the small-scale agri-food supply chain, providing information on the kind of indicators selected, on the benefits of the management systems of alternative resources and on the possible common strategies for pursuing sustainable development.