Next Article in Journal
The Impact of Future Offshore Wind Farms on Wind Power Generation in Great Britain
Previous Article in Journal
Wind Energy Integration through District Heating. A Wind Resource Based Approach
Article Menu

Export Article

Resources 2015, 4(1), 128-154; doi:10.3390/resources4010128

Article
Towards a Theoretical Grounding of Climate Resilience Assessments for Smallholder Farming Systems in Sub-Saharan Africa
Jami L. Dixon ,* and Lindsay C. Stringer
Sustainability Research Institute, School of Earth and Environment, University of Leeds, Leeds LS2-9JT, UK
These authors contributed equally to this work.
*
Author to whom correspondence should be addressed; Tel.: +44-113-343-5572.
Academic Editor: Richard Kazmierczak Jr.
Received: 29 September 2014 / Accepted: 23 January 2015 / Published: 13 March 2015

Abstract

: Resilience assessments are increasingly used to inform management decisions and development interventions across sub-Saharan Africa (SSA). In light of current and future climate change and variability, there is growing interest in applying such tools and frameworks to assess and strengthen the climate resilience of smallholder farming systems. However, these assessments are often undertaken without explicit consideration of the resilience thinking in which they are grounded. This makes it difficult to understand how the conceptual aspects of resilience are translating into resilience assessment practice. This paper provides an important first step in tackling this gap, by identifying and using key characteristics of resilience thinking to evaluate existing resilience assessment tools and frameworks and drawing insights for assessing the climate resilience of smallholder farming systems. We find that power, politics, and agency, identified as important in the resilience literature, are not fully incorporated within current tools and frameworks. This leads to inadequate consideration of spatial and temporal trade-offs. We propose six recommendations for assessing the climate resilience of smallholder farming systems in SSA in order to enhance the linkages between resilience theory and practice. These are: (1) better integrate vulnerability and resilience; (2) recognize that resilience does not equal development or poverty reduction; (3) recognize the benefits and limitations of adopting flexible, participatory approaches; (4) integrate issues of power into assessment tools; (5) target specific systems; and (6) encourage knowledge sharing, empirical studies, and critical evaluation. Our findings contribute to improved understanding of applications of resilience thinking to enhance natural resource management.
Keywords:
agriculture; vulnerability; adaptive capacity; adaptation; climate change; development

1. Introduction

Researchers, practitioners, and policy makers employ a range of tools and frameworks to enhance the design, implementation and monitoring of policies and programs to increase resilience. Such tools play a crucial role in enabling strategic choices about funding priorities. They also provide evidence for natural resource management and for monitoring and accountability purposes, regarding where investments can build resilience. Smallholder farming systems represent one arena in which there is increasing interest in developing and applying such tools, largely due to concerns about food security and the impacts of climate change on natural resources. This reflects a broader shift to applying resilience thinking to develop multiple tools, frameworks and methodologies, with a view to assessing and strengthening resilience [1,2].

While resilience is accepted as important and, despite critiques, is generally seen as a positive attribute [3], several gaps remain in moving between resilience thinking and resilience assessment. One particularly pressing challenge is that of scarce empirical evidence that critically evaluates the processes and outcomes from resilience assessments [4]. Another challenge is the lack of incorporation of published research on the key aspects of resilience thinking into a widely applicable resilience measurement framework [5]. Indeed, research has identified important factors that strengthen resilience of a range of different systems [6], yet this information is often only marginally used among practitioners or in policy domains [7]. This paper contributes towards addressing this gap, and aims to provide a more substantive theoretical grounding to the development of future climate resilience assessments for smallholder farming systems.

Scientific awareness and understanding about the potential consequences of climate change and variability have improved significantly since the 1980s, with wide acknowledgement of the potential for climate impacts to affect agricultural production and food security [8,9]. Rain-fed agriculture has been repeatedly identified as particularly sensitive to climate changes and variability [10,11,12,13]. Sub-Saharan Africa (SSA) is projected to be disproportionately affected by the impacts of future climate change and variability due to a high dependence on rain-fed agriculture for food, income and economic growth [14]. At the same time, there are growing concerns that smallholder farmers across SSA, who depend on rain-fed agriculture as their main source of livelihood, may not possess the necessary capacity to cope with and adapt to current and future climate impacts [15,16]. One approach to assist smallholder farmers is to increase their social, economic, and ecological resilience.

Farming systems, also referred to as agro-ecosystems, are considered as social-ecological systems (SESs) in the resilience literature, and incorporate both biophysical and human components [17]. A farming system comprises multiple diverse, individual, but interacting, individual farms [18]. Understood in this way, a farming system is a unit of analysis above individual farm systems [18]. The production and social structures of individual farms are highly diverse, suggesting that the ability of smallholder farmers to respond to climate change and variability will be similarly varied, resulting in different farm-scale climate impacts [19]. Strengthening the resilience of individual farms therefore forms part of a broader goal to increase the resilience of smallholder farming systems in the face of both current and future climate change and variability.

Decision makers are increasingly interested in strengthening the climate resilience of smallholder farming systems. Yet it is unclear what kinds of tools and frameworks that focus on climate resilience already exist and the extent to which they reflect current resilience thinking. Furthermore, there is a growing number of resilience assessments and a growing interest in linking resilience theory and practice [20]. There is thus potential to draw out insights from how characteristics of current resilience thinking are reflected in other practically-applied resilience assessments such that gaps can be addressed in the ongoing development of climate resilience assessments. Herein, tools are defined as including methodologies and approaches aimed at providing practical assessments or measurements of resilience, and frameworks as collections of concepts or ideas that may have been used or applied to inform the development of practical assessments.

The overall aim of this paper is to analyze the resilience literature alongside existing resilience assessment tools and frameworks with a view to guiding a more comprehensive application of resilience thinking. In particular, this paper makes recommendations for future assessment tools and frameworks that focus on assessing the climate resilience of smallholder farming systems in SSA. The paper’s objectives are to: (1) summarize current resilience thinking, setting out the key characteristics relating to resilience in the literature; (2) use a systematic approach to identify existing assessment tools and frameworks from a range of different disciplines and sectors; and (3) evaluate the extent to which tools and frameworks in our sample reflect the key characteristics of resilience thinking identified in objective 1. Key characteristics of resilience that are captured within assessment tools are highlighted and gaps are identified. The paper concludes by drawing out recommendations that reflect the key characteristics of resilience thinking that could better ground the development of future tools that seek to assess the climate resilience of smallholder farming systems in SSA.

2. Literature Review

2.1. Unpacking Resilience

Multiple definitions of resilience exist [21]. It is commonly defined as the ability to bounce back after an external shock or stress [22], where the resilience of a system is demonstrated by undergoing disturbance, maintaining system functions and controls, and returning to a stable state [23]. Such definitions of resilience have roots in ecology and ecosystem dynamics [24].

Resilience is often presented as an antonym of vulnerability [25], where vulnerability to climate change is defined by the Intergovernmental Panel on Climate Change (IPCC) as the degree to which a system is susceptible to, or unable to cope with, adverse effects of climate change, including climate variability and extremes [26]. Empirical studies demonstrate that the relationship between resilience and vulnerability is nevertheless complex [7], as systems can be both resilient and vulnerable at the same time depending on the particular shocks and stresses [27]. For example, a farm close to a water body may be vulnerable to flooding yet due to its geographical location be highly resilient to drought. This demonstrates that the type of shock that is experienced matters [28,29].

Resilience has evolved to include the dynamics of social and ecological systems and has been applied as a lens through which we can understand and examine how SESs respond to shocks, stresses or perturbations [30]. This highlights the importance of understanding the resilience of a system (or part of a system) in relation to a specific shock, stress or perturbation [25], also referred to as specified resilience [31]. However, building specified resilience has limitations. For example, focusing on a specific shock or part of a system can cause the system to lose resilience in other ways or be less resilient to other shocks. It also relies on knowing or being able to predict the nature of a specific shock, which is particularly problematic in the face of increasingly variable, dynamic and uncertain climate conditions [32].

While it is possible to assess resilience after a shock or stress, it can also be assessed before exposure takes place through focus on the “inherent characteristics or qualities of social systems that create the potential for harm” [5] (p. 599). Resilience can be considered an intrinsic system property or process [33]. Berkes and Seixas [34] refer to this as “general resilience”. General resilience does not define a particular part of the system or identify specific shocks [31]. Empirical studies of general resilience in SESs demonstrate that the social, economic and ecological context of a system is important [35] and further highlight the centrality of financial, political, and institutional factors [36]. Building general resilience is considered highly desirable in coping with and adapting to climate change and variability [20], and many resilience assessment results are used to inform investments in general resilience.

To strengthen the resilience of an individual or system, past exposure to shocks and stresses is considered essential [37]. Where exposure has taken place, resilience research demonstrates that historically, individuals and systems have been able to successfully negotiate challenges or adverse events [38]. Central to this understanding is the notion that resilience is maintained by disturbing and probing at its boundaries [24,39,40]. This implies that all individuals or systems can learn from past exposure, suggesting that processes such as social learning are important [41,42].

Indeed, many authors note that in its recent application, resilience requires flexibility, learning and ability to deal with change [43,44]. Here a distinction is drawn between adaptability and transformability as different aspects of resilience [31,45]. Adaptability refers to the capacity to deal with change and maintain system functions and structures, whereas transformability is “the capacity to create a fundamentally new system when ecological, economic, or social structures make the existing system untenable” [45] (p. 5). Studies demonstrate the importance of including transformability as a characteristic of a resilient system [31,46]. In line with current resilience thinking, this paper understands resilience to have three dimensions: persistence, adaptability and transformability [16,31]. It proposes that general climate resilience, rather than resilience to specific climate shocks, is a desirable farming system characteristic in the face of changing climatic conditions. It also recognizes that strengthening such climate resilience could contribute to building general resilience.

Resilience thinking recognizes the interconnectedness and interdependence of dynamic and interacting factors that comprise SES [3,44,47]. SESs are predisposed to change rather than equilibrium and are therefore an example of a complex adaptive system, characterized by uncertainty and surprise [48]. The adaptive cycle, which identifies phases of exploitation, conservation, release and reorganization, is a key contribution of resilience thinking, and helps to describe the ability of SESs to deal with uncertainty and surprise. Multiple adaptive cycles, which involve feedback loops and interplay between cycles across scales, are nested within a SES. The existence of such cycles has been described as panarchy [23]. The adaptive cycle and panarchy can be applied to understand the resilience of SESs [21,49], in particular how SESs deal with uncertainty and surprise [48].

A key criticism of resilience thinking is that it often presents resilience as a positive attribute, yet this overlooks that a resilient system may bounce back to an undesirable state [50,51]. For example, areas depleted of natural resources are often extremely resilient but may provide little in terms of income, food, or other ecosystem services. Moreover, a system with highly polluted water supplies or governed under a dictatorship may be highly resilient, but unjust, undesirable, or bear high economic and social costs [52]. Defining resilience as robustness or ability to bounce back to a prior state can maintain the status quo and allow unsustainable or socially unjust practices to continue [46]. In practice, system transformations may be required to overcome social and economic injustices [46]. This highlights the growing normative debates that emphasize the importance understanding the resilience for whom [4] and of including transformability in our understanding of resilience.

Resilience thinking has been further criticized for its inability to adequately capture social dynamics related to power and agency [48]. This is because agency, the ability of individuals to exercise a degree of choice or autonomy over their own lives, is often veiled in resilience assessments that focus on a household, community or system scale [5]. For example, a household may act to strengthen their overall resilience, but at the detriment of an individual’s well-being [53]. Similarly, resilience assessments may take place at the system level, masking impacts at other, smaller scales. This limitation is common to all system-oriented approaches [4], where the focus is on the SES, rather than on the choices made by individuals or groups within the system [53]. This suggests it is vital to highlight the importance of scale and the notion of “winners and losers” when assessing resilience [54].

In the context of climate change, actions that erode longer-term resilience may be termed “mal-adaptations” [55], defined as any response that is not sustainable or which increases vulnerability [27]. In the context of smallholder farming, the sale of farm assets during drought to enable coping over the short term, for example, can undermine smallholders’ abilities to re-engage in future agricultural production. Resilience assessments need to recognize that both spatial and temporal trade-offs may be encountered [38].

Resilience is also sometimes confused with development. The literature suggests that resilience can act as a barrier to development, whilst development may undermine resilience [56], through for example, the construction of houses on flood plains. Recognizing and understanding the limitations to resilience is therefore imperative. Yet, building resilience is increasingly recognized as a central development objective alongside poverty reduction and economic growth [56].

This section has demonstrated that resilience research highlights a number of considerations that should be borne in mind when using resilience thinking as a conceptual framing for climate resilience assessments. A summary of the key characteristics of resilience that emerge from our literature review is presented in Table 1. They were identified as reoccurring characteristics across multiple papers in the reviewed literature and were distilled to reflect the current state of resilience thinking. Identifying characteristics grounded in current resilience thinking provided the boundaries for this study.

The next section explores approaches to assessment, including the assessment of resilience, before focusing on the climate resilience of smallholder farming systems.

Table 1. Characteristics of resilience identified from the resilience literature to reflect the current state of resilience thinking.
Table 1. Characteristics of resilience identified from the resilience literature to reflect the current state of resilience thinking.
Key CharacteristicsIndicative References
  • Holistic approaches are required to understand interactions, interconnectedness and interdependence between human and biophysical components of a single complex system

[6,44]
2.

Resilience is not only an antonym of vulnerability

[7,25]
3.

Resilience can be an intrinsic system property or process, independent of exposure to a shock or stress

[33,45]
4.

Resilience indicators should include social, financial, political, and institutional considerations

[35,36]
5.

Resilience requires flexibility, learning and transformability, where the ability of individuals and/or systems to learn from past exposure is important

[31,37,48,50,57]
6.

The adaptive cycle (exploitation, conservation, release and reorganization) and panarchy can be applied to understand the resilience of SES

[33,49]
7.

Resilience is not always a positive attribute

[3,45]
8.

Social dynamics and issues of power and agency should be included

[3,4,48]
9.

Resilience has temporal dimensions and may require trade-offs (e.g., strengthening resilience in the short term may reduce resilience in the long term)

[38,55]
10.

Spatial scale is important in recognizing there may be “winners and losers” and trade-offs between resilience and well-being

[3,53]

2.2. Approaches to Assessing Resilience and Their Application in Farming Systems

Given the complex and dynamic nature of resilience, some researchers have suggested that it is not something that can be meaningfully quantified or measured [58]. Critics consider the findings from resilience assessments to be too contextual to be useful in informing wider scale decisions. Despite this, several relevant tools aimed at assessing resilience at different scales have been developed [1] and applied across a wide range of sectors, including agriculture. The benefits of adopting a resilience framework in understanding farming and other natural resource based systems are well established in the literature [44,50,51,59] and incorporate a range of different assessment methods. This section outlines common assessment approaches.

Assessments of issues such as natural resource degradation note the importance of capturing both the social and ecological context [60] and that participatory methods can play a key role in achieving this [61]. Research from around the world shows the benefits of stakeholder participation and local community engagement in generating contextually relevant information and implementing projects that meet local needs [62]. Furthermore, the benefits of local knowledge and community participation in monitoring and assessing social and environmental changes, including changes in resilience, are well established in the literature [63,64,65]. Indicators based on local data, developed through participatory means, provide a practical form of assessment, making it possible to identify gaps and monitor progress on the ground [66]. This offers the potential for the flexible, participatory and integrated approaches required to deal with climate changes [67]. Such approaches have been widely applied in the assessment of SESs [36,68]. They can also foster learning which, as noted earlier, can help to build resilience [41].

Systems-approaches first emerged in the 1980s to understand and explore smallholder farming across the developing world [69,70]. Assessments that employ systems-approaches recognize interconnectedness of social, economic, political and institutional processes across temporal and spatial scales. Furthermore, such approaches have also been linked to farmer participation, innovation and learning [71]. Resilience thinking inspires systems-based approaches to analyzing systems with both social and ecological components, and in doing so, recognizes the importance of interconnectedness and interdependency and the context in which they are embedded [44]. The benefits of bringing together farming systems approaches and resilience thinking have been established by existing studies [72,73]. There is potential to build on this further to strengthen climate resilience of smallholder farming systems.

In the context of climate change and variability, climate resilience, particularly in rain-fed systems, is an important property of smallholder farming systems. Whilst general resilience may also act as a barrier to change [50,51], climate resilience may better enable smallholder farming to survive and thrive in the face of the future climate uncertainties. We define climate resilience as the resilience of a system or part of a system to climate-related shocks and stresses, i.e., the ability to survive, recover from, and even thrive in changing climatic conditions [74], and in the process, maintain essential functions, identities and structures [75].

3. Research Design and Methodology

This section outlines the multi-step approach we took to identifying resilience assessment tools and frameworks and explores how the key characteristics of resilience identified in our literature review have been incorporated therein (Figure 1).

Alongside the literature review of resilience research (Phase 1 in Figure 1); an international stakeholder workshop on resilience assessment tools was convened in May 2013 (Step 3 in Figure 1). International expert participants (both academics and practitioners) were asked to suggest an initial sample of relevant tools and frameworks for this research, both during the workshop and as part of an ongoing process (May–December 2013). Workshop approaches have been used effectively in previous research to bring together different knowledges, providing a valuable starting point for further work [76].

Tools that were identified by participants during and after the workshop had varying degrees of relevance for farming systems but nevertheless provided a useful insight into some of the tools, frameworks and approaches used in practical assessments. To identify further tools and frameworks, we selected search terms from the literature review (Step 4 in Figure 1) and used these to conduct a systematic internet search using a search engine (Google, November 2013). The strengths of using this kind of systematic process are outlined in the literature [77].

Figure 1. Flow chart showing systematic approach to tool/framework identification and analysis. Processes are in white text boxes, number (n) of tools/frameworks in circles.
Figure 1. Flow chart showing systematic approach to tool/framework identification and analysis. Processes are in white text boxes, number (n) of tools/frameworks in circles.
Resources 04 00128 g001 1024

Google was initially selected as a search engine over an academic database in order to capture those tools and frameworks that had been or are being developed and used by practitioners, not just academics. To create a manageable sample through the search process, we considered the first 20 results returned by Google that emerged during each iteration of the search terms. This yielded results from sectors as diverse as psychology and business to climate resilient cities. To narrow down the sample for further analysis and enhance its relevance to the assessment of resilience in farming systems, we next applied a set of criteria (Table 2). To be included for further analysis, tools and frameworks had to meet two or more of the criteria set out in Table 2. In total, our combined searches and the expert workshop resulted in an overall set of 25 relevant tools and frameworks. We narrowed the sample further by focusing on tools and frameworks that had supporting written documents that were available in English, resulting in 23 relevant tools and frameworks.

Table 2. Selection criteria and search terms used to identify tools/frameworks.
Table 2. Selection criteria and search terms used to identify tools/frameworks.
Selection CriteriaJustification for Selection
  • Applicability in smallholder farming systems in sub-Saharan Africa (SSA)

SSA is projected to be disproportionately affected by the impacts of future climate change and variability and have a high proportion of smallholder farmers.
2.

Evidence of use in multiple countries/used by international organizations

Provides an indication of the coverage, utility and/or acceptability of the tool.
3.

Specific to agriculture/farm systems/climate resilience

Agriculture is an important sector, both in terms of adaptation and mitigation, and in terms of food security. Climate resilience is one way to reduce vulnerability to the uncertainties surrounding future climate change.
4.

Evidence of peer review

Peer review implies that an attempt has been made to link theory and practice.

Using a non-academic search engine to generate results and applying a series of relevance criteria provided a sample of tools and frameworks that have been used in practice or explicitly aim to inform practical assessments. Using a systematic approach to this ensured that the breadth of tools and frameworks used in practice was captured. Not all of the tools focused specifically on resilience, but we kept them in the sample as they matched other relevance criteria. Including this wider sample provided a way to identify lessons from a wider range of relevant assessment tools.

We analyzed the content of the 23 selected tools and frameworks (Phase 3 in Figure 1), categorizing them according to their scale of analysis, their goal and to the extent to which they focused on climate, vulnerability and/or resilience. This analysis provided an overview of the characteristics of existing tools and frameworks that have been used in practice or explicitly aim to inform practical assessments.

The next step was to search for each tool/framework in Web of Science (Phase 4, in Figure 1). Evidence of academic peer review was used as an indicator of an explicit attempt to link theory and practice. By the end of this process, two frameworks remained that specifically focused on resilience. The content of these frameworks was qualitatively analyzed in relation to the key characteristics identified from the resilience literature. The selected frameworks had both been identified in the Google search or by experts, and had undergone peer review, as evident from the Web of Science search. Focusing in on these two specific frameworks which have attempted to link resilience theory and practice allowed us to highlight important recommendations for the design and implementation of more theoretically grounded assessments. If either framework did not address one of the key issues identified in the literature, we referred back to our larger sample of tools and frameworks to identify how they incorporate the issue. This process was useful in enabling comparisons across tools and frameworks. Learning from other tools and frameworks provides a means to address the limitations of current resilience assessments.

4. Results: Frameworks Linking Resilience Theory and Practice

The two peer-reviewed resilience frameworks that attempt to operationalize resilience thinking and that formed the basis of our analysis were those developed by Tyler and Moench (2012) [32] and Cabell and Oelesfe (2012) [20]. Although one of the frameworks is not directly concerned with resilience of smallholder farming systems, which are predominantly located in rural areas, it explicitly links resilience theory and practice. Both of our case study frameworks identify and discuss elements, indicators and characteristics of resilient systems. This section presents results from the qualitative content analysis of these case study frameworks to provide insight into how resilience theory has been operationalized in practice. Comparing these with the key characteristics identified from our review of the resilience literature enables the identification of existing gaps, potential ways to strengthen links between resilience theory and practice, and, following referral back to our larger sample of tools, allows identification of recommendations for the further development of resilience frameworks and assessment tools. The section begins with a short description of each framework.

Tyler and Moench [32] developed a practical, conceptual framework for assessing urban climate resilience (Tool 11 in Appendix Table A1). The framework highlights three elements of urban resilience: systems, agents and institutions. For each of these, they propose resilience characteristics from a diverse body of literature, which they go on to describe and exemplify to develop a conceptual framework to operationalize in the context of local, urban planning. Characteristics of a resilient system are flexibility and diversity; redundancy and modularity; and safe failure. Agents in a resilient system should be: responsive and capable of reorganization, planning and responding in a timely manner; resourceful and able to mobilize assets for action; and have the capacity to learn and internalize past experiences. Institutions should: provide an inclusive structure for rights and entitlements; have transparent, representative and accountable decision-making processes; provide relevant information; and encourage the application of evidence and new knowledge. These characteristics of resilient urban systems are presented as guidelines for thinking about complex urban systems rather than technical prescriptions. The urban climate framework was found to be used by Asian Cities Climate Change Resilience Network (ACCRN), yet limited review or evaluation of the practical testing or application of this framework was found.

In the second framework, Cabell and Oelesfe (2012) compiled 13 behavior-based indicators of agro-ecosystem resilience from the literature, to include both ecological and social elements (Tool 22 in Appendix Table A1). The indicators are: socially self-organized; ecologically self-regulated; appropriately connected; high degree of functional and response diversity; optimally redundant; high degree of spatial and temporal heterogeneity; carefully exposed to disturbance; responsibly coupled with local natural capital; reflected and shared learning; globally autonomous and locally interdependent; honors legacy while investing in the future; builds human capital; and reasonably profitable. Each of the indicators is linked to phases in the adaptive cycle: growth/exploitation, conservation, release, and reorganization/renewal [23] For each of the 13 indicators, a description and a practical example is provided.

Indicators are used to determine resilience of the agro-ecosystem and can be used to monitor changes; for example an absence or disappearance of an indicator suggests vulnerabilities and movements away from resilience. The framework does not focus on assessing the resilience of the system to a specific shock or stress, and instead identifies rules of thumb to guide farmers and other stakeholders in the agricultural sector [20]. It integrates indicators into the adaptive cycle, can be applied at multiple spatial scales, and is appropriate for both current and future assessments. Although the list of indicators is theoretically grounded, we found no evidence that this framework has been operationalized or tested.

The results from qualitative content analysis of the frameworks in relation to the key characteristics identified in the resilience literature are presented in Table 3. Analysis shows that both frameworks recognize the interactions and interconnectedness between the human and biophysical components of a complex system. Both frameworks conceptualize resilience as an antonym of vulnerability, and thus do not fully reflect the breadth of thinking encountered across the resilience literature. We interrogated the wider sample and found that whilst other tools that focus on either resilience or vulnerability may recognize that the relationship between resilience and vulnerability is complex, this was not captured in the indicators, elements or characteristics set out in the tools and frameworks.

Both case study frameworks partially consider financial and institutional considerations, yet political considerations are largely excluded. Whilst agency is recognized as an integral part of a system (Tyler and Moench, 2012), power relations are excluded from indicators and analysis. Tyler and Moench (2012) justify this choice on the grounds that these are not issues specific to climate resilience. We note that both frameworks provide limited space for consideration of power structures and their influence on resilience, suggesting this is a key gap that future resilience assessment tools and frameworks should address. To identify opportunities to learn from other frameworks and tools, we referred back to the larger sample and found that others explicitly incorporate considerations of power and politics, for example the Household Economy Approach (HEA) [78] and recent initiatives framed around the Resilience Assessment Workbook [22] and Climate Smart Agriculture (CSA) [79].

The HEA uses a political economy analysis to enable a deeper understanding of the causes of poverty and food insecurity [78]. The tool includes questions that capture information on policies, institutions and processes and also explicitly considers political and economic interests of different actors. HEA is sensitive to power relationships and conflict in both programming interventions and when making recommendations. The Resilience Assessment Workbook also includes the identification of key formal and informal institutions, facilitates discussion about ability to influence decision-making, and the mapping of power relations and conflicts [22]. The participatory planning process outlined in the CSA framework promotes change and the empowerment of local stakeholders, with a focus on the least powerful [79].

Table 3. Qualitative content analysis of frameworks compared with characteristics identified from the resilience literature. Convergence between the frameworks and literature are not shaded, characteristics that are partially covered by the frameworks are shaded in light grey, and if not represented in the framework, shaded in dark grey.
Table 3. Qualitative content analysis of frameworks compared with characteristics identified from the resilience literature. Convergence between the frameworks and literature are not shaded, characteristics that are partially covered by the frameworks are shaded in light grey, and if not represented in the framework, shaded in dark grey.
Characteristics from the Literature (Table 1)Tyler and Moench, 2012Cabell and Oelosfe, 2012
  • Holistic approaches are required to understand interactions, interconnectedness and interdependence between human and biophysical components of a single complex system

Interconnections within institutions, agents and systems are explicit, but linear relationships between these elements are implied. Feedbacks are not considered.Captured by multiple indicators. Feedbacks are not considered.
2.

Resilience is not only conceptualized as an antonym of vulnerability

Resilience is conceptualized as an antonym of vulnerability.Resilience is conceptualized as an antonym of vulnerability.
3.

Resilience can be an intrinsic system property or process, independent of exposure to a shock or stress

Though climate exposure is highlighted, the conceptual framework implies that resilience can be assessed independent of exposure.Resilience as an intrinsic system property underpins the conceptual framework and all indicators.
4.

Resilience indicators should include social, financial, political, and institutional considerations

Political dimensions excluded. Both formal and informal institutional dimensions are considered. Financial assets included.Political issues conceptualized as external to the system. Both formal and informal institutional dimensions are considered. Financial considerations captured in “reasonably profitable” indicator, specifically focuses on financial independence from subsidies.
5.

Resilience requires flexibility, learning and transformability, where the ability of individuals and/or systems to learn from past exposure is important.

Learning is included on an individual level not mentioned in relation to system. Flexibility is recognized as important and captured by multiple resilience characteristics. Transformability not considered.Learning is captured on a system and individual level, by the “reflective and shared learning” indicator. Flexibility is captured by multiple indicators. Transformability forms part of the definition of resilience, but it is not explicitly captured by any of the indicators.
6.

The adaptive cycle (exploitation, conservation, release and reorganization) and panarchy can be applied to understand the resilience of SES.

Panarchy is not included. The adaptive cycle is not explicitly considered, but learning and governance are recognized as important elements of enabling a system to reorganize.Recognizes that agro-ecosystems move through four phases in adaptive cycle. Links each indicator with phase in the adaptive cycle.
7.

Resilience is not always a positive attribute

Issue recognized in the main text but not captured in the characteristics.Issue recognized in the main text but not captured in the indicators.
8.

Social dynamics and issues of power and agency should be included

Agency is recognized as an integral part of the system, yet power relations are explicitly excluded.Not specified–words power and agency do not appear in paper. Actors are mentioned, but not the power relationships between them.
9.

Resilience has temporal dimensions and may require trade-offs (e.g., strengthening resilience in the short term may reduce resilience in the long term)

Not includedIssue recognized in the main text. It is implied that the proposed indicators overcome this.
10.

Spatial scale is important in recognizing there may be “winners and losers” and trade-offs between resilience and well-being

Indicators capture multi- scalar dimensions of system. Consideration of winners and losers not included.Indicators capture multi- scalar dimensions of system. Focuses on system rather than individuals or groups of winners and losers. However, trade-offs between indicators are noted, but it is unclear what the implications of this are for resilience.

Qualitative content analysis of our two case study frameworks shows that flexibility and learning and change are embedded in both, along with recognition that resilience can be thought of as an intrinsic system property, independent of exposure to a shock or stress. System transformability is considered by Cabell and Oelsofe [20], although they do not explicitly identify which indicators are important for adaptability or transformability. Tyler and Moench [32] do not consider transformability. The Resilience Assessment Workbook provides space to consider if transformation is needed and a way of identifying and assessing what kind of strategies may be needed to bring about transformational change [22]. There is potential to learn from the processes outlined in the Resilience Assessment Workbook to enhance future climate resilience assessments.

Whilst the frameworks may provide useful ways to assess resilience and operationalize concepts such as flexibility and participation in particular systems, they do not explicitly state how they fit within a wider process of building general resilience. For example, they do not explicitly identify how different stakeholder interests and locally relevant knowledge or bottom-up approaches could be integrated. The expert-led, top-down nature of the frameworks may limit their ability to measure what is important in reality, such that contextual information is missed. Given the dynamic nature of farming systems, which Cabell and Oelsofe [20] describe as like aiming at a moving target, generating contextually relevant information to enhance our understanding of resilience is paramount if the resilience of farming systems is to be strengthened. Of the other tools, the Resilience Assessment Workbook recognizes multi-level governance and includes a process to map power relations [22], offering potential to learn from this to inform future climate resilience assessments.

Conceptually, both case study frameworks recognize the interconnectedness and interdependence of systems, and therefore take into account multi-scalar dimensions. However, limited consideration is given to how certain individuals or groups may be “winners” or “losers”. Tyler and Moench [32] conclude that the central issue of “whose resilience” is not clearly articulated in the urban resilience framework. Referring back to the wider sample, the HEA focuses on social protection and identifying who is or may be marginalized, using a political economic analysis to understand underlying causes and root drivers of marginalization [78]. Similarly, the Social-ecological Inventory provides a structured way to identify key stakeholders and analyze the relationships between them [80].

Neither of our case study frameworks provides a space to reflect upon the potential trade-offs, or synergies, between indicators, or their implications for resilience. Additionally, trade-offs between resilience and poverty reduction or development are not explicitly considered. Although both case study frameworks recognize that resilience is not always positive and that strengthening resilience in the short term may undermine future resilience, neither specifically addresses this when developing indicators or characteristics. Referring back to the wider sample, many of the other tools also fail to identify trade-offs. One exception is the Sustainability Assessment of Food and Agriculture systems tool, which provides a structured way to analyze trade-offs and synergies between elements of sustainability [81]. This tool could usefully inform future climate resilience assessments, guiding the consideration of trade-offs and synergies in both future resilience assessments and assessments targeted at farming systems [82].

Results from the analysis of two case study frameworks therefore demonstrate that there are gaps between resilience thinking and is application through assessment frameworks and tools. Referring back to the wider sample enabled identification of potential ways to learn from other tools and frameworks in order to strengthen the climate resilience of smallholder farming systems.

5. Discussion: Recommendations for Assessing the Climate Resilience of Smallholder Farming System in Sub-Saharan Africa

This section draws on our findings to provide recommendations to strengthen the theoretical grounding of future resilience tools and frameworks that aim to assess the climate resilience of smallholder farming systems in SSA. The potential benefits of this have both theoretical and applied relevance. Before discussing our recommendations, it is important to recognize that other assessment tools and frameworks currently under development may have been excluded from our sample frame. We also recognize that we do not capture all of the practical lessons from relevant tools and frameworks, as our assessment was limited by the amount of data and information available online. This suggests that further lessons may be revealed should an alternative approach be taken, and this represents an important avenue for further work.

Our recommendations are as follows, in no particular order:

(1)

There is considerable potential to better integrate vulnerability and resilience approaches, both in theory and in practice. In the reviewed frameworks and tools, resilience is conceptualized as an antonym of vulnerability, with the links between vulnerability and resilience being overlooked or oversimplified. As such, the frameworks and tools do not capture the range of debates in the academic literature. Furthermore, the complementarities between vulnerability and resilience and the potential to bring them together are ignored in the two case study frameworks, as well as within the wider sample. Although vulnerability and resilience are rooted in different epistemological traditions in the natural and social sciences, there are overlaps in the theory, methodology, and application of the concepts [7].Adger [83] (p. 269) argues that “the points of convergence are more numerous and more fundamental than the points of divergence”. In practice, more consideration is needed regarding how to better identify and build on the synergies between resilience and vulnerability and how this can be integrated into tools that assess the climate resilience of smallholder farming systems in SSA.

(2)

Explicit recognition is needed that resilience is not the same as development or poverty reduction. The complex relationship between resilience and poverty reduction is noted in the resilience literature [3]. Strengthening specific resilience, such as climate resilience, may contribute to increasing overall or general resilience. However, in strengthening climate resilience we should not assume that poverty is reduced [53,56]; this complexity should be addressed by future tools and frameworks. This brings us back to the problem of systems bouncing back to undesirable states, where populations live in poverty, and demonstrates the need to increase capacity for transformation [31]. In practice, we need to be explicit about what strengthening resilience will achieve. This requires a clear definition of what resilience is and what (and who) it includes and excludes. Further research is also required to understand the implications of resilience frameworks and tools and how labelling an individual, community or system as resilient can lead to discursive traps and be used to justify inaction [84]. In light of this, we encourage further debate surrounding the relationship between resilience, development and other development goals, e.g., poverty reduction or sustainable livelihoods.

(3)

Participatory, flexible and learning approaches to planning, implementation and monitoring and evaluation are important; benefits and limitations should be explicit. Resilience approaches foster learning [50], flexibility [85], participation and empowerment [57,86]. Although these characteristics are considered by some of the frameworks in our wider sample, reflection on how to operationalize and measure progress is lacking. Furthermore, fostering participatory processes in practice, and benefits and limitations of participation are often neglected and should be made explicit. One possibility is to integrate processes of learning and social learning into both the development and application of resilience frameworks and tools [43]. Putting this into practice may require multi-disciplinary research teams and wider stakeholder engagement [87]. Smallholder farmers often have significant expertise when it comes to managing their farm system [88]. There is potential to learn from farmers, and where possible, their experience and knowledge should be integrated and enhanced to strengthen the resilience of farming systems. Contextually-relevant information and locally-identified indicators provide a practical way to monitor progress and also increase the potential of generating contextually relevant solutions that can not only increase resilience but also empower farmers in the process [89]. This local level understanding could contribute to Recommendation 4.

(4)

Better integrate issues of power, change and transformation into tools and frameworks. From our analysis of the two case study resilience frameworks and the wider sample, we observed that issues of power, politics and agency are underrepresented. From the wider sample, we found that issues of power and agency are explicitly considered in tools such as the HEA, which do not have such an explicit resilience focus. Learning from these tools could provide lessons on integrating such considerations into climate resilience assessments. This reduces the risk of framing climate change debates in terms of technical and apolitical solutions that ignore notions of equality, social justice and power [7,90]. Pelling [46] proposes that resilience cannot be defined as buffering alone, therefore considerations of power and social justice must be considered and conceptualizations of resilience should capture the capacity for change. In order to assess this ability, certain system properties or capacities, including social, ecological and institutional components, are important and need to be incorporated into assessments. Capacities at an individual or farm level should also be considered. This links to broader questions of how to link resilience, agency and SESs [53,91]. Addressing such issues could also feed into Recommendation 5.

(5)

Resilience tools and frameworks targeting specific systems are needed, recognizing spatial and temporal dynamics and trade-offs. The resilience literature highlights the importance of the contextual factors that shape resilience [2]. However, we identified only one peer reviewed framework specific to agro-ecosystems, and found little empirical evidence to demonstrate how relevant or useful such a framework is in practice. Whilst resilience tools and frameworks designed for other contexts, e.g., urban environments, provide some insights for linking resilience theory and practice, they have largely been tested in a limited geographical area (Asia) and may not be applicable to contextual complexities of elsewhere. We therefore highlight the need for more empirical research and testing to guide which indicators are necessary and to identify to which systems they apply. We also highlight an opportunity for future practical tools and frameworks that focus on specific systems and a space for tools and frameworks that focus on the individual, household or farm level.

Our findings further demonstrate that existing tools and frameworks fail to capture the temporal and spatial dynamics of SESs. Moreover, they do not adequately address or incorporate the spatial and temporal trade-offs that may be required in dealing with climate variability and change. Future frameworks and tools should be theoretically grounded, and also empirically tested to identify the “win wins” for resilience, i.e., the indicators or characteristics which ensure resilience across a range of spatial and temporal scales.

(6)

Encourage knowledge sharing, empirical studies and critical evaluations of resilience and resilience tools and frameworks. Limited data availability on how resilience frameworks and tools are used in practice made it difficult to fully assess some aspects of existing tools and frameworks, for example their geographical coverage and application. There is scope to extend the approach used in this paper to identify further tools and frameworks, and compare the outcomes with the findings presented in this paper. For example, a search engine for academic papers such as Scopus, or different search criteria, could be used to identify additional tools and frameworks.

We found limited critical reflection or evaluation of tools and frameworks that have been used in practice. Given the complex nature of resilience, and other practical constraints such as time and funding, this is perhaps unsurprising. We nevertheless encourage practitioners and academics to carry out and publish critical evaluations of their tools and frameworks and to develop partnerships in order to facilitate learning across tools and reduce duplication of efforts. The wider resilience literature critically engages with the limits to resilience e.g., [3,4,92], yet this was missing in our sample of tools and frameworks. Such practical insights to the limits of resilience could contribute useful additional understanding to resilience thinking.

Our recommendations provide an important first-step to guide climate resilience assessments in smallholder farming systems in SSA. To build the theoretical grounding of resilience thinking, and bridge the gap between resilience thinking and assessment practice, future assessments should continue to consider the interconnections and interactions of human components and natural resources; thus building on one of the major strengths of using a resilience approach. We highlight that there is also potential to learn from other assessment tools and frameworks that address some of the current gaps in resilience assessment, for example, the way in which HEA and the Resilience Workbook incorporate issues of power, marginalization and agency. We also highlight a need to integrate the knowledge and expertise of smallholder farmers, as the major agents in smallholder farming systems. Strengthening climate resilience could be part of a wider participatory and learning process to empower smallholder farmers with the capacity for change and transformation. However, we also note that future assessments need to critically consider what resilience is, how it relates to vulnerability, development and poverty reduction, and what (and who) it includes and excludes. Incorporating such critical analysis into resilience assessments would provide scope to identify and address potential trade-offs between spatial and temporal scales; thus contributing to the advancement of resilience thinking and practice and enhanced natural resource management.

6. Conclusions

The range of peer-reviewed articles, frameworks and tools that mention resilience highlights its increasing popularity as an area of academic enquiry as well as goal in development practice. Due to the multifaceted nature of the concept, any attempts to assess resilience face the challenge of how to capture the dynamic and interconnected nature of the physical, social, institutional, economic, and ecological dimensions [5] of SES across spatial and temporal scales [1]. In this paper we analyzed a broad sample of tools and frameworks, including resilience assessment frameworks, two of which underwent further in depth analysis. In doing this, we compared how resilience theory has been operationalized to inform the development of practical assessment tools and frameworks. Findings clearly highlight some key gaps between resilience theory and its operationalization. This demonstrates a lack of shared understanding about what resilience is, how to build it, and linked to that, how to measure or characterize it. This lack of shared understanding highlights the potential difficulties encountered when developing and implementing a climate resilience framework that simultaneously tackles complexity, context-dependency and temporal and spatial dynamics.

Through comparing resilience theory and how it has informed the development of resilience frameworks, we have proposed six recommendations for future resilience assessments. Our recommendations highlight the current real-world application of resilience thinking and represent a first step to enhance our understanding of how theory and practice can be connected. Our findings can be used inform the development of future resilience assessment tools and frameworks, and provide a potential starting point for further empirical studies of farming system resilience. For smallholder farming systems in particular, we note that specific assessment tools and frameworks are required that integrate farmers’ knowledge and promote participation, flexibility and learning. This would enable assessments to better integrate contextually-relevant information and locally-held knowledge to empower smallholder farmers, which will be essential in building the resilience of smallholder farming systems.

Acknowledgments

We acknowledge the contributions of John Choptiany, Benjamin Graub and Suzanne Phillips to the development of the paper. We also thank Andrew Dougill, Susannah Sallu, Lisa Ficklin and Claire Quinn for their comments on an earlier draft and the journal’s anonymous reviewers for their constructive feedback, comments and suggestions. Work included in this paper was supported by the Food and Agriculture Organization of the United Nations and the development of this paper received funding from the European Union through the Improved Global Governance for Hunger Reduction Programme. The views expressed herein can in no way be taken to reflect the official opinion of the European Union or the Food and Agriculture Organization.

Author Contributions

The authors contributed equally to the design of the research and the writing of the manuscript. Jami L Dixon also conducted the initial literature search, data analysis, and data interpretation.

Appendix

Table A1. Overview of existing 25 tools/frameworks identified through selection process. Those presented at the stakeholder workshop in Burkina Faso are in white boxes, those identified through the internet search in are shaded in light grey boxes and additional expert recommendations are in dark grey boxes. Tools marked with a star (*) were under development at the time of analysis and the two frameworks assessed in detail in the text are highlighted in bold.
Table A1. Overview of existing 25 tools/frameworks identified through selection process. Those presented at the stakeholder workshop in Burkina Faso are in white boxes, those identified through the internet search in are shaded in light grey boxes and additional expert recommendations are in dark grey boxes. Tools marked with a star (*) were under development at the time of analysis and the two frameworks assessed in detail in the text are highlighted in bold.
No.Name of the Tool/FrameworkSourcePurpose (As Stated by Authors)
1Household Economy Approach (HEA/AEM)Holzmann, Boudreau, Holt, Lawrence and O’Donnell [78]To improve the predictive ability of short-term assessments of changes in food access based on an analysis of peoples’ access to the goods and services that they require to survive.
2Climate-Smart Agriculture (CSA) SourcebookFAO [79]To develop the technical, policy and investment conditions to achieve sustainable agricultural development for food security under climate change.
3*Sustainability Assessment of Food and Agriculture systems (SAFA)FAO [81]To enable people and companies undertaking the self-assessment to identify areas of high sustainability and areas where action is needed to improve sustainability.
4Climate proofing for Development (CP4Dev)Hahn and Fröde [93]To make development interventions more efficient and resilient. Provide a methodological approach to analyze development measures with regard to the current and future challenges and opportunities presented by climate change.
5MApping System and Services for Canal Operation Techniques (MASSCOTE)FAO [94]To evaluate and analyze different components of irrigation and canal systems in order to develop a modernization plan.
6Land Degradation Assessment in Drylands (LADA)FAO [95]To assess land degradation at the sub-regional, regional, national and global scales.
7Community-based Risk Screening Tool–Adaptation and Livelihoods (CRiSTAL)IISD [96]To systematically assess the impacts of a project on some of the local determinants of vulnerability and exposure, so that project planners and managers can design activities that foster climate adaptation (i.e., adaptation to climate variability and change).
8Climate Vulnerability and Capacity Analysis (CCVA)Care International [97]To present a new participatory methodology for Climate Vulnerability and Capacity Analysis.
9Climate Resilience and Food Security in Central America (CREFSCA)IISD [98]To strengthen the long-term food security of vulnerable populations in Central America by improving the climate resilience of food systems at different spatial and temporal scales.
10Climate Resilient Agriculture Module (CRAM)CCAFS [99]To bring together a group of participatory research tools to support research and development partners in gathering information that will help them design inclusive and gender sensitive programs in climate resilient agriculture.
11Climate Resilience Framework (CRF)Tyler and Moench [32 ]To build networked resilience that is capable of addressing emerging, indirect and slow-onset climate impacts and hazards.
12iResilience (including other assessment tools & quizzes like this)Robertsoncooper [100]To provide a comprehensive understanding of personal resilience and give examples of how this could impact on users responses to demanding work situations.
13International Strategy for Disaster ReductionUNISDR [101]To assist disaster reduction efforts by the cities and local governments that has signed up to the global "Making Cities Resilient" Campaign.
14Climate Resilient CitiesWorld Bank [102]To aid city governments in the East Asia Region to understand better how to plan for climate change impacts and impending natural disasters through sound urban planning to reduce vulnerabilities.
15A Self-Assessment To Address Climate Change Readiness in Your CommunityMinnesota Sea Grant [103]To provide community leaders, administrators, planners, engineers, public work directors, and/or natural resource managers with a simple and inexpensive method to review their communities potential vulnerabilities to climate trends and to begin the conversation of how and when to incorporate these trends into planning and projects within our communities.
16ADAPTWorld Bank [104]A screening tool designed to bring together climate databases and expert assessment of the threats and opportunities arising from climate variability and change.
17The Resilience ToolFAO [105]To provide a framework for understanding the most effective combination of short and long term strategies for lifting families out of cycles of poverty and hunger.
18Rapid Assessment[106]FAO [107]To assist investment project formulation practitioners in incorporating climate change considerations into agricultural investment projects and programs.
19Resilience Assessment Workbook: Assessing Resilience in Social-Ecological SystemsResilience Alliance [22]To provide a step-by-step approach to assessing resilience of a social-ecological system with the long term goal of sustainable delivery of environmental benefits linked to human well-being.
20Social-Ecological InventorySchultz, Plummer and Purdy [80]To identify existing knowledge and activities already underway in an area or sector, as well as the key actors involved with particular issues.
21Participatory Monitoring, Evaluation, Reflection and Learning for Community-based Adaptation (PMERL)Care International [106]To build the resilience of vulnerable individuals, households, communities and societies from the ground up.
22Analyzing Urban Digital Infrastructure Interventions from a Resilience LensHeeks and Ospina [108]To develop a well-conceptualized model of resilience that can be used in both research and practice to understand and evaluate climate change and other interventions in urban settlements.
23Indicator Framework for Assessing Agro-ecosystem ResilienceCabell and Oelofse [20 ]To present an index of behavior-based indicators that, when identified in an agro-ecosystem, suggest that it is resilient and endowed with a capacity for adaptation and transformation.
24*Resilience Index Measurement and AnalysisFAO [109]To provide decision-makers with clear indications of where and how to intervene to strengthen resilience. RIMA identifies populations most in need in order to frame policy, investment and response options in terms of resilience. RIMA also enables monitoring and evaluation of the impact of interventions to achieve greater accountability towards affected populations.
25*Self-evaluation and Holistic Assessment of the Resilience of farmers and Pastoralists (SHARP)Gräub and Choptiany [110]To allow farmers and pastoralists to self-assess their climate resilience in order to identify areas of improvement. Results from a rapid assessment are discussed with facilitators who are provided with potential actions and guidance documents to improve resilience of farmers and pastoralists. Includes governance, environmental, practices, social and economic questions.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Frazier, T.G.; Thompson, C.M.; Dezzani, R.J.; Butsick, D. Spatial and temporal quantification of resilience at the community scale. Appl. Geogr. 2013, 42, 95–107. [Google Scholar] [CrossRef]
  2. Ventura, A.D.O.B.; Lana, P.D.C. A new empirical index for assessing the vulnerability of peri-urban mangroves. J. Environ. Manag. 2014, 145, 289–298. [Google Scholar] [CrossRef]
  3. Béné, C.; Newsham, A.; Davies, M.; Ulrichs, M.; Godfrey-Wood, R. Resilience, poverty and development. J. Int. Dev. 2014, 26, 598–623. [Google Scholar] [CrossRef]
  4. Leach, M. Re-Framing Resilience: A Symposium Report; Institute of Development Studies: Brighton, UK, 2008. [Google Scholar]
  5. Cutter, S.L.; Barnes, L.; Berry, M.; Burton, C.; Evans, E.; Tate, E.; Webb, J. A place-based model for understanding community resilience to natural disasters. Global Environ. Chang. 2008, 18, 598–606. [Google Scholar] [CrossRef]
  6. Biggs, R.; Schluter, M.; Biggs, D.; Bohensky, E.L.; BurnSilver, S.; Cundill, G.; Dakos, V.; Daw, T.M.; Evans, L.S.; Kotschy, K.; et al. Toward principles for enhancing the resilience of ecosystem services. Annu. Rev. Environ. Resour. 2012, 37, 421–448. [Google Scholar] [CrossRef]
  7. Miller, F.; Osbahr, H.; Boyd, E.; Thomalla, F.; Bharwani, S.; Ziervogel, G.; Walker, B.; Birkmann, J.; van der Leeuw, S.; Rockström, J.; et al. Resilience and vulnerability: Complementary or conflicting concepts? Ecol. Soc. 2010, 15, Article 11. [Google Scholar]
  8. Challinor, A.; Wheeler, T.; Garforth, C.; Craufurd, P.; Kassam, A. Assessing the vulnerability of food crop systems in africa to climate change. Clim. Chang. 2007, 83, 381–399. [Google Scholar] [CrossRef]
  9. Thornton, P.K.; van de Steeg, J.; Notenbaert, A.; Herrero, M. The impacts of climate change on livestock and livestock systems in developing countries: A review of what we know and what we need to know. Agric. Syst. 2009, 101, 113–127. [Google Scholar] [CrossRef]
  10. Morton, J.F. The impact of climate change on smallholder and subsistence agriculture. Proc. Natl. Acad. Sci. USA 2007, 104, 19680–19685. [Google Scholar] [CrossRef] [PubMed]
  11. Muller, C.; Cramer, W.; Hare, W.L.; Lotze-Campen, H. Climate change risks for african agriculture. Proc. Natl. Acad. Sci. USA 2011, 108, 4313–4315. [Google Scholar] [CrossRef] [PubMed]
  12. Salami, A.; Kamara, A.B.; Brixiova, Z. Smallholder Agriculture in East Africa: Trends, Constraints and Opportunities; African Development Bank: Tunis, Tunisia, 2010. [Google Scholar]
  13. Mendelsohn, R.; Dinar, A. Climate change, agriculture, and developing countries: Does adaptation matter? World Bank Res. Obs. 1999, 14, 277–293. [Google Scholar] [CrossRef]
  14. IPCC. Climate change 2014: Impacts, adaptation, and vulnerability. In Part B: Regional aspects. Contribution of Working Group ii to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change; Barros, V.R., Field, C.B., Dokken, D.J., Mastrandrea, M.D., Mach, K.J., Bilir, T.E., Chatterjee, M., Ebi, K.L., Estrada, Y.O., Genova, R.C., et al, Eds.; Cambridge University Press: Cambridge, UK; New York, NY, USA, 2014. [Google Scholar]
  15. Parry, M.L.; Canziani, O.F.; Palutikof, J.P.; van der Linden, P.J.; Hanson, C.E. Climate Change 2007: Impacts, Adaptation and Vulnerability; Cambridge University Press: Cambridge, UK, 2007. [Google Scholar]
  16. IPCC. Summary for policymakers. In Climate Change 2014: Impacts, Adaptation, and Vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change; Field, C.B., Barros, V.R., Dokken, D.J., Mach, K.J., Mastrandrea, M.D., Bilir, T.E., Chatterjee, M., Ebi, K.L., Estrada, Y.O., Genova, R.C., et al, Eds.; Cambridge University Press: Cambridge, UK; New York, NY, USA, 2014; pp. 1–32. [Google Scholar]
  17. Keating, B.A.; McCown, R.L. Advances in farming systems analysis and intervention. Agric. Syst. 2001, 70, 555–579. [Google Scholar] [CrossRef]
  18. Giller, K.E. Guest editorial: Can we define the term “farming systems”? A question of scale. Outlook Agric. 2013, 42, 149–153. [Google Scholar] [CrossRef]
  19. Antwi-Agyei, P.; Fraser, E.D.G.; Dougill, A.J.; Stringer, L.C.; Simelton, E. Mapping the vulnerability of crop production to drought in ghana using rainfall, yield and socioeconomic data. Appl. Geogr. 2012, 32, 324–334. [Google Scholar] [CrossRef]
  20. Cabell, J.F.; Oelofse, M. An indicator framework for assessing agroecosystem resilience. Ecol. Soc. 2012, 17, Article 18. [Google Scholar]
  21. Cote, M.; Nightingale, A.J. Resilience thinking meets social theory: Situating change in socio-ecological systems (SES) research. Progress Hum. Geogr. 2012, 36, 475–489. [Google Scholar] [CrossRef]
  22. Resilience Alliance. Assessing Resilience in Social-Ecological Systems: Workbook for Practitioners, Version 2.0, Available online: http://www.redagres.org/Assessing%20Resilience%20in%20Social-Ecological%20Systems.pdf (accessed on 27 January 2015).
  23. Gunderson, L.H.; Holling, C.S. Panarchy: Understanding Transformation in Human and Natural Systems; Island Press: Washington, DC, USA, 2002. [Google Scholar]
  24. Holling, C.S. Resilience and stability of ecological systems. Ann. Rev. Ecol. Syst. 1973, 4, 1–23. [Google Scholar] [CrossRef]
  25. Folke, C.; Carpenter, S.; Elmqvist, T.; Gunderson, L.; Holling, C.; Walker, B.; Bengtsson, J.; Berkes, F.; Colding, J.; Danell, K.; et al. Resilience and Sustainable Development: Building Adaptive Capacity in a World of Transformations; Environmental Advisory Council to the Swedish Government: Stockholm, Sweden, 2002. [Google Scholar]
  26. IPCC. Climate change 2007: The Physical Science Basis; Solomon, S., Qin, D., Manning, M., Chen, Z., Marquis, M., Averyt, K.B., Tignor, M., Miller, H.L., Eds.; Cambridge University Press: Cambridge, UK, 2007. [Google Scholar]
  27. Vincent, K.; Cull, T.; Chanika, D.; Hamazakaza, P.; Joubert, A.; Macome, E.; Mutonhodza-Davies, C. Farmers’ responses to climate variability and change in southern Africa—Is it coping or adaptation? Clim. Dev. 2013, 5, 194–205. [Google Scholar] [CrossRef]
  28. Ericksen, P.J. , Conceptualizing food systems for global environmental change research. Glob. Environ. Chang. 2008, 18, 234–245. [Google Scholar] [CrossRef]
  29. Turner, B.L.; Kasperson, R.E.; Matson, P.A.; McCarthy, J.J.; Corell, R.W.; Christensen, L.; Eckley, N.; Kasperson, J.X.; Luers, A.; Martello, M.L.; et al. A framework for vulnerability analysis in sustainability science. Proc. Natl. Acad. Sci. USA 2003, 100, 8074–8807. [Google Scholar] [CrossRef]
  30. O’Brien, K.; Eriksen, S.; Nygaard, L.P.; Schjolden, A.N.E. Why different interpretations of vulnerability matter in climate change discourses. Clim. Policy 2007, 7, 73–88. [Google Scholar] [CrossRef]
  31. Folke, C.; Carpenter, S.R.; Walker, B.; Scheffer, M.; Chapin, T.; Rockström, J. Resilience thinking: Integrating resilience, adaptability and transformability. Ecol. Soc. 2010, 15, Article 20. [Google Scholar]
  32. Tyler, S.; Moench, M. A framework for urban climate resilience. Clim. Dev. 2012, 4, 311–326. [Google Scholar] [CrossRef]
  33. Folke, C. Resilience: The emergence of a perspective for social-ecological systems analyses. Glob. Environ. Chang. 2006, 16, 253–267. [Google Scholar] [CrossRef]
  34. Berkes, F.; Seixas, C.S. Building resilience in lagoon social-ecological systems: A local-level perspective. Ecosystems 2005, 8, 967–974. [Google Scholar] [CrossRef]
  35. Crane, T.A. Of models and meanings: Cultural resilience in social & ecological systems. Ecol. Soc. 2010, 15, Article 19. [Google Scholar]
  36. Twyman, C.; Fraser, E.D.G.; Stringer, L.C.; Quinn, C.; Dougill, A.J.; Ravera, F.; Crane, T.A.; Sallu, S.M. Climate science, development practice, and policy interactions in dryland agroecological systems. Ecol. Soc. 2011, 16, Article 19. [Google Scholar]
  37. Berkes, F.; Folke, C. Back to the future: Ecosystem dynamics and local knowledge. In Panarchy: Understanding Transformations in Human and Natural Systems; Gunderson, L.H., Holling, C.S., Eds.; Island Press: Washington, DC, USA; London, UK, 2002. [Google Scholar]
  38. Dixon, J.L.; Stringer, L.C.; Challinor, A.J. Farming system evolution and adaptive capacity: Insights for adaptation support. Resources 2014, 3, 182–214. [Google Scholar] [CrossRef]
  39. Holling, C.S. Resilience of ecosystems: Local surprise and global change. In Sustainable Development and the Biosphere; Clark, W.C., Munn, R.E., Eds.; Cambridge University Press: Cambridge, UK, 1986; pp. 292–317. [Google Scholar]
  40. Walker, B.; Gunderson, L.; Kinzig, A.; Folke, C.; Carpenter, S.; Schultz, L. A handful of heuristics and some propositions for understanding resilience in social-ecological systems. Ecol. Soc. 2006, 11, Article 13. [Google Scholar]
  41. Reed, M.S.; Evely, A.C.; Cundill, G.; Fazey, I.; Glass, J.; Laing, A.; Newig, J.; Parrish, B.; Prell, C.; Raymond, C.; et al. What is social learning? Ecol. Soc. 2010, 15, Response 1. [Google Scholar]
  42. Stringer, L.C.; Dougill, A.J.; Fraser, E.; Hubacek, K.; Prell, C.; Reed, M.S. Unpacking participation in the adaptive management of socialecological systems: A critical review. Ecol. Soc. 2006, 11, Article 39. [Google Scholar]
  43. Tschakert, P.; Dietrich, K.A. Anticipatory learning for climate change adaptation and resilience. Ecol. Soc. 2010, 15, Article 11. [Google Scholar]
  44. Berkes, F.; Colding, J.; Folke, C. Navigating the Dynamics of Social-Ecological Systems: Building Resilience for Complexity and Change; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
  45. Walker, B.; Holling, C.S.; Carpenter, S.R.; Kinzig, A.P. Resilience, adaptability and transformability in social-ecological systems. Ecol. Soc. 2004, 9, Article 5. [Google Scholar]
  46. Pelling, M. Adaptation to Climate Change: From Resilience to Transformation; Routledge: Abingdon, UK, 2011. [Google Scholar]
  47. Peterson, G. Political ecology and ecological resilience: An integration of human and ecological dynamics. Ecol. Econ. 2000, 35, 323–336. [Google Scholar] [CrossRef]
  48. Olsson, P.; Galaz, V.; Boonstra, W.J. Sustainability transformations: A resilience perspective. Ecol. Soc. 2014, 19, Article 1. [Google Scholar]
  49. Holling, C.S.; Gunderson, L.H. Resilience and adaptive cycles. In Panarchy: Understanding Transformations in Human and Natural Systems; Gunderson, L.H., Holling, C.S., Eds.; Island Press: Washington, DC, USA, 2002; pp. 25–62. [Google Scholar]
  50. Carpenter, S.; Walker, B.; Anderies, J.M.; Abel, N. From metaphor to measurement: Resilience of what to what? Ecosystems 2001, 4, 765–781. [Google Scholar] [CrossRef]
  51. Walker, B.H.; Anderies, J.M.; Kinzig, A.P.; Ryan, P. Exploring resilience in social-ecological systems through comparative studies and theory development: Introduction to the special issue. Ecol. Soc. 2006, 11, Article 12. [Google Scholar]
  52. Stringer, L.C.; Harris, A. Land degradation in dolj county, southern romania: Environmental changes, impacts and responses. Land Degrad. Dev. 2014, 25, 17–28. [Google Scholar] [CrossRef]
  53. Coulthard, S. Can we be both resilient and well, and what choices do people have? Incorporating agency into the resilience debate from a fisheries perspective. Ecol. Soc. 2012, 17, Article 4. [Google Scholar]
  54. Kates, R. Cautionary tales: Adaptation and the global poor. In Societal Adaptation to Climate Variability and Change; Kane, S., Yohe, G., Eds.; Springer: Dordrecht, The Netherlands, 2000; pp. 5–17. [Google Scholar]
  55. Suckall, N.; Tompkins, E.; Stringer, L. Identifying trade-offs between adaptation, mitigation and development in community responses to climate and socio-economic stresses: Evidence from zanzibar, tanzania. Appl. Geogr. 2014, 46, 111–121. [Google Scholar] [CrossRef]
  56. Béné, C.; Godfrey-Wood, R.; Newsham, A.; Davies, M. Resilience: New utopia or new tyranny? Reflection about the potentials and limits of the concept of resilience in relation to vulnerability reduction programmes. IDS Work. Pap. 2012, 2012, 1–61. [Google Scholar]
  57. Fazey, I.; Fazey, J.A.; Fischer, J.; Sherren, K.; Warren, J.; Noss, R.F.; Dovers, S.R. Adaptive capacity and learning to learn as leverage for social–ecological resilience. Front. Ecol. Environ. 2007, 5, 375–380. [Google Scholar] [CrossRef]
  58. Walker, B.; Salt, D. Resilience Practice: Building Capacity to Absorb Disturbance and Maintain Function; Island Press: Washington, DC, USA, 2012. [Google Scholar]
  59. Darnhofer, I. Resilience and why it matters for farm management. Eur. Rev. Agric. Econ. 2014, 41, 461–484. [Google Scholar] [CrossRef]
  60. Warren, A. Land degradation is contextual. Land Degrad. Dev. 2002, 13, 449–459. [Google Scholar] [CrossRef]
  61. Stringer, L.C.; Reed, M.S. Land degradation assessment in southern africa: Integrating local and scientific knowledge bases. Land Degrad. Dev. 2007, 18, 99–116. [Google Scholar] [CrossRef]
  62. Dyer, J.; Leventon, J.; Stringer, L.; Dougill, A.; Syampungani, S.; Nshimbi, M.; Chama, F.; Kafwifwi, A. Partnership models for climate compatible development: Experiences from zambia. Resources 2013, 2, 1–25. [Google Scholar] [CrossRef]
  63. Dougill, A.J.; Fraser, E.D.G.; Holden, J.; Hubacek, K.; Prell, C.; Reed, M.S.; Stagl, S.; Stringer, L.C. Learning from doing participatory rural research: Lessons from the peak district national park. J. Agric. Econ. 2006, 57, 259–275. [Google Scholar] [CrossRef]
  64. Raymond, C.M.; Fazey, I.; Reed, M.S.; Stringer, L.C.; Robinson, G.M.; Evely, A.C. Integrating local and scientific knowledge for environmental management. J. Environ. Manag. 2010, 91, 1766–1777. [Google Scholar] [CrossRef]
  65. Vernooy, R.; Ruiz, M. Access to and benefit sharing of plant genetic resources: Novel field experiences to inform policy. Resources 2013, 2, 96–113. [Google Scholar] [CrossRef]
  66. Reed, M.S.; Fraser, E.D.G.; Dougill, A.J. An adaptive learning process for developing and applying sustainability indicators with local communities. Ecol. Econ. 2006, 59, 406–418. [Google Scholar] [CrossRef]
  67. Amaru, S.; Chhetri, N.B. Climate adaptation: Institutional response to environmental constraints, and the need for increased flexibility, participation, and integration of approaches. Appl. Geogr. 2013, 39, 128–139. [Google Scholar] [CrossRef]
  68. Fraser, E.D.G.; Dougill, A.J.; Hubacek, K.; Quinn, C.H.; Sendzimir, J.; Termansen, M. Assessing vulnerability to climate change in dryland livelihood systems: Conceptual challenges and interdisciplinary solutions. Ecol. Soc. 2011, 16, Article 3. [Google Scholar]
  69. Dixon, J.A.; Gobbon, D.P.; Gulliver, A. Farming Systems and Poverty: Improving Farmers’ Livelihoods in a Changing World; FAO: Rome, Italy; World Bank: Washington, DC, USA, 2001. [Google Scholar]
  70. Chambers, R.; Jiggins, J. Agricultural research for resource-poor farmers part I: Transfer-of-technology and farming systems research. Agric. Adm. Ext. 1987, 27, 35–52. [Google Scholar]
  71. Leeuwis, C. Communication for Rural Innovation: Rethinking Agricultural Extension, 3rd ed.; Wiley-Blackwell: Oxford, UK, 2004. [Google Scholar]
  72. Darnhofer, I.; Fairweather, J.; Moller, H. Assessing a farm’s sustainability: Insights from resilience thinking. Int. J. Agric. Sustain. 2010, 8, 186–198. [Google Scholar] [CrossRef]
  73. Milestad, R.; Darnhofer, I. Building farm resilience: The prospects and challenges of organic farming. J. Sustain. Agric. 2003, 22, 81–97. [Google Scholar] [CrossRef]
  74. Rockefeller Foundation. Building Climate Change Resilience, 2009. Available online: http://www.rockefellerfoundation.org/uploads/files/c9725eb2-b76e-42eb-82db-c5672a43a097-climate.pdf (accessed on 27 January 2015).
  75. IPCC. Climate change 2014: Impacts, adaptation, and vulnerability. Part A: Global and Sectoral Aspects. Contribution of Working Group II to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change; Field, C.B., Barros, V.R., Dokken, D.J., Mach, K.J., Mastrandrea, M.D., Bilir, T.E., Chatterjee, M., Ebi, K.L., Estrada, Y.O., Genova, R.C., et al, Eds.; Cambridge University Press: Cambridge, UK; New York, NY, USA, 2014. [Google Scholar]
  76. Stringer, L.C.; Twyman, C.; Gibbs, L.M. Learning from the south: Common challenges and solutions for small-scale farming. Geogr. J. 2008, 174, 235–250. [Google Scholar] [CrossRef]
  77. Lorenz, S.; Berman, R.; Dixon, J.; Lebel, S. Time for a systematic review: A response to bassett and fogelman’s “déjà vu or something new? The adaptation concept in the climate change literature”. Geoforum 2014, 51, 252–255. [Google Scholar] [CrossRef]
  78. Holzmann, P.; Boudreau, T.; Holt, J.; Lawrence, M.; O’Donnell, M. The Household Economy Approach: A Guide for Programme Planners and Policy-Makers; Carroll, J., Ed.; Save the Children: London, UK, 2008. [Google Scholar]
  79. Food and Agriculture Organization (FAO). Climate Smart Agriculture Sourcebook; FAO: Rome, Italy, 2013. [Google Scholar]
  80. Schultz, L.; Plummer, R.; Purdy, S. Applying a Social-Ecological Inventory: A Workbook for Finding the Key Actors and Engaging them; Stockholm Resilience Centre: Stockhom, Sweden, 2011. [Google Scholar]
  81. Food and Agriculture Organization (FAO). Sustainability Assessment of Food and Agriculture Systems (SAFA): Guidelines Version 3.0; FAO: Rome, Italy, 2013. [Google Scholar]
  82. Klapwijk, C.J.; van Wijk, M.T.; Rosenstock, T.S.; van Asten, P.J.A.; Thornton, P.K.; Giller, K.E. Analysis of trade-offs in agricultural systems: Current status and way forward. Curr. Opin. Environ. Sustain. 2014, 6, 110–115. [Google Scholar] [CrossRef]
  83. Adger, W.N. Vulnerability. Global Environ. Chang. 2006, 16, 268–281. [Google Scholar] [CrossRef]
  84. MacGregor, S. Vulnerable victims or resilient subjects? Dismantling the discursive traps of climate policy. In Proceedings of the Gender and Climate Change: Women, Research and Action, International Conference, Prato, Italy, 15–16 September 2011.
  85. Wardekker, J.A.; de Jong, A.; Knoop, J.M.; van der Sluijs, J.P. Operationalising a resilience approach to adapting an urban delta to uncertain climate changes. Technol. Forecast. Soc. Chang. 2010, 77, 987–998. [Google Scholar] [CrossRef]
  86. Tompkins, E.; Adger, W. Adaptive capacity and learning to learn as leverage for social-ecological resilience. In Working Paper; Tyndall Centre for Climate Change Research: Norwich, UK, 2003; Volume 39. [Google Scholar]
  87. Parsons, R.; Lederwasch, A.; Moffat, K. Clermont preferred future: Stakeholder reflections on a community foresight and planning initiative. Resources 2013, 2, 528–554. [Google Scholar] [CrossRef]
  88. Chambers, R.; Ghildyal, B.P. Agricultural Research for Resource-Poor Farmers: The Farmer-First-and-Last Model; Ford Foundation: New Delhi, India, 1984. [Google Scholar]
  89. Reed, M.S.; Dougill, A.J.; Taylor, M.J. Integrating local and scientific knowledge for adaptation to land degradation: Kalahari rangeland management options. Land Degrad. Dev. 2007, 18, 249–268. [Google Scholar] [CrossRef]
  90. Mathur, V.N.; Afionis, S.; Paavola, J.; Dougill, A.J.; Stringer, L.C. Experiences of host communities with carbon market projects: Towards multi-level climate justice. Clim. Policy 2013, 14, 42–62. [Google Scholar] [CrossRef]
  91. Westley, F.R.; Tjornbo, O.; Schultz, L.; Olsson, P.; Folke, C.; Crona, B.; Bodin, Ö. A theory of transformative agency in linked social-ecological systems. Ecol. Soc. 2013, 18, Article 27. [Google Scholar]
  92. Adger, W.N. Social and ecological resilience: Are they related? Progress Hum. Geogr. 2000, 24, 347–364. [Google Scholar] [CrossRef]
  93. Hahn, M.; Fröde, A. Climate proofing for development (cp4dev): Adapting to climate change, reducing risk; Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ): Reinheim, Germany, 2011. [Google Scholar]
  94. Food and Agriculture Organization (FAO). Masscote: Mapping system and services for canal operation techniques; FAO: Rome, Italy, 2007. [Google Scholar]
  95. Food and Agriculture Organization (FAO). Land degradation assessment in drylands; FAO: Rome, Italy, 2007. [Google Scholar]
  96. International Institute for Sustainable Development (IISD). Cristal user’s manual version 5; IISD: Winnipeg, Canada, 2012. [Google Scholar]
  97. Care International. Climate Vulnerability and Capacity Analysis (ccva) Handbook; Care International: London, UK, 2009. [Google Scholar]
  98. IISD. Climate resilience and food security in central america. Available online: http://www.iisd.org/adaptation/crefsca/ (accessed on 8 November 2014).
  99. Climate Change, Agriculture, Food Security (CCAFS). Climate resilient agriculture module; CCAFS: Copenhangen, Denmark, 2012. [Google Scholar]
  100. Robertsoncooper Iresilience. Available online: http://www.robertsoncooper.com/iresilience/ (accessed on 19 October 2013).
  101. UNISDR. How to Make Cities More Resilient: A Handbook for Local Government Leaders; United Nations Office for Disaster Risk Reduction: Geneva, Switzerland, 2012. [Google Scholar]
  102. Neeraj, P.; Federica, R.; Fatima, S.; Zoe, T.; Earl, K.; Ravi, S. Climate Resilient Cities: A Primer on Reducing Vulnerabilities to Disasters; World Bank: Washington, DC, USA, 2009. [Google Scholar]
  103. A self-assessment to address climate change readiness in your community. Available online: http://www.cmap.illinois.gov/documents/10180/14193/Appendix+B+--+Self-Assessment+to+Address+Climate+Change+Readiness+in+Your+Community.pdf/11377862-ee6c-4d8e-84a5-b3a138c486d4 (accessed on 9 November 2013).
  104. World Bank. Adapt: A Tool to Screen Development Projects: A Tool to Screen Development Projects for Climate Risk; World Bank: Washington, DC, USA, 2009. [Google Scholar]
  105. FAO. Measuring Resilience—A Concept Note; Food and Agriculture Organization: Rome, Italy, 2010. [Google Scholar]
  106. Care International. Participatory Monitoring, Evaluation, Reflection and Learning for Community-Based Adaptation (Pmerl): A Manual for Local Practicioners; Care International: London, UK, 2012. [Google Scholar]
  107. Food and Agriculture Organziation (FAO). Incorporating climate change considerations into agricultural investment programmes: A guidance document; Food and Agriculture Organziation: Rome, Itay, 2012. [Google Scholar]
  108. Heeks, R.; Ospina, A.V. Analysing urban digital infrastructure interventions from a resilience lens. In Working Paper Series; Centre for Development Informatics, Institute for Development Policy and Management, SEED, University of Manchester: Manchester, UK, 2013. [Google Scholar]
  109. Food and Agriculture Organziation. Resilience index measurement and analysis (rima) model. Available online: http://www.foodsec.org/web/resilience/measuring-resilience/resilience-model/en/ (accessed on 3 May 2014).
  110. Gräub, B.; Choptiany, C. Food producers at the center of the water-food-climate nexus: Implementing sharp to increase climate resilience. In Proceedings of Nexus 2014: Water, Food, Climate and Energy, University of North Carolina, Chapel Hill, NC, USA; 2014. [Google Scholar]
Resources EISSN 2079-9276 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top