Three Dimensions of Transformative Impact and Capacity: A Conceptual Framework Applied in Social Innovation Practice

Round 1

Reviewer 1 Report

This is an interesting and relevant focus for a study.  However, the article as it is has quite a few areas which would need to be addressed in order to fulfil requirements related to research, ethics and appeal to readers.  The following are offered humbly for your consideration. 

1. Flow and organisation of the article.  This seems unnecessary complex.  Perhaps it would be helpful to simplify it and present it as the process of evaluating and refining the original 3D Conceptual Framework. in other words - make the purpose of the article that which is currently your second purpose (lines 62-63).  This would, i believe, enable an article which has better sense of structure and flow - and thus more appealing to the reader.  The changes made to the 3D Conceptual Framework are included in footnotes - these make sense.  I believe it would serve the reader better to have this information within the narrative of the study - showing how the study led to the change. 
2. Strengths

• Lines 111-133 read well and are helpful to make sense of the study/article.  Appreciated point (line 138-139) outing the relationship between impact and transformation.  
• The care taken throughout to define terminology - e.g., "by usefulness we mean (line 272)
• Section 3.2. The process of developing diagnostic questions from the original components of the CF.  However, perhaps readers would be better served if - within the article, the authors provide them with a generic description of the process - with some examples. 

3. Weaknesses 

• This is not an example of a 'Mixed method approach' (line 179).  All aspects of the research are qualitative in nature. A ‘mixed method approach’ is when researchers collect and analyse both quantitative and qualitative data within the same study – in different phases – one to establish some findings and the second to test or elaborate on those findings.   It would be more accurate to describe it as a multimethod study.
• The methodology section, and references throughout the article, need to be much clearer, accurate, address ethical issues that relate to the fact that one researcher was working in their own context.  in particular:
  • have a more theoretically grounded presentation on the multi methods used - e.g., auto ethnography (line 323-327)- this is appropriate for the study - but there is no theoretical background provided for this method, nor is there any indication who was involved in this, nor if any of the data quoted in the article flow from this method.  
  • This section needs clarification in terms of which are 'methods' and which are 'data collection tools' within that method.  For example  - if it truly has an 'embedded action research approach' - then perhaps it would be best to write that up as a singular case study. 
  • There are no references to the ethical issues or considerations within the study. 
  • Line 321 - diagnostic questions were used for "some of the interviews" ; What were used for the others?  How would a reader know what data came from the different tools?
  • It is not clear who undertook the participant observation, semi-structured interviews? – did the one researcher conduct all of these?Including that of their own context?  Similarly, for document analysis. Focus group Workshops – to what degree focus group and to what degree workshop?
• Working with data.  It is not possible for the reader to know which data (ie the source - participant observation, interview, focus group, auto ethnographic journal etc) contributed to refinements of the CF.  It would be helpful - from the methodology section to clearly indicate sources of data - e.g., IntGreen to indicate the data came from an interview from Green Office, or FGEC to indicate a Focus Group with participants from ECOLISE.    This lack does not build confidence for the reader in terms of data usage.  Examples of better working with data include:

  Line 693 – “An ECOLISE leader……”

  Line718-720 – clear it came from (GEN event) – more info would be good – Focus Group? Autoethnographic journal?

  Here are some specific examples of unhelpful working with data to which i refer:

  Line 392 – where did this information come from?  Is it reliable data?

  Line 400 – changed their view of students – but in what way?  How might raw data from the project be used to illustrate this.

  Line 407-411 – similar to above – needs examples.

  Line 417-418 – what is the source of the quote

  Line 427 – many cases – give examples from raw data.

  Line 431 – one interviewee – from what context?

  Line 515 – what people? Is this raw data?

  Line 626 – many alumni – how do you know?  Who – particular subgroups? 

  Line 635-637 – this framework element was found difficult – good point – but do we know why or how – and what is the source of words in quote marks.  How was this addressed in the changes made.

  Line 670  key issues are . . . . . according to whom?  Does this emerge from the data?  From literature? 

• Section 4.6, line 738 – very interested in what?  Is it line 789?  Use raw data to illustrate, build a case for change/refinement/usefulness etc

• Line 495ff – new topic, not clear how it relates to study, where is this discussed, mentioned by whom? 

• Overall the writing accuracy is acceptable.  a few errors

  Some incomplete sentences – line 421 – 422;  line 510-511

  Word missing – line 735 –

  Unhelpful sentence structure – line 797-799

  Line 837 – feel? Felt?  More info would be helpful.  Why? Was this addressed in refinements?

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear authors,

The empirical work presented in this paper is valuable and the theoretical work used to put it in context is relevant. The RQs and the objectives are clearly stated; I found Appendix 2 – Diagnostic questions very useful.

Here are my recommendations:

1."By testing the conceptual framework in practice, some of these capacities have been revised or complemented by additional ones (L 156-157)”: Why did you consider relevant to add new elements compared to the initial approach? Why “Reconciliation and healing of trauma” element is considered now worthy of consideration?

2.“… during the application process we chose to use ‘capacities’ instead of ‘competences’, as it became evident that the term ‘competences’ unnecessarily increases the conceptual density of the framework, making it more difficult to understand and use in practice” (L 160-162): This statement needs further explanations, the rationale is not clear.

3.“In comparison to these other typologies, we suggest (and tested in this study) that the three dimensions offers fairly simple, comprehensive and intuitive meta categories that can make these aspects more easy to use in practice, while also integrating the length dimension which is missing or less explicit in the other conceptualisations that are more focused on the width and depth dimensions” (174-177). A broader discussion on the contribution of this study to the literature must be included. Moreover, lines 905-913 (from Conclusion) can be integrated here.

4. Subsection 3.1.1, 3.1.2, 3.1.3 (Case-cluster 1, 2,3): What are the criteria (apart from the purpose for which some of these were created) based on which you selected the meta-networks, member- networks, and citizen-networks?

5. Subsection 3.2.: It will be useful to have (in an Annex) a table/figure that displays how each of the interview questions responds to the RQs .

6. More research would be required to describe these interactions in more detail in various case contexts, and to identify the extent to which generalizable patterns of interactions can be observed across case contexts” (L 858-860). These lines dedicated to the limitations should be enriched.

7. I'm not sure the Conclusion chapter was distinct enough from the Discussion to warrant having it as a separate section. There seemed to be a decent amount of overlap with the Discussion and I'm wondering if the Conclusion section could just be removed and any unique content folded into the Discussion.

8. In general, sentences are often longer than necessary (this remark goes also for the abstract).

9. There are several grammar mistakes. See, for example: “In comparison to these other typologies, we suggest (and tested in this study) that the three dimensions offers fairly simple, comprehensive and intuitive meta categories that can make these aspects more easy  easier to use in practice.”

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

The article under review presents an interesting perspective about transformative impact of social innovation. Method and results are adequately presented. However authors, should better introduce the gaps in literature and, therefore, the need of the proposed conceptual framework in the specific context.

Even if the idea of the 3D framework sounds of a good degree of novelty, in the paper it is not well developed a logic flow towards the theme explored by the authors. More in detail, there is opacity about the epistemiological perspective adopted, given the complexity existing in social innovation studies (as well as geographical views about the specific theme). More over, it should useful for the reader to highlight if other conceptual framworks addressing the same topic have been explored or adopted.

In order to suggest a starting point to build a more robust overview of concepts that should be better developed, I would indicate the following sources:

1) Kaletka, C., & Schröder, A. (2017). A global mapping of social innovations: Challenges of a theory driven methodology. European Public & Social Innovation Review, 2(1), 78-92.

2) Ziegler, R. (2017). Social innovation as a collaborative concept. Innovation: The European Journal of Social Science Research, 30(4), 388-405.

3) Lindberg, M., & Portinson Hylander, J. (2017). Boundary dimensions of social innovation: Negotiating conflicts and compatibilities when developing a national agenda. Innovation: The European Journal of Social Science Research, 30(2), 168-181.

4) K. Smith, W., Erez, M., Jarvenpaa, S., Lewis, M. W., & Tracey, P. (2017). Adding complexity to theories of paradox, tensions, and dualities of innovation and change: Introduction to organization studies special issue on paradox, tensions, and dualities of innovation and change.

5) Parés, M., Ospina, S. M., & Subirats, J. (Eds.). (2017). Social innovation and democratic leadership: communities and social change from below. Edward Elgar Publishing.

6) Loorbach, D., Wittmayer, J., Avelino, F., von Wirth, T., & Frantzeskaki, N. (2020). Transformative innovation and translocal diffusion. Environmental Innovation and Societal Transitions.

7) Visser, W. (2018). Creating Integrated Value Through Sustainable Innovation: A Conceptual Framework. In Sustainable Business Models (pp. 129-150). Springer, Cham.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

Review of

Three dimensions of Transformative Impact and Capacity: a Conceptual Framework Applied in Social Innovation Practice

I have read the article and I have very much enjoyed it. The article explores the usefulness and potential applications of the three dimensions of transformative impact for social innovation. The article introduces the relevant theory for the 3D framework and provides examples of its use and usefulness. Despite being interesting, the article has serious flaws that prevent me from recommending it for publication. Here are some comments that might help the authors in improving the article:

1. I feel like the first paragraph of the introduction is a little bit too subjective and slightly political. This is not a major issue and I acknowledge my own bias in this, but perhaps the authors could rewrite the paragraph in a more neutral tone? I’d suggest refraining from advocating “radical change” or demonizing currently existing systems as somehow intrinsically bad.
2. Figure 1 uses several stock images. Are they really necessary? What is the copyright situation with these images? Perhaps Figure 1 could be reworked into a simpler representation (although a similar figure was used by the authors https://doi.org/10.3390/su11051304 ).
3. Section numbering on p. 4 is confusing. Add another section name for the text starting at line 135.
4. Figure 2 is a recycled version of the figure used in https://doi.org/10.3390/su11051304 ; so either provide a reference to the original figure, or consider omitting it altogether. Same is true of Figure 3.
5. Sustainability issues are often politicized, and a qualitative approach adds another layer of possible bias and subjectivity to the whole issue. Bearing in mind that this article has a qualitative part, I would like to see an in-depth reflection on the personal political and social biases of the authors. This will add to the transparency of the article. This is a very important point.
6. There is a lot of overlap between the theory of this article and the authors’ previous work. It may be bordering on self-plagiarism.
7. The methods section is insufficient, and it is unclear which method (or a combination of methods) was used to derive themes and, ultimately, conclusions. It seems that the authors used data from interviews (I assume they transcribed them?).
8. Was any software used for data analysis? Was it done by hand? What type of procedure was used for identification of themes? This process needs to be detailed in such a way that a reader would be able to replicate the study perfectly.
9. The results section is unacceptable (how were the themes identified? What examples are there to ground the themes?). It is not enough to just jump into the interpretation.
10. Missing statements on the reliability of the results. Did at least two of the authors do the analysis? What is their agreement rate? How were disagreements reconciled?
11. Transcripts should be provided for independent verification of the findings.
12. The researchers’ personal relationship with the data, as well as their preconceptions prior the analysis are not made clear to the reader.
13. Subjective and value-laden statements are being made as if they are objective and impartial.
14. Qualitative data are not sufficiently discussed in the context of available quantitative knowledge.
15. Bearing in mind that the results section already consists of the authors’ interpretation of the results, many things in the Discussion part of the article are redundant and already stated when discussing the results. I would suggest the authors provide a Results section that actually presents results impartially through identifying themes and supporting examples.
16. Conclusions should not be made so easily bases on qualitative data. Especially when the qualitative approach through which these conclusions were arrived upon is so flawed.

Ultimately, I believe that after being rewritten to suit standard quality criteria for qualitative work, the article would be more suited for a journal that publishes qualitative research exclusively. While the article attempts at a theoretical contribution based on qualitative data, the data are gathered and interpreted in ways that do not meet standard quality criteria for such research (see these sources: https://doi.org/10.1177/1359105306066643 ; https://doaj.org/article/8f67c52f6e9c438c97d39e7d4faeb729 ; https://doi.org/10.1093/jpepsy/jsw032 ; https://ebmh.bmj.com/content/ebmental/2/4/99.full.pdf ). The article lacks transparency and entirely skips essential parts of a qualitative study.

I hope that this review does not discourage the authors. Take this as an opportunity to hone your skills at writing up qualitative research. For methods that have a high degree of subjectivity, the researcher must be held to a high standard of quality and a high degree of scrutiny. The nature of qualitative work requires one to be very thorough and transparent in one’s work, the reliability and validity of the work needs to be grounded in robust examples.

I wish the authors the best of luck in their ongoing and future work.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The substantial changes made in response to reviewers' feedback have improved this article in terms of its sense of flow, engagement with scholarship and research rigor.  The abstract is much clearer and reads well. The methodology section, ethics considerations, role of first author and use of data source  indicators throughout the article increases the rigor and trustworthiness of the study itself as well as making it more narrative in nature.  Changes to sub headings are helpful indicators to support the reader's progression through the lengthy article.

The following are offered for consideration as needful corrections and considerations for polishing of the article:

Line 309 - spelling for practitioners

Lines 328-329 - would "access to internal documents" be better than "provision of" ?

Lines 376-377 - Simplify sentence - e.g., The abbreviations as shown in Table 6 were used in the results section indicate data sources. 

Line 419 - check name and use of caps for German sustainability council.   is  this the German Council for Sustainable Development?

Check for consistency in terms of the use of Transition  - sometimes it seems to need Transition Network,  inconsistent capitalisation for transition initiatives (lines 420, 535, 538, 581, 582, 568619, 740, 742, 816, 885

Lines 449-450 Remove duplicate 'their' and check if according should be accordingly 

Consistency of non capitalisation of first word after 1., 2., 3. etc.  Line 517 - change The to the - then would be consistency all the way through article (e.g., lines 916-917)

Line 548 - takes several readings to get meaning 

Line 632 - remove the word 'are' 

Lines 690-691 “Some created programs, offers, or organisational conditions to make their models and activities accessible to and inclusive of ….. doesn’t make sense – think the issue is with the word offers  - needs verb before offers and organisational conditions –  the verb created before programs could apply to organisational conditions – but ‘created offers’ ????

Line 745 - do you 'live up' to essential principles?

Line 930 - are words 'in this article' necessary.  Seems the changes apply beyond the article

All the very best with your ongoing work and continued research in the field. 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Recommendation: Accept in present form.

Author Response

see attachment for changes made in response to Reviewer 1, as well as some additional minor corrections.

Author Response File: Author Response.pdf

Reviewer 3 Report

Author Response

see attachment for changes made in response to Reviewer 1, as well as some additional minor corrections.

Author Response File: Author Response.pdf

Reviewer 4 Report

Review of the revision of

Three dimensions of Transformative Impact and Capacity: a Conceptual Framework Applied in Social Innovation Practice

I have read the revised article and can definitely see improvements. Most of my comments were addressed adequately. I still have methodological concerns about the study, but qualitative research can be conducted in many ways and I acknowledge my own bias in this regard. With the issues now mostly addressed and the method, data sources, and authors’ personal relationship with the data being made fairly transparent and clear, I believe that the article can be published.

I wish the authors the best of luck in their ongoing and future research.

Author Response

see attachment for changes made in response to Reviewer 1, as well as some additional minor corrections.

Author Response File: Author Response.pdf