A Method for Measuring Systems Thinking Learning
Abstract
:1. Introduction
1.1. Reductionism and Systemism
1.2. Systems Literacy and Systems Education
2. Background
2.1. What is a System?
2.2. What is Thinking?
2.3. What is Systems Thinking?
- Distinctions Rule (D): Any idea or thing can be distinguished from the other ideas or things it is with;
- Systems Rule (S): Any idea or thing can be split into parts or lumped into a whole;
- Relationship Rule (R): Any idea or thing can relate to other things or ideas; and
- Perspectives Rule (P): Any thing or idea can be the point or the view of a perspective.
2.4. Systems Thinking Education
2.5. Systems Literacy Roadmap
2.6. Systems Thinking Learning Model
3. Methodology
3.1. Data Collection
3.2. Qualitative Methods
3.2.1. Classifying Distinctions (Elements)
- Elements that are visible or invisible. For example, a plant is visibly identifiable in a fish-tank system, whereas bacteria are not visibly identifiable (i.e., cannot be seen with the naked eye).
- Elements inside or outside the system boundary (i.e., the walls of the fish tank). For example, rocks and filters are found inside the system boundary while a human and a thermostat are found outside the system boundary.
- Elements that are critical to the system’s definition. For example, a fish, a tank, and water are necessary to define a system as a fish-tank system, whereas the absence of a filter or a heater does not preclude defining a system that includes a fish, a tank, and water as a fish-tank system.
- Elements that are underlying causes of the problem situation. In the context of the green-water problem for a fish-tank system, some elements can be identified as underlying causes of that problem while many elements cannot be identified as underlying causes. The most likely underlying causes of green water in a fish-tank system include excess light or sunlight, excess fish waste, decaying organisms, overfeeding resulting in uneaten food, and a broken or dirty (ineffective) filter or filtration system [44].
- Elements that are labeled as systems. For example, the filter is only one element that can belong to the greater filtration system which may also contain filter media, filter tubes, an impeller, and other elements.
3.2.2. Classifying Relationships (Interactions)
3.2.3. Classifying Perspectives (Roles/Purposes)
- The first pattern was drawn from the mental model called “egocentric” [19]. This mental model is characterized by adopting the perspective of an observer of a fish-tank system. In the case of this research study, the observer of the system was the student drawing the fish-tank system.
- The second pattern was drawn from the mental model called “simple healthy fish” [19]. This mental model is characterized by adopting the perspective of a fish in the fish-tank system. In other words, all roles/purposes for elements focus on helping the fish in some way.
- The third pattern was drawn from the mental model called “good tank” [19]. This mental model is characterized by adopting the perspective of the fish-tank system as a whole. In other words, all roles/purposes for elements focus on helping the fish-tank system in some way.
3.2.4. Inter-Coder Agreement
3.3. Quantitative Methods
4. Results
4.1. Distinctions (Elements) Results
4.2. Relationships (Interactions) Results
4.3. Perspectives (Roles/Purposes) Results
4.4. Totals Results
5. Discussion
5.1. Research Findings
5.2. Systems Thinking Learning
5.3. Expected Learning and Systems Thinking Learning
5.4. Future Research Needs
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Appendix A
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) |
---|---|---|---|---|---|---|---|
Air and/or Bubbles | Concrete | Internal | No | No | No | Secondary | Literacy |
Air or Water Pump | Concrete | Internal | No | No | No | Secondary | Literacy |
Algae | Concrete | Internal | No | No | No | Secondary | Literacy |
Ammonia (NH3) | Conceptual | Internal | No | No | No | Advanced | Capability |
Bacteria | Conceptual | Internal | No | No | No | Advanced | Capability |
Bio-filter | Concrete | Internal | No | No | No | Secondary | Literacy |
Broken/Dirty Filter | Concrete | Internal | No | Yes | No | Advanced | Capability |
Carbon Dioxide (CO2) | Conceptual | Internal | No | No | No | Advanced | Capability |
Coral | Concrete | Internal | No | No | No | Secondary | Literacy |
Dead Organisms | Concrete | Internal | No | Yes | No | Advanced | Capability |
Denitrifying Bacteria | Conceptual | Internal | No | No | No | Advanced | Capability |
Dirt, Dust, and Debris | Concrete | Internal | No | No | No | Secondary | Literacy |
Electricity | Conceptual | External | No | No | No | Advanced | Capability |
Excess Fish Waste Products | Concrete | Internal | No | Yes | No | Advanced | Capability |
Filter | Concrete | Internal | No | No | No | Secondary | Literacy |
Filter Feeder Fish | Concrete | Internal | No | No | No | Secondary | Literacy |
Filtration System | Concrete | Internal | No | No | Yes | Advanced | Capability |
Fish | Concrete | Internal | Yes | No | No | Essential | Sensibility |
Fish Food | Concrete | Internal | No | No | No | Secondary | Literacy |
Fish Net | Concrete | External | No | No | No | Secondary | Literacy |
Fish Waste | Concrete | Internal | No | No | No | Secondary | Literacy |
Free Nitrogen Gas (N2) | Conceptual | Internal | No | No | No | Advanced | Capability |
Fungi | Conceptual | Internal | No | No | No | Advanced | Capability |
Green Water | Concrete | Internal | No | No | No | Secondary | Literacy |
Heater | Concrete | Internal | No | No | No | Secondary | Literacy |
Human | Concrete | External | No | No | No | Secondary | Literacy |
Impeller | Concrete | Internal | No | No | No | Secondary | Literacy |
Lighting/Lights | Concrete | External | No | Yes | No | Advanced | Capability |
Nitrate (NO3-) | Conceptual | Internal | No | No | No | Advanced | Capability |
Nitrate Bacteria | Conceptual | Internal | No | No | No | Advanced | Capability |
Nitrite (NO2-) | Conceptual | Internal | No | No | No | Advanced | Capability |
Nitrite Bacteria | Conceptual | Internal | No | No | No | Advanced | Capability |
Oxygen (O2) | Conceptual | Internal | No | No | No | Advanced | Capability |
Ornaments/Decorations | Concrete | Internal | No | No | No | Secondary | Literacy |
Other Animals | Concrete | Internal | No | No | No | Secondary | Literacy |
Plant Fragments | Concrete | Internal | No | No | No | Secondary | Literacy |
Plants | Concrete | Internal | No | No | No | Secondary | Literacy |
Rocks | Concrete | Internal | No | No | No | Secondary | Literacy |
Silicone | Concrete | Internal | No | No | No | Secondary | Literacy |
Substrate | Concrete | Internal | No | No | No | Secondary | Literacy |
Sunlight | Concrete | External | No | Yes | No | Advanced | Capability |
Tank | Concrete | Internal | Yes | No | No | Essential | Sensibility |
Tank Cover (Lid) | Concrete | External | No | No | No | Secondary | Literacy |
Tank Stand | Concrete | External | No | No | No | Secondary | Literacy |
Tank Wall Scraper | Concrete | External | No | No | No | Secondary | Literacy |
Thermometer | Concrete | Internal | No | No | No | Secondary | Literacy |
Thermostat | Concrete | External | No | No | No | Secondary | Literacy |
Uneaten Fish Food | Concrete | Internal | No | Yes | No | Advanced | Capability |
Water | Concrete | Internal | Yes | No | No | Essential | Sensibility |
(1) | (2) | (3) | (4) | (5) | (6) | (7) | (8) |
---|---|---|---|---|---|---|---|
Battery | Concrete | External | No | No | No | Secondary | Literacy |
Bucket | Concrete | External | No | No | No | Secondary | Literacy |
Cat | Concrete | External | No | No | No | Secondary | Literacy |
Chemicals | Concrete | External | No | No | No | Secondary | Literacy |
Electrical Cord/Outlet | Concrete | External | No | No | No | Secondary | Literacy |
Kool-Aid Packet | Concrete | External | No | Yes | No | Advanced | Capability |
Nuclear Waste | Concrete | External | No | Yes | No | Advanced | Capability |
Oil | Concrete | Internal | No | No | No | Secondary | Literacy |
Smoke | Concrete | External | No | No | No | Secondary | Literacy |
Trash | Concrete | External | No | No | No | Secondary | Literacy |
Tree | Concrete | External | No | No | No | Secondary | Literacy |
Appendix B
Rule # | Rule Description for Classifying and Recording Elements | Example (If Necessary) |
---|---|---|
1 | An element must be drawn or described using words to be recorded. | |
2 | An element does not need to be labeled using words to be recorded. | |
3 | An element must be drawn, labeled, and/or described in a “distinguishable” manner to be recorded. A “distinguishable” element is recognizable or identifiable at first glance without extra effort. | |
4 | An element that is drawn and labeled using words shall be recorded exactly as the label is written. | Suppose the element “air/bubbles” are drawn and labeled as “oxygen”. Therefore, the element “oxygen” is recorded and not the element “bubbles”. |
5 | Elements must be classified independently for each drawing (i.e., elements classified in Drawing A cannot influence the elements classified in Drawing B, and vice versa, for the same student or between students). | Suppose student j draws an undistinguishable “blob” in Drawing A with no label. In Drawing B, the student draws the same “blob” with the label of “food”. The “blob” in Drawing A should not be recorded as the element “food” since it is an undistinguishable element when evaluated independently |
6 | If an element is not drawn, labeled and/or described in a distinguishable manner, or if the element is not relevant for a fish-tank system, the element shall not be recorded. | The element “cat” is not considered relevant to a fish-tank system unless it is explicitly connected to other elements or the system. For example, if a student describes how a “cat” tries to get the fish, this means the cat is relevant to the system and can be classified (if necessary) and recorded. |
7 | If two (2) or more instances of the same element are drawn, labeled, and/or described, only one (1) instance of that element shall be recorded per drawing. | If a student draws multiple elements that look like “plants”, only one (1) instance of the element “plants” is recorded. |
8 | If an element that is not included in the defined list of elements (see Table A1 and Table A2 in Appendix A) prior to analysis is drawn, labeled, and/or described, and the element is relevant to a fish-tank system, the element shall be classified using the flowchart (Figure 3) and recorded. | The element “cat” as described in rule #5. |
An element classified at the sensibility level is assigned a score of one (1). | For sensibility: fish, tank, water | |
An element classified at the literacy level is assigned a score of two (2). | For literacy: filter, human, fish food, plants, etc. | |
9 | An element classified at the capability level is assigned a score of three (3). | For capability: electricity, bacteria, broken filter, filtration system, etc. |
10 | The “tank” element shall always be recorded for Drawing B because it is pre-drawn on the worksheet, even if the “tank” is not labeled and/or described. | |
11 | The “tank cover (lid)” element shall only be recorded if it is described or labeled in words as “tank cover” or “tank lid”. A “tank cover (lid)” element that is only drawn shall not be recorded. | |
12 | The “filtration system” element shall only be recorded if it is described or labeled in words as “filter system” or “filtration system”. | |
13 | The “impeller” element shall only be recorded if it is drawn, labeled, or described separately from a “filter” or “filtration system” element. | Suppose an impeller is drawn inside a filter, therefore the “impeller” element shall not be recorded as a separate element. |
Rule # | Rule Description for Classifying and Recording Interactions | Example (If Necessary) |
---|---|---|
1 | An interaction must involve two (2) or more distinguishable elements to be recorded. | “The fish swims” is not an interaction because only one element (“fish”) is involved. This statement would not be recorded as an interaction. |
2 | An interaction must be denoted (drawn) either with arrows/lines or described using words to be recorded (an interaction might be denoted using both arrows/lines and written descriptions). | A line drawn between the elements of “fish” and “food” denotes an interaction between these two elements (at the sensibility level). The description “the fish eats the food” also denotes an interaction between the elements of “fish” and “food” (at the literacy level). |
3 | An interaction must be drawn and/or described in a “distinguishable” manner to be recorded. A “distinguishable” interaction is recognizable or identifiable at first glance without extra effort. | |
4 | Only one (1) interaction shall be assigned between two elements. If multiple interactions are assigned between two elements, then only the interaction classified at the higher systems thinking learning level shall be recorded. | Suppose two interactions, “fish eat food to gain energy” and “fish play with food”, have been assigned between the two elements of “fish” and “food”. Since the interaction of “fish eat food to gain energy” is classified at a capability level (both an action and a reaction are defined), only this interaction shall be recorded between the two elements “fish” and “food”. |
5 | Interaction descriptions must be phrased using verbs, with the exception of the verbs: add, make, give, gave, get, got, keep, help, is, are, allow, or provide. Descriptions using these verbs shall be recorded as a role/purpose and not an interaction. | The description “the fish swims in the water” is recorded as an interaction. However, the description “the fish is swimming in the water” is recorded as a role/purpose because the linking verb “is” is used in the description. |
6 | A part of a drawing or a description that is recorded as an interaction cannot also be recorded as a role/purpose unless both an interaction and a role/purpose are distinguishable. | Suppose a student includes two descriptions that are identical, such as “fish eat food”, where one description is denoted using the word “interaction” and the other description is denoted using the words “role/purpose”. Therefore, the interaction between the elements “fish” and “food” is recorded as “fish eat food” and the role/purpose of the element “fish” is recorded as “fish eat food”. |
7 | If two (2) or more instances of the same interaction are drawn and/or described, only one (1) instance of that interaction shall be recorded per drawing. | If a student draws multiple “fish” elements and connects them all to the element of “food”, only one (1) instance of the “fish-food” interaction is recorded. |
An interaction classified at the sensibility level is assigned a score of one (1). | For sensibility: A line connecting the elements “fish” and “food”. | |
An interaction classified at the literacy level is assigned a score of two (2). | For literacy: A line connecting the elements of “fish” and “food” and the description “eating” which defines an action between the elements. | |
8 | An interaction classified at the capability level is assigned a score of three (3). | For capability: A line connecting the elements of “fish” and “food” and the description “eats to get energy” which defines an action and a reaction between the elements. |
Rule # | Rule Description for Classifying and Recording Roles/Purposes | Example (If Necessary) |
---|---|---|
1 | A role/purpose must be assigned to at least one (1) distinguishable element or system to be recorded. | The description “the tank is for holding everything” can be recorded as a role/purpose, whereas the statement “holds everything” cannot be recorded as a role/purpose unless the statement is assigned to a distinguishable element (see Rule #2). |
2 | A role/purpose must be written in words. A role/purpose can either be assigned as a written label connected to a distinguishable element using lines/arrows or as a written description next to a distinguishable element. | |
3 | A role/purpose must be labeled and/or described in a “distinguishable” manner to be recorded. A “distinguishable” role/purpose is recognizable or identifiable at first glance without extra effort. | |
4 | Only one (1) role/purpose shall be assigned to each element. If multiple roles/purposes are assigned to the same element, then only the role/purpose classified at the higher systems thinking learning level shall be recorded. | Suppose the element “fish” has been assigned both the role/purpose of “to swim around the tank” and “to be a pet”. Since the role/purpose of “to swim around the tank” is classified at a literacy level (from the element’s point-of-view), only this role/purpose shall be recorded for the element “fish”. |
5 | A label or description that includes the prepositions to or for or the pronoun so is always recorded as a role/purpose. | A line connecting the distinguishable element of “fish” to the description “to swim” is recorded as “the role/purpose of the fish is to swim”. The description of “so fish can live” written next to the distinguishable element of “water” is recorded as “the role/purpose of water is so fish can live”. |
6 | Descriptions that include the verbs add, make, give, gave, get, got, keep, help, is, are, allow, or provide are recorded as a role/purpose. Descriptions using other verbs are recorded as an interaction and not a role/purpose. | The description “the fish swims in the water” is recorded as an interaction. However, the description “the fish is swimming in the water” is recorded as a role/purpose because the linking verb “is” is used in the description. |
7 | A part of a drawing or a description that is recorded as a role/purpose cannot also be recorded as an interaction unless both a role/purpose and an interaction are distinguishable. | Suppose a student includes two descriptions that are identical, such as “fish eat food”, where one description is denoted using the word “interaction” and the other description is denoted using the words “role/purpose”. Therefore, the interaction between the elements “fish” and “food” is recorded as “fish eat food” and the role/purpose of the element “fish” is recorded as “fish eat food”. |
8 | If two (2) or more instances of the same role/purpose are labeled or described, only one (1) instance of that interaction shall be recorded per drawing. | If a student draws multiple “fish” elements and labels each with the description “fish make waste”, only one instance of the role/purpose of a fish is to “make waste” is recorded. |
A role/purpose classified at the sensibility level is assigned a score of one (1). | For sensibility: “Rocks are decoration” is a role/purpose assigned to the element “rocks” from the observer’s point-of-view. | |
A role/purpose classified at the literacy level is assigned a score of two (2). | For literacy: “Rocks allow fish to hide” is a role/purpose assigned to the element “rocks” from the element’s point-of-view. | |
9 | A role/purpose classified at the capability level is assigned a score of three (3). | For capability: “The air pump helps circulate the water to improve tank and water health” is a role/purpose assigned to the element “pump” from the system’s point-of-view. |
References
- Suppe, F. The Structure of Scientific Theories, 2nd ed.; University of Illinois Press: Urbana, IL, USA, 1977. [Google Scholar]
- Uebel, T. Vienna Circle. In The Stanford Encyclopedia of Philosophy; Zalta, E.N., Ed.; Available online: https://plato.stanford.edu/archives/spr2019/entries/vienna-circle/ (accessed on 6 August 2019).
- Sloane, E.H. Reductionism. Psychol. Rev. 1945, 52, 214–223. [Google Scholar] [CrossRef]
- Jackson, M.C. Systems Thinking: Creative Holism for Managers; John Wiley & Sons Ltd.: Chichester, UK, 2003. [Google Scholar]
- Dori, D.; Sillitto, H.; Griego, R.M.; McKinney, D.; Arnold, E.P.; Godfrey, P.; Martin, J.; Jackson, S.; Krob, D. System definition, system worldviews, and systemness characteristics. IEEE Syst. J. 2019, 1–11. [Google Scholar] [CrossRef]
- Bunge, M. Systemism: The alternative to individualism and holism. J. Socio-Econ. 2000, 29, 147–157. [Google Scholar] [CrossRef]
- Rousseau, D. Three general systems principles and their derivation: Insights from the philosophy of science applied to systems concepts. In Disciplinary Convergence: Implications for Systems Engineering Research, Proceedings of the 15th Annual Conference on Systems Engineering Research, Redondo Beach, CA, USA, 23–25 March 2017; Madni, A.M., Boehm, B., Erwin, D.A., Ghanem, R., Wheaton, M.J., Eds.; Systems Engineering Research Center (SERC): Hoboken, NJ, USA, 2017. [Google Scholar]
- Checkland, P.B. Systems Thinking, Systems Practice; John Wiley & Sons Ltd.: Chichester, UK, 1981. [Google Scholar]
- Stember, M. Advancing the social sciences through the interdisciplinary enterprise. Soc. Sci. J. 1991, 28, 1–14. [Google Scholar] [CrossRef]
- Hofkirchner, W. Transdisciplinarity needs systemism. Systems 2017, 5, 15. [Google Scholar] [CrossRef] [Green Version]
- Bosch, O.J.H.; Nguyen, N.C.; Maeno, T.; Yasui, T. Managing complexity issues through evolutionary learning laboratories. Syst. Res. Behav. Sci. 2013, 30, 116–135. [Google Scholar] [CrossRef] [Green Version]
- Kahneman, D. Thinking Fast and Slow; Farrar, Straus, and Giroux: New York, NY, USA, 2011. [Google Scholar]
- Crowell, F.A. Systems Literacy and the Literate Design of Educational Systems. In Proceedings of the Thirty-Sixth Annual Meeting for the International Society for the System Sciences (ISSS), Denver, CO, USA, 12–17 July 1992. [Google Scholar]
- Tuddenham, P. Observations on systems literacy at the international society for systems sciences (ISSS) 2016 conference. Syst. Res. Behav. Sci. 2017, 34, 625–630. [Google Scholar] [CrossRef]
- Cabrera, D.; Cabrera, L. Systems Thinking Made Simple: New Hope for Solving Wicked Problems; Odyssean Press: Ithaca, NY, USA, 2015. [Google Scholar]
- Ison, R.; Shelley, M. Governing in the anthropocene: Contributions from systems thinking in practice? Syst. Res. Behav. Sci. 2016, 33, 589–594. [Google Scholar] [CrossRef]
- NGSS Lead States. Next Generation Science Standards: For States, By States; The National Academies Press: Washington, DC, USA, 2013. [Google Scholar]
- Hmelo-Silver, C.E.; Eberbach, C.; Jordan, R. Technology-supported inquiry for learning about aquatic ecosystems. Eurasia J. Math. Sci. Technol. Educ. 2014, 10, 405–413. [Google Scholar] [CrossRef] [Green Version]
- Hmelo-Silver, C.E.; Liu, L.; Gray, S.; Jordan, R. Using representational tools to learn about complex systems: A tale of two classrooms. J. Res. Sci. Teach. 2015, 52, 6–35. [Google Scholar] [CrossRef]
- Liu, L.; Hmelo-Silver, C.E. Promoting complex systems learning through the use of conceptual representations in hypermedia. J. Res. Sci. Teach. 2009, 46, 1023–1040. [Google Scholar] [CrossRef]
- Cabrera, D.A. Systems Thinking. Ph.D. Thesis, Cornell University, Ithaca, NY, USA, 2006. [Google Scholar]
- Cabrera, D.; Colosi, L.; Lobdell, C. Systems thinking. Eval. Program Plan. 2008, 31, 299–310. [Google Scholar] [CrossRef] [PubMed]
- François, C. International Encyclopedia of Systems and Cybernetics, 2nd ed.; K.G. Saur Verlag GmbH: München, Germany, 2004. [Google Scholar]
- Rousseau, D.; Billingham, J.; Calvo-Amodio, J. Systemic semantics: A systems approach to building ontologies and concept maps. Systems 2018, 6, 32. [Google Scholar] [CrossRef] [Green Version]
- Deming, W.E. The New Economics: For Industry, Government, Education, 2nd ed.; Massachusetts Institute of Technology, Center for Advanced Educational Services: Cambridge, MA, USA, 1994. [Google Scholar]
- Ackoff, R.L. On learning and the systems that facilitate it. Reflect. Soc. Organ. Learn. J. 1999, 1, 14–24. [Google Scholar] [CrossRef]
- Buckle, P.B.; Chen, W.-C. Systems thinking: Common ground or untapped territory? Syst. Res. Behav. Sci. 2012, 28, 470–483. [Google Scholar] [CrossRef]
- Creative Learning Exchange (CLE). In Creative Learning Exchange Brochure; Available online: http://static.clexchange.org/ftp/CLEBrochure_2013.pdf (accessed on 7 August 2019).
- HeHfafOpen University (OU). In MSc in Systems Thinking in Practice; Available online: http://www.openuniversity.edu/courses/postgraduate/qualifications/f47 (accessed on 7 August 2019).
- Buckle, P. Maturity models for systems thinking. Systems 2018, 6, 23. [Google Scholar] [CrossRef] [Green Version]
- Hmelo, C.E.; Holton, D.L.; Kolodner, J.L. Designing to learn about complex systems. J. Learn. Sci. 2000, 9, 247–298. [Google Scholar] [CrossRef]
- Jordan, R.C.; Hmelo-Silver, C.; Liu, L.; Gray, S.A. Fostering reasoning about complex systems: Using the aquarium to teach systems thinking. Appl. Environ. Educ. Commun. 2013, 12, 55–64. [Google Scholar] [CrossRef]
- Sensibility. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/sensibility (accessed on 30 July 2019).
- Literacy. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/literacy (accessed on 30 July 2019).
- Literate. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/literate#h1 (accessed on 30 July 2019).
- Capability. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/capability (accessed on 30 July 2019).
- Capable. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/capable (accessed on 30 July 2019).
- Comprehensive. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/comprehensive (accessed on 30 July 2019).
- Comprehension. In Merriam Webster; Available online: https://www.merriam-webster.com/dictionary/comprehension (accessed on 30 July 2019).
- Taylor, S.; Calvo-Amodio, J.; Well, J. A Proposed Methodology for Developing Systems Thinking Lessons By and For Non-Experts. In Proceedings of the American Society for Engineering Management (ASEM) 2018 International Annual Conference, Coeur d’Alene, ID, USA, 17–20 October 2018; Ng, E.-H., Nepal, B., Schott, E., Keathley, H., Eds.; American Society for Engineering Management (ASEM): Huntsville, AL, USA, 2018. [Google Scholar]
- Bacaër, N. Verhulst and the logistic equation (1838). In A Short History of Mathematical Population Dynamics; Springer: London, UK, 2011; pp. 35–39. [Google Scholar]
- Leedy, P.D.; Ormrod, J.E. Practical Research Planning and Design, 11th ed.; Pearson Education, Inc.: New York City, NY, USA, 2016. [Google Scholar]
- Mills, D. The Encyclopedia of the Marine Aquarium; Crescent Books: New York, NY, USA, 1987. [Google Scholar]
- Sharpe, S. Causes and cures for green aquarium water. The Spruce Pets. Available online: https://www.thesprucepets.com/causes-and-cures-for-green-aquarium-water-1378633 (accessed on 27 July 2019).
- Hmelo-Silver, C.E.; Pfeffer, M.G. Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cogn. Sci. 2004, 28, 127–138. [Google Scholar] [CrossRef]
- Lombard, M.; Snyder-Duch, J.; Bracken, C.C. Content analysis in mass communication: Assessment and reporting of intercoder reliability. Hum. Commun. Res. 2002, 28, 587–604. [Google Scholar] [CrossRef]
- Krippendorff, K. Computing Krippendorff’s Alpha-Reliability. In University of Pennsylvania Scholarly Commons: Annenberg School of Communication; Available online: https://repository.upenn.edu/asc_papers/43/ (accessed on 10 September 2019).
- Neuendorf, K.A. The Content Analysis Guidebook; Sage: Thousand Oaks, CA, USA, 2002. [Google Scholar]
- Montgomery, D.C. Design and Analysis of Experiments, 8th ed.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2013. [Google Scholar]
- Conover, W.J. Practical Nonparametric Statistics, 3rd ed.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 1999. [Google Scholar]
Pattern | Classification | Definition |
---|---|---|
(1) Elements that are visible or invisible | Concrete Elements | A concrete element is visible (i.e., can be seen with the naked human eye). Examples: fish, plants, filter. |
Conceptual Elements | A conceptual element is invisible (i.e., cannot be seen with the naked human eye). Examples: bacteria, oxygen, nitrogen. | |
(2) Elements inside or outside the system boundary | Internal Elements | An internal element is located primarily inside of the system boundary (i.e., the walls of the fish tank). Examples: rocks, fish food, air/water pump. |
External Elements | An external element is located primarily outside of the system boundary (i.e., the walls of the fish tank) in the environment. Examples: fish net, sunlight, tank stand. | |
(3) Elements that are critical to the system’s definition | Essential Elements | An essential element is crucial data or information used to define what a fish-tank system is. Without an essential element, the system cannot be defined as a fish-tank system. For a fish-tank system, the essential elements are fish, tank, and water. |
(4) Elements that are underlying causes of the problem situation | Secondary Elements | A secondary element enhances knowledge about how a fish-tank system works. The addition or removal of a secondary element does not affect the definition of a fish-tank system. Examples: filter, algae, human. |
(5) Elements that are labeled as systems | Advanced Elements | An advanced element enhances understanding about why the fish-tank system is behaving a certain way. Advanced elements include all conceptual elements, elements that are underlying causes of the problem situation, and elements that are labeled as sub-systems within the greater fish-tank system. Examples: ammonia, bacteria, filtration system, dead organisms. |
Learning Level | Classification | Description |
---|---|---|
Sensibility | 1. Concrete, Internal, Essential Elements | All elements at the sensibility level are essential meaning these elements are necessary for defining what a fish-tank system is. Additionally, all elements at this level are visible and are found inside the fish-tank system boundary. |
Literacy | 1. Concrete, Internal, Secondary Elements | All elements at the literacy level are secondary meaning these elements enhance knowledge about how the fish-tank system works. Additionally, all elements at this level are visible and can be found both inside and outside the fish-tank system boundary. |
2. Concrete, External, Secondary Elements | ||
Capability | 1. Concrete, Internal, Advanced Elements | All elements at the capability level are advanced meaning these elements enhance understanding about why the fish-tank system is behaving the way it is. Additionally, elements at this level can be either visible or invisible and can be found both inside and outside the fish-tank system boundary. |
2. Concrete, External, Advanced Elements | ||
3. Conceptual, Internal, Advanced Elements | ||
4. Conceptual, External, Advanced Elements |
Pattern | Classification | Definition |
---|---|---|
(1) Non-action and non-reaction interactions | Non-action Interactions | A non-action interaction does not explain how or why two or more elements are interacting. No clear action is defined. |
Non-reaction Interactions | A non-reaction interaction does not explain the effect(s) of an interaction between two or more elements. No clear reaction is defined. | |
Simple Interactions | A simple interaction demonstrates awareness that two or more elements relate, or that the elements are interacting in some way, but no clear action and no clear reaction is defined. | |
(2) Action and non-reaction interactions | Action Interactions | An action interaction does explain how two or more elements are interacting. A clear action is defined. |
(3) Non-action and reaction interactions | Intermediate Interactions | An intermediate interaction demonstrates awareness that two or more elements relate, or that the elements are interacting in some way, and also demonstrates knowledge about either the action (how two or more elements are interacting) or the reaction (the effect(s) of an interaction between two or more elements). |
Reaction Interactions | A reaction interaction does explain the effect(s) of an interaction between two or more elements. A clear reaction is defined. | |
(4) Action and reaction interactions | Advanced Interactions | An advanced interaction demonstrates both the awareness that two or more elements relate and the knowledge of actions and reactions, and also demonstrates the understanding of the interplay between the action and reaction (i.e., the cause-and-effect relationship). |
Learning Level | Classification | Description |
---|---|---|
Sensibility | 1. Non-action, Non-reaction, Simple Interactions | All interactions at the sensibility level are simple, meaning interactions only demonstrate an awareness that two or more elements relate. Interactions at this level do not explain how two or more elements are interacting or the effect(s) of the interaction between two or more elements. No clear action or reaction is defined. |
Literacy | 1. Action, Non-reaction, Intermediate Interactions | All interactions at the literacy level are intermediate, meaning interactions demonstrate awareness that two or more elements relate and knowledge about either how two or more elements are interacting or the effect(s) of the interaction between two or more elements. In case (1) where a clear action is defined, no clear reaction is defined. Conversely, in case (2) where a clear reaction is defined, no clear action is defined. |
2. Non-action, Reaction, Intermediate Interactions | ||
Capability | 1. Action, Reaction, Advanced Interactions | All interactions at the capability level are advanced, meaning interactions demonstrate both awareness that two or more elements relate and knowledge about the actions and reactions between two or more elements, and also an understanding of why the action causes the reaction. A clear cause-and-effect relationship is defined. |
Pattern | Classification | Learning Level | Definition |
---|---|---|---|
(1) Observer’s point-of-view | Individualistic Roles/Purposes | Sensibility | A role/purpose at the sensibility level demonstrates awareness that an element in a fish-tank system has a role/purpose, but that role is individualistically focused. Roles/purposes at this level are viewed from the observer’s point-of-view of the fish-tank system. |
(2) Element’s point-of-view | Elementalistic Roles/Purposes | Literacy | A role/purpose at the literacy level demonstrates knowledge about the role/purpose of an element in a fish-tank system, but that role or purpose is elementalistically focused. Roles/purposes at this level are viewed from the element’s point-of-view to explain how that role/purpose relates to other elements in the fish-tank system. |
(3) System’s point-of-view | Systemic Roles/Purposes | Capability | A role/purpose at the capability level demonstrates understanding about the role/purpose of an element in a fish-tank system, but that role is systemically focused. Roles/purposes at this level are viewed from the system’s point-of-view and explain why elements exist in the fish-tank system. |
Item Coded | Part #1 | Part #2 | Part #3 | |||
---|---|---|---|---|---|---|
(5 Students; 10 Drawings) | (5 Students; 10 Drawings) | (25 Students; 50 Drawings) | ||||
Elements (E) | A: 67% | A + B: 70% | A: 88% | A + B: 91% | A: 90% | A + B: 91% |
B: 74% | B: 95% | B: 93% | ||||
Interactions (I) | A: 100% | A + B: 68% | A: 100% | A + B: 87% | A: 96% | A + B: 85% |
B: 37% | B: 73% | B: 74% | ||||
Roles/Purposes (R) | A: 80% | A + B: 65% | A: 100% | A + B: 72% | A: 96% | A + B: 88% |
B: 50% | B: 44% | B: 80% | ||||
Totals (T) | A: 82% | A + B: 68% | A: 96% | A + B: 83% | A: 94% | A + B: 88% |
B: 53% | B: 71% | B: 82% |
Drawing | [S] | [S] | [L] | [L] | [C] | [C] | [T] | [T] | [T] | [T] |
---|---|---|---|---|---|---|---|---|---|---|
A | 2.47 | 3 | 2.97 | 3 | 0.19 | 0 | 5.63 | 5 | 8.97 | 8 |
B | 2.92 | 3 | 4.73 | 4 | 0.63 | 0 | 8.28 | 8 | 14.27 | 13 |
B—A | 0.44 | 0 | 1.76 | 2 | 0.44 | 0 | 2.65 | 3 | 5.30 | 5 |
Test | Null Hypothesis | Test Statistic (T) | Critical Value (zcritical) | p-Value | |
---|---|---|---|---|---|
Two-tailed (x) | 81 | 7.35 | 1.96 | <0.001 | |
Two-tailed (z) | 86 | 7.45 | 1.96 | <0.001 | |
Left-tailed (x) | 81 | 7.35 | 1.64 | >0.999 | |
Left-tailed (z) | 86 | 7.45 | 1.64 | >0.999 | |
Right-tailed (x) | 81 | 7.35 | 1.64 | <0.001 | |
Right-tailed (z) | 86 | 7.45 | 1.64 | <0.001 |
Drawing | [S] | [S] | [L] | [L] | [C] | [C] | [T] | [T] | [T] | [T] |
---|---|---|---|---|---|---|---|---|---|---|
A | 0.14 | 0 | 0.07 | 0 | 0.03 | 0 | 0.25 | 0 | 0.38 | 0 |
B | 0.66 | 0 | 1.30 | 0 | 0.06 | 0 | 2.02 | 1 | 3.44 | 2 |
B—A | 0.52 | 0 | 1.23 | 0 | 0.03 | 0 | 1.77 | 1 | 3.06 | 2 |
Test | Null Hypothesis | Test Statistic (T) | Critical Value (zcritical) | p-Value | |
---|---|---|---|---|---|
Two-tailed (x) | 57 | 6.61 | 1.96 | <0.001 | |
Two-tailed (z) | 60 | 6.72 | 1.96 | <0.001 | |
Left-tailed (x) | 57 | 6.61 | 1.64 | >0.999 | |
Left-tailed (z) | 60 | 6.72 | 1.64 | >0.999 | |
Right-tailed (x) | 57 | 6.61 | 1.64 | <0.001 | |
Right-tailed (z) | 60 | 6.72 | 1.64 | <0.001 |
Drawing | [S] | [S] | [L] | [L] | [C] | [C] | [T] | [T] | [T] | [T] |
---|---|---|---|---|---|---|---|---|---|---|
A | 0.00 | 0 | 0.11 | 0 | 0.00 | 0 | 0.11 | 0 | 0.23 | 0 |
B | 0.53 | 0 | 1.58 | 1 | 0.00 | 0 | 2.10 | 2 | 3.68 | 4 |
B—A | 0.53 | 0 | 1.46 | 1 | 0.00 | 0 | 1.99 | 2 | 3.45 | 3 |
Test | Null Hypothesis | Test Statistic (T) | Critical Value (zcritical) | p-Value | |
---|---|---|---|---|---|
Two-tailed (x) | 74 | 7.37 | 1.96 | <0.001 | |
Two-tailed (z) | 74 | 7.32 | 1.96 | <0.001 | |
Left-tailed (x) | 74 | 7.37 | 1.64 | >0.999 | |
Left-tailed (z) | 74 | 7.32 | 1.64 | >0.999 | |
Right-tailed (x) | 74 | 7.37 | 1.64 | <0.001 | |
Right-tailed (z) | 74 | 7.32 | 1.64 | <0.001 |
Drawing | [S] | [S] | [L] | [L] | [C] | [C] | [T] | [T] | [T] | [T] |
---|---|---|---|---|---|---|---|---|---|---|
A | 2.62 | 3 | 3.15 | 3 | 0.22 | 0 | 5.99 | 5 | 9.58 | 8 |
B | 4.10 | 3 | 7.61 | 7 | 0.69 | 0 | 12.40 | 12 | 21.39 | 19 |
B—A | 1.48 | 1 | 4.45 | 4 | 0.47 | 0 | 6.41 | 6 | 11.81 | 10 |
Test | Null Hypothesis | Test Statistic (T) | Critical Value (zcritical) | p-Value | |
---|---|---|---|---|---|
Two-tailed (x) | 94 | 8.42 | 1.96 | <0.001 | |
Two-tailed (z) | 95 | 8.46 | 1.96 | <0.001 | |
Left-tailed (x) | 94 | 8.42 | 1.64 | >0.999 | |
Left-tailed (z) | 95 | 8.46 | 1.64 | >0.999 | |
Right-tailed (x) | 94 | 8.42 | 1.64 | <0.001 | |
Right-tailed (z) | 95 | 8.46 | 1.64 | <0.001 |
Systems Thinking Concept | (1) Expected Learning (EL) | (2) Systems Thinking Learning (STL) | (3) EL/STL Score Threshold | (4) Number (%) of Students with Score > Threshold 1 | (5) EL Average Score % 2 | (6) STL Average Score % 2 |
---|---|---|---|---|---|---|
Elements | 70.11% | 29.89% | 3.72 = 4.00 | 53 (54.64%) | 68.41% | 31.59% |
Interactions | 18.65% | 81.35% | 0.57 = 1.00 | 53 (54.64%) | 34.19% | 65.81% |
Roles/Purposes | 10.77% | 89.23% | 0.37 = 1.00 | 71 (73.20%) | 30.32% | 69.68% |
Totals | 50.55% | 49.45% | 5.97 = 6.00 | 78 (80.41%) | 58.91% | 41.09% |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Taylor, S.; Calvo-Amodio, J.; Well, J. A Method for Measuring Systems Thinking Learning. Systems 2020, 8, 11. https://doi.org/10.3390/systems8020011
Taylor S, Calvo-Amodio J, Well J. A Method for Measuring Systems Thinking Learning. Systems. 2020; 8(2):11. https://doi.org/10.3390/systems8020011
Chicago/Turabian StyleTaylor, Seth, Javier Calvo-Amodio, and Jay Well. 2020. "A Method for Measuring Systems Thinking Learning" Systems 8, no. 2: 11. https://doi.org/10.3390/systems8020011