The Influence of Problem Construction on Undergraduates’ Success with Stoichiometry Problems

Abstract

Although there are numerous studies that aim to reveal the source of student failure in problem solving in STEM fields, there is a lack of attention on testing different methods to identify what works best in improving students’ problem-solving performance. In this study, the authors examined the influence of the type of problem construction intervention and compared it to the effect of traditional practice on 38 general chemistry students’ comprehension of problem-solving process as well as overall success with given stoichiometric problems. To determine students’ success with each subtopic involved in stoichiometric problems and to better understand the source of difficulty at a finer level rather than focusing on the end product as practiced in most studies, students’ solutions were examined using the COSINE (Coding System for Investigating Sub-problems and Network) method. The findings revealed that students who practiced the problem-construction method outperformed their counterparts in the control group who followed a traditional approach during their study session. An in-depth analysis also showed that the experimental group improved their success with seven out of nine subtopics while three topics observed an increase in the control group. The practical implication of the problem-construction method was discussed for a wider adoption by textbook publishers and educators across different disciplines.

1. Introduction

As the information age advances, problem solving is becoming an increasingly vital skill set required in every stage of our academic and professional lives [1]. Businesses are becoming less interested in employees specializing in one specific area but instead prefer hiring individuals who have interdisciplinary experience and excellent problem-solving abilities [2]. Introductory science courses, including general chemistry courses, could contribute to developing this important 21st century skill if students are taught using research-based effective methods. In these courses, a lack of an in-depth understanding of the presented topics encourages students to memorize the facts and use the same strategies for a wide range of problems [3]. Mikula and Heckler [4] pointed out this common issue observed in many STEM classes and mentioned that “many students have difficulties with basic, [but very] essential skills in STEM courses”. Stoichiometry is one of the first topics in chemistry curriculum that challenges students’ problem-solving ability and understanding of chemical concepts [5]. Some of the challenges students face can be explained by exploring students’ chemistry self-efficacy and motivation levels [3,6,7]. Although it is important to target and examine as many of these variables as possible to better identify the source and the nature of the challenges, the goal of this study was to investigate students’ conceptual understanding of chemistry topics, as measured by their ability to connect knowledge pieces and changes in students’ problem-solving performances associated with the presented methods. Students’ epistemological beliefs and their perception of learning and conceptual change impact the way that they learn and approach their education [1,3]. Students have different conceptions about the structure of knowledge. Many view it as one large block and some perceive it as a highly complex system built with many richly connected small pieces [3,5]. As students’ beliefs and perceptions on how learning occurs and knowledge is constructed are critical in developing problem-solving skill, this study focused on making the knowledge assembly process more explicit by encouraging the students to investigate the connections between different subtopics of stoichiometry. As emphasized in the constructivist theory, improving students’ ability of connecting all the studied concepts, not only within the same chapter but across the whole curriculum and even over different courses helps them develop a better conceptual understanding of chemistry topics, and, in turn, excel at solving problems [5]. Differences in students’ success with solving chemistry problems are related to the idea of concept building and how students navigate conceptual tasks [5,6]. The Complete Success Rate (CSR) formula generated by Gulacar et al. [8] and tested in previous studies [9,10,11,12] to reveal the connection between students’ conceptual understanding and problem-solving performance was utilized in this study as well. The authors shared the details regarding the determination of the CSR scores and their analysis in the methods section.

One of the biggest conceptual barriers in solving stoichiometry problems among all subtopics presented in stoichiometry, the mole concept, has been identified as a major source of struggle yielding unsuccessful attempts in solving related questions [13]. By spending their attention and time trying to memorize how to find moles from given examples and worksheets, students are not able to build their problem-solving skills because they are not spending time understanding the complex interactions amongst chemistry concepts and how they are connected in a solution. Rather, they tend to focus on superficial elements that distract them from the actual task when dealing with problems [14]. This aspect clearly differentiates novices from experts, who have a more thorough understanding of the subject and categorize questions based on underlying principles without being barred by the surface elements of the questions [15,16,17]. Mikula and Heckler [4] believe that this difference is one of the fundamental causes behind novice’s poor performance in STEM fields as the mastery of this elementary skill is vital for meaningful learning. When students identify the type of problem, they become more successful with the use of proper strategies and are able to put the given information together to generate correct solutions instead of being distracted by surface elements presented in the problem [14]. Gulacar et al. [5] and Day et al. [18] found that the main difference between low- and high-achieving students in stoichiometry problems was their ability to interpret the given information in the question body and connect the necessary steps to solve the problem. In Day et al.’s study [18], eye-tracking technology was used to discover which part of the stoichiometry problems students were struggling most to solve. What they discovered was that low-achieving students could accurately identify the useful information within a problem but could not figure out how to combine the given information to compute an answer [18]. After analyzing the data collected through think-aloud protocols, Gulacar et al. [5] also discovered that students struggle to link the necessary subproblems together. For example, in several instances, students remembered how to convert the grams of reactant into moles and find the mass of desired product; however, they forgot that they needed to test for the limiting reagent between these two steps [5]. These studies indicate that students need more help with developing a better conceptual understanding and bridging the pieces of information given in the problem. In other words, they need to see how the subproblems come together to generate a successful solution.

Special attention and time need to be allotted to improve students’ problem-solving strategies, which is not readily happening in today’s classrooms. Instead, educators rely heavily on standardized tests, which mainly measure student memorization of content knowledge [19]. Therefore, students are not focusing on developing a strong conceptual understanding but, instead, are prioritizing receiving a good grade on their tests; in essence, high scores are not always associated with a meaningful understanding [19,20]. Standardized tests are ineffective at revealing students’ true understanding of concepts and pinpointing weaknesses in their conceptual knowledge or problem-solving abilities [20]. Many students do not realize that problem solving is a great learning tool because it is mostly used as part of summative assessments to measure their level of understanding [21]. Therefore, they think their sole goal is to get an answer without worrying about the process used to get the answer or reflecting on how the solutions are obtained at a deeper level [22]. They do not realize that problem solving is what we do when we do not know what to do [23]. While solving problems, consciously or unconsciously, the problem solver starts to make connections between different concepts involved in the questions, which is seen as synonymous with the process of learning defined by many learning theorists as the reorganization of information we have in our minds [24,25]. As students solve problems, they become more aware of the discrepancies that exist between their own understanding of topics and scientifically accepted theories and refine their knowledge [26]. At the same time, problem solving encourages students to retrieve needed information from their long-term memory and process it every time, which promotes a better understanding of topics and helps them learn new information by molding it into a frame that makes sense to them. However, research shows that many students struggle to retrieve needed pieces of information during problem solving [27]. There is a need for more effective strategies assisting students to create rich, well-connected, and organized knowledge structures, so they can become more successful at applying their knowledge to various problem situations.

In a study conducted by Angawi [28] to improve student’s understanding of proton NMR, a method called skillful thinking approach was used to teach the students the thought-process behind each step which was taken to solve the problem instead of just providing a set of cues and signals to use in interpreting NMR spectroscopy data. Here, the researchers aimed to identify if students have the conceptual framework to identify the steps needed to solve problems. Another study also examined conceptual frameworks of students by analyzing the questions students asked out loud during classroom activities with a goal of understanding student’s conceptual comprehension and to provide insight to teachers about specific topics students needed help [29]. Chen et al. [27] specifically investigated the difficulties students experience in drawing connections between problem-solving and conceptual thinking. They provided students with solutions which incorporated mistakes to help them identify errors in the solutions and improve their problem-solving ability. Researchers in these two studies pointed out the importance of providing students with the conceptual tool set so that they can identify sub questions that they should be considering and figure out why they are following certain steps in solving a problem [27,28,30].

In addition, there are studies looking into the effects of students grading their peer’s work on their ability to identify the required knowledge and strategies in solving problems. For example, by evaluating their peers’ work, Scott [31] determined that the students were able to improve their overall problem-solving skills as measured with pre- and post-tests which included questions similar to the ones found in the peer-assessment activities. This study also concluded that the students avoided common sources of error by engaging in these activities [31]. Learning about the benefits of evaluating other students’ work encouraged several researchers [32,33] to study the effects of students’ generating their own questions on their problem-solving performance. They particularly aimed to determine how these interventions help them think through the conceptual framework and retrieve the concepts they learned in the classroom to generate a successful solution. Botelho et al. [34], in this specific study conducted in Hong Kong, asked freshman to create multiple-choice questions with the intention of helping them see the connections among different concepts in classes and develop a stronger conceptual understanding. However, the study did not result in strong conclusions on the effects of these practices due to some observed limitations such as the students’ unfamiliarity with the nature of multiple-choice questions and in developing them, thus they failed to see the purpose behind their own and peer generated questions [33]. In addition, this study only analyzed post surveys completed by students to determine how the students perceived this activity, heavily emphasizing the student’s opinions, thus, providing limited data on the effect of creating multiple-choice questions on their problem-solving achievement.

Overall, these studies [27,28,31,32,33] have evaluated the efficacy of various methods in enhancing students’ conceptual understanding of chemistry topics along with disclosing the benefits of evaluating peer’s work and multiple-choice question creation to some extent. Considering the advantages and the limitations of the methodologies presented in these studies, a new approach, problem construction, also called reverse engineering, was proposed and tested in this study. In this approach, students were provided with solutions to chemistry questions and asked to create a question that corresponds with that solution.

Angawi [28] and Chen et al. [27] aimed to help students develop conceptual understanding which is seen as synonymous to being able to connect different knowledge components as practiced in problem solving [34,35,36]. It was predicted that creating open-ended questions as practiced in problem construction would be a more effective way to help students see the connections between different chemistry concepts assessed with a wide range of subproblems in stoichiometry. On the other hand, Bothelo et al. [33] investigated the effects of creation of multiple-choice questions and Scott [31] examined the influence of peer evaluation on students’ problem-solving performance. Considering the complexities with generating multiple-choice questions, our intervention required that students generate open-ended questions using the given solution as a guide. In our study, it was hypothesized that problem construction would provide students a chance to evaluate previously developed solutions, mimicking peer-evaluation, and go beyond multiple-choice question creation and encouraging them to be more creative with constructing open-ended questions, which in turn is expected to be more effective in helping them build stronger connections between chemistry concepts and applying them across different stoichiometry problems. It was also noted that the impact of open-ended question creation has not received sufficient attention from the scientific community yet. Therefore, the goal of our investigation was to compare the effects of a common practice method that students are most familiar and use to get an answer to given problems to those of a naïve approach, problem construction, on their success with solving stoichiometry problems.

The aforementioned studies also lack an in-depth analysis on the effects of their methods on students’ overall problem-solving performance and almost totally ignore the interaction between students’ success with individual subproblems and getting a correct answer to the overall problem in their analysis. The analysis of the student-generated questions in our study was done at a finer scale; instead of looking at how problem construction improved student’s scores, we studied how it helped improve each sub step within the problems and their ability in connecting them.

2. Methods

2.1. Research Questions

Among the many effective methods used in problem solving studies, this study focuses on problem construction to answer the following questions:

• How does the problem-construction intervention affect student’s performance with problem solving?
• How do high- and low- achieving students benefit from the implementation of problem-construction intervention? Do their performances with stoichiometry problems change differently?

2.2. Participants

The study was conducted at a public research university that is on a quarter system and located in Northern California. After getting the IRB approval, students registered for the second course (CHE 2B) in the General Chemistry series were invited to complete an online survey to determine their grades in CHE 2A, the previous course. Out of 854 students, 99 indicated an interest to participate in the research project. Considering their performance in CHE 2A, 70 of them were invited to join the study and split into experimental and control groups with very similar grade distribution. Since the students who earn a grade lower than a C in CHE 2A cannot move to CHE 2B, in this study, we only had students who earned an A, B, or C. The team did everything with its power to achieve homogenous sampling. However, 32 students quit at different stages of the study due to various reasons, which changed homogeneity of the groups and limited the investigations of differences between control and experimental groups. Some of the common reasons for quitting included having discomfort with revealing their weaknesses during the think-aloud protocols, the time pressure in a fast-paced quarter system, and health issues. Therefore, their data were excluded from the analysis. While the study started with an equal number of participants in each group, these separations mostly affected the control group.

2.3. Design and Instrument

The study took place in the Winter Quarter about ten weeks after the topic of interest, stoichiometry, was covered. The data collection was fulfilled in three stages consisting of pre-test, intervention, and post-test as shown in Figure 1. All the participants in each group, the experimental and control groups, completed all three stages, but the control group did not have the same intervention.

The students in both groups solved the same pre- and post-test questions while thinking aloud during the weeks of 6–7 and weeks of 9–10, respectively. In these sessions, the participants were asked to make their thinking process audible as much as possible. When they forget to speak, they were reminded to reveal what is going on in their mind. The total time allotted for them to complete the questions was 30 min. During the interviews, if participants got stuck at the first step and could not determine a chemical equation, they were provided standardized hints on chemical formulas and symbols to help them move forward. At the end of the pre-test session, the participants were invited to come back after a week for the intervention without giving much detail about the nature of exercises they would complete.

During the intervention session, instead of presenting questions to work on, the students in the experimental group were given solutions accompanied by detailed steps and asked to construct and write questions that would generate those expected solutions. The goal of this reverse-engineering problem set was to challenge students to creatively devise questions that frame the numerical values, steps, and elemental substances included in the questions. Similar to the pre-test, certain hints were provided as necessary to allow the students to move forward with their work. At the end of the session, each participant wrote 11 questions within 75 min. On the other hand, the control group intervention mimicked what is commonly performed by students during their study time. They were provided with the same number of questions and asked to solve them. After completing each question, they were provided with the corresponding solutions and encouraged to compare their own solutions to those provided by the research team before moving to the next question.

Due to students’ limited availability, the post-test had to be scheduled one week after the interventions were implemented. The post-test included modified versions of the questions used in the pre-test. To minimize and eliminate the effect of memorization on their ability to solve problems, some of the chemicals and the numbers in the questions were changed. It was ensured that these modifications did not cause any additional difficulty to the questions. All the sessions were audio recorded and transcribed for coding and analysis. Think-aloud protocols led the team to capture participants’ thinking process, struggles, and unique challenges [27]. These transcripts were triangulated with participants’ solutions and interviewers’ observations.

The pre- and post-test included five stoichiometry questions with varying levels of difficulty. As a topic, stoichiometry was selected because its mastery is imperative in understanding the relationship between reactants and products in a reaction and learning how to manipulate the given information to determine a successful solution is a prerequisite to following topics covered in general and advanced chemistry courses [27]. Since stoichiometry is a multi-faceted topic that necessitates understanding of several sub-topics ranging from conversion of mass to limiting reagents, the questions were designed in a way that students can show their performance with each subtopic, and the research team can identify their challenges with them. Considering all these factors, understanding students’ performance and challenges with solving overall stoichiometric problems and the subproblems involved was deemed to be critical to help a large group of students struggling in the general chemistry series [12,34,35].

2.4. Data Analysis

Upon the completion of the interviews, students’ solutions were scanned digitally and uploaded to Gradescope, an online grading site, while all the audio recordings were transcribed verbatim.5 A sample coding screen from Gradescope was shared in Figure 2. After organizing all the data, the team applied the COSINE (Coding System for Investigating Sub-Problems and Network) [8,10,11] to systematically analyze students’ success with each subproblem and pinpoint the exact steps that students got lost and frustrated.

In this study, a subproblem was defined as one-step calculation that usually involves one stoichiometric subtopic (e.g., mole concept, limiting reagent, or percent yield). There was a total number of 29 subtopics from all of the questions. Since each student did the same question twice, both in pre- and post-tests, 58 data points were collected for each participant regardless of their groups.

The COSINE comprises 8 codes: S (successful), NR (Not Required), DD (Did not Know to Do), DSE (Did Something Else), CD (Could Not Do), UG (Unsuccessful-Guessed), URH (Unsuccessful-Received Hint), UDI (Unsuccessful-Did Incorrectly) [37]. Broken up into three different sections, the 8 codes of COSINE are categorized as successful, neutral, and unsuccessful codes. Within the successful codes category, only one code exists: S (Successful). Of the remaining seven codes, NR (Not Required), DD (Did Not Know to Do), and DSE (Did Something Else) fall into the neutral codes. On the other hand, the unsuccessful codes include CD (Could Not Do), UG (Unsuccessful-Guessed), URH (Unsuccessful-Received Hint), and UDI (Unsuccessful-Did Incorrectly). Organizing each code into its respective category allows us to identify the nature and the sources of students’ challenges.

Examples of how each code was applied are included below in

Table 1. Examples for the application of the COSINE codes *.

Codes	Subtopic	Students’ Thinking Process	Student Work **
S	MC (Finding Moles of Carbon)	In this solution, the participant correctly identifies that he needs to use stoichiometry to go from grams to moles. The participant says, “I am going to convert everything I see here into moles. So, 40 g of carbon, I am going to use stoichiometry. Carbon is 12.011 g and then that equals to 1 mole. I am going to take 40 and divide that by 12.01 to get 3.33 moles of Carbon.”
DD	LR (Identifying Limiting Reactant)	The participant states he does not know how to do this problem and he also does not realize that he needed to calculate the limiting reactant using the two compounds. He thinks both would equally contribute the overall mass of N2. He states, “I started by writing down the two compounds given […] the question says it wants you to find the grams of N2 gas […] and now I am trying to find the grams of Nitrogen in this compound, so the 100 g is the total mass of N2H4 and I am trying to find Nitrogen [from it]. I am dividing the mass of nitrogen by the total mass to get the percentage of the mass of nitrogen in N2H4”. “I don’t know how to do this, but with my math knowledge it makes sense to say that there is the same amount of nitrogen in both of them [the two compounds].”
DSE	EF (Empirical Formula Ratio)	The participant multiplies the mole ratios by a common multiple to change the fraction into a whole number. His reasoning for manipulating the mole quantities is not the way specified in the traditional method. In doing so, he does not realize that he is changing the empirical formula he calculated. As they state, “carbon turned out to be 2.67 moles, we know that that is equal to two and two thirds, so we multiply by three to get rid of the fraction. So, you have the formula should be C8H12O12.”

* The detailed explanations for each code can be found in Gulacar et al. [10]. ** Highlighted region represents the subproblem being coded for that question.

and

Table 2. Examples for the application of the COSINE codes *.

Codes	Subtopic	Students’ Thinking Process	Student Work **
CD	MC (Finding Moles of N2H4)	The participant clearly recognizes that this is a limiting reactant problem by stating “limiting reactant” when comprehending the question. She says she needs to “figure out which is the limiting” however she has a hard time recalling the exact steps. It is observed that she knows what needs to be done and as she mentions how there was a “proper procedure” however it “does not stick in my head.”
UG	PY (Calculating Percent Yield)	In this subproblem, the participant clearly guesses. He states, “I don’t know, I’m just gonna say 100% yield”.
URH	WEQ (Writing the Equation)	The participant is struggling to write the formula for strontium halide and the interviewer actually provides it. The interviewer states, “But I’ll just help you a little bit. […] the charge of strontium is plus two so in the halide it is a negative one so it’ll be a SrX2.”
UDI	MC (Finding Moles of Carbon)	The participant does the incorrect stoichiometric calculation to go from grams to moles carbon. Instead of dividing the grams of carbon they state that “I have to multiply the mass by the percentage and then add them up to see if it makes the total mass”.

* The detailed explanations for each code can be found in Gulacar et al. [10]. ** Highlighted region represents the subproblem being coded for that question.

. The team developed a detailed key for the questions used in the think-aloud interviews and used it as a rubric to assign the codes. The answer key showed the most common way to approach and solve each sub-problem and helped the coders compare students’ solutions to the expected solutions and assign the codes. While coding the subproblems, the coders checked students’ solutions and transcripts at the same time to determine the best code for the work completed. Each subproblem was coded independently without interference from the preceding and subsequent subproblems. In several cases, regardless of the correctness of the final answer, the intermediate steps were coded as successful when found appropriate. The coders not only evaluated the overall answer students reported but also scrutinized their performance with the individual sub-problems. Before the three coders started coding assigned students’ solutions, the inter-rater reliability was determined by utilizing Krippendorf’s alpha, [37] which was determined to be 0.73.

Upon finishing the coding both the pre- and post-tests, the codes were used in two formulas, Attempt Success Rate (ASR) and the Complete Success Rate (CSR), to enhance the analysis of hundreds of assigned codes and make the changes in students’ performances clearer.

The Attempt Success Rate (ASR) takes students’ scores into consideration and removes the codes assigned when students completely forgot or did not know to do a specific step to continue to the correct solution (DD) or did an irrelevant calculation (DSE) which was not appropriate to get the right answer. In other words, the ASR formula does not factor in the neutral codes (DD and DSE). On the other hand, the Complete Success Rate (CSR) takes all possible mistakes into consideration and factors in all the codes of the COSINE. It should be noted that code NR was not included in either formula because this code was only applied when the students used a new subproblem within a correct method different from the one used in the rubric. To keep the coding consistent across all participants, their presence was not involved in the calculations, but novelty in their methods was documented for further analysis.

$$\mathrm{ASR}=\frac{\mathrm{S}}{\left(\mathrm{S}+\mathrm{CD}+\mathrm{UG}+\mathrm{UDI}+\mathrm{URH}\right)}$$

$$\mathrm{CSR}=\frac{\mathrm{S}}{\left(\mathrm{S}+\mathrm{DD}+\mathrm{DSE}+\mathrm{CD}+\mathrm{UG}+\mathrm{UDI}+\mathrm{URH}\right)}$$

The values obtained from the ASR formula usually appear to be higher than the values obtained from the CSR formula [36]. It is hypothesized that after a successful intervention the gap between ASR and CSR will be narrowed down and even closed, which means students become more successful with connecting sub-problems without doing any unnecessary calculations indicated by low number of the code DSE and with generating a problem schema that has all the sub-problems in the best network possible. The difference between the ASR and CSR formula is crucial in determining if there is a correlation and increase in success rate from the intervention. Additionally, the difference provides substantial data in revealing whether students are connecting sub-problems in stoichiometry and understanding the relationship. If the difference between the ASR and CSR is close to 0 for problems, it is interpreted that the students are becoming more successful in figuring out what subproblems required for the given problem and putting them in a correct schema to get an accurate answer [36]. In other words, students are becoming more successful with doing individual sub-problems and connecting them. If the difference between the ASR and CSR is far from 0 for a specific problem, the student has gone off track and either done random calculations (DSE) or could not figure out what exactly needs to be done (DD), which could be an important sign for poor conceptual understanding. To help the reader find the meaning of the acronyms used in this paper easily,

Table 3. Acronyms for stoichiometric concepts and COSINE codes.

Acronym	Stoichiometry Topics	Acronym	COSINE CODES and Formulas
BEQ	Balancing Chemical Equation	S	Successful
WEQ	Writing Chemical Equation	UG	Unsuccessful Guessed
MC	Mole Concept	UDI	Unsuccessful Did Incorrectly
SR	Stoichiometric Ratio	URH	Unsuccessful Received Hint
PY	Percent Yield	CD	Could not Do
CM	Conservation of Mass	DSE	Did Something Else
EF	Empirical Formula	DD	Did not know to Do
MF	Molecular Formula	NR	Not Required
LR	Limiting Reagent	ASR	Attempt Success Rate
		CSR	Complete Success Rate

was prepared.

3. Results and Discussions

The analysis of our findings focused on examining the impact of the problem-construction method on students’ problem-solving performance and to determine if any student group (e.g., low- and high-achievers) benefited from this method more than any other group did. In doing so, a special emphasis was placed on the variations of students’ success with individual subtopics that make up a subproblem and their achievement with the overall solutions. This approach was taken because a student may arrive at an incorrect answer for as many reasons as the number of sub steps involved in the questions or even more. Doing an in-depth analysis should reveal more about the students’ learning deficiencies and challenges with individual subtopics. Additionally, examining the success of different student groups would enable us to determine more efficient methods to close the achievement gap between the students in STEM fields.

3.1. Examining the Effect of Problem Construction on Student’s Problem-Solving Performance

The first part of our analysis utilized Attempt Success Rate (ASR) and Complete Success Rate (CSR) scores obtained by using the data generated with the application of COSINE codes to each subproblem in students’ solutions. Investigating the trends in changes between the pre- to the post-test scores enabled us to determine how students improved their understanding of individual stoichiometry topics in detail and how their success changed with the integration, synthesis, and use of all these topics to generate a correct solution to the given problems.

3.1.1. Evaluating the Changes in the Complete Success Rate (CSR) Scores

To determine the overall effectiveness of the intervention on students’ problem-solving achievement with stoichiometry problems, the CSR scores were first determined for the students in both experimental and control groups considering their performance with all 64 subproblems involved in the pre- and post-tests. The total data points for control group and experimental group were 448 and 1984, respectively. These numbers should be kept in mind while reading the analysis. Even though the sample size was small and not suitable for running widely acceptable statistical tests, it is believed that the differences between the groups presented in the results and discussion section are meaningful and give some interesting facts about the nature of interventions and show where they were most helpful and what the limitations were associated with them.

Table 4. Descriptive data for the changes in the CSR scores after the interventions.

	Experimental (N = 31)				Control (N = 7)
	M	SD	Min	Max	M	SD	Min	Max
∆ CSR	0.04	0.12	−0.17	0.44	−0.07	0.08	−0.19	0.00

summarizes the changes in overall CSR scores, Δ CSR, for both groups. The Δ CSR value was calculated for each of the students for all five problems and averaged to give the Δ value of 0.04 for the experimental group and –0.07 for the control group.

The experimental group overall improved on the post-test, signified by the positive Δ CSR. On the other hand, the control group’s Δ value was determined to be negative indicating that the participants did worse on the post-test than the pre-test. When the changes in individual students’ CSR scores were examined, it was determined that, in experimental group, 32% of students did worse on the post-test, 55% improved their scores, and the rest did not show either advancement or decline in their scores. However, in the control group, 71% of students did poorly after the traditional practice while half of the remaining students showed a better performance and half of them protected their original score. Although the sample size is relatively small and the difference between delta values is not huge, these variances could be important evidence revealing the effectiveness of the intervention on students’ overall success with getting the correct solutions for stoichiometric problems.

3.1.2. Interpreting the Variations in the Attempt Success Rate (ASR) Scores

To further understand where the greatest improvements occurred to a student’s problem solving ability, a closer subproblem analysis was performed using ASR scores for each stoichiometric subtopic. Figure 3 shows the pre- and post- ASR values for the experimental group, categorized by each subtopic involved in the stoichiometry problems. It was determined that ASR values went up from pre- to post-test for 7 out of 9 subtopics. Thus, the difference between pre-test and post-test indicates that the intervention was effective in helping students perform better in the majority of the subproblems tested. Considering the documented students’ challenges with many of these subtopics including BEQ (Balancing Chemical Equations) [38], LR (Limiting Reagent) [39,40], MC (Mole Concept) [41], and SR (Stoichiometric Ratio) [42,43], these results could be considered as promising. Problem-construction has potential in helping students see how different components related to these topics are put together to create successful solutions.

Although the experimental group overall showed great progress with regard to their problem solving ability, this was not the case for the control group. While the experimental group was able to improve their performance with 7 subtopics, this number stayed at 3 for the control group. Figure 4 compares delta ASR values of the control and experimental groups. It should be noted that there are three types of trends in this figure for different groups of subtopics. The first trend indicates that the experimental group improved their performance with three subtopics, WEQ (Writing Equations), LR (Limiting Reactant), and SR (Stoichiometric Ratio), while the control group’s success with these subtopics worsened after the traditional study session.

It can be argued that the problem-construction method helps students more with these subtopics as each of them demands an analysis of multiple components, evaluating their interactions, and carefully connecting them to each other. Additionally, it was noticed that these topics require different types of knowledge and abilities. For example, WEQ is more about declarative knowledge and students’ ability of memorization [11]. So, it may not necessitate higher-order thinking skills as much as other subtopics. The subproblems about WEQ can be categorized under Remembering or Understanding level at maximum according to Bloom’s Taxonomy [44]. On the other hand, LR would require application level of thinking and proportional reasoning ability so that determined moles of substances are assessed and the LR is identified before moving to the next step in a stoichiometry problem [40,45]. In a similar way, the success with the SR would be related to students’ proportional reasoning ability to some extent. While some students are accustomed to using the dimensional analysis without thinking much about the conceptual background of the method, some use mole ratio deliberately to solve the problems [46,47].

The second trend involves three subtopics, PY (Percent Yield), BEQ (Balancing Equations), and MC (Mole Concept). The ASR scores went up for these three subtopics for both experimental and control groups. Out of these three topics, the control group improved their performance in MC and PY more than the experimental group did. Even though the differences for MC and BEQ are minimal, 0.01 and 0.04, respectively, the PY showed the greatest difference between two groups. It should be noted that, in this study, the PY subtopic is defined as being able to identify actual yield from the given question and plug it together with theoretical yield determined from another sub step into the PY formula. It could be argued that the problem construction method mainly aiming to show how different subproblems or subtopics are connected to each other in a solution is not very effective in helping students do algorithmic steps. It is also possible that this naïve approach caused retroactive interference [48]. Being exposed to a new way of thinking about stoichiometric problem solving could possibly resulted in relatively bigger changes in their knowledge system, which happens in conceptual change [49,50,51]. This gaining of these new memories may have in turn resulted in the inability of the experimental students to recall previously learned information, such as the PY subtopic (REF). On a deeper level, an explanation for this phenomenon could be due to negative priming. The experimental group was subjected to a more recent exposure of the PY subtopic in the intervention than the control group, for whom the most recent exposure for the PY subtopic was on the pre-test. The more recent exposure of the PY subtopic could have negatively influenced the future response to the same stimulus, resulting in a greater error rate for the experimental group compared to the control group. On the other hand, for the experimental group, the second greatest change was observed in the ASR score of the MF (Molecular Formula) with a delta value of 0.13 while the control group’s pre- and post-test scores stayed the same. In our coding, the MF subtopic was related to being able to recognize the relationship between the EF (Empirical Formula) and the MF and carry out the calculations to determine the new chemical formula with the correct molar mass. Although it can be argued that this operation also requires significant algorithmic problem-solving skills, it was somehow more challenging for the control group. The difference between the delta values of PY and MF needs to be investigated further to find out other possible variables affecting students’ scores here.

Finally, the last category highlights subtopics that both the control and experimental groups showed poorer performance in the post-test. The EF (Empirical Formula) and CM (Conservation of Mass) are two subtopics that the experimental group did worse after the intervention. The same trend was observed for the control group. It is speculated that students struggled with Conservation of Mass because it is not a conceptual topic, but instead requires algorithms and the use of calculations to solve for the correct answer. So, either traditional practice or problem-construction is not effective in improving students’ algorithmic skills. A result of this weak math foundational understanding is that students are not able to apply advanced problem-solving methods and instead rely on novice methods [52]. Additionally, it should be noted that this topic is mainly associated with figuring out the mass differences among the steps in a solution. So, another reason for this decrease may be due to the lack of attention to details. The steps that involve subtopic CM could be perceived to be trivial steps compared to other major steps in getting the correct answer for the given problems. While students are focusing on getting other relatively more complex calculations done, their working memory could be overloaded, [53,54] resulting in failure at acknowledging the need for doing simple math involving the subtopic CM. Otherwise, it is difficult to claim that students are not able to do basic addition or subtraction. Furthermore, students typically do not have a strong conceptual understanding of conservation of mass because they are accustomed to simply following an algorithm or equation to solve a stoichiometric problem without understanding the conceptual reasoning behind it [38]. In this study, the subtopic EF is defined as dividing the mole numbers of the elements by the smallest value and determining the simplest ratio among them. So, finding an empirical formula successfully strongly correlates with one’s proportional reasoning skills [46]. Starting from elementary schools, educators aim to promote the development of these skills as they affect their performance not only in math but also science [55]. Despite all these efforts, this study shows that students still have not developed their proportional reasoning skills to tackle the problems presented in chemistry classes. Heavy dependence on factor-label methods or dimensional analysis in traditional education could also contribute to the lack of desired improvements in students’ skills [46,55].

3.1.3. Bringing Hidden Differences to Daylight: Inspecting the Changes in the Codes

Following the analysis of students’ performance with the subtopics by utilizing the changes in their ASR scores, a deeper analysis was executed with the help of the codes assigned to each step in the experimental group’s solutions. Figure 5 highlights the specific changes seen in three categories of the COSINE codes from the pre- to the post-test for all the subtopics investigated in this study.

With this analysis, it was intended to find out how exactly problem-construction practice helps students and whether unsuccessful or neutral codes decrease more if the students improve their scores after the intervention. As described in the methodology section, there are three categories of codes, successful (S), unsuccessful (CD, UDI, UG, and URH), and neutral codes (DSE and DD). A close examination of the changes in unsuccessful codes indicates the presence of three categories. In the first category, there are 6 stoichiometric topics, which saw a decrease in the number of unsuccessful codes. Within the second group, there are only two stoichiometry topics, Stoichiometric Ratio (SR) and Conservation of Mass (CM), with the same number of unsuccessful codes before and after the study. So, the overall increase in ASR score of SR is explained with the transformation of neutral codes into successful codes. On the other side, the overall negative delta CSR value of CM was the result of the conversion of successful codes into neutral codes. It seems that the participants either forgot to carry out this step and received DD code or did inappropriate calculations and received DSE code, which support our previous hypothesis related to working memory capacity being an important factor in determination of students’ performance with this subtopic. It is obvious that neutral codes play an important role in determining students’ overall success with problem solving, and they should be considered in the studies examining students’ challenges with problem solving and in everyday grading practices. These findings should encourage educators to modify current assessment methods and go beyond the use of simple points that do not clearly reveal the nature of mistakes made in solving problems. Finally, the third category includes only one subtopic, EF, that observed an increase in unsuccessful codes after the intervention. In addition to this increase, it was also noted that the number of neutral codes went up. This finding points out the urgent need to address the weakness in students’ proportional reasoning abilities and employ effective techniques to improve it [46]. This observation agrees with the trends depicted in Figure 3 and Figure 4 where the overall ASR score decreased after the intervention.

3.2. Examining the Influence of the Intervention on High and Low Achieving Students

3.2.1. An In-Depth Analysis of Changes in CSR and ASR Scores

In the previous section, a thorough analysis of changes in CSR and ASR scores and variations in the code distribution was performed to determine the impact of problem-construction practice on all experimental group’s success with individual stoichiometry topics and overall problems. Here, the goal was to find out if the problem-construction intervention helped any specific student group more than others. First, students were grouped based on success levels in their general chemistry course.

Table 5. Descriptive data of students grouped by success level in General Chemistry I.

	A Students (N = 12)				B Students (N = 9)				C Students (N = 10)
	M	SD	Min	Max	M	SD	Min	Max	M	SD	Min	Max
∆ CSR	0.01	0.06	−0.14	0.10	0.01	0.15	−0.17	0.26	0.09	0.14	−0.09	0.44

displays the descriptive data for each student group. Although these delta values are small, for comparison purposes, they still provide a good method for judging the effectiveness of the intervention for students who performed differently in the prior chemistry course.

A and B students have an average mean of 0.01 for the delta CSR indicating overall small improvement in their problem-solving performance after the intervention. Interestingly, the greatest delta value, 0.09, was observed for the C students, which could be interpreted as the C students received the greatest benefit of participating in the problem-construction intervention compared to the students of stronger academic backgrounds. A possible explanation is that the lower performing students are the ones who may have been struggling with chemistry topics more as evidenced by their low grades, and the intervention helped them develop their problem-solving performance the most and possibly showed them what was lacking in their approaches to the problems [56]. Likewise, in the study conducted by Hardy et al. [32] using peer generated multiple-choice questions to improve understanding, lower intermediate achieving students improved more than the higher achieving students by engaging in the PeerWise activities.

3.2.2. Looking into the Distribution of Unsuccessful, Neutral, and Successful Codes

To better understand how exactly the intervention helped each group of students, particularly C students, Figure 6 was prepared with all the code breakdowns for each subtopic. Like the previous code analysis, the primary focus was placed on the changes in the unsuccessful and neutral codes. The changes in the successful code were not directly investigated but interpreted indirectly in relation to the changes in unsuccessful and neutral codes. The ideal scenario would compromise situations where both neutral and unsuccessful codes decrease or disappear after the intervention, resulting in a higher number of successful codes and, in turn, greater CSR and ASR scores. Each grade level’s performance was investigated in depth and compared to that of other grade levels. In the analysis of code breakdown of the students with a grade A, the first thing was noticed that out 9 topics, in only one, PY, a great transformation was observed, all neutral codes (8%) transformed to successful ones. Another interesting finding regarding the changes in A students’ codes is that 5 topics saw a decrease in the number of neutral codes while only two topics had a lower number of unsuccessful codes in the post-test. It seems that the intervention is more effective in addressing issues captured by DD and DSE codes than those marked by UDI, CD, UG, or URH codes.

Additionally, it should be noted there were a few occasions like in subtopics EF and LR, the number of neutral codes were transformed into unsuccessful ones after the intervention. It is hard to argue whether this is a better or a worse thing, but this point deserves further investigation. Finally, it is worth mentioning that the insignificant progress depicted with a low mean delta value, 0.01, is not surprising after seeing these students’ high percentage of successful codes even before the intervention. It was obvious that there was little room, or no room like in the subtopic MF for the improvement.

Unlike A students, B students did not have a subtopic that had 100% successful codes before or after the intervention. However, this group of students showed much better improvement in their problem solving with the subtopic PY. The percentage of unsuccessful codes went from 33% to 0% and the number of neutral codes decreased by 50%, but it was not enough to give them perfect ASR score. The same decrease in the number of unsuccessful codes was observed for the subtopic MF, which was the second greatest decrease in the number of unsuccessful codes for the entire student population after the C students’ success with the subtopic LR, in which unsuccessful codes went from 70% to 30%. When comparing the changes in this group to those in the group of A students, a similar trend was observed. Out of 9 topics, three topics saw a decrease in the number of neutral codes versus four topics that observed a decrease in the unsuccessful codes. Again, the results indicate that the intervention is better at addressing two issues, forgetting to do the necessary steps and doing irrelevant calculations. Further investigation of the codes for B students indicated that both the number of unsuccessful and neutral codes of the subtopic SR went up after the intervention, which was not the case for any subtopic in the group of A students. On the other hand, the number of both codes decreased at the same time for one subtopic, MC, which was not the case for A students, either. Overall, B students had relatively less successful codes in total compared to those of A students, but higher than those documented for C Students. This finding supports the idea that there is a correlation between the students’ overall standing in the course and performance with problems presented in the study. It could be also used to endorse the use of COSINE codes in similar studies aiming to understand students’ challenges with problem solving.

The final portion of analysis involved the students who had relatively low achievement in the previous general chemistry course. It is correct that the number of successful codes overall for C students was fewer than those received by A and B students, but a closer analysis shows that they were the ones who benefited from the intervention most. The number of unsuccessful codes dropped for seven subtopics out of nine. For two subtopics, BEQ and MC, both the number of unsuccessful and neutral codes went down, which did not happen before in any other grade group. It should be noted that the number of neutral codes decreased only for three topics for C students. This number is the lowest one observed among the three groups. It can be argued that the intervention helps students with relatively low achievement in the course fix their shortcomings related to unsuccessful codes but assist those who do better in the course with their challenges documented by the neutral codes. This is another aspect of the study that needs further examination. It is interesting to see for the first time that the unsuccessful codes assigned to a subtopic, CM, completely become neutral after the intervention. It was observed that 10% of unsuccessful codes were transformed into neutral codes. Opposite transformations were documented earlier, but as said before, there is no clear evidence for either transformation being more favorable or showing a better progress in reaching 100% successful codes. This topic will be under the radar of the researchers as well.

4. Limitation

The first and the most critical limitation was the number of students in the control group. The team had planned carefully to have reasonably high and equal numbers of students in both control and experimental groups, so certain statistical tests can be run with sufficient power. However, the study did not go as planned, and several students had to quit. In future iterations, the team is planning to invite significantly more students to start the project. Another limitation can be mentioned about the number of interventions. If the real impact of the interventions needs to be assessed, students should be exposed to those methods for an extended period rather than one time. Possibly, a better trend would be observed if longer interventions, more than 75 min, are implemented involving more than one chapter. Although there is not any evidence for its impact, it is suggested that the post-test given right after the interventions are implemented to better determine their impact on students’ problem-solving abilities. It is almost impossible to control what students do between the intervention and the post-test and rule out the effects of different practices students do on their own. Among other limitations, it should be noted that Krippendorf’s alpha score is slightly lowered than the recommended value, 0.80, to minimize the negative influence of inter-rater coding process on the data analysis. The result could be higher if we had only two coders instead of three. Finally, it is recommended that visual data is collected during the think-aloud protocols so it will be easier to determine what the participants refer to when they use words like “this” or “that” in their explanations or thinking process.

5. Conclusions

Conducting studies to understand the nature of problem-solving processes and find where students struggle most has been an important goal in chemistry education since the field was established. Many researchers have worked in this area and helped the community advance their comprehension of challenges and their roots associated with problem solving involving a wide range of science topics. Even though many issues were well studied and described in previous studies, there is a shortage in the number of studies aiming to mitigate those common problems. In this study, the effects of a promising and relatively naïve method, problem construction, were investigated and compared to those of a traditional practice method on improving college students’ problem-solving skills. The differences in the CSR values pointed out that the problem-construction intervention was more effective in improving students’ success with solving overall stoichiometry problems. Higher CSR scores obtained in the post-test indicate that not only did students recognize the proper steps to solving the problem, but also their relation to other sub steps. Additionally, increased ASR values revealed that the same intervention is efficient in helping students do better with subproblems involving different stoichiometric subtopics such as Percent Yield and Limiting Reagent as well as decreasing the number of unsuccessful and neutral codes for many subtopics. This change would reduce the gap between the ASR and CSR scores, indicating that students become proficient at doing individual subproblems and overall problems at the same time. This finding is encouraging as the previous studies [5,57] found out that students are, in general, successful with doing subproblems, but fail to figure out the correct schema that maps out every subproblem needed for getting the right answer to the given problems. A zero difference between ASR and CSR scores could also be interpreted as the students are getting a more organized knowledge system, which should make the knowledge retrieval process more efficient and increase the chance of getting the correct answers [10,11,12]. These positive changes are better understood when the nature of reverse problem solving is considered. During that process, students challenge themselves by taking a step further in the problem by using the given information and manipulating it in such a way that they can formulate a specific question with all the required stoichiometric subtopics, which are linked to each other considering the conceptual framework of the question. Using the method of reverse problem solving, instructors can help students connect concepts more successfully and see the underlying principles and schemas that certain types of problems carry. Sufficient problem-construction practice should also teach students how to break down the information given in the question, relate them to appropriate topics learned, and then complete the puzzle by putting all the pieces back together. Examining the differences in student groups based on overall performance in the previous chemistry course also discloses that problem-construction method helps all, but low-performing students benefit most. If educators want to close the achievement gap in stoichiometric problem solving and in other areas, they should design homework assignments utilizing problem-construction method and especially use it as a form of intervention to those who show relatively poorer performance in the class. Problem construction method can also be turned into a group activity as well. Students in the group can interpret the pieces of the given solutions and step by step create the questions collectively that will result in the same solution.

As highlighted in the limitation and participants sections, the number of students in the experimental group changed abruptly after the study began. It is planned to use the problem construction method with a larger and a more diverse group of students. In this study, with limited power, the effects of this method on students of different achievement groups were investigated. In the new studies, more detailed information on students’ demographics (e.g., socioeconomic and ethnic backgrounds) will be collected to bring depth to the analysis and find out if the method benefits a specific group more than others. Additionally, future studies will focus not only on stoichiometry but also on other general chemistry and advanced topics such as acid-base chemistry [58,59], chemical kinetics [60,61], NMR spectroscopy [28], which are known to be challenging to many students.