Skip to Content
  • Proceeding Paper
  • Open Access

26 October 2026

8 Pages

Improving Logical Reasoning of Small-Scale Language Models †

,
and
1
Department of Information and Computer Science, Faculty of Engineering, Kyoto Tachibana University, Kyoto 607-8175, Japan
2
Department of Knowledge Science, Division of Information and Quantum Sciences, SANKEN, University of Osaka, Osaka 567-0047, Japan
*
Author to whom correspondence should be addressed.
†
Presented at the 6th International Conference on Electronic Communications, Internet of Things and Big Data (ICEIB 2026), New Taipei, Taiwan, 24–26 April 2026.

Abstract

To improve the inference performance of small-scale language models, we proposed a stepwise phased process design that divides problem-solving into multiple phases and evaluated it using the Grade School Math 8K arithmetic word problem dataset. We adopted a structure in which the system first generates a brief solution strategy to compute and then performs the calculations to arrive at the final answer. The evaluation results showed that while the final accuracy rate increased slightly when the phased approach was used compared to when it was not, no significant difference was observed. However, upon examining the intermediate outputs of the phased process, we found many instances where the solution strategy itself was correct even for problems that resulted in incorrect answers. Furthermore, the number of incorrect answers decreased with the phase division, and the number of unanswered questions increased. This suggests that having the phased procedure made it harder for the model to produce incorrect answers.

1. Introduction

Large language models (LLMs) show dramatic performance improvements across a wide range of tasks. However, this progress comes with practical challenges, such as the massive consumption of computational resources and rising operational costs. Furthermore, since many LLMs process data on the provider’s servers, there are concerns regarding the risk of user input being incorporated into the model’s training and security risks associated with communication with third-party servers. The adoption of LLMs is advancing in companies and local governments. There are many situations where organizations wish to avoid these risks.
Against this backdrop, small-scale language models (SLMs) capable of running on local PCs are gaining attention. If operations are confined to the local environment, confidential information can be handled without relying on external servers, significantly reducing security risks. However, SLMs face the challenge of limited model scale in terms of the number of parameters and computational resources, resulting in poor inference capabilities. Consequently, their performance tends to be insufficient for logical tasks, making them impractical for real-world use as they stand. Overcoming this weakness is a critical research challenge for realizing secure and lightweight AI applications.
This study aims to improve the insufficient reasoning capabilities that are a weakness of SLMs. Specifically, we introduce a stepwise phased process design that divides the thought process leading to an answer into multiple phases, thereby reducing the amount of reasoning required in a single generation. By doing so, we aim to address the reasoning limitations of SLMs and improve answer accuracy. The effectiveness of this approach is evaluated using the Grade School Math 8K arithmetic word problem dataset (GSM8K) to demonstrate the extent of performance improvement.

3. Method

3.1. Mechanism

The method proposed in this study introduces a mechanism that processes the target logical problem in small, sequential phases rather than addressing it all at once. The primary objective is to make complex problems solvable by SLMs by reducing the volume of information processed during a single inference step. The execution workflow follows a structured sequence: receiving the task, formulating a solution procedure, and executing calculations according to that procedure. Through this approach, we evaluate whether decomposing the task improves reasoning accuracy within small-scale architectures.
Specifically, the task of solving arithmetic word problems is divided into two distinct phases: Phase 1 (formulate a calculation plan) and Phase 2 (execute calculations according to the plan), designed to derive the final answer in sequential order. By establishing a clear solving strategy in Phase 1, the model in Phase 2 can focus exclusively on numerical execution. The prompts for each phase are structured systematically. The prompt configuration for Phase 1 is illustrated in Figure 1. The highlighted box defines the operational constraints and objectives, incorporating a step-by-step prompting technique [7] that is widely recognized for enhancing language model performance. Some explicit instructions are provided in the blue box.
Figure 1. Phase 1 prompt (CoT): Some explicit instructions are provided in the blue box, followed by examples in the yellow box. The problem to be solved ({problem}) is then presented in the red box.
Additionally, we employ few-shot prompting, providing several example input-output pairs to facilitate the model’s task execution, as highlighted in the yellow box in Figure 1. These examples demonstrate the expected format for responding to a user query. After presenting five such examples, the target {problem} is introduced in the red box, directing the language model to generate a response using the prior examples as structural references. Subsequently, the prompt design for Phase 2 is shown in Figure 2. Similar to Phase 1, explicit instructions are provided in the blue box, followed by five examples in the yellow box, which are omitted from the figure for brevity. The problem to be solved ({problem}) is then presented in the red box. Crucially, the generated output from Phase 1 is injected into the {hint} section to serve as contextual guidance for this phase.
Figure 2. Phase 2 prompt (CoT): Some explicit instructions are provided in the blue box, followed by examples in the yellow box. The problem to be solved ({problem}) is then presented in the red box. The generated output from Phase 1 is injected into the {hint} section to serve as contextual guidance for this phase.
A key characteristic of this research is its focus on SLMs that operate efficiently in local environments, eliminating reliance on large-scale server infrastructure. While these models offer significant advantages regarding computational efficiency and deployment costs, they inherently struggle with complex logical reasoning. To mitigate these limitations, this study integrates specific operational techniques, including decomposing text into shorter structural segments, enforcing strict output formatting constraints, and requiring explicit justification. These combined mechanisms enable small-scale models to execute systematic, step-by-step reasoning, ultimately achieving a level of accuracy suitable for practical deployment. The overarching goal of this study is to propose an optimized framework that makes such structured reasoning approaches viable even within resource-constrained architectures.
Dividing a task into several phases and considering them sequentially is similar to the CoT approach. However, CoT illustrates the thought process in a single generation. In contrast, this method divides the generation process into multiple phases, passing the results of one generation to the next to reduce the amount of reasoning (the amount of thinking) required in a single generation.

3.2. Testing Procedure

To evaluate logical reasoning performance, this study employed the GSM8K benchmark, a dataset of grade-school arithmetic word problems requiring both linguistic comprehension and numerical calculation. Model performance was assessed through accuracy, which measured the ability to produce correct numerical answers, and reasoning validity, which examined whether intermediate steps were logically consistent with the source text and conclusions. Because inspecting the entire dataset was impractical, outcomes were grouped into correct, incorrect, and unanswered, with 20 cases randomly sampled from each for manual inspection. This process-oriented evaluation enabled granular error analysis, identifying where reasoning succeeded or failed beyond binary correctness.
The experimental setup focused on enhancing the reasoning capabilities of SLMs in a local environment, defined as computation on a consumer-grade laptop. All experiments were conducted on a MacBook Pro (Apple Inc., Cupertino, CA, USA) with an M3 chip and 16 gigabytes of unified memory. The model tested was the creator/organization large language model Meta AI-3.1 8B, with 8 billion parameters and instruction-tuned, deployed in GPT-generated unified format with Q6_K quantization. No fine-tuning or supplementary training was performed; optimization relied entirely on prompt engineering and phased inference. For reproducibility, decoding parameters were fixed at a temperature of 0.8, a top_p of 0.9, and a maximum token limit of 256.

4. Results and Discussion

We compared performance between executions with and without the phased approach using the first 150 questions from the GSM8K arithmetic word problem dataset. For the non-phased prompt, the same Phase 2 prompt shown in Figure 2 was used, but the hint section referring to Phase 1 output was removed, leaving the rest unchanged. Differences emerged in accuracy and response tendencies.
With phase division, the correct answer rate was 20.7%, slightly higher than the 17.3% without phasing. The error rate dropped to 35.3% with phasing compared to 50.0% without, but the non-response rate increased to 44.0 from 32.7%. Thus, the phased approach improved accuracy and reduced incorrect answers, though it led to more unanswered responses. This may be because Phase 1 outputs constrained arbitrary responses, suppressing hallucinations but sometimes causing breakdowns in output generation.
Failure case analysis results revealed common patterns across both methods. Problems requiring multiple calculation steps often produced errors, with breakdowns occurring during intermediate reasoning and misidentification of the final quantity. In one example (Figure 3), Phase 1 established a valid strategy and Phase 2 proceeded smoothly until the third line, but subsequent calculations involving new_house_value became incoherent, leading to an incorrect answer. Misinterpretation of problem statements also contributed to errors in both methods, particularly when conditions such as “X times a week,” “weekdays vs. weekends,” or “X% of” were misread, resulting in flawed calculations.
Figure 3. Example of output with phase division for a problem requiring multi-step calculations.
In the problem shown in Figure 4, the condition “three times a week, three sprints per session” must be interpreted correctly, but SLM misread it as three sprints spread over seven days, totaling 21, and produced the wrong answer. Errors were also common in unit conversion tasks, such as length, time, or quantity. For example, in Figure 5, the model needed to apply the conversion “one dozen = 12.” Although the number 12 appeared in the calculation, suggesting awareness of the conversion, the subsequent computation was incoherent, revealing failure.
Figure 4. Example output with phase division for a problem with difficult text interpretation.
Figure 5. Example of output without phased breakdown for problems involving unit conversion.
Overall, both methods struggled with multi-step calculations. In the phased approach, visual inspection of Phase 1 outputs showed that about two-thirds of incorrect or unanswered problems contained a generally reasonable strategy, while one-third displayed misguided procedures or corrupted outputs. A key weakness is that errors in earlier phases are difficult to detect and correct later, raising the challenge of preventing flawed strategies from leading to wrong answers.
Comparing incorrect and unanswered outputs, the phased approach reduced incorrect answers but increased unanswered ones. Many “no answer” cases were not intentional refusals but rather output failures, such as incomplete text, corruption, or missing final formats. While this suggests hallucinations may be suppressed, improvements could come from regeneration or external verification rather than simply accepting unanswered outputs.
Despite enhanced accuracy, the phased approach offers notable benefits. It makes the reasoning process transparent, allowing failures to be isolated—whether due to flawed strategies, calculation mistakes, or corrupted outputs. This helps identify bottlenecks, such as weaknesses in computational ability, and clarifies directions for improvement. Moreover, even small-scale models often produced correct Phase 1 strategies, showing that they can grasp problem structure and required steps even when final answers are wrong.
The experiment results highlight limitations and potential for improvement in SLMs. Limitations include weak computational ability, frequent arithmetic errors, difficulty with long sentences and multi-step reasoning, and misinterpretation of complex expressions like “X per…” or “X% of….” Yet the ability to decompose problems and outline solution steps suggests untapped potential. If instability in generation—such as arithmetic mistakes or formatting issues—can be reduced, accuracy rates should improve.
Current SLMs struggle with misreading, arithmetic errors, and unstable outputs, but their ability to devise procedures exceeds their accuracy rates. This indicates room for performance gains through stronger output control and verification mechanisms.

Limitations and Potential for Improvement in SLMs

The experiment results highlight both the limitations and the potential for improvement of SLMs in solving arithmetic word problems. A key limitation lies in their weak computational ability: the models occasionally produced errors even in basic operations such as multiplication and division, for example, miscalculating 48 ÷ 6 as 7. They also struggled with long sentences and multi-step reasoning, often failing to construct solution procedures that accounted for all conditions in the problem statement. Complex expressions such as “X per …” or “X% of …” were frequently misinterpreted, and the models showed limited ability to structure information, leading to misunderstandings caused by misreading or skipping essential details.
At the same time, the potential for improvement is evident. In problems approached with phased breakdowns, Phase 1 often produced correct procedural steps even when the final answer was wrong or missing. This suggests that the models can decompose problem structures and identify necessary steps more effectively than their accuracy rates imply. In other words, the models often understand the logic of the calculation process, but instability in generation—such as arithmetic mistakes or formatting errors—undermines execution. Addressing these instabilities could significantly improve accuracy.
Current SLMs face challenges with misinterpretation, arithmetic errors, and unstable outputs, but they show a stronger ability to devise solution procedures than their final accuracy rates suggest. This indicates that performance can be enhanced by introducing output control and verification mechanisms, which would help stabilize execution and reduce errors.

5. Conclusions and Future Directions

In this study, we examined the effectiveness of segmenting the generation process of an SLM when applied to 150 GSM8K arithmetic word problems. Accuracy improved modestly, rising from 17.3% without segmentation to 20.7% with segmentation, but the phased approach also reduced the frequency of incorrect answers caused by forced outputs. Instead of producing arbitrary wrong values, the model often refrained from completing responses when contradictions or difficulties arose. While this did not result in explicit outputs such as “I can’t solve this,” corrupted text or repeated sequences were more common, making hallucinations easier to detect and correct during post-processing. This can be considered a secondary benefit of the phased approach.
Another important observation was that Phase 1, which focused on devising a solution procedure, frequently produced accurate strategies even when the final answer was incorrect. This suggests that SLMs possess latent logical reasoning capabilities, and improvements in basic computational skills could lead to higher accuracy. However, when Phase 1 generated flawed or incomplete strategies, Phase 2 was unable to recover, highlighting the need for mechanisms to verify and correct Phase 1 outputs before proceeding.
These results provide valuable insights into how SLMs can be adapted for complex tasks. Although the performance gains were limited, the phased approach demonstrates the potential of prompt engineering to unlock additional capabilities. The study also revealed challenges, including weak computational power, difficulties in reading comprehension, and an increase in non-responses. Addressing these issues will be crucial for practical implementation. Ensuring that errors in one phase do not propagate to subsequent phases, exploring further subdivisions of the process, and adopting hybrid approaches—such as delegating computation to external tools [9]—are promising directions.
The phased process design improves accuracy and makes the model’s reasoning transparent and structured, enabling human oversight and intervention. By combining verification, tool integration, and model coordination, future work can enhance the performance of SLMs and extend these benefits to larger language models tackling high-difficulty tasks.

Author Contributions

Conceptualization, Y.K. and M.N.; methodology, Y.K.; software, Y.K.; validation, Y.K., K.K. and M.N.; formal analysis, Y.K.; investigation, Y.K.; resources, Y.K. and M.N.; data curation, Y.K.; writing—original draft preparation, Y.K. and M.N.; writing—review and editing, M.N. and K.K.; visualization, Y.K.; supervision, M.N. and K.K.; project administration, M.N.; funding acquisition, M.N. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not application.

Data Availability Statement

No new data were created or analyzed in this study. Data sharing is not applicable to this article.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.V.; Zhou, D. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv 2022, arXiv:2201.11903. [Google Scholar]
  2. Lightman, H.; Kosaraju, V.; Burda, Y.; Edwards, H.; Baker, B.; Lee, T.; Leike, J.; Schulman, J.; Sutskever, I.; Cobbe, K. Let’s Verify Step by Step. arXiv 2023, arXiv:2305.20050. [Google Scholar]
  3. Liu, N.F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; Liang, P. Lost in the Middle: How Language Models Use Long Contexts. Trans. Assoc. Comput. Linguist. 2024, 12, 157–173. [Google Scholar] [CrossRef] [Scilit]
  4. Saparov, A.; He, H. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. arXiv 2023, arXiv:2210.01240. Available online: https://arxiv.org/abs/2210.01240 (accessed on 21 October 2026).
  5. Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. Training Verifiers to Solve Math Word Problems. arXiv 2021, arXiv:2110.14168. [Google Scholar]
  6. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; Narasimhan, K. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv 2023, arXiv:2305.10601. [Google Scholar]
  7. Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-K.; Nakhost, H.; Fujii, Y.; Ratner, A.; Krishna, R.; Lee, C.-Y.; Pfister, T. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In Findings of the Association for Computational Linguistics: ACL 2023; Association for Computational Linguistics: Stroudsburg, PA, USA, 2023; pp. 8003–8017. Available online: https://aclanthology.org/2023.findings-acl.507/ (accessed on 21 October 2026).
  8. Yu, W.; Jiang, Z.; Dong, Y.; Feng, J. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. arXiv 2020, arXiv:2002.04326. Available online: https://arxiv.org/abs/2002.04326 (accessed on 21 October 2026).
  9. Shimura, M.; Sakurai, S. Learning Arithmetic Problem Solver. Proc. AAAI 1986, 1036–1040. Available online: https://cdn.aaai.org/AAAI/1986/AAAI86-170.pdf (accessed on 21 October 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.