# Validation of Rubric Evaluation for Programming Education

^{*}

## Abstract

**:**

## 1. Introduction

- RQ1: How do we identify statistical methods to assess the properties of rubrics?
- RQ2: How can rubrics be evaluated and analyzed to identify improvements?
- RQ3: By evaluating and analyzing the rubrics, can we confirm the generality of the evaluation?

## 2. Backgrounds

#### 2.1. Definition and Required Properties of Rubrics

#### 2.2. Evaluating the Reliability of Rubrics

#### 2.3. About the Rubric to Be Evaluated

## 3. Characteristic Evaluation Framework for Rubrics

#### 3.1. Rubric Characterization Methods

#### 3.2. Factor Response Rate

#### 3.3. Integration Within

_{i}: variance of each question item, σ

_{x}: variance of the total of each question item.

#### 3.4. Inter-Item Correlation

## 4. Experiment and Evaluation

#### 4.1. Rubrics and Data to Be Covered

#### 4.2. Class Details

#### 4.3. Application Results of Evaluation Framework

#### 4.3.1. Factor Loadings in Evaluation Results

#### 4.3.2. Internal Consistency in Each Evaluation Result

## 5. Discussion

#### 5.1. Answers to RQs

#### 5.1.1. RQ1: How Do We Identify Statistical Methods to Evaluate Rubric Properties?

#### 5.1.2. RQ2: How Can Rubrics Be Evaluated and Analyzed to Identify Improvements?

#### 5.1.3. RQ3: Can We Confirm the Generality of Evaluation by Evaluating and Analyzing Rubrics?

#### 5.2. General Comments on the Application of the Evaluation Framework

#### 5.3. Threats of Validity

## 6. Related Works

## 7. Conclusions and Future Work

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Appendix A

Category | Item | Stage 5 | Stage 4 | Stage 3 | Stage 2 | Stage 1 |
---|---|---|---|---|---|---|

Understanding of the algorithm (the concept of the algorithm) | Sequence | It can be tied to understand the sequence of ideas with other ideas | Multiple things the in’s order, can be considered in conjunction | It can be multiple things the have’s order, respectively | It is possible that it is the order for one thing | Do not understand the concept of sequence |

Branch | Able to connect the idea of branching to other ideas. | Consider the plurality of conditions for a matter can be branched in combination | It can be branched in a plurality of conditions for a matter | It can be branched at one condition for a certain matter | Do not understand the concept of branch | |

Repetition | The concept of repetition can be combined with other ideas. | It can be used to combine multiple iterations (double loop) | You can notice from one procedure to the plurality of iterations | You can notice a certain procedure or one iteration | Do not understand the concept of repetition | |

Variable | It is possible to link the concept of variables and other ideas | It is possible to make multiple variables associated with a certain matter | It is possible to create multiple variables for a certain matter | It is possible to make one of the variables for a certain matter | Do not understand the concept of variable | |

Array | It is possible to link the idea of the sequence with other ideas | It can be in unity associated with some of the elements as | Can some elements into a plurality of separate unity | It can be some of the elements in one of unity | Do not understand the concept of array | |

Function | It is possible to link the concept of a function and other ideas | Summarized the several steps in the form associated with each of the plurality of elements | It is summarized a few steps into a plurality of elements | It is summarized a few steps to one of the elements | Do not understand the concept of function | |

Recursion | It is possible to link the concept of recursion to the other way of thinking | Notice cannot be a comeback can function, use recursion for recursion can function | For some functions, it can be issued to call the function itself in the function | For some functions, Kizukeru call the function itself in the function | Do not understand the concept of recursion | |

Sort | It is possible to link the idea of the sort to other ideas | It can be sorted in an optimal way associated with some of the elements to the element | It is possible to sort each with some of the elements of a plurality of different ways | It can be sorted in one way some elements | Do not understand the concept of sorting |

## Appendix B

Category | Item | Stage 5 | Stage 4 | Stage 3 | Stage 2 | Stage 1 |
---|---|---|---|---|---|---|

Thinking in the design and creation of the program | Subdivision of the problem | Can subdivide the problem, it is possible to make the solutions and other things subdivided problem | It can be divided into several smaller problems associated with major problems | It can be separated from a big problem into smaller problems | It is possible to find one small problem from a big problem | Cannot subdivide the problem |

Analysis of events | It is the analysis of the events, the results of the analysis can be used in problem-solving and other things | It can be found several factors (causes) related to an event | You can find multiple factors (causes) for a certain event | You can find one factor (Cause) about a certain event | Cannot analyze events | |

Extraction operation | Operation can be extracted, the extracted operation can be utilized to solve problems and other things | It can be withdrawn a plurality of operations associated with existing matter | It is possible to extract a plurality of operations from the existing things | It is possible to extract one of the operations from the existing things | Cannot extract operation | |

Construction of the operation | Following the purpose to build an operation, it can be used in problem-solving and other things | You can build more of the operations related to suit your purpose | You can build a plurality of operations following the purpose | You can build one operation by the purpose | Cannot build operation | |

functionalization | Big thing can function reduction, can be utilized for their problem-solving and other things that | Can be summarized into several smaller steps associated with large things that | For large things that can be summarized into several smaller steps | Can be summarized for the big things that, one in small steps | Cannot function of | |

Generalization | Be generalized to various things, it can be used for problem-solving and other things | It is possible to combine the common parts of the various things in the big concept | It can represent multiple things to a large concept | It can express one thing as a great concept | Cannot be generalized |

## Appendix C

Category | Item | Stage 5 | Stage 4 | Stage 3 | Stage 2 | Stage 1 |
---|---|---|---|---|---|---|

Thinking in the design and creation of the program | Abstraction | Big thing can abstract for, can be used for their problem-solving and other things that | Be focused on a plurality of elements associated with large things that | For large things that can focus on several important factors | For large things that can focus on one important element | Cannot be abstracted |

inference | Hypothesized cause to the problem, it can be utilized to resolve methods (deduction and induction) were derived problem-solving and other things on the basis thereof | Hypothesized cause to the problem, derivable how to solve based thereon (deductive, inductive) | You can both the following (independent form) 1. is a hypothesis of the cause for the problem. 2. derivable how to solve the problem | You are either the following 1. is a hypothesis of the cause for the problem. 2. derivable how to solve the problem | Cannot be inferred | |

Logical algebra | It can be used in conjunction with logical algebra and other ideas | It can be used in combination logical sum, logical product, a logical negation | Logical sum, logical product, and understand more about the logical negation | Is the logical sum, logical product, one of the logical NOT understand | Do not understand the logic of algebra | |

operator | It is possible to use the operator in conjunction with other ideas | Can be used in conjunction to understand several types of operators | We are familiar with several types of operators | We are familiar with one type of operator (assignment operators, arithmetic operators, comparison operators, Boolean operators, bitwise) | Do not understand the operator | |

Understanding of the program (reading, editing, and evaluation) | Understanding of the program | Comprehension of procedures and operations, evaluation, to understand the editing, can be utilized in problem-solving and other things | Comprehension of procedures and operations, evaluation, can be associated with the editing | Possible reading of procedures and operations, evaluation, editing each | It is possible to read the procedures and operations | Do not understand the program |

## References

- Stegeman, M.; Barendsen, E.; Smetsers, S. Designing a rubric for feedback on code quality in programming courses. In Proceedings of the Koli Calling 2016: 16th Koli Calling International Conference on Computing Education Research, Koli, Finland, 24–27 November 2016; pp. 160–164. [Google Scholar]
- Alves, N.D.C.; von Wangenheim, C.G.; Hauck, J.C.R.; Borgatto, A.F. A large-scale evaluation of a rubric for the automatic assessment of algorithms and programming concepts. In Proceedings of the SIGCSE ‘20: The 51st ACM Technical Symposium on Computer Science Education, Portland, OR, USA, 12–14 March 2020; pp. 556–562. [Google Scholar]
- Chen, B.; Azad, S.; Haldar, R.; West, M.; Zilles, C. A validated scoring rubric for explain-in-plain-english questions. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education, Portland, OR, USA, 12–14 March 2020; pp. 563–569. [Google Scholar]
- Saito, D.; Kaieda, S.; Washizaki, H.; Fukazawa, Y. Rubric for Measuring and Visualizing the Effects of Learning Computer Programming for Elementary School Students. J. Inf. Technol. Educ. Innov. Pract.
**2020**, 19, 203–227. [Google Scholar] [CrossRef] - Sadhu, S.; Laksono, E.W. Development and validation of an integrated assessment for measuring critical thinking and chemical literacy in chemical equilibrium. Int. J. Instr.
**2018**, 11, 557–572. [Google Scholar] [CrossRef] - Demir, K.; Akpinar, E. The effect of mobile learning applications on students’ academic achievement and attitudes toward mobile learning. Malays. Online J. Educ. Technol.
**2018**, 6, 48–59. [Google Scholar] [CrossRef] - Christmann, A.; Van Aelst, S. robust estimation of Cronbach’s Alpha. J. Multivar. Anal.
**2006**, 97, 1660–1674. [Google Scholar] [CrossRef] [Green Version] - Angell, K. The application of reliability and validity measures to assess the effectiveness of an undergraduate citation rubric. Behav. Soc. Sci. Libr.
**2015**, 34, 2–15. [Google Scholar] [CrossRef] - Reising, D.L.; Carr, D.E.; Tieman, S.; Feather, R.; Ozdogan, Z. Psychometric testing of a simulation rubric for measuring interprofessional communication. Nurs. Educ. Perspect.
**2015**, 36, 311–316. [Google Scholar] [CrossRef] [PubMed] - ISTE. Standards for Students. Available online: https://www.iste.org/standards/for-students (accessed on 25 January 2021).
- CSTA K-12. Computer Science Standards. 2017. Available online: https://www.doe.k12.de.us/cms/lib/DE01922744/Centricity/Domain/176/CSTA%20Computer%20Science%20Standards%20Revised%202017.pdf (accessed on 25 January 2021).
- Van Solingen, R.; Basili, V.; Caldiera, G.; Rombach, H.D. Goal question metric (GQM) approach. In Encyclopedia of Software Engineering; John Wiley & Sons: Hoboken, NJ, USA, 2002. [Google Scholar]
- Tsuda, N.; Washizaki, H.; Honda, K.; Nakai, H.; Fukazawa, Y.; Azuma, M.; Komiyama, T.; Nakano, T.; Suzuki, T.; Morita, S.; et al. Wsqf: Comprehensive software quality evaluation framework and benchmark based on square. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Montreal, QC, Canada, 25–31 May 2019. [Google Scholar]
- Brown, T.A. Confirmatory Factor Analysis for Applied Research; The Guilford Press: New York, NY, USA, 2006. [Google Scholar]
- DeVellis, R.F. Scale Development: Theory and Applications; SAGE Publications: Thousand Oaks, CA, USA, 2003. [Google Scholar]
- Jackson, J.E. Oblimin rotation. In Wiley StatsRef: Statistics Reference Online; John Wiley & Sons: Hoboken, NJ, USA, 2014. [Google Scholar]
- Arcuria, P.; Morgan, W.; Fikes, T.G. Validating the use of LMS-derived rubric structural features to facilitate automated measurement of rubric quality. In Proceedings of the 9th International Learning Analytics & Knowledge Conference, Tempe, AR, USA, 4–8 March 2019; pp. 270–274. [Google Scholar]
- Catete, V.; Lytle, N.; Barnes, T. Creation and validation of low-stakes rubrics for k-12 computer science. In Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, Larnaca, Cyprus, 29 June–3 July 2018; pp. 63–68. [Google Scholar]
- Mustapha, A.; Samsudin, N.A.; Arbaiy, N.; Mohammed, R.; Hamid, I.R. generic assessment rubrics for computer programming courses. Turk. Online J. Educ. Technol.
**2016**, 15, 53–68. [Google Scholar] - Allen, S.; Knight, J. A method for collaboratively developing and validating a rubric. Int. J. Scholarsh. Teach. Learn.
**2009**, 3, 10. [Google Scholar] [CrossRef] [Green Version]

**Figure 5.**Puyo Puyo Scratch * Copyright of Puyo Puyo belongs to SEGA (The authors have obtained permission for use). © SEGA.

Learning Perspective | Learning Goals | |||||
---|---|---|---|---|---|---|

Category | Item | Stage 5 | Stage 4 | Stage 3 | Stage 2 | Stage 1 |

Thinking in the design and creation of the program | Subdivision of the problem | Can subdivide the problem, it is possible to make the solution and other things a subdivided problem | It can be divided into several smaller problems associated with major problems | It can be reduced from a big problem into smaller problems | It is possible to find one small problem from a big problem | Cannot subdivide the problem |

Analysis of events | It is the analysis of the events, the results of the analysis can be used in problem-solving and other things | Several factors (causes) related to an event can be found | You can find multiple factors (causes) for a certain event | You can find one factor (cause) about a certain event | Cannot analyze events |

Name | Definition | Details |
---|---|---|

Factor correspondence rate | X = A/B | A: Number of factors in the rubric that matches the evaluation objective B: Number of concepts in the evaluation objective |

Intrinsic Consistency | X = A/(A − 1) + (1 − B/C) | A: Number of items B: Total variance of each item C: Variance of the total score |

Item Correlation | Xn = A − Bn | A: Overall alpha coefficient Bn: Alpha coefficient when the nth item is deleted |

Label | Subject | Number of Students | Number of Quizzes |
---|---|---|---|

A | Arithmetic | 47 | 13 |

B | Science | 36 | 9 |

C | Information | 18 | 10 |

Q1 | Q2-1 | Q2-2 | Q2-3 | Q3-1 | Q3-2 | Q3-3 | Q3-4 | Q4-1 | Q4-2 | Q5 | Q6-1 | Q6-2 | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Sequence | 2 | 2 | |||||||||||

Branch | 3 | ||||||||||||

Repetition | 2 | 2 | |||||||||||

Subdivision of the problem | 4 | 4 | 4 | 2 | 2 | 2 | 2 | 2 | |||||

Analysis of events | 2 | 2 | 2 | 2 | |||||||||

Extraction operation | 3 | 3 | |||||||||||

Construction of the operation | 3 | 3 | |||||||||||

functionalization | 2 | 2 | 2 | ||||||||||

Generalization | 4 | 4 | 4 | ||||||||||

Abstraction | 3 | 3 | 3 | ||||||||||

Inference | 2 | 2 | 2 | ||||||||||

Operator | 2 | 2 |

Q1 | Q2-1 | Q2-2 | Q2-3 | Q3-1 | Q3-2 | Q4 | Q5 | Q6-1 | Q6-2 | |
---|---|---|---|---|---|---|---|---|---|---|

Sequence | 2 | 2 | ||||||||

Branch | 2 | 3 | ||||||||

Repetition | 2 | 2 | ||||||||

Subdivision of the problem | 2 | 2 | ||||||||

Extraction operation | 3 | 3 | ||||||||

Construction of the operation | 3 | 3 | ||||||||

Operator | 2 | 2 |

Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Q7 | Q8 | Q9 | Q10 | |
---|---|---|---|---|---|---|---|---|---|---|

Sequence | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |

Branch | 3 | 3 | 3 | 3 | 3 | 3 | ||||

Repetition | 3 | 3 | 3 | 3 | 3 | 3 | ||||

Variable | 3 | 3 | 3 | 3 | 3 | 3 | ||||

Array | 3 | |||||||||

Function | 2 | |||||||||

Analysis of events | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |

Construction of the operation | 2 | |||||||||

Functionalization | 2 | |||||||||

Understanding of the program | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |

A | B | C | |
---|---|---|---|

KMO value | 0.519 | 0.709 | 0.541 |

Q | Factor1 | Factor2 | Factor3 |
---|---|---|---|

Q1 | 0.403 | 0.099 | −0.153 |

Q2-1 | −0.047 | 0.050 | 0.658 |

Q2-2 | 0.163 | −0.015 | 0.700 |

Q2-3 | 0.025 | 0.499 | 0.446 |

Q3-1 | 0.971 | −0.087 | 0.024 |

Q3-2 | 0.971 | −0.087 | 0.024 |

Q3-3 | 0.928 | 0.091 | 0.008 |

Q3-4 | 0.928 | 0.091 | 0.008 |

Q4-1 | −0.126 | 0.869 | −0.001 |

Q4-2 | 0.095 | 0.840 | −0.052 |

Q5 | 0.134 | 0.263 | −0.006 |

Q6-1 | −0.010 | −0.126 | 0.684 |

Q6-2 | 0.352 | 0.485 | −0.076 |

Q | Factor1 | Factor2 |
---|---|---|

Q1-1 | 0.18 | 0.30 |

Q1-2 | −0.10 | 0.30 |

Q2 | −0.11 | 0.58 |

Q3-1 | 0.94 | −0.01 |

Q3-2 | 0.99 | −0.01 |

Q4 | 0.43 | 0.07 |

Q5 | 0.03 | 0.55 |

Q6-1 | −0.01 | 0.88 |

Q6-2 | 0.40 | 0.46 |

Q | Factor1 | Factor2 |
---|---|---|

Q1 | 0.10 | 0.68 |

Q2 | 0.48 | 0.11 |

Q3 | 0.10 | 0.84 |

Q4 | 0.37 | 0.28 |

Q5 | 0.80 | 0.03 |

Q6 | 0.62 | −0.42 |

Q7 | 0.48 | −0.44 |

Q8 | 0.81 | 0.14 |

Q9 | 0.30 | 0.28 |

Q10 | 0.72 | 0.12 |

A | B | C | |||
---|---|---|---|---|---|

Q | α | Q | α | Q | α |

Overall | 0.841 | Overall | 0.767 | Overall | 0.778 |

Q1 | 0.84 | Q1-1 | 0.723 | Q1 | 0.778 |

Q2-1 | 0.841 | Q1-2 | 0.759 | Q2 | 0.756 |

Q2-2 | 0.832 | Q2 | 0.728 | Q3 | 0.779 |

Q2-3 | 0.831 | Q3-1 | 0.685 | Q4 | 0.760 |

Q3-1 | 0.809 | Q3-2 | 0.677 | Q5 | 0.723 |

Q3-2 | 0.809 | Q4 | 0.721 | Q6 | 0.769 |

Q3-3 | 0.803 | Q5 | 0.717 | Q7 | 0.788 |

Q3-4 | 0.803 | Q6-1 | 0.696 | Q8 | 0.714 |

Q4-1 | 0.847 | Q6-2 | 0.685 | Q9 | 0.776 |

Q4-2 | 0.835 | Q10 | 0.733 | ||

Q5 | 0.847 | ||||

Q6-1 | 0.842 | ||||

Q6-2 | 0.831 |

Statistical Methods | A | B | C |
---|---|---|---|

Factor correspondence ratio | 1 | 0.86 | 0.89 |

Internal consistency | 0.841 | 0.767 | 0.778 |

Inter-item correlation | Q4-1/Q5/Q6-1 | N/A | Q7 |

Question | A | B | C |
---|---|---|---|

Q1: Is the rubric-based assessment consistent with the purpose of the assessment? | Consistency. | Generally consistent | Generally consistent |

Q2: Is it possible to evaluate the items for evaluation purposes using the rubric? | Evaluated | Generally evaluated | Generally evaluated |

Q3: Do the rubric’s assessment items evaluate a consistent concept? | Evaluated | Generally evaluated | Generally evaluated |

Q4: Are the rubric’s evaluation items appropriate in terms of consistency? | Generally appropriate | Reasonable | Generally appropriate |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Saito, D.; Yajima, R.; Washizaki, H.; Fukazawa, Y.
Validation of Rubric Evaluation for Programming Education. *Educ. Sci.* **2021**, *11*, 656.
https://doi.org/10.3390/educsci11100656

**AMA Style**

Saito D, Yajima R, Washizaki H, Fukazawa Y.
Validation of Rubric Evaluation for Programming Education. *Education Sciences*. 2021; 11(10):656.
https://doi.org/10.3390/educsci11100656

**Chicago/Turabian Style**

Saito, Daisuke, Risei Yajima, Hironori Washizaki, and Yoshiaki Fukazawa.
2021. "Validation of Rubric Evaluation for Programming Education" *Education Sciences* 11, no. 10: 656.
https://doi.org/10.3390/educsci11100656