How the Mastery Rubric for Statistical Literacy Can Generate Actionable Evidence about Statistical and Quantitative Learning Outcomes

Statistical literacy is essential to an informed citizenry; and two emerging trends highlight a growing need for training that achieves this literacy. The first trend is towards “big” data: while automated analyses can exploit massive amounts of data, the interpretation—and possibly more importantly, the replication—of results are challenging without adequate statistical literacy. The second trend is that science and scientific publishing are struggling with insufficient/ inappropriate statistical reasoning in writing, reviewing, and editing. This paper describes a model for statistical literacy (SL) and its development that can support modern scientific practice. An established curriculum development and evaluation tool—the Mastery Rubric—is integrated with a new, developmental, model of statistical literacy that reflects the complexity of reasoning and habits of mind that scientists need to cultivate in order to recognize, choose, and interpret statistical methods. This developmental model provides actionable evidence, and explicit opportunities for consequential assessment that serves students, instructors, developers/reviewers/accreditors of a curriculum, and institutions. By supporting the enrichment, rather than increasing the amount, of statistical training in the basic and life sciences, this approach supports curriculum development, evaluation, and delivery to promote statistical literacy for students and a collective quantitative proficiency more broadly.


Introduction
Statistical literacy (SL) is widely described as important for full social participation (see [1]; elementary curricula, e.g., [2,3]; higher education and beyond, e.g., [4][5][6]).Although this is true for all students, there is a special relationship between statistics and scientific research that amplifies the importance of developing appropriate statistical literacy in undergraduate or graduate/post-graduate students in the sciences.
Empirical research relies on statistical methods, and statistics is a wide, dynamic field perpetually propelled by new and improved methods.This far outstrips the capacities of other fields to fully adapt to these innovations, much less to incorporate all "relevant" methods in their own PhD curricula.Recently, Weissgerber et al. (2016) [7] correctly articulate that-and the myriad empirical arguments why-basic scientists need training in statistics (see also [8][9][10][11][12][13][14][15][16]; see also [17]).In fact, science PhD programs face a nearly Sisyphusian task: to adapt to some or any new methods, or even to prepare their students to adapt, so that their non-statistical discipline may exploit the power of new, or justify selecting established, statistical methods.Learning all statistical methods is clearly not feasible; even a focus on "just" those that are currently relevant for the discipline may impede adoption of newer, more efficient methods in the future.However, initiating the development of statistical literacy and orienting science students to value quantitative methods, which empowers them to seek additional training when needed, might be an achievable goal.
Exemplifying the special importance of statistical literacy for scientists as they are trained is the Carnegie model of the doctorate wherein PhD programs prepare graduates to be/become "stewards" of their scientific disciplines (see [18] (pp.[9][10][11][12][13][14]).The definition of a steward of a discipline is "someone who will creatively generate new knowledge, critically conserve valuable and useful ideas, and responsibly transform those understandings through writing, teaching, and application" [18] (p.5).Consistent with the disciplinary stewardship model, Henson et al. (2010) [12] propose a "collective quantitative proficiency" (CQP) model explicitly linking the valuation of quantitative methods within the culture of a scientific discipline to the training in these methods that is provided to the future researchers in (stewards of) that discipline.The CQP was described originally for education researchers, but the argument and model are appropriate to all sciences.In fact, Weissgerber et al. (2016) [7] review only the most recent literature representing the damage that weak or incomplete (or incorrect) knowledge of statistics and statistical methods is currently having on the rigor, interpretability and reproducibility of scientific work across the basic and life sciences.Established scientific practitioners must become more statistically literate to effectively model this competency for their mentees and students, to teach effectively, and to promote competence in writing, reviewing, and editing across the sciences.As Shulman noted, "(b)oth scholarship and teaching in any field reflect the character of inquiry, the nature of community, and the ways in which research and teaching are conducted in that particular discipline or disciplinary intersection" [19] (p.xii).Students at all levels need to know (and observe) that their scientific mentors also value-and contribute to-the collective quantitative proficiency (CQP) that disciplinary stewardship requires.
Despite the importance of statistical literacy and competency for the practicing scientist and the steward, doctoral programs may struggle with recommendations to add statistical training (see [20][21][22]).Many science PhD programs include no formal statistical training, or just a single course (see [23]; see also [7]).Two emerging trends in the basic and life sciences are highlighting a growing need for the addition-and integration-of statistical training in these disciplines.The first trend is towards "big" data across basic and life sciences; where the potential to automate-and thereby remove from active consideration-statistical inferences across datasets could ultimately exclude formal training and reasoning in statistics and experimental design.While some PhD programs contemplate adding statistical training to their programs, there is also movement to integrate "big data" into training future or current stewards of the biomedical sciences-without attention to reproducibility, experimental design, inferential statistics, or statistical literacy (e.g., [24]).While automated analyses can exploit massive amounts of data, without statistical literacy, the interpretation-and possibly more importantly, the replication, of results is challenged.However, "statistical literacy" is not included as a key competency in most fields (e.g., bioinformatics [25]; biology [26]) and where it is discussed, it relates to undergraduate single-course educational requirements (compliance) or to something less concretely defined (e.g., [27][28][29]).These arguments focus on undergraduate and PhD level programs because at the Master's level, the course load is usually rigidly fixed; however, those seeking or completing Master's level preparation are also challenged when it comes to statistical literacy.
It seems impossible to achieve the goal of a "collective quantitative proficiency" [12] among disciplinary stewards given the resistance to (or lack of time for, or lack of opportunities/interest in) coursework beyond introductory statistical training (e.g., [22]).However, adding or retaining one course in "introductory statistics" is also unlikely to achieve sufficient statistical literacy for modern scientific practice-as either a producer or a consumer of argument that relies on quantitation and data.A one-course approach to statistical literacy for PhD programs implies that: (A) the single course is sufficient to teach the critical-and complex-set of skills that encompasses "the ways in which research . . .(is) conducted in that particular discipline [19]; and (B) the single course will support the level of consumption and production of statistical arguments representing competent stewardship of a discipline that uses these methods.
The one-course-done model of statistical training exemplifies the comment by Henson et al. (2010) [12] (p.235) that "(f)aculty and students often perceive quantitative methods as a static field to be mastered".Moreover, the current conceptualizations of statistical literacy are grounded in the satisfaction of an undergraduate requirement (e.g., [30,31], for example, the Guidelines for Assessment and Instruction in Statistics Education (GAISE [31]; see also e.g., [6]).The scientist, professional, and/or instructor must be considered to have statistical literacy needs that differ qualitatively and quantitatively from those of undergraduates whose use for, or application of, statistical reasoning and methods is not yet known.For professional scientists, statistical literacy must support the responsible stewardship of their disciplines, producing and consuming statistical arguments (see e.g., [12]; [32] (p.xiii); see also [23]).This is a complex set of skills required for literature review, documenting the background and contextual (apart from the statistical) significance of one's work, and for writing and reviewing manuscripts.Instead of reinforcing the perception that quantitative methods are "static", an explicitly developmental model of statistical literacy directs attention of PhD scientists (students and mentors alike) towards their own awareness of the importance of, and variety in, quantitative method options for their research and discipline.Because the model is developmental, it can be augmented to accommodate learners earlier (than the PhD; see [33]) in their training.This conceptualization of statistical literacy as developmental could fulfill the objectives of increasing statistical sophistication for scientists, reviewers, and faculty/mentors who are training future scientists, reviewers and faculty/mentors.Moreover, although other models have separated statistical "literacy", "reasoning", and "thinking" [34], these are actually three stages in a developmental trajectory that describes a deepening of sophistication with respect to data and principles of statistics.A more explicit statement of this development is intended to promote the "cultural" shift towards CQP in PhD training in basic and life sciences like biology, physiology, biochemistry, and genetics-towards a more holistic, reflective, and adaptive view of statistical literacy (SL).A curriculum development and evaluation tool, the Mastery Rubric (described in the next section), can be used to create, evaluate, or revise curricula that can generate actionable evidence (see [35]) of performance by students, instructors, and institutions.In this manuscript, a new Mastery Rubric for Statistical Literacy (MR-SL) is presented, and its potential to generate actionable evidence of growth and development in understanding of fundamental statistical concepts, and reasoning with them, is explored.

The Mastery Rubric
A traditional rubric is assignment-specific and lists the skills the grader requires in the work product, along with performance levels from poorest to best [36] (Chapter 1).The Mastery Rubric is similar, but outlines the knowledge, skills and abilities (KSAs) to be developed within the curriculum (or over time), together with performance levels that characterize the learner moving from novice to expert [37,38].
Related to the Mastery Rubric is the concept of a "learning progression" (e.g., [39] (p.1)) which describes shifts from naïve to "more expert understanding" and is based on how children learn the concepts of interest (but see [40] for an example with law students).Whereas a learning progression represents a curricular segment (e.g., Schwarz et al. 2009 [41]), the Mastery Rubric [37,42,43] represents the entire (predominantly) post-baccalaureate curriculum.Like the Mastery Rubric for Statistical Literacy (MR-SL, Table 2), two others were designed to capture and encourage development throughout the career [42,43].Additionally, unlike a learning progression, the Mastery Rubric is public: explication of curricular objectives, and what work products look like then these are met, facilitates the identification by faculty, mentors, or evaluators of strengths and weaknesses in the curriculum itself.This also formalizes the evidence that any individual may elicit (instructor/institution) or present (student) to support their claim of achieving a target performance level throughout the curriculum.This can also support faculty in other courses to create opportunities to generate this evidence, and instruction supporting the same objectives from diverse contexts and perspectives.Explicit and public description of the necessary evidence can, in turn promote learners to self-monitor, and spur the individual (student, instructor, or institution) to seek (or create) opportunities to generate such evidence [42].
The Mastery Rubric represents the perspective of Messick (1994) [44]: articulating what KSAs students should possess at the end of the curriculum; what behaviours by the students will reveal these KSAs; and what tasks will elicit these specific behaviours.Toohey (1999) [45] refers to this outcomes-based approach as "systems-" or "performance-based", and every Mastery Rubric follows this approach.Thus, by design, any Mastery Rubric supports assessable curriculum development, evaluation, and delivery because learning objectives are articulated and public so that each can be explicitly aligned to individuals' progress and development along the articulated continuum from novice to expert.Then, conversations about curricular objectives, and actionable evidence of whether or not they are being met, are possible for all stakeholders.
In the next sections, the MR-SL is presented and described, and its alignment with principles of learning outcomes documentation [46] is analyzed.

Materials and Methods
Every Mastery Rubric is constructed with two dimensions: performance levels that represent a developmental trajectory (columns) and knowledge, skills, and abilities that represent the targets of the teaching and/or learning (rows; [38]).The methods by which each dimension of the MR-SL was constructed are articulated below.A degrees of freedom analysis [47][48][49] was used to create a matrix to permit examination of alignment of features of the MR-SL with the Principles for Learning Outcomes articulated by the National Institution for Learning Outcomes Assessment (NILOA [46]).

The Mastery Rubric for Statistical Literacy (MR-SL): Establishing a Developmental Trajectory
As noted, one of the two essential elements in the creation of a Mastery Rubric (MR) is the articulation of a developmental trajectory.Much of the research in statistical literacy has focused on understanding how students or experts think about data (e.g., [50][51][52])-which means that the two ends of the "developmental trajectory" in this discussion to date are "completing the undergraduate course" and "being an expert".
The Mastery Rubric for Statistical Literacy (MR-SL) was designed from the opposite perspective, namely, to articulate what is common across middle stages of engagement with data (consumption and production), with desired entry and exit criteria for each stage, along an explicit continuum from more naïve to more expert.This is achieved by explicit reference to Bloom's Taxonomy of Educational Objectives [53]; see also [54].Moreover, the MR-SL was constructed synthesizing a developmental view of Bloom's taxonomy with a long-standing model of the development of general literacy [55], focusing on the knowledge, skills, and abilities specific to statistical literacy arising by consensus from the literature (e.g., [50][51][52][56][57][58].
Table 1 presents the Bloom's Taxonomic context of the MR-SL.Expert reviewer-capable of stewardship of the not-statistics discipline.
Expert review, diagnosis and recommender of remediation; capable steward of a statistical discipline.
Table 1 includes an important column that does not actually appear in the Mastery Rubric but is included here because it is such a common stage across the biomedical and life sciences (and across some social and educational sciences as well): the pre-literate non-reasoner.This individual is described consistently in critiques of the quality of journal and grant reviewing (see also e.g., [27,29]), and is identified specifically by the lack of skills in the recent review by Weissgerber et al. (2016) [7].The difference between a scientist who functions at this level and one who functions even at the Beginning Literacy level is profound-and their effects undermining the rigor and reproducibility of scientific research are increasingly less tolerable (e.g., [8][9][10][11][12][13][14][15][16]; see also [17]).Recognition that some reviews provided for journal editorial decisions, as well as grant funding, represent functioning at this level should be highlighted in these important decision-making contexts (i.e., even this structure represents actionable evidence).
The MR-SL can promote remediation of these identified weaknesses by individuals seeking to generate evidence they are "on the right track" or at least at the Beginning Literacy stage-and by institutions seeking to provide opportunities to achieve learning outcomes consistent with performance at this stage (or beyond).PhD students and scientists are often not operating at even the lowest Bloom's taxonomy [53] level (knowledge-the main level targeted by most statistical training, see e.g., [21,27]) while professionally, they must function at the highest level (e.g., [10][11][12]; see also [32] (p.xiii)).
Evidence of this (perhaps surprisingly low) level of functioning with respect to statistical and quantitative argumentation comes from a variety of sources (e.g., [7,8,[14][15][16]); the pre-literate non-reasoner is common and problematic.If evidence is found that an institution is training people to this level (and not beyond), action must be taken to remediate the situation or to reconfigure curriculum or learning objectives that purposefully aim at this level of performance.The MR-SL treats statistical literacy in a similar manner to general literacy [54]: comprising a set of learnable, improvable skills.In order to promote development of a CQP by initiating the learning and improving of this skillset, the MR-SL could be used to promote curricular or institutional remediation.

The Mastery Rubric for Statistical Literacy (MR-SL): KSAs for SL
The second dimension of a Mastery Rubric is the articulation of knowledge, skills, and abilities (KSAs) that are to be targeted and grown throughout the developmental trajectory.For the MR-SL, the list of KSAs representing statistical literacy was derived by synthesizing several models of statistical literacy with the more active "empirical enquiry" model of Wild and Pfannkuch ([57]; see also [58]).Because the developmental trajectory for these KSAs describes change from more naïve to more expert performance, the qualification of how these KSAs are executed is captured (and described) in the row that outlines growth and development in each KSA over time/training.The SL KSAs were synthesized from "A four-dimensional framework for statistical thinking in empirical enquiry" [52] (p.19) and the "Statistical Thinking" facility described in [58] (p.218) into the new MR-SL shown in Table 2.  Recognizes need for pilot studies and asks for appropriate assistance in the design and analysis.Pilot results are seen to be useful in addressing scalability issues.May seek assistance with scalability based on pilot results.Does not recognize when design or review demands are beyond their skill set.
Independently conceptualizes pilot studies that address relevant design issues.May seek expert advice for design, power, and analysis planning for their own work, and consistently recognizes when reviewing demands are beyond their skill set.
Expertly designs and analyzes pilot studies, utilizing the data for full study design, analysis planning and power, within their own and others' work.
Diagnoses and remediates each of these across contexts.

Discerning "exploratory", "planned", and "unplanned" data analysis
Does not perceive differences between "planned", and "unplanned" data analysis in their own or others' work.Does not recognize that exploratory analyses can be planned or unplanned and that these should be described as such.
In their own work, can differentiate between exploratory analysis and hypothesis testing, but not "planned" and "unplanned" analyses.May incorrectly characterize "exploratory" analysis as hypothesis testing (planned or unplanned).
Perceives differences between "planned", and "unplanned" data analysis in their own work, but not in others' work unless it is identified.May not recognize that exploratory analyses can be planned or unplanned, does not know why it might matter to communicate which they are doing/reporting.
Recognizes differences between "planned", and "unplanned" data analysis in their own and others' work, even when others do not recognize it in their own work.Knows that exploratory analyses can be planned or unplanned, and can identify which is included in their own and others' work.
Clearly and consistently differentiates planned and unplanned analyses in their own work and that of others.Utilizes all types of analysis appropriately in support of coherent contributions to science.Consistently requires others to do the same, and can diagnose and remediate each of these across contexts in order to support scientific integrity and competence.

Hypothesis generation based on planned and unplanned analyses
Uses the default settings of software to guide analysis planning (and execution in the unplanned analysis case  The model of statistical thinking articulated by Wild and Pfannkuch ([57] discussed in [52] (pp.[18][19][20], captures the features of literacy, reasoning, and thinking that are relevant for graduate science curricula (as noted by [58]; see also [50]) and beyond.Thus, this model embodies " . . .value on the integration of quantitative methods as part of the substantive enterprise of doctoral education" [12] (p.236).The KSAs (rows) in the MR-SL are:

•
Define a problem based on critical literature review; These KSAs generally define the scientific method-and also require content knowledge.The initiation and development of these KSAs could therefore be integrated across multiple content course areas, and also for those who are practicing scientists-whether or not they completed PhD-level training.The MR-SL serves to link instruction in statistical methods with the application, and reasoning with, those methods and results.Thus, it can support the initiation of the development of this set of KSAs and their continued promotion within, and beyond the ending of, formal education.There are six mutually exclusive performance level descriptors for each of these KSAs in the MR-SL (Table 2); the integration of the Bloom's level functioning at different stages with the features of statistical literacy are explicit.

Results
The MR-SL in Table 2 co-articulates cognitive perspectives on the development (columns) derived from extant literature with context-appropriate and explicit, but flexible, descriptions of a complex set of knowledge skills and abilities KSAs (rows) that represent statistical literacy as a learnable, improvable skill set.Table 3 provides a rough alignment of the KSAs in the MR-SL and definitions of statistical "literacy", "thinking", and "reasoning" in prior models.Piloting, analysis and interpretation.

Do final study; collect and present the data;
analyze and interpret the data.Hypothesis generation based on planned and unplanned analyses.

Interpretation of results.
"To think statistically means that one can: 1. Read data, critically and with comprehension; 2. Produce data that provide clear answers to important questions; 3. Draw trustworthy conclusions based on data" [58] (p.220).

Statistical reasoning.
Draw and contextualize conclusions. Communicate.
Table 3 shows that the alignment of the MR-SL KSAs is tightest with the model of statistical thinking outlined by Bishop and Talbot (2001) [58] but it is also similar to the statistical thinking model of Wild and Pfannkuch (1999) [57].The definition of "statistical literacy" given in Garfield, delMas and Chance (2003) [34] may be foundational to engaging in any of the KSAs, but "statistical thinking" as they defined it may be more aligned with the consumption of statistical reasoning and is not (according to Table 3) involved in production.However, considering the alignment of their definition of "statistical reasoning" with the interpretation of results and drawing of conclusions suggests that the ability to reason statistically can be developed without a focus on data collection or analysis (which are key aspects of the other two models of "statistical thinking").For scientists who are either in training or in practice, both the production and consumption of statistical argumentation are essential and these can be leveraged as two types of important practice for ensuring that the learning in statistics coursework is sustainable (endures beyond the end of teaching and can be applied in different contexts than where it was learned).
It is not necessary that all learners progress on all KSAs simultaneously; with the co-articulation of KSAs with developmental stages of performance on each one, instructors, institutions and students can leverage their time and effort in order to ensure that all KSAs are performed, at some point (e.g., midway through a degree program), at a target level.The co-articulation of the MR-SL also supports the generation of actionable evidence for learners who identify one or another KSA as most challenging, as well as for institutions or instructors that identify performance on one KSA or another as least-consistent across a student cohort.The co-articulation also both captures an explanation for why reproducibility and peer review in science are widely perceived to be weak (i.e., because people do "function" at insufficiently-sophisticated levels on some, if not all, of these important KSAs) and also provides an approach, to an individual, instructor or the institution, for addressing this weakness.
Table 4 is a degrees of freedom analysis [47][48][49] evaluating the alignment of the MR-SL and its potential to support evidence-based decision-making in higher education as well as the five Principles for the documentation of learning outcomes [46].
It can be seen in Table 4 that four of five Principles [46] are addressed by the creation of the MR-SL and its adoption to promote statistical literacy that is appropriate for PhD science students and anyone who will consume or produce scientific argumentation that depends on data or quantitation.One of the five principles (collaborative) is not addressed at all by the MR-SL; however, the MR-SL KSAs are articulated based on multiple models of statistical thinking and reasoning, and on real-world requirements for applied statistical literacy by scientists in their daily work.Explicit articulation allows learners to see what is expected of them and institutions/instructors to support learners in their achievement of these expectations.Thus, the MR-SL is "representative", but not necessarily "collaborative".Its implementation at any institution would need to be based on all stakeholders agreeing, so "collaboration" might become relevant in that (implementation) context.
The alignment of the MR-SL with one of the principles ("Outcomes are focused on improvement") does not differentiate between the learner and the instructor/institution.The instructor/institution can obtain actionable evidence about how courses or training support improvement in key outcomes, and the learner can obtain actionable evidence about what other information or training is needed in order to achieve a targeted performance level on each KSA.With a Mastery Rubric self-monitoring is focused on, "what training do we offer to promote growth or development of this KSA, what else can we offer to support if for all learners?"(institution/instructor) and "how well do I do/know this KSA, what do I need to do to become more proficient?"(learner).These perspectives are sufficiently similar to warrant collapsing over instructor/learner in Table 4. Outcomes are focused on improvement The explicit articulation of expected growth and development in the target KSAs focuses all stakeholders on improvement of these KSAs-emphasizing they are not static.

Outcomes document learning and its extent
Learners generate evidence of their achievement and ongoing development of KSAs.
Instructors/institutions structure training/teaching to generate documentation of learning and the achievement of articulated learning objectives.

Outcomes provide evidence of quality of learning
A portfolio can be created articulating the extent and quality of learning.
Assessment opportunities that document the achievement and quality of learning can be developed.

Expectations are explicit in the outcomes
The MR-SL makes explicit the expectation that the learner takes some responsibility for self-assessment and ensuring ongoing development until the target performance level is achieved.
The MR-SL makes explicit the institutional obligation to provide learning opportunities that can and do promote growth and development in the target KSAs.
Evidence from the outcomes is externally relevant Portfolios documenting the achievement of learning outcomes (in statistical literacy) can be used to document readiness/qualification to review.
Statistical literacy is known to be weak; institutions that adopt the MR-SL and use it to guide curriculum development or evaluation can document their alignment of learning outcomes with the improvement of statistical literacy and/or contributions to the collective quantitative proficiency.
Four additional rows are included in Table 4.These are not "principles" for documenting learning per se, but they are relevant to a discussion about promoting statistical literacy with a Mastery Rubric approach and they are also discussed in the NILOA policy statement [46].These additional considerations are that:

•
Outcomes document learning and its extent; • Outcomes provide evidence of quality of learning; • Expectations are explicit in the outcomes; and • Evidence from the learning outcomes is externally relevant.

Discussion
A Mastery Rubric emphasizes habits of mind as they transition from more novice to more expert, along a Bloom's-compatible developmental trajectory [37].The developmental stages of the MR-SL map onto those articulated for the development of general literacy [55], and the potential for explicit articulation of performance levels for each KSA are aligned with the self-efficacy argument of Bandura [59] (Chapter 2).The MR-SL captures key features of engagement in scientific inquiry (e.g., [57,58]); it is consistent with four of five NILOA principles for learning outcomes, and has an additional four features that generate actionable evidence by both the learner and the instructor/institution.Overall, these features suggest that the MR-SL is strongly supportive of the perspective that documenting learning matters.
Undergraduate statistical literacy is fundamentally different from that required for applied science and for doctoral level work, but it is not expertise in statistics that is targeted with the MR-SL-it is expertise, or movement towards it, in this particular type of literacy that is targeted.Perhaps especially, explicitly describing what the KSAs are and how they should be developing can promote the recognition that/when additional training (institution) or information (learner) is needed.This approach is supportive of the identification of strengths and weaknesses-in the student and in the curriculum-thereby promoting creation or revision of training opportunities to address the identified gaps.This in turn can promote the concrete articulation by learners/trainees of how statistical training experiences have actually promoted observable changes in their own SL strengths and weaknesses.
The one NILOA principle for documenting learning outcomes with which the MR-SL is not closely aligned, that outcomes are articulated via collaboration with all stakeholders, is a significant limitation to the applicability of this work-because buy-in from faculty across courses and possibly disciplines or departments is critical for institutional adoption of the MR-SL.It is possible, however, that a focus on the representativeness of the KSAs of what is commonly defined as "statistical literacy", and the alignment of the developmental trajectory with other well-established models (e.g., [54,57,58]; [59] (Chapter 2)), can facilitate consideration of how the MR-SL can best be adopted or adapted to achieve institutional objectives in support of the collective quantitative proficiency and statistical literacy that modern scientific practice requires.

Conclusions
Many PhD science programs require a single statistics course, and although this may suffice for undergraduates (see [31]), statistical literacy to support responsible stewardship of a scientific discipline differs fundamentally from that of undergraduates (see, e.g., [58]).Any syllabus can be compared to the MR-SL to determine the stage at which learners would be able/are expected to function, as well as evaluating the extent to which the learning objectives articulated in the syllabus are supportive of growth in statistical literacy.Moreover, the MR-SL can be used like other Mastery Rubrics have been to revise an existing curriculum [43], or to create new training opportunities that can promote the initiation of, and sustainable development in, a target set of KSAs [42].
It is not possible for degree and training programs to teach every quantitative method.At a minimum, because the developmental trajectory and KSAs are specified in this article, institutions can use it to determine the highest level a graduate in any program (undergraduate or graduate) can expect to attain given the existing statistical training and practice opportunities.Individual scientists may use the MR-SL to seek new quantitative learning opportunities by placing themselves on the developmental trajectory with respect to each KSA.With a focus on their metacognitive awareness of their own statistical literacy, individuals in or beyond their formal education setting can discern their growth and/or the need for more training.
The National Institute for Learning Outcomes Assessment [46] articulates that . . ."students need a postsecondary education that will prepare them to meet the challenges of the 21st century" (see also [60]).This paper describes a model for statistical literacy (SL) and its development that can support the dynamics of practicing modern science, starting with either graduate or undergraduate training.The MR-SL does so by generating actionable evidence about learning outcomes in statistical literacy from institutional, instructor, and student performance.The developmental framework around SL promotes the learner's understanding of his/her own statistical reasoning, as well as growth and depth of their knowledge, skills, and abilities relating to data and statistical analysis.This feature is an important element of education that can prepare learners "to meet the challenges of the 21st century", because knowledge is increasing at a rate we simply cannot keep up with.A crucial aspect of 21st century education is preparing individuals to continue learning-part of which involves self-assessment.
The Mastery Rubric provides explicit opportunities for consequential assessment that serves students, instructors, developers/reviewers/accreditors of a curriculum, and institutions.By supporting the enrichment, rather than increasing the amount, of statistical training in the sciences, the MR-SL supports evaluable curriculum development, evaluation, and delivery to promote statistical literacy for students and a collective quantitative proficiency more broadly.This model for promoting SL can be adopted by an individual for their own learning, or by a department or discipline, to promote ongoing and integrated teaching and learning in statistical reasoning.The extent that the model is adopted can support a cultural shift across scientific disciplines towards a collective quantitative proficiency that enables scientists and students alike to determine which methods to learn about and also how to know if they have learned enough about the chosen methods for professional-level engagement in modern life and science.
The model, described in the next section, is intended to: [12]on et al. (2010)etween this statistical literacy and the "collective quantitative proficiency" ofHenson et al. (2010)[12]; and C.represent statistical literacy training that could be integrated into-or at least initialized within-any PhD science program (and possibly earlier).

Table 1 .
Performance levels in the developmental trajectory of "statistical literacy": Given a research question, proposal, manuscript, report, or grant, this reader will/is.
Not yet on the Bloom's trajectory.Bloom's 1 remembering, understanding.Bloom's 2, 3, understand and apply but only apply what you're told to apply.Bloom's 3-5 Choose and apply techniques.Analyze & interpret.Identify limitations, but not sophisticated enough to independently review literature, proposals, grants.Bloom's 5-6 evaluate (review) and synthesize for new methods but not for evaluation of others.Bloom's 6 synthesize for new methods, and evaluation of others.Not a careful consumer.Becoming a careful Consumer.A careful consumer.Becoming a careful producer.Expert consumer, expert producer.No or limited capacity to critique.Requires external "validation" to believe what is presented (e.g., "it was published in JAMA!" "Cochrane Reviews are correct").Developing: capacity to evaluate; sense of what is/is not appropriate; ability to critique; opinions on debates (e.g., application of multi-model inference; Bayesian vs. frequentist; when to use multiple-comparisons corrections).

Table 3 .
Alignment of models of statistical reasoning, thinking, and literacy with the MR-SL KSAs.

Table 4 .
Alignment of Principles for documenting and improving assessment with features of the MR-SL from student and institutional perspectives.