The Difficulty with Setting a Standard Based on Difficulty: The Role of Validity in Determining Assessment Standards

Abstract

When setting standards for high-stakes assessments, it is necessary to be clear whether the primary criterion should be importance or difficulty. While standard setting practices frequently rely on the difficulty to determine passing or failing thresholds, this can undermine the intended inferences and uses of assessments. Standard setting based on difficulty conflates importance with subjective perceptions of difficulty, misguiding decisions about what candidates need to demonstrate for safe and effective practice. We explore several reasons why difficulty does not align with the aims of assessment and how prioritising importance can better reflect the intended uses and inferences of a test. Our analysis incorporates: (1) the relevance of validity to setting standards, (2) potential misunderstandings of norm- and criterion referencing, (3) critical differentiation between importance and difficulty, (4) reasons for prioritising validity, (5) the variability of candidates’ difficulty perceptions, (6) challenges of assessor judgments, (7) the need for clear definitions of competence, and (8) the appropriate use of ‘would’ vs. ‘should’ in establishing standards. Ultimately, a valid standard reflects the ability to demonstrate performance of what is important, not difficult, in the workplace.

1. Validity as the Central Framework for Standard Setting

Validity in assessment refers to the degree to which an evaluation accurately measures what it intends to measure, ensuring that the results reflect the intended purpose (Plake & Wise, 2014). When determining a passing standard, the emphasis should be placed on the importance of measuring the relevant competencies or knowledge through valid methods. In high-stakes assessments, the validity of inferences and decisions hinges on ensuring that the outcomes reflect candidates’ ability to demonstrate necessary knowledge, skills, and behaviours (Bandaranayake, 2008). Validity in this context should be conceptualised as the alignment of assessment content, performance expectations, and passing standards with the intended uses of the test. Specifically, when setting standards, the focus should be on what is essential for candidates to know and be able to do (importance), not how difficult the content is perceived to be (difficulty).

2. The Role of Norm- and Criterion Referencing in Setting Standards

Glaser (1963) was the first to use the terms norm- and criterion referencing. Norm referencing determines the outcome for each candidate based on their performance against the other candidates’ performance. This is in contrast to criterion referencing, which determines the outcome for each candidate based on their performance against predefined requirements. Established mainstream standard setting methods can theoretically be placed at different points along a continuous spectrum between these two extremes. Norm referencing can be considered to depend on an evaluation of difficulty, and criterion referencing to depend on an evaluation of importance. For example, Angoff (1971) is often claimed to be a criterion method, with Cohen et al. (1999) representing a normative method, and Ebel (1972), which involves judgement of both difficulty and importance, being considered a combined method. It can be contended that a problem with Angoff-based methods is that they have long purported to produce a criterion-based standard (Hambleton, 1993; Goodwin, 1996), while they are in practice conflated by norm referencing.

At the advent of criterion-referenced standard setting using minimal competence, there was much exposition concerning its use that is often neglected and worth recalling. The required performance standards should be clear in advance of any instruction (Popham, 1971), and the subsequent performance should be evaluated without reference to the performance of others (Wiersma & Jurs, 1976). Mager identified the need to stipulate a minimum performance requirement for every objective, rather than achievement of an overall performance standard representing mastery (Mager & Peatt, 1962). More specifically, criterion referencing can be distinguished as representing a behavioural task rather than a standard (Jackson, 1970). To elaborate further, criteria should be referenced to the degree of achievement measured during a test on a continuum of performance, which demonstrates the capability to successfully complete tasks (Glaser, 1963). When acquisition occurs on a continuum, it follows that testing all candidates at the same time is unlikely to be optimal from their perspective: the choice of any specific criteria can ultimately be undermined when consideration is given to the extent to which such criteria are an arbitrary sample (Glass, 1978), and a standard defined by criteria can be further eroded by the inclusion of any buffer in the requirement to account for lapses by those who are ‘masters’ (Tyler, 1971).

3. Differentiating Importance and Difficulty

Importance can be defined as ‘necessary or of great value, meaning or effect’ (McIntosh, 2022b). In contrast, difficulty can be defined as ‘the fact of not being easy to do or understand’ (McIntosh, 2022a). There are different and well-established methods used to set standards, such as Angoff (1971); Ebel (1972); Cohen et al. (1999), with a growing wealth of research reviewing their use (Boulet et al., 2003; Bandaranayake, 2008; Cohen-Schotanus & van der Vleuten, 2010; McKinley & Norcini, 2014). However, as already contended, these all involve at least a degree of norm referencing, relying on judgements of how difficult candidates will find items rather than how important it is for them to have the knowledge to answer correctly. A wealth of newer alternative standard setting methods have been developed, using body of work, bookmarking, item descriptor matching, briefing book, and benchmarking (reviewed by Cizek, 2012). However, it is still a novel challenge to conceive of setting a standard that is decoupled from an evaluation of difficulty. ‘Difficulty’ has been used to refer to various distinctly different constructs that have long been established, including: assessor judgments of difficulty based on expert knowledge typically before a test (Busch & Jaeger, 1990), perceived candidate difficulty based on perceived effort during a test (Sweller, 1988), and statistical item difficulty based on candidate performance after a test (Ebel & Frisbie, 1991).

Unfortunately, the different constructs of difficulty are often treated interchangeably, because standard setting methods do not generally draw a distinction or use a definition that should not be a consideration in standard setting. The point has been well made elsewhere that concepts like difficulty (Cizek & Bunch, 2007) and borderline (Shulruf et al., 2018) are often poorly defined. The expected performance of a group on an item is not the same as difficulty in terms of how easy or otherwise an individual finds an item. Whether or not an individual student or group of students would answer an item correctly or not (student-level property) should not determine whether they should be able to answer it correctly, or whether they need to be able to (importance; a test-level property).

All conceptualisations of the difficulty construct can be considered to be irrelevant when trying to determine whether someone has achieved what is important (e.g., when assessing knowledge that has implications for the health and safety of others). All questions could be judged for inclusion on the basis that they are important to practice, irrespective of difficulty (e.g., as can be achieved with assessments that adapt by content; Burr et al., 2022). This can result in questions being perceived to be of varying difficulties. However, difficulty is a subjective and variable construct and will vary by the individual in a way in which the importance of a given piece of knowledge or skill does not. Difficulty is further confounded by an individual’s familiarity with a task and their perceptions of their likely attainment (Li et al., 2007). If attempting to balance the difficulty of items, whether based on ‘expert’ judgement or analysis of cohort performance (with the associated issues that this norm referencing introduces), the perception of variable difficulty is likely to persist, and no method can reasonably account for individual differences in knowledge and experience that influence perceptions of difficulty. In general, difficulty pulls us toward normative information, whereas criterion referencing pulls us toward test content. With greater mastery, such as at postgraduate level, difficulty can have more of a bearing on distinguishing ability, as specialisation becomes associated with having the expertise to deal with issues others find difficult; but even here, ‘difficulty’ is a relative (normative) term defined by the experience of one or another group rather than an independent, external criteria.

4. Types of Validity in Assessment

Broadly, standards can relate to content, construct, criterion, or face validity. Content validity ensures that the assessment comprehensively covers the key components of the construct that it is intended to measure (Crocker & Algina, 1986). In the context of standard setting, it is paramount that the content reflects the core competencies that are necessary for performance. Difficulty should not overshadow the necessity of covering the full range of essential knowledge and skills, as missing key content areas would undermine the assessment’s validity (Ebel, 1972; Mehrens & Lehmann, 1978).

Construct validity assesses whether the assessment truly measures the trait or skill that it aims to evaluate (Messick, 1995). When determining the standard, it is crucial to focus on whether the assessment reflects the central construct, regardless of how difficult or easy individual items might be. The core competency being assessed should take precedence over the perceived difficulty of the test. Criterion-related validity gauges how well assessment scores predict real-world outcomes or behaviours (Anastasi, 1988). When setting standards, it is essential to align the assessment with relevant real-world criteria, ensuring that candidates who meet the standard are likely to succeed in practice (Downing, 2003b). While difficulty can affect predictive accuracy, the primary concern must be the alignment of the assessment with the intended outcomes.

Face validity refers to how relevant and appropriate the assessment appears to the candidates and other stakeholders (Murphy & Davidshofer, 1988). While difficulty can influence perceptions of fairness or appropriateness, face validity is ultimately determined by whether the assessment measures what it is intended to measure (Downing, 2003b). Thus, for face validity, it is the perceived importance of the content being assessed that should dominate rather than how difficult individual items appear to be.

5. Candidates Have Varied Perceptions of Difficulty

The concept of fairness has many forms (Verma & Rubin, 2018), and the perception of fairness in assessment often depends on candidates experiencing questions of equal difficulty (Denny et al., 2019). The perception of fairness has been found to depend on whether an assessment involves authentic tasks, reasonable demands, realistic contexts, and the need to develop a range of skills, and whether it is perceived to have long-term benefits (Sambell et al., 1997); notably, a role for difficulty in the perception of fairness is not always evident. Candidates do not complain about differences between questions when presented with different questions to each other (Emeka, 2019); rather, complaints are linked to performance and, with random allocation of questions, that their performance is dependent on luck (Graham, 1991). If one candidate in a cohort is perceived to receive easier or harder questions than another candidate in the same cohort, then this is considered unfair, except, arguably, where test equating methods are used to deal with these differences (Bandaranayake, 2008). However, we are considering item-level, not test-level difficulty; and if the difficulty of the tests is ‘equated’, then the issue of difficulty being a candidate property rather than an item property is compounded. It is necessary to consider whether the tests are equated by seeing how candidates scored on a subset of items and whether those candidates have identical experiences to each other. If not, then the items will have different apparent difficulties to different candidates, and identifying questions to use to equate could be perceived to be unfair. In contrast, importance should be constant across candidates.

In many settings, candidates find item difficulties to be rank-ordered in a fashion that is highly correlated across candidates (Cattell, 1963; Cattell & Horn, 1978; Keith & Reynolds, 2010). Clearly, item difficulty similarities across candidates tend to show a high correlation, based on intelligence (Cattell, 1987; Carroll, 1993). This is to be expected where candidates with different levels of ability in a given domain have followed the same syllabus taught via the same curriculum and are assessed using tests on which domain-general intelligence facilitates performance—higher intelligence leading to higher performance and a sense of greater ease of items. However, while most candidates who are selectively enrolled in a programme that is subject to demanding professional body requirements may be of similar intelligence, they will have many different experiences, in placement opportunities and self-directed learning, and so their perceptions of difficulty can be expected to differ as a function of this variation in experience rather than intelligence.

The different difficulty of the questions experienced by two different candidates depends on their different understanding of the content, which in turn is determined by their capability, application, any mitigating circumstances, and any demographic biases. Student understanding of a topic will be affected by the quality of their learning experiences. Learning experiences will be focussed on what an individual teaching or learning something considers important (Koriat, 1997). Role modelling will reflect what students then consider important, and frequency of coming across the same knowledge in different learning experiences will reinforce the importance of that knowledge and make a test-wise student more focussed on retaining and being able to apply it.

Question difficulty is not exclusively a property of the question, but rather, a function of the question and the candidate’s experience (R. C. Thomas et al., 2016). It depends on when different candidates cover the content, how often they encounter it, and whether they recall and understand it. The learning experiences provided by a programme are only ever likely to be found to be successful and satisfying by some students (Glass, 1978). Question difficulty depends on their educational experiences and work–life background, attendance, engagement with timetabled activities, use of resources, and self-directed learning. Difficulty is a subjective perception of challenge and unique to the individual, and using taxonomy based multiple-choice questions in first-year medical assessments has demonstrated this point (Blumberg et al., 1982). If some individuals find a question easier (or harder), that does not make that question any more or less important for candidates to be able to answer correctly. Unlike difficulty, importance is a property of the question content, assessment purpose, and alignment to the wider curriculum. These are independent of candidate properties.

6. Assessor Judgements of Difficulty Can Vary

Following blueprinting to ensure that the content is important and aligned to requirements, the determination of a threshold that is required to pass an assessment is predominantly determined by how difficult the questions are, not how important the content is. The use of Angoff is endemic in high-stakes assessments and often accepted as the ‘gold standard’, especially to set standards where professional, statutory and regulatory bodies are concerned (Burr et al., 2017). Assessors may feel uncomfortable with the conceptualisation of what would be found difficult by a ‘minimally competent’ candidate (Burr et al., 2021). It has long been known that judges are better at determining relative difficulty than absolute difficulty, where there is regression to the central tendency (Tinkelman, 1947; Lorge & Kruglov, 1953), with judgements tending to underestimate difficulty, combined with a shrinkage of variance (Lorge & Diamond, 1954).

It has been repeatedly shown that neither assessors nor candidates tend to be able to accurately estimate difficulty (Impara & Plake, 1998; van de Watering & van der Rijt, 2006). Furthermore, typical practices across the world for a wide range of different types of programme result in pass-marks that are usually somewhere between 40 and 60% (Imrie, 1983; Downs, 2009; Petr & McArdle, 2012). Unconscious bias in standard setting is likely to drive the pass-mark towards whatever is accepted as the local social norm (Bloxham & Boyd, 2007). Of course, as more assessors and questions are included to increase reproducibility (Schuwirth & Ash, 2013), the outcome also regresses towards the mean (Squire, 1978), and so, regardless of the content, a candidate who gets any half of the questions incorrect will typically pass and be considered fit to progress.

Judgements of minimal competence can be harrowingly variable. Comparison of passing standards derived by the Ebel method have been shown to differ considerably from those derived by the Nedelsky method (Nedelsky, 1954; Andrew & Hecht, 1976). This implies that including specific consideration of importance alters who is considered to have performed satisfactorily, bringing into question the conceptualisation of minimal competence (Glass, 1978). Another flaw in the logic of Angoff is that whether a candidate passes or fails can depend on the questions in the test rather than their ability (Glass, 1978). This can be improved by personalising the questions to the ability of each candidate (Burr et al., 2023).

Relative changes in achievement (i.e., improvement or decline in performance) have been proposed as preferable to absolute criteria, although it is recognised that how much change requires definition as a criterion in itself (Glass, 1978). Thus, value judgments based on comparative evidence can be considered more cogent and fair (Glass, 1978). To some extent, progress testing addresses this by using repeated testing and monitoring growth in knowledge towards the final target (Neeley et al., 2016). However, a final target still requires definition, with recognition that candidates starting at a high level will not need their knowledge to grow by as much as others.

7. The ‘Just Good Enough’ Candidate: Defining Competence Based on What Is Required

The abilities that any newly qualified professional or practitioner must have are critical to determine when beginning a standard setting process. This has been referred to variously as ‘just good enough’ (Burr et al., 2017), ‘just qualified’ (Plake et al., 2000), and ‘just competent’ (Ben-David, 2000). However, where conceptualisation has slipped from ‘minimally competent’ to ‘borderline’ it has been recognised to represent a shift from criterion to norm referencing (Hurtz & Auerbach, 2003; Chapman, 2014).

If different individuals are struggling with a subject, they may struggle with different aspects because of their different educational histories, quality of teaching, recent experiences, aptitude and effort. In short, it is self-evident that individuals have different strengths and weaknesses. For example, higher-performing candidates have fewer weaknesses, but lower-performing candidates could have poorer learning strategies and not have good breadth of knowledge but could have strengths in areas that are deemed less important or not have enough breadth in their areas of strength. In qualifying assessments, all questions that are set will be set to a similar standard. For a normal distribution of scores for questions answered in an assessment, the bulk of candidates will answer most questions correctly. For most questions, only a subset of candidates will answer them incorrectly. By focussing on certain topic areas in revision, students are thinking about what pieces of knowledge are important to them, whilst also favouring areas of a curriculum that they find easier to understand, as well as those areas that they think are more likely to be tested. This leads to questions about the rationale of basing a standard on the perceived difficulty experienced by candidates who are minimally competent, and methods of equating tests of different ‘difficulty’ (Bandaranayake, 2008). How can assessors reasonably conceptualise and judge how all struggling, ‘just good enough’, candidates would perform when asked a particular question? How do assessors know that this sub-group of candidates are uniform in ability, or representative of other candidates to decide who should clearly pass or fail? How could such judgements be empirically corroborated? It is difficult to see how any of these questions can be adequately answered. Borderline ‘just good enough’ candidates are defined by the process and so cannot be confirmed by their performance relative to others (Burr et al., 2017).

8. ‘Would’ and ‘Should’ Need to Be Used Appropriately

It is important to consider whether the difference between what candidates would get and what they should get really matters. With most current methods, if most candidates answer a question incorrectly (either using facility values from previous cohorts, or using norm referencing), or if standard setters think that most candidates would get a question incorrect (using Angoff), then the standard that is required to pass is lowered. Using an extreme but not impossible example to clarify this point, if an assessment included questions such as ‘how many lungs does the average human have?’, or more realistically, ‘What would be the most appropriate management after diagnosing a cerebellar stroke?’, and the majority of candidates were to get such questions incorrect, this would lead to lower standards being derived. The assessment would be considered ‘more difficult’, and therefore, the score required by candidates to pass would be lowered. This results in candidates needing to demonstrate less knowledge in order to pass, when in reality, such questions may be critical for safe practice. This is compounded by compensation, where poorer performance in one area removes some of the jeopardy of assessment for the candidate, but by favouring expertise in some areas while permitting weaknesses in other areas. Clearly any conflation of ‘should’ with ‘would’ is problematic for any professional discipline, from an engineer failing to calculate that a bridge will undergo structural failure when bearing an expected load, to a lawyer failing to identify mitigating circumstances that would be pivotal in reducing their client’s sentence. Clearly, it can be important for candidates to answer difficult questions correctly.

Ensuring the Validity of Standard Setting: Focus on Importance

For some considerable time now, the quality of assessments and the validity of the decisions based on them has depended on item response theory (Lord & Novick, 1968; Downing, 2003a; M. L. Thomas, 2019), decision accuracy frameworks (Livingston & Lewis, 1995), and contemporary validity argument structures (Kane, 1994; DeLuca, 2011). Emphasising importance over difficulty still utilises this framework. Item-level performance data, establishing optimal cut points for classification, and justifying the process to stakeholders all remain relevant.

For validity, the setting of standards needs to support the intended outcomes of a programme. This involves discussing the kinds of knowledge, skills, and behaviours that any beginning practitioner must have, as a critical conversation, when beginning a standard setting process. Using difficulty can detract from those goals. Assessment content should be aligned to the curriculum and its purpose. Finding low-importance items is a good quality assurance aspect of the method, and something which just rating a hypothetical candidate’s knowledge does not capture. Moreover, lots of low-importance items, which presumably candidates would not be expected to know, lead to a lower pass-mark and hence more scope to compensate for lacking important knowledge.

Questions should be set based on the knowledge required at a particular point in time in a candidate’s educational stage, for example the knowledge that is needed for a newly qualified professional in a given field of expertise (healthcare, engineering, law, etc.) to meet a minimum competency. A minimum competence test should have a high pass-mark, with the required score accounting for the extent of repeat opportunities to achieve any given outcome within a test, in essence ensuring that all successful candidates demonstrate an acceptable level of broad knowledge. However, it may still be theoretically possible for a candidate to avoid demonstrating a required outcome through compensation. An example might be a doctor graduating who has never answered any question about heart failure correctly and has avoided learning or successfully demonstrating any knowledge of heart failure.

If judging what a candidate should answer correctly, as opposed to what they are expected to answer correctly or what they actually answer correctly, the conceptualisation shifts away from difficulty towards importance. Deciding what proportion of candidates should get something correct typically becomes a dichotomous judgement, especially if there is a clear set of learning outcomes which define what needs to be known. A candidate either needs to know it or they do not need to know it, although categories of importance are also possible. If the purpose of assessment is to determine whether a candidate can perform to a criterion standard, then pass-marks determined by difficulty are exposed as flawed, because the use of difficulty in standard setting prioritises perception of fairness over achieving the criteria. It follows that candidates should be required to achieve the prescribed learning outcomes, and not a pass-mark. If the purpose is to ensure successful demonstration of performance in all content areas, then the pass-mark becomes 100%. Achieving this can be facilitated through the use of sequential assessments, where unachieved content is reassessed. Thus, the final target of demonstrating all required content can be achieved progressively (e.g., using content adaptive testing) or through remediation and repetition.

Difficulty can be considered a proxy for differentiation to assess potential to be a high achiever or to rank candidates. However, when producing an assessment which contributes towards a professional qualification, the importance and breadth of knowledge is more relevant to standard setting than difficulty. There should be more concern for candidates to demonstrate that they can perform across a range of knowledge areas and skills to make them safe and competent practitioners rather than ensuring an assessment can differentiate between the attainment of candidates in order to rank them. Thus, it should not be decided who is safe to work based on how difficult a candidate finds a task; instead, it should be decided who is safe to work based on whether a candidate can actually do what is important. Surely what matters is not what a sub-group would be able to do, but rather what all should be able to do.

9. Conclusions

The determination of a passing standard must prioritise validity over difficulty. While difficulty can be a factor in differentiating candidate abilities, it should not overshadow the core purpose of the assessment: measuring the competencies that are essential for success. The standard should reflect what is important, regardless of the perceived difficulty of individual items by either expert assessors or candidates. It should not matter how difficult something is if a candidate needs to be able to do it. If something is important, then they can either do it or not; doing it well becomes a professionalism judgement beyond a minimum competence. Setting a standard based on difficulty can be considered to be a dangerous dogma not fit for making high-stakes decisions about the adequacy of performance. By focusing on the importance of the constructs being measured, assessments can produce valid, reliable, and equitable outcomes. The validity of a test—ensuring it accurately measures what it is supposed to—is the primary concern, and test difficulty should only serve the purpose of maintaining or enhancing validity rather than being an end in itself.