Uniting Psychometric Modelling and Poisson Distributions: A Metrological Study of Elementary Counting

Abstract

A study of elementary counting (of simple clouds of dots by the Munduruku indigenous people of Brazil) is reanalysed in order to compare and contrast three kinds of probability mass functions (PMFs): (i) quantitative response to a discrete range of counts; (ii) the classic Poisson distribution of miscounts; and (iii) psychometric (Rasch) distributions of counting task difficulty and person counting ability. This reanalysis highlights how best to handle PMFs which provide a means of defining—for discrete and qualitative data—the basic metrics, viz. location and dispersion, of metrology—quality-assured measurement, as increasingly required since the turn of the millennium in topical and challenging quality-assurance applications, amongst others, in the human sciences and in Artificial Intelligence. PMF-based metrics, useful in ‘clinical’ and other applications where meaning and value are sought, complement the traditionally dominating role played by the corresponding probability density functions (PDF) in ‘analytical’, quantitative and continuous Metrology in Physics. New insights are provided when benchmarking the Rasch Poisson Counts Model, which has received less attention in modern metrology, against full psychometric Rasch modelling.

1. Introduction

Probability mass functions (PMFs) (Section 2.1) provide a means of defining—for discrete and qualitative data—the basic metrics, viz. location (Equation (5)) and dispersion (Equation (6)), of metrology when assuring the quality of measurement in terms of traceability and uncertainty. Quality assurance of those metrics, following satisfaction of constraints concerning statistical sufficiency and additivity, in turn supports, respectively, interoperability and defining limits on the quality of products and services of all kinds [1,2,3]. Deploying PMFs for discrete and qualitative data thereby complements the dominating role for these location and dispersion metrics traditionally played by PDFs, the corresponding probability density functions, in quantitative and continuous Metrology in Physics.

Location and dispersion metrics for discrete and qualitative data have been increasingly required since the turn of the millennium in topical and challenging quality-assurance applications, amongst others, in biology and medicine [4], the human sciences [5], sustainability [6] and Artificial Intelligence [7].

The metrology of PMFs remains so far relatively undeveloped, and the present work presents a reanalysis of a study of elementary counting (of simple clouds of dots by the Munduruku indigenous people of Brazil [8,9]) as a means of comparing and contrasting the metrology of three kinds of probability mass functions (PMFs) for the four scenarios shown in

Table 1. Discrete and continuous ranges versus qualitative and quantitative scales.

	Discrete	Continuous
Qualitative	Instrument response (Section 5.4): $P_{\mathrm{success}}$ per category/class	Clinical performance (bullet 2): Instrument ability, $\theta$, u($\theta$) Task difficulty, $\delta$, u($\delta$)
Quantitative	Analytical counting (Section 3.3) How many dots in object? Counting errors Limit of detection	Analytical measure (bullet 1): How much of a quantity in object? Measurement errors and uncertainties Trueness & precision

: (i) quantitative response to a discrete range of counts (Section 3.3) [10]; (ii) the classic Poisson distribution of miscounts (Section 4.3.3); and (iii) psychometric (Rasch) distributions of counting task difficulty and person counting ability (Section 5) for one of the conceptually simplest cases of measurement. The analyses illustrate the typical challenges normally faced when determining quality-assured metrics with the different kinds of PMF.

The concluding Discussion (Section 6) provides the motives for choosing an elementary construct. Classification, such as counting dots, despite sometimes having relatively large uncertainties and being considered ‘trivial’ or not even ‘metrology’ [11,12], is one of the conceptually simplest metrological cases with a well-defined measurand, as will be illustrated. In a summary of this and other limitations in the present study, perspectives will be given on future work, including consideration of other measurement systems where a Human as a Measurement Instrument is replaced by, for instance, an AI agent and multidimensional measurement theories. This includes proper accounting for ‘anti-racist’, ‘culturally responsive’ and ‘culturally specific’ measurement applications being developed, for example, by Mallinson [13] and Sul [14,15,16].

While in the 20th century metrological studies were dominated by cases in the bottom-right entry in Table 1, that is, of ‘analytical’ measurement of physical properties in trade and industry [17], the first quarter century of the millennium has seen an emergence [3] of metrology in all the other entries in that table, discrete and (for want of a better term) ‘clinical’ measurements (Section 3.2), as will be exemplified in this work for conceptually elementary counting. This reflects a growing trend towards more accountability in terms of meaning, value and effectiveness [18], and not mere numbers and efficiency, which might be met with a new domain of Metrology for Classification.

2. PMFs and Measurement

Examples of PMFs abound in the human sciences, education, sustainability and the health sciences (in surveys, sensory panels and educational examinations) and, by extension, increasingly in the area of Artificial Intelligence as well as other applications outside of traditional physical metrology [3]. Models used when quality-assuring PMFs, particularly those based on qualitative observations, such as item response theory and log-odds Equation (17) in the educational, medical and social sciences, are not exclusive to human agents but apply in many cases equally well (Section 4.3.1) to more technical and physical measurement systems, where they have so far not been regarded as belonging to mainstream metrology.

2.1. Probability Mass Functions

The discrete counterparts of PDFs—probability mass functions (PMFs)—are distributions of probability density (ordinate) over one or more discrete ranges of a variable (abscissa).

Let X be a discrete random variable with a range of C categories:

$$R_X=x_1,x_2,\dots ,x_C.$$

(1)

An event $A=X=x_c$ is defined as the set of outcomes, s, in the sample space, S, for which the corresponding value of X is equal to $x_c$ for a response in category c. In particular, the event

$$A = s\in S|X\left(s\right)=x_c.$$

(2)

The probabilities of events $X=x_c$ are formally shown by the probability mass function (PMF) of X:

$$P_X\left(x\right)=\left\{\begin{array}{cc}P(X=x_c)& ifx\in R_X\\ 0& ifx\notin R_X.\end{array}\right\}$$

(3)

for the range $R_X=x_1,x_2,\dots$ (finite or countably infinite).

The probabilities for the different categories, c, $P(X=x_c)=N_c/N_S$, that is, the occupancy, $N_c$, of a bin for category c as a fraction (or corresponding probability) of the sample size, $N_S$, must be non-negative and for PMFs are normalised to sum up to 1, respectively:

$$P_X\left(x\right)\ge 0$$

and

$$\sum _x{P}_X\left(x\right)=1.$$

(4)

In the examples referred to in this paper, mention will be made of the following:

• Probabilities
  The probabilities can be either derived from actual frequencies (occupancy per bin) or estimated in terms of prior knowledge or even judgment when scores are set. Examples of PMFs are given: for (i) an analytical case, (Section 3.3); (ii) for a binomial distribution (Section 4.1); (iii) for a Poisson distribution (Section 4.2); and (iv) for psychometric quantities (Section 5.4). When setting response scores as probabilities P, choices have to be made about how to define the scale-end scores of 0 and 100. Inferentially stable measurement also requires that the score varies monotonically across the scale in a manner allowing summarisation, which exhausts the data of all available information in a minimally sufficient statistic [19,20,21]. (These requirements [22] are subsequently checked (Section 5.5).)
• Ranges

  –

  Intrinsically discrete ranges, $R_X$ (Equation (1)) (such as when counting dots (Section 3.3)).

  –

  Discrete ranges chosen for convenience (such as in sensory panel responses), where the observed variable is on a continuous scale but the large uncertainties make it practical to round off the score, say, to the nearest integer.

  –

  Ranges can be as short as two, such as for the binomial distribution from binary Bernoulli trials (Section 4.1), as well as long, polytomous ranges spanning several categories (Section 4.3.2).

• Scales
  In different application areas, the scales associated with both X (abscissa axis) and P (ordinate axis) of a PMF can be

  –

  Fully quantitative (Section 3.3);

  –

  More qualitative (Section 5).

  These scales can range, respectively, from ratio and interval scales to ordinal and nominal scales.

There is no obvious one-to-one correlation between discrete and continuous ranges versus qualitative and quantitative scales [23]. Table 1 shows a matrix of possible correlations.

2.2. Accuracy of Classification

The accuracy of classification, in terms of both precision and trueness ISO 5725 [24], involves, respectively, differences between observed and ‘correct’ in terms of the location, Equation (5), and the dispersion of PMFs, Equation (6) (Equation (2.8) in [2]) (Section 4.1.1): where (provided distances between different categories are known, Section 4.1.2)

$$\mu ={\widehat{p}}_x={\displaystyle \sum _x}x·p_x;$$

(5)

where $p_x=P(X=x_c)$.

$$\sigma \left({\widehat{p}}_x\right)={\displaystyle \frac{1}{C-1}}\sqrt{\sum _x\left({(x·p_x-{\widehat{p}}_x)}^2\right)}.$$

(6)

where (provided distances between different categories are known (Section 4.1.2)).

Here, trueness is a measure of closeness of the mean to the ‘true’ value, while precision is the dispersion of repeated responses (without reference to a ‘true’ value).

The concept of accuracy can be considered for both analytic and clinical performance metrics (Section 3.2) and for the various cells of Table 1:

• Analytical accuracy.
  In the ‘analytical’ scenario 1, the ‘accuracy’ estimation of the ‘measurand’, the quantitative number of each set of discrete objects (with the number of dots increasing from 1 to 10 in the present counting case), can be expressed in terms of

  –

  (i) Trueness—estimated as the analytical difference between the measured $\mu$ and true dot count, ${\mu }_0$;

  –

  (ii) Precision—estimated in terms of analytical dispersion, such as $\sigma$ dots (Section 3.3).

• Clinical accuracy.
  In the ‘clinical’ scenario 2, a definition of ‘error’ in classification—such as that needed when estimating accuracy (Section 2.2)—is the difference ‘distance’ between the ‘response categorisation’ and ‘input (true) categorisation’, as given in ([2] Equation (2.8)). The challenge with clinical data is that the distances between different categories of classification on the abscissa of a PMF (such as when measuring the difference before or after an intervention or when estimating dispersion measures) may not be fully known or even meaningful (such as on nominal scales (Section 4.1)). Such effects can arise when measurement quality is limited, as dealt with in Measurement System Analysis, described further in (Section 3.1). Appropriate methods for dealing with such scales include log-odds ratio transformations, including GLMM and the Rasch psychometric theory with which measurands, such as counter ability and task difficulty, can be identified, as dealt with later in the paper in our clinical counting example (Section 5).

Alongside concepts about PMFs found in the wider literature, such as so-called ‘measure theoretic formulations’, a proper metrological approach describes how PMFs should be associated with various stages in the measurement process (Section 3.1). A major aim of this paper is to exemplify all the steps when establishing a measurement result—(i) data collection (e.g., counting responses); (ii) construction of PMFs from observed frequencies; (iii) selection of modelling approach (analytical (Section 3.3), Poisson (Section 4.2), or Rasch (Section 4.3)); (iv) parameter estimation (Section 4.3.3; and (v) interpretation of results—for one of the most conceptually simple cases of elementary counting (Section 5).

3. Analytical and Clinical Performance Metrics

3.1. PMF and Measurement System Analysis (MSA)

PMFs are a common tool of statistics, but here there are also metrological aspects. Uncertainty in counting, for instance, arises both from the identification of what is being counted and from the experimental counting procedure [12,25].

A proper metrological approach therefore describes how PMFs should be associated at various stages in the measurement process, corresponding to the propagation stage of establishing a measurement result. This means consideration of how measurement information is ‘transmitted’ (as in any communication system) through a measurement system [object–instrument–observer–environment and method (Figure 14 in [26]) and cf. [2] p. 198)] with a particular focus on an object as ‘probed’ with an instrument of some kind (Figure 1). This MSA analysis will guide the formulation and propagation stages for PMFs in establishing a measurement result from initial observations through to restitution of the measurand(s) [27]. For the present case of measurement systems involving discrete PMFs, inputs, such as $X_1$, when restituting the measurands, can be characterised in terms of informational entropy (Figure 5.4 in [2]), assigned on the basis of available knowledge (Section 5.3).

3.2. Analytical and Clinical Performance Criteria

Different attributes associated with various elements (MSA object–instrument–observer–environment and method, (Figure 1) of the measurement system (Section 3.1) can be analysed in terms of PMFs. For each MSA element attribute, two principal kinds of performance criteria—‘analytical’ and ‘clinical’ as in the various cells of Table 1—can be identified, which Hofmann [28] places as essential intermediaries between, on the one hand, epistemic (state-of-knowledge) relevance and, on the other, meaningfulness (ultimately for patient health). This can be topically exemplified in the context of regulation and conformity assessment of in vitro devices, specified in EU regulations [29,30], although equally applicable to our elementary case of counting dots (Figure 2):

• ‘Analytical’ performance criteria for determining, e.g., how much (quality characteristic: concentration) of a particular analyte (MSA object) is present in a sampled object (by ‘variable’), such as, first, analytical method accuracy (trueness and precision [24]) (Section 2.2) and, second, sensitivity, such as instrument limit of detection (MSA measurement instrument), as exemplified in the analytical interpretation of the elementary counting case of the present study (Section 3.3). According to [29], “…Analytical performance focuses on the gathering of evidence that the measurement instrument in question reliably, accurately and consistently measures and or detects an analyte”. This is closely related to terminology in acceptance sampling standards [31] (Section 3.1), where inspection by variables is inspection by measuring the magnitude(s) of a characteristic(s) of an item.
• ‘Clinical’ performance, according to [29], “aims to demonstrate that the measurement instrument can achieve clinically relevant outputs through predictable and reliable use by the intended users”. This is closely related to terminology in the definition in Section 3.1.3 of [32], where inspection by attribute is “inspection whereby either the item is classified simply as conforming or nonconforming with respect to a specified requirement or set of specified requirements, or the number of nonconformities in the item is counted”. Commonly used clinical performance metrics of the measurement systems include: selectivity (Equation (23)) and sensitivity (Equation (22), not to be confused with analytical instrument sensitivity, bullet 1), which are plotted against each other on receiver operating characteristic curves [33], when sampling by ‘attribute’. A psychometric treatment of these clinical performance metrics [34,35] yields quantitative estimates for quality characteristics, such as task difficulty (MSA object) and agent ability (MSA measurement instrument), as attributes of the different elements of the measurement system illustrated in Figure 1, corresponding to the top-right entry in Table 1.

3.3. PMFs for Counting and Related Tasks: ‘Analytical’ and ‘Sampling by Variable’

The elementary counting of dots by less numerate agents (e.g., children or indigenous people [8,9], Figure 2) presents a case of the metrology of discrete counting as a typical ‘analytical’ or ‘sampling by variable’ procedure [11,12] (bullet 1 and bottom-left entry in Table 1).

This study of elementary counting has several advantages, including the opportunity of analysing the measurement response where the analytical measurand (quantity intended to be measured)—the number of dots ($1\dots 10$)—is

• (i) known exactly;
• (ii) Conceptually simple.

In fact, this elementary case can exemplify each of the elements of Table 1, as will be shown throughout this paper.

As the number of dots to be counted exceeds about 5 (the number of digits on one hand traditionally used by the agents for counting), each estimated count becomes successively in error [8] (Figure 2). Similar observations of limitations in counting ability can be found in children and educated adults when they are given only a limited time to perform each counting task. The initial ‘analytical’ scenario 1 is the estimation of the measurand: the quantitative number of a set of discrete objects (with the number of dots increasing from 1 to 10) in the data shown in Figure 2. A count of 5 dots, where perceived counts deviate substantially from the known, true number of dots, could be considered a ‘limit of detection’ as an analytical performance metric.

These successive count errors can be described in terms of PMFs distributed over an increasing number of counting categories, as shown in Figure 3. Analytical PMFs of distributions of perceived counting errors can be plotted, for example, for three counts: (1 dot, 6 dots and 8 dots) in the study [8] of elementary counting tasks (‘items’, j) of sets of clouds of increasing numbers of dots and task difficulty. The lines drawn for each count PMF assume a Gaussian distribution of counting error, $p_x=N((\mu -{\mu }_0),{\sigma }^2)$, $\mu =$ perceived count, Equation (5); $\sigma =1$ dot, perceived counting dispersion, Equation (6); ${\mu }_0=$ true count.

4. Case Studies of PMFs: Quantitative Statistical Process Control

PMFs have been invoked for many decades for a number of well-known sampling distributions in the context of trials for statistical process control (SPC [36]) as one major area of application. In general, the parameters of a production process are unknown and can change with time. Procedures to estimate the parameters of probability distributions and solve other inference or decision-oriented problems related to them need to be developed. In this section, we will motivate how SPC PMFs can be deployed when tackling the metrology of clinical performance (Section 2), with a particular focus on the elementary counting example, Figure 2.

4.1. Binomial Distribution and Dichotomous Bernoulli Trials in SPC

A basic kind of PMF is the dichotomous Bernoulli variant shown in Figure 4. For an elementary counting case, a counter may initially estimate the number of dots as one or another of two adjacent integers, say, ‘9’ or ‘10’ when counting a cloud of ten dots, corresponding to the top-left entry in Table 1.

For a dichotomous [38] ‘trial’, where a ‘success’ is scored 1 and a ‘failure’ 0, let the probability of categorising a ‘successful’ response be p. Values on the ordinate (vertical) axis of the binomial PMF shown in Figure 4 are probability density values:

$$P_X\left(x\right)=\left\{\begin{array}{cc}P(X=x_c)={\displaystyle \frac{coun{t}_c}{N_{sample}}}={\displaystyle \frac{{\sum }_{i=1}^{N_{sample}}[z_i=c]}{N_{sample}}}& ifx\in R_X\\ 0& ifx\notin R_X.\end{array}\right\}$$

(7)

On a quantitative scale on the abscissa, the PMF plotted in Figure 4 (based on Monte Carlo (MC) simulations of sampling trials [37]) has the mean and standard deviation $\widehat{p}=0.5,\sigma \left(\widehat{p}\right)=0.28$, while Equations (5) and (6) yield

$$\widehat{p}={\displaystyle \sum _{c=0}^{c=1}}c·p_c=0.5$$

(8)

$$\sigma \left(\widehat{p}\right)={\displaystyle \frac{\sqrt{{\sum }_{c=0}^{c=1}{(c·p_c-\widehat{p})}^2}}{C-1}}=0.3.$$

(9)

The binomial distribution is a limiting form of the Poisson distribution (Section 4.2) for an infinite number of Bernoulli trials and an infinitely small probability of non-conforming products [36].

4.1.1. Dichotomous Decision-Making with Uncertainty

A typical decision of conformity is whether the attribute of an entity (e.g., a product or sample) can be classified as above (‘positive’) or below (‘negative’) a specification limit, SL, for that quality characteristic. However, measurement uncertainty leads generally to the risk of incorrect decisions and classifications in such conformity assessments, as described in [17] JCGM 106:2012. As recalled in [4], in the context of laboratory accreditation with uncertainty, “Unlike ISO 17025, [39], ISO 15189 [40]…categorically requires the use of positive and negative samples (Point 7.3.4 letter f) for the uncertainty of qualitative results based on thresholds”.

An observed response in a performance test is typically scored binarily (binomial PMF, Figure 4) in one of two categories on the abscissa, as $c=1$ for correct and as $c=0$ for incorrect, and presented on the PMF ordinate as $P_{success,i,j}$ for each instrument, i, to a specific object, j, when examined in terms of Measurement System Analysis (Section 3.1). This simplest binary classification can be extended to polytomous decision-making over a range of classification categories (Section 4.2). In the elementary counting case, this would correspond to ‘guess-estimating’ the count to lie in a range of integer numbers of dots for a given cloud as an analytic measurand, as is evident in the results shown in Figure 3. A corresponding clinical performance study (Section 2) would identify two measurands: the (i) ability of the counter and the (ii) difficulty of each counting task derived from the probability of correct classification, corresponding, respectively, to the top-right and top-left entries in Table 1. In any case, the decision quandary reflects uncertainty. We share the views of Pradella [4], who writes: “‘doubt’ is not only a specialised concept in psychology [41], but a term of common language and everyday experience, attested in general vocabulary”, and which goes beyond the traditional ‘operational’ definition of ‘standard uncertainty’ as a standard deviation [42]. The next sections take this even further, by presenting psychometric modelling capable of dealing with the identification of performance measurands of ability and difficulty in the presence of uncertainty, culminating in the Rasch measurement model (Section 4.3) as a method of choice for the Metrology of Classification [43].

4.1.2. Distances on Categorical Scales: Counted Fractions

Distances on categorical scales, such as on a binomial PMF (Figure 4) (but also PMFs for other distributions, such as the Poisson and psychometric), can be challenging to evaluate correctly in the case of hidden, unrecognised non-linearity or simply a non-numerical scale (Section 2.1). This ‘scale ordinality’ obliges relatively more emphasis to be placed on the PMF ordinate values compared with more quantitative cases with PDFs such as the analytical (bullet 1). Typical clinical performance (bullet 2) cases include assessing the effects of an intervention or the relative performance of two counters (Figure 3) in terms of distances between PMFs.

As defined in Equation (4), the probabilities for the different categories of any PMF, c, $P(X=x_c)$, must be non-negative and sum up to 1: $P_X\left(x\right)\ge 0$ and ${\sum }_x{P}_X\left(x\right)=1$. A general feature of categorical data in which fractions on a finite scale are counted on any bounded scale is referred to as the ‘counted fraction’ effect [44]. Put simply, any score on a limited scale bounded by 0 and 100% becomes increasingly non-linear as either extreme of the scale is approached. A difference of, for example, 1% in a score $P_{success}$ of successful classification is a different amount at mid-scale (50%) compared with scores at 5% or 95%. This effect will be explained and compensated for as follows:

An early statement by Pearson [45] is given: “Beware of attempts to interpret correlations between ratios whose numerators and denominators contain common parts”.

The counted fraction effect has been explained in terms of the occurrence of the same term, $Z_j$, in the numerator and denominator of the score $Z_j=Z_j/\left({\sum }_{k=1}^K{Z}_k\right)$.

Tukey [44] wrote further: “As a general consequence we should expect that scales which have a finite range are likely to give us trouble unless all our observations tend to be safely away from any ends which are present. Hence the fact that percentages go only from one end (at 0%) to another (at 100%) suggests that, whenever even moderately extreme percentages are likely to occur, we are likely to have to ‘stretch the tails’, while, if really extreme percentages occur, we may have to stretch hard enough so that there are no ends (at any finite values).”

Filzmoser [46] stated for basically the same concept in the context of mineral compositional data studies: “If not all variables or components have been analysed, this constant sum property, (viz., ${\sum }_x{P}_X\left(x\right)=1$ (Section 2.1)) makes the relation between variables not ‘real’ but ‘forced’…For example, if the concentrations of chemical elements are measured in soil samples, and if an element like SiO₂ has a big proportion of, say, 70%, then automatically the sum of the remaining element concentrations is at most 30%. Increasing values of SiO₂ automatically lead to decreasing values of the other elements, and even if not all elements of the soil have been measured, the correlations will be mainly driven by the constant sum constraint.”

Non-linearity in the raw response scale, such as that which arises from the counted fraction effect [47], can be readily revealed by plotting, as in Figure 5 and Figure 6, the marginal probabilities for the different items j (cloud of dots to be counted, for example), given by ${\displaystyle \frac{{\Sigma }_{i=1}^{N_{TP}}{y}_{i,j}}{N_{TP}}}$, against the more quantitative scales from Rasch Measurement Theory (RMT) [48] for the task difficulties, ${\delta }_j$, (logits), as will be described below. As is clear from both figures and as expected from the counted-fraction effect, while substantially linear in the central region around 50 percent successful classification, there is increasing non-linearity towards the scale extremes (low scores in Figure 6 and for both low and high scores in Figure 5, exactly as said by Tukey [44]).

Scale non-linearity caused by the counted-fraction effect is ubiquitous to all PMFs because of the limited percentage scale (Equation (4)), but if not recognised (as in Classical Test Theory, CTT, [49] which is still widely used, even in the most recent literature [12]) it can lead to serious underestimation of true scores at either end of the response scale, as shown in Figure 5 and Figure 6, compared with the substantially linear region close to mid-scale (Equation (10) (Section 4.1.3)). Without compensation —”stretching the tails”, in the words of Tukey [44]—the counted-fraction effect will invalidate even the simplest mathematical and statistical expressions, such as those used when calculating an average or expressing a law of probability (Equation (1) in [12]).

4.1.3. Compensating for Counted-Fraction Scale Non-Linearity

There is a substantial and growing body of literature ([2], pp. 88–89) for a diversity of applications, such as psychometry [48] and compositional data analysis [46,50], where the counted fraction effect is recognised. The ‘counted fraction’ effect can be compensated for using a log-odds (GLMM) approach (Equation (11)):

Ordinate values compensating for the counted fractions effect are given as a straight line $y=m·z+c$, as plotted in Figure 5 and Figure 6, where two statistics are known:

• The intercept $c=0.5$ at $z=0$.
• The slope.

  $$m={\displaystyle \frac{dp}{dz}}=p\left(z\right)·(1-p\left(z\right))=0.25$$

  (10)

  at $z=0,p=0.5$.

These two parameters, which apply to all bounded-scale fractions, define the anchor (at $P_{success}=$ 50%) and the span (the slope) of the linear, quantitative scale resulting from a log-odds transformation Equation (11), thus giving the corrections to raw performance data for the counted-fraction effect for any PMF.

Ordinality in responses generally compensates for the effects of counted fractions by taking log-odds ratios (ORs) [51], i.e.,

$$log\left(OR\right)=log({\displaystyle \frac{x_j}{1-x_j}})=z.$$

(11)

The RHS of Equation (11) is referred to as a ‘link function’ in the literature in the Generalised Linear Measurement Model (GLMM) approach [52] and provides the means for common-person and common-item equating in psychometrics, following Wright’s expansions of Rasch’s models ([53] pp. 109–117). For current application, see [54].

4.2. Poisson Distributions in SPC

The Poisson is another widely applied distribution used to model count data and random occurrences over time or space, such as arrivals in a queue, radioactive decay events or rare defects in SPC production [55]. We will also examine in (Section 4.3.3) how the Poisson distribution can be useful when defining clinical performance metrics for the elementary counting case (Figure 2).

Let $\lambda$ denote the expected number of events per interval, and assume $\lambda$ is known. Then, the distribution of X is Poisson with parameter $\lambda$:

$$X\sim \mathrm{Poisson}\left(\lambda \right).$$

The PMF for X is

$$p_X\left(x\right)=\left\{\begin{array}{cc}\hfill {\displaystyle \frac{{\lambda }^x{e}^{-\lambda }}{x!},}& x=0,1,2,\dots ,\hfill \\ \hfill 0,& \mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

X is the quality characteristic, x, being classified (e.g., number of defects) in n observations.

X has expectation and variance ([36], p. 55)

$$E\left(X\right)=\lambda ,V\left(X\right)=\lambda .$$

(13)

The Poisson and binomial distributions are closely related: The Poisson distribution can be regarded as a limiting form of the binomial distribution (Section 4.1). Allowing the binomial parameters $n\to \infty$ and $p\to 0$ in such a way that $n·p=\lambda$ results in a Poisson distribution [36]. According to Meredith [56], the Poisson distribution “also occurs as a limit distribution for a number of discrete distributions other than the binomial. Two of the more important instances of this are the negative binomial or Pascal distribution and the hypergeometric distribution.”

As for the binomial (dichotomous Bernoulli trials) case (Section 4.1), values on the ordinate (vertical) axis of the Poisson PMF are probability density values for each x:

$$p_x=coun{t}_x/N_{sample}={\displaystyle \sum _{i=1}^{N_{sample}}}[z_i=x]/N_{sample}.$$

(14)

On a quantitative scale on the abscissa, each Poisson PMF—plotted in Figure 7 for the elementary counting case—has the mean and standard deviation given by

$${\widehat{p}}_x={\displaystyle \sum _{x=0}^{x=X}}x·p_x;$$

(15)

$$\sigma \left({\widehat{p}}_x\right)=1/(X-1)·\sqrt{\sum _{x=0}^{x=X}\left({(x·p_x-{\widehat{p}}_x)}^2\right)}.$$

(16)

4.3. Rasch Psychometric Model: Principle of Specific Objectivity

The output of the uncertainty model (Section 3.1) will ultimately provide (through ‘restitution’), from an MSA perspective (Figure 1), our best estimates of the instrument attribute (typically an ‘ability’ of a rater to make a ‘correct’ classification), alongside the object attribute, which together constitute a ‘conjugate’ pair of measurands. This procedure for clinical properties involving ordinal (and nominal) PMFs has the same role as the corresponding procedure for data on fully quantitative scales, when calibrating the sensitivity of, for instance, a weighing machine in order to metrologically determine the mass of an unknown weight. Providing separate estimates of measurement system attributes from the analysis of PMFs on the less quantitative scales will require dealing with possible scale non-linearity (Figure 6). Again, a psychometric model will be useful when defining clinical performance metrics for the elementary counting case (Figure 2) and corresponding to the top-right entry in Table 1.

Amongst a family of log-odds transformation models (through restitution from the ordinal or nominal response scores, $Y=P_{success}$, of the measurement system), a particularly metrological approach is the psychometric [48] modern measurement (dichotomous) theory [43,58], as a special case of the log-odds (Equation (11)):

$$y_{i,j}=P_{success,i,j}={\displaystyle \frac{e^{({\theta }_i-{\delta }_j)}}{1+e^{({\theta }_i-{\delta }_j)}}}$$

(17)

or in the form of the log-odds:

$$ln({\displaystyle \frac{P_{success,i,j}}{(1-P_{success,i,j})}})={\theta }_i-{\delta }_j$$

(18)

This not only compensates for counted fraction non-linearity (Section 4.1.2) but also allows separate estimates by logistic regression of Equation (18) to the response data (often using a maximum likelihood criterion) of the quantitative, continuous variables ${\delta }_j$—task difficulty of each set of objects, j—and ${\theta }_i$—agent ability of each of the set (‘cohort’) of instruments, with i as the principal clinical measurands, for instance, in our elementary case. (Note that this logistic regression is made on the whole matrix of responses of i instruments and j objects similar to, for instance, a least-squares subdivision, as opposed to bilateral, pairwise comparisons more commonly found in metrology (Section 4.1.1 in [2])).

The principle of specific objectivity is one major approach to providing separate estimates of these attributes as measurands [48]. Following the Principle of Specific Objectivity, MSA responses are not measures of the instrument’s (classifier’s or decision maker’s) ability, nor the classification task (object) difficulty, but they depend on both. Rasch [48] attempted to motivate his principle by drawing analogies to relations between several variables and the universal equations of Physics (such as Newton’s second law).

The MSA approach of engineering metrology is felt to be a closer point of departure [43] (Section 3.1). In accordance with Rasch’s principle of specific objectivity, task difficulty and agent ability need to be treated separately:

• ‘Agent’ ability, $\theta$.
• ‘Task’ difficulty, $\delta$.

In making separate estimates of the measurands, ability and task difficulty, following the Principle of Specific Objectivity, when reliability and validity are to be assessed, the normal rules of statistics apply, as well as more metrological requirements, which go beyond requiring mere numbers to making judicious and representative sampling across the full range of measurement scales (Section 5.5).

Alongside his better-known log-odds expression (Equation (17)), the Poisson PMF distribution (Figure 7) of SPC was chosen by Rasch [57] when originally proposing his psychometric approach (Section 4.3.3) to determine the ability of individual cohort members to perform psychometric tests, such as reading. Misreadings in such tests correspond with the typical application of the Poisson distribution (Section 4.2) in quality control as a model of the number of defects or non-conformities that occur in a ‘unit of product’ [36]. In Rasch’s psychometric models [48,57], the Poisson rate factor $\lambda =k/h$, where ‘task’ difficulty, $\delta =log\left(k\right)$, and ‘agent’ ability, $\theta =log\left(h\right)$ [59,60], are recognised as “a necessary condition for specific objectivity of comparisons between tests and between students”, with “the result that for the Poisson distribution the multiplicative structure (IX:2) of its parameter A is both necessary and sufficient for obtaining specific objectivity can be extended to the whole class of additive exponential models….” ([48], p. 92). How the Poisson distribution can be useful when defining clinical performance metrics for the elementary counting case (data shown in Figure 2), particularly the case in the top-right cell of Table 1, is further examined in (Section 4.3.3).

4.3.1. Agnostic Rasch Models

Most applications of Rasch’s measurement models [48,57] have to date been made in education, health care, and survey research, involving persons who act as instruments [43], such as in the elementary counting example (Section 5). But that model is not mathematically limited to psychometrics or human respondents but should also be applicable to more technical agents, as Rasch [61] recognised in his retirement lecture, which he closed by saying that the potential for use of the models he developed “stretches to all sciences where the subjects are comparisons that must be objective”.

Diverse examples include

• A material hardness indenter [62] to make an impression (JCGM VIM [63] EXAMPLE 1 Rockwell C hardness);
• A hospital to provide a service (A & E [64], surgery [65]);
• In psychophysics, that is, where various physical fields and forces impinge on the five senses of an agent [43,66].

These cases are amenable to an analysis in terms of the following measurands: (i) an ‘ability’ of an agent and the ‘difficulty’ of a task, or, alternatively, (ii) in usability studies [67], the ‘leniency’ of an agent and the ‘quality’ of a product or service, by exploiting the agnostic aspect of the Rasch measurement model (Section 4.3).

4.3.2. Polytomous Measurement Models

Rasch’s dichotomous measurement model version (Equation (17)) has subsequently been complemented by the corresponding polytomous measurement model variants as well as multilevel [68,69], multidimensional [70], multifaceted [71,72], mixture [73], and other models. As an example, JCGM VIM [63] EXAMPLE 4, Subjective level of abdominal pain on a scale from zero to five, belongs here.

Registering the response, $P_{success}$, of the measurement system when instrument resolution is limited or levels of uncertainty are high in the measurement process is often made more practically in terms of a Probability Mass Function (PMF) with discrete categories rather than as a continuous response scale (so-called ‘visual (analogue) score’) by rounding off to the nearest integer, even when the object quantity varies on a continuous scale.

Polytomous classification is the term used when more than two categories are assigned. In the elementary counting case, Figure 2, this would correspond to ‘guessing’ the count to lie in a range of integer numbers of dots for a given cloud, as a clinical performance study (Section 2). There is an extensive body of literature—for instance, in the educational assessment field [74]—where guessing can be modelled in much the same way as the MSA approach (Section 3.1); in classical measurement, engineering parametrises measurement errors such as bias.

Approaches to polytomous response analysis include the so-called ‘partial credit model’ (PCM [75,76]) in which the binary (dichotomous) log-odds ratio (OR) models are extended to polytomous cases, as summarised as follows:

Probability $q_{i,j,k}$ of response $x_{i,j}$ of instrument (agent) i, object (item) j, over k categories for the PCM:

$$q_{i,j,k}={\displaystyle \frac{exp{\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c})}{exp{\sum }_{k=0}^{C_j}·\left({\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c})\right)}}.$$

(19)

An alternative to the PCM is the Rasch Rating Model [77]:

$$Pr(Y_{i,j}=q)=q_{i,j,k}={\displaystyle \frac{exp{\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c}-{\tau }_c)}{exp{\sum }_{k=0}^{C_j}·\left({\sum }_{c=0}^k({\theta }_i-{\delta }_{j,c}-{\tau }_c)\right)}}$$

(20)

where ${\tau }_c$ is the threshold of category c.

This formalism applies even in the present case of elementary counting, where, for each task of counting a number of dots, an individual cohort member is making a polytomous decision: “how many dots do I see?” among the set of potential dot numbers (Section 4.3.3).

4.3.3. Poisson PMFs for Counting

At about the same time as Rasch [48] formulated his psychometric model (Equation (17)), he also considered [57] an alternative formulation referring to a probabilistic Poisson distribution (Section 4.2) well known from SPC as a model of the number of defects or nonconformities that occur in a unit of product when classifying it:

$$p\left(x\right)={\displaystyle \frac{e^{-\lambda }{\lambda }^x}{x!}};x=0,1,\dots ,n$$

(21)

where x is the ‘quality characteristic’ being classified (number of defects) in n observations.

The rate parameter $\lambda$ is equal directly both to the mean and variance of the Poisson distribution ([36] p. 55 and (Section 4.2)). In [57] a Rasch Poisson PMF was given as a model for the number of ‘misreadings’ by a child (i) for a text (j) of a particular level of difficulty. As quoted in [60], and references therein: “Although it is one of the earliest models that Rasch has developed, the Rasch Poisson Counts Model has received less attention than other binary or polytomous Rasch models”.

A much earlier observation by Meredith [56] noted: “a thorough understanding and application of such a particularly simple [Poisson] model is often the key to further development of suitable stochastic models in scientific work”.

The fundamental value obtained via the Poisson counts model was only belatedly realised by Rasch ([78] p. 66) when, after a 1959 conversation with a colleague, he suddenly realised “that the possibility of separating two sets of parameters must be a fundamental property of a very important class of models,” and that his study of children’s misreadings had led him to devise a new “class of probability models [having] the property in common with the Multiplicative Poisson Model, that one set of parameters can be eliminated by means of conditional probabilities while attention is concentrated on the other set, and vice versa.” Andrich [79] then showed that the Poisson distribution is necessary and sufficient for constructing measures from discrete observations.

In the present work, the example of the elementary counting of dots, akin to the Rasch [57] counting study, not only provides an ‘analytical, sampling by variable’ PMF (Section 3.3) but will also exemplify how Poisson PMFs can be benchmarked against full psychometric studies as ‘clinical, sampling by attribute’ PMF studies (Section 5).

In the case study of dot counting by the Munduruku (indigenous people of Brazil) [8,9], Poisson PMFs, as ‘clinical’ cases in the top-left cell of Table 1, can be derived with Equation (21) since task difficulty can be calculated from first principles (Equation (26)), from the number of dots being counted, thanks to the conceptual simplicity of the task [80] (p. 66ff).

Table 2. Empirical PMFs for three counting tasks: 1 dot, 6 dots and 8 dots.

x	1	2	3	4	5	6	7	8	9	10
$P(X=1)$	33.72	10.77	2.29	0.37	0.05	0.0	0.0	0.0	0.0	0.0
$P(X=6)$	3.09	7.89	13.43	17.15	17.53	14.92	10.89	6.95	3.95	2.02
$P(X=8)$	1.83	5.27	10.10	14.51	16.68	15.97	13.12	9.42	6.02	3.46

shows the PMF values from the Poisson rate, $\lambda$, calculated by entering the simulated counting difficulty $\delta$ into Equation (21), where $k_j=e^{{\delta }_j}$ and taking a cohort-average ability, ${\theta }_{mean}$, yields $h_i=e^{{\theta }_i}$.

Figure 7 shows the resulting plots of PMFs for three cases (true counts of dots: (1), (6) and (8)). These PMFs show the distribution of probabilities for each object count (no. of dots), with a shifting maximum—e.g., at $x=5$ misclassifications for eight dots—corresponding to the most likely value of x, the quality characteristic being classified (number of misclassifications) in n observations, where those miscounts are those shown in the analytical plot of Figure 3.

• In contrast to Rasch’s [48] psychometric model, it is important to note that Poisson [81] placed important restrictions on the applicability of his model: particularly that the mis-classification events were to be “very rare”, as should apply to the PMFs shown in Figure 7. Although there appears in the literature to be no exact threshold where the Poisson approximation breaks down, a ‘rule of thumb’ is as follows: $\lambda =n·p$ is moderate (typically $n·p\le 5-10$ for good accuracy) (Figures 2–24 in [36]). Our psychometric simulations (Section 5) of task difficulty and counter ability do not suffer from the same restrictions and allow the choice of response level, $P_{success}$, over a complete range (shown later in Figure 9): from the most mis-classifications (for a low-ability counter attempting a difficult counting task) to the least number of mis-classifications, where the Poisson approximation should be valid in the latter case (i.e., for a high-ability counter performing an easy counting task) [82]. The Poisson ‘rule of thumb’ range can be seen to be comparable with the mis-classification rates shown in Figure 7. However, to make a full comparison between the classic ‘rule of thumb’ limit to the Poisson distribution and the present psychometric simulations requires a proper account of the applicability of the ergodic principle: that is, to what extent the classic group-statistic Poisson approach [36,81] corresponds to the individual statistics of Rasch [57] psychometric modelling. For instance, the Poisson rate $\lambda =n·p$ has in some way to be interpreted in psychometric cases where $n=1$, i.e., just one counting agent (MSA: Instrument), while, at the same time, the overall number of degrees of freedom needs to be sufficiently large to achieve adequate reliability and validity (Section 5.5).
• The Poisson PMF number of mis-classifications shown in Figure 7 has no information about which numbers are perceived correctly or not, but should in principle correspond—for each item (j)—to the occupancies of the analytical PMFs shown in Figure 3. Similarly, the Poisson PMF number of mis-classifications has no obvious relation to the corresponding ‘clinical’ metrics, that is, the counting task difficulty and the counter ability. (In principle, one can estimate the expected count by inverting Equation (21) since—in the present case of an elementary construct—the relation between task difficulty and the number of dots is known (Equation (26))). Future work is intended on this novel approach to benchmarking of the Rasch Poisson Counts Model against full psychometric Rasch modelling, as mentioned in the final Discussion (Section 6).

5. ‘Clinical’ Psychometric Study: Counting Dots

The elementary counting of dots by less numerate agents (e.g., children or indigenous people [8,9]) (data shown in Figure 2) also presents a case typical of the metrology of counting as a ‘clinical’ or ‘sampling by attribute’ procedure (Section 2) corresponding to the top row of Table 1, as follows.

5.1. Dichotomous and Polytomous Mis-Classification Probabilities. CTT and Rasch Measurement Theory

Regularly used clinical performance metrics (Section 2) of measurement systems include: ‘sensitivity’ (Equation (22)) and ‘selectivity’ (Equation (23)), which are plotted against each other on receiver operating characteristic curves [33], when sampling by ‘attribute’, and are often analysed with CTT despite its known limitations (Section 4.1.2).

Dichotomous mis-classification probabilities [83,84] due to binary decision quandary arising from uncertainties are

• $FAP=P[Y=1|X=0];$ False Acceptance (or positive, FPR) Probability;

• $FRP=P[Y=0|X=1];$ False Rejection (or negative, FNR) Probability.

These two probabilities lie on the off-diagonal of a 2 × 2 decision matrix for a binary test, where the true positive and true negative classification probabilities lie on the diagonal. Each pair of probabilities—positive (TP, FP) and negative (FN, TN)—can be plotted as binary PMFs, like the binomial plots shown in Figure 4.

Performance metrics can be defined for the case of a binary decision A or B about an entity when assessed for conformity with respect to a specification limit, SL, for the quantity, z, of interest, based on a signal (SN with noise) in the presence of noise, N [33]:

True positive rate (TPR), a.k.a. sensitivity or recall:

$$P(A\mid SN)={\displaystyle \frac{{\sum }_{z>SL}{N}_{z,\mathrm{detected}}}{{\sum }_z{N}_{z,\mathrm{detected}}}}.$$

(22)

True negative rate (TNR), a.k.a. selectivity (or specificity in clinical/diagnostic contexts):

$$P(B\mid N)={\displaystyle \frac{{\sum }_{z<SL}{N}_{z,\mathrm{not}\mathrm{detected}}}{{\sum }_z{N}_{z,\mathrm{detected}}}}.$$

(23)

The elementary counting case, Section 5, corresponds to a polytomous extension (Section 4.3.2) of each of these classic clinical performance metrics.

Recall that “clinical performance aims to demonstrate that the measurement instrument can achieve clinically relevant outputs through predictable and reliable use by the intended users”, scenario 2 [29]. Considering the known (but not always recognised) limitations in CTT (Section 4.1.2), a more reliable approach to clinical performance assessment [35,85,86,87] is therefore to deploy Rasch Measurement Theory (Section 4.3), which:

• (i) Compensates for counted-fraction scale non-linearity (Section 4.1.3);
• (ii) Exchanges the analytical measurands (such as the number of dots, Section 3.3) for the clinical measurands: counting task difficulty and counter ability.

In the ‘clinical’ scenario 2, the level of difficulty attributed to each object counted and the ability attributed to each counter (the person or, in MSA terms 1, the ‘instrument’) both contribute in some way to the probability of successful counting. For a proper metrological analysis of the clinical analysis of counting, PMFs (occupancy: probability of responses in each category (number of dots)) for a range of tasks of increasing levels of difficulty can be analysed psychometrically, where the quantities (measurands) of interest are primarily the ability of each counter and the level of difficulty of a certain number of dots (rather than the actual number of dots—which of course can be determined by any capable counter).

5.2. Counted-Fraction Scale Non-Linearity When Counting Dots: ICC

Figure 8 exemplifies for the elementary counting case (Section 5) a so-called ICC (Item Characteristic Curve [88]) for a particular item, calculated for a selected counting task (difficulty ${\delta }_j$, $j=4$), using PCM modelling of polytomous data distributed across a cohort of test agents (persons counting) calculated as follows:

Figure 8. ICC (red, solid line)—together with upper and lower confidence intervals (grey, solid lines above and below) from logistic regression) of partial credit model (Equation (19), to polytomous data (crosses), 10 categories, $0,\dots ,9$) to raw scores on the y-axis; calculated with the Rasch formula for $P_{success,i,j}$ (Equation (17)) across a simulated cohort of varying (x-axis) counting ability, ${\theta }_i$, for a selected counting task (difficulty ${\delta }_j$, $j=4$, shown in Figure 9) of elementary counting tasks (‘items’, j) of a set of clouds of increasing number of dots $1,\dots ,10$ [8] ($N_{TP}=500)$.

Figure 8. ICC (red, solid line)—together with upper and lower confidence intervals (grey, solid lines above and below) from logistic regression) of partial credit model (Equation (19), to polytomous data (crosses), 10 categories, $0,\dots ,9$) to raw scores on the y-axis; calculated with the Rasch formula for $P_{success,i,j}$ (Equation (17)) across a simulated cohort of varying (x-axis) counting ability, ${\theta }_i$, for a selected counting task (difficulty ${\delta }_j$, $j=4$, shown in Figure 9) of elementary counting tasks (‘items’, j) of a set of clouds of increasing number of dots $1,\dots ,10$ [8] ($N_{TP}=500)$.

For an item with categories $c=0,1,2,\dots ,k$ and Andrich thresholds ${\tau }_c$ (Equation (20)), with ${\tau }_0$ chosen to be 0, the following holds: At location x on the latent variable (relative to the item difficulty), the probability of observing category c is $P_c\left(x\right)={\displaystyle \frac{e^{c·x-{\sum }_{c=0}^k{\tau }_c}}{sump\left(x\right)}}$, where $sump\left(x\right)={\sum }_{c=0}^k{e}^{c·x-sum\left({\tau }_c\right)}$ for $c=0$ to k. The expected score y at location x is given by $y={\sum }_{c=0}^k(c·P_c\left(x\right))$.

Further publications referring to the simulation of psychometric ICC models can be found in, for example, [89,90]. In the present work, the simulated values as described in (Section 4) simply use the usual dichotomous Rasch expression (Equation (17)) (which for benign data sets should give at least as good estimates as rounding to the nearest category, c, as described by [90]).

5.3. Communication of Measurement Information Throughout the Measurement Process: Amount of Entropy

To meet the challenge posed by PMF axes being often on less quantitative or even nominal scales, where even the most basic arithmetic operations cannot be assumed to always work (Section 4.1.3), a viable alternative to calculating distances (even on ordinal and nominal scales) of PMF plots is to take differences in informational (Shannon) entropy.

The amount of measurement information on the categorical scales of signals in each classification category, c, at any one point and state in the measurement process (from object, through instrument and operator to restitution (Figure 1) can be expressed as a [91] ‘surprisal’ $-ln\left(q_c\right)$, while the relative contribution to the total entropy is weighted with the relative occupancy, q_c (chapter 5 in [2]).

The propagation of measurement information depicted in Figure 1, where information can be lost, distorted or gained, is described more readily with sums and differences of informational entropy than with the corresponding combination of probabilities and PMFs. An example of additions of entropy is found in the so-called Mutual information: $I(Z;Y)=H\left(Z\right)-H\left(Z\right|Y)$, which has been used by Benish [92], where an individual’s disease state is denoted Z while the diagnostic test result is Y. The well-known Kullback–Leibler (K-L) metric $d_{KL}(P,Q)$ for a response Y, PMF $Q\left(y\right)$, to an object attribute Z, PMF $P\left(z\right|A)$, is

$$d_{KL}(P,Q)=\int -z·d{P}_{success}=-[P_{success}·log\left(P_{success}\right)+(1-P_{success})·log(q-P_{success})]=H(P,Q)-H\left(P\right).$$

(24)

The K-L parameter, Equation (24), is admittedly a valid metric only for infinitesimally small distances and is only one of several candidates for PMF distance metrics widely used, for example, in automatic image classification [93]. It can also be shown (Section 5.5.3 in [2]) that log-odds transformed data correspond to entropy (Equation (11)).

The start of the measurement process has a deficit in overall entropy (that is, the amount of ‘useful’ measurement information initially available), which, for a discrete PMF, is $\Delta H\left(P\right)=-{\sum }_k{p}_k·ln\left(p_k\right)$, where $p_k$ is the occupancy of category k from the actual quantity attributed to the measurement object. For conceptually simple tasks—such as the counting of dots—this calculated entropy will provide a metrological reference for calibration of classification properties, as illustrated in the current case study (Section 5.4.1). From a measurement system perspective, the response of the instrument (often a score, $P_{success}$, made by a ‘rater’) will provide an input, $X_1$, when establishing the measurement result (Section 3.1).

5.4. Simulated PMFs for the Elementary Counting Case

PMFs are shown in Figure 9 for simulated data for a set of clouds of dots (of increasing (a) (red histograms) task (’object’) counting difficulty, based on entropy theory (Equation (26), Section 5.4.1) and (b) (blue histograms) counter ability (Section 5.4.2), as the conjugate pair of psychometric measurands [8,9]. The blue bars in Figure 9b illustrating the occupancy PMF indicate the numbers of persons at that logit $\theta$ value, and the red dots in Figure 9a indicate the number of items at the corresponding logit $\delta$ values. When $\theta$ and $\delta$ are equal (vertically aligned), the logit difference between them is 0, and the success odds are 50–50. The odds of success increase the further to the right a blue bar is in relation to a red dot, as the positive difference between $\theta$ and $\delta$ grows. Conversely, the further to the left a blue bar is in relation to a red dot, the lower the odds of success become, and the differences between $\theta$ and $\delta$ become negative.

5.4.1. Counting Task Difficulty Simulated: Object (Task) Entropy

For the present guide on PMF simulations, ‘seed’ values will be considered in a study of elementary counting (of dots by Munduruku indigenous people in Brazil, [8]). A set of clouds of dots was presented visually for each counter as a sequence of counting tasks (‘objects’). Task difficulty is in general proportional to entropy—a more ordered (less entropy) task is easier to perform [80,94].

For the current purpose of demonstrating best practice when simulating uncertainties associated with PMFs, of particular interest will be cases where attributes—for instance, the level of difficulty of tasks—are sufficiently simple conceptually so that simulations can be ‘seeded’ with realistic start values. Beyond numerical simulation, it will be particularly valuable if ab initio estimates of categorical quantities such as task difficulty are available. Recent work has demonstrated how such knowledge-informed seed values can indeed be calculated even for categorical properties in a manner analogous to establishing metrological reference values in the more quantitative metrology in chemistry and the material sciences, as demonstrated in the neurodegeneration studies of legacy memory tests [95], where recall task difficulty could be expressed with the so-called Construct Specification Equation (CSE [96,97,98,99]):

$$\delta ={\Sigma }_k{\beta }_k·Z_k$$

(25)

in terms of entropy-based explanatory variables $Z_k$. The PMFs plotted (red histograms) in Figure 9 against the measurand, counting task difficulty, ${\delta }_j$, are those predicted to increase discretely, reflecting the increasing number, G, of integer dots in successive clouds according to the basic Hartley entropy [100]:

$${\delta }_j=ln\left(G_j\right)$$

(26)

assuming a random distribution of identical dots in each cloud counted. (The addition of some symmetry to a cloud of dots would make the counting task easier, in terms of a lowering in the overall task entropy, as has been studied for elementary memory recall tests [80], p. 68.)

5.4.2. Counting Classifier Ability Simulated

• Excel:

  $$NORM.INV\left(RAND\right();0;2)$$

  (27)

  returns a vector of random numbers having the Normal distribution of the measurand counter abilities across the cohort shown with the blue histograms in Figure 9.

5.5. Reliability and Validity

In making separate estimates of the conjugate attributes, ${\delta }_j$ and ${\theta }_i$—which are not merely the measurands, but also for ‘clinical’ purposes the quality characteristics, when conformity assessment is made—the normal rules of statistics of course apply—including making sufficient numbers of observations of items and instruments—to ensure minimum levels of reliability and power, as these are relevant in fit-for-purpose designs distinguishing between the varying demands of screening, diagnostic, accountability, and research applications. Additional, more metrological requirements go beyond requiring mere numbers to making judicious and representative sampling across the full range of measurement scales. For metrological purposes—concerning both precision and trueness (for metrological invariance, comparability and interoperability)—the underlying scale needs to be unidimensional, linear, quantitative, well targeted, free of differential item functioning, etc., and there is a whole battery of conventional tests of model validity [94].

The standard uncertainty in the ‘height’ (i.e., occupancy, $P_{success}$) of a PMF column is

$$u\left(P_{success}\right)=\sqrt{{\displaystyle \frac{P_{success}·(1-P_{success})}{n}}},$$

(28)

derived from the underlying binomial distribution (Equation (7)). One should, however, always respect the inherent non-linearity of the scale, such as due to the counted-fractions effect (Section 4.1.2). A more proper estimation of this uncertainty is to first calculate the uncertainties in the link function, as given by the GLMM log-odds approach (Equation (11)).

Uncertainties (so-called Standard Errors, SE, i.e., standard uncertainties) in Rasch analyses with the logistic regression (Equation (18)) are those quoted by the WINSTEPS^® program (section 21.120), where, according to that program’s manual (p. 805),

$$SE({\theta }_i,\tilde{\delta })=\sqrt{{\displaystyle \frac{1}{{\Sigma }_{j=1}^{N_{items}}{P}_{success,i,j}·(1-P_{success,i,j})}}}$$

(29)

and

$$SE(\tilde{\theta },{\delta }_j)=\sqrt{{\displaystyle \frac{1}{{\Sigma }_{i=1}^{N_{TP}}{P}_{success,i,j}(1-P_{success,i,j})}}}.$$

(30)

Thereafter, one can apply the inverse of the Rasch transformation in case the end-user wants to relate to the original PMFs. (Uncertainty intervals in $P_{success}$ calculated in that way will become increasingly asymmetric the closer the score is to either end of the scale, which is how it should be due to the counted-fractions effect.)

Reliability means assessing the influence of measurement uncertainty as an estimate of limited measurement quality. In psychometrics [53], a reliability coefficient, $R_{\beta }$, for a Rasch variable $\beta ={\beta }^{\prime }+{ϵ}_{\beta }$ (for either Rasch attribute: $\beta =\theta$ or $\delta$), including an error term, ${ϵ}_{\beta }$, is defined as

$$R_{\beta }={\displaystyle \frac{var\left({\beta }^{\prime }\right)}{var\left(\beta \right)}}={\displaystyle \frac{var\left(\beta \right)-var\left({ϵ}_{\beta }\right)}{var\left(\beta \right)}}$$

(31)

where the ‘true’ variance is $var\left({\beta }^{\prime }\right)$ and the observed variance is $var\left(\beta \right)$.

The consequences of the decisions to be made determine what the actual limits of reliability are, for instance, by setting a maximum permissible uncertainty. Traditional psychometric limits for so-called ‘high-stakes’ decisions are typically $R_{\beta }>0.8$ (Equation (31)), which corresponds to at most half of the observed dispersion being assigned to measurement uncertainty.

5.6. Validation of Simulated Logistic Regressions

The results of logistic regressions of the partial credit model (Equation (19)) to raw scores (y-axes of Figure 8; calculated with the dichotomous Rasch formula for task difficulty ${\delta }_j$, Equation (17)) across a simulated cohort of varying (x-axes) counting abilities, ${\theta }_i$, (Figure 9b) for a pair of selected counting tasks (difficulties ${\delta }_j$ shown in Figure 9a) of elementary counting tasks, are presented. Using the WINSTEPS® program for this logistic regression, the raw scores were coded into categories by integer rounding of the % scores, as is particularly evident in the steps shown in Figure 8.

The validity of this procedure is investigated in Figure 10 and Figure 11 by comparing the restituted estimates (y-axes) of $\delta$ and $\theta$ with the original simulated values (x-axes) plotted, respectively.

A second validation was made by comparing (Figure 12) simulated (y-axis) and original experimental data (x-axis) counting task difficulties from psychometric analysis (Rasch Equation (17)).

6. Discussion and Conclusions

The choice of construct—quantities attributed to an elementary counting process—in the current case study is made deliberately. The conceptual simplicity ensures the unidimensionality required in metrological psychometrics [48,94,101]. Additionally, it offers the opportunity to define—from first principles—metrological references (’gold standards’) [80] for counting task difficulty in terms of Construct Specification Equations [96,97,98,99,102] with explanatory variables based on informational entropy [103,104], as demonstrated in the simulation of counting task difficulty (Section 5.4.1). Such metrological references for calibration of categorical properties in turn enable interoperability and defining limits on quality of products and services of all kinds. (Other explanatory approaches beyond CSE include [105,106].)

This conceptual simplicity facilitates contrasting the metrology of three kinds of probability mass functions (PMFs) (Table 1) presented in this work: (i) quantitative response to a discrete range of counts (Figure 3); (ii) the classic Poisson distribution of miscounts (Figure 5); and (iii) psychometric (Rasch) distributions of counting task difficulty and person counting ability (Figure 9). New insights are provided when benchmarking the Rasch Poisson Counts Model (Section 4.3.3), which has received less attention, against full psychometric Rasch modelling (Section 5). Limits (traditionally rules of thumb on small probabilities and a large number of observations) on the validity of the Poisson distribution can be tested with access to psychometric simulations across the full range of task difficulty and counter ability.

A human being acting as a counter (MSA: A human as a Measurement Instrument [43]) is a prototype for more advanced applications, including counting, decision-making and similar classification tasks performed by Artificial Intelligence agents [7,43,80,107,108,109]. Logistic regression (Section 4.3) provides an essential complement when handling discrete and sometimes qualitative observations, in particular classifications [43], for example, to a typical use of Machine Learning in metrology where traditionally regular regression tasks are made on a continuous quantity [110]. The proposed methods will be useful when the traditional role of Physics in metrology is to be complemented by a typical ‘turn-of-the-millennium’ focus on the new metrology of ‘clinical’ and other applications where meaning and value, and not just numbers and analytics, are sought.

Future work is needed to extend the present studies to include less elementary and more complex constructs. There is much current research addressing multidimensional measurement theory and related topics in this field [35,111,112,113,114].