A New Approach to Model Verification, Falsification and Selection

Abstract

This paper shows that a qualitative analysis, i.e., an assessment of the consistency of a hypothesized sign pattern for structural arrays with the sign pattern of the estimated reduced form, can always provide decisive insight into a model’s validity both in general and compared to other models. Qualitative analysis can show that it is impossible for some models to have generated the data used to estimate the reduced form, even though standard specification tests might show the model to be adequate. A partially specified structural hypothesis can be falsified by estimating as few as one reduced form equation. Zero restrictions in the structure can themselves be falsified. It is further shown how the information content of the hypothesized structural sign patterns can be measured using a commonly applied concept of statistical entropy. The lower the hypothesized structural sign pattern’s entropy, the more a priori information it proposes about the sign pattern of the estimated reduced form. As an hypothesized structural sign pattern has a lower entropy, it is more subject to type 1 error and less subject to type 2 error. Three cases illustrate the approach taken here.

1. Introduction

In general, a demonstration that economics is a science would be comprised of showing that economics organizes its subject matter in some fashion that limits the configurations that may be taken on by the data. Organizing the subject matter in a fashion that limits the permissible realizations of the data may be termed “theory”. If the limits on the data required by the theory are not found, then the theory has been falsified (Popper [1]). The potential for falsification provided by propositions about the economy is termed by Samuelson [2] as the source of economics’ “meaningful theorems”.

For example, an allocation-of-time model might produce the result that more schooling results in greater lifetime earnings, ceteris paribus. It is the all-other-things-equal clause that produces the wrinkle. Angrist and Krueger [3] suggested that children’s dates of birth sort children into groups with more or less education for a given chronological age, and therefore we can observe the effect of schooling on earnings independent of ability. Butcher and Case [4] argue that it is gender of first-born-sibling that can be used to identify the value of schooling. These two papers might be thought of as (non-)nested alternatives. 1 Barron, Ewing and Waddell [6] consider the effect of high school athletics participation on lifetime earnings differences. Since they do not account for a quarter of birth or gender of first-born-siblings, their theory imposes zero restrictions on the theories of the first two papers. Not surprisingly, all three papers report statistically significant returns to educational activity, be it sports participation or years of schooling. All three of these papers use instrumental variables estimation. None of them have asked to what extent the alternative structural models restrict the outcomes in the first stage of estimation. The implicit first stage reduced form is not fraught with the statistical difficulties inherent in reporting structural results common to all three papers and would therefore be the proper seat for model falsification. 2 Determination of the permissible signs for the reduced form coefficients, given the structural model, is known as a “qualitative analysis”. To state the issue succinctly: It is quite possible to fit alternative “models” or “theories” to the data and find statistically significant results even though the proposed structural models could not possibly have generated the observed data. 3

This paper shows that a qualitative analysis can always be used in evaluating a model’s validity both in general and compared to other hypothesized models. Issues that can be approached via qualitative analysis are identified and manners of assessing them are illustrated. To do this, algorithmic approaches are used to process a structure’s sign pattern information in order to reach conclusions about a model’s scientific content, its potential for falsification (and associated type 1 error) and issues at stake in its acceptance (and associated type 2 error). 4 It is demonstrated that such an analysis can always potentially falsify the hypothesized structural sign patterns or support an analysis of the relative likelihoods of alternative structural hypotheses, if neither are falsified. It is also noted that a partially specified structural hypothesis can be sometimes falsified by estimating as few as one reduced form equation. Additionally, zero restrictions in the structure can themselves be falsified; and, when so, current practice proposes estimated structural arrays that are impossible. It is further shown how the information content of the hypothesized structural sign patterns can be measured using Shannon’s [7] concept of entropy. In general, the lower the hypothesized structural sign pattern’s entropy, the more a priori information it proposes about the sign pattern of the estimated reduced form. As an hypothesized structural sign pattern has a lower entropy, it is more subject to type 1 error and less subject to type 2 error. We want to emphasize that the techniques presented here are proposed as complements to, rather than replacements for, currently accepted econometric practice.

Following Samuelson [2], we propose that economic “theory” about some feature of the economy is expressed by an enumeration of n-many endogenous variables Y to be explained by m-many exogenous variables Z. Further, relationships among these variables are expressed by n-many relationships,

f ⁱ (Y, Z) = 0, i = 1, 2, …, n

(1)

A propos of the returns to schooling papers cited above, the issue of whether or not the system (1) presents a refutable hypothesis is studied by the method of comparative statics, as specified by a linear system of differentials associated with disturbing a solution (which we assume always exists) to the system (1),

$${\displaystyle \mathrm{\sum }_{j=1}^n}\frac{\partial f^i}{\partial y_j}d{y}_j + {\displaystyle \mathrm{\sum }_{k=1}^m}\frac{\partial f^i}{\partial z_k}d{z}_k = 0, i = 1,2,\mathrm{\dots },n$$

(2)

For the econometrician to bring the system (2) to the data, it is often assumed that the relationships in (2) are (at least locally) linear. In particular, that over the range of the data at issue, the signs of the partial derivatives in (2) are invariant. Adding unobserved disturbance terms expressed by the n-vector U, (2) is given by the equation below,

βY = γZ + δU

(3)

where β, γ and δ are appropriately dimensioned matrices. 5 Equation (3) is usually called the structural form. The disturbance U is a random vector with zero means and finite, positive definite covariance matrix. 6 Bringing Equation (3) to the data requires manipulation. To avoid a variety of statistical issues the system to be estimated is expressed as,

$$\mathrm{Y}=\mathrm{\pi } \mathrm{Z}+\mathrm{\psi }\mathrm{U},$$

(4)

where π = β⁻¹γ. (4) is usually called the reduced form, from which the unknowns in Equation (3) are subsequently derived. Given this, the issue of falsification becomes that of the limitations on the entries of the estimated π = β⁻¹γ, denoted by $\stackrel{⌢}{\mathrm{\pi }}$, as derived from the theoretical proposals about the entries of {β, γ}.

When Samuelson [2] considered the issue in exactly these terms, he first noted that the theory might specify the directions of influence among the endogenous and exogenous variables, i.e., the sign patterns of β and γ. Given this, due to a “qualitative analysis”, it might be possible to go through the algebra of calculating π = β⁻¹γ and finding that certain entries of π must have particular signs, based entirely upon the sign patterns of β and γ, independent of the magnitudes of their entries. If so, then when a $\stackrel{⌢}{\mathrm{\pi }}$ is realized, if the required signs do not show up, the theory has been falsified. 7 On reflection, Samuelson proposed that as a practical matter, a qualitative analysis was extremely unlikely to be successful. To work, all of the potentially millions of terms in the expansions of β’s determinant and cofactors necessary in computing β⁻¹ would all have to have the same sign, independent of magnitudes. Samuelson viewed this as too unlikely to be taken seriously. Accordingly, he proposed other sources for limitations on the signs, of π, e.g., that β is a stable matrix or is derived from the second order conditions to a (perhaps constrained) optimization problem.

Nevertheless, a literature on the conditions such that a successful qualitative analysis could be conducted was developed (well summarized in Hale et al. [9]). For example, a liberal sprinkling of zeros among β’s entries can drastically reduce the number of nonzero terms in the expansions of its determinant and cofactors. Still, even so, the conditions for a successful qualitative analysis were found to be severe and are rarely satisfied. 8 Accordingly, there is virtually no tradition of assessing a model’s qualitative properties that might falsify entries in sgn $\stackrel{⌢}{\mathrm{\pi }}$. Actually, Samuelson’s [2] other criteria are rarely explored as well. It is tempting to worry that the degree to which economic models are “scientific”; that is, falsifiable based on the data, is really not well in-hand.

The purpose of this paper is to demonstrate that the above view of the potential usefulness of qualitative analysis is miscast: starting with Samuelson’s [2] pessimism through the development of the extremely restrictive conditions such that entries of π can be shown to have particular signs, based upon a qualitative analysis. Instead, we show that a qualitative analysis can always be successfully conducted and that it provides important insight into the “scientific” content of an economic model and can always enumerate outcomes for sgn $\stackrel{⌢}{\mathrm{\pi }}$ that would falsify hypothesized sign patterns for the structural arrays.

In particular, it will be shown that:

-

a specification of the sign patterns of the structural arrays always limits the possible sign patterns that can be taken on by the estimated reduced form;

-

the falsifiability of hypothesized sign patterns for the structural arrays is independent of identifiability 9;

-

zero restrictions specified for the structural arrays can be falsified independent of the sign pattern of the nonzero entries. When this happens, multi-stage least squares will always be in error when providing estimates of the structural arrays, i.e., the sign patterns of the estimated structural arrays with zero restrictions imposed are impossible, given the sign pattern of the estimated reduced form;

-

a partially specified structural hypothesis sometimes can be falsified by a small number of estimated reduced form equations, even as few as one;

-

the conditional probability that an hypothesized structural sign pattern is actually true when not falsified can be estimated using Bayes’ formula;

-

the relative likelihood of competing structural hypotheses can be assessed, given the sign pattern of the estimated reduced form, based upon their qualitative characteristics; and,

-

hypothesized structural arrays can be assessed for their information content using Shannon’s [7] measure of entropy.

In the next section, a method of qualitative falsification is presented, including using a measure of entropy to measure the information content of a structural hypothesis. In the section following that, issues in “verifying” a structural hypothesis that is not falsified are discussed. Section 4 works through three case studies. The last section provides a summary. An appendix provides additional detail about our method of analysis.

2. Qualitative Falsification

Lady and Buck [8] (2011) and Buck and Lady [11] (2012) showed that the falsification of a qualitative specification of a model does not require that any individual signs of β⁻¹ (or more generally sgn π) be signable based upon sgn β (and more generally, also sgn γ). Instead, patterns of signs in sgn π may not be possible, even though no individual entry is signable. In this section, this principle is generalized compared to this earlier work. In the next section, the new issue of verification is presented.

An easy example to use for explication is β: 2 × 2 without zero entries. For this and all of the examples below, it is assumed that β is irreducible 10 and not singular. There are 16 possible sign patterns that can be hypothesized for β. Of these, eight have an entry with a sign the opposite of the other three. For these cases, the determinant of β is signable, independent of the magnitudes of its entries. Since the adjoint of β is signable, for these cases there is only one possible sign pattern for β⁻¹. For the other eight cases, there are only two possible sign patterns for β⁻¹; however, no entry of sgn β⁻¹ is the same for these two possibilities, since they are the negative of each other. Accordingly, for any of these eight cases, an hypothesized sgn β would be falsified by sgn $\stackrel{⌢}{\mathrm{\pi }}$ (assuming for the moment that γ = I), if it took on one of the 14 sign patterns that are not possible for the hypothesized sgn β, even though no individual entry of sgn β⁻¹ is signable.

For larger arrays (and/or for γ ≠ I), although the same principles apply, the algorithmic burden of revealing them quickly becomes more difficult. To cope with this, we developed a Monte Carlo algorithm for investigating the possible sign patterns of the reduced form based upon a qualitative specification of the structural form. 11 In brief, sgn β and sgn γ (if need be) are randomly given quantitative values for their entries consistent with the hypothesized sign pattern, π = β⁻¹γ is computed, and the sign pattern outcome of this computation is saved. The iterations/repetitions can be done millions of times. For sufficiently small systems, the set of all possible sign patterns that the reduced form might take on, given the hypothesized sign pattern of the structural arrays, can be tabulated with a vanishingly small potential for error as the number of repetitions increases. 12 A central point of the analysis is that this set of allowed sign patterns always limits the sign patterns the reduced form can take on. Given this, there are always sign patterns that the reduced form array might take on that are impossible, given the hypothesized structural sign patterns, and if so, the given structure is thus falsified.

Consider as the next example the case of hypothesizing that sgn β is 3 × 3 and all positive and γ = I. 13 In principle, there are 512 such 3 × 3 sign patterns that sgn β⁻¹ can take (barring zeros as we always do for the examples we provide, although this is not necessary). Of these, only 102 were found by the Monte Carlo in tens of millions of iterations. Given this, it appears that 410 sign patterns for sgn β⁻¹ are impossible for the hypothesized sgn β. The analytic basis for some sign patterns being impossible can be quickly developed. Let the qualitative inverse of sgn β be defined as:

Definition (qualitative inverse). Sgn π is a qualitative inverse of sgn β if and only if there exist magnitudes for the entries of β, consistent with the given sgn β, such that sgn β⁻¹ = sgn π.

Given this, it is immediate that the proposed sign patterns must be such that magnitudes can be assigned so that:

• Sgn π is a qualitative inverse of sgn β only if it is possible that i) βπ = I; and, ii) πβ = I.

This possibility can be algorithmically investigated directly. For β 3 × 3 and all positive, of the 512 possible 3 × 3 sign patterns that sgn π might take on, only 216 sign patterns satisfy i) or ii) individually; and, there are only 102 sign patterns that satisfy both. These are what the Monte Carlo found. Since i) and ii) are clearly necessary for the case here, their satisfaction also turned out to be sufficient for sgn π to be a qualitative inverse for the hypothesized sgn β. As we will see below, satisfaction of both of these conditions may not always be sufficient. From the standpoint of falsification for β all positive and of any dimension, it is immediate that a proposed, i.e., estimated, sgn π cannot satisfy both i) and ii) unless it has a positive and negative entry in each row and column. More generally, sgn π cannot be a qualitative inverse of a proposed sgn β unless, for i) above the i^th row of β and the j^th column of π share a pair of common nonzero entries of the same sign for i = j and both this and another pair of common nonzero entries of the opposite sign for i ≠ j; and, also for the i^th row of π and the j^th column of β for ii) above.

For the special case that we are considering, why not simply set $\widehat{\mathrm{\beta }} = {\stackrel{⌢}{\mathrm{\pi }}}^{-1}$ and compare sgn $\widehat{\mathrm{\beta }}$ with the hypothesized sgn β? If any signs are different, then the hypothesized sgn β has been falsified and there is no need to worry about the derivations immediately above. This could indeed be done for the exactly identified case, which given our simplifying assumptions requires that β have no zero restrictions. However, even for this very special case, if sgn $\widehat{\mathrm{\beta }}$ ≠ sgn β, this outcome might not be taken as falsification. It might be that sgn $\stackrel{⌢}{\mathrm{\pi }}$ is a qualitative inverse of both the data based sgn $\widehat{\mathrm{\beta }}$ and the hypothesized sgn β. Given this, further analysis, not to speak of empirical work, might be viewed as necessary. This situation is discussed in the next section. Generally, for n ≠ m and/or γ ≠ I setting $\widehat{\mathrm{\beta }} = {\stackrel{⌢}{\mathrm{\pi }}}^{-1}$ is not possible and/or appropriate. For these more general cases, if (3) is exactly identified, then unique arrays $\widehat{\mathrm{\beta }} and \widehat{\mathrm{\gamma }}$ can be found based upon $\stackrel{⌢}{\mathrm{\pi }}$; nevertheless, as pointed out above, this may not lead to falsification, at least not without further analysis.

For the special case at issue here, if zero restrictions are imposed upon sgn β, then (3) is over identified and the zero restrictions will be imposed upon the derivation of $\widehat{\mathrm{\beta }}$. Under these circumstances, there is more than one way to recover $\widehat{\mathrm{\beta }}$ from $\stackrel{⌢}{\mathrm{\pi }}$. For these, sgn $\widehat{\mathrm{\beta }}$ may not be the same. Even more serious, the zero restrictions themselves can be falsified. If this is the case for the sgn $\stackrel{⌢}{\mathrm{\pi }}$ that was found, whatever method is used to recover $\widehat{\mathrm{\beta }}$ results in a mistake if these zero restrictions are imposed; namely, the estimated array $\widehat{\mathrm{\beta }}$ is impossible, given sgn $\stackrel{⌢}{\mathrm{\pi }}$; i.e., sgn $\stackrel{⌢}{\mathrm{\pi }}$ is not a qualitative inverse of sgn $\widehat{\mathrm{\beta }}$.

As an easy to follow example, let the hypothesized β be given by,

$$\text{sgn} \beta = \left[\begin{array}{ccc}?& ?& ?\\ ?& ?& ?\\ 0& ?& ?\end{array}\right].$$

For this array, there is the single zero restriction, β₃₁ = 0. The remaining entries marked “?” are nonzero, but can be either positive or negative. For this case, assume that, based on the data 14

$$\text{sgn} \widehat{\pi } = \left[\begin{array}{ccc}?& ?& ?\\ +& -& ?\\ +& +& ?\end{array}\right].$$

This result cannot be the qualitative inverse for any array for which β₃₁ = 0, since the (1,3) cofactor is signable, independent of magnitudes. Of the 512 possible sign patterns that sgn $\stackrel{⌢}{\mathrm{\pi }}$ might have taken on from the data, 32 of them have the above 2 × 2 sign pattern corresponding to the (1,3) cofactor. As discussed above, altogether there are eight such signable sign patterns for this cofactor. As a result, 256 of the possible 512 sign patterns that sgn $\stackrel{⌢}{\mathrm{\pi }}$ might take on falsify the given zero restriction, independent of the signs of the other entries of the proposed sgn β or estimated sgn $\stackrel{⌢}{\mathrm{\pi }}$. If any of these show up for sgn $\stackrel{⌢}{\mathrm{\pi }}$, then however $\widehat{\mathrm{\beta }}$ is derived, it is a mistake, i.e., impossible, if the zero restriction is imposed. 15

To dramatize the potential for the zero restrictions themselves to falsify a structural hypothesis, and potentially lead to an error when deriving $\widehat{\mathrm{\beta }}$ from $\stackrel{⌢}{\mathrm{\pi }}$, consider as an example an inference structure for β (assumed to be irreducible 16) that contains only one cycle of inference. Consider this case for n = 5,

$$\text{sgn} \mathrm{\beta }=\left[\begin{array}{ccccc}?& 0& 0& 0& ?\\ ?& ?& 0& 0& 0\\ 0& ?& ?& 0& 0\\ 0& 0& ?& ?& 0\\ 0& 0& 0& ?& ?\end{array}\right]$$

As before, the entries marked “?” are nonzero, but may be of any sign. The number of possible qualitative inverses for this array can be derived directly. In particular, the expansion of each cofactor of β includes only one term, since there is only one path of inference between any pair of (endogenous) variables. There are 10 nonzeros which can take on 2¹⁰ = 1024 distinct sign patterns. Accordingly, adjoint β can only take on 1024 distinct sign patterns. These in turn can be divided into two groups for which the patterns in one group are the negative of the patterns in the other group. As it happens, the determinant is signed for half of the 1024 cases, but this does not make any difference. Given the possible sign patterns of the adjoint, there are only 1024 possible sign patterns for sgn β⁻¹, regardless of how the determinant’s (nonzero) sign plays out. Yet, for a 5 × 5 array, there are 33,554,432 possible sign patterns, barring zeros. For this case, of these, 33,553,408 are impossible, based upon the zero restrictions on β alone (plus γ = I). For this inference structure, the proportion of qualitative inverses among the possible sign patterns decreases significantly as n increases, since there are 2ⁿⁿ possible sign patterns for which only 2²ⁿ are qualitative inverses. It is worth emphasizing the points just made. If the zero restrictions of the structural hypothesis are falsified by the sign pattern of the estimated reduced form, then the estimated structural arrays with these zero restrictions imposed are impossible, given the sign pattern of the estimated reduced form. Since current practice does not include conducting a qualitative analysis, this leaves the prospect of the outcome of many estimations as unknowingly being utterly impossible.

If two or more entries of the adjoint of β are signable, which could well be the case if the nonzeros in β are sparse, then a criterion in addition to i) and ii) above for the falsification of β based upon sgn $\stackrel{⌢}{\mathrm{\pi }}$ is in-hand; namely, the entries of sgn $\stackrel{⌢}{\mathrm{\pi }}$ that correspond to the signable entries of the adjoint must always have the same or opposite signs as appropriate, or the hypothesized β is falsified. This circumstance can be robustly detected by the Monte Carlo and algorithmically applied directly. To organize this idea formally, let B = [B_ij] be the adjoint of β.

Definition (adjoint-consistent). If sgn B_ij = sgn B_uv (resp. sgn B_ij = −sgn B_uv) independent of magnitudes, then sgn π is adjoint-consistent with sgn β if and only if sgn π_ij = sgn π_uv (resp. sgn π_ij = −sgn π_uv).

Thus, a third criterion would be,

iii)

An estimated sgn $\stackrel{⌢}{\mathrm{\pi }}$ is a qualitative inverse of sgn β only if the estimated sgn $\stackrel{⌢}{\mathrm{\pi }}$ is adjoint consistent with sgn β.

Once two or more signable entries in sgn B are enumerated, satisfaction of iii) can be algorithmically investigated directly. Taken together, i), ii), and iii) are necessary. As shown in the example below, they may not be sufficient. The ultimate requirement for qualitative inverses of sgn β would be the existence of a solution to a system of inequalities, as given below.

For a given sgn $\stackrel{⌢}{\mathrm{\pi }}$, consider the system(s) of inequalities as written out symbolically as the expansions of β’s cofactors and determinant,

$$\text{sgn} \mathrm{B} = \text{sgn} \widehat{\mathit{\pi }}, \text{and} \text{det} \mathrm{\beta } > 0; \text{and}/\text{or}, \text{sgn} \mathrm{B} = -\text{sgn} \widehat{\mathit{\pi }}, \text{and} \text{det} \mathrm{\beta } < 0$$

(5)

It is immediate that a given empirical sgn $\widehat{\mathit{\pi }}$ is a qualitative inverse of a proposed sgn β if and only if at least one of the systems in (5) has a solution. Even if i), ii), and iii) are satisfied, a given empirical sgn $\stackrel{⌢}{\mathrm{\pi }}$ may not be such that one or both of the systems (5) has a solution. The necessary and sufficient algorithmic method for assessing if (5) is satisfied, if it exists, is beyond the scope of this paper. It was the resolution of this issue (among others) for which the Monte Carlo was developed as a heuristic. For a given empirical sgn $\stackrel{⌢}{\mathrm{\pi }}$, if the hypothesized sgn β is quantitatively sampled repeatedly and β⁻¹ computed, and it is repeatedly found that, given this, the given sgn $\stackrel{⌢}{\mathrm{\pi }}$ ≠ sgn β⁻¹, then the inference is that the given sgn $\stackrel{⌢}{\mathrm{\pi }}$ is not a qualitative inverse of the hypothesized sgn β. The chances of incorrectly falsifying the hypothesized structure are made ever smaller by increasing the number of quantitative Monte Carlo iterations taken, but the chances are never absolutely zero. For this reason, once falsification is strongly suspected based upon the Monte Carlo, a direct investigation of (5) can be attempted, hoping to analytically confirm what the Monte Carlo suggests. Examples of this approach are given in the appendix of Buck and Lady [11].

As an example of applying all of the criteria presented above, consider the case (with γ = I),

$$\text{sgn} \mathrm{\beta } = \left[\begin{array}{cccc}-& +& +& 0\\ +& -& +& 0\\ 0& +& -& +\\ 0& +& +& -\end{array}\right]$$

This sign pattern, negative main diagonal entries and non-negative off diagonal entries, is an example of a Metzler [12] matrix, a form corresponding to the excess demand functions for multimarket equilibria for which all commodities are (weakly, if zeros are allowed) gross substitutes. The four zero restrictions are imposed here for the purpose of demonstrating all of the falsification criteria discussed above.

Application of the above criteria revealed the following: Of the 65,536 possible 4 × 4 sign patterns for sgn β⁻¹, only 4096 are possible, given the zero restrictions, independent of the signs of the nonzeros. To reiterate this point yet again, if the sign pattern of the estimated reduced form had any of the other 61,440 sign patterns, then any sign pattern proposed for sgn $\widehat{\mathrm{\beta }}$ derived from the observed reduced form with these zero restrictions imposed would be a mistake. Direct analysis revealed that only 625 of the possible sign patterns for sgn β⁻¹ satisfied i) above, 400 satisfied ii), and only 28 satisfied both. Further analysis revealed that entries of β adjoint were signable, as given below,

$$\text{Adjoint} \mathrm{\beta } = \left[\begin{array}{cccc}*& *& a& a\\ b& b& a& a\\ a& a& c& c\\ a& a& *& *\end{array}\right]$$

In the above, entries marked “a”, “b”, and “c” always have the same sign. Taking this into account for the 28 sign patterns that satisfied both i) and ii), only 20 satisfied iii), i.e., presented the pattern of equal signs given above. Of these, the Monte Carlo only found 18. Since the outcome of i), ii), and iii) can be inspected directly, the findings are clearly necessary. Of 20 sign patterns that satisfied all of these necessary conditions, two were nevertheless not solutions of (5). For these two, further analysis would be appropriate to confirm that the finding falsified the hypothesized structure, i.e., to uncover the contradiction that prevents the satisfaction of (5).

A feature of the above analysis was the complete specification of the qualitative structure of (3) and (4). It should be noted that falsification might be the case if only parts of the structure are qualitatively specified and less than all of the reduced form equations are estimated. For example, if the sign pattern of only one column of β is hypothesized, then if only one row of π is estimated, the (partial) structural hypothesis has been falsified if the signs of the entries of that row are the opposite from the corresponding nonzeros in the hypothesized column of β, since the criterion ii) above would not be satisfied. Other examples of falsifying partially specified structural models can be readily proposed.

The simplifying assumptions for the analysis so far, n = m and γ = I, were made to permit the principles of the analysis to be readily revealed. Relaxing these simplifications increases the analytic burden of the analysis; nevertheless, a qualitative specification of the structural form will inevitably impose restrictions on β⁻¹ that always result in these restrictions being translated to the reduced form in the more general cases.

For the structural form exactly- or over-identified, estimates $\widehat{\mathrm{\beta }} and \widehat{\mathrm{\gamma }}$ can be derived, although they may not be unique and may be otherwise problematical. Accordingly, the sign patterns of such estimates may not be viewed as decisive from the standpoint of falsification via hypothesis testing; unlike qualitative falsification, which is decisive (absent large sampling errors). For the under-identified structural form, such estimates cannot be made; nevertheless, qualitative falsification can proceed as before.

Consider the structural hypothesis:

$$\text{sgn} \mathrm{\beta } = \left[\begin{array}{ccc}+& +& +\\ +& +& +\\ +& +& +\end{array}\right] and \text{sgn} \mathrm{\gamma } = \left[\begin{array}{ccc}+& +& +\\ +& +& +\\ +& +& +\end{array}\right]$$

For this under-identified structural form, estimates for $\widehat{\mathrm{\beta }} and \widehat{\mathrm{\gamma }}$ cannot be made. Nevertheless, qualitative analysis shows that of the 512 3 × 3 reduced form sign patterns, only 343 are possible for this sign pattern for the structural form. The structural hypothesis would be falsified if any of the other 169 sign patterns were estimated for the reduced form.

Qualitative Falsification and Type 1 Error. As noted above, a structural hypothesis may be rejected because the sign pattern of the estimated reduced form is not found among those admissible reduced forms generated by the Monte Carlo algorithm that we have employed. Given this, further investigation is called for in order to reveal the inconsistency as related to a solution to (5). If this is found, i.e., if the sign pattern of the estimated reduced form is demonstrably impossible, given the hypothesized structural sign pattern, the result may be due to errors in the data, and to reject the structural hypothesis would constitute a type 1 error. The issue of this subsection is to assess the propensity for a structural hypothesis to be incorrectly rejected.

As an example (to be used in the next section as well), let the structural sign patterns given below be termed “system #1”:

$$\text{sgn} \mathrm{\beta } = \left[\begin{array}{cccc}+& +& +& 0\\ +& +& +& +\\ +& +& +& +\\ 0& 0& +& +\end{array}\right] and \text{sgn} \mathrm{\gamma } = \left[\begin{array}{cccc}+& 0& 0& 0\\ 0& +& 0& 0\\ 0& 0& +& 0\\ 0& 0& 0& +\end{array}\right]$$

Of the 65536 4 × 4 sign patterns that the estimated reduced form might take on, simulation results report that only 535 are possible, which constitute less than 1% of the total number. The Monte Carlo simulation draws the absolute values of the nonzeros in the above arrays from a uniform distribution over the open interval [0,10]. This is sufficient to potentially find all possible reduced form sign patterns. However, suppose the distributional rule for populating β and γ is not otherwise supported by data or analysis, which can make a difference as pointed out below. An indicative method to portray the potential for type 1 one error due to sampling error is to draw “error terms” from a normal distribution with mean zero and add these to each nonzero entry used to populate the β and γ arrays, and then compute $\tilde{\pi }$ = β⁻¹γ. If the sign pattern found for $\tilde{\pi }$ is not one of the (535 for system #1) possible sign patterns, then the associated falsification is a type 1 error due to sampling error 17. For this example, this can only happen if the error term that was drawn puts a negative value in one of the array entries when added to the sampled quantitative value that was supposed to be positive. Such a “wrong” sign is intended to represent errors in the data. As the number of possible sign patterns permitted by the structural sign patterns is smaller, the potential for type 1 error due to sampling errors is larger. For system #1 above, with the standard deviation of the “error term” = 0.2, there were 4.68% of 500,000 quantitative Monte Carlo repetitions which falsified system #1’s sign pattern. This result is dependent in part on the distributional assumption employed in the Monte Carlo repetitions. The influence of this approach can be further investigated by the Monte Carlo simulation, but it is beyond our scope to do so.

For the purpose of comparison, a second structural system was formulated, basically by moving three zero restrictions from γ to β and otherwise changing two signs in β. Given this, system #2 is specified as:

$$\text{sgn} \mathrm{\beta } = \left[\begin{array}{cccc}+& -& 0& 0\\ 0& +& +& 0\\ +& 0& +& +\\ -& 0& +& +\end{array}\right] and \text{sgn} \mathrm{\gamma } = \left[\begin{array}{cccc}+& -& 0& 0\\ -& +& 0& 0\\ 0& -& +& 0\\ 0& 0& 0& +\end{array}\right]$$

The Monte Carlo reports that for system #2, that there are only 65 possible sign patterns for the reduced form. From the above discussion, since there are fewer possible reduced form sign patterns (or better yet, a smaller proportion of the total), the potential for type 1 error would be larger for system #2 compared to system #1. This result is reflected in the results tabulated in Table 1 below.

Table 1. Proportion of Type 1 Error.

Standard Deviation of the Error Distribution	Proportion Falsified (500,000 Samples)
	System #1	System #2
0.2	0.0468	0.0672
0.4	0.0976	0.1395
0.6	0.1420	0.1964
0.8	0.1847	0.2604
1	0.2366	0.3256

Table 1. Proportion of Type 1 Error.

Standard Deviation of the Error Distribution	Proportion Falsified (500,000 Samples)
	System #1	System #2
0.2	0.0468	0.0672
0.4	0.0976	0.1395
0.6	0.1420	0.1964
0.8	0.1847	0.2604
1	0.2366	0.3256

Type 1 Error and Entropy. The result of the last subsection can be generalized somewhat. Consider that for the sign pattern of an n × m reduced form array, barring zeros, that there are nm bits of information (say “1” for “+” and “0” for “−“), one bit for each entry. Given a hypothesized structural sign pattern, the possible reduced form sign patterns are limited to Q < 2^nm; and for these, given the algebra of computing π = β⁻¹γ and the distributional rules for assigning values to the nonzeros of {β, γ}, each possible reduced form sign pattern has a particular likelihood of occurrence. Let F_i be the frequency of occurrence of the i^th possible reduced form sign pattern (see below for how “i” is computed). Given this, the entropy of the distribution (Shannon [7], see also Cover and Thomas [13]) is given by:

$$Entropy(\text{sgn}\mathrm{\beta }, \text{sgn}\mathrm{\gamma }) = - {\displaystyle \mathrm{\sum }_{i \in Q}}{F}_i\text{log}(F_i)$$

(6)

where log(F_i) is to the base 2. For example, for one simulation of system #1 above with 3,000,000 samples taken, the entropy of the resulting distribution of (the 535 possible) reduced form sign patterns was 8.18. The unit of this measure is bits. It measures the information achieved by estimating the reduced form. The maximum entropy is 16 for the 4 × 4 sign pattern (barring zeros) and would be the model’s entropy if all of the 65,536 4 × 4 reduced form sign patterns were possible and equally likely. The minimum entropy is zero if only one sign pattern for the reduced form is possible. Our algorithm for drawing the quantitative realizations of the structural sign pattern is designed to find the possible reduced form sign patterns, but has no econometric justification for the distributional (uniform) rule used in the iterations. Nevertheless, we will use the frequency distributions found in the analysis below. If a default entropy is computed which “simply” treats all of the possible reduced form sign patterns as equally likely, then doing this for system #1 causes the entropy measure to increase a small amount to 9.06. The key item driving the measure is the number of possible sign patterns, as opposed to the frequency distribution of the possibilities.

As the entropy measure is high, the structural hypothesis provides less information about the expected outcome of the reduced form’s estimated sign pattern, and conversely. To revise the measure to be larger as the information provided by the structural hypothesis is greater, the following amended measure is proposed,

$$INFO\%(\text{sgn}\mathrm{\beta }, \text{sgn}\mathrm{\gamma })) = 100(1 - \frac{Entropy(\text{sgn}\mathrm{\beta }, \text{sgn}\mathrm{\gamma })}{nm})$$

(7)

For system #1 this comes out to 48.94% and for system #2 (which is more limiting) it comes out to 66.4%. Accordingly, system #2 provides more a priori information about the expected outcome of the estimated reduced form’s sign pattern (only 65 are possible compared to 535 for system #1) and thus is more susceptible to type 1 error compared to system #1.

3. Qualitative Verification

Although contingent upon the propensity for type 1 error, qualitative falsification is otherwise entirely decisive. Given the hypothesized sign pattern of the structural form, either the estimated reduced form sign pattern is possible, or it is not. If not, subject to error, the structural hypothesis has been falsified.

On the other hand, if the sign pattern of a structural hypothesis is not falsified, this does not establish that the structural hypothesis is “true.” Indeed, as Samuelson (op. cit.) pointed out, if degenerate hypotheses are allowed, there are a virtually unlimited number of hypothesized structural forms that could be consistent with, i.e., not falsified by, a particular outcome for the estimated reduced form. In fact, science does not establish “the” true hypothesized system. Instead, a system may be used that is persistently consistent with the data, perhaps chosen by, say, the principle of Occam’s Razor; but always, such a system awaits its replacement by another system that matches its explanatory power to the present point, but provides further explanation of data that the initial system cannot explain, e.g., such as features of the orbit of Mercury around the Sun that were explained by general relativity, but were not explained by Newtonian physics. 18 As a result, what we will term a model’s “verification” is an open-ended, in some sense never-ending, exercise of continuous testing, evaluation, and potential comparison with competing hypotheses. Open-ended or not, the point of this section is to present a number of procedures due to which a qualitative analysis can help evaluate the validity of an hypothesis and compare it to competitors.

The analysis that we develop here will be conditional upon the lists of endogenous and exogenous variables. It will always be assumed that β is irreducible and that γ is otherwise configured such that the n × m reduced form has no logical zeros; and further, we will assume that there are no zeros otherwise in the estimated reduced form. In this frame of reference, a type 2 error is the acceptance of a hypothesized structural form due to its not being falsified, even though the hypothesized structure is not correct. The propensity for a type 2 error can be assessed in general or in comparison to one or more competing hypotheses.

The latter issue might be addressed by assessing the likelihood that a hypothesized structural sign pattern might fail to be falsified by accident, i.e., if the sign pattern of the estimated reduced form were in some sense random and was found to be a member of the permissible set of reduced forms. From this perspective, it is intuitive that system #2 is less likely to be randomly accepted compared to system #1, since the number of possible outcomes for the sign pattern of the reduced form is substantially less for system #2. There is nothing wrong with this insight, but the analysis can be pushed further, specifically in terms of estimating the conditional probability for each system’s validity, given the outcome of the estimated reduced form sign pattern, p (system #1 or #2 sgn $\stackrel{⌢}{\pi }$). Given this, the likelihood of type 2 error is one minus this conditional probability.

For the two systems proposed for illustration here, of the possible reduced form sign patterns for each, 28 are common to both of them. We will develop our illustration in terms of these 28 sign patterns. To start, the Monte Carlo (3,000,000 repetitions for each system) provides an immediate estimate of the conditional probability of a reduced form sign pattern for each system as provided by the frequency with which that sign pattern appeared. These results are given in Table 2 below for the two hypothesized systems. The reduced form sign pattern for the first row of Table 2 is:

$$\text{sgn} \widehat{\mathrm{\pi }}(8914) = \left[\begin{array}{cccc}-& -& +& -\\ -& -& +& -\\ +& +& -& +\\ -& -& +& -\end{array}\right]$$

The base 2 index for this sign pattern is 0010001011010010 (“0” for “−” and “1” for “+”) which equals 8914 in base 10. The base 10 indices in the other rows are computed in the same way and are found in the first column of the table. In the second column of the table, the sign pattern of the reduced form at issue is written out row by row, laid end-to-end.

Not surprisingly, the frequencies for system #2 are uniformly larger than system #1 since significantly fewer sign patterns are possible for this system, although exceptions are possible.

Table 2. Frequency of Reduced Form Sign Patterns in Sample.

$\stackrel{⌢}{\pi }$ Base 10 Index	Sign Pattern of the Reduced Form Row by Row	Frequency in Sample of 3,000,000
		System #1		System #2
		Count	Frequency: p($\stackrel{⌢}{\pi }$|sys #1)	Count	Frequency: p($\stackrel{⌢}{\pi }$|sys #2)
8914	− − + − − − + − + + − + − − + −	3101	0.00103	84455	0.02815
8915	− − + − − − + − + + − + − − + +	1661	0.00055	70168	0.02339
9878	− − + − − + + − + − − + − + + −	2448	0.00082	17047	0.00568
9879	− − + − − + + − + − − + − + + +	3642	0.00121	28049	0.00935
9938	− − + − − + + − + + − + − − + −	2329	0.00078	14731	0.00491
9939	− − + − − + + − + + − + − − + +	4576	0.00153	84842	0.02828
21925	− + − + − + − + + − + − − + − +	1933	0.00064	122556	0.04850
26262	− + + − − + + − + − − + − + + −	10809	0.00360	51052	0.01702
26263	− + + − − + + − + − − + − + + +	6590	0.00220	42724	0.01424
38189	+ − − + − + − + − − + − + + − +	1703	0.00057	18889	0.0063
38249	+ − − + − + − + − + + − + − − +	879	0.00029	36755	0.01225
39273	+ − − + + − − + − + + − + − − +	7996	0.00267	77355	0.02578
40237	+ − − + + + − + − − + − + + − +	2442	0.00081	8728	0.00291
40297	+ − − + + + − + − + + − + − − +	4291	0.00143	16501	0.0055
41562	+ − + − − − + − − + − + + − + −	560	0.00019	44339	0.01478
41563	+ − + − − − + − − + − + + − + +	1061	0.00035	99335	0.03311
41682	+ − + − − − + − + + − + − − + −	1137	0.00038	43087	0.01436
41683	+ − + − − − + − + + − + − − + +	1713	0.00057	28838	0.00961
42586	+ − + − − + + − − + − + + − + −	2884	0.00096	20251	0.00675
42587	+ − + − − + + − − + − + + − + +	7693	0.00256	165480	0.05516
42646	+ − + − − + + − + − − + − + + −	5229	0.00174	182054	0.06068
42647	+ − + − − + + − + − − + − + + +	10518	0.00351	98504	0.03283
42706	+ − + − − + + − + + − + − − + −	7833	0.00261	48281	0.01609
42707	+ − + − − + + − + + − + − − + +	26405	0.00880	75320	00.02511
43610	+ − + − + − + − − + − + + − + −	2029	0.00068	70759	0.02359
43611	+ − + − + − + − − + − + + − + +	1690	0.00056	76690	0.02556
54573	+ + − + − + − + − − + − + + − +	1280	0.00043	56623	0.01887
56621	+ + − + + + − + − − + − + + − +	7170	0.00239	55421	0.01847

Table 2. Frequency of Reduced Form Sign Patterns in Sample.

$\stackrel{⌢}{\pi }$ Base 10 Index	Sign Pattern of the Reduced Form Row by Row	Frequency in Sample of 3,000,000
		System #1		System #2
		Count	Frequency: p($\stackrel{⌢}{\pi }$|sys #1)	Count	Frequency: p($\stackrel{⌢}{\pi }$|sys #2)
8914	− − + − − − + − + + − + − − + −	3101	0.00103	84455	0.02815
8915	− − + − − − + − + + − + − − + +	1661	0.00055	70168	0.02339
9878	− − + − − + + − + − − + − + + −	2448	0.00082	17047	0.00568
9879	− − + − − + + − + − − + − + + +	3642	0.00121	28049	0.00935
9938	− − + − − + + − + + − + − − + −	2329	0.00078	14731	0.00491
9939	− − + − − + + − + + − + − − + +	4576	0.00153	84842	0.02828
21925	− + − + − + − + + − + − − + − +	1933	0.00064	122556	0.04850
26262	− + + − − + + − + − − + − + + −	10809	0.00360	51052	0.01702
26263	− + + − − + + − + − − + − + + +	6590	0.00220	42724	0.01424
38189	+ − − + − + − + − − + − + + − +	1703	0.00057	18889	0.0063
38249	+ − − + − + − + − + + − + − − +	879	0.00029	36755	0.01225
39273	+ − − + + − − + − + + − + − − +	7996	0.00267	77355	0.02578
40237	+ − − + + + − + − − + − + + − +	2442	0.00081	8728	0.00291
40297	+ − − + + + − + − + + − + − − +	4291	0.00143	16501	0.0055
41562	+ − + − − − + − − + − + + − + −	560	0.00019	44339	0.01478
41563	+ − + − − − + − − + − + + − + +	1061	0.00035	99335	0.03311
41682	+ − + − − − + − + + − + − − + −	1137	0.00038	43087	0.01436
41683	+ − + − − − + − + + − + − − + +	1713	0.00057	28838	0.00961
42586	+ − + − − + + − − + − + + − + −	2884	0.00096	20251	0.00675
42587	+ − + − − + + − − + − + + − + +	7693	0.00256	165480	0.05516
42646	+ − + − − + + − + − − + − + + −	5229	0.00174	182054	0.06068
42647	+ − + − − + + − + − − + − + + +	10518	0.00351	98504	0.03283
42706	+ − + − − + + − + + − + − − + −	7833	0.00261	48281	0.01609
42707	+ − + − − + + − + + − + − − + +	26405	0.00880	75320	00.02511
43610	+ − + − + − + − − + − + + − + −	2029	0.00068	70759	0.02359
43611	+ − + − + − + − − + − + + − + +	1690	0.00056	76690	0.02556
54573	+ + − + − + − + − − + − + + − +	1280	0.00043	56623	0.01887
56621	+ + − + + + − + − − + − + + − +	7170	0.00239	55421	0.01847

The estimates in columns 4 and 6 can be used in computing the desired conditional probability using Bayes’ formula:

$$p(system\#1 or\#2 |\text{sgn}\widehat{\mathrm{\pi }})=\frac{p(\text{sgn}\widehat{\mathrm{\pi }} |system\#1 or \#2)p(system\#1 or \#2)}{p(\text{sgn} \widehat{\mathrm{\pi }})}$$

(8)

To use this formula, in addition to the estimates of the conditional probabilities provided in Table 2, estimates of the prior probabilities for each system and the reduced form sign pattern must also be made. Presumably, in a particular empirical context, there would be reasons for assigning these values. Here, for the sake of illustration, we will proceed as follows:

Consider that for the 32 entries in β and γ, the two hypothesized systems agree on the signs for 23 of them. Given this, for all combinations of the nine disputed signs, there are 512 (= 2⁹) possible structural sign patterns. Assume that all of these are equally likely; hence, the prior probability for each such system = 1/512 = 0.00195, including systems #1 and #2. These 512 structural sign patterns might be termed the “universe” from which the specific structural sign patterns were selected, i.e., as what the theory dictates and (evidently) leaves undecided. 19 This universe can be sampled and the frequency found for each of the 28 reduced form patterns enumerated in Table 2. This was done for 300,000,000 Monte Carlo iterations and the results are reported in the Appendix (along with the work up for computing the conditional probabilities for each system for each reduced form sign pattern as given in Equation (8)). The estimates found for these conditional probabilities are given in Table 3 below.

Table 3. Estimated Conditional Probabilities for System #1 and System #2.

sgn $\stackrel{⌢}{\pi }$ Base 10 Index	P(System #1|sgn $\stackrel{⌢}{\pi }$)	P(System #2|sgn $\stackrel{⌢}{\pi }$)
8914	0.00044	0.01214
8915	0.00014	0.00597
9878	0.00188	0.01303
9879	0.00085	0.00656
9938	0.00108	0.00679
9939	0.00020	0.00372
21925	0.00010	0.00725
26262	0.00291	0.01377
26263	0.00052	0.00334
38189	0.00278	0.03071
38249	0.00018	0.00773
39273	0.00221	0.02130
40237	0.00255	0.00915
40297	0.00216	0.00831
41562	0.00016	0.01259
41563	0.00013	0.01225
41682	0.00028	0.01057
41683	0.00009	0.00152
42586	0.00082	0.00580
42587	0.00067	0.01454
42646	0.00065	0.02276
42647	0.00044	0.00413
42706	0.00206	0.01270
42707	0.00074	0.00211
43610	0.00043	0.01508
43611	0.00023	0.01043
54573	0.00015	0.00676
56621	0.00057	0.00439

Table 3. Estimated Conditional Probabilities for System #1 and System #2.

sgn $\stackrel{⌢}{\pi }$ Base 10 Index	P(System #1|sgn $\stackrel{⌢}{\pi }$)	P(System #2|sgn $\stackrel{⌢}{\pi }$)
8914	0.00044	0.01214
8915	0.00014	0.00597
9878	0.00188	0.01303
9879	0.00085	0.00656
9938	0.00108	0.00679
9939	0.00020	0.00372
21925	0.00010	0.00725
26262	0.00291	0.01377
26263	0.00052	0.00334
38189	0.00278	0.03071
38249	0.00018	0.00773
39273	0.00221	0.02130
40237	0.00255	0.00915
40297	0.00216	0.00831
41562	0.00016	0.01259
41563	0.00013	0.01225
41682	0.00028	0.01057
41683	0.00009	0.00152
42586	0.00082	0.00580
42587	0.00067	0.01454
42646	0.00065	0.02276
42647	0.00044	0.00413
42706	0.00206	0.01270
42707	0.00074	0.00211
43610	0.00043	0.01508
43611	0.00023	0.01043
54573	0.00015	0.00676
56621	0.00057	0.00439

Inspection of Table 3 reveals that for each of the 28 reduced form sign patterns, system #2 is more likely than system #1. Since the prior probabilities of each system and the reduced form sign pattern are the same for each system, this inequality will be the same as the conditional probabilities for each reduced form sign pattern with respect to each system. Accordingly, relative likelihoods for the systems can be determined based upon the Table 2 frequency distributions even if the prior probabilities of the systems and the reduced form sign pattern cannot be estimated. The generality brought forward by these results is that lower entropy systems, which provide more information about the reduced form sign patterns, will be more likely if not falsified and their acceptance will be less prone to type 2 error regardless of the distribution of the structural error.

4. Examples

Three examples of the above analytic point of view are presented in this section. The first is a (famously) under-identified model of the relationships among advertising, market concentration, and price-cost margins. The second is the Oil Market Simulation (OMS), a model of multi-market equilibria that was utilized by the U.S. Department of Energy to determine the world oil price as an input to its long term forecasting system. Remarkably, the sign pattern of the reduced form for this model is entirely determined by the hypothesized sign patterns of the structural arrays. The third example is Klein’s Model I. This is a concise, macroeconomic model that has been widely utilized for pedagogical purposes in both the econometrics literature and the literature on qualitative analysis. We consider estimates of the structural forms for all of these models. Utilizing the Monte Carlo to simulate the corresponding signs and zero restrictions hypothesized for the structural arrays, we found that the sign pattern of the estimated reduced form was not possible given the hypothesized sign patterns of the structural arrays. Further, in all three cases, we found that the sign patterns of the estimated reduced forms were not possible for any structural sign patterns that imposed the zero restrictions proposed for the structural arrays, regardless of the signs of the nonzero entries in these arrays. In some, but not all cases, we can point to the inconsistencies responsible for rejecting the estimated reduced form sign pattern. For the cases in which we are unable to account for the falsification, we note that the point of using the Monte Carlo is to enable such an analysis numerically, when a more explicit, analytical approach is not readily practicable.

4.1. An Under-Identified Model of Industrial Organization

A model of the relationships among advertising, concentration, and price-cost margin was presented by Strickland and Weiss [15]. In the econometrics literature, e.g., Amemiya [16], Greene [17] and Gujarati [18] the paper is cited because the model fails the rank test for identification. The model is also presented in texts in industrial organization, e.g., Greer [19], Scherer and Ross [20], and Carlton and Perloff [21]. The model is also subjected to a qualitative analysis in Buck and Lady [22]. The model is expressed as βY = γZ, where the Y vector of endogenous variables has three entries: advertising divided by sales (Ad/S), concentration (C), and the price cost margin (M); and, the Z vector of exogenous variables has six entries: growth of the industry (Gr), a dummy for durable goods (Dur), minimum efficient scale divided by sales (MES/S), capital stock divided by sales (K/S), geographic dispersion of the industry (GD), and the percentage of industry sales that is represented by consumer demand (CD/S). The sign patterns of the structural arrays proposed by Strickland and Weiss are:

$$\text{sgn} \mathrm{\beta } = \left[\begin{array}{ccc}-& +& +\\ +& -& 0\\ +& +& -\end{array}\right]; and, \text{sgn} \mathrm{\gamma } = \left[\begin{array}{cccccc}-& ?& 0& 0& 0& -\\ 0& 0& -& 0& 0& 0\\ -& 0& -& -& +& 0\end{array}\right]$$

The “?” for γ₁₂ indicates that the authors were not sure about this sign; and, for the analysis reported on here, this entry is set both positive and negative in the quantitative Monte Carlo repetitions, each about half the time. In Buck and Lady [22], the reduced form of the model was estimated using data derived from the 1992 Census of Manufacturers and the corresponding National Input-Output Tables. 20 The sign pattern of this result is:

$$\text{sgn} \widehat{\mathrm{\pi }} = \left[\begin{array}{cccccc}+& +& -& -& -& +\\ +& -& +& +& +& +\\ +& -& +& +& -& +\end{array}\right]$$

In Buck and Lady [22], 10 of the 18 entries of the reduced form could be signed assuming that β was a stable matrix (which meant that its inverse would have all negative entries). Given this, the analysis proceeded based upon traditional qualitative principles directed at the signs in the reduced form individually. This was not done here. Instead, the Monte Carlo proceeded with quantitative draws for the structural sign patterns above, including the ambiguous (nonzero) sign of γ₁₂. It was found that no entry of the reduced form was signable analytically. Nevertheless, in tens of millions of iterations, the Monte Carlo only found 18 sign patterns for the corresponding reduced form. This result shows that the qualitatively specified structural hypothesis imposes extreme limits upon the possible reduced form sign pattern, since in principle there are 262,144 possible 3 × 6 sign patterns, barring zeros. As a default, assuming that each possible sign pattern is equally likely, the information measure yields a value of INFO% = 76.8, i.e., the hypothesized structural sign patterns (if true) provide 76.8% of the information provided by the fully specified reduced form sign pattern. As discussed, for a hypothesis with low entropy (i.e., high information content), the likelihood of type 1 error is relatively high and the likelihood of type 2 error is relatively low. Significantly, the sign pattern of the estimated reduced form was not one of the 18 found, falsifying the structural hypothesis. 21

We continued the analysis by assessing the limitations on the reduced form sign pattern provided by the zero restrictions in the structural arrays. This was done by setting all of the nonzeros in β and γ above to “?”. The quantitative Monte Carlo draws then proceeded to set these entries both positive and negative, independently, each about half the time. The imposed restrictions on the structure were of course reduced, compared to working with the hypothesized signs. Nevertheless, in tens of millions of samples only 4096 sign patterns for the reduced form were found (about 1.5% of the 262,144 possible 3 × 6 sign patterns for the reduced form). Treating these sign patterns as equally likely gave INFO% = 33.3 for the zero restrictions alone. Significantly, the sign pattern of the estimated reduced form was not found among these 4096 sign patterns, a somewhat decisive falsification of the structural hypothesis. Or, at the very least, an outcome that should prompt more analysis concerning the structural hypothesis.

4.2. The Oil Market Simulation (OMS)

The OMS (System’s Science, Inc. [23], EIA [24]) was used in the 1990s by the Department of Energy’s Energy Information Administration (EIA) in conjunction with a model of domestic energy markets in the preparation of annual long term forecasts and special studies as requested. The model’s output is the world oil price (WOP), given estimates of supply and demand in some number of regions world-wide (seven in the version of the model used by EIA). Four other equations give the demand for OPEC production as needed to balance world-wide supply and demand; and, additionally express the relationship between the WOP in terms of OPEC’s rate of capacity utilization. In general, the relationships used were nonlinear.

This model was subjected to a traditional (i.e., entry by entry of the reduced form) qualitative analysis by Hale and Lady [10]. They found that all the entries in the reduced form were signable, based upon the hypothesized sign patterns for the structural arrays. Further, this result was independent of the level of aggregation specified for the model, i.e., the number of regions considered, so long as the algebraic form and sign patterns of the regional supply and demand relationships remained the same. This analysis was reiterated in Hale et al. [9]. Lady and Buck [8] aggregated the model to only one non-OPEC region and we will work here with that aggregated version of the model. In these terms, the structural form of the model is expressed as,

$$\left[\begin{array}{cccccc}1& 0& 0& 0& 0& {\mathrm{\beta }}_{16}\\ 0& 1& 0& 0& 0& {\mathrm{\beta }}_{26}\\ {\mathrm{\beta }}_{31}& {\mathrm{\beta }}_{32}& 1& 0& 0& 0\\ 0& 0& {\mathrm{\beta }}_{43}& 1& 0& 0\\ 0& 0& 0& {\mathrm{\beta }}_{54}& 1& 0\\ 0& 0& 0& 0& {\mathrm{\beta }}_{65}& 1\end{array}\right] \left[\begin{array}{c}dD\\ dS\\ dDO\\ dCAPUT\\ dR\\ dWOP\end{array}\right] = \left[\begin{array}{cccc}{\mathrm{\gamma }}_{11}& 0& 0& 0\\ 0& {\mathrm{\gamma }}_{22}& 0& 0\\ 0& 0& 0& 0\\ 0& 0& {\mathrm{\gamma }}_{43}& 0\\ 0& 0& 0& 0\\ 0& 0& 0& {\mathrm{\gamma }}_{64}\end{array}\right] \left[\begin{array}{c}d{D}_{-1}\\ d{S}_{-1}\\ dMaxCap\\ dWO{P}_{-1}\end{array}\right]$$

For this notation, the lower case d is the differential operator, D is world oil demand, S is non-OPEC world oil supply, DO is the demand for OPEC oil, CAPUT is the rate of OPEC capacity utilization, R is the percentage change (in decimal) of the current WOP over last year’s, MaxCap is maximum OPEC capacity and WOP is the world oil price. The hypothesized sign patterns for the structural arrays are given below,

$$\text{sgn} \mathrm{\beta } = \left[\begin{array}{cccccc}+& 0& 0& 0& 0& +\\ 0& +& 0& 0& 0& -\\ -& +& +& 0& 0& 0\\ 0& 0& -& +& 0& 0\\ 0& 0& 0& -& +& 0\\ 0& 0& 0& 0& -& +\end{array}\right], and \text{sgn} \mathrm{\gamma } = \left[\begin{array}{cccc}+& 0& 0& 0\\ 0& +& 0& 0\\ 0& 0& 0& 0\\ 0& 0& -& 0\\ 0& 0& 0& 0\\ 0& 0& 0& +\end{array}\right]$$

Following Hale and Lady’s [10] qualitative methodology, the reduced form for this aggregated version of the model must have the following sign pattern,

$$\text{sgn} \mathrm{\pi } = \left[\begin{array}{cccc}+& +& +& -\\ +& +& -& +\\ +& -& +& -\\ +& -& -& -\\ +& -& -& -\\ +& -& -& +\end{array}\right]\text{}$$

In brief, this is because an analysis of the expansion of det β reveals that all terms have the same sign; and further, for each cofactor in β’s adjoint, all terms are the same sign, positive or negative depending. All of this is based entirely upon the hypothesized structural sign patterns, independent of the magnitudes of the nonzero entries. For this result, INFO% = 100. If the hypothesized structure is true, nothing about the reduced form sign pattern is learned by estimating it. Lady and Buck [8] then estimated the reduced form based upon recent data with the following result (Table 4),

Table 4. OMS Estimated (1983–2006) Reduced Form Sign Pattern.

	dD-1	dS-1	dMaxCap	dWOP-1
dD	+	+	+	−
dS	−*	+	−	−*
dDO	+	+*	+	+*
dCaput	−*	−	+*	+*
dR	+	−	+*	−
dWOP	+	−	−	−*

“*” indicates a “wrong” sign compared to the predicted reduced form sign pattern.

Table 4. OMS Estimated (1983–2006) Reduced Form Sign Pattern.

	dD-1	dS-1	dMaxCap	dWOP-1
dD	+	+	+	−
dS	−*	+	−	−*
dDO	+	+*	+	+*
dCaput	−*	−	+*	+*
dR	+	−	+*	−
dWOP	+	−	−	−*

“*” indicates a “wrong” sign compared to the predicted reduced form sign pattern.

There are nine entries with different signs than those required by the structural hypothesis. The basis for the inconsistencies corresponding to these terms for this model is immediate, given the outcome of the qualitative analysis of the structural arrays.

The zero restrictions in the structural hypothesis (as above) can be tested by replacing each nonzero in the structural arrays with “?”. As before, in the quantitative Monte Carlo iterations, these terms are independently chosen positive or negative, each about half of the time. Based on this, the Monte Carlo was repeated tens of millions of times. For this, only 4099 reduced form sign patterns were found, out of the possible 2²⁴ = 16,777,216 outcomes for the 6 × 4 sign patterns that the estimated reduced form could take on (barring zeros). The zero restrictions thus provide significant limitations on the outcome of the reduced form estimation and INFO% = 50 for this case. Significantly, the estimated reduced form sign pattern was not among those found. As before, this outcome should at least prompt significant reconsideration of the hypothesized structure.

4.3. Klein’s Model I

Klein’s Model I (Klein [25]) is an over-identified model that has appeared in the econometrics literature, e.g., Berndt [26] and Greene [17] for a variety of pedagogical reasons. It has also appeared in a number of papers on qualitative analysis, e.g., Maybee and Weiner [27], Hale et al. [9], Lady [28], and Buck and Lady [22]. In a qualitative context, the model was chosen to illustrate that for sufficiently sparse structural arrays (perhaps with other convenient features) entries in the reduced form would be signable due to adding a small number of (as called) “side conditions” on the relative values of some of the nonzero entries proposed for the structure. The idea, in brief, is that an inspection of the terms in the expansions of β’s determinant and cofactors might find that “almost” all are of the same sign. Given this, the embellished structural hypothesis, i.e., sign pattern plus side condition(s), would result in (at least some) entries in the corresponding reduced form being signable; and, this in turn could be tested by comparison to the sign pattern of the estimated reduced form.

Suppressing the error vector, Klein’s model is expressed as,

$$\mathrm{\beta }Y=\mathrm{\gamma }Z$$

where

$$\left[\begin{array}{ccccccc}-1& 0& a_1& 0& a_2& 0& 0\\ 0& -1& 0& 0& b_1& 0& 0\\ 0& 0& -1& 0& 0& 0& c_1\\ 1& 1& 0& -1& 0& 0& 0\\ 0& 0& 0& 1& -1& -1& 0\\ 0& 0& 1& 0& 0& -1& 0\\ 0& 0& 0& 1& 0& 0& -1\end{array}\right]\left[\begin{array}{c}C\\ I\\ W_1\\ Y\\ P\\ W\\ E\end{array}\right]=\left[\begin{array}{ccccccc}-a_1& -a_3& 0& 0& 0& 0& 0\\ 0& -b_2& -b_3& 0& 0& 0& 0\\ 0& 0& 0& -c_2& -c_3& 0& 0\\ 0& 0& 0& 0& 0& 1& -1\\ 0& 0& 0& 0& 0& 0& 0\\ -1& 0& 0& 0& 0& 0& 0\\ 1& 0& 0& 0& 0& -1& 0\end{array}\right]\left[\begin{array}{c}{W}_2\\ P_{-1}\\ K_{-1}\\ E_{-1}\\ Year\\ TX\\ G\end{array}\right]$$

In this structure the endogenous variables are private consumption (C), investment (I), the private wage bill (W₁), income (Y), profits or nonwage income (P), the sum of private and government wages (W), and private product (E); and the exogenous variables are the government wage bill (W₂), lagged profits (P₋₁), end of last period capital stock (K₋₁), lagged private product (E₋₁), years since 1931 (Year), taxes (TX), and government consumption (G).

The sign patterns of the arrays proposed by Klein [25] are as follows,

$$\text{sgn} \mathrm{\beta }=\left[\begin{array}{ccccccc}-& 0& +& 0& +& 0& 0\\ 0& -& 0& 0& +& 0& 0\\ 0& 0& -& 0& 0& 0& +\\ +& +& 0& -& 0& 0& 0\\ 0& 0& 0& +& -& -& 0\\ 0& 0& +& 0& 0& -& 0\\ 0& 0& 0& +& 0& 0& -\end{array}\right]\text{and} \text{sgn} \mathrm{\gamma }=\left[\begin{array}{ccccccc}-& -& 0& 0& 0& 0& 0\\ 0& -& +& 0& 0& 0& 0\\ 0& 0& 0& -& -& 0& 0\\ 0& 0& 0& 0& 0& +& -\\ 0& 0& 0& 0& 0& 0& 0\\ -& 0& 0& 0& 0& 0& 0\\ +& 0& 0& 0& 0& -& 0\end{array}\right]$$

The sign pattern of the estimated reduced form of Klein’s model is reported in Goldberger [29], using data from 1921–1941, as

$$Estimated \text{sgn} \widehat{\mathrm{\pi }} = \left[\begin{array}{ccccccc}-& +& -& +& +& -& +\\ -& +& -& -& +& -& +\\ -& +& -& +& +& -& +\\ -& +& -& +& +& -& +\\ -& +& -& +& +& -& +\\ -& +& -& +& +& -& +\\ -& +& -& +& +& -& +\end{array}\right]$$

Without side conditions, no entry in the reduced form is signable, based upon the hypothesized sign patterns for the structural arrays. The size of this model is too large for our (PC-based) computing platform to enumerate all possible reduced form sign patterns, but repeated quantitative samples of the hypothesized structure can be taken and the outcome for the corresponding reduced form can be compared to a proposed reduced form, such as that above as reported in Goldberger. This was done in Buck and Lady [8]. Tens of millions of samples were taken and the above reduced form sign pattern was not found. Buck and Lady [8] (p. 2826) then found inconsistencies in the hypothesized structural arrays that made certain of the signs in the estimated reduced form impossible. It was patterns of signs that were at issue, since no individual entry was signable, e.g., although not individually signable, π₂₄ and π₂₅ cannot have opposite signs.

We have gone further and assessed the zero restrictions and the estimable entries of the structural arrays. As before, nonzeros were replaced by “?” for which both positive and negative values were chosen, each about half the time. The structural arrays were configured for the analysis as given below,

$$\text{sgn} \mathrm{\beta }=\left[\begin{array}{ccccccc}-& 0& ?& 0& ?& 0& 0\\ 0& -& 0& 0& ?& 0& 0\\ 0& 0& -& 0& 0& 0& ?\\ +& +& 0& -& 0& 0& 0\\ 0& 0& 0& +& -& -& 0\\ 0& 0& +& 0& 0& -& 0\\ 0& 0& 0& +& 0& 0& -\end{array}\right]\text{and} \text{sgn} \mathrm{\gamma }=\left[\begin{array}{ccccccc}?& ?& 0& 0& 0& 0& 0\\ 0& ?& ?& 0& 0& 0& 0\\ 0& 0& 0& ?& ?& 0& 0\\ 0& 0& 0& 0& 0& +& -\\ 0& 0& 0& 0& 0& 0& 0\\ -& 0& 0& 0& 0& 0& 0\\ +& 0& 0& 0& 0& -& 0\end{array}\right],$$

Tens of millions of samples were taken and the sign pattern of the reduced form sign pattern estimated by Goldberger was not found. This not only calls the hypothesized structure into question, but also makes the continuation of the analysis by estimating the values of appropriate entries of the hypothesized structure problematical. In particular, if the zero restrictions (and values for certain of the other entries) are imposed when constructing the structural coefficient estimates, the result could not possibly have generated the data upon which the estimated reduced form is based (absent errors). Specifically, with the zero restrictions imposed, all possible sign patterns for the structure could not have generated the sign pattern of the estimated reduced form.

5. Conclusions

Economics as a science should produce meaningful theorems that when true limit the range of data outcomes. The purpose of this paper is to demonstrate the potential of a qualitative analysis in evaluating models when confronted with data. A qualitative analysis is proposed as a complement to other econometric methods that depend upon more quantitative features of the data to be analyzed. The algorithmic methods used in illustrating our points are indicative and not definitive.

The use of qualitative methods can be compelling to the degree to which the theory does not usually provide a great deal of information beyond a system’s inference structure and the directions of influence among its variables. Unlike the bulk of the literature on such analyses, we show that insightful, and in the case of falsification, decisive, results are always possible and do not depend upon unusually restrictive or uncommon features of the mathematical content of a model. Indeed, even a partially specified structural model can be falsified by as few as one reduced form result.

The process involves using a model’s structural sign pattern information to assess its potential for falsification (and its associated type 1 error) and for its acceptance (and its associated type 2 error). Nevertheless, in the particular case of an estimated reduced form’s sign pattern falsifying the zero restrictions of a hypothesized structural form, the estimation of the structural arrays with the zero restrictions imposed will always lead to a mistake, i.e., the reduced form sign pattern is impossible for estimated structural arrays with the zero restrictions imposed.

The proposed qualitative analysis can enable the information content of a model to be estimated, based upon the entropy of the distribution of possible reduced form sign patterns that corresponds to the hypothesized structural sign patterns. As a model’s information content is greater, i.e., as it imposes greater limits on the possible outcomes of the reduced form sign patterns, the model’s falsification is more subject to type 1 error and the model’s acceptance is less subject to type 2 error.