# A New Approach to Model Verification, Falsification and Selection

^{*}

## Abstract

**:**

## 1. Introduction

^{i}(Y, Z) = 0, i = 1, 2, …, n

^{−1}γ. (4) is usually called the reduced form, from which the unknowns in Equation (3) are subsequently derived. Given this, the issue of falsification becomes that of the limitations on the entries of the estimated π = β

^{−1}γ, denoted by $\stackrel{\u2322}{\mathrm{\pi}}$, as derived from the theoretical proposals about the entries of {β, γ}.

^{−1}γ and finding that certain entries of π must have particular signs, based entirely upon the sign patterns of β and γ, independent of the magnitudes of their entries. If so, then when a $\stackrel{\u2322}{\mathrm{\pi}}$ is realized, if the required signs do not show up, the theory has been falsified. 7 On reflection, Samuelson proposed that as a practical matter, a qualitative analysis was extremely unlikely to be successful. To work, all of the potentially millions of terms in the expansions of β’s determinant and cofactors necessary in computing β

^{−1}would all have to have the same sign, independent of magnitudes. Samuelson viewed this as too unlikely to be taken seriously. Accordingly, he proposed other sources for limitations on the signs, of π, e.g., that β is a stable matrix or is derived from the second order conditions to a (perhaps constrained) optimization problem.

- -
- a specification of the sign patterns of the structural arrays always limits the possible sign patterns that can be taken on by the estimated reduced form;
- -
- the falsifiability of hypothesized sign patterns for the structural arrays is independent of identifiability 9;
- -
- zero restrictions specified for the structural arrays can be falsified independent of the sign pattern of the nonzero entries. When this happens, multi-stage least squares will always be in error when providing estimates of the structural arrays, i.e., the sign patterns of the estimated structural arrays with zero restrictions imposed are impossible, given the sign pattern of the estimated reduced form;
- -
- a partially specified structural hypothesis sometimes can be falsified by a small number of estimated reduced form equations, even as few as one;
- -
- the conditional probability that an hypothesized structural sign pattern is actually true when not falsified can be estimated using Bayes’ formula;
- -
- the relative likelihood of competing structural hypotheses can be assessed, given the sign pattern of the estimated reduced form, based upon their qualitative characteristics; and,
- -
- hypothesized structural arrays can be assessed for their information content using Shannon’s [7] measure of entropy.

## 2. Qualitative Falsification

^{−1}(or more generally sgn π) be signable based upon sgn β (and more generally, also sgn γ). Instead, patterns of signs in sgn π may not be possible, even though no individual entry is signable. In this section, this principle is generalized compared to this earlier work. In the next section, the new issue of verification is presented.

^{−1}. For the other eight cases, there are only two possible sign patterns for β

^{−1}; however, no entry of sgn β

^{−1}is the same for these two possibilities, since they are the negative of each other. Accordingly, for any of these eight cases, an hypothesized sgn β would be falsified by sgn $\stackrel{\u2322}{\mathrm{\pi}}$ (assuming for the moment that γ = I), if it took on one of the 14 sign patterns that are not possible for the hypothesized sgn β, even though no individual entry of sgn β

^{−1}is signable.

^{−1}γ is computed, and the sign pattern outcome of this computation is saved. The iterations/repetitions can be done millions of times. For sufficiently small systems, the set of all possible sign patterns that the reduced form might take on, given the hypothesized sign pattern of the structural arrays, can be tabulated with a vanishingly small potential for error as the number of repetitions increases. 12 A central point of the analysis is that this set of allowed sign patterns always limits the sign patterns the reduced form can take on. Given this, there are always sign patterns that the reduced form array might take on that are impossible, given the hypothesized structural sign patterns, and if so, the given structure is thus falsified.

^{−1}can take (barring zeros as we always do for the examples we provide, although this is not necessary). Of these, only 102 were found by the Monte Carlo in tens of millions of iterations. Given this, it appears that 410 sign patterns for sgn β

^{−1}are impossible for the hypothesized sgn β. The analytic basis for some sign patterns being impossible can be quickly developed. Let the qualitative inverse of sgn β be defined as:

**Definition (qualitative inverse)**. Sgn π is a qualitative inverse of sgn β if and only if there exist magnitudes for the entries of β, consistent with the given sgn β, such that sgn β

^{−1}= sgn π.

- Sgn π is a qualitative inverse of sgn β only if it is possible that i) βπ = I; and, ii) πβ = I.

^{th}row of β and the j

^{th}column of π share a pair of common nonzero entries of the same sign for i = j and both this and another pair of common nonzero entries of the opposite sign for i ≠ j; and, also for the i

^{th}row of π and the j

^{th}column of β for ii) above.

_{31}= 0. The remaining entries marked “?” are nonzero, but can be either positive or negative. For this case, assume that, based on the data 14

_{31}= 0, since the (1,3) cofactor is signable, independent of magnitudes. Of the 512 possible sign patterns that sgn $\stackrel{\u2322}{\mathrm{\pi}}$ might have taken on from the data, 32 of them have the above 2 × 2 sign pattern corresponding to the (1,3) cofactor. As discussed above, altogether there are eight such signable sign patterns for this cofactor. As a result, 256 of the possible 512 sign patterns that sgn $\stackrel{\u2322}{\mathrm{\pi}}$ might take on falsify the given zero restriction, independent of the signs of the other entries of the proposed sgn β or estimated sgn $\stackrel{\u2322}{\mathrm{\pi}}$. If any of these show up for sgn $\stackrel{\u2322}{\mathrm{\pi}}$, then however $\widehat{\mathrm{\beta}}$ is derived, it is a mistake, i.e., impossible, if the zero restriction is imposed. 15

^{10}= 1024 distinct sign patterns. Accordingly, adjoint β can only take on 1024 distinct sign patterns. These in turn can be divided into two groups for which the patterns in one group are the negative of the patterns in the other group. As it happens, the determinant is signed for half of the 1024 cases, but this does not make any difference. Given the possible sign patterns of the adjoint, there are only 1024 possible sign patterns for sgn β

^{−1}, regardless of how the determinant’s (nonzero) sign plays out. Yet, for a 5 × 5 array, there are 33,554,432 possible sign patterns, barring zeros. For this case, of these, 33,553,408 are impossible, based upon the zero restrictions on β alone (plus γ = I). For this inference structure, the proportion of qualitative inverses among the possible sign patterns decreases significantly as n increases, since there are 2

^{nn}possible sign patterns for which only 2

^{2n}are qualitative inverses. It is worth emphasizing the points just made. If the zero restrictions of the structural hypothesis are falsified by the sign pattern of the estimated reduced form, then the estimated structural arrays with these zero restrictions imposed are impossible, given the sign pattern of the estimated reduced form. Since current practice does not include conducting a qualitative analysis, this leaves the prospect of the outcome of many estimations as unknowingly being utterly impossible.

_{ij}] be the adjoint of β.

**Definition (adjoint-consistent).**If sgn B

_{ij}= sgn B

_{uv}(resp. sgn B

_{ij}= −sgn B

_{uv}) independent of magnitudes, then sgn π is adjoint-consistent with sgn β if and only if sgn π

_{ij}= sgn π

_{uv}(resp. sgn π

_{ij}= −sgn π

_{uv}).

- iii)
- An estimated sgn $\stackrel{\u2322}{\mathrm{\pi}}$ is a qualitative inverse of sgn β only if the estimated sgn $\stackrel{\u2322}{\mathrm{\pi}}$ is adjoint consistent with sgn β.

^{−1}computed, and it is repeatedly found that, given this, the given sgn $\stackrel{\u2322}{\mathrm{\pi}}$ ≠ sgn β

^{−1}, then the inference is that the given sgn $\stackrel{\u2322}{\mathrm{\pi}}$ is not a qualitative inverse of the hypothesized sgn β. The chances of incorrectly falsifying the hypothesized structure are made ever smaller by increasing the number of quantitative Monte Carlo iterations taken, but the chances are never absolutely zero. For this reason, once falsification is strongly suspected based upon the Monte Carlo, a direct investigation of (5) can be attempted, hoping to analytically confirm what the Monte Carlo suggests. Examples of this approach are given in the appendix of Buck and Lady [11].

^{−1}, only 4096 are possible, given the zero restrictions, independent of the signs of the nonzeros. To reiterate this point yet again, if the sign pattern of the estimated reduced form had any of the other 61,440 sign patterns, then any sign pattern proposed for sgn $\widehat{\mathrm{\beta}}$ derived from the observed reduced form with these zero restrictions imposed would be a mistake. Direct analysis revealed that only 625 of the possible sign patterns for sgn β

^{−1}satisfied i) above, 400 satisfied ii), and only 28 satisfied both. Further analysis revealed that entries of β adjoint were signable, as given below,

^{−1}that always result in these restrictions being translated to the reduced form in the more general cases.

**Qualitative Falsification and Type 1 Error.**As noted above, a structural hypothesis may be rejected because the sign pattern of the estimated reduced form is not found among those admissible reduced forms generated by the Monte Carlo algorithm that we have employed. Given this, further investigation is called for in order to reveal the inconsistency as related to a solution to (5). If this is found, i.e., if the sign pattern of the estimated reduced form is demonstrably impossible, given the hypothesized structural sign pattern, the result may be due to errors in the data, and to reject the structural hypothesis would constitute a type 1 error. The issue of this subsection is to assess the propensity for a structural hypothesis to be incorrectly rejected.

^{−1}γ. If the sign pattern found for $\tilde{\pi}$ is not one of the (535 for system #1) possible sign patterns, then the associated falsification is a type 1 error due to sampling error 17. For this example, this can only happen if the error term that was drawn puts a negative value in one of the array entries when added to the sampled quantitative value that was supposed to be positive. Such a “wrong” sign is intended to represent errors in the data. As the number of possible sign patterns permitted by the structural sign patterns is smaller, the potential for type 1 error due to sampling errors is larger. For system #1 above, with the standard deviation of the “error term” = 0.2, there were 4.68% of 500,000 quantitative Monte Carlo repetitions which falsified system #1’s sign pattern. This result is dependent in part on the distributional assumption employed in the Monte Carlo repetitions. The influence of this approach can be further investigated by the Monte Carlo simulation, but it is beyond our scope to do so.

Standard Deviation of the Error Distribution | Proportion Falsified (500,000 Samples) | |
---|---|---|

System #1 | System #2 | |

0.2 | 0.0468 | 0.0672 |

0.4 | 0.0976 | 0.1395 |

0.6 | 0.1420 | 0.1964 |

0.8 | 0.1847 | 0.2604 |

1 | 0.2366 | 0.3256 |

**Type 1 Error and Entropy.**The result of the last subsection can be generalized somewhat. Consider that for the sign pattern of an n × m reduced form array, barring zeros, that there are nm bits of information (say “1” for “+” and “0” for “−“), one bit for each entry. Given a hypothesized structural sign pattern, the possible reduced form sign patterns are limited to Q < 2

^{nm}; and for these, given the algebra of computing π = β

^{−1}γ and the distributional rules for assigning values to the nonzeros of {β, γ}, each possible reduced form sign pattern has a particular likelihood of occurrence. Let F

_{i}be the frequency of occurrence of the i

^{th}possible reduced form sign pattern (see below for how “i” is computed). Given this, the entropy of the distribution (Shannon [7], see also Cover and Thomas [13]) is given by:

_{i}) is to the base 2. For example, for one simulation of system #1 above with 3,000,000 samples taken, the entropy of the resulting distribution of (the 535 possible) reduced form sign patterns was 8.18. The unit of this measure is bits. It measures the information achieved by estimating the reduced form. The maximum entropy is 16 for the 4 × 4 sign pattern (barring zeros) and would be the model’s entropy if all of the 65,536 4 × 4 reduced form sign patterns were possible and equally likely. The minimum entropy is zero if only one sign pattern for the reduced form is possible. Our algorithm for drawing the quantitative realizations of the structural sign pattern is designed to find the possible reduced form sign patterns, but has no econometric justification for the distributional (uniform) rule used in the iterations. Nevertheless, we will use the frequency distributions found in the analysis below. If a default entropy is computed which “simply” treats all of the possible reduced form sign patterns as equally likely, then doing this for system #1 causes the entropy measure to increase a small amount to 9.06. The key item driving the measure is the number of possible sign patterns, as opposed to the frequency distribution of the possibilities.

## 3. Qualitative Verification

$\stackrel{\u2322}{\pi}$ Base 10 Index | Sign Pattern of the Reduced Form Row by Row | Frequency in Sample of 3,000,000 | |||
---|---|---|---|---|---|

System #1 | System #2 | ||||

Count | Frequency: p($\stackrel{\u2322}{\pi}$|sys #1) | Count | Frequency: p($\stackrel{\u2322}{\pi}$|sys #2) | ||

8914 | − − + − − − + − + + − + − − + − | 3101 | 0.00103 | 84455 | 0.02815 |

8915 | − − + − − − + − + + − + − − + + | 1661 | 0.00055 | 70168 | 0.02339 |

9878 | − − + − − + + − + − − + − + + − | 2448 | 0.00082 | 17047 | 0.00568 |

9879 | − − + − − + + − + − − + − + + + | 3642 | 0.00121 | 28049 | 0.00935 |

9938 | − − + − − + + − + + − + − − + − | 2329 | 0.00078 | 14731 | 0.00491 |

9939 | − − + − − + + − + + − + − − + + | 4576 | 0.00153 | 84842 | 0.02828 |

21925 | − + − + − + − + + − + − − + − + | 1933 | 0.00064 | 122556 | 0.04850 |

26262 | − + + − − + + − + − − + − + + − | 10809 | 0.00360 | 51052 | 0.01702 |

26263 | − + + − − + + − + − − + − + + + | 6590 | 0.00220 | 42724 | 0.01424 |

38189 | + − − + − + − + − − + − + + − + | 1703 | 0.00057 | 18889 | 0.0063 |

38249 | + − − + − + − + − + + − + − − + | 879 | 0.00029 | 36755 | 0.01225 |

39273 | + − − + + − − + − + + − + − − + | 7996 | 0.00267 | 77355 | 0.02578 |

40237 | + − − + + + − + − − + − + + − + | 2442 | 0.00081 | 8728 | 0.00291 |

40297 | + − − + + + − + − + + − + − − + | 4291 | 0.00143 | 16501 | 0.0055 |

41562 | + − + − − − + − − + − + + − + − | 560 | 0.00019 | 44339 | 0.01478 |

41563 | + − + − − − + − − + − + + − + + | 1061 | 0.00035 | 99335 | 0.03311 |

41682 | + − + − − − + − + + − + − − + − | 1137 | 0.00038 | 43087 | 0.01436 |

41683 | + − + − − − + − + + − + − − + + | 1713 | 0.00057 | 28838 | 0.00961 |

42586 | + − + − − + + − − + − + + − + − | 2884 | 0.00096 | 20251 | 0.00675 |

42587 | + − + − − + + − − + − + + − + + | 7693 | 0.00256 | 165480 | 0.05516 |

42646 | + − + − − + + − + − − + − + + − | 5229 | 0.00174 | 182054 | 0.06068 |

42647 | + − + − − + + − + − − + − + + + | 10518 | 0.00351 | 98504 | 0.03283 |

42706 | + − + − − + + − + + − + − − + − | 7833 | 0.00261 | 48281 | 0.01609 |

42707 | + − + − − + + − + + − + − − + + | 26405 | 0.00880 | 75320 | 00.02511 |

43610 | + − + − + − + − − + − + + − + − | 2029 | 0.00068 | 70759 | 0.02359 |

43611 | + − + − + − + − − + − + + − + + | 1690 | 0.00056 | 76690 | 0.02556 |

54573 | + + − + − + − + − − + − + + − + | 1280 | 0.00043 | 56623 | 0.01887 |

56621 | + + − + + + − + − − + − + + − + | 7170 | 0.00239 | 55421 | 0.01847 |

^{9}) possible structural sign patterns. Assume that all of these are equally likely; hence, the prior probability for each such system = 1/512 = 0.00195, including systems #1 and #2. These 512 structural sign patterns might be termed the “universe” from which the specific structural sign patterns were selected, i.e., as what the theory dictates and (evidently) leaves undecided. 19 This universe can be sampled and the frequency found for each of the 28 reduced form patterns enumerated in Table 2. This was done for 300,000,000 Monte Carlo iterations and the results are reported in the Appendix (along with the work up for computing the conditional probabilities for each system for each reduced form sign pattern as given in Equation (8)). The estimates found for these conditional probabilities are given in Table 3 below.

sgn $\stackrel{\u2322}{\pi}$ Base 10 Index | P(System #1|sgn $\stackrel{\u2322}{\pi}$) | P(System #2|sgn $\stackrel{\u2322}{\pi}$) |
---|---|---|

8914 | 0.00044 | 0.01214 |

8915 | 0.00014 | 0.00597 |

9878 | 0.00188 | 0.01303 |

9879 | 0.00085 | 0.00656 |

9938 | 0.00108 | 0.00679 |

9939 | 0.00020 | 0.00372 |

21925 | 0.00010 | 0.00725 |

26262 | 0.00291 | 0.01377 |

26263 | 0.00052 | 0.00334 |

38189 | 0.00278 | 0.03071 |

38249 | 0.00018 | 0.00773 |

39273 | 0.00221 | 0.02130 |

40237 | 0.00255 | 0.00915 |

40297 | 0.00216 | 0.00831 |

41562 | 0.00016 | 0.01259 |

41563 | 0.00013 | 0.01225 |

41682 | 0.00028 | 0.01057 |

41683 | 0.00009 | 0.00152 |

42586 | 0.00082 | 0.00580 |

42587 | 0.00067 | 0.01454 |

42646 | 0.00065 | 0.02276 |

42647 | 0.00044 | 0.00413 |

42706 | 0.00206 | 0.01270 |

42707 | 0.00074 | 0.00211 |

43610 | 0.00043 | 0.01508 |

43611 | 0.00023 | 0.01043 |

54573 | 0.00015 | 0.00676 |

56621 | 0.00057 | 0.00439 |

## 4. Examples

#### 4.1. An Under-Identified Model of Industrial Organization

_{12}indicates that the authors were not sure about this sign; and, for the analysis reported on here, this entry is set both positive and negative in the quantitative Monte Carlo repetitions, each about half the time. In Buck and Lady [22], the reduced form of the model was estimated using data derived from the 1992 Census of Manufacturers and the corresponding National Input-Output Tables. 20 The sign pattern of this result is:

_{12}. It was found that no entry of the reduced form was signable analytically. Nevertheless, in tens of millions of iterations, the Monte Carlo only found 18 sign patterns for the corresponding reduced form. This result shows that the qualitatively specified structural hypothesis imposes extreme limits upon the possible reduced form sign pattern, since in principle there are 262,144 possible 3 × 6 sign patterns, barring zeros. As a default, assuming that each possible sign pattern is equally likely, the information measure yields a value of INFO% = 76.8, i.e., the hypothesized structural sign patterns (if true) provide 76.8% of the information provided by the fully specified reduced form sign pattern. As discussed, for a hypothesis with low entropy (i.e., high information content), the likelihood of type 1 error is relatively high and the likelihood of type 2 error is relatively low. Significantly, the sign pattern of the estimated reduced form was not one of the 18 found, falsifying the structural hypothesis. 21

#### 4.2. The Oil Market Simulation (OMS)

dD_{-}_{1} | dS_{-}_{1} | dMaxCap | dWOP_{-1} | |
---|---|---|---|---|

dD | + | + | + | − |

dS | −* | + | − | −* |

dDO | + | +* | + | +* |

dCaput | −* | − | +* | +* |

dR | + | − | +* | − |

dWOP | + | − | − | −* |

^{24}= 16,777,216 outcomes for the 6 × 4 sign patterns that the estimated reduced form could take on (barring zeros). The zero restrictions thus provide significant limitations on the outcome of the reduced form estimation and INFO% = 50 for this case. Significantly, the estimated reduced form sign pattern was not among those found. As before, this outcome should at least prompt significant reconsideration of the hypothesized structure.

#### 4.3. Klein’s Model I

_{1}), income (Y), profits or nonwage income (P), the sum of private and government wages (W), and private product (E); and the exogenous variables are the government wage bill (W

_{2}), lagged profits (P

_{−1}), end of last period capital stock (K

_{−1}), lagged private product (E

_{−1}), years since 1931 (Year), taxes (TX), and government consumption (G).

_{24}and π

_{25}cannot have opposite signs.

## 5. Conclusions

## Acknowledgment

## Conflicts of Interest

## Appendix: A Qualitative Tool Kit

^{−1}γ is computed. Given this, it is a simple procedure to count the number of times each entry of π is positive and the number of times negative (if zero the sample is discarded). This is an extremely robust method to determine if any of the entries are signable, since if so they will always turn out to have the same sign, regardless of the size of the sample. Further, these simple counts will also reveal if entries of π always have the same, or different signs, as dictated by entries of β’s adjoint being signable, since if so the counts of such will be the same, independent of the size of the sample. If the sign pattern of an estimated reduced form is in-hand, it is additionally straight-forward to see if the sign patterns of any of the sampled reduced forms are the same as the estimated reduced form. The size of the arrays is not particularly limiting for this “simple search”, except for the time needed to construct samples of a given size. If the sign pattern of the estimated reduced form is not found after repeated Monte Carlo draws, then the counts of positive and negative entries in the sampled reduced form may reveal the reason, i.e., due to signable entries or entries required to always have the same or different signs. Typically (unfortunately), the sampled reduced forms will not provide these regularities and call for further analysis if the sign pattern of the estimated reduced form is not found, suggesting that the structural hypothesis is falsified.

^{−1}γ is one of several obvious features of the analysis that calls for substantial development and algorithmic support. One immediate technique is to check to see if subgroupings of the reduced form sign pattern are resulting in the apparent falsification. This is accomplished by only checking such a subgrouping when comparing the sampled reduced forms to the estimated reduced form. When such subgroupings are enumerated, then the algebra of computing the reduced form can be written out with the appropriate focus to determine the problem of satisfying the systems (5). This was done in Lady and Buck [8] and Buck and Lady [11].

^{mn}possible reduced form sign patterns. The (long) integer used in our computing platform is limited to +/− 2

^{31}. Accordingly, we cannot tabulate indexed counts of the reduced form sign patterns except for mn < 30. This limitation can be mitigated by using other computing platforms or indexing schemes.

sgn $\stackrel{\u2322}{\pi}$ Base 10 Index | P(sgn $\stackrel{\u2322}{\pi}$) | P(Sys#1) | P(sgn $\stackrel{\u2322}{\pi}$|Sys #1) | P(Sys#1/sgn $\stackrel{\u2322}{\pi}$) |
---|---|---|---|---|

8914 | 0.00452 | 0.00195 | 0.00103 | 0.00044 |

8915 | 0.00764 | 0.00195 | 0.00055 | 0.00014 |

9878 | 0.00085 | 0.00195 | 0.00082 | 0.00188 |

9879 | 0.00278 | 0.00195 | 0.00121 | 0.00085 |

9938 | 0.00141 | 0.00195 | 0.00078 | 0.00108 |

9939 | 0.01481 | 0.00195 | 0.00153 | 0.00020 |

21925 | 0.01304 | 0.00195 | 0.00064 | 0.00010 |

26262 | 0.00241 | 0.00195 | 0.00360 | 0.00291 |

26263 | 0.00831 | 0.00195 | 0.00220 | 0.00052 |

38189 | 0.00040 | 0.00195 | 0.00057 | 0.00278 |

38249 | 0.00309 | 0.00195 | 0.00029 | 0.00018 |

39273 | 0.00236 | 0.00195 | 0.00267 | 0.00221 |

40237 | 0.00062 | 0.00195 | 0.00081 | 0.00255 |

40297 | 0.00129 | 0.00195 | 0.00143 | 0.00216 |

41562 | 0.00229 | 0.00195 | 0.00019 | 0.00016 |

41563 | 0.00527 | 0.00195 | 0.00035 | 0.00013 |

41682 | 0.00265 | 0.00195 | 0.00038 | 0.00028 |

41683 | 0.01232 | 0.00195 | 0.00057 | 0.00009 |

42586 | 0.00227 | 0.00195 | 0.00096 | 0.00082 |

42587 | 0.00740 | 0.00195 | 0.00256 | 0.00067 |

42646 | 0.00520 | 0.00195 | 0.00174 | 0.00065 |

42647 | 0.01549 | 0.00195 | 0.00351 | 0.00044 |

42706 | 0.00247 | 0.00195 | 0.00261 | 0.00206 |

42707 | 0.02324 | 0.00195 | 0.00880 | 0.00074 |

43610 | 0.00305 | 0.00195 | 0.00068 | 0.00043 |

43611 | 0.00478 | 0.00195 | 0.00056 | 0.00023 |

54573 | 0.00544 | 0.00195 | 0.00043 | 0.00015 |

56621 | 0.00821 | 0.00195 | 0.00239 | 0.00057 |

sgn $\stackrel{\u2322}{\pi}$ Base 10 Index | P(sgn $\stackrel{\u2322}{\pi}$) | P(Sys#2) | P(sgn $\stackrel{\u2322}{\pi}$|Sys #2) | P(Sys#2/sgn $\stackrel{\u2322}{\pi}$) |
---|---|---|---|---|

8914 | 0.00452 | 0.00195 | 0.02815 | 0.01214 |

8915 | 0.00764 | 0.00195 | 0.02339 | 0.00597 |

9878 | 0.00085 | 0.00195 | 0.00568 | 0.01303 |

9879 | 0.00278 | 0.00195 | 0.00935 | 0.00656 |

9938 | 0.00141 | 0.00195 | 0.00491 | 0.00679 |

9939 | 0.01481 | 0.00195 | 0.02828 | 0.00372 |

21925 | 0.01304 | 0.00195 | 0.04850 | 0.00725 |

26262 | 0.00241 | 0.00195 | 0.01702 | 0.01377 |

26263 | 0.00831 | 0.00195 | 0.01424 | 0.00334 |

38189 | 0.00040 | 0.00195 | 0.0063 | 0.03071 |

38249 | 0.00309 | 0.00195 | 0.01225 | 0.00773 |

39273 | 0.00236 | 0.00195 | 0.02578 | 0.02130 |

40237 | 0.00062 | 0.00195 | 0.00291 | 0.00915 |

40297 | 0.00129 | 0.00195 | 0.0055 | 0.00831 |

41562 | 0.00229 | 0.00195 | 0.01478 | 0.01259 |

41563 | 0.00527 | 0.00195 | 0.03311 | 0.01225 |

41682 | 0.00265 | 0.00195 | 0.01436 | 0.01057 |

41683 | 0.01232 | 0.00195 | 0.00961 | 0.00152 |

42586 | 0.00227 | 0.00195 | 0.00675 | 0.00580 |

42587 | 0.00740 | 0.00195 | 0.05516 | 0.01454 |

42646 | 0.00520 | 0.00195 | 0.06068 | 0.02276 |

42647 | 0.01549 | 0.00195 | 0.03283 | 0.00413 |

42706 | 0.00247 | 0.00195 | 0.01609 | 0.01270 |

42707 | 0.02324 | 0.00195 | 00.02511 | 0.00211 |

43610 | 0.00305 | 0.00195 | 0.02359 | 0.01508 |

43611 | 0.00478 | 0.00195 | 0.02556 | 0.01043 |

54573 | 0.00544 | 0.00195 | 0.01887 | 0.00676 |

56621 | 0.00821 | 0.00195 | 0.01847 | 0.00439 |

## References

- K. Popper. The Logic of Scientific Discovery. New York, NY, USA: Harper and Row, 1959. [Google Scholar]
- P. Samuelson. Foundations of Economic Analysis. Cambridge, MA, USA: Harvard University Press, 1947. [Google Scholar]
- J.D. Angrist, and A.B. Krueger. “Does compulsory school attendance affect schooling and earnings? ” Q. J. Econ. 106 (1991): 979–1014. [Google Scholar] [CrossRef]
- K.F. Butcher, and A. Case. “The effect of sibling composition on women’s education and earnings.” Q. J. Econ. 109 (1994): 531–563. [Google Scholar] [CrossRef]
- M.R. Rosenzweig, and K.I. Wolpin. “Natural “natural experiments” in Economics.” J. Econ. Lit. 38 (2000): 827–874. [Google Scholar] [CrossRef]
- J.M. Barron, T.E. Ewing, and G.R. Waddell. “The Effect of High School Athletic Participation on Education and Labor Market Outcomes.” Rev. Econ. Statist. 82 (2000): 409–421. [Google Scholar] [CrossRef]
- C.E. Shannon. “A Mathematical Theory of Communication.” Bell Syst. Techn. J. 27 (1948): 379–423. [Google Scholar] [CrossRef]
- G. Lady, and A. Buck. “Structural Models, Information and Inherited Restrictions.” Econ. Model. 28 (2011): 2820–2831. [Google Scholar] [CrossRef]
- D. Hale, G. Lady, J. Maybee, and J. Quirk. Nonparametric Comparative Statics and Stability. Princeton, NJ, USA: Princeton University Press, 1999. [Google Scholar]
- D. Hale, and G. Lady. “Qualitative Comparative Statics and Audits of Model Performance.” Linear Algebra Its Appl. 217 (1995): 141–154. [Google Scholar] [CrossRef]
- A.J. Buck, and G. Lady. “Structural Sign Patterns and Reduced Form Restrictions.” Econ. Model. 29 (2012): 462–470. [Google Scholar] [CrossRef]
- L. Metzler. “Stability of Multiple Markets: The Hicks Conditions.” Econometrica 13 (1945): 277–292. [Google Scholar] [CrossRef]
- T.M. Cover, and J.A. Thomas. Elements of Information Theory. New York, NY, USA: Wiley Series in Telecommunications, 1991. [Google Scholar]
- W. Isaacson. Einstein, His Life and Universe. New York, NY, USA: Simon and Schuster, 2007. [Google Scholar]
- A.D. Strickland, and L.W. Weiss. “Advertising concentration, and price-cost margins.” J. Polit. Econ. 84 (1976): 1109–1121. [Google Scholar] [CrossRef]
- T. Amemiya. Advanced Econometrics. Cambridge, MA, USA: Harvard University Press, 1985. [Google Scholar]
- W. Greene. Econometric Analysis, 5th ed. Upper Saddle River, NJ, USA: Prentice Hall, 2003. [Google Scholar]
- D. Gujarati. Basic Econometrics. New York, NY, USA: McGraw-Hill Higher Education, 2003. [Google Scholar]
- D.F. Greer. Industrial Organization and Public Policy, 2nd ed. New York, NY, USA: Macmillan, 1984. [Google Scholar]
- F.M. Scherer, and D. Ross. Industrial Market Structure and Economic Performance. Boston, MA, USA: Houghton Mifflin, 1990. [Google Scholar]
- D.W. Carlton, and J.M. Perloff. Modern Industrial Organization, 3rd ed. Glenview, IL, USA: Scott, Foresman, 2000. [Google Scholar]
- A.J. Buck, and G. Lady. “Falsifying Economics Models.” Econ. Model. 22 (2005): 777–810. [Google Scholar] [CrossRef]
- System’s Science Inc. The Oil Market Simulation Model, Documentation Report. Washington, DC, USA: System’s Science Inc., 1985. [Google Scholar]
- Energy Information Administration. Oil Market Simulation User’s Manuel; Washington, DC, USA: Energy Information Administration, 1990.
- L. Klein. Economic Fluctuations in the U.S., 1921–1941. Cowles Commission for Research in Economics Monograph No. 11; New York, NY, USA: John Wiley and Sons, 1950. [Google Scholar]
- E.R. Berndt. The Practice of Econometrics, Classic and Contemporary. New York, NY, USA: Addison-Wesley, 1996. [Google Scholar]
- J. Maybee, and G. Wiener. “From qualitative matrices to quantitative restrictions.” Linear Multilinear Algebra 22 (1988): 229–248. [Google Scholar] [CrossRef]
- G. Lady. “Topics in nonparametric comparative statics and stability.” Int. Adv. Econ. Res. 6 (2000): 67–83. [Google Scholar] [CrossRef]
- A.S. Goldberger. Econometric Theory. New York, NY, USA: John Wiley and Sons, 1964. [Google Scholar]
- K. Lancaster. “The Solution of Qualitative Comparative Statics Problems.” Q. J. Econ. 53 (1966): 278–295. [Google Scholar] [CrossRef]
- L. Bassett, J. Maybee, and J. Quirk. “Qualitative Economics and the Scope of the Correspondence Principal.” Econometrica 36 (1968): 544–563. [Google Scholar] [CrossRef]
- G. Ritschard. “Computable Qualitative Comparative Statics Techniques.” Econometrica 51 (1983): 1145–1168. [Google Scholar] [CrossRef]
- S. Maybee. “A Method for Identifying Sign Solvable Systems.” Master Thesis, University of Colorado, Boulder, CO, USA, 1986. [Google Scholar]
- G. Lady. SGNSOLVE.EXE Analysis Package; prepared as a job of work for the Energy Information Administration; Washington, DC, USA: U.S. Department of Energy, 1993.
- “RF Finder Web Site.” Available online: http://astro.temple.edu/~gmlady/RF_Finder/Finder_Page.htm (accessed on 18 June 2015).

^{2}All three papers report results for a single structural equation estimated by IV methods without specifying the signs and zero restrictions in the other equations of the implicit structural model. By ignoring this step their models cannot be falsified. As a consequence of the absence of the ability to falsify, these papers cannot claim to have modelled causation. Rather, they have at best provided more precise conditional partial correlations. This is a common shortcoming in the “natural experiment” literature.^{3}At the risk of belaboring the point, consider a model car. The model car is not an exact replica of a real car, but we could ask what part of the real car is correctly described by the model. What we show in this paper is that empirical implementation of the model car may lead us to believe that we have correctly described some part of the real car, even though the model could not possibly have generated the observed data. See Section 4 below for examples of well-known models that could not possibly have generated the data used to estimate model parameters.^{4}We show below that even in the absence of sampling error it is possible to commit type 1 and type 2 errors.^{6}In hypothesis testing there is either an appeal to the Central Limit Theorems or an assumption of normality in order to apply the usual test statistics. For the purposes of the work presented here the actual distribution of the error is of no importance as long as the variances of the random variables in U are not excessively large. In simulations not reported here we have found that the error vector variance has to be orders of magnitude greater than the variation in the independent variables before there is any impact on the falsification procedures presented here. One might even argue that if the structural error is orders of magnitude larger than the variation in the independent variables then the model is incompletely specified, in which case a mistaken falsification is forgivable.^{7}Sampling error is an issue that may lead one to conclude that an estimate is different from, say, zero when in fact it is not because the test statistic is not constrained to fall in a certain part of the distribution’s support. An exception would be testing for unit roots, in which the parameter of interest is bounded away from one under the null hypothesis and as a consequence the test statistic is also bounded. In the realm of the present paper, if the specified model is TRUE in its assigned zero restrictions and sign patterns then in expectation the reduced form coefficients estimators mathematically cannot take the wrong signs. The sampling error occurs because the reduced form error is a weighted average of the structural disturbance and might admit falsifiable signs. We reiterate the point in footnote 5, that in Mont Carlo simulations the structural error must be orders of magnitude larger than the variation in Z to induce a rejection of a true null. The more important point remains that the admissible outcomes for the reduced form estimates can be small in number and structural forms other than the one specified can produce an admissible reduced form. In short, the world in which theory informs model specification is a good deal more restricted than classical statistics.^{9}In this paper we deal only with identification made possible through zero restrictions imposed on β and γ of Equation (3). In an earlier paper we dealt with identifiability and falsification of a structural VAR using the error covariance; see Lady and Buck [8].^{10}A matrix is reducible if and only if it can be placed into block upper-triangular form by simultaneous row/column permutations. A square matrix which is not reducible is said to be irreducible. The inverse of an irreducible matrix has no entries that must be equal to zero, a characteristic that we make use of when running the Monte Carlo. Limiting the analysis to irreducible systems is convenient, but not necessary for any of our analytical points.^{11}This algorithmic approach and its use are briefly described in the Appendix. Suffice it to say that the Monte Carlo is run for millions of repetitions.^{12}Regardless of the actual number of possible sign patterns for an inverse, the outcomes of the inversion have a multinomial distribution. The Monte Carlo is a maximum likelihood estimator of the parameters of that multinomial distribution. For further discussion of this as well as the bias in our estimator for the information content of the model see Lady and Buck [8].^{13}This is not a trivial example even though the system is exactly identified. In standard practice the estimates of the structural coefficients can be solved from the reduced form using indirect least squares. In spite of being exactly identified the proposed model can still be wrong, produce a set of first stage results that would by our methodology falsify the proposed model, yet produce estimates of some of the proposed model coefficients that were of the proposed sign and be statistically significant.^{14}Strictly, sgn a = 1, −1, 0 as a > 0, a < 0, a = 0. We will use the signs themselves as a felicitous convention.^{15}For our example, the condition γ = I is also imposed. Accordingly, it is this condition plus β_{31}= 0 that is being falsified, not just the zero restriction on β.^{16}Recall that there can be no logical zeros in the inverse of an irreducible matrix.^{17}There is fundamentally no difference between what we have done to model sampling error and an alternative approach in which one might first use a Monte Carlo to populate an admissible π and then perturb those π_{i,j}to simulate sampling error.^{18}See Isaacson [14] (Chapter 9).^{19}It might be noted that this hypothesized “universe” can itself be falsified. For example, an all positive reduced form is impossible for any of the 512 structural sign patterns.^{20}We were not able to obtain the dataset used by Strickland and Weiss [15].^{21}In an actual application, if a reduced form sign pattern is not found by the Monte Carlo, additional analysis would be appropriate to isolate the particular inconsistencies presented by the hypothesized sign patterns for the structural arrays. We will set that as beyond our scope, accepting the results of the numerical method employed. The chances of missing a sign pattern in tens of millions of quantitative samples for the size of the arrays discussed here is presumably quite small. A means of estimating the likelihood of making such a mistake is given in Lady and Buck [8] (p. 2825).

© 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license ( http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Buck, A.J.; Lady, G.M.
A New Approach to Model Verification, Falsification and Selection. *Econometrics* **2015**, *3*, 466-493.
https://doi.org/10.3390/econometrics3030466

**AMA Style**

Buck AJ, Lady GM.
A New Approach to Model Verification, Falsification and Selection. *Econometrics*. 2015; 3(3):466-493.
https://doi.org/10.3390/econometrics3030466

**Chicago/Turabian Style**

Buck, Andrew J., and George M. Lady.
2015. "A New Approach to Model Verification, Falsification and Selection" *Econometrics* 3, no. 3: 466-493.
https://doi.org/10.3390/econometrics3030466