# The Relative Merits of Observational and Experimental Research: Four Key Principles for Optimising Observational Research Designs

## Abstract

**:**

## 1. Introduction

“Despite the potential for observational studies to yield important information, clinicians tend to be reluctant to apply the results of observational studies into clinical practice. Methods of observational studies tend to be difficult to understand, and there is a common misconception that the validity of a study is determined entirely by the choice of study design.”[15] (p. 860)

## 2. Certainty, Risk and Uncertainty in Experimental and Observational Research

“… as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know … it is the latter category that tends to be the difficult ones.”[21] (p. 1)

#### 2.1. What We Know (Group 1—Certainty)

“As we explained in the last section, researchers do not derive causal relations from an SEM. Rather, the SEM represents and relies upon the causal assumptions of the researcher. These assumptions derive from the research design, prior studies, scientific knowledge, logical arguments, temporal priorities, and other evidence that the researcher can marshal in support of them. The credibility of the SEM depends on the credibility of the causal assumptions in each application.”[26] (p. 309)

^{79}= 4.5%. This very low level of probability is not a marginal outcome, and it is based upon a universally accepted probability calculation [28] and an authoritative account in support of SEM describing how SEM uses information with a high p value to establish causality [26]. It becomes even more alarming when one considers that once published, such research can then be used as a ‘credible’ secondary causal assumption input to further related SEM based primary research with its reliability/validity as Group 1 information ‘prior research’ readjusted up from 4% to 100%.

#### 2.2. What We Know We Don’t Know (Group 2—Risk)

^{2}full factorial with six levels of each variable and 30 observations in each cell would require 480 observations to fully compare the relationships between 2 independent variables and one dependent variable. By contrast a 4

^{4}full factorial would require 7680 observations to study the relationships between four independent variables and one dependent variable to the same standard.

#### 2.3. What We Don’t Know We Don’t Know (Group 3—Uncertainty)

## 3. Managing Risk and Uncertainty in Experimental and Observational Research—Fisher’s Principals

“… the problem of designing economical and effective field experiments is reduced to two main principles (i) the division of the experimental area into plots as small as possible …; (ii) the use of [experimental] arrangements which eliminate a maximum fraction of soil heterogeneity, and yet provide a valid estimate of residual errors.”[40] (p. 510)

## 4. Certainty, Risk, Uncertainty and the Relative Merits of Experimentation and Observational Research

**Principal 1.**

**Principle 2.**

**Principle 3.**

**Principle 4.**

## 5. Conclusions

## 6. Final Thought: The Application of Fisher’s Principles to Recall Bias and within Individual Variation

“There are some features on methods of data collection in nutritional studies that require attention, for example recall bias or within individual variation. The authors did not mention these at all.”

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- Drummond, K.E.; Reyes, A.; Cooke, N.K.; Stage, V.C. Nutrition Research: Concepts and Applications; John Wiley: Hoboken, NJ, USA, 2022. [Google Scholar]
- Andrew, C.O.; Hildebrand, P.E. Applied Agricultural Research: Foundations and Methodology; Routledge: Abingdon, UK, 2019. [Google Scholar]
- Hillison, J. The origins of agriscience: Or where did all that scientific agriculture come from? J. Agric. Educ.
**1996**, 37, 8–13. [Google Scholar] [CrossRef] - Longchamps, L.; Tremblay, N.; Panneton, B. Observational Studies in Agriculture: Paradigm Shift Required. In Proceedings of the 15th International Conference on Precision Agriculture, Montreal, QC, Canada, 24–27 June 2018; Available online: https://www.ispag.org/proceedings/?action=download&item=5436 (accessed on 1 July 2022).
- Hamlin, R.P. The rise and fall of the Latin Square in marketing:a cautionary tale. Eur. J. Mark.
**2005**, 39, 328–350. [Google Scholar] [CrossRef] - Djurisic, S.; Rath, A.; Gaber, S.; Garattini, S.; Bertele, V.; Ngwabyt, S.N.; Hivert, V.; Neugebauer, E.A.M.; Laville, M.; Hiesmayr, M.; et al. Barriers to the conduct of randomised clinical trials within all disease areas. Trials
**2017**, 18, 360. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Pounis, G. Design of Observational Nutrition Studies. In Analysis in Nutrition Research; Pounis, G., Ed.; Academic Press: Cambridge, MA, USA, 2019; pp. 3–22. [Google Scholar]
- Lee, N.; Broderick, A.J. The past, present and future of observational research in marketing. Qual. Mark. Res.
**2007**, 10, 121–129. [Google Scholar] [CrossRef] - Ejima, K.; Li, P.; Smith, D.L., Jr.; Nagy, T.R.; Kadish, I.; van Groen, T.; Allison, D.B. Observational research rigour alone does not justify causal inference. Eur. J. Clin. Investig.
**2016**, 46, 985–993. [Google Scholar] [CrossRef] [Green Version] - Hilton Boon, M.; Burns, J.; Craig, P.; Griebler, U.; Heise, T.L.; Vittal Katikireddi, S.; Bero, L. Value and Challenges of Using Observational Studies in Systematic Reviews of Public Health Interventions. Am. J. Public Health
**2022**, 112, 548–552. [Google Scholar] [CrossRef] - Bailey, R.L.; Sahni, S.; Chocano-Bedoya, P.; Daly, R.M.; Welch, A.A.; Bischoff-Ferrari, H.; Weaver, C.M. Best practices for conducting observational research to assess the relation between nutrition and bone: An international working group summary. Adv. Nutr.
**2019**, 10, 391–409. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Hu, Z.; Qin, J. Generalizability of causal inference in observational studies under retrospective convenience sampling. Stat. Med.
**2018**, 37, 2874–2883. [Google Scholar] [CrossRef] [PubMed] - Beyerbach, J.; Stadelmaier, J.; Hoffmann, G.; Balduzzi, S.; Bröckelmann, N.; Schwingshackl, L. Evaluating Concordance of Bodies of Evidence from Randomized Controlled Trials, Dietary Intake, and Biomarkers of Intake in Cohort Studies: A Meta-Epidemiological Study. Adv. Nutr.
**2022**, 13, 48–65. [Google Scholar] [CrossRef] - Anglemyer, A.; Horvath, H.T.; Bero, L. Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials. Cochrane Database Syst. Rev.
**2014**, 2014, MR000034. [Google Scholar] [CrossRef] - Gershon, A.S.; Jafarzadeh, S.R.; Wilson, K.C.; Walkey, A.J. Clinical knowledge from observational studies. Everything you wanted to know but were afraid to ask. Am. J. Respir. Crit. Care Med.
**2018**, 198, 859–867. [Google Scholar] [CrossRef] - Elston, D.M. Letter from the Editor: Participation bias, self-selection bias, and response bias. J. Am. Acad. Dermatol.
**2021**. [Google Scholar] [CrossRef] - Mindell, J.S.; Giampaoli, S.; Goesswald, A.; Kamtsiuris, P.; Mann, C.; Männistö, S.; Tolonen, H. Sample selection, recruitment and participation rates in health examination surveys in Europe–experience from seven national surveys. BMC Med. Res. Methodol.
**2015**, 15, 1–19. [Google Scholar] [CrossRef] [Green Version] - Mulvany, J.L.; Hetherington, V.J.; VanGeest, J.B. Survey research in podiatric medicine: An analysis of the reporting of response rates and non-response bias. Foot
**2019**, 40, 92–97. [Google Scholar] [CrossRef] - Morton, S.M.; Bandara, D.K.; Robinson, E.M.; Carr, P.E.A. In the 21st Century, what is an acceptable response rate? Aust. N. Z. J. Public Health
**2012**, 36, 106–108. [Google Scholar] [CrossRef] [PubMed] - Ronsebaum, P.R. Observational Studies; Springer: Berlin, Germany, 2002. [Google Scholar]
- Defense Gov. News Transcript: DoD News Briefing—Secretary Rumsfeld and Gen. Myers; United States Department of Defense (defense.gov): Washington, DC, USA, 2002. Available online: https://archive.ph/20180320091111/http://archive.defense.gov/Transcripts/Transcript.aspx?TranscriptID=2636 (accessed on 9 May 2022).
- Seely, H. Pieces of Intelligence: The Existential Poetry of Donald H. Rumsfeld; Simon & Schuster: New York, NY, USA, 2010. [Google Scholar]
- Ware, J.H.; Mosteller, F.; Delgado, F.; Donnelly, C.; Engelfinger, J.A. P Values. In Medical Uses of Statistics; Bailer, J.C., Hoaglin, D.C., Eds.; CRC Press: Boca Raton, FL, USA, 2019; pp. 181–200. [Google Scholar]
- Sorkin, J.D.; Manary, M.; Smeets, P.A.; MacFarlane, A.J.; Astrup, A.; Prigeon, R.L.; Tobias, D.K. A guide for authors and readers of the American Society for Nutrition journals on the proper use of P values and strategies that promote transparency and improve research reproducibility. Am. J. Clin. Nutr.
**2021**, 114, 1280–1285. [Google Scholar] [CrossRef] [PubMed] - Babin, B.J.; Svensson, G. Structural equation modeling in social science research: Issues of validity and reliability in the research process. Eur. Bus. Rev.
**2012**, 24, 320–330. [Google Scholar] [CrossRef] - Bollen, K.A.; Pearl, J. Eight Myths About Causality and Structural Equation Models. In Handbook of Causal Analysis for Social Research; Springer: Berlin, Germany, 2013; pp. 301–332. [Google Scholar]
- Maitland, S.B.; Brauer, P.; Mutch, D.M.; Royall, D.; Klein, D.; Tremblay, A.; Jeejeebhoy, K. Evaluation of Latent Models Assessing Physical Fitness and the Healthy Eating Index in Community Studies: Time-, Sex-, and Diabetes-Status Invariance. Nutrients
**2021**, 13, 4258. [Google Scholar] [CrossRef] - Parker, R.E. Introductory Statistics for Biology; Cambridge University Press: Cambridge, UK, 1991. [Google Scholar]
- Rees, D.G. Essential Statistics; Chapman and Hall/CRC: London, UK, 2018. [Google Scholar]
- Fisher, R.A. Statistical Methods for Research Workers; Oliver & Boyd: Edinburgh, UK, 1925. [Google Scholar]
- Davies, O.L. The Design and Analysis of Industrial Experiments; Oliver & Boyd: London, UK, 1954. [Google Scholar]
- Banks, S. Experimentation in Marketing; McGraw-Hill: New York, NY, USA, 1965. [Google Scholar]
- Tabachnik, B.G.; Fidell, L.S. Using Multivariate Statistics, 7th ed.; Pearson: London, UK, 2018. [Google Scholar]
- Dey, A.; Mukerjee, R. Fractional Factorial Plans; John Wiley & Sons: Hoboken, NJ, USA, 2009. [Google Scholar]
- Pelzer, E. The potential of conjoint analysis for communication research. Commun. Res. Rep.
**2019**, 36, 136–147. [Google Scholar] [CrossRef] - Rao, V.R. Applied Conjoint Analysis; Springer: New York, NY, USA, 2014. [Google Scholar]
- Louviere, J.J. Conjoint Analysis. In Advanced Methods of Marketing Research; Blackwell Publications: Cambridge, MA, USA, 1994; pp. 223–259. [Google Scholar]
- Fisher, R.A. The Design of Experiments; Oliver & Boyd: Edinburgh, UK, 1935. [Google Scholar]
- Verdooren, L.R. History of the statistical design of agricultural experiments. J. Agric. Biol. Environ. Stat.
**2020**, 25, 457–486. [Google Scholar] [CrossRef] - Fisher, R.A. Statistical Methods for Research Workers, 10th ed.; Oliver & Boyd: Edinburgh, UK, 1950. [Google Scholar]
- Punch, J. Johannes Poncius’s Commentary on John Duns Scotus’s Opus Oxoniense, book III, dist. 34, q. 1. In John Duns Scotus Opera Omnia; Luke Wadding: Louvain, Belgium, 1639; Volume 15, p. 483. [Google Scholar]
- Van Den Berg, H.A. Occam’s razor: From Ockham’s via moderna to modern data science. Sci. Prog.
**2018**, 101, 261–272. [Google Scholar] [CrossRef] [PubMed] - Langton, S.D. Avoiding edge effects in agroforestry experiments; the use of neighbour-balanced designs and guard areas. Agrofrestry Syst.
**1990**, 12, 173–185. [Google Scholar] [CrossRef] - Youden, W.J.; Hunter, J.S. Partially replicated Latin squares. Biometrics
**1955**, 11, 399–405. [Google Scholar] [CrossRef] - Hamlin, R. How small sample size and replication can increase accuracy in experiments: Lessons that marketing may learn from agricultural scientific method. Australas. Mark. J.
**2017**, 25, 166–174. [Google Scholar] [CrossRef] - Hamlin, R.; McNeill, L. Does the Australasian “health star rating” front of pack nutritional label system work? Nutrients
**2016**, 8, 327. [Google Scholar] [CrossRef] [Green Version] - Hamlin, R.; McNeill, L. The impact of the Australasian ‘Health Star Rating’, front-of-pack nutritional label, on consumer choice: A longitudinal study. Nutrients
**2018**, 10, 906. [Google Scholar] [CrossRef] - Abramson, J.H.; Abramson, Z.H. Research Methods in Community Medicine: Surveys, Epidemiological Research, Programme Evaluation, Clinical Trials, 6th ed.; Wiley: Hoboken, NJ, USA, 2008. [Google Scholar]

**Figure 2.**Graphical representation of the interaction of risk, uncertainty and unreliability as a function of experimental sample size.

Knowledge Group | Description | Definition |
---|---|---|

1. What we know “… things we know we know …” | The information available via earlier research and observation. Not actually a certainty (p = 0), but routinely treated as such. | Certainty |

2. What we know we don’t know “… we know there are some things we do not know …” | The target relationship(s) of the research, and potentially a small number of other relationships and interactions. Usually quantified and described in the reporting process via a p value (p < x) | Risk |

3. What we don’t know we don’t know “… the ones we don’t know we don’t know …” | All other relationships and interactions within the proposed dataset, including interactions of these unknown variables with the variables in Group 2 above. These cannot be specifically described or quantified. Additionally their potential impact is not usually discussed in any depth, or not at all, at any stage in the research design or reporting process. | Uncertainty |

**Table 2.**The impact of Group 3 uncertainty variables on experimental and observational research outcomes.

Experimental Study | Observational Study |
---|---|

Design: 2^{4} factorial design—480 subjects recruited as eight matched groups of 60 on the basis of parental education, household income and gender. Within each group 30 randomly allocated to a high fructose diet and 30 to a low one, and attention span observed. | Design: 480 subjects recruited as eight matched groups of 60 on the basis of parental education, household income and gender. Each group of 60 divided up into two groups of 30 (high and low) on the basis of their reported fructose consumption and attention span observed. |

Random impact Group 3 uncertainty input (school attended): The school attended effect will uniformly increase variation within the two randomly allocated experimental groups for high and low fructose diet. This increase in variation will end up in the error term of the analysis of variance, reducing the F ratio for fructose intake and for parental education, income and gender (trending to a Type I error).As the groups for parental education, household income and the child’s gender are not randomly allocated, the school effect will either end up in the error term of the analysis of variance thereby depressing the F ratio for parental education, income and gender if it is not colinear, or it will end up in the error that is related to these variables, and thus increase the F ratio if it is colinear. Therefore, results could trend towards a Type I or Type II error with regard to any or all of these Group 2 variables, depending on the level of and nature of the collinearity between them and the Group 3 variable. The school effect would be likely to be strongly colinear with all of these three Group 2 variables if the attention span crusading school was perceived to be the ‘good’ school in the area. | Random impact Group 3 uncertainty input (school): The school attended variable will impact upon the parental education, household income and child’s gender variables exactly as it does in the experimental design opposite.The impact of the school attended variable upon the fructose intake variable will depend upon its degree of collinearity with it. If it is not collinear, then the allocation to the two groups will effectively be random, and the variation will thus end up in the error term depressing the F ratio for fructose intake, and tending towards a Type I error. If school attended has any collinearity with fructose intake, then the allocation will not be random and the impact of school attended will be apportioned into the variation associated with fructose intake. Depending whether the effect of school attended is complementary or anticomplementary to the effect of fructose intake, the result is a trend towards either a Type I (suppressed F ratio) or a Type II error (increased F ratio). |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Hamlin, R.
The Relative Merits of Observational and Experimental Research: Four Key Principles for Optimising Observational Research Designs. *Nutrients* **2022**, *14*, 4649.
https://doi.org/10.3390/nu14214649

**AMA Style**

Hamlin R.
The Relative Merits of Observational and Experimental Research: Four Key Principles for Optimising Observational Research Designs. *Nutrients*. 2022; 14(21):4649.
https://doi.org/10.3390/nu14214649

**Chicago/Turabian Style**

Hamlin, Robert.
2022. "The Relative Merits of Observational and Experimental Research: Four Key Principles for Optimising Observational Research Designs" *Nutrients* 14, no. 21: 4649.
https://doi.org/10.3390/nu14214649