Epistemological Framework for Computer Simulations in Building Science Research: Insights from Theory and Practice

: Computer simulations are widely used within the area of building science research. Building science research deals with the physical phenomena that a ﬀ ect buildings, including heat and mass transfer, lighting and acoustic transmission. This wide usage of computer simulations, however, is characterized by a divergence in thought on the composition of an epistemological framework that may provide guidance for their deployment in research. This paper undertakes a fundamental review of the epistemology of computer simulations within the context of the philosophy of science. Thereafter, it reviews the epistemological framework within which computer simulations are used in practice within the area of building science research. A comparison between the insights obtained from the realms of theory and practice is made, which then interrogates the adequacy of the epistemological approaches that have been employed in previously published simulation-based research. These insights may help in informing a normative composition of an adequate epistemological framework within which computer simulation-based building science research may be conducted.


Introduction
Computer simulations are widely used within the area of building science research. Building science research deals with the physical phenomena that affect buildings, including heat and mass transfer, lighting and acoustic transmission. Among others, the computer simulations may be used for the purposes of building theory by way of prediction, retrodiction and explanation. They may also be used for the purposes of testing existing theories. The use of simulations in lieu of conventional experiments may be necessitated by the unavailability of data, which they may then be called upon to provide [1].
The wide usage of computer simulations, however, is characterized by a divergence in thought on the composition of an epistemological framework that may provide guidance for their deployment in research. This lack of consensus may lead to inconsistent use of computer simulations in building science research, ultimately leading to an erosion of trust and confidence in the knowledge that originates from the simulations.
This paper seeks to achieve three main objectives. First, it seeks to undertake a fundamental review of the epistemology of computer simulations within the context of the philosophy of science. Thereafter, it seeks to review the epistemological framework within which computer simulations are used in practice within the area of building science research. A comparison between the insights obtained from the realms of theory and practice is made, which then interrogates the adequacy of of making a claim that a particular event will occur with certain probability in the near future [9]. Simulations may also be used for purposes of explaining phenomena, in which case they may be seeking to provide any of three types and levels of explanation, namely full explanation, partial explanation and potential explanation [9]. Simulations can also be used for proving theories.
According to Winsberg [1], simulations may be necessitated where there is an unavailability of data, which the simulations are thus called upon to provide, replacing conventional experiments and observations as sources of data about the world. This may particularly occur under two conditions, namely where data on analytically resolving mathematical models is unavailable and where natural experimentation is inappropriate for practical reasons or unattainable for physical reasons [10].

Epistemological Framework for Building and Testing Theory Using Computer Simulations
The epistemological framework at the core of this study is an ideological construct that must address the question of how acceptable theories can be built and tested using computer simulations. In addressing this question, the framework must provide insights into the nature of the knowledge that may be obtainable through computer simulations. It is also important that this framework must provide insights into the possible challenges that may be anticipated in the pursuit of the said theory and suggest means of dealing with the challenges. Adequate usage of computer simulations in the process of building and testing theory must then be evaluated on the basis of this framework.

Nature of Knowledge Obtained from Computer Simulations
The knowledge that is produced by computer simulations is the result of inferences that must conform to three characters, namely that the knowledge must be downward, motley and autonomous [11]. The inferences must be downward in a way that demonstrates a top-down transition from high theory to particular phenomena. For instance, with regard to the earlier example of CFD simulations given in Section 2, the inferences must draw from a general understanding of the theory governing the flow of fluids in space and time. This general understanding must then be applied to a specific situation, such as a building whose indoor and outdoor wind environments need to be analyzed. They must also be motley by way of drawing from a multiplicity of sources, such as the simulationist's own knowledge of the simulation scenario, simulation best practice guidelines and previous simulation experiences among others. Finally, the inferences must be autonomous such that when they are made, there must be enough ground to believe in them.
In spite of the understanding that mathematical modeling forms part of the core of computer simulations, the knowledge that is obtained from an evaluation of simulations is different from the kind of knowledge that follows from an evaluation of mathematical equations. In the latter, the transparency obtained from the formal presentation of the relationship between the mathematical terms enables understanding without the need for calculations. On the other hand, the understanding that follows from an evaluation of simulations is only feasible as a function of calculations. It is characterized by epistemic opacity, rendering it, to a larger extent, pragmatist. This understanding is a certain threshold about an actual solution such that its use may best be suited to interventions and predictions, rather than theoretical explanations [3]. How to set this threshold about the actual solutions becomes an important epistemic consideration that affects the significance of simulations within the realm of the philosophy of science. In a world where any given problem defined by a mathematical model can have a feasible solution, a satisfactory solution, an optimum solution or no solution at all, computer simulation mathematical models are primarily directed toward finding satisfactory solutions to practical problems [12].
There is wide disagreement on the nature of simulations bordering on whether they may be regarded as experiments or not, with far-reaching consequences on the nature of the kind of understanding that they provide. A fair amount of research work has sought to compare computer simulations and traditional experiments [13][14][15][16][17][18], with other researchers casting doubt on the epistemological power of the former relative to the latter, on the basis of materiality. Morgan [16] suggests that inferences about target systems are more justified when the experimental and target system are made of the same material than when they are made of different materials, as is the case in computer simulations. However, Parker [6] argues that, with respect to epistemic adequacy comparisons between the two, the focus on materiality is misplaced, as it is relevant materiality, and not materiality in general, that ultimately matters. Nonetheless, Parker [6] agrees with Morgan's [16] suggestion that ontological equivalence provides epistemological power in so far as this suggestion remains consistent with the relevance of similarity.

Hierarchical Order of Computer Simulations
According to [19], the core of the practice of computer simulations consists in the construction of a hierarchy of models. This hierarchy consists of mechanical models, dynamical models, computational models and phenomena models in that order. The mechanical models place the phenomena of interest within an existing fundamental theoretical context. The dynamical models seek to localize the simulation within specific contexts of the phenomena under investigation and may include data on the appropriate boundary and initial conditions. The computational models are a discretized version of the mechanical models, through which continuous equations are solved numerically. The phenomena models constitute a higher-level representation of real-world phenomena, complete with data that may need to be interpreted to enhance understanding of the phenomena. In essence, computer simulations take a tiered structure, including input conditions, mechanism conditions and output conditions [8].

Challenges of Computer Simulations
The epistemological trustworthiness of computer simulations has been severally called to question in the literature [2]. This is perhaps due to the essence of the act of obtaining knowledge through simulations, which consists in, among others, approximations, similarities and mathematical modeling. The approximations and similarities mean that simulations build theory by way of inductive claims to generalization. Since simulationists have direct access only to their own peculiar and limited set of experiences, it is difficult to understand how generalizations would be made beyond the simulationists' empirical domain [20]. There have been questions on computer simulations' degree of accuracy, confidence in the inferences that are drawn from their explanations and reliability of simulation-based decisions. Lenhard [3] notes that these pervasive doubts on the epistemological adequacy of simulations might be the cause for their slower uptake in the sciences. Among others, these doubts have been promulgated by those who hold the amplifier view of computer simulations as being merely an extension of mathematical modeling and thus inferior to theory, ultimately incapable of yielding any philosophical novelty of their own.

Validation, Verification and Robustness of Computer Simulations
In view of the epistemological challenges that computer simulations face, a number of strategies have been developed with an aim of evaluating the simulations' adequacy and enhancing the confidence with which they are used in research. In general, such strategies take the form of validation, verification and a measure of robustness.
Validation and verification focus on the output and the process by which solutions to the simulation model's continuous equations are estimated respectively [21]. Validation is the process of ensuring that the equations at the core of the simulation model accurately represent the target system [2]. On the other hand, verification entails an attempt at ensuring that the simulation's numerical solution is close enough to an analytical solution. According to Sargent [22], validation and verification exercises are attempts at addressing concerns on the correctness of simulation models, with the former being a substantiation of the correctness of the model's representation of the real world, and the latter, the substantiation of the correctness of the fundamental components of the simulation model and their implementation.
Another strategy that may be used in addressing the epistemological weakness of simulations is a measure of the robustness of the simulations. Robustness is a measure of how much value may be placed in a computer simulation as a function of the adequacy of the model's representation of the target phenomenon under varying input conditions and parameters [9]. D'Arms et al. [23] suggest that a simulation result may be robust if it is achieved across a variety of different starting conditions and parameters. This may be seen to underscore the need for sensitivity analysis for simulation-based studies. Closely related to the measure of robustness is the practice of using falsifications whose success and reliability do not amount to the truth, but can enhance the credibility of simulations [1]. These falsifications can be implemented through the use of principles that do not purport to offer true accounts of the nature of the target phenomena [1]. The falsifications must be proven to have no impact on the simulation results by ensuring that the simulation remains robust despite deliberate variations over repeated runs [8].
Validation in simulations is closely linked with the problem of induction where empiricism is necessary to justify a theory [24]. Typically, a match between the simulation results and those from empirical testing lends to the confidence of predictions and explanations from the simulations [9]. In the pursuit of validation, three positions have emerged in the literature, namely the justificationism or objectivist, anti-justificationism or relativist, and a blend of the two [24]. The justificationist position seeks to eliminate aspects of human judgment in the validation process, limiting it to empiricism and rationalism. The anti-justificationist position, on the other hand, makes room for human judgment in the validation of simulations, noting the difficulty that exists for a simulationist to completely detach themselves and their judgments from the simulation process. Purely empirically-oriented validation standards cannot be met in most instances [24].
In view of the major differences that exist as to what evidentiary requirement is necessary to validate computer simulations, Kleindorfer and Ganeshan [20] suggest that the simulationists should be left free to decide which methodological validation regime they would like to use, so long as they do it in an ethical and professional manner. The wide latitude that this position accords the simulationists, however, might be problematic, especially in so far as scientific epistemology is concerned. On the other hand, the strict objectivist position on the validation of simulations faces insufficiency, especially where empirical data that is characteristic of phenomena under investigation either currently does not exist or may be hard to obtain. The strict empiricism also goes against the autonomous nature of simulations, as advanced by Winsberg [11]. In order to adequately address this problem of the unavailability of empirical data, Winsberg [19] suggests that the simulation process needs to be justified internally. The confidence that a simulationist places in the simulation results must derive from their knowledge of computers, the adequacy of their assumptions, their ability to calibrate models against other simulation tools and empirical results where possible, and their ability to make judgments about the degree of resemblance between different sets of data. On the same point, Winsberg [1] notes that one of the conditions that necessitates the usage of simulations is the unavailability of empirical data, and that the simulations are thus meant to replace experiments and observations as sources of data about the world. Where such data about the world is unavailable, it may not be possible to evaluate simulations simply by comparing them to the world; rather, their credibility can derive from the credentials of their underlying theory and the credentials of the techniques employed by the simulationists. Further, the simulation models must be supported by demonstrated evidence of adequate performance under given hardware and software environments [9].
With verification of simulations, the problem is that simulations usually seek to solve problems whose analytical solutions are hard to get in the first place [2], a thing that renders analytical verification highly intractable. Where there is an expectation that the simulation model's code may be used in many subsequent studies, the verification can be streamlined to enable usage of the code in an off-the-shelf manner for different applications [21]. This may imply that where such is the case, as is typical when using commercially developed code, explicit verification may not be necessary for model evaluation. However, some process input assumptions may still need to be evaluated in the light of best practices that may have been developed to guide usage of the model's code.

Conditions for the Failure of Computer Simulations
A number of conditions will lead to the epistemological failure of computer simulations. Simulations fail when they are unable to properly correspond with reality [8]. To simulate means to build a likeness where the accuracy of the likeness constitutes one of the fundamental considerations in evaluating the simulation's success [24]. Correspondence between simulations and their targets may range from nearly-identical scale models to abstractions or idealizations [25]. However, on this point, it is noted that the notion of correspondence is highly plastic and relative depending on specific situations and particular aspects of the representation and the representandum [8]. The evaluation of correspondence may consider aspects of spatiotemporal relations, levels of abstraction and the relevance of representations [8].
Another condition for the failure of simulations is the usage of inadequate techniques, such as human error and faulty equipment [8]. Further, they may also fail by way of producing results that run counter to an existing favored theory. Such failure may, however, constitute a point of scientific progress, where the existing theory is successfully disproved. Inadequate representation of conditions as required by existing theory may also lead to failure of simulations.
In guarding against failure while enhancing the epistemological adequacy of computer simulations, several authors [17,19,26] have suggested the use of the same, albeit subtly modified, strategies that Franklin [27][28][29] proposed for use in lending credence to experimental results. These strategies are presented in Table 1 below.  [21].

Simulation Model Evaluation Strategies Simulation Code Evaluation Strategies
Apparatus gives other results that match known results Simulation output fits closely enough with various observational data It is important to note that these strategies are neither meant to be exhaustive, nor is there any one or fixed combination of strategies that may always be necessary to enhance the credibility of computer simulations.

Epistemological Framework for Computer Simulations in the Practice of Building Science Research
In order to obtain insights into the epistemological framework within which computer simulations are used in practice within the realm of building science research, this study undertook a review of scholarly work that has been published in the Journal of Building Performance Simulation. This journal was selected on account of its primary interest in simulation-based research work within the built environment. For purposes of the review, a total of 39 research papers, published in the journal over a period of 13 years between 2008, when the journal's first volume was published, and the present day in 2020, were analyzed, as shown in Table 2. These papers were selected following a systematic random sampling technique where three papers were randomly selected from each of the 13 years.  The review of the published scholarly work shows that within the realm of building science research, simulations are predominantly used for two purposes, namely the prediction and proving of theories. Where the simulations are used for the latter purpose, there is a desire to either prove or disprove an a priori theoretical proposition.
From the review, a consistent characterization of the epistemological framework for simulation-based research proves hard to obtain. There is an observed wide variation in terms of strategies that different researchers routinely adopt in order to enhance the epistemological adequacy of their simulation-based research work. In spite of the variation, five distinct epistemic strategies appeared to occur prominently within the body of the published research. These included verification, experimental validation, sensitivity analysis, inter-model comparison and justification for use of simulation over conventional experiments. These five strategies may be understood to form part of the epistemological framework within which computer simulations are presently used in practice within the area of building science research.
There were three different observed ways through which the verification approach was deployed. Researchers may have presented an analytical evaluation of the mathematical models at the core of the simulations in rare instances where this may have been possible. They may have also simply presented all the mathematical equations that form the core of the simulations. In yet other instances, the researchers may have resorted to using commercial simulation codes that have undergone rigorous verification checks prior to their availability for public use. The experimental validation strategy was implemented by a comparison between the simulation results and experimental results, which were obtained as part of the same study or from the literature. The sensitivity analysis, on the other hand, was undertaken by varying the simulation parameters where a fairly stable output result, despite the parametric variations, indicated the solution's independence of extraneous variables. The inter-model comparison involved two or more differently developed simulation codes being deployed to solve the same problem, with a desire that the solutions did not vary widely. In some instances, a justification, more often than not a logistical one, was provided as part of the study for why simulations, and not the conventional experiment, had been used.
The five epistemic strategies were used in 14 different combinations as shown in Table 3. A simulation-based researcher will typically deploy at least one of the 14 combinations. Of these combinations, some appeared to enjoy wider usage than others, as shown in Figure 1 below. The combination 'AB', featuring the use of verification and experimental validation, appeared to be the most widely used. This points toward some remote indication that in order for the epistemological framework for simulation-based research to be considered adequate, it ought to include verification checks and experimental validation at the minimum. This, however, is in spite of the fact that it was not uncommon to find the combination 'E' that only features a provision of justification for using simulations over conventional experiments in its epistemological framework, without conducting any verification checks and experimental validation.
The five distinct epistemological strategies appeared with different prevalence rates in the 14 combinations that are presented in Figure 1. The verification strategy was used in more than 90% of the simulation-based research studies' epistemological framework, while sensitivity analysis enjoyed the least prevalence rate, as shown in Figure 2. This suggests that many researchers regard verification as the single most important element of the epistemological framework for simulation-based research, followed by experimental validation. The lack of a normative idealization of the adequate epistemological framework for computer simulations appears to be a long-running matter in the literature. Over the years, as computing power and simulation understanding and capabilities have grown, there have remained clear differences in terms of the number of strategies that researchers deployed in efforts to enhance the epistemic adequacy of their work, as can be seen in Figure 3 below. Nonetheless, in general, a bigger proportion of researchers seems to have deployed, at the very least, two strategies in efforts to enhance the epistemological framework of their simulation-based work, as shown in Figure 4 below. The review shows that the strategies that are put forward in the theoretical realm as being necessary for dealing with the trustworthiness challenges of computer simulations are actually put to use in practice, albeit in an inconsistent manner, across research studies that have been undertaken within the area of building science. It is important to emphasize that each of the five epistemic strategies that are used in practice advances a uniquely different and important aspect of the simulation regime. For this reason, they must be used as a total package. Where only a few selected strategies are used, this potentially compromises the epistemological adequacy of the simulations. In the present scenario, where validation and verification appeared to be widely used exclusively of the other strategies, there might potentially be problems with influences from extraneous variables and faulty models, which may affect the credibility of the simulation results. While the provision of justification for use of computer simulations in lieu of conventional experiments may be necessary to demonstrate that the latter may not be practicable, this may be seen as an affront to the autonomy of the simulations as an equally robust source of data about the world.

Conclusions
The study found agreement between the theory of computer simulations and the practice of computer simulation-based research in terms of the purposes of use of the computer simulations. However, while in both realms computer simulations were deployed to build and test theories, in practice, the act of building theory was predominantly by way of prediction, leaving out the others, such as retrodiction and explanation.
There was also another observed agreement between theory and practice with regard to the epistemological framework for computer simulations. While the theory has avoided providing neither an exhaustive framework nor an evaluative standard for adequacy of the framework, the practice of simulation-based building science research has shown some indication of the existence of an unwritten standard for evaluation of the adequacy of the epistemological framework. From the many different epistemic strategy combinations that have been deployed in the research, it was observed that a significant body of the research work deployed, at the very least, two individual epistemic strategies, with verification being the single most widely deployed strategy, followed by experimental validation.
In spite of this unwritten standardization of the epistemological framework, there was wide variation in usage about the same in the body of the research work. This suggests that computer simulation-based building science research proceeds under very varied epistemological frameworks and resultant varying measures of the adequacy of the methodological strategies. Such inconsistency is problematic as it may lead to questions on the confidence and trust that may be placed in simulation-based research.
While an exhaustive list of epistemological framework strategies may be hard to obtain and equally hard to be wholly implemented in research, an evaluative minimum standard would significantly help to ensure consistency in the use of simulation, while lending credence to simulation-based research.
This study recommends that the computer simulationist would do well to ensure that, at the very minimum, the verification, experimental validation, sensitivity analysis and inter-model comparison strategies are deployed as a total package, forming part of the epistemological framework for computer simulations. In the same manner, publishers would do well to require that prior to publication of computer simulation-based research, authors can demonstrate adherence to this minimum evaluative standard at the very least.