Machine 1 is a retrofit FFF system [
12], which is a custom-built 6-degree-of-freedom robotic arm with an extruder attached to the end. This machine is most suitable for single-use purposes, as it can be repurposed later. Machine 2 is a conventional gantry system. The extruder can be moved using either the Cartesian or the polar coordinate systems. This is the most commonly occurring FFF system, as most desktop systems are of this kind. Machine 3 is an assembled robot [
13] and involves buying a whole pre-built robotic arm setup. The benefit of this is cost versus convenience. It is more suitable for large companies that want to use the system for mass production. Machine 4 is a belt gantry system [
7]. It is a customization on the conventional gantry system, where instead of a traditional build plate, a moving belt is used to create an infinite axis. This is most suitable for bulk production or for unidirectionally large parts. This system is more expensive due to the customization of the conventional gantry system. Finally, machine 5 is a fixed nozzle system [
14]. This is a novel take on the retrofit system where, instead of the extruder being attached to the robotic arm, the build plate is attached to the robotic arm. The extruder instead is fixed. This machine is most suitable for cases where multiple manufacturing processes are to be conducted on the same part.
The weights and scores are from Technical Experts (TEs); these are subjective and can vary from one expert to another even though they may be largely in agreement [
15]. Specifically, these subjective values may vary based on the experiences of each expert as well as their frame of mind when evaluating the case. To test the sensitivity of the rankings obtained to changes in these subjective values, a Monte Carlo simulation [
16] of 10,000 iterations was performed on the model. This was performed by modeling the scores as a triangular distribution with the assigned score as the mode and the lower and upper limits as 1 and 10, respectively. The rationale for the model is that the score assigned by the TEs may differ, but the probability of these differences decreases with increasing difference. This ensures each iteration of the simulation has randomized scores, with the assigned score being the most frequently appearing score.
3.4. Sensitivity Analysis
Given that the scores and weights are subjective inputs, it becomes important to assess how sensitive the rankings provided by the methodology are, particularly when the top ranked ones are close to each other. For example, an input from a decision-maker may not precisely represent their opinion and may be somewhat approximate. The challenge therefore is to assess how the rankings for the machines change if the scores and weights are changed. Therefore, sensitivity analysis was performed using Monte Carlo simulations for all three cases.
Figure 6 shows the number of times a specific ranking of the five machines occurred for Case 1 in 10,000 runs of Monte Carlo simulation. It can be observed from the figure that the ranking [
4,
6,
8,
20,
21] occurred 445 times, i.e., M2 was the top, followed by M4, M5, M1, and M3 at the bottom. This is consistent with the baseline ranked order of M2 gantry, M4 belt gantry, M5 fixed nozzle, M1 retrofit, and M3 assembled robot. Furthermore, in
Figure 6, it is observed that there was a significant drop in rank frequency when there was a change in the top two machines. If we compare the data in
Figure 6 with those in
Table 4, it is observed that the scores of the gantry and the belt gantry system were significantly higher than the other machines, which is further confirmed by the data in
Figure 6. Therefore, in this case, the rank order can be considered robust to small variations in human input.
Similarly,
Figure 7 and
Figure 8 show Cases 2 and 3, respectively, with the number of times a particular rank order is obtained after 10,000 runs of Monte Carlo simulation. It is observed that the top two machines that have been recommended based on baseline human input match the ones from the simulation about 40% of the time. In
Figure 7, there are three machines that are rotating in the top two recommended machines. This can be attributed to scores that are relatively close to each other in
Table 6, further confirming that the baseline human decision is robust in this case.
In
Table 8, the belt gantry system has the best score by a significant margin from the other machines, whose scores are all close to each other. From
Figure 8, it is observed that the belt gantry system is top ranked most often. Since the other systems have scores close to each other, there is no significant difference between them, and the baseline human decision can be considered robust in this case also.
From the Monte Carlo simulation of the scores, the recommended printer matched the ones from the case studies. From this, it is evident that the TEs’ assigned scores display some level of flexibility and the overall methodology can be considered robust. The third case study is the only one that shows a deviation in the ranking of printers, albeit the rankings of the worst were interchanged. This is acceptable, as their scores are relatively close to each other.
From the Monte Carlo simulations of the varied weights, we can observe that the ranking of the printers did not change with perturbations. In particular, the machine that was top ranked remained the same regardless of the amount of perturbation or direction of perturbation, as shown in
Table 9. This is because the methodology depends on the overall ratio of all parameters rather than on the weight of a particular parameter, which makes it robust.
Another aspect of robustness is the choice of probability distribution used to characterize variation in human input and the number of simulation runs: e.g., whether the top ranked machine remains the same for 1000 simulations, 5000 simulations, and 10,000 simulations. All the above Monte Carlo simulations model human input variation as a triangular distribution. Alternatively, instead of triangular distribution, normal or beta distributions can be considered.
Figure 9 illustrates a histogram of 10,000 computed samples for triangular distribution for machine 1, attribute 1, with a specified mode of 5, an upper limit of 10, and a lower limit of 1. Similarly,
Figure 10 and
Figure 11 illustrate histograms for normal and beta distributions. The specified (and computed) parameters of the triangular distribution are mean 5 (5.34), mode 5 (5.13), and standard deviation 1.5 (1.82). The specified (and computed) parameters of the normal distribution are mean 5 (4.99), mode 5 (4.93), and standard deviation 1.5 (1.53). The specified (and computed) parameters of the beta distribution are mean 5 (5.0), mode 5 (4.95), and standard deviation 1.5 (1.34). Given the close agreement between specified and computed values of the parameters, the Monte Carlo simulation program can be considered an accurate computational model.
The impact of triangular, normal, and beta distributions were evaluated by running 10,000 simulations for the case of Nema 23 plate. Specifically, for the triangular distribution, the mode was set to the score assigned by the TEs along with lower and upper bounds of 1 and 10, respectively. For the normal distribution, the mean is the TE-assigned score and the standard deviation is 1.5. For the beta distribution, to have a mean of 5, mode of 5, and standard deviation of 1.5, the shape parameters specified are α = 4.44, β = 5.56, with a concentration of 10. Minor adjustments are made in the Monte Carlo simulation program to ensure that scores computed are bounded between 1 and 10 by rounding up/down to the nearest bound. As shown in
Table 10, the top two machines were consistent regardless of the distribution used. Therefore, in this case, the methodology can be considered robust to the probability distribution used to model the variation in human input.
The triangular distribution had the lowest frequency of occurrence of the top two recommended machines. From the frequency plots, all three distributions followed the same trend. The difference in the frequencies between different recommendations was more severe in the normal and beta distributions. There was very little difference between the orders within the top two machines, which was observed in the triangular distribution.
Figure 12 provides the frequency of the top eight recommended systems for each distribution.
To evaluate the sensitivity of the ranked order to the number of simulation runs, Monte Carlo simulations were run for 1000, 5000, and 10,000 runs. As shown in
Table 11, the top two machines were consistently M2 and M4 for all runs. Therefore, in this case, the methodology can be robust to the number of simulation runs beyond 1000.