Using Genetic Algorithms for Research Software Structure Optimization

Abstract

Our goal is to generate restructuring recommendations for research software systems based on software architecture descriptions that were obtained via reverse engineering. We reconstructed these software architectures via static and dynamic analysis methods in the reverse engineering process. To do this, we combined static and dynamic analysis for call relationships and dataflow into a hierarchy of six analysis methods. For generating optimal restructuring recommendations, we use genetic algorithms, which optimize the module structure. For optimizing the modularization, we use coupling and cohesion metrics as fitness functions. We applied these methods to Earth System Models to test their efficacy. In general, our results confirm the applicability of genetic algorithms for optimizing the module structure of research software. Our experiments show that the analysis methods have a significant impact on the optimization results. A specific observation from our experiments is that the pure dynamic analysis produces significantly better modularizations than the optimizations based on the other analysis methods that we used for reverse engineering. Furthermore, a guided, interactive optimization with a domain expert’s feedback improves the modularization recommendations considerably. For instance, cohesion is improved by 57% with guided optimization.

1. Introduction

Remodularization refers to the process of reorganizing or restructuring the modular components of a software system [1]. This is typically done to improve code quality, maintainability, scalability, or to better align with evolving requirements. Remodularization involves changing the way a software system is divided into modules. This could include splitting large modules into smaller, more cohesive ones; merging modules that are tightly coupled; or reassigning responsibilities among modules to reduce dependencies. We optimize the module structure

• To improve maintainability (easier to understand, test, and modify parts of the software system),
• To enhance modularity (reduce coupling and increase cohesion),
• To better adapt to change (support new features), and
• To reduce technical debt (clean up legacy structures that no longer make sense).

Such an optimization is a recommendation for remodularization, whereby we do not change the semantics of the software.

We employ genetic algorithms for multi-objective optimization of module structures. Multi-objective optimization involves finding the best solutions for problems with two or more conflicting objectives that cannot be simultaneously improved. Our optimized objectives are low coupling and high cohesion. Thus, these metrics for coupling and cohesion are used as fitness functions in the genetic algorithm. We use six different analysis methods to compute these metrics based on the previous static and dynamic analysis of the existing software for reconstructing their software architecture. To fill the research gap, we analyze and discuss the impact of these analysis methods on the optimization results. Our specific application domain is research software [2], particularly Earth System Models (ESMs).

As contributions of this paper, we provide:

• Based on the recovered software architectures of existing research software [3], we show that genetic algorithms allow to generate recommendations for restructuring this research software.
• Specific remodularization experiments for two ESMs, namely the University of Victoria Earth System Climate Model (UVic) [4] and the MIT General Circulation Model (MITgcm) [5] are presented and discussed.
• To evaluate these recommendations for restructuring the ESMs, we collected feedback from domain experts who are involved in climate research as research software engineers. This feedback is also used for a guided, interactive optimization process.

In Section 2, we start with a look at the specific requirements of our application domain, which are the Earth System Models MITgcm and UVic. Then, we describe the reverse engineering process for architecture recovery in Section 3, which we applied in previous work [3] to MITgcm and UVic. Our genetic optimization process to generate restructuring recommendations is introduced in Section 4. The optimizations of MITgcm and UVic are presented in Section 5 and Section 6, respectively. We discuss our results in Section 7, threats to validity in Section 8 and related work in Section 9, before we conclude the paper in Section 10 with an outlook to future work.

2. Our Application Domain: Earth System Models

Research software is software that is designed and developed to support research activities. It can be used to collect, process, analyze, and visualize data, as well as to model complex phenomena and run sophisticated simulations. Research software engineering research aims at understanding and improving how software is developed for research [2]. Various categories of research software exist. Category 1.1 of [6] describes “Modeling and Simulation” as a role for research software. Such simulation software is usually continually modified and enhanced to provide new insights for specific research questions.

Earth System Models (ESMs) are such complex software systems used to simulate the earth’s climate and understand its effects on, e.g., oceans, agriculture, and habitats. They comprise different sub-models addressing specific aspects of the earth system, such as ocean circulation. Their code is, partly, decades old. Such scientific models can start as small software systems, which often evolve into large, complex models or are integrated into other models. Software engineering for such scientific models has specific characteristics [7], and involves dedicated roles such as model developers and research software engineers [8].

We chose UVic and MITgcm for our analysis, as they are used in many climate research projects, such as SOLAS [9], PalMod [10,11], CMIP6 [12], and the IPCC report [13]. Both ESMs are implemented in Fortran, run on Unix workstations without specific hardware requirements, and we collaborate with domain experts working with both models as research software engineers. MITgcm is referred to by the domain experts collaborating with us as a good example of a modern ESM, while UVic is a representative of a more traditionally developed ESM, whose development began in the 1960s.

A note on the term ‘model’ in climate science: In software engineering and information systems research, modeling is also an essential activity. Conversely to climate science, where climate models are the software systems to simulate the Earth, in software engineering and in information systems research, models are built as abstractions of the software systems themselves. A software architecture description is an example of such a model in software engineering [14], not to be confused with a (software) model of the Earth.

The ESMs are continually modified and enhanced to provide new insights for specific research questions. This leads to numerous variants and versions for an ESM, which must be maintained as other scientists intend to base their research on new features and reproduce results. Thus, ESMs are long-living software [15], and face typical risks, e.g., blurring module boundaries and architectural erosion, due to changes in functionality, hardware and language features. The resulting architectural debts [16] can limit further research and may harm the validity of scientific results. Furthermore, the lack of documentation and the loss of knowledge, as scientists move to other positions, limit the maintainability of ESMs. Therefore, the long-term development of ESMs faces unique challenges. Our goal is to determine how software engineering techniques and tools that have been successfully applied in other areas can assist in the development of ESMs.

To ensure the maintainability of ESMs, an understanding of the software architecture is necessary [17]. Documentation is usually rare in this domain. In particular, architecture documentation is, if it exists at all, often outdated and incomplete. Thus, rediscovering the architecture of ESMs is a key step to ensure their longevity, as it allows guiding architecture improvements and identifying areas that should be restructured. Architectural analyses provide an overview of the ESM’s structure and dependencies. They allow to identify interfaces that support developers in extending a model or adding alternative sub-models to an ESM. In addition, they help new scientists understand the model implementation. Furthermore, based on structural optimizations, developers can improve the architecture over time to facilitate future developments.

Thus, reconstructing the architecture of the simulation software via reverse engineering (see Section 3) is a key step to ensure its longevity, as it is a basis for guiding architecture improvements and identifying areas that should be restructured.

3. Reverse Engineering the Software Architecture

A prerequisite for a structural software optimization is the availability of a software architecture description of the existing software. Therefore, the first step is to reconstruct the software architecture of the software via reverse engineering, which can then be optimized. Here, we briefly summarize the relevant parts of our analysis approach; the full approach is detailed in [3]. Our analysis takes static and dynamic analysis, control flow and data flow into account to construct the required software architecture descriptions. Figure 1 presents an overview of our reverse engineering process. We start with recovering the architecture via reverse engineering. Then, the results of the dynamic and static analysis are combined. The recovered software architecture is then used as input for the subsequent optimization process, see Figure 2. Here, the recovered architecture is restructured via recombination and mutation. The obtained restructured architectures are then selected based on the fitness functions. The optimization process iterates until the involved metrics are below a given threshold (good enough).

We combine static and dynamic analysis techniques. Our analysis process can be configured to take any combination of static method-call analysis, dynamic method-call analysis, or static dataflow analysis into account, by merging the results of the individual analyses (this is the Combine Architectures step of Figure 1).

We categorize these methods into six different analysis methods, displayed in Figure 3. These methods form a hierarchy, e.g., the analysis method combined/call is obtained by merging the results of dynamic/call (only dynamic analysis) and static/call (only static analysis). The method static/both is obtained by merging the results of static/call and static/dataflow. At the top, all methods are combined into combined/both. These six analysis methods were chosen to systematically explore their impact on the optimization results. This way, we intend to improve the analysis of additional software systems.

The output of our analysis process, insofar as relevant for our optimization, is a coupling graph as a software architecture description of the analyzed software: The nodes of this graph are the compilation units of the analyzed software and its edges are call- or dataflow relations. There is an edge between functions f and g if the function g is called from the function f, or if data flows from f to g (this can be by a parameter, a return value, or by shared access of global data).

Reverse engineering MITgcm and UVic revealed that the architecture of an ESM differs significantly from that of, e.g., web-based systems, where the differences between dynamic and static analyses are much more significant [18]. A possible reason for this is that the design of an ESM can anticipate the way the code will execute much better than the design of an event-based system, where much of the actual behavior depends on events that cannot be predicted at design time.

4. Genetic Optimization to Generate Restructuring Recommendations

The output of our optimization process is recommendations to improve the module structure of the existing software systems. In addition, the optimization process also provides the developers with an overview of both the optimization potential and the involved trade-offs between the modularization options. We describe our genetic optimization approach in Section 4.1. The specific fitness functions for optimizing the module structure are introduced in Section 4.2.

4.1. Genetic Algorithms for Multi-Objective Optimization

Genetic algorithms are inspired by biology, and use recombination, mutation, and selection based on fitness functions to approximate optimal solutions. Figure 2 illustrates our architecture optimization process. The recovered architecture from the preceding reverse engineering process is first restructured via recombination and mutation. The obtained restructured architectures are then selected based on the fitness functions. The process can be configured to run either for a specified period of time, for a fixed number of generations, until the involved metrics are below a given threshold, or until no significant progress is achieved anymore in each step (good enough). Genetic algorithms are suitable to our domain, because they can significantly reduce the search space for optimal modularizations. For the results in the present paper, we let each optimization run for a maximum of two hours, with a population size of 1000, mutation and crossover rates of 0.125 each.

If one of these termination criteria is met, we generate improvement recommendations. Otherwise, we iterate with the next evolution step. We started initially with unguided optimizations, and subsequently complemented this with guided, interactive optimizations. This guided, interactive optimization takes expert feedback into account to restrict the search space. In our case, the fitness functions are the software quality metrics, which are introduced in the following subsection.

4.2. Software Quality Metrics as Fitness Functions

Our optimization process can be configured to use a wide range of different optimization metrics as fitness functions. For optimizing the modularization, we use coupling and cohesion metrics [1] to measure the quality of both the original software structure and the suggested optimizations. These metrics use the coupling graph, which is the output of our reverse engineering process (Section 3), to measure the fitness of the software’s modularization:

StrCoup: 

Structural coupling is defined as the average fan-out of components, i.e., the average number of edges connecting a program unit in the current component with a program unit in another component.

LStrCoh: 

Lack of structural cohesion is defined as the average number of pairs of unrelated units in a component, i.e., the number of pairs $(u,v)$ where both u and v are program units, and there is no edge connecting u and v in the coupling graph.

Both of these metrics are chosen such that small values are better (as is usual in multi-objective optimization).

As an example for the computation of our metrics StrCoup and LStrCoh, consider the coupling graph in Figure 4: The system consists of seven program units (e.g., Fortran files), grouped into two components: Component A contains the units $A_1,\dots ,A_4$, Component B contains the units $B_1,B_2,$ and $B_3$. The modularization (i.e., mapping of program units to components) is also indicated by positioning of the nodes in the component boxes and the color-coding in the graph. Edges between the nodes indicate calls in the program code: The edge $B_1\to A_2$ indicates that code in the unit $A_2$ is called from code in $B_1$.

We now show how to compute our software quality metrics for this example. For LStrCoh, we count the number of unrelated pairs in each component, i.e., the number of (undirected) pairs of units that are not connected with an edge in either direction. For the Component B, there is only a single unrelated pair $\{B_1,B_3\}$. Since there are 3 undirected pairs in B, the LStrCoh score of the Component B is therefore ${\displaystyle \frac{1}{3}}$. The component A has the two unrelated pairs $\{A_1,A_2\}$ and $\{A_2,A_3\}$ and 6 undirected pairs altogether, leading to the LStrCoh score ${\displaystyle \frac{2}{6}}={\displaystyle \frac{1}{3}}$. The LStrCoh score of the entire system is the average of the component scores, and is therefore also ${\displaystyle \frac{1}{3}}$.

For StrCoup, we compute the average fan-out of each component, i.e., the number of outgoing edges. Both components have two outgoing edges ($A_2\to B_3$ and $A_3\to B_3$ for Component A, $B_1\to A_2$ and $B_2\to A_2$ for Component B). Therefore, the fan-out of each component is 2, and the StrCoup value of the system (defined as the average fan-out of the components) is 2 as well.

We now perform a simple re-modularization of the system, by moving the unit $A_2$ to the Component B. The result can be found in Figure 5—the change is indicated by moving $A_2$ closer to the units in Component B and changing its color. The edges in the graph of course remain unchanged, since moving a program unit to another component does not change the caller–callee relationships between the units. Similarly as above, we now compute the metrics of the new system (recall that $A_2$ now belongs to the Component B). Component B still has the single unrelated pair $\{B1,B_3\}$, but since the package now consists of 4 units, it has 6 undirected pairs total, so its LStrCoh score is now ${\displaystyle \frac{1}{6}}$. Component A now has no unrelated pairs anymore, leading to a LStrCoh score of 0. The LStrCoh score of the entire system is the average of the component scores, i.e., ${\displaystyle \frac{1}{12}}$. The StrCoup score of the system has also improved: Each component now has a single outgoing edge ($A_3\to B_3$ for component A and $A_2\to A_4$ for component B, to which $A_2$ now belongs), leading to a StrCoup score of 1.

Therefore, moving the unit $A_2$ to the Component B improved both coupling and cohesion—the reason for this is that $A_2$ is more closely connected to the units in Component B than to those in A.

Obviously, there is a trade-off between coupling and cohesion, since one can easily reduce coupling by merging different components, which then often leads to less cohesive components (see Section 5.1 for examples). As usual when studying optimizations with multiple criteria, we call a modularization m undominated, if there is no other modularization that is strictly better than m with regard to one criterion (StrCoup or LStrCoh), and at least as good as m with respect to the other. There can be multiple undominated modularizations, the set of these is called the Pareto front. In multi-objective optimization, the Pareto front is the set of all Pareto efficient solutions. A situation is called Pareto efficient or Pareto optimal if all possible Pareto improvements have already been made; in other words, there are no longer any ways left to make one metric better off without making another metric worse. These modularizations represent an optimal trade-off between the involved metrics, in the sense that improving one metric is impossible without worsening the other metric. To compute an approximation of the Pareto front, we adapt the approach from Candela et al. [1] and apply the genetic algorithm NSGA-II [19].

In the following two sections, we apply our methods to the Earth System Models MITgcm and UVic.

5. Optimization of MITgcm Global Ocean

In the following, we compare optimizations of the MITgcm variant global-ocean-cs32x15 based on the different analysis techniques depicted in Figure 3. Results for other MITgcm variants can be found in the Reference [20]. Section 5.1 presents the results of our initial unguided optimization, and Section 5.2, the subsequent guided, interactive optimization.

5.1. Unguided Optimization of MITgcm Global Ocean

We present our first optimization result in Figure 6. This plot shows the result of running our genetic algorithm on the model obtained from the MITgcm variant global-ocean-cs32x15 by the dynamic/call analysis method. The result is a large set of new modularizations of the software. In the above plot, each modularization is represented as a dot expressing its StrCoup and LStrCoh values on the two axes. The original modularization is indicated by a red triangle .

In the table on the right-hand side of Figure 6, we present the values of our software quality metrics for some specifically chosen modularizations. Here, we can observe some “extreme” optimizations:

1.

The modularization opt-1238 achieves a coupling degree of zero, at the price of very low cohesion (recall that LStrCoh measures the lack of cohesion, so high values indicate low cohesion). Manual inspection of this modularization shows that opt-1238 merges components to obtain a modularization that consists of two unrelated components (and hence obtains coupling degree zero).

2.

An even more extreme modularization is opt-1260, which merges all components into a single one—with the “benefit” of fewer components, but at the price of a very low cohesion.

3.

A third “extreme” modularization is opt-180, which achieves full cohesion (i.e., zero lack of cohesion) while still improving the coupling value over the original modularization (1.72 instead of 2.84). However, a manual inspection of this solution shows that it distributes 96 units into 83 components, which are then mostly one-element components. Such components are cohesive, but obviously this is not a good way to structure a software system.

Obviously, modularizations such as the extreme ones opt-180, opt-1238, and opt-1260 are not our goal. More generally, optimizations that, compared to the original modularization, improve one of the two quality metrics (LStrCoh or StrCoup) and worsen the other are usually not relevant for practice. Therefore, we will now only consider optimization results that improve over the original modularization with regard to both StrCoup and LStrCoh, and focus on the solutions that only modestly increase the number of components.

In order to focus on modularizations that only use a reasonable number of components, we use larger and more colorful symbols for solutions introducing few additional components, and smaller, less colorful dots for solutions introducing many components (this is better visible in Figure 7, which zooms into the part of Figure 6 that contains the most relevant optimizations). As an example, results shown with a purple circle denote optimization results that introduce at most a single additional component compared to the original modularization of the software, while blue star-shaped symbols represent results that introduce three additional components, and the small blue dot stands for optimizations introducing at least 15 and at most 29 additional components (the complete list of symbols is given in the legend of the plots). In this way, the plots illustrate not only the (expected) tradeoff between the quality metrics LStrCoh and StrCoup, but also between the conflicting goals of improving these metrics and introducing a few new components.

As discussed above, the optimizations in Figure 7 contain many promising suggestions, an example is opt-467, which improves on both quality metrics compared to the original modularization (LStrCoh decreases from 68 to $28.85$; StrCoup decreases from $2.84$ to $2.05$) while introducing only a single additional component. In order to study the effect of the choice of the analysis method (see Figure 3) on the quality of the obtained optimizations, we also performed optimizations based on the other five analysis methods.

We start, in Figure 8, with the analysis method combined/both, i.e., the method taking all available data (both static and dynamic analysis, both call and dataflow relationships) into account. It is obvious from Figure 8 that the optimization results are of a much lower quality than the ones obtained from our dynamic-only analysis (Figure 7 above): all results that actually improve on the original modularization introduce many additional components, whereas Figure 7 shows many relevant improvements introducing only a single component (represented by the large purple dots).

Therefore, comparing the two optimization runs using dynamic/call (Figure 7) and combined/both (Figure 8), the two figures look very different, with the former yielding much more satifsfying results. To investigate the reason for these differences, we performed similar optimization runs with the remaining four analysis methods. Out of the three “base” methods (i.e., the lowest three in the presentation from Figure 3), the method static/dataflow is conceptually most different from dynamic/call, since it uses static instead of dynamic data, and dataflow instead of method–call relationships. We present the results of the corresponding optimization in Figure 9. It is obvious that the optimizations obtained from this method are problematic in the same way as those obtained from the combined/both method (Figure 8): there are again very few modularizations that improve on both our quality metrics without introducing many new components.

The third “base” analysis method from Figure 3 is static/call, which uses method–call relationships (as dynamic/call) based on static code analysis (as static/dataflow). The optimization results can be found in Figure 10. It is obvious that these results again share the same problems as the ones presented in Figure 8 and Figure 9, with no modularizations that significantly improve our quality metrics without introducing too many new components.

The two second-level analysis methods from Figure 3 lead to similar results and can be found in Figure 11 for static/both and in Figure 12 for combined/call.

As a consequence of our analysis of the optimization runs based on our six different analysis methods, it is obvious that the dynamic-only analysis dynamic/call stands out—it contains many more “interesting” optimization results, i.e., ones that are both an improvement over the original modularization and also do not add too many new components to the software. There are some differences between the modularizations based on the other five analysis types, but none of them yields improvements like dynamic/call. This shows that, as stated in the abstract, pure dynamic analysis produces significantly better modularizations than the optimizations based on the other analysis methods that we used for reverse engineering.

5.2. Guided, Interactive Optimization of MITgcm Global Ocean

The optimizations reported on in the preceding Section 5.1 took the entire model of the software (e.g., the MITgcm variant under analysis) into account and attempt to obtain a new global structure for the software. However, as confirmed in our discussion with ESM developers, applying the result of such a global optimization to a software system with a long development history and a large number of developers is not realistic. The ocean scientists were much more interested in more “local” optimizations, where the goal is not to find a completely new structure for the entire software, but to decompose an existing component into very few (ideally only two) new components.

To address this, we used our optimization software to decompose a single component (for MITgcm, the ‘src’ component was suggested to us as a realistic target, for UVic, the ‘common’ component plays a similar role). In the following, we present the results of these guided, interactive optimizations for the two analysis methods dynamic/call (Figure 13) and combined/both (Figure 14). The results for the remaining four analysis methods can be found in Figure 15, Figure 16, Figure 17 and Figure 18.

Since the optimization now only changes a single component, the search space and the optimization potential are very restricted. Surprisingly, this restricted search space leads to much better optimizations than the unguided optimizations reported in Section 5.1.

It is again obvious that the dynamic-only analysis dynamic/call stands out: The plot is much more sparsely populated than the other ones, i.e., the algorithm found fewer improvements here than for the other analysis methods. This is not surprising, since the number of units found in the src component is much smaller compared to the other analysis methods. The dynamic-only analysis found 51 units in the selected ‘src’ component, while the other analysis found between 371 and 393 units (see [3] for more details). Therefore, the search space for the dynamic-only analysis was much smaller than for the other analysis types.

The difference between the other five analysis techniques is again smaller than the difference between the dynamic-only analysis and the other five.

Another interesting observation is that results for our guided, interactive optimization are, in parts, even better than the results for the global optimization: Comparing, e.g., Figure 8 and Figure 14 (both using the static/call analysis method): The numerically best optimization opt-0 has almost the same StrCoup value ($12.44$ vs. $12.41$) in both results, but the single-component optimization has a much better LStrCoh value ($82.75$ vs. $166.74$). The best “single added component” optimization has both a better StrCoup value ($82.52$ vs. $91.26$), and a better LStrCoh value ($2411.93$ vs. $4225.30$) in the local optimization. The results are similar when comparing other lines in the corresponding tables: Even though the one-component optimization has a severely restricted search space, the optimization results are not clearly inferior to those of the global optimization problem.

A possible reason for this is that with the smaller search space, the algorithm can find the optimal values in this restricted space sooner. To test this hypothesis, we also let one global optimization problem run for a longer time (24 h instead of the 2 h for the other optimization runs). The result is presented in Figure 19 and shows that the optimal solution with respect to both StrCoup and LStrCoh (opt-0) is in fact much better with regard to both metrics than the shorter runs reported in Figure 8 and Figure 14.

6. Optimization of UVic

As a comparison to the MITgcm ESM studied above, we also applied our methods to the UVic ESM. Similarly to MITgcm above, we first performed an unguided optimization, taking the entire ESM into account (Section 6.1), and then, based on feedback from modelers, restricted our optimizations to the common component (Section 6.2). The results are similar to those for MITgcm presented above; we, therefore, keep the presentation short and highlight the differences. Again, Section 6.1 presents the results of our initial unguided optimization, and Section 6.2 the subsequent guided, interactive optimization.

6.1. Unguided Optimization of UVic

We present the optimization runs based on the two most “extreme” analysis methods, taking the least (dynamic/call, Figure 20) and the most (combined/both, Figure 21) information into account. The results of the other four analysis methods can be found in Figure 22, Figure 23, Figure 24 and Figure 25.

It is obvious that the differences between these two optimization runs are much smaller than between the corresponding methods in the MITgcm case, and that both optimizations suffer from the same problem as most of our optimizations of MITgcm: There are hardly any optimizations that improve both our quality metrics and do not add too many new components. The difference to the MITgcm case is that the optimization run based on dynamic/call in Figure 20 does not stand out as much against the other runs here as was the case for MITgcm. The reason for this is that, as discussed in [3], our dynamic analysis of UVic yielded a much more complete architecture model than that of MITgcm.

6.2. Guided, Interactive Optimization of UVic

Restricting the optimizations again to a single component of UVic has similar effects as in the case of MITgcm: The search space for the optimizations is very restricted, resulting in quite similar results for the different analysis techniques. We present the optimizations based on our different analysis techniques in Figure 26, Figure 27, Figure 28, Figure 29, Figure 30 and Figure 31.

The results of applying our methods to the Earth System Models MITgcm and UVic are discussed in the following sections.

7. Discussion

In general, our results confirm the applicability of genetic algorithms for optimizing the module structure of ESMs. We can draw two main observations from our experiments:

• The pure dynamic analysis produces significantly better modularizations than the optimizations based on the other analysis methods that we used for reverse engineering. This may be a specific observation for our application domain of research software: if we use the scientific model for a specific experiment, we can optimize the software architecture for this specific use case. Only those parts of the software that are actually used in the experiment are represented in the reconstructed architecture, which is then optimized.
• The guided, interactive optimization provides good modularizations. Thus, we can recommend feeding expert feedback into the optimization process.

We also observed differences between our example systems, MITgcm and Uvic. As demonstrated by the analysis, the existing legacy modularization of UVic is very poor with respect to the metrics LStrCoh and StrCoup. This coincides with the experience communicated by UVic developers.

A specific aspect of applying our methods to Earth System Models, which makes the problem different from general software remodularization is the fact that we are working with Fortran 95, which only need flat modularizations. Thus, we do not consider deeper nested structures, such as local classes. We discuss the technical challenges analyzing Fortran 95 software in [3]. Furthermore, refactoring Fortran software can be more difficult than refactoring, e.g., a Java 25 project, due to limited tool support.

For domain experts, support for program comprehension was another key takeaway, in addition to the restructuring recommendations. They found program comprehension helpful for their daily work, especially connecting the interface descriptions with the code base. From our discussions with the domain experts, we also obtained suggestions for other ESMs on which to apply our methods. This will be an area for future work.

8. Threats to Validity

Threats to validity are factors that can affect the conclusions drawn from our experiments. We discuss the threats to the internal, construct and external validity of our experiments below.

Internal validity refers to whether the observed effects in a software engineering study are genuinely caused by the experimental treatment (e.g., a new tool, method or technique) rather than by other factors. In our study, the effects of combining static and dynamic analyses in certain ways may be relevant. We address this possible threat by systematically combining our six analysis methods to explore their impact on the optimization results (see also Figure 3).

Construct validity refers to whether the study truly measures what it claims to measure. Besides static analysis, our approach relies on dynamic analysis. However, dynamic analysis approaches often have drawbacks, such as incomplete execution paths, which could lead to partial or biased architectural recovery. To mitigate this risk when obtaining execution traces, we selected representative test suites from the two ESMs. However, we are aware that representative traces cannot be guaranteed for every project.

External validity refers to whether the results of our study can be generalized beyond the study setting. Our experiments are applied to two ESMs. It remains unclear how well the proposed approach can be scaled up to other systems or types of research software. This issue may be addressed by applying our methods to additional software systems in future work.

9. Related Work

Praditwong et al. [21] investigated the application of multi-objective genetic algorithms to software restructuring. Mkaouer et al. [22] also applied genetic algorithms, with the additional goal of minimizing the resulting changes to the considered software. To the best of our knowledge, we are the first to apply genetic algorithms for restructuring ESMs. Sudhakar and Nithiyanandam [23] use clustering algorithms like K-Means and MAD-ENRNN to restructure object-oriented software. Maikantis et al. [24] also use genetic algorithms for restructuring software. As far as we know, our paper is the first one to compare optimizations obtained by dynamic and static analysis techniques.

Cortellessa et al. [25] employ genetic algorithms to optimize software architectures with respect to performance and reliability, while we optimize the module structures with respect to cohesion and coupling. Similar to our work, they integrate interaction with engineers into the optimization process.

Pourasghar et al. [26] also employ genetic algorithms to optimize module structures with respect to cohesion and coupling. Different from our approach, they integrate control flow into their software analysis, while we integrate data flow in software modularization.

10. Conclusions and Future Work

We applied our architecture optimizations to the ESMs UVic and MITgcm. The optimizations computed significant numerical improvements for restructuring recommendations for both ESMs. The pure dynamic analysis produces significantly better modularizations. Our methods and tools provide recommendations for subsequent restructuring activities.

In particular, guided, interactive optimization, such as restricting to a single component, helps to get better results, as demonstrated by the difference between the results in Section 5.1 and those in Section 5.2. In addition to the actual optimization results, getting an overview of the optimization potential is very helpful for the developers. As an example, compare Figure 8 and Figure 14, reporting the results from the guided/unguided optimization of MITgcm using combined/both: The best optimization adding only a single component (which is what, from our discussions, developers are most interested in) only a single component has LStrCoh/StrCoup values $4225.30$/$91.26$ in the unguided case (Figure 8), whereas there is such an optimization with metrics $2411.93/82.52$ in the guided case (Figure 14, recall that for our metrics, smaller values are better). Thus, cohesion is improved by 57% with guided optimization.

In the future, we plan to take into account additional information for our optimization recommendations. Our analysis so far was based on calls and dataflow relationships between operations. However, there are additional dependencies that we can utilize. One example for this is semantic coupling, which measures the similarity of, e.g., operation names. If the software under analysis uses a well-designed naming scheme, it can be beneficial to keep operations with similar names (or common prefixes in the names) in the same components. We implemented such a metric (using the edit distance from Levenshtein [27]) and used it to enrich the coupling graph with additional edges reflecting common prefixes of method names. First results indicate that this method is more beneficial for MITgcm than for UVic, possibly indicating that the former’s naming scheme for operations is more helpful for optimization. A preliminary take-away is that metrics beyond the standard coupling and cohesion metrics should be chosen very carefully, and this choice must take knowledge of the structure and possibly the development conventions (such as naming schemes) of the examined software into account. Other metrics that we plan to evaluate are test metrics discussed in [28] and clustering metrics from [29], which we anticipate can be used both in optimization and in interface discovery. We also intend to add, as a further quality metric, the edit distance between the generated solutions and the original modularization. We hope that this metric can serve as an approximation of the implementation cost of a modularization.

Another interesting area for future work is a thorough comparison of our optimization methods with alternative optimization techniques, like simulated annealing, hill climbing, multi-criteria decision-making, machine learning-based methods, or the Bunch clustering approach [30].