ArchJIT: Just-in-Time Warning of Architecture Smells via Incremental Dependency Graph Evolution and Learning-Based Prioritization

Abstract

Software architecture tends to degrade during evolution, yet most studies analyze architectural decay offline at the version level. This limits early intervention when developers submit changes. We propose ArchJIT, a just-in-time warning framework for architecture smells that combines incremental dependency graph evolution with learning-based warning prioritization. ArchJIT updates a package-level dependency graph from changed source files, detects commit-induced smell events, generates warning candidates, and uses a lightweight binary retention model to select final warnings. We evaluate ArchJIT on multiple open-source Java repositories for detection effectiveness, incremental efficiency, structural discriminability, warning usefulness, and retention learnability. ArchJIT identifies smell-inducing commits at commit granularity, remains highly consistent with full reconstruction while reducing analysis cost, and emits concise warnings with interpretable evidence. Using calibrated retention labels, the retention model achieves 0.91 accuracy and a 0.89 F1-score on held-out candidates, and a small-scale expert study shows that final warnings are understandable and often comment-worthy. These results suggest that ArchJIT is a practical AI-for-software-engineering method for commit-time architecture warning.

1. Introduction

Software architecture is central to software quality [1]. It shapes modularity, maintainability, and changeability [2,3]. However, architecture does not remain stable as software evolves [4,5]. As developers submit new changes, they also change dependency structure, responsibility distribution, and module boundaries. Over time, these changes may accumulate and lead to architectural decay [6,7].

This problem is important because architectural decay usually does not emerge in one large step. It often grows through many small, ordinary commits. A developer may add one dependency, move one responsibility, or increase an existing coupling relation [8,9]. Each local change may appear acceptable. Yet repeated local changes may gradually create cycles, unstable dependencies, or overly central modules. As a result, systems may become harder to understand, modify, and maintain [10].

Many studies have shown that architectural smells and architectural decay can be studied from software history [7]. These studies have improved our understanding of how architectural quality degrades and how decay can be forecast from past evolution. However, most existing approaches work in an offline setting. They usually analyze releases, versions, or already integrated changes [11]. This setting is useful for retrospective assessment. It is less useful when developers need support during normal commit activity.

A similar observation can be made from defect prediction research. Prior studies have shown that fine-grained prediction can guide quality assurance efforts [12,13]. They also show that practical usefulness depends on more than predictive accuracy alone. Cost, effort, and output quality are also critical [14,15]. These lessons are directly relevant to architecture warning. A useful just-in-time warning approach for architecture smells must operate during normal commit activity, update its analysis efficiently, and produce a small number of relevant warnings that developers can inspect in practice.

This paper addresses this gap. We study just-in-time warning of architecture smells as an artificial intelligence application in software engineering. Our focus is not limited to smell detection. Our focus is the identification of smell events caused by commits, the efficient update of architectural state, and the generation of concise developer-facing warnings through learning-based prioritization. This focus differs from traditional architecture smell detection. It also differs from version-level forecasting of architectural decay. We study change-induced architectural events instead of only architectural states, and we further model final warning retention as a learnable decision problem over structured architectural evidence.

To support this goal, we propose ArchJIT, a hybrid framework based on incremental dependency graph evolution and learning-based warning prioritization. The framework replays the repository history commit by commit. For each commit, it updates a package-level dependency graph from changed Java files, compares the graph state before and after the change, detects architecture smell events, and generates warning candidates. To reduce subjective configuration, the framework estimates smell thresholds from repository-specific structural distributions and selects warning thresholds from validation data. Based on the resulting warning candidates, ArchJIT then applies a lightweight machine-learning-based binary retention model to determine whether a candidate should be retained in the final warning set.

This study is guided by two practical concerns. The first concern is efficiency. A just-in-time approach is useful only if it avoids the cost of rebuilding the full architectural state for every change. The second concern is usefulness. A warning mechanism is useful only if its output is concise, interpretable, and focused on relevant change events. Therefore, this paper evaluates not only whether smell-inducing commits can be identified, but also whether the analysis is efficient, whether the warnings are practical, and whether final warning-retention decisions can be effectively supported by artificial intelligence in a realistic software-engineering setting.

This paper makes the following contributions.

• We formulate just-in-time architecture smell warning as a repository-level problem based on commit-induced smell events.
• We propose an incremental dependency graph approach that identifies architecture smell events introduced or intensified by each commit.
• We introduce a hybrid warning strategy that combines calibrated architectural rules with a lightweight machine-learning-based binary retention model for warning prioritization.
• We conduct a multi-perspective empirical evaluation on open-source Java repositories. The evaluation examines effectiveness, efficiency, structural characteristics, warning usefulness, and the learnability of warning-retention decisions.

The results show that ArchJIT is a feasible AI-for-software-engineering application. The proposed framework preserves consistency with full reconstruction while reducing analysis cost. It also produces concise warnings with interpretable evidence. Moreover, the learned retention model shows that calibrated warning-retention decisions can be predicted effectively from structured warning features, while the developer study provides a separate small-scale check on the final warnings. These findings suggest that architecture analysis can move closer to daily development, support earlier intervention during repository evolution, and provide a practical AI-based warning mechanism for software engineering.

The rest of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the proposed approach. Section 4 describes the evaluation and reports the results. Section 5 discusses implications, limitations, and threats to validity. Section 6 concludes the paper.

2. Related Work

Recent research on software architecture has continued to examine architectural smells, architectural decay, and architecture analysis infrastructure. Garcia et al. [6] studied how architectural decay can be forecast from evolutionary history. Their work showed that historical structural and semantic information can be used to predict future architectural degradation. This line of research is closely related to the motivation of our study. However, their analysis is version-oriented and prediction-oriented. It does not target commit-time warning during repository evolution.

Gnoyke et al. [10] analyzed the evolution of software architecture smells in long-running systems. Their results showed that architecture smells remain relevant during system evolution and that some smell types, especially cyclic dependencies, are strongly related to structural degradation. Their study provides important empirical evidence on how architecture smells evolve over time. However, it remains an offline evolutionary analysis rather than a commit-level warning approach.

More recently, Kuo et al. [16] studied software architecture smell refactoring in 2024. Their work focused on architecture-smell-driven refactoring analysis and on prioritizing refactoring paths. Their work strengthens the practical motivation of our study, but it focuses on refactoring support, not commit-time warning during repository evolution.

Jolak et al. [17] further studied the empirical impact of architectural smells on maintainability in 2025. Their results provide additional evidence that several smell families are meaningfully associated with maintainability-related quality attributes. This recent result is relevant to our study because it reinforces the practical importance of early architecture-smell detection, but it still does not target commit-time warning during repository evolution.

Zhong et al. [7] investigated the impacts, causes, and solutions of architectural smells in microservices through an industrial study. Their work is important because it connects architectural smells with practical development problems and remediation concerns. However, the focus is on understanding the industrial effects of smells in microservices, not incrementally identifying smell-inducing commits in repository history.

Garcia et al. [11] proposed a shared infrastructure for software architecture analysis and maintenance. Their work improves interoperability and reuse across architecture analysis tools and datasets. This contribution is important from a tooling perspective, since architecture research often suffers from fragmented infrastructures. However, the work does not address commit-time smell warning, incremental dependency graph updating for change analysis, or calibrated developer-facing warning generation.

A second related line of work comes from just-in-time software quality prediction. Zhuang et al. [12] studied just-in-time defect prediction through AST-based change embedding. Their work showed that richer change-level representations can improve defect identification. This finding is relevant because it supports the value of commit-level analysis. However, their target is defect proneness, not architectural degradation.

Tabassum et al. [13] studied cross-project online just-in-time defect prediction. Their work is especially relevant because it emphasizes online updating and changing data distributions during software evolution. This perspective is closely related to the practical motivation of ArchJIT. However, their method remains defect-oriented and does not model architectural state, architectural smells, or architectural change events.

Sahar et al. [18] recently proposed IRJIT, a simple online just-in-time defect prediction approach based on information retrieval instead of expensive retraining. Their work is relevant because it highlights the practical importance of online updating speed and developer-facing explanation quality. However, the target remains defect prediction, not architecture smell warning, and the method does not operate on an explicit architectural dependency graph.

Ju et al. [19] proposed JIT-CF in 2025, which integrates contrastive learning with feature fusion for just-in-time defect prediction. Their work is relevant because it shows how recently learned representations continue to improve commit-level prediction quality. However, the target remains defect prediction, not architecture warning, and the method does not model repository-level architectural state or smell events.

Chen et al. [14] proposed a multi-objective just-in-time defect prediction method that combines expert metrics and semantic metrics. Zheng et al. [15] studied interpretability in just-in-time defect prediction and discussed how model outputs can be made more understandable in practice. These studies show how machine-learning models can support just-in-time software quality decisions. In contrast, our work applies a lightweight learning component to architecture-specific warning retention and models explicit architectural state, incremental dependency graph evolution, and architecture smell events.

Overall, existing architectural studies are strong in smell analysis, decay understanding, and architecture tooling [6,7,10,11]. Existing just-in-time quality prediction studies are strong in commit-level modeling, online adaptation, practical prediction output, and learning-based prioritization [12,13,14,15]. In contrast, ArchJIT combines these two perspectives. It analyzes architecture smell events at commit granularity, incrementally updates the dependency graph, and applies learning-based prioritization to final warning retention.

Table 1. Comparison with representative recent work.

Work	Main Focus	Arch. Specific	Cmt. Level	Incr. Analysis	Dev. Warn.	Learning Priority
Garcia et al. [6]	Forecasting architectural decay from history	Yes	No	No	No	No
Gnoyke et al. [10]	Evolutionary analysis of architecture smells	Yes	No	No	No	No
Zhong et al. [7]	Architectural smells in microservices	Yes	No	No	No	No
Garcia et al. [11]	Shared infrastructure for analysis	Yes	No	Partial	No	No
Zhuang et al. [12]	AST-change-embedding just-in-time defect prediction	No	Yes	No	No	Yes
Tabassum et al. [13]	Cross-project just-in-time defect prediction	No	Yes	Yes	No	Yes
Chen et al. [14]	Multi-objective just-in-time defect prediction	No	Yes	No	No	Yes
Zheng et al. [15]	Interpretable just-in-time defect prediction	No	Yes	No	Partial	Yes
Sahar et al. [18]	Online information-retrieval just-in-time defect prediction	No	Yes	Yes	Partial	Yes
Ju et al. [19]	Contrastive-learning just-in-time defect prediction	No	Yes	No	No	Yes
ArchJIT	Just-in-time architecture smell warning with learning-based prioritization	Yes	Yes	Yes	Yes	Yes

summarizes this comparison.

3. Approach

Our approach, ArchJIT, consists of four main steps. First, it replays repository history commit by commit and incrementally updates the package-level dependency graph from changed Java files. Second, it compares the graph state before and after each commit to identify architecture smell events and estimate their structural severity. Third, it calibrates smell thresholds and warning thresholds from repository data, so that both detection and warning selection are adapted to the current project. Fourth, it transforms detected events into developer-facing warnings through risk scoring, structural aggregation, root-cause deduplication, and learning-based warning prioritization. These four steps together allow ArchJIT to connect repository evolution, architectural risk detection, and AI-supported warning selection in one unified analysis process.

3.1. Commit Replay and Incremental Dependency Graph Construction

ArchJIT analyzes repository history in commit order. Dependency graph construction begins at the start of this commit replay process. Each commit is treated as one architectural change step. For a target commit, ArchJIT first extracts the changed Java files from version control history. It then parses the package declaration and import statements of these files and maps them to package-level dependencies. Based on these dependencies, ArchJIT updates one repository-level dependency graph incrementally for the current snapshot.

This step enables efficient architectural analysis at commit time and constructs the structural representation used in later stages of ArchJIT. In ArchJIT, each commit is represented not only as a code change, but also as a graph transformation over the architectural state. This representation is important because learning-based methods require structured inputs that preserve both local change information and global architectural context. At each replay step, ArchJIT maintains one directed repository-level graph snapshot. This graph may contain multiple disconnected components, but it is still treated as one evolving architectural state for the repository.

Let

$$G_{t-1}=\langle V_{t-1},E_{t-1}\rangle$$

(1)

denote the package-level dependency graph before commit $c_t$. Each node represents a package. Each directed edge represents a dependency from one package to another package. After processing commit $c_t$, the graph becomes

$$G_t=\langle V_t,E_t\rangle .$$

(2)

A direct solution is to rebuild the whole dependency graph from all source files after every commit. However, this solution is costly and weakens the practical value of commit-time analysis. ArchJIT therefore updates the graph only from changed files. For each changed file, it removes dependency contributions derived from the old file version and adds dependency contributions derived from the new file version. In this way, ArchJIT preserves architectural context from previous commits while avoiding repeated full reconstruction.

For a changed file, let p denote its package and let $\mathcal{I}$ denote the set of imported packages in that file. The dependency contribution of the file is defined as

$$D=\left\{\right(p,q)\mid q\in \mathcal{I},p\ne q\}.$$

(3)

If the file exists before the commit, ArchJIT computes its old dependency contribution $D^{-}$. If the file exists after the commit, ArchJIT computes its new dependency contribution $D^{+}$. The graph update then removes all edges in $D^{-}$ and adds all edges in $D^{+}$.

Algorithm 1 presents the incremental graph update procedure in ArchJIT.

Algorithm 1 Incremental dependency graph update in ArchJIT

Require:  Previous graph $G_{t-1}=\langle V_{t-1},E_{t-1}\rangle$, changed file set $F_t$ Ensure:  Updated graph $G_t=\langle V_t,E_t\rangle$   1: $V_t\leftarrow V_{t-1}$   2: $E_t\leftarrow E_{t-1}$   3: for each file $f\in F_t$ do   4:     parse the old version of f and obtain package $p^{-}$ and import set ${\mathcal{I}}^{-}$   5:     parse the new version of f and obtain package $p^{+}$ and import set ${\mathcal{I}}^{+}$   6:     if the old version of f exists then   7:         $D^{-}\leftarrow \{(p^{-},q)\mid q\in {\mathcal{I}}^{-},p^{-}\ne q\}$   8:         for each edge $e\in D^{-}$ do   9:            remove e from $E_t$ 10:         end for 11:     end if 12:     if the new version of f exists then 13:         $D^{+}\leftarrow \{(p^{+},q)\mid q\in {\mathcal{I}}^{+},p^{+}\ne q\}$ 14:         add $p^{+}$ and all target packages in $D^{+}$ to $V_t$ 15:         for each edge $e\in D^{+}$ do 16:            add e to $E_t$ 17:         end for 18:     end if 19: end for 20: remove isolated packages from $V_t$ if they no longer appear in any source file 21: return  $G_t=\langle V_t,E_t\rangle$

ArchJIT processes only changed files, which reduces analysis cost and supports commit-level replay. This procedure also preserves the evolving global graph across commits, which is important because architecture smells depend on structural context, not only on local file edits. In the current implementation, isolated packages are removed after every commit update, once dependency deletions and additions have been applied, so that packages no longer present in the current source snapshot do not persist in the graph.

Figure 1 shows a small generated graph example from the Hive repository. The left subgraph is the local package graph before commit 7baa55c6. The right subgraph is the graph after the incremental update. In this case, the added dependency from serde2 to objectinspector closes a two-package cycle. This local graph delta is the structural basis for later smell-event detection and warning generation.

This step produces the structural signals needed by later learning-based analysis. The graph snapshot before a commit, the graph snapshot after a commit, and the graph delta together form a commit-level architectural representation. This representation can support graph-based learning and risk ranking in later stages of the method. In this sense, incremental graph construction is not only an efficiency mechanism in ArchJIT. It is also the representation layer that links software evolution to data-driven architecture analysis.

3.2. Architecture Smell Event Detection and Severity Estimation

After the dependency graph is updated, ArchJIT determines whether the current commit changes the architectural risk state of the repository. The goal of this step is not only to detect whether a smell exists. The goal is to determine whether the current commit causes a smell event and how strong this event is.

ArchJIT focuses on three dependency-related architecture smells: cyclic dependency, hub-like dependency, and unstable dependency.

Let $S_{t-1}$ denote the smell state before commit $c_t$, and let $S_t$ denote the smell state after commit $c_t$. ArchJIT compares these two states and derives architecture smell events from their difference. Each smell event belongs to one of the following types:

• INTRODUCE. A smell instance appears after the commit and did not exist before.
• INTENSIFY. A smell instance exists before the commit and becomes more severe after the commit.
• MITIGATE. A smell instance becomes weaker or disappears after the commit.

In ArchJIT, INTRODUCE and INTENSIFY are treated as positive smell-inducing events. In the current implementation, any positive increase in severity is treated as an INTENSIFY event. ArchJIT does not apply an additional minimum change threshold at this stage. Instead, the magnitude of severity change is preserved as a continuous signal and is used later in risk scoring and warning ranking. These events can therefore serve as commit-level supervision signals for later learning-based analysis.

For a smell instance x, let $se{v}_{t-1}\left(x\right)$ and $se{v}_t\left(x\right)$ denote its severity before and after the current commit. ArchJIT defines the event type as

$$event\left(x\right)=\left\{\begin{array}{cc}\mathtt{INTRODUCE},\hfill & x\notin S_{t-1}\wedge x\in S_t\hfill \\ \mathtt{INTENSIFY},\hfill & x\in S_{t-1}\wedge x\in S_t\wedge se{v}_t\left(x\right)>se{v}_{t-1}\left(x\right)\hfill \\ \mathtt{MITIGATE},\hfill & x\in S_{t-1}\wedge x\notin S_t\hfill \\ \varnothing ,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

This formulation shifts the task from static smell detection to commit-induced architectural change analysis.

A cyclic dependency exists when a set of packages forms a strongly connected component of size greater than one in the directed dependency graph. Let C denote such a component. ArchJIT defines the severity of the cyclic dependency as $se{v}_{cycle}\left(C\right)$, which quantifies how structurally serious the cycle is in the current graph state. The severity combines two factors, namely the size of the cyclic region and the density of dependency relations inside it. Formally, ArchJIT computes it as

$$se{v}_{cycle}\left(C\right)=\alpha ·\left|C\right|+\beta ·|E_{in}\left(C\right)|$$

(5)

where $\left|C\right|$ is the number of packages in the cycle and $|E_{in}\left(C\right)|$ is the number of internal edges inside the component. In the current implementation, ArchJIT uses $\alpha =1$, $\beta =1$.

This definition gives higher severity to larger and denser cyclic structures.

A hub-like dependency exists when one package becomes too central in the dependency graph. For a package p, let $fanin\left(p\right)$ be the number of incoming dependencies and let $fanout\left(p\right)$ be the number of outgoing dependencies. Package p is treated as a hub when

$$fanin\left(p\right)\ge {\tau }_{in}\wedge fanout\left(p\right)\ge {\tau }_{out}$$

(6)

where ${\tau }_{in}$ and ${\tau }_{out}$ are repository-specific thresholds calibrated later. Its severity is defined as

$$se{v}_{hub}\left(p\right)=fanin\left(p\right)+fanout\left(p\right)$$

(7)

This measure reflects the degree to which structural responsibility is concentrated in one package.

For a package p, ArchJIT computes instability as

$$I\left(p\right)={\displaystyle \frac{fanout\left(p\right)}{fanin\left(p\right)+fanout\left(p\right)}}$$

(8)

A dependency from package $p_i$ to package $p_j$ is treated as unstable when the target package is less stable than the source package by a sufficient margin. Let

$$gap(p_i,p_j)=I\left(p_j\right)-I\left(p_i\right)$$

(9)

Then the dependency is treated as unstable when

$$gap(p_i,p_j)>{\tau }_u$$

(10)

where ${\tau }_u$ is the instability gap threshold. Its severity is defined as

$$se{v}_{unstable}(p_i,p_j)=gap(p_i,p_j)$$

(11)

A larger gap indicates a stronger violation of the desirable dependency direction.

Algorithm 2 presents the smell event detection procedure in ArchJIT.

Algorithm 2 Architecture Smell Event Detection in ArchJIT

Require:  Graphs $G_{t-1}$ and $G_t$ Ensure:  Smell event set ${\mathcal{E}}_t$   1: detect cycle instances in $G_{t-1}$ and $G_t$   2: detect hub instances in $G_{t-1}$ and $G_t$   3: detect unstable dependency instances in $G_{t-1}$ and $G_t$   4: ${\mathcal{E}}_t\leftarrow \varnothing$   5: for each smell family do   6:     for each smell instance x in the union of before and after states do   7:         compute $se{v}_{t-1}\left(x\right)$ and $se{v}_t\left(x\right)$   8:         if $x\notin S_{t-1}$ and $x\in S_t$ then   9:               add INTRODUCE event of x to ${\mathcal{E}}_t$ 10:         else if $x\in S_{t-1}$ and $x\in S_t$ and $se{v}_t\left(x\right)>se{v}_{t-1}\left(x\right)$ then 11:               add INTENSIFY event of x to ${\mathcal{E}}_t$ 12:         else if $x\in S_{t-1}$ and $x\notin S_t$ then 13:               add MITIGATE event of x to ${\mathcal{E}}_t$ 14:         end if 15:     end for 16: end for 17: return  ${\mathcal{E}}_t$

This step serves two roles in ArchJIT. It determines whether the current commit changes the structural health of the repository, and it also produces event labels and severity signals that support later learning-based warning prioritization. As a result, each analyzed commit is associated with both discrete event types and continuous structural strength. These outputs are then used in the next step for threshold calibration and later warning-stage learning.

3.3. Threshold Calibration for Smell Detection and Warning Selection

Thresholds affect both smell detection and warning output. If thresholds are fixed by hand, the method may become unstable across repositories. A threshold that is suitable for one repository may be too weak or too strict for another repository. This problem is important in ArchJIT because the detected smell events are not only analysis outputs. They also serve as supervision signals for later risk estimation and learning-based analysis. For this reason, ArchJIT calibrates thresholds from repository data.

ArchJIT uses two calibration processes. The first process calibrates thresholds for smell detection. The second process calibrates thresholds for warning selection. The first process determines how structural abnormality is recognized. The second process determines which warning candidates are finally retained.

For hub-like dependency, ArchJIT defines structural abnormality relative to the current dependency graph instead of by a fixed constant. Let

$${\mathcal{F}}_{in}^{\left(t\right)}=\{fanin\left(p\right)\mid p\in V_t\}$$

(12)

and

$${\mathcal{F}}_{out}^{\left(t\right)}=\{fanout\left(p\right)\mid p\in V_t\}$$

(13)

denote the fan-in and fan-out values of all packages in graph $G_t$, where $V_t$ is the package set of the dependency graph at commit step t, and p denotes a package in that graph. ArchJIT estimates the hub thresholds as percentile values of these distributions

$${\tau }_{in}^{\left(t\right)}=Q_{{\gamma }_{in}}\left({\mathcal{F}}_{in}^{\left(t\right)}\right)$$

(14)

$${\tau }_{out}^{\left(t\right)}=Q_{{\gamma }_{out}}\left({\mathcal{F}}_{out}^{\left(t\right)}\right)$$

(15)

where $Q_{\gamma }$ denotes the percentile operator, ${\gamma }_{in}$ is the percentile level used for the fan-in threshold, and ${\gamma }_{out}$ is the percentile level used for the fan-out threshold. A package is treated as hub-like only if

$$fanin\left(p\right)\ge {\tau }_{in}^{\left(t\right)}\wedge fanout\left(p\right)\ge {\tau }_{out}^{\left(t\right)}$$

(16)

This definition makes hub detection depend on the repository-specific structure. A package is therefore judged against the current graph distribution instead of against a universal constant.

For unstable dependency, ArchJIT first computes the set of positive instability gaps in the current graph

$${\mathcal{G}}_u^{\left(t\right)}=\left\{I\left(p_j\right)-I\left(p_i\right)\mid (p_i,p_j)\in E_t\wedge I\left(p_j\right)-I\left(p_i\right)>0\right\}$$

(17)

where $E_t$ is the dependency edge set of graph $G_t$, $(p_i,p_j)$ denotes a directed dependency from package $p_i$ to package $p_j$, and $I\left(p\right)$ is the package instability defined in the smell-detection step.

ArchJIT then estimates the instability threshold as

$${\tau }_u^{\left(t\right)}=Q_{{\gamma }_u}\left({\mathcal{G}}_u^{\left(t\right)}\right)$$

(18)

where ${\gamma }_u$ is the percentile level used for unstable dependency calibration. A dependency from $p_i$ to $p_j$ is treated as unstable only if

$$I\left(p_j\right)-I\left(p_i\right)>{\tau }_u^{\left(t\right)}$$

(19)

This strategy reduces subjective threshold selection. Moreover, it stabilizes label construction across repositories with different structural densities and dependency scales. As a result, the smell events detected by ArchJIT are better aligned with repository-specific evolution patterns.

Smell detection and warning generation are related but different tasks. A detected smell event does not automatically imply that the event should be shown as a final warning. If the warning policy is too permissive, ArchJIT may emit many low-value warnings. If the warning policy is too strict, it may suppress useful architectural signals. Therefore, ArchJIT calibrates warning thresholds separately from smell thresholds.

For a repository with a commit sequence

$$C=\{c_1,c_2,\dots ,c_n\}$$

(20)

ArchJIT divides the sequence into two ordered regions: a calibration region and an evaluation region. The calibration region is used only to select the warning policy. The evaluation region is used only to produce final warnings and to support empirical analysis.

Each warning candidate is associated with a risk score. ArchJIT calibrates two output parameters: K, the maximum number of warnings retained for each commit, and ${\tau }_r$, the minimum risk score required for emission.

A warning policy is therefore defined as $\pi =(K,{\tau }_r)$. Let ${\mathcal{W}}_{cand}$ denote the set of warning candidates in the calibration region, and let ${\mathcal{W}}_{emit}\left(\pi \right)$ denote the set of warnings retained under policy $\pi$. Let ${\mathcal{E}}_{pos}$ denote the set of positive smell events in the calibration region, that is, events labeled as INTRODUCE or INTENSIFY. ArchJIT evaluates each candidate policy by two criteria.

The first criterion is event coverage

$$Cov\left(\pi \right)={\displaystyle \frac{|{\mathcal{E}}_{cov}\left(\pi \right)|}{|{\mathcal{E}}_{pos}|}}$$

(21)

where ${\mathcal{E}}_{cov}\left(\pi \right)$ is the subset of positive smell events that remain covered by the retained warnings under policy $\pi$.

The second criterion is warning reduction

$$Red\left(\pi \right)=1-{\displaystyle \frac{|{\mathcal{W}}_{emit}\left(\pi \right)|}{|{\mathcal{W}}_{cand}|}}$$

(22)

A good warning policy should preserve coverage while reducing warning volume. ArchJIT therefore defines the ideal behavior as

$$Cov\left(\pi \right)\to 1\mathrm{and}Red\left(\pi \right)\to 1$$

(23)

To balance these two objectives, ArchJIT selects the warning policy that minimizes the distance to the ideal point

$$Dist\left(\pi \right)=\sqrt{{(1-Cov\left(\pi \right))}^2+{(1-Red\left(\pi \right))}^2}$$

(24)

The selected policy is

$${\pi }^{*}=arg\underset{\pi \in \Pi }{min}Dist\left(\pi \right)$$

(25)

where $\Pi$ is the search space of candidate warning policies, that is, the set of all candidate combinations of K and ${\tau }_r$ considered during calibration.

This formulation gives ArchJIT a simple but explicit optimization target. A policy is preferred when it retains more positive architectural signals and emits fewer warnings at the same time. In this sense, warning selection is treated as a calibrated decision problem instead of as a fixed reporting rule.

In the current repositories, this calibration objective often yields several tied or near-tied policies instead of one isolated optimum. Across the eight repositories with nonempty warning-policy grids, the selected trial and the second-best trial show the same average distance to the ideal point, and they also preserve the same average event coverage and warning reduction. This result suggests that the selected warning policy should be interpreted as one representative trade-off point under the current objective, not as a uniquely optimal policy in a stronger global sense.

The calibration objective also does not assume that uncovered events are harmless. Coverage has a direct complement, namely the residual miss rate of a policy:

$$Miss\left(\pi \right)=1-Cov\left(\pi \right)$$

(26)

where $Cov\left(\pi \right)$ is the event coverage defined in Equation (21). This value gives an explicit upper bound on the positive-event fraction not retained by the warning policy under the current event labels. Therefore, a selected policy should be read as a measured coverage-reduction trade-off, not as a no-harm guarantee for suppressed warnings.

This calibration step has two roles in ArchJIT. It improves methodological rigor by grounding thresholds in observed repository data. It also improves learning readiness by stabilizing both label construction and warning output. As a result, the architectural events detected by ArchJIT are more consistent across repositories, and the final warnings are better aligned with the practical goal of concise developer-facing feedback.

3.4. Warning Generation, Consolidation, and Ranking

After smell events are identified and thresholds are calibrated, ArchJIT transforms architectural signals into a small set of relevant and interpretable developer-facing warnings. This step also connects architecture analysis with artificial intelligence: final warning selection combines calibrated architectural rules with learning-based retention.

For each positive smell event $e\in {\mathcal{E}}_{pos}$, ArchJIT constructs one warning candidate, denoted by $w=\langle type,event,comp,root,files,evd,score\rangle$, where $type$ denotes the smell type, $event$ denotes the event type, $comp$ denotes the affected components, $root$ denotes the root-cause dependency edges, $files$ denotes the related changed files, $evd$ denotes the supporting evidence, and $score$ denotes the warning risk score. This representation preserves architectural semantics and ranking information, so it supports both warning generation and learning-based prioritization. The root-cause edge set is smell-specific: cyclic dependency uses new or removed dependency edges, unstable dependency uses the offending dependency, and hub-like dependency uses the event-specific root edges stored in the evidence. Structural aggregation then groups candidates by smell type and canonicalized affected components. Root-cause deduplication is applied only after this aggregation step.

ArchJIT treats warning generation as risk estimation over commit-induced graph events. The risk score combines four signals: event type, severity change, structural impact, and detection confidence. Event type distinguishes newly introduced problems from intensified existing problems. Severity change captures how much the smell grows after the commit. Structural impact reflects the size of the affected component region and contributing dependency relations. Detection confidence reflects how strongly the graph evidence supports the detected smell event.

Formally, let $s_{evt}\left(e\right)$ denote the event-related signal, $s_{sev}\left(e\right)$ denote the normalized severity-change signal, $s_{imp}\left(e\right)$ denote the structural-impact signal, and $s_{conf}\left(e\right)$ denote the detection-confidence signal. The final risk score is defined as a weighted combination of these signals. Its value increases when the event is newly introduced, when the severity increase is larger, when the affected graph region is broader, and when the structural evidence is stronger. In the current version of ArchJIT, the severity signal receives the largest weight, followed by event type and structural impact, while confidence serves as an additional stabilizing factor. This design keeps the score aligned with architectural meaning, not with a single metric.

The confidence signal is smell-specific. For cyclic dependency, confidence becomes higher when clear root-cause edges are observed and when the cycle involves more than one component. For hub-like dependency, confidence becomes higher when fan-in and fan-out exceed the calibrated thresholds by a larger margin. For unstable dependency, confidence becomes higher when the instability gap exceeds the calibrated threshold more strongly. As a result, the risk score is not only a reflection of structural change size. It also reflects how reliably the current evidence supports the detected event.

Raw warning candidates may contain several candidates that describe the same architectural issue. Emitting all of them would increase redundancy and reduce usefulness. ArchJIT therefore performs structural aggregation before the final output. Two warning candidates are placed in the same aggregation group when they belong to the same smell type and refer to the same structural scope. The structural scope is defined by the canonical form of the affected component set. Under this rule, an INTRODUCE event and an INTENSIFY event may be aggregated together if they refer to the same structural region. In other words, the first aggregation key is the pair of smell type and canonicalized structural scope, not the event type itself. This behavior is important because a developer usually needs to inspect one architectural issue instead of several local variants of the same issue.

Let

$$G=\{w_1,w_2,\dots ,w_m\}$$

(27)

denote one aggregation group, where $w_1,w_2,\dots ,w_m$ are the warning candidates assigned to that group. ArchJIT merges the evidence, related files, root-cause edges, and event labels of all candidates in this group and constructs one aggregated warning, denoted by $\overline{w}$. Here, $\overline{w}$ is the merged warning after structural aggregation. The score of the aggregated warning is determined mainly by the strongest member in the group, with a small upward adjustment when multiple events contribute to the same structural issue. This rule preserves the dominant signal while acknowledging that a repeated or mixed event pattern often indicates a more important risk.

Even after structural aggregation, different warning groups may still describe the same architectural cause. ArchJIT therefore performs root-cause deduplication. For each warning $\overline{w}$, let $R\left(\overline{w}\right)$ denote its root-cause edge set. The similarity between two warnings is measured by Jaccard similarity

$$J({\overline{w}}_i,{\overline{w}}_j)={\displaystyle \frac{|R\left({\overline{w}}_i\right)\cap R\left({\overline{w}}_j\right)|}{|R\left({\overline{w}}_i\right)\cup R\left({\overline{w}}_j\right)|}}$$

(28)

where ${\overline{w}}_i$ and ${\overline{w}}_j$ are two aggregated warnings. ArchJIT treats them as duplicates only when root-cause similarity is sufficiently high and affected components overlap:

$$J({\overline{w}}_i,{\overline{w}}_j)\ge {\tau }_d\mathrm{and}com{p}_i\cap com{p}_j\ne \varnothing$$

(29)

where ${\tau }_d$ is the deduplication threshold, and $com{p}_i$ and $com{p}_j$ denote the affected component sets. This rule keeps the final warning set focused on distinct structural causes instead of repeated symptoms.

A commit may touch many files, but not all of them are equally relevant to a given architectural warning. If all changed files are attached to every warning, the output becomes difficult to inspect. At the same time, attaching only the root-cause neighborhood can be too narrow for cyclic or hub-like smells. ArchJIT therefore uses a two-tier related-file context. Let $P_{root}\left(w\right)$ denote the set of packages appearing in the root-cause edges of warning w, let $P_{aff}\left(w\right)$ denote the set of affected components of warning w, and let $P_{chg}\left(w\right)$ denote the set of packages mapped from changed files in the same commit. The primary package region is

$$P_{pri}\left(w\right)=P_{root}\left(w\right)\cup (P_{aff}\left(w\right)\cap P_{chg}\left(w\right))$$

(30)

The secondary context region contains changed packages outside $P_{pri}\left(w\right)$ that are adjacent to the affected region in the current dependency graph or share the same top-level package prefix with an affected package. Let ${\mathcal{F}}_{pri}\left(w\right)$ and ${\mathcal{F}}_{ctx}\left(w\right)$ denote the changed files mapped to the primary region and the secondary context region, respectively. The final attached file set is

$${\mathcal{F}}_{rel}\left(w\right)=\mathrm{TopM}({\mathcal{F}}_{pri}\left(w\right)\cup {\mathcal{F}}_{ctx}\left(w\right),M)$$

(31)

where $\mathrm{TopM}$ ranks primary files first and then ranks secondary context files by graph proximity and package-prefix overlap. In the evaluation, $M=12$. This rule expands the file context beyond the root-cause neighborhood for wide smells, but still bounds the number of files that developers need to inspect for one warning.

After aggregation and deduplication, ArchJIT sorts the remaining warnings by risk score in descending order. Let ${\mathcal{W}}_{rank}$ denote the ranked warning set for the current commit. The calibrated warning policy

$${\pi }^{*}=(K,{\tau }_r)$$

(32)

is then applied to retain only warnings with sufficient risk and only the top-ranked warnings within the per-commit budget. Formally, the final warning set is

$${\mathcal{W}}_{final}=TopK\left(\left\{w\in {\mathcal{W}}_{rank}\mid score\left(w\right)\ge {\tau }_r\right\},K\right)$$

(33)

where $TopK(·,K)$ denotes the operator that keeps the highest-ranked K warnings from the filtered set. This definition separates ranking from output control. Ranking determines which warnings are more important. Calibration determines how many of them are finally shown.

ArchJIT also treats warning retention as a supervised learning problem. For each warning candidate w, it derives a feature vector $x\left(w\right)$ from architectural semantics and change context. The features include smell type, event type, heuristic risk score, affected component count, root-cause edge count, related file count, evidence count, source event count, and commit-level structural measures such as changed file count, added edge count, and impacted package count. The retention label is defined by the calibrated final warning output:

$$y_{ret}\left(w\right)=\left\{\begin{array}{cc}1,\hfill & w\in {\mathcal{W}}_{final}\hfill \\ 0,\hfill & w\notin {\mathcal{W}}_{final}\hfill \end{array}\right.$$

(34)

Based on these instances, ArchJIT applies a lightweight binary retention model

$$f_{\theta }\left(x\left(w\right)\right)=P(y_{ret}\left(w\right)=1\mid x\left(w\right))$$

(35)

to estimate whether a warning candidate should be retained. We use logistic regression because the candidate set is small, the label is binary, and feature effects remain inspectable. This model tests whether warning retention is learnable from structured architectural evidence without introducing a high-capacity opaque learner. It is trained on warning candidates from the calibration region and evaluated on candidates from the evaluation region. Larger comparisons among more complex learners require larger candidate-level datasets.

This learning component makes the warning stage an explicit AI-for-software-engineering step. Instead of treating final warning retention only as a reporting rule, ArchJIT models it as a learnable decision over structured architectural evidence. Thus, the warning stage combines calibrated architectural analysis with AI-based prioritization.

The output of this step is the final warning set for each commit, together with a candidate-level supervision structure that supports learning-based retention analysis. Each final warning contains the smell type, event information, affected components, root-cause edges, related files, evidence, and final risk score. These warnings are the main commit-level outputs of ArchJIT and are used in the empirical evaluation reported in the next section.

4. Evaluation

4.1. Study Objects and Research Questions

We evaluated ArchJIT on 10 open-source Java repositories as the main benchmark. These repositories were selected because they provide accessible evolutionary history, analyzable package dependency structure, and sufficient architectural variation for commit-level analysis. For each repository, ArchJIT replays the selected commits in chronological order and constructs commit-level architectural results.

Table 2. Study objects used in the evaluation.

Repository	Analyzed Commits	HEAD Java KLOC	Latest Packages
ActiveMQ	26	612 K	113
Ant	19	258 K	2
Cassandra	25	1313 K	26
Druid	19	1604 K	65
Flink	14	2223 K	106
Hive	19	2041 K	27
Pulsar	20	795 K	69
RocketMQ	8	323 K	107
Tomcat	17	539 K	74
Zookeeper	12	171 K	8
Total	179	–	–

reports the main benchmark. The table includes the number of analyzed commits, the Java source code scale at the current HEAD revision, and the number of packages in the latest analyzed dependency graph. In total, the dataset contains 179 analyzed commits. ArchJIT replays all selected commits for each repository. However, the main empirical results reported for RQ1 to RQ4 are computed on the evaluation region only. In these experiments, ArchJIT uses the earliest 20% of replayed commits as the calibration region and the remaining 80% as the evaluation region. This separation avoids using the same region both to choose the policy and to evaluate the final warning behavior. The selected repositories vary substantially in size, which helps assess ArchJIT under different architectural scales and reconstruction costs.

In addition to the main benchmark, we ran a supplementary transfer check on three additional local Java repositories, namely Tablesaw, SkyWalking, and Dubbo. To keep this extension lightweight and reproducible under local availability constraints, we selected up to the earliest 20 non-merge commits for each repository and used the same analysis pipeline as in the main study. As in the main pipeline, commits without Java-file changes were skipped during replay. The supplementary runs are not merged into the main RQ1 to RQ4 aggregates, because they use a smaller and more availability-driven sample.

Table 3. Supplementary transfer check.

Repository	Analyzed Commits	Replay-Level Smell Events	Positive Commits	Warnings
Tablesaw	20	10	5	0
SkyWalking	11	2	2	1
Dubbo	12	14	2	1

reports this extension. Across these three repositories, ArchJIT completed 43 commit-level analyses successfully. These results suggest that the pipeline transfers to additional open-source Java repositories, while also confirming that signal density varies substantially in small early-history samples.

The evaluation is organized around four research questions.

RQ1 examines whether ArchJIT can identify smell-inducing commits at commit granularity. This question evaluates the basic effectiveness of the method.

RQ2 examines whether incremental dependency graph updating can reduce analysis cost while preserving consistency with full reconstruction. This question evaluates the practical value of the incremental design.

RQ3 examines whether graph-derived and commit-level structural characteristics exhibit discriminative power for smell-inducing commits. This question evaluates whether the structural representation produced by ArchJIT is informative enough for later prediction and ranking.

RQ4 examines whether the generated architecture warnings are sufficiently concise, interpretable, and practical for developer-facing use, and whether the final warning-retention decision can be learned effectively through a lightweight machine-learning model. This question evaluates not only the usefulness of the final warning mechanism, but also the feasibility of learning-based warning prioritization.

4.2. RQ1. Can ArchJIT Identify Smell-Inducing Commits at Commit Granularity?

RQ1 examines whether ArchJIT can identify commits that introduce or intensify architecture smells. This question is fundamental to the whole study. If commit-level identification is not reliable, then the later analysis on efficiency and warning usefulness becomes less meaningful. In contrast, if smell-inducing commits can be identified at commit time, architectural governance can move from offline diagnosis to earlier intervention during repository evolution.

To answer this question, ArchJIT analyzed 140 commits in the evaluation region. Among them, 14 commits were identified as positive commits, that is, commits that introduced or intensified at least one architectural smell. This corresponds to a positive commit ratio of 10.00%. Across the same 140 commits, ArchJIT detected 55 architecture smell events in total and generated 19 warnings. These numbers describe different levels of analysis. The 140 commits are the evaluation instances, the 14 positive commits are the subset identified as smell-inducing, and the 55 smell events are the architectural events observed within these commits. The 10.00% positive ratio shows that smell-inducing commits form a minority class, which is consistent with the practical setting of commit-time quality analysis.

To further assess identification correctness, we conducted an independent dual-review manual validation on a broader sampled set from the full analyzed commit pool. The reviewed set contained all 24 commits labeled as positive by ArchJIT and 50 randomly sampled commits labeled as negative, for a total of 74 commits. Two independent reviewers examined the sampled commits separately and assigned one of three labels, namely positive, negative, or uncertain. One reviewer was a senior Java developer with 8 years of experience, and the other was a software architect with 10 years of experience. The protocol used partial blinding: the reviewers worked independently, could not see each other’s labels, and did not inspect the final warning outputs when assigning commit-level validation labels. Across the 74 reviewed commits, the two reviewers achieved 93.24% exact agreement, with a Cohen’s kappa of 0.89. After excluding uncertain cases, the two reviewers achieved 100% agreement on the remaining 62 commits. Importantly, no reviewed commit received a direct positive versus negative conflict between the two reviewers. All disagreements involved uncertain versus definite labels. The agreement pattern indicates that the manual review protocol was stable and that the main ambiguity came from borderline or weakly described commits instead of contradictory human interpretations.

Figure 2 shows the distribution of smell types among the 55 detected smell events in the evaluation region. CyclicDependency is the most frequent smell type, with 30 detected events, accounting for 54.55% of all events. UnstableDependency appears 20 times and accounts for 36.36%. HubLikeDependency appears 5 times and accounts for 9.09%. If only positive events are considered, cyclic dependency and unstable dependency still dominate. The architectural degradation observed in the current dataset is therefore mainly expressed through increasing cyclic structures and undesirable dependency directions.

Figure 3 shows the distribution of event types among the 55 detected smell events. ArchJIT detected 29 INTRODUCE events, 15 INTENSIFY events, and 11 MITIGATE events. This result is important for two reasons. First, it shows that ArchJIT not only detects the existence of smells. It detects change-induced smell events and uses them to characterize whether a commit worsens or mitigates architectural conditions. Second, it shows that repository evolution is not purely degradational. Although positive events dominate, some commits also mitigate architectural problems. This makes the event-based formulation more faithful to actual architectural evolution.

We further tested whether the positive labels are sensitive to small severity increases.

Table 4. Sensitivity to intensification threshold.

Minimum Severity Increase	Positive Events	Positive Commits
Default, $>0$	44	14
$>0.5$	35	11
$>1.0$	31	8

reports the results of applying stricter minimum severity-change thresholds to INTENSIFY events. Under the default setting, the evaluation region contains 44 positive events and 14 positive commits. When the minimum severity increase is raised to 0.5, these counts decrease to 35 positive events and 11 positive commits. When it is raised to 1.0, they decrease to 31 positive events and 8 positive commits. This sensitivity check shows that small intensification events do affect the number of positive labels. Therefore, the default setting should be interpreted as a recall-oriented operating point that preserves weak but observable structural degradation, while stricter settings provide higher-severity subsets.

The results also suggest that smell-inducing commits are not evenly distributed across repositories. Some repositories contain several positive commits in the evaluation region, while others contain none in the same analysis window. This observation indicates that architectural risk is repository-dependent and tends to concentrate in specific evolution phases instead of appearing uniformly over time. Although this repository-level variation is not the main focus of RQ1, it further supports the need for commit-level analysis instead of coarse version-level diagnosis.

Overall, the results support the effectiveness of commit-time architecture smell identification in ArchJIT. Positive commits are sparse but structurally meaningful, and the corresponding smell events exhibit clear semantic differences across smell families and event types. More importantly, the independent dual-review validation shows high inter-rater consistency and indicates that manual disagreement is concentrated in uncertain boundary cases, not in direct positive versus negative conflicts.

In summary, ArchJIT can identify smell-inducing commits at commit granularity. Across 10 repositories and 140 evaluation commits, it identified 14 positive commits, and these commits were associated with 44 positive smell events in total. In addition, independent dual-review validation on 74 sampled commits achieved 93.24% exact agreement and a Cohen’s kappa of 0.89, with no direct positive versus negative reviewer conflicts. These results support the effectiveness of commit-time architecture smell warning in ArchJIT while also showing that a small subset of commits remains genuinely uncertain under human inspection.

4.3. RQ2. To What Extent Does Incremental Dependency Graph Updating Reduce Analysis Cost While Preserving Detection Consistency?

RQ2 examines whether the incremental analysis in ArchJIT can reduce analysis cost while preserving the correctness of architectural detection. This question is important because a commit-time architecture warning is only practical if the architectural state can be updated efficiently. If each commit requires a full reconstruction of the whole dependency graph, then the method becomes costly for large repositories. At the same time, efficiency alone is not sufficient. The incremental result must remain consistent with the full reconstruction result. Therefore, RQ2 evaluates both efficiency and consistency.

Figure 4 summarizes the runtime difference between incremental analysis and full reconstruction. In each stacked bar, the lower segment represents the average incremental runtime, and the upper segment represents the additional runtime that would be required to perform full reconstruction. The percentage shown inside the upper segment indicates the proportion of runtime saved by the incremental procedure relative to the full reconstruction baseline.

Across the 10 repositories, the figure shows substantial runtime reduction in all cases, although the magnitude varies. The smallest gain appears in Ant, where the speedup is 9.30 times. The largest gain appears in ActiveMQ, where the speedup reaches 318.80 times. Tomcat also shows a strong reduction, with a speedup of 196.58 times. The average repository-level total speedup is 103.38 times. The pattern indicates that the benefit of incremental updating becomes more substantial as repository size and full reconstruction cost increase.

To better separate I/O-oriented cost from core architectural analysis cost, we further inspected the per-commit runtime breakdown. Across the 179 replayed commits, the average incremental total runtime is 2388.78 ms. Of this total, changed-file loading and diff extraction account for 2374.57 ms, or 99.41%, while parsing, graph updating, and smell detection together account for only 14.20 ms. In contrast, the average full-rebuild total runtime is 26,566.13 ms, of which full parsing alone accounts for 26,508.67 ms, or 99.78%. The breakdown clarifies that the observed end-to-end gap is mainly driven by avoiding repeated full parsing of repository snapshots.

Table 5. Runtime breakdown under different views.

Runtime View	Incremental ms	Full Rebuild ms	Full/Inc.
End-to-end replay	2388.78	26,566.13	11.12
Cached file list	14.20	26,512.62	1866.87
Core graph only	5.03	3.95	0.79

further reports three runtime views. The end-to-end row uses the complete replay path. The cached-file-list row removes changed-file loading from the incremental path and file scanning from the full-rebuild path, which approximates a setting where repository file lists are already available. The core graph-only row includes only graph updating or graph building, plus smell detection. Under the cached-file-list view, incremental analysis remains much faster because full reconstruction still needs to parse the whole snapshot. Under the core graph-only view, both paths are already millisecond-scale: incremental analysis takes 5.03 ms, and full reconstruction takes 3.95 ms. This result shows that the main practical benefit of incremental updating comes from avoiding repeated full-snapshot parsing and reconstruction during commit replay, not from faster isolated in-memory graph operations.

Consistency with full reconstruction is critical for the practical value of incremental analysis. Even if the incremental procedure is much faster, it would be of limited use if it produced noticeably different architectural results from the full reconstruction baseline. For this reason, we compare the two analysis paths in terms of both graph structure and detected smell events.

Consistency remains very high across the dataset. The average graph match rate is 0.98, and the average smell match rate is 0.99. Most repositories achieve perfect consistency for both graph structure and smell detection, while only a small number of cases show partial mismatch. Overall, these results indicate that the current incremental implementation preserves architectural analysis results very well in most repositories. At the same time, the non-perfect cases also suggest that incremental updating is not automatically equivalent to full reconstruction under all change patterns, and some repository-specific corner cases still warrant further investigation.

In summary, ArchJIT substantially reduces analysis cost while preserving high consistency with full reconstruction. Across 10 repositories, the average repository-level total speedup is 103.38 times, while the pooled ratio of mean end-to-end runtimes is 11.12 times. The runtime breakdown shows that the main saved cost is repeated full-snapshot parsing and reconstruction, not every isolated in-memory graph operation. At the same time, the average graph match rate is 0.98, and the average smell match rate is 0.99. These results support the practical value of incremental dependency graph updating for commit-time architecture smell warning.

4.4. RQ3. Do Graph-Derived and Commit-Level Structural Characteristics Exhibit Discriminative Power for Smell-Inducing Commits?

RQ3 examines whether smell-inducing commits exhibit structural characteristics that are clearly different from other commits. This question is important because ArchJIT is not intended only as a rule-based detector. It also aims to construct commit-level structural representations that are informative for later prediction and ranking. If positive and negative commits do not show meaningful structural separation, then the value of graph-derived features would be limited. In contrast, if positive commits consistently exhibit stronger structural disturbance, then the use of architecture-aware representations becomes well-motivated.

The dataset contains 24 positive commits and 155 negative commits. Figure 5 compares positive and negative commits along three structural dimensions, namely changed file count, added dependency edge count, and impacted package count. Because all three variables are strongly long-tailed, we use log-scaled empirical cumulative distribution curves instead of ordinary boxplots. This visualization makes the separation between the two groups clearer and avoids compressing most negative commits near zero.

The figure shows a clear and consistent separation between the two groups. For changed files, the median of positive commits is 121, whereas the median of negative commits is only 1. The difference remains large in the upper tail: the 90th percentile is 642 for positive commits but only 7 for negative commits. This means that smell-inducing commits are not merely slightly larger than ordinary commits. They are usually associated with substantially broader code-level disturbance.

A similar pattern appears for added dependency edges. The median of positive commits is 12, while the median of negative commits is 0. At the 90th percentile, positive commits reach 447 added edges, whereas negative commits remain at 0. This is an especially strong result, because it shows that the positive class is distinguished not only by larger change size, but also by explicit structural expansion in the dependency graph. In the current dataset, most negative commits do not introduce new dependency edges at all.

The same separation is also visible for impacted packages. The median of positive commits is 17, compared with 1 for negative commits. At the 90th percentile, positive commits affect 106 packages, while negative commits affect only 4. This result indicates that smell-inducing commits are associated with a much broader architectural footprint. In other words, the positive class is characterized not only by more file edits, but also by a larger structural propagation region in the package dependency graph.

To test whether this separation is explained only by commit size, we conducted a size-controlled baseline analysis on all 179 sampled commits using deterministic five-fold cross-validation. We also added a simple commit-message keyword baseline, which predicts a positive commit when the message contains architecture-related terms such as dependency, cycle, refactor, module, package, layer, component, coupling, interface, split, move, or moved.

Table 6. Baselines for smell-inducing commits.

Baseline	Accuracy	Precision	Recall	F1
Commit-message keywords	0.75	0.14	0.17	0.15
Changed-file count	0.92	0.66	0.88	0.75
Graph features	0.96	0.84	0.88	0.86
Size plus graph features	0.96	0.84	0.88	0.86

reports the results. The commit-message baseline performs poorly, with 0.75 accuracy and 0.15 F1. A size-only logistic baseline using changed file count achieves 0.92 accuracy and 0.75 F1. In contrast, a graph-only baseline using added dependency edges and impacted packages achieves 0.96 accuracy and 0.86 F1, and the combined size-plus-graph baseline achieves the same 0.96 accuracy and 0.86 F1. The main improvement is fewer false positives: 11 for the size-only baseline and 4 when graph-derived features are used, with recall remaining 0.88. The baseline comparison suggests that the discriminative signal is not explained by changed file count alone and is much stronger than a simple commit-message heuristic.

Overall, these three distributions support the central design of ArchJIT. Positive commits and negative commits are distinguishable not only at the label level, but also at the level of structural behavior. The additional baseline analysis further shows that graph-derived features remain useful after controlling for commit size. At the same time, the result should still be interpreted carefully. The current analysis shows a strong discriminative association, not a full causal explanation. Nevertheless, the consistent shift across all three dimensions indicates that the structural representation produced by ArchJIT captures signals that are closely associated with architectural risk.

Figure 6 provides a repository-level view of positive commit ratios. The distribution is clearly uneven across repositories. Hive shows the highest positive commit ratio, at 36.8%. Zookeeper, Druid, ActiveMQ, and Pulsar form a second group, with values around 15%. In contrast, Cassandra and Flink remain below 10%, and Ant has no positive commit in the current sample. This repository-level pattern suggests that smell-inducing commits are not uniformly distributed across project history. Instead, they are repository-dependent and tend to cluster in specific projects or evolution phases.

Overall, the results show that graph-derived and commit-level structural characteristics have strong discriminative power for smell-inducing commits. The difference is visible not only in average magnitude, but also across the full empirical distribution, especially in the upper tail. The size-controlled baseline analysis further shows that graph-derived features improve precision and F1 beyond a size-only baseline. This observation supports the use of graph-based structural representations for later prediction and ranking tasks.

In summary, smell-inducing commits are associated with substantially larger structural disturbance than negative commits. Across the current dataset, positive commits involve many more changed files, added dependency edges, and impacted packages, and the repository-level positive ratios further show that these events are unevenly distributed across projects. In addition, graph-derived baselines achieve 0.86 F1, compared with 0.75 for a size-only baseline. These results support the learning readiness of the structural representations constructed by ArchJIT.

4.5. RQ4. Are the Generated Architecture Warnings Sufficiently Concise, Interpretable, and Practical for Developer-Facing Use?

RQ4 examines whether ArchJIT produces concise, interpretable, and practical developer-facing warnings. Warning selection combines a calibrated heuristic policy with a lightweight binary retention model trained on calibration-region warning candidates and tested on evaluation-region candidates. We also conducted a small developer study in which two experienced participants independently evaluated the final warning set. Because commit-time warnings are useful only when developers can inspect them with reasonable effort, RQ4 evaluates warning consolidation, AI-supported warning prioritization, and small-scale developer-facing usefulness.

To answer this question, we analyzed the RQ4 results of 10 repositories. Among these repositories, four repositories produced neither warning candidates nor final warnings in the evaluation region.

Figure 7 shows the effect of warning consolidation. For visual clarity, the four repositories without warning activity are omitted from the figure, but they remain part of the study scope. Among the repositories shown, ArchJIT reduces raw warning candidates to a smaller final warning set in almost all cases. Flink has the largest reduction, from 21 candidates to 10 final warnings. Hive is reduced from 11 to 4, ActiveMQ from 4 to 2, Pulsar from 3 to 1, and Druid from 3 to 2. Zookeeper has one candidate but no final warning after filtering, showing that the calibrated policy can suppress low-value candidates completely.

Overall, the number of warning candidates decreased from 43 to 19, meaning that 24 candidates were filtered out, and the reduction ratio was 55.81%. At the repository level, the reduction ratio ranged from 33.33% in Druid to 66.67% in Pulsar and reached 100% in Zookeeper. This reduction pattern shows that warning consolidation plays a central role in controlling developer-facing output volume.

The reduction also creates an explicit coverage trade-off.

Table 7. Coverage of warning policy.

Stage	Positive Commits	Positive Events	Output Count
Warning candidates	14/14	43/44	43
Final warnings	9/14	20/44	19

reports this trade-off at the commit and event levels. Warning candidates are produced for all 14 positive commits in the evaluation region. After the final policy selection, warnings are emitted for 9 of these 14 positive commits, or 64.29%. At the event level, the final warnings cover 20 of the 44 positive events, or 45.45%. Thus, the residual miss rate is 35.71% at the positive-commit level and 54.55% at the positive-event level under the current policy. This result is important because it makes the cost of warning reduction explicit. The selected policy reduces warning volume, but it should not be interpreted as preserving every positive architectural signal.

Figure 8 shows the relation between warning quality and relative warning scope. The horizontal axis is the average ratio between related files attached to a warning and all changed files in the commit. The vertical axis is the average warning risk score, and the bubble size reflects the number of final warnings. This figure evaluates the two-tier related-file context used in the final warnings. Each warning can attach up to 12 related files. Across all 19 final warnings, the average attached-file count is 11.63, and the median is 12. Seventeen warnings use the full 12-file budget, while the other two attach 7 and 10 files. The average related-file ratio is 0.20, and the median ratio is only 0.02, showing that the method expands context beyond root-cause edges while keeping a bounded file subset for broad commits.

Warning scope also varies by repository. Flink has the largest number of final warnings, with an average risk score of 0.82 and an average related-file ratio of 0.02, because its warned commit changes 642 files, and each warning keeps a bounded context subset. Hive and Druid have average risk scores of 0.79 and 0.77, with related-file ratios of 0.11 and 0.27. Pulsar and ActiveMQ have higher related-file ratios, 0.71 and 0.95, because their warned commits are much smaller. These results show that related-file attachment is a bounded context-expansion mechanism, rather than a root-neighborhood-only explanation.

Together, the heuristic warning results show that ArchJIT balances warning reduction and interpretability. It reduces warning volume while keeping warning scope bounded, and repositories with stronger architectural issues, such as Hive and Druid, still retain higher-risk warnings after consolidation.

To further evaluate AI-supported warning prioritization, we trained a binary retention model on 92 warning candidates from the calibration region, of which 48 were retained by the calibrated ArchJIT policy. The model was then evaluated on 43 warning candidates from the evaluation region, of which 19 were retained. On the held-out evaluation candidates, the model achieved an accuracy of 0.91, a precision of 0.89, a recall of 0.89, and an F1-score of 0.89. These results indicate that the final warning-retention decision in ArchJIT is largely predictable from structured warning evidence. However, the target label is the calibrated ArchJIT retention decision, not direct developer acceptance. Therefore, this result is evidence of policy-consistent retention learnability and false-positive control, not independent proof of developer utility.

The most influential features in the learned model are the heuristic risk score, warning candidate rank, smell type, changed file count, and root-cause edge count. This pattern suggests that final warning retention depends jointly on local warning severity and broader structural disturbance. At the same time, repository-level prediction becomes less stable when only a very small number of warning candidates are available. Therefore, the current machine-learning result should be interpreted as evidence of retention learnability, not as a complete replacement for the calibrated warning policy.

We also compared the learned retention model against a simpler risk-score-only baseline. Using the calibration-region candidates to choose a single risk threshold, the best risk-only baseline selects a threshold of 0.591. On held-out warning candidates, this baseline achieves 0.81 accuracy and 0.82 F1. In comparison, the learned retention model achieves 0.91 accuracy and 0.89 F1. The main difference lies in the false-positive control. The risk-only baseline produces 7 false positives, whereas the learned model produces only 2. This result suggests that the learning-based component captures useful retention structure beyond a single score threshold.

To examine the stability of this comparison under the small held-out candidate set, we further computed bootstrap confidence intervals over the 43 evaluation warning candidates.

Table 8. Retention robustness on held-out candidates.

Model	Accuracy	Precision	Recall	F1
Risk-score baseline	0.81 [0.70, 0.93]	0.72 [0.54, 0.89]	0.95 [0.82, 1.00]	0.82 [0.67, 0.93]
Learned retention	0.91 [0.81, 0.98]	0.89 [0.74, 1.00]	0.89 [0.73, 1.00]	0.89 [0.77, 0.98]

reports the point estimates and 95 percent bootstrap intervals for the risk-only baseline and the learned retention model. The intervals are necessarily wide because the held-out set is small. Nevertheless, the learned model preserves a higher point estimate for accuracy, precision, and F1, while the risk-only baseline obtains higher recall by retaining more false positives. This result supports the use of the learned model as a precision-improving warning-retention component, but it also shows that the learning result should be interpreted as preliminary evidence, not as statistically final evidence of generalization.

To complement the repository-level indicators, we conducted a small developer study on the 19 final warnings emitted in the evaluation region. The study was not used to train or tune the retention model. Two experienced participants, a senior Java developer with 8 years of experience and a software architect with 10 years of experience, independently reviewed all warnings. Across 38 warning assessments, the average understanding score is 4.50 out of 5, relevance is 4.34, trust is 3.87, fix intention is 3.34, and review effort is 3.11. In addition, 27 assessments, or 71.05%, indicate that the participant would raise a review comment. The two participants agree on the comment decision for 18 of the 19 warnings, corresponding to 94.74% agreement and a Cohen’s kappa of 0.87. These results suggest that the final warnings are concise, understandable, and actionable enough to trigger review attention in a small expert-facing evaluation.

Finally, we conducted a retrospective PR-oriented case study using the GitHub-style Markdown comments exported by ArchJIT. All nine warned commits in the evaluation region had PR-style comment artifacts. In ActiveMQ commit 687bf43d, the generated comment reported one cyclic-dependency warning with risk 0.80, explicit root-cause evidence, and a focused related-file list, and both participants said they would comment. In Hive commit 8bc0920b, the comment combined a high-risk cyclic-dependency warning with a hub-like hotspot warning, and both participants again said they would comment. A weaker case appeared in Pulsar commit c2fc29f3, where the warning was localized to the discovery-service packages, but only one participant said they would comment. These cases suggest that exported PR-style warnings can support retrospective review and that stronger architectural signals are more consistently actionable.

In summary, the generated warnings are concise and practically interpretable for developer-facing use, and the calibrated final warning-retention decision is learnable from structured warning evidence. Across the repositories with nonzero warning activity, ArchJIT reduces warning volume by 55.81%, but this reduction also leaves a measurable residual miss rate at the final-warning stage. The learned retention model improves false-positive control over a risk-only baseline, and the small developer study shows that the final warnings are understandable, relevant, and often comment-worthy. These results suggest that ArchJIT improves output usability while providing a practical AI-based warning mechanism for software engineering.

5. Discussion

5.1. Implications

The results of this study support the main design idea of ArchJIT. Architectural degradation can be analyzed at commit granularity instead of only at version granularity. The results of RQ1 show that smell-inducing commits can be identified at commit granularity and that the detected events are structurally meaningful. The results of RQ2 further show that this analysis can be performed efficiently through incremental dependency graph updating, while still remaining highly consistent with full reconstruction in most repositories. The runtime breakdown also clarifies the source of this gain: ArchJIT mainly avoids repeated full-snapshot parsing and reconstruction, while isolated in-memory graph operations are already small for both paths. Together, these findings suggest that commit-time architecture analysis is feasible in practice when efficiency is evaluated at the replay and reconstruction level.

The results also have an important methodological implication. Existing research often studies architectural decay as a state of the system. In contrast, ArchJIT studies architectural decay as an evolving sequence of structural events. This distinction is important because architectural problems are often introduced gradually through ordinary commits. By focusing on smell-inducing events, ArchJIT moves the analysis closer to the actual process of architectural degradation. This event-based view is also more suitable for early intervention, since developers can be warned when a risky change is submitted instead of after degradation has already accumulated.

The results of RQ3 support the use of graph-based structural representations. Positive commits and negative commits differ substantially in changed file count, added dependency edges, and impacted package count. These differences show that smell-inducing commits are not only a label-level distinction. They are also associated with stronger structural disturbance. The size-controlled baseline analysis further shows that graph-derived features achieve higher precision and F1 than a changed-file-count baseline, which means that the observed separation is not explained by raw change size alone. This finding supports the design choice of ArchJIT as a learning-ready architecture analysis method. At the same time, the results should still be interpreted carefully, because structural association does not automatically imply full architectural causality under all repository histories.

The results of RQ4 highlight the practical importance of warning control. Raw architectural signals must be consolidated and selected before they become developer-facing output. The two-tier related-file context also changes warning consolidation from a root-only explanation into a bounded context-expansion step. Warning reduction has a measurable cost: the final warning policy covers 9 of 14 positive commits and 20 of 44 positive events in the evaluation region. This makes the trade-off explicit and prevents the selected policy from being interpreted as a no-harm optimum. The developer study and retrospective PR-oriented cases provide preliminary evidence that retained warnings can support review attention. From an AI-for-software-engineering perspective, learning-based retention functions as a core component of warning prioritization, while the current model still learns calibrated policy decisions instead of direct developer acceptance labels.

5.2. Limitations

This study has four main limitations. First, the evaluation scale is still limited. The main benchmark contains 179 analyzed commits and 140 evaluation commits, with 14 positive commits in the evaluation region. The learning-based retention analysis uses 92 calibration candidates and 43 held-out candidates. The supplementary transfer check on Tablesaw, SkyWalking, and Dubbo broadens the repository scope, but larger and more diverse evaluations are still needed. Therefore, the current results should be interpreted as evidence of feasibility and retention learnability, not broad statistical generalization.

Second, ArchJIT operates on a package-level dependency graph derived from Java source files and import relations. This abstraction is stable and efficient, but it may miss finer class-level or method-level relations, as well as dynamic architectural behavior such as reflection, dynamic proxies, and dependency injection effects. As a result, some architecturally relevant dependencies may not be fully visible in the current representation.

Third, several design choices in ArchJIT are fixed empirically in the current implementation. These include the definition of intensification, the chronological split for warning threshold calibration, and the scoring and deduplication settings. We report sensitivity analyses for warning calibration, intensification, and retention robustness, but the main pipeline still uses one operating point. Raising the minimum severity increase for INTENSIFY reduces evaluation positive commits from 14 to 11 at a threshold of 0.5 and to 8 at a threshold of 1.0. The selected warning policy is also often tied with nearby policies under the current calibration objective, and it leaves residual miss rates of 35.71% at the positive-commit level and 54.55% at the positive-event level. These results improve transparency, but they also show that some outcomes remain configuration-dependent. The binary retention model is trained on calibrated ArchJIT retention labels, not external developer preference labels. Logistic regression is used because the candidate set is small and interpretability is important, but larger candidate-level datasets are needed before comparing more complex learners reliably.

Fourth, the evaluation of warning usefulness remains small. RQ4 combines repository-level indicators with a developer study, but the study includes only two experienced participants and 19 final warnings. The results, therefore, provide preliminary evidence, not complete validation of developer utility in deployment settings. The related-file rule now expands beyond the root-cause neighborhood through secondary context files, but it remains bounded by a 12-file budget. This improves actionability compared with root-neighborhood-only attachment, but very wide cyclic or hub-like smells may still require additional context. A related limitation applies to RQ1: although the dual-review protocol shows high inter-rater agreement, some commits remain uncertain under human inspection.

5.3. Threats to Validity

A threat to internal validity comes from the implementation of dependency extraction and smell event detection. If parsing errors, file reconstruction errors, or edge construction errors occur, they may affect both the incremental and the full reconstruction results. We reduced this threat by using the same extraction logic in both analysis paths and by explicitly comparing incremental results with full reconstruction. However, the current study does not quantify the observed rate of parsing or extraction failures as a separate empirical variable. Another threat comes from threshold calibration. Although percentile-based calibration is more principled than fixed constants, it still depends on the observed repository distribution and may behave differently in repositories with unusual structural patterns. A threat to construct validity concerns the operational definition of architectural risk. ArchJIT detects INTRODUCE, INTENSIFY, and MITIGATE events, but smell-inducing commits are defined only through INTRODUCE and INTENSIFY, because MITIGATE represents improvement. This captures an important part of architectural decay, but not all forms of architectural quality degradation. Warning usefulness is evaluated through reduction, risk, scope indicators, and a small developer study, which cannot replace larger developer-level validation. Manual commit validation also remains approximate because some commits are difficult to classify decisively. Finally, the binary retention model learns calibrated ArchJIT retention behavior, not direct developer acceptance or actionability.

A threat to external validity concerns the generalizability of the results. The current results may not directly transfer to non-Java systems, mixed-language repositories, or industrial systems with different architectural conventions. The current package-level formulation may also behave differently in systems whose design structure is not well reflected by package organization. We reduced this threat slightly by adding a small supplementary open-source extension on Tablesaw, SkyWalking, and Dubbo, which shows that the pipeline can be replayed successfully beyond the original 10 repositories. However, this extension is still limited to Java open-source projects and does not substitute for broader industrial validation.

A threat to conclusion validity comes from repository heterogeneity. Some repositories contain many positive commits and warnings, while others contain few or none. This variation is itself an interesting empirical result, but it also means that aggregate results should be interpreted carefully. Strong performance in one group of repositories should not be over-generalized to all software projects.

6. Conclusions

This paper presented ArchJIT, a hybrid warning framework for architecture smells and an AI-for-software-engineering application based on incremental dependency graph evolution and learning-based prioritization. By combining commit-level dependency graph updating, architecture-smell event detection, threshold calibration, warning construction, structural consolidation, and machine-learning-based retention mechanisms, ArchJIT supports repository-level architectural monitoring during software evolution.

The empirical evaluation showed that ArchJIT can identify smell-inducing commits at commit granularity, maintain high consistency with full reconstruction while reducing analysis cost, and generate concise warnings with interpretable evidence. The size-controlled analysis, commit-message baseline, and runtime-breakdown analysis further show that graph-derived features contribute beyond raw commit size and simple message heuristics, and that efficiency gains mainly come from avoiding repeated full parsing of repository snapshots. These results support the feasibility of commit-time architecture warning and show that architectural degradation can be analyzed as an evolving sequence of structural events.

This study also shows that artificial intelligence can function as a core methodological component in software-engineering warning systems. At the warning-selection stage, ArchJIT uses a lightweight binary retention model to learn whether a warning candidate should be retained by the calibrated policy. The model outperforms a risk-score-only baseline and achieves strong held-out performance, showing that calibrated warning-retention decisions are learnable from structured architectural evidence. The developer study and retrospective PR-oriented case study further show that retained warnings are understandable enough to support review attention. Thus, ArchJIT is both an incremental architecture-analysis framework and an AI-enabled warning-prioritization method, although larger developer-labeled datasets are still needed to validate direct developer acceptance.

Overall, ArchJIT shows that software architecture analysis, repository evolution, and artificial intelligence can be integrated into a coherent just-in-time warning framework. The supplementary extension to Tablesaw, SkyWalking, and Dubbo suggests that the pipeline can transfer beyond the original 10-project benchmark, although broader industrial validation is still needed. Future work will extend this work toward finer-grained dependency models, richer language support, developer-labeled warning datasets, and more advanced graph-based and learning-based prioritization methods.