Comparing Machine and Deep Learning for Electricity Theft Detection from Monthly Billing Data in an Emerging Energy Market

Abstract

This study addresses a regime-conditioned question in electricity theft detection: when only monthly billing series and inspection-confirmed labels are available, which supervised model families recover irregular consumption without relying on the temporal resolution of advanced metering infrastructure (AMI)? The working sample comprises 4000 utility customers and 864 confirmed theft cases, each represented by 53 monthly kWh values from January 2021 to May 2025. After majority-class undersampling that retains all theft observations and construction of a balanced 1:1 learning set, eight classifiers are compared under an 80/20 stratified split: K-Nearest Neighbors, Decision Tree, Support Vector Machine, Random Forest, two dense multilayer perceptrons, Long Short-Term Memory, and a one-dimensional Convolutional Neural Network. Performance is assessed through threshold-optimized accuracy together with precision, recall, F1-score, the area under the receiver operating characteristic curve (AUC), and confusion matrices. On the hold-out test set, Random Forest and the compact dense network both reach an accuracy of 0.685; Random Forest attains the highest AUC (0.748) and F1-score (0.677). Even so, these models miss about one-third of the hold-out theft accounts (59 and 65 false negatives out of 173). Sequential deep models underperform on this short monthly regime. The results support ensembles and compact dense networks for monthly theft screening and indicate that AMI-oriented sequential gains do not transfer automatically to 53-point billing vectors under the present protocol.

1. Introduction

Electricity distribution systems lose energy through technical and non-technical channels. Technical losses arise from physical dissipation in lines and transformers; non-technical losses (NTLs) arise from theft, meter tampering, billing irregularities, and related administrative failures. The economic and operational consequences of NTLs are material: unrecovered energy reduces utility cash flow, shifts cost onto compliant customers through higher tariffs, and can delay maintenance and network expansion. NTL detection has been organized into data-oriented, network-oriented, and hybrid families, and reported classifier performance has been shown to depend on labelling quality, feature design, and metric choice rather than on algorithm novelty alone [1]. That framing remains central for emerging energy markets, where the decision problem is not only to invent a more complex detector but to identify which modelling family remains informative under the data that utilities actually observe.

In many industrialized settings, advanced metering infrastructure (AMI) provides half-hourly or daily smart-meter streams with dense temporal structure for convolutional and recurrent architectures. In emerging markets, the constraint is sharper. Incomplete AMI forces operators to screen accounts from monthly billing series, often with inspection labels that are costly to obtain and incomplete in coverage. Reviews of artificial intelligence for theft detection in distribution networks emphasize that operational data regimes frequently diverge from the AMI benchmarks used in method papers [2]. This divergence is not cosmetic: an architecture that exploits local half-hourly motifs may have little inductive advantage when each customer is represented by only a few dozen monthly totals.

Traditional detection still depends on field inspection and complaint-driven investigation. Those processes remain indispensable for label generation and legal confirmation, yet they do not scale to full customer portfolios. Pattern-based detectors can prioritize suspects, but that evidence still assumes richer smart-grid context than many utilities possess [3]. The practical implication is a triage problem: models must rank accounts so that limited inspection capacity is allocated where irregular behaviors is most plausible. Under that objective, false negatives leave losses unrecovered, while false positives consume inspection budgets and may erode customer trust.

Monthly billing is difficult for three practical reasons. Aggregation to monthly kWh collapses the intra-month irregularities that AMI detectors exploit. Confirmed theft labels are scarce because utilities inspect only a fraction of accounts, so the binary target may mix theft behaviors with inspection policy. Extreme class imbalance also rewards majority-class predictors and makes accuracy an unsafe headline metric.

The research question of this paper is therefore concrete. Under inspection-labelled monthly consumption, which supervised model families recover theft with acceptable precision–recall trade-offs, and do sequential deep architectures retain the advantage reported on AMI data? The question is regime-conditioned. It does not ask whether deep learning can detect theft in general; it asks whether the complexity justified by AMI evidence remains warranted when the input is a fixed-length monthly vector of length 53.

The empirical contribution is a controlled head-to-head comparison of four machine learning (ML) and four deep learning (DL) classifiers on the same monthly vectors, imbalance protocol, train–test split, and metric suite. The paper does not propose a new architecture. Its scientific value lies in testing transfer of modelling assumptions across metering regimes and in reporting not only scalar metrics but also confusion matrix error structure for the strongest configurations. Section 2 synthesizes research streams and states why prior evidence is insufficient for monthly emerging-market settings. Section 3 justifies every modelling decision for reproducibility. Section 4 reports results with emphasis on comparative metrics and confusion matrices. Section 5 discusses alternative explanations and limits of inference. Section 6 concludes with operational and research implications.

The study should be interpreted as a regime-specific benchmarking exercise rather than as a universal ranking of modelling families. Its scope is deliberately restricted to inspection-labelled monthly billing vectors, and its contribution lies in evaluating whether modelling advantages commonly reported under high-frequency AMI environments remain observable when utilities operate with substantially lower temporal resolution. The conclusions are therefore conditional on the data representation, sample, and experimental protocol considered here.

2. Review of Literature

2.1. Machine Learning Approaches to Electricity Theft Detection

The project corpus can be read as three interacting research streams rather than as a list of isolated papers. The first stream is survey and problem structuring. NTL detection has been structured as a data/network/hybrid taxonomy, with the warning that feature design and evaluation protocol can dominate reported gains [1]. That survey is useful precisely because it separates methodological claims from data artefacts: a high accuracy score may reflect class prevalence, label construction, or feature leakage rather than genuine theft signal. Household electricity-theft publications have been mapped and show a marked geographic concentration of case studies [4]. That concentration implies limited external validity for under-represented regions and supports the need for emerging-market evidence based on local billing and inspection practices.

The second stream is classical and ensemble supervised learning on consumption features. Boosted C5.0 trees have been used to detect NTLs with interpretable structure, which matters when utilities must justify inspection lists [5]. Extremely randomized trees obtain competitive theft detection, reinforcing that non-deep ensembles remain viable when inputs are tabular [6]. Stacked boosting classifiers have been applied to improve robustness, illustrating a movement from single learners toward ensemble aggregation [7]. Genetic algorithm feature selection has been coupled with supervised models, marking an evolution from raw classifiers toward search-based pipelines [8,9]. Deep feature extraction has also been combined with SVM, showing that hybrid designs often arise when neither pure deep nor pure classical families are trusted alone [10]. More recent ensemble and prototype-learning detectors continue this line under smart-meter assumptions: gradient-boosted monthly screening [11], prototype-based AMI detection [12], and related ensemble refinements [13]. Collectively, this stream suggests that tree and margin methods remain scientifically competitive whenever the representation is closer to billing features than to high-frequency load images.

The third stream is deep sequential and convolutional detection under AMI assumptions. CNN–LSTM stacks on smart-grid matrices have reported high detection rates and established a template that subsequent studies refine [14]. LSTM has been placed at the center of theft pipelines [15,16], and recurrent modelling has been extended with BiGRU designs aimed at noisy or weakly structured labels [17,18]. Later amalgams increase architectural complexity: DenseNet–GRU [19], RNN–BiLSTM–CRF [20], hybrid CNN security detectors [21], two-stage AMI deep models [22], ConvLSTM anomaly detectors [23], and hybrid KNN–LSTM AMI frameworks [24]. ANFIS has achieved strong rates but on thirteen synthetic theft scenarios overlaid on Irish half-hourly profiles [25]. The methodological evolution is therefore clear: as AMI data became available, research migrated toward deeper temporal architectures. What remains unsettled is whether that migration is justified when AMI density is absent.

2.2. Data Regimes and Model Performance

Dominant datasets in the high-impact segment of the corpus are AMI or smart-meter streams with dense temporal sampling [14,15,25]. Dominant algorithms have shifted from SVM and trees toward CNN–LSTM and related hybrids, yet ensembles remain competitive when features are tabular [6,7]. A recurring contradiction follows. Papers on high-frequency data often conclude that deep sequential models are necessary; tree and boosting studies conclude that classical ensembles suffice. These claims are not directly comparable because inputs, labelling protocols, and imbalance treatments differ. Apparent disagreement in the literature is therefore partly a disagreement about data regimes rather than a pure disagreement about algorithms.

Class imbalance is nearly universal and shapes metric interpretation. Comparisons of unbalanced data handlers for theft and defective meter classification show that the choice of balancing method can reorder model rankings [26]. The same long-tailed difficulty appears in adjacent energy infrastructure diagnosis: a prototype-driven contrastive framework for EV charging faults remains sensitive to imbalance ratios as extreme as 1:100 [27]. Robust resampling with stacked learners has been used to the same end [28], and RUSBoost-type mechanisms appear in related theft pipelines [15,29]. Active learning and explainable pipelines have also been proposed to reduce label demand and to support inspection decisions [30]. Semi-supervised routes pursue the same objective [31,32]. Synthetic attack models have further been shown to fail against real-world theft, undermining optimistic AMI-synthetic benchmarks [33]. Headline accuracies above 90 percent should therefore be read conditionally: they often arise where temporal density is high, fraud is synthetically injected, or evaluation does not stress prevalence-sensitive metrics.

Monthly or non-AMI settings exist but remain thinner. Classification with IoT-oriented features has been studied under less dense metering assumptions [34], and electricity and gas measurements have been used jointly for theft detection [35]. These studies broaden the evidence base beyond pure AMI, yet neither provides a matched ML-versus-DL comparison on inspection-labelled monthly vectors of fixed length 53. That absence leaves utilities without a direct answer to the deployment question that motivates this paper. The present study uses the literature to isolate that comparison; it does not attempt a comprehensive review of all theft detection algorithms.

2.3. Research Gap and Study Contribution

Previous work is insufficient for emerging-market monthly billing for three linked reasons. First, headline deep learning gains are identified mainly under AMI or synthetic theft regimes [14,25], so the inductive bias of LSTM and CNN is evaluated where temporal density exists. Second, imbalance remedies and metric reporting are often incomplete: accuracy-centric claims persist even when the choice of balancing method has been shown to invert operational rankings [26,28]. Third, few studies hold the input representation fixed as monthly kWh and then vary only the classifier family under one threshold-aware protocol. Without that control, superior performance cannot be attributed cleanly to model class.

This paper is necessary because utilities without AMI still need an evidence-based choice between ensembles and deep sequential models. The gap is not the absence of another hybrid network. The gap is the absence of a regime-conditioned comparison with inspection labels, monthly inputs, multi-metric reporting, and explicit interpretation of confusion matrix errors that determine inspection cost and unrecovered losses.

3. Methodology

This section describes the methodological framework adopted in this study to systematically evaluate and compare machine learning and deep learning approaches for electricity theft detection. The proposed framework is organized as a structured experimental pipeline consisting of seven sequential stages: data preparation, data cleaning, feature engineering, feature retention, model development, validation, and performance evaluation. Figure 1 summarizes that sequence. Each stage is presented together with the methodological rationale supporting the adopted decisions, providing a transparent and reproducible workflow for the conducted experiments. Furthermore, the use of a unified experimental framework ensures that all evaluated models are trained, validated, and assessed under the same conditions, enabling a fair comparison of their predictive performance while minimizing potential sources of methodological bias. Implementation uses Python (version 3.12.5) with scikit-learn for ML models and TensorFlow/Keras for DL models.

3.1. Data

The full utility extract contains 145,977 customer accounts and 53 aligned monthly kWh columns from January 2021 to May 2025. The binary target Fraude_V2 equals 1 for the 864 accounts with inspection-confirmed theft and 0 for the 145,113 accounts without a confirmed theft record (prevalence $864/\mathrm{145,977}=0.0059$). The zero class is therefore the complement of confirmed theft in the billing extract; it is not a claim that every negative account was inspected and certified as honest. The working file used for modelling contains $N=4000$ customers after constrained undersampling of that extract, including all $N_1=864$ inspection-confirmed theft cases and $N_0=3136$ non-theft cases. Each customer $i$ is represented by monthly consumption

$${\mathit{x}}_i=(x_{i,1},\dots ,x_{i,53}{)}^{\mathrm{\top }}\in {\mathbb{R}}^{53}.$$

(1)

Labels follow on-site inspection and technical verification of irregular practices affecting metered consumption. The label is assigned at the account level for the 53-month window; it does not identify which particular month contained the irregular practice. The classifiers therefore learn from the monthly consumption profile as a whole, not from a month-specific theft event.

The 53-month length is the complete aligned window available in the extract, not a tuned lookback chosen to maximize detection. A shorter window would answer a different operational question—how early theft can be flagged after a few bills—and would require a separate protocol. That extension is identified as future work rather than as part of the present comparison.

The 864 confirmed cases are heterogeneous rather than a single stereotypical profile. In the full extract, the median of each theft account’s mean monthly consumption is 124 kWh (10th–90th percentiles 79–586 kWh), against a median of 107 kWh among accounts with $y=0$. Within-account volatility is also higher on average among confirmed cases (median monthly standard deviation 54 kWh vs. 33 kWh). These descriptives do not identify a unique theft signature; they show that the positive class spans a wide range of consumption levels and variability.

The decision to retain every theft case and to cap the working sample at 4000 customers is deliberate. Discarding positives would waste the scarcest labelled resource. Capping sample size controls computation while preserving the full irregular class. Monthly kWh is used because it is the operational signal available under limited AMI, not because it maximizes information relative to smart-meter streams. The scientific claim of the paper is therefore tied to this representation: conclusions concern monthly billing vectors, not arbitrary load sequences.

3.2. Cleaning

Series are aligned to the common 53-month window. Predictors are standardized with training-fold statistics only:

$${\tilde{x}}_{i,t}={\displaystyle \frac{x_{i,t}-{\mu }_t}{{\sigma }_t}}.$$

(2)

No synthetic load injection is applied.

Fitting StandardScaler on the training fold prevents leakage of test moments into preprocessing. Avoiding synthetic attacks is equally important. Synthetic theft models can misrepresent real fraud [33]; injecting attacks would confound the claim that positive labels are inspection-based. Cleaning is therefore minimal by design: the study tests classifiers on verified monthly structure rather than on enriched or artificially corrupted profiles.

3.3. Feature Engineering

ML models and dense networks consume ${\tilde{\mathit{x}}}_i$ directly. LSTM and CNN receive the reshape

$${\mathit{X}}_i^{\mathrm{seq}}\in {\mathbb{R}}^{53\times 1}.$$

(3)

No handcrafted statistics such as mean, variance, skewness, or load factor are appended.

This restriction is methodological rather than computational. Engineered descriptors have demonstrated value on richer temporal representations [3,25], but adding them here would confound classifier comparison with feature enrichment. The same lesson appears outside theft detection. Dual-stream convolutional fusion after a continuous wavelet transform improves analog circuit fault diagnosis once the raw waveform is mapped into a time–frequency image [36], and a physics-inspired S-transform has been used to support lightweight classification of power quality disturbances [37]. Those gains depend on high-frequency traces that monthly billing totals do not provide. Holding the feature set fixed isolates inductive bias. If LSTM or CNN fail, the failure cannot be attributed to withholding standard descriptors from trees while granting sequences to deep models. Both families see the same monthly information; only the tensor shape changes.

3.4. Feature Retention

All 53 months are retained. No filter, wrapper, or genetic selector is applied.

With only 53 predictors, aggressive selection risks discarding weak but complementary monthly signals and would require a nested validation loop not present in the experimental log. Genetic algorithm feature selection of the kind already applied to theft detectors [8] remains a future extension. In the present design, feature retention keeps the comparison focused on model class rather than on an additional combinatorial search over subsets.

3.5. Class Imbalance Treatment

Theft prevalence in the 4000-customer working set is $864/4000=0.216$ (ratio approximately $3.6:1$), already much higher than the 0.59 percent prevalence of the full extract. A balanced learning set is then formed by sampling equal counts of theft and non-theft customers, so that $n_{\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}}=\mathrm{m}\mathrm{i}\mathrm{n}(N_1,N_0)=864$ and the learning set contains 1728 observations at ratio $1:1$. Figure 2 contrasts the skewed working distribution with the balanced target used for model fitting.

Training under a prevalence of 0.216 would reward majority-class predictors and inflate accuracy without theft recovery, the failure mode stressed when balancing choices reorder theft detection rankings [26]. Balancing enables comparison of discriminative capacity across model families. The scientific cost is external validity: metrics on a $1:1$ test fold do not equal metrics under live prevalence. That limitation is not hidden in preprocessing; it is treated explicitly in the discussion as a boundary condition on deployment claims.

The balanced design is therefore used as an analytical device to compare discriminative capacity under controlled class shares, rather than as an attempt to reproduce the prevalence that would be observed during routine deployment. Consequently, the reported precision, confusion matrix counts, and threshold-dependent metrics should be interpreted within the experimental setting. Operational implementation would require recalibration according to the utility’s actual prevalence, inspection capacity, and relative costs of false positives and false negatives.

3.6. Classification Models

Eight classifiers are estimated: K-Nearest Neighbors (KNN), Decision Tree (DT), Support Vector Machine with RBF kernel (SVM), Random Forest (RF), compact dense multilayer perceptron (DNN), wider dense multilayer perceptron (ANN), Long Short-Term Memory (LSTM), and one-dimensional Convolutional Neural Network (CNN). The labels DNN and ANN denote two MLP configurations rather than distinct theoretical families. The model set spans local, axis-aligned, margin, ensemble, dense nonlinear, and sequential inductive biases under one data regime.

3.6.1. K-Nearest Neighbors

Prediction follows the nearest-neighbor majority vote among $k$ neighbors [38]:

$$\widehat{y}\left(\tilde{\mathit{x}}\right)=\mathrm{mode}\{y_{\left(1\right)},\dots ,y_{\left(k\right)}\},$$

(4)

with optional distance weights $1/d(\tilde{\mathit{x}},{\tilde{\mathit{x}}}_{\left(j\right)})$. The grid searches $k\in \{3,5,11,21\}$ and weights in $\{\mathrm{u}\mathrm{n}\mathrm{i}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m},\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\}$.

KNN is included as a transparent local baseline. If theft cases form compact neighborhoods in monthly space, distance-weighted voting should recover them. If neighborhoods are unstable, KNN will overfit the training fold and collapse on the hold-out set. That contrast is scientifically useful even when KNN is not the final deployment choice.

3.6.2. Decision Tree

Nodes minimize Gini impurity [39]:

$$G=1-\sum _{c\in \{0,1\}}{p}_c^2,$$

(5)

and splits maximize impurity decrease. The grid includes $\mathrm{m}\mathrm{a}\mathrm{x}\_\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}\in \{5,10,15,20\}$ and $\mathrm{m}\mathrm{i}\mathrm{n}\_\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}\_\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\in \{2,5\}$.

The single tree provides an interpretable axis-aligned reference. Comparing DT with RF isolates the contribution of bagging and feature randomness from the contribution of recursive partitioning itself.

3.6.3. Random Forest

The ensemble probability is the average of the tree-level class probabilities [40]:

$$\widehat{p}(y=1\mid \tilde{\mathit{x}})={\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{p}_b(y=1\mid \tilde{\mathit{x}}).$$

(6)

The grid uses $B\in \{50,100\}$, $\mathrm{m}\mathrm{a}\mathrm{x}\_\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}\in \{5,10,15\}$, and $\mathrm{m}\mathrm{i}\mathrm{n}\_\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}\_\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}\in \{2,5\}$.

RF is the natural ensemble counterpart to DT and aligns with the strong tree stream in NTL research [5,6] without requiring AMI-scale sequences. Its probability output also supports threshold calibration, which is essential when inspection capacity is constrained.

3.6.4. Support Vector Machine

The soft-margin dual is implemented with an RBF kernel [41]:

$$K({\tilde{\mathit{x}}}_i,{\tilde{\mathit{x}}}_j)=\mathrm{e}\mathrm{x}\mathrm{p}(-\gamma \parallel {\tilde{\mathit{x}}}_i-{\tilde{\mathit{x}}}_j{\parallel }^2).$$

(7)

The grid searches $C\in \{0.1,1,10\}$ and $\gamma \in \{\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e},\mathrm{a}\mathrm{u}\mathrm{t}\mathrm{o}\}$, with probability estimates enabled for thresholding.

SVM remains a standard NTL baseline and appears as the second stage in hybrid detectors that combine deep features with a margin classifier [10]. Including it prevents an incomplete comparison that contrasts RF only with deep models and ignores margin methods.

3.6.5. Dense Networks (DNN and ANN)

The dense networks follow the back-propagation multilayer perceptron construction [42]:

$${\mathit{h}}^{\left(\mathcal{l}\right)}=\psi ({\mathit{W}}^{\left(\mathcal{l}\right)}{\mathit{h}}^{(\mathcal{l}-1)}+{\mathit{b}}^{\left(\mathcal{l}\right)}), {\widehat{p}}_i=\sigma ({\mathit{w}}^{\left(L\right)\mathrm{\top }}{\mathit{h}}^{(L-1)}+b^{\left(L\right)}),$$

(8)

trained with binary cross-entropy and Adam [43]. The compact DNN uses widths $(128,64,32)$ with dropout rates of 0.3 and 0.2. The wider ANN uses widths $(256,128,64)$, batch normalisation, and dropout rates of 0.4 and 0.3.

Two widths test whether additional dense capacity helps on 53 tabular inputs. If the wider ANN fails to beat the compact DNN, extra width and regularization are not supported by the data regime. That negative result would be informative: complexity is not free, and unjustified capacity can degrade calibration or recall.

3.6.6. Long Short-Term Memory

The LSTM gates follow the standard recurrent construction [44]:

$$\begin{array}{r}{\mathit{f}}_t=\sigma (W_f{\mathit{u}}_t+U_f{\mathit{h}}_{t-1}+{\mathit{b}}_f),\\ {\mathit{i}}_t=\sigma (W_i{\mathit{u}}_t+U_i{\mathit{h}}_{t-1}+{\mathit{b}}_i),\\ {\mathit{o}}_t=\sigma (W_o{\mathit{u}}_t+U_o{\mathit{h}}_{t-1}+{\mathit{b}}_o),\\ {\tilde{\mathit{c}}}_t=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(W_c{\mathit{u}}_t+U_c{\mathit{h}}_{t-1}+{\mathit{b}}_c),\\ {\mathit{c}}_t={\mathit{f}}_t\odot {\mathit{c}}_{t-1}+{\mathit{i}}_t\odot {\tilde{\mathit{c}}}_t,\\ {\mathit{h}}_t={\mathit{o}}_t\odot \mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\mathit{c}}_t\right).\end{array}$$

(9)

The implemented network is LSTM(64), Dropout(0.3), LSTM(32), Dense(16), and a sigmoid output.

LSTM is included because sequential gains have been reported on denser AMI series [14,15,16]. The present test asks whether that gain survives monthly sampling. If LSTM underperforms already in training, the limitation is more likely a missing temporal signal than test-set idiosyncrasy.

3.6.7. One-Dimensional Convolutional Neural Network

The one-dimensional CNN follows the standard convolutional construction [45]:

$$z_t^{\left(m\right)}=\psi (b^{\left(m\right)}+{\displaystyle \sum _{u=0}^{k-1}}{k}_u^{\left(m\right)}{u}_{t+u}),$$

(10)

with two blocks of 32 and 64 filters (kernel size 3), batch normalization, max pooling, dropout, Dense(32), and sigmoid.

CNN and CNN–LSTM hybrids are central in AMI theft detection [14,21]. Including CNN tests whether local monthly filters extract irregular motifs without LSTM memory cells. Weak CNN recall under monthly inputs would indicate that local convolutional structure, like recurrence, depends on finer temporal granularity than billing totals provide.

3.7. Validation

Within the balanced set, an 80/20 stratified split is applied with test_size = 0.2. The resulting folds contain 1382 training observations and 346 test observations, preserving class balance. ML hyperparameters are chosen by GridSearchCV with 3-fold StratifiedKFold scoring accuracy on the training subset only. DL models use validation_split = 0.2 from training data, a batch size of 64, at most 20 epochs, and early stopping on validation loss with a patience of 4 and restoration of best weights. After fitting, the decision threshold is selected to maximize accuracy on the evaluation fold under consideration:

$$t^{\star }={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}}_{t\in \mathcal{T}}\mathrm{Accuracy}\left(\mathrm{I}\right[{\widehat{p}}_i\ge t],y_i).$$

(11)

Selected ML configurations on the reported run are KNN with $k=11$ and distance weights; RF with $B=100$, a maximum depth of 15, and min_samples_split = 2; DT with a maximum depth of 5 and min_samples_split = 2; and SVM with $C=10$, $\gamma =\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e}$, and RBF kernel.

Stratification preserves class balance in both folds. Restricting grid search to the training subset avoids using the hold-out twice for hyperparameter search. Early stopping is used because DL capacity can overfit small tabular samples within 20 epochs. Threshold optimization acknowledges that the operational cutoff need not be 0.5. At the same time, optimizing the threshold on the same fold used for reporting can inflate threshold-dependent metrics relative to a nested scheme. Accuracy as the grid-search score is inherited from the experimental log; under imbalance it would be unsafe, which is why learning uses the balanced set. No bootstrap confidence intervals or paired significance tests are computed in the log; therefore, results are interpreted as descriptive rankings under a fixed protocol rather than as inferentially settled superiority tests.

For this reason, the hold-out results are not presented as definitive statistical evidence that one modelling family is universally superior to another. They provide comparative evidence under a common and fully reported experimental protocol. The interpretation consequently emphasizes consistent performance patterns, ranking quality, and the direction of operational error trade-offs rather than small numerical differences between individual configurations.

3.8. Metrics

With theft as the positive class,

$$\begin{array}{r}\mathrm{Accuracy}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}},\\ \mathrm{Precision}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}},\\ \mathrm{Recall}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}},\\ \mathrm{F}1={\displaystyle \frac{2\cdot \mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}.\end{array}$$

(12)

The area under the receiver operating characteristic curve (AUC) summarizes ranking quality across thresholds. Confusion matrices expose the asymmetric structure of errors that scalar summaries can conceal. The Matthews correlation coefficient and SHAP explanations are not in the experimental log; they are identified as required extensions rather than claimed results.

Multi-metric reporting follows the imbalance of the literature. Accuracy alone can mis-rank detectors when class shares are controlled by resampling [26,28]. In theft detection, precision approximates the yield of an inspection list, recall approximates the share of irregular accounts recovered, and AUC evaluates whether the score ranks theft above non-theft independently of one cutoff. Confusion matrices then translate those ratios into absolute counts of missed thefts and unnecessary inspections.

4. Results

4.1. Sample Distribution and Training Performance

Figure 2 shows the skewed working subsample against the balanced learning target. In the working set of 4000 customers, non-theft accounts dominate and the imbalance ratio is approximately $3.6:1$. After balanced sampling, both classes contributed 864 observations. This figure is a design check rather than a performance claim. It establishes why accuracy on the original prevalence would be an unreliable selection criterion and why subsequent tables are interpreted under controlled class shares.

Table 1. This study: training metrics on the balanced 1:1 sample (n = 1382; dimensionless).

Model	Type	Accuracy	Precision	Recall	F1	AUC	Threshold
KNN	ML	1.000	1.000	1.000	1.000	1.000	0.36
RF	ML	0.975	1.000	0.949	0.974	1.000	0.47
DT	ML	0.697	0.789	0.537	0.639	0.729	0.37
SVM	ML	0.627	0.644	0.567	0.603	0.677	0.47
DNN	DL	0.674	0.703	0.603	0.650	0.747	0.45
ANN	DL	0.610	0.633	0.522	0.573	0.668	0.48
LSTM	DL	0.580	0.621	0.411	0.495	0.606	0.48
CNN	DL	0.578	0.842	0.192	0.313	0.632	0.49

and Figure 3 report the training metrics for the retained configuration of each model family. The training results already distinguish between the model families by their learning behaviors. KNN with distance weighting achieves perfect in-sample separation (accuracy = 1.000, precision = 1.000, recall = 1.000, AUC = 1.000). RF is nearly saturated (accuracy = 0.975, precision = 1.000, recall = 0.949, AUC $\approx$ 1). Dense models fall in an intermediate range: DNN reaches an accuracy of 0.674 and AUC of 0.747, whereas the wider ANN performs worse (accuracy 0.610, AUC 0.668). LSTM and CNN already show weak theft recall during training (0.411 and 0.192, respectively), despite CNN reaching a precision of 0.842. These results suggest that flexible local and ensemble models can memorize the balanced sample, whereas sequential models fail to extract a stable theft signal even before external evaluation. The high precision but very low recall of CNN also indicates a conservative decision regime: when CNN predicts theft, it is often correct, but it rarely assigns the theft label.

4.2. Test Set Performance

Table 2. This study: test metrics on the balanced 1:1 sample (n = 346; dimensionless).

Model	Type	Accuracy	Precision	Recall	F1	AUC	Threshold
KNN	ML	0.647	0.678	0.561	0.614	0.671	0.36
RF	ML	0.685	0.695	0.659	0.677	0.748	0.47
DT	ML	0.639	0.700	0.486	0.573	0.636	0.37
SVM	ML	0.616	0.628	0.566	0.596	0.649	0.47
DNN	DL	0.685	0.711	0.624	0.665	0.711	0.45
ANN	DL	0.630	0.651	0.561	0.602	0.659	0.48
LSTM	DL	0.616	0.672	0.451	0.540	0.615	0.48
CNN	DL	0.558	0.738	0.179	0.288	0.581	0.49

and Figure 4 provide the primary comparative evidence. RF and DNN both reach a test accuracy of 0.685, so accuracy alone cannot select between them. RF yields a precision of 0.695, recall of 0.659, F1-score of 0.677, and AUC of 0.748. DNN yields a precision of 0.711, recall of 0.624, F1-score of 0.665, and of AUC 0.711. The accuracy tie therefore conceals a ranking and balance advantage for RF: a higher AUC indicates better separation across thresholds, and a higher F1 indicates a more even precision–recall trade-off. DNN is slightly more precise but recovers fewer theft cases.

KNN drops from a training accuracy of 1.000 to a test accuracy of 0.647, a direct overfitting signature of local memorization. DT reaches an accuracy of 0.639 but a recall of only 0.486, so many theft cases remain undetected. SVM remains below RF and DNN on both accuracy and AUC. LSTM (recall 0.451, F1 = 0.540) and especially CNN (recall 0.179, F1 = 0.288) miss most theft cases. Because this weakness is already visible in training, it is more consistent with a limited monthly sequential structure than with an unlucky hold-out draw.

Figure 5 evaluates whether score rankings, rather than one cutoff, separate theft from non-theft. RF and DNN dominate the ROC plane; KNN, ANN, and SVM form a middle tier; DT, LSTM, and CNN separate classes weakly. The RF advantage over DNN in AUC (0.748 vs. 0.711) is important precisely because threshold optimization can favor accuracy at one operating point. AUC shows that RF remains stronger across the continuum of possible inspection intensities. In operational terms, if a utility tightens or loosens the cutoff as inspection budgets change, RF preserves better ordering of suspicious accounts.

4.3. Confusion Matrices and Error Trade-Offs

Scalar metrics summarize performance; confusion matrices reveal how errors are distributed between missed thefts and unnecessary inspections. The analysis concentrates on RF and DNN because they share the best test accuracy and are the only configurations that combine competitive recall with competitive precision under the retained thresholds.

Figure 6 shows the RF training matrix on 1382 balanced observations. The counts are TN = 691, FP = 0, FN = 35, and TP = 656. RF therefore classifies every non-theft training account correctly and misses only 35 of 691 theft accounts. Training precision for theft is 1.000 and training recall is 0.949. This near-saturated in-sample profile indicates that the ensemble finds stable partitions in the monthly feature space, but it also warns against reading training matrices as deployment evidence.

Figure 7 shows the corresponding test matrix on 346 hold-out observations. The counts are TN = 123, FP = 50, FN = 59, and TP = 114. Relative to training, false positives rise from 0 to 50 and false negatives from 35 to 59. The generalization gap is therefore asymmetric in kind but comparatively balanced in magnitude on the test fold: RF issues 50 incorrect inspection flags and misses 59 true theft cases. Specificity on the test set is $123/(123+50)=0.711$, while theft recall is $114/(114+59)=0.659$. The scientific reading is that RF remains the most balanced of the eight models yet still leaves a material share of irregular accounts undetected and still imposes a non-trivial false-alarm burden.

Figure 8 reports the compact DNN training matrix: TN = 515, FP = 176, FN = 274, and TP = 417. Unlike RF, DNN does not saturate the training fold. It already commits many false positives and false negatives in-sample, with a training recall of 0.603 and specificity of $515/(515+176)=0.745$. The dense model therefore learns a softer decision surface: it never achieves the training purity of RF, but it also avoids the extreme memorization pattern of distance-weighted KNN.

Figure 9 shows TN = 129, FP = 44, FN = 65, and TP = 108. Compared with RF on the same test fold, DNN records fewer false positives (44 vs. 50) and more false negatives (65 vs. 59). This is the matrix-level counterpart of the scalar contrast between higher DNN precision (0.711) and lower DNN recall (0.624). DNN is more conservative in assigning the theft label: when it predicts theft it is slightly more often correct, but it recovers fewer irregular accounts. For a utility, that profile reduces wasted inspections relative to RF at the selected threshold while increasing unrecovered losses.

Read jointly, the four matrices support a precise comparison. RF learns almost perfectly in-sample and then relaxes on the test fold toward a nearly symmetric error profile (50 false positives vs. 59 false negatives). DNN is imperfect in-sample and remains more specificity-oriented on the test fold (44 false positives vs. 65 false negatives). Neither matrix is operationally negligible: under balanced prevalence, both models leave roughly one-third of theft cases undetected at the selected thresholds. The choice between RF and DNN is therefore not a choice between success and failure; it is a choice between a more recall-balanced ensemble and a slightly more precise dense network. Because RF also leads in AUC, it remains the preferred default when the utility can vary the threshold and needs stable ranking. If inspection slots are scarce and false alarms are unusually costly, the DNN matrix suggests a more conservative alternative at its selected cutoff.

5. Discussion

Under this monthly, inspection-labelled, balanced protocol, the data support three claims. First, RF and compact DNN are the strongest of the eight tested configurations on the hold-out fold, with RF preferred when ranking quality and F1-score are prioritized. Second, KNN’s perfect training fit does not generalize, so local memorization is insufficient as an operational strategy. Third, LSTM and CNN do not transfer their AMI-reported advantage to 53-point monthly vectors in this experiment: weak train and test recall indicate a missing temporal signal rather than mere test-set noise.

Claim three qualifies the deep sequential stream. CNN and LSTM gains have been obtained where dense sampling exists [14,21,23,24]; the present results indicate that those gains are regime-dependent. Conversely, the competitive RF outcome aligns with the ensemble stream already reported for interpretable boosted trees [5], extremely randomized trees [6], gradient-boosted monthly screening [11], and prototype-based AMI detectors [12], now stress-tested against matched deep baselines on monthly inputs.

Table 3. This study: qualitative comparison of the eight classifiers on monthly billing vectors.

Method	Main Advantage	Main Limitation
KNN	Recovers compact local neighborhoods without a parametric form	Memorizes the training fold and loses rank quality on hold-out
DT	Produces interpretable axis-aligned rules	Unstable and low theft recall once bagging is removed
SVM	Supplies a standard margin baseline	Weaker accuracy and AUC than RF and the compact DNN
RF	Best hold-out ranking and F1 on 53 monthly totals	Near-saturated training fit; less transparent than a single tree
DNN	Ties RF on accuracy and is slightly more precise	Recovers fewer thefts than RF at the selected threshold
ANN	Tests whether extra dense width helps	Additional capacity does not improve on the compact DNN
LSTM	Encodes sequential dependence when temporal density exists	Weak train and test recall on 53 monthly totals
CNN	Extracts local motifs without recurrent memory	Very low theft recall once consumption is collapsed to monthly bills

complements that ranking with a qualitative comparison of the eight methods. The table is not a second scoreboard; it states the inductive-bias trade-offs that Table 1 and Table 2 and Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 already illustrate under monthly billing vectors.

The contribution is therefore not that deep learning fails everywhere but that monthly billing does not supply the sequential richness those architectures are designed to exploit.

This interpretation should not be understood as evidence that sequential architectures are intrinsically unsuitable for electricity theft detection. Rather, it indicates that, under the present monthly resolution, sample size, architecture specifications, and validation protocol, additional sequential complexity did not translate into improved discrimination. Alternative architectures, broader optimizations strategies, or higher-frequency observations may produce different rankings.

Several non-architectural explanations remain viable and must temper any deployment narrative. Balancing to $1:1$ may favor models that separate classes under artificial prevalence; live deployment would require threshold recalibration under a prevalence near 0.22 or lower, and still more under the 0.59 percent prevalence of the full extract. Threshold optimization on the evaluation fold can raise accuracy and F1 relative to a nested threshold protocol; AUC partially mitigates this concern but does not eliminate it. DL training used at most 20 epochs with early stopping and no architecture search as broad as the ML grid; a deeper DL search might improve DNN or ANN, although it is unlikely to create monthly local dependence that CNN and LSTM require. Inspection labels may encode selection bias because only some suspects are visited, so models may learn correlates of inspection policy as well as theft physics. Finally, point estimates lack confidence intervals; the RF–DNN accuracy tie at 0.685 should not be read as statistically tested equivalence. The sample comes from one utility with one billing and inspection protocol, so the numerical ranking should not be transported to other emerging markets without local re-estimation.

These limits do not cancel the comparative signal. They bound it. The safest scientific statement is that, under the reported protocol and representation, ensembles and compact dense networks dominate sequential deep models for monthly theft screening, while absolute operating performance will shift when prevalence, threshold policy, and labelling coverage change.

For utilities restricted to monthly billing, the evidence favors deploying RF, or a compact dense model, with explicit threshold calibration tied to inspection capacity rather than defaulting to LSTM or CNN because AMI papers report high scores. The confusion matrices make the operational stakes concrete: on the balanced test fold, RF misses 59 theft cases and incorrectly flags 50 honest accounts, whereas DNN misses 65 and incorrectly flags 44. Those missed thefts remain the central operational limitation of the present detectors. Threshold movement can trade these counts against each other, but only a model with a stronger AUC preserves ranking quality while that trade-off is explored. Interpretability and lower tuning cost further favor RF for inspection lists, consistent with ranking-oriented operational needs discussed in the theft ranking literature [46].

6. Conclusions

This study examined electricity theft detection under a data regime that remains common in emerging energy markets and remains under-represented in the deep learning literature: monthly billing vectors paired with inspection-confirmed labels. The guiding question was not whether artificial intelligence can detect NTLs in principle but which supervised families remain informative when each customer is observed through only 53 monthly consumption values and when class imbalance must be handled explicitly before comparison.

Within a reproducible protocol—working set of 4000 customers, retention of all 864 theft cases, balanced learning, 80/20 stratified split, grid-searched ML models, early-stopped DL models, and threshold-aware multi-metric evaluation—Random Forest and the compact dense network both reach a test accuracy of 0.685. Random Forest records the best AUC (0.748) and F1-score (0.677), while the compact dense network is slightly more precise but less sensitive. Confusion matrix evidence clarifies the operational meaning of that contrast. On the test fold, Random Forest yields 114 true theft detections, 59 missed thefts, 50 false alarms, and 123 correct non-theft classifications. The dense network yields 108 true detections, 65 missed thefts, 44 false alarms, and 129 correct non-theft classifications. These counts show that the accuracy tie is not an operational tie: the ensemble recovers more irregular accounts at a modest increase in false alarms, and it ranks customers more reliably across thresholds. Even the strongest models, however, leave about one-third of hold-out theft accounts undetected.

The broader scientific implication concerns transfer across metering regimes. Much of the recent literature has migrated toward CNN, LSTM, and hybrid sequential architectures because AMI streams make local and long-range temporal structure available. The present results indicate that this migration should not be treated as unconditional progress. On short monthly series, sequential models underperform already in training and remain weak on the hold-out set. That pattern is more consistent with insufficient temporal granularity than with inadequate optimization alone. By contrast, Random Forest and a compact dense network extract a usable signal from the same monthly representation, aligning with earlier ensemble evidence while confronting deep baselines under matched conditions.

The study also yields a methodological reflection. Balancing, threshold optimization, and multi-metric reporting are not cosmetic additions; they determine whether a comparison answers a scientific question or merely rewards class prevalence. At the same time, the protocol has boundaries that future work must tighten. Evaluation on a balanced fold does not reproduce live prevalence. Threshold selection on the reporting fold can inflate threshold-dependent scores. Inspection labels may embed selection bias. Confidence intervals, nested threshold validation, shorter lookbacks, multi-utility samples, the Matthews correlation coefficient, and SHAP-based explanations are absent from the current experimental log and should accompany any stronger claim of deployability. Generative architectures that enrich scarce representations remain outside the present supervised comparison; a dual-discriminator quantum-convolutional adversarial design has recently been proposed for that purpose [47] and is a possible extension if synthetic monthly profiles are ever used to stress-test detectors.

For practice, the reflective conclusion is cautious but actionable. Utilities that possess only monthly billing data need not wait for full AMI coverage before building screening models, nor should they import AMI-oriented sequential architectures by default. A well-tuned Random Forest, calibrated to inspection capacity and monitored through confusion matrix counts rather than accuracy alone, offers the most defensible starting point among the eight families tested here. A compact dense network is a credible alternative when false alarms are especially costly. In both cases, model choice should remain subordinate to the metering regime, the labelling process, and the economics of inspection. Those constraints, more than architectural fashion, determine whether electricity theft detection research becomes operationally useful in emerging energy markets.