Next Article in Journal
Optimizing Schedule Duration of Repetitive Construction Considering Reducing Overtime Hours
Previous Article in Journal
The Application of Bio-Banding in Youth Soccer: A Systematic Review of Crossover Controlled Trials
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 16
Issue 9
10.3390/app16094301
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Benchmarking Sparse and Dense Models for Deception Detection in Negotiation: A Context-Aware and Imbalance-Sensitive Approach

by
Jae-Uk Kim
1,
Beom Jun Go
2,
Hwan Soo Yu
3 and
Soo Young Cho
1,*
1
Department of Game Studies, Kwangwoon University, Seoul 01897, Republic of Korea
2
Game Department, Kwangwoon University, Seoul 01897, Republic of Korea
3
Defense Acquisition Program, Kwangwoon University, Seoul 01897, Republic of Korea
*
Author to whom correspondence should be addressed.
Appl. Sci. 2026, 16(9), 4301; https://doi.org/10.3390/app16094301
Submission received: 5 April 2026 / Revised: 21 April 2026 / Accepted: 26 April 2026 / Published: 28 April 2026
Browse Figures
Versions Notes

Abstract

Automatic detection of deceptive intent in negotiation dialogue remains difficult because deceptive utterances are rare, context-dependent, and pragmatically subtle. This study develops a deployment-oriented evaluation pipeline for negotiation analytics using the Diplomacy corpus and compares sparse, dense, and imbalance-aware neural models under a unified protocol. The pipeline integrates context-window benchmarking, validation-based threshold selection, 10-seed robustness analysis, model-agnostic explanation case studies, and controlled perturbation stress tests. Across binary speaker-intention and receiver-perception tasks, contextualized inputs consistently outperform isolated utterances, confirming that deception-related interpretation is inherently sequential. The sparse term frequency–inverse document frequency (TF-IDF) model remains the strongest and most efficient overall benchmark, whereas stronger imbalance-aware neural baselines can improve minority deceptive-instance sensitivity at substantially higher computational cost. Error analysis further shows that socially mediated deception-quadrant prediction is markedly harder than direct binary in-tent prediction, with many failures collapsing toward majority straightforward cases. Controlled perturbation tests show that sparse modeling is especially stable under light-weight surface corruption, while neural robustness remains architecture-dependent. The main contribution is therefore a practical decision framework for selecting among efficient sparse monitoring, dense baselines, and minority-sensitive neural detection under operational constraints.

1. Introduction

Deception is a pervasive feature of strategic interaction. In bargaining, coalition formation, alliance maintenance, and multi-party coordination, speakers often benefit from selectively revealing information, overstating commitment, or presenting misleading plans in order to influence the beliefs and actions of others. In that sense, deception is not merely a psychological phenomenon; it is also a communicative and decision-theoretic process situated at the intersection of incentives, pragmatics, and relational management [1,2,3,4,5,6,7,8].
Prior work on deception has described deceptive behavior as a socially managed act associated with impression management, ambiguity control, and departures from conversational expectations [9,10]. Computational studies subsequently showed that deceptive language leaves detectable traces in review spam, online profiles, computer-mediated communication, and courtroom testimony [11,12,13,14,15,16,17,18]. Recent corpora and benchmark efforts extend this line toward more generalizable deception modeling, including cross-domain corpora and modern NLP evaluation settings [19,20,21].
Negotiation dialogue is a particularly valuable test bed because deceptive statements are embedded in sequential interaction rather than in isolated documents. Their interpretation depends on previous commitments, expected reciprocity, and the strategic consequences of future actions. This makes negotiation corpora fundamentally different from single-text deception datasets. In the NLP literature, negotiation has been studied through end-to-end bargaining dialogue, strategy-conditioned generation, and human-oriented negotiation corpora such as Deal or No Deal and CaSiNo [22,23,24,25,26]. Diplomacy is especially appropriate because alliances are negotiated through language and later enacted on the board, making deception strategically rational rather than incidental [27,28,29,30,31].
Methodologically, many recent NLP pipelines assume that dense contextual representations and transformer-derived sentence embeddings will dominate older sparse baselines [32,33,34,35,36,37]. Nevertheless, lexical models remain difficult to beat in many classification settings when the signal is sparse, domain-specific, and tightly linked to local wording patterns [38,39,40,41,42,43,44]. In highly imbalanced settings, moreover, accuracy alone can be misleading, and model assessment should include class-sensitive metrics, calibration-aware analysis, and threshold selection [45,46,47,48,49,50].
The present study addresses six questions. First, does local conversational context improve deception-related prediction in negotiation dialogue. Second, how do sparse lexical models compare with dense sentence embeddings in binary deception tasks and in a harder multi-class formulation. Third, can imbalance-aware transformer tuning improve deceptive minority sensitivity without collapsing to the majority truthful class. Fourth, do stronger neural encoders materially change the sparse-versus-neural trade-off. Fifth, after removing artificial separator tokens introduced during context concatenation, do interpretable lexical cues and model-agnostic explanations remain stable enough to support applied auditing. Sixth, how robust are the candidate models under lightweight perturbations that mimic surface corruption or context degradation.
In this study, speaker intention denotes whether the speaker intended to deceive in a given utterance, whereas receiver perception denotes whether the addressee perceived that utterance as deceptive; the deception quadrant combines these two dimensions into socially mediated outcome labels. To answer the questions above, we benchmark term frequency–inverse document frequency (TF-IDF) + logistic regression, Sentence–BERT embeddings + logistic regression, a compact imbalance-aware DistilRoBERTa extension, and a stronger DeBERTa-v3-base extension. Rather than treating the paper as a leaderboard comparison, we interpret each model–threshold pair as a candidate deployment policy whose value depends jointly on minority-sensitive predictive utility, calibration, robustness, interpretability, and computational cost. This study makes six contributions. First, it benchmarks sparse, dense, compact-neural, and stronger-neural models under a unified contextual evaluation protocol for negotiation analytics. Second, it corrects threshold evaluation by selecting operating points on the validation fold and applying them once to the held-out test fold. Third, it expands robustness analysis through a ten-seed study for the compact transformer and a controlled perturbation stress test across sparse and neural baselines. Fourth, it adds a stronger neural reference model (DeBERTa-v3-base) to reduce over-interpretation of compact-transformer limitations. Fifth, it complements cleaned sparse lexical analysis with model-agnostic Local Interpretable Model-agnostic Explanations (LIME) case studies for dense and neural baselines. Sixth, it clarifies when an efficient sparse monitor is preferable and when a minority-sensitive neural detector may justify its additional cost.
Figure 1 illustrates the end-to-end experimental workflow, beginning with the Diplomacy corpus and local context construction, followed by sparse, dense, and imbalance-aware transformer modeling, and concluding with threshold-based evaluation and interpretability analysis. The workflow is designed to compare candidate deployment policies in terms of predictive performance, minority-class sensitivity, robustness, and efficiency.

2. Materials and Methods

2.1. Datasets and Task Formulation

The primary dataset was the Diplomacy deception corpus introduced by Peskov et al. and distributed through ConvoKit (version 4.1.0) [28,32]. The corpus contains more than 17,000 private player-to-player messages from human Diplomacy games and provides utterance-level labels for speaker intention, receiver perception, and deception quadrants. Official training, validation, and test folds supplied with the corpus were retained throughout the study. Splits were respected before context concatenation so that contextual windows were built only within already assigned folds, thereby reducing the risk of cross-fold leakage.
Speaker-intention detection was selected as the principal benchmark because it offers the most direct setting for comparing an efficiency-oriented sparse baseline with imbalance-aware neural classifiers under severe class imbalance. Receiver-perception prediction and deception-quadrant prediction were retained as auxiliary tasks to examine whether the same evaluation logic transfers from direct deceptive intent to more socially mediated interpretations of deception. For clarity, speaker intention refers to the speaker’s annotated deceptive intent, whereas receiver perception refers to whether the interlocutor perceived the utterance as deceptive.
Table 1 summarizes the dataset configuration used for reproducibility, including the official split policy, leakage control, and context-window settings.

2.2. Context Construction and Preprocessing

Each utterance was evaluated in two forms: isolated message text and message text augmented with short local context. Context was created by concatenating up to the previous n turns from the same conversation. For the context-window ablation, n was set to 0, 1, 3, and 5. The primary speaker-intention comparison uses n = 3 as the main operating point because it offers a practical compromise between global performance, minority-class utility, and computational cost.
During the interpretability stage, artificial separator tokens used during context concatenation were removed and their lowercase derivatives were filtered from the sparse vectorizer. This produced a cleaned lexical analysis that could be interpreted as substantive rather than as an artifact of preprocessing.

2.3. Sparse and Dense Baselines

The sparse baseline uses TF-IDF features with unigram and bigram weighting followed by logistic regression. This design follows strong classical baselines in short-text classification and retains the advantage of direct coefficient-based interpretation [38,39,40,41,42,43,44].
The dense baseline uses Sentence–BERT embeddings (all-MiniLM-L6-v2) followed by logistic regression. This setup isolates the representation effect while keeping the downstream classifier simple. It therefore allows sparse-versus-dense comparison without confounding the results with additional classifier complexity [33,36].

2.4. Imbalance-Aware Transformer Extensions and Stronger Neural Reference

To address severe class imbalance in deceptive-utterance prediction, we implemented imbalance-aware transformer extensions for binary sequence classification with two output labels (Lie and Truth). The compact neural baseline used DistilRoBERTa, whereas the stronger reference model used DeBERTa-v3-base. Because the deceptive class is rare, standard fine-tuning can collapse to the majority truthful class. We therefore extended the neural pipeline with weighted random sampling, class-weighted cross-entropy, early stopping on validation Lie-class F1, and validation-based threshold selection [34,35,50,51].
The compact DistilRoBERTa configuration was retained as the principal robustness-tested neural baseline because it makes the cost-sensitive sparse-versus-neural trade-off most visible, while DeBERTa-v3-base was added as a stronger reference point to test whether the conclusions change under a larger encoder. This distinction allows the paper to compare compact-neural behavior, stronger-neural behavior, and sparse efficiency within a single deployment-oriented framework.
Figure 2 illustrates the neural classification pipeline, beginning with raw Diplomacy messages and local contextual input, followed by tokenization, contextual encoding, pooled representation, and linear classification into Lie or Truth labels. It also highlights weighted random sampling, weighted cross-entropy loss, validation-based early stopping, and deployment-oriented threshold tuning used to mitigate majority-class collapse. The stronger DeBERTa-v3-base baseline follows the same training logic with a larger encoder backbone.

2.5. Evaluation, Calibration, and Validation-Based Threshold Selection

We report accuracy, macro-F1, weighted F1, balanced accuracy, Matthews correlation coefficient (MCC), deceptive-class precision, deceptive-class recall, deceptive-class F1, area under the precision–recall curve (AUPRC), and Brier score where probabilistic outputs are available. This metric set was chosen to support applied comparison under severe class imbalance, where global accuracy alone may conceal operational failure on rare deceptive cases.
To formalize the deployment-oriented perspective, each model–threshold pair was treated as a candidate decision policy whose value depends jointly on minority-sensitive predictive utility (e.g., balanced accuracy, Lie-class F1, and AUPRC) and operational cost (e.g., training time, inference burden, and model size). In the experiments, decision thresholds were selected exclusively on the validation fold and then applied to the held-out test fold. The reported threshold curves should therefore be interpreted as validation-based operating analyses rather than as test-set tuning or post hoc optimization of the final evaluation partition.

2.6. Implementation Details, Robustness, and Explanation Analyses

Experiments employed Python 3.12.4, ConvoKit (version 4.1.0) for corpus access and conversational data processing, scikit-learn (version 1.8.0) for the sparse baselines, sentence-transformers (version 5.0.0) for sentence embedding generation, and the Hugging Face Transformers library (version 5.6.2) with PyTorch (version 2.9.0+cu128; CUDA 12.8) and GPU acceleration for transformer fine-tuning. The overall workflow was organized modularly, covering preprocessing, baseline modeling, imbalance-aware fine-tuning, validation-based threshold selection, robustness analysis, and interpretability, thereby facilitating stepwise reproducibility.
Beyond the main comparison, the revised study includes three additional analyses. First, the compact weighted DistilRoBERTa baseline was evaluated across ten random seeds to separate best-case operating performance from robustness. Second, model-agnostic LIME case studies were generated for Sentence-BERT (SBERT), DistilRoBERTa, and DeBERTa-v3-base on representative true-positive, false-negative, false-positive, and true-negative cases [52,53,54]. Third, controlled perturbation stress tests (lowercasing, punctuation removal, character-level typo noise, word dropout, and context truncation) were used to compare surface robustness across the sparse and neural models.
Table 2 summarizes the core preprocessing steps and model hyperparameters used in the present study.

3. Results

3.1. Context-Window Effects and Baseline Representation Trade-Offs

The context-window ablation shows that local dialogue history helps global speaker-intention performance, but the optimal window depends on the evaluation target. For TF-IDF, overall accuracy rises from 0.846 at n = 0 to 0.877 at n = 5, whereas Lie-class F1 declines from 0.186 at n = 0 to 0.133 at n = 5. This indicates that longer local history may stabilize majority truthful predictions while attenuating minority deceptive discrimination. SBERT exhibits the opposite tendency: balanced accuracy and Lie-class recall improve relative to raw utterances, but the model remains consistently below the sparse baseline in overall performance.
From an applied perspective, n = 3 is a useful compromise operating point. We selected n = 3 not as the absolute best setting for every metric, but as a deployment-oriented compromise that preserves strong overall sparse performance (accuracy = 0.871, balanced accuracy = 0.540, AUPRC = 0.153) while avoiding the sharper minority-class decline observed at n = 5. This operating choice is therefore motivated by practical trade-offs rather than by a single headline metric.
Figure 3 illustrates the effect of the previous-turn context window on both global and minority-sensitive performance in the speaker-intention task. Panel (a) compares global metrics, including accuracy and balanced accuracy, whereas panel (b) focuses on minority-class behavior through Lie-class F1 and AUPRC. The results show that longer local context tends to improve overall accuracy for the sparse baseline, but can weaken minority-sensitive detection, especially at larger window sizes.
Table 3 presents the context-window ablation results for the primary speaker-intention task.

3.2. Validation-Based Threshold Selection for the Sparse Baseline

The TF-IDF model was trained on the training fold, the operating threshold was selected on the validation fold, and the chosen threshold was then applied once to the held-out test fold. The validation-selected threshold was 0.40. On the test fold, this protocol yielded 0.803 accuracy, 0.578 balanced accuracy, 0.213 Lie-class F1, and 0.153 AUPRC. Relative to the default 0.50 threshold reported in the original submission, the revised protocol preserves the practical threshold trade-off while avoiding test-set optimization.
This analysis confirms that the sparse benchmark remains useful as a family of deployment operating points, but it now does so under a methodologically cleaner procedure. The validation-based threshold selection improves deceptive minority sensitivity without relying on blind test-set tuning, thereby addressing the main concern regarding optimistic bias in the original threshold presentation.
Figure 4 illustrates the validation-based threshold trade-off of the TF-IDF model for the speaker-intention task. The vertical marker indicates the threshold selected on the validation fold and subsequently applied to the held-out test fold. The curves compare accuracy, balanced accuracy, Lie-class F1, and Lie-class recall across candidate thresholds. The results show that validation-selected operating points can improve minority-sensitive detection without relying on test-set tuning.

3.3. Auxiliary Tasks: Receiver Perception, Deception Quadrant, and Error Analysis

The auxiliary tasks reinforce two consistent findings. First, contextualization im-proves performance in both receiver-perception and deception-quadrant prediction. Second, the deception-quadrant task remains substantially harder than the binary tasks. With contextual input, TF-IDF reaches 0.892 accuracy and 0.513 macro-F1 on receiver perception after cleaned lexical processing, whereas deception-quadrant prediction reaches only 0.620 accuracy and 0.240 macro-F1. Per-class analysis shows that the model performs reasonably on Straightforward (F1 = 0.774) but remains weak on Cassandra (0.072), Caught (0.087), Deceived (0.118), and Unknown (0.147).
Error analysis shows that the largest confusion flows are Straightforward → Un-known (313 cases), Unknown → Straightforward (188), Deceived → Straightforward (120), and Straightforward → Deceived (117). These patterns suggest that the socially mediated quadrant labels are not merely ‘harder’ in the abstract; rather, the text-only models tend to collapse ambiguous or interaction-dependent cases toward the dominant Straightforward class. This makes the quadrant task a useful stress test for the limits of text-only deception modeling.
Table 4 summarizes the deception-quadrant results for the sparse and dense baselines.

3.4. Validation-Selected Comparison Across Sparse, Dense, Compact-Neural, and Stronger-Neural Baselines

Table 5 reports the main comparison under validation-selected thresholds. The sparse TF-IDF model remains highly competitive and efficient, but the stronger neural reference materially changes the interpretation of the earlier compact-transformer result. SBERT provides only modest minority-sensitive gains, the weighted DistilRoBERTa baseline improves over SBERT on minority-oriented metrics, and the weighted DeBERTa-v3-base baseline provides the strongest neural result in the current study.
The comparison shows that neural conclusions should not be stated as a blanket limitation of all neural models. Instead, a more defensible reading is that compact neural baselines do not automatically dominate strong sparse baselines, whereas stronger encoders can improve balanced accuracy, MCC, Lie-class F1, and AUPRC at substantially higher computational cost. Validation-selected thresholds for TF-IDF, SBERT, DistilRoBERTa, and DeBERTa-v3-base were 0.40, 0.65, 0.50, and 0.10, respectively.
Figure 5 illustrates the trade-off between minority-sensitive performance and computational cost across the sparse, dense, compact-neural, and stronger-neural baselines under validation-selected thresholds. Panel (a) summarizes minority-sensitive metrics, including balanced accuracy, Lie-class F1, and AUPRC, whereas panel (b) compares computational cost in terms of training time and model size on a logarithmic scale. The results show that DeBERTa-v3-base provides the strongest neural minority-sensitive performance, while the sparse TF-IDF baseline remains substantially more efficient and computationally economical.

3.5. Ten-Seed Robustness of the Weighted DistilRoBERTa Baseline

The weighted DistilRoBERTa baseline was re-evaluated across ten random seeds under the revised validation-threshold protocol. Using validation-selected thresholds, the model achieved a mean accuracy of 0.795 ± 0.046, mean macro-F1 of 0.537 ± 0.012, mean balanced accuracy of 0.565 ± 0.026, mean MCC of 0.098 ± 0.028, mean Lie-class F1 of 0.191 ± 0.028, and mean AUPRC of 0.139 ± 0.014. These results confirm that the compact neural model can avoid total collapse, but they also show that performance remains meaningfully variable across seeds.
Figure 6 illustrates the variability of the weighted DistilRoBERTa baseline across ten random seeds under the validation-selected threshold protocol. The three panels summarize variation in balanced accuracy, Lie-class F1, and AUPRC, respectively. The results show that the compact neural baseline avoids universal collapse across seeds, but still exhibits non-trivial variability in minority-sensitive performance.
Table 6 summarizes the ten-seed robustness results for the weighted DistilRoBERTa baseline under validation-selected thresholds.

3.6. Cleaned Lexical Interpretation

After separator-token cleanup, the major sparse-model findings remain stable: speaker intention reaches 0.865 accuracy and 0.543 macro-F1; receiver perception reaches 0.892 accuracy and 0.513 macro-F1; and deception quadrant reaches 0.613 accuracy and 0.233 macro-F1. The persistence of these values indicates that the benchmark’s main conclusions are not artifacts of separator leakage.
Representative cleaned n-grams reveal a consistent contrast between relationship-managing, plan-framing language, and concrete tactical coordination language. Lie-associated cues in the speaker-intention task include terms such as “honest”, “plan”, “sorry”, and “deal”, whereas truth-associated cues are more likely to reference concrete board-state coordination such as “next”, “italy”, “hold”, or “with you”. This interpretability signal is useful in applied settings because it allows model outputs to be audited rather than treated as opaque scores.
Figure 7 illustrates representative cleaned lexical cues extracted from the sparse model after separator-token removal. The figure summarizes the most salient lexical indicators associated with deceptive and truthful predictions in the speaker-intention and receiver-perception tasks. These cues suggest a contrast between relationship-managing and plan-framing language on the deceptive side and more concrete, board-grounded coordination language on the truthful side.
Table 7 lists representative cleaned lexical cues used for post hoc interpretation.

3.7. Controlled Perturbation Robustness

Controlled perturbation stress tests were added to examine whether the principal deployment conclusions remain stable under lightweight input corruption. The perturbations included lowercasing, punctuation removal, character-level typo noise, word drop-out, and context truncation to the focal utterance. All robustness evaluations used the same validation-based threshold protocol as the main comparison.
Under these stress tests, TF-IDF was the most stable model at the surface form level: lowercasing produced no measurable degradation, punctuation removal and light char-acter-typo noise had negligible effects, and the main drop arose only when contextual history was truncated to the focal utterance. The compact DistilRoBERTa baseline was more fragile, especially under punctuation removal and character-level noise, whereas the stronger DeBERTa-v3-base baseline improved both minority-sensitive performance and partial robustness relative to DistilRoBERTa. These results do not constitute a full cross-corpus domain-shift study, but they do show that deployment conclusions change when lightweight corruption is introduced.
Figure 8 illustrates the robustness of the main deployment candidates under controlled perturbation stress tests. The curves summarize Lie-class F1 under lowercasing, punctuation removal, character-level typo noise, word dropout, and context truncation. The results show that the sparse TF-IDF baseline is the most stable under lightweight surface corruption, the compact DistilRoBERTa baseline is the most fragile, and the stronger DeBERTa-v3-base baseline improves robustness relative to DistilRoBERTa, albeit at substantially higher computational cost.

4. Discussion

The present results support a deployment-oriented view of deception detection in negotiation dialogue. Rather than identifying a single universally superior model, the re-vised experiments show that model choice depends on the balance among overall accuracy, minority-class sensitivity, interpretability, robustness, and computational efficiency. In this respect, the study contributes not only a benchmark comparison, but also a practical decision framework for selecting model–threshold policies under operational constraints.
A first important finding is that local conversational context consistently improves deception-related prediction, although the benefit depends on the metric of interest. In the primary speaker-intention task, increasing the context window generally improved overall accuracy for the sparse baseline while at the same time reducing Lie-class F1 at longer windows. This pattern suggests that additional context can stabilize majority truthful predictions while diluting local lexical signals that help identify rare deceptive utterances. The result is important for applied settings because it shows that more context is not automatically better detection; instead, context size is itself a deployment decision.
A second major finding is that the sparse TF-IDF baseline remains highly competitive even after the corrected threshold protocol and the new robustness analyses are introduced. Under validation-selected thresholds, TF-IDF preserves strong minority-sensitive performance while remaining vastly cheaper than the neural baselines. It is also the most stable model under lightweight perturbation stress. This combination of efficiency, transparency, and surface robustness makes the sparse model an attractive first-line monitoring option in trust-sensitive or human-in-the-loop settings.
A third finding concerns how the neural story changes once both a stronger encoder and a more careful evaluation protocol are added. The compact DistilRoBERTa baseline no longer supports a broad claim that neural models are uniformly limited; rather, it shows that compact neural baselines may remain unstable, seed-sensitive, and fragile to lightweight corruption. By contrast, DeBERTa-v3-base improves balanced accuracy, MCC, Lie-class F1, and AUPRC relative to both DistilRoBERTa and SBERT, but these gains come with substantially higher computational cost and only partial robustness gains. The practical implication is therefore conditional: stronger neural encoders can be preferable when missing deceptive minority cases is more costly than reduced efficiency, whereas sparse monitoring remains preferable when transparency, stability, and resource constraints dominate.
The auxiliary-task and interpretability analyses refine the scientific interpretation of these trade-offs. Deception-quadrant prediction remains much harder than direct binary intention detection because text-only models tend to collapse socially mediated outcomes toward the dominant Straightforward class. At the same time, cleaned sparse lexical cues such as ‘honest’, ‘plan’, ‘sorry’, and ‘deal’ are consistent with reassurance, commitment management, and strategic expectation shaping, whereas truth-associated cues more often encode concrete board-grounded coordination [55,56,57,58,59,60]. The added LIME case studies for dense and neural models suggest that successful deceptive detections also depend on local planning and coalition cues, although neural explanations are often less compact and more context-dispersed than sparse lexical coefficients.

5. Conclusions

This study developed a deployment-oriented benchmark for deception detection in negotiation dialogue. Under a corrected validation-threshold protocol, local conversational context improved predictive performance, TF-IDF remained the most efficient and surface-stable overall benchmark, and deception-quadrant prediction proved substantially harder than direct binary deception detection. In addition, stronger neural modeling changed the original compact-transformer interpretation: while weighted-sampler Distil-RoBERTa remained variable and perturbation-sensitive, weighted-sampler DeBERTa-v3-base provided the strongest neural minority-sensitive result in the current study, albeit at substantially higher computational cost.
The main contribution of this study is therefore not the identification of a universal model winner, but the proposal of a practical decision framework for choosing among sparse monitoring, dense representation baselines, compact-neural extensions, and stronger-neural detectors under operational constraints. More broadly, the results suggest that deception-related dialogue analytics should be evaluated not only in terms of global accuracy, but also with respect to class imbalance, calibration, robustness, interpretability, and computational efficiency.
The findings should nevertheless be interpreted within several scope conditions: the experiments remain centered on a single strategic-dialogue corpus, rely on text-only inputs without explicit board-state or partner-history features, and do not yet include a true cross-corpus transfer evaluation. The perturbation-based stress tests help reduce robust-ness uncertainty, but they do not replace external validation on other negotiation datasets. Future work should therefore evaluate additional negotiation corpora, incorporate richer discourse-state and game-state representations, and extend robustness analysis beyond the present controlled perturbation setting.

Author Contributions

Conceptualization, S.Y.C.; methodology, J.-U.K.; software, H.S.Y.; validation, S.Y.C.; formal analysis, J.-U.K.; investigation, B.J.G.; resources, S.Y.C.; data curation, J.-U.K.; writing—original draft preparation, J.-U.K. and B.J.G.; writing—review and editing, S.Y.C. and H.S.Y.; visualization, J.-U.K.; supervision, S.Y.C.; project administration, S.Y.C. All authors have read and agreed to the published version of the manuscript.

Funding

The present research has been conducted by the Research Grant of Kwangwoon University in 2025.

Institutional Review Board Statement

Not applicable. The study used publicly available textual data and did not involve a new intervention with human participants.

Informed Consent Statement

Not applicable.

Data Availability Statement

The study analyzed publicly available datasets, including the Diplomacy deception corpus distributed through ConvoKit. Processed outputs, summary tables, figure-generation assets, and note-book-based analysis scripts are available at: https://github.com/Brom-git/Benchmarking-Sparse-and-Dense-Models-for-Deception-Detec-tion-in-Negotiation (accessed on 4 April 2026). Raw source data remain available through the original Diplomacy/ConvoKit distribution.

Acknowledgments

All outputs were reviewed, edited, and validated by the authors, who take full responsibility for the analysis, interpretation, and final manuscript content.

Conflicts of Interest

The authors declare no conflicts of interest.

Abbreviations

AUPRCArea under the precision–recall curve
MCCMatthews correlation coefficient
SBERTSentence–BERT/sentence–transformer representation
TF-IDFTerm frequency-inverse document frequency

References

  1. Goffman, E.; Naegele, K.D. The presentation of self in everyday life. In The Presentation of Self in Everyday Life; Doubleday & Company, Inc.: Garden City, NY, USA, 1959; Volume 259, p. 2002. [Google Scholar]
  2. Grice, H.P. Logic and conversation. In Speech Acts; Brill: Leiden, The Netherlands, 1975; pp. 41–58. [Google Scholar]
  3. Brown, P. Politeness: Some Universals in Language Usage; Cambridge University Press: Cambridge, UK, 1987. [Google Scholar]
  4. Levinson, S.C. Pragmatics; Cambridge University Press: Cambridge, UK, 1983. [Google Scholar]
  5. Myerson, R.B. Game Theory; Harvard University Press: Cambridge, MA, USA, 2013. [Google Scholar]
  6. Osborne, M.J.; Rubinstein, A. A Course in Game Theory; MIT Press: Cambridge, MA, USA, 1994. [Google Scholar]
  7. Raiffa, H. The Art and Science of Negotiation; Harvard University Press: Cambridge, MA, USA, 1985. [Google Scholar]
  8. Camerer, C.F. Progress in behavioral game theory. J. Econ. Perspect. 1997, 11, 167–188. [Google Scholar] [CrossRef]
  9. Ekman, P. Telling Lies: Clues to Deceit in the Marketplace Politics and Marriage; WW Norton & Company: New York, NY, USA, 2009. [Google Scholar]
  10. Vrij, A. Detecting Lies and Deceit: Pitfalls and Opportunities; John Wiley & Sons: Hoboken, NJ, USA, 2008. [Google Scholar]
  11. Newman, M.L.; Pennebaker, J.W.; Berry, D.S.; Richards, J.M. Lying words: Predicting deception from linguistic styles. Personal. Soc. Psychol. Bull. 2003, 29, 665–675. [Google Scholar] [CrossRef] [PubMed]
  12. Zhou, L.; Burgoon, J.K.; Nunamaker, J.F.; Twitchell, D. Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communications. Group Decis. Negot. 2004, 13, 81–106. [Google Scholar] [CrossRef]
  13. Hancock, J.T.; Curry, L.E.; Goorha, S.; Woodworth, M. On lying and being lied to: A linguistic analysis of deception in computer-mediated communication. Discourse Process. 2007, 45, 1–23. [Google Scholar] [CrossRef]
  14. Mihalcea, R.; Strapparava, C. The lie detector: Explorations in the automatic recognition of deceptive language. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers; Association for Computational Linguistics: Stroudsburg, PA, USA, 2009; pp. 309–312. [Google Scholar]
  15. Ott, M.; Choi, Y.; Cardie, C.; Hancock, J.T. Finding deceptive opinion spam by any stretch of the imagination. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics: Stroudsburg, PA, USA, 2011; pp. 309–319. [Google Scholar]
  16. Feng, S.; Banerjee, R.; Choi, Y. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2012; Short Papers; Volume 2, pp. 171–175. [Google Scholar]
  17. Toma, C.L.; Hancock, J.T. What lies beneath: The linguistic traces of deception in online dating profiles. J. Commun. 2012, 62, 78–97. [Google Scholar] [CrossRef]
  18. Pérez-Rosas, V.; Abouelenien, M.; Mihalcea, R.; Xiao, Y.; Linton, C.; Burzo, M. Verbal and nonverbal clues for real-life deception detection. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2015; pp. 2336–2346. [Google Scholar]
  19. Vargas, F.; Jonas, D.A.; Rabinovich, Z.; Benevenuto, F.; Pardo, T. Rhetorical structure approach for online deception detection: A survey. In Proceedings of the Thirteenth Language Resources and Evaluation Conference; Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; pp. 5906–5915. [Google Scholar]
  20. Velutharambath, A.; Klinger, R. UNIDECOR: A unified deception corpus for cross-corpus deception detection. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis; Association for Computational Linguistics: Stroudsburg, PA, USA, 2023; pp. 39–51. [Google Scholar]
  21. Hazra, S.; Majumder, B.P. To tell the truth: Language of deception and language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics: Stroudsburg, PA, USA, 2024; Long Papers; Volume 1, pp. 8506–8520. [Google Scholar]
  22. Lewis, M.; Yarats, D.; Dauphin, Y.; Parikh, D.; Batra, D. Deal or no deal? End-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2017; pp. 2443–2453. [Google Scholar]
  23. He, H.; Chen, D.; Balakrishnan, A.; Liang, P. Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2018; pp. 2333–2343. [Google Scholar]
  24. Chawla, K.; Ramirez, J.; Clever, R.; Lucas, G.; May, J.; Gratch, J. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 3167–3185. [Google Scholar]
  25. Kwon, D.; Hae, J.; Clift, E.; Shamsoddini, D.; Gratch, J.; Lucas, G. ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning via Tool-integrated Action for Dynamic Offer Optimization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2025; pp. 16217–16238. [Google Scholar]
  26. Park, J.; Rahman, H.A.; Suh, J.; Hussin, H. A study of integrative bargaining model with argumentation-based negotiation. Sustainability 2019, 11, 6832. [Google Scholar] [CrossRef]
  27. Niculae, V.; Kumar, S.; Boyd-Graber, J.; Danescu-Niculescu-Mizil, C. Linguistic harbingers of betrayal: A case study on an online strategy game. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2015; Long Papers; Volume 1, pp. 1650–1659. [Google Scholar]
  28. Peskov, D.; Cheng, B.; Elgohary, A.; Barrow, J.; Danescu-Niculescu-Mizil, C.; Boyd-Graber, J. It takes two to lie: One to lie, and one to listen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 3811–3854. [Google Scholar]
  29. Meta Fundamental AI Research Diplomacy Team (FAIR); Bakhtin, A.; Brown, N.; Dinan, E.; Farina, G.; Flaherty, C.; Fried, D.; Goff, A.; Gray, J.; Hu, H.; et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 2022, 378, 1067–1074. [Google Scholar]
  30. Wongkamjan, W.; Gu, F.; Wang, Y.; Hermjakob, U.; May, J.; Stewart, B.M.; Kummerfeld, J.; Peskoff, D.; Boyd-Graber, J. More victories, less cooperation: Assessing cicero’s diplomacy play. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2024; Long Papers; Volume 1, pp. 12423–12441. [Google Scholar]
  31. Wongkamjan, W.; Wang, Y.; Gu, F.; Peskoff, D.; Kummerfeld, J.K.; May, J.; Boyd-Graber, J.L. Should I Trust You? Detecting Deception in Negotiations using Counterfactual RL. In Findings of the Association for Computational Linguistics: ACL 2025; Association for Computational Linguistics: Stroudsburg, PA, USA, 2025; pp. 25099–25113. [Google Scholar]
  32. Chang, J.P.; Chiam, C.; Fu, L.; Wang, A.; Zhang, J.; Danescu-Niculescu-Mizil, C. Convokit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 57–60. [Google Scholar]
  33. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar]
  34. Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; long and short papers; Volume 1, pp. 4171–4186. [Google Scholar]
  35. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv 2019, arXiv:1907.11692. [Google Scholar]
  36. Reimers, N.; Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 3982–3992. [Google Scholar]
  37. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 2020, 33, 1877–1901. [Google Scholar]
  38. Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient estimation of word representations in vector space. arXiv 2013, arXiv:1301.3781. [Google Scholar]
  39. Pennington, J.; Socher, R.; Manning, C.D. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP); Association for Computational Linguistics: Stroudsburg, PA, USA, 2014; pp. 1532–1543. [Google Scholar]
  40. Thorstan, J. Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the European Conference on Machine Learning; Springer Berlin Heidelberg: Berlin, Heidelberg, 1998; pp. 137–142. [Google Scholar]
  41. Salton, G.; Buckley, C. Term-weighting approaches in automatic text retrieval. Inf. Process. Manag. 1988, 24, 513–523. [Google Scholar] [CrossRef]
  42. Wang, S.I.; Manning, C.D. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2012; Short Papers; Volume 2, pp. 90–94. [Google Scholar]
  43. Forman, G. An extensive empirical study of feature selection metrics for text classification. J. Mach. Learn. Res. 2003, 3, 1289–1305. [Google Scholar]
  44. Aggarwal, C.C. Mining text data. In Data Mining: The Textbook; Springer International Publishing: Cham, Switzerland, 2015; pp. 429–455. [Google Scholar]
  45. Davis, J.; Goadrich, M. The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA, 25–29 June 2006; pp. 233–240. [Google Scholar]
  46. Saito, T.; Rehmsmeier, M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 2015, 10, e0118432. [Google Scholar] [CrossRef] [PubMed]
  47. Elkan, C. The foundations of cost-sensitive learning. In Proceedings of the International Joint Conference on Artificial Intelligence; Lawrence Erlbaum Associates Ltd.: Mahwah, NJ, USA, 2001; pp. 973–978. [Google Scholar]
  48. Lipton, Z.C.; Elkan, C.; Narayanaswamy, B. Thresholding classifiers to maximize F1 score. arXiv 2014, arXiv:1402.1892. [Google Scholar] [CrossRef]
  49. Chicco, D.; Jurman, G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom. 2020, 21, 6. [Google Scholar] [CrossRef] [PubMed]
  50. Powers, D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv 2020, arXiv:2010.16061. [Google Scholar] [CrossRef]
  51. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision; IEEE: New York, NY, USA, 2017; pp. 2980–2988. [Google Scholar]
  52. Ribeiro, M.T.; Singh, S.; Guestrin, C. “Why should I trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016; pp. 1135–1144. [Google Scholar]
  53. Lundberg, S.M.; Lee, S.-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 2017, 30, 4766–4777. [Google Scholar]
  54. Molnar, C. Interpretable Machine Learning; Lulu. com: Durham, NC, USA, 2020. [Google Scholar]
  55. Danescu-Niculescu-Mizil, C.; Sudhof, M.; Jurafsky, D.; Leskovec, J.; Potts, C. A computational approach to politeness with application to social factors. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics; Association for Computational Linguistics: Stroudsburg, PA, USA, 2013; Long Papers; Volume 1, pp. 250–259. [Google Scholar]
  56. Rashkin, H.; Choi, E.; Jang, J.Y.; Volkova, S.; Choi, Y. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing; Association for Computational Linguistics: Stroudsburg, PA, USA, 2017; pp. 2931–2937. [Google Scholar]
  57. Austin, J.L. How to Do Things with Words; Harvard University Press: Cambridge, MA, USA, 1975. [Google Scholar]
  58. Searle, J.R. Speech Acts: An Essay in the Philosophy of Language; Cambridge University: Cambridge, UK, 1969. [Google Scholar]
  59. Thompson, L.L. The Mind and Heart of the Negotiator; Pearson Education: New York, NY, USA, 2020. [Google Scholar]
  60. Jurafsky, D.; Martin, J.H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition; Prentice Hall PTR: Upper Saddle River, NJ, USA, 2000. [Google Scholar]
Applsci 16 04301 g001
Figure 1. Overall workflow of the proposed deception–detection pipeline.
Figure 1. Overall workflow of the proposed deception–detection pipeline.
Applsci 16 04301 g001
Applsci 16 04301 g002
Figure 2. Architecture of the imbalance-aware DistilRoBERTa extension.
Figure 2. Architecture of the imbalance-aware DistilRoBERTa extension.
Applsci 16 04301 g002
Applsci 16 04301 g003
Figure 3. Context-window ablation for the speaker-intention task. Panel (a) compares global metrics, including accuracy and balanced accuracy, whereas panel (b) focuses on minority-class behavior through Lie-class F1 and area under the precision–recall curve (AUPRC). The results show that longer local context tends to improve overall accuracy for the sparse baseline, but can weaken minority-sensitive detection, especially at larger window sizes.
Figure 3. Context-window ablation for the speaker-intention task. Panel (a) compares global metrics, including accuracy and balanced accuracy, whereas panel (b) focuses on minority-class behavior through Lie-class F1 and area under the precision–recall curve (AUPRC). The results show that longer local context tends to improve overall accuracy for the sparse baseline, but can weaken minority-sensitive detection, especially at larger window sizes.
Applsci 16 04301 g003
Applsci 16 04301 g004
Figure 4. Validation-based threshold trade-off for the TF-IDF speaker-intention model.
Figure 4. Validation-based threshold trade-off for the TF-IDF speaker-intention model.
Applsci 16 04301 g004
Applsci 16 04301 g005
Figure 5. Comparison across sparse, dense, compact-neural, and stronger-neural baselines under validation selected thresholds.
Figure 5. Comparison across sparse, dense, compact-neural, and stronger-neural baselines under validation selected thresholds.
Applsci 16 04301 g005
Applsci 16 04301 g006
Figure 6. Ten-seed robustness of the weighted DistilRoBERTa baseline under validation-selected thresholds. The blue circles indicate the metric values obtained from individual random seeds. The orange horizontal line inside each box indicates the median value. The box represents the interquartile range (IQR), and the whiskers indicate the spread of the distribution.
Figure 6. Ten-seed robustness of the weighted DistilRoBERTa baseline under validation-selected thresholds. The blue circles indicate the metric values obtained from individual random seeds. The orange horizontal line inside each box indicates the median value. The box represents the interquartile range (IQR), and the whiskers indicate the spread of the distribution.
Applsci 16 04301 g006
Applsci 16 04301 g007
Figure 7. Representative cleaned lexical cues extracted from the sparse model after separator-token removal. Panel (a) shows Lie-associated cues for speaker intention, panel (b) shows Truth-associated cues for speaker intention, panel (c) shows perceived-lie cues for receiver perception, and panel (d) shows perceived-truth cues for receiver perception. The figure summarizes the most salient lexical indicators associated with deceptive and truthful predictions in the speaker-intention and receiver-perception tasks. These cues suggest a contrast between relationship-managing and plan-framing language on the deceptive side and more concrete, board-grounded coordination language on the truthful side.
Figure 7. Representative cleaned lexical cues extracted from the sparse model after separator-token removal. Panel (a) shows Lie-associated cues for speaker intention, panel (b) shows Truth-associated cues for speaker intention, panel (c) shows perceived-lie cues for receiver perception, and panel (d) shows perceived-truth cues for receiver perception. The figure summarizes the most salient lexical indicators associated with deceptive and truthful predictions in the speaker-intention and receiver-perception tasks. These cues suggest a contrast between relationship-managing and plan-framing language on the deceptive side and more concrete, board-grounded coordination language on the truthful side.
Applsci 16 04301 g007
Applsci 16 04301 g008
Figure 8. Controlled perturbation robustness for the main deployment candidates.
Figure 8. Controlled perturbation robustness for the main deployment candidates.
Applsci 16 04301 g008
Table 1. Dataset summary used for reproducibility.
Table 1. Dataset summary used for reproducibility.
ItemValue
Primary sourceDiplomacy deception corpus distributed through ConvoKit/ACL 2020 release
Primary prediction unitUtterance-level classification with local conversational context
Official split policyTrain/validation/test folds supplied with the corpus
Leakage controlContext windows constructed only within the assigned fold and conversation
Primary benchmarkSpeaker intention (binary Lie vs. Truth)
Auxiliary tasksReceiver perception (binary) and deception quadrant (multi-class)
Held-out class ratio (speaker intention test)240 Lie vs. 2501 Truth utterances (8.8% vs. 91.2%)
Context-window settingsn_prev = 0, 1, 3, 5; deployment-oriented operating point emphasized at n = 3
Table 2. Core preprocessing and model hyperparameters.
Table 2. Core preprocessing and model hyperparameters.
ComponentKey Settings
TF-IDF + logistic regressionmax_features = 20,000; ngrams = (1,2); min_df = 2; class_weight = balanced; max_iter = 2000
SBERT + logistic regressionall-MiniLM-L6-v2 encoder; batch size = 64 for embedding; downstream logistic regression with class_weight = balanced
Compact neural baselinedistilroberta-base; weighted random sampling; max_length = 256; early stopping on validation Lie F1
Stronger neural referencemicrosoft/deberta-v3-base; weighted random sampling; max_length = 256; same validation-threshold protocol
Neural optimizationlearning rate = 2 × 10−5; weight_decay = 0.01; epochs = 3; batch sizes = 8 (train)/16 (eval); early stopping patience = 2
Threshold selectionvalidation-only selection; selected threshold applied once to the held-out test set
Robustness analysis10 random seeds for weighted DistilRoBERTa; mean ± SD and per-seed outputs
Perturbation stress testlowercase; punctuation removal; character typo noise; word dropout; context truncation
Explainabilitycleaned sparse lexical coefficients plus LIME case studies for SBERT and neural baselines
Table 3. Context-window ablation for the primary speaker-intention task. The table compares sparse and dense baselines across four local-context settings.
Table 3. Context-window ablation for the primary speaker-intention task. The table compares sparse and dense baselines across four local-context settings.
Context WindowModelAccuracyBalanced AccuracyLie F1AUPRCTrain Time (s)
0TF-IDF + LR0.8460.5540.1860.1510.97
0SBERT + LR0.6600.5840.2020.1203.25
1TF-IDF + LR0.8550.5510.1810.1401.18
1SBERT + LR0.6730.5740.1960.1273.78
3TF-IDF + LR0.8710.5400.1580.1531.84
3SBERT + LR0.6620.5760.1960.1275.63
5TF-IDF + LR0.8770.5290.1330.1362.63
5SBERT + LR0.6360.5440.1720.1016.93
Table 4. Deception-quadrant results. Context improves both model families, but the multi-class task remains substantially more difficult than binary deception detection.
Table 4. Deception-quadrant results. Context improves both model families, but the multi-class task remains substantially more difficult than binary deception detection.
InputModelAccuracyMacro-F1Weighted F1
Utterance onlyTF-IDF + LR0.5360.2290.580
Utterance onlySBERT + LR0.2460.1560.320
Utterance + contextTF-IDF + LR0.6200.2400.632
Utterance + contextSBERT + LR0.2560.1600.333
Table 5. Main comparison under validation-selected thresholds across sparse, dense, compact-neural, and stronger-neural baselines.
Table 5. Main comparison under validation-selected thresholds across sparse, dense, compact-neural, and stronger-neural baselines.
ModelAccuracyBalanced AccuracyMCCLie PrecisionLie RecallLie F1AUPRCTrain Time (s)Model Size (MB)
TF-IDF + LR0.8030.5780.1190.1640.3040.2130.1532.020.08
SBERT + LR0.8300.5380.0670.1400.1830.1590.1275.6986.65
WS DistilRoBERTa0.8120.5500.0820.1450.2330.1790.137209.24313.26
WS DeBERTa-v3-base0.8600.5730.1430.2150.2250.2200.168571.30703.52
Table 6. Ten-seed robustness summary for the weighted DistilRoBERTa baseline under validation-selected thresholds.
Table 6. Ten-seed robustness summary for the weighted DistilRoBERTa baseline under validation-selected thresholds.
MetricMean ± SD
Accuracy0.795 ± 0.046
Macro-F10.537 ± 0.012
Balanced Accuracy0.565 ± 0.026
MCC0.098 ± 0.028
Lie Precision0.150 ± 0.013
Lie Recall0.285 ± 0.106
Lie F10.191 ± 0.028
AUPRC0.139 ± 0.014
Table 7. Representative cleaned lexical cues used for post hoc interpretation.
Table 7. Representative cleaned lexical cues used for post hoc interpretation.
TaskDirectionRepresentative Cleaned Cues
Speaker intentionLie-associatedalright; think; honest; plan; sorry; deal
Speaker intentionTruth-associatednext; italy; hold; with you; moscow; opening
Receiver perceptionPerceived lieturn; scandinavia; army would; you plan; plan; austria is
Receiver perceptionPerceived truthturkey; tell; you should; retreat; germany is; thinking
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Kim, J.-U.; Go, B.J.; Yu, H.S.; Cho, S.Y. Benchmarking Sparse and Dense Models for Deception Detection in Negotiation: A Context-Aware and Imbalance-Sensitive Approach. Appl. Sci. 2026, 16, 4301. https://doi.org/10.3390/app16094301

AMA Style

Kim J-U, Go BJ, Yu HS, Cho SY. Benchmarking Sparse and Dense Models for Deception Detection in Negotiation: A Context-Aware and Imbalance-Sensitive Approach. Applied Sciences. 2026; 16(9):4301. https://doi.org/10.3390/app16094301

Chicago/Turabian Style

Kim, Jae-Uk, Beom Jun Go, Hwan Soo Yu, and Soo Young Cho. 2026. "Benchmarking Sparse and Dense Models for Deception Detection in Negotiation: A Context-Aware and Imbalance-Sensitive Approach" Applied Sciences 16, no. 9: 4301. https://doi.org/10.3390/app16094301

APA Style

Kim, J.-U., Go, B. J., Yu, H. S., & Cho, S. Y. (2026). Benchmarking Sparse and Dense Models for Deception Detection in Negotiation: A Context-Aware and Imbalance-Sensitive Approach. Applied Sciences, 16(9), 4301. https://doi.org/10.3390/app16094301

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop