The Small Sample Challenge: Unsupervised Authorship Clustering in Croatian

Abstract

Many forensic methods have been developed for uncovering authors of threatening letters to get rid of their anonymous nature. With the development of new tools for the digital creation of threatening letters, such methods became obsolete, yet new options for their digital analysis also became available. This paper covers a method for digital analysis of threatening letters written in the Croatian language using machine learning algorithms. First the techniques for extraction of style elements of a text are covered, focusing on the most influential elements, and then transforming the extracted data into a vectorized form which can be used in machine learning models. The models are built upon the autoencoder architecture in combination with two text style vectorization methods. Multiple such models are implemented and their results are compared and analyzed, focusing on their effectiveness and the quality of their results. Finally, additional possibilities for further development of machine learning models for the analysis of threatening letters written in the Croatian language are covered and their possible effectiveness explained.

1. Introduction

Threatening letters are a common tool to manipulate people into executing the author’s wishes, but their specialty is their anonymity, because of which the victim does not know how and from who to defend themselves. Many forensic methods for analysis of authorship exist, but their primary focus is on the physical characteristics of the written text instead of the text itself. Hence, today, when letters can be digitally written, we are in need of new methods which can analyze the letters in a more profound manner, through the text itself.

There is already abundant research about the topic itself, but not enough research has been done on letters written in Croatian. This paper covers the combination of techniques for forensic-style analysis with the use of machine learning models, with the goal of detecting common authors from a corpus of unknown threatening letters written in Croatian. Yet, this paper is written as an introductory work on applying such techniques on Croatian threatening letters, because of which we only have a small, unlabelled (i.e., of unknown authors) corpus of 13 threatening letters at our disposal for this paper.

The developed solution is described in two parts: the algorithm for the extraction of useful information about the letters’ styles from the text, and the explanation of the developed machine learning models. Then the algorithmically extracted data is analyzed using the machine learning models to compare the stylistic similarities and presumed authorships of the corresponding letters.

This paper is organized as follows. In Section 2, we discuss the related work in the field. Section 3 presents the research methodology. Later, in Section 4, results are presented along with a discussion of the results. The paper is concluded with Section 5.

2. Materials and Methods

2.1. Related Work

The research by Yanamandram Kuppuraju et al. significantly advances the field of Cyber Threat Intelligence (CTI) by developing an automated framework that integrates Natural Language Processing (NLP) with machine learning to process unstructured data. A primary contribution of the study is the implementation of advanced Named Entity Recognition (NER) to efficiently extract critical indicators of compromise from diverse and voluminous security sources. By automating threat classification, the authors demonstrate a substantial reduction in detection latency, shifting the defensive paradigm from a reactive to a proactive stance. Furthermore, the paper provides a comparative analysis of various machine learning models, establishing scalable methodologies for real-time threat assessment in increasingly complex digital environments [1].

The study by Fedotova et al. examines the effectiveness of authorship attribution in Russian by comparing formal literary texts with informal social media posts. The authors developed a robust feature selection methodology capable of identifying key stylometric parameters even in morphologically complex languages. A central contribution is the demonstration that the precise selection of character and word n-grams can achieve high attribution accuracy despite stylistic variations across different platforms. Their findings offer critical guidance for addressing data sparsity, a challenge of paramount importance in the forensic analysis of short-form texts [2,3,4].

Qian et al. explore the application of various deep learning architectures, specifically Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs), for authorship identification. A key contribution of their work is the comparative analysis of character-level versus word-level modeling, illustrating how different levels of granularity influence the detection of an author’s unique stylistic signature. The authors propose a hybrid approach utilizing Long Short-Term Memory (LSTM) networks to manage long-range dependencies, which is essential for distinguishing between multiple candidates. Their results indicate that deep learning models can surpass traditional stylometric methods by automatically capturing complex features without the need for manual linguistic engineering [5].

Research by Huang et al. presents a novel framework for authorship verification that utilizes authorial language models to differentiate between human-written and AI-generated content. A primary contribution of the study is a methodology that treats each author as a distinct linguistic system, enabling the detection of subtle stylistic inconsistencies that current AI detectors often overlook. The authors demonstrate that fine-tuning large language models on specific authorial datasets can create a “stylistic fingerprint” capable of verifying authenticity even in brief text samples. Additionally, the paper provides a rigorous evaluation of how authorial idiosyncrasies persist across different genres, carrying significant implications for forensic linguistics and digital content integrity [6].

The paper by Sari et al. introduces an innovative approach to authorship attribution using continuous n-gram representations, blending traditional stylometric features with modern word embeddings. A key contribution of this research is the development of a model that learns low-dimensional vectors for character and word n-grams, effectively capturing stylistic nuances while mitigating the dimensionality issues associated with sparse data. The authors show that these continuous representations significantly outperform standard discrete n-gram models, particularly when handling numerous candidate authors. Their findings are especially relevant for forensic linguistics, as the method maintains high accuracy even when training data per author is limited [7].

The research by Layton et al. addresses the challenges of authorship attribution within micro-message environments, specifically focusing on the constraints of short-form platforms like Twitter. A significant contribution of this work is the adaptation of the Source Code Author Profiling (SCAP) method to natural language, proving that character-level n-gram analysis remains effective even with minimal textual input. The authors demonstrate that, despite the lack of traditional linguistic cues in short messages, their methodology achieves high accuracy in identifying authors from a candidate pool. This provides a critical foundation for forensic investigations where evidence consists of brief or highly constrained samples [8].

Comaniciu et al. introduce the Mean-shift algorithm as a versatile, non-parametric technique for ascending density gradients to locate modes in feature space. A significant contribution of this work is its ability to perform clustering and image segmentation without the prior specification of the number of clusters, making it highly adaptable to complex data distributions. The authors highlight the algorithm’s robustness to outliers through a kernel-based approach that identifies high-density regions as cluster centers. This research established a fundamental framework for feature space analysis that remains a cornerstone of statistical pattern recognition [9,10].

Building upon density-based concepts [11]. introduced HDBSCAN, a clustering algorithm that integrates the advantages of DBSCAN with a hierarchical framework. A key contribution of this research is the development of a method to extract flat clustering from a condensed cluster tree based on cluster stability across varying density levels. The authors demonstrate that this approach effectively handles clusters of varying densities and is highly robust to noise, eliminating the need for a globally optimal density parameter. Their work represents a significant advancement in data mining by providing a more flexible and automated solution for identifying natural groupings within complex datasets [11].

In the context of applying machine learning techniques to complex pattern recognition problems, recent studies emphasize the crucial role of robust feature representation learning. One such example is [12], where machine learning is applied for recognizing human activities from Channel State Information (CSI) obtained from Wi-Fi signals, with a special emphasis on its privacy-preserving property. The study evaluates four fine-tuned versions of existing machine learning techniques, with [13], proposing a novel GRU-RNN architecture specifically designed for Wi-Fi CSI-based human interaction recognition. Although operating in a different domain, these methodologies share fundamental representation learning principles with textual digital forensics. Both domains face the challenge of extracting distinctive, noise-resilient fingerprints from high-dimensional, sparse, or constrained inputs. The development and superior performance of a specialized machine learning model in such a complex pattern recognition problem suggests that similarly specialized models may also prove beneficial in the domain of textual digital forensics. In text analysis, particularly when dealing with morphologically rich and less-resourced languages like Croatian, latent feature representations, similar to the latent embeddings constructed for activity recognition, allow the extraction of deep stylistic patterns that go beyond surface-level word frequencies [13,14].

2.2. Research Methodology

The studied problem can be generalized as a clustering problem: the goal is to partition letters into an unknown number of groups based on their author, so that all letters of the same group are written by the same author. Thus, we forensically and statistically analyze the data of the letters, then encode the data into a multidimensional space using machine learning, and finally cluster the resulting data based on deduced authorship. This section provides a detailed explanation of each step and the associated technical parameters for reproducibility and the justification of the used methods. Letter parameters are presented in

Table 1. Descriptive statistics for the letter corpus.

Corpus Size: $\mathit{N}=13$
Variable	Mean	Median	SD	Min	Max
Word count	142.15	90	112.15	38	453
Sentence count	8.92	8	4.18	4	17
Unique words	181.23	109	138.68	58	568
Error count	6.62	6	4.58	1	19
Total unique words	1019

.

2.2.1. Preprocessing and Statistical Data Extraction

Useful data about the letter’s style can be separated into two parts: errors and corrected text (the message that the author meant to write). These two distinctions are made because for the style analysis it is important to consider both the mistakes the author made and the original message and exact words the author meant to communicate. Hence, first we detect the errors the author made, which we also later use as useful information, and then correct them, and only then analyze the full, corrected letter again to try and encompass every exact word the author originally meant to use. Using such a method, both the data about the errors and the specific words used are included in the analysis.

For error detection, a separate, specialized algorithm should be used. In this paper, for simplicity and the goal of general method effectiveness testing, we used Ispravi.me [15], a publicly available Croatian text correction service. Following that an algorithm was developed for generating the corrected version of the letter using the fetched results. It is important to note however, that this service is made for analysis of generally more academically styled text instead of the crude and rustical text that we find in the threatening letters we are analyzing; hence, it would be best to develop a separate algorithm specialized for such a style of the Croatian language. The main difference which impacts its effectiveness the most is the use of many unusual words, acronyms, personal names, colloquial phrases and the common occurrence of grammatically incorrect word order.

Afterwards, words and related data are extracted from the corrected letters by use of a simplified version of the hrLex dictionary [16,17,18]. Certain metrics are defined to calculate the most influential statistical data from the extracted words, which are then iteratively collected, e.g., by calculating the frequency of a certain word in a letter and in the whole corpus. Concretely, the chosen metrics are: word, sentence, name and incorrect word count, certain part of speech counts, and the ratio of correctly written capital letters at the start of new sentences. The optimal choices of such metrics themselves should be further researched, as these are chosen based on educated guesses for the sake of testing the general effectiveness of the methods later used for analysis.

Because we are analyzing an unlabelled (i.e., no ground truth) set, the collected statistical data is normalized, i.e., the frequency with regards to the corpus is used instead of the absolute values. This eliminates the possible huge value differences, e.g., when a letter has a lot more words, which can cause the difference to affect the reasoning of the machine learning model to focus on that dimension too much, for we know for sure that the length of the letter is not a sure metric to identify the author without considering other factors. This data is then not further analyzed itself but used in machine learning models later.

2.2.2. Machine Learning

The idea of using machine learning to analyze the extracted data from the letters’ contents is to vectorize the data into a latent space through which the model can deduce the most influential information for identifying the author likeliness of two letters.

Letter Vectorization

For vectorization of the letters’ contents, the TF-IDF metric [19] is used. The content is transformed into a dictionary which tracks the frequency of occurrence of each word in the specific letter, combined with a value of inverse-frequency of its occurrence in the whole corpus (

Table 2. Examples of values for words in the last (#13) letter of the analyzed corpus.

Word	Frequency in Document	Frequency in Corpus	IDF Value	Final Value in Dictionary
Da	7	32	1.24	7.52
Će	6	25	1.34	8.02
novinari	4	4	2.59	11.78
Vi	3	17	1.44	4.33
drugih	3	5	2.25	6.76
Što	2	14	1.44	2.88
Život	2	3	2.54	5.08
Protiv	1	3	2.54	2.54
likvidacija	1	1	2.95	2.95
janje	0	1	2.95	0

). Using this dictionary, we can normalize the effects of common words on the model learning process, preventing frequent words from having too strong of an effect on the final encoded form of the letter [19].

Here we already have comparable data for each letter, e.g., by the sum of the differences between the frequency of each word in two letters. But the data is in a format which is made of a very high number of dimensions, one for each word in the whole corpus, which then creates a few problems. Firstly, the range of possible resulting similarity values between letters is too big, which means the model will most likely be overfitted and not learn to generalize. Secondly, we only compare data for equal words in between letters, which then actually does not analyze the context of words, but just the words themselves. And lastly, most of the dictionary is empty data, i.e., most words in the dictionary have a frequency value of zero for each letter. Therefore, we need a way to extract only the most influential information for deducing the authors of the letters.

Encoding Data Using the Model

The next step is encoding the vectorized data into a latent high-dimensional space of a machine learning model. The goal is to reduce the dimensionality of the data and the elimination of noise created by the fact that most of the frequency values in the dictionaries are equal to zero, as was mentioned at the end of the last section.

The model’s purpose is to determine the most influential elements of differentiating between different author’s styles and general attributes in the way they communicate. For that we train a model built upon the autoencoder architecture using the unsupervised learning method. If we used supervised learning with labelled data, the model would only learn to differentiate between the authors that were included in the training corpus and would not be able to differentiate between new authors it was not trained on. Furthermore, the autoencoder architecture is chosen as an effective method for such unsupervised learning because it gives the model the ability to decode the resulting vector back into the original letter and hence enables us to check how precisely it encoded the given data [20,21].

The encoder part of the model is generally of the form [Linear layer to ReLU], while the decoder is then [ReLU to Linear layer to Sigmoid activation layer], with an additional layer in between for further model parameterization (Figure 1). The model is trained by receiving minibatches of the TF-IDF dictionary vectorized form of the letters from the training corpus as input, many times through many epochs, each time in a randomized order. Then it is expected to recreate the original dictionary format of the letter as the output, after transforming it into its latent space through the linear layer, the success of which is judged by a defined criterion function. The dimensionality of the latent layer is defined to be 100, while the dimensionality of the input depends on the given TF-IDF vector, where it is equal to the total number of unique words, i.e., the vocabulary size of the analyzed letter.

Word Embedding

An alternative approach was also implemented by using an already generated word embedding matrix instead of encoding TF-IDF dictionaries of letters from scratch. The matrix was generated from the hrWaC corpus [22], and is used here in combination with IDF values from the TF-IDF dictionaries to once again normalize the effects of common words on the final result. Because the embedding matrix is generated from a much more massive and relevant corpus of Croatian words, we expect this model to perform better in the long run.

Identifying Common Authors Through Vectorized Letter Analysis

As mentioned at the start of this section, the last step is partitioning the resulting vectorized data into groups based on stylistic and authorship similarities in the content of the corresponding letters. For that purpose the L2 distances between the vectorized representations of letters were calculated and analyzed. Then two clustering algorithms were used to group the data according to the calculated distances: Mean-Shift [22], chosen for its simplicity and as a baseline, and HDBSCAN [23], chosen for its adaptability and capacity to generate a hierarchical tree of clusters, providing multiple partitionings with varying numbers of groups. Both algorithms were also selected because they do not require a predefined number of clusters, with the additionally useful ability of HDBSCAN to let the user decide on the final result of the clustering, including the number of groups, i.e., different authors.

To ensure reproducibility and keeping the goal of testing the methods used in mind, default parameters were used for the algorithms. For the Mean-Shift algorithm, the kernel bandwidth parameter was dynamically estimated using the $estimat{e}_{bandwidth}$ functionality implemented inside the 1.7.1 version of the scikit-learn module within Python 3.13.5. For HDBSCAN, the minimum cluster size was set to 2, reflecting the extremely small scale of the test dataset, using the Euclidean metric over the 100-dimensional latent vectors, which is equivalent to the L2 distance measure analyzed previously.

2.3. Implementation

TF-IDF Autoencoder

The described TF-IDF autoencoder model was implemented, with three versions created, each based on a different learning corpus:

• Letters: The first version uses the learning corpus of the letters themselves. This way we know there will be no words missing in the resulting corpus dictionary. Yet the big flaw of this approach is that such a model will not be as effective when analyzing new letters with different words. Hence, the most effective way to use this approach is to retrain the model on each corpus of threatening letters in which we want to differentiate the authors. As was already mentioned, the corpus contains 13 different letter samples, which were not separated into a validation set because there were so few samples.
• CroLTeC (CROatian Learner TExt Corpus): The second version uses the CroLTeC corpus [24], publicly available essays written by foreign students learning the Croatian language. This corpus was chosen for the many spelling and grammatical errors it contains in its letters, as threatening letters often do. However, a major flaw is that there are many aggressive phrases and personal names that are not included which often appear in threatening letters. A big note about this model is that the TF-IDF dictionaries are generated by analyzing the n-grams instead of the words themselves, in an attempt to capture the mistakes and the context of the words for a more effective style analysis. The corpus contains 1724 samples of written essays, separated into a training set of 1500 and a validation set of 224.
• Wikipedia: The third version consists of 4.5 million characters gathered from the top featured Wikipedia articles written in the Croatian language (in September of 2025) [25]. This corpus was chosen as the largest and most unbiased source of Croatian words and hence has a role as a control group. The corpus contains 243 samples of the chosen articles’ introductory texts, separated into a training set of 210 and a validation set of 33.

Each model was trained for 50 epochs using the minibatch methodology, separating samples into minibatches of 64 randomly chosen samples (without repeating) from the corpus per epoch. The results are presented in Figure 2. The criterion used is Mean Squared Error loss, which measures the average squared difference between the original sample and the model-generated output. The main goal of the criterion in the TF-IDF autoencoder model is to judge how different the output of the model is from the original sample. That is why this criterion was chosen, as it is a slightly more complex and meaningful version of a basic, direct difference criterion (e.g., Manhattan distance), and it also aligns well with the later use of L2 distances for comparing the resulting latent space embeddings. The optimizer used is the Adaptive Moment Estimation (also known as “ADAM”) with the starting learning rate of ${10}^{-3}$. It was chosen as an industry standard choice, known for its stability and adaptability.

The average validation loss of both models trained with validation was too small to be shown on a graph: $3.26\times {10}^{-6}$ for the CroLTeC corpus and $1.44\times {10}^{-6}$ for the Wikipedia corpus.

The second model is created by using the word embedding method, using the IDF dictionary values for common word normalization, as described before. Hence, the model was not trained but uses the CLARIN.SI-embed.hr [26] publicly accessible embedding matrix, which is the result of a model trained by the skip-gram method upon the hrWaC corpus. The TF-IDF dictionary is generated from the original corpus of threatening letters that we aim to analyze.

The model analyzes the letters by iteratively summing the embedded value of each word, multiplied by its IDF value in the generated TF-IDF dictionary. Through this approach we get the model’s understanding of the letters, which we can then use to compare the letters’ styles and possibly authors.

3. Results

Two metrics are used for interpreting the results: grouping letters under shared authors using clustering algorithms and calculating similarities using L2 distances between vectorized letters.

3.1. Clustering Algorithms

Using the clustering algorithms, outlines of groups were generated which contain letters the model deduced are written by the same, but still anonymous, author (Figure 3). The HDBSCAN algorithm also generates a hierarchy of clusters in the form of a tree structure, which then enables the parameterization of the point at which we should cut the tree using the $\lambda$ parameter, which could be effective to calculate through a separate method or model.

3.2. Distance

For letter similarity analysis, L2 distances between each letter pair were calculated and triangular distance matrices were constructed from the obtained values, one for each model (Figure 4). Since we do not know the exact correlation between the distance of a pair of letters and their real similarity/chance of sharing the same author, a threshold value of 20 was decided, for clearer review purposes. For the same reason, relative distance matrices were also constructed from normalized distance values, which provide us with a more intuitive understanding of the differences between different letter pairs’ similarities.

For further development of this method, we need to derive concrete, ground truth values of the real similarity between letters and their authors through use of either a corpus of labelled letters (i.e., with known authorship) or already existing proven methods for calculating letter similarities. Then we would be able to compare the similarity expressed in the L2 distance calculated by the models with ground truth similarity values.

4. Discussion

4.1. Clustering

A big problem with using a clustering algorithm to decide on common authors can be quickly observed; the algorithms assume they are analyzing data with fine granularity and the existence of at least one group with multiple elements, which does not need to be the case. The letters could all be written by different authors since they are randomly sent by the public. We can observe the problem in each of the collected results; often there are only two or three groups, which is statistically quite unlikely for the aforementioned reason. This problem could be minimized by using the hierarchical clustering results of the HDBSCAN algorithm, by “cutting” the tree structure at the point which calculates the optimal groupings. Yet HDBSCAN results seem worse. It groups by at most 3 authors, which is again very unlikely considering we are analyzing threatening letters from the public. Mean-shift groups by up to 5 different authors, which may be more useful data about the similarities of the letters in this case.

4.2. Distance

The noticeably big difference between L2 distance values obtained by the TF-IDF autoencoder and the word embedding model arises from the word embedding matrix and its model’s output being normalized, while the TF-IDF autoencoder’s outputs are left in the latent space the model created, to keep the bigger range for expression. It is interesting to note how all three TF-IDF autoencoder models mostly agree about the relative similarities between certain pairs of letters, which we can see in, e.g., the second, fourth and fifth rows of the relative distance matrices. Furthermore, the distance matrices from the CroLTeC corpus and Wikipedia articles are quite similar, even though they substantially differ in the way they analyze the texts. The relative difference matrix from the hrWaC embedding matrix visibly greatly differs from the rest, which probably arises from the fact that the autoencoders analyze encoded dictionaries, while the latter only analyzes the letters themselves.

But what about the word order? By using the TF-IDF dictionary approach, we only consider the frequency of the occurrence of the words, while we do not capture the order of the words, which could logically be extremely relevant for style analysis. The only case where the order is considered is when we generate the TF-IDF dictionaries by analyzing the n-grams instead of the words, which we did for the model constructed upon the CroLTeC corpus.

Yet it would be very useful to additionally analyze the word order in the letters and include this data as an extra dimension when calculating the distances between letters. A simple example of such an analysis would be using the formal grammar based on the Croatian language, in combination with part of speech data for each word in the letter. Thus, we would obtain the letter’s grammatical composition data, but such data is also difficult to universally vectorize.

The specialty of using the machine learning approach to natural language processing is that the constructed model can “learn” the meanings of words based on the context they are in when training. That is why the most important part in training such a model is deciding on which training corpus to use. Hence, for this case, existing models will not be effective in analyzing the threatening letters, because they were taught on mostly academic and formal texts, but no model was built upon a corpus of threatening letters written in Croatian, which is very error-prone and rustic in nature. But further development of such a specialized model could prove very effective if we could gather a large and varied enough corpus of such threatening letters to train it on.

4.3. Limitations of Such a Small Corpus Scale

A primary constraint that was highlighted many times throughout this research is the reliance on a very small corpus of 13 unlabelled threatening letters. These letters were provided as a simple sample for the development of computer-based processing techniques for authorship clustering, with the long-term goal of headstarting such a field of research, specifically for threatening letters written in the Croatian language. Consequently, the statistical validity of the reported clusters must be interpreted within an exploratory context rather than as a definitive diagnostic tool. Small sample sizes inherently heighten the risk of sample variance bias, where unique idiosyncratic errors of a single document, such as mentions of names, slang, and grammatical errors, might be overrepresented. Furthermore, without ground-truth labels (i.e., confirmed author identities), classic supervised metrics such as Precision, Recall, or F1-Score cannot be computed. Hence, the framework serves as a proof-of-concept demonstrating how unsupervised latent encoding can compress stylistic variance of the Croatian language to assist forensic examiners in preliminary evidence triage.

4.4. Possibilities with a Bigger, Labelled Letter Corpus

If we had a bigger and labelled letter corpus at our disposal, we could calculate the final step of the analysis, to exactly deduce which letters are written by the same authors. The letters in the corpus would not necessarily need to be labelled with exact authors, but just with a label that identifies which letters should be grouped together under the same author. After vectorizing the corpus using the models implemented through this paper, we could further develop a model which would be trained in a supervised manner using the vectorized letter data and the labels as inputs and expected outputs, respectively. Such a model would especially excel in isolating the most important aspects in deducing which letters share the same author.

But there is a certain issue if we look at this approach from a different perspective: the model would learn to only recognize the similarities between letters of the authors included in the training corpus. That means that if we use that model to then analyze letters of new authors not included in the training corpus, we would obtain incorrect deductions. That would happen because some of these new authors could have similar style elements as some of the authors from the training corpus, but very different in certain other aspects that the model did not learn to consider. Thus, it would wrongly put letters of multiple different new authors in the same group, which would correspond to the author from the training corpus. The only way to minimize the effect of this possible issue is to use as much a varied training corpus as possible, which would capture the styles of many different authors.

4.5. Practical Applicability

In regards to practical applicability, the TF-IDF autoencoder model is scalable to any corpus size, with the ability to arbitrarily set the dimension of the latent space. The chosen dimension of 100 worked well for the used corpora and converged quickly, so it could have been set to a smaller value. The longest duration it took to train a model on each of the corpora was around two minutes, with the largest corpus containing 4 million characters. Analyzing the main sample of 13 letters using a learned model takes around 10 s, and analyzing a test sample of ten thousand generated letters takes at most 30 s. The main problem detected during performance testing is the memory requirements; the model trained on the Wikipedia-based corpus is 1.5 gigabytes in size, and because the dimension of the latent space is defined as 100, memory allocation becomes an obstacle when analyzing the test sample of ten thousand generated letters together at the same time. Yet in practical application it can be expected that the number of samples analyzed together at once will not reach such numbers, because there is no practical use of analyzing shared authorship at such a large scale. Also, as mentioned, the dimension of the latent space could have been freely reduced based on the complexity of the analyzed threatening letters, which directly reduces the memory requirements. A special case of memory requirements becomes apparent when using the predefined embedding matrix, whose size is around 3 gigabytes. Yet such sizes are not unusual for machine learning models, and hence it is an acceptable cost if the model proves effective.

The biggest issue with the model is its explainability, because the technique of encoding a corpus into a latent space and extracting the relevant features for determining authorship ends up being a type of “black box”. In the state developed and analyzed in this paper, there is no method of explaining which features the model chose as relevant and why, because it was not in the scope of the paper, but this must be further developed for a model to be usable in practical forensic analysis. Determining the authorship of a threatening letter must be based on provable data, especially in cases of criminal prosecution, which this model does not provide in its current form. Another big issue is the dependence of the model on the corpus it was trained on, as explained in the previous Section 4.4.

The model was developed and tested specifically on threatening letters and corpora written in the Croatian language, but in its general form could be adapted and used on data written in any language.

4.6. Forensic Usability

The data generated by a further developed version of the proposed model could be used in forensics to show derived similarities between analyzed letters. As mentioned in the introduction, there are existing forensic methods for the analysis of authorship, and this model could provide additional information gathered from a large amount of statistical information that the model would have been trained on. The resulting data could be used in the same ways as it was analyzed in this paper; latent vectors of the target letters would be generated and then directly compared to form a sort of similarity matrix (as in Figure 4), which would give an idea of which letters to focus on when comparing. Or, a clustering algorithm could be added to the output of the model so that it produces direct decisions on shared authorship through the generated clusters of the latent vectors. In this paper, HDBSCAN was used for this purpose, because it also generates a dendrogram, which does not explicitly cluster the data, but gives a tree of the procedural generation of clusters based on a threshold variable (as in Figure 3). Such a manner of generic output instead of explicit groupings would be ideal, because we cannot expect a machine learning model to have perfect accuracy, and false positives in the field of forensics and criminal prosecution in general can be extremely catastrophic. In the case of the model developed for this paper, nothing can be said of its accuracy because the letters of the main analyzed challenge sample are of unknown authorship.

5. Conclusions

This paper analyzed and implemented a possible solution for recognizing common authors in a corpus of threatening letters written in the Croatian language, using machine learning models. First an algorithm for forensic analysis of the styles of digitally written letters was described, with a special focus on elements that should differ the most between different authors.

Then two methods for vectorization of the contents of the letters for later compatibility with machine learning models were described: TF-IDF dictionary and word embedding. The autoencoder architecture of the models used was explained, along with how the model encodes the vectorized data into its latent space. Two versions of the described model were implemented, each using one of the different mentioned vectorization methods. The different training corpora used are listed and the reasons are explained for why they were chosen. Finally, the results are shown and compared, using a combination of the L2 distance metric and clustering algorithms, with comments about the usefulness and effectiveness of the obtained results in the context of the goal of this paper. Evaluations show that models of the same letter vectorization type give similar results, but very clearly differ from the other type. Yet there are still consistently high or low similarity scores observed when comparing certain pairs of letters, which points to the possible effectiveness of the use of these models for the goal of detecting common authors of threatening letters written in the Croatian language.

Following the results and their evaluations, the flaws of using a small, unlabelled corpus are listed and described, with an additional focus on the lack of sufficiently developed methods for the analysis of threatening letters, especially written in the Croatian language. Afterwards, further possibilities for development of a more effective model for the detection of common authors among a corpus of threatening letters written in the Croatian language are analyzed.