Social Sentiment Sensor in Twitter for Predicting Cyber-Attacks Using ℓ1 Regularization

In recent years, online social media information has been the subject of study in several data science fields due to its impact on users as a communication and expression channel. Data gathered from online platforms such as Twitter has the potential to facilitate research over social phenomena based on sentiment analysis, which usually employs Natural Language Processing and Machine Learning techniques to interpret sentimental tendencies related to users’ opinions and make predictions about real events. Cyber-attacks are not isolated from opinion subjectivity on online social networks. Various security attacks are performed by hacker activists motivated by reactions from polemic social events. In this paper, a methodology for tracking social data that can trigger cyber-attacks is developed. Our main contribution lies in the monthly prediction of tweets with content related to security attacks and the incidents detected based on ℓ1 regularization.


Introduction
Online Social Networks (OSNs) are platforms designed as communication channels for information exchange in real time. These platforms may generate over 1 billon posts per month around the world. For example, Twitter statistics [1,2] report the generation of 313 million posts monthly, better known as tweets, over different countries.
Different topics in Twitter may reflect polarized opinions from celebrities, corporations, and regular users about daily life aspects [3], some of them with well defined geographic embedded data (e.g., assisted GPS coordinates). Streams of tweets generate valuable information that can be modeled as a social sentiment sensor for real-world event prediction [4] by analyzing clustered topics, such as in rumour spreading analysis [5], human mobility sensing [6], spam & botnet detection [7], and disaster response [8].
Within the context of cyber-security, the large volumes of data that can be collected over different time intervals from Twitter have the potential to facilitate the understanding of the motivation behind cyber-attacks by sentiment analysis of tweets. Specifically, any underlying correlation among the sentimental polarity of various groups of Twitter users can be interpreted by probabilistic and classification models [9], whose results are predictive by nature and can be used as a social behavior warning tool. For example, in [10], an early warning process related to abnormal behavior is developed relating intrusion techniques and terrorist attacks.
Regional language and lexical variations derived from tweets are key factors in searching patterns related to sentimental tendencies. Natural language processing has shown that negative-oriented textual features [11] related to information security lexicons used by hacktivists groups can be used as warning alarms to mitigate possible cyber-attacks. Therefore, important political, religious, and cultural events can serve as targets for data extraction in Twitter to predict such attacks [12].
This paper focuses on sentiment analysis extracted from tweets, which are processed with probabilistic techniques [13] in order to measure the correlation between the sentiment of user groups and possible cyber-attacks. Specifically, we propose a methodology for predicting possible cyber-attacks based on scraping and classifying Twitter data. This is done by employing a supervised learning algorithm [14][15][16][17] on a daily corpora of tweets. The methodology classifies tweets in order to obtain monthly sentimental scores that are fitted into an 1 regression algorithm to predict potential cyber-attacks.
Although current advances on information security have improved the trust on information handling mechanisms by means of e.g., cryptographic and data protection algorithms [18][19][20], cyber-attacks are still an important issue to tackle. Our methodology has the potential to aide in the prevention of cyber-attacks based on sentiment analysis of tweets.
The rest of the paper is organized as follows. Section 2 provides a review of the related work. Section 3 describes the proposed methodology. Section 4 describes the data gathering and pre-processing mechanisms used. Sections 5-7 detail the supervised classification used in this work. Section 8 describes the statistical analysis. Section 9 provides the experimental results. Finally, Section 10 concludes this work.

Related Work
According to [21], cyber-attacks are increasing as a result of global insurgency given geopolitical contexts. These attacks pose major concerns due to their potential effects on denial-of-service, data leaking, and application compromising. Alternative security measures, like forecasting threatening security events, are thus gaining credibility.
Data from OSNs are useful for extending capabilities from intrusion detection systems (IDSs) and intrusion prevention systems (IPSs) from outer-level networks. In [22], a Latent Dirichlet Allocation (LDA)-based model is proposed to discover semantically related concepts to analyze cyber-crime forensics. More recently, a bipartite and monopartite network analysis is achieved by crawling hackers forums to identify members by specific malicious tool usage [23]. A list of anti-threat strategies is proposed in [24] to prevent and visualize common practices regarding privacy, spamming and malicious attacks. In [25], the authors present a relationship of social unrest between countries and directed cyber-attacks. These works prove that Arbor Network data are useful to determine if attacks such as Distributed Denial-of-Service (DDoS) attacks are expected to grow if radical or extremist sentiments from users are perceived in streams of OSN posts.
Predictive analysis is particularly advantageous in Twitter due to the fact that certain functionalities, such as retweets, favorites, and replies, can be characterized and, together with the polarity of the text, can provide data that increase the forecasting of events such as political elections and product outcomes [26]. According to [27], the predictive power of social networks can be exploited by the inspection of published data and statistical modelling, which may result in the detection of a statistical relationship between a social media-based measure (e.g., number of re-tweets or sentiment analysis scores) and the outcome of interest (e.g., economic growth or presidential approval rates). For example, in [28], an 1 regularized regression model is presented in order to predict influenza-like illness by training data from Twitter and comparing outcomes with official health reports.

Proposed Methodology
The work flow of the proposed methodology is depicted in Figure 1. First, a query is requested from the Twitter search endpoint. The resulting response containing blocks of tweets is then processed by a web scrapping engine and stored on a local database. A set of pre-selected tweets is prepared for training a classifier using supervised learning [14][15][16][17]. Finally, sentimental scores of the classified tweets are fed to an 1 regularization algorithm to obtain predictive results.

Data Acquisition
Data gathering schemes are designed for querying Twitter endpoints to obtain chronological tweets. Recent works on sentiment analysis [29][30][31] use a public information streaming platform known as Twitter Standard Search API, which is an interface that has capabilities for information retrieval in chronological order for no longer than seven days [32]. In this paper, we use an approach proposed in [33] for historical retrieval by querying Twitter search endpoints. The web crawling tasks are done with web spiders' engines designed for document scraping in an automated and efficient manner. Information is processed by Scrapy, a Python Web Scraping Framework that extracts embedded text in HTML tags and simultaneously uses recursive functions to analyze each link to follow other tweets. This data gathering scheme is depicted in Figure 2. Collecting data is achieved by querying the endpoints in time intervals sorted by days. Each query q is based on n-grams (set of co-occurring words within a given text) bags-of-words related to specific events defined as q = [{ 1-gram, 2-gram, 3-gram, . . . , n-gram }, {date}].
Queries responses are processed by a web spider towards the endpoint and redirected to a Scrapy download layer. Unprocessed data are then fed into the Scrapy engine in order to strip hypertext tags and retrieve each tweet in plain text. As depicted in Figure 3, the retrieved text is processed independently in Scrapy pipes that handle data streams into objects to be stored on a relational database. The set of retrieved queries is the corpus of tweets, C, and its size is directly proportional to the daily number of tweets stored for the query. Each tweet can be represented as a structure comprising fundamental attributes, as tabulated in Table 1. Each tweet t is stored with its own id as a primary key that is used to sort them in a sequentially and non-repeatable way. Each tweet in set C is then denoted by C(q) = t i ∈ {t id , t text , t date } n i=1 .

Tokenization and Noise Removal
A cleaning task is applied on C to generate individual arrays of words (i.e., tokens) for each tweet. A normalization step is required to transform each token into lower case words; a dimensionality reduction [34] of C is important to reduce textual noise. Noise is considered as frequent uni-grams or stop-words (very commonly used words) that do not provide valuable information as candidate textual markers. In the case of the English language, sets of stop-words widely applied in Natural Language Processing are used in text cleaning tasks. This work uses the publicly-available English stop-words set published in [35], and each word is weighted by textual and lexical functions in a sentence [36]. URL patterns are removed from the corpus. Other non-informative expressions, such as retweets RT and appearances of @username, are also deleted.

Lexical Derivations
Textual markers have lexical derivations as part of ungrammatical text structures written by most users. Grammatical restriction is performed to stem each token, thus avoiding repeated samples from the same grammatical root and bias in the training step for classification. An example of stemming is shown in Table 2.
We use a Snowball Stemmer for lexicographical lemmatisation, which is a set of probabilistic algorithms based on Porter stemmer [37] of Indo-European languages and has been shown to attain high capabilities for searching pattern inflections into roots from composed words [38].

Pre-Classification and Class Labeling
Supervised classification provides predefined class labels given specific inputs, where each class must be independent from the others. Selecting relevant and high impact tweets are important for good training performance due to the fact that some words give most information about a particular context. We use The Stanford sentiment corpus [39] along with tweets crawled by our own scraping approach, tweets are labeled as negative (neg) or positive (pos) based on the user's emotions.
A second set of tweets related to cyber-security and cyber-attacks topics is scraped by querying terms contained in The Glossary of Common Cybersecurity Terminology [40], and other manually annotated hacker-activists terms [11]. It is important to mention that hacktivism, according to [41], is a type of activity among hackers with specific political motivations and ideologies, such as religion or jigonism. In general, there are four motivations, i.e., revenge, financial, notoriety and curiosity, [42] related to hacktivism. This work considers all of these motivations.
Crawled tweets are labeled by a sec(security − oriented) tag. The set of labels is then denoted by label = {pos, neg, sec} and the corpus for training is denoted by where τ i is the ith tweet text and label in the training set. Figure 4 depicts some examples regarding class labeling.

Supervised Classifier
Building a supervised classifier is achieved by first transforming each input of textual markers into features, followed by a training step with labels. Features extracted from T contain basic information that allows for C to be successfully classified. The work flow is graphically depicted in Figure 5.
Features and labels from T are processed by the supervised learning algorithms [14][15][16][17] to generate a classifier model. A feature extractor computes features based on words by the term frequency-inverse document frequency (Tf-idf ) algorithm [43]. A label for each tweet of C is then predicted.

Feature Extraction and Selection
Features are based on sentimental relevance; i.e., words that better describe a user's sentiment towards a specific context are selected. As proposed in [44], identifying raw n-grams is more useful for feature extraction than using speech tagging because supervised classifiers tend to attain a higher accuracy with grammatical and positional independence in sentences.
In order to avoid over-fitting, we perform a model selection procedure to split data into random matrices for training and testing. By performing a train-test selection procedure with Python sklearn library, we divide T into 80% training and 20% validation subsets. Training and validation tweets from regular users merged with security oriented users are denoted by X T , which contains pre-processed text from tweets, while y denotes their respective labels. Resulting subsets from T are denoted by X T , y T , which are the training subset tuples, and X V , y V , the validation subsets tuples selected to evaluate the classifier model. Word particles contained in tweets from the training set are extracted and transformed into Tf-idf term weights [45] by using a sklearn Tf-idf vectorizer; then, each resulting vector is normalized by an 2 norm.

Classification Baseline
Choosing a good classifier is an important task to generate a robust model for testing corpus C. In other words, results must be accurate enough to eventually find relationships between the users sentiments and cyber-attacks. In [39,46,47], different classifiers such as Naive Bayes, Maximum Entropy and Support Vector Machines are proposed and evaluated; results show that, for noisy labels and the case of emotions in tweets, Support Vector Machines attain better results than those of other text classifiers.

Naive Bayes Classifier
Classifiers based on the Bayes theorem are widely used in text classification [14] for short messages like tweets because of the simplicity in computing probabilistic evidence for class prediction given independent text features. This method contrasts with those that employ Bernoulli models [48], which are based on document counts for each class. Having a label set for C classes, we can define parameters to calculate the probability of a class c given a tweet by: where t is a tweet, c a class (label), f i ∈ f (X T ) is the feature, and n i(t) is a word presence given t and m is the number of features.

Support Vector Machines
Support Vector Machines [15] are suitable for bounding data in linear and non-linear ways. Inherently, SVM is a binary classifier, meaning that data are separated into two labeled classes. For a multi-class approach for the training set (X T , y T ) with labels y T ∈ {0, 1, 2}, an optimization approach is proposed by solving: Constrained to (w y · t i ) . . , , m ∈ {1, . . . , k}, thus we can find an optimized decision function by finding the saddle point of the Lagrangian: where w is the hyper plane, α i is the non-negative Variable Lagrange Multiplier, y i is the ith input class (label) from the label set, t are input tweets, b denotes the hyper-plane parameters (bias), ξ is a slack variable (0 < ξ ≤ i is the point between the margin and the correct side of the hyper-plane with ξ > 1 denoting a misclassified point) and C is the regularization parameter.

Maximum Entropy Classifier
Maximum Entropy classifiers are widely used for learning from input features in a weighted manner to generate a discriminative model that evaluates possible values from possible classes [16,17]. The model is represented by: where c denotes the class (label), t is a tweet, λ is the weight vector (considering that a higher weight assumes a strong indicator about the class), Z(t) is the normalization function given t, and F i,c is the feature-class function for a feature f i ∈ f (X T ).

Prediction-Statistical Analysis 1 Regularized Regression
Regression is suitable for predicting events given multiple inputs, better known as observations, that are linearly independent from each other [49]. A linear model is interpreted as: where: 1. X C is the observation matrix of all classified tweets from corpus C 2. X C pos and X C neg are the observations with positive and negative scores, respectively, 3. y C security_oriented is the fitted security-oriented response from regression coefficients [β 1 , Because of the negative effect on computing regression by ordinary least squares over highly correlated observations and an increase of variance, a regularized regression using selection and reduction is proposed. Regression based on vector norm 1 can adjust the linear model by making some coefficients zero, which is suitable for large multivariate observation matrices. LASSO (Least Absolute Shrinkage and Selection Operator) is an adaptation to linear models that minimizes the error in the limit of absolute values from prediction coefficients: where λ is the tuning parameter for shrinking coefficients [β]. To solve the 1 penalization problem, the Forward Stagewise algorithm [50] is computed. The proposed solution of Equation (6) is given by tacking the subgradient: where g j is the subbgradient of the 1 norm, g j = sign(β j ) if β j = 0, g j ∈ [−1, 1], otherwise. The algorithms for the proposed system are shown in Algorithms 1-4. 3: return y C security_oriented 4: end function

Experimental Results
This section shows the evaluation of the proposed sensor for sentiment analysis using a total of 1,800,000 tweets in English. One million tweets are extracted using the method proposed in [33] from regular and cyber-security related accounts and 800,000 belonging to The Stanford dataset [51]. In Table 3, some well identified Twitter accounts related to hacktivists, cyber-security feeds, researchers, and enthusiasts users are tabulated.

Experimental Results
This section shows the evaluation of the proposed sensor for sentiment analysis using a total of 1,800,000 tweets in English. One million tweets are extracted using the method proposed in [33] from regular and cyber-security related accounts and 800,000 belonging to the Stanford dataset [51]. In Table 3, some well identified Twitter accounts related to hacktivists, cyber-security feeds, researchers, and enthusiasts users are tabulated.  Table 4 tabulates the classification results attained by the Support Vector Machine (SVM), Naive Bayes (NB), and Maximum Entropy (ME) classifiers. Bold rows represent the best classification results. These results are obtained using parameters related to document frequency (df ), which is a threshold for support applied to weight terms where the minimum and maximum support are in the interval [0.5, 0.95].

A Case Study
During the 2016 United States of America (USA) presidential campaigns and post election time, an important set of polarized opinions was generated by Donald Trump polemic speeches. Speculations about the winning candidate increased by adding financial, political, immigration, religious, and sexist comments towards his opponent, Hillary Clinton, during the campaign. Hacktivists generated public threats towards Donald Trump using hash-tags like #OpTrump and #OpDrump f . In addition, rumors about hackers manipulating electoral campaigns increased users' negative reactions towards both candidates. Table 5 tabulates classification results of the three classifiers used in this work for a number of tweets generated by users who tweet regularly and hacktivists. These tweets are contained in C. As specified before, we denote by X C the testing set to perform this case study. In order to better appreciate the sentimental average scores, i.e., X C neg , X C pos and y C security_oriented , the 486 tracked days between 9 January 2016 and 1 May 2017 are divided into six time-intervals.

Regularized Regression
Prediction over high volumes of scores can be difficult with ordinary regression due to unbiassed coefficients. By employing LASSO [52], we can shrink coefficients in order to optimize our prediction model. Moreover, regularized regression tasks can be only implemented in multivariate sets. As tabulated in Table 4, the ME classifier attains the best accuracy results, so we use 1 normalization on ME [53]. We divide the normalized scores from X C into monthly prediction tasks to precisely analyse the presidential campaign period. A statistical report containing the following measures is tabulated in Table 6: • Mean Squared Error (M.S.E.): shows the difference or loss of the predicted scores with the inputs, i.e., between the actual scores, y C security_oriented , and the predictions, y C security_oriented .
• p-value (probability value): determines how well the observations (X C neg , X C pos ) are adjusted in the predictive model, thus rejecting the null hypothesis that relates to the low effectiveness of the samples. The lower the probability value (p-value ≈ 0), the greater the adjustment in the model.
• R 2 (coefficient of determination): explains the proportion of adjustment from the observations, (X C neg , X C pos ), with respect to the outputs, y C security_oriented .
• Detected Attacks : the total number of cyber-attacks detected.
Bold rows represent the maximum correlation between users sentiment and a security oriented response given by R 2 . Historical data extracted from Google News can help to determine if R 2 values related to users' sentiments are correlated with cyber-attacks. During mid-March 2016, Trump's comments and behavior regarding abortion, the violence on his rallies, and his declarations about the Brussels terrorist attacks, increase users' negative opinions towards him and, in retaliation, hacktivists started a raid under the banner of OpTrump threatening election sites, voice-mails, and public information. June 2016 was also a hard month during the election; rumors about hackers hijacking elections by cyber-intrusions increase people's reaction by posting DNC compromised servers revealing Hillary Clinton's private emails. The observations obtained in these time series show that there is a correlation between the negative opinions expressed in tweets of hacktivists and cyber-attacks. A chronological time-line of tweets classified by ME as negative, positive, and security-oriented, as well as the index of important security related incidents, as reported by Google News (see Table 7), is presented in Figure 6. Based on the results, note that it is possible to define thresholds for predicting possible cyber-attacks, for example when the coefficient of determination, R 2 , increases above 80%. Figure 7 depicts a PoC (Proof of Concept) of this idea. Specifically, this figure depicts the cyber-attacks perpetrated from January to April 2016 when the coefficient of determination, R 2 , is greater than or equal to 80%. At the end of February 2016, the hacktivist Anonymous made the following statement: Suspend campaign...or face consequences. Anonymous also created sites like www.optrump2016.com (now redirected to www.donaldjdrumpf.com) with a counter for the time left before hacking sites related to the presidential campaign. Simultaneously, the number of tweets with an associated sentiment increased considerably from February to March, 2016, approximately 78% for negative tweets, 87% for positives tweets, and 37% for security-related tweets. This confirms that the correlation computed by Equation (6) is useful to predict possible cyber-attacks.

Conclusions
This paper presented a methodology to predict cyber-attacks by using a Social Sentiment Sensor in Twitter. The methodology collects historical tweets and classifies them as negative, positive and security-oriented. By using 1 regularization on the classified tweets, cyber-attacks can be predicted when the corresponding coefficient of determination reaches a certain value. The methodology is evaluated within the context of the 2016 USA presidential campaigns, during which politicians appear to have influenced the sentiment of Tweeter users and in response, hacktivists reacted as part of the opposition by threatening public information. Specifically, we have shown that the proposed methodology can serve as a warning mechanism to detect possible cyber-attacks.
The proposed methodology is not limited to cyber-attacks. Our future work includes testing and tailoring the proposed methodology to predict other real-life events such as pandemics, political alignment, and market events.