Research on the Credulity of Spear-Phishing Attacks for Lithuanian Education Institutions’ Employees

Abstract

Organizational security assurance is a complex and multi-dimensional task. One of the biggest threats to an organization is the credulity of phishing attacks for its employees. To prevent attacks, employees must maintain cyber security hygiene and increase their awareness of the cyberattack landscape. In this paper, we investigate how selected Lithuanian education system employees are vulnerable to spear-phishing attacks. In various education organizations, spear-phishing attacks were imitated, and user responses to received emails were monitored and analyzed. Each organization needs a different attention because employee behavior varies. Employees’ reaction time dimension is explored in the research. Based on these results, it appears that the organization has no time for delayed responses. Employees in the education system are highly affected by spear-phishing attacks and need less than one minute to provide attacker-requested data. This illustrates that automated e-mail filtering systems are a key element in the fight against these kinds of attacks.

1. Introduction

Social engineering attacks aim to fool the user and extract some benefit from them—obtain the needed data or motivate the user to take actions [1]. Since 2019, the number of phishing attacks has increased by 150% yearly [2]. Especially effective are spear-phishing attacks, which are aimed specifically at a dedicated target (usually a person or company employee) [3]. The inclusion of some already gathered data, selection of a suitable time and personalized spear-phishing messages lead to a higher attack success. The automation of spear-phishing attacks [4] results in high-impact and effective attacks that affect not only the employee but also the company.

To fight against phishing attacks, both technical and human-oriented mitigation solutions are used [5,6]. The idea of eliminating or mitigating human vulnerability, rather than preventing attacks from reaching the user, is highly critical [7]. As a result, user training and credulity estimation are a crucial part of organizations’ security management [8]. However, human-related attacks and their mitigation are very sensitive to many different factors, which are challenging to control. What works for one target group might have a different effect on another. Therefore, this research investigates the credulity of spear-phishing attacks for Lithuanian education institutions’ employees. It is important to understand the area of the credulity of spear-phishing attacks for Lithuanian education institutions seeking to develop a set of best practices and tools to fight this threat and increase the resistance.

Narrowing the area to a specific country, organization type and spear-phishing attacks should help gather more accurate data. This will eliminate the effect of different work specifics, national regulations and other possible factors. At the same time we cannot be sure that these conditions define the lower variance of the results; therefore, the first research question (RQ1) is oriented to estimate whether the vulnerability of education organization employees is dependent on the organization and its cyber security management policy. Experiments with different Lithuanian education institutions should lead to the answer to whether the same tendencies can be applied to all organizations of the same education sector. We raise a hypothesis (H01) that Lithuanian education system employees display similar behavior when responding to spear-phishing emails, and the organization type has no effect on the credulity of this kind of attack for employees. If this null hypothesis is confirmed, all Lithuanian educational system organizations could use the same security management policy.

It is important to consider the time period during which the organization needs to react to a spear-phishing attack when designing the strategy for preventing and mitigating its effects. In this regard, we raise the second question (RQ2): when will employees respond to spear-phishing emails sent to them? Our null hypothesis (H02) is that no data will be revealed at least within the first 5 min after the spear-phishing emails are sent. At least 5 min to react could ensure at least a partial information technology support center response to such emails.

If employees respond faster than 5 min to reveal the data requested in the spear-phishing email, the information technology support center will not be able to respond. Other types of direction are needed to mitigate spear-phishing attacks in such cases. One of the solutions is employee training. Accordingly, the third research question (RQ3) is to determine how effective cyber security training is for Lithuanian education organizations’ employees. The following hypothesis (H03) is that security training affects the credulity level of spear-phishing attacks for educational institution employees. If it is confirmed, training should be incorporated into the organization’s security management policy as a must.

The research questions and hypotheses supported a more justified security management policy regarding spear-phishing attacks (see Figure 1). This is relevant to most organizations fighting employee-affected cyber security attacks, but in this research, attention is devoted to Lithuanian education institutions. The employees of this sector are often associated with being highly educated and conscious; therefore, it is imperative to determine whether this applies to cyber security.

These three research questions in combination allow one to have a clearer vision of what should be the Lithuanian education security management policy recommendations: RQ1 allows one to define the need for specific recommendations, rather than one, unified for all institutions; RQ2 highlights the importance of the technical solutions fighting against phishing attacks; RQ3 defines how much attention should be devoted to employees’ training and soft solutions fighting against phishing attacks.

The timing perspective in phishing attacks is a new dimension, not analyzed in previous works, while research on Lithuanian educational institution types and the cyber security training policies used is not available too. In this way this research brings additional insights for Lithuanian education institutions as well as presenting a framework, which could be repeated by other countries or institutions, to estimate their fighting strategy against phishing attacks.

2. Related Work

Phishing attacks are challenging to fight because of their wide variety of forms and situations. While the initial phishing attacks were mostly focused on email messages, current attacks are closely related to web technologies and different communication channels [9]. Web pages that are used for the gathering of phishing-attack data are challenging to find (identify newly created and not well-known web pages), but because of their nature, they are easy to recognize as fraudulent [10,11]. In most cases, the web pages used for phishing-attack data collection are determined by collecting a portion of phishing messages. These messages are linked to the web page. Therefore, the more critical task is to identify phishing messages and distinguish them from other legitimate messages. This is more relevant than finding fraudulent websites.

The current trend in phishing attack detection is mostly oriented toward machine learning [12,13]. These solutions identify phishing message patterns. To achieve better detection accuracy, natural language processing is used [14] to understand the message’s meaning and potential intention. In addition, the visual identity can be estimated [15]. All the solutions can identify the phishing web page or message with up to 99% accuracy. However, most research authors admit that spear-phishing attacks are more sophisticated, are adapted to individual persons or organizations, and are more difficult to detect [16].

In cases when automated phishing detection solutions do not work and when organization security does not rely on technical solutions only, human-oriented solutions are needed. Employee training is actively used [17]. Different training programs are developed to address specific phishing targets [18]. While some organizations notice situation improvement after phishing-related training [19], some research paper results indicate that even the training might not be enough to fight phishing attacks. The biggest affect is on employees’ habits, phishing email type [20] and potentially other properties. Therefore, the identification of the most affected employees and implementation of individual measures might be the way to achieve more resistant protection against phishing attacks.

However, the credulity of phishing attacks for and vulnerability of employees are not easy to determine. Researchers face challenges when testing employees’ ability to detect fraudulent messages [21] and estimate their credulity level [22]. Therefore, imitations of phishing attacks are used to test how employees react and act under these kinds of attacks [23]. For this purpose, both laboratory settings [24] and real environments [25] are used.

In the analysis of phishing attacks, different factors are analyzed. Most of it defines the weakest links in the security management process, based on employees’ age and gender [26]. The language effect is also investigated [27]. The specifics of different sectors are discussed too [28,29,30,31]. Research indicates that a wide range of properties, such as optimism [32], country [33], social impact [34], conscientiousness [35], socio-economic status [36] and others affect phishing cases’ credulity and reporting.

There are, however, a number of different tendencies among organizations [37] that must be reevaluated individually for each organization. The summary of phishing-email-link click rates, investigated during the last 5-year research works indicate the variety of results. This is because each sector is different, and there are different credulity levels in relation to phishing attacks for the persons in it. Even the type of industry plays a significant role in phishing attack identification [38]. Some research works were conducted to identify the level of credulity of phishing attacks in relation to education institutions [39,40,41]. Separate attention should be given to this sector because it is significant. However, there are no research studies investigating whether the same tendencies apply to different educational organizations in the same country. This answer would lead to understanding why the idea to create a common phishing management policy for education institutions is relevant and could be useful. Governments or local institutions, responsible for education systems, could consider to request all educational institutions to follow some main guidelines, dedicated to proper phishing management. At the same time, the existing research is mostly oriented towards the analysis of limited data fields—user properties and actions, not considering the time perspective. For example, Tianhao Xu and Prashanth Rajivan [42] designed a dataset of user reactions to phishing emails, where each person labeled defined emails as “respond immediately”, “flag and respond later”, “leave in the mailbox”, “delete” and “delete and block the sender”. This dataset could be used as a source to understand the relative timing of phishing email reaction; however, it does not provide quantitative data on how much time the organization has to react to the phishing email before the employees start submitting the data.

3. Materials and Methods

The analysis of related works revealed the importance of all three research questions that we raised in the research. It also illustrated that the existing research results are not answering it. Therefore, we designed new research, indicated to gather the needed data and confirm or reject the hypotheses that we defined in the introduction to the paper.

3.1. Scope of the Research

The research was conducted systematically since 2018. During the period, 5 imitated phishing attacks were executed (see Figure 2). Each email, used in the phishing attack imitation, was created taking into account some related institutions, but using publicly available data (upcoming organization event, name of the organization managing person, title of the organization department, etc.). Multiple versions of emails were created by the experiment authors trying to keep similar organizational specific data and the purpose of the email in order to eliminate the effect of the email type on employees’ behavior. The email generation approach was oriented toward an invitation to register for a free present, refund claim for data breach accident and requirement to change a password—calling for actions which are targeted at common human vulnerabilities and not specifically oriented towards a specific target group.

A different date was selected for each of the simulated spear-phishing attacks. This was to reduce the employees’ ability to determine whether it was a real or imitated attack. However, all the employees were informed about planned security training initiatives at least one week before the mock attacks. This was necessary to comply with the legal requirements. A one-week period was selected to assure that the employees would forget about the announcement in the internal system and would not associate the attack with more general reminders of planned security-related training initiatives.

The attacks were conducted on Lithuanian education institutions and employees only (experiments on how students respond to phishing attacks are not included in the research). Three attacks were carried out on Vilnius Gediminas Technical University (technical university) [40]. This was conducted repeatedly, and between attacks, dedicated training sessions were arranged [8]. One imitated attack was executed on another higher education institution (arts university). This organization has a different profile—more focused on arts than technologies. The last attack targeted all Lithuanian secondary schools, but only the main staff, not all the employees.

The arts university and secondary school had no systematic training, while the technical university practiced phishing management using experiments and training sessions. Based on these three examples of different types of education institutions in Lithuania, it should be possible to identify the differences between them as well as the impact of phishing training on them. Based on the systemic literature analysis of S. Zhuo et al. [23], our research falls into a large-group-sample-size simulated phishing experiment. It sent 6054 emails during the whole period to all the institutions. In this kind of experiment, the medium size of emails is 273, which is smaller than any of our individual experiments.

3.2. Design of the Phishing Experiments

Imitated attacks are one of the key components of proactive risk management. It serves several purposes: the evaluation of the current situation; identification of weak links; identification of attack properties that affect the credulity of phishing attacks for employees; evaluation of the applied mitigation actions (if they were applied); training by experience; evidence collection for employees’ training; and others. Therefore, it is imperative to design and execute social engineering attacks that are appropriate. The key aspects of the experiment design are realistic attacks, GDPR compliance and trustworthy results.

The general principle of the imitated social engineering attack and its data logging is presented in Figure 3. To ensure the evaluation of the attack type, applied mitigations are taken into account during the analysis, the properties of which must be discrete and clearly identified. This is in the development of imitation attacks. If the social engineering attack landscape is summarized in a taxonomy or another form, the modeling of potential attacks becomes less affected by human factors. It can be expressed clearly as a set of properties and their gained values. We used an email-based phishing attack taxonomy [39] to specify each email generated for the experiment. For example, in the development of the phishing email content, a set of properties was revised and based on its possible values from the taxonomy; therefore, each generated email was defined by reference to the discrete values of the aim of the imitated attack (benefit, legitimate requirement, important information, possible failure/lost, other); type (legal or spoofed) of email address; usage of attached files (yes or no); email text language (human or robot created); usage of individual or group emails.

Close to the social engineering attack content, technical aspects have been considered. This is needed to assure that the needed data for further analysis will be collected. Data recorded during the imitated attack included the employee’s identity (ability to trace a specific employee), what kind of actions were taken, what data were revealed, and when the actions were taken. To implement all data logging, the following implementation was carried out:

• Generate unique resources for each employee, to ensure traceability.
• For resource request identification and logging, include embedded sources.
• Log all identified user actions with all available data (time, IP address, used device, etc.).
• Assured employee-provided data confidentiality, preventing attack data leakage.

The architecture used to ensure phishing experiment data logging and the traceability of users are presented in Figure 4. The idea is to create emails with unique links and embedded images with unique URLs in them. The server generates the links and images programmatically; therefore, when the user loads the resource (by viewing it in the email or in the web browser), the programming code logs the access and all the possible data (user ID, based on the created URL, timestamp, action type, IP address, etc.). In the case of web form data filling, the data of the form are stored as well. As URL addresses were generated for each employee individually, the mapping between URLs and users (identified by their email) was maintained. Therefore, all user personal data can be mapped from the internal employee system as well.

This architecture is suitable for gathering the needed phishing experiment data. To ensure the privacy of personal data, the web server is protected at the needed security level. Personal user data are not stored on the server, just the randomly generated unique user URLs. The user-entered data are encrypted, to prevent private data leakage in the case of the low likelihood of webserver hacking.

3.3. Analyzed Research Data

To obtain data for the situation analysis, the following time-related data were logged on the server:

• Time when the email was sent to the user. The email contained unique identifiers dedicated to user action traceability. The time was logged on the email-sending server.
• Time when the user read the email. This was logged every time the user opened the email. Email opening was implemented by integrating an additional external resource, which was automatically loaded. The loading request initiated the creation of an email-opening journal record.
• Time when the user loaded the image embedded in the email. Time logging was implemented based on the same principle, as in the email-opening case.
• Time when the user opened the website, mentioned in the email. The journal record was created on the website.
• Time when the user opened the data entry form. Analog to URL visiting was implemented.
• Time when the user filled in the data. The journal record was saved in the form submission action.

All these data were used for timing analysis and the logging of user actions. The experiment duration was 3 days. The limit was added to ensure natural reactions. A longer period would cause employees to share their experiences and could reflect false positive results.

The user IP address was logged while gathering timing data. Later analysis was conducted to see where the employee opened the emails, i.e., inside or outside the office. Based on the user identifier, employees were identified and personal data were mapped to actions. We used employee age, gender, working position, workload and participation to analyze whether there was any relation to the credulity level of phishing attacks. We also included the employees’ data while preparing the training for the target group in the security training. In this research, personal data were not analyzed because the positions in different institutions vary considerably and could not be accurately matched. Age and gender factors have been analyzed in other research papers [20,30,40].

In this research, we added an additional parameter—the score of credulity level in relation to the user. The credulity score CS was calculated based on the principal schema presented in Figure 5 and Equation (1).

$$CS=\underset{i=0}{\overset{n_{im}}{\bigvee }}{im}_i+2·\underset{i=0}{\overset{n_{wl}}{\bigvee }}{wl}_i+3·\underset{i=0}{\overset{n_{ff}}{\bigvee }}{ff}_i-2·\underset{i=0}{\overset{n_{hd}}{\bigvee }}{hd}_i$$

(1)

there CS is the credulity level in relation to an employee, im_i is the i-th image loading in the email action (there i = 0 for the default value and i > 0 for all remaining n_im email-reading log entries), wl_i is the i-th web-loading action (there i = 0 for the default value and i > 0 for all remaining n_wl web-loading log entries), ff_i is the i-th form-filling action (there i = 0 for the default value and i > 0 for all remaining n_ff form-filling log entries), hd_i is the i-th email reporting to HelpDesk action (there i = 0 for the default value and i > 0 for all remaining n_hd email-reading log entries).

Each email contained a link to a website. If the user clicked on the link and opened the website, he or she received a new wl_i = 1 record, there i is the i-th web-loading action for this specific user. The same principle applies to web form filling data—if the user filled the data field with the correct data (this was estimated by comparing users’ personal data with the requested name and surname or email address data in the web form), he or she had ff_i = 1 for the i-th form-filling action, or ff_i = 0 if the data were falsely submitted.

Some users were conscious and took some actions. If the user reported the phishing attack to the HelpDesk (in any step of the research and using any means—reporting system, email, phone call, etc.), hd_i = 1 for the i-th reporting-to-HelpDesk action.

The weights for the steps were selected to be step-by-step increasing, while reporting to HelpDesk had a negative weight as it illustrates the awareness of the employee and the value is selected to be smaller than the one for data form filling. This was performed to reflect the fact that reporting a spear-phishing email does not eliminate the potential harm, obtained by revealing sensitive data in the spear-phishing email provided form.

Multiple logs might occur as the same person can execute the same task several times. Therefore, in Formula (1), the maximum score is used for each of the actions and multiplied based on some weight, illustrating the effect on the credulity. The timing analysis only took into account the first action time when defining the user’s fastest action.

4. Results

The data obtained in the research can be analyzed from different perspectives. To better address the research aim, the results are presented based on the research questions.

4.1. Is the Vulnerability of Education Organization Employees Dependent on the Organization and Its Cyber Security Management Policy?

To indicate the employee vulnerability to social engineering attacks, the percentage of employees that received the phishing email was analyzed. In the technological university (taking the last experiment results only), 1785 emails were sent to employees, where within 3 days, 679 employees read the email (38%). The art university had the lowest email read rate—24% (160 out of 656 employees). The administration of the secondary school demonstrated the highest email read rate—40% (141 out of 347 employees viewed the email). The statistics do not reflect the organization’s security level, but rather its response time—less than half of employees read emails at least once in a three-day period.

Meanwhile, analyzing what actions were taken by the employees who read the spear-phishing emails, the specifics of the education organization can be seen (see Figure 6).

It is clearly seen that the secondary school employees downloaded the image twice as often as higher education institutions. The most images were downloaded by the art university employees. And the results seem logical as the visual information in the email might seem relevant to art-related persons. Meanwhile, the low secondary school image download rate can be explained by technical social engineering mitigation solutions. A user must have additional knowledge to enable images that are automatically blocked. Therefore, despite the difference in image download ratio being visible, it is not representative in the sense of awareness of social engineering attacks.

The above-mentioned insight is strengthened by further numbers—the secondary school employees led by URL visiting and data reveal actions. Of all the secondary school employees who read the email, 70% opened the provided URL link (with a 95% confidence interval, the values ranged from 57% to 85%) and 17% (with a 95% confidence interval [11%; 25%]) provided the requested personal data. Technical university employees were eager to see what data were presented on the website—61% of them visited the URL (with a 95% confidence interval [55%; 67%]). While the matching percentage between downloaded images and visited websites might lead to the idea that the images were automatically downloaded, this was not the case—only 20% of employees performed both actions.

The arts university employees led in the image downloads; however, they had the lowest URL visits and data reveal ratios. Additionally, the numbers cannot be directly linked to the social engineering attack awareness level, but they decreased the likelihood of being attacked.

The last two actions are directly indicative of the awareness in the cyber security area. The percentage of employees who filled in the form with the requested data reached 17% in the secondary school (with a 95% confidence interval [11%; 25%]), 8% in the technical university (with a 95% confidence interval [6%; 11%]) and 4% in the arts university (with a 95% confidence interval [1%; 10%]). Similar trends were seen with the reports of phishing attacks. The secondary school employees led. Of those who read the email, 20% reported it as an incident (with a 95% confidence interval [13%; 29%]). In the technical university, the ratio was 11% (with a 95% confidence interval [8%; 13%]), and in the arts university, the ratio was 4% (with a 95% confidence interval [1%; 6%]). Despite the similarity between these two actions, several cases were noticed where the employee filled in the form and then reported it. These two actions were rarely carried out by the same person.

To estimate the significance of the differences, a two-sample t-test was used [43]. All the numbers and the p-value comparison between each type of institution for certain actions indicate significant differences (see

Table 1. p-value for the difference between the two rates with a 95% confidence interval.

Employee Actions	Comparison Between Different Types of Education Institution
	Technical vs. Arts University	Technical University vs. Secondary School	Arts University vs. Secondary School
Emails read	p < 0.0001	p = 0.4729	p < 0.0001
Image downloaded	p = 0.0295	p < 0.0001	p = 0.0160
URL visited	p < 0.0001	p = 0.0655	p < 0.0001
Filled form with data	p = 0.0008	p = 0.0027	p < 0.0001
Reported the incident	p < 0.0001	p = 0.0032	p < 0.0001

). Statistically no difference can be seen between the technical university and the secondary school, but just in the email read and URL-visiting actions. The arts university acted differently to the other institutions.

Taking into account the fact that the most important factor for credulity and organization resistance to phishing attacks was form filling with the requested data, we cannot confirm the first of our hypotheses (H01) that Lithuanian education system employees display similar behavior when responding to spear-phishing emails, and the organization type has no effect on the credulity of this kind of attack for employees. This means one framework for spear-phishing attack mitigation or that employee training on this topic should be individualized, as one management strategy is not necessary for all types of educational institution.

Considering the differences in employee behavior of the analyzed education institutions, it is difficult to determine the organization’s resistance to social engineering attacks. To express it, we asked two security experts to discuss and express a joint opinion on how important each ratio is to the final organization vulnerability level. The image download rate importance is 5%, while the URL visit rate importance is 10%. Incident reporting data rate had a 25% importance and data reveal rate of 60%. Assuming that the high incident reporting ratio is a positive score and the other ratios are better when lower, the final organization social engineering attack resistance S can be determined by using Formula (2).

$$S=0.05·\left(1-r_{im}\right)+0.10·\left(1-r_{wl}\right)+0.60·\left(1-r_{ff}\right)+0.25·r_{hd}$$

(2)

there S is the social engineering attack resistance score, r_im is the image download ratio (image download cases divided by the number of opened emails), r_wl is the website viewed ratio (URL visited cases divided by the number of opened emails), r_ff is the data reveal ratio for form-filling cases divided by the number of opened emails), and r_hd is the incident reporting ratio (incident reported cases divided by the number of opened emails).

Based on the formula and imitated spear-phishing attack logging data, the social engineering attack resistance scores suggest that the most resistant is the arts university (the score is 0.68), the technical university is second (score is 0.64), while the secondary school is third (score is 0.61). Hence, the difference in social engineering attack resistance scores is not significant. Despite the fact that the social engineering attack resistance score can be used as a metric for ranking organizations, but taking into account the fact that all four parameters used for its calculation have a different distribution between institutions, this score should not be considered as the metric for understanding the difference between different institutions.

The evaluation of each employee’s score regarding the credulity is presented in Figure 7. The numbers indicate that the majority of employees have a score of 1. This would indicate the situation when the user reveals some of the data to investigate whether the message really is fraudulent or not.

Analyzing the highest and lowest scores of employees, there are seven who did all the actions expected by the imitated hacker. They had no doubts or at least did not report the incident. Twelve employees were completely resistant to the spear-phishing attack and after reading the email, reported the incident, with no further investigation of the email’s intentions. By using the employee’s credulity score, the list of employees who require the most attention and dedicated training could be identified.

4.2. When Will Employees Respond to Spear-Phishing Emails Sent to Them?

The analysis of the timing of phishing attacks reveals a new perspective. Upon analyzing the timing results of the last experiment at the technical university, it was observed that employees took an average of 8.5 h to find and open the phishing email (see Figure 8, left boxplot diagram). We should consider that the waiting time was 72 h (3 days), but during the period the longest time was 42.4 h. This might be related to the requirement for employees to respond to received emails within 3 working days. Half of the emails were read within 4.88 h, while the quickest time to open them was 15 s.

The average time to click on the link in the email and open the web page was 86 s (see Figure 8, middle boxplot diagram). The average is highly affected by the five outlier values. These values aim to reach up to 10 min, while the upper quartile values do not reach 1.5 min. Data filling in the form requires some time, but the values usually fit into 5 min.

By analyzing the timing of those 57 employees who filled in the form with the requested data, we found that the average data entry time from the moment when the email was sent was 808 s. Taking the standard deviation values (std = 478) and comparing them to the 300 s interval estimated in our second hypothesis, we see that between those values, there are significant differences (p < 0.0001). The time from the moment when the phishing email was sent to when the user filled in the form with the requested data with a 95% confidentiality had a very broad range. Considering that the maximum mean time was 934 s and the standard deviation was 478, the lowest value of the range in a 95% confidentiality interval was lower than 300 s. Hence, hypothesis (H02) that no employee will reveal their data in 5 min or sooner is rejected. The research results indicate that eight data forms were submitted within 5 min faster than before. This was about 14% of the cases. Meanwhile, the fastest submitted form was received 45 s after the phishing email was sent. The fastest form filling could be even faster as the sum of the lowest values for each step would be 27 s. This highlights the importance of automated email filtering and employee awareness regarding social engineering attacks as they become more prevalent. If at least one employee provides all the required data for the hacker in less than 5 min, there will be no time for the cybersecurity management team to react. In this experiment, the first incident registration was received less than 6 min after the email had been read.

The development of new, usually AI-based solutions for phishing websites [44] and emails [45] indicate that we are approaching a high accuracy (reaching 96–99% F-score) in regard to identifying phishing attacks and potentially could achieve results with even higher accuracy in the future. However, at the same time, it is important to use the up-to-date phishing filtering solutions and guarantee that legal user emails will not be blocked.

4.3. How Effective Is Cyber Security Training for Lithuanian Education Organization Employees?

The application of a system that mimics social engineering attacks allows the evaluation of the organization’s mitigation mechanisms. In 2018, the first imitated attack was conducted at the technical university. The organization’s employee social engineering attack resistance score (1), based on the received data was 0.49 (1648 emails were emailed, 708 employees viewed them). Then a dedicated cyber security training program was implemented. As a result of the program, a second imitation of a phishing attack was conducted. Based on its results, the social engineering attack resistance score increased to 0.74 (1618 emails were sent, 931 employees read them). The same training material was used for the second training session. In the third imitation session, the resistance score to social engineering attacks decreased to 0.64 (1785 emails were emailed, 679 employees read them). The numbers affecting the score are presented in Figure 9.

If we take the organization’s employee social engineering attack resistance scores as proportions of the security, where the sample size is the number of employees who read the email, we would see a significant resistance score increase in comparison to the first imitation attack (p < 0.0001 comparing the first and second imitation and the first and third imitation). This indicates that training and repetition of imitation can be effective. However, the difference between the second and third imitations is significant as well (p < 0.0001). As employee training was repeated, this could potentially be affected because the training material was not renewed.

The lack of training material updates is visible in all areas: image download and website visits jumped to an even higher ratio than it was in the initial experiments in 2018; reveals of the data are more than three times smaller in comparison to the first imitation, but increased in comparison to the second imitation experimental data when employees first underwent the dedicated training program; reporting of the incident dropped twice in comparison to the second imitation, but was ten times higher than in the initial stage. All this can be summarized as facts, that the applied training program is effective as it is oriented to the two metrics, mostly affecting the social engineering attack resistance score (prevention of data reveal via form filling and reporting of the suspicious emails); however, it is not effective any more to address the importance of image downloading and website visiting.

Hypothesis (H03) is that security training affects the credulity level of spear phishing attacks for educational institution employees. At the same time, an increase in the organization’s resistance score is visible compared to the situation when no training was performed. Meanwhile, if the training material is not updated to reflect the constantly changing nature of phishing attacks, repetitive training will not increase the resistance score and even lead to its decrease, because of false positive confidence, caused by a lack of knowledge of the most advanced spear-phishing methods.

5. Discussion

The research results confirm the complex nature of organization resistance to spear-phishing attacks. The assumption that educational sector organizations’ employees in Lithuania should reveal similar behavior during imitated phishing attacks was not confirmed. The technical university and secondary school employees shared some similarities, but not in regard to data disclosure and reporting, which are the most important. The behavior of arts university employees differed from that of the two other types of educational institution. This indicates the need for an individual phishing attack mitigation plan for each organization.

It might be possible to cluster different types of educational institution based on the behavior of some groups that may share the same security policy by including more similar types of educational institution in the imitating spear-phishing attack experiments. Taking into account the need for organization agreement to execute the experiments and needed resources to prepare for suitable imitation execution, a wide range of education organizations’ resistance to spear-phishing attack estimation has a limited chance to be accomplished without national-level decisions.

The phishing attack mitigation plan should not rely on the HelpDesk team’s ability to respond to reported suspicious emails. The first reported incident occurred 5 min after the first data reveal case. The time needed to reveal the data was less than a minute from the time when the person read the email. The number highlights the need for automated solutions against phishing attacks, while to mitigate spear-phishing attacks, employees’ training should be organized. It is not enough to prepare learning material and train employees. The experiment shows that the material used for training one year ago is no longer effective—the resistance score does not increase, but decreases significantly. This could be related to false confidence and ignorance of spear-phishing techniques’ always changing nature. In addition, it is important to update the training materials, including the most advanced techniques and educating employees to have the ability to identify even new instances of spear phishing that have not been reported.

6. Conclusions

The executed experiments illustrate the need for an individual phishing-attack strategy in different Lithuanian education institutions, as their employees’ resistance scores varied. The proposed resistance score can be used to analyze the progress of the same organization, repeating imitated phishing attacks and enabling numerical comparison-based analysis. This could lead to even more accurate reaction in regard to social engineering attack risk management success, utilizing systematic testing of employees and comparing their resistance change.

The phishing attack prevention policy cannot be based just on employees’ resistance and HelpDesk ability to react to reported cases. The numbers of cases in each Lithuanian education institution where employees provided all the information requested in phishing emails indicate that none of the organizations are safe as the ratio of such employees varied from 4% to 17%. The time usually taken for the employees to follow the phishing email instructions and fill in the data was less than 5 min, while the reporting required more than 5 min for the employees. Such a situation does not provide a chance for the HelpDesk to prevent such emails, if no efficient and automated phishing attack filtering options are used and a simplified option for phishing email reporting exists.

Furthermore, data analysis confirms the need for training material renewal. When employees are continually reminded about the importance of not visiting untrusted websites and downloading attachments, they lose their awareness of the impact on the security of the organization. The importance of personal data protection and the need for incident reporting are becoming more noticeable. Without an update of the training material, understanding among some employees will vanish, but the decrease is not as high as in cases of untrusted website visiting and file downloads.