Personalized Advertising Computational Techniques: A Systematic Literature Review, Findings, and a Design Framework

: This work conducts a systematic literature review about the domain of personalized advertisement, and more speciﬁcally, about the techniques that are used for this purpose. State-of-the-art publications and techniques are presented in detail, and the relationship of this domain with other re-lated domains such as artiﬁcial intelligence (AI), semantic web, etc., is investigated. Important issues such as (a) business data utilization in personalized advertisement models, (b) the cold start problem in the domain, (c) advertisement visualization issues, (d) psychological factors in the personalization models, (e) the lack of rich datasets, and (f) user privacy are highlighted and are pinpointed to help and inspire researchers for future work. Finally, a design framework for personalized advertisement systems has been designed based on these ﬁndings.


Advertisement Domain
Electronic commerce applications play an important role in everyday life for millions of people [1]. Peoples' daily activities in many sectors such as financial transactions, navigation, and entertainment have changed [2]. A popular domain that has benefited from this evolution is marketing and advertising [3][4][5]. This can be illustrated by the fact that Facebook's worldwide advertising revenue from the previous year is estimated at USD 69 billion while Google's is estimated at USD 134 billion. Nowadays, this domain is combined with other emerging software-related domains such as artificial intelligence, semantic web, internet of things (IoT), and hardware-related technologies (sensors, RFIDs, augmented reality-AR, etc.) [6,7]. Researchers and industries focus on collecting and utilizing contextual knowledge, every piece of information that is useful for inferring a user's situation and determining his/her requirements to display the right advertisement at the right time (contextual-based advertising) [8,9]. Such information could be (a) user environment (location, activities, etc.), (b) user browsing history and cookies (e.g., pages visited, searching for products, etc.), or (c) user profile (social media data, preferences, check-ins, etc.) [10]. Such approaches offer new opportunities for advertisers, users/potential customers, and personalized advertisement mediator systems. More efficient personalized systems can be designed, and better customer relationships and more interactive communication between consumers and businesses can be built [11][12][13].

Related Reviews and Motivation
There are some very interesting reviews in the literature that have inspired our work. These works can be classified into two major categories. The first category includes works that focus on studying the computational techniques that have been used by researchers to display relevant ads [8,14]. These works present and discuss state-of-the-art algorithms and techniques that are used in personalized advertisement domains, such as those applied when ranking advertisements, deciding whether to display an advertisement, or real-time bidding (RTB).
For the work included in the second category, many researchers focus on non-computational/theoretical issues that are important when designing personalization models in their reviews. Cleff 2010 focuses on privacy and discusses fundamental principles and information practices that are used for protecting the private data of individuals [15]. Shankar et al., 2010, propose a design framework that comprises three key entities: the consumer, the mobile, and the retailer [11]. In the same spirit, Boerman et al., 2017, provide an overview of the empirical findings by developing a framework that identifies and integrates all of the factors that can explain consumer responses toward behavioral advertising (e.g., level of personalization, individual characteristics, etc.) [16]. The framework addresses key related issues such as mobile consumer activities, mobile consumer segments, mobile properties, key retailer mobile marketing activities, etc. Bhat et al., 2019, conducted a systematic review of social marketing and happiness management [17]. They summarize the key findings of different research articles and identify important research gaps. Additionally, Cherubino et al., 2019, study neurophysiological measures and how they affect consumer behavior [18]. Swani et al., 2020, focus on B2B advertising and areas such as the applied theories; the message, content, and strategies used; media; performance metrics; and budgeting [19]. Furthermore, Bauer et al., 2011, andGrewal et al., 2016, both discuss technical factors (e.g., context variables, computational experimental analysis, etc.) and other non-technical aspects in their works to examine advertising in a more general way [20,21]. Aspects such as methodological approach (design-oriented approach, social science experiment with users, computational experimental analysis, and survey), advertisement goals, metrics, and other factors are discussed.
Beginning with the first category, to the best of our knowledge, reviews that focus on computational models in personalized advertisements have been conducted for a few years; thus, it is useful to collect and discuss findings from the past few years. Artificial intelligence, big data, internet of things, etc., are now used consistently for this purpose [22][23][24]. Additionally, works that examine the domain in a more broadened way (second group above) provide interesting findings that could be used in personalized advertisement systems to enhance them. For example, by taking these findings, into consideration systems can improve their architecture, their user interface/design, have more efficient computational models, etc. Not only personalized advertisement but also "personalized marketing" could be achieved, improving the user experience.

Objective and Contribution
The objective of this work is to provide a systematic literature review in the field of personalized advertising and to combine both the above worlds, spot interesting findings, and present a design framework that could be adopted by personalized advertisement systems.
Upon conducting this extended analysis in the literature, the following questions will be answered: What types of techniques, methodologies, and approaches (design-oriented, system implementation, computational techniques, etc.) are used to achieve better personalized advertisements? b.
What kind of issues, challenges and limitations exist in the domain? c.
How can we combine the personalized advertising domain with other domains such as recommender systems, social networking analysis, semantic web, IoT, marketing, business, etc., and bridge the gap between them?
The contributions of the present work can be summarized through the following points: 1. This work performs a systematic literature study in the field of personalized advertising techniques and presents them in groups based on the focus and the approaches that are used. Because of the commercial value of the domain, many researchers and companies do not reveal much information.
Inclusion in the review.
Information 2021, 12, x FOR PEER REVIEW 4 of 38 Regarding the screening process, a detailed study was performed on these titles and abstracts to identify the relevance with the topic (personalized advertisement techniques). Papers that focus on other areas, although the terms "advertising" and "marketing" appear in the paper, were excluded. We also excluded those that focus on areas other than the scope of our study even if the terms "computational", "personalized", and "mobile" appeared in the paper. Finally, over 1600 records were removed. 417 articles qualified for derailed study. In this step, 116 works were excluded since their focus was in other areas.
Step 3. Paper inclusion Finally, 301 papers from the most well-known journals and conferences of the domain are included in the presented study. These papers have been grouped into six categories (relative to the focus of the study) and are presented in detail in the next section. Useful information regarding the techniques that were used, and the focus of the study were drawn from each paper. This information is presented in Section 3 of this manuscript. Figure 2 presents the number of papers that are included from various sources/scientific databases, and Figure 3 displays the number of publications by year.   In more detail:

Step 1. Work Identification
To being, quick research was performed using review papers from related domains to spot the all the well-known scientific databases [27,28]. After that, we defined all of the keywords that were used to retrieve articles. The keywords that we used were "mobile advertising", "computational advertising", and "personalized advertising". An extensive search was conducted in all of the related and well-known scientific databases, such as ACM, IEEE, Elsevier, Springerlink, Microsoft Academic database, Hindawi, Taylor and Francis, DOAJ, Wiley, Web of Science, Emerald, etc. Google Scholar was also used to retrieve other works that may have been omitted. It is worth mentioning that a filter to only return papers that were published after 2005 was set. Regarding the selection of 2005 as a starting point, it was chosen (a) because it was a milestone (the mobile commerce area started in 2005, but our research does not only focus on mobile commerce advertising) and (b) to achieve a balance between presenting up-to-date works and including a representative sample that shows the evolution of the domain through the years. Titles and abstracts from the returned results were collected (over 2000). Then, all duplicates were removed.

Step 2. Screening
Regarding the screening process, a detailed study was performed on these titles and abstracts to identify the relevance with the topic (personalized advertisement techniques). Papers that focus on other areas, although the terms "advertising" and "marketing" appear in the paper, were excluded. We also excluded those that focus on areas other than the scope of our study even if the terms "computational", "personalized", and "mobile" appeared in the paper. Finally, over 1600 records were removed. 417 articles qualified for derailed study. In this step, 116 works were excluded since their focus was in other areas.

Step 3. Paper inclusion
Finally, 301 papers from the most well-known journals and conferences of the domain are included in the presented study. These papers have been grouped into six categories (relative to the focus of the study) and are presented in detail in the next section. Useful information regarding the techniques that were used, and the focus of the study were drawn from each paper. This information is presented in Section 3 of this manuscript. Figure 2 presents the number of papers that are included from various sources/scientific databases, and Figure

Categorization of Research Literature Based on the Topic and State-of-the-Art Techniques
After studying the selected works, the approach that was followed in this manuscript was to group them based on the aim/topic and the approaches/techniques used in each work. These works have been grouped into the following six categories ( Figure 4): 1. Works where the aim is to design and implement a personalized advertisement model. Personalized advertisement models that utilize user context, profile, history etc., and that then try to predict user interests about the available advertisements, usually by predicting CTR (click-through rate) or by ranking them. To achieve this, they apply computational techniques such as neural networks, distance function, probabilistic-based techniques, among other techniques. 2. Works where the topic is a content-based analysis and text matching. Researchers in this domain focus on techniques that extract textual information from (a) a web page (or even a mobile application or a video), (b) the candidate advertisements, and (c) the user profile. This information is usually represented by keywords and tags. Finally, researchers try to find the most suitable advertisement based on this content/keyword matching. 3. Works that are interested in studying consumer behavior and identifying the personalization factors that affect user attitudes towards advertisement acceptance. Usually, surveys and real-user experiments are used for these purposes. Although the vast majority of these works focus on the design step of a model or a framework and does not include implementation, to the best of our knowledge, their findings are very

Categorization of Research Literature Based on the Topic and State-of-the-Art Techniques
After studying the selected works, the approach that was followed in this manuscript was to group them based on the aim/topic and the approaches/techniques used in each work. These works have been grouped into the following six categories ( Figure 4):

1.
Works where the aim is to design and implement a personalized advertisement model. Personalized advertisement models that utilize user context, profile, history etc., and that then try to predict user interests about the available advertisements, usually by predicting CTR (click-through rate) or by ranking them. To achieve this, they apply computational techniques such as neural networks, distance function, probabilistic-based techniques, among other techniques.

2.
Works where the topic is a content-based analysis and text matching. Researchers in this domain focus on techniques that extract textual information from (a) a web page (or even a mobile application or a video), (b) the candidate advertisements, and (c) the user profile. This information is usually represented by keywords and tags. Finally, researchers try to find the most suitable advertisement based on this content/keyword matching.

3.
Works that are interested in studying consumer behavior and identifying the personalization factors that affect user attitudes towards advertisement acceptance. Usually, surveys and real-user experiments are used for these purposes. Although the vast majority of these works focus on the design step of a model or a framework and does not include implementation, to the best of our knowledge, their findings are very useful and can be used to enhance existing state-of-the-art personalization models (e.g., as input variables, for better context perception).

4.
Works that consider advertiser bidding and the scheduling of advertisements and that focus on maximizing their profits. The topic of these works is the optimal scheduling of the advertisements based on the advertiser's budget.

5.
Works that focus on privacy and security regarding the personal data of the users. Privacy and security are very important factors when developing advertisement personalization systems, and consequently, many researchers focus on the design and the implementation of related techniques. 6.
Works that focus on advertisement positioning, interactivity, and visualization. Their aim is to identify optimal advertisement positioning, design techniques for image and message customization, and other related issues.
The selected works were categorized into the above six categories because these six categories (a) have a different topic/aim (highlighted as important in the literature, having a sufficient number of related works), (b) could potentially be combined in an advertising service consisting of different components, and (c) are mutually exclusive (apart from very Information 2021, 12, 480 6 of 36 few exceptions that had a two-fold objective). The number of articles that belong to each category is displayed in Figure 4. All of these categories are presented below.
6. Works that focus on advertisement positioning, interactivity, and visualization. Their aim is to identify optimal advertisement positioning, design techniques for image and message customization, and other related issues.
The selected works were categorized into the above six categories because these six categories (a) have a different topic/aim (highlighted as important in the literature, having a sufficient number of related works), (b) could potentially be combined in an advertising service consisting of different components, and (c) are mutually exclusive (apart from very few exceptions that had a two-fold objective). The number of articles that belong to each category is displayed in Figure 4. All of these categories are presented below.

Research in Computational Models That Predict User Interest towards Ads
As discussed above, the first and most popular category concerns personalized advertisement models that utilize user context to predict his/her interest about the available advertisements [29]. The purpose of these models is to predict CTR and to rank the available advertisements [30]. By studying these works, the aim is to review the techniques that have been used during CTR prediction and to identify the state-of-the-art techniques.
The first step for implementing these models is to collect useful information about the user. Techniques from context-based systems, IoT, or semantic web are used to infer and represent the user profile and context [31]. To become more specific, mobile programming development technologies (e.g., Java, Kotlin, Swift) provide developers with the capability of gathering user information and implementing the interface [32]. Additionally, the evolution of IoT platforms such as Node-RED (https://nodered.org/, 10 November 2021), Google Cloud (https://cloud.google.com/, 10 November 2021), Particle.io (https://www.particle.io/, 10 November 2021), and many others can provide a huge boost to this domain since data from many devices and services can be extracted. [27]. Variables such as time, location, history, demographics, activity, device type, mood, blood pressure, etc., are utilized in the models as inputs [33]. Semantic web technologies such as ontologies and rules (e.g., Sesame, Jena and other tools are used for this purpose) can be applied

Research in Computational Models That Predict User Interest towards Ads
As discussed above, the first and most popular category concerns personalized advertisement models that utilize user context to predict his/her interest about the available advertisements [29]. The purpose of these models is to predict CTR and to rank the available advertisements [30]. By studying these works, the aim is to review the techniques that have been used during CTR prediction and to identify the state-of-the-art techniques.
The first step for implementing these models is to collect useful information about the user. Techniques from context-based systems, IoT, or semantic web are used to infer and represent the user profile and context [31]. To become more specific, mobile programming development technologies (e.g., Java, Kotlin, Swift) provide developers with the capability of gathering user information and implementing the interface [32]. Additionally, the evolution of IoT platforms such as Node-RED (https://nodered.org/, 10 November 2021), Google Cloud (https://cloud.google.com/, 10 November 2021), Particle.io (https://www. particle.io/, 10 November 2021), and many others can provide a huge boost to this domain since data from many devices and services can be extracted. [27]. Variables such as time, location, history, demographics, activity, device type, mood, blood pressure, etc., are utilized in the models as inputs [33]. Semantic web technologies such as ontologies and rules (e.g., Sesame, Jena and other tools are used for this purpose) can be applied to model these entities, manipulate data, and solve potential interoperability issues (when transferring data between different services) [34].
The second step is to apply computational techniques to achieve the desired purpose. Many well-known programming languages have be used (e.g., Java, PHP, R etc.), but Python is adopted by the vast majority of researchers to implement these computational techniques [35]. Python provides standard libraries that can be incorporated and modified by researchers/developers to implement computational techniques. A lot of techniques have been used for these purposes. Some well-known categories that could be spotted in the literature are the following:

•
Graph-based methods (Gh), social-e.g., PageRank: Mathematical structures used to model pairwise relationships between objects. A graph consists of vertices that are connected by edges [36,37]. • Classification methods (Cs)-e.g., naive Bayes, k-NN, logistic regression, decision trees, random forest, support vector machines, factorization machines-based models, etc.: Classification algorithms are used to categorize data into a class or category for a given example of input data. There are three types of classification that are possible: binary classification, multiclass classification, and multilabel classification [38].
• Knowledge-based (Kb) or Rule-based methods: These systems are a form of AI and aim to capture the knowledge of human experts for support decision-making. A knowledge-based system has two distinguishing features: a knowledge base to represent the existing knowledge and an inference engine to derive new knowledge [39]. • Clustering (Cl)-e.g., fuzzy clustering, k-means: These approaches try to identify similar classes of objects. Clustering is an unsupervised machine learning task. It involves automatically discovering natural grouping in the data [40]. • Distance functions and weight-based algorithms (Wd): Mathematical formulas that are based on distance functions and weights. A distance function is a function that gives a distance between each pair of point elements in a set [41]. • Statistical and probability-based (Sp)-e.g., Markov-based: Mathematical formulas that are based on statistics and probabilities [42]. • Collaborative filtering techniques for ranking (Cf)-e.g., matrix factorization, principal component analysis-PCA, SVD, similarity-based: These techniques are used consistently in recommender systems, and they aim to identify similarities among users [43]. When they provide recommendations to a user, the core philosophy is to identify users with similar behavior (e.g., purchased items) and recommend items from their lists.

•
Neural networks (NNs)-e.g., deep learning in the vast majority of methods, such as deep Boltzmann machines, deep recurrent neural networks, deep reinforcement learning, etc.: NNs are computing systems inspired by the biology and the neurons that constitute animal brains. Every NN system is a web of interconnected entities that are known as nodes where each node is responsible for a simple computation. Node elements receive inputs and deliver outputs based on their predefined activation functions. Typically, NNs are aggregated into layers [44].
Thus, every work in this domain is assigned to one of the above categories depending on the approach that is used. In some rare cases, where a combination of techniques is used in a paper, both techniques are considered. Figure 5 below displays (grouped) these approaches through the years. The time periods were selected to be almost equally distributed. It can be noticed that during the period of 2006-2013, the graph-based techniques were used a lot, showing a trend towards social-based recommendations during this period (eight works are included for this time period, and afterwards, their number was reduced significantly). From the same figure, we can conclude that in the past few years, classification and neural network-based approaches have been used to most. Indeed, factorization machine-based models and deep learning have been highlighted as the most popular and promising techniques in the domain, having overcome many challenges [29]. Presently, researchers have focused on this domain. Additionally, the use of graph-based and statistical methods has decreased. It is worth mentioning that the datasets found e.g., in Kaggle (Avazu, Avito) (https://www.kaggle.com/datasets, 10 November 2021), Criteo (https: //labs.criteo.com/2014/02/download-kaggle-display-advertising-challenge-dataset/, 10 November 2021), Meitu, among others (some not publicly available), have been used to evaluate models [45]. Table A1 in Appendix A includes all of the works that have been collected and grouped together into this category in more detail along with the techniques that are used in each one. Moreover, a few related reviews are included in this category (but omitted in the diagram below).

Research in Content-Based Analysis and Text Mining
An important part of the literature includes works that focus on content-based analysis and text mining. Researchers extract textual information from a webpage (or even a mobile application or a video), the candidate advertisements and the user profile (usually as keywords), and they then try to find the most suitable advertisement based on the content of the profile/keyword matching. By studying these works, the objective is to review the techniques that have been used in this domain and to identify the current state of the art. Researchers usually use natural language processing (NLP) techniques to extract related entities that can be used for sentiment analysis on reviews, for aspect mining, topic modeling, or even for message generation. Classification techniques, clustering techniques such as latent Dirichlet allocation, neural networks, among others, are used consistently for these purposes. Additionally, distance functions, weighted-based algorithms, etc., are used for keyword matching. Sometimes all of the above techniques are combined with semantic web technologies such as ontologies to enhance the whole process (e.g., for synonyms, for interoperability).
Information 2021, 12, x FOR PEER REVIEW 8 of 38 Presently, researchers have focused on this domain. Additionally, the use of graph-based and statistical methods has decreased. It is worth mentioning that the datasets found e.g., in Kaggle (Avazu, Avito) (https://www.kaggle.com/datasets, 10 November 2021), Criteo (https://labs.criteo.com/2014/02/download-kaggle-display-advertising-challenge-dataset/, 10 November 2021), Meitu, among others (some not publicly available), have been used to evaluate models [45]. Table A1 in Appendix A includes all of the works that have been collected and grouped together into this category in more detail along with the techniques that are used in each one. Moreover, a few related reviews are included in this category (but omitted in the diagram below).

Research in Content-Based Analysis and Text Mining
An important part of the literature includes works that focus on content-based analysis and text mining. Researchers extract textual information from a webpage (or even a mobile application or a video), the candidate advertisements and the user profile (usually as keywords), and they then try to find the most suitable advertisement based on the content of the profile/keyword matching. By studying these works, the objective is to review the techniques that have been used in this domain and to identify the current state of the art. Researchers usually use natural language processing (NLP) techniques to extract related entities that can be used for sentiment analysis on reviews, for aspect mining, topic modeling, or even for message generation. Classification techniques, clustering techniques such as latent Dirichlet allocation, neural networks, among others, are used consistently for these purposes. Additionally, distance functions, weighted-based algorithms, etc., are used for keyword matching. Sometimes all of the above techniques are combined with semantic web technologies such as ontologies to enhance the whole process (e.g., for synonyms, for interoperability).
In order for the implementation to be more specific, first of all, standard web development technologies (such as HTML, CSS and JS) or mobile development technologies (e.g., Kotlin) can be used to create the user interface, interact with the user, and extract information [46]. Once again, many well-known programming languages have been used to build computational models, but Python is adopted by the vast majority [35]. Semantic web technologies such as Sesame, Jena, among others, are sometimes used to model related entities and to conceive semantic similarity.
In Figure 6, all of these techniques are grouped, and the number of works that belong to each group is displayed (if a combination of techniques is used in a work, both techniques are considered). Table A2 in Appendix A displays these works and the techniques used in each case in detail. More sophisticated techniques such as deep networks, classification, and clustering seem to outperform distance-based formulas and are used more consistently the past few years. In order for the implementation to be more specific, first of all, standard web development technologies (such as HTML, CSS and JS) or mobile development technologies (e.g., Kotlin) can be used to create the user interface, interact with the user, and extract information [46]. Once again, many well-known programming languages have been used to build computational models, but Python is adopted by the vast majority [35]. Semantic web technologies such as Sesame, Jena, among others, are sometimes used to model related entities and to conceive semantic similarity.
In Figure 6, all of these techniques are grouped, and the number of works that belong to each group is displayed (if a combination of techniques is used in a work, both techniques are considered). Table A2 in Appendix A displays these works and the techniques used in each case in detail. More sophisticated techniques such as deep networks, classification, and clustering seem to outperform distance-based formulas and are used more consistently the past few years.
Information 2021, 12, x FOR PEER REVIEW 9 of 3 Figure 6. Approaches on content-based analysis and text mining.

Works That Identify Factors That Affect Advertisement Acceptance
A lot of researchers focus on identifying the most important factors that affect use attitude towards advertisements [47].By studying these works, the aim is to spot and writ

Works That Identify Factors That Affect Advertisement Acceptance
A lot of researchers focus on identifying the most important factors that affect user attitude towards advertisements [47].By studying these works, the aim is to spot and write down these factors and to provide suggestions on how personalized advertisement models can use these findings to become even more effective. Usually, their methodology consists of three phases. In phase 1, they collect the data; in phase 2, they apply computational techniques to these data to extract important factors and findings; and in phase 3, they present these findings, usually by designing a framework. Regarding the data collection process, researchers typically use questionnaires or conduct real user experiments to collect data. In the past few years, more sophisticated experiments that involve neuroscience technologies (e.g., Electroencephalogram, eye tracking, facial coding etc.) have been conducted [48]. Tools such as Versatile EEG, EyeWorks, Observer XT, and many other are used for these purposes [49]. Some other approaches also use business datasets to extract useful information.
Regarding data management and the factor extraction process (phase 2), the most common practice is to apply statistical analysis techniques such as ANOVA, correlation, factor analysis, etc., to the data. SPSS software is widely adopted, but many other tools are used as well [50]. Moreover, more sophisticated machine learning techniques (classifiers, Support Vector Machines, etc.) have been used in a few cases when more data are available (e.g., in cases where researchers utilize business data). Finally, many factors/findings have been highlighted as being important in the literature, such as entertainment, gamification, informativeness, social ties, psychological factors, etc. [51]. Table A3 in Appendix A displays all of these factors along with the techniques used to extract them in detail. Context-aware systems that display advertisements can infer these factors and can include them as input in the models to enhance them.

Research in Real-Time Bidding (RTB) and Advertisement Scheduling
This section includes a group of works that consider advertiser bidding and the scheduling of advertisements, focusing on maximizing their profits. User interest towards an advertisement is combined with advertiser bidding/budget to find the optimal strategy for them. By studying these works, the aim is to review the techniques that have been used and to identify the state of the art. Many techniques have been used for this purpose such as: Optimization functions (e.g., minimax-based functions, game theory-based, etc.); • Classification (e.g., logistic regression); • Statistical and probability based; • Knowledge-based and rule-based methods (e.g., Apriori); • Shallow and deep learning neural networks (deep Boltzmann machines, deep reinforcement learning, etc.).
Similar to above, the Python programming language is used by most developers for implementation. Figure 7 below displays these approaches grouped by category. Not surprisingly, optimization functions are of the vast majority in this category. Deep learning and regression methods seem to fit well and could also be used more frequently. Table A4 in Appendix A provides more information about the works and the techniques used.

Privacy-Based Approaches
In the literature, privacy is highlighted as a very important factor when designing advertisement services, especially with the new regulations. By studying these works, the aim is to review the state-of-the-art techniques that have been used in this domain regarding privacy and security so that personalized advertisement systems can adopt them. Researchers usually apply a privacy-focused software architecture, encryption techniques, and other methods in rare situations. Common web and mobile development technologies are used for the system implementation. Table A5 in Appendix A presents works from this domain that were spotted in the literature and the approach that they use. To the best of our knowledge, it can be determined by that there are not enough works that utilize the knowledge-based techniques, rule-based approaches, and neural networks. These approaches have been used for fraud detection, privacy, etc., in other domains, and it would be interesting to examine if any of the above techniques (classification, NNs, etc.) can be used in advertising (e.g., image recognition, detect privacy violations, etc.) in future research. Similar to above, the Python programming language is used by most developers for implementation. Figure 7 below displays these approaches grouped by category. Not surprisingly, optimization functions are of the vast majority in this category. Deep learning and regression methods seem to fit well and could also be used more frequently. Table A4 in Appendix A provides more information about the works and the techniques used.

Privacy-Based Approaches
In the literature, privacy is highlighted as a very important factor when designing advertisement services, especially with the new regulations. By studying these works, the aim is to review the state-of-the-art techniques that have been used in this domain regarding privacy and security so that personalized advertisement systems can adopt them. Researchers usually apply a privacy-focused software architecture, encryption techniques, and other methods in rare situations. Common web and mobile development technologies are used for the system implementation. Table A5 in Appendix A presents works from this domain that were spotted in the literature and the approach that they use. To the best of our knowledge, it can be determined by that there are not enough works that utilize the knowledge-based techniques, rule-based approaches, and neural networks. These approaches have been used for fraud detection, privacy, etc., in other domains, and it would be interesting to examine if any of the above techniques (classification, NNs, etc.) can be used in advertising (e.g., image recognition, detect privacy violations, etc.) in future research.

Research in Advertisement Interactivity and Visualization
Last but not least, various works have been spotted in the literature that focus on the presentation and the visualization of advertisements. The aim here is to spot the state-ofthe-art techniques that have been used in this area and to provide suggestions on how they can enhance personalized advertisement systems. These works are grouped into the following categories based on the purpose they have:


Optimal ad positioning;  Advertisement (message or image) customization;  Infer user emotion and intention when viewing the advertisement;  Advertisement interactivity.
More specifically, regarding the first category, many researchers focus their interest on optimal ad positioning [52]. Probabilistic, software architecture-based approaches, and

Research in Advertisement Interactivity and Visualization
Last but not least, various works have been spotted in the literature that focus on the presentation and the visualization of advertisements. The aim here is to spot the state-of-the-art techniques that have been used in this area and to provide suggestions on how they can enhance personalized advertisement systems. These works are grouped into the following categories based on the purpose they have:

•
Optimal ad positioning; • Advertisement (message or image) customization; • Infer user emotion and intention when viewing the advertisement; • Advertisement interactivity.
More specifically, regarding the first category, many researchers focus their interest on optimal ad positioning [52]. Probabilistic, software architecture-based approaches, and other techniques are used for this purpose. Regarding the second category, a common practice is to customize advertisement images or messages as well [53]. Statistics (usually in data from real user experiments), fuzzy logic, neural network-based, and other methods are used in this direction. Moreover, in the past few years, other approaches have gained popularity and can extract user emotions by applying neuroscience techniques or use new hardware and software-based techniques like Augmented Reality to present the advertisement in a more interactive way [54,55].
Regarding implementation, standard web/mobile development, and neuroscience technologies (that are discussed in detail above) are used to obtain information, create the user interface, and develop the related algorithms. It is worth mentioning the evolution of augmented reality, which has gained huge popularity in the past few years. Tools such as Vuforia, ARKit, etc., are used to develop such functionalities [56]. Table A6 in Appendix A below presents all of the related works (including the approach/techniques which are used and their purpose), and the diagram in Figure 8 displays all of these grouped into the above four categories. From the findings, it can be spotted that neural network-based techniques and optimization functions could be used more often in ad positioning.
augmented reality, which has gained huge popularity in the past few years. Tools such as Vuforia, ARKit, etc., are used to develop such functionalities [56]. Table A6 in Appendix A below presents all of the related works (including the approach/techniques which are used and their purpose), and the diagram in Figure 8 displays all of these grouped into the above four categories. From the findings, it can be spotted that neural network-based techniques and optimization functions could be used more often in ad positioning.

General Discussion, Challenges and Future Trends
After conducting this extensive research and thoroughly studying the existing solutions, the following general points/challenges can be highlighted:


Existing approaches in the literature that focus on implementing personalized advertising systems are completely user-centered and do not take advantage of existing business/advertiser data to infer the useful information about advertisers.
Usually, businesses either (a) analyze their data using their models and deploy their marketing strategy by providing explicit specifications about their target group to an external personalized advertisement system; (b) send their ads to an advertisement network and rely on its personalization system; or (c) use a combination of these two approaches to improve the results [57]. In the first case, the usually omit factors that affect the result, and their specifications are relatively simple (e.g., demographics, location, time, etc.). Regarding the second case, the personalization system lacks some information that possibly affects the result because these systems only focus on analyzing user data. Apart from using only users' contextual and historical data in the personalization models, an extra layer can be added. This layer can contain contextual and historical data belonging to the advertiser to improve the results. To be clearer, one can imagine a scenario where a coffee

General Discussion, Challenges and Future Trends
After conducting this extensive research and thoroughly studying the existing solutions, the following general points/challenges can be highlighted:

•
Existing approaches in the literature that focus on implementing personalized advertising systems are completely user-centered and do not take advantage of existing business/advertiser data to infer the useful information about advertisers.
Usually, businesses either (a) analyze their data using their models and deploy their marketing strategy by providing explicit specifications about their target group to an external personalized advertisement system; (b) send their ads to an advertisement network and rely on its personalization system; or (c) use a combination of these two approaches to improve the results [57]. In the first case, the usually omit factors that affect the result, and their specifications are relatively simple (e.g., demographics, location, time, etc.). Regarding the second case, the personalization system lacks some information that possibly affects the result because these systems only focus on analyzing user data. Apart from using only users' contextual and historical data in the personalization models, an extra layer can be added. This layer can contain contextual and historical data belonging to the advertiser to improve the results. To be clearer, one can imagine a scenario where a coffee shop would like to place an advertisement into a personalized advertisement system/network. The system can obtain this coffee shop's check-ins or reviews to infer useful information about this business/potential advertiser (e.g., its popularity depending on the time) and can take this information into account using the personalization model along with other factors such as user historical data, context, etc. By combining both layers, the results can be improved significantly. This approach can also be very helpful in cold start cases where limited data or even no data are available for the user [58]. Relative APIs (e.g., Google API) exist so that personalized advertisement services can extract useful information. It is worth mentioning that no privacy issues exist, as no personal user information is revealed (only an id is visible). Additionally, semantic web technologies can be used for this purpose to provide a general representation standard and to bypass data heterogeneity limitations.

•
Exploit Semantic Web technologies to a great extent regarding all of the related aspects.
To the best of our knowledge, the semantic web technologies that are mentioned above are mostly used (a) in approaches that use keyword matching to enhance the search results and the matching of ads or (b) to enhance contextual knowledge and data representation. This domain has a lot more potential. First, representing advertisements with a formal representation standard could make them machine independent and interoperable. For example, businesses could upload their ads in a formal way on their website (e.g., use JSON, RuleML format, etc.), and various personalized advertisement networks could retrieve them [59]. Additionally, the same approach can be used to upload their data or even their preferences. Advertisement networks can retrieve and use business data (e.g., check-ins, target group, etc.) in their personalization approaches to improve the results.

•
Cold start (or even no data at all).
A very popular problem where the output of the system depends on the history of inputs is the cold start. For example, when a new user or a new item is inserted into the system [60,61]. Because of the limited historical data, systems and their personalization models have trouble providing accurate information [62]. Many researchers in the recommender systems domain focus on developing personalized models specifically for these purposes. Surprisingly, almost none of the studies focusing on existing approaches to implement personalization models for advertisements focus on this problem. Cold start is very important in the advertisement domain because user data will often not be available (e.g., user does not give consent, user has limited interaction/historical data, a new advertisement is added to the network, etc.). Existing state-of-the-art techniques from the cold start recommender system domain should be tested for advertisements, or new approaches that are specialized in the cold start advertisement recommendation domain should be implemented [63].

•
Exploit the capabilities of relatively new technologies such as big data, IoT, and augmented reality (AR).
After thoroughly studying the literature, to the best of our knowledge, these technologies are still immature in the advertisement and marketing domains; thus, they have a lot of potential. Even more could be achieved. They offer advertisers the opportunity to conceive the user context in more detail, to create interactive experiences for their potential customers, and build better relationships with them.
Beginning with the big data and IoT domains, relatively few studies focus on providing personalized advertising models based on these technologies [26,54]. New technologies have emerged and provide a lot of potential. For example, apart from the IoT platforms that are mentioned above (Node-RED, Google Cloud, etc.), new messaging protocols such MQTT, hardware technologies (e.g., RFID), big data platforms (e.g., Spark, Hadoop), and many others provide a lot of capabilities [64]. Collecting data not only from PCs or mobile phones but also from various home appliances such as TVs, air conditioners, kitchens, etc., will lead to a new digital innovation in the next few years [21,65]. Future personalized advertisement models should be capable of collecting and analyzing these data and of providing more efficient results/advertisements to users (e.g., they should also collect data on the physical purchases that users make in order to display relevant ads).
Augmented reality is a new and exciting technology that can be used to provide better user experiences and to build superior customer-advertiser relationships as well. By using the AR tools that are referred to in the previous section, advertisements can act as "meaningful" media. More specifically, AR creates meaningful brand experiences by addressing the user's mind, emotions, and physical experience [56].

•
Most of the personalization models do not include the factors that have been highlighted in the literature as important in consumer decision making (e.g., user centric approaches, user involvement, self-esteem, gamification, etc.).
One important category in the proposed systematic review is surveys that try to explore factors that affect consumer decision making and that use questionnaires, experiments, or even cognitive neuroscience techniques in the field of decision making. These findings have great research potential, especially when they are combined with other domains. First of all, some psychological factors that have been highlighted by researchers can be utilized to enhance existing personalization models. Inferring the psychological characteristics of users and inserting them as input variables into a personalization model could significantly improve the results. New technologies such as IoT and semantic web can help in this direction, as discussed above, by providing an overall "extended" context perception. Furthermore, AR can be used to cover factors regarding user experience and involvement as discussed in the above point [66]. Last but not least, new business models have been adopted over the past few years in this direction. For example, business models that reward users for watching ads (e.g., get some credits/points relative to the app used) [67]. More research should be conducted in this direction, regarding the business perspective as well.
The main limitation here is the necessity of having knowledge in various domains such as psychology, business, data mining, software engineering, and neuroscience.

•
Lack of rich public datasets.
In the previous years, researchers pinpointed the lack of datasets in the domain. A lot of works did not provide their datasets to the public [68]. Others suffer from a small set of test subjects, a short time frame, diversity in locations, etc. Although t some qualitative datasets have been released (they are referred above) in the past few years, to the best of our knowledge, there is still a need for richer datasets that contain many variables, especially those that are context related. Often, personalized advertisement systems have trouble evaluating their approaches (due to lack of relative datasets or adjust their input variables/factors accordingly), omitting useful information.

•
Apart from finding the most suitable advertisements, there are other domains that need more in-depth study, such as the frequency of the advertisement, advertisement message, and image customization, providing detailed explanations to the user about the advertisements, etc.
To begin with, explaining recommendations to the user (why an item is recommended) is a popular domain of recommender systems [69]. From our study, we noticed that the related literature lacks approaches that focus on explaining the advertisement to the user in a more efficient way [70]. No explanations or explanations that are too simple are provided. Explaining ad messages is important because it helps to provide a better user experience, builds better relationships with the user/potential customer, creates a better attitude towards advertisements, etc. Applying machine learning and knowledge-based techniques in advertising personalization systems can help.
Furthermore, another area that needs to be studied more extensively is advertisement customization depending on user context (e.g., different image, different text message, etc.). Depending on the unique characteristics of the advertised product and the user context, (a) the advertising message may require different key attributes to attract the attention of the user and be more efficient, and (b) the advertisement image could be selected relative to the user. Although some works were spotted and included in this study, the vast majority of the studies focused on understanding the general attitude of users/customers toward mobile advertising and did not address the issue of spotting important advertising attributes for products and services. Techniques such as sentiment analysis and NLP, image analysis, and others can be used for these purposes [71]. For example, advertisements can provide the products or services that they would like to advertise, and the message could be generated automatically relative to the user.
Regarding optimal advertisement positioning, we noted that the literature lacks methods that are based on optimization functions. Moreover, deep learning could be used more for this purpose. Such methods seem to fit well for this purpose, and it would be interesting to test them.

•
Few knowledge-based approaches regarding personalized advertisement systems exist in the literature.
After studying the literature, it can be noticed that in personalized advertisement systems, knowledge-based approaches are a minority. Although knowledge-based approaches have some drawbacks, these technologies can be combined with the other approaches to enhance the results [72]. Furthermore, we can combine rule-based and semantic web technologies (SWRL, RuLeML, Sesame, Jena etc.) with innovative interfaces to help advertisers deploy their strategy or to help users filter their preferences [46]. Semantic web provides many more capabilities apart from improving search results. Since rules are declarative, advertisers or users/potential customers can easily conceive, create, and edit them. Consequently, they can customize the service(s) according to their needs. Moreover, combining data from various sources (e.g., user profiles, social networks, points of interest, etc.), rules, and ontologies provide advertisers with great flexibility and advanced expressiveness. Therefore, advertisers and users can easily define their preferences, cover a lot of possible situations, and enjoy better quality personalized information according to their requirements.

•
The line between privacy and advertising effectiveness.
One important issue that has been highlighted in the literature and that has to be discussed is the line between privacy and advertising effectiveness. Cross-domain knowledge and collaboration are needed for this purpose. Furthermore, personalized advertisement systems should always consider privacy issues and, consequently, privacy-oriented frameworks and architectures should be investigated further. Moreover, some of the techniques that are presented in the manuscript and are used in other areas (knowledge-based, classification, NNs, etc.) should be tested for this purpose (e.g., image recognition, detect privacy violations, etc.).

A Proposed Design Framework for Personalized Advertisement Systems
Taking all of these into consideration, a design framework for personalized advertising systems has been designed and illustrated in Figure 9. The purpose of the framework is to incorporate the findings of this extensive review and to help developers design personalized advertising systems. It discusses the steps that are necessary for providing personalized advertisements in detail along with the techniques and technologies that could be used in each step. The framework consists of four layers that are described in detail below: Information 2021, 12, x FOR PEER REVIEW 15 of 38 Figure 9. The design framework.

Data collection layer
The first step is to collect all of the necessary information regarding users and advertisers. The proposed framework provides two extra additions to existing approaches to Figure 9. The design framework.

• Data collection layer
The first step is to collect all of the necessary information regarding users and advertisers. The proposed framework provides two extra additions to existing approaches to enhance this process. First of all, it suggests that personalized advertisement systems should not only collect all of the related context-related attributes of the user (profile, history, social, time, location, connected devices, etc.) but also combine them with psychological factors that can positively affect the attitude towards ad acceptance. These factors can significantly enhance the results. This is defined in this work as "extended user context". The second addition regarding existing approaches is the utilization of advertiser data to a great extent. More specifically, apart from collecting only advertisement-related data (e.g., text, image, etc.) and advertiser specifications for the target group, it is recommended to utilize the advertisers' related information, such as check-ins, reviews, and every piece of available information, if possible. Various sources exist and can be used for these purposes (e.g., Google Places API, DBpedia, etc.). In this direction, IoT platforms and the Semantic Web can provide a huge boost. IoT platforms can provide a better context perception by collecting data from various sources. Additionally, the semantic web should play a more important role in this layer. Its vision is based on data exchange by providing a formal representation standard. Information can be easily read and used by other services; thus, the data collection process could be boosted. Platforms such as Node-RED, Google Cloud, etc., can be combined with semantic web technologies and tools (RDF, OWL, Sesame, Protégé, etc.) to implement this step based on the needs of the service [73]. As mentioned previously, many sources in the web that can be used to obtain useful data exist (e.g., Social Media data, Google, etc.).

• Data management layer
This layer includes data representation, storage, and management. The use of semantic web, big data, and IoT technologies is recommended used for this purpose. As previously discussed in detail, it is important to exploit the full potential of these technologies. Semantic web technologies and tools (e.g., Sesame, Jena, Virtuoso) are suitable for data representation and knowledge management (e.g., complex queries, relationships, reasoning, etc.). Additionally, IoT, big data, and other cloud platforms can be used for data management, storage, and processing (e.g., Spark, Hadoop, Firebase, etc.) [64]. Such technologies help to manage huge amounts of data more efficiently.

• Advertisement selection layer
The advertisement selection layer includes data analysis and computational techniques that are used to select the most suitable ads depending on the user. State-of-the-art technologies such as deep learning models, graph-based models, and other approaches should be used for these purposes. As it was discussed above, deep learning techniques seem to be the most promising for such purposes. Additionally, cold start recommender models should be incorporated for situations when there is a relatively new user or even a new advertisement, which is a very common phenomenon (e.g., many users do not give their consent about personal data, new advertisements being added, etc.). For the implementation of these models, the Python programming language and Django framework that are referred to in the previous section are convenient and can be used in most systems because they have many open-source libraries for that can be incorporated or modified (e.g., DeepCTR (https://deepctr-doc.readthedocs.io/en/latest/, 10 November 2021) library provides implementations for the most state-of-the-art deep NN algorithms).

• Visualization layer
The visualization layer includes the presentation of the advertisement to the end user. Common web or mobile development technologies and tools that are based on the developers' expertise can used in this step. For example, programming languages such as HTML, JS, CSS, Java, etc., and tools such as the angular framework, React JS, Android Studio, etc., can be used, and these tools help developers use these programming languages to design applications more efficiently [46]. This layer includes not only the way that the advertisement is delivered and its position but also image and message customization. Regarding the presentation of the advertisement, AR tools (e.g., ARKit, ARCore etc.) and other related techniques can be used to enhance user entertainment, participation, etc. [66]. Such factors create a positive attitude towards advertising acceptance. In terms of advertisement positioning, it is suggested that optimization functions and deep neural networking be tested to a greater extent. Additionally, advertisement messages, image customization, and explanation to the end user should always be taken into consideration. NLP techniques, deep learning networks, and other state-of-the-art techniques should be used for these purposes.
As previously mentioned, the proposed design framework provides some guidelines and some points that can be considered by developers when they design such systems. However, system implementation and real-world experiments should be conducted to prove its viability and to verify it. Some possible matters/limitations should be taken into account before starting the implementation phase. First of all, because it combines many different technologies, a system's size and implementation time should be examined when following the framework. Moreover, implementation requires a large team of developers with a broad range of experience. Last but not least, possible performance issues regarding speed, agility, etc., should be investigated.

Conclusions and Future Directions
This manuscript systematically analyzes and evaluates existing literature in the field of personalized advertising techniques. Apart from this, it pinpoints some interesting findings and proposes a design framework (based on these findings) that could be adopted when designing personalized advertisement systems to achieve highly targeted marketing.
All of the research questions that have been posed were answered. Specifically, Section 3 answers the first research question. A total of 301 works and the state-of-the-art approaches are presented in a condensed, succinct, and evaluative way to support researchers in orienting further research in the domain. Additionally, Section 4 answers the second research question by spotting and discussing some interesting issues/findings for future research, most of which are being highlighted for the first time (business data utilization in personalized advertisement models, the cold start problem, advertisement visualization issues, inclusion of psychological factors in the models, the lack of rich datasets, privacy). Finally, Section 5 answers the third research question and builds bridges between advertising and other related emerging domains (AI, semantic web, IoT, augmented reality, etc.) by presenting a design framework that incorporates all of these findings.
This systematic review opens future paths for more research in this domain. To begin with, the works and the techniques that are presented this manuscript provide a good background for researchers to find useful information or even can inspire them for future work. Moreover, some key points that have been spotted in this work (such as using advertisement provider data and user personality traits as inputs in the models) could lead to better personalized advertisement systems. To the best of our knowledge, it would be an interesting topic for future research to examine the performance of state-of-the-art models in such datasets (if available). Furthermore, spotting the state-of-the-art models and recent trends will help developers and researchers to improve existing solutions even further. For example, literature review analysis showed that deep NN models have dominated graph-based and statistical approaches over the past few years, and this domain has gained the focus of the researchers who are working on improving these models. Finally, combining these models with other domains (e.g., big data, IoT, AR, semantic, etc.) could help researchers and developers to upgrade these systems [74]. For example, richer datasets (exploiting IoT and big data technologies) and better user experience using AR technologies could lead to improved systems.
The proposed design framework tries to incorporate all of the findings of this research. In the future, we plan to design and the implement a personalized advertisement system that is based on this framework. After that, experiments using real users that verify and validate the system will be made to show its worthiness and to investigate it for possible limitations. Furthermore, an issue that is spotted and discussed in detail is the cold start. Users do not usually provide information for privacy reasons; this, it is difficult to send them relevant advertisements. Special focus will be given to this domain, and we plan to design a methodology and to develop a system that focuses specifically on cold start advertisement recommendations in the future.

Acknowledgments:
We would like to thank Greece and the European Union (European Social Fund-ESF) that made it possible for the research to be conducted, and we also thank all the reviewers and editors that have contributed their insights to this work.