Trend-Based Categories Recommendations and Age-Gender Prediction for Pinterest and Twitter Users

: Category suggestions or recommendations for customers or users have become an essential feature for commerce or leisure websites. This is a growing topic that follows users’ activity in social networks generating a huge quantity of information about their interests, contacts, among many others. These data are usually collected to analyze people’s behavior, trends, and integrate a complete user proﬁle. In this sense, we analyze a dataset collected from Pinterest to predict the gender and age by processing input images using a Convolutional Neural Network. Our method is based on the meaning of the image rather than the visual content. Additionally, we propose a heuristic-based approach for text analysis to predict users’ age and gender from Twitter. Both of the classiﬁers are based on text and images and they are compared with various similar approaches in the state of the art. Suggested categories are based on association rules conformed by the activity of thousands of users in order to estimate trends. Computer simulations showed that our approach can recommend interesting categories for a user analyzing his current interest and comparing this interest with similar users’ proﬁles or trends and, therefore, achieve an improved user proﬁle. The proposed method is capable of predicting the user’s age with high accuracy, and at the same time, it is able to predict gender and category information from the user. The certainty that one or more suggested categories be interesting to people is higher for those users with a large number of publications.


Introduction
Multiple studies have been done in order to explore how to predict age, gender, authorship, region, and other characteristics of the author of a text. In years, along with the rise of social networks, a lot of works have focused on authorship and attributes identification in social media posts, by analyzing text, images, and posting behavior. However, most work is focused on text analysis due to the amount of text generated per minute in social networks as Twitter [1]. At present, several statistics, people similarities, and trends are computed from the user activity in social networks, including the author profiling, which remains as an open multifactor challenge. In this context, the most prominent conference about author profiling (PAN at CLEF) has been studying, since 2013, these topics; however, it was until 2018 when images were included as part of the training sets to explore the age and gender identification using images and text information.
The use of images for author profiling provides additional information and, therefore, more data diversity to create a model. For instance, the authors in [2,3] explored, in 2014, the possibility of predicting users' gender using Scale-Invariant Feature Transform or SIFT visual descriptor on images posted by the user, from Twitter and Pinterest, respectively. The authors in [2] classified such posted images into categories before predicting gender and, in [3], they directly computes gender prediction using a Logistic Regression Classifier. Due to several image descriptors spent high computational time and show more complexity, Convolutional Neural Networks (CNN) have influenced most of the methods for gender and age prediction because of their high performance results. For instance, more recently, during PAN 2018 [4], multiple techniques were explored to predict user's gender on twitter posts including images, the best result was reported by Takahashi et al. [4] who used the CNN model VGG16 to extract the features from images and then combining them with textual features. Additionally, in 2018 Álvarez Carmona [5] used a similar method extracting features with a VGG16 for classifying age and gender with a Support Vector Machine (SVM) on Twitter images. Later, Bravo-Marmolejo [6] made a comparison between different classifier for age and gender using features that were extracted from a ResNet-50 on Pinterest images.
This work focuses on the analysis of users' publications in social networks, specifically Pinterest and Twitter, with the aim of establishing a representative user model based on age and gender. The main idea behind our work is to recommend trend categories to Pinterest users by analyzing the images they have pinned and, with this information, enrich the typical user profile consisting of gender and age user prediction. To do this, we propose two separate classifiers: (1) for predicting age and gender and (2) for classifying images pinned by users on Pinterest. Moreover, for the age and gender classifier, two methods are implemented, one of them is text-based and the other is image-based. Therefore, we analyze distinct and reliable features in tweets or in images published on Twitter or Pinterest respectively, in order to predict age and gender. We evaluate two different types of datasets (from Twitter and Pinterest) to compare text and image features performance separately for predicting an initial profile for each user in the dataset (age and gender). The second classifier predicts the category for any pinned image on Pinterest along with the other 32 predefined categories. The proposed image classifier helps to describe the interest of users and, at the same time, create a set of user preferences. Finally, additional categories can be recommended to the user according to similar image pins published and the association rules based on the activity of thousands of users. Thus, the Pinterest categories predicted for a user are combined and evaluated following proposed association rules.
Based on the results that were obtained using VGG16 as feature extractor, we performed some test to find out if better results are obtained by using a specific form of machine learning technique called Transfer Learning. This technique is used to avoid training deep networks from scratch, instead, using a pre-trained network on a similar and bigger dataset, and then retraining specific layers for the new output. It is worth noting that Bravo-Marmolejo [6] and Álvarez Carmona [5] used a type of transfer learning, however they did not retrained the network. Thus, we re-trained some of the layers to improve classification performance. This re-training was performed because the extracted features should be more specialized for age and gender recognition than for general image classification. A collected dataset from Pinterest (labeled as DB1) is used to perform experimental tests for age and gender prediction.
Concerning text analysis, we want to explore a different approach from those techniques that are only focused on the processing of plain text. This traditional techniques try to analyze the semantics and relationship between words [4,7] using techniques like word2vec, bag of words or TF-IDF [8].
In this work, we assume that there is some information from the user that can be extracted (i.e., age and gender) by analyzing the hashtags, links, and tags from a tweet, and also in how the users use punctuation signs. We use the database from PAN 2015 based on Twitter users (labeled as DBT) to perform the experimental tests by text analysis.
Moreover, we want to find similarities between users by analyzing their preferred categories on Pinterest, and then built a module capable of recommending new categories to a user based on the general Pinterest users preferences. A different dataset of Pinterest is used, because we require a huge number of images from different users and categories to set up the association rules. These rules are generated by the Apriori algorithm applied to recommend additional categories to Pinterest users. The dataset used to generate association rules is labeled as DB2. This document is structured, as follows: Section 2 describes our global methodology used to predict age-gender, category, and recommendations for social network users. Section 3 describes the obtained results for different datasets. Finally, Section 4 shows the conclusions and further perspectives for user profile.

Method Description
The proposed category recommendation method for Pinterest users consists in predict an initial profile based on the age and gender of the user using tweets or images datasets. Most of the existing methods combine both datasets to improve gender and age classification results while using the same social network. As we know, Twitter allows users to publish mainly text while Pinterest allows mostly images. However, in addition to text messages, Twitter users can also publish images. Unlike texts, images are posted with less frequency than text and, therefore, images in Twitter could be less representative than text for predicting the user profile. In this context, our first goal is to predict age and gender by analyzing user text and images separately, as such data come from social networks with different approaches. Figure 1 shows the proposed methodology for age and gender prediction using images. Our method requires a set of images and it is based on transfer learning. Thus, given the DB1 database of posted images in Pinterest, the first step is to resize the images. Next, the full image database is split to create the training set and the testing set. The images in the training set go through a VGG16 Convolutional Neural Network (CNN) and, finally, to age and gender classifiers by means of a multi-output layer. With this technique, some layers of the network are trained after initialization.
Appl. Sci. 2020, xx, 5 3 of 12 huge number of images from different users and categories to set up the association rules. These rules are generated by the Apriori algorithm applied to recommend additional categories to Pinterest users. The dataset used to generate association rules is labeled as DB2. This document is structured, as follows: Section 2 describes our global methodology used to predict age-gender, category, and recommendations for social network users. Section 3 describes the obtained results for different datasets. Finally, Section 4 shows the conclusions and further perspectives for user profile.

Method Description
The proposed category recommendation method for Pinterest users consists in predict an initial profile based on the age and gender of the user using tweets or images datasets. Most of the existing methods combine both datasets to improve gender and age classification results while using the same social network. As we know, Twitter allows users to publish mainly text while Pinterest allows mostly images. However, in addition to text messages, Twitter users can also publish images. Unlike texts, images are posted with less frequency than text and, therefore, images in Twitter could be less representative than text for predicting the user profile. In this context, our first goal is to predict age and gender by analyzing user text and images separately, as such data come from social networks with different approaches. Figure 1 shows the proposed methodology for age and gender prediction using images. Our method requires a set of images and it is based on transfer learning. Thus, given the DB1 database of posted images in Pinterest, the first step is to resize the images. Next, the full image database is split to create the training set and the testing set. The images in the training set go through a VGG16 Convolutional Neural Network (CNN) and, finally, to age and gender classifiers by means of a multi-output layer. With this technique, some layers of the network are trained after initialization. The age and gender classifier that is based on tweets analysis will be described in Section 2.5. After predicting the age and gender of the user, a second classifier to predict the category of a pinned The age and gender classifier that is based on tweets analysis will be described in Section 2.5. After predicting the age and gender of the user, a second classifier to predict the category of a pinned image in Pinterest is implemented. This classifier is trained using the same method that is described in Figure 1, however instead of using a multi-output layer this model uses a single-output layer with 32 outputs, that is, one output for each category in the dataset. Once the image category is obtained, we evaluate a set of rules generated by the Apriori Algorithm using a dataset with a large number of proactive users in the social network. Ideally, Apriori rules are used to extract only the most frequent item-sets to find their behavior and, in most cases, extract important information that can be used to improve the accuracy of the classifier. In our scenario, we used Apriori rules to know the behavior for all the items in our dataset. The validation to recommend additional categories to users is performed using two proposed methods A and B according to their pinned images. This validation includes the analysis of a dataset with information from more than 400,000 users. This approach is explained in detail in Section 3.
The method that is proposed in this work extracts additional information from social networks when compared with classic methods typically used in the state of the art. Therefore, our method is capable of predicting the user's age and gender with high accuracy and, at the same time, it is able to predict category information from the user without violating privacy issues.

Description of Pinterest Database
For our first experimental tests of age and gender classification, we use the image dataset that was collected by López-Santamaría et al. [8]. This dataset includes 548,761 pins from 256 users extracted directly from the Pinterest website. In this case, the ages of the users are similar to those of PAN 2015: 18-24, 25-34, 35-49, and 50+. Pinterest social network provides 32 predefined categories in which the user can classify their posts. Table 1 shows the list of categories found in Pinterest and the number of images we have in our database per category. Therefore, uncategorized pins were removed, and the final dataset DB1 contains 307,225 pins from 241 users.

VGG16
According to Alvarez-Carmona [5] and Takahashi et al. in [4], the CNN with 16 layers, VGG, performs well for age or gender prediction on social network images. Hence, we use this architecture for our work. This network was originally proposed by Simonyan & Zisserman [9] in 2015 with an architecture consisting of 13 convolutional layers divided in groups of 2 and 3, where each group is separated by a pooling layer and each convolutional layer uses multiple 3 × 3 kernels to perform the convolution. Small modifications were made to the original architecture that has two fully connected layers and an output layer for 1000 classes. In this study, the output layer is modified to include a multi-output layer: one for the gender and the other one for the age, totally independent of each other. The two fully connected layers are independent from each other but connected to the same network. This configuration allows training age and gender predictor using the same network but with independent outputs, as illustrated in Figure 1. We use Transfer learning because the training of this network architecture implies high computational time and resources. That is, a pre-trained network is utilized to obtain similar results fast.

Transfer Learning
The use of the weights of a pre-trained network to initialize a new CNN is a technique that has reported good results for age-gender prediction [4,5]. In this work, we load the weights of a VGG16 trained on the image dataset Imagenet (http://image-net.org/about-overview) and replace the last layer with our own multi-output layer (as mentioned in Section 2.3). These new outputs are independent of each other, as shown in Figure 1. The weights of the 13 convolutional layers remain fixed while the fully connected layers are trained. The original VGG16 network has an input size of 224 × 224, thus all our input images were resized.

Gender and Age Prediction by Text
This section considers an heuristic text-based approach to propose a second strategy for gender and age prediction without using CNN. The second set of tests includes the analysis of text collected from Twitter. For our text approach, we used the database DBT coming from PAN 2015 [10] that gathers 152 users (in the english dataset) with 100 tweets each and contains labels for age and gender for each user.
According to [11,12] the use of symbols, linguistic structures and styles in blogs or social networks is directly related to the users' age. Recently, new features, like links, tags, and hashtags, appearing in social networks have been introduced to address the users age prediction problem in social networks [13,14]. Moreover, these textual features are also analyzed to predict the gender of users, based on correlations between words and phrases shown in texts [15]. The analysis of different styles, punctuation, or symbols used in tweets replaces the classical method to analyze the whole text posted. In our method, we count the number of links, hashtags, tags, words, and other features instead of just directly processing the text. Table 2 describes the features used in this work. Observe, that we assume that the frequency of appearance of the feature in Table 2 is directly related to the users' age and gender. Because we want to know the number of words in the user vocabulary, we merge all of the tweets per user and obtain a polynomial of the frequency of appearance of every word. In this work, the feature vector length is 17 per user (most of the features have a length, with the exception of the polynomial vocabulary). After the feature vectors have been built, the feature classification is performed by a Bag of Trees classifier.

User Profile and Category Suggestion Approach
Researchers have proven that it is possible to cluster users according to their behavior in social networks. The authors in [5,16] showed that users from the same gender group tend to show a preference for similar types of image categories. Furthermore, providing a recommendation that is based on one or several choices is a strategy largely used by browsers or e-commerce pages to assist user navigation. Following the same idea, we should be able to identify the relationship between different pinned categories and determine whether or not a new category could be interesting for a user, based on previous categories pinned.
A great amount of information is required to extract such rules, for that reason we used a different database of Pinterest users. The DB2 database was taken from [17] that was collected by repeatedly visiting the pages of the 32 Pinterest categories (similar categories reported in Table 1) and retrieving the most popular pins. For our purposes, we used the images, its category label and the list of users with the number of pins published in each category. This database consists of 176,092 images, belonging to 401,241 users, it is important to note that one image can be repined by multiple users.
Several algorithms have been studied and proposed for association rule mining for both small and big datasets, but, because our amount of available categories is limited, we used the Apriori algorithm [18] to get those rules. The purpose of this algorithm is to find which items are related to each other. In other words, this algorithm provides rules to indicate which categories can appear once another set of categories is present. Each rule consists of an item-set with an appearance percentage, called support. The item-set is divided in base and add, where the add is a subset of items which probability of appearance can be influenced by the presence of the base. After running the Apriori algorithm, around 10,000 rules were found and sorted according to the support they have.
Given the category of an image from a user, we used two methods to recommend additional categories that are typically pined by similar users. In the first method, method A, we inspect the list of rules looking for the item-set with the highest support, which includes the category published by the user. Once we identify the item-set, any other element of the set can be recommended to the user as a new category. In the second method (called method B), we use the confidence value, defined as the probability of appearance of the add once the base has appeared. To do this, the category of the image is searched on the base of the rules and the rule with the highest confidence is selected, any category on the add set can be recommended to the user. Table 3 shows some examples of real rules mined by the algorithm using DB2 dataset. Suppose that we have an image from a user with category 'Weddings', applying both methods A and B we obtain the rules that are shown in Table 3. With method A, we look the rule with highest support that includes 'Weddings', for this example the best result is {Home&Decor, Weddings, Hair&Beauty} with a support value of 0.05, then we randomly recommend 'Home&Decor' or 'Hair&Beauty'. Note that, for this method we do not take into account base or confidence parameters provided for each rule. With method B we look in the base of the item-set, then we choose the rule with highest confidence; therefore, the recommendation is the category given by add set, in this case 'History'.
Notice that the Apriori algorithm states that the entirety of the base must be present to ensure the confidence of the appearance of the add. However, in our scenario with Pinterest in many cases is nearly impossible to find a rule with exactly the same base and relevant confidence. For such reason, we only look for one or two categories present in the base. The percentages of successful recommendation are described in the next section.
In some special cases, some users do not have a category for all the images and boards in Pinterest. Therefore, an image classifier is needed to handle these special cases. Such classifier should be able to distinguish images from each one of the 32 Pinterest categories. As we mentioned above, this classifier is trained using the same method described in Figure 1, except that instead of using a multi-output layer we use a single output layer with 32 outputs.

Results for Age and Gender Prediction Using Images from Pinterest
In the first part of the experimental tests, we used the DB1 Pinterest dataset described in Section 2.2 for age and gender prediction. It is well known that balanced datasets can improve classification. In our case, images from men users were only 1/5 of the original dataset (307,225 labeled images). For this reason, only 60,000 pins were used and, consequently, the number of users was reduced to 126 distributed in 61 men and 65 women. In order to have ∼30,000 images in each gender and ∼15,000 in each age range, we took men users' images from each age range as possible until we reached 30,000 images. In our dataset, men images were unbalanced for age, as is shown in Table 4, so we filled the ages categories with images from women users trying to reach 60,000 images with ∼15,000 on each age range. Notably, we do not have the same pins from men and women per age range; however, we look for a balance on pins number per category while maintaining the same number of pins provided by men and women users.
As before, all of the images were resized to 224 × 224 pixels and the dataset was split into 85% for training and 15% for testing. The training and validation of the last fully connected layers of the network were performed 10 times and, for each training and validation, the images in the dataset were randomly shuffled. For this test, we obtain a mean accuracy of 72% for gender and 49% for age using DB1 dataset. Further metrics of the experimental tests and a comparison with the related works are shown in Table 5. The best accuracy (81%) is obtained by Takahashi et al. [19], however, they use a different dataset (images from Twitter) and only provide gender prediction. For age prediction, our method achieves the highest score 49%, which is 8% higher than [6] that uses Pinterest dataset and 10% higher than [5] for a Twitter dataset. Furthermore, we include the F1 metric to compare our approach with those methods that provide it, although this metric is more representative than accuracy for unbalanced datasets. We achieve the highest score for F1 giving a reliable percentage of recognition per class.

Results for Gender and Age Prediction Using Text from Twitter
In this section, we discuss the results of the methodology that is described in Section 2.5 for text analysis with the PAN 2015 database DBT. The experimental tests were performed 10 times and we obtained, on average, 64% of accuracy for gender prediction and 67% for age. Additional performance metrics for the results are 64% and 65% of precision and recall, respectively, for gender and 58% and 65% of precision and recall, respectively, for age. In general, the age classification is unbalanced and produces values for the precision and the F1 norm that are lower than those values for accuracy and recall. As we mentioned above, social networks like Twitter or Pinterest are essentially used for different purposes. In addition to pinned images, Pinterest users can also publish short text messages added to their images. Contrary to Twitter, text are posted with less frequency than images and, therefore, text in Pinterest could be less representative than images for predicting the user profile. In this context, a comparison with the related works is shown in Table 6 using different datasets. It is important to point out that authors in [5] used the Twitter PAN 2014 with additional images. While the accuracy of our model (using text and images) is not better than Takahashi et al. [19] in the state of the art, the benefit of our model is that we also predict age of users using the same architecture of the CNN. The accuracy of our results and the F1 metric for age prediction (67%) outperform the approaches presented in Table 6. In addition, our strategy could be easily extended to combine text and image information to enrich the vector features in the CNN, which could improve the accuracy results for age and gender prediction. However, we expect to combine text and image published by the same user from different social networks to perform further tests.

Trends Analysis and Category Recommendations
Our results for category recommendations are divided in two parts. First, we show the results from the pin classification when the category of the image is missing. Second, we discuss the recommendation results when a category is known. An accuracy of 42% was obtained for the Pinterest database DB1 with 32 categories. Further tests were performed for DB2 database, from where we obtained the rules described in Section 3, obtaining a 51% accuracy. Different metrics computed for classification performance are illustrated on Table 7. Note that metrics for the dataset DB2 are slightly different, we assume that this behavior is due to the unbalanced category of the dataset DB1 for some categories. In the second part of experimental tests, we used the DB1 dataset to test our category recommendation methods. We identified the users who posted images in at least five different categories and at the most 25. The number of users with this specific number of categories was 132 from the 256 users. Given a specific category from a user, a successful recommendation is made when a new category (recommended using method A or B illustrated in Table 3) is found in the list of categories for that user. The overall performance is obtained by calculating the percentage of successful recommendations for all of the posted categories per user, then averaging this percentage for all the 132 users. The first test was performed using only one category as a seed to look into the rules. For this test, the overall performance was 29% of successful recommendations for method A, and 30% for method B.
The second test was performed using two categories taken as seed. Thus, for each user, all of the combinations of two categories were tested and validated with the rest of the posted categories of the user. In this test, we achieved an overall performance of 19% with method A and 17% with method B. However, most of the systems that provide recommendations give more than one option to the user. To explore this possibility, we tested recommending more than one category to the user. In this sense, we considered a successful recommendation when at least one of them was in the posted categories. The results for 1, 2, and 3 suggestions are summarized in Table 8. As we expect, expanding the number of categories in the seed delivers a low percentage of successful recommendation, because it is more complicated to find a particular pair of categories than to find one in the item-set for method A or in the Base for the method B. Moreover, providing two or three suggested categories improves the probability of one of given suggestions can be chosen by user with an initial seed. From these results, we confirm that more suggestions provide higher probabilities that one or more categories could be chosen by a Pinterest user once he has a certain item-set or base of categories.

Discussion and Conclusions
We propose a method to predict gender, age, and suggested category to construct an enriched profile of users of Twitter and Pinterest. To do this, we develop two strategies to predict the gender and age of users analyzing text and images separately. The innovation in our proposed text-based model is the semantic and personality aspects considered to construct the feature vector that achieves reliable score without using a CNN as a classifier. From the review of age and gender results, it is more convenient to analyze text than images based on our achieved score. Nevertheless, the image dataset acts as a bridge between the age-gender predictor and the categories that can be recommended to the user according to his images pinned. Thus, our image classifier utilizes the same dataset as the age and gender model and it is only applied to label uncategorized images in the user wall of Pinterest. Once all the images belong to one of the 32 categories of Pinterest, the association rules analyze categories posted by the user and suggest one or more categories to him. The experimental results show that a loss of accuracy is present in some rules with very low support or confidence. This, in turn, highlights the diversity of Pinterest' categories and indicates that the rule should not be popular.
Further modifications to this work consist in combining text and images published by the same user from different social networks. An alternative way to improve the percentages of categories suggestions can be obtained by increasing the current support and confidence values for the association rules. Future work also includes the integration of a complete user profile based on the users' activity on social networks and cluster similar users' profile by search algorithms. At the same time, we could associate groups of users with the same interest and construct population trends. Our particular attention is to create clusters of users with similar interests that are based on representative models and provide this information to National Statistic Institutions.