# Automatic 3D Modeling and Reconstruction of Cultural Heritage Sites from Twitter Images

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Related Work

#### 2.1. Event Detection on Twitter

#### 2.2. Content-Based Filtering

## 3. An Overview of the Proposed Approach

## 4. Image Collection as a Twitter Event Detection Problem

#### 4.1. Tweet-Post Characterization

#### 4.2. Twitter-Based Information Metrics

#### 4.3. Defining the Similarity Metric

#### 4.4. Multiassignment Graph Partitioning Approach to Event Detection in Twitter

## 5. Representative Image Selection as a Content-Based Filtering Problem

#### 5.1. Visual Modeling and Image Representation onto Multidimensional Manifolds

#### 5.2. Additional Content Refinement and Extraction of Representative Views

#### 5.3. Twitter Content Copyright Issues

## 6. Experimental Results

#### 6.1. Evaluation of Event Detection on Twitter

#### 6.1.1. Evaluation under a Controlled Environment

_{gt}. To generate the words within an event, we develop a word generator. This generator produces tweets that contain the particular word based on a probability that indicates the percentage of the tweets of a particular category (high, medium, low importance) that have posted the specific word. In our experiments, this probability follows a Gaussian distribution; the mean value corresponds to the probability of the event this word is assigned to (average number of tweet-post words of this event over the total number of tweets), while the standard deviation $\mathsf{\sigma}$ regulates the coherence degree that the respective event has in terms of word appearance. Small values for the standard deviation means that almost all the words within an event are synchronized in time, since their appearance probability is quite similar for every time interval. The opposite is held for high standard deviation values.

#### 6.1.2. Evaluation on Real-World Data

#### 6.1.3. Computational Complexity

#### 6.1.4. Reconstruction Efficiency

## 7. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Labadi, S. UNESCO, Cultural Heritage, and Outstanding Universal Value: Value-Based Analyses of the World Heritage and Intangible Cultural Heritage Conventions; Rowman & Littlefield: Lanham, MD, USA, 2013. [Google Scholar]
- Bowitz, E.; Ibenholt, K. Economic impacts of cultural heritage—Research and perspectives. J. Cult. Herit.
**2009**, 10, 1–8. [Google Scholar] [CrossRef] - Dimoulas, C.A.; Kalliris, G.M.; Chatzara, E.G.; Tsipas, N.K.; Papanikolaou, G.V. Audiovisual production, restoration-archiving and content management methods to preserve local tradition and folkloric heritage. J. Cult. Herit.
**2014**, 15, 234–241. [Google Scholar] [CrossRef] - Doulamis, A.; Voulodimos, A.; Doulamis, N.; Soile, S.; Lampropoulos, A. Transforming Intangible Folkloric Performing Arts into Tangible Choreographic Digital Objects: The Terpsichore Approach.; SCITEPRESS: Setubal, Portugal, 2017; Volume 6, pp. 451–460. [Google Scholar]
- Stentoumis, C.; Livanos, G.; Doulamis, A.; Protopapadakis, E.; Grammatikopoulos, L.; Zervakis, M. Precise 3D Reconstruction of Cultural Objects Using Combined Multi-component Image Matching and Active Contours Segmentation. In Advances in Visual Computing; Bebis, G., Boyle, R., Parvin, B., Koracin, D., Li, B., Porikli, F., Zordan, V., Klosowski, J., Coquillart, S., Luo, X., et al., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2013; pp. 148–157. ISBN 978-3-642-41938-6. [Google Scholar]
- Wahbeh, W.; Nebiker, S.; Fangi, G. Combining public domain and professional panoramic imagery for the accurate and dense 3D reconstruction of the destroyed Bel Temple in Palmyra. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci.
**2016**, 3, 81. [Google Scholar] [CrossRef] - Voulodimos, A.; Doulamis, N.; Fritsch, D.; Makantasis, K.; Doulamis, A.; Klein, M. Four-dimensional reconstruction of cultural heritage sites based on photogrammetry and clustering. J. Electron. Imaging
**2016**, 26, 011013. [Google Scholar] [CrossRef] - Doulamis, A.; Doulamis, N.; Protopapadakis, E.; Voulodimos, A.; Ioannides, M. 4D Modelling in Cultural Heritage. In Advances in Digital Cultural Heritage; Springer: Cham, Switzerland, 2018; pp. 174–196. [Google Scholar] [CrossRef]
- Doulamis, A.; Liarokapis, F.; Petridis, P.; Miaoulis, G. Serious Games for Cultural Applications. In Intelligent Computer Graphics 2011 (Studies in Computational Intelligence); Springer: Berlin/Heidelberg, Germany, 2012; pp. 97–115. ISBN 978-3-642-22906-0. [Google Scholar]
- Sun, A.; Hu, M. Query-Guided Event Detection from News and Blog Streams. IEEE Trans. Syst. Manand Cybern. Part A Syst. Hum.
**2011**, 41, 834–839. [Google Scholar] [CrossRef] - Petcu, D.; Vazquez-Poletti, J.L. European Research Activities in Cloud Computing; Cambridge Scholars Publishing: Newcastle upon Tyne, UK, 2012. [Google Scholar]
- Kazienko, P.; Chawla, N. Applications of Social Media and Social Network Analysis; Springer: Cham, Switzerland, 2015. [Google Scholar]
- Huang, T.S.; Netravali, A.N. Motion and structure from feature correspondences: A review. Proc. IEEE
**1994**, 82, 252–268. [Google Scholar] [CrossRef] - Makantasis, K.; Doulamis, A.; Doulamis, N.; Ioannides, M. In the wild image retrieval and clustering for 3D cultural heritage landmarks reconstruction. Multimed. Tools Appl.
**2016**, 75, 3593–3629. [Google Scholar] [CrossRef] - Aizawa, A. The Feature Quantity: An Information Theoretic Perspective of Tfidf-like Measures. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens Greece, 24–28 July 2000; ACM: New York, NY, USA, 2003; pp. 104–111. [Google Scholar]
- Xue, X.B.; Zhou, Z.H. Distributional Features for Text Categorization. IEEE Trans. Knowl. Data Eng.
**2009**, 21, 428–442. [Google Scholar] - Counting Characters. Available online: https://developer.twitter.com/en/docs/basics/counting-characters (accessed on 18 May 2020).
- Sankaranarayanan, J.; Samet, H.; Teitler, B.E.; Lieberman, M.D.; Sperling, J. TwitterStand: News in Tweets. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Seattle, DC, USA, 4–6 November 2009; ACM: New York, NY, USA, 2009; pp. 42–51. [Google Scholar]
- Grinev, M.; Grineva, M.; Boldakov, A.; Novak, L.; Syssoev, A.; Lizorkin, D. Sifting Micro-blogging Stream for Events of User Interest. In Proceedings of the 32Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Boston, MA, USA, 19–23 July 2009; ACM: New York, NY, USA, 2009; p. 837. [Google Scholar]
- Cha, M.; Haddadi, H.; Benevenuto, F.; Gummadi, P.K. Measuring user influence in twitter: The million follower fallacy. ICWSM
**2010**, 10, 30. [Google Scholar] - Duan, Y.; Jiang, L.; Qin, T.; Zhou, M.; Shum, H.-Y. An Empirical Study on Learning to Rank of Tweets. In Proceedings of the 23rd International Conference on Computational Linguistics, Beijing, China, 23–27 August 2010; Association for Computational Linguistics: Stroudsburg, PA, USA, 2010; pp. 295–303. [Google Scholar]
- Weng, J.; Lee, B.-S. Event detection in twitter. ICWSM
**2011**, 11, 401–408. [Google Scholar] - Aiello, L.M.; Petkos, G.; Martin, C.; Corney, D.; Papadopoulos, S.; Skraba, R.; Göker, A.; Kompatsiaris, I.; Jaimes, A. Sensing Trending Topics in Twitter. IEEE Trans. Multimed.
**2013**, 15, 1268–1282. [Google Scholar] [CrossRef] - Blei, D.M.; Ng, A.Y.; Jordan, M.I. Latent Dirichlet Allocation. J. Mach. Learn. Res.
**2003**, 3, 993–1022. [Google Scholar] - Bellaachia, A.; Al-Dhelaan, M. NE-Rank: A Novel Graph-Based Keyphrase Extraction in Twitter. In Proceedings of the 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, Macau, China, 4–7 December 2012; Volume 1, pp. 372–379. [Google Scholar]
- Bellaachia, A.; Al-Dhelaan, M. Learning from Twitter Hashtags: Leveraging Proximate Tags to Enhance Graph-Based Keyphrase Extraction. In Proceedings of the 2012 IEEE International Conference on Green Computing and Communications, Besancon, France, 20–23 November 2012; pp. 348–357. [Google Scholar]
- Kim, Y.; Shim, K. TWILITE: A recommendation system for Twitter using a probabilistic model based on latent Dirichlet allocation. Inf. Syst.
**2014**, 42, 59–77. [Google Scholar] [CrossRef] - Yang, M.-C.; Rim, H.-C. Identifying interesting Twitter contents using topical analysis. Expert Syst. Appl.
**2014**, 41, 4330–4336. [Google Scholar] [CrossRef] - Sizov, S. GeoFolk: Latent Spatial Semantics in Web 2.0 Social Media. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, New York, NY, USA, 3–6 February 2010; ACM: New York, NY, USA, 2010; pp. 281–290. [Google Scholar]
- Yin, Z.; Cao, L.; Han, J.; Zhai, C.; Huang, T. Geographical Topic Discovery and Comparison. In Proceedings of the 20th International Conference on World Wide Web, Hyderabad, India, 28 March–1 April 2011; ACM: New York, NY, USA, 2011; pp. 247–256. [Google Scholar]
- Gao, D.; Li, W.; Cai, X.; Zhang, R.; Ouyang, Y. Sequential Summarization: A Full View of Twitter Trending Topics. IEEE ACM Trans. Audio Speech Lang. Proc.
**2014**, 22, 293–302. [Google Scholar] [CrossRef] - Zhou, X.; Chen, L. Event detection over twitter social media streams. VLDB J.
**2014**, 23, 381–400. [Google Scholar] [CrossRef] - He, Q.; Chang, K.; Lim, E.-P. Analyzing Feature Trajectories for Event Detection. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, 23–27 July 2007; ACM: New York, NY, USA, 2007; pp. 207–214. [Google Scholar]
- Kaneko, T.; Yanai, K. Visual event mining from geo-tweet photos. In Proceedings of the 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), San Jose, CA, USA, 15–19 July 2013; pp. 1–6. [Google Scholar]
- Musleh, M. Spatio-temporal Visual Analysis for Event-specific Tweets. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, Snowbird, UT, USA, 22–27 June 2014; ACM: New York, NY, USA, 2014; pp. 1611–1612. [Google Scholar]
- Kaneko, T.; Yanai, K. Event photo mining from Twitter using keyword bursts and image clustering. Neurocomputing
**2016**, 172, 143–158. [Google Scholar] [CrossRef] - Cai, H.; Yang, Y.; Li, X.; Huang, Z. What Are Popular: Exploring Twitter Features for Event Detection, Tracking and Visualization. In Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia, 26–30 October 2015; ACM: New York, NY, USA, 2015; pp. 89–98. [Google Scholar]
- Kaleel, S.B.; Abhari, A. Cluster-discovery of Twitter messages for event detection and trending. J. Comput. Sci.
**2015**, 6, 47–57. [Google Scholar] [CrossRef] - Voulodimos, A.S.; Patrikakis, C.Z.; Karamolegkos, P.N.; Doulamis, A.D.; Sardis, E.S. Employing Clustering Algorithms to Create User Groups for Personalized Context Aware Services Provision. In Proceedings of the 2011 ACM Workshop on Social and Behavioural Networked Media Access, Scottsdale, AZ, USA, 1 December 2011; pp. 33–38. [Google Scholar] [CrossRef]
- Doulamis, A.; Katsaros, G. 3D modelling of cultural heritage objects from photos posted over the Twitter. In Proceedings of the 2016 IEEE International Conference on Imaging Systems and Techniques (IST), Chania, Greece, 4–6 October 2016; pp. 389–394. [Google Scholar]
- Chum, O.; Philbin, J.; Sivic, J.; Isard, M.; Zisserman, A. Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval. In Proceedings of the IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, 14–20 October 2007; IEEE Computer Society: Washington DC, USA, 2007; pp. 1–8. [Google Scholar]
- Kosmopoulos, D.I.; Doulamis, A.; Makris, A.; Doulamis, N.; Chatzis, S.; Middleton, S.E. Vision-based production of personalized video. Signal Process. Image Commun.
**2009**, 24, 158–176. [Google Scholar] [CrossRef][Green Version] - Halkos, D.; Doulamis, N.; Doulamis, A. A secure framework exploiting content guided and automated algorithms for real time video searching. Multimed. Tools Appl.
**2009**, 42, 343–375. [Google Scholar] [CrossRef] - Papadakis, N.; Doulamis, A.; Litke, A.; Doulamis, N.; Skoutas, D.; Varvarigou, T. MI-MERCURY: A mobile agent architecture for ubiquitous retrieval and delivery of multimedia information. Multimed. Tools Appl.
**2008**, 38, 147–184. [Google Scholar] [CrossRef] - Kekre, D.H.B.; Sarode, T.K.; Thepade, S.D.; Vaishali, V. Improved texture feature based image retrieval using Kekre’s fast codebook generation algorithm. In Thinkquest~2010; Pise, S.J., Ed.; Springer: New Delhi, India, 2011; pp. 143–149. ISBN 978-81-8489-988-7. [Google Scholar]
- Min, R.; Cheng, H.D. Effective image retrieval using dominant color descriptor and fuzzy support vector machine. Pattern Recognit.
**2009**, 42, 147–157. [Google Scholar] [CrossRef] - Angelov, P.; Sadeghi-Tehran, P. Look-a-Like: A Fast Content-Based Image Retrieval Approach Using a Hierarchically Nested Dynamically Evolving Image Clouds and Recursive Local Data Density. Int. J. Intell. Syst.
**2017**, 32, 82–103. [Google Scholar] [CrossRef][Green Version] - Srivastava, P.; Khare, A. Integration of wavelet transform, Local Binary Patterns and moments for content-based image retrieval. J. Vis. Commun. Image Represent.
**2017**, 42, 78–103. [Google Scholar] [CrossRef] - Doulamis, N.D.; Doulamis, A.D.; Kokkinos, P.; Varvarigos, E.M. Event Detection in Twitter Microblogging. IEEE Trans. Cybern.
**2016**, 46, 2810–2824. [Google Scholar] [CrossRef] [PubMed] - Ullman, S.; Brenner, S. The interpretation of structure from motion. Proc. R. Soc. London. Ser. B Biol. Sci.
**1979**, 203, 405–426. [Google Scholar] - Protopapadakis, E.; Doulamis, A. Semi-Supervised Image Meta-Filtering Using Relevance Feedback in Cultural Heritage Applications. Int. J. Herit. Digit. Era
**2014**, 3, 613–627. [Google Scholar] [CrossRef] - Protopapadakis, E.; Voulodimos, A.; Doulamis, A.; Camarinopoulos, S. A Study on the Use of Kinect Sensor in Traditional Folk Dances Recognition via Posture Analysis. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, Rhodes, Greece, 21–23 June 2017; ACM: New York, NY, USA, 2007; pp. 305–310. [Google Scholar]
- Protopapadakis, E.; Voulodimos, A.; Doulamis, A.; Camarinopoulos, S.; Doulamis, N.; Miaoulis, G. Dance Pose Identification from Motion Capture Data: A Comparison of Classifiers. Technologies
**2018**, 6, 31. [Google Scholar] [CrossRef][Green Version] - Bach, F.R.; Jordan, M.I. Learning spectral clustering. In Proceedings of the Advances in Neural Information Processing Systems, Whistler, BC, Canada, 13–16 December 2004; MIT Press: British Columbia, Canada, 2005; pp. 305–312. [Google Scholar]
- Fan, K. Maximum Properties and Inequalities for the Eigenvalues of Completely Continuous Operators. Proc. Natl. Acad. Sci. USA
**1951**, 37, 760–766. [Google Scholar] [CrossRef][Green Version] - Doulamis, N.D.; Kokkinos, P.; Varvarigos, E. Resource Selection for Tasks with Time Requirements Using Spectral Clustering. IEEE Trans. Comput.
**2014**, 63, 461–474. [Google Scholar] [CrossRef] - Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the 2011 IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain, 6–13 November 2011; pp. 2564–2571. [Google Scholar]
- Lv, Q.; Josephson, W.; Wang, Z.; Charikar, M.; Li, K. Multi-probe LSH: Efficient Indexing for High-dimensional Similarity Search. In Proceedings of the 33rd International Conference on Very Large Data Bases, Vienna, Austria, 23–27 September 2007; VLDB Endowment: Vienna, Austria, 2007; pp. 950–961. [Google Scholar]
- Cox, M.A.A.; Cox, T.F. Multidimensional Scaling. In Handbook of Data Visualization; Springer Handbooks Comp. Statistics; Springer: Berlin/Heidelberg, Germany, 2008; pp. 315–347. ISBN 978-3-540-33036-3. [Google Scholar]
- Cayton, L. Algorithms for manifold learning. Univ. Calif. San Diego Tech. Rep
**2005**, 12, 1. [Google Scholar] - Birant, D.; Kut, A. ST-DBSCAN: An algorithm for clustering spatial—Temporal data. Data Knowl. Eng.
**2007**, 60, 208–221. [Google Scholar] [CrossRef] - Themistocleous, K. Model reconstruction for 3d vizualization of cultural heritage sites using open data from social media: The case study of Soli, Cyprus. J. Archaeol. Sci. Rep.
**2017**, 14, 774–781. [Google Scholar] [CrossRef] - Kierkegaard, S. Twitter thou doeth? Comput. Law Secur. Rev.
**2010**, 26, 577–594. [Google Scholar] [CrossRef] - VisualSFM: A Visual Structure from Motion System. Available online: http://ccwu.me/vsfm/index.html (accessed on 17 May 2020).
- Paige, C.C. Computational variants of the Lanczos method for the eigenproblem. IMA J. Appl. Math.
**1972**, 10, 373–381. [Google Scholar] [CrossRef]

**Figure 1.**Distribution of inliers/outliers using images projected onto a 2D space when applying the classical multidimensional scaling (cMDS) algorithm. Illustration is for a given monument.

**Figure 2.**Distribution probability over 20 time intervals (periods) for three different events regarding the most important tweets.

**Figure 3.**Precision–recall curve for the three proposed metrics using both wavelet and nonwavelet representation and for a standard deviation σ equal to 1.

**Figure 4.**Precision versus standard deviation $\mathsf{\sigma}$ for the three tweet-post characterization metrics when we (do not) apply the wavelet representation for a recall value RE = 0.7.

**Figure 5.**Precision versus standard deviation $\mathsf{\sigma}$ for the three tweet-post characterization metrics when we (do not) apply the wavelet representation for a recall value RE = 0.4.

**Figure 6.**Precision–recall curve for the three tweet-post characterization metrics when we (do not) apply the wavelet representation of Section 4.2. The results were obtained using cross-correlation distance on real-life tweet posts.

**Figure 8.**The effect of the number of the outliers in reconstruction performance using the SfM algorithm. (

**a**) 100 inliers, 0 outliers. (

**b**) 90 inliers, 10 outliers. (

**c**) 70 inliers, 30 outliers. (

**d**) 60 inliers, 40 outliers.

**Figure 9.**The effect of the number of the outliers in reconstruction performance using the SfM algorithm. (

**a**) 100 inliers, 0 outliers. (

**b**) 90 inliers, 10 outliers. (

**c**) 70 inliers, 30 outliers. (

**d**) 60 inliers, 40 outliers.

100 images | 1013.2 secs |

90 images | 883 secs |

80 images | 715 secs |

70 images | 635 secs |

60 images | 571 secs |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Doulamis, A.; Voulodimos, A.; Protopapadakis, E.; Doulamis, N.; Makantasis, K. Automatic 3D Modeling and Reconstruction of Cultural Heritage Sites from Twitter Images. *Sustainability* **2020**, *12*, 4223.
https://doi.org/10.3390/su12104223

**AMA Style**

Doulamis A, Voulodimos A, Protopapadakis E, Doulamis N, Makantasis K. Automatic 3D Modeling and Reconstruction of Cultural Heritage Sites from Twitter Images. *Sustainability*. 2020; 12(10):4223.
https://doi.org/10.3390/su12104223

**Chicago/Turabian Style**

Doulamis, Anastasios, Athanasios Voulodimos, Eftychios Protopapadakis, Nikolaos Doulamis, and Konstantinos Makantasis. 2020. "Automatic 3D Modeling and Reconstruction of Cultural Heritage Sites from Twitter Images" *Sustainability* 12, no. 10: 4223.
https://doi.org/10.3390/su12104223