WARCProcessor: An Integrative Tool for Building and Management of Web Spam Corpora
AbstractIn this work we present the design and implementation of WARCProcessor, a novel multiplatform integrative tool aimed to build scientific datasets to facilitate experimentation in web spam research. The developed application allows the user to specify multiple criteria that change the way in which new corpora are generated whilst reducing the number of repetitive and error prone tasks related with existing corpus maintenance. For this goal, WARCProcessor supports up to six commonly used data sources for web spam research, being able to store output corpus in standard WARC format together with complementary metadata files. Additionally, the application facilitates the automatic and concurrent download of web sites from Internet, giving the possibility of configuring the deep of the links to be followed as well as the behaviour when redirected URLs appear. WARCProcessor supports both an interactive GUI interface and a command line utility for being executed in background. View Full-Text
Share & Cite This Article
Callón, M.; Fdez-Glez, J.; Ruano-Ordás, D.; Laza, R.; Pavón, R.; Fdez-Riverola, F.; Méndez, J.R. WARCProcessor: An Integrative Tool for Building and Management of Web Spam Corpora. Sensors 2018, 18, 16.
Callón M, Fdez-Glez J, Ruano-Ordás D, Laza R, Pavón R, Fdez-Riverola F, Méndez JR. WARCProcessor: An Integrative Tool for Building and Management of Web Spam Corpora. Sensors. 2018; 18(1):16.Chicago/Turabian Style
Callón, Miguel; Fdez-Glez, Jorge; Ruano-Ordás, David; Laza, Rosalía; Pavón, Reyes; Fdez-Riverola, Florentino; Méndez, Jose R. 2018. "WARCProcessor: An Integrative Tool for Building and Management of Web Spam Corpora." Sensors 18, no. 1: 16.
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.