Skip to main content
SpringerLink
Log in

Mirkwood: An Online Parallel Crawler

  • Conference paper
  • First Online:
International Joint Conference: 12th International Conference on Computational Intelligence in Security for Information Systems (CISIS 2019) and 10th International Conference on EUropean Transnational Education (ICEUTE 2019) (CISIS 2019, ICEUTE 2019)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 951))

  • 584 Accesses

Abstract

In this research we present Mirkwood, a parallel crawler for fast and online syntactic analysis of websites. Configured by default to behave as a focused crawler, analysing exclusively a limited set of hosts, it includes seed extraction capabilities, which allows it to autonomously obtain high quality sites to crawl. Mirkwood is designed to run in a computer cluster, taking advantage of all the cores of its individual machines (virtual or physical), although it can also run on a single machine. By analysing sites online and not downloading the web content, we achieve crawling speeds several orders of magnitude faster than if we did, while assuring that the content we check is up to date. Our crawler relies on MPI, for the cluster of computers, and threading, for each individual machine of the cluster. Our software has been tested in several platforms, including the Supercomputer Calendula. Mirkwood is entirely written in Java language, making it multi–platform and portable.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Apache nutch (2018). http://nutch.apache.org/. Accessed 15 Nov 2018

  2. Google terms of service (2018). policies.google.com. Accessed 15 Nov 2018

  3. Heritrix (2018). https://github.com/internetarchive/heritrix3. Accessed 15 Nov 2018

  4. jsoup: Java html parser (2018). https://jsoup.org/. Accessed 15 Nov 2018

  5. Mirkwood (2018). http://bit.do/eKLo2. Accessed 15 Nov 2018

  6. Ninja plugin (2018). https://www.internetmarketingninjas.com/seo-tools/get-urls-grease. Accessed 15 Nov 2018

  7. Open mpi (2018). https://www.open-mpi.org. Accessed 15 Nov 2018

  8. Chakrabarti, S.: Focused web crawling. In: Encyclopedia of Database Systems, pp. 1147–1155. Springer (2009)

    Google Scholar 

  9. Chakrabarti, S., Punera, K., Subramanyam, M.: Accelerated focused crawling through online relevance feedback. In: Proceedings of the 11th International Conference on World Wide Web, pp. 148–159. ACM (2002)

    Google Scholar 

  10. Dikaiakos, M., Zeinalipour-Yiazti, D.: WebRACE: a distributed www retrieval, annotation, and caching engine. In: Proceedings of PADDA01: International Workshop on Performance-Oriented Application Development for Distributed Architectures, April 2001

    Google Scholar 

  11. Dong, H., Hussain, F.K.: SOF: a semi-supervised ontology-learning-based focused crawler. Concurrency Comput. Pract. Experience 25(12), 1755–1770 (2013)

    Article  Google Scholar 

  12. Fisher, J.: Dwarves, spiders, and murky woods. In: CS Lewis and the Inklings: Discovering Hidden Truth, pp. 104–115 (2012)

    Google Scholar 

  13. García, J.F., Carriegos, M.: On parallel computation of centrality measures of graphs. J. Supercomput., 1–19 (2018)

    Google Scholar 

  14. Jamali, M., Sayyadi, H., Hariri, B.B., Abolhassani, H.: A method for focused crawling using combination of link structure and content similarity. In: Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence, pp. 753–756. IEEE Computer Society (2006)

    Google Scholar 

  15. Menczer, F.: ARACHNID: adaptive retrieval agents choosing heuristic neighborhoods for information discovery. In: Machine Learning-International Workshop Then Conference, pp. 227–235. Morgan Kaufmann Publishers, Inc. (1997)

    Google Scholar 

  16. Munzert, S., Rubba, C., Meißner, P., Nyhuis, D.: Automated Data Collection with R. JW & Sons (2014)

    Google Scholar 

  17. Ramya, P., Sindhura, V., Sagar, P.V.: Testing using selenium web driver. In: 2017 Second International Conference on Electrical, Computer and Communication Technologies (ICECCT), pp. 1–7. IEEE (2017)

    Google Scholar 

  18. Risvik, K.M., Michelsen, R.: Search engines and web dynamics. Comput. Netw. 39(3), 289–302 (2002)

    Article  Google Scholar 

  19. Shkapenyuk, V., Suel, T.: Design and implementation of a high-performance distributed web crawler. In: 2017 Second International Conference on Data Engineering, Proceedings, pp. 357–368. IEEE (2002)

    Google Scholar 

Download references

Acknowledgements

This work has been partially supported by the Spanish National Cybersecurity Institute (Instituto Nacional de Ciberseguridad, INCIBE). This research uses the resources of the Centro de Supercomputación de Castilla y León (SCAYLE, www.scayle.es), funded by the “European Regional Development Fund (ERDF)”.

Author information

Authors and Affiliations

  1. RIASC, Instituto CC, Aplicadas a la Ciberseguridad, Universidad de León, León, Spain

    Juan F. García & Miguel V. Carriegos

Authors
  1. Juan F. García
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miguel V. Carriegos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan F. García .

Editor information

Editors and Affiliations

  1. Data Science and Big Data Lab, Pablo de Olavide University, Seville, Spain

    Francisco Martínez Álvarez

  2. Data Science and Big Data Lab, Pablo de Olavide University, Seville, Spain

    Alicia Troncoso Lora

  3. University of Salamanca, Salamanca, Spain

    José António Sáez Muñoz

  4. Department of Industrial Engineering, University of A Coruña, A Coruña, Spain

    Héctor Quintián

  5. University of Salamanca, Salamanca, Spain

    Emilio Corchado

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

García, J.F., Carriegos, M.V. (2020). Mirkwood: An Online Parallel Crawler. In: Martínez Álvarez, F., Troncoso Lora, A., Sáez Muñoz, J., Quintián, H., Corchado, E. (eds) International Joint Conference: 12th International Conference on Computational Intelligence in Security for Information Systems (CISIS 2019) and 10th International Conference on EUropean Transnational Education (ICEUTE 2019). CISIS ICEUTE 2019 2019. Advances in Intelligent Systems and Computing, vol 951. Springer, Cham. https://doi.org/10.1007/978-3-030-20005-3_5

Download citation

Publish with us

Policies and ethics

Search

Navigation