A task scheduling strategy based on weighted round robin for distributed crawler. (26th November 2015)

• Record Type: Journal Article
• Authors: Ge, Dajie; Ding, Zhijun; Ji, Hongfei
• Other Names: Frincu Mark guestEditor.; Bósa Károly guestEditor.; Rong Chunming guestEditor.; Liu Lu guestEditor.; Chen Guolong guestEditor.
• Is Part Of: Concurrency and computation. Volume 28:Number 11(2016)
• Page Start: 3202
• Page End: 3212
• Publication Date: 2015-11-26
• Subjects: distributed -- crawlers -- scheduling -- weighted round robin; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3701
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: With the rapid development of the network, stand‐alone crawlers are finding hard to find and gather information. Distributed crawlers are gradually accepted to solve this problem. This paper proposes a task scheduling strategy based on weighted round robin for small‐scale distributed crawler with formula weights for the current node based on crawling efficiency, implements a distributed crawler system with multithreading support and deduplication which takes the algorithm as core, and discusses some possible extensions and details. The design of the error recovery mechanism and the node table allows crawling nodes have flexible scalability and fault tolerance. Finally, we conducted some experiments to prove the good load balancing performance of the system. Concurrency and Computation: Practice and Experience, 2015.© 2015 Wiley Periodicals, Inc. Copyright © 2015 John Wiley & Sons, Ltd.