Abstract

Multi-Agent theory which is used for communication and collaboration among focused crawlers has been proved that it can improve the precision of returned result significantly. In this paper, we proposed a new organizational structure of multi-agent for focused crawlers, in which the agents were divided into three categories, namely F-Agent (Facilitator-Agent), As-Agent (Assistance-Agent) and C-Agent (Crawler-Agent). They worked on their own responsibilities and cooperated mutually to complete a common task of web crawling. In our proposed architecture of focused crawlers based on multi-agent system, we emphasized discussing the collaborative process among multiple agents. To control the cooperation among agents, we proposed a negotiation protocol based on the contract net protocol and achieved the collaboration model of focused crawlers based on multi-agent by JADE. At last, the comparative experiment results showed that our focused crawlers had higher precision and efficiency than other crawlers using the algorithms with breadth-first, best-first, etc.

Keywords

Cite this article

@article{cai-1590,
  author  = {Yajun Du and Yong Xu and Min Wang},
  title   = {A Novel Cooperation and Competition Strategy Among Multi-Agent Crawlers},
  journal = {Computing and Informatics},
  volume  = {35},
  number  = {5},
  pages   = {1050--1078},
  year    = {2016},
  issn    = {1335-9150},
  url     = {https://cai.type.sk/content/2016/5/a-novel-cooperation-and-competition-strategy-among-multi-agent-crawlers/}
}
Download .bib