Abstract

Latent Semantic Indexing (LSI) has been widely used in information retrieval due to its efficiency in solving the problems of polysemy and synonymy. However, LSI is notably a computationally intensive process because of the computing complexities of singular value decomposition and filtering operations involved in the process. This paper presents MR-LSI, a MapReduce based distributed LSI algorithm for scalable information retrieval. The performance of MR-LSI is first evaluated in a small scale experimental cluster environment, and subsequently evaluated in large scale simulation environments. By partitioning the dataset into smaller subsets and optimizing the partitioned subsets across a cluster of computing nodes, the overhead of the MR-LSI algorithm is reduced significantly while maintaining a high level of accuracy in retrieving documents of user interest. A genetic algorithm based load balancing scheme is designed to optimize the performance of MR-LSI in heterogeneous computing environments in which the computing nodes have varied resources.

Keywords

Cite this article

@article{cai-995,
  author  = {Yang Liu and Maozhen Li and Mukhtaj Khan and Man Qi},
  title   = {A MapReduce Based Distributed LSI for Scalable Information Retrieval},
  journal = {Computing and Informatics},
  volume  = {33},
  number  = {2},
  pages   = {259--280},
  year    = {2014},
  issn    = {1335-9150},
  url     = {https://cai.type.sk/content/2014/2/a-mapreduce-based-distributed-lsi-for-scalable-information-retrieval/}
}
Download .bib