Abstract

The data de-duplication system not only pursues the high de-duplication rate, which refers to the aggregate reduction in storage requirements gained from de-duplication, but also the de-duplication speed. To solve the problem of random parameter-setting brought by Content Defined Chunking (CDC), a self-adaptive data chunking algorithm is proposed. The algorithm improves the de-duplication rate by conducting pre-processing de-duplication to the samples of the classified files and then selecting the appropriate algorithm parameters. Meanwhile, FastCDC, a kind of content-based fast data chunking algorithm, is adopted to solve the problem of low de-duplication speed of CDC. By introducing de-duplication factor and acceleration factor, FastCDC can significantly boost de-duplication speed while not sacrificing the de-duplication rate through adjusting these two parameters. The experimental results demonstrate that our proposed method can improve the de-duplication rate by about 5

Keywords

Cite this article

@article{cai-1687,
  author  = {Xingjun Zhang and Guofeng Zhu and Endong Wang and Scott Fowler and Xiaoshe Dong},
  title   = {Data De-Duplication with Adaptive Chunking and Accelerated Modification Identifying},
  journal = {Computing and Informatics},
  volume  = {35},
  number  = {3},
  pages   = {586--614},
  year    = {2016},
  issn    = {1335-9150},
  url     = {https://cai.type.sk/content/2016/3/data-de-duplication-with-adaptive-chunking-and-accelerated-modification-identifying/}
}
Download .bib