METHOD OF ADAPTIVE DEDUPLICATION WITH MULTILEVEL BLOCK INDEXING
Data deduplication is a lossless compression technology that has been widely used in the large-scale storage systems. In a data storage system, the performance of data deduplication algorithms are varying on the condition of file contents. This paper presents a method of an adaptive deduplication that uses both file-level and block-level deduplication techniques. This approach gives the ability to balance computational workload with regulated dedupe ratios. The module of adaptive deduplication uses file metadata, system settings and statistics. For improving the performance of the method we developed a specialized algorithm of multilevel block indexing and file classification. Files are automatically separated into classes according to the degree of similarity, that is based on application of Jaccard index and Broder theorem. The experimental evaluation driven by the real world datasets and developed software shows that our proposed approach effectively balance workload.