Nowadays the digital universe becomes larger and larger. The data created everyyear has been up to ZB level. How to store thedata in many normal servers is a critical issue. Although a distributed file system alleviates thestorage problem, there is still a need to reducethe storage and speed up transmission of largescale data. The distributed file system like HDFSalready offers compression schemes to cater tothe need, however, when the workload and dataformat change, configuring the compression withonly one kind of algorithm is not always effective. In this paper, we proposed a model calledPACM (Prediction-based Auto-adaptive Compression Model) to optimize the storage and performance by using different algorithms, e.g. quicklz, zlib, snappy according to variable data formatand workload. We also implemented the model inHadoop and our empirical evaluation shows thatby using PACM, the write throughput has beenimproved by 2-5 times.
['Ruijian Wang', 'Chao Wang', 'Li Zha']
PACM: A Prediction-Based Auto-Adaptive Compression Model for HDFS