Explanation of the default "auto" values for DeepSpeed stage 3?
Hi, I would like to know how the default values of the variousdefault DeepSpeed stage 3 parameterswere determined when using"auto"fields. They seem to work quite well, but I can’t find any documentation of their origins.What experiments or computations were done to land on thereduce_bucket_size,stage3_prefetch_bucket_size, andstage3_param_persistence_thresholdformulas below?self.fill_only("zero_optimization.reduce_bucket_size", hidden_size * hidden_size) 
 if self.is_zero3(): 
     # automatically assign the optimal config values based on model config 
     self.fill_only("zero_optimization.stage3_prefetch_bucket_size", 0.9 * hidden_size * hidden_size) 
     self.fill_only("zero_optimization.stage3_param_persistence_threshold", 10 * hidden_size)Maybe@stas? I saw that you have worked on much of thedeepspeed.pycode.Thank you!
2023-08-22T20:41:24Z
A response can be foundhere