Total_flos vs C = 6 * N * D
As far as I am aware,total_flos, returned bytrainer.train, is supposed to indicate the total number of floating point operations performed to train some given model. However, this value can vary substantially compared to that of the commonly used formula  C \approx 6 * N * D, where N is number of model parameters and D is model size.For instance, as a quick experiment, training a freshly initialized 106M parameter Llama model on 52M tokens results in total_flos=2.7e16, whereas using the formula we obtain C=3.3e16 FLOPS.My question is whether the returnedtotal_flosis indeed accurate (specially for Llama-type models).Tagging@tevensince I believe he wrote the relevant code.
2023-12-13T12:31:38Z
Currently, trainer does not include the embedding parameters in the FLOP computation, which is the source of the discrepancy observed