Seq2Seq fnetuning wandb issue
When using seq2seqfinetune.pyI am getting this warning:wandb: WARNING Step must only increase in log calls.and because of this issue, it is not reporting all the stats in my wandb console.I found this which seems to be relevant:https://github.com/wandb/client/issues/1530Here is a more complete log:Epoch 1:  50%|█████     | 1/2 [00:00<00:00,  2.62it/s, lowandb: WARNING Step must only increase in log calls.  Step 1 < 3; dropping {'val_avg_loss': 3.4621164798736572, 'val_avg_rouge1': 0.0, 'val_avg_rouge2': 0.0, 'val_avg_rougeL': 0.0, 'val_avg_rougeLsum': 0.0, 'val_avg_gen_time': 0.008627939224243163, 'val_avg_gen_len': 3.0, 'step_count': 3, 'epoch': 1}.
Epoch 2:  50%|█████     | 1/2 [00:00<00:00,  2.65it/s, lowandb: WARNING Step must only increase in log calls.  Step 2 < 4; dropping {'val_avg_loss': 2.1513829231262207, 'val_avg_rouge1': 19.3801, 'val_avg_rouge2': 8.7686, 'val_avg_rougeL': 18.0454, 'val_avg_rougeLsum': 19.2525, 'val_avg_gen_time': 0.11329512596130371, 'val_avg_gen_len': 66.0, 'step_count': 4, 'epoch': 2}.
Epoch 3:  50%|█████     | 1/2 [00:00<00:00,  2.61it/s, lowandb: WARNING Step must only increase in log calls.  Step 3 < 5; dropping {'val_avg_loss': 1.903280258178711, 'val_avg_rouge1': 27.2808, 'val_avg_rouge2': 11.1652, 'val_avg_rougeL': 23.9808, 'val_avg_rougeLsum': 27.4063, 'val_avg_gen_time': 0.10656294822692872, 'val_avg_gen_len': 61.0, 'step_count': 5, 'epoch': 3}.
Epoch 4:  50%|█████     | 1/2 [00:00<00:00,  2.71it/s, lowandb: WARNING Step must only increase in log calls.  Step 4 < 6; dropping {'val_avg_loss': 1.3527400493621826, 'val_avg_rouge1': 13.7018, 'val_avg_rouge2': 7.5888, 'val_avg_rougeL': 11.3654, 'val_avg_rougeLsum': 11.3237, 'val_avg_gen_time': 0.20975174903869628, 'val_avg_gen_len': 121.0, 'step_count': 6, 'epoch': 4}.Any thoughts on this?
2020-12-03T20:23:15Z
Looks like we need to use the latest version of Pytorch-lightening:https://github.com/PyTorchLightning/pytorch-lightning/issues/4811