Logging & Experiment tracking with W&B
For people interested in tools for logging and comparing different models and training runs in general,Weights & Biasesis directly integrated withTransformers.You just need to havewandbinstalled and logged in.It automatically logs losses, metrics, learning rate, computer ressources, etc.image1176×718 56.5 KBHere is another coolexamplewhere I ran a sweep to fine-tune GPT-2 on tweets.image1281×957 678 KBFinally you can use your runs to create cool reports. See for example myhuggingtweets report.Seedocumentationfor more details orthis colab.At the moment it is integrated withTrainerandTFTrainer.If you use Pytorch Lightning, you can useWandbLogger. SeePytorch Lightning documentation.Let me know if you have any questions or ideas to make it better!
2020-09-01T02:44:20Z
@borisI have a few questions for the HF Transformers integration:It looks likewandbis charting the loss, learning rate, and epoch for a given run ofTrainer.train(). Are there other things that would be useful to have charted for a finetuning run?It also looks likewandbis using thelogging_stepsvalue inTrainerArguments. Is this right?Is it preferred to setwandbbehavior through the environment variables or in the finetuning script directly?
2020-09-02T16:11:39Z
Hi@aclifton314,aclifton314:It looks likewandbis charting the loss, learning rate, and epoch for a given run ofTrainer.train(). Are there other things that would be useful to have charted for a finetuning run?It also logs validation loss and all the task dependent metrics defined against your validation dataset.Then you can decide to log losses/metrics against epoch using it as x-axis in W&B interface (instead of the step) if it makes more sense for your use case.aclifton314:It also looks likewandbis using thelogging_stepsvalue inTrainerArguments. Is this right?That is correct. It can also log evaluation loss/metrics at the end of training if theevaluateloop is called at the end of your script (usually the case when you use one of the “examples” script from the library).aclifton314:Is it preferred to setwandbbehavior through the environment variables or in the finetuning script directly?There is no preferred way. When I write my own fine-tuning script, I like to pass explicitly my variables as the code looks more clear to me but either way should work perfectly fine.
2020-09-02T19:28:29Z
@boris. Thanks for your reply! One more quick question. I see wherewandbgets initialized in the HF integration. Suppose I want to also log some other metric or value that isn’t automatically logged in the integration. Is there a way to call the samewandbobject from my script that the HF inmplementation is using to include this value being logged?
2020-09-02T19:32:01Z
Yes, you can just callwandb.loganytime manually like sowandb.log({"my_metric":1.2}, step=trainer.global_step)Passing the step is not mandatory but preferred when logging at different places.
2020-09-03T15:29:13Z
@sguggerCan you make top post a wiki to update docs link and add relevant updates when needed?
2020-09-03T16:07:46Z
It’s done!
2020-09-03T16:25:18Z
I’m thinking that we could add a way to log & track datasets and trained models.We could either do it with an environment variable or another parameter inTrainingArguments.
2020-09-04T09:35:26Z
@borisHow did you manage to create such a sweep? Have you used Google Colab or any local machine?  (I’m currently struggling with setting up one in Colab)In the docs it says that there are troubles by using wandb.agent with GPU-Support
2020-09-04T23:30:21Z
@katharinafluchThere are many ways to run sweeps but I actually ran mine in the console.There is a new version of wandb coming up soon that will better support sweeps in Colab so I can make a demo for it then!
2020-09-28T16:51:39Z
@borisI haveexport WANDB_MODE='dry_run'andWANDB_WATCH='all'setup in my environment. I ran a training session and synced the dryrun usingwandb local. When I view the results on the localhost, I don’t see any information about the parameters. Just plots about learning rate, epoch, and loss.Do you know where I can view this information about the parameters and/or if I have done something wrong to prevent them being recorded?
2020-09-28T17:10:11Z
Is it a tensorflow model? I believe watch works mainly for Pytorch models.Also watch does not work with TPU.
2020-09-28T17:28:32Z
It is a pytorch model and I am running on a single gpu.
2020-09-28T17:50:47Z
Ok then maybe it’s because you have less than 100 training steps. Watch logs every 100 steps by default. Try training for more epochs.
2020-09-28T18:44:59Z
I checked theTrainArgumentsobject and it was set to 12,000 steps.
2020-09-30T14:56:27Z
@boris, should the information about the parameters appear alongside the learning rate, epoch, and loss or are they somewhere else in the wandb dashboard?
2020-09-30T18:10:09Z
@aclifton314It appears in separate sections.wandb.ai_borisd13_huggingface_runs_1zdvv8or_workspace=user-borisd131186×814 54.1 KBBy default, you get gradients logged under “gradients” (as long as you have more than 100 training steps).You can also log parameters by settingWANDB_WATCHto 'all` and you will get both parameters and gradients (seedocumentation).I made ademo colabwhich also logs both gradients and parameters.Feel free if you have any other questions!
2020-10-01T18:04:11Z
@borisoh awesome! Thank you for pointing that out. I had totally missed it.I am not seeing the gradients section, though. Is there something I need to modify in my setup?
2020-10-01T18:11:57Z
Do they appear when you use my colab?It worked onmy runYou just need to have at least 100 training steps.
2020-10-01T18:17:23Z
I haveexport WANDB_WATCH='all'in my.bashrc. Do you think the formatting of that needs to be changed?
No date available
No reply text available