Wandb.watch in accelerate library
When I’m nominally usingwandb, in there documentation for pytorch integration there is a suggested call to thewatchmethod:wandb.watch(my_model, log='all', log_freq=8)With the following enviroment variable set:export WANDB_WATCH="all"I’m able to record the values of the gradients and the parameter values throughout training. Is there a way to record the gradients and parameter values throughout training by using the accelerate library?
2022-06-22T10:04:18Z
Did you try callingwandb.watch()just after initialising the wandb run in Acclerate? It might still work as long as its called before you start logging in your script
2022-06-23T20:19:01Z
morgan:It might still work as long as its called before you start logging in your scriptFor code of what@morganmeans, it’d look something like this:accelerate = Accelerator(log_with="wandb")

accelerate.init_trackers("my_projectname")
wandb.watch()
2022-06-30T18:00:37Z
@morgan@muellerzrThat works perfectly! I tested it out and you can callwandb.watch()even right before the training loop starts (before one starts logging metrics to wandb).
2022-06-30T18:50:30Z
One last thing. I noticed that if I runacceleratoron multiple gpu (say 3) and choose my tracker to bewandbthat it will output 3 files to sync towandb. Is there a way to some how aggregate all those files into one to get a single view of the entire training loop?
2022-07-18T16:16:15Z
@aclifton314we’re aware of this and will be working on it, for now  you could pass agroupargument to the wandb init kwargs, then you can group by this value in the  UI.Group Runs - Documentation
2024-05-01T17:29:05Z
aclifton314:@morgan@muellerzrThat works perfectly! I tested it out and you can callwandb.watch()even right before the training loop starts (before one starts logging metrics to wandb).did you put auguments in watch()? I meanaccelerate.init_trackers("my_projectname")
wandb.watch(model, ...)