Error when instantiating dataloaders outside training_function
Currently testing out HuggingFace Accelerate on TPUs on Colab based on theexample notebook.My question: Do you have to instantiate the dataloaders inside the training function? Because I didn’t, and then I get this error:UnboundLocalError: local variable 'train_dataloader' referenced before assignment
Exception in device=TPU:5: local variable 'train_dataloader' referenced before assignment
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 274, in __call__
    self.launcher(*args)
  File "<ipython-input-20-001611a4ea01>", line 33, in training_function
    model, optimizer, train_dataloader, val_dataloader, test_dataloader
UnboundLocalError: local variable 'train_dataloader' referenced before assignment
Exception in device=TPU:4: local variable 'train_dataloader' referenced before assignmentIt’s weird, becauseaccelerator.prepareis defined within thetraining_function. Thetrain_dataloader is a global variable, is that the problem?cc@sgugger
2021-06-29T14:52:43Z
Update: apparently you have to, because it only works when instantiating the dataloaders inside thetraining_function.
2021-06-29T15:12:00Z
Yes we were just investigating that with@lewtun. Weirdly, it works if you name the prepared dataloader differently (train_dlfor instance) but not if it has the same name as the global variable.
2021-06-30T07:01:50Z
Hmm yes that’s weird. Also, how can you access the model after you’ve trained it? Because themodelvariable is only defined within the training function.
2021-06-30T07:39:16Z
Update: I tried by addingmodel.save_pretrained(directory_name)within the function, such that you can access it after training, but this gives me an error:Configuration saved in /content/config.json
---------------------------------------------------------------------------
ProcessExitedException                    Traceback (most recent call last)
<ipython-input-28-a91f3c0bb4fd> in <module>()
      1 from accelerate import notebook_launcher
      2 
----> 3 notebook_launcher(training_function)

3 frames
/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    134                     error_pid=failed_process.pid,
    135                     exit_code=exitcode,
--> 136                     signal_name=name
    137                 )
    138             else:

ProcessExitedException: process 7 terminated with signal SIGKILLThis is probably because the model is still replicated on 8 TPU cores?
2021-06-30T11:37:54Z
You need to specifically use the save function PyTorch XLA gives you to avoid error, so to useaccelerator.save. See thedoc on savingfor mor context but basically:# Wait for all the processes to arrive
accelerator.wait_for_everyone()
# Save the real model, not the DDP wrapper
unwrapped_model = accelerator.unwrap_model(model)
# Use accelerator.save to save
model.save_pretrained(local_folder, save_function=accelerate.save)See alsothis scriptfor another example.