I am facing some error, when I SFTTrainer
hy, when i start training they showing the following error.trainer.train()here is the errorRuntimeError: unscale_() has already been called on this optimizer since the last update().StepTraining Loss╭───────────────────────────────Traceback(most recent call last)────────────────────────────────╮│in<cell line: 1>:1││││/usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1645intrain││││1642│   │inner_training_loop = find_executable_batch_size(││1643│   │   │self._inner_training_loop,self._train_batch_size, args.auto_find_batch_size││1644│   │)││❱1645│   │returninner_training_loop(││1646│   │   │args=args,││1647│   │   │resume_from_checkpoint=resume_from_checkpoint,││1648│   │   │trial=trial,││││/usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1987in_inner_training_loop││││1984│   │   │   │   │   │   │   │args.max_grad_norm,││1985│   │   │   │   │   │   │)││1986│   │   │   │   │   │else:││❱1987│   │   │   │   │   │   │self.accelerator.clip_grad_norm_(││1988│   │   │   │   │   │   │   │model.parameters(),││1989│   │   │   │   │   │   │   │args.max_grad_norm,││1990│   │   │   │   │   │   │)││││/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:1893inclip_grad_norm_││││1890│   │   │# `accelerator.backward(loss)` is doing that automatically. Therefore, its i││1891│   │   │# We cannot return the gradient norm because DeepSpeed does it.││1892│   │   │returnNone││❱1893│   │self.unscale_gradients()││1894│   │returntorch.nn.utils.clip_grad_norm_(parameters, max_norm, norm_type=norm_type)││1895│││1896│defclip_grad_value_(self, parameters, clip_value):││││/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:1856inunscale_gradients││││1853│   │   │foroptinoptimizer:││1854│   │   │   │whileisinstance(opt, AcceleratedOptimizer):││1855│   │   │   │   │opt = opt.optimizer││❱1856│   │   │   │self.scaler.unscale_(opt)││1857│││1858│defclip_grad_norm_(self, parameters, max_norm, norm_type=2):││1859│   │"""││││/usr/local/lib/python3.10/dist-packages/torch/cuda/amp/grad_scaler.py:275inunscale_││││272│   │optimizer_state =self._per_optimizer_states[id(optimizer)]││273│   │││274│   │ifoptimizer_state["stage"]isOptState.UNSCALED:││❱275│   │   │raiseRuntimeError("unscale_() has already been called on this optimizer sin││276│   │elifoptimizer_state["stage"]isOptState.STEPPED:││277│   │   │raiseRuntimeError("unscale_() is being called after step().")││278│╰──────────────────────────────────────────────────────────────────────────────────────────────────╯RuntimeError:unscale_()has already been called on this optimizer since the lastupdate().
2023-06-28T17:58:40Z
Try installing from main, should be fixed there:!pip install git+https://github.com/huggingface/transformers git+https://github.com/huggingface/accelerate