Saving weights and checkpoints
I used to use checkpoint callback in Keras, Is there any alternative in Huggingface?If I re-run the training cell it continues from the last loss so it is automatically saved?Could anyone please explain more about how Huggingface saves partial checkpoints so I can continue later from this point?
2022-04-13T13:39:09Z
Yes, you can control how to deal with checkpoints within theTrainerclass. Have a read through thedocumentation, which should help you.
2022-04-14T07:19:48Z
Thanks, BramVanroyI think this documentation is for PyTorch and I am currently using TensorFlow.So that means there is no such solution now in TensorFlow
2022-04-14T08:56:00Z
I am no familiar with the TF code base of the libraries, but it seems that some checkpointing is implemented:github.comhuggingface/transformers/blob/195fbbb6cfc6c6279cef6be12b05a53d589b0de8/src/transformers/trainer_tf.py#L502-L526with self.args.strategy.scope():self.create_optimizer_and_scheduler(num_training_steps=t_total)folder = os.path.join(self.args.output_dir, PREFIX_CHECKPOINT_DIR)ckpt = tf.train.Checkpoint(optimizer=self.optimizer, model=self.model)self.model.ckpt_manager = tf.train.CheckpointManager(ckpt, folder, max_to_keep=self.args.save_total_limit)iterations = self.optimizer.iterationsepochs_trained = 0steps_trained_in_current_epoch = 0if self.model.ckpt_manager.latest_checkpoint:logger.info(f"Checkpoint file {self.model.ckpt_manager.latest_checkpoint} found and restoring from checkpoint")ckpt.restore(self.model.ckpt_manager.latest_checkpoint).expect_partial()self.global_step = iterations.numpy()epochs_trained = self.global_step // self.steps_per_epochsteps_trained_in_current_epoch = self.global_step % self.steps_per_epochThis file has been truncated.show originalMaybe someone else can chime in, who knows more about TF.