Saving Checkpoint on S3 using Trainer
I am using Trainer and in the TraningArgument I am passing a s3 location, My Sagemaker notebook has access to S3 and I am able to save data directly to s3 from the notebook. but when I pass the same in the Trainer output_dir it create the same folder structure in the current director. and does not send it to s3. Is there a way I can prohibit the Trainer to create dir struct. Because when I delete the folder that it send the run details to the s3 but At the same time whenever it save a checkpoint it again checks for the folder and as it is not present it creates a new one.training_args = Seq2SeqTrainingArguments(output_dir=‘s3://path/to/folder/’,evaluation_strategy=EVALUATION_STRATEGY,save_strategy = SAVE_STRATEGY,learning_rate=LEARNING_RATE,per_device_train_batch_size=BATCH_SIZE,per_device_eval_batch_size=BATCH_SIZE,weight_decay=WEIGHT_DECAY,save_total_limit=SAVE_TOTAL_LIMIT,num_train_epochs=EPOCH,predict_with_generate=PREDICT_WITH_GENERATE,fp16=FP16,# resume_from_checkpoint=RESUME_FROM_CHECKPOINT,load_best_model_at_end = LOAD_BEST_MODEL_AT_END,)I am doing this as I do not have save the file on notebook disk