Sage maker training failing for many models

Models
1

Hi,

As I try to train them in sage maker as per the provided code in models like Llama or gpt2 they all fail with error:

UnexpectedStatusException: Error for Training job huggingface-pytorch-training-2024-06-23-15-29-04-155: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage "raise ImportError(
 ImportError: This example requires a minimum version of 4.37.0, but the version found is 4.36.0.
 Check out https://github.com/huggingface/transformers/tree/main/examples#important-note for the examples corresponding to other versions of HuggingFace Transformers."
Command "/opt/conda/bin/python3.10 run_qa.py --model_name_or_path openai-community/gpt2 --output_dir /opt/ml/model", exit code: 1

I have tried

  1. running sagemaker update; which succeeds but no luck
  2. Manually set it to transformers_version=‘4.36.0’ but still no luck

Does anyone know how to deal with this issue?