Understanding of TrainingJobAnalytics
Hello guysI’m working on a Text Classification project, therefore I’m testing different models (with different hyper-parameters config) to see which one can give me the best results.To be able to compare the different models, I’d like to retrieve the metrics computed during the training jobs. I followed what is explained in the different notebooks with theTrainingJobAnalyticsmethod.However, I’m not sure to understand which parameter controls the log step (if there’s any). Let me explain, I have this config:parser.add_argument("--epochs", type=int, default=5)
parser.add_argument("--train-batch-size", type=int, default=16)
parser.add_argument("--eval-batch-size", type=int, default=32)
parser.add_argument("--model_name", type=str)
parser.add_argument("--learning_rate", type=str, default=2e-5)
parser.add_argument("--weight_decay", type=str, default=0.01)
parser.add_argument("--gradient_accumulation_steps", type=int, default=1)with anepocheval_strategy and logging_strategy. Therefore, I expect to have in output of theTrainingJobAnalytics5 rows for each metric representing the value at each epoch as it’s displayed during the training process:image2038×510 62.2 KBHowever what I have in output is that:image598×872 33.1 KBIs there a way to retrieve the same metrics displayed during the training job when calling theTrainingJobAnalytics? Or the values retrieved are only based on the frequency at which Sagemaker sends events to Cloudwatch (which seems to be every 60s) ?Thank you in advance
2022-07-14T08:18:13Z
Hello@YannAgora,There are two different options how you could progress:Use Weights & Biases or Tensorboard integration withTransformersand track you experiments like that.To use Sagemaker-experiments/modify the information extraction you need to defineregexpattern from the information you want to extract from the logs and the provide them when creating your Training, see thisexampleIs there a way to retrieve the same metrics displayed during the training job when calling theTrainingJobAnalytics? Or the values retrieved are only based on the frequency at which Sagemaker sends events to Cloudwatch (which seems to be every 60s) ?The Sagemaker Experiment Metrics can be display in real time in Amazon SageMaker Studio or you can write a custom polling method yourself.For W&B and TB they are reported in real-time to either the Hub or W&B platform.
2022-07-15T09:04:29Z
Thank you@philschmidfor the options you gave, I’ll try to dig into theseMeanwhile I found workaround by capturing the training logs and then applying regex on it to extract the different dictionaries. (Certainly not the most efficient solution but simple and working)%%capture log
huggingface_estimator.logs()thenitems = re.findall(r"({.+})", log.stdout)