Training Metrics in AWS SageMaker
Hi,in the notebook06_sagemaker_metrics/sagemaker-notebook.ipynb, there is the code to get training and eval metrics at the end of the training from the HuggingFaceEstimator.How we can get them DURING the training?Great, but I don’t understandhow we can get them DURING the training to check how good (or not) the training is(for example, to detect overfitting and then, stop training before the last epoch).My idea was to create a duplicate notebook (without runningfit()in this duplicated one) for that purpose. The following text in the notebook seems to say that it is possible but how can we getspecifiying the exact training job name in theTrainingJobAnalyticsAPI call? Thanks.Note that you can also copy this code and run it from a different place (as long as connected to the cloud and authorized to use the API), by specifiying the exact training job name in theTrainingJobAnalyticsAPI call.)Problem: “Warning: No metrics called eval_loss found”I have a second question.I used the metrics code (copy/paste) from06_sagemaker_metrics/sagemaker-notebook.ipynbin a NER finetuning notebook on AWS SageMaker.The code of my NER notebook uses directly the scriptrun_ner.pyfrom github (through the argumentgit_configin my Hugging Face Estimator).metric_definitions=[
    {'Name': 'loss', 'Regex': "'loss': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'learning_rate', 'Regex': "'learning_rate': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_loss', 'Regex': "'eval_loss': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_accuracy', 'Regex': "'eval_accuracy': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_f1', 'Regex': "'eval_f1': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_precision', 'Regex': "'eval_precision': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_recall', 'Regex': "'eval_recall': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_runtime', 'Regex': "'eval_runtime': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'eval_samples_per_second', 'Regex': "'eval_samples_per_second': ([0-9]+(.|e\-)[0-9]+),?"},
    {'Name': 'epoch', 'Regex': "'epoch': ([0-9]+(.|e\-)[0-9]+),?"}]

git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.12.3'} 

huggingface_estimator = HuggingFace(
    entry_point          = 'run_ner.py',
    source_dir           = './examples/pytorch/token-classification',  
    git_config           = git_config,
    (...),
    metric_definitions   = metric_definitions,
)I have no problem of training butwhen I want to display the metrics, most of them were not found(see the following screen shot):metrics2038×940 59.8 KBI compared the code relative to the logs in the 2 scripts and they are different.In thetrain.py:# Set up logging
   logger = logging.getLogger(__name__)

   logging.basicConfig(
        level=logging.getLevelName("INFO"),
        handlers=[logging.StreamHandler(sys.stdout)],
        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    )In therun_ner.py:logger = logging.getLogger(__name__)

# Setup logging
logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )This is the reason of the problem?
2021-12-06T07:38:55Z
pierreguillou:How we can get them DURING the training?You find them in the AWS Management console at the SageMaker Service → Training → Training job → Details View. Or you can use SageMaker Studio to inspect those.AWS Blog postProblem: “Warning: No metrics called eval_loss found”If you are usingmetrics_definitionssagemaker is scanning thestdoutbased on the regex schemes defined inmetrics_definitionsthe scheme doesn’t match the output I cannot find any. So I guess forrun_ner.pyyou might need to adjust themetrics_definitionsregex scheme. Here is more documentation for that:Amazon SageMaker
2021-12-09T16:07:55Z
@pierreguillou- how to find the metrics - in addition to dashboarding them in Cloudwatch via the path provided by Philipp (job detail page in the console, then “algorithm metrics” link in the bottom), you can also pull them in real time with the SDKfrom sagemaker.analytics import TrainingJobAnalytics

df = TrainingJobAnalytics( training_job_name="jobname").dataframe()Capture d’écran 2021-12-09 à 17.07.251144×1134 93.4 KB
2021-12-09T18:12:02Z
Hi@OlivierCR.You’re right: I did create another notebook in my AWS notebook instance where a training notebook is running and I copied/pasted the metrics code from theHF sagemaker notebook about this topic.The main one is what you posted:from sagemaker.analytics import TrainingJobAnalytics
df = TrainingJobAnalytics( training_job_name="jobname").dataframe()My problem was as following:In my training notebook, I did create ajob_nameas@philschmiddid in the notebookworkshop_1_getting_started_with_amazon_sagemaker/lab_1_default_training.ipynb,job_namethat I used in myHuggingFaceEstimatoras following:# define Training Job Name 
import time
job_name = f'huggingface-workshop-{time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())}'

# create the Estimator
huggingface_estimator = HuggingFace(
    base_job_name        = job_name,          # the name of the training job
   (...)
)For example, this code gavejob_name = 'huggingface-workshop-2021-12-09-18-03-30'However, thisjob_namedid not work in the metrics code (at the top of the post).Then, I was toAWS SageMaker console >> Training Jobsand saw a name slightly different that I copied/pasted asjob_name… and it worked!Why the AWS SageMaker console does change the training jobs names?
2021-12-09T18:34:26Z
oh yes good catch! with theEstimatorclass (and its children like HuggingFaceEstimator) of the high-level Python SDK, you can’t set the job name ; you only set a prefix withbase_job_name. The SDK then appends a timestamp to make it unique. SM Training jobs have to be unique, and I guess the SDK team preferred to handle that uniqueness management instead of forcing users to deal with it.if you want to control exact job name, you need to launch jobs with boto3create_training_jobas shown hereif you want to know the full job name of a job launched from an SDK Estimator you can doEstimator.latest_training_job.name
2021-12-09T18:45:34Z
OlivierCR:if you want to know the full job name of a job launched from an SDK Estimator you can doEstimator.latest_training_job.nameI love this last solutionMany thanks for your explanation and help!