How to integrate an AzureMLCallback for logging in Azure?
Hi!I saw that@sguggerrecently refactored the way in whichtransformersintegrates with tools to visualize logs in a more helpful way:https://github.com/huggingface/transformers/pull/7596As I am running in Azure and using AzureML, I was trying to see if I could do something similar.Prior to the PR above, I could add a pair of very simplesnippetsthat allowed to send information to Azure viaazureml.core.Run class - Azure Machine Learning Python | Microsoft LearnI tried to replicate the above with the new approach, but I may be missing something obvious.I created a new callback class inintegrations.pyclass AzureMLCallback(TrainerCallback):

    def __init__(self, azureml_run=None):
        assert (
            _has_azureml
        ), "AzureMLCallback requires azureml to be installed. Run `pip install azureml-sdk`."
        self.azureml_run = azureml_run

    def on_init_end(self, args, state, control, **kwargs):
        if self.azureml_run is None and state.is_world_process_zero:
            self.azureml_run = Run.get_context()

    def on_log(self, args, logs=None, **kwargs):
        if self.azureml_run:
            for k, v in logs.items():
                if isinstance(v, (int, float)):
                    self.azureml_run.log(k, v, description=k)and did anotherbunch of other minor changes.Upon installing on a machine my fork of the library withpip install git+https://github.com/davidefiocco/transformers.git@c32718170899d1110a77ab116a2a60bbe326829e --quietwhen runningpython run_glue.py --model_name_or_path bert-base-cased \
                    --task_name CoLA \
                    --do_train \
                    --do_eval \
                    --train_file ./glue_data/CoLA/train.tsv \
                    --validation_file ./glue_data/CoLA/dev.tsv \
                    --max_seq_length 128 \
                    --per_device_train_batch_size 32 \
                    --learning_rate 2e-5 \
                    --num_train_epochs 3.0 \
                    --output_dir output \
                    --evaluation_strategy steps \
                    --logging_steps 8 \
                    --eval_steps 4I get the error:Traceback (most recent call last):File “run_glue.py”, line 417, inmain()File “run_glue.py”, line 352, in mainmodel_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else NoneFile “/usr/local/lib/python3.6/dist-packages/transformers/trainer.py”, line 792, in trainself._maybe_log_save_evaluate(tr_loss, model, trial, epoch)File “/usr/local/lib/python3.6/dist-packages/transformers/trainer.py”, line 853, in _maybe_log_save_evaluatemetrics = self.evaluate()File “/usr/local/lib/python3.6/dist-packages/transformers/trainer.py”, line 1291, in evaluateself.log(output.metrics)File “/usr/local/lib/python3.6/dist-packages/transformers/trainer.py”, line 1044, in logself.control = self.callback_handler.on_log(self.args, self.state, self.control, logs)File “/usr/local/lib/python3.6/dist-packages/transformers/trainer_callback.py”, line 366, in on_logreturn self.call_event(“on_log”, args, state, control, logs=logs)File “/usr/local/lib/python3.6/dist-packages/transformers/trainer_callback.py”, line 382, in call_event**kwargs,TypeError: on_log() got multiple values for argument ‘logs’So there’s likely something wrong in myAzureMLCallback… can someone help me spot the issue?If you wish to replicate the behavior you can use this notebookGoogle Colabwhile the source code ishttps://github.com/davidefiocco/transformers/tree/c32718170899d1110a77ab116a2a60bbe326829e
2020-10-23T16:29:13Z
Hi there! Glad to see you try the new callbacks! The mistake is that you did not leavestateandcontrolwhich are positional arguments. Just replace youon_logdefinition by:def on_log((self, args, state, control, logs=None, **kwargs):and you’ll be fine!
2020-10-23T16:47:45Z
Indeed! Aw, I don’t know why I messed with the function signature when copying the available examples!Thanks for spotting that and for the new tricks!
2020-10-25T08:34:46Z
might be interesting to add this snippet tohttps://github.com/huggingface/transformers/blob/master/src/transformers/integrations.py@davidefiocco
2020-10-26T08:22:01Z
Cool@julien-c! I will review a couple of things and aim to send a PR to you by this week.