Eval freezes on local multi GPU Deepspeed run
Environment infotransformersversion: 4.6.0.dev0Platform: Linux-4.19.112±x86_64-with-Ubuntu-18.04-bionicPython version: 3.7.10PyTorch version (GPU?): 1.8.1+cu101 (True)Tensorflow version (GPU?): 2.4.1 (True)Using GPU in script?: <2,4>Using distributed or parallel set-up in script?:InformationI’m working on wav2vec2.0 using the following official script of huggingface.github.comhuggingface/transformers/blob/main/examples/research_projects/wav2vec2/run_common_voice.py#!/usr/bin/env python3
import json
import logging
import os
import re
import sys
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Union

import datasets
import numpy as np
import torch
import torchaudio
from packaging import version
from torch import nn

import transformers
from transformers import (
    HfArgumentParser,
    Trainer,This file has been truncated.show originalI am trying to finetune huggingface model with multiple gpus using deepspeed.deepspeed --num_gpus=1 run_common_voice.py --deepspeed ds_config.json --do_train --do_evalworks, butdeepspeed --num_gpus=2 run_common_voice.py --deepspeed ds_config.json --do_train --do_evalstops working and freezes at the end of eval.The progress bar is 100% done but the eval result is not returned and it freezes.To reproduceThis is how to reproduce!colab.research.google.comGoogle ColaboratorySteps to reproduce the behavior:Install deepspeedAddwith autocast():after line 481 in run_common_voice.pySet param:--deepspeed ds_config.json --do_train --do_evalRun run_common_voice.py using deepspeed with 1> gpusds_config has the following parameters.{
  "fp16": {
    "enabled": "true",
    "loss_scale": 0,
    "loss_scale_window": 1000,
    "hysteresis": 2,
    "min_loss_scale": 1,
    "opt_level": "O3"
  },
  "steps_per_print": 100,
  "wall_clock_breakdown": "false"
}Expected behaviorThe finetuning eval should be executed without freezing.
2021-04-26T09:20:56Z
This is how to reproduce!colab.research.google.comGoogle Colaboratory
2021-04-26T14:36:06Z
cc@stasfor deepspeed.
2021-04-26T16:53:49Z
deepspeed doesn’t work withautocast, it has its own way of dealing with mixed precision, if you look in thetrainer.pyit’s carefully bypassed.If after removingautocastthe problem persistes, please let’s use an Issue for debugging problems, so it’s easy to track.Edit: I see it was already filed:[wav2vec] deepspeed eval bug in the case of >1 gpus · Issue #11446 · huggingface/transformers · GitHubThank you.
2021-04-28T08:53:47Z
Thanks for replying here and on GitHub! I’ll make further reply on the issue.