Metrics of of mdeberta-v3-base training stuck on same level
Hey there,I’ve tried to train a model of mdeberta-v3-base on Google Colab (Premium GPU). However, there is an issue, that the training does not proceed or the model does notlearn. The metrics won’t change for all epochs.Training of DeBERTaTraining Arguments:output_dir='./model-results',
num_train_epochs=10,
per_device_train_batch_size=8,
per_device_eval_batch_size=64,
weight_decay=0.01,                   # defaults to pytorch
logging_dir=os.path.join('logs', 'DeBERTa-training-small'),
logging_steps=100,                   # defaults to 500
disable_tqdm=False,
evaluation_strategy='epoch',
save_strategy='epoch',
load_best_model_at_end=True,
metric_for_best_model='eval_f1',
save_total_limit=5,
fp16=True,
report_to="wandb",
learning_rate=0.000005 )Dataset language: germanI’ve trained with the exact same data and parameters XLM-RoBERTa and BERT without any issue.Training of XLM-RoBERTaEpochTraining LossValidation LossAccuracyF1PrecisionRecall10.1785000.1097580.9718640.9606480.9338430.98903720.0659000.0959730.9817940.9741660.9601850.98856130.0768000.0772410.9860970.9800660.9758980.98427140.0228000.0809630.9856010.9795050.9683280.99094450.0446000.0754880.9879180.9827140.9764710.98903760.0232000.0891650.9859320.9799860.9683570.99189770.0185000.0971320.9869250.9814070.9693170.99380480.0292000.1109280.9841110.9775280.9604420.99523490.0160000.1062960.9864280.9807240.9675320.994280100.0107000.1046990.9865940.9809550.9679810.994280Additionally there is an issue that if I increase the batch size to 16, I ran out of CUDA memory (40GB available).Initialization of Tokenizer:tokenizer = AutoTokenizer.from_pretrained('microsoft/mdeberta-v3-base', use_fast=True, max_length=1024)Initialization of Model:model = AutoModelForSequenceClassification.from_pretrained('microsoft/mdeberta-v3-base', num_labels=2)Data formatting[CLS] sequence 1 [SEP] sequence 2 [SEP]The code is running on an Jupyter Notebook on Google Colab. The assigned task is a binary classification to identify sequence pairs.Is there any issue with the model or did I make a mistake?Thanks in advanceBest regardsSacrumDeus
2023-02-26T10:57:52Z
The training with the modelmicrosoft/deberta-v3-baseand the exact same parameters worked. It performed pretty well, although the modelmicrosoft/deberta-v3-basewas not pretrained on German data
2023-05-23T00:59:03Z
I ran into the same issue. Turns out mdeberta does not support fp16 training yet as described inthis issue. Simply turn off fp16 in your training argument should fix the issue.
2023-05-23T08:47:12Z
Hey@bingyinhthanks for your reply. Same happend to me with the modelT5.Since I usedmicrosoft/deberta-v3-baseas model for training, the error does not occur.I will mark it as respolvedBest regardsSacrumDeus