Fine-Tune dit-large-finetuned-rvlcdip
I’m trying tofine-tune“microsoft/dit-large-finetuned-rvlcdip”on my custom dataset which is made up of three document classes (“Altro”, “Fattura”, “Prescrizione”).Below the training parameters:training_args = TrainingArguments(
        auto_find_batch_size=True,
        output_dir=str(output_location), 
        remove_unused_columns=False,
        evaluation_strategy="epoch",
        save_strategy="epoch",
        num_train_epochs=10,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        gradient_accumulation_steps=2,
        save_total_limit=3,
        seed=42,
        load_best_model_at_end=True,
        logging_steps=10,
        optim="adamw_torch",
        bf16=False,
        fp16=False,
        learning_rate=1e-5,
        warmup_ratio=0.1,
        report_to='tensorboard'
    )Training is performed on 2-GPUs (NVIDIA GeForce RTX 2070).As can be seen in the figure below, the problem is thatvalidation loss and validation metrics remain stationarystarting from the second epoch.Screenshot 2023-12-21 1101221105×326 19.8 KBWhat could be the cause of this behavior?I expected at least a small change in loss and validation metrics.I get the same behavior by using more or less training data and both enabling and disabling different data augmentation strategies.
2023-12-21T14:57:40Z
Hi@elisasylvie,If you’re using ignore_mismatched_sizes=True, we recently fixed a bug which might be related to this. Could you try by installing Transformers from the main branch (pip install --upgrade git+https://github.com/huggingface/transformers.git) and try again?
2023-12-21T16:15:57Z
Hi@nielsr, thanks for your quick response.I followed your suggestion and installed Transformers from the main branch.Unfortunately I’m not able to start training due to the following error:RuntimeError: CUDA has been initialized before thenotebook_launchercould create a forked subprocess. This likely stems from an outside import causing issues once thenotebook_launcher()is called. Please review your imports and test them when running thenotebook_launcher()to identify which one is problematic and causing CUDA to be initialized.I specify that there is nothing in the code outside of notebook_launcher() that explicitly initializes the CUDA context. The version for accelerate is 0.25.0.
2023-12-21T23:06:09Z
Hi. I do see same issue is openedhere. But I don’t know well what is the current state on  this issue.If possible, the following information will be very helpful:previously (before you installed transformers main branch), what was the accelerate version (the one that didn’t haveCUDA has been initializedissue)if you are running google colab, could you share it (probably use a public dataset instead of your custom dataset)otherwise, could you share your env. info, like gpu type, number of gpu, pytorch version, cuda versioin etc.  (you can runtransformers-cli envto get this)