Trainer and Accelerate
What are the differences and if Trainer can do multiple GPU work, why need Accelerate?Accelerate use only for custom code? (add or remove something)
2023-06-28T22:19:01Z
I assume accelerate was added later and has more features like:"""
Accelerate is a library that enables the same PyTorch code to be run across any distributed configuration by adding just
four lines of code!

tldr; handles all from cpu-gpu(s)-multi-node-tpu-tpu + deepseed + mixprecision in one simple wrapper without complicated
calls e.g. that ddp has to do for multi gpus.

ref: my notes: https://www.evernote.com/shard/s410/sh/f1158fa5-4122-0d17-d6eb-a920461e12b6/g47Qtu6j1F58zvMnJ3fWY8v6pFFWi3I_krn5155UigRUmBzr-D8td5HaQA
"""
2023-06-28T22:20:31Z
related:Trainers.train() with accelerate
2023-06-28T23:46:14Z
brando:I assume accelerate was added later and has more features like:TheTrainernow usesaccelerateas the backbone for it (our work the last few months) so it’s "do you want raw accelerate? Or theTrainerAPI). The capabilities are the same overall
2023-07-12T22:21:53Z
just saw this. Pasting this as a ref:Hugging Face Trainer? · Issue #144 · huggingface/accelerate · GitHub
2023-07-12T23:27:28Z
answer:Since the trainer already has created an accelerator obj inside it’s own code you have to dono code changesexcept for writing your own accelerate config and calling it as :accelerate launch --config_file {path/to/config/my_config_file.yaml} {script_name.py} {--arg1} {--arg2} ...An example config is given at the end.Long answerMy assumption was that there would be code changes, since every other accelerate tutorial showed that e.g.,+ from accelerate import Accelerator
  from transformers import AdamW, AutoModelForSequenceClassification, get_scheduler

+ accelerator = Accelerator()

  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  optimizer = AdamW(model.parameters(), lr=3e-5)

- device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
- model.to(device)

+ train_dataloader, eval_dataloader, model, optimizer = accelerator.prepare(
+     train_dataloader, eval_dataloader, model, optimizer
+ )

  num_epochs = 3
  num_training_steps = num_epochs * len(train_dataloader)
  lr_scheduler = get_scheduler(
      "linear",
      optimizer=optimizer,
      num_warmup_steps=0,
      num_training_steps=num_training_steps
  )

  progress_bar = tqdm(range(num_training_steps))

  model.train()
  for epoch in range(num_epochs):
      for batch in train_dataloader:
-         batch = {k: v.to(device) for k, v in batch.items()}
          outputs = model(**batch)
          loss = outputs.loss
-         loss.backward()
+         accelerator.backward(loss)

          optimizer.step()
          lr_scheduler.step()
          optimizer.zero_grad()
          progress_bar.update(1)but those code changes are already inside the Trainer. Their integration is so seamless it’s unclear, or perhaps it’s just not in the tutorials so one has to look at their trainer code e.g.,if is_accelerate_available():
    from accelerate import __version__ as accelerate_version

    if version.parse(accelerate_version) >= version.parse("0.16"):
        from accelerate import skip_first_batches

    from accelerate import Accelerator
    from accelerate.utiSo just make an accelerate config and run it e.g.,# -----> see this ref: https://huggingface.co/docs/accelerate/package_reference/cli#accelerate-config
# ref for fsdp to know how to change fsdp opts: https://huggingface.co/docs/accelerate/usage_guides/fsdp
# ref for accelerate to know how to change accelerate opts: https://huggingface.co/docs/accelerate/basic_tutorials/launch
# ref alpaca accelerate config: https://github.com/tatsu-lab/alpaca_farm/tree/main/examples/accelerate_configs

main_training_function: main  # <- change

deepspeed_config: { }
distributed_type: FSDP
downcast_bf16: 'no'
dynamo_backend: 'NO'
# seems alpaca was based on: https://huggingface.co/docs/accelerate/usage_guides/fsdp
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  #  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer  # <-change
  fsdp_transformer_layer_cls_to_wrap: FalconDecoderLayer  # <-change
#  fsdp_min_num_params:  7e9 # e.g., suggested heuristic: num_params / num_gpus = params/gpu, multiply by precision in bytes to know GBs used
gpu_ids: null
machine_rank: 0
main_process_ip: null
main_process_port: null
megatron_lm_config: { }
#mixed_precision: 'bf16'
#mixed_precision: 'no'
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_name: null
tpu_zone: null
use_cpu: falsestackoverflow.comHow does one use accelerate with the hugging face (HF) trainer?pytorch, nlp, huggingface-transformers, huggingfaceanswered byCharlie Parkeron11:25PM - 12 Jul 23 UTC
2023-07-15T20:27:09Z
Hi can someone help me out?I am working in Google Colab to learn how to finetune a ViT model, following this tutorialVision Transformers (ViT) Explained + Fine-tuning in Python - YouTube.Trying to define the following argumentstraining and testing datasetfeature extractormodelcollate functionevaluation metricBut I have error defining TrainingArguments and Trainer, where it saysImportError: Using theTrainerwithPyTorchrequiresaccelerate>=0.20.1: Please runpip install transformers[torch]orpip install accelerate -UWhere do I need to add accelerate? And how do I incorporate it with this code?nbviewer.jupyter.orgJupyter Notebook ViewerCheck out this Jupyter notebook!
2023-07-26T09:42:30Z
Have you solved this? I meet the same question:colab.research.google.comGoogle Colaboratory
2023-07-26T09:46:48Z
didi-ai:youAfter installing you need to restart the runtime so it loads the newer version
2023-07-26T09:59:08Z
Thanks, I changed the runtime type to GPU and do not change any code, then the code is works.The code still didn’t work on CPU.Do you know why?
2023-07-26T10:47:53Z
didi-ai:Do you know why?On the CPU version, dopip install accelerate -U, make sure it shows the latest, and do Runtime → Restart Runtime, then run the code again (skipping the install). Let me know if this still errors out
2023-07-27T10:25:31Z
Thanks, It works on CPU as you say.
2023-09-27T21:48:00Z
@muellerzrif I want to use only fsdp, do I need HF accelerate? how would I run my script?
2024-09-19T12:02:29Z
I have set accelerate configuration using ‘accelerate config’ in bash. Now does Trainer create some other accelerate configuration by its own?