Accelerate FSDP config prompts
Hello. I’m using accelerate to leverage FSDP as per the tutorialhere. I would like to finetune CodeBert usingrun_mlm_no_trainer.pyon a custom dataset. I’ve runaccelerate configbut I’m not sure what the prompt →transformer_layer_cls_to_wrapentails - be it for my model or even in the examples provided.
2022-08-05T13:10:02Z
cc@smangrul
2022-08-05T15:27:21Z
Hello@ablam, the blog post is outdated as the FSDP features have been upgraded in PyTorch version 1.12.0. As such, all these new features have been integrated into HF Accelerate. For transformer-based models, PyTorch teams suggested using thetransformer_auto_wrappolicy. In this policy, the user has to specify the case-sensitive name of an encoder/decoder block comprising of the Multi-Head Attention layer followed by the Feedforward layer. For example, in the T5 model,T5Blockis the name for the attention block used by the model for N such layers/blocks in the encoder and decoder. Similarly, for the BERT model, it isBertLayerand for GPT2 it isGPT2Block. Below is an example of the accelerate config for thebert-base-casedmodel :compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_offload_params: false
  fsdp_sharding_strategy: 1
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: BertLayer
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 2
use_cpu: falseYou will have to print the model and check the name for the attention block and pass it as the value fortransformer_layer_cls_to_wrap. I hope this helps.
2022-08-09T20:26:39Z
Yes, it does. Found the corresponding reference in my model. Thank you!
2023-07-16T16:14:30Z
Hi,I have a question aboutfsdp_transformer_layer_cls_to_wrap: How can I know the transformer layer for different models? If I want to accelerate LLaMa, can I directly set this parameter toLlamaLayer?
2023-09-15T09:59:56Z
Hello@Colorful, the docs have been update here recentlyFully Sharded Data Parallel (huggingface.co)answering your question. Specifically, the highlighted part below.Screenshot 2023-09-15 at 3.29.40 PM1060×497 73.2 KB