Exact difference between Transformers' and Accelerate's DeepSpeed integrations?
Hello,I’m trying to use DeepSpeed with Transformers, and I see there are two DeepSpeed integrations documented on HF:(a) Transformers’ DeepSpeed integration:DeepSpeed Integration(b) Accelerate’s DeepSpeed integration:DeepSpeedHowever, I’m a bit confused by these two.They have separate documentations, but are they really two completely separate integrations?After examining the codes, I’ve realized thatAccelerator._prepare_deepspeedcallsdeepspeed.initializewhile there is no calling ofdeepspeed.initializeon Transformers’ end, which seems to be contradicting with documentation (a) as “Trainer Deepspeed Integration” shouldn’t require the user to calldeepspeed.initializemanually.Could someone clarify it for me, please?Thank you!
2023-12-23T21:34:02Z
The Trainer uses Accelerate under the hood so it’s called when the Trainer callsaccelerator.prepare()
2023-12-23T21:45:07Z
Thank you for your response.Does that mean Transformers’ DeepSpeed integration relies on Accelerate’s DeepSpeed integration?
2023-12-23T21:46:56Z
Yes indeed
2023-12-25T01:59:22Z
so in an example where the script includestokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

training_args = TrainingArguments(
    ...
    deepspeed='ds_config.json'
)

trainer = Trainer(
        model=model,
        tokenizer=tokenizer,
        args=training_args
)
trainer.train()We would usedeepspeed train.pyinstead ofaccelerate launch --config_file config.yaml train.py?Or would those two be equivalent?
2024-02-13T22:43:17Z
Equivalent