`run_translation.py` example is erroring out with the recommended settings
I am on 4 V100 GPUs, 32 CPUs, and 208 GBs of RAM.ds_report:--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [YES] ...... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.7/site-packages/torch']
torch version .................... 1.11.0+cu113
torch cuda version ............... 11.3
torch hip version ................ None
nvcc version ..................... 11.0
deepspeed install path ........... ['/opt/conda/lib/python3.7/site-packages/deepspeed']
deepspeed info ................... 0.6.2+0149bd4f, 0149bd4f, master
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3, hip 0.0Versions:torch: 1.11.0+cu113
transformers: 4.18.0.dev0
deepspeed: 0.6.2+0149bd4fI’m runningdeepspeed run_translation.py \
  --deepspeed ds_config_zero2.json \
  --model_name_or_path t5-small --per_device_train_batch_size 1 \
  --output_dir output_dir --overwrite_output_dir --fp16 \
  --do_train --max_train_samples 500 --num_train_epochs 1 \
  --dataset_name wmt16 --dataset_config "ro-en" \
  --source_lang en --target_lang torun_translation.pyandds_config_zero2.jsonwere sourced from thetransformersrepository.This leads to:Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f726bc1add0>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 97, in __del__
Exception ignored in:     self.ds_opt_adam.destroy_adam(self.opt_id)
<function DeepSpeedCPUAdam.__del__ at 0x7f9a62c11dd0>AttributeError
: Traceback (most recent call last):
'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 97, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7fa18e555dd0>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 97, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7efe2cd62dd0>
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/deepspeed/ops/adam/cpu_adam.py", line 97, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'This is howdeepspeedwas installed from source:git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="7.0" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 pip install . \
--global-option="build_ext" --global-option="-j8" --no-cache -v \
--disable-pip-version-check 2>&1 | tee build.log@stas@sgugger
2022-04-04T06:26:15Z
Solved withAttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' · Issue #1846 · microsoft/DeepSpeed · GitHub