Training Instruct-pix-2-pix with my own dataset: torch error
Hello everyone,I am trying to train Instruct-pix-2pix pipeline with this guidelinesdiffusers/examples/instruct_pix2pix/README.md at main · huggingface/diffusers · GitHub. I have created my own dataset. But I got an errortorch.autograd.backward( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: GET was unable to find an engine to execute this computationCould not load library libcudnn_cnn_train.so.8. Error: /usr/local/cuda/lib64/libcudnn_cnn_train.so.8: undefined symbol: _ZN5cudnn3cnn34layerNormFwd_execute_internal_implERKNS_7backend11VariantPackEP11CUstream_stRNS0_18LayerNormFwdParamsERKNS1_20NormForwardOperationEmb, version libcudnn_cnn_infer.so.8Maybe it’s incompatibility between my libraries?I am running on NVIDIA A100 40GBtorch  2.2.0, torchvision 0.17.0, xformers 0.0.24, and cuda 12.1Thank you!