Issues with building extensions in Deepspeed
I get this error even after following the instructions on Deepspeed installation page and HF Trainer docs. Can anyone suggest how to fix this ?I’ve followed@stas’s replies on GH issues, but this error keeps coming up. pybind11 is installed on my machine, not sure what is this indicating ?ds1305×675 166 KB
2021-05-23T06:23:40Z
@prajjwal1- something is off in your python env.The mantra should always be - let’s see if others have asked this question already, so here tryhttps://www.google.com/search?q=pybind11%2Fpybind11.h+no+such+file+or+directoryPlease avoid posting images for tracebacks - instead copy-n-paste the output using code blocks - this is because it’s impossible to copy-n-paste from the image to do the search for you.That’s said if the issue continues after you tried the solutions offered at the top matching pages - please post the details atIssues · microsoft/DeepSpeed · GitHub
2021-05-23T14:27:14Z
Okay, I think I should have provided more details in the sense that you think I didn’t look elsewhere, so here it is.The issue comes when theTrainertries to usecpu_adamextension. Now there are two ways, JIT way of building extension on fly, and prebuilding the extensions. JIT way and prebuilding fails because of the error trace related to pybind11. So I search forcpu_adam, I land upon thisissue. Now this issue is open, you provided some details which I did follow.Trainer docs, local rank doesn’t need to configured as you said. I did setPATHandLD_LIBRARY_PATHas well.I did seethisalso, where in I tried pre-building:git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 python setup.py build_ext -j8 bdist_wheelThis doesn’t seem to work for the same reason. Looked into1,2as well.
2021-05-23T15:50:58Z
Thank you for sharing the details on what you have already tried - that helps a lot.I assume prebuilding failing with the same error of missingpybind11/pybind11.h, correct?So, next, let’s do a very simple thing? Check that you havepybind11/pybind11.hin your python environment that you use to build deepspeed.I see that it comes bundled withtorch, so e.g. I have it under my conda env:/mnt/nvme1/anaconda3/envs/py38-pt18/lib/python3.8/site-packages/torch/include/pybind11/pybind11.hSo in your case there should be one under/usr/lib/python3/dist-packages/torch/includesince this is one of-Is include statements in your error log. So for some reason you probably lack:/usr/lib/python3/dist-packages/torch/include/pybind11/pybind11.hand if it is so, then your package is borked.  I have no idea how you came about this package, it appears like some non-conda/non-pip distribution package - but is it possible that it hastorchandtorch-devpackages? In which case you’re missing the dev side of things, which would have all the required headers. At least that’s how all.debpackages are made.So if you installed this package as:apt install torchyou now need:apt install torch-devAt this point you probably want to tell us what OS/dist you’re on and how you installed torch.
2021-05-23T16:49:13Z
Thanks for replying. So I don’t seepybind11in the include directory. I seeATen,c10,c10d,caffe2,fp16,TH,THC,THCUNNandtorch. I do not have root rights on the server. Server has 20.04 Ubuntu. I installedtorchvia pip. I’m not sure why do we need to useaptfor this or maybe you’re just showing it as an example if i understand correctly.I want to trainCTRLon a 24x4 GPUs but since it doesn’t have model parallel built in, it won’t even fit on one GPU with single batch size.
2021-05-23T17:18:07Z
Most likely you’re not using pytorch you installed but a system-wide installed one. If you don’t have sudo access you won’t have been able to install torch into system-wide dirs, and your error message shows that it a system-wide torch you’re trying to use.Before you edited this last comment - you shared how you installed it - so based on that info we can validate that the distributed package is whole:$ wget https://download.pytorch.org/whl/cu111/torch-1.8.1%2Bcu111-cp38-cp38-linux_x86_64.whl
$ unzip torch-1.8.1+cu111-cp38-cp38-linux_x86_64.whl
$ find torch | grep pybind11.h
torch/include/pybind11/pybind11.hThe header is there.Whoever installedtorchon that system, didn’t do a full install. And you’re attempting to use the system-wide torch and not thetorchyou presumably installed yourself.I recommend you create a dedicated conda env (or any other preferred python virt env of your choice), activate it, install torch and whatever else you need and everything will just work.Of course, make sure you’re inside the virtual environment when you run your code.Alternatively contact your sysadmin and request installing the missing files. I was guessing that this was some kind of custom .deb package which only installed the essentials and left out any dev files, hence the discussion ofapt.Of course, you can also dopip install pybind11to solve the immediate problem, but chances are that some other required by torch header files will be missing. So it’s best to solve this problem at the root.
2021-05-23T21:04:13Z
Thank you very much for swift and detailed replies. I figured out about the torch installation after I see thatpybind11is not present. Will reinstall it on my end. Hopefully it should be fixed once I verify that the pybind11 headers are present.EDIT: Installingtorch 1.8worked. I ensuredpybind11is present.
2022-05-14T20:30:58Z
FWIW I also got this error. On ubuntu, I fixed it by apt install python3-pybind11