Backends for distributed setups like gloo and nccl are not supported by the mps device which means you can only train on a single GPU with the MPS backend.
You can learn more about the MPS backend in the Introducing Accelerated PyTorch Training on Mac blog post.