Running inference on tuned models across multiple gpus
Im having a tough time running my tuned model across multiple gpusI have various pt files that i tuned with torchtune. I can inference with their generate function on lora but not full precision as one of my cards cant hold the whole model. Wondering the right approach to do this I have tried various methods but am struggling>REDACTED_TOKEN_0001_2.ptREDACTED_TOKEN_0002_2.ptREDACTED_TOKEN_0003_2.ptREDACTED_TOKEN_0004_2.pt