Why is there no cross-gpu negative sample gathering for CLIP model in multiple-gpu training?
Hello, I notice that in transformers code, when training CLIP, negative samples are not gathered across all devices in multiple-gpu training scenario. I think gathering all negative samples across all devices is what the original paper does. Why is there no such implementation?
2024-03-17T17:31:43Z
Hi,The CLIP training script is mainly meant for demonstration purposes but is not optimized for multi-GPU setups.I’d recommend taking a look atOpenCLIPwhich provides this functionality.
2024-03-18T02:15:40Z
Thanks! I will have a look at that.