Functorch with transformers
I want to accelerate per-sample gradient computations withfunctorch. How do we compile models from transformers to use with functorch?Thanks
2023-12-19T20:18:03Z
Hi@jpcorb20, were you able to solve this issue? I’m trying to do the same right now, getting batched per-sample gradients using functorch (for BERT, ViT, and ResNet)
2023-12-21T16:40:12Z
@pkadambihello, unfortunately no, I ended up recomputing per-sample gradients on the side in pure torch, which is far from efficient but my research only required the actual numbers in the end …