GPU is far slower than CPU for patch embedding
Hello guys!I am playing around with ViT inference speed. I have tested the cost time of embedding and encoding on CPU and GPU separately.The results, out of my expectation, are:time spent(ms)GPUCPUembedding3311encoding472details:GPU model = RTX 3090TiCPU model = Intel i9-12900KFPretrained model weights = google/vit-base-patch16-224-in21kI can understand that GPU is faster than CPU for encoding. But why CPU is faster than GPU for embedding since both embedding and encoding are some DL neural networks and do matrix multiplying operations?Thanks a lot!