If we split the weight matrix A column-wise across N GPUs and perform matrix multiplications XA_1 through XA_n in parallel, 
then we will end up with N output vectors Y_1, Y_2, , Y_n which can be fed into GeLU independently: