(x)
return x * cdf
Implementation of the gelu activation function.
        XLNet is using OpenAI GPT's gelu (not exactly the same as BERT)
        Also see https://arxiv.org/abs/1606.08415
Implementation of the gelu activation function.
        XLNet is using OpenAI GPT's gelu (not exactly the same as BERT)
        Also see https://arxiv.org/abs/1606.08415
Implementation of the gelu activation function . XLNet is using OpenAI GPT s gelu ( not exactly the same as BERT ) Also see https : // arxiv . org / abs / 1606 . 08415
def gelu(x):
    """ Implementation of the gelu activation function.
        XLNet is using OpenAI GPT's gelu (not exactly the same as BERT)
        Also see https://arxiv.org/abs/1606.08415
    """
    cdf = 0.5 * (1.0 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
    return x * cdf
def gelu ( x ) : cdf = 0.5 * ( 1.0 + torch . tanh ( math . sqrt ( 2 / math . pi ) * ( x + 0.044715 * torch . pow ( x , 3 ) ) ) ) return x * cdf