Is there any difference between GPT-J and GPT-2?
All i know is GPT-J is GPT-3 like model, and GPT-3 is bigger implementation of GPT-2 architecture. Then is there any technical difference between GPT-2 and GPT-J?
2023-03-31T12:30:52Z
A answer comes from new bing:GPT-J and GPT-2 are both transformer models that can generate text from a prompt. However, they have some differences in their size, architecture, training data and performance.GPT-J is a newer and larger model than GPT-2. It has6 billionparameters, while GPT-2 has1.5 billionparameters¹. GPT-J also usesRotary Position Embedding (RoPE)to encode the position of each token, while GPT-2 usessinusoidal position embedding².GPT-J was trained onthe Pile, a large-scale curated dataset created by EleutherAI, which contains diverse sources of text such as books, Wikipedia, GitHub and Reddit². GPT-2 was trained onWebText, a smaller dataset scraped from the internet by OpenAI, which mainly consists of news articles and blog posts³.GPT-J is claimed to approach GPT-3-level performance on some natural language processing tasks, such as arithmetic, logic, quantitative reasoning and pseudocode generation. GPT-2 is less capable of these tasks and somewhat underperforms equal-sized GPT-3 models¹.Source:(2) EleutherAI/gpt-j-6B · Hugging Face.EleutherAI/gpt-j-6B · Hugging FaceAccessed 2023/3/31.(3) EleutherAI claims new NLP model approaches GPT-3-level performance.EleutherAI claims new NLP model approaches GPT-3-level performance | VentureBeatAccessed 2023/3/31.
2023-09-06T03:37:58Z
while the above answer is helpful, I believe GPT-J usessinusoidal position embeddingas well according to hugging facehttps://github.com/huggingface/transformers/blob/172f42c512e1bf32554ef910fe82f07916b4d4af/src/transformers/models/gptj/modeling_gptj.py#L115