Fine-tuning GPT-J on Articles (Wikipedia)
HelloI’m using GPT-J (EleutherAI/gpt-j-6B) as a chatbot. To increase the knowledge of the model in specific areas, I would like to fine-tune it on specific data, such as Wikipedia articles (e.g., the Wikipedia page about Marie Curie). For fine-tuning on a TPU, I’m following the following guide:mesh-transformer-jax/howto_finetune.md at master · kingoflolz/mesh-transformer-jax · GitHubHow can GPT-J be fine-tuned on a Wikipedia article (or a collection of articles)? Can I just provide the plain text file of the Wikipedia article during fine-tuning?
2022-11-19T16:04:29Z
Does anybody have ideas? Any help is highly welcome.