Train T5 model for commit message generation
T5 for commit message generationCommit message generation is known to be a difficult task due to the following reasons:No available clean and large dataset for pre-training and fine-tuning(Almost) No available pre-trained seq2seq models on both programming and natural languages corporaContext limitations (sequence length, external knowledge, etc.)In the recent research [1] [2] [3] it has been shown that pre-trained transformer models (like T5 or BART) can capture code understanding domain and fine-tune for various seq2seq programming downstream tasks (e.g. code summarization, code documentation generation, commit message generation, etc).Nevertheless, each of the listed downstream tasks deserves more thorough research. I’ve chosen commit message generation to be such a task because it has many practical applications, such as auto version control systems.To wrap up, these are the main goals of this research project:Release a publicly available dataset for programming language models pre-trainingRelease a publicly available dataset for commit message generation fine-tuningRelease a publicly available (pre)-trained T5 model for commit message generationRelease all pre-processing, post-processing, and training scripts for further research2. LanguageWe’ll use English and Python languages for training.This research is limited in time, so it would be great to show a strong baseline for at least one pair of languages. Later we’ll be able to extend the model for more language pairs.3. ModelWe’ll be using a random T5 model similar tot5-basein configuration.4. DatasetsPossible links to publicly available datasets include:bigquery-public-data:github_reposC4Please note that the BigQuery GitHub dump needs to be cleaned up before being used for pre-training/fine-tuning. We should also consider using only those repos whose license allows us to do so.5. Training scriptsWe can make use ofrun_summarization_flax.pyto train the model.6. (Optional) ChallengesThe main challenges are to prepare the dataset and set up training tasks in Flax. Besides, further challenges include facing the short context sizes and transferring knowledge across domains. I suppose we could try using sparse attention blocks and techniques like RAG for bringing the domain knowledge.7. (Optional) Desired project outcomeThe desired project outcome is to achieve the listed goals and have fun! A demo would be a simple application (e.g., VSCode extension) that generates commit messages given the changed python files.8. (Optional) ReadsThe following links can be useful to better understand the project andwhat has previously been done in the research community.Contextualized Code Representation Learning for Commit MessageGenerationA Transformer-based Approach for Source Code SummarizationPYMT5: multi-mode translation of natural language and PYTHON code with transformersUnsupervised Translation of Programming Languages
2021-06-24T11:45:41Z
This idea is definitely quiteinterestingI would love to join in and be a part of thisI have worked on Seq-2-Seq tasks like Machine Translation on low-resource languages, Character Transliteration for code-mixed conversations, Built and Trained Transformer models in both PyTorch as well as TensorFlow before. I have been meaning to find another seq-2-seq task to work on and this is justideal.
2021-06-24T14:44:25Z
Hey, thanks for being interested in the project! I’ll be super happy to see you on board!
2021-06-24T16:16:36Z
This seems very interesting can I please join this team?
2021-06-24T18:43:24Z
Hey! Sure, welcome on board!
2021-06-24T18:46:40Z
Does anybody know TypeScript btw?Looking right now at VSCode Extensions API – it’s awesome! I’m sure we’ll be able to integrate the model into the commit message generation pipeline within an extension.
2021-06-25T08:44:42Z
I’m very interested and would like to join as well
2021-06-25T11:41:05Z
There are already T5 models that was trained for this task:huggingface.coModels - Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.This repo should help you to extend the pretrained CodeTrans models for more tasks or languages:GitHubGitHub - agemagician/CodeTrans: Pretrained Language Models for Source codePretrained Language Models for Source code. Contribute to agemagician/CodeTrans development by creating an account on GitHub.Good luck.
2021-06-25T17:25:36Z
That’s a great project description - excited about the results!
2021-06-25T18:07:38Z
Hey@agemagician!Thanks for pointing out great resources for this task.I noticed that the paper CodeTrans as well as the Git Commit Message Generator model available on HuggingFace Hub is only trained forJava.I think since Java and Python have fundamentally different languages with different synatctical structure and semantics, fine-tuning that model may not yeild the best results. Though I would not mind trying at all and comparing the results, if we have time.
2021-06-25T20:02:27Z
ProtTrans actually has models which were trained on 9 programming languages + English language using self-supervised learning.You can easily fine-tune them on any task and almost all famous programming languages.For example:huggingface.coSEBIS/code_trans_t5_large_transfer_learning_pretrain · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coSEBIS/code_trans_t5_small_transfer_learning_pretrain · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coSEBIS/code_trans_t5_base_transfer_learning_pretrain · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2021-06-25T21:44:51Z
Thanks,@patrickvonplaten! Shall we consider forming two teams under this project? We have both pre-training and fine-tuning research experiments that can be parallelized across the teams.
2021-06-25T22:19:36Z
Hey@agemagician,CodeTrans is a great research project! Our research is highly backed by their results. Anyway, we still have space for improving their ideas to make pre-trained models stronger domain learners.Besides, we’re missing publicly available high-quality data and benchmarks for us and other researchers to improve upon existing ideas and bring new ones.We believe that the HuggingFace ecosystem is a perfect place to make research open for all, and hopefully, our research can enforce these values further.Regards,Eugene
2021-06-29T14:17:27Z
Very exciting project - putting it downSince one team is good here! Since you are more than 5, we might have 2 TPUs for you!
2021-06-29T16:08:51Z
Amazing news, thank you! Just a quick question – can we add more participants since we have a few more slots available?
2021-06-29T17:51:52Z
Hi, I would like to join this team if spots are still available! I have some experience in Software Engineering and GitHub APIs that may help
2021-06-30T08:32:25Z
Hey, join us on ourDiscord channel!
2021-06-30T12:35:49Z
Yes! This is fine
2021-07-01T10:01:04Z
added you
2021-07-02T11:03:47Z
Hey@jxuhf, welcome on board! Please, join our Discord channel (#t5-commit-message).