Using transformers (BERT, RoBERTa) without embedding layer
I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different.In the language of proteins, I have 20 characters instead of the normal 26 characters used in english (it is 26 right? :D), so that is rather similar. The big difference is that you don’t really combine the characters in proteins to actual words, but rather just keep each character as a distinct token or class.Hence essentially my input to the transformer model could just be a list of numbers ranging from 0-19. However that would mean that my input would only have 1 feature if I did that, and I’m not sure a transformer could work with that?I’m thinking of just doing a onehot encoding of these characters, which would give me 20 input features. However this is of course still very low in comparison to how normal transformers are trained, where d_model is somewhere in the range of 128-512 if I understand correctly.Does anyone have any experience with anything like this? any good advice for how it is most likely to work?
2020-12-13T21:34:43Z
Hey,I’d recommend taking a look at this repo:GitHub - agemagician/CodeTrans: Pretrained Language Models for Source codeby@agemagician. This repo uses transformer models for protein sequences if I understand it correctly.Also, taking a look at those models:huggingface.coRostlab (Rostlab)We’re on a journey to advance and democratize artificial intelligence through open source and open science.might help. Not sure if there is a notebook on doing protein sequence LM, maybe@agemagicianhas a good pointer by chance
2020-12-13T22:00:24Z
Hi@tueboesen,Yes, it will work. It can give you a very close results compared to MSA methods, sometimes even better results. If you combine it with MSA, it will even give you a better results compared to MSA methods alone.We have trained (Transformer XL, XLNet, Bert, Albert, Electra and T5) for Uniref100 and BFD dataset. I would recommend to simply use on of these models, because it requires tremendous amount of computing power to reach good results.You can find them here:GitHubGitHub - agemagician/ProtTrans: ProtTrans is providing state of the art...ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models. - GitH...huggingface.coRostlab (Rostlab)We’re on a journey to advance and democratize artificial intelligence through open source and open science.You can find more details on our paper:bioRxiv – 21 Jul 20ProtTrans: Towards Cracking the Language of Life’s Code Through...Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we...Facebook also trained Roberta using Unrief50 dataset:GitHubGitHub - facebookresearch/esm: Evolutionary Scale Modeling (esm): Pretrained...Evolutionary Scale Modeling (esm): Pretrained language models for proteins - GitHub - facebookresearch/esm: Evolutionary Scale Modeling (esm): Pretrained language models for proteinsUnfortunately, we don’t have a notebook for training from scratch, but you can find more details to replicate our results here:github.com/agemagician/ProtTransSource code of the modelsopened11:49PM - 10 Oct 20 UTCclosed08:01PM - 11 Oct 20 UTChadimquestionDo you have the source code of the various pre-trained models?@patrickvonplaten:You meant :GitHubGitHub - agemagician/ProtTrans: ProtTrans is providing state of the art...ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models. - GitH...Not :GitHubGitHub - agemagician/CodeTrans: Pretrained Language Models for Source codePretrained Language Models for Source code. Contribute to agemagician/CodeTrans development by creating an account on GitHub.ProtTrans: Provides the SOT pre-trained models for protein sequences.CodeTrans: Provides the SOTpre-trained models for computer source code.
2020-12-16T15:35:15Z
agemagician:gy and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontieWow this is an amazing response, thank you so much for this. I will need some time to digest it all, but this is exactly what I need!
2020-12-16T16:21:56Z
Is there a way for me to use any of the models to return probability distributions?More specifically I would like to see how exactly the model has learned and test it out a bit. To this effect I would love to be able to feed it a protein sequence where I have masked out some of the amino acids, and then have it return a probability distribution for the full returned protein.I’m sure this is possible, after all this is how the model was trained in the first place, but I’m just a bit overwhelmed by all the models, so I haven’t managed to figure out how to do this.
2020-12-16T17:37:43Z
You can find an answer to your question here:https://github.com/agemagician/ProtTrans/issues/5
2020-12-16T19:19:29Z
Hmm that still doesn’t quite do it unless I’m missing something.This does allow masking of a sequence, but you can only mask 1 amino acid in the sequence, and it doesn’t give the actual probabilities on output, but only the top5 probabilities for that single masked amino acid.
2020-12-16T19:55:38Z
You can send “top_k” parameter to “fill-mask” method, to return more/all tokens.Check here:github.comhuggingface/transformers/blob/1c1a2ffbff2052100053cddb3a87d45fb9d210ca/src/transformers/pipelines.py#L1184"""def __init__(self,model: Union["PreTrainedModel", "TFPreTrainedModel"],tokenizer: PreTrainedTokenizer,modelcard: Optional[ModelCard] = None,framework: Optional[str] = None,args_parser: ArgumentHandler = None,device: int = -1,top_k=5,task: str = "",):super().__init__(model=model,tokenizer=tokenizer,modelcard=modelcard,framework=framework,args_parser=args_parser,device=device,binary_output=True,If it is still doesn’t fit your use-case, then you have to implement it your self.
2020-12-16T21:00:52Z
Something like that could be a good starting point for you:colab.research.google.comGoogle Colaboratory