How to use T5 for sentence embedding?
is there any way to use encoder part of T5 model for representation learning?
2020-09-12T13:11:28Z
Hi@banucoolYou can initialize theT5Modelclass and only forward pass through it’s encoder. The first element of the returned tuple is the final hidden states.model = T5Model.from_pretrained("t5-small")
tok = T5Tokenizer.from_pretrained("t5-small")

enc = tok("some text", return_tensors="pt")

# forward pass through encoder only
output = model.encoder(
    input_ids=enc["input_ids"], 
    attention_mask=enc["attention_mask"], 
    return_dict=True
)
# get the final hidden states
emb = output.last_hidden_stateThe shape ofembwill be(batch_size, seq_len, hidden_size)
2020-09-12T14:06:41Z
thanks a lot@valhalla
2020-09-12T15:33:12Z
can we use pruned version of bert for feature extraction?does it make sense?
2020-09-12T18:08:55Z
To clarify, the above code just returns the final hidden state of each token and not whole sentence embedding.for sentence embedding you can trysentence-bert.https://huggingface.co/sentence-transformers
2022-05-31T17:01:27Z
valhalla:model = T5Model.from_pretrained("t5-small")
tok = T5Tokenizer.from_pretrained("t5-small")

enc = tok("some text", return_tensors="pt")

# forward pass through encoder only
output = model.encoder(
    input_ids=enc["input_ids"], 
    attention_mask=enc["attention_mask"], 
    return_dict=True
)
# get the final hidden states
emb = output.last_hidden_stateHi, I’m interested in using T5 to generate word embeddings. I tried the code supplied above. Unfortunately, got this error message:---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-40-5f6e22d1ad1e> in <module>()
      1 model = T5Model.from_pretrained("t5-small")
----> 2 tok = T5Tokenizer.from_pretrained("t5-small")
      3 
      4 enc = tok("some text", return_tensors="pt")
      5 

TypeError: 'NoneType' object is not callableDo you have any thoughts on resolving this error message?Thank you in advance for your help.
2023-05-27T02:25:02Z
arXiv.orgSentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text ModelsWe provide the first exploration of sentence embeddings from text-to-text
transformers (T5). Sentence embeddings are broadly useful for language
processing tasks. While T5 achieves impressive performance on language tasks
cast as sequence-to-sequence...