Multiple texts as inputs to Transformers models
I would like to use multiple texts as inputs to a model, let’s say I have a dataset with 10 columns each column is a text (sentence or two), how can I fit all these inputs to the model and do a classification for example ?I can see it’s possible to just concatenate all texts in one, but seems that for me, I need a very large data to be apple to achieve good accuracy.Maybe using multiple models (BERT) in parallel, taking last hidden state, concatenate them and classify ? But the problem is that there’s so many values order of 30 texts.Any idea how to tackle this ?
2020-11-26T07:15:01Z
You should take the same approach as Extractive text summarization :Concatenate all your sentences, separated with a special token (CLSfor example), then use theCLStoken representation to do classification.From thePresumm paperimage1533×484 100 KB
2020-11-26T11:47:05Z
Hi@astariul, thank you for your reply.I understand what you suggested, the problem is that I don’t have only texts as inputs I have also  some floats values, is converting this values to text would be sufficient ?
2020-11-26T23:33:50Z
I see…I never encountered this case myself, but maybe you can directly input the float values in the last classifier ?Since it’s not text, there is no need for BERT to encode it (?)
2020-12-07T22:18:18Z
Hi@ZackI don’t know whether you’ve tried / considered  the multimodal toolkit (blog post,github)- takes in tabular data (text, numbers, categorical data) and can use them as inputs to develop models.Haven’t tried it myself, but looks quite promising.
2023-07-11T15:10:49Z
Hello@astariul,Thank you for your answer. I am also trying to do something similar but I just had a question though. So, even after concatenating the data will the different input data be provided with different weights and biases in this case? or the whole input text itself will be provided with weights and biases?
2023-07-12T00:09:42Z
You can do both.In the case of BERT for summarization, they just use sentence-specific representation :[CLS] sen1 [SEP] [CLS] sen2 [SEP] [CLS] sen3 [SEP]Then use each CLS token as a representation of each sentence.But if you want a general representation for the whole text, you can just train your model with an additional token at the beginning :[CLS2] [CLS] sen1 [SEP] [CLS] sen2 [SEP] [CLS] sen3 [SEP]Then useCLSfor sentence representation,CLS2for token representation.
2023-07-25T12:41:20Z
Hello@astariul, Thank you for your reply again. I was just curious to know if I can build a model using huggingface transformers which can give me multiple outputs?
2023-07-25T13:52:11Z
Yes you can.Regular models have a single output (usually a LM head or a classification head) on top of the Transformer stack, but you can just add several different heads on top of the Transformer stack to get multiple outputs.
2024-09-13T11:50:32Z
@astariul, Hi I am also working on similar project. My use case has only text with multiple inputs and multiple outputs. If you know resources around it please help. Currently using hugging face’s trainer giving me better accuracy