Token Classification Models on (Very) Long Text
Hi everyone,From what I have seen, most token classification models out there have max token lengths less than 1k. Are there any models out there that can be used (i.e. customized) to be used with very long texts (long-form documents?Assuming a model’s max token length is customizable, I assume its memory footprint has to be light for it to be able to batch a large number of embeddings&weights in GPU memory?Any help/recommendation would be greatly appreciated in tackling this problem.Thanks!
2022-08-31T13:25:02Z
HI@nottakumasatoMost models have a 512 tokens limit and cannot extrapolate to longer sequences.Memory footprint also increases quadratically with sequence length because standard attention is O(n²).Best way to handle long sequences is to use a custom attention mecanism.You can try thisrepowith a small model and a small block size, you should be able to process 16k tokens sequences.
2022-08-31T15:43:57Z
The BART model goes up to 1024 tokens.Then there are models which can take up to 16k tokens but they’re more custom and not always available out of the box on HuggingFace. One of these is theLongformerfor example. Their model can be accessed via HuggingFace as shownhere. You may also want to take a look atthis recent paperfrom Google. It is a model specific for text generation (not exactly classification as you asked, but gives you an idea for what’s possible) and they have also made their code available (you can see more detailshereandhere- there is still an open PR which will be merged into the main HuggingFace branch soon, so right now you’d have to take their code from the fork)
2022-09-02T10:38:54Z
Thank you for the replies@ccdv&@AndreaSottana!So I guess there are two ways to tackle this:Split up the input text into segments that are less than the model’s max sequence (token) lengthFind a model like Pegasus-X or Longformer that can handle all the samples (based on their sequence length) in my datasetOption#1seems more plausible and will give it a try.Thanks!
2022-09-20T01:33:49Z
Is it also possible to use RNN (non-Transformer) based models? I assume the tradeoff is model “accuracy” vs the max sequence length?
2022-09-20T03:20:16Z
Best way to handle long sequences is to use a custom attention mecanism.Is there a specific reason that you didn’t recommend using earlier RNN-based models? Since they don’t have an attention mechanism, their memory footprint should theoretically be linear to the sequence lenght, right?
2022-09-20T03:21:25Z
You can try thisrepoIs there a paper about this LSG attention mechanism? Looks interesting and any further info would be appreciated to understand it a bit more.
2023-03-07T06:22:53Z
Hey did you try the paper if possible can you share us the results of how did it work
2023-03-09T17:30:55Z
Unfortunately not