How to specify sequence length when using "feature-extraction"
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast("./vocab.txt",
        unk_token="<unk>",
        sep_token="</s>",
        cls_token="<s>",
        pad_token="<pad>",
        mask_token="[MASK]"
)
features = pipeline(
        "feature-extraction",
        model="./model",
        tokenizer=tokenizer
        )
search_features = features(text)#IndexError:index out of range in self
2021-04-27T13:08:22Z
what happens if you specifymodel_max_len=512when you load the tokenizer? i’d try that and do a sanity check withtokenizer(text)to make sure the truncation is working as expected
2021-04-27T22:12:38Z
lewtun:model_max_len=512I’m getting the following error: IndexError: index out of range in self
2021-04-28T09:42:31Z
ah indeed it seems that no truncation is enabled in the basePipelineclass:transformers/base.py at 8d43c71a1ca3ad322cc45008eb66a5611f1e017e · huggingface/transformers · GitHubone alternative would be to extract the features directly from the model as described in this thread:Truncating sequence -- within a pipelinethis way you can enforcetruncation=Truewith your tokenizer and pass the truncated inputs to the model