Pipeline very slow
I just trained aBertForSequenceClassificationclassifier but come on problems when trying to predict.When I use thepredictmethod of trainer on encodings I precomputed, I’m able to obtain predictions for ~350 samples from test set in less than 20 seconds.trainer.predict(test_encodings)However, when I load the model from storage and use a pipeline, the code runs for more than 10 mins, even adding batching doesn’t seem to help.classifier = pipeline("text-classification", model=model,tokenizer=tokenizer,max_length=512, truncation=True)
classifier(X_test.Text.to_list(),batch_size=10)What can explain this difference ?
2023-05-05T13:52:15Z
Can’t offer any solutions, but I’m experiencing a similar problem training a model in the HuggingFace hub and trying to load it with thefrom_pretrainedmethod to make predictions locally. I’m averaging 6s / observation - not something I can ever use in production.ThisSO postreports basically the same issue withfrom_pretrained…