Inference without gradient computation?
Hi all,I have some very inefficient code that just runs once, so it doesn’t matter to me for now. It’s just inference of about 14k data points.My code looks like that:tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
# loop over dataset:
inputs = tokenizer(preprocess_sentence(train_data['notes'][i]), return_tensors="pt", max_length=510)
sentence_vector = model(**inputs).pooler_outputI recognized that I run out of memory very quickly and put the inference part into awith torch.no_grad():block, which helps.Is it generally the case like in all other Pytorch models to run the inference withmodel.eval()andwith torch.no_grad():? I haven’t found any reference in the Transformers lib docs about that and thought it might be really useful for newbies.
2023-01-04T14:43:38Z
I happened to find my own question again by searching for some similar question.To all of you that come here for an answer.Yes, for inference you should set the model top eval mode withmodel.eval(), andYes, if you have resource issues or want to keep them low also gowith torch.no_grad():Example code:# set model to eval mode
model.eval()
inputs = tokenizer(…)

# set torch to no grad to save resources
with torch.no_grad():
    outputs = model(**inputs)
…