Pretrained Models for Inferencing not using gpu
#Themodel codemodel_path = “martin-ha/toxic-comment-model”tokenizer = AutoTokenizer.from_pretrained(model_path)model = AutoModelForSequenceClassification.from_pretrained(model_path)model.to(device)#Thedataset codeclass GetData2(Dataset):definit(self, texts, tokenizer, max_length):self.texts = textsself.tokenizer = tokenizerself.max_length = max_lengthdef __len__(self):
    return len(self.texts)

def __getitem__(self, idx):
    encoded_text = self.tokenizer.encode_plus(
        self.texts[idx],
        add_special_tokens=True,
        max_length=self.max_length,
        padding='max_length',
        return_tensors="pt",
        truncation=True
    )
    return {
        "input_ids": encoded_text["input_ids"].squeeze(),
        "attention_mask": encoded_text["attention_mask"].squeeze()
    }#Creationof the Datasettest_2 = GetData2(list(df_pd[‘body’]), tokenizer, 512)Creation of the dataloaderCreate a DataLoader and move data to GPUdataloader = DataLoader(test_2, batch_size=128, shuffle=False)Perform inference on GPUmodel.eval()predictions =with torch.no_grad():for batch in dataloader:batch = {key: value.to(device) for key, value in batch.items()}outputs = model(**batch)probabilities = torch.softmax(outputs.logits, dim=1)predictions.extend(probabilities.cpu().numpy())And this is the task manager pic:Task_manager1920×917 83.4 KBI have tried almost everything but for some reason this does not work. I have tried a different code where i create my own pytorch model and train and test it using gpu. It works over there and it is in the same environment. I do not want to fine-tune, i just want to use the pretained model on my gpu so that the predicitons can be faster. Any help would be appreciated. Thank you.