Overhead caused by moving eos_token_id to gpu mem
Hi, I’m considering using transformers to real-time service. (call model.generate every seconds or more).But in greedy search, there’s a code line which move eos token to gpu mem and It takes 170ms.(A100GPU)The exact line of code iseos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else NoneinhereCost of 170ms in every 1s inference is quite large, and I think moving same token to gpu mem is inefficient.How can I avoid this?
2024-01-28T12:36:56Z
Pretty sure thateos_token_idis an integer here, not a torch tensor. The conversion from an integer to a list then to a torch tensor viatorch.tensor(eos_token_id)is the more likely reason to why that line is taking up quite some time. However, I think that the overhead should not be that significant (aka the ratio of time taken to compute the line you mentioned andmodel.generate()is too large).Based on my experimentations:@timer
def test1():
    eos_token_id_tensor = torch.tensor([model.generation_config.eos_token_id]).to(input_ids.device)

@timer
def test2():
    output = model.generate(input_ids.unsqueeze(0), max_new_tokens = 1)

test1()
test2()Output:Total execution time for test1: 1 ms
Total execution time for test2: 1029 msMy ratio is 1:1029, in comparison to your 170:1000. I think you should test it out again.
2024-01-29T06:35:20Z
Which GPU do you use? In my A100 GPU, following code returns 260ms of latency at first load() call.import torchfrom datetime import datetimedef load():start = datetime.now()token = [0]torch.tensor(token).to(“cuda:0”)end = datetime.now()print((end - start).total_seconds())returnfor i in range(5):load()
2024-01-29T08:57:07Z
For the initial performance overhead, it’s a known issue with PyTorch. It will save the result into the cache, which allows for the next few similar tensors to be created much faster. Unless you are runningmodel.generate()withmax_new_tokens =1, this overhead isn’t exactly noticeable as the tokens generated increases.I can also reproduce your problem on a T4 Google colab GPU, but technically speaking it isn’t a problem if you’re generating a lot of tokens, and the significant overhead will no longer be there when you runmodel.generate()for a second time.
2024-02-01T05:21:37Z
Yeah I agree with you comment but in my case, I have to runmodel.generate()every seconds or more maybe, since I’m considering real-time application with whisper.The model is served by pytorch and the overhead arises every run. This is the problem.
2024-02-01T06:40:00Z
Can you trymodel.generate()on randominput_idsfor a couple of iterations, and see whether the overhead remains even after the first iteration? I don’t exactly know how you are running your model, but do you also reset the cache/instance whenever you want to get an inference?
2024-02-01T09:34:24Z
Yeah I tried test code the way you explained, as below(code 1), but the result was same. Every first tensor movement to gpu costs 0.8s.I modified transformers code(at generation/utils.py) to l find latency(code 2)Code 1.model: WhisperForConditionalGeneration = WhisperForConditionalGeneration.from_pretrained(pretrained_model_name_or_path=model_folder_path).to("cuda:0")

with open(feature_path,'rb') as f:
    input_features = pickle.load(f)
audio_feature: torch.Tensor = torch.as_tensor(input_features).to("cuda:0")
for _ in range(5):
    model.generate(audio_feature)Code 2.eos_token_id_tensor_start = time.perf_counter()
eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None
eos_token_id_tensor_end = time.perf_counter()
2024-02-01T13:25:09Z
Could you try this method where you synchronize the GPU while timing the speed of the code via what was mentionedhere?Also, may I ask why are you usingpickle.loadfor your input features? What is the data type of the input features before converting it viatorch.as_tensor?
2024-02-02T06:21:14Z
There’s no specific purpose of usingpickel.loadin my test code, just for extracted audio feature from real data, instead of randomly generated tensor.The type of data is List[float], with shape of (1,80,3000).  Since request body of my server (fastapi) should be List[float],(request from feature-extraction server) I build test code with list type.BTW, I don’t understand what you mentioned above. Add torch.cuda.synchronize() in transformers code will improve performance, you mean?
2024-02-02T07:22:02Z
torch.cuda.synchronize()will not improve performance, but it does ensure that the timing that we have recorded is consistent. The function waits for all kernels in all CUDA streams to complete, which is why it may also introduce overhead during kernel launch.
2024-02-02T07:27:54Z
For code 2 (from transformers code), maybe you could convert the tensor to the correct device (cuda) directly viaeos_token_id_tensor = torch.tensor(eos_token_id, device = input_ids.device) if eos_token_id is not None else None? I don’t have a GPU with me right now so I can’t test this hypothesis out.
2024-02-02T07:31:25Z
I breakdown transformers code and got some findings.There’s 2 tensor process across cpu and gpu memory and thery are bottlenecks.First one is in _prepare_attention_mask_for_generation function, find whether int(cpu) in  tensor(gpu) → (pad_token_id in inputs) : 0.8s in RTX A2000 (laptop)The second one is what I mentioned above, move eos token to gpu. (0.8s)Strange thing is, latency in 2 gets low with next inference iterations but 3 stays the same.I guess gpu warmup works at 2 and encoding process (which is 1.3s at first try and decreased to 12ms. The exact code ismodel_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs)but at 3, it seems no improvement in next same processes.
2024-02-02T07:47:13Z
I noticed an issue similar to yours that has remained open at PyTorch’s repo:model->to(device) costs over a millisecond when doing nothing · Issue #23865 · pytorch/pytorch · GitHub
2024-02-06T08:00:15Z
Thank you for reply!As far as I understand, we can’t avoid overhead, am I right?I guess we should take that amount of latency in real-time inference scenario.
2024-02-07T02:37:32Z
Yep, seems to be the case as of now.