How to modify Model Class with AutoModelForCausalLM.from_config
I’m working with deepspeed and transformers for distributed inference. The model is LLAMA-7B@FP16. I’m loading model withAutoModelForCausalLM.from_config.I want to profile decoding stage. So I have to addtorch.profileraround decoding and omit prefilling. What I suppose could achieve this is to modify funtions intransformers.generation.utilssuch assample()orgreedy_search(). But I don’t wanna to change source code oftransfomers.Any idea that I could addtorch.profileringreedy_search()? Or any other better method that I cound profiling decoding stage?