Convert pre-trained MHA weights to GQA weights
In LlamaConfig, the fieldnum_key_value_headsstated that: “When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group.” Is there any transformers implementation for this conversion?
2024-09-29T09:10:36Z
I also have this question now. Is there a solution?