

"attention": The names of the attention layers to fuse in the following order: query, key, value and output projection layer. If you don't want to fuse these layers, pass an empty list.
"layernorm": The names of all the LayerNorm layers you want to replace with a custom fused LayerNorm. If you don't want to fuse these layers, pass an empty list.
"mlp": The names of the MLP layers you want to fuse into a single MLP layer in the order: (gate (dense, layer, post-attention) / up / down layers).
"use_alibi": If your model uses ALiBi positional embedding.
"num_attention_heads": The number of attention heads.
"num_key_value_heads": The number of key value heads that should be used to implement Grouped Query Attention (GQA). If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA), otherwise GQA is used.
"hidden_size": The dimension of the hidden representations.