text
A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers. 
A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers. 
new_line
new_line
text
Key points:
Key points:
new_line
text
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
new_line
text
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
new_line
text
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.
new_line
new_line
text
Congrats to the authors for their work!
Congrats to the authors for their work!
new_line
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2403.03853
paper
2403.03853
https://huggingface.co/papers/2403.03853
ShortGPT: Layers in Large Language Models are More Redundant Than You
  Expect (2403.03853)
new_line
new_line
A recent paper titled "ShortGPT: Layers in Large Language Models are More Redundant Than You Expect" proposes a simple and effective approach to pruning Large Language Models (LLMs) by removing redundant layers. 

Key points:
* Discovers significant redundancy across layers in LLMs, with some layers playing a negligible role for the final performance.
* Defines a new metric called Block Influence (BI) to quantify the importance of each layer in an LLM.
* Removes layers with low BI scores, achieving up to 25% reduction in parameters and computation while maintaining 92% of the LLM's performance.

Congrats to the authors for their work!

Paper: https://huggingface.co/papers/2403.03853


👍
Dlbk emran86 dev7halo MexIvanov Pretam
❤️
emran86