jeffmeloy commited on
Commit
d113a30
·
verified ·
1 Parent(s): 3aa3288

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -0
README.md CHANGED
@@ -15,4 +15,13 @@ tags:
15
  Optimized Layer Merging (OLM)
16
  Is a transformer optimization framework implementing automated layer recombination.
17
 
 
 
 
 
 
 
 
 
 
18
  https://github.com/jeffmeloy/olm
 
15
  Optimized Layer Merging (OLM)
16
  Is a transformer optimization framework implementing automated layer recombination.
17
 
18
+ Olm create Frankenstein's monster out of language models by cherry-picking the best performing layers across different models to create a superior hybrid.
19
+ The core mechanism:
20
+
21
+ - Takes multiple language models as input
22
+ - Uses a base model as the foundation
23
+ - Iteratively replaces individual layers, evaluating performance on specified datasets
24
+ - Keeps the best performing layer at each position based on metrics like perplexity, exact match, and a custom "quality" score
25
+ - Builds a fusion model layer-by-layer while maintaining or improving performance
26
+
27
  https://github.com/jeffmeloy/olm