Sin2pi commited on
Commit
5d32abc
·
verified ·
1 Parent(s): e3cfe0c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -0
README.md CHANGED
@@ -9,6 +9,7 @@ Updated. Was having some issues there with the hybrid attention and tensor shari
9
 
10
  Drop-in enhanced givens rotary block -- Its like a rubiks cube of embbedings :)
11
 
 
12
 
13
  class CombinedRotaryEmbedding(nn.Module):
14
  def __init__(self, n_state, n_head, num_rotations, base=10000, checkpointing=False):
 
9
 
10
  Drop-in enhanced givens rotary block -- Its like a rubiks cube of embbedings :)
11
 
12
+ Think of regular rope embeddings as a rotating 3d block in space.. Now add columns and rows that rotate and then rotate the faces of each resulting cube :
13
 
14
  class CombinedRotaryEmbedding(nn.Module):
15
  def __init__(self, n_state, n_head, num_rotations, base=10000, checkpointing=False):