Update README.md
Browse files
README.md
CHANGED
@@ -9,6 +9,7 @@ Updated. Was having some issues there with the hybrid attention and tensor shari
|
|
9 |
|
10 |
Drop-in enhanced givens rotary block -- Its like a rubiks cube of embbedings :)
|
11 |
|
|
|
12 |
|
13 |
class CombinedRotaryEmbedding(nn.Module):
|
14 |
def __init__(self, n_state, n_head, num_rotations, base=10000, checkpointing=False):
|
|
|
9 |
|
10 |
Drop-in enhanced givens rotary block -- Its like a rubiks cube of embbedings :)
|
11 |
|
12 |
+
Think of regular rope embeddings as a rotating 3d block in space.. Now add columns and rows that rotate and then rotate the faces of each resulting cube :
|
13 |
|
14 |
class CombinedRotaryEmbedding(nn.Module):
|
15 |
def __init__(self, n_state, n_head, num_rotations, base=10000, checkpointing=False):
|