Juliushanhanhan
commited on
Update README.md
Browse files
README.md
CHANGED
@@ -2,7 +2,13 @@
|
|
2 |
library_name: saelens
|
3 |
---
|
4 |
|
5 |
-
# SAEs
|
|
|
|
|
|
|
|
|
|
|
|
|
6 |
|
7 |
This repository contains the following SAEs:
|
8 |
- blocks.25.hook_resid_post
|
@@ -12,4 +18,4 @@ Load these SAEs using SAELens as below:
|
|
12 |
from sae_lens import SAE
|
13 |
|
14 |
sae, cfg_dict, sparsity = SAE.from_pretrained("Juliushanhanhan/llama-3-8b-it-res", "<sae_id>")
|
15 |
-
```
|
|
|
2 |
library_name: saelens
|
3 |
---
|
4 |
|
5 |
+
# Llama-3-8B SAEs (layer 25, Post-MLP Residual Stream)
|
6 |
+
|
7 |
+
We train a Gated SAE on the post-MLP residual stream of the 25th layer of [Llama-3-8b-instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) model. The width of SAE hidden dimensions is 65536 (x16).
|
8 |
+
|
9 |
+
The SAE is trained with 500M tokens from the [OpenWebText corpus](https://huggingface.co/datasets/Juliushanhanhan/openwebtext-1b-llama3-tokenized-cxt-1024).
|
10 |
+
|
11 |
+
Feature visualizations are hosted at https://www.neuronpedia.org/llama3-8b-it. The wandb run is recorded [here](https://wandb.ai/jiatongg/sae_semantic_entropy/runs/ruuu0izg?nw=nwuserjiatongg).
|
12 |
|
13 |
This repository contains the following SAEs:
|
14 |
- blocks.25.hook_resid_post
|
|
|
18 |
from sae_lens import SAE
|
19 |
|
20 |
sae, cfg_dict, sparsity = SAE.from_pretrained("Juliushanhanhan/llama-3-8b-it-res", "<sae_id>")
|
21 |
+
```
|