SAELens
Juliushanhanhan commited on
Commit
6ce0ac6
·
verified ·
1 Parent(s): e940bb4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -2
README.md CHANGED
@@ -2,7 +2,13 @@
2
  library_name: saelens
3
  ---
4
 
5
- # SAEs for use with the SAELens library
 
 
 
 
 
 
6
 
7
  This repository contains the following SAEs:
8
  - blocks.25.hook_resid_post
@@ -12,4 +18,4 @@ Load these SAEs using SAELens as below:
12
  from sae_lens import SAE
13
 
14
  sae, cfg_dict, sparsity = SAE.from_pretrained("Juliushanhanhan/llama-3-8b-it-res", "<sae_id>")
15
- ```
 
2
  library_name: saelens
3
  ---
4
 
5
+ # Llama-3-8B SAEs (layer 25, Post-MLP Residual Stream)
6
+
7
+ We train a Gated SAE on the post-MLP residual stream of the 25th layer of [Llama-3-8b-instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) model. The width of SAE hidden dimensions is 65536 (x16).
8
+
9
+ The SAE is trained with 500M tokens from the [OpenWebText corpus](https://huggingface.co/datasets/Juliushanhanhan/openwebtext-1b-llama3-tokenized-cxt-1024).
10
+
11
+ Feature visualizations are hosted at https://www.neuronpedia.org/llama3-8b-it. The wandb run is recorded [here](https://wandb.ai/jiatongg/sae_semantic_entropy/runs/ruuu0izg?nw=nwuserjiatongg).
12
 
13
  This repository contains the following SAEs:
14
  - blocks.25.hook_resid_post
 
18
  from sae_lens import SAE
19
 
20
  sae, cfg_dict, sparsity = SAE.from_pretrained("Juliushanhanhan/llama-3-8b-it-res", "<sae_id>")
21
+ ```