6.7m (6,700,128) param GPT-J model.
```
n_positions - 128
n_embd - 64
n_layer - 4
n_head - 8
rotary_dim - 64
tokenizer - gpt-j
```

First, trained on 4,194,304 samples from the [c4](https://hf.co/datasets/c4) dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048. A cosine learning rate schedule was used starting at 1e-3.
{"language": ["en"], "license": "bigscience-bloom-rail-1.0", "library_name": "transformers", "tags": ["causal-lm", "gpt-j"], "datasets": ["c4"]}
transformers pytorch safetensors gptj text-generation causal-lm gpt-j en dataset:c4 license:bigscience-bloom-rail-1.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #gptj #text-generation #causal-lm #gpt-j #en #dataset-c4 #license-bigscience-bloom-rail-1.0 #autotrain_compatible #endpoints_compatible #region-us 

6.7m (6,700,128) param GPT-J model.


First, trained on 4,194,304 samples from the c4 dataset, at a length of 128 tokens each, that comes out to 536,870,912 (0.53B) tokens seen during training. A batch size of 16 with 128 gradient accumulation steps was used, making the effective batch size 2048. A cosine learning rate schedule was used starting at 1e-3.
TAGS
#transformers #pytorch #safetensors #gptj #text-generation #causal-lm #gpt-j #en #dataset-c4 #license-bigscience-bloom-rail-1.0 #autotrain_compatible #endpoints_compatible #region-us 

TAGS
#transformers #pytorch #safetensors #gptj #text-generation #causal-lm #gpt-j #en #dataset-c4 #license-bigscience-bloom-rail-1.0 #autotrain_compatible #endpoints_compatible #region-us 
