
**(!) _Don't forget to preprocess unknown_tokens and substitute them with <|endoftext|>. Otherwise the \<unk\> tokens in dataset will be split into the '<', 'unk' and '>' tokens_**


- Full context (1024) perplexity on test set: **13.68**

**Dependence of the cross entropy loss on the length of the context for prediction**

- x-axis*128 = context length
- y-axis = cross entropy


![image/png](https://cdn-uploads.huggingface.co/production/uploads/63c1ac8cc58fcfeac186bda2/Dpc_d3buivfBd5_-A03Vb.png)


{"language": ["en"], "datasets": ["wikitext", "wikitext-103-v1"], "metrics": ["perplexity", "cross_entropy"]}
transformers pytorch gpt2 text-generation en dataset:wikitext dataset:wikitext-103-v1 autotrain_compatible endpoints_compatible text-generation-inference region:us