
# Model description

paper: [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://arxiv.org/abs/2210.13569)

This is a gpt2-small-like decoder-only transformer model trained on a the [wikitext-103 dataset](https://paperswithcode.com/dataset/wikitext-103).

# Usage

You can download and load the model as follows:

```python
from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained("Kristijan/gpt2_wt103_12-layer")

```

Alternatively, if you've downloaded the checkpoint files in this repository, you could also do:

```python
from transformers import GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained(path_to_folder_with_checkpoint_files)

```

## BPE Tokenizer

You should first pretokenize your text using the [MosesTokenizer](https://pypi.org/project/mosestokenizer/):

```python
from mosestokenizer import MosesTokenizer

with MosesTokenizer('en') as pretokenize:
    pretokenized_text = " ".join(pretokenize(text_string))
```

Then, to BPE tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-103_tokenizer_v2):

```python
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("Kristijan/wikitext-103-tokenizer_v2")
tokenized_text = tokenizer.tokenize(pretokenized_text)

```

# Intended uses

This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.
{"language": ["en"], "library_name": "pytorch", "tags": ["language-model", "gpt2", "transformer", "wikitext-103"], "model-index": [{"name": "gpt2_wt103-40m_12-layer", "results": [{"task": {"type": "language-modeling"}, "dataset": {"name": "Wikitext-103", "type": "wikitext"}, "metrics": [{"type": "perplexity", "value": 40.6}]}]}]}
pytorch gpt2 language-model transformer wikitext-103 en arxiv:2210.13569 model-index region:us
TAGS
#pytorch #gpt2 #language-model #transformer #wikitext-103 #en #arxiv-2210.13569 #model-index #region-us 


# Model description

paper: Characterizing Verbatim Short-Term Memory in Neural Language Models

This is a gpt2-small-like decoder-only transformer model trained on a the wikitext-103 dataset.

# Usage

You can download and load the model as follows:



Alternatively, if you've downloaded the checkpoint files in this repository, you could also do:



## BPE Tokenizer

You should first pretokenize your text using the MosesTokenizer:



Then, to BPE tokenize your text for this model, you should use the tokenizer trained on Wikitext-103:



# Intended uses

This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.
# Model description

paper: Characterizing Verbatim Short-Term Memory in Neural Language Models

This is a gpt2-small-like decoder-only transformer model trained on a the wikitext-103 dataset. # Usage

You can download and load the model as follows:



Alternatively, if you've downloaded the checkpoint files in this repository, you could also do: ## BPE Tokenizer

You should first pretokenize your text using the MosesTokenizer:



Then, to BPE tokenize your text for this model, you should use the tokenizer trained on Wikitext-103: # Intended uses

This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.
TAGS
#pytorch #gpt2 #language-model #transformer #wikitext-103 #en #arxiv-2210.13569 #model-index #region-us 
 # Model description

paper: Characterizing Verbatim Short-Term Memory in Neural Language Models

This is a gpt2-small-like decoder-only transformer model trained on a the wikitext-103 dataset. # Usage

You can download and load the model as follows:



Alternatively, if you've downloaded the checkpoint files in this repository, you could also do: ## BPE Tokenizer

You should first pretokenize your text using the MosesTokenizer:



Then, to BPE tokenize your text for this model, you should use the tokenizer trained on Wikitext-103: # Intended uses

This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.
TAGS
#pytorch #gpt2 #language-model #transformer #wikitext-103 #en #arxiv-2210.13569 #model-index #region-us 
# Model description

paper: Characterizing Verbatim Short-Term Memory in Neural Language Models

This is a gpt2-small-like decoder-only transformer model trained on a the wikitext-103 dataset.# Usage

You can download and load the model as follows:



Alternatively, if you've downloaded the checkpoint files in this repository, you could also do:## BPE Tokenizer

You should first pretokenize your text using the MosesTokenizer:



Then, to BPE tokenize your text for this model, you should use the tokenizer trained on Wikitext-103:# Intended uses

This checkpoint is intended for research purposes, for example those interested in studying the behavior of transformer language models trained on smaller datasets.