Answer the following expert-level machine learning question.
Background:
Large language models, such as GPT-3, typically use Byte-Pair Encoding (BPE) as tokenizers, resulting in a fixed vocabulary size of around 50,000 tokens (source: GPT-2 paper). However, the embedding dimension of these large language models keeps growing, from 512 in the original Transformer model (2017), to 1,600 in the GPT-2 model (2019), then to 12,288 in the GPT-3 model (2020), and seems to strongly correlate with model performance.
Question:

1. What are the benefits of using a super high dimensional space (e.g. dimensionality > 10k) to embed 50k tokens?
2. Can you explain NAME_1–Lindenstrauss lemma to me? Does the observed phenomenon of ever-increasing embedding dimension contradict NAME_1–Lindenstrauss lemma?
3. Given enough training data, computing power and storage access, will it be beneficial to use even higher embedding dimensions (e.g. embedding dimensionality larger than vocabulary size, say 100k vs. 50k) in large language models?