Question on byte pair encoding (chapter 6.5)
Hi! After having read thepageon Byte Pair Encoding, I thought I understood the idea but I’m confused by the list of merges of the Mistral-7B tokenizer… Inhttps://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2/raw/main/tokenizer.json, I see that the first merges listed are:0: ▁ t
 1: i n
 2: e r
 3: ▁ a
 4: h e
 5: o n
 6: r e
 7: ▁ s
 8: e n
 9: a t
10: o r
11: ▁t he
12: ▁th e
13: ▁ the
14: e s
15: ▁ w
16: a n
17: ▁ c
18: i s
19: i t
20: o u
21: ▁ d
22: a l
23: a r
24: ▁ p
25: ▁ f
26: e d
27: ▁ b
28: in g
29: i ngWhat I don’t understand is how merge#12can ever be useful because the_thtoken cannot have been created at this point of the encoding process (there are no_ thor_t hmerges before#12). It’s the same for#13and#29. I count that 10312 merges (out of 58980 merges in total) are in this situation.Can someone please help me understand?