Dynamic Programming for Byte-level BPE
Could anyone explain the rationale behind equation (1) inNeural Machine Translation with Byte-Level Subwords?Besides, what does it exactly mean byThe design of UTF-8 encoding ensures the uniqueness of this recovery process: for a character UTF-8 encoded with multiple bytes, its trailing bytes will not make a valid UTF-8 encoded character?How exactly are the hexadecimal digits being derived in Figure 1 ?