

CANINE uses no less than 3 Transformer encoders internally: 2 "shallow" encoders (which only consist of a single
  layer) and 1 "deep" encoder (which is a regular BERT encoder). First, a "shallow" encoder is used to contextualize
  the character embeddings, using local attention. Next, after downsampling, a "deep" encoder is applied. Finally,
  after upsampling, a "shallow" encoder is used to create the final character embeddings. Details regarding up- and
  downsampling can be found in the paper.
CANINE uses a max sequence length of 2048 characters by default. One can use [CanineTokenizer]
  to prepare text for the model.
Classification can be done by placing a linear layer on top of the final hidden state of the special [CLS] token
  (which has a predefined Unicode code point). For token classification tasks however, the downsampled sequence of
  tokens needs to be upsampled again to match the length of the original character sequence (which is 2048). The
  details for this can be found in the paper.