SwitchTransformers uses the [T5Tokenizer], which can be loaded directly from each model's repository.
The released weights are pretrained on English Masked Language Modeling task, and should be finetuned.

Resources

Translation task guide
Summarization task guide