

Wav2Vec2-Conformer follows the same architecture as Wav2Vec2, but replaces the Attention-block with a Conformer-block
  as introduced in Conformer: Convolution-augmented Transformer for Speech Recognition.
For the same number of layers, Wav2Vec2-Conformer requires more parameters than Wav2Vec2, but also yields 
an improved word error rate.
Wav2Vec2-Conformer uses the same tokenizer and feature extractor as Wav2Vec2.
Wav2Vec2-Conformer can use either no relative position embeddings, Transformer-XL-like position embeddings, or
  rotary position embeddings by setting the correct config.position_embeddings_type.