[DistilBertConfig] displays all the default attributes used to build a base [DistilBertModel]. All attributes are customizable, creating space for experimentation. For example, you can customize a default model to:

Try a different activation function with the activation parameter.
Use a higher dropout ratio for the attention probabilities with the attention_dropout parameter.