Easy way to implement annealing temperature softmax
Hi friends,I wonder if there’s a easy way to implement annealing temperature softmax:For example, I want to change314in modeling_bert.py fromattention_probs = nn.Softmax(dim=-1)(attention_scores)toattention_probs = nn.Softmax(dim=-1)(attention_scores/temp)where “temp” is a variable decaying as the training process goes on according to a scheduler like the learning rate.Thank you!
2021-01-04T13:01:01Z
You can just do that changeModel files are kept completely independent from each other just so you can easily tweak them for experiments like this.