

TITLE
Flaky Whisper PT-TF & PT-Flax Equivalence Test

COMMENTS
1

REACTIONS
+1: 2
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

transformers 4.29.0 dev

### Who can help?

@ArthurZucker @sanchit-gandhi @ydshieh 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Flaky test, so not reproducible. Example run where the error occurred: 

* https://app.circleci.com/pipelines/github/huggingface/transformers/64100/workflows/b4463c5d-b3dc-4b00-a7cd-19acd096cb07/jobs/792381
* https://app.circleci.com/pipelines/github/huggingface/transformers/64111/workflows/dc9092c4-0673-46c7-b89f-f805bc20128c/jobs/792557
* https://app.circleci.com/pipelines/github/huggingface/transformers/64230/workflows/e2d42ca4-f367-4a85-9054-a0ea99e49849/jobs/794534

Occasionally, the PT-TF and PT-Flax whisper equivalence test fails. The tolerance was increased in #23257 and #23288 but the reason for recent failures has not yet been found. 
 
### Expected behaviour

Equivalence tests reliably pass. 