Issue in the Documentation of transformers for BiET
Hello,I have been reading the documentation of Biet modelhere. In the sectionpooler outputthis is what is writtenLast layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for the BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretrainingAfter going through the code (here)[https://github.com/huggingface/transformers/blob/master/src/transformers/models/beit/modeling_beit.py#L666-L667], the pooler output is actually the mean of all hidden states and not a linear projection on theCLStoken.Is it possible to update the documentation as it creates confusion while going through it?Note: I thought of raising the issue in the GitHub repo but couldn’t find how to do it incase of documentation.
2021-10-21T19:32:27Z
Hi,Thanks for reporting. Indeed,BeitModelcurrently returns an output of typeBaseModelOutputWithPooling. This is a generic class that automatically generates the documentation for the model, definedhere. However, in this case, it might be better to define a customBeitModelOutput, that better describes the outputs of the model.Do you mind opening a PR for this? This would mean defining a new dataclass withinmodeling_beit.py.Otherwise, I’ll do it
2021-10-24T21:04:48Z
Hello,I raised a pull request here:Added Beit model ouput class by lumliolum · Pull Request #14133 · huggingface/transformers · GitHubCan you tell me why theCircleCI - check code qualitytest is failing?