Transformers Huge Community Feedback
So last week we shared the first feedback request ontransformers. The community was pretty amazingly involved in this with close to 900 detailed feedback forms to analyze and dive into, containing more than 50 thousand words of open answersIn this post, I would like, first, to deeply thank the community for this amazing feedback and for the carefully crafted answers, people were so keen to share. It’s really humbling to read all these encouraging and more critical words, from the thank-you to the detailed critics of the library in which people tried to be so helpful I wanted to thank you all one by one if only I had all your email addresses.In the following post, I’d like to try to summarize and share the most interesting takeaways from this huge corpus.Let me first start by our userstransformers hasthreebig user communities of roughly equal sizes (in the respondents):image1398×796 49.4 KBResearchers (blue)ML engineers (red)Data scientists (green)Comparing the answers from each community can give some hints:Researchers:Theoldestand core usersThey probably oftendevelop or studymodelsThey are more oftenusing directly masterand wantverboseandeasy to customizemodel and examples. Some would like to be able totrain from scratch. They usually don’t want to use high-level training abstraction and encapsulated examples.ML engineers:They often joined a bit after the researchers communityThey probably oftenpush models in productionapplicationsThey are more oftenusing recent PyPi versions. They are interested infast inference, fp16, ONNX, TPU, multi-GPU,production-ready models. Some of them like training abstractions, some don’t.Data scientists:Themost recent communityof users (many have been using it since less than 3 months)They probably oftenuse models for data-analytics(i.e. no strong reqs for perf)They are often beginning to use transformers models. They are interested infast prototypingtool and examples that they can easily and quickly adapt to their use cases. They often don’t care much about the model internals or having access to training details but they are very interested in diving into/masteringdata-processing.There are also a lot of common points between these communities (they have mostly common interests) so don’t be distracted by these apparent differences, almost all of our users want recent (SOTA) models in an easy-to-use APIFor how long have our users been using the libraryimage2130×740 98.2 KBThere is a significant influx of new users (green + purple are < 3 months users). One-third of the respondents have been using the library for less than 3 months!The longest users are researchers followed by ML engineers. Data scientists are more recent users (40% of them have been using transformers for less than 3 months).work orfunimage2124×676 62.8 KBMost users are usingtransformers for work (80% overall)Researchers are somewhat the more serious community (>90% using it for work)Data scientists are using it more for fun than the other communities at the moment (maybe also because they are still discovering it).Which versionimage2130×740 153 KBMany users are on the latest or two latest versions (blue+red+green+purple)Researchers use master (red) more than the other communities (maybe because they tweak the models more).ML engineers are somewhat a bit more conservative (more users on 2.11.X (green))Data scientists tend to use master (red) less than the other communities (maybe because they customize the models less than the other groups)Would you recommend the library?1488×650 40.5 KBImportance of various features in the examplesimage2192×970 170 KBUser specific interests:1600×885 96.3 KBThree features are rated as most essentialin the examples overall by all user-communities:Full/transparent access todata-processingFull access totraining loopSimpledata-downloading and processingPeople care less aboutTPU supportSome more community-specific interests:Researcherswant more strongly to avoid encapsulated training than the other communitiesML engineersare more interested in FP16, multi-GPU, TPU than the other communitiesData Scientistscare less about training and optimization than the other communities (more ok with encapsulated trainer logic) and care more about the data processingWhat to prioritize1600×649 121 KBUsers ask for more priority on:Adding examplesfor NLP tasks + easier to adapt to morereal-life scenariosKeep addingSOTA modelsLess interesting for most users:Refactor the code toward modularizationWhat do you like the most1310×1274 491 KBMost frequent reasons are:Easy to useand simplicityManySOTA pretrained modelsCommunityShort summary of the top 300 strongest likes (apart from above mentioned top 3):Pipelines <= many people like themAutoModels <= many people like them as wellEasy to tweak - self-contained-verbose - non-modularization - no code reuseGood docModel hubPyTorch and TF interoptTransparencyWhat do you dislike the most1310×1274 518 KBSome top dislikes are:Needmore examples,more real-use-casesand on how to load your own datasets for certain tasks - More examples on using transformers for text or token classificationMore tutorialsand lack of guidance for simple examplesToo frequentbreaking changesToo muchmodularization- Model code is spread across too many filesExamples aretoo encapsulated– examples are hard to unpack sometimesTrainerLess support for Tensorflow 2.0model hub is a bit messy, it’s hard to find the right model…Open-feedback1310×1274 499 KBThe most noticeable one:Thanks! <=…
2020-07-08T20:13:44Z
I am positively impressed by the number of responses you got! Looking at the “would you recommend” graph, most people seem to have taken questionnaire seriously, too. (Not too many trolls.)Some things that I am surprised by:I had expected more data scientists and ML engineers and less researchersI am surprised that there are so many long-time users because it often feels that the library is continuously growing with many new users but less interaction from older users (perhaps the forums will change that!)67, 9, 123 as a an answer to a “work or fun” questionBig one: it seems that there is quite a preference for DataParallel over DistributedDataParallel. I wonder why this is. What are use cases where you can use DP but not DDP? In all other cases, DDP should perform better in terms of speed.I do understand the comments about modularization, that the code base seems to be spread over too many files. On the one hand I think this can be partially solved by using a folder structure for the models, tokenizers, and utils. A directory structure already makes the code base more readable. On the other hand, though, since there is a lot of inheritance (e.g. PretrainedModel, BertModel, RobertaModel), you cannot get around separate building blocks. It is necessary to allow for custom models and rapid research where other building blocks can be reused.The most important thing that I see in all of this, though, is the highlighted keyword in the likes:community. I definitely agree with it. The catch phrase of HF is todemocratize NLP, and I strongly believe that this is exactly what you guys are doing. Let’s have the forum be another step into that direction. A big congrats on all the work that you do!
2020-07-08T20:48:13Z
One of the questions for me coming out of this is how useful our examples are to people, especially in the data scientist / student crowd. Does theBig table of taskshit the spot for people who are looking for beginner-oriented examples of how to use transformers on various tasks with different datasets, or do we need to build some more walkthrough type examples in the docs directly, e.g. in theKeras style?
2020-07-09T01:52:52Z
I would love to see some notebooks for token classification. I was recently going through the Trainer run_ner.py example and was not having a great time. The preprocessing/data formatting is not very clear.I had a really great time going through some of the other notebooks that used transformers onhttps://notebooks.quantumstat.com/, however they are often out-of-date. If you could keep an up to date version where you touch each portion of what the pipeline handles for you, I would have a much easier time recommending transformers to my colleagues who are maybe aren’t able to find outside resources to learn from.
2020-07-09T17:19:17Z
jeremyaRD:The preprocessing/data formatting is not very clear.+1 I think we need to document what files/format users need to supply in README.md for examples. I tried for seq2seq:https://github.com/huggingface/transformers/blob/353b8f1e7a7361c0afd9e391381bc226b4a5ca8f/examples/seq2seq/README.md#L9but I’d definitely appreciate help.I think in general we don’t experiment/document using private data because we full-time open source developers/researchers mostly use benchmark datasets. We should do better.
2022-09-19T07:36:07Z
my friend is having some problem in using the transformers library from hugging face, so please can you help to create a more better library by linking some emotional qualities which correspond with the sentences meaning:(-)ve Qualities are:Anger, Attachment, Greed, Ego, Lust.(+)ve Qualities are:Calm, Love, Kindness, Humble, Healthy.