Open Source survey results [Jan 2022]
Open Source survey results [Jan 2022]lend-a-hand-transparent1024×1024 26.6 KBBack in December, we shared the first feedback request for Hugging Face’s Open-Source Ecosystem. While the first two requests for feedback were centered ontransformers, this one aimed to gather feedback on all components making up the Hugging Face ecosystem.Once again, it is thanks to all of your answers that we’re able to steer the directions of libraries and the ecosystem in a direction that fits you all; so we wanted to thank you for sharing your thoughts and a little of your time to let us know what you think, what you like and what you dislike. As a community-focused endeavor it is amazing to see that you all care about the direction in which we’re heading, to see the outpour of positive and encouraging comments, and the very constructive feedback you all are ready to give. From all of us at Hugging Face: Thank you!With this feedback summary, we aim for something a bit different than for the previous editions: we have crafted roadmaps for three aspects of the feedback request on which you had a lot of comments.These are the TensorFlow support in thetransformerslibrary, thedatasetslibrary direction, and the current state of the documentation oftransformers.TensorFlow roadmapWe were delighted to see that the appreciation for the TensorFlow side of thetransformerslibrary has greatly improved between this survey and the previous one. The net promoter score nearly doubled, from 14 to 27. We aim to continue in this direction, and the following roadmap was tailored according to your comments and expectations:Recurring feedback is that while the TensorFlow side of the library has improved over the past months, it is still not on the same level as PyTorch. Two aspects are at play here:A non-negligible number of model architectures available in PyTorch are not so in TensorFlowSome TensorFlow functionality is not working as well as the same in PyTorchIn order to work through this, we’ve identified a few key models and utilities on which we expect progress in the coming months.TensorFlow is lacking model architecturesSee below for the architectures that are yet to be implemented in TensorFlow and that have been asked in the survey/identified by the team as important points of focus.LayoutLM-v2DeBERTa-v3 (now supported!)GPT-JDETR, alongside backbonesWav2Vec2 and Hubert, while available in TensorFlow, are lacking aspeech_recognition_ctc.pyexample scriptWavLMUtilitiesTensorFlow usage is sometimes sub-par to PyTorch as some features are not as supported, lacking, or unoptimized. In order to improve this for users, here is what we’ll be focusing on in the coming months:Thegeneratemethod is not as customizable as its PyTorch/Flax counterpart, and it is slower.Rework the TensorFlowgeneratemethod so that it is readable, understandable, and customizable. This has been addressed bytransformers#15562MakegenerateXLA-compatible. This is being addressed bytransformers#15786andtransformers#15793Transformers’ tokenizers do not accept TensorFlow string tensors as input, which adds an additional necessary transformation.Datasets roadmapFor the first edition of the survey containing feedback requests relative todatasets, we were excited to see the majority of respondents leveragedatasetsin their daily workflows, even independently oftransformers.See below for a detailed roadmap of the next steps regarding datasets.DocumentationWe understand that a lot of issues mentioned were linked to lacking, hard to find, or incomplete documentation. We’re in the process of switching from Sphinx to our own frontend and customizing it so that search is better integrated. Alongside it, we aim to:Add anexamplesfolder in the datasets repository with examples of loading/processing/preparing datasets in multiple modalities.Document better how to manage large datasets for optimal performanceDocs and examples specifically tailored around vision, speech and time-series datasetsWe aim to improve dataset search on the hub with better tags/tasks.Load and share your own dataThe dataset scripts remain one of the pain points to contribute datasets on the hub. In order to help users to contribute their datasets in a simpler manner, we’ll work on relaxing the constraints on dataset scripts: having generic/standard loaders, sharing structure among datasetsStreaming datasetsStreaming datasets are currently lacking features that traditionalDatasetobjects have. We aim to reduce this disparity, by:Aligning themapfor streaming and non-streaming datasetsAdding additional features for streaming datasets:filterColumn manipulationcastothersMetrics shouldn’t be in datasetsThemetricswere mentioned as a very positive aspect ofdatasets, but several respondents raised the point of whether they should be contained indatasetsor be split.We’re happy to mention that we’ve been thinking about splitting themetricscomponent ofdatasetsin a separate, standalone project.Transformers Documentation roadmapWe’ve been looking forward to migrating the current documentation toDiátaxis, a new information architecture (IA). Information architecture refers to how content is organized, which impacts how you interact with the documentation.Diátaxisis a user-first framework. It focuses on howyouinteract with the documentation instead of arranging it based on a product’s features. Based on the feedback, some users have difficulty finding what they’re looking for. The new documentation structure will help address this issue, in addition to makingTransformers more accessible to a broader background of users.Coming to a browser near you soon, you will see changes to the documentation structure. It will be easier for you tofind exactly what you’re looking for, when you need it. The documentation will be separated into four categories:Tutorialswill teach you the basic skills you need to useTransformers. If you are new to the library, we recommend starting here!How-to guideswill show you how to apply your skills to solve specific problems.Concept guideswill explain things and help you understand a topic better. This way you can focus ondoing -in the tutorial and how-to guides - without getting too distracted by explanations and abstractions.Referencewill help you as you work, allowing you to look up what you need to know and describe how things work. We will continue our work on ensuring the API documentation is accurate, up-to-date, and includes examples for training and inference.Thank you once again for helping us steer the ecosystem in a direction that fits your needs. As always, we welcome any and all contributions to our libraries, to the datasets hub, and to the model hub, and we’ll be very happy to guide you if you would like to help on any of the bullet points of the roadmaps defined above, as well as any other (code or other!) contribution.Please comment on this post if you’d like to help!
2022-03-10T08:07:46Z
There are some great development plans in here! I particularly love the direction of datasets. A wonderful tool that I, too, frequently use, but whose documentation is sometimes hard to fiddle through. Giving more attention to users who wish to load their own, local, datasets with it, is an excellent development as well! Splitting off of the metrics, finally, seems like a sensible idea indeed but probably won’t change too much to how they work (?), so I can only encourage it.Thanks for doing these surveys from time to time, it shows that HF still has the open-source community in mind, even after the successful funding rounds. Kudos!