Tokenizers v0.8.0 is out!
Highlights of this releaseWe can now encode both pre-tokenized inputs, and raw strings. This is especially usefull whenprocessing datasets that are already pre-tokenized like for NER (Name Entity Recognition), and helpswhile applying labels to each word.Full tokenizer serialization. It is now easy to save a tokenizer to a single JSON file, to laterload it back with just one line of code. That’s what sharing a Tokenizer means now: 1 line of code.With the serialization comes the compatibility withPickle! The Tokenizer, all of its components,Encodings, everything can be pickled!Training a tokenizer is now even faster (up to 5-10x) than before!Compatibility withmultiprocessing, even when using theforkstart method. Since this librarymakes heavy use of the multithreading capacities of our computers to allows a very fast tokenization,this led to problems (deadlocks) when used withmultiprocessing. This version now allows todisable the parallelism, and will warn you if this is necessary.And a lot of other improvements, and fixes.Fixed#286: Fix various crash when training a BPE model#309: Fixed a few bugs related to additional vocabulary/tokensAdded#272: Serialization of theTokenizerand all the parts (PreTokenizer,Normalizer, …).This adds some methods to easily save/load an entire tokenizer (from_str,from_file).#273:Tokenizerand its parts are now pickable#289: Ability to pad to a multiple of a specified value. This is especially useful to ensureactivation of the Tensor Cores, while ensuring padding to a multiple of 8. Use withenable_padding(pad_to_multiple_of=8)for example.#298: Ability to get the currently set truncation/padding params#311: Ability to enable/disable the parallelism using theTOKENIZERS_PARALLELISMenvironmentvariable. This is especially usefull when usingmultiprocessingcapabilities, with theforkstart method, which happens to be the default on Linux systems. Without disabling the parallelism,the process dead-locks while encoding. (Cf [#187] for more information)ChangedImproved errors generated during truncation: When the provided max length is too low arenow handled properly.#249encodeandencode_batchnow accept pre-tokenized inputs. When the input is pre-tokenized,the argumentis_pretokenized=Truemust be specified.#276: Improve BPE training speeds, by reading files sequentially, but parallelizing theprocessing of each file#280: Useonigfor byte-level pre-tokenization to remove all the differences with the originalimplementation from GPT-2#309: Improved the management of the additional vocabulary. This introduces an optionnormalized, controlling whether a token should be extracted from the normalized version of theinput text.