This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the [huggingfaceNLP course exercise](https://huggingface.co/learn/nlp-course/chapter6/2). It uses the method `train_new_from_iterator()`
{"license": "cc0-1.0", "library_name": "transformers", "tags": ["text-generation", "code", "python"], "datasets": ["code_search_net"]}
transformers text-generation code python dataset:code_search_net license:cc0-1.0 endpoints_compatible region:us