
Microlang was designed to test text generation architectures

It consists of 16 tokens:
  1. special (implicit):

    <> 

  2. noun:
  
    bob, tom, bike, speech
  
  3. transitive verb:
  
    take, use
  
  4. intransitive verb: 

    talk, go
  
  5. adjective:
  
    good, active
  
  6. adverb:
  
    not
  
  7. conjunction:
  
    and, then, but
    
  8. punctuation:
 
    .


The tokenizer can be found on `umarzein/microlang-utils` and can be loaded this:

```python
import transformers
tokenizer = transformers.PreTrainedTokenizerFast.from_pretrained("umarzein/microlang-utils")
```


language:en license:mit region:us
{"language": ["en"], "license": "mit"}
TAGS
#language-English #license-mit #region-us 


Microlang was designed to test text generation architectures

It consists of 16 tokens:
  1. special (implicit):

    <> 

  2. noun:
  
    bob, tom, bike, speech
  
  3. transitive verb:
  
    take, use
  
  4. intransitive verb: 

    talk, go
  
  5. adjective:
  
    good, active
  
  6. adverb:
  
    not
  
  7. conjunction:
  
    and, then, but
    
  8. punctuation:
 
    .


The tokenizer can be found on 'umarzein/microlang-utils' and can be loaded this:




TAGS
#language-English #license-mit #region-us 

passage: TAGS
#language-English #license-mit #region-us 
