HPLT
/

hplt_bert_base_en

Model card Files Files and versions Community

davda54 commited on Aug 6, 2024

Commit

1108d9b

·

verified ·

1 Parent(s): 968a138

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +5 -3

tokenizer.json CHANGED Viewed

@@ -1102,9 +1102,11 @@
         "use_regex": true
       },
       {
-        "type": "Metaspace",
-        "replacement": "▁",
-        "add_prefix_space": false
       },
       {
         "type": "Strip",

         "use_regex": true
       },
       {
+        "type": "Replace",
+        "pattern": {
+          "Regex": "▁"
+        },
+        "content": " "
       },
       {
         "type": "Strip",