region:us
# GlobalTokenizer

GlobalTokenizer tokenize all languages of the earth.

## usage
```python
from transformers import AutoModelForMaskedLM, AutoTokenizer
model_path = "JuanbaiAI/GlobalTokenizer"

tokenizer = AutoTokenizer.from_pretrained(model_path)
print(tokenizer("Hello, 世界"))

```

## corpus: 
Wikipedia, Wikisource, SlimRedPajama, baidubaike, daizhige, Wanjuan.

languages(200+) coverage:
* English:33%
* Chinese:33%
* Global (other langauges exclude English/Chinese):33%

## train
* algorithm: BBPE
* code: https://huggingface.co/learn/nlp-course/chapter6/8

## benchmark
| **Corpus**   |              | **UnicodeTokenizer** |               | **Bloom**         |               | **GlobalTokenizer** |               |
|--------------|--------------|----------------------|---------------|-------------------|---------------|---------------------|---------------|
|              | num of bytes | avg bytes /tokens    | num of tokens | avg bytes /tokens | num of tokens | avg bytes /tokens   | num of tokens |
| RedPajama    | 4.2E+10      | 2.731478             | 1.54E+10      | 4.370739          | 9.6E+09       | 3.96642             | 1.06E+10      |
| baidubaike   | 4.97E+09     | 3.980441             | 1.25E+09      | 4.127816          | 1.2E+09       | 3.922987            | 1.27E+09      |
| daizhige     | 2.29E+09     | 3.473577             | 6.59E+08      | 3.244604          | 7.05E+08      | 3.174714            | 7.21E+08      |
| TextBook-cn  | 1.1E+09      | 3.480186             | 3.16E+08      | 3.299681          | 3.33E+08      | 3.241768            | 3.39E+08      |
| WebText-cn   | 4.2E+10      | 4.177369             | 1.01E+10      | 4.381168          | 9.6E+09       | 4.281016            | 9.82E+09      |
| ab           | 1237796      | 6.056468             | 204375.9      | 2.37433           | 521324.3      | 3.014965            | 410550.7      |
| ace          | 1574258      | 3.166403             | 497175.5      | 2.920242          | 539084.7      | 2.865722            | 549340.7      |
| af           | 1.24E+08     | 2.842203             | 43798967      | 3.089724          | 40290184      | 3.024023            | 41165548      |
| als          | 49452211     | 2.957404             | 16721491      | 2.899396          | 17056040      | 2.794853            | 17694029      |
| alt          | 2910477      | 5.346916             | 544328.2      | 2.811496          | 1035206       | 3.755865            | 774915.3      |
| am           | 10720027     | 5.450505             | 1966795       | 1.651675          | 6490396       | 2.368297            | 4526471       |
| ami          | 2261859      | 2.511871             | 900467.9      | 2.999494          | 754080.3      | 2.741438            | 825063        |
| an           | 29573020     | 2.638144             | 11209781      | 3.678932          | 8038480       | 3.001739            | 9851963       |
| ang          | 1789811      | 3.09878              | 577585.8      | 2.648479          | 675788.2      | 2.541121            | 704339.2      |
| anp          | 3355979      | 6.35562              | 528033.3      | 8.648999          | 388019.4      | 3.51896             | 953684.9      |
| ar           | 1.79E+09     | 4.512798             | 3.98E+08      | 6.1454            | 2.92E+08      | 4.111721            | 4.37E+08      |
| ary          | 4672257      | 4.115167             | 1135375       | 4.571392          | 1022064       | 3.667588            | 1273932       |
| arz          | 3.17E+08     | 4.658518             | 68097110      | 5.049221          | 62827827      | 5.280836            | 60072224      |
| as           | 38558469     | 7.573861             | 5090992       | 8.281984          | 4655704       | 3.34377             | 11531436      |
| ast          | 2.71E+08     | 2.864269             | 94682610      | 3.76335           | 72062514      | 3.128894            | 86674857      |
| av           | 2573436      | 5.750956             | 447479.7      | 2.826744          | 910388.9      | 3.639347            | 707114.7      |
| avk          | 7984474      | 2.768189             | 2884367       | 2.775794          | 2876465       | 2.650919            | 3011964       |
| awa          | 1269824      | 6.413487             | 197992.8      | 7.927757          | 160174.4      | 3.513209            | 361442.8      |
| ay           | 1756131      | 2.819794             | 622787        | 2.824251          | 621804        | 2.463072            | 712983.9      |
| az           | 2.51E+08     | 3.781892             | 66373457      | 2.594857          | 96736449      | 2.82334             | 88907897      |
| azb          | 46739926     | 4.962435             | 9418748       | 3.655348          | 12786724      | 4.257964            | 10977061      |
| ba           | 1.23E+08     | 5.780001             | 21210345      | 2.623517          | 46729559      | 4.177275            | 29348274      |
| ban          | 6715876      | 3.355344             | 2001546       | 3.239442          | 2073158       | 2.977388            | 2255627       |
| bar          | 21799389     | 2.91939              | 7467105       | 2.870996          | 7592972       | 2.713014            | 8035119       |
| bat-smg      | 3348765      | 3.503523             | 955827.9      | 2.320313          | 1443238       | 2.480687            | 1349935       |
| bcl          | 11369234     | 2.875691             | 3953567       | 3.320823          | 3423619       | 3.084119            | 3686380       |
| be           | 3.11E+08     | 5.476524             | 56838635      | 3.017893          | 1.03E+08      | 4.068214            | 76514692      |
| be-x-old     | 1.14E+08     | 5.694565             | 20075035      | 2.982457          | 38330337      | 4.088079            | 27963890      |
| bg           | 5.27E+08     | 4.992976             | 1.06E+08      | 3.51584           | 1.5E+08       | 4.492216            | 1.17E+08      |
| bh           | 5880458      | 5.962781             | 986193.8      | 7.715468          | 762164.8      | 3.517409            | 1671815       |
| bjn          | 3323032      | 3.253983             | 1021220       | 3.749782          | 886193.4      | 3.105171            | 1070161       |
| blk          | 8028430      | 6.456261             | 1243511       | 1.551035          | 5176175       | 3.901145            | 2057968       |
| bn           | 3.44E+08     | 8.168696             | 42098026      | 9.874329          | 34826264      | 3.375484            | 1.02E+08      |
| bo           | 38851784     | 5.471347             | 7100954       | 2.369802          | 16394527      | 3.550876            | 10941464      |
| bpy          | 6568483      | 7.031357             | 934170        | 6.442238          | 1019596       | 3.316048            | 1980816       |
| br           | 49768597     | 2.596823             | 19165184      | 2.847565          | 17477598      | 2.618958            | 19003205      |
| bs           | 1.17E+08     | 3.034931             | 38645253      | 2.676185          | 43825689      | 2.806575            | 41789603      |
| bxr          | 3078699      | 5.557568             | 553965.1      | 2.724922          | 1129830       | 3.588642            | 857900.8      |
| ca           | 1.12E+09     | 2.739052             | 4.1E+08       | 4.175102          | 2.69E+08      | 3.151386            | 3.56E+08      |
| cdo          | 1982914      | 2.605206             | 761135.1      | 2.549999          | 777613.7      | 2.208196            | 897979        |
| ce           | 88393330     | 4.908585             | 18007906      | 2.941855          | 30046803      | 5.186398            | 17043298      |
| ceb          | 8.28E+08     | 2.780197             | 2.98E+08      | 3.081311          | 2.69E+08      | 3.753617            | 2.21E+08      |
| ckb          | 42964544     | 5.585014             | 7692826       | 2.731714          | 15728052      | 3.510231            | 12239804      |
| co           | 5794731      | 2.716548             | 2133123       | 3.019771          | 1918931       | 2.767274            | 2094021       |
| crh          | 3654461      | 3.631442             | 1006339       | 2.507428          | 1457454       | 2.738573            | 1334440       |
| cs           | 1.1E+09      | 3.254687             | 3.38E+08      | 2.60566           | 4.22E+08      | 2.818314            | 3.91E+08      |
| csb          | 2055233      | 3.303751             | 622090.8      | 2.390176          | 859867        | 2.315591            | 887563.2      |
| cv           | 29640641     | 4.05758              | 7305005       | 2.837606          | 10445649      | 3.770825            | 7860520       |
| cy           | 1.14E+08     | 2.799797             | 40541764      | 2.564199          | 44266735      | 3.051617            | 37196248      |
| da           | 3.39E+08     | 2.976211             | 1.14E+08      | 2.994664          | 1.13E+08      | 3.050977            | 1.11E+08      |
| dag          | 9026986      | 2.625832             | 3437762       | 3.41197           | 2645681       | 3.140065            | 2874777       |
| de           | 6.08E+09     | 3.265201             | 1.86E+09      | 3.293052          | 1.85E+09      | 3.295854            | 1.85E+09      |
| diq          | 7616839      | 3.015742             | 2525693       | 2.618003          | 2909408       | 2.480188            | 3071074       |
| dsb          | 1931937      | 3.149291             | 613451.5      | 2.481379          | 778573.9      | 2.486934            | 776834.9      |
| dty          | 2521250      | 7.573179             | 332918.3      | 8.289923          | 304134.3      | 3.395063            | 742622.5      |
| dv           | 5283133      | 7.507441             | 703719.5      | 1.082612          | 4879986       | 2.04329             | 2585601       |
| dz           | 2583520      | 5.389404             | 479370.3      | 2.35514           | 1096971       | 3.561721            | 725357.2      |
| el           | 7.41E+08     | 5.303681             | 1.4E+08       | 2.740865          | 2.7E+08       | 3.639245            | 2.04E+08      |
| eml          | 1689575      | 2.63297              | 641699.2      | 2.679583          | 630536.4      | 2.404491            | 702674.7      |
| en           | 1.3E+10      | 2.784381             | 4.67E+09      | 4.317653          | 3.01E+09      | 4.016363            | 3.24E+09      |
| eo           | 3E+08        | 2.934561             | 1.02E+08      | 2.982057          | 1.01E+08      | 2.910416            | 1.03E+08      |
| es           | 3.72E+09     | 2.865625             | 1.3E+09       | 4.372629          | 8.5E+08       | 3.423013            | 1.09E+09      |
| et           | 2.68E+08     | 3.519661             | 76075834      | 2.766091          | 96801265      | 2.704236            | 99015440      |
| eu           | 2.8E+08      | 3.412258             | 82006870      | 4.233857          | 66093076      | 2.948036            | 94920346      |
| ext          | 2761099      | 2.848068             | 969463.7      | 3.367813          | 819849.2      | 2.909613            | 948957.6      |
| fa           | 7.93E+08     | 4.328803             | 1.83E+08      | 4.326995          | 1.83E+08      | 4.276889            | 1.85E+08      |
| fat          | 1124684      | 2.635375             | 426764.4      | 2.885461          | 389776.2      | 2.563495            | 438730.7      |
| ff           | 1087702      | 3.073717             | 353871.9      | 2.283122          | 476410        | 2.181656            | 498567        |
| fi           | 7.14E+08     | 3.875986             | 1.84E+08      | 2.826093          | 2.53E+08      | 2.847687            | 2.51E+08      |
| fiu-vro      | 2434159      | 3.292221             | 739366.9      | 2.463464          | 988104.1      | 2.407473            | 1011085       |
| fo           | 8857239      | 3.002534             | 2949922       | 2.697937          | 3282967       | 2.633042            | 3363881       |
| fr           | 4.66E+09     | 2.812367             | 1.66E+09      | 4.357305          | 1.07E+09      | 3.327677            | 1.4E+09       |
| frp          | 1914046      | 2.802151             | 683063        | 3.104856          | 616468.5      | 2.716365            | 704635.1      |
| frr          | 5317694      | 2.855094             | 1862529       | 2.727262          | 1949829       | 2.649299            | 2007208       |
| fur          | 2421238      | 2.630931             | 920297        | 3.050013          | 793845.2      | 2.810208            | 861586.7      |
| fy           | 76002257     | 2.748455             | 27652722      | 2.854965          | 26621080      | 2.831785            | 26838989      |
| ga           | 34136733     | 2.907793             | 11739740      | 2.763848          | 12351159      | 2.647309            | 12894878      |
| gag          | 1331866      | 3.32828              | 400166.4      | 2.67359           | 498156.4      | 2.659286            | 500836        |
| gan          | 1508844      | 3.566718             | 423034.3      | 3.216688          | 469067.6      | 3.075602            | 490585        |
| gcr          | 1345482      | 2.66169              | 505499.1      | 2.859282          | 470566.5      | 2.564873            | 524580.3      |
| gd           | 7190137      | 2.708183             | 2654967       | 2.723874          | 2639673       | 2.507579            | 2867363       |
| gl           | 2.95E+08     | 2.91577              | 1.01E+08      | 3.951435          | 74610154      | 3.256364            | 90535697      |
| glk          | 2382997      | 4.282207             | 556488.1      | 3.740417          | 637094        | 3.520784            | 676837.1      |
| gn           | 3806548      | 3.111337             | 1223444       | 2.798959          | 1359987       | 2.494229            | 1526142       |
| gom          | 11306217     | 6.703585             | 1686593       | 6.073611          | 1861531       | 3.33379             | 3391400       |
| gor          | 2101154      | 3.124766             | 672419.7      | 3.949286          | 532033.9      | 3.423681            | 613712        |
| gpe          | 1141261      | 2.811199             | 405969.5      | 4.171368          | 273594        | 3.974151            | 287171        |
| gu           | 39554078     | 7.279748             | 5433441       | 8.638276          | 4578932       | 3.18546             | 12417069      |
| gur          | 1068954      | 2.313831             | 461984.5      | 2.921246          | 365924        | 2.644545            | 404211        |
| gv           | 3347095      | 2.782664             | 1202838       | 2.629385          | 1272957       | 2.491743            | 1343275       |
| ha           | 43131815     | 2.643041             | 16319013      | 2.953407          | 14604086      | 2.807367            | 15363795      |
| hak          | 1878558      | 2.424355             | 774869.2      | 2.560305          | 733724.2      | 2.09133             | 898260        |
| he           | 1.5E+09      | 4.460416             | 3.36E+08      | 2.26048           | 6.63E+08      | 3.514656            | 4.26E+08      |
| hi           | 2.38E+08     | 6.332328             | 37558795      | 9.035875          | 26321147      | 3.568483            | 66648667      |
| hif          | 2715682      | 2.700324             | 1005687       | 3.324587          | 816848        | 3.056977            | 888355.3      |
| hr           | 3.14E+08     | 2.997707             | 1.05E+08      | 2.677169          | 1.17E+08      | 2.779891            | 1.13E+08      |
| hsb          | 7437491      | 3.240744             | 2294995       | 2.492049          | 2984488       | 2.576124            | 2887086       |
| ht           | 21993952     | 2.672347             | 8230202       | 2.976064          | 7390281       | 2.873894            | 7653015       |
| hu           | 9.67E+08     | 3.478646             | 2.78E+08      | 2.690766          | 3.59E+08      | 2.782708            | 3.47E+08      |
| hy           | 5.09E+08     | 5.823873             | 87346563      | 2.294067          | 2.22E+08      | 3.956965            | 1.29E+08      |
| hyw          | 27450541     | 5.907639             | 4646618       | 2.240724          | 12250745      | 3.683785            | 7451722       |
| ia           | 8237640      | 2.872221             | 2868039       | 3.820258          | 2156304       | 3.441554            | 2393581       |
| id           | 6.02E+08     | 3.171613             | 1.9E+08       | 4.526109          | 1.33E+08      | 3.31139             | 1.82E+08      |
| ie           | 3019044      | 2.769531             | 1090092       | 3.798603          | 794777.4      | 3.400945            | 887707.4      |
| ig           | 34663540     | 2.788113             | 12432617      | 3.572043          | 9704121       | 2.880903            | 12032178      |
| ilo          | 7352572      | 2.660928             | 2763161       | 3.001911          | 2449297       | 2.8403              | 2588660       |
| inh          | 1279524      | 5.404367             | 236757.4      | 2.922974          | 437747.4      | 3.828947            | 334171.3      |
| io           | 17106040     | 2.648579             | 6458572       | 3.071699          | 5568918       | 2.878832            | 5942006       |
| is           | 63906249     | 3.209898             | 19909118      | 2.661164          | 24014398      | 2.652901            | 24089191      |
| it           | 3.05E+09     | 2.906538             | 1.05E+09      | 3.41333           | 8.93E+08      | 3.293968            | 9.25E+08      |
| ja           | 4.03E+09     | 4.420311             | 9.12E+08      | 3.207393          | 1.26E+09      | 3.529492            | 1.14E+09      |
| jv           | 40261285     | 3.217514             | 12513166      | 3.222019          | 12495670      | 2.891515            | 13923939      |
| ka           | 2.4E+08      | 8.898958             | 26968063      | 2.879661          | 83338866      | 4.77234             | 50287207      |
| kaa          | 2913134      | 3.610466             | 806858.3      | 2.580794          | 1128774       | 2.572442            | 1132439       |
| kab          | 2580584      | 2.713354             | 951067.8      | 2.518844          | 1024511       | 2.306864            | 1118654       |
| kbd          | 1304580      | 5.821687             | 224089.7      | 2.34828           | 555547        | 3.118805            | 418294.8      |
| kbp          | 1806400      | 3.173719             | 569174.6      | 2.053971          | 879467.2      | 2.129424            | 848304.7      |
| kk           | 1.81E+08     | 6.057511             | 29839073      | 2.795574          | 64655964      | 4.276387            | 42267107      |
| km           | 35567417     | 8.744804             | 4067263       | 2.495309          | 14253715      | 2.932378            | 12129205      |
| kn           | 2.2E+08      | 9.743682             | 22624591      | 9.264455          | 23794905      | 3.269786            | 67419352      |
| ko           | 1.09E+09     | 4.188618             | 2.59E+08      | 2.060378          | 5.27E+08      | 2.865548            | 3.79E+08      |
| koi          | 1888392      | 4.277403             | 441480.9      | 2.810405          | 671928.8      | 3.239775            | 582877.5      |
| krc          | 2022144      | 5.998891             | 337086.3      | 3.09196           | 654000.8      | 4.020692            | 502934.3      |
| ks           | 1094458      | 3.276181             | 334065.2      | 3.128696          | 349812.8      | 2.783333            | 393218.5      |
| ksh          | 2009928      | 2.806068             | 716279.2      | 2.734781          | 734950.4      | 2.657422            | 756344.9      |
| ku           | 22938233     | 3.011483             | 7616923       | 2.604819          | 8806074       | 2.522653            | 9092899       |
| kv           | 3690978      | 4.57134              | 807417.1      | 2.772082          | 1331482       | 3.439921            | 1072983       |
| kw           | 2711398      | 2.640414             | 1026884       | 2.64011           | 1027002       | 2.545511            | 1065168       |
| ky           | 63947035     | 5.787278             | 11049588      | 2.961551          | 21592413      | 4.034032            | 15851891      |
| la           | 4.28E+08     | 3.151159             | 1.36E+08      | 3.296501          | 1.3E+08       | 3.134588            | 1.37E+08      |
| lad          | 2754531      | 2.871327             | 959323.3      | 3.365023          | 818577.1      | 3.048921            | 903444.7      |
| lb           | 50515020     | 2.947684             | 17137189      | 3.09739           | 16308900      | 2.929474            | 17243714      |
| lez          | 3864848      | 5.783752             | 668225        | 2.998299          | 1289014       | 3.824453            | 1010562       |
| lfn          | 5207546      | 2.493424             | 2088512       | 3.404739          | 1529499       | 3.007587            | 1731470       |
| lg           | 3708097      | 3.12612              | 1186166       | 3.009781          | 1232015       | 2.576595            | 1439146       |
| li           | 18919316     | 2.827532             | 6691106       | 2.869629          | 6592949       | 2.722327            | 6949687       |
| lij          | 6255409      | 2.686963             | 2328060       | 2.764908          | 2262429       | 2.464669            | 2538033       |
| lld          | 13866243     | 2.570061             | 5395297       | 3.028007          | 4579330       | 3.179828            | 4360690       |
| lmo          | 19142356     | 2.505112             | 7641316       | 3.033964          | 6309355       | 2.74959             | 6961896       |
| ln           | 1122138      | 2.82705              | 396928.9      | 3.03398           | 369856.7      | 2.58079             | 434804.1      |
| lo           | 5646554      | 7.447939             | 758136.4      | 1.612251          | 3502279       | 2.910082            | 1940342       |
| lt           | 2E+08        | 3.406909             | 58560670      | 2.625878          | 75978739      | 2.659482            | 75018710      |
| lv           | 1.3E+08      | 3.446303             | 37645920      | 2.589252          | 50106832      | 2.619658            | 49525246      |
| mai          | 6180231      | 7.075233             | 873502.1      | 7.641407          | 808781.8      | 3.416389            | 1808995       |
| map-bms      | 2377123      | 3.182913             | 746838.9      | 3.934144          | 604228.8      | 3.132915            | 758757.6      |
| mdf          | 1725294      | 4.607865             | 374423.7      | 3.031015          | 569213.3      | 3.428923            | 503159.2      |
| mg           | 22117304     | 2.868733             | 7709783       | 2.830482          | 7813972       | 2.795544            | 7911627       |
| mhr          | 6902162      | 4.916981             | 1403740       | 3.066632          | 2250730       | 3.901038            | 1769314       |
| min          | 25691303     | 3.246645             | 7913186       | 3.440872          | 7466509       | 3.782731            | 6791734       |
| mk           | 2.76E+08     | 5.178571             | 53362784      | 3.540047          | 78061941      | 4.591763            | 60182317      |
| ml           | 2.13E+08     | 10.76792             | 19737657      | 8.952587          | 23739904      | 3.170749            | 67029451      |
| mn           | 41521786     | 5.443966             | 7627121       | 2.750759          | 15094662      | 3.709353            | 11193807      |
| mni          | 2208525      | 6.843953             | 322697.3      | 1.090132          | 2025925       | 2.764841            | 798789        |
| mnw          | 13765461     | 5.687051             | 2420492       | 1.661704          | 8283943       | 3.330587            | 4133044       |
| mr           | 95291716     | 7.609766             | 12522293      | 8.816335          | 10808540      | 3.347845            | 28463595      |
| mrj          | 3283618      | 4.500504             | 729611.2      | 2.582179          | 1271646       | 3.173599            | 1034667       |
| ms           | 2.1E+08      | 3.193519             | 65804920      | 4.299357          | 48879231      | 3.278002            | 64108953      |
| mt           | 18686521     | 2.879751             | 6488937       | 2.590851          | 7212503       | 2.472526            | 7557665       |
| mwl          | 11521563     | 2.844074             | 4051077       | 3.572892          | 3224716       | 2.995551            | 3846225       |
| my           | 85497205     | 8.523657             | 10030577      | 1.67488           | 51046763      | 3.759618            | 22740930      |
| myv          | 4600620      | 5.171224             | 889657.9      | 3.152488          | 1459362       | 3.787347            | 1214734       |
| mzn          | 5419390      | 4.135287             | 1310523       | 3.992757          | 1357305       | 3.829374            | 1415216       |
| nah          | 1191779      | 3.676918             | 324124.4      | 2.495853          | 477503.7      | 2.294145            | 519487.2      |
| nap          | 3188122      | 2.527682             | 1261283       | 2.929708          | 1088205       | 2.642238            | 1206599       |
| nds          | 48106879     | 2.942096             | 16351225      | 3.016395          | 15948466      | 2.987598            | 16102194      |
| nds-nl       | 8354427      | 2.883845             | 2896975       | 2.871939          | 2908985       | 2.806673            | 2976630       |
| ne           | 37548833     | 7.989947             | 4699509       | 9.557693          | 3928650       | 3.460884            | 10849493      |
| new          | 20517810     | 7.302558             | 2809675       | 6.637754          | 3091077       | 3.412097            | 6013255       |
| nia          | 1086670      | 2.666779             | 407484.2      | 2.791992          | 389209.5      | 2.556508            | 425060.2      |
| nl           | 1.46E+09     | 2.980084             | 4.92E+08      | 3.117232          | 4.7E+08       | 3.19817             | 4.58E+08      |
| nn           | 1.35E+08     | 2.921273             | 46127799      | 2.97112           | 45353896      | 3.012606            | 44729344      |
| no           | 5.92E+08     | 2.99826              | 1.98E+08      | 3.018857          | 1.96E+08      | 3.069458            | 1.93E+08      |
| nqo          | 3508645      | 4.800661             | 730867        | 1.01531           | 3455738       | 2.313888            | 1516342       |
| nrm          | 1507257      | 2.643664             | 570139.5      | 3.056702          | 493099.1      | 2.62216             | 574815        |
| nv           | 3304031      | 3.214593             | 1027823       | 2.34207           | 1410731       | 2.970995            | 1112096       |
| oc           | 64043588     | 2.800003             | 22872683      | 3.548549          | 18047826      | 3.003005            | 21326497      |
| olo          | 1724315      | 3.46881              | 497091.3      | 2.630054          | 655619.6      | 2.532276            | 680934.9      |
| om           | 1982849      | 3.146444             | 630187.3      | 2.669185          | 742866.9      | 2.485367            | 797809.2      |
| or           | 34054918     | 7.503656             | 4538443       | 8.050034          | 4230407       | 3.019334            | 11278949      |
| os           | 5572799      | 3.930341             | 1417892       | 2.741069          | 2033075       | 3.398942            | 1639569       |
| pa           | 84036831     | 6.039642             | 13914207      | 7.968192          | 10546536      | 3.347806            | 25102062      |
| pam          | 4277038      | 2.979156             | 1435654       | 3.518991          | 1215416       | 3.234084            | 1322488       |
| pap          | 2435005      | 2.619218             | 929668.8      | 3.336065          | 729903.4      | 2.959935            | 822654.9      |
| pcd          | 3145572      | 2.547703             | 1234670       | 3.264388          | 963602.5      | 2.748718            | 1144378       |
| pcm          | 1160762      | 2.454551             | 472902        | 3.379737          | 343447.4      | 3.187808            | 364125.4      |
| pfl          | 1971214      | 3.069015             | 642295.3      | 2.853351          | 690841.8      | 2.743806            | 718423.1      |
| pl           | 1.8E+09      | 3.372125             | 5.34E+08      | 2.596431          | 6.93E+08      | 2.862203            | 6.29E+08      |
| pms          | 21712699     | 2.483721             | 8742006       | 2.994908          | 7249872       | 2.768085            | 7843943       |
| pnb          | 1.33E+08     | 3.850482             | 34610271      | 4.844648          | 27507933      | 3.869398            | 34441083      |
| ps           | 53312545     | 3.863741             | 13798167      | 3.120696          | 17083543      | 3.5782              | 14899263      |
| pt           | 1.64E+09     | 2.850676             | 5.76E+08      | 4.279796          | 3.84E+08      | 3.262223            | 5.03E+08      |
| qu           | 7688106      | 3.070997             | 2503456       | 2.76082           | 2784719       | 2.523484            | 3046623       |
| rm           | 10483970     | 2.82811              | 3707058       | 3.396868          | 3086364       | 3.049769            | 3437627       |
| ro           | 5.48E+08     | 2.959287             | 1.85E+08      | 2.898551          | 1.89E+08      | 2.946163            | 1.86E+08      |
| roa-tara     | 4003095      | 2.822698             | 1418180       | 2.927105          | 1367595       | 2.66504             | 1502077       |
| ru           | 9.48E+09     | 5.401585             | 1.75E+09      | 3.537656          | 2.68E+09      | 4.602799            | 2.06E+09      |
| rue          | 6346106      | 5.189003             | 1222991       | 3.239176          | 1959173       | 4.218571            | 1504326       |
| rw           | 6623388      | 3.115312             | 2126076       | 3.386084          | 1956062       | 2.580994            | 2566216       |
| sa           | 2.56E+08     | 9.344079             | 27376241      | 6.500521          | 39351579      | 3.328375            | 76856060      |
| sah          | 29330820     | 6.091066             | 4815383       | 2.686451          | 10918055      | 3.771827            | 7776289       |
| sat          | 15428584     | 6.501258             | 2373169       | 1.095737          | 14080552      | 3.694191            | 4176444       |
| sc           | 7711996      | 2.789137             | 2765011       | 3.222008          | 2393537       | 2.871126            | 2686053       |
| scn          | 10223816     | 2.814459             | 3632604       | 2.91792           | 3503802       | 2.710677            | 3771684       |
| sco          | 24287944     | 2.814675             | 8629040       | 3.556268          | 6829617       | 3.322475            | 7310197       |
| sd           | 17591997     | 3.820602             | 4604510       | 3.189349          | 5515859       | 3.323255            | 5293605       |
| se           | 1816006      | 3.639336             | 498993.8      | 2.519246          | 720853        | 2.374435            | 764816.1      |
| sh           | 2.66E+08     | 3.109088             | 85677639      | 2.810773          | 94770818      | 3.086501            | 86304614      |
| shi          | 1359828      | 2.603696             | 522268.4      | 2.388864          | 569236.3      | 2.228096            | 610309.3      |
| shn          | 8163231      | 6.818837             | 1197159       | 1.563793          | 5220148       | 3.548365            | 2300561       |
| si           | 54229127     | 6.861562             | 7903321       | 1.605319          | 33780901      | 3.228928            | 16794776      |
| simple       | 1.57E+08     | 2.604897             | 60227038      | 4.105208          | 38216143      | 3.746763            | 41872201      |
| sk           | 2.42E+08     | 3.185025             | 75896153      | 2.660064          | 90874192      | 2.743562            | 88108506      |
| skr          | 9978607      | 3.390208             | 2943361       | 4.619296          | 2160201       | 3.69874             | 2697840       |
| sl           | 5.1E+08      | 2.776806             | 1.84E+08      | 2.65154           | 1.92E+08      | 2.679793            | 1.9E+08       |
| smn          | 2813872      | 3.460772             | 813076.5      | 2.543944          | 1106106       | 2.433844            | 1156143       |
| sn           | 4979456      | 3.124034             | 1593919       | 3.120518          | 1595715       | 2.655855            | 1874898       |
| so           | 7940363      | 2.992185             | 2653701       | 2.708393          | 2931762       | 2.508277            | 3165664       |
| sq           | 1.17E+08     | 2.979585             | 39248921      | 2.73659           | 42734026      | 2.815558            | 41535465      |
| sr           | 7.84E+08     | 4.795495             | 1.64E+08      | 3.16455           | 2.48E+08      | 4.331577            | 1.81E+08      |
| stq          | 2884429      | 2.993801             | 963467.2      | 2.787199          | 1034884       | 2.690831            | 1071947       |
| su           | 19806020     | 3.211327             | 6167549       | 3.37651           | 5865827       | 3.11505             | 6358171       |
| sv           | 9.91E+08     | 3.20815              | 3.09E+08      | 2.957759          | 3.35E+08      | 3.228023            | 3.07E+08      |
| sw           | 35936177     | 2.91445              | 12330346      | 3.787084          | 9489141       | 2.725329            | 13185994      |
| szl          | 7347967      | 3.358912             | 2187603       | 2.544334          | 2887973       | 2.958125            | 2483995       |
| szy          | 6192815      | 2.928042             | 2115002       | 3.039297          | 2037581       | 2.753526            | 2249049       |
| ta           | 3.06E+08     | 10.13499             | 30165058      | 10.02704          | 30489830      | 3.312023            | 92306924      |
| tay          | 1232811      | 2.576051             | 478566.2      | 2.76919           | 445188.3      | 2.525171            | 488209        |
| tcy          | 4611006      | 8.512955             | 541645.8      | 6.638766          | 694557.7      | 3.144578            | 1466335       |
| te           | 2.52E+08     | 8.549729             | 29447150      | 9.063892          | 27776717      | 3.237454            | 77766408      |
| tg           | 49825647     | 5.281643             | 9433740       | 2.829083          | 17611941      | 3.940163            | 12645580      |
| th           | 3.96E+08     | 8.271261             | 47823543      | 2.862464          | 1.38E+08      | 4.089545            | 96724945      |
| tk           | 7383654      | 3.667693             | 2013160       | 2.523091          | 2926431       | 2.4631              | 2997708       |
| tl           | 45797527     | 2.775242             | 16502176      | 3.24525           | 14112173      | 3.02519             | 15138727      |
| tly          | 1070456      | 3.351578             | 319388.7      | 2.374073          | 450894.2      | 2.408261            | 444493.3      |
| tn           | 1708110      | 2.639858             | 647046.2      | 3.149103          | 542411.6      | 2.677334            | 637989.2      |
| tr           | 5.87E+08     | 3.482104             | 1.69E+08      | 2.764463          | 2.12E+08      | 2.937163            | 2E+08         |
| trv          | 2706664      | 2.665858             | 1015307       | 2.623241          | 1031802       | 2.404338            | 1125742       |
| tt           | 1.29E+08     | 4.933108             | 26178434      | 2.920663          | 44216357      | 4.736173            | 27266963      |
| tum          | 5459856      | 3.069905             | 1778510       | 3.047064          | 1791842       | 2.854557            | 1912681       |
| tw           | 4118530      | 2.631451             | 1565118       | 2.860678          | 1439704       | 2.522578            | 1632667       |
| tyv          | 6528290      | 5.510621             | 1184674       | 2.975143          | 2194278       | 3.877014            | 1683845       |
| udm          | 2982821      | 5.038611             | 591992.7      | 2.958814          | 1008114       | 3.557863            | 838374.3      |
| ug           | 17741860     | 6.217129             | 2853706       | 2.574344          | 6891799       | 3.036012            | 5843805       |
| uk           | 2.3E+09      | 5.424781             | 4.24E+08      | 3.209132          | 7.16E+08      | 4.452019            | 5.16E+08      |
| ur           | 1.68E+08     | 4.031675             | 41577728      | 5.905548          | 28384813      | 4.152323            | 40369661      |
| uz           | 2.1E+08      | 3.270747             | 64285816      | 2.768941          | 75936117      | 2.790661            | 75345110      |
| vec          | 19276543     | 2.667171             | 7227336       | 3.076717          | 6265297       | 3.048724            | 6322824       |
| vep          | 6423002      | 3.340101             | 1922996       | 2.645025          | 2428333       | 2.570651            | 2498590       |
| vi           | 7.47E+08     | 2.828246             | 2.64E+08      | 4.607117          | 1.62E+08      | 3.510764            | 2.13E+08      |
| vls          | 6985406      | 2.779891             | 2512834       | 2.846496          | 2454037       | 2.700319            | 2586882       |
| vo           | 6582571      | 2.806364             | 2345587       | 2.703518          | 2434817       | 2.661834            | 2472946       |
| wa           | 9550310      | 2.458642             | 3884384       | 2.794199          | 3417906       | 2.460766            | 3881031       |
| war          | 1.05E+08     | 3.290553             | 31784461      | 3.186653          | 32820783      | 3.690181            | 28342365      |
| wo           | 2094574      | 2.400075             | 872712        | 2.73538           | 765734.2      | 2.446062            | 856304.7      |
| wuu          | 15985963     | 3.970011             | 4026680       | 4.025088          | 3971581       | 3.612661            | 4424983       |
| xh           | 2505472      | 5.03599              | 497513.3      | 2.826526          | 886413.9      | 2.368852            | 1057674       |
| xmf          | 12948576     | 8.046115             | 1609295       | 2.743687          | 4719408       | 4.326371            | 2992942       |
| yi           | 26905047     | 4.673416             | 5757040       | 2.147134          | 12530677      | 3.193248            | 8425605       |
| yo           | 8861465      | 2.830447             | 3130765       | 3.656377          | 2423564       | 2.580695            | 3433751       |
| zea          | 2620396      | 2.786056             | 940539.5      | 2.82913           | 926219.7      | 2.65124             | 988366.2      |
| zh           | 5.03E+09     | 3.687222             | 1.36E+09      | 3.552345          | 1.42E+09      | 3.42038             | 1.47E+09      |
| zh-classical | 10098073     | 3.569372             | 2829090       | 3.35671           | 3008324       | 3.177208            | 3178285       |
| zh-min-nan   | 47410572     | 2.36058              | 20084289      | 2.552651          | 18573074      | 2.519768            | 18815450      |
| zh-yue       | 64950815     | 3.802575             | 17080745      | 3.706028          | 17525721      | 3.302348            | 19668068      |
| zu           | 3792429      | 3.822076             | 992243.3      | 3.093608          | 1225892       | 2.477525            | 1530733       |


## test
```python

from transformers import AutoModelForMaskedLM, AutoTokenizer
model_path = "JuanbaiAI/GlobalTokenizer"

tokenizer = AutoTokenizer.from_pretrained(model_path)
print(tokenizer("Hello, 世界"))

doc = [
    "                                                          ",
    "ننصحك أن تستخدم كلمة سر فريدة لا تستخدمها على أي موقع شبكي آخر.",
    "แก๊สมีสกุล หรือ แก๊สมีตระกูล (ในอดีตเรียกว่า แก๊สเฉื่อย หรือบางครั้งใช้ชื่อว่า aerogens) เป็นกลุ่มของธาตุทางเคมีที่มีสมบัติคล้ายกัน ภายใต้ภาวะมาตรฐานสำหรับอุณหภูมิและความดันธาตุเหล่านี้ต่างไม่มีกลิ่น ไม่มีสี เป็นแก๊สอะตอมเดี่ยวซึ่งไม่มีความว่องไวต่อปฏิกริยาเคมี แก๊สมีสกุลที่เกิดในธรรมชาติทั้งหกธาตุ ได้แก่ ฮีเลียม (He), นีออน (Ne), อาร์กอน (Ar), คริปทอน (Kr), ซีนอน (Xe) (ในภาพตามลำดับ) และเรดอน (Rn)  โอกาเนสซอน (Og) เป็นธาตุสังเคราะห์มีความเป็นกัมมันตรังสีสูงมาก แม้ว่า IUPAC จัดโอกาเนสซอนเป็นแก๊สมีสกุลหรือหมู่ที่ 18 มันอาจไม่เฉื่อยทางเคมีเหมือนธาตุอื่นในหมู่เดียวกัน และถูกทำนายว่าจะสลายและแผ่กัมมันตรังสีเนื่องจากปรากฏการณ์สัมพัทธ์ เนื่องจากครึ่งชีวิตที่สั้นเพียง 0.7 ไมโครวินาทีของไอโซโทปตัวเดียว ทำให้คุณสมบัติทางเคมีของโอกาเนสซอนยังไม่มีการศึกษามากนัก     首先8.88设置 st。art_new_word=True 和 output=[açaí]，output 就是最终� no such name",
    "ᠠᠩᠬ᠎ᠠ ᠳᠤᠮᠳᠠᠳᠤ ᠶᠢᠨ ᠮᠣᠩᠭᠤᠯ ᠬᠡᠯᠡᠨ ᠤ ᠬᠢᠴᠢᠶᠡᠯ ᠤᠨ ᠪᠡᠯᠡᠳᠭᠡᠮᠵᠢ ᠮᠢᠨᠣ ᠳᠤᠰᠬᠠᠢ ᠰᠡᠳᠤᠪ ᠤᠨ ᠭᠤᠣᠯ ᠠᠭᠤᠯᠭ᠎ᠠ ᠨᠢ ᠠᠩᠬ᠎ᠠ ᠳᠤᠮᠳᠠᠳᠤ ᠶᠢᠨ ᠮᠣᠩᠭᠤᠯ ᠬᠡᠯᠡᠨ ᠤ ᠬᠢᠴᠢᠶᠡᠯ ᠳᠤ ᠳᠠᠭᠠᠯᠳᠤᠭᠤᠯᠤᠨ ᠵᠣᠬᠢᠶᠠᠭᠰᠠᠨ ᠺᠣᠣᠷᠰᠸᠠᠢᠷ ᠪᠣᠯᠤᠨ᠎ᠠ᠃ ᠲᠤᠰ ᠺᠣᠣᠷᠰᠸᠠᠢᠷ ᠲᠤ ᠪᠠᠨ ᠰᠤᠷᠤᠭᠴᠢ ᠶᠢᠨ ᠰᠤᠷᠬᠤ ᠤᠷᠮ᠎ᠠ ᠪᠠᠬ᠎ᠠ ᠶᠢ ᠳᠠᠳᠠᠬᠤ᠂ ᠬᠢᠴᠢᠶᠡᠯ ᠤᠨ ᠭᠤᠣᠯᠳᠠᠯᠭ᠎ᠠ ᠬᠦᠴᠢᠷᠳᠡᠯᠭᠡ ᠶᠢ ᠳᠠᠭᠤᠯᠵᠤ ᠬᠢᠴᠢᠶᠡᠯᠯᠡᠭᠡ ᠶᠢᠨ ᠶᠠᠪᠤᠴᠠ ᠪᠠᠨ ᠪᠦᠷᠢᠨᠵᠢᠬᠦᠯᠵᠤ᠂ ᠮᠡᠳᠡᠯᠭᠡ ᠶᠢᠨ ᠥᠷᠭᠡᠳᠭᠡᠯ ᠢᠶᠠᠷ ᠳᠠᠮᠵᠢᠨ ᠵᠢᠭᠠᠨ ᠰᠤᠷᠭᠠᠬᠤ ᠴᠢᠨᠠᠷ ᠴᠢᠨᠰᠠᠭ᠎ᠠ ᠪᠠᠨ ᠳᠡᠭᠡᠭᠰᠢᠯᠡᠭᠦᠯᠬᠦ ᠬᠡᠰᠡᠨ ᠵᠣᠷᠢᠯᠭ᠎ᠠ ᠲᠠᠢ᠃ ᠲᠤᠰ ᠬᠥᠮᠦᠨ ᠤ ᠬᠤᠪᠢ ᠶᠢᠨ ᠬᠡᠪ ᠴᠢᠨᠠᠷ᠂ ᠮᠡᠳᠡᠯᠭᠡ ᠴᠢᠳᠠᠪᠤᠷᠢ ᠶᠢᠨ ᠮᠡᠬᠦᠰ ᠠᠴᠠ ᠪᠣᠯᠵᠤ ᠪᠡᠯᠡᠳᠭᠡᠮᠵᠢ ᠶᠢᠨ ᠳᠤᠮᠳᠠ ᠠᠰᠠᠭᠤᠳᠠᠯ ᠣᠷᠤᠰᠢᠵᠤ ᠪᠠᠢᠭ᠎ᠠ ᠨᠢ ᠯᠠᠪᠳᠠᠢ᠃ ᠠᠩᠬᠠᠷᠤᠯ ᠶᠢᠨ ᠬᠠᠨᠳᠤᠭᠤᠯᠤᠭᠰᠠᠨ ᠲᠠ ᠪᠦᠬᠦᠨ ᠢᠯᠡᠭᠦᠬᠡᠨ ᠰᠠᠨᠠᠯ ᠵᠥᠪᠯᠡᠯᠭᠡ ᠥᠭᠬᠦ ᠶᠢ ᠬᠦᠰᠡᠵᠤ ᠪᠠᠢᠨ᠎ᠠ᠃    的输出คุณจะจัดพิธีแต่งงานเมื่อไรคะ탑승 수속해야pneumonoultramicroscopicsilicovolcanoconiosis",
    "四色定理是一个著名的数学定理：如果在平面上划出一些邻接的有限区域，那么可以用四种颜色来给这些区域染色，使得每两个邻接区域染的颜色都不一样。被称为邻接的两个区域是指它们有一段公共的边界，而不仅仅是一个公共的交点。四色问题最早是由英国数学家法兰西斯·古德里在1852年提出的。人们发现，要证明宽松一点的“五色定理”很容易，但四色问题却出人意料地异常困难。1976年，数学家凯尼斯·阿佩尔和沃夫冈·哈肯借助电子计算机首次得到一个完全的证明，四色问题也终于成为四色定理。这是首个主要借助计算机证明的定理。",
    """est 𗴂𗹭𘜶𗴲𗂧, ou "phiow-bjij-lhjij-lhjij", ce que l'on peut traduire par « pays-grand-blanc-élevé » (白高大夏國). """,
    "蜂蜜とはミツバチが花の蜜を採集し、巣の中で加工、貯蔵したものをいう。約8割の糖分と約2割の水分によって構成され、ビタミンとミネラル類などの栄養素をわずかに含む。味や色は蜜源植物によって様々である。  本来はミツバチの食料であるが、しばしば他の生物が採集して食料としている。人類も「蜂蜜の歴史は人類の歴史」ということわざがあるように、古来、食用、薬用など様々な用途に用いている。人類は初め、野生のミツバチの巣から蜂蜜を採集していたが、やがてミツバチを飼育して採集すること（養蜂）を身に付けた。人類による蜂蜜の生産量は、世界全体で年間約120万tと推定される。",
    "[2]: Different from the embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models. For example, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 documents to get the final top-3 results.    ",
    "오스트레일리아청개구리(Australian green tree frog)는 오스트레일리아와 뉴기니 원산의 청개구리과에 속한 청개구리의 일종이다. 미국과 뉴질랜드에도 외래종으로 흘러들어갔는데, 뉴질랜드에서는 멸종된 것으로 여겨진다. 학명 명명자 존 화이트의 이름을 따 화이트청개구리(White's tree frog), 특유의 생김새에서 유래한 시무룩청개구리(dumpy tree frog) 등의 별명이 있으며, 학명은 리토리아 카에룰레아(Litoria caerulea)이다. 형태학적으로 오스트레일리아청개구리속의 다른 청개구리, 특히 예쁜청개구리와 왕청개구리와 많이 닮았다. ",
    "Элой была придумана студией Guerrilla Games, которая хотела создать сильного и героического женского персонажа в рамках своей совершенно новой игры. Лицо героини было создано на основе внешности нидерландской актрисы Ханны Хукстры; роль персонажа при озвучивании на английском языке и в сценах захвата движения исполняет американская актриса Эшли Бёрч. ",
    "網上瘋傳一張照片，指在港鐵的機場快綫列車有床蝨蹤影。. 通訊事務管理局公布，香港寬頻 (01310.HK)於1至4月期間發生三宗涉及固定和流動電訊服務用戶的錯誤收帳事件。. 三宗事件共影響超過1萬名客戶",
    "하는데 카운터가 어디에 있어요ꆃꎭꆈꌠꊨꏦꏲꅉꆅꉚꅉꋍꂷꂶꌠلأحياء تمارين تتطلب من [MASK] [PAD] [CLS][SEP]" "def normalize(text, maxlen=0, isolate_digits=False):     text = unicodedata.normalize('NFC', text)     if maxlen > 0:         if isolate_digits:             regex = '\d|[^\n\d]{,%d}\n{1,100}|[^\n\d]{1,%d}' % (maxlen, maxlen)         else:             regex = '.{,%d}\n{1,100}|.{1,%d}' % (maxlen, maxlen)     else:         if isolate_digits:             regex = '\d|[^\n\d]*\n+|[^\n\d]+'         else:             regex = '.*\n+|.+'     return [t.encode() for t in re.findall(regex, text)]",
    "ꃛꏢꄧꍏꇩꇭꍣꄈ꒜ꀉꒉꊰꏁꃢꌠꇩꏤꑭꊂꁧꎁ꒜ꈿꅉꇬꄉꅇꄜꌠ",
]
for i,line in enumerate(doc):
    print(i,len(tokenizer.tokenize(line)))
```
