
This model has the safetensors weights for the [Madlad-400](https://github.com/google-research/google-research/tree/master/madlad_400) 8B param **language model**.

The HF transformers code to run inference is not ready yet. The [original implementation](https://github.com/google/flaxformer/blob/ea17eb012a1d340ddff017b7a534c2162aaec34c/flaxformer/architectures/t5/t5_architecture.py#L1484) is in JAX/Flaxformer.

The model architecture is the same as [Palm 8B](https://arxiv.org/pdf/2204.02311.pdf).

It's a decoder-only T5 with 32 layers, 16 query heads, 1 KV head, and 4096 embedding size.

These are the main differences relative to the original T5 architecture:

- SwiGLU Activation
- Parallel Layers
- Multi-Query Attention
- RoPE Embeddings
- Shared Input-Output Embeddings
- No biases
- Bidirectional attention
- Layer Norm with `center_scale_at_zero` and final layer with `use_scale=False`

If you are looking for the language models models, here are the available versions:
- [3B](https://huggingface.co/jbochi/madlad400-3b-mt)
- [7B](https://huggingface.co/jbochi/madlad400-7b-mt)
- [7B-BT](https://huggingface.co/jbochi/madlad400-7b-mt-bt)
- [10B](https://huggingface.co/jbochi/madlad400-10b-mt)


Article: [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://arxiv.org/abs/2309.04662)

Abstract:

> We introduce MADLAD-400, a manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss the limitations revealed by self-auditing MADLAD-400, and the role data auditing had in the dataset creation process. We then train and release a 10.7B-parameter multilingual machine translation model on 250 billion tokens covering over 450 languages using publicly available data, and find that it is competitive with models that are significantly larger, and report the results on different domains. In addition, we train a 8B-parameter language model, and assess the results on few-shot translation. We make the baseline models available to the research community.




{"language": ["en", "ru", "es", "fr", "de", "it", "pt", "pl", "nl", "vi", "tr", "sv", "id", "ro", "cs", "zh", "hu", "ja", "th", "fi", "fa", "uk", "da", "el", "no", "bg", "sk", "ko", "ar", "lt", "ca", "sl", "he", "et", "lv", "hi", "sq", "ms", "az", "sr", "ta", "hr", "kk", "is", "ml", "mr", "te", "af", "gl", "fil", "be", "mk", "eu", "bn", "ka", "mn", "bs", "uz", "ur", "sw", "yue", "ne", "kn", "kaa", "gu", "si", "cy", "eo", "la", "hy", "ky", "tg", "ga", "mt", "my", "km", "tt", "so", "ku", "ps", "pa", "rw", "lo", "ha", "dv", "fy", "lb", "ckb", "mg", "gd", "am", "ug", "ht", "grc", "hmn", "sd", "jv", "mi", "tk", "ceb", "yi", "ba", "fo", "or", "xh", "su", "kl", "ny", "sm", "sn", "co", "zu", "ig", "yo", "pap", "st", "haw", "as", "oc", "cv", "lus", "tet", "gsw", "sah", "br", "rm", "sa", "bo", "om", "se", "ce", "cnh", "ilo", "hil", "udm", "os", "lg", "ti", "vec", "ts", "tyv", "kbd", "ee", "iba", "av", "kha", "to", "tn", "nso", "fj", "zza", "ak", "ada", "otq", "dz", "bua", "cfm", "ln", "chm", "gn", "krc", "wa", "hif", "yua", "srn", "war", "rom", "bik", "pam", "sg", "lu", "ady", "kbp", "syr", "ltg", "myv", "iso", "kac", "bho", "ay", "kum", "qu", "za", "pag", "ngu", "ve", "pck", "zap", "tyz", "hui", "bbc", "tzo", "tiv", "ksd", "gom", "min", "ang", "nhe", "bgp", "nzi", "nnb", "nv", "zxx", "bci", "kv", "new", "mps", "alt", "meu", "bew", "fon", "iu", "abt", "mgh", "mnw", "tvl", "dov", "tlh", "ho", "kw", "mrj", "meo", "crh", "mbt", "emp", "ace", "ium", "mam", "gym", "mai", "crs", "pon", "ubu", "fip", "quc", "gv", "kj", "btx", "ape", "chk", "rcf", "shn", "tzh", "mdf", "ppk", "ss", "gag", "cab", "kri", "seh", "ibb", "tbz", "bru", "enq", "ach", "cuk", "kmb", "wo", "kek", "qub", "tab", "bts", "kos", "rwo", "cak", "tuc", "bum", "cjk", "gil", "stq", "tsg", "quh", "mak", "arn", "ban", "jiv", "sja", "yap", "tcy", "toj", "twu", "xal", "amu", "rmc", "hus", "nia", "kjh", "bm", "guh", "mas", "acf", "dtp", "ksw", "bzj", "din", "zne", "mad", "msi", "mag", "mkn", "kg", "lhu", "ch", "qvi", "mh", "djk", "sus", "mfe", "srm", "dyu", "ctu", "gui", "pau", "inb", "bi", "mni", "guc", "jam", "wal", "jac", "bas", "gor", "skr", "nyu", "noa", "sda", "gub", "nog", "cni", "teo", "tdx", "sxn", "rki", "nr", "frp", "alz", "taj", "lrc", "cce", "rn", "jvn", "hvn", "nij", "dwr", "izz", "msm", "bus", "ktu", "chr", "maz", "tzj", "suz", "knj", "bim", "gvl", "bqc", "tca", "pis", "prk", "laj", "mel", "qxr", "niq", "ahk", "shp", "hne", "spp", "koi", "krj", "quf", "luz", "agr", "tsc", "mqy", "gof", "gbm", "miq", "dje", "awa", "bjj", "qvz", "sjp", "tll", "raj", "kjg", "bgz", "quy", "cbk", "akb", "oj", "ify", "mey", "ks", "cac", "brx", "qup", "syl", "jax", "ff", "ber", "tks", "trp", "mrw", "adh", "smt", "srr", "ffm", "qvc", "mtr", "ann", "kaa", "aa", "noe", "nut", "gyn", "kwi", "xmm", "msb"], "license": "apache-2.0", "library_name": "transformers", "tags": ["text-generation-inference"], "datasets": ["allenai/MADLAD-400"]}
transformers safetensors t5 text-generation text-generation-inference custom_code en ru es fr de it pt pl nl vi tr sv id ro cs zh hu ja th fi fa uk da el no bg sk ko ar lt ca sl he et lv hi sq ms az sr ta hr kk is ml mr te af gl fil be mk eu bn ka mn bs uz ur sw yue ne kn kaa gu si cy eo la hy ky tg ga mt my km tt so ku ps pa rw lo ha dv fy lb ckb mg gd am ug ht grc hmn sd jv mi tk ceb yi ba fo or xh su kl ny sm sn co zu ig yo pap st haw as oc cv lus tet gsw sah br rm sa bo om se ce cnh ilo hil udm os lg ti vec ts tyv kbd ee iba av kha to tn nso fj zza ak ada otq dz bua cfm ln chm gn krc wa hif yua srn war rom bik pam sg lu ady kbp syr ltg myv iso kac bho ay kum qu za pag ngu ve pck zap tyz hui bbc tzo tiv ksd gom min ang nhe bgp nzi nnb nv zxx bci kv new mps alt meu bew fon iu abt mgh mnw tvl dov tlh ho kw mrj meo crh mbt emp ace ium mam gym mai crs pon ubu fip quc gv kj btx ape chk rcf shn tzh mdf ppk ss gag cab kri seh ibb tbz bru enq ach cuk kmb wo kek qub tab bts kos rwo cak tuc bum cjk gil stq tsg quh mak arn ban jiv sja yap tcy toj twu xal amu rmc hus nia kjh bm guh mas acf dtp ksw bzj din zne mad msi mag mkn kg lhu ch qvi mh djk sus mfe srm dyu ctu gui pau inb bi mni guc jam wal jac bas gor skr nyu noa sda gub nog cni teo tdx sxn rki nr frp alz taj lrc cce rn jvn hvn nij dwr izz msm bus ktu chr maz tzj suz knj bim gvl bqc tca pis prk laj mel qxr niq ahk shp hne spp koi krj quf luz agr tsc mqy gof gbm miq dje awa bjj qvz sjp tll raj kjg bgz quy cbk akb oj ify mey ks cac brx qup syl jax ff ber tks trp mrw adh smt srr ffm qvc mtr ann aa noe nut gyn kwi xmm msb dataset:allenai/MADLAD-400 arxiv:2204.02311 arxiv:2309.04662 license:apache-2.0 autotrain_compatible has_space region:us