Difference in dimensions of T0 vs T5 models
Why is the dimension of T0 3B = 2048huggingface.coconfig.json · bigscience/T0_3B at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.whereas the dimension of T5 3B = 1024 (same as in original paper)huggingface.coconfig.json · t5-3b at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.although T0 models should be based on corresponding T5 models?
2022-11-17T14:09:37Z
cc@VictorSanh
2022-11-17T15:25:55Z
Hi@cookiemonster,that’s a great question!T0 is based onT5+LMwhich itself is based onT5 v1.1.T5+LM is essentially T5 but with some additional steps of autoregressive language modeling.
2022-11-17T15:50:28Z
Thanks Victor this was really helpful!I updated the model description in case it makes things clearer for someone elsehuggingface.cobigscience/T0_3B · update base models description for more clarityWe’re on a journey to advance and democratize artificial intelligence through open source and open science.