T5 Fine Tuning - Text to Text Generation
I was working on an interesting problem of generating inferences from the excel data. I wrote a python program to generate rules from the data in the form of RDF Triple and now training using T5-Base model. with some 10k training data of rdf rules and inferences I was able to get some 80% to 85% test accuracy. I’m using ADAMW optimizer with lr of 1e-5.One issue I have seen is the model is not able to generalize well on new numbers. eg. if I pass a rule of“Critical” | priority_ticketshare | “23.09%”to the model, it returns inference asCritical priority tickets accounted for 22.09% of total tickets.While the statement is correct, the number it has taken is wrong. Any idea how to solve this
2021-04-07T15:12:40Z
You are expecting a model to behave like a database :-). It will never do that buddy. Not even GPT-3. Use t5-small for less memory footprints. I gurantee you, there won’t be much drop n accuracy and you get better inference time.
2021-04-07T16:11:32Z
I will definitely try with t5-small. I got this thought from WEBNLG2017 dataset where many of the training samples were numerical relations like “ Aargas airport | runwayslength | 24.5” and where T5-base showed a sota results. Hence, thought of applying same concept on reporting automations.