Building a custom Squad 2.0 style dataset, is it worth it?
Was wondering what the experts think and whether this is a sensible approach. The pre-trained Squad 2.0 models perform well in a custom domain, but can be greatly improved, given the target domain is rather narrow and the vocabulary is different but there is overlap.Do you think it is worth obtaining a custom dataset, say 1000 observations, using the same methodology as Squad v2.0 but derived from data of the target domain?Is 1000 observation enough for the fine-tuning?
2020-07-20T15:30:50Z
Hi@swayson, not an expert here but fine-tuning on your domain should give better results. I can’t comment on if 1000 examples will e enough or not, you’ll probably need to experiment.Also have look at this question generationmodels. You can try to create synthetic QA corpora using these models. Synthetic QA corpora has shown to improve results for QA.
2020-07-20T15:50:54Z
Thank you@valhalla; I am going to give the synthetic QA models a shot and see if I can get some improvements.
2020-07-20T16:06:43Z
Here’s a relevantpaper. See table 2 for Synthetic QA results.