Multiple choice with variable number of choices
Hi all,Similar question toMultiple choice with variable length options🤗TransformersHello! I have a beginner question. I am trying to create a model that makes predictions on the QAngaroo dataset with DistilBert. In this dataset, we get a list of supports and some candidate answers (between 2~100), and we need to choose the right answer for the model. Right now, I am trying to use TFDistilBertForMultipleChoice, but I am running into a problem since num_choices is a value that is fixed with the entire batch size. I was wondering how I could go about making that value dynamic. 
I…The reply there is to go full-blown text-to-text—which is a great idea!—but I’m interested in getting a discriminative BERT-esque baseline if possible (due to the dataset’s particular size, structure, and text content).Since multiple choice models (likeRobertaForMultipleChoice) detect the number of questions dynamically per batch, it seems like the main challenge is getting each batch to have a consistent number of choices.Going off of therun_swag.pyexample, there are two main user-provided data processing functions:apreprocess_function()— for adding new features to the Dataseta collate function — for turning a raw batch into tensorsUnfortunately, while it’s no problem to add the number of choices for an example in 1., by the time 2. comes around, we’ve already been given a batch, so it’s too late to ensure they all have the same number of choices.In other words, it seems like thesampleris the place where we’d make sure the number of choices is consistent per batch.To my delight, I found the--group_by_lengthand--length_column_nameoptions, which enable the(Distributed)LengthGroupedSampler. This opens up a potential way for doing this:Add a feature with the number of choicesPass this feature as the--length_column_nameand use--group_by_lengthUnfortunately for me, this constraint is a soft one rather than a hard one. (This makes sense for the original purpose, of course, which is just helping with padding.) This means that some batches do end up with multiple “lengths” (choices).I wrote a quick test that injects a number from 1-4 as the feature and checked how many batches had multiple “lengths.” I was hoping it would be just three batches (at the borders between 1-2, 2-3, 3-4). Over 500 batches, there were 14 that ended up with mixed numbers. I could truncate these batches, which would only lose ~3% of the data. Not a huge loss, but it makes me wonder whether I can do better!So, to complete the very long-winded question,I wonder whether anyone more familiar with Huggingface Transformers can recommend an approach to implement multiple choice with a variable number of choices. Right now my main options are:Write my own sampler to do this. Given this is just for a baseline, and my trepidation at debugging a custom distributed sampler, I worry this might not be worth the investment.Create multiple Dataset objects, each with a consistent number of choices. Do an outer training loop. (This would be less ideal because each number of choices corresponds to a question format, so this would increase coarse patterns into training / reduce how shuffled it is.)Just throw away the ~3% of mixed-data batches (less if we take the majority, so maybe ~1.5%).??? (better approach I can’t see?)Huge thanks for your time!
2021-08-19T17:40:51Z
In case anyone in the future is reading this, the above does work pretty well, but only for training. For evaluation, the (Distributed)LengthGroupedSampler is not used. Furthermore, even if it was, by nature of throwing some data away in the collator, we skip some of the evaluation set (which is a no-no for comparing results between methods).I provided an example implementation of a batch sampler that groups based on a provided feature in the following comment:Option for `(Distributed)LengthGroupedSampler` to treat groups as a hard constraint · Issue #12995 · huggingface/transformers · GitHubIt also requires a change to Transformers itself to support a batch sampler, which is in a PR linked to that issue.