Using truncated fragments as input samples in training
Hi!I am using thetokenizerslibrary, roughly following therun_mlm.pyscript to train a Masked Language Model (MobileBert) from scratch.Since I am training an unsupervised model using truncated sentences, I was wondering if the truncated (left-out) fragments are included by default in the dataset for training since they would be valid examples for my use case (MLM setting). If they are not used (which I believe to be the case) I wanted to ask if there is any easy way in which I might include them in my training dataset (maybe by usingreturn_overflowing_tokensandstridein a smart way?).As an additional related question, I would like to know if there is any native way of sorting by length before batching to reduce the dataset size to the minimum. Something along these lines:pommedeterresautee gistandMcCormickML blogpost.EDIT: The best way I have found to do the smart batching is to create an ‘sample_length’ column and use the.sortmethod to sort by that column before tokenizing.Thanks in advance!
2021-06-21T13:06:41Z
By default, those are not included (unless you use the--line_by_lineoption which will concatenate all the samples then create block of the size you picked). Usingreturn_overflowing_tokensis definitely an option to get those truncated part!strideis only if you want some overlap between the two parts of a long sentence, which is useful for question answering, but not necessarily for masked language modeling pretraining.For the sorting by length before batching, we have the--group_by_lengthoption in theTrainer, though it’s for the dataset so it happens after tokenization, which may not be what you are looking for.
2021-06-30T21:27:21Z
Thanks Sylvain! As per yourvideo, I understand that the--group_by_lengthoption is compatible with theDataCollatorWithPadding. Is it also compatible with theDataCollatorForLanguageModelling?I understand it is, since according to thedocstheDataCollatorForLanguageModellingdynamically pads to make batches even.
2021-07-01T12:38:42Z
Yes, it’s compatible with any data collator, it changes the sampler of the dataset only.