Using mixup on RoBERTa
Hello everyone!I tried to apply the technique of data augmentation, mixup, popularly used on computer vision, but in this case on NLP.The algorithm developed is in two phases:The first phase gets the representation for each sentence of the batch, computing the mean of the correspondent hidden states of the last layer. The fragment below shows the corresponding module.class LanguageModel(nn.Module):
 
  def __init__(self, pretrained_model_name, device="cuda:0", anonymized_tokens=False):
    super(LanguageModel, self).__init__()
    # Load tokenizer
    self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
    # Load model
    self.config = AutoConfig.from_pretrained(pretrained_model_name)
    self.config.output_hidden_states = True
    self.model = AutoModel.from_pretrained(pretrained_model_name, config=self.config).to(device)

  def forward(self, input_ids, attention_mask):
    outputs = self.model(
        input_ids=input_ids,
        attention_mask=attention_mask,
    )
    activations = torch.mean(outputs[0], axis=1)
    return activationsAfter that, it generates the mixup examples using the function proposed on theoriginal code, but  being  the input, the representations computed on the previous step, instead the images, like originally.One time the mixup examples are generated, the second phase realizes the predictions (the fragment below shows the corresponding module). Finally, is computed the loss, in the same way as in the original work.class ClassifierLayer(nn.Module):
 
  def __init__(self, num_classes, dropout_rate=0.1, petrained_size=768, device="cuda:0"):
    super(ClassifierLayer, self).__init__()
    self.layer = nn.Linear(petrained_size, num_classes, bias=True).to(device)
    self.drop = nn.Dropout(dropout_rate)

  def forward(self, z):
    activations = self.layer(self.drop(z))
    return activationsIn the fragment of the code below, is shown a summary of the training loop proposed, however the full script used ishere:for idx_epoch in range(0, args.num_train_epochs):
        language_model.train()
        classifier_layer.train()
        accs = 0; ps = 0; rs = 0; f1s = 0; lss = 0
        for (idx_batch, train_batch) in enumerate(train_dataloader):
            # 0: input_ids, 1: attention_mask, 2:token_type_ids, 3: labels
            batch_train = tuple(data_.to(device) for data_ in train_batch)
            labels_train = batch_train[-1]
            inputs = {
                'input_ids': batch_train[0],
                'attention_mask': batch_train[1],
            }
            optimizer.zero_grad()
            # 1st phase: conextual embeddings
            contextual_embeddings = language_model(
                input_ids=inputs['input_ids'],
                attention_mask=inputs['attention_mask'],
            )
            # 2nd phase: mixup
            inputs, targets_a, targets_b, lam = mixup_data(contextual_embeddings, labels_train, args.alpha_mixup, use_cuda)
            inputs, targets_a, targets_b = map(Variable, (inputs, targets_a, targets_b))
            predictions = classifier_layer(inputs)
            loss = mixup_criterion(criterion, predictions, targets_a, targets_b, lam)
            
            # 2nd phase: standard
            # predictions = classifier_layer(contextual_embeddings)
            # loss = criterion(predictions, labels_train)
            
            lss += loss
            loss.backward()
            optimizer.step()
            scheduler.step()Experimenting with this approach, the results obtained are very poor…Have any of you worked on an approximation similar to this one with successful/good results?Thanks.
2020-07-15T20:55:49Z
Hi@franborjavalero!This is really interesting. I remember@sguggergot a little bump using mixup after embeddings with ULMFiT. Would be really awesome to share this code as implementation for this is not trivial.
2020-07-15T21:08:00Z
It wasn’t for transformers, but ULMFiT. Didn’t get the chance to try it on transformers model.Also, I was using themanifold mixupversion, which applies the mixup at a random layer (not necessarily the embedding), though this could also mess up the attention mechanism in tansformers.
2020-07-15T21:16:17Z
Thanks for sharing@sgugger.Data augmentation for text classification really is a tough one. Is there anything you consider promising?@franborjavaleroyou might want to checkout thisthread
2020-07-15T21:23:22Z
Haven’t found anything that really stands out for now, so no magic trick on my side
2020-07-15T21:26:46Z
Syntactic Data Augmentation Increases Robustness to Inference Heuristicsdiscussed in the other thread seems interesting for NLI
2020-07-21T18:20:25Z
You might find our work on Cost-Sensitivity to be of interest. We found it to be a good alternative to data augmentation.  [Paper hereandCode here]