What is the context as per run_clm?
I am using a dataset fromdatasetlibrary for text generation task. For each example in dataset, I want to provide sentence1 and label as context, and allow network to generatesentence2and this should be used for loss calculation. In therun_clmscript, I’m not able to find this distinction as to what is being used as context.Based on my understandingattention_maskandtoken_type_idsare responsible for controlling what network sees and what not. So from this, I wantlabel+sentence1to have 0s andsentence2to have 1s inattention_mask.def tokenize_function(examples):
    tokenized_input = tokenizer(LABELS[examples['label']], examples['sentence1'])
    # not sure how to manage examples['sentence2']
    return tokenized_inputI want the output to be[CLS] [SEP] label [SEP] sentence 1 [SEP] sentence 2 (to be generated) [SEP]Can anyone clarify ?
2021-01-29T09:22:00Z
If I understand your question correctly, I think you want to calculate loss using only the generated text.This is not handled byrun_clmscript.To handle this, you can prepare thelabelssuch that, all the tokens will have -100 as value except the ones you want to include in loss, that way the -100 tokens will be ignored by theCrossEntropyloss function.
2021-01-29T14:59:02Z
Thanks@valhallafor your response.You mean like this ? Is that where 100 goes ?# inside group_texts function
marker =  [i for i in enumerate(examples.input_ids) if i==985][1] # 985 is id of ']', we take second occurence.
result["labels"] = result["input_ids"].copy()
result["labels"][:marker+1] = [-100]*(marker+1)I’m a bit confused about labels here. In the the script I seeresult["labels"] = result["input_ids"].copywhich just means we are taking everything as label and then perform CLM on both sentences. Also what aboutattention_maskandtoken_type_ids. I’m not clear on where to specify the context, and where to specify label (i.esentence2).Can you please clarify how to go about it ?
2021-01-29T18:46:50Z
Based on your suggestion, this is what I did.def group_texts(examples):
        # Concatenate all texts.
        second_sentence_start_pos = [i for i, n in enumerate(examples['input_ids']) if n == 985][1]
        token_type_ids = examples["token_type_ids"].copy()
        labels = examples["input_ids"].copy()

        token_type_ids[:second_sentence_start_pos+1] = [0]*(second_sentence_start_pos+1)
        token_type_ids[second_sentence_start_pos+1:] = [1]*(len(examples['input_ids'])-second_sentence_start_pos)
        labels[:second_sentence_start_pos+1] = [-100]*(second_sentence_start_pos+1)

        examples["token_type_ids"] = token_type_ids
        examples["labels"] = labels
        print(examples)
        return examplesIs this what you meant ? Is this the right way to deal withattention_masksandtoken_type_ids?
2021-02-01T06:44:11Z
Since GPT-2 is an auto-regressive decoder, we need to feed both the context and target as the input text.In the the script I seeresult["labels"] = result["input_ids"].copywhich just means we are taking everything as label and then perform CLM on both sentencesSince it’s a CLM, during training it predicts next token for every token in the input.So we pass thelabeld=input_idsand thenlablesare shifted to rightinside the model.i.e ifinput_ids = [BOS] tok1 tok2 tok3....thenlabels= tok1 tok2 tok3and all the tokens in the labels are considered when calculating the loss.Now if we want to use a CLM for tasks that require context and target (let’s say a summarization task) and we only want the model to generate a target (summary) thenDuring Trainig:we need to concatenate both the context (source_text) and target (summary) as  one single input texti.einput = [BOS] source_text [SEP] summaryand preparelabelssuch that tokens corresponding to the context (source text) will have -100 as the value. This way we ensure that the loss is only calculated for summary tokens so the model will (hopefully) only learn to generate summary (or target).And During inferencewe would feed the source text with sep and ask the model to generate the next tokens which will be summary in this case i.einput = [BOS] source text [SEP]Also, theattention_maskhere is the same asattention_maskin every other model, it specifies which tokens in the input should be attended and which should be ignored.For reference you could refer to this conv ai example which uses this approach for Conversational agent with GPT2github.comhuggingface/transfer-learning-conv-ai/blob/d4c760731e8f062abd3d61ccdc8fa29ccb5e3485/train.py#L56-L69def build_input_from_segments(persona, history, reply, tokenizer, lm_labels=False, with_eos=True):""" Build a sequence of input from 3 segments: persona, history and last reply. """bos, eos, speaker1, speaker2 = tokenizer.convert_tokens_to_ids(SPECIAL_TOKENS[:-1])sequence = [[bos] + list(chain(*persona))] + history + [reply + ([eos] if with_eos else [])]sequence = [sequence[0]] + [[speaker2 if (len(sequence)-i) % 2 else speaker1] + s for i, s in enumerate(sequence[1:])]instance = {}instance["input_ids"] = list(chain(*sequence))instance["token_type_ids"] = [speaker2 if i % 2 else speaker1 for i, s in enumerate(sequence) for _ in s]instance["mc_token_ids"] = len(instance["input_ids"]) - 1instance["lm_labels"] = [-100] * len(instance["input_ids"])if lm_labels:instance["lm_labels"] = ([-100] * sum(len(s) for s in sequence[:-1])) + [-100] + sequence[-1][1:]return instanceHope this helps
2021-02-01T16:04:12Z
Thanks for the response. I am using CTRL for my task and not much is available on it regarding how to finetune it properly. I have opened up a seperate thread but no response.Soattention_maskstays intact because we require the model to attend to the context. As you’re saying we do control the learning entirely by[-100]entries in labels, our attention_mask is basically a tensor filled with 1s. Coming totoken_type_ids, I am making the parts that need to be generated 1s, rest 0s as per BERT documentation.Now forlabels, I’ve made everything [-100] except the parts that needs to be generated and shifting automatically happens inside the model. Having done this, I’m still not able to get good generations.For CTRL, I don’t think we use aSEPtoken, I didn’t see it being used in original repo. Also the tokenizer (BPE based) is working quite differently i.e it is not adding special tokens as one would expect.So my sentences were like[Custom control Code][Sentence1][Sentence2].The documentation says thisThe PyTorch models can take the past as input, which is the previously computed key/value attention pairs.but this is only applicable during inference right ?
2021-02-02T16:16:19Z
Thanks@valhalla, I think the most important part was dealing with labels. I was performing shifting manually. Now generations have improved.