How to train TFT5ForConditionalGeneration model?
My code is as follows:batch_size=8
sequence_length=25
vocab_size=100
import tensorflow as tf
from transformers import T5Config, TFT5ForConditionalGeneration
configT5 = T5Config(
    vocab_size=vocab_size,
    d_ff =512, 
)  
model = TFT5ForConditionalGeneration(configT5)

model.compile(
    optimizer = tf.keras.optimizers.Adam(),
    loss = tf.keras.losses.SparseCategoricalCrossentropy()
)
input = tf.random.uniform([batch_size,sequence_length],0,vocab_size,dtype=tf.int32)
labels = tf.random.uniform([batch_size,sequence_length],0,vocab_size,dtype=tf.int32)
input = {'inputs': input, 'decoder_input_ids': input}
model.fit(input, labels)It generates an error:logits and labels must have the same first dimension, got logits shape [1600,64] and labels shape [200] [[node sparse_categorical_crossentropy_3/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at C:\Users\FA.PROJECTOR-MSK\Google Диск\Colab Notebooks\PoetryTransformer\experiments\TFT5.py:30) ]] [Op:__inference_train_function_25173] Function call stack: train_functionI dont understand - why the model returns a tensor of [1600, 64]. According toT5model returns [batch_size, sequence_len, vocab_size].
2020-08-28T15:32:29Z
Pinging@patrickvonplaten
2020-09-01T10:26:47Z
Thanks for posting this! I’ll open an issue about it on work on it asap.Issue:https://github.com/huggingface/transformers/issues/6876
2020-09-06T12:02:58Z
Using high-level keras to fine-tune T5 seems tricky due to limited flexibility and code controls. I wrote something on fine-tuning T5 with customized training loop in TF2.0:https://github.com/wangcongcong123/ttt. Hope this helps.
2020-10-01T15:56:26Z
This should helphttps://github.com/huggingface/transformers/pull/7428
2020-11-21T10:19:00Z
Thanks a lot. Am I right that I have to override train_step() to make TFT5ForConditionalGeneration working ?