GPT2 returns sequence of <|endoftext|> after finetuning
Here’s I tried to finetune gpt2github.comtempdeltavalue/temp_l/blob/main/finetune_seq2seq.ipynb{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "225b0a06-4dd3-4c84-89ed-86de4adcd24b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\tempdelta\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# https://huggingface.co/docs/peft/index\n",
    "\n",This file has been truncated.show originalbut after few iterations I started to get same sequences, for ex.prompt: Prime number ismodel output: <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>By the way, my text divided into sentences without [EOS] tokens, in other words every item is separate sentence
2024-01-27T00:20:23Z
upd:It’s reproducible even without finetuningScreenshot 2024-01-27 0211201017×643 51.7 KB
2024-01-31T11:05:56Z
one top screenshot - correct outputScreenshot 2024-01-31 1240151001×242 18.3 KBon bottom - eos sequenceScreenshot 2024-01-31 123959221018×570 45.6 KBBut code generally the same