Warm start with BigBird
I am interested in taking a Roberta model and turning into a BigBird model that can handle long sequences. In the BigBird paper, they mention doing a warm start from the Roberta checkpoint, which I interpret to mean they loaded all the pre-trained weights into the new model before doing more training on long sequences. It looks like all of the model components are identical, apart from the full attention being replaced with block sparse attention and the positional embeddings being different sizes.The main problem I have right now is dealing with the positional embeddings. BigBird’s embeddings will be much larger than Roberta’s because the positional embeddings have shape (max_position_embeddingsxhidden_size)Roberta hasmax_position_embeddings= 514BigBird hasmax_position_embeddings= 4096Any ideas on how I should handle this?My only ideas right now are:Reinit all positional embeddingsUse Roberta positional embeddings for the first 514 and then reinit the remaining.Use the pretrained BigBird positional embeddings (but this feels like cheating and is definitely not what the original authors did).
2022-07-22T14:56:57Z
For anyone who interested, you can tile the roberta position embeddings 8 times to be the right shape for bigbird[512, 512, 512… ]
2024-04-11T14:34:34Z
Hey! How did you do this exactly? I’m trying to do the exact same thing
2024-05-02T00:00:19Z
Follow thisgithub.comallenai/longformer/blob/master/scripts/convert_model_to_long.ipynb{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aad_1s7ybD5o"
   },
   "source": [
    "# `RoBERTa` --> `Longformer`: build a \"long\" version of pretrained models\n",This file has been truncated.show original
2024-05-02T12:00:31Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.