Sliding window for Long Documents
Hi,Is there any way to chunk a large document with left and right context? The default param in tokenizer provide only left context, is there a way we can provide right context also and predict for the central part not for the context.flert-paper1536×780 240 KBSimilar to the approach mention in this paper:https://arxiv.org/pdf/2011.06993.pdf
2023-02-09T22:34:33Z
return_over_flowing_tokensmight help:github.comhuggingface/notebooks/blob/main/examples/question_answering.ipynb{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X4cRE8IbIrIV"
   },
   "source": [
    "If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets. Uncomment the following cell and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "MOsHUjgdIrIW",This file has been truncated.show original