Longformer: The Long-Document Transformer
deep learning machine learning nlp natural language processing machine translation arxiv attention mechanism attention transformer bert roberta mlm convolution memory linear sliding dilated sparse
The Longformer extends the Transformer by introducing sliding window attention and sparse global attention. This allows for the processing of much longer documents than classic models like BERT.

Paper: https://arxiv.org/abs/2004.05150
Code: https://github.com/allenai/longformer

Abstract:
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.

Authors: Iz Beltagy, Matthew E. Peters, Arman Cohan

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at Longformer, the long document transformer by Is Beltaji, Matthew Peters and Armin Cohen of Allen AI. So the longformer is a variant of the transformer as you might have guessed. The longformer is a transformer that can deal with long documents, so it's aptly named. So I am going to discuss what differentiates the longformer from the transformer. If you don't know what a transformer is, watch the video on attention is all you need. I have a video on that. And I would also suggest you watch the video on BERT, because a lot of the architecture and training here is based on the BERT or variants of BERT. So I'll basically explain what makes the longformer different such that it gets long documents, right, so she can be applied to long documents. So what is the problem with the original transformer? If you have a transformer model and let's say you're doing an NLP task, which usually is where transformers are used, and you want to have a paragraph like this one right here, the abstract of the paper, and maybe you want to predict whether the paper gets accepted at a conference or not. Now the classic transformers, they have a limit, a very harsh limit, on the amount of tokens that they can look at at the same time. So what you would do in a classic transformer is you couldn't process this entire thing, let's say, you would divide it in chunks, you'd say, okay, here's my first chunk from here to here, my second chunk from here to here, and then here to here, and so on. So you go through the documents, split it up in chunks, process each of the chunks individually, and then maybe aggregate the predictions. But of course the drawback is that the model cannot make specific connections between, let's say, some word here, like operation, and somewhere down here, like language. It cannot connect the two on a neural level, at least not in the classic transformer architectures. Now there are ways to try to alleviate this, but classically, if you split up your documents into individual samples, they become independent, you cannot do attention, this attention mechanism cannot operate over across the boundaries of these chunks. So the long former, the goal is to actually just be able to put this entire document here into the model at the same time. So let's look a bit closer into this. In a classic transformer model, what you'll have is you'll have layers of what is called attention mechanism. I'm gonna draw six units here, and the units are actually the input sequence. So in a transformer, other than like a classic neural network, you don't actually have numbers of units in the layers, but you can input as many as long sequences as you want, until your memory limit is reached basically. So these units, they expose something called keys on the lower layer, and these are vectors that point somewhere, and the upper layer will produce what are called queries. And again, I invite you to look at the attention is all you need video if you want more explanation. And basically the keys and queries, they decide where information gets routed to. So the routing of information is what makes the transformer the transformer. So for example, this here is probably going to be routed to this here. So the information is routed like this, and then this here is going to be routed like this. You see the routing is according to the dot product of the keys and queries. So in essence, if you have an input sequence tokens, and you usually transform in a transformer, you transform the things into same length sequences. That has to do a lot also with how you want to pre-train things and so on. So we're not really going to change that part. If you have n input sequence and n tokens on the next layer, and everything can attend to everything, so all the inner products are computed, right? Everything is connected to everything. That means that you're going to end up with an O of n squared memory requirement, because you have n squared connections. The way to alleviate this is much much like you would alleviate this in a classic neural network. So in a classic neural network, imagine you have this MLP, a multi-layer perceptron, or what usually known as a fully connected layer, right? So here I have the same thing, but it's not a transformer. It's a classic neural network, fully connected. So I have D units right here, and D units in this first hidden layer. And I'll have a weight matrix in here, right? And the weight matrix means everything is connected to everything, right? Everything connects to everything else. Again, my memory requirement here is D squared. Now how do we deal with this in a classic neural network? We go to what is called a convolutional neural network. At least that's one of the methods. So let's make this again, but let's now make this a convolutional neural network. What we'll have is we'll have a convolutional kernel. In this case, it's just of length 3, right? So we just have 3 units here, and they will do the same fully connected pattern, but only over these 3 units right here. And then we slide the kernel over, right? Now it's in this position. It's still the same 3 units, but now these 3 things are connected to these 3 things that they're now over, right? And so you keep sliding this over across the lower layer until you're finally at the end here. And now you've reduced the memory consumption from D squared to just D times, and if this is usually the kernel size, it's called K, to D times K. And K you can keep pretty much constant, so that's O of D, right? The same goes for the long former. So in the long former, the idea is that you have a so-called sliding window attention. It's exactly the same as it is in the convolution, except that you don't have these hidden units here, but these are actually parts of the input sequence, and instead of the weight matrix here, you have the attention mechanism over the keys, queries, and values. But the idea is similar. So you can basically say this is a sort of a convolution, and we've already had this in the video about axial attention a bit. Now of course this is your trade-off memory for performance, because before, right, before, I'm gonna draw, let's draw it on top of this fully connected layer, before all the units could attend to all the units, right? And now the unit can only attend to its immediate neighborhood, right? This green unit here can only attend to itself in the lower layer and its immediate neighbors if the kernel size is 3. But consider what happens in the next layer. So in the next layer I have, for example, this unit right here. This is same unit, right, on the next layer. It can attend to these two and itself in the lower layer, but these two themselves can attend to all of these, right, so that the one on the right can attend to one more. So in the first layer this particular unit had information from these three units, but in the second layer the same unit has now information across these five, right, and this is kind of this cone of attention. It gets bigger and bigger as you go through the layers. So you lose the information to incorporate wide ranges of information in a single layer, but you regain it through depth, right? The deeper you go the more a single unit gets information, right, this unit gets information from this unit over here through the layers, through the layers. It can't watch the unit right here in this layer. That's not possible, but it gets the information through the layers. Of course there's still a trade-off, like a fully connected layer could just do this in one step and then in the next layer it could do it again, right, it can do much more complex computation. But if you believe that the most important information is actually in the neighborhoods of the individual tokens, which is conceivable in something like a convolutional neural network, you know that, you know, in an image usually you have localized information, right, if there's a cat here then the nose and the eyes of the cat are pretty close together. So in order to recognize it's a cat you mostly want local information, more and more local information. So in an image that makes sense and in a text it also makes sense to a degree in that usually words close together in a sentence, they are important for each other, right, but the power of the transformer was initially that it could attend to everything in a sentence, right. So for example if you have again the paragraph here, the power of the transformer, at least that was said, is the fact that this piece of text here could make a connection to this piece of text here and therefore the understanding of the entire paragraph could be reliant on this connection being made, which a local model can't do. But if you go through depth that you might be able to recover that. So the longformer is basically what the convolutional neural network does for MLPs, it does it for transformers, right. So instead of n by n giving you n squared now you go into this way where you have, so if you do the same for the transformer you go to o n times, let's call it w, and w being your window size in this case. They have an illustration of this right here. So in a original transformer this is an attention matrix. So here you have your n units in a sequence and drawn in is which unit can attend to which other unit in a given layer. So you'll see this particular unit i here can attend of course to itself, right, can attend to unit i. But it can also attend to this unit or to this unit or to this unit to any unit, right. And that's what gives you this n squared attention because any unit can attend to any unit. Now in this sliding window attention pattern, and this is one of the core components of the longformer, you see that the i-th unit here right here can attend to itself, right, but also to this and to this, this, but no more. It can only attend to the i-th unit or to i minus w to i plus w, right. And this here is a window of size w. This is this sliding window. So a given unit can only attend to itself or its neighbors in one layer, right. And this is exactly what a convolution is. Like if you see if you see this pattern, this is a this is a convolutional pattern. Now the second core component is they expand on this idea in that they make they create these dilated sliding windows. Now you see you already know what a sliding window is. Now they're saying well if you if you have this sliding window it might take quite a number of layers in order to you know get your attention of the entire sequence incorporated. We saw before it took like three layers to get halfway through this sequence of what was it like six tokens and it took us like three layers and with so basically if you go if you go one layer up right one layer up you gain one more context window in each direction, right. So it's not you'd have to go very very deep in order to incorporate the information from these very long sequences and the sliding the dilated sliding window helps this where they say well technically now any any any sequence here so again if we have this sequence and this is the next layer actually let's just draw so this unit right here it will be able to attend this and this but not this and not this but it will also be able to attend this and this but not this and not this sorry not this so it'll skip one so right these these attention patterns they will always kind of skip skip one and the idea is that now you have a vastly greater window of attention right your your window size is now way bigger that means you can incorporate information way faster across the layers like global information but of course now they're kind of arguing against each other in when they do this sliding window they say well we pose that mostly local information is important for NLP right the words right around the word are important and now if they say this here they basically say oh well it's not so important that we miss this word right here which is right next to the word that they are attending from which is counter counter to what they just said that probably the most important information is around the word they they do get around this by saying well if we have different layers in a transformer and in the lower layers will use this sliding window fully local and in the higher layers will use this dilated window and therefore in the in the lower layers we postulate that local information is actually what's needed to understand local features and then in the higher layers we want more global information because it will incorporate features from the from the local informations of the lower layers all right I can I can get the argumentation but I feel that's just something they've thrown in there to make it work better after they tried it out and the the last idea here in the long former is what they call global attention and these global attention is sparse what it means is that there are some special units here so in this this this and this unit and these special units as you can see from the attention pattern these are these can actually attend to everything so this unit can attend for example to this one or to this one or to anything these can attend to anything and any unit can attend to those right any unit can attend to the the first unit right here right so these are your special tokens your special units and they have global attention and the reason for this particularly is that sometimes this is needed and this is an engineering choice right the example I can give is let's say you have a question answering task in a question answering task what you usually have is a question and a paragraph and let's say the task here is to answer yes or no is the is the question so the question might be a statement right I don't know King James was King of England from 1120 to 1140 and then the paragraph will be the Wikipedia entry for King James and the the question is yes or no is the is the question true or not is the statement made true or not how you would feed this to a birdmold to a transformer is you concatenate these two things quest statement in paragraph right these are the tokens right here and then you would separate them using a special token called the separator token this is just to inform the model that here is where the first thing stops and the next thing starts and then at the beginning you would put a special token called the CLS token now usually what you do is you send these things through your transformer and now in the last layer right you end up as we've seen before because you always transform a sequence into a sequence you end up with a sequence again but you just want a single thing you just want yes or no so you designate you say this particular unit here that corresponds to the CLS token that's what I'm going to throw into a logistic regression and that's what will give me my yes or no answer and that's how you train it right so you you don't want to single out any of these any of these as like special so you simply already include a special token at the beginning that then you take the classification from right it's pretty smart but also you say ah this is such a special token I want that to be able to attend to anything right even though for example this unit right here it can only attend to its neighbors right it has this cone thing and this unit right here has this cone thing this unit right here can always attend to anything at each of the layers right it can attend to anything and anything can attend to it so it can get information from anywhere routed to it in each of the layers and it can send information to any of the other units. This is an engineering choice. So at the beginning, you as an engineer have to say which one of these tokens are special tokens. For these tokens, you'll actually then do full attention. It can attend to and from anything. What are our new memory requirements? What this will give us is, first of all, we have N tokens. And here W is our window size. So we have N times W memory. But then we also add the global attention. So plus the number of special tokens times, if there's a special token, it will have N times 2 memory requirement, because it can attend from and to in each layer. And this entire thing, sorry, with the plus, this entire thing times the number of layers. So this is your new attention memory requirements. And as you can see here, N plus N. So this is going to be order of N, instead of much smaller than order of N squared, as we had for the original transformer. Right. So this is what the longformer basically does. Now they have written custom CUDA kernels for doing this dilated attention and so on, which is pretty cool. And they have code available for the model. They test this on a number of language tasks. And what I find interesting is, actually, they start from the Roberta checkpoint, which Roberta, where is it said? Somewhere, oh yeah, this Roberta model right here is a variant of BERT. Right, you can see the name in here. It's a variant of BERT. And that's their baseline. And they start from these checkpoints, as far as I understand, and they kind of copy over the position embeddings and so on. And therefore, they only need to train not very much past the Roberta. Now the reason why they can copy it over actually is, and this I find very interesting, is they use a window size of 512. So until I read this, I got away from reading the paper thinking that this window size here might be fairly small. Right. So this window size, it might be, you know, maybe 10, 20, 30 tokens or something, right? But actually, this window size is 512 in their formulation, which basically means that this is as much as one of the classic models could take as a document. Right. So, sorry, let's go over. So this here is 512. So this is what a classic model could take as an entire document. And in the classic model, you simply split up the document, feed chunks, right? And then aggregate over them. Now the longformer basically has this. So right now, for now, I said it has less memory requirements. Actually, it has the same memory requirements as a classic model, but it is also able, because of these global attention, to kind of incorporate information from the surrounding things. So that's the new part. Because if you think about it, if this W here is 512, 512 was the original N. So 512 was the N0. Whatever the old models had as an N. So right now, if I replace this, and let's not take care of this. If I replace this, it's actually N times N0. And that regresses to the classic model if you plug in N0 here, right? So the new part really is the fact that you have this sliding window, and the global attention is able to incorporate information from these special tokens as well. Because sliding window, that was done before. So I just don't want to get to you the wrong impression that now we can run transformers on like very small memory machines. We can't. But we can run them on the same memory machines, because this is the same length, right? But also feed in longer documents, and have some information of the entire document be propagated to these blocks, which before we couldn't. Before we could just simply feed these blocks as one and not have global information. So that's the new thing. At least they haven't tested it on the smaller things, which is cool from an engineering point, right? You would want to, because if you want to show that you're better, you would want to basically be able to be as powerful as the old model, but then be more powerful. And that's what they do. All right. So if you want to check out the experiments and the ablations, it's very interesting because they turn on and off a lot of things in their model, and kind of check out where things come from, what helps, what doesn't. And I'll leave this to you, and I'll link it. And with that, thanks for listening, watching, and bye-bye.
 Hi there, today we're looking at Longformer, the long document transformer by
 Is Beltaji, Matthew Peters and Armin Cohen of Allen AI. So the longformer is a
 variant of the transformer as you might have guessed. The longformer is a
 transformer that can deal with long documents, so it's aptly named. So I am
 going to discuss what differentiates the longformer from the transformer. If you
 don't know what a transformer is, watch the video on attention is all you need. I
 have a video on that. And I would also suggest you watch the video on BERT,
 because a lot of the architecture and training here is based on the BERT
 or variants of BERT. So I'll basically explain what makes the longformer
 different such that it gets long documents, right, so she can be applied
 to long documents. So what is the problem with the original transformer?
 If you have a transformer model and let's say you're doing an NLP task, which
 usually is where transformers are used, and you want to have a paragraph like
 this one right here, the abstract of the paper, and maybe you want to predict
 whether the paper gets accepted at a conference or not. Now the classic
 transformers, they have a limit, a very harsh limit, on the amount of tokens that
 they can look at at the same time. So what you would do in a classic
 transformer is you couldn't process this entire thing, let's say, you would divide
 it in chunks, you'd say, okay, here's my first chunk from here to here, my second
 chunk from here to here, and then here to here, and so on. So you go through the
 documents, split it up in chunks, process each of the chunks individually, and then
 maybe aggregate the predictions. But of course the drawback is that the model
 cannot make specific connections between, let's say, some word here, like operation,
 and somewhere down here, like language. It cannot connect the two on a neural
 level, at least not in the classic transformer architectures. Now there are
 ways to try to alleviate this, but classically, if you split up your
 documents into individual samples, they become independent, you cannot do
 attention, this attention mechanism cannot operate over across the boundaries
 of these chunks. So the long former, the goal is to actually just be able
 to put this entire document here into the model at the same time. So let's look
 a bit closer into this. In a classic transformer model, what you'll have is
 you'll have layers of what is called attention mechanism. I'm gonna draw six
 units here, and the units are actually the input sequence. So in a transformer,
 other than like a classic neural network, you don't actually have numbers of units
 in the layers, but you can input as many as long sequences as you
 want, until your memory limit is reached basically. So these units, they expose
 something called keys on the lower layer, and these are vectors that
 point somewhere, and the upper layer will produce what are called queries. And
 again, I invite you to look at the attention is all you need video if you
 want more explanation. And basically the keys and queries, they decide where
 information gets routed to. So the routing of information is what makes
 the transformer the transformer. So for example, this here is probably going to
 be routed to this here. So the information is routed like this, and then
 this here is going to be routed like this. You see the routing is according to
 the dot product of the keys and queries. So in essence, if you have
 an input sequence tokens, and you usually transform in a transformer, you transform
 the things into same length sequences. That has to do a lot also with how you
 want to pre-train things and so on. So we're not really going to change that
 part. If you have n input sequence and n tokens on the next layer, and everything
 can attend to everything, so all the inner products are computed, right?
 Everything is connected to everything. That means that you're going to end up
 with an O of n squared memory requirement, because you have n squared
 connections. The way to alleviate this is much much like you would alleviate this
 in a classic neural network. So in a classic neural network, imagine you have
 this MLP, a multi-layer perceptron, or what usually known as a fully connected
 layer, right? So here I have the same thing, but it's not a transformer. It's a
 classic neural network, fully connected. So I have D units right here, and D units
 in this first hidden layer. And I'll have a weight matrix in here, right? And the
 weight matrix means everything is connected to everything, right? Everything
 connects to everything else. Again, my memory requirement here is D squared. Now
 how do we deal with this in a classic neural network? We go to what is called a
 convolutional neural network. At least that's one of the methods. So let's
 make this again, but let's now make this a convolutional neural network. What we'll
 have is we'll have a convolutional kernel. In this case, it's just of length 3, right?
 So we just have 3 units here, and they will do the same fully connected pattern,
 but only over these 3 units right here. And then we slide the kernel over, right?
 Now it's in this position. It's still the same 3 units, but now these 3 things
 are connected to these 3 things that they're now over, right? And so you keep
 sliding this over across the lower layer until you're finally at the end here. And
 now you've reduced the memory consumption from D squared to just D
 times, and if this is usually the kernel size, it's called K, to D times K. And K
 you can keep pretty much constant, so that's O of D, right? The same goes for
 the long former. So in the long former, the idea is that you have a so-called
 sliding window attention. It's exactly the same as it is in the convolution,
 except that you don't have these hidden units here, but these are actually
 parts of the input sequence, and instead of the weight matrix here, you have the
 attention mechanism over the keys, queries, and values. But the idea is
 similar. So you can basically say this is a sort of a convolution, and we've
 already had this in the video about axial attention a bit. Now of course this
 is your trade-off memory for performance, because before, right, before, I'm gonna
 draw, let's draw it on top of this fully connected layer, before all the units
 could attend to all the units, right? And now the unit can only attend to its
 immediate neighborhood, right? This green unit here can only attend to itself in
 the lower layer and its immediate neighbors if the kernel size is 3. But
 consider what happens in the next layer. So in the next layer I have, for example,
 this unit right here. This is same unit, right, on the next layer. It can attend to
 these two and itself in the lower layer, but these two themselves can attend to
 all of these, right, so that the one on the right can attend to one more. So in
 the first layer this particular unit had information from these three units, but
 in the second layer the same unit has now information across these five, right,
 and this is kind of this cone of attention. It gets bigger and bigger as
 you go through the layers. So you lose the information to incorporate wide
 ranges of information in a single layer, but you regain it through depth, right?
 The deeper you go the more a single unit gets information, right, this unit gets
 information from this unit over here through the layers, through the layers.
 It can't watch the unit right here in this layer. That's not possible, but it
 gets the information through the layers. Of course there's still a trade-off, like
 a fully connected layer could just do this in one step and then in the next
 layer it could do it again, right, it can do much more complex computation. But if
 you believe that the most important information is actually in the
 neighborhoods of the individual tokens, which is conceivable in something like a
 convolutional neural network, you know that, you know, in an image usually you
 have localized information, right, if there's a cat here then the nose and the
 eyes of the cat are pretty close together. So in order to recognize it's a
 cat you mostly want local information, more and more local information. So in
 an image that makes sense and in a text it also makes sense to a degree in that
 usually words close together in a sentence, they are important for
 each other, right, but the power of the transformer was initially that it could
 attend to everything in a sentence, right. So for example if you have again the
 paragraph here, the power of the transformer, at least that was said, is
 the fact that this piece of text here could make a connection to this
 piece of text here and therefore the understanding of the entire paragraph
 could be reliant on this connection being made, which a local model can't do.
 But if you go through depth that you might be able to recover that. So the
 longformer is basically what the convolutional neural network does for
 MLPs, it does it for transformers, right. So instead of n by n giving you n squared
 now you go into this way where you have, so if you do the same for the
 transformer you go to o n times, let's call it w, and w being your window size
 in this case. They have an illustration of this right here. So in a original
 transformer this is an attention matrix. So here you have your n units in a
 sequence and drawn in is which unit can attend to which other unit in a given
 layer. So you'll see this particular unit i here can attend of course to itself,
 right, can attend to unit i. But it can also attend to this unit or to this unit
 or to this unit to any unit, right. And that's what gives you this n squared
 attention because any unit can attend to any unit. Now in this sliding window
 attention pattern, and this is one of the core components of the longformer, you
 see that the i-th unit here right here can attend to itself, right, but also to
 this and to this, this, but no more. It can only attend to the i-th unit or to i
 minus w to i plus w, right. And this here is a window of size w. This is this
 sliding window. So a given unit can only attend to itself or its neighbors in one
 layer, right. And this is exactly what a convolution is. Like if you see if you
 see this pattern, this is a this is a convolutional pattern. Now the second
 core component is they expand on this idea in that they make they create these
 dilated sliding windows. Now you see you already know what a sliding window is.
 Now they're saying well if you if you have this sliding window it might take
 quite a number of layers in order to you know get your attention of the entire
 sequence incorporated. We saw before it took like three layers to get halfway
 through this sequence of what was it like six tokens and it took us like
 three layers and with so basically if you go if you go one layer up right one
 layer up you gain one more context window in each direction, right. So it's
 not you'd have to go very very deep in order to incorporate the information
 from these very long sequences and the sliding the dilated sliding window helps
 this where they say well technically now any any any sequence here so again if we
 have this sequence and this is the next layer actually let's just draw so this
 unit right here it will be able to attend this and this but not this and
 not this but it will also be able to attend this and this but not this and
 not this sorry not this so it'll skip one so right these these attention
 patterns they will always kind of skip skip one and the idea is that now you
 have a vastly greater window of attention right your your window size is
 now way bigger that means you can incorporate information way faster
 across the layers like global information but of course now they're
 kind of arguing against each other in when they do this sliding window they
 say well we pose that mostly local information is important for NLP right
 the words right around the word are important and now if they say this here
 they basically say oh well it's not so important that we miss this word right
 here which is right next to the word that they are attending from which is
 counter counter to what they just said that probably the most important
 information is around the word they they do get around this by saying well if we
 have different layers in a transformer and in the lower layers will use this
 sliding window fully local and in the higher layers will use this dilated
 window and therefore in the in the lower layers we postulate that local
 information is actually what's needed to understand local features and then in
 the higher layers we want more global information because it will incorporate
 features from the from the local informations of the lower layers all
 right I can I can get the argumentation but I feel that's just something they've
 thrown in there to make it work better after they tried it out and the the last
 idea here in the long former is what they call global attention and these
 global attention is sparse what it means is that there are some special units
 here so in this this this and this unit and these special units as you can see
 from the attention pattern these are these can actually attend to everything
 so this unit can attend for example to this one or to this one or to anything
 these can attend to anything and any unit can attend to those right any unit
 can attend to the the first unit right here right so these are your special
 tokens your special units and they have global attention and the reason for
 this particularly is that sometimes this is needed and this is an engineering
 choice right the example I can give is let's say you have a question answering
 task in a question answering task what you usually have is a question and a
 paragraph and let's say the task here is to answer yes or no is the is the
 question so the question might be a statement right I don't know King James
 was King of England from 1120 to 1140 and then the paragraph will be the
 Wikipedia entry for King James and the the question is yes or no is the is the
 question true or not is the statement made true or not how you would feed this
 to a birdmold to a transformer is you concatenate these two things quest
 statement in paragraph right these are the tokens right here and then you would
 separate them using a special token called the separator token this is just
 to inform the model that here is where the first thing stops and the next
 thing starts and then at the beginning you would put a special token called the
 CLS token now usually what you do is you send these things through your
 transformer and now in the last layer right you end up as we've seen before
 because you always transform a sequence into a sequence you end up with a
 sequence again but you just want a single thing you just want yes or no so
 you designate you say this particular unit here that corresponds to the CLS
 token that's what I'm going to throw into a logistic regression and that's
 what will give me my yes or no answer and that's how you train it right so you
 you don't want to single out any of these any of these as like special so
 you simply already include a special token at the beginning that then you
 take the classification from right it's pretty smart but also you say ah this is
 such a special token I want that to be able to attend to anything right even
 though for example this unit right here it can only attend to its neighbors
 right it has this cone thing and this unit right here has this cone thing this
 unit right here can always attend to anything at each of the layers right it
 can attend to anything and anything can attend to it so it can get information
 from anywhere routed to it in each of the layers and it can send information
 to any of the other units.
 This is an engineering choice.
 So at the beginning, you as an engineer have to say which one of these tokens are special tokens.
 For these tokens, you'll actually then do full attention.
 It can attend to and from anything.
 What are our new memory requirements?
 What this will give us is,
 first of all, we have N tokens.
 And here W is our window size.
 So we have N times W memory.
 But then we also add the global attention.
 So plus the number of special tokens times,
 if there's a special token,
 it will have N times 2 memory requirement,
 because it can attend from and to in each layer.
 And this entire thing, sorry, with the plus,
 this entire thing times the number of layers.
 So this is your new attention memory requirements.
 And as you can see here, N plus N.
 So this is going to be order of N,
 instead of much smaller than order of N squared,
 as we had for the original transformer.
 Right.
 So this is what the longformer basically does.
 Now they have written custom CUDA kernels
 for doing this dilated attention and so on,
 which is pretty cool.
 And they have code available for the model.
 They test this on a number of language tasks.
 And what I find interesting is,
 actually, they start from the Roberta checkpoint,
 which Roberta, where is it said?
 Somewhere, oh yeah, this Roberta model right here
 is a variant of BERT.
 Right, you can see the name in here.
 It's a variant of BERT.
 And that's their baseline.
 And they start from these checkpoints,
 as far as I understand,
 and they kind of copy over the position embeddings and so on.
 And therefore, they only need to train not very much
 past the Roberta.
 Now the reason why they can copy it over actually is,
 and this I find very interesting,
 is they use a window size of 512.
 So until I read this,
 I got away from reading the paper
 thinking that this window size here might be fairly small.
 Right.
 So this window size, it might be, you know,
 maybe 10, 20, 30 tokens or something, right?
 But actually, this window size is 512 in their formulation,
 which basically means that this is as much
 as one of the classic models could take as a document.
 Right.
 So, sorry, let's go over.
 So this here is 512.
 So this is what a classic model could take as an entire document.
 And in the classic model,
 you simply split up the document, feed chunks, right?
 And then aggregate over them.
 Now the longformer basically has this.
 So right now, for now, I said it has less memory requirements.
 Actually, it has the same memory requirements as a classic model,
 but it is also able,
 because of these global attention,
 to kind of incorporate information from the surrounding things.
 So that's the new part.
 Because if you think about it,
 if this W here is 512, 512 was the original N.
 So 512 was the N0.
 Whatever the old models had as an N.
 So right now, if I replace this,
 and let's not take care of this.
 If I replace this, it's actually N times N0.
 And that regresses to the classic model if you plug in N0 here, right?
 So the new part really is the fact that you have this sliding window,
 and the global attention is able to incorporate information from these special tokens as well.
 Because sliding window, that was done before.
 So I just don't want to get to you the wrong impression
 that now we can run transformers on like very small memory machines.
 We can't.
 But we can run them on the same memory machines,
 because this is the same length, right?
 But also feed in longer documents,
 and have some information of the entire document be propagated to these blocks,
 which before we couldn't.
 Before we could just simply feed these blocks as one and not have global information.
 So that's the new thing.
 At least they haven't tested it on the smaller things,
 which is cool from an engineering point, right?
 You would want to,
 because if you want to show that you're better,
 you would want to basically be able to be as powerful as the old model,
 but then be more powerful.
 And that's what they do.
 All right.
 So if you want to check out the experiments and the ablations,
 it's very interesting because they turn on and off a lot of things in their model,
 and kind of check out where things come from,
 what helps, what doesn't.
 And I'll leave this to you, and I'll link it.
 And with that, thanks for listening, watching, and bye-bye.