Author Interview - Transformer Memory as a Differentiable Search Index
#neuralsearch #interview #google

This is an interview with the authors Yi Tay and Don Metzler.
Paper Review Video: https://youtu.be/qlB0TPBQ7YY

Search engines work by building an index and then looking up things in it. Usually, that index is a separate data structure. In keyword search, we build and store reverse indices. In neural search, we build nearest-neighbor indices. This paper does something different: It directly trains a Transformer to return the ID of the most relevant document. No similarity search over embeddings or anything like this is performed, and no external data structure is needed, as the entire index is essentially captured by the model's weights. The paper experiments with various ways of representing documents and training the system, which works surprisingly well!

OUTLINE:
0:00 - Intro
0:50 - Start of Interview
1:30 - How did this idea start?
4:30 - How does memorization play into this?
5:50 - Why did you not compare to cross-encoders?
7:50 - Instead of the ID, could one reproduce the document itself?
10:50 - Passages vs documents
12:00 - Where can this model be applied?
14:25 - Can we make this work on large collections?
19:20 - What's up with the NQ100K dataset?
23:55 - What is going on inside these models?
28:30 - What's the smallest scale to obtain meaningful results?
30:15 - Investigating the document identifiers
34:45 - What's the end goal?
38:40 - What are the hardest problems currently?
40:40 - Final comments & how to get started

Paper: https://arxiv.org/abs/2202.06991

Abstract:
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

Authors: Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, Donald Metzler

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 This is an interview with the authors of the paper transformer memory as a differentiable search index. I have done a comprehensive review of this paper yesterday. I've released it just before this video. So be sure to check that out. The authors today have actually seen my review and we'll dive right into the matter during this interview. You will not only learn much more about the paper itself, but also the research project itself, what went well, what didn't, and what the authors think of the future of the field. This is super duper interesting. It's an absolute pleasure to interview all of these people and that's possible because of you. So continue to let me know in the comments, what you think, how I can make this content better. Thank you to everyone who shares out these videos, to everyone who's part of our discord community, to all the supporters on Patreon and so on. And without further ado, let's get into the video. Hello everyone. Today I'm here with Yite and Don Metzler, who are authors of the paper Transformer Memory as a Differentiable Search Index, which I find really cool, really inspiring, very creative and very happy that you are here. Welcome to the channel. Yeah, thanks for having us. Thanks for having us. This paper is a bit special, right? Because it takes a little bit of thinking outside the box, I think, to overcome or to arrive at the conclusion, hey, let's just store the entire data set into transformer weights or you can frame it in whatever way you want, but it is not an obvious idea. How did you get the idea that you want to try something like this? Yeah, so maybe I'll just share a little bit from my point of view and Don can go next about his thoughts. So I think from my side, I'm mainly interested in understanding the properties of transformers and how many documents can transformers encode in the parameters. And then obviously retrieval is a good way to test whether a model is able to generalize and digest what it has encoded in memory. So I think from my point of view, it's more of trying to see what transformers are capable of and pushing the limits of memorization. And yeah, so I think that's from my point of view. One of the reasons why we thought of this at the start, maybe Don can share some thoughts as well. Yeah, so I'm taking just a sort of a step back. This paper is somewhat tied to this paper that we published sometime last year called Rethinking Search, which laid out kind of our vision for how we can bring the latest and greatest in machine learning, natural language understanding to bear on sort of information retrieval problems. There's been a lot of interest in this space recently. And so one of the things that we talked about in that paper was this, I mean, essentially this idea, how to essentially take these large language models that exist, which understand relationships between sequences of tokens and imbue them with an understanding of documents. Because usually these sequences of tokens come from documents. But I've never seen anyone explicitly model that. And so from my point of view, sort of more as a kind of IR researcher, and it's great that Yi sort of has more of the machine learning and LP background. We decided to come together and say, like, hey, what can we actually do here to study this? Is this a crazy idea? Is this even possible? And so one of the things that we'd hope to do is actually see if we can build like this idea of not even like an evolution of language models that are more of like corpus type of models, right? Where you have documents now and in these types of models, potentially not, we didn't do it necessarily here, but in the future, right, you can have models that actually understand relationships between documents. And, you know, we established this, OK, how can you model relationships between token sequences of tokens and documents? But I think you can take this sort of a step further. And yeah, so that's kind of like a broader framing and how we came up with this. Then also, I mean, obviously a super cool problem from like machine learning, natural language understanding side things as well. I think it's been long suspected, said, however you want to call it, that transformers, especially the large language models, they essentially regurgitate their training examples and kind of interpolate their training examples. Was this in your mind as you went about that research or how does that connect to people saying, well, all GPT-3 does is essentially, you know, kind of reproduce a bunch of its training data sets. This is like a good question, but I guess beyond memorization, we are also kind of trying to test for whether a model can make use of the memory because if it's like, you know, you give a model a prompt and it generates from that prompt, it's like associative memory and stuff. But like, you know, maybe understanding of documents is like maybe slightly beyond that. And we want to like just probe this a bit more and see what kind of data sets are slightly beyond that and we want to like just probe this ability of the models because, you know, if you can do zero-shot retrieval here, it kind of, you know, implies that the model has, you know, understands like reasons a little bit with what it has memorized. So I guess from an ML point of view is at least some kind of benchmark like type of task to kind of probe for this type of ability in a model. Now, I had a bunch of questions, maybe technical questions about the model. So I suggest we kind of clarify these first before we go into more the broad or the meanings behind the things. You have this contrastive objective here that you present in the dual encoders and you have the fully differentiable search index. Have you tried or there are these things called cross encoders, right, where I input a query and a document and I try to predict some sort of a score of how they go together. These usually work a lot better than the dual encoders themselves. What is the reason that you chose to not compare to any cross encoder type setups here? Yeah, that's a great question. I can take that. So the reason why we don't compare with cross encoders is because generally cross encoders are pretty expensive because you cannot like cache the documents in advance and you have like, you always have to, you know, compute for every query that comes in, you have to always compute with all the documents. So there's some latency and some compute cost restrictions for cross encoders. So within the scope of DSI, because DSI is basically generating doc ID. So we kind of put that in the same ballpark as a similar compute cost as, you know, like instead of doing a ****, like you kind of, instead of that, you kind of decode one document. So we consider that the compute cost like to be more fair than, you know, having to pass through a pipeline of like, and not like usually there's another re-ranker that does this cross attention stuff and then that definitely improves the performance. And I don't think that at this point of time, like we would beat a cross attention encoder. But, you know, basically cross encoders are just expensive. So that's why we consider it like out of scope for this. That makes sense. You hear you very elegantly, you output just a list of document IDs. I was wondering, have you ever tried to actually produce the document itself that you're searching for instead of the document ID? Because I was wondering, because the model needs to learn this association between the input and the document ID and it kind of needs to remember what text is in that document, right? There's no other way for it to really learn to associate text with document IDs. And I was wondering, is it a harder or an easier task for the model to directly output the text of the document? What do you think? I think there's a lot of challenges with decoding the document. I mean, you can obviously constrain your beam search to only generate stuff that is within a certain memory and stuff. And then that's definitely possible, or at least maybe the title of documents. But then I think that would, like, we have not tried that in this work. And then I think this is definitely interesting and it's a good point that you brought up. But I think that at least within the scope of this, we wanted to keep the compute low. And we have already in related a lot of possibilities in representing the doc IDs. And then that will probably be a different class of style of doc ID representation, like using natural language that can be a follow-up work. But the reason why we mainly don't explore that now is because there's a lot of additional challenges that we need to think about. And so we will consider that slightly out of scope for now. But that's definitely a great suggestion. And we think that it's also potentially quite viable as well. The only other thing I quickly add here, going back to also your question about the cross-encoders, these models typically have limited ability to essentially monopont text length, right? So you're limited usually to passages or parts of documents, right? By sort of modeling the document ID sort of as itself, you sort of open up the ability to model larger, more complex documents that you wouldn't be able to do sort of if you were treating everything as sequences of tokens, which again, sort of the standard. From the IR perspective, it's been, again, my very biased opinion, very unsatisfying, the move away from sort of documents that are very trivial to more passage retrieval that has happened recently. And a lot of that is just because the models have not been able to handle these longer sequences like they did before. So this takes us a little bit back to that. And obviously, if you have longer documents and whatnot, it'd be even more challenging to potentially decode that entire document. Though, isn't it a bit because if I think of information retrieval in the, let's say the olden days, what I actually retrieved was keywords, right? And then I simply looked up which documents the keywords belonged to. And I had some heuristics of how I combined for an entire document, all the keywords that were inside of it. Couldn't this also the move to passages be viewed as an expansion rather than a reduction in the scope of what I'm looking at? Do you see what I mean? Yeah, for sure. Obviously, there's always a way to aggregate from the passage level to the document level. And this is a very standard trick that people have done. People even did that back in the olden days when you just had sort of keyword-based indexes as well. So for sure, but then you also do have considerations of efficiency, right? If you're going to then go and have to score dozens of passages per document, that suddenly explodes the cost versus just scoring sort of at the document. So there's definitely trade-offs here. What this introduces is a level of redirection or a level of indirection in what the model needs to learn. So we no longer map sentences with the same meanings to each other, for example. We now have to learn this indirection almost like addressing a document by a variable name. Even with your semantically meaningful identifiers, still, I believe a large part as a model, I need to remember just this identifier means something. It stands for a particular document. Do you see this applicable in maybe a broader context? You already allude in your paper that this could be part of a differentiable architecture. Where do you see these types of indirection-based models going? Yeah, that's a great question. Actually, I was waiting to talk about this because it's something I'm really excited about. So the doc IDs, using the doc IDs, as you mentioned, is some indirection. You store the information in some address, and then later on, you can just use that address in the place of a long document and stuff. So I think one possible avenue here is you can imagine prompt tunings. This few shots in context learning might require you might need to stuff 10 prompts, 10 examples in this large language model. So if this memory addressing type of architecture allows you to compress stuff to doc IDs, and then you can use that as for prompt tuning, or you can use that for retrieval augmentation. So I think that might be more use cases that can be explored beyond retrieval. So this is more of a fundamental. I think that you got it really very accurate where it's a class of models that uses this memory addressing stuff that may have more wider applications. So yeah, we are also quite excited about that. So everything that you can be like, on top of my head is mainly like maybe like prompt tuning or retrieval augmented models that could benefit from this. But yeah, as of now, we don't know that for sure. But yeah, this is just a guess. In your paper, you describe the performance of your models and the trend seems to be, if I recall this correctly, at least if we go to the results section real quick, that the larger models do perform better. However, the larger the data set gets, the less the improvements of, let's say, the DSI compared to the dual encoders are, if I understood this correctly. And in your data sets, you're still in the realm of 300,000 documents. For an IR problem, that is not really a large scale problem. Do you think that in the future, people might be able to expand these models to also become better on larger document or collection instances? Or do you think that the application of these types of things might be, as you say, much more as a differentiable component in something, maybe in a reinforcement learning agent or something like this? How do you deal with the fact that as you seem to scale the document collection size, the benefits get weaker and weaker? Yeah, so that's a good question. So we kind of think that it gets harder and harder to do the same thing. It gets harder and harder as you increase more documents. I think that's also because the model has to memorize or link documents to much more identifiers. So to be honest, the interplay between memorizing and retrieval is actually quite tough for the model to learn. And as you can see, you need an XSL model to almost do well on these tasks. But I think that to cope with larger documents, there are multiple ways. One of them potentially is sparse models, make sure experts, where you can just increase the parameter size significantly without increasing the compute. So we think that those are also promising to scale these models up to maybe a few million docs at least. This is like estimate. We don't have the results yet to show this. But this is what we think right now. And yeah, it's true that it gets harder and harder eventually. So we are not sure where the limit is yet. And we are also excited to find out where does this end and where's the limit of this. Do you have an idea of how these things scale? If I have double the amount of documents, do I need double the amount of parameters or do I need an order of magnitude more parameters? Is it related linearly, exponentially? Do you have any idea of how this scales? Off the top of my head, I'm unable to put a number on it right now. It's mainly like the intuition is... And it also depends on... There's one part which is the memorizing capability because I believe that beyond this paper, we have actually tried brute force memorizing a couple million documents. The model does memorize, but then there's another... If you need to factorize other part of how well the model is able to make use of this information. So it depends on... The data set depends on many factors. So it's very hard to say. But at least on NQ, we don't have currently we don't have beyond 300K documents, but going from 100K to 320K documents it wasn't really exactly trivial. So we expect that going to a 1 million docs in a retrieval context would be... If I had to put a number on it, it probably may need to go to 32 billion parameters or something like that. If I had to give a guess and estimate. Obviously, this is the standard feedback we get when people take a look at the paper. Lots of questions about the experiments, other data sets, scaling it up. I don't want to give too much away. Obviously, we're aware of this. We're working on this. We hope to be able to have better answers to all of these questions sometime soon and also demonstrate that this works more than just on NtU, on some larger data sets. And hopefully have much better empirical basis for understanding limitations and scalability of these approaches. I have to ask just for... It's a detailed question, but this NQ100K data set it seems to be just out of place a little bit. The numbers, they're just kind of off. It looks really good with the 10K data set and the 320K data set. You can see things either get better or worse, maybe as you'd expect. But then the 100K data set, it's just like, for example, the BM25 is all of a sudden a lot better than either on the 10K data set and the 320K data set. And likewise, in a bunch of the other numbers, it's just sort of out of place. Do you have an idea of what's going on with the data set as such? Yeah, sure. I think if you look at the numbers right now, one of the points that stand out the most is the bucket of the atomic doc IDs. The second stuff. Even you look at NQ320K, you see a 6.9 there randomly. So the fact is that for atomic doc IDs, there were a lot of training instability issues that we had to overcome. So there's a lot of variance and a lot of trainability issues. And we tried our best to overcome those. So sometimes you get a base model doing better than a... It's more of optimization and the interplay between the retrieval and memorization sometimes. I mean, I think coming from my experience of running many of these logical reasoning or memorizing tasks, sometimes the model gets it in the end, and then sometimes it just doesn't get it at the end by the end of the training. So I think there's generally... Especially for atomic doc IDs because we initialize... The softmax layer is initialized from scratch, and we use the pre-trained models. And also depending on the warm-up and everything. So it was already a challenge to optimize for the atomic doc IDs. That's why you see generally even on all three sets, there's a very... I think the rest of them scales pretty more nicely than the atomic doc IDs, but that is actually a big challenge that we had. I'm not sure if we actually explicitly point out this instability issue too much in the paper, but I think I remember mentioning somewhere, but at least the middle bucket is really hard to train. The second bucket is... You do mention it, yes. The other thing to mention... If you look at the BM25 number, that's not trained in any way. It also obviously demonstrates very different performance there. The other thing is just... There is variance when you subsample the documents. So if you go from 320,000 to 100,000, you're subsampling. Maybe that was just a very lucky, good set of documents that somehow was much more amenable and much more relevant in some way. So if you do this with any sort of, I think, standard IR system, you just start subsampling documents in different ways. You're going to get very different performance. I mean, probably the best thing would have been to subsample like five or six times, get some sort of error bars there to get a sense of what the variance is. So I suspect that probably it's a mix of the instability plus the fact that maybe this is a luckier, sort of different sample of documents in the 320k and the 10k. I actually have an answer about the... There's one point which is a bit implicit. It's not like... It's mentioned, but it's not very obvious. But for NQ10k and NQ100k, these are subsampled sets from NQ, right? And then NQ320k uses the official validation set, right? So there's like 10k and 100k is subsampled. And then I'm not exactly sure how the validation set was constructed in NQ, but so 10k and 100k uses a similar way of sampling. It's just random, but when you go to 320k, it's actually using the official validation set. So I don't know, maybe it's a bit cleaner or like there's some difference in the way this... So 10k, 100k and 320k came from the official validation set. So there might be some differences in the way we sample and how the other people sample. So I believe that you mentioned the training instabilities also at points throughout, and that might also explain a little bit as well why different methods are good at different tasks, right? You have, there's quite a bit of variance in which methods are better here or there, quite a bit of variance in the numbers itself. Although what I see is very thoroughly the case is that the larger models tend to do better in general. Whenever a model wins here with whatever way, it tends to be the larger models that outperform the smaller models within the same buckets. Do you think that is a property of the larger models being pre-trained better? Because larger models also exhibit better language modeling behavior, right? And given that these are pre-trained, I guess T5 style checkpoints, that might be an improvement because as far as I understand it, your retrieval performance also in a part depends on the models being pre-trained to actually understand language, especially the zero shot ones. Or do you think that is mainly a, the main contributor is that with more parameters I can memorize more documents? So could you comment on that? And maybe also a little bit on what do you think intuitively is going on inside of these models that they are even able to retrieve those IDs? So I think the pre-training definitely does contribute, like I wouldn't be able to say like how many, put a number on like how many percent it contributes to that. But I definitely think that like one way to tell is like probably to just rerun all the experiments with like randomly initialized T5 style models, right? I think at a very early stage, I mean, it's not in the paper, but we did run some early experiments with like no pre-trained models. And these models actually like, it's way harder to learn without the pre-training. And this is a common finding across, not only in this context, but in broader NLP and machine learning in general. So we think that the pre-training does a lot of like heavy lifting and also the size, like with a larger model, you also benefit more from, like it's the composition of two different things helping each other. So because you are pre-trained and then you also larger and then you benefit more from pre-training when you are for this T5 XXL models. So I think that also probably contributes to like the zero shot and stuff like that. So yeah, just to answer the question, especially I think that the pre-training does contribute a lot to this. Yeah. Yeah, I think the other thing we don't have a good understanding of is, after we fine tune on these, the DSI tasks, what sort of the model, what knowledge the model retains or does not retain, right? What was the nature of the model at that point? As others have sort of asked this question and I think it's a great question. I do suspect that some of the knowledge that sort of obviously pick up during pre-training is helping as you suggested, but there may be other pre-training tasks that are even more amenable to sort of DSI than sort of the standard T5 pre-training. Did you, have you attempted at introspecting these models in some way? To kind of see whether you can find the documents, whatever it means inside of these weights. Like, you know, I imagine since I can query these models and they give me a doc ID that I need to be able to go and look inside the weights or something and find traces of these documents or something. Like, is there something you can say about the inner workings or is there something one can see in the attention maps or in the weights? I have a very disappointing answer because I wish I knew where to look in the model as well. But the unfortunate thing is that I don't know where this is safe in the model. Is it in the decoder layers? But I think intuitively it seems like, because the decoder learns to output doc IDs, I think the decoder does quite a lot of heavy lifting in the model, but which weight is in? And there's also the feed-forward layers, key value memories and stuff like that. And then you can somehow probe that. I think this is interesting for a lot, but unfortunately we don't know where it's safe now in the model. Yeah. What do you think, if people want to get started with this, what do you think is the smallest scale thing that would still give meaningful insights into the technique? Because a certain scale is necessary if found or stand this correctly, right? But what would be the minimal setup for anyone to get into this type of research, like differentiable indexing and things like this? Yeah, that's a very good question, actually. So at what point where this gets getting meaningful, which scale does it get meaningful? I guess that's my personal opinion. This is just my personal opinion, obviously, this is my sense of things. But I think starting at around XL, 3B, is probably a reasonable scale to start. Because actually, I don't really know why 3B, but this is just from my experience running the experiments. Because 3B and 11B has slightly different training dynamics compared to Bayes and Lodge. So it's very hard to characterize this. It's very latent within me. But I think 3B, somewhere around 3B, is medium scale models. But small and Bayes probably will not be that meaningful. But I guess starting from 3B would be pretty nice. So that is not exactly small, right? I can't really run this on my 1080 at home. But it's still, I guess, maybe accessible to more people than just the biggest companies. Here you have a pretty interesting thing in your hierarchical document IDs. And I understand this is not the end all be all. This is like an attempt at forging meaningful document IDs. And you make very interesting requirements here. You have two requirements that they retain some semantics, which the clustering, I would say, gives you. It gives you a little bit of semantic thing. But then also you want to reduce the search space with each decoding step, which is a property of autoregressive decoding. The first decoding step only needs to care about the big picture, the next one, the smaller and the smaller. Do you have an idea how much these two things play together? Or which one is kind of the important one? Because one could also, I think in the review, I raised the issue, you could reverse this in this document ID, which would give you the same meaningful document identifier, but without this property of autoregressive decoding. Do you have an insight of which of the two properties might be the more important one here? And which one is, or are they interacting with each other? So we have thought like really like factorized both of them. Intuitively, I think that segmenting the search space is more beneficial, but I think they help each other. I think this is possible to also come up with ways of ablating this, but I think we did not try those yet. If you look maybe a bit more high level, no, wait, I have one more question. Yeah, this L right here, right? Because you have this very interesting graph that shows this thing right here, which document representations make the most sense and direct indexing. I also find it interesting that in your paper, you try out a lot of things, and then at the end, it seems like often the simpler things work better, which is a neat finding, I guess, an encouraging finding for a lot of people. Although I was surprised to see that if you index fewer tokens of the documents, it tends to perform better. Because that shouldn't be, right? What's the problem here? What's the problem that prevents us from indexing longer sequences of the documents? So this is just like my thoughts on this is that like going up to 128 and above makes the training harder. We also observe this in memorization, looking at the training accuracy of memorization. So I think by, and there's going to be quite some examples, we don't know how many examples, but there's going to be some examples that can be solved easily by the first 32 tokens or 65 tokens. So I think that the model, okay, this is just a guess, I'm not really 100% sure about this, but it's like the model prioritizes getting the one in most correctly rather than trying to fit 256 tokens and then not being able to solve anything, even the easy ones. So I think this might be what's happening. And then this 32, I will not over-index on this 64 or 32, because it's probably going to be very dataset dependent. And also the inverted index, I saw on your review that you were surprised that the inverted index didn't work. But this might be an artifact of this dataset. And it's maybe the simpler approach here, but when we scale up, when we go to something harder or more documents, or just the structure of the dataset is different, then perhaps the inverted index would help. So I think that there's a lot here that we are just showing a slice of the data points, but I wouldn't over-index or like, oh, DSI only works when the document length is short or something. But I think this is dataset dependent. And for sure, I believe that for other datasets, you need longer sequence length. If you look ahead a little bit, and you came into this, you told me at least that you just wanted to know certain things, like you had some questions, is this even possible and so on. My question is, is there an end goal here? If you look into the future, maybe two, three, five years or so, you develop this a little bit, hardware gets better and so on. What's the outlook? What's the North Star that this could lead to? Yeah, so I'm going to share a bit, and then I think Don surely has thoughts about this as well. So I will leave some for him. So I think one of the North Star here is because retrieval is generally slightly decoupled away from other NLP tests. People are unifying models, they are going for T5, everything is 6 to 6. But when it comes to retrieval, you always have this separate infrastructure of dual encoders, and then you have to compute ranking metrics, and then the whole infrastructure is always very different from machine translation or text generation stuff. So I think this, at least for me, one aspect of it is to be able to conveniently do retrieval in a way that you don't need to have a separate infrastructure. You can just co-train your retrieval, get all the metrics you need, get a competitive performance to dual encoders while still being able to do machine translation at the same time. So maybe machine translation may not be the best example, but maybe you want some NLU, some question answering model, end-to-end, or synthesizing. From the doc IDs, you can generate doc IDs together with text, and then maybe substantiating the text with doc IDs, like learning to cite and stuff like that. So I think these are the visions that I'm pretty excited about. Maybe Dawn can chime in. Going back to what I mentioned at the start, this is part of this exploration of what's possible. If you play this forward, we have no idea what's going to happen. One potential outcome is that it turns out that this is a great way of actually modeling a lot of the things that the IR community in the past has modeled in terms of documents and terms of all of this, and that this type of approach could be a way of unifying retrieval and scoring. You mentioned cross encoders. Today, usually, as you mentioned earlier, you have this cascaded approach where you do retrieval first and then you do scoring next. So this does everything together, jointly. That kind of simplifies things. It would be nice, I think, in the future to be able to have a way of doing that all end-to-end in a highly differentiable way. The other thing that is obvious here is that there's a lot of attention and interest recently with retrieval, augmented, everything. The idea being fewer parameters and more reliance on external memory or storage in some way. This is diametrically opposed to that. I think there's pros and cons to both of the approaches, and it will be very interesting to see as we continue to explore both directions what are the benefits of each of these things and how maybe the two of them can come together, as you were suggesting. Maybe DSI could be an inner loop on a retrieval, augmented approach in the future. If you look ahead maybe a bit more short term, what are the hardest problems that are still outstanding to make the next steps of progression here? There's actually a lot. It's good, right? As a researcher. There's a lot of things that we want to solve and there's still a lot of things that keep me up at night. I think there are a couple of pressing ones, like how do you update documents, and then solving the trainability issue and then solving the scale. I'm hoping that going to sparse models, something like switch transformer, you can just handle 20-30 million docs out of the bat. I think scaling is a more short term to mid term thing that we want to solve. So updating, scaling, and also the interplay between retrieval and understanding a little bit more about this zero-shot behaviour, and also understanding where it is in the model, as you mentioned. Understanding this behaviour of these models, I think these are immediate next steps that I think to take this idea further, these things need to be to some extent solved, or at least figured out somehow. Obviously, some of the questions you brought up here are things that are actively being thought about and explored. One of the things that we were just talking about was indexing the first 32 tokens. So just understanding the properties of the model across more datasets, and what are the best practices here, I think are also very immediate term things that we'll need to do to just get a basic understanding of this beyond this initial proof of concept, if you will, that this crazy idea is even feasible. Is there anything else that maybe we haven't touched on yet that you would like people to take away from the paper that they shouldn't go without knowing? That's a good question. Nothing that I can do. Yeah, I can't think of anything right now. Even if the models are large, could people get into this? Is the code somewhere available or are you planning to make it? This is subject to approval, but we do have plans to make the code available sometime in Q2 of this year. But this is all subject to approval. We have not gotten the approval yet as of now, but this is our plan to release it in Q2. The fight with the lawyers. Excellent. We have a history of open sourcing. You've reviewed several of our papers in the past. We do have a history of being able to release the code. It's just a matter of checking various boxes, and we're committed to this. We've already had folks reaching out, trying to replicate this, and we want to make it easy for everyone so that they can get going with this. I think it's a really interesting area, and hopefully this will stimulate some additional fun research. I was in Google for a while. I know it can be a hassle to open source anything and the amount of approvals you need to get. Props that you even want to go through with it. It's pretty cool. All right. Don and Yi, thank you very much for being here. This was very enlightening, and I hope people had fun. I hope to see you again soon. Thanks for inviting me. This was great. It was great, yeah.
 This is an interview with the authors of the paper transformer memory as a
 differentiable search index.
 I have done a comprehensive review of this paper yesterday.
 I've released it just before this video.
 So be sure to check that out.
 The authors today have actually seen my review and we'll dive right into the
 matter during this interview.
 You will not only learn much more about the paper itself, but also the research
 project itself, what went well, what didn't, and what the authors think of the
 future of the field.
 This is super duper interesting.
 It's an absolute pleasure to interview all of these people and that's possible
 because of you.
 So continue to let me know in the comments, what you think, how I can make
 this content better.
 Thank you to everyone who shares out these videos, to everyone who's part of
 our discord community, to all the supporters on Patreon and so on.
 And without further ado, let's get into the video.
 Hello everyone.
 Today I'm here with Yite and Don Metzler, who are authors of the paper
 Transformer Memory as a Differentiable Search Index, which I find really cool,
 really inspiring, very creative and very happy that you are here.
 Welcome to the channel.
 Yeah, thanks for having us.
 Thanks for having us.
 This paper is a bit special, right?
 Because it takes a little bit of thinking outside the box, I think, to
 overcome or to arrive at the conclusion, hey, let's just store the entire data
 set into transformer weights or you can frame it in whatever way you want, but
 it is not an obvious idea.
 How did you get the idea that you want to try something like this?
 Yeah, so maybe I'll just share a little bit from my point of view and Don can
 go next about his thoughts.
 So I think from my side, I'm mainly interested in understanding the
 properties of transformers and how many documents can transformers encode in
 the parameters.
 And then obviously retrieval is a good way to test whether a model is able to
 generalize and digest what it has encoded in memory.
 So I think from my point of view, it's more of trying to see what transformers
 are capable of and pushing the limits of memorization.
 And yeah, so I think that's from my point of view.
 One of the reasons why we thought of this at the start, maybe Don can share
 some thoughts as well.
 Yeah, so I'm taking just a sort of a step back.
 This paper is somewhat tied to this paper that we published sometime last
 year called Rethinking Search, which laid out kind of our vision for how we can
 bring the latest and greatest in machine learning, natural language understanding
 to bear on sort of information retrieval problems.
 There's been a lot of interest in this space recently.
 And so one of the things that we talked about in that paper was this, I mean,
 essentially this idea, how to essentially take these large language models that
 exist, which understand relationships between sequences of tokens and imbue
 them with an understanding of documents.
 Because usually these sequences of tokens come from documents.
 But I've never seen anyone explicitly model that.
 And so from my point of view, sort of more as a kind of IR researcher, and
 it's great that Yi sort of has more of the machine learning and LP background.
 We decided to come together and say, like, hey, what can we actually do here to study
 this? Is this a crazy idea? Is this even possible?
 And so one of the things that we'd hope to do is actually see if we can build
 like this idea of not even like an evolution of language models that are more
 of like corpus type of models, right?
 Where you have documents now and in these types of models, potentially not, we
 didn't do it necessarily here, but in the future, right, you can have models that
 actually understand relationships between documents.
 And, you know, we established this, OK, how can you model relationships between
 token sequences of tokens and documents?
 But I think you can take this sort of a step further.
 And yeah, so that's kind of like a broader framing and how we came up with this.
 Then also, I mean, obviously a super cool problem from like machine learning,
 natural language understanding side things as well.
 I think it's been long suspected, said, however you want to call it, that
 transformers, especially the large language models, they essentially regurgitate
 their training examples and kind of interpolate their training examples.
 Was this in your mind as you went about that research or how does that connect to
 people saying, well, all GPT-3 does is essentially, you know, kind of reproduce
 a bunch of its training data sets.
 This is like a good question, but I guess beyond memorization, we are also kind of
 trying to test for whether a model can make use of the memory because if it's
 like, you know, you give a model a prompt and it generates from that prompt, it's
 like associative memory and stuff.
 But like, you know, maybe understanding of documents is like maybe slightly
 beyond that.
 And we want to like just probe this a bit more and see what kind of data sets
 are slightly beyond that and we want to like just probe this ability of the
 models because, you know, if you can do zero-shot retrieval here, it kind of,
 you know, implies that the model has, you know, understands like reasons a little
 bit with what it has memorized.
 So I guess from an ML point of view is at least some kind of benchmark like
 type of task to kind of probe for this type of ability in a model.
 Now, I had a bunch of questions, maybe technical questions about the model.
 So I suggest we kind of clarify these first before we go into more the broad or
 the meanings behind the things.
 You have this contrastive objective here that you present in the dual encoders
 and you have the fully differentiable search index.
 Have you tried or there are these things called cross encoders, right, where I
 input a query and a document and I try to predict some sort of a score of how
 they go together.
 These usually work a lot better than the dual encoders themselves.
 What is the reason that you chose to not compare to any cross encoder type
 setups here?
 Yeah, that's a great question.
 I can take that.
 So the reason why we don't compare with cross encoders is because generally
 cross encoders are pretty expensive because you cannot like cache the
 documents in advance and you have like, you always have to, you know, compute
 for every query that comes in, you have to always compute with all the
 documents.
 So there's some latency and some compute cost restrictions for cross encoders.
 So within the scope of DSI, because DSI is basically generating doc ID.
 So we kind of put that in the same ballpark as a similar compute cost as,
 you know, like instead of doing a ****, like you kind of, instead of that, you
 kind of decode one document.
 So we consider that the compute cost like to be more fair than, you know,
 having to pass through a pipeline of like, and not like usually there's
 another re-ranker that does this cross attention stuff and then that
 definitely improves the performance.
 And I don't think that at this point of time, like we would beat a cross
 attention encoder.
 But, you know, basically cross encoders are just expensive.
 So that's why we consider it like out of scope for this.
 That makes sense.
 You hear you very elegantly, you output just a list of document IDs.
 I was wondering, have you ever tried to actually produce the document itself
 that you're searching for instead of the document ID?
 Because I was wondering, because the model needs to learn this
 association between the input and the document ID and it kind of needs to
 remember what text is in that document, right?
 There's no other way for it to really learn to associate text with document
 IDs.
 And I was wondering, is it a harder or an easier task for the model to
 directly output the text of the document?
 What do you think?
 I think there's a lot of challenges with decoding the document.
 I mean, you can obviously constrain your beam search to only generate stuff
 that is within a certain memory and stuff.
 And then that's definitely possible, or at least maybe the title of
 documents.
 But then I think that would, like, we have not tried that in this work.
 And then I think this is definitely interesting and it's a good point that
 you brought up.
 But I think that at least within the scope of this, we wanted to keep the
 compute low.
 And we have already in related a lot of possibilities in representing the
 doc IDs.
 And then that will probably be a different class of style of doc ID
 representation, like using natural language that can be a follow-up work.
 But the reason why we mainly don't explore that now is because there's a
 lot of additional challenges that we need to think about.
 And so we will consider that slightly out of scope for now.
 But that's definitely a great suggestion.
 And we think that it's also potentially quite viable as well.
 The only other thing I quickly add here, going back to also your question
 about the cross-encoders, these models typically have limited ability to
 essentially monopont text length, right?
 So you're limited usually to passages or parts of documents, right?
 By sort of modeling the document ID sort of as itself, you sort of open up
 the ability to model larger, more complex documents that you wouldn't be
 able to do sort of if you were treating everything as sequences of tokens,
 which again, sort of the standard.
 From the IR perspective, it's been, again, my very biased opinion, very
 unsatisfying, the move away from sort of documents that are very
 trivial to more passage retrieval that has happened recently.
 And a lot of that is just because the models have not been able to handle
 these longer sequences like they did before.
 So this takes us a little bit back to that.
 And obviously, if you have longer documents and whatnot, it'd be even
 more challenging to potentially decode that entire document.
 Though, isn't it a bit because if I think of information retrieval in the,
 let's say the olden days, what I actually retrieved was keywords, right?
 And then I simply looked up which documents the keywords belonged to.
 And I had some heuristics of how I combined for an entire document, all
 the keywords that were inside of it.
 Couldn't this also the move to passages be viewed as an expansion rather than a
 reduction in the scope of what I'm looking at?
 Do you see what I mean?
 Yeah, for sure.
 Obviously, there's always a way to aggregate from the passage level to the
 document level.
 And this is a very standard trick that people have done.
 People even did that back in the olden days when you just had
 sort of keyword-based indexes as well.
 So for sure, but then you also do have considerations of efficiency, right?
 If you're going to then go and have to score dozens of passages per document,
 that suddenly explodes the cost versus just scoring sort of at the document.
 So there's definitely trade-offs here.
 What this introduces is a level of redirection or a level of indirection in
 what the model needs to learn.
 So we no longer map sentences with the same meanings to each other, for example.
 We now have to learn this indirection almost like addressing a document by a
 variable name.
 Even with your semantically meaningful identifiers, still, I believe a large
 part as a model, I need to remember just this identifier means something.
 It stands for a particular document.
 Do you see this applicable in maybe a broader context?
 You already allude in your paper that this could be part of a
 differentiable architecture.
 Where do you see these types of indirection-based models going?
 Yeah, that's a great question.
 Actually, I was waiting to talk about this because it's something I'm really
 excited about.
 So the doc IDs, using the doc IDs, as you mentioned, is some indirection.
 You store the information in some address, and then later on, you can just
 use that address in the place of a long document and stuff.
 So I think one possible avenue here is you can imagine prompt tunings.
 This few shots in context learning might require you might need to stuff 10
 prompts, 10 examples in this large language model.
 So if this memory addressing type of architecture allows you to compress
 stuff to doc IDs, and then you can use that as for prompt tuning, or you can
 use that for retrieval augmentation.
 So I think that might be more use cases that can be explored beyond retrieval.
 So this is more of a fundamental.
 I think that you got it really very accurate where it's a class of models
 that uses this memory addressing stuff that may have more wider applications.
 So yeah, we are also quite excited about that.
 So everything that you can be like, on top of my head is mainly like maybe
 like prompt tuning or retrieval augmented models that could benefit from this.
 But yeah, as of now, we don't know that for sure.
 But yeah, this is just a guess.
 In your paper, you describe the performance of your models and the trend seems to be,
 if I recall this correctly, at least if we go to the results section real quick,
 that the larger models do perform better.
 However, the larger the data set gets, the less the improvements of, let's say,
 the DSI compared to the dual encoders are, if I understood this correctly.
 And in your data sets, you're still in the realm of 300,000 documents.
 For an IR problem, that is not really a large scale problem.
 Do you think that in the future, people might be able to expand these models to
 also become better on larger document or collection instances?
 Or do you think that the application of these types of things might be,
 as you say, much more as a differentiable component in something,
 maybe in a reinforcement learning agent or something like this?
 How do you deal with the fact that as you seem to scale the document collection size,
 the benefits get weaker and weaker?
 Yeah, so that's a good question.
 So we kind of think that it gets harder and harder to do the same thing.
 It gets harder and harder as you increase more documents.
 I think that's also because the model has to memorize or link documents to
 much more identifiers.
 So to be honest, the interplay between memorizing and retrieval
 is actually quite tough for the model to learn.
 And as you can see, you need an XSL model to almost do well on these tasks.
 But I think that to cope with larger documents, there are multiple ways.
 One of them potentially is sparse models, make sure experts,
 where you can just increase the parameter size significantly without increasing the compute.
 So we think that those are also promising to scale these models up
 to maybe a few million docs at least.
 This is like estimate.
 We don't have the results yet to show this.
 But this is what we think right now.
 And yeah, it's true that it gets harder and harder eventually.
 So we are not sure where the limit is yet.
 And we are also excited to find out where does this end
 and where's the limit of this.
 Do you have an idea of how these things scale?
 If I have double the amount of documents,
 do I need double the amount of parameters
 or do I need an order of magnitude more parameters?
 Is it related linearly, exponentially?
 Do you have any idea of how this scales?
 Off the top of my head, I'm unable to put a number on it right now.
 It's mainly like the intuition is...
 And it also depends on...
 There's one part which is the memorizing capability
 because I believe that beyond this paper,
 we have actually tried brute force memorizing
 a couple million documents.
 The model does memorize, but then there's another...
 If you need to factorize other part of how well the model
 is able to make use of this information.
 So it depends on...
 The data set depends on many factors.
 So it's very hard to say.
 But at least on NQ, we don't have
 currently we don't have beyond 300K documents,
 but going from 100K to 320K documents
 it wasn't really exactly trivial.
 So we expect that going to a 1 million docs
 in a retrieval context would be...
 If I had to put a number on it,
 it probably may need to go to 32 billion parameters
 or something like that.
 If I had to give a guess and estimate.
 Obviously, this is the standard feedback we get
 when people take a look at the paper.
 Lots of questions about the experiments,
 other data sets, scaling it up.
 I don't want to give too much away.
 Obviously, we're aware of this.
 We're working on this.
 We hope to be able to have better answers to all of these questions
 sometime soon and also demonstrate that
 this works more than just on NtU,
 on some larger data sets.
 And hopefully have much better empirical basis
 for understanding limitations and scalability of these approaches.
 I have to ask just for...
 It's a detailed question, but this NQ100K data set
 it seems to be just out of place a little bit.
 The numbers, they're just kind of off.
 It looks really good with the 10K data set
 and the 320K data set.
 You can see things either get better or worse,
 maybe as you'd expect.
 But then the 100K data set,
 it's just like, for example, the BM25 is all of a sudden
 a lot better than either on the 10K data set
 and the 320K data set.
 And likewise, in a bunch of the other numbers,
 it's just sort of out of place.
 Do you have an idea of what's going on
 with the data set as such?
 Yeah, sure.
 I think if you look at the numbers right now,
 one of the points that stand out the most
 is the bucket of the atomic doc IDs.
 The second stuff.
 Even you look at NQ320K, you see a 6.9 there randomly.
 So the fact is that for atomic doc IDs,
 there were a lot of training instability issues
 that we had to overcome.
 So there's a lot of variance and a lot of trainability issues.
 And we tried our best to overcome those.
 So sometimes you get a base model doing better than a...
 It's more of optimization and the interplay between
 the retrieval and memorization sometimes.
 I mean, I think coming from my experience of running
 many of these logical reasoning or memorizing tasks,
 sometimes the model gets it in the end,
 and then sometimes it just doesn't get it at the end
 by the end of the training.
 So I think there's generally...
 Especially for atomic doc IDs because we initialize...
 The softmax layer is initialized from scratch,
 and we use the pre-trained models.
 And also depending on the warm-up and everything.
 So it was already a challenge to optimize for the atomic doc IDs.
 That's why you see generally even on all three sets,
 there's a very...
 I think the rest of them scales pretty more nicely
 than the atomic doc IDs,
 but that is actually a big challenge that we had.
 I'm not sure if we actually explicitly point out
 this instability issue too much in the paper,
 but I think I remember mentioning somewhere,
 but at least the middle bucket is really hard to train.
 The second bucket is...
 You do mention it, yes.
 The other thing to mention...
 If you look at the BM25 number, that's not trained in any way.
 It also obviously demonstrates very different performance there.
 The other thing is just...
 There is variance when you subsample the documents.
 So if you go from 320,000 to 100,000, you're subsampling.
 Maybe that was just a very lucky, good set of documents
 that somehow was much more amenable and much more relevant in some way.
 So if you do this with any sort of, I think, standard IR system,
 you just start subsampling documents in different ways.
 You're going to get very different performance.
 I mean, probably the best thing would have been to subsample
 like five or six times,
 get some sort of error bars there to get a sense of what the variance is.
 So I suspect that probably it's a mix of the instability
 plus the fact that maybe this is a luckier,
 sort of different sample of documents in the 320k and the 10k.
 I actually have an answer about the...
 There's one point which is a bit implicit.
 It's not like...
 It's mentioned, but it's not very obvious.
 But for NQ10k and NQ100k, these are subsampled sets from NQ, right?
 And then NQ320k uses the official validation set, right?
 So there's like 10k and 100k is subsampled.
 And then I'm not exactly sure how the validation set was constructed in NQ,
 but so 10k and 100k uses a similar way of sampling.
 It's just random, but when you go to 320k,
 it's actually using the official validation set.
 So I don't know, maybe it's a bit cleaner
 or like there's some difference in the way this...
 So 10k, 100k and 320k came from the official validation set.
 So there might be some differences in the way we sample
 and how the other people sample.
 So I believe that you mentioned the training instabilities
 also at points throughout,
 and that might also explain a little bit as well
 why different methods are good at different tasks, right?
 You have, there's quite a bit of variance
 in which methods are better here or there,
 quite a bit of variance in the numbers itself.
 Although what I see is very thoroughly the case
 is that the larger models tend to do better in general.
 Whenever a model wins here with whatever way,
 it tends to be the larger models that outperform
 the smaller models within the same buckets.
 Do you think that is a property of the larger models
 being pre-trained better?
 Because larger models also exhibit better language
 modeling behavior, right?
 And given that these are pre-trained,
 I guess T5 style checkpoints, that might be an improvement
 because as far as I understand it,
 your retrieval performance also in a part depends on
 the models being pre-trained to actually understand language,
 especially the zero shot ones.
 Or do you think that is mainly a,
 the main contributor is that with more parameters
 I can memorize more documents?
 So could you comment on that?
 And maybe also a little bit on what do you think intuitively
 is going on inside of these models
 that they are even able to retrieve those IDs?
 So I think the pre-training definitely does contribute,
 like I wouldn't be able to say like how many,
 put a number on like how many percent it contributes to that.
 But I definitely think that like one way to tell is like
 probably to just rerun all the experiments with like
 randomly initialized T5 style models, right?
 I think at a very early stage,
 I mean, it's not in the paper,
 but we did run some early experiments
 with like no pre-trained models.
 And these models actually like,
 it's way harder to learn without the pre-training.
 And this is a common finding across,
 not only in this context, but in broader NLP
 and machine learning in general.
 So we think that the pre-training does a lot of like
 heavy lifting and also the size,
 like with a larger model, you also benefit more from,
 like it's the composition of two different things
 helping each other.
 So because you are pre-trained and then you also larger
 and then you benefit more from pre-training
 when you are for this T5 XXL models.
 So I think that also probably contributes to like
 the zero shot and stuff like that.
 So yeah, just to answer the question,
 especially I think that the pre-training
 does contribute a lot to this.
 Yeah.
 Yeah, I think the other thing we don't have
 a good understanding of is,
 after we fine tune on these, the DSI tasks,
 what sort of the model,
 what knowledge the model retains or does not retain, right?
 What was the nature of the model at that point?
 As others have sort of asked this question
 and I think it's a great question.
 I do suspect that some of the knowledge that
 sort of obviously pick up during pre-training
 is helping as you suggested,
 but there may be other pre-training tasks
 that are even more amenable to sort of DSI
 than sort of the standard T5 pre-training.
 Did you, have you attempted at introspecting
 these models in some way?
 To kind of see whether you can find the documents,
 whatever it means inside of these weights.
 Like, you know, I imagine since I can query these models
 and they give me a doc ID that I need to be able
 to go and look inside the weights or something
 and find traces of these documents or something.
 Like, is there something you can say
 about the inner workings or is there something
 one can see in the attention maps or in the weights?
 I have a very disappointing answer because I wish I knew
 where to look in the model as well.
 But the unfortunate thing is that I don't know
 where this is safe in the model.
 Is it in the decoder layers?
 But I think intuitively it seems like,
 because the decoder learns to output doc IDs,
 I think the decoder does quite a lot of heavy lifting
 in the model, but which weight is in?
 And there's also the feed-forward layers,
 key value memories and stuff like that.
 And then you can somehow probe that.
 I think this is interesting for a lot,
 but unfortunately we don't know where it's safe now
 in the model.
 Yeah.
 What do you think, if people want to get started with this,
 what do you think is the smallest scale thing
 that would still give meaningful insights into the technique?
 Because a certain scale is necessary if found
 or stand this correctly, right?
 But what would be the minimal setup for anyone
 to get into this type of research,
 like differentiable indexing and things like this?
 Yeah, that's a very good question, actually.
 So at what point where this gets getting meaningful,
 which scale does it get meaningful?
 I guess that's my personal opinion.
 This is just my personal opinion, obviously, this is my sense of things.
 But I think starting at around XL, 3B,
 is probably a reasonable scale to start.
 Because actually, I don't really know why 3B,
 but this is just from my experience running the experiments.
 Because 3B and 11B has slightly different training dynamics
 compared to Bayes and Lodge.
 So it's very hard to characterize this.
 It's very latent within me.
 But I think 3B, somewhere around 3B, is medium scale models.
 But small and Bayes probably will not be that meaningful.
 But I guess starting from 3B would be pretty nice.
 So that is not exactly small, right?
 I can't really run this on my 1080 at home.
 But it's still, I guess, maybe accessible to more people
 than just the biggest companies.
 Here you have a pretty interesting thing in your hierarchical document IDs.
 And I understand this is not the end all be all.
 This is like an attempt at forging meaningful document IDs.
 And you make very interesting requirements here.
 You have two requirements that they retain some semantics,
 which the clustering, I would say, gives you.
 It gives you a little bit of semantic thing.
 But then also you want to reduce the search space with each decoding step,
 which is a property of autoregressive decoding.
 The first decoding step only needs to care about the big picture,
 the next one, the smaller and the smaller.
 Do you have an idea how much these two things play together?
 Or which one is kind of the important one?
 Because one could also, I think in the review, I raised the issue,
 you could reverse this in this document ID,
 which would give you the same meaningful document identifier,
 but without this property of autoregressive decoding.
 Do you have an insight of which of the two properties
 might be the more important one here?
 And which one is, or are they interacting with each other?
 So we have thought like really like factorized both of them.
 Intuitively, I think that segmenting the search space is more beneficial,
 but I think they help each other.
 I think this is possible to also come up with ways of ablating this,
 but I think we did not try those yet.
 If you look maybe a bit more high level,
 no, wait, I have one more question.
 Yeah, this L right here, right?
 Because you have this very interesting graph that shows this thing right here,
 which document representations make the most sense and direct indexing.
 I also find it interesting that in your paper, you try out a lot of things,
 and then at the end, it seems like often the simpler things work better,
 which is a neat finding, I guess, an encouraging finding for a lot of people.
 Although I was surprised to see that
 if you index fewer tokens of the documents, it tends to perform better.
 Because that shouldn't be, right?
 What's the problem here?
 What's the problem that prevents us from indexing longer sequences of the documents?
 So this is just like my thoughts on this is that like going up to 128 and above
 makes the training harder.
 We also observe this in memorization, looking at the training accuracy of memorization.
 So I think by, and there's going to be quite some examples,
 we don't know how many examples,
 but there's going to be some examples that can be solved easily by the first 32 tokens or 65 tokens.
 So I think that the model, okay, this is just a guess,
 I'm not really 100% sure about this,
 but it's like the model prioritizes getting the one in most correctly
 rather than trying to fit 256 tokens and then not being able to solve anything, even the easy ones.
 So I think this might be what's happening.
 And then this 32, I will not over-index on this 64 or 32,
 because it's probably going to be very dataset dependent.
 And also the inverted index,
 I saw on your review that you were surprised that the inverted index didn't work.
 But this might be an artifact of this dataset.
 And it's maybe the simpler approach here,
 but when we scale up, when we go to something harder or more documents,
 or just the structure of the dataset is different, then perhaps the inverted index would help.
 So I think that there's a lot here that we are just showing a slice of the data points,
 but I wouldn't over-index or like, oh, DSI only works when the document length is short or something.
 But I think this is dataset dependent.
 And for sure, I believe that for other datasets, you need longer sequence length.
 If you look ahead a little bit, and you came into this,
 you told me at least that you just wanted to know certain things,
 like you had some questions, is this even possible and so on.
 My question is, is there an end goal here?
 If you look into the future, maybe two, three, five years or so,
 you develop this a little bit, hardware gets better and so on.
 What's the outlook? What's the North Star that this could lead to?
 Yeah, so I'm going to share a bit, and then I think Don surely has thoughts about this as well.
 So I will leave some for him.
 So I think one of the North Star here is because retrieval is generally slightly decoupled away from other NLP tests.
 People are unifying models, they are going for T5, everything is 6 to 6.
 But when it comes to retrieval, you always have this separate infrastructure of dual encoders,
 and then you have to compute ranking metrics,
 and then the whole infrastructure is always very different from machine translation or text generation stuff.
 So I think this, at least for me, one aspect of it is to be able to conveniently do retrieval
 in a way that you don't need to have a separate infrastructure.
 You can just co-train your retrieval, get all the metrics you need, get a competitive performance to dual encoders
 while still being able to do machine translation at the same time.
 So maybe machine translation may not be the best example, but maybe you want some NLU,
 some question answering model, end-to-end, or synthesizing.
 From the doc IDs, you can generate doc IDs together with text,
 and then maybe substantiating the text with doc IDs, like learning to cite and stuff like that.
 So I think these are the visions that I'm pretty excited about.
 Maybe Dawn can chime in.
 Going back to what I mentioned at the start, this is part of this exploration of what's possible.
 If you play this forward, we have no idea what's going to happen.
 One potential outcome is that it turns out that this is a great way of actually modeling
 a lot of the things that the IR community in the past has modeled in terms of documents and terms
 of all of this, and that this type of approach could be a way of unifying retrieval and scoring.
 You mentioned cross encoders. Today, usually, as you mentioned earlier, you have this cascaded approach
 where you do retrieval first and then you do scoring next.
 So this does everything together, jointly. That kind of simplifies things.
 It would be nice, I think, in the future to be able to have a way of doing that all end-to-end
 in a highly differentiable way.
 The other thing that is obvious here is that there's a lot of attention and interest recently
 with retrieval, augmented, everything.
 The idea being fewer parameters and more reliance on external memory or storage in some way.
 This is diametrically opposed to that.
 I think there's pros and cons to both of the approaches, and it will be very interesting
 to see as we continue to explore both directions what are the benefits of each of these things
 and how maybe the two of them can come together, as you were suggesting.
 Maybe DSI could be an inner loop on a retrieval, augmented approach in the future.
 If you look ahead maybe a bit more short term, what are the hardest problems that are still outstanding
 to make the next steps of progression here?
 There's actually a lot.
 It's good, right? As a researcher.
 There's a lot of things that we want to solve and there's still a lot of things that keep me up at night.
 I think there are a couple of pressing ones, like how do you update documents,
 and then solving the trainability issue and then solving the scale.
 I'm hoping that going to sparse models, something like switch transformer,
 you can just handle 20-30 million docs out of the bat.
 I think scaling is a more short term to mid term thing that we want to solve.
 So updating, scaling, and also the interplay between retrieval and understanding a little bit more
 about this zero-shot behaviour, and also understanding where it is in the model, as you mentioned.
 Understanding this behaviour of these models, I think these are immediate next steps
 that I think to take this idea further, these things need to be to some extent solved,
 or at least figured out somehow.
 Obviously, some of the questions you brought up here are things that are actively being thought about and explored.
 One of the things that we were just talking about was indexing the first 32 tokens.
 So just understanding the properties of the model across more datasets,
 and what are the best practices here, I think are also very immediate term things that we'll need to do
 to just get a basic understanding of this beyond this initial proof of concept, if you will,
 that this crazy idea is even feasible.
 Is there anything else that maybe we haven't touched on yet that you would like people to take away from the paper
 that they shouldn't go without knowing?
 That's a good question.
 Nothing that I can do.
 Yeah, I can't think of anything right now.
 Even if the models are large, could people get into this?
 Is the code somewhere available or are you planning to make it?
 This is subject to approval, but we do have plans to make the code available sometime in Q2 of this year.
 But this is all subject to approval.
 We have not gotten the approval yet as of now, but this is our plan to release it in Q2.
 The fight with the lawyers. Excellent.
 We have a history of open sourcing.
 You've reviewed several of our papers in the past.
 We do have a history of being able to release the code.
 It's just a matter of checking various boxes, and we're committed to this.
 We've already had folks reaching out, trying to replicate this, and we want to make it easy for everyone
 so that they can get going with this.
 I think it's a really interesting area, and hopefully this will stimulate some additional fun research.
 I was in Google for a while.
 I know it can be a hassle to open source anything and the amount of approvals you need to get.
 Props that you even want to go through with it. It's pretty cool.
 All right. Don and Yi, thank you very much for being here.
 This was very enlightening, and I hope people had fun.
 I hope to see you again soon.
 Thanks for inviting me.
 This was great.
 It was great, yeah.