A Deepdive into Tantivy
Masurel, Paul
Tantivy is a search engine library, akin to Lucene but for Rust. In this talk, we will walk through the building blocks that make a scalable full-text search engine. While this talk will be centered on tantivy, most of the concepts introduced should apply to other inverted-list search engines like Lucene.
 I'm going to start by a very short call about me. I'm a friend to Rokori in Japan and started this project around three years ago. Right now I'm working as a law and I've been asked specifically to be clear about how this project is not a really complicated system. I've been asked a lot of questions about the way this project is done. What's the purpose of this project? Within this community, 20D is closer to UC. I don't know if everyone is familiar with Rust. This project is very popular, for those who are familiar with Rust, it's a program in English that is kind of young. You can consider Rust as a policy post. The code that you write is basically safe. You get the great ecosystem and great building and the same method that it seems to be learning. What is the developers whose goal iseted so far? So you can apply it on platform that are not x86, 64 bit, so you can actually run it on R, at number four. And I actually even provide a concept for whether something wants. So that's something that I probably don't talk to you all too much because there's not much performance training right now in the web assembly, but at one point, it will be an advantage if you run such an engineer worker. It's still cute, it's very important for me to keep some of our clients quite small, there's only 30,000 minds of all, cool, even if you include all the different crates that I had to write for the concept. It's very important for me because as I told you, it's a pet project and you write for pet pet projects, it's quite small and cute. And also ideally I would love in some future to have companies that are considering between their own such engine parts which they have for one reason or another, if they can actually receive for that project. Rather than starting from scratch and the idea, I'd love them to at least consider using them. And I think that feeling is not the best, it's the best for people who actually embrace the code and own it, get it, own it and work with it. It's fast, so I put a link on the benchmark here, I think we want that then to go through the benchmark, but there's an interesting benchmark that, for example, the performance of Tansili, per voice, it can look like for the shank of the way. Usually it's doing really well, it can only focus on different versions of the NCD server. But what's really interesting is that you can actually identify, oh, for this way, the scene is actually faster, what's going on there. So that's a great way for me to explore different optimizations that are in the scene and do not have to set it in particular. I can know what would be the next step for Pansili. I thought there was a way around for some query like Pansili is abnormally faster than this scene. It's a good thing that there is no transition that's happening. So I suspect that everybody in the room is more interested in this scene than in the search. Some of you are just wondering, what's the best way to run for the next generation? So I'm not gonna look too much about the rest actually, but I find that from now on, we're gonna build a search engine together. And as a matter of fact, what you do is only by using, all of the servers they're gonna talk from now on are very valid for using the screen as a scene design. So hopefully at the end of this talk, you can see everybody wants to understand the scene. Maybe a little bit better. So yeah, there's really a search engine together. Most of the time, our goal will be, we want the search engine that is, I'm trying to come up with a scale here because we are still coding both. The search engine that is sitting in one machine. But we want to be able to search an index, terabytes of data, a lot of data, on machine that is, the only gigabyte is one. So we want to make sure that all of the stuff that we are doing are solid and they don't require you to have quite a gigantic amount of RAM. And so, until you can do that, I actually tested it, that's something that I did. So right now we are at 20.8 and I did test the first case, this one is your.6. And what I did is that, so this is probably a very good data set for its command code. And what it is is kind of a snapshot of the core of the web. And it's a pretty large one, a very decent one. It's around 3 billion pages. So I took this snapshot, I downloaded it with, to be accurate, I downloaded half of it. I downloaded half of it using my own just known internet connection. It's using my internet provider base. It took around one month full time. I mean, Japan, so we have like, optical fiber and stuff, so it is pretty good in the impression that it's good. And I wanted to test if I could index it, see what happens because like, it's a very good test to do that in your home, especially like, I bought the house in Japan and the electrical network was actually great for my usage. I mean, I'm French, so I drink coffee and I have a ghost in the morning and when I would switch ghosts at the same time, I would already have a dog in my home. That's a great test point for me. Thanks, thanks, also. Yeah, so I did that and I thought, why some are really on the process to, and basically the goal of the Python library is to extract, like you give it a query and there is a placeholder. So for instance, you will say, French people are and then you ask for all of the adjectives that follow these centers. And so, two libraries together, one of these is the small NLB library, made it possible to look for the internet, the most common adjectives that was French people. Oh, man. I think it's pretty accurate. Okay, so let's move on. So, to use something that was perfect, so should we use a graph to, we have to use almost the same way. Can you check if you know that, please? I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. Green. That's green. All right. All right. All right, better. Okay, so usually that means that you're going to want to build an inverted index. An inverted index is simply something that associates to each term a list of sorted lists of the queries. And the simple data structure is sufficient to actually compute any kind of Boolean query. So I'm going to explain rapidly how our intersection works. So actually means that we are trying to find all of the documents that are matching the query, run and migrate. We are going to have like this two posting lists. We take the first one. The first document of the first one is two. And we are going to seek into the second posting list for the document two. So by seeking what I mean is that we are going to do exactly as if we are advancing, going through the scanning through the second posting list. And we are going to stop at the moment where we find the document that is greater or equal to the document. If it's equal, then that means that two is in the intersection. If it's greater, then it's not. So here it's not. And we found the document five. And we are going to like do symmetrically. We are going to seek for five in the first posting list. Here we go. Five was found. So it's going to be in our output. We can advance the first posting list. And now we are seeking for seven in the second list and so on so forth until we reach the end of one of the posting lists. So the only takeaway there is that if you want fast intersection, you need to have a fast way to seek through the posting list. For union, we could use the same strategy. Basically, it's like the merge. Everybody has a different name for this algorithm, but we could merge those two posting lists and get the union. But that's actually inefficient. So I'm using the same trick as Lucine does. The idea is that you prepare in advance some kind of bit set here. And so just for the sake of this slide, the bit set is actually on offense eight, but in Don TV, we're using 4,096 bits, so it's much larger. And what we do is that we are going to take the first posting list and we are going to append all of the docs that are between two and two plus. So this horizon, in this case, eight, but in Don TV, 4,096. And we append those documents in this bit set. So in this bit set, the first bit will mean two, the second bit will mean three, and so on and so forth. We do that for the first posting list, and we do that on the second time, on the second posting list. Here's what we get for bit set, and now we can flush it. So we transform basically this bit set into the document that it represented. So it can be done quite fast because your CPU usually has an instruction to pop the lowest significant bit out of a 64-bit world. So you just pop it and so on and forth. All right. So that's how an inverted index is used to compute Polyanquery. Now the question is, how do we represent that on disk? So Don TV is relying on M-Map for all of its I.O. And when you start Don TV, the only thing that it does is basically mapping all of the file of the index, so it goes really, really fast, and it's ready to go. So we don't have to load any data structure and put that in anonymous memory and have some kind of hash map in anonymous memory. Everything is on M-Map. So the startup time is very, very nice. But that means that we need to have data on the disk that is usable as is for Java people. Anonymous memory is more or less like the heap. So the first data structure that we are going to have to you have in our index is the term dictionary. So a term dictionary, maybe our term dictionary will be broken into two steps. One step will associate the terms, like the sequence of bytes associated with the term to some kind of term ID. And then we're going to have another data structure that associates the term ID to some kind of term info structure that basically have pointers to files and the beginning of the pushing this. I'm not going to talk about the second data structure because it's going, but let's talk about the first one. So how do we go from terms, sequence of bytes to term ID? So if you are actually trying to build your own search engine, you will have two broad kind of family for this solution. One could be like hash-based in general. Maybe you will go fancy and have like perfect hash or anything like that. It's a very nice solution in the sense that you will get like very fast lookups and especially if you don't have much RAM and everything is sitting on your hard disk, you will probably require very little I.O. to do that. Your hash map will be able to send you directly in the right place on your disk. Another solution is using like a tree-based solution or like a trial. This will use slightly more CPU and you will have a lot more RAM than I.O. so that kind of depends on the layout of your data, but you tend to jump from one node of your try to another node of your try. So it might be a lot of six if your data is on your disk. But it has a lot of benefits. So one benefit is you can iterate through a round of keys. You can, so naturally you will probably use term ordinals. So by term ordinals, I mean if Arabic is your first word in your dictionary, then it will get term ID zero. The first word when sorted the x-y-cography plane and then the second word might be something starting by a B, then it will get term ordinals B, term ordinals two and then one's fourth. So term ordinals will, your term IDD will be sorted exactly the same as your terms. That's a very nice property. Importantly, you can do an intersection of your try or your try light structure with a DFA. So are you guys familiar with what a DFA is? OK, we'll still explain it a little. I actually made a mistake in the DFA here, but so that's one. So DFA stands for Deterministic Finite Automaton, and that's one way to implement regular expression. So you can transform any regular expression into an automaton like this one. And the way it works is once you have it into this shape, then matching a string on this automaton means that you're going to consume every charge in your string. So you start in the state over there. I was hoping I could point out so that you can jump. So you start in the white state over there, and let's say that you are trying to match carousel. Your first charge is C. You look at the outbound arrows that are emitted by the state you are in. Our one is labeled with a C, so that means that after consuming the C, I'm in the state, so yellow one. And then carousel, second letter is an A. I'm going to follow this letter, this arrow, and I'm in the blue state, R. I end up in this state, and Usel, until the Use, I guess, without the L, will be following the arrow with a star over there, and then the last L will bring me to the end state. It's really nice because I'm advancing one charge at a time. I just have to look at the end state of my string to say whether I matched or not. I mark the state that matched by a double circle here. So our try looks like this, and it's actually possible to match this DFA over the try very efficiently. So I'm going to show how it's done. So here, if we consume the C, we end up in the yellow state, just like for carousel. If I consume the A, I end up in the blue state. If I get a B, then I end up in this gray state, which happens to be a sink. So the sink is a dead end. We will never match if we reach this state. We don't need to see what happens with the following characters. And that's great because that means that all of this subtree, we don't care about it anymore. So for the purpose of fitting soon into the slide, this try is very simple. But you can imagine that maybe there is a gigantic tree that is a child of this node. And we just cut that, so that's much, much faster. And now we want to match M. We don't have to recompute what is the state required for A. We can just look at the state of A before. So that's blue. And we see that we are in the sink state again. And so on and so forth. So we are just putting colors of our try to guess which term in our dictionary are matching our deterministic finite automaton. That means that I can go through all of the terms that match a regular expression. But also I can get very rapidly all of the terms that are at Levenstein's distance or edit distance of one or two. That means that if you are asking me, please stream me all of the terms that are one type away from what I typed, I can do that very, very efficiently. So Tantivi is using a tree-based solution. It's actually using a finite state transducer within those two. And I didn't have to code anything. The Rust ecosystem is nice enough that somebody already coded a very nice implementation of finite state transducer. I will not explain how it works because I didn't code it. But the thing that you might want to know about it is pretty much like a try except that nodes can share suffixes. So a try would never have an error that goes like this. So because it can share suffixes, you end up with something that is slightly more compact than a try. And that's always a very nice feature when you're being spaced with your RAM. Now let's talk a little bit about how we are going to do the same thing. We're going to use one code, posting lists. So posting lists are a lot of integers. It's going to take a lot of the size of your index. And you're going to want to compress them. Integer compression is a field that has been well studied. There is a lot of solutions to do integer compression. I put a chart over there. My point here is you have a trade-off between something that is compressing a lot. So it's always like that. You have to choose with something that compresses a lot and something that is very fast. Another thing that I need to point out is basically all of the algorithm over there and all of these, all of the best algorithms, they are all using CMD instructions. The CMD instructions are instructions on your CPU that makes it possible for you to process four or eight integers at a time. So that's something that actually, Tantivi is using a lot. So Tantivi is actually using this code there. So it's a compression scheme that has been designed by Daniel Lemire. I recommend you to read this blog. It's always very interesting if you're interested in things related with search or data structures. I used to depend on his library actually in C and I removed it because I prefer to be entirely in Rust and I re-implemented it entirely in Rust. The gist of it is like many of those scheme, your posting lists are increasing so it starts by doing something called tell-down coding. So instead of compressing your integer directly, what you do is that you take the difference between two consecutive integers and you get something that is much smaller. So you take a pack of those. So in my case, blocks are 128 integer long and out of those 128 deltas, you look at the one that is the largest. So I think in this example, there should be this one, that's 16. And you notice that it can be represented using five bits. So what you do then is that you represent all of your deltas over five bits and you just concatenate this version. So that's called bit packing. So I describe what was the solution for the scalar version of bit packing. What I do is actually I use a CMD instruction for that and I do that with four integer at a time. And the algorithm really looks like this trick that we were using at school to avoid writing lines when we are punished by our professor. So it's exactly the same algorithm as the scalar solution. You just use CMD instruction in place of those scalar instructions. It's very, very simple. And there is an interesting improvement compared to the methods that existed before Daniel and Mir's paper, which is the algorithm also take advantage of the fact that as we are decoding those integer, we end up at one point where we have one register with the delta that we are decoding and we can decode the delta using CMD instruction as well. So this is also CMD instruction. And we do that at the point where they are already in the registers. It goes really, really fast. And we're talking about, when I say fast to give you an idea, I'm talking about four billion integers a second. So we are already very, very, very, we are flirting with the bandwidth of your run, basically. So I told you that Rust was nice and was able to compile very efficient code that it was very close to C++. So I wrote that in Rust and this is a summary that is generated. I'm not going to tell you that I understand what's in there. But the important point here is that all of the long-looking instruction over there, every single one of those is working on four bytes at a time. And you will see scalar instruction maybe here, here, here. So the assembly code that is generated is really as good as possible. It's very, very close to, it's probably exactly the same as what we had in C++. So I think we talked this morning a little about BM25 and TFI-DFF. This requires to, so those are like the scoring, the default scoring function of Lucine. Tantivi comes in with BM25 also. This requires to have access to the term frequency as we search. So when we match a document, we need to have access to the number of time a term appears in the document. And it's nice to have good locality for that. So we actually interleave blocks of the KDs with block of term frequencies so that we are likely to have it in a nice cache at the moment where we are reading the, as we are doing the match, as we are running the query. So we have one block of the KDs and then one block of term frequency and so on and so forth. Each block is representing 128 blocks. So unfortunately, posting is not always exactly a multiple of 128 blocks. So the last one is using a different compressing scheme. That is not interesting. And so we also need to have the information of how many bits are used to encode each block. And we also would like to have some way to avoid decompressing this block if we are running an intersection, for instance. Maybe the document is not in this block and in that case we would like to entirely avoid decompressing it. So for that, we have another structure on top of that that allows us to precisely skip and decompress only the blocks that might be interesting. We are going to talk about something else now. So I said in the beginning of the talk that we wanted to be able to index an arbitrary large amount of data, which is about terabytes, and that's a big problem because if we have a reduced amount of RAM, how are we going to be able to build this nice data structure on this? We have another problem, which is people might create an index and they want to add new documents. We want something that is dynamic. We want to be able to add documents to an existing index. That's another problem. The solution is the same for both of these problems. Just to refine the idea of why it's difficult to add new documents in an index like that. The data structure that I described to you on disk is extremely compact. Everything is super compressed. Everything is laid out on disk one after the other. So if it's on the bookshelf, it looks like this. If you're in front of a bookshelf like that and somebody gives you a book and tells you to put that in the library, you're going to suffer. You're going to have to move everything away. It's a nightmare. What you want is to have something that looks like that if you want to update your bookshelf all of the time. So generally speaking, there is a trade-off between being dynamic and being compact. So there is a nice trick that exists in many database that is called the logarithmic method. The idea will be we're going to have compact bookshelves like that, but we are going to have many of them. So the way it works is the user will tell Tantivi or will tell Lucine, hey, Tantivi, I will give you a budget of, let's say, 300 megabytes. And these 300 megabytes will be used for the dynamic bookshelf here. So that's going to be a big bookshelf. And people will add books into that until the bookshelf is full. Once the bookshelf is full, we are going to transform it into a very compact bookshelf. So I'm going to stop with the major, but basically we serialize our dynamic data structure into the static data structure that I described before. And this piece of very compact index is called a segment. So that's a word that you probably heard in Lucine, but that's a segment. And both in Lucine and in Tantivi, there is a strong architecture choice. A segment is basically an independent index. So you could literally pick a segment and copy it into another index, and it will run just the same as long as the schemas are the same. That's a very interesting property, actually. So if we do that, we are going to end up with a lot of compact segments. And maybe that's not optimal. If we have a 1 terabyte of data, that would be a lot of small segments. And when we do a search, we are going to have to do as many lookups in the dictionary. And that's very inefficient, right? So on the background, we also have a process that merge those segments together. And you can actually control, you can define on your own as a user what is a strategy used to merge these segments. But it comes with a default strategy called the Long Merge Policy. And the heuristic behind it is just we try to merge, by default, it's going to be eight segments that have about the same size. And yeah. So this technique is also really nice because that means that we can do multi-threaded indexing very, very easily. So, potentially, I ask you how much data you give as a budget and also how many indexing threads you want to use. And then when you open the document, what you are doing is that you are pinning the document to a document queue. And in the background, you don't control it, but there is your given number of indexing threads that are consuming this document queue and that are populating these dynamic structures that I was talking about. And once they reach their capacity, then they are going to serialize the segment. The serial segment is the very compact data structure that we use in the end. And some merging threads will merge those very transparently. Yes. Thank you. So there is a bunch of pros and cons to this. One problem that is very evident when you use Tantilli and I think it's still true for, to some extent, it's true with Lucine is that when you add a document, it's not searchable right away. So it's quite puzzling for people who are using SQL database. Because we decided to have our segments to be independent index, they do not share any dictionary. And sometimes that makes specific use of search difficult not to be able to actually merge the result from the different segment very easily. Yes. There is not one dictionary. I didn't talk about deletes and updates, but those are a natural nightmare. It's very complicated. Yes. That's pretty much all I have on this slide. And then the pros will be the index throughput is actually excellent if you can batch stuff. If you don't have to commit all of the time, you can reindex a gigantic amount of data. So seeing on my laptop, I index Wikipedia in, I thought, I restocked it for two or three minutes, but that's the amount of English Wikipedia. But that's the speed we're talking about. Wikipedia is like a tiny benchmark in search. It's not a big dataset. Yes, so segments are independent index. So you could decide to actually do your indexing on Hadoop and just copy the file of the segment in the same place. And everything will work just the same. Or you could decide to do a distributed search by sending the segment and dispatching them. It's free just copying files. It's very transparent. And because we write files and then we never touch them again, we just write very large files and then they're read only. There is no problem of flux or anything. Readers just read the file and nobody is touching them. So it simplifies a lot of my work. I have a bunch of slides there. Let's keep then so that we have a bit of time to ask questions. So do you guys have questions? Yeah. I'll try to say that about Lucille is a little bit of a corner case for Java. It's like basically a library which is difficult for Java to handle because there are lots of low level start going on here. So one question. Your project and Rust, do they fit together or did you find trouble using Rust as beautiful? I wanted to see the little card. So the question is Rust and building a search engine, a good fit, I guess, especially in contrast with Lucille and Java. So I would say yes. So the main benefit that I get from Rust, I guess, compared to Java is first I can access SIMD instruction. Second, I have a lot of control of, I know when I write code whether I'm going to get static dispatch or dynamic dispatch. That's something that is extremely hard in Java. You never know what happens. Also all of the things like M-map being a nightmare in Java, it's something that I don't have any problem about. I don't know if everyone knows what I'm talking about. But basically when you're working with Orfit memory in Java, like if you are mapping a huge amount of data and you have very little data that is on your heap, your data will be unmapped only at the moment where the object that is holding the M-map thing is garbage collected. And garbage collection happens only if your heap is actually kind of full. I think that's a big problem with Java. So some JVM give you a non-official Java API to unmap stuff. Some people just decide to use GNI code to do the M-map. Well, I never have that kind of problem with Rust. Everything, I'm working with C or C++. And when I started this project, I did not know Rust at all. And I started on TV to actually learn Rust, which is a bit stupid and crazy. So I already knew C++ and I had done a lot of Java as well. And after around two weeks, you don't have to believe me, but I was more productive in Rust and I felt safer writing code in Rust. I was in both Java and C++. Can you explain the name and the logo of the words? So Pantyby is an English word. I often do that. Actually, I like choosing English words that nobody knows. As project names. They're actually not bad at all for SEO and yeah, people learn a new word. Everybody is happy, right? So Pantyby means at full gallop. Hands off of. And I drew the horse myself. I think that's one of the best achievements in the project. Yes? Thank you.
 I'm going to start by a very short call about me.  I'm a friend to Rokori in Japan and started this project around three years ago.  Right now I'm working as a law and I've been asked specifically to be clear about how this project is not a really complicated system.  I've been asked a lot of questions about the way this project is done.  What's the purpose of this project?  Within this community, 20D is closer to UC.  I don't know if everyone is familiar with Rust.  This project is very popular, for those who are familiar with Rust, it's a program in English that is kind of young.  You can consider Rust as a policy post.  The code that you write is basically safe.  You get the great ecosystem and great building and the same method that it seems to be learning.  What is the developers whose goal iseted so far?  So you can apply it on platform that are not x86,  64 bit, so you can actually run it on R,  at number four.  And I actually even provide a concept  for whether something wants.  So that's something that I probably don't talk  to you all too much because there's  not much performance training right now  in the web assembly, but at one point,  it will be an advantage if you run  such an engineer worker.  It's still cute, it's very important for me  to keep some of our clients quite small,  there's only 30,000 minds of all,  cool, even if you include all the different crates  that I had to write for the concept.  It's very important for me because  as I told you, it's a pet project  and you write for pet pet projects,  it's quite small and cute.  And also ideally I would love in some future  to have companies that are considering  between their own such engine parts  which they have for one reason or another,  if they can actually receive for that project.  Rather than starting from scratch and the idea,  I'd love them to at least consider using them.  And I think that feeling is not the best,  it's the best for people who actually embrace the code  and own it, get it, own it and work with it.  It's fast, so I put a link on the benchmark here,  I think we want that then to go through the benchmark,  but there's an interesting benchmark that, for example,  the performance of Tansili, per voice,  it can look like for the shank of the way.  Usually it's doing really well,  it can only focus on different versions of the NCD server.  But what's really interesting is that you can actually  identify, oh, for this way,  the scene is actually faster, what's going on there.  So that's a great way for me to explore  different optimizations that are in the scene  and do not have to set it in particular.  I can know what would be the next step for Pansili.  I thought there was a way around for some query  like Pansili is abnormally faster than this scene.  It's a good thing that there is no transition  that's happening.  So I suspect that everybody in the room  is more interested in this scene than in the search.  Some of you are just wondering,  what's the best way to run for the next generation?  So I'm not gonna look too much about the rest actually,  but I find that from now on,  we're gonna build a search engine together.  And as a matter of fact,  what you do is only by using,  all of the servers they're gonna talk from now on  are very valid for using the screen as a scene design.  So hopefully at the end of this talk,  you can see everybody wants to understand the scene.  Maybe a little bit better.  So yeah, there's really a search engine together.  Most of the time, our goal will be,  we want the search engine that is,  I'm trying to come up with a scale here  because we are still coding both.  The search engine that is sitting in one machine.  But we want to be able to search an index,  terabytes of data, a lot of data,  on machine that is,  the only gigabyte is one.  So we want to make sure that all of the stuff  that we are doing are solid and they don't require you  to have quite a gigantic amount of RAM.  And so,  until you can do that,  I actually tested it, that's something that I did.  So right now we are at 20.8 and I did test  the first case,  this one is your.6.  And what I did is that,  so this is probably a very good data set  for its command code.  And what it is is kind of a snapshot  of the core of the web.  And it's a pretty large one, a very decent one.  It's around 3 billion pages.  So I took this snapshot, I downloaded it with,  to be accurate, I downloaded half of it.  I downloaded half of it using my own  just known internet connection.  It's using my internet provider base.  It took around one month full time.  I mean, Japan, so we have like,  optical fiber and stuff, so it is pretty good  in the impression that it's good.  And I wanted to test if I could index it,  see what happens because like,  it's a very good test to do that in your home,  especially like, I bought the house in Japan  and the electrical network was actually great for my usage.  I mean, I'm French, so I drink coffee  and I have a ghost in the morning  and when I would switch ghosts at the same time,  I would already have a dog in my home.  That's a great test point for me.  Thanks, thanks, also.  Yeah, so I did that and I thought,  why some are really on the process to,  and basically the goal of the Python library  is to extract,  like you give it a query and there is a placeholder.  So for instance, you will say,  French people are and then you ask for all of the adjectives  that follow these centers.  And so, two libraries together,  one of these is the small NLB library,  made it possible to look for the internet,  the most common adjectives that was French people.  Oh, man.  I think it's pretty accurate.  Okay, so let's move on.  So, to use something that was perfect,  so should we use a graph to,  we have to use almost the same way.  Can you check if you know that, please?  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  Green.  That's green.  All right.  All right.  All right, better.  Okay, so usually that means  that you're going to want to build an inverted index.  An inverted index is simply something that associates  to each term a list of sorted lists of the queries.  And the simple data structure is sufficient to actually  compute any kind of Boolean query.  So I'm going to explain rapidly how our intersection works.  So actually means that we are trying to find all of the  documents that are matching the query,  run and migrate.  We are going to have like this two posting lists.  We take the first one.  The first document of the first one is two.  And we are going to seek into the second posting list for  the document two.  So by seeking what I mean is that we are going to do exactly  as if we are advancing, going through the scanning  through the second posting list.  And we are going to stop at the moment where we find the  document that is greater or equal to the document.  If it's equal, then that means that two is in the  intersection.  If it's greater, then it's not.  So here it's not.  And we found the document five.  And we are going to like do symmetrically.  We are going to seek for five in the first posting list.  Here we go.  Five was found.  So it's going to be in our output.  We can advance the first posting list.  And now we are seeking for seven in the second list and so on  so forth until we reach the end of one of the posting lists.  So the only takeaway there is that if you want fast  intersection, you need to have a fast way to seek through  the posting list.  For union, we could use the same strategy.  Basically, it's like the merge.  Everybody has a different name for this algorithm, but we  could merge those two posting lists and get the union.  But that's actually inefficient.  So I'm using the same trick as Lucine does.  The idea is that you prepare in advance some kind of bit set  here.  And so just for the sake of this slide, the bit set is  actually on offense eight, but in Don TV, we're using 4,096  bits, so it's much larger.  And what we do is that we are going to take the first posting  list and we are going to append all of the docs that are  between two and two plus.  So this horizon, in this case, eight, but in Don TV, 4,096.  And we append those documents in this bit set.  So in this bit set, the first bit will mean two, the second  bit will mean three, and so on and so forth.  We do that for the first posting list, and we do that on the  second time, on the second posting list.  Here's what we get for bit set, and now we can flush it.  So we transform basically this bit set into the document that  it represented.  So it can be done quite fast because your CPU usually has  an instruction to pop the lowest significant bit out of a 64-bit  world.  So you just pop it and so on and forth.  All right.  So that's how an inverted index is used to compute  Polyanquery.  Now the question is, how do we represent that on disk?  So Don TV is relying on M-Map for all of its I.O.  And when you start Don TV, the only thing that it does is  basically mapping all of the file of the index, so it goes  really, really fast, and it's ready to go.  So we don't have to load any data structure and put that in  anonymous memory and have some kind of hash map in anonymous  memory.  Everything is on M-Map.  So the startup time is very, very nice.  But that means that we need to have data on the disk that is  usable as is for Java people.  Anonymous memory is more or less like the heap.  So the first data structure that we are going to have to you  have in our index is the term dictionary.  So a term dictionary, maybe our term dictionary will be broken  into two steps.  One step will associate the terms, like the sequence of bytes  associated with the term to some kind of term ID.  And then we're going to have another data structure that  associates the term ID to some kind of term info structure  that basically have pointers to files and the beginning of  the pushing this.  I'm not going to talk about the second data structure because  it's going, but let's talk about the first one.  So how do we go from terms, sequence of bytes to term ID?  So if you are actually trying to build your own search engine,  you will have two broad kind of family for this solution.  One could be like hash-based in general.  Maybe you will go fancy and have like perfect hash or  anything like that.  It's a very nice solution in the sense that you will get  like very fast lookups and especially if you don't have  much RAM and everything is sitting on your hard disk,  you will probably require very little I.O. to do that.  Your hash map will be able to send you directly in the right  place on your disk.  Another solution is using like a tree-based solution or like a  trial.  This will use slightly more CPU and you will have a lot more  RAM than I.O. so that kind of depends on the layout of your  data, but you tend to jump from one node of your try to another  node of your try.  So it might be a lot of six if your data is on your disk.  But it has a lot of benefits.  So one benefit is you can iterate through a round of keys.  You can, so naturally you will probably use term ordinals.  So by term ordinals, I mean if Arabic is your first word in  your dictionary, then it will get term ID zero.  The first word when sorted the x-y-cography plane and then  the second word might be something starting by a B,  then it will get term ordinals B, term ordinals two and  then one's fourth.  So term ordinals will, your term IDD will be sorted exactly  the same as your terms.  That's a very nice property.  Importantly, you can do an intersection of your try or your  try light structure with a DFA.  So are you guys familiar with what a DFA is?  OK, we'll still explain it a little.  I actually made a mistake in the DFA here, but so that's one.  So DFA stands for Deterministic Finite Automaton,  and that's one way to implement regular expression.  So you can transform any regular expression into an automaton  like this one.  And the way it works is once you have it into this shape,  then matching a string on this automaton means that you're  going to consume every charge in your string.  So you start in the state over there.  I was hoping I could point out so that you can jump.  So you start in the white state over there, and let's say that  you are trying to match carousel.  Your first charge is C. You look at the outbound arrows that  are emitted by the state you are in.  Our one is labeled with a C, so that means that after consuming  the C, I'm in the state, so yellow one.  And then carousel, second letter is an A.  I'm going to follow this letter, this arrow,  and I'm in the blue state, R.  I end up in this state, and Usel, until the Use, I guess,  without the L, will be following the arrow with a star over there,  and then the last L will bring me to the end state.  It's really nice because I'm advancing one charge at a time.  I just have to look at the end state of my string to say  whether I matched or not.  I mark the state that matched by a double circle here.  So our try looks like this, and it's actually possible to match  this DFA over the try very efficiently.  So I'm going to show how it's done.  So here, if we consume the C, we end up in the yellow state,  just like for carousel.  If I consume the A, I end up in the blue state.  If I get a B, then I end up in this gray state,  which happens to be a sink.  So the sink is a dead end.  We will never match if we reach this state.  We don't need to see what happens with the following characters.  And that's great because that means that all of this subtree,  we don't care about it anymore.  So for the purpose of fitting soon into the slide,  this try is very simple.  But you can imagine that maybe there is a gigantic tree that  is a child of this node.  And we just cut that, so that's much, much faster.  And now we want to match M.  We don't have to recompute what is the state required for A.  We can just look at the state of A before.  So that's blue.  And we see that we are in the sink state again.  And so on and so forth.  So we are just putting colors of our try to guess which  term in our dictionary are matching our deterministic finite  automaton.  That means that I can go through all of the terms that match  a regular expression.  But also I can get very rapidly all of the terms that are at  Levenstein's distance or edit distance of one or two.  That means that if you are asking me, please stream me all of  the terms that are one type away from what I typed,  I can do that very, very efficiently.  So Tantivi is using a tree-based solution.  It's actually using a finite state transducer within those two.  And I didn't have to code anything.  The Rust ecosystem is nice enough that somebody already coded a  very nice implementation of finite state transducer.  I will not explain how it works because I didn't code it.  But the thing that you might want to know about it is pretty  much like a try except that nodes can share suffixes.  So a try would never have an error that goes like this.  So because it can share suffixes, you end up with something  that is slightly more compact than a try.  And that's always a very nice feature when you're  being spaced with your RAM.  Now let's talk a little bit about how we are going to  do the same thing.  We're going to use one code, posting lists.  So posting lists are a lot of integers.  It's going to take a lot of the size of your index.  And you're going to want to compress them.  Integer compression is a field that has been well studied.  There is a lot of solutions to do integer compression.  I put a chart over there.  My point here is you have a trade-off between something  that is compressing a lot.  So it's always like that.  You have to choose with something that compresses a lot  and something that is very fast.  Another thing that I need to point out is basically all of  the algorithm over there and all of these, all of the best  algorithms, they are all using CMD instructions.  The CMD instructions are instructions on your CPU that  makes it possible for you to process four or eight integers  at a time.  So that's something that actually, Tantivi is using a lot.  So Tantivi is actually using this code there.  So it's a compression scheme that has been designed by  Daniel Lemire.  I recommend you to read this blog.  It's always very interesting if you're interested in things  related with search or data structures.  I used to depend on his library actually in C and I removed  it because I prefer to be entirely in Rust and I  re-implemented it entirely in Rust.  The gist of it is like many of those scheme, your posting lists  are increasing so it starts by doing something called  tell-down coding.  So instead of compressing your integer directly, what you do  is that you take the difference between two consecutive  integers and you get something that is much smaller.  So you take a pack of those.  So in my case, blocks are 128 integer long and out of those  128 deltas, you look at the one that is the largest.  So I think in this example, there should be this one,  that's 16.  And you notice that it can be represented using five bits.  So what you do then is that you represent all of your deltas  over five bits and you just concatenate this version.  So that's called bit packing.  So I describe what was the solution for the scalar version  of bit packing.  What I do is actually I use a CMD instruction for that  and I do that with four integer at a time.  And the algorithm really looks like this trick that we were  using at school to avoid writing lines when we are  punished by our professor.  So it's exactly the same algorithm as the scalar solution.  You just use CMD instruction in place of those scalar  instructions.  It's very, very simple.  And there is an interesting improvement compared to the  methods that existed before Daniel and Mir's paper,  which is the algorithm also take advantage of the fact that  as we are decoding those integer, we end up at one point  where we have one register with the delta that we are  decoding and we can decode the delta using CMD instruction  as well.  So this is also CMD instruction.  And we do that at the point where they are already in the  registers.  It goes really, really fast.  And we're talking about, when I say fast to give you an idea,  I'm talking about four billion integers a second.  So we are already very, very, very, we are flirting with the  bandwidth of your run, basically.  So I told you that Rust was nice and was able to compile  very efficient code that it was very close to C++.  So I wrote that in Rust and this is a summary that is generated.  I'm not going to tell you that I understand what's in there.  But the important point here is that all of the long-looking  instruction over there, every single one of those is working  on four bytes at a time.  And you will see scalar instruction maybe here, here,  here.  So the assembly code that is generated is really as good as  possible.  It's very, very close to, it's probably exactly the same as  what we had in C++.  So I think we talked this morning a little about BM25 and  TFI-DFF.  This requires to, so those are like the scoring, the default  scoring function of Lucine.  Tantivi comes in with BM25 also.  This requires to have access to the term frequency as we  search.  So when we match a document, we need to have access to the  number of time a term appears in the document.  And it's nice to have good locality for that.  So we actually interleave blocks of the KDs with block of  term frequencies so that we are likely to have it in a nice  cache at the moment where we are reading the, as we are doing  the match, as we are running the query.  So we have one block of the KDs and then one block of term  frequency and so on and so forth.  Each block is representing 128 blocks.  So unfortunately, posting is not always exactly a multiple of  128 blocks.  So the last one is using a different compressing scheme.  That is not interesting.  And so we also need to have the information of how many bits are  used to encode each block.  And we also would like to have some way to avoid  decompressing this block if we are running an intersection,  for instance.  Maybe the document is not in this block and in that case we  would like to entirely avoid decompressing it.  So for that, we have another structure on top of that that  allows us to precisely skip and decompress only the blocks that  might be interesting.  We are going to talk about something else now.  So I said in the beginning of the talk that we wanted to be able  to index an arbitrary large amount of data,  which is about terabytes, and that's a big problem because  if we have a reduced amount of RAM, how are we going to be able  to build this nice data structure on this?  We have another problem, which is people might create an index  and they want to add new documents.  We want something that is dynamic.  We want to be able to add documents to an existing index.  That's another problem.  The solution is the same for both of these problems.  Just to refine the idea of why it's difficult to add new  documents in an index like that.  The data structure that I described to you on disk is  extremely compact.  Everything is super compressed.  Everything is laid out on disk one after the other.  So if it's on the bookshelf, it looks like this.  If you're in front of a bookshelf like that and somebody  gives you a book and tells you to put that in the library,  you're going to suffer.  You're going to have to move everything away.  It's a nightmare.  What you want is to have something that looks like that  if you want to update your bookshelf all of the time.  So generally speaking, there is a trade-off between being  dynamic and being compact.  So there is a nice trick that exists in many database  that is called the logarithmic method.  The idea will be we're going to have compact bookshelves like  that, but we are going to have many of them.  So the way it works is the user will tell Tantivi or will  tell Lucine, hey, Tantivi, I will give you a budget of,  let's say, 300 megabytes.  And these 300 megabytes will be used for the dynamic  bookshelf here.  So that's going to be a big bookshelf.  And people will add books into that until the bookshelf is full.  Once the bookshelf is full, we are going to transform it into  a very compact bookshelf.  So I'm going to stop with the major, but basically we  serialize our dynamic data structure into the static data  structure that I described before.  And this piece of very compact index is called a segment.  So that's a word that you probably heard in Lucine,  but that's a segment.  And both in Lucine and in Tantivi, there is a strong  architecture choice.  A segment is basically an independent index.  So you could literally pick a segment and copy it into  another index, and it will run just the same as long as the  schemas are the same.  That's a very interesting property, actually.  So if we do that, we are going to end up with a lot of compact  segments.  And maybe that's not optimal.  If we have a 1 terabyte of data, that would be a lot of  small segments.  And when we do a search, we are going to have to do as many  lookups in the dictionary.  And that's very inefficient, right?  So on the background, we also have a process that merge those  segments together.  And you can actually control, you can define on your own as a  user what is a strategy used to merge these segments.  But it comes with a default strategy called the Long Merge  Policy.  And the heuristic behind it is just we try to merge, by default,  it's going to be eight segments that have about the same size.  And yeah.  So this technique is also really nice because that means that  we can do multi-threaded indexing very, very easily.  So, potentially, I ask you how much data you give as a budget  and also how many indexing threads you want to use.  And then when you open the document, what you are doing is  that you are pinning the document to a document queue.  And in the background, you don't control it, but there is your  given number of indexing threads that are consuming this  document queue and that are populating these dynamic  structures that I was talking about.  And once they reach their capacity, then they are going to  serialize the segment.  The serial segment is the very compact data structure that we  use in the end.  And some merging threads will merge those very  transparently.  Yes.  Thank you.  So there is a bunch of pros and cons to this.  One problem that is very evident when you use Tantilli and  I think it's still true for, to some extent, it's true with  Lucine is that when you add a document, it's not searchable  right away.  So it's quite puzzling for people who are using SQL database.  Because we decided to have our segments to be independent  index, they do not share any dictionary.  And sometimes that makes specific use of search  difficult not to be able to actually merge the result from  the different segment very easily.  Yes.  There is not one dictionary.  I didn't talk about deletes and updates, but those are a  natural nightmare.  It's very complicated.  Yes.  That's pretty much all I have on this slide.  And then the pros will be the index throughput is actually  excellent if you can batch stuff.  If you don't have to commit all of the time, you can  reindex a gigantic amount of data.  So seeing on my laptop, I index Wikipedia in, I  thought, I restocked it for two or three minutes, but that's  the amount of English Wikipedia.  But that's the speed we're talking about.  Wikipedia is like a tiny benchmark in search.  It's not a big dataset.  Yes, so segments are independent index.  So you could decide to actually do your indexing on Hadoop and  just copy the file of the segment in the same place.  And everything will work just the same.  Or you could decide to do a distributed search by sending the  segment and dispatching them.  It's free just copying files.  It's very transparent.  And because we write files and then we never touch them again,  we just write very large files and then they're read only.  There is no problem of flux or anything.  Readers just read the file and nobody is touching them.  So it simplifies a lot of my work.  I have a bunch of slides there.  Let's keep then so that we have a bit of time to ask questions.  So do you guys have questions?  Yeah.  I'll try to say that about Lucille is a little bit of a corner  case for Java.  It's like basically a library which is difficult for Java to handle  because there are lots of low level start going on here.  So one question.  Your project and Rust, do they fit together or did you find  trouble using Rust as beautiful?  I wanted to see the little card.  So the question is Rust and building a search engine,  a good fit, I guess, especially in contrast with Lucille and Java.  So I would say yes.  So the main benefit that I get from Rust, I guess, compared to Java is first I can  access SIMD instruction.  Second, I have a lot of control of, I know when I write code whether I'm going  to get static dispatch or dynamic dispatch.  That's something that is extremely hard in Java.  You never know what happens.  Also all of the things like M-map being a nightmare in Java, it's something  that I don't have any problem about.  I don't know if everyone knows what I'm talking about.  But basically when you're working with Orfit memory in Java, like if you are  mapping a huge amount of data and you have very little data that is on your  heap, your data will be unmapped only at the moment where the object that is  holding the M-map thing is garbage collected.  And garbage collection happens only if your heap is actually kind of full.  I think that's a big problem with Java.  So some JVM give you a non-official Java API to unmap stuff.  Some people just decide to use GNI code to do the M-map.  Well, I never have that kind of problem with Rust.  Everything, I'm working with C or C++.  And when I started this project, I did not know Rust at all.  And I started on TV to actually learn Rust, which is a bit stupid and crazy.  So I already knew C++ and I had done a lot of Java as well.  And after around two weeks, you don't have to believe me, but I was more  productive in Rust and I felt safer writing code in Rust.  I was in both Java and C++.  Can you explain the name and the logo of the words?  So Pantyby is an English word.  I often do that.  Actually, I like choosing English words that nobody knows.  As project names.  They're actually not bad at all for SEO and yeah, people learn a new word.  Everybody is happy, right?  So Pantyby means at full gallop.  Hands off of.  And I drew the horse myself.  I think that's one of the best achievements in the project.  Yes?  Thank you.
A_Deepdive_into_Tantivy_NIL677uIaOs_0_143.png A_Deepdive_into_Tantivy_NIL677uIaOs_144_443.png A_Deepdive_into_Tantivy_NIL677uIaOs_444_1499.png A_Deepdive_into_Tantivy_NIL677uIaOs_1500_8579.png A_Deepdive_into_Tantivy_NIL677uIaOs_8580_9731.png A_Deepdive_into_Tantivy_NIL677uIaOs_9732_14255.png A_Deepdive_into_Tantivy_NIL677uIaOs_14256_14603.png A_Deepdive_into_Tantivy_NIL677uIaOs_14604_16499.png A_Deepdive_into_Tantivy_NIL677uIaOs_16500_21131.png A_Deepdive_into_Tantivy_NIL677uIaOs_21132_21383.png A_Deepdive_into_Tantivy_NIL677uIaOs_21384_21515.png A_Deepdive_into_Tantivy_NIL677uIaOs_21516_21563.png A_Deepdive_into_Tantivy_NIL677uIaOs_21564_21863.png A_Deepdive_into_Tantivy_NIL677uIaOs_21864_22019.png A_Deepdive_into_Tantivy_NIL677uIaOs_22020_22283.png A_Deepdive_into_Tantivy_NIL677uIaOs_22284_22439.png A_Deepdive_into_Tantivy_NIL677uIaOs_22440_22787.png A_Deepdive_into_Tantivy_NIL677uIaOs_22788_23063.png A_Deepdive_into_Tantivy_NIL677uIaOs_23064_23135.png A_Deepdive_into_Tantivy_NIL677uIaOs_23136_23315.png A_Deepdive_into_Tantivy_NIL677uIaOs_23316_23435.png A_Deepdive_into_Tantivy_NIL677uIaOs_23436_23591.png A_Deepdive_into_Tantivy_NIL677uIaOs_23592_23639.png A_Deepdive_into_Tantivy_NIL677uIaOs_23640_27143.png A_Deepdive_into_Tantivy_NIL677uIaOs_27144_32783.png A_Deepdive_into_Tantivy_NIL677uIaOs_32784_34175.png A_Deepdive_into_Tantivy_NIL677uIaOs_34176_34307.png A_Deepdive_into_Tantivy_NIL677uIaOs_34308_34439.png A_Deepdive_into_Tantivy_NIL677uIaOs_34440_34511.png A_Deepdive_into_Tantivy_NIL677uIaOs_34512_36023.png A_Deepdive_into_Tantivy_NIL677uIaOs_36024_36143.png A_Deepdive_into_Tantivy_NIL677uIaOs_36144_36263.png A_Deepdive_into_Tantivy_NIL677uIaOs_36264_36983.png A_Deepdive_into_Tantivy_NIL677uIaOs_36984_38615.png A_Deepdive_into_Tantivy_NIL677uIaOs_38616_40895.png A_Deepdive_into_Tantivy_NIL677uIaOs_40896_42083.png A_Deepdive_into_Tantivy_NIL677uIaOs_42084_44147.png A_Deepdive_into_Tantivy_NIL677uIaOs_44148_45155.png A_Deepdive_into_Tantivy_NIL677uIaOs_45156_45779.png A_Deepdive_into_Tantivy_NIL677uIaOs_45780_45839.png A_Deepdive_into_Tantivy_NIL677uIaOs_45840_46271.png A_Deepdive_into_Tantivy_NIL677uIaOs_46272_48419.png A_Deepdive_into_Tantivy_NIL677uIaOs_48420_48551.png A_Deepdive_into_Tantivy_NIL677uIaOs_48552_48803.png A_Deepdive_into_Tantivy_NIL677uIaOs_48804_51479.png A_Deepdive_into_Tantivy_NIL677uIaOs_51480_52979.png A_Deepdive_into_Tantivy_NIL677uIaOs_52980_54863.png A_Deepdive_into_Tantivy_NIL677uIaOs_54864_57119.png A_Deepdive_into_Tantivy_NIL677uIaOs_57120_57167.png A_Deepdive_into_Tantivy_NIL677uIaOs_57168_57215.png A_Deepdive_into_Tantivy_NIL677uIaOs_57216_62315.png A_Deepdive_into_Tantivy_NIL677uIaOs_62316_62363.png A_Deepdive_into_Tantivy_NIL677uIaOs_62364_62411.png A_Deepdive_into_Tantivy_NIL677uIaOs_62412_62459.png A_Deepdive_into_Tantivy_NIL677uIaOs_62460_62531.png A_Deepdive_into_Tantivy_NIL677uIaOs_62532_63335.png A_Deepdive_into_Tantivy_NIL677uIaOs_63336_63450.png
I'm going to start by a very short call about me. I'm a friend to Rokori in Japan and started this project around three years ago. Right now I'm working as a law and I've been asked specifically to be clear about how this project is not a really complicated system. I've been asked a lot of questions about the way this project is done. What's the purpose of this project? Within this community, 20D is closer to UC. I don't know if everyone is familiar with Rust. This project is very popular, for those who are familiar with Rust, it's a program in English that is kind of young. You can consider Rust as a policy post. The code that you write is basically safe. You get the great ecosystem and great building and the same method that it seems to be learning. What is the developers whose goal iseted so far? So you can apply it on platform that are not x86, 64 bit, so you can actually run it on R, at number four. And I actually even provide a concept for whether something wants. So that's something that I probably don't talk to you all too much because there's not much performance training right now in the web assembly, but at one point, it will be an advantage if you run such an engineer worker. It's still cute, it's very important for me to keep some of our clients quite small, there's only 30,000 minds of all, cool, even if you include all the different crates that I had to write for the concept. It's very important for me because as I told you, it's a pet project and you write for pet pet projects, it's quite small and cute. And also ideally I would love in some future to have companies that are considering between their own such engine parts which they have for one reason or another, if they can actually receive for that project. Rather than starting from scratch and the idea, I'd love them to at least consider using them. And I think that feeling is not the best, it's the best for people who actually embrace the code and own it, get it, own it and work with it. It's fast, so I put a link on the benchmark here, I think we want that then to go through the benchmark, but there's an interesting benchmark that, for example, the performance of Tansili, per voice, it can look like for the shank of the way. Usually it's doing really well, it can only focus on different versions of the NCD server. But what's really interesting is that you can actually identify, oh, for this way, the scene is actually faster, what's going on there. So that's a great way for me to explore different optimizations that are in the scene and do not have to set it in particular. I can know what would be the next step for Pansili. I thought there was a way around for some query like Pansili is abnormally faster than this scene. It's a good thing that there is no transition that's happening. So I suspect that everybody in the room is more interested in this scene than in the search. Some of you are just wondering, what's the best way to run for the next generation? So I'm not gonna look too much about the rest actually, but I find that from now on, we're gonna build a search engine together. And as a matter of fact, what you do is only by using, all of the servers they're gonna talk from now on are very valid for using the screen as a scene design. So hopefully at the end of this talk, you can see everybody wants to understand the scene. Maybe a little bit better. So yeah, there's really a search engine together. Most of the time, our goal will be, we want the search engine that is, I'm trying to come up with a scale here because we are still coding both. The search engine that is sitting in one machine. But we want to be able to search an index, terabytes of data, a lot of data, on machine that is, the only gigabyte is one. So we want to make sure that all of the stuff that we are doing are solid and they don't require you to have quite a gigantic amount of RAM. And so, until you can do that, I actually tested it, that's something that I did. So right now we are at 20.8 and I did test the first case, this one is your.6. And what I did is that, so this is probably a very good data set for its command code. And what it is is kind of a snapshot of the core of the web. And it's a pretty large one, a very decent one. It's around 3 billion pages. So I took this snapshot, I downloaded it with, to be accurate, I downloaded half of it. I downloaded half of it using my own just known internet connection. It's using my internet provider base. It took around one month full time. I mean, Japan, so we have like, optical fiber and stuff, so it is pretty good in the impression that it's good. And I wanted to test if I could index it, see what happens because like, it's a very good test to do that in your home, especially like, I bought the house in Japan and the electrical network was actually great for my usage. I mean, I'm French, so I drink coffee and I have a ghost in the morning and when I would switch ghosts at the same time, I would already have a dog in my home. That's a great test point for me. Thanks, thanks, also. Yeah, so I did that and I thought, why some are really on the process to, and basically the goal of the Python library is to extract, like you give it a query and there is a placeholder. So for instance, you will say, French people are and then you ask for all of the adjectives that follow these centers. And so, two libraries together, one of these is the small NLB library, made it possible to look for the internet, the most common adjectives that was French people. Oh, man. I think it's pretty accurate. Okay, so let's move on. So, to use something that was perfect, so should we use a graph to, we have to use almost the same way. Can you check if you know that, please? I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. Green. That's green. All right. All right. All right, better. Okay, so usually that means that you're going to want to build an inverted index. An inverted index is simply something that associates to each term a list of sorted lists of the queries. And the simple data structure is sufficient to actually compute any kind of Boolean query. So I'm going to explain rapidly how our intersection works. So actually means that we are trying to find all of the documents that are matching the query, run and migrate. We are going to have like this two posting lists. We take the first one. The first document of the first one is two. And we are going to seek into the second posting list for the document two. So by seeking what I mean is that we are going to do exactly as if we are advancing, going through the scanning through the second posting list. And we are going to stop at the moment where we find the document that is greater or equal to the document. If it's equal, then that means that two is in the intersection. If it's greater, then it's not. So here it's not. And we found the document five. And we are going to like do symmetrically. We are going to seek for five in the first posting list. Here we go. Five was found. So it's going to be in our output. We can advance the first posting list. And now we are seeking for seven in the second list and so on so forth until we reach the end of one of the posting lists. So the only takeaway there is that if you want fast intersection, you need to have a fast way to seek through the posting list. For union, we could use the same strategy. Basically, it's like the merge. Everybody has a different name for this algorithm, but we could merge those two posting lists and get the union. But that's actually inefficient. So I'm using the same trick as Lucine does. The idea is that you prepare in advance some kind of bit set here. And so just for the sake of this slide, the bit set is actually on offense eight, but in Don TV, we're using 4,096 bits, so it's much larger. And what we do is that we are going to take the first posting list and we are going to append all of the docs that are between two and two plus. So this horizon, in this case, eight, but in Don TV, 4,096. And we append those documents in this bit set. So in this bit set, the first bit will mean two, the second bit will mean three, and so on and so forth. We do that for the first posting list, and we do that on the second time, on the second posting list. Here's what we get for bit set, and now we can flush it. So we transform basically this bit set into the document that it represented. So it can be done quite fast because your CPU usually has an instruction to pop the lowest significant bit out of a 64-bit world. So you just pop it and so on and forth. All right. So that's how an inverted index is used to compute Polyanquery. Now the question is, how do we represent that on disk? So Don TV is relying on M-Map for all of its I.O. And when you start Don TV, the only thing that it does is basically mapping all of the file of the index, so it goes really, really fast, and it's ready to go. So we don't have to load any data structure and put that in anonymous memory and have some kind of hash map in anonymous memory. Everything is on M-Map. So the startup time is very, very nice. But that means that we need to have data on the disk that is usable as is for Java people. Anonymous memory is more or less like the heap. So the first data structure that we are going to have to you have in our index is the term dictionary. So a term dictionary, maybe our term dictionary will be broken into two steps. One step will associate the terms, like the sequence of bytes associated with the term to some kind of term ID. And then we're going to have another data structure that associates the term ID to some kind of term info structure that basically have pointers to files and the beginning of the pushing this. I'm not going to talk about the second data structure because it's going, but let's talk about the first one. So how do we go from terms, sequence of bytes to term ID? So if you are actually trying to build your own search engine, you will have two broad kind of family for this solution. One could be like hash-based in general. Maybe you will go fancy and have like perfect hash or anything like that. It's a very nice solution in the sense that you will get like very fast lookups and especially if you don't have much RAM and everything is sitting on your hard disk, you will probably require very little I.O. to do that. Your hash map will be able to send you directly in the right place on your disk. Another solution is using like a tree-based solution or like a trial. This will use slightly more CPU and you will have a lot more RAM than I.O. so that kind of depends on the layout of your data, but you tend to jump from one node of your try to another node of your try. So it might be a lot of six if your data is on your disk. But it has a lot of benefits. So one benefit is you can iterate through a round of keys. You can, so naturally you will probably use term ordinals. So by term ordinals, I mean if Arabic is your first word in your dictionary, then it will get term ID zero. The first word when sorted the x-y-cography plane and then the second word might be something starting by a B, then it will get term ordinals B, term ordinals two and then one's fourth. So term ordinals will, your term IDD will be sorted exactly the same as your terms. That's a very nice property. Importantly, you can do an intersection of your try or your try light structure with a DFA. So are you guys familiar with what a DFA is? OK, we'll still explain it a little. I actually made a mistake in the DFA here, but so that's one. So DFA stands for Deterministic Finite Automaton, and that's one way to implement regular expression. So you can transform any regular expression into an automaton like this one. And the way it works is once you have it into this shape, then matching a string on this automaton means that you're going to consume every charge in your string. So you start in the state over there. I was hoping I could point out so that you can jump. So you start in the white state over there, and let's say that you are trying to match carousel. Your first charge is C. You look at the outbound arrows that are emitted by the state you are in. Our one is labeled with a C, so that means that after consuming the C, I'm in the state, so yellow one. And then carousel, second letter is an A. I'm going to follow this letter, this arrow, and I'm in the blue state, R. I end up in this state, and Usel, until the Use, I guess, without the L, will be following the arrow with a star over there, and then the last L will bring me to the end state. It's really nice because I'm advancing one charge at a time. I just have to look at the end state of my string to say whether I matched or not. I mark the state that matched by a double circle here. So our try looks like this, and it's actually possible to match this DFA over the try very efficiently. So I'm going to show how it's done. So here, if we consume the C, we end up in the yellow state, just like for carousel. If I consume the A, I end up in the blue state. If I get a B, then I end up in this gray state, which happens to be a sink. So the sink is a dead end. We will never match if we reach this state. We don't need to see what happens with the following characters. And that's great because that means that all of this subtree, we don't care about it anymore. So for the purpose of fitting soon into the slide, this try is very simple. But you can imagine that maybe there is a gigantic tree that is a child of this node. And we just cut that, so that's much, much faster. And now we want to match M. We don't have to recompute what is the state required for A. We can just look at the state of A before. So that's blue. And we see that we are in the sink state again. And so on and so forth. So we are just putting colors of our try to guess which term in our dictionary are matching our deterministic finite automaton. That means that I can go through all of the terms that match a regular expression. But also I can get very rapidly all of the terms that are at Levenstein's distance or edit distance of one or two. That means that if you are asking me, please stream me all of the terms that are one type away from what I typed, I can do that very, very efficiently. So Tantivi is using a tree-based solution. It's actually using a finite state transducer within those two. And I didn't have to code anything. The Rust ecosystem is nice enough that somebody already coded a very nice implementation of finite state transducer. I will not explain how it works because I didn't code it. But the thing that you might want to know about it is pretty much like a try except that nodes can share suffixes. So a try would never have an error that goes like this. So because it can share suffixes, you end up with something that is slightly more compact than a try. And that's always a very nice feature when you're being spaced with your RAM. Now let's talk a little bit about how we are going to do the same thing. We're going to use one code, posting lists. So posting lists are a lot of integers. It's going to take a lot of the size of your index. And you're going to want to compress them. Integer compression is a field that has been well studied. There is a lot of solutions to do integer compression. I put a chart over there. My point here is you have a trade-off between something that is compressing a lot. So it's always like that. You have to choose with something that compresses a lot and something that is very fast. Another thing that I need to point out is basically all of the algorithm over there and all of these, all of the best algorithms, they are all using CMD instructions. The CMD instructions are instructions on your CPU that makes it possible for you to process four or eight integers at a time. So that's something that actually, Tantivi is using a lot. So Tantivi is actually using this code there. So it's a compression scheme that has been designed by Daniel Lemire. I recommend you to read this blog. It's always very interesting if you're interested in things related with search or data structures. I used to depend on his library actually in C and I removed it because I prefer to be entirely in Rust and I re-implemented it entirely in Rust. The gist of it is like many of those scheme, your posting lists are increasing so it starts by doing something called tell-down coding. So instead of compressing your integer directly, what you do is that you take the difference between two consecutive integers and you get something that is much smaller. So you take a pack of those. So in my case, blocks are 128 integer long and out of those 128 deltas, you look at the one that is the largest. So I think in this example, there should be this one, that's 16. And you notice that it can be represented using five bits. So what you do then is that you represent all of your deltas over five bits and you just concatenate this version. So that's called bit packing. So I describe what was the solution for the scalar version of bit packing. What I do is actually I use a CMD instruction for that and I do that with four integer at a time. And the algorithm really looks like this trick that we were using at school to avoid writing lines when we are punished by our professor. So it's exactly the same algorithm as the scalar solution. You just use CMD instruction in place of those scalar instructions. It's very, very simple. And there is an interesting improvement compared to the methods that existed before Daniel and Mir's paper, which is the algorithm also take advantage of the fact that as we are decoding those integer, we end up at one point where we have one register with the delta that we are decoding and we can decode the delta using CMD instruction as well. So this is also CMD instruction. And we do that at the point where they are already in the registers. It goes really, really fast. And we're talking about, when I say fast to give you an idea, I'm talking about four billion integers a second. So we are already very, very, very, we are flirting with the bandwidth of your run, basically. So I told you that Rust was nice and was able to compile very efficient code that it was very close to C++. So I wrote that in Rust and this is a summary that is generated. I'm not going to tell you that I understand what's in there. But the important point here is that all of the long-looking instruction over there, every single one of those is working on four bytes at a time. And you will see scalar instruction maybe here, here, here. So the assembly code that is generated is really as good as possible. It's very, very close to, it's probably exactly the same as what we had in C++. So I think we talked this morning a little about BM25 and TFI-DFF. This requires to, so those are like the scoring, the default scoring function of Lucine. Tantivi comes in with BM25 also. This requires to have access to the term frequency as we search. So when we match a document, we need to have access to the number of time a term appears in the document. And it's nice to have good locality for that. So we actually interleave blocks of the KDs with block of term frequencies so that we are likely to have it in a nice cache at the moment where we are reading the, as we are doing the match, as we are running the query. So we have one block of the KDs and then one block of term frequency and so on and so forth. Each block is representing 128 blocks. So unfortunately, posting is not always exactly a multiple of 128 blocks. So the last one is using a different compressing scheme. That is not interesting. And so we also need to have the information of how many bits are used to encode each block. And we also would like to have some way to avoid decompressing this block if we are running an intersection, for instance. Maybe the document is not in this block and in that case we would like to entirely avoid decompressing it. So for that, we have another structure on top of that that allows us to precisely skip and decompress only the blocks that might be interesting. We are going to talk about something else now. So I said in the beginning of the talk that we wanted to be able to index an arbitrary large amount of data, which is about terabytes, and that's a big problem because if we have a reduced amount of RAM, how are we going to be able to build this nice data structure on this? We have another problem, which is people might create an index and they want to add new documents. We want something that is dynamic. We want to be able to add documents to an existing index. That's another problem. The solution is the same for both of these problems. Just to refine the idea of why it's difficult to add new documents in an index like that. The data structure that I described to you on disk is extremely compact. Everything is super compressed. Everything is laid out on disk one after the other. So if it's on the bookshelf, it looks like this. If you're in front of a bookshelf like that and somebody gives you a book and tells you to put that in the library, you're going to suffer. You're going to have to move everything away. It's a nightmare. What you want is to have something that looks like that if you want to update your bookshelf all of the time. So generally speaking, there is a trade-off between being dynamic and being compact. So there is a nice trick that exists in many database that is called the logarithmic method. The idea will be we're going to have compact bookshelves like that, but we are going to have many of them. So the way it works is the user will tell Tantivi or will tell Lucine, hey, Tantivi, I will give you a budget of, let's say, 300 megabytes. And these 300 megabytes will be used for the dynamic bookshelf here. So that's going to be a big bookshelf. And people will add books into that until the bookshelf is full. Once the bookshelf is full, we are going to transform it into a very compact bookshelf. So I'm going to stop with the major, but basically we serialize our dynamic data structure into the static data structure that I described before. And this piece of very compact index is called a segment. So that's a word that you probably heard in Lucine, but that's a segment. And both in Lucine and in Tantivi, there is a strong architecture choice. A segment is basically an independent index. So you could literally pick a segment and copy it into another index, and it will run just the same as long as the schemas are the same. That's a very interesting property, actually. So if we do that, we are going to end up with a lot of compact segments. And maybe that's not optimal. If we have a 1 terabyte of data, that would be a lot of small segments. And when we do a search, we are going to have to do as many lookups in the dictionary. And that's very inefficient, right? So on the background, we also have a process that merge those segments together. And you can actually control, you can define on your own as a user what is a strategy used to merge these segments. But it comes with a default strategy called the Long Merge Policy. And the heuristic behind it is just we try to merge, by default, it's going to be eight segments that have about the same size. And yeah. So this technique is also really nice because that means that we can do multi-threaded indexing very, very easily. So, potentially, I ask you how much data you give as a budget and also how many indexing threads you want to use. And then when you open the document, what you are doing is that you are pinning the document to a document queue. And in the background, you don't control it, but there is your given number of indexing threads that are consuming this document queue and that are populating these dynamic structures that I was talking about. And once they reach their capacity, then they are going to serialize the segment. The serial segment is the very compact data structure that we use in the end. And some merging threads will merge those very transparently. Yes. Thank you. So there is a bunch of pros and cons to this. One problem that is very evident when you use Tantilli and I think it's still true for, to some extent, it's true with Lucine is that when you add a document, it's not searchable right away. So it's quite puzzling for people who are using SQL database. Because we decided to have our segments to be independent index, they do not share any dictionary. And sometimes that makes specific use of search difficult not to be able to actually merge the result from the different segment very easily. Yes. There is not one dictionary. I didn't talk about deletes and updates, but those are a natural nightmare. It's very complicated. Yes. That's pretty much all I have on this slide. And then the pros will be the index throughput is actually excellent if you can batch stuff. If you don't have to commit all of the time, you can reindex a gigantic amount of data. So seeing on my laptop, I index Wikipedia in, I thought, I restocked it for two or three minutes, but that's the amount of English Wikipedia. But that's the speed we're talking about. Wikipedia is like a tiny benchmark in search. It's not a big dataset. Yes, so segments are independent index. So you could decide to actually do your indexing on Hadoop and just copy the file of the segment in the same place. And everything will work just the same. Or you could decide to do a distributed search by sending the segment and dispatching them. It's free just copying files. It's very transparent. And because we write files and then we never touch them again, we just write very large files and then they're read only. There is no problem of flux or anything. Readers just read the file and nobody is touching them. So it simplifies a lot of my work. I have a bunch of slides there. Let's keep then so that we have a bit of time to ask questions. So do you guys have questions? Yeah. I'll try to say that about Lucille is a little bit of a corner case for Java. It's like basically a library which is difficult for Java to handle because there are lots of low level start going on here. So one question. Your project and Rust, do they fit together or did you find trouble using Rust as beautiful? I wanted to see the little card. So the question is Rust and building a search engine, a good fit, I guess, especially in contrast with Lucille and Java. So I would say yes. So the main benefit that I get from Rust, I guess, compared to Java is first I can access SIMD instruction. Second, I have a lot of control of, I know when I write code whether I'm going to get static dispatch or dynamic dispatch. That's something that is extremely hard in Java. You never know what happens. Also all of the things like M-map being a nightmare in Java, it's something that I don't have any problem about. I don't know if everyone knows what I'm talking about. But basically when you're working with Orfit memory in Java, like if you are mapping a huge amount of data and you have very little data that is on your heap, your data will be unmapped only at the moment where the object that is holding the M-map thing is garbage collected. And garbage collection happens only if your heap is actually kind of full. I think that's a big problem with Java. So some JVM give you a non-official Java API to unmap stuff. Some people just decide to use GNI code to do the M-map. Well, I never have that kind of problem with Rust. Everything, I'm working with C or C++. And when I started this project, I did not know Rust at all. And I started on TV to actually learn Rust, which is a bit stupid and crazy. So I already knew C++ and I had done a lot of Java as well. And after around two weeks, you don't have to believe me, but I was more productive in Rust and I felt safer writing code in Rust. I was in both Java and C++. Can you explain the name and the logo of the words? So Pantyby is an English word. I often do that. Actually, I like choosing English words that nobody knows. As project names. They're actually not bad at all for SEO and yeah, people learn a new word. Everybody is happy, right? So Pantyby means at full gallop. Hands off of. And I drew the horse myself. I think that's one of the best achievements in the project. Yes? Thank you.
there's really a search engine together. Most of advancing, going through the scanning through the for this solution. One could be like hash-based the blocks that might be interesting. We are in the beginning of the talk that we wanted to be the question is Rust and building a search that is on your heap, your data will be unmapped