Playing with casync @ instagram
Alvaro Leiva Geisse
In Instagram, we have been experimenting with casync as an alternative package format for deployment of the site. This talks describe our findings Last year when Lennart presented in ASG about casync, we were excited to check it out. So we checked what was necessary to deploy parts of our sites with it. and we spend some times experimenting... This talk show our results, so far...
 Hi. Hello. So my name is Elvaro Leyva. I'm a production engineer at Facebook and Instagram and I'm the only thing standing between you and lunch. So let's get this over. So basically my talk is about how we saw Lennar stock last year about TA Think and we saw that it was a really cool project. So we start finding out what can we do with it and we found this problem. So we try to solve it with TA Think. So basically I will start saying, like, why did we want to experiment with this? How was our strategy? And then our results, yeah. Okay, cool. So as a race of hand, can the system that you work on, can anybody raise their hand if that system is deployed once a week? Okay, cool. Once a day? Cool. Okay. Twice a day. 10 times a day. 20. Yeah. So the reason why I say this is because at Instagram we deploy, since two years ago, we deployed more than 50 times a day. Basically what we try to do is deploy each single commit that a developer sends to master. We try to deploy that directly into production, have it enough time in production so it gives a signal if that commit will break or not break and then move to the next one. This works really good for us because it allows us to find things that will break or security or vulnerabilities and stuff like that really quick. So the way that this works, it's simple. A developer commits its code. We pack it into our own internal tooling. We run tests into it and then we send it into production. And this really works really well because it allows us to chip small changes. A developer, after he lands his code, it's an hour until it's in production. So he's around. If he breaks production, he can help us fix it. And it's really easy to roll back to the previous date. Cool. So basically how we do this? You have to imagine that this is version A and this is kind of a representation of what would be our source tree. For those who don't know, Instagram is mainly at Django shop. So that's Python. Most of these are just plain text files. So we basically have this package that is a representation. We strip things that we don't want. We compile a few things that are C. We convert Python to byte code and then we package into our format. Then a developer comes, make a commit, make a small change, and then we do the same process and we end up with a package that is really similar to A but has all the components. And then we also have C that probably also changes different things. And you can see that if we do this a lot of times a day, the sum of A, B, and C gets really big. So this was like a really interesting problem to solve with C async. And I will explain a little bit how we view C async. That is like for reasons of gravity and for what the abstraction of our problem is, maybe oversimplified. But, okay. So the way that we work is that we take this version A and then what do you think does it will take and divide it in little chunks of data. The magic of what you think is that these chunks are variable length. So that means that this piece over here can weight like 10K, but this one can weight 20K and stuff like that. So we take those packages and then see I think we'll output two things. The chunks, there are these files and an index file. That index file is basically a recipe on how we're going to take these chunks and then reconstruct them into creating the directory. So this is the service session. And then the opposite process is that you take your index file, then you grab whatever chunks it says that you have, you assemble them in the right order, and then you have your package back. The cool thing about this is that if we now have a package V that has a small change of it, the service session result will be really similar between A and B. And maybe we'll have one extra or two extra chunks there. And we'll yield a different index file. So now we don't have to think in terms of versions. We just sync or we just have all these, oh, hello. We just have all these stores, all these chunks storing in a single location. And then what we distribute, it's the index file. And the index file is what it will become our version. So that is basically how we use CAsync. So it's really simple. The way that we work on this is that we put an intern who was really good at his job. And we ask him to come with abstraction and stuff like that. First of all, we wanted to create an abstract definition of package that will englobe this idea of having stores and having index, but also will not be subjected to just file systems and maybe instead of syncing the index file as a file in directory, we want to sync it as a database record because this blends to be a key value stuff. So the first thing that we did is that we changed the idea of index to an idea of manifest. And the reason is really simple. What CAsync gives you as an index file, it's basically a recipe to reconstruct your package. But it doesn't give you any information of how the package was constructed. Who constructed, where it constructed, did it use certain compilers, what version, what hash of the Mercurial repository. And then we put it into stores. So this is really important for us. It's like the information about the package is almost as important as the package itself. So we created this tool called CAPackage to make an abstraction over CAsync. And the way that you will get the index file is something like that. You give CAPackage stage and then it gives you a URI. That URI in this case is an SQL query, so we can store all the things into SQL. And this will be our manifest. Really here, the data, it's an encoded version of what you will get into index file. And then you see that we put all the other information that we look here about. Stuff like, for instance, what is the package name? And Instagram, of course, we deploy our Instagram package. But how about if you want to deploy virtual environments? Or we want to deploy other binaries like this? Not just file system, but simple binaries. We have version. Like the same package, we built it multiple times. So it will be really good to have versions. All the things. Finally, we basically have a store adapter that we can, if we want to save a Retriever store from an HTTP server, we can, if you want to store it from local disk, we can also do torrent. So that is basically how we did it. So let's see a few experiments that we did to see how was the results. And then we will be done. So the first thing that we did is we work on the creation. That's why I put a little cookie there, because it's like, how do we create the package? The first experiment that we did, we took a hundred versions of Instagram and we created in our regular format, and then we created with CA package that is basically an abstraction over Csync. The first thing that we did find out is that we save about 90% of space. And this is kind of obvious, because this is the idea of Csync at the end of the day. And our regular model, basically each version is a full package container of the things. While when you do it with Csync, you are just including the new stuff or whatever extra it is. And in terms of resources and time, creating the things took about the same time. And it makes sense, because we use the same technologies, we serialize the same way, we compress and uncompress using the same libraries. So basically the big win was in space. You can see if you deploy more than 50 times a day, by the end of the day, you are saving a lot of network, because you are not keeping all the components, you are saving in space, and eventually reconstructed will be faster. So the second part of this experiment was to actually get these packages and put it into production in the same way that we deploy our normal system. So in parallel, we basically had a few, a handful of machines that when a commit came and landed into master, we created the C package and then we chip it into production. We measure the total bandwidth, we measure the time it took to state, and the resource usage. The total download was, again, like 90% safe, makes sense. We are basically downloading the stuff, because moving from version A to B, it's really pain-free. But the cool thing is that moving from A to C without going through B, it's also really pain-free. The state's time was faster, and the resource usage was basically the same again, because we used the same technologies. So that concludes basically what we did. What's next? We want to try on binary heavy distributions. So again, we say Python application is basically mostly text files, but we want to try it with virtual environments that basically has a lot of binary things. We want to stop shipping chunks through HTTP and start using Torrent, because if you have a big infrastructure, you can leverage the fact that most of your machines already have them, the chunks. We want to, right now, we just shell out and execute it. We would really like to start using CACync as a library instead of just chilling out to it. And finally, we want to also try other toolings different than CACync, because the AVA is really cool and we would like to stress it against other things in the market. So basically that's it. I finished. Yeah, don't worry. The question there will be the questions. And yeah, also, if we run out of time, I'm going to be here, and I have stickers if people want. So the question is, just the last point that you raised, that you want to stop chilling out. Do you want to turn CACync into a library, or do you want to re-implement the code? So CACync, again, I don't want to overstep my boundaries here, but CACync works, or it's written in a way that it resembles a lot like a library. It's just it's in version two, so I don't know if the API is going to be stable or not. And basically what we want is to take that same thing and put it into a Python binding. Go ahead. Does it work? Yeah. So my intention was always that it was supposed to be a library, and that's why it's written in a library style, but I haven't come around to make it a library yet. My other question was actually just that, what's the size of the images? The things that we produce? Yeah, the stuff that you actually store, what's the average size? You mean the full size? Okay, I don't know if I can say that the size of my... Just a graph address? Okay, so it's like, I don't know, I would say like 50 megabytes to 200 megabytes depends on the size. These are like text files, so you will basically, even if it's really big when you compress it, it goes really string. So you create a lot of packages per day. What do you do with all chunks after a while? Do you garbage collect them or do you keep everything all the time? So the cool thing about that is that since we deploy every commit, by the time that we deploy, I'm going to say a number, commit number 150, there's no point of going back to anything before that because we know that we are in a good place, right? So we perch all chunks all the time. And the way that you do that is that you have a list of all the index that you... I mean, all the chunks that compose your whatever you want to keep in the back, and then you just serialize it and find all the chunks that doesn't belong to that list. But the more because you have a linear history that you get, because if you have different branches that you... Oh, yeah, yeah, yeah. But you cannot deploy this speed of having different branches. You always deploy from master. Yes. I would like to ask if something like that can be achieved using Git annex or Git LFS and if not, what are the advantages of using CI I think? So that's what we are going to discover in the next step. We're going to start using other tooling. The good thing that I really like about CI I think is that is general purpose. It's not the best on particular, like this particular problem, but it's really general purpose so we can apply the same techniques and put it into binary distribution instead of just text. Or maybe we want to tip like an entire file system with this. And it works really good while Git and all the things tends to be like more single into problem. But I cannot say that for sure because we haven't tried yet. So one more question. Go ahead. He was trying to make a question. So you have an incremental way to ship packages. Do you also have an incremental way to build if there's only small changes every time? So let me see if I can kind of explain. Not really because when you do A and B, you have to think that this, don't think of this as incremental. That's kind of the first thing that I tried to get out of my mind. Think of this like I build A, I build B, and it happens to have components of A and components of B are similar. But this could be like two different applications. They don't have to be. So don't think of this as incremental of A to B. With that in mind, you still need to serialize your whole directory and compare the chunks and then realize which one you actually have to build. But when you are on that point, then you already wasted like 90% of your time on just doing this realization and doing this realization. I'm going to stay here. So if people want to ask me questions, you can do it after this. Thanks a lot. Thank you.
 Hi.  Hello.  So my name is Elvaro Leyva.  I'm a production engineer at Facebook and Instagram and I'm the only thing standing between you  and lunch.  So let's get this over.  So basically my talk is about how we saw Lennar stock last year about TA Think and we saw  that it was a really cool project.  So we start finding out what can we do with it and we found this problem.  So we try to solve it with TA Think.  So basically I will start saying, like, why did we want to experiment with this?  How was our strategy?  And then our results, yeah.  Okay, cool.  So as a race of hand, can the system that you work on, can anybody raise their hand  if that system is deployed once a week?  Okay, cool.  Once a day?  Cool.  Okay.  Twice a day.  10 times a day.  20.  Yeah.  So the reason why I say this is because at Instagram we deploy, since two years ago,  we deployed more than 50 times a day.  Basically what we try to do is deploy each single commit that a developer sends to master.  We try to deploy that directly into production, have it enough time in production so it gives  a signal if that commit will break or not break and then move to the next one.  This works really good for us because it allows us to find things that will break or security  or vulnerabilities and stuff like that really quick.  So the way that this works, it's simple.  A developer commits its code.  We pack it into our own internal tooling.  We run tests into it and then we send it into production.  And this really works really well because it allows us to chip small changes.  A developer, after he lands his code, it's an hour until it's in production.  So he's around.  If he breaks production, he can help us fix it.  And it's really easy to roll back to the previous date.  Cool.  So basically how we do this?  You have to imagine that this is version A and this is kind of a representation of what  would be our source tree.  For those who don't know, Instagram is mainly at Django shop.  So that's Python.  Most of these are just plain text files.  So we basically have this package that is a representation.  We strip things that we don't want.  We compile a few things that are C. We convert Python to byte code and then we package into  our format.  Then a developer comes, make a commit, make a small change, and then we do the same process  and we end up with a package that is really similar to A but has all the components.  And then we also have C that probably also changes different things.  And you can see that if we do this a lot of times a day, the sum of A, B, and C gets really  big.  So this was like a really interesting problem to solve with C async.  And I will explain a little bit how we view C async.  That is like for reasons of gravity and for what the abstraction of our problem is, maybe  oversimplified.  But, okay.  So the way that we work is that we take this version A and then what do you think does  it will take and divide it in little chunks of data.  The magic of what you think is that these chunks are variable length.  So that means that this piece over here can weight like 10K, but this one can weight 20K  and stuff like that.  So we take those packages and then see I think we'll output two things.  The chunks, there are these files and an index file.  That index file is basically a recipe on how we're going to take these chunks and then  reconstruct them into creating the directory.  So this is the service session.  And then the opposite process is that you take your index file, then you grab whatever  chunks it says that you have, you assemble them in the right order, and then you have  your package back.  The cool thing about this is that if we now have a package V that has a small change of  it, the service session result will be really similar between A and B. And maybe we'll have  one extra or two extra chunks there.  And we'll yield a different index file.  So now we don't have to think in terms of versions.  We just sync or we just have all these, oh, hello.  We just have all these stores, all these chunks storing in a single location.  And then what we distribute, it's the index file.  And the index file is what it will become our version.  So that is basically how we use CAsync.  So it's really simple.  The way that we work on this is that we put an intern who was really good at his job.  And we ask him to come with abstraction and stuff like that.  First of all, we wanted to create an abstract definition of package that will englobe this  idea of having stores and having index, but also will not be subjected to just file systems  and maybe instead of syncing the index file as a file in directory, we want to sync it  as a database record because this blends to be a key value stuff.  So the first thing that we did is that we changed the idea of index to an idea of manifest.  And the reason is really simple.  What CAsync gives you as an index file, it's basically a recipe to reconstruct your package.  But it doesn't give you any information of how the package was constructed.  Who constructed, where it constructed, did it use certain compilers, what version, what  hash of the Mercurial repository.  And then we put it into stores.  So this is really important for us.  It's like the information about the package is almost as important as the package itself.  So we created this tool called CAPackage to make an abstraction over CAsync.  And the way that you will get the index file is something like that.  You give CAPackage stage and then it gives you a URI.  That URI in this case is an SQL query, so we can store all the things into SQL.  And this will be our manifest.  Really here, the data, it's an encoded version of what you will get into index file.  And then you see that we put all the other information that we look here about.  Stuff like, for instance, what is the package name?  And Instagram, of course, we deploy our Instagram package.  But how about if you want to deploy virtual environments?  Or we want to deploy other binaries like this?  Not just file system, but simple binaries.  We have version.  Like the same package, we built it multiple times.  So it will be really good to have versions.  All the things.  Finally, we basically have a store adapter that we can, if we want to save a Retriever  store from an HTTP server, we can, if you want to store it from local disk, we can also  do torrent.  So that is basically how we did it.  So let's see a few experiments that we did to see how was the results.  And then we will be done.  So the first thing that we did is we work on the creation.  That's why I put a little cookie there, because it's like, how do we create the package?  The first experiment that we did, we took a hundred versions of Instagram and we created  in our regular format, and then we created with CA package that is basically an abstraction  over Csync.  The first thing that we did find out is that we save about 90% of space.  And this is kind of obvious, because this is the idea of Csync at the end of the day.  And our regular model, basically each version is a full package container of the things.  While when you do it with Csync, you are just including the new stuff or whatever extra  it is.  And in terms of resources and time, creating the things took about the same time.  And it makes sense, because we use the same technologies, we serialize the same way, we  compress and uncompress using the same libraries.  So basically the big win was in space.  You can see if you deploy more than 50 times a day, by the end of the day, you are saving  a lot of network, because you are not keeping all the components, you are saving in space,  and eventually reconstructed will be faster.  So the second part of this experiment was to actually get these packages and put it into  production in the same way that we deploy our normal system.  So in parallel, we basically had a few, a handful of machines that when a commit came  and landed into master, we created the C package and then we chip it into production.  We measure the total bandwidth, we measure the time it took to state, and the resource  usage.  The total download was, again, like 90% safe, makes sense.  We are basically downloading the stuff, because moving from version A to B, it's really pain-free.  But the cool thing is that moving from A to C without going through B, it's also really  pain-free.  The state's time was faster, and the resource usage was basically the same again, because  we used the same technologies.  So that concludes basically what we did.  What's next?  We want to try on binary heavy distributions.  So again, we say Python application is basically mostly text files, but we want to try it with  virtual environments that basically has a lot of binary things.  We want to stop shipping chunks through HTTP and start using Torrent, because if you have  a big infrastructure, you can leverage the fact that most of your machines already have  them, the chunks.  We want to, right now, we just shell out and execute it.  We would really like to start using CACync as a library instead of just chilling out  to it.  And finally, we want to also try other toolings different than CACync, because the AVA is really  cool and we would like to stress it against other things in the market.  So basically that's it.  I finished.  Yeah, don't worry.  The question there will be the questions.  And yeah, also, if we run out of time, I'm going to be here, and I have stickers if people  want.  So the question is, just the last point that you raised, that you want to stop chilling  out.  Do you want to turn CACync into a library, or do you want to re-implement the code?  So CACync, again, I don't want to overstep my boundaries here, but CACync works, or it's  written in a way that it resembles a lot like a library.  It's just it's in version two, so I don't know if the API is going to be stable or not.  And basically what we want is to take that same thing and put it into a Python binding.  Go ahead.  Does it work?  Yeah.  So my intention was always that it was supposed to be a library, and that's why it's written  in a library style, but I haven't come around to make it a library yet.  My other question was actually just that, what's the size of the images?  The things that we produce?  Yeah, the stuff that you actually store, what's the average size?  You mean the full size?  Okay, I don't know if I can say that the size of my...  Just a graph address?  Okay, so it's like, I don't know, I would say like 50 megabytes to 200 megabytes depends  on the size.  These are like text files, so you will basically, even if it's really big when you compress  it, it goes really string.  So you create a lot of packages per day.  What do you do with all chunks after a while?  Do you garbage collect them or do you keep everything all the time?  So the cool thing about that is that since we deploy every commit, by the time that  we deploy, I'm going to say a number, commit number 150, there's no point of going back  to anything before that because we know that we are in a good place, right?  So we perch all chunks all the time.  And the way that you do that is that you have a list of all the index that you...  I mean, all the chunks that compose your whatever you want to keep in the back, and  then you just serialize it and find all the chunks that doesn't belong to that list.  But the more because you have a linear history that you get, because if you have different  branches that you...  Oh, yeah, yeah, yeah.  But you cannot deploy this speed of having different branches.  You always deploy from master.  Yes.  I would like to ask if something like that can be achieved using Git annex or Git LFS  and if not, what are the advantages of using CI I think?  So that's what we are going to discover in the next step.  We're going to start using other tooling.  The good thing that I really like about CI I think is that is general purpose.  It's not the best on particular, like this particular problem, but it's really general  purpose so we can apply the same techniques and put it into binary distribution instead  of just text.  Or maybe we want to tip like an entire file system with this.  And it works really good while Git and all the things tends to be like more single into  problem.  But I cannot say that for sure because we haven't tried yet.  So one more question.  Go ahead.  He was trying to make a question.  So you have an incremental way to ship packages.  Do you also have an incremental way to build if there's only small changes every time?  So let me see if I can kind of explain.  Not really because when you do A and B, you have to think that this, don't think of this  as incremental.  That's kind of the first thing that I tried to get out of my mind.  Think of this like I build A, I build B, and it happens to have components of A and components  of B are similar.  But this could be like two different applications.  They don't have to be.  So don't think of this as incremental of A to B. With that in mind, you still need to  serialize your whole directory and compare the chunks and then realize which one you  actually have to build.  But when you are on that point, then you already wasted like 90% of your time on just doing  this realization and doing this realization.  I'm going to stay here.  So if people want to ask me questions, you can do it after this.  Thanks a lot.  Thank you.
Playing_with_casync___instagram_9o5mFTE6KJc_0_155.png Playing_with_casync___instagram_9o5mFTE6KJc_156_287.png Playing_with_casync___instagram_9o5mFTE6KJc_288_407.png Playing_with_casync___instagram_9o5mFTE6KJc_408_455.png Playing_with_casync___instagram_9o5mFTE6KJc_456_659.png Playing_with_casync___instagram_9o5mFTE6KJc_660_803.png Playing_with_casync___instagram_9o5mFTE6KJc_804_3407.png Playing_with_casync___instagram_9o5mFTE6KJc_3408_6383.png Playing_with_casync___instagram_9o5mFTE6KJc_6384_7391.png Playing_with_casync___instagram_9o5mFTE6KJc_7392_7475.png Playing_with_casync___instagram_9o5mFTE6KJc_7476_9227.png Playing_with_casync___instagram_9o5mFTE6KJc_9228_11315.png Playing_with_casync___instagram_9o5mFTE6KJc_11316_11423.png Playing_with_casync___instagram_9o5mFTE6KJc_11424_12959.png Playing_with_casync___instagram_9o5mFTE6KJc_12960_16043.png Playing_with_casync___instagram_9o5mFTE6KJc_16044_16847.png Playing_with_casync___instagram_9o5mFTE6KJc_16848_17339.png Playing_with_casync___instagram_9o5mFTE6KJc_17340_17495.png Playing_with_casync___instagram_9o5mFTE6KJc_17496_18635.png Playing_with_casync___instagram_9o5mFTE6KJc_18636_19931.png Playing_with_casync___instagram_9o5mFTE6KJc_19932_21383.png Playing_with_casync___instagram_9o5mFTE6KJc_21384_21467.png Playing_with_casync___instagram_9o5mFTE6KJc_21468_22787.png Playing_with_casync___instagram_9o5mFTE6KJc_22788_22928.png
Hi. Hello. So my name is Elvaro Leyva. I'm a production engineer at Facebook and Instagram and I'm the only thing standing between you and lunch. So let's get this over. So basically my talk is about how we saw Lennar stock last year about TA Think and we saw that it was a really cool project. So we start finding out what can we do with it and we found this problem. So we try to solve it with TA Think. So basically I will start saying, like, why did we want to experiment with this? How was our strategy? And then our results, yeah. Okay, cool. So as a race of hand, can the system that you work on, can anybody raise their hand if that system is deployed once a week? Okay, cool. Once a day? Cool. Okay. Twice a day. 10 times a day. 20. Yeah. So the reason why I say this is because at Instagram we deploy, since two years ago, we deployed more than 50 times a day. Basically what we try to do is deploy each single commit that a developer sends to master. We try to deploy that directly into production, have it enough time in production so it gives a signal if that commit will break or not break and then move to the next one. This works really good for us because it allows us to find things that will break or security or vulnerabilities and stuff like that really quick. So the way that this works, it's simple. A developer commits its code. We pack it into our own internal tooling. We run tests into it and then we send it into production. And this really works really well because it allows us to chip small changes. A developer, after he lands his code, it's an hour until it's in production. So he's around. If he breaks production, he can help us fix it. And it's really easy to roll back to the previous date. Cool. So basically how we do this? You have to imagine that this is version A and this is kind of a representation of what would be our source tree. For those who don't know, Instagram is mainly at Django shop. So that's Python. Most of these are just plain text files. So we basically have this package that is a representation. We strip things that we don't want. We compile a few things that are C. We convert Python to byte code and then we package into our format. Then a developer comes, make a commit, make a small change, and then we do the same process and we end up with a package that is really similar to A but has all the components. And then we also have C that probably also changes different things. And you can see that if we do this a lot of times a day, the sum of A, B, and C gets really big. So this was like a really interesting problem to solve with C async. And I will explain a little bit how we view C async. That is like for reasons of gravity and for what the abstraction of our problem is, maybe oversimplified. But, okay. So the way that we work is that we take this version A and then what do you think does it will take and divide it in little chunks of data. The magic of what you think is that these chunks are variable length. So that means that this piece over here can weight like 10K, but this one can weight 20K and stuff like that. So we take those packages and then see I think we'll output two things. The chunks, there are these files and an index file. That index file is basically a recipe on how we're going to take these chunks and then reconstruct them into creating the directory. So this is the service session. And then the opposite process is that you take your index file, then you grab whatever chunks it says that you have, you assemble them in the right order, and then you have your package back. The cool thing about this is that if we now have a package V that has a small change of it, the service session result will be really similar between A and B. And maybe we'll have one extra or two extra chunks there. And we'll yield a different index file. So now we don't have to think in terms of versions. We just sync or we just have all these, oh, hello. We just have all these stores, all these chunks storing in a single location. And then what we distribute, it's the index file. And the index file is what it will become our version. So that is basically how we use CAsync. So it's really simple. The way that we work on this is that we put an intern who was really good at his job. And we ask him to come with abstraction and stuff like that. First of all, we wanted to create an abstract definition of package that will englobe this idea of having stores and having index, but also will not be subjected to just file systems and maybe instead of syncing the index file as a file in directory, we want to sync it as a database record because this blends to be a key value stuff. So the first thing that we did is that we changed the idea of index to an idea of manifest. And the reason is really simple. What CAsync gives you as an index file, it's basically a recipe to reconstruct your package. But it doesn't give you any information of how the package was constructed. Who constructed, where it constructed, did it use certain compilers, what version, what hash of the Mercurial repository. And then we put it into stores. So this is really important for us. It's like the information about the package is almost as important as the package itself. So we created this tool called CAPackage to make an abstraction over CAsync. And the way that you will get the index file is something like that. You give CAPackage stage and then it gives you a URI. That URI in this case is an SQL query, so we can store all the things into SQL. And this will be our manifest. Really here, the data, it's an encoded version of what you will get into index file. And then you see that we put all the other information that we look here about. Stuff like, for instance, what is the package name? And Instagram, of course, we deploy our Instagram package. But how about if you want to deploy virtual environments? Or we want to deploy other binaries like this? Not just file system, but simple binaries. We have version. Like the same package, we built it multiple times. So it will be really good to have versions. All the things. Finally, we basically have a store adapter that we can, if we want to save a Retriever store from an HTTP server, we can, if you want to store it from local disk, we can also do torrent. So that is basically how we did it. So let's see a few experiments that we did to see how was the results. And then we will be done. So the first thing that we did is we work on the creation. That's why I put a little cookie there, because it's like, how do we create the package? The first experiment that we did, we took a hundred versions of Instagram and we created in our regular format, and then we created with CA package that is basically an abstraction over Csync. The first thing that we did find out is that we save about 90% of space. And this is kind of obvious, because this is the idea of Csync at the end of the day. And our regular model, basically each version is a full package container of the things. While when you do it with Csync, you are just including the new stuff or whatever extra it is. And in terms of resources and time, creating the things took about the same time. And it makes sense, because we use the same technologies, we serialize the same way, we compress and uncompress using the same libraries. So basically the big win was in space. You can see if you deploy more than 50 times a day, by the end of the day, you are saving a lot of network, because you are not keeping all the components, you are saving in space, and eventually reconstructed will be faster. So the second part of this experiment was to actually get these packages and put it into production in the same way that we deploy our normal system. So in parallel, we basically had a few, a handful of machines that when a commit came and landed into master, we created the C package and then we chip it into production. We measure the total bandwidth, we measure the time it took to state, and the resource usage. The total download was, again, like 90% safe, makes sense. We are basically downloading the stuff, because moving from version A to B, it's really pain-free. But the cool thing is that moving from A to C without going through B, it's also really pain-free. The state's time was faster, and the resource usage was basically the same again, because we used the same technologies. So that concludes basically what we did. What's next? We want to try on binary heavy distributions. So again, we say Python application is basically mostly text files, but we want to try it with virtual environments that basically has a lot of binary things. We want to stop shipping chunks through HTTP and start using Torrent, because if you have a big infrastructure, you can leverage the fact that most of your machines already have them, the chunks. We want to, right now, we just shell out and execute it. We would really like to start using CACync as a library instead of just chilling out to it. And finally, we want to also try other toolings different than CACync, because the AVA is really cool and we would like to stress it against other things in the market. So basically that's it. I finished. Yeah, don't worry. The question there will be the questions. And yeah, also, if we run out of time, I'm going to be here, and I have stickers if people want. So the question is, just the last point that you raised, that you want to stop chilling out. Do you want to turn CACync into a library, or do you want to re-implement the code? So CACync, again, I don't want to overstep my boundaries here, but CACync works, or it's written in a way that it resembles a lot like a library. It's just it's in version two, so I don't know if the API is going to be stable or not. And basically what we want is to take that same thing and put it into a Python binding. Go ahead. Does it work? Yeah. So my intention was always that it was supposed to be a library, and that's why it's written in a library style, but I haven't come around to make it a library yet. My other question was actually just that, what's the size of the images? The things that we produce? Yeah, the stuff that you actually store, what's the average size? You mean the full size? Okay, I don't know if I can say that the size of my... Just a graph address? Okay, so it's like, I don't know, I would say like 50 megabytes to 200 megabytes depends on the size. These are like text files, so you will basically, even if it's really big when you compress it, it goes really string. So you create a lot of packages per day. What do you do with all chunks after a while? Do you garbage collect them or do you keep everything all the time? So the cool thing about that is that since we deploy every commit, by the time that we deploy, I'm going to say a number, commit number 150, there's no point of going back to anything before that because we know that we are in a good place, right? So we perch all chunks all the time. And the way that you do that is that you have a list of all the index that you... I mean, all the chunks that compose your whatever you want to keep in the back, and then you just serialize it and find all the chunks that doesn't belong to that list. But the more because you have a linear history that you get, because if you have different branches that you... Oh, yeah, yeah, yeah. But you cannot deploy this speed of having different branches. You always deploy from master. Yes. I would like to ask if something like that can be achieved using Git annex or Git LFS and if not, what are the advantages of using CI I think? So that's what we are going to discover in the next step. We're going to start using other tooling. The good thing that I really like about CI I think is that is general purpose. It's not the best on particular, like this particular problem, but it's really general purpose so we can apply the same techniques and put it into binary distribution instead of just text. Or maybe we want to tip like an entire file system with this. And it works really good while Git and all the things tends to be like more single into problem. But I cannot say that for sure because we haven't tried yet. So one more question. Go ahead. He was trying to make a question. So you have an incremental way to ship packages. Do you also have an incremental way to build if there's only small changes every time? So let me see if I can kind of explain. Not really because when you do A and B, you have to think that this, don't think of this as incremental. That's kind of the first thing that I tried to get out of my mind. Think of this like I build A, I build B, and it happens to have components of A and components of B are similar. But this could be like two different applications. They don't have to be. So don't think of this as incremental of A to B. With that in mind, you still need to serialize your whole directory and compare the chunks and then realize which one you actually have to build. But when you are on that point, then you already wasted like 90% of your time on just doing this realization and doing this realization. I'm going to stay here. So if people want to ask me questions, you can do it after this. Thanks a lot. Thank you.
talk is about how we saw Lennar stock last year do with it and we found this problem. So we try production, have it enough time in production so really simple. What CAsync gives you as an index of how the package was constructed. Who for instance, what is the package name? And Instagram, of course, we deploy our Instagram package. But how about if you want to deploy container of the things. While when you do it