How to shard MariaDB like a Pro?
Tezuysal, Alkin
In this talk, I'd like to give brief information about how to shard your data under MariaDB topologies and the possibility of using frameworks such as Vitess. While discussing the pros and cons of sharding I would like to showcase how structured horizontal sharding can scale your database almost infinitely. The audience will benefit from how others sharding to scale unlimited under both cloud and Kubernetes realm. In this short talk, I'd like to demo a case study that can be an example to get started for many. Vitess is a database clustering system for horizontal scaling of MySQL through sharding. By enabling shard-routing logic, Vitess allows application code and database queries to remain agnostic to the distribution of data onto multiple shards. With Vitess, you can split, merge, and migrate shards as your needs grow. With its compatibility to development frameworks and integration to open-source tools, Vitess has been a core component of several high traffic OLTP sites around the world and serving data across different platforms. Vitess is also the first Linux Foundation/Cloud Native Computing Foundation graduated open-source database project.
 Welcome. My name is Alkin and I'd like to share some experiences with Vitesse. A little bit about myself. I am a senior technical manager at Planet Scale. I am one of the maintainers for the Vitesse project. I'm an open source database evangelist, previously working at Paracord at Pythion and some of the background and enterprise. I'm also an avid sailor. If you want to talk about sailing, please do find me on Instagram or Twitter. A little bit about Planet Scale because of the relations between the Vitesse project and the company as founded in 2018 by the co-creators of Vitesse at YouTube, later on Google, currently about 45 to 50 employees located in California and it's a complete remote team due to the pandemic. On the Vitesse project, what is Vitesse question can be answered by its database clustering system for horizontal scaling of MySQL or MariaDB databases. Why we say that? It's not a database, but it's a framework that actually manages the databases that exists. So we actually take advantage of the boat's technologies on the framework and the database back end site. Vitesse does not make any changes to the existing running databases like MySQL or MariaDB. It's a CNCF-graduated project. In fact, it is one of the first database projects in CNCF, very important. That's an open source and with an Apache tool that all licensed and it has contributors from around the community with the large users of Vitesse. Today's agenda is going to be an architectural overview of Vitesse. I'm going to talk about what Vitesse is. So you have an idea and Vitesse use cases, a little bit on sharpening and where we can actually meet MariaDB as the MariaDB Dev Room. Thank you for inviting me over. On the Vitesse architecture basics, we actually enable the database infrastructure operations to be more smooth and coming from single land. The glossary for the terminology in the Vitesse world is a key space, a logical database, whether it's sharded or unsharpened, you would actually have a key space with an ID and a primary index and a Vitesse index which points to an index. So there is a little bit of an additional terms that applies for the Vitesse, which we can explain a little bit more on the proxy server as a VT gate. A VT tablet is a back end server that actually works like a sidecar. There's also a topology data store which we manage by its ID ZooKeeper or console. Consider a common replication cluster where we have a primary end replicas. VT tablet attaches to the MySQL D process in each tablet and manages the MySQL D in the back end. So these coupled database structure with a replication cluster is a tablet. And in production, you would have multiple clusters and VT gate manages them. So if we go back in time over here, each MySQL server assigned to a VT tablet and then each MySQL VT tablet is managed by a VT gate where actually it's at state this proxy, it doesn't manage but it actually routes the traffic to the VT tablets. And you can have multiple VT gates. You're not bound to single one of them. So there's no single point of failure. Also, you can scale out the number of connections and that points to the clusters that you have in the back end. VT gate also routes the traffic to the correct clusters depending on the shouting scheme. Since we are intending to shard our data set using VTES, eventually, you would have to know where your data sits. So you don't have to maintain it yourself but the VTES maintains by VT gate. And then the queries are rooted based on the schema and the shouting scheme. In this example, let's say we have a commerce database, which is a key space, and then it's sharded. There's an also internal on sharded. This is important. So you don't have to shard everything but you can still have an on sharded key spaces in your topology. And a query comes in that points to the key space and VT gates know where that customer ID ranges. Topo is another component in this architecture actually stores the state of the schemas, shards, sharding scheme and tablets, can be used by HCD. We currently actually use HCD. There are others who use console or zookeeper. We don't recommend console for some reasons. But the key space is actually used by the console for some reasons. But you may choose to do so. And this has a very small dataset and it's mostly cashed by the VT gate. VT CTLD is another component that's in our architecture, which is a control daemon and runs the ad hoc operations to the database and API server and retries the topo, which also operates on tablets. So these are the components of that. And then VTest topology knows all the schemas, shards, clusters, and it keeps a state of the latest and the greatest information because in the real production environment, these will be changing and you would need to keep them up to date as much as possible. So VTest control plane not only includes these components, but also includes some of the normal database operations you would do. So in normal production environment, you would have some sort of a proxy server, which is covered by VT gate. You would actually have a backup and recovery, which is covered by the VT gate and VT CTLD, it would test using backups and recoveries. And in this case, we could actually drive extra backup for it for different data stores. Integrated failover that is actually known as an orchestrator, and in which case for the VTest topology, it is called VTALK. And sharding schemes that you can actually horizontal or vertical that you actually apply to that. There is an also advanced replication options, which actually allows you to replicate from one key space to the other. And VStream, there is going to be an online EDL operation, which is an experimental stage at this point, but we will be driving Goal Store Perkono toolkits online schema change tools. And there's also more for that. In summary, if you have an application that's actually driving through a load balancer, you can point to VT gate and a topology server knows your sharding scheme and points to the to the tablets that you have your data sets. Let's go over for the supported backend database. This is a very popular question that has been asked in the open source community, what databases and what engine it supports. VTest is a very MySQL-centric framework, and it works perfectly fine with with my score 5.7. And it's slowly being migrated to 80. And it also works on my score eight compatible. And we already have very large users in the community that are heavily invested in the MySQL backend for hundreds of thousands of shards. And also, it is known that it works for MariaDB for the compatibility purpose. The frequently asked question whether it will work for PostgreSQL or not, it will not work for PostgreSQL. There is going to be a need for another set of implementation that needs to be managed outside of the MySQL infrastructure. And let's talk about the VTest use cases and sharding. So in many cases, what we see is not always dropping replacement for your entire application. In some cases, your application does only need to scale out on certain parts. And either you have the entire application pointed to VTest, or keep and charted and unsharded key spaces in your topology, or you just scale out where it's where you need to. And also, you can point and create a link to your existing MySQL topology and use just VTest as your management of your MySQL topology without ever changing anything on your back end. You could do sharding and resharding. That's the key point over here. So in normal cases, once you shard, that's it. There's no actually coming back from sharding. With VTest, since it knows how you sharded, it's able to reshard or unshard your data sets into different key spaces. And with this, if you scaled out with VTest and simplified and minimized your backup and recovery scenarios by keeping a smaller data sets, this is actually a big advantage for when we see a terabytes of data in a single database and having issues for both backup and recovery scenarios. There are some other use cases that you can think of, obviously, but these are the things that I just came to my mind. Where we come to the MariaDB compatibility, MariaDB, there are no extensive work has been done, as far as I know. And we are looking for contributors and users. At this point, we have not received much feedback on MariaDB usage. And my scale, MariaDB compatibility for 10.4 is still pending, and there's an open issue at GitHub. And I will be following up with Fort MariaDB and the community, see if there's anything we can do to improve that area. So for more resources, please visit VTest.io. And there's a user guide and docs over there. It's an open source. I can rephrase that, and it's open. And you can see the code. If you are willing to contribute, please see the contributor guide at the document. And there's also VTest Slack where you can actually join and create a discussion. I can create a MariaDB room if it doesn't exist, and we can discuss further on that. Let's take a look at how we can drive MariaDB through VTest. Okay, now we're at the demo section. I have built a local Docker image. As I mentioned earlier, there are a couple of options that you can test VTest against MariaDB. One of them is to build a Docker image. The other one is building locally, compiling from source. The third option is the VTest operator. There's an open source VTest operator that you can test against Minicube or your favorite Kubernetes environment like GKE. In this example, I want to demonstrate a simple use case for scaling out MariaDB database using VTest. So the Docker image already built to save time, but I want to show you the Docker file and there's going to be some dependencies installed. We set those dependencies per the flavor of the backend engine we use and then build up a cluster that is an example of this demonstration. Over here, we also have the file for the installing local dependencies and we use HCD for the topo server. I will run this Docker image and build a simple e-commerce application database. Let's say we have an e-commerce database that is struggling to scale and it has an order table. It keeps customer and product information for the sake of this example. Then we actually want to not only shard but also manage this environment without having to go any downtime. Basically, we're going to do a small migration over here against the schema. The initial database that is simply unsharded and we want to experiment sharding over it. Before getting into that, I want to show you what we are running against. We are building this image against MariaDB version 10.3. If I connect to VTGate, which in this case is our proxy for the backend instances, and take a look at the version over here, it's the MariaDB and we are actually in the VTest client space that actually connects. Then we have different commands over here that you're not familiar with. If I said show VTest tablets, it would give me an example for the cell that is the zone. It's a key space, which is the schema and the database. There are no sharding, it's unsharded. There are different tablet types. Of course, there's a primary replica and read-only tablets that is attached for this cluster. There is a little bit of an extended information over here, but basically it mimics a primary replica scenario that you would normally run in your cluster. Let's move forward with the example over here. We have some pre-set scripts over here, which you can find this under VTest IO user guides. There are different examples for both local install operator, and then you can run through this scenario to experiment yourself. I want to continue on this one, showing the customer tablets. Basically, in order to get from commerce to the customer, we need to initialize a new shard. I will run this and then it will create a new customer tablet for us so that we can migrate our commerce database, which is unsharded, to a customer key space, which will be sharded eventually. It's going to initialize. Basically, this is a driver of instances. As the database instance is coming up, we can connect and basically see those tablets coming up. This operation will bring a new cluster, which is completely configured automatically in order for us to continue our example. Next up is to moving the tables to the new key space. As you can see over here from the script, I don't want to just run it, so it's not hidden. We TCTL client with the move tables from commerce to customer, it will actually migrate this over to that new key space. I will run this and it executed already. Next up will be moving these switching reads from commerce to customer key space. Basically, we're currently switching reads that are coming into the commerce key space into the replicas that are built for the new customer key space. I will run this. Then a traffic switcher, which is our VT gate, which actually knows about all this top of server, keeps track of all this information. It can actually take all the incoming traffic and migrate into the new space. Now that this will look a little bit awkward and how to visualize, we have some web interface that you can connect to. I don't want to get into that too much because we have a new VT admin UI is coming up very soon and that is built by our contributors at Slack and will be announcing that soon. Let's continue with the example over here. Now that we switched reads, we want to look at switching writes. With this command, VT CTL client switch writes customer to commerce, we are going to switch the writes also incoming writes to that. Since there's no traffic, things are going to be fast. Of course, this is not going to be the case at your example. We have a commerce still at the state of serving. Now that we switched, they're both serving, but actually all the reads and writes are going to the customer key space. We migrated into customer, but we didn't do what we didn't do. We haven't started this space. We just prepared the topology to be able to start. We can actually drop sources for the customer and for the commerce and then we can clean up the commerce space. It's going to drop the tablet and drop the tables from the source. Now that we want the customer space, and I want to take a look at that real quick, it is going to create a sequence for that. Now we have a concept of sequences, which it doesn't exist in our MISG RMAVV space in too much, but the sequences are required for the auto increment values. This way, the VT test topology server and the VT gate can control the auto increment values that goes into this charted environment. Now we run this, it will apply the sequences and it will create the V schema for that and it will apply the sharding scheme and it will create the sharded customer space. If you look at these, you can see these files that will actually be looking like this. It will be creating a table customer sequence, which is going to be a VT test sequence category. We will be running the customer sharded environment. Now that we created the customer sharded environment, that is ready for our sharding. If you look at the V schema objects that are created, there's going to be two sequences created, one for customer, one for the order. Then the V schema objects that are created for a sharded scheme that's on hash and a customer ID on hash. You can change that and then you have the column and the sequence ID, those are attached to those values so that the test knows what the sharding scheme is. For this example, the next one will be creating the new shards that are going to be needed. Let's take a look at that. It's going to create the new shards for that and this will be the shards and it will bring up those tablets with this. Then it will initialize those shard master to the related sharding scheme. It has created those tablets, those are probably coming off. Yes, we can see the new tablets coming for the sharded customer scheme. Just to reiterate, we had a commerce key space that we actually switched over to customer and then we started our sharding scheme against the customer key space and then built a cluster that is going to be a sharded against the customer. The new shards are created. Now, within the customer space, we need to reshard from the unsharded customer key space to a sharded customer key space. Okay, run that example over here. We were able to reshard that and then what we want to do is we want to switch the reads to the new sharded customer space. Okay, and we also want to switch the writes to that customer space by the traffic switcher over here. Okay. Okay. All right. Now that we have switched, the primary customer unsharded key space is no longer serving and the switch has been made over to this. Okay. Now, we can basically get rid of that unsharded customer space and delete that. No longer need that. It's going to delete those tablets and free them up. Okay. And that's it. I wish we had more time to play with the data. We don't have too much data to show and there's also a visual example of this. Again, please visit Vites.io website under user guides. There's more comprehensive steps and implementation for this. And again, find me on Vites.lac or Twitter or other medium to ask questions, maybe even contribute. Thank you very much for listening.
 Welcome. My name is Alkin and I'd like to share some experiences with Vitesse.  A little bit about myself. I am a senior technical manager at Planet Scale. I am one of the  maintainers for the Vitesse project. I'm an open source database evangelist, previously working  at Paracord at Pythion and some of the background and enterprise. I'm also an avid sailor. If you  want to talk about sailing, please do find me on Instagram or Twitter. A little bit about Planet  Scale because of the relations between the Vitesse project and the company as founded in 2018 by the  co-creators of Vitesse at YouTube, later on Google, currently about 45 to 50 employees located in  California and it's a complete remote team due to the pandemic. On the Vitesse project, what is Vitesse  question can be answered by its database clustering system for horizontal scaling of  MySQL or MariaDB databases. Why we say that? It's not a database, but it's a framework that actually  manages the databases that exists. So we actually take advantage of the boat's technologies on the  framework and the database back end site. Vitesse does not make any changes to the existing running  databases like MySQL or MariaDB. It's a CNCF-graduated project. In fact, it is one of the first  database projects in CNCF, very important. That's an open source and with an Apache tool that all  licensed and it has contributors from around the community with the large users of Vitesse.  Today's agenda is going to be an architectural overview of Vitesse. I'm going to talk about  what Vitesse is. So you have an idea and Vitesse use cases, a little bit on sharpening  and where we can actually meet MariaDB as the MariaDB Dev Room. Thank you for inviting me over.  On the Vitesse architecture basics, we actually enable the database infrastructure operations  to be more smooth and coming from single land. The glossary for the terminology in the Vitesse  world is a key space, a logical database, whether it's sharded or unsharpened, you would actually  have a key space with an ID and a primary index and a Vitesse index which points to an index.  So there is a little bit of an additional terms that applies for the Vitesse, which we can explain  a little bit more on the proxy server as a VT gate. A VT tablet is a back end server that  actually works like a sidecar. There's also a topology data store which we manage by  its ID ZooKeeper or console. Consider a common replication cluster where we have a primary  end replicas. VT tablet attaches to the MySQL D process in each tablet and manages the MySQL  D in the back end. So these coupled database structure with a replication cluster is a tablet.  And in production, you would have multiple clusters and VT gate manages them. So if we go back in  time over here, each MySQL server assigned to a VT tablet and then each MySQL VT tablet is managed  by a VT gate where actually it's at state this proxy, it doesn't manage but it actually routes  the traffic to the VT tablets. And you can have multiple VT gates. You're not bound to single  one of them. So there's no single point of failure. Also, you can scale out the number of connections  and that points to the clusters that you have in the back end. VT gate also  routes the traffic to the correct clusters depending on the shouting scheme. Since we are  intending to shard our data set using VTES, eventually, you would have to know where your  data sits. So you don't have to maintain it yourself but the VTES maintains by VT gate.  And then the queries are rooted based on the schema and the shouting scheme. In this example,  let's say we have a commerce database, which is a key space, and then it's sharded. There's  an also internal on sharded. This is important. So you don't have to shard everything but you  can still have an on sharded key spaces in your topology. And a query comes in that points to  the key space and VT gates know where that customer ID ranges. Topo is another component in this  architecture actually stores the state of the schemas, shards, sharding scheme and tablets,  can be used by HCD. We currently actually use HCD. There are others who use console or zookeeper.  We don't recommend console for some reasons. But the key space is actually used by the  console for some reasons. But you may choose to do so. And this has a very small dataset and  it's mostly cashed by the VT gate. VT CTLD is another component that's in our architecture,  which is a control daemon and runs the ad hoc operations to the database and API server and  retries the topo, which also operates on tablets. So these are the components of that. And then  VTest topology knows all the schemas, shards, clusters, and it keeps a state of the latest and  the greatest information because in the real production environment, these will be changing  and you would need to keep them up to date as much as possible. So VTest control plane not only  includes these components, but also includes some of the normal database operations you would do.  So in normal production environment, you would have some sort of a proxy server,  which is covered by VT gate. You would actually have a backup and recovery, which is covered by the  VT gate and VT CTLD, it would test using backups and recoveries. And in this case, we could actually  drive extra backup for it for different data stores. Integrated failover that is actually known as an  orchestrator, and in which case for the VTest topology, it is called VTALK. And sharding schemes  that you can actually horizontal or vertical that you actually apply to that. There is an also  advanced replication options, which actually allows you to replicate from one key space to the other.  And VStream, there is going to be an online EDL operation, which is an experimental stage at this  point, but we will be driving Goal Store Perkono toolkits online schema change tools. And there's  also more for that. In summary, if you have an application that's actually driving through a load  balancer, you can point to VT gate and a topology server knows your sharding scheme and points to  the to the tablets that you have your data sets. Let's go over for the supported backend database.  This is a very popular question that has been asked in the open source community, what databases  and what engine it supports. VTest is a very MySQL-centric framework, and it works perfectly  fine with with my score 5.7. And it's slowly being migrated to 80. And it also works on my  score eight compatible. And we already have very large users in the community that are heavily  invested in the MySQL backend for hundreds of thousands of shards. And also, it is known that  it works for MariaDB for the compatibility purpose. The frequently asked question whether it will  work for PostgreSQL or not, it will not work for PostgreSQL. There is going to be a need for another  set of implementation that needs to be managed outside of the MySQL infrastructure. And let's  talk about the VTest use cases and sharding. So in many cases, what we see is not always  dropping replacement for your entire application. In some cases, your application does only need  to scale out on certain parts. And either you have the entire application pointed to VTest,  or keep and charted and unsharded key spaces in your topology, or you just scale out where  it's where you need to. And also, you can point and create a link to your existing MySQL topology  and use just VTest as your management of your MySQL topology without ever changing anything  on your back end. You could do sharding and resharding. That's the key point over here. So  in normal cases, once you shard, that's it. There's no actually coming back from sharding.  With VTest, since it knows how you sharded, it's able to reshard or unshard your data sets  into different key spaces. And with this, if you scaled out with VTest and simplified and minimized  your backup and recovery scenarios by keeping a smaller data sets, this is actually a big  advantage for when we see a terabytes of data in a single database and having issues for both  backup and recovery scenarios. There are some other use cases that you can think of, obviously,  but these are the things that I just came to my mind. Where we come to the MariaDB compatibility,  MariaDB, there are no extensive work has been done, as far as I know. And we are looking for  contributors and users. At this point, we have not received much feedback on MariaDB usage.  And my scale, MariaDB compatibility for 10.4 is still pending, and there's an open issue at  GitHub. And I will be following up with Fort MariaDB and the community, see if there's anything we  can do to improve that area. So for more resources, please visit VTest.io. And there's a user guide  and docs over there. It's an open source. I can rephrase that, and it's open. And you can see the  code. If you are willing to contribute, please see the contributor guide at the document. And  there's also VTest Slack where you can actually join and create a discussion. I can create a  MariaDB room if it doesn't exist, and we can discuss further on that.  Let's take a look at how we can drive MariaDB through VTest. Okay, now we're at the demo section.  I have built a local Docker image. As I mentioned earlier, there are a couple of options that you  can test VTest against MariaDB. One of them is to build a Docker image. The other one is building  locally, compiling from source. The third option is the VTest operator. There's an open source VTest  operator that you can test against Minicube or your favorite Kubernetes environment like GKE.  In this example, I want to demonstrate a simple use case for scaling out MariaDB database using  VTest. So the Docker image already built to save time, but I want to show you the Docker file  and there's going to be some dependencies installed. We set those dependencies per the flavor of the  backend engine we use and then build up a cluster that is an example of this demonstration. Over  here, we also have the file for the installing local dependencies and we use HCD for the topo  server. I will run this Docker image and build a simple e-commerce application database.  Let's say we have an e-commerce database that is struggling to scale and it has an order table.  It keeps customer and product information for the sake of this example. Then we actually  want to not only shard but also manage this environment without having to go any downtime.  Basically, we're going to do a small migration over here against the schema. The initial database  that is simply unsharded and we want to experiment sharding over it. Before getting into that,  I want to show you what we are running against. We are building this image against MariaDB version  10.3. If I connect to VTGate, which in this case is our proxy for the backend instances,  and take a look at the version over here, it's the MariaDB and we are actually in the VTest  client space that actually connects. Then we have different commands over here that you're not  familiar with. If I said show VTest tablets, it would give me an example for the cell that is the  zone. It's a key space, which is the schema and the database. There are no sharding, it's unsharded.  There are different tablet types. Of course, there's a primary replica and read-only  tablets that is attached for this cluster. There is a little bit of an extended information  over here, but basically it mimics a primary replica scenario that you would normally run  in your cluster. Let's move forward with the example over here. We have some pre-set scripts  over here, which you can find this under VTest IO user guides. There are different examples for both  local install operator, and then you can run through this scenario to experiment yourself.  I want to continue on this one, showing the customer tablets. Basically, in order to get from  commerce to the customer, we need to initialize a new shard. I will run this and then it will  create a new customer tablet for us so that we can migrate our commerce database, which is  unsharded, to a customer key space, which will be sharded eventually. It's going to initialize.  Basically, this is a driver of instances. As the database instance is coming up, we can connect  and basically see those tablets coming up. This operation will bring a new cluster,  which is completely configured automatically in order for us to continue our example.  Next up is to moving the tables to the new key space. As you can see over here from the script,  I don't want to just run it, so it's not hidden. We TCTL client with the move tables  from commerce to customer, it will actually migrate this over to that new key space.  I will run this and it executed already. Next up will be moving these switching reads  from commerce to customer key space. Basically, we're currently switching reads that are coming  into the commerce key space into the replicas that are built for the new customer key space.  I will run this. Then a traffic switcher, which is our VT gate, which actually knows about  all this top of server, keeps track of all this information. It can actually take all the incoming  traffic and migrate into the new space. Now that this will look a little bit awkward and how to  visualize, we have some web interface that you can connect to. I don't want to get into that too  much because we have a new VT admin UI is coming up very soon and that is built by our contributors  at Slack and will be announcing that soon. Let's continue with the example over here.  Now that we switched reads, we want to look at switching writes. With this command,  VT CTL client switch writes customer to commerce, we are going to switch the writes also incoming  writes to that. Since there's no traffic, things are going to be fast. Of course,  this is not going to be the case at your example. We have a commerce still at the state of serving.  Now that we switched, they're both serving, but actually all the reads and writes are going to  the customer key space. We migrated into customer, but we didn't do what we didn't do. We haven't  started this space. We just prepared the topology to be able to start. We can actually drop sources  for the customer and for the commerce and then we can clean up the commerce space.  It's going to drop the tablet and drop the tables from the source.  Now that we want the customer space, and I want to take a look at that real quick,  it is going to create a sequence for that. Now we have a concept of sequences, which it  doesn't exist in our MISG RMAVV space in too much, but the sequences are required for the  auto increment values. This way, the VT test topology server and the VT gate can control  the auto increment values that goes into this charted environment. Now we run this,  it will apply the sequences and it will create the V schema for that and it will apply the  sharding scheme and it will create the sharded customer space. If you look at these, you can see  these files that will actually be looking like this. It will be creating a table customer  sequence, which is going to be a VT test sequence category. We will be running the customer sharded  environment. Now that we created the customer sharded environment,  that is ready for our sharding. If you look at the V schema objects that are created, there's  going to be two sequences created, one for customer, one for the order. Then the V schema  objects that are created for a sharded scheme that's on hash and a customer ID on hash. You  can change that and then you have the column and the sequence ID, those are attached to those  values so that the test knows what the sharding scheme is.  For this example, the next one will be creating the new shards that are going to be needed.  Let's take a look at that. It's going to create the new shards for that and this will be the  shards and it will bring up those tablets with this. Then it will initialize those shard master  to the related sharding scheme. It has created those tablets, those are probably coming off.  Yes, we can see the new tablets coming for the sharded customer scheme. Just to reiterate,  we had a commerce key space that we actually switched over to customer and then we started  our sharding scheme against the customer key space and then built a cluster that is going  to be a sharded against the customer. The new shards are created. Now, within the customer space,  we need to reshard from the unsharded customer key space to a sharded customer key space.  Okay, run that example over here. We were able to reshard that  and then what we want to do is we want to switch the reads to the new sharded customer space.  Okay, and we also want to switch the writes to that customer space by the traffic switcher over here.  Okay.  Okay. All right. Now that we have switched, the primary customer  unsharded key space is no longer serving and the switch has been made over to this.  Okay. Now, we can basically get rid of that  unsharded customer space and delete that. No longer need that. It's going to delete those  tablets and free them up. Okay. And that's it. I wish we had more time to play with the data.  We don't have too much data to show and there's also a visual example of this. Again, please visit  Vites.io website under user guides. There's more comprehensive steps and implementation for this.  And again, find me on Vites.lac or Twitter or other medium to ask questions, maybe even contribute.  Thank you very much for listening.
mariadb_shard_k_0_131.png mariadb_shard_k_132_1559.png mariadb_shard_k_1560_5591.png mariadb_shard_k_5592_5819.png mariadb_shard_k_5820_13883.png mariadb_shard_k_13884_20399.png mariadb_shard_k_20400_21575.png mariadb_shard_k_21576_21635.png mariadb_shard_k_21636_21935.png mariadb_shard_k_21936_22103.png mariadb_shard_k_22104_22991.png mariadb_shard_k_22992_23387.png mariadb_shard_k_23388_23699.png mariadb_shard_k_23700_25427.png mariadb_shard_k_25428_25619.png mariadb_shard_k_25620_26927.png mariadb_shard_k_26928_27083.png mariadb_shard_k_27084_27575.png mariadb_shard_k_27576_29303.png mariadb_shard_k_29304_31739.png mariadb_shard_k_31740_31823.png mariadb_shard_k_31824_32603.png mariadb_shard_k_32604_33047.png mariadb_shard_k_33048_34343.png mariadb_shard_k_34344_34775.png mariadb_shard_k_34776_34871.png mariadb_shard_k_34872_34991.png mariadb_shard_k_34992_35075.png mariadb_shard_k_35076_35663.png mariadb_shard_k_35664_36059.png mariadb_shard_k_36060_36856.png
Welcome. My name is Alkin and I'd like to share some experiences with Vitesse. A little bit about myself. I am a senior technical manager at Planet Scale. I am one of the maintainers for the Vitesse project. I'm an open source database evangelist, previously working at Paracord at Pythion and some of the background and enterprise. I'm also an avid sailor. If you want to talk about sailing, please do find me on Instagram or Twitter. A little bit about Planet Scale because of the relations between the Vitesse project and the company as founded in 2018 by the co-creators of Vitesse at YouTube, later on Google, currently about 45 to 50 employees located in California and it's a complete remote team due to the pandemic. On the Vitesse project, what is Vitesse question can be answered by its database clustering system for horizontal scaling of MySQL or MariaDB databases. Why we say that? It's not a database, but it's a framework that actually manages the databases that exists. So we actually take advantage of the boat's technologies on the framework and the database back end site. Vitesse does not make any changes to the existing running databases like MySQL or MariaDB. It's a CNCF-graduated project. In fact, it is one of the first database projects in CNCF, very important. That's an open source and with an Apache tool that all licensed and it has contributors from around the community with the large users of Vitesse. Today's agenda is going to be an architectural overview of Vitesse. I'm going to talk about what Vitesse is. So you have an idea and Vitesse use cases, a little bit on sharpening and where we can actually meet MariaDB as the MariaDB Dev Room. Thank you for inviting me over. On the Vitesse architecture basics, we actually enable the database infrastructure operations to be more smooth and coming from single land. The glossary for the terminology in the Vitesse world is a key space, a logical database, whether it's sharded or unsharpened, you would actually have a key space with an ID and a primary index and a Vitesse index which points to an index. So there is a little bit of an additional terms that applies for the Vitesse, which we can explain a little bit more on the proxy server as a VT gate. A VT tablet is a back end server that actually works like a sidecar. There's also a topology data store which we manage by its ID ZooKeeper or console. Consider a common replication cluster where we have a primary end replicas. VT tablet attaches to the MySQL D process in each tablet and manages the MySQL D in the back end. So these coupled database structure with a replication cluster is a tablet. And in production, you would have multiple clusters and VT gate manages them. So if we go back in time over here, each MySQL server assigned to a VT tablet and then each MySQL VT tablet is managed by a VT gate where actually it's at state this proxy, it doesn't manage but it actually routes the traffic to the VT tablets. And you can have multiple VT gates. You're not bound to single one of them. So there's no single point of failure. Also, you can scale out the number of connections and that points to the clusters that you have in the back end. VT gate also routes the traffic to the correct clusters depending on the shouting scheme. Since we are intending to shard our data set using VTES, eventually, you would have to know where your data sits. So you don't have to maintain it yourself but the VTES maintains by VT gate. And then the queries are rooted based on the schema and the shouting scheme. In this example, let's say we have a commerce database, which is a key space, and then it's sharded. There's an also internal on sharded. This is important. So you don't have to shard everything but you can still have an on sharded key spaces in your topology. And a query comes in that points to the key space and VT gates know where that customer ID ranges. Topo is another component in this architecture actually stores the state of the schemas, shards, sharding scheme and tablets, can be used by HCD. We currently actually use HCD. There are others who use console or zookeeper. We don't recommend console for some reasons. But the key space is actually used by the console for some reasons. But you may choose to do so. And this has a very small dataset and it's mostly cashed by the VT gate. VT CTLD is another component that's in our architecture, which is a control daemon and runs the ad hoc operations to the database and API server and retries the topo, which also operates on tablets. So these are the components of that. And then VTest topology knows all the schemas, shards, clusters, and it keeps a state of the latest and the greatest information because in the real production environment, these will be changing and you would need to keep them up to date as much as possible. So VTest control plane not only includes these components, but also includes some of the normal database operations you would do. So in normal production environment, you would have some sort of a proxy server, which is covered by VT gate. You would actually have a backup and recovery, which is covered by the VT gate and VT CTLD, it would test using backups and recoveries. And in this case, we could actually drive extra backup for it for different data stores. Integrated failover that is actually known as an orchestrator, and in which case for the VTest topology, it is called VTALK. And sharding schemes that you can actually horizontal or vertical that you actually apply to that. There is an also advanced replication options, which actually allows you to replicate from one key space to the other. And VStream, there is going to be an online EDL operation, which is an experimental stage at this point, but we will be driving Goal Store Perkono toolkits online schema change tools. And there's also more for that. In summary, if you have an application that's actually driving through a load balancer, you can point to VT gate and a topology server knows your sharding scheme and points to the to the tablets that you have your data sets. Let's go over for the supported backend database. This is a very popular question that has been asked in the open source community, what databases and what engine it supports. VTest is a very MySQL-centric framework, and it works perfectly fine with with my score 5.7. And it's slowly being migrated to 80. And it also works on my score eight compatible. And we already have very large users in the community that are heavily invested in the MySQL backend for hundreds of thousands of shards. And also, it is known that it works for MariaDB for the compatibility purpose. The frequently asked question whether it will work for PostgreSQL or not, it will not work for PostgreSQL. There is going to be a need for another set of implementation that needs to be managed outside of the MySQL infrastructure. And let's talk about the VTest use cases and sharding. So in many cases, what we see is not always dropping replacement for your entire application. In some cases, your application does only need to scale out on certain parts. And either you have the entire application pointed to VTest, or keep and charted and unsharded key spaces in your topology, or you just scale out where it's where you need to. And also, you can point and create a link to your existing MySQL topology and use just VTest as your management of your MySQL topology without ever changing anything on your back end. You could do sharding and resharding. That's the key point over here. So in normal cases, once you shard, that's it. There's no actually coming back from sharding. With VTest, since it knows how you sharded, it's able to reshard or unshard your data sets into different key spaces. And with this, if you scaled out with VTest and simplified and minimized your backup and recovery scenarios by keeping a smaller data sets, this is actually a big advantage for when we see a terabytes of data in a single database and having issues for both backup and recovery scenarios. There are some other use cases that you can think of, obviously, but these are the things that I just came to my mind. Where we come to the MariaDB compatibility, MariaDB, there are no extensive work has been done, as far as I know. And we are looking for contributors and users. At this point, we have not received much feedback on MariaDB usage. And my scale, MariaDB compatibility for 10.4 is still pending, and there's an open issue at GitHub. And I will be following up with Fort MariaDB and the community, see if there's anything we can do to improve that area. So for more resources, please visit VTest.io. And there's a user guide and docs over there. It's an open source. I can rephrase that, and it's open. And you can see the code. If you are willing to contribute, please see the contributor guide at the document. And there's also VTest Slack where you can actually join and create a discussion. I can create a MariaDB room if it doesn't exist, and we can discuss further on that. Let's take a look at how we can drive MariaDB through VTest. Okay, now we're at the demo section. I have built a local Docker image. As I mentioned earlier, there are a couple of options that you can test VTest against MariaDB. One of them is to build a Docker image. The other one is building locally, compiling from source. The third option is the VTest operator. There's an open source VTest operator that you can test against Minicube or your favorite Kubernetes environment like GKE. In this example, I want to demonstrate a simple use case for scaling out MariaDB database using VTest. So the Docker image already built to save time, but I want to show you the Docker file and there's going to be some dependencies installed. We set those dependencies per the flavor of the backend engine we use and then build up a cluster that is an example of this demonstration. Over here, we also have the file for the installing local dependencies and we use HCD for the topo server. I will run this Docker image and build a simple e-commerce application database. Let's say we have an e-commerce database that is struggling to scale and it has an order table. It keeps customer and product information for the sake of this example. Then we actually want to not only shard but also manage this environment without having to go any downtime. Basically, we're going to do a small migration over here against the schema. The initial database that is simply unsharded and we want to experiment sharding over it. Before getting into that, I want to show you what we are running against. We are building this image against MariaDB version 10.3. If I connect to VTGate, which in this case is our proxy for the backend instances, and take a look at the version over here, it's the MariaDB and we are actually in the VTest client space that actually connects. Then we have different commands over here that you're not familiar with. If I said show VTest tablets, it would give me an example for the cell that is the zone. It's a key space, which is the schema and the database. There are no sharding, it's unsharded. There are different tablet types. Of course, there's a primary replica and read-only tablets that is attached for this cluster. There is a little bit of an extended information over here, but basically it mimics a primary replica scenario that you would normally run in your cluster. Let's move forward with the example over here. We have some pre-set scripts over here, which you can find this under VTest IO user guides. There are different examples for both local install operator, and then you can run through this scenario to experiment yourself. I want to continue on this one, showing the customer tablets. Basically, in order to get from commerce to the customer, we need to initialize a new shard. I will run this and then it will create a new customer tablet for us so that we can migrate our commerce database, which is unsharded, to a customer key space, which will be sharded eventually. It's going to initialize. Basically, this is a driver of instances. As the database instance is coming up, we can connect and basically see those tablets coming up. This operation will bring a new cluster, which is completely configured automatically in order for us to continue our example. Next up is to moving the tables to the new key space. As you can see over here from the script, I don't want to just run it, so it's not hidden. We TCTL client with the move tables from commerce to customer, it will actually migrate this over to that new key space. I will run this and it executed already. Next up will be moving these switching reads from commerce to customer key space. Basically, we're currently switching reads that are coming into the commerce key space into the replicas that are built for the new customer key space. I will run this. Then a traffic switcher, which is our VT gate, which actually knows about all this top of server, keeps track of all this information. It can actually take all the incoming traffic and migrate into the new space. Now that this will look a little bit awkward and how to visualize, we have some web interface that you can connect to. I don't want to get into that too much because we have a new VT admin UI is coming up very soon and that is built by our contributors at Slack and will be announcing that soon. Let's continue with the example over here. Now that we switched reads, we want to look at switching writes. With this command, VT CTL client switch writes customer to commerce, we are going to switch the writes also incoming writes to that. Since there's no traffic, things are going to be fast. Of course, this is not going to be the case at your example. We have a commerce still at the state of serving. Now that we switched, they're both serving, but actually all the reads and writes are going to the customer key space. We migrated into customer, but we didn't do what we didn't do. We haven't started this space. We just prepared the topology to be able to start. We can actually drop sources for the customer and for the commerce and then we can clean up the commerce space. It's going to drop the tablet and drop the tables from the source. Now that we want the customer space, and I want to take a look at that real quick, it is going to create a sequence for that. Now we have a concept of sequences, which it doesn't exist in our MISG RMAVV space in too much, but the sequences are required for the auto increment values. This way, the VT test topology server and the VT gate can control the auto increment values that goes into this charted environment. Now we run this, it will apply the sequences and it will create the V schema for that and it will apply the sharding scheme and it will create the sharded customer space. If you look at these, you can see these files that will actually be looking like this. It will be creating a table customer sequence, which is going to be a VT test sequence category. We will be running the customer sharded environment. Now that we created the customer sharded environment, that is ready for our sharding. If you look at the V schema objects that are created, there's going to be two sequences created, one for customer, one for the order. Then the V schema objects that are created for a sharded scheme that's on hash and a customer ID on hash. You can change that and then you have the column and the sequence ID, those are attached to those values so that the test knows what the sharding scheme is. For this example, the next one will be creating the new shards that are going to be needed. Let's take a look at that. It's going to create the new shards for that and this will be the shards and it will bring up those tablets with this. Then it will initialize those shard master to the related sharding scheme. It has created those tablets, those are probably coming off. Yes, we can see the new tablets coming for the sharded customer scheme. Just to reiterate, we had a commerce key space that we actually switched over to customer and then we started our sharding scheme against the customer key space and then built a cluster that is going to be a sharded against the customer. The new shards are created. Now, within the customer space, we need to reshard from the unsharded customer key space to a sharded customer key space. Okay, run that example over here. We were able to reshard that and then what we want to do is we want to switch the reads to the new sharded customer space. Okay, and we also want to switch the writes to that customer space by the traffic switcher over here. Okay. Okay. All right. Now that we have switched, the primary customer unsharded key space is no longer serving and the switch has been made over to this. Okay. Now, we can basically get rid of that unsharded customer space and delete that. No longer need that. It's going to delete those tablets and free them up. Okay. And that's it. I wish we had more time to play with the data. We don't have too much data to show and there's also a visual example of this. Again, please visit Vites.io website under user guides. There's more comprehensive steps and implementation for this. And again, find me on Vites.lac or Twitter or other medium to ask questions, maybe even contribute. Thank you very much for listening.
Welcome. My name is Alkin and I'd like to share Vitesse project and the company as founded in Vitesse project, what is Vitesse question can be answered by its database clustering system for horizontal scaling of MySQL or MariaDB databases. In fact, it is one of the first database projects has contributors from around the community with I'm going to talk about what Vitesse is. So you VT tablets. And you can have multiple VT gates. end. VT gate also routes the traffic to the important. So you don't have to shard everything component in this architecture actually stores using backups and recoveries. And in this case, different data stores. Integrated failover that the tablets that you have your data sets. Let's is a very popular question that has been asked in it works for MariaDB for the compatibility With VTest, since it knows how you sharded, it's against Minicube or your favorite Kubernetes environment like GKE. In this example, I want to e-commerce database that is struggling to scale tablets, it would give me an example for the cell customer tablets. Basically, in order to get from both serving, but actually all the reads and that and it will apply the sharding scheme and it sharded against the customer. The new shards are