Gluster Container Storage
Madappa, Kaushal
Gluster Container Storage (GCS) is a new initiative from the Gluster community. It integrates several different old and new Gluster projects, building on top of the experience gained from the Heketi and gluster-kubernetes projects, and aims to provide the best persistent storage solution for containers. Gluster Container Storage (GCS) is a new initiative from the Gluster community, that aims to build the best persistent storage solution for containers. GCS integrates different Gluster projects, built on top of the experience gained from the Heketi and gluster-kubernetes projects and provides users with simple and robust tools to deploy, monitor and manage Gluster as well-behaved citizens of the Kubernetes world. GCS introduces the new Anthill and gluster-csi-driver projects. Anthill is Kubernetes operator, that deploys and manages GlusterFS with GlusterD2 and the gluster-csi-driver as containerized applications in a Kubernetes cluster. The gluster-csi-driver implements the Container Storage Interface API, and interacts with GlusterD2 to automatically provision persistent volumes for Kubernetes applications when needed. In this session we'll learn how all of these projects work together and form GCS.
 All right, we'll stay on the topic of containers. This time with Gluster, give a hand for Kaushal. Thanks. Thank you everyone. Okay, so today I'll be talking about Gluster container storage. This is that is storage for containers running in your containers, similar to what Rook is doing with Ceph. So this is what Gluster does for containers. Right, so I'm Kaushal. I'm a senior engineer at Red Hat and I work on the Gluster project. I'm a maintainer and I work mainly on the Gluster management frameworks and right now I'm contributing to the container story around Gluster. Right, so in today's talk, I'll talk about what GCS is. We'll go into a little bit more of a detail about what GCS looks like, how it performs as in how it does what it does, and I'll let you know how you can try out GCS. Okay, so just a quick question first up. So has anyone already heard of Gluster in containers with something called Hiketi? Right, so yeah, this is GCS is built on our experiences that we gained with the Hiketi project and running Gluster in containers with Hiketi. So I'll touch a little bit about Hiketi and try to compare what GCS does differently from the Hiketi-based stack in the stock, but yeah, so let's get started. So Gluster container storage, I'll be calling it GCS for short. Please don't get confused with GCS because in Kubernetes there is already GCS which stands for Google Cloud Storage, it's not Google Cloud Storage, it's Gluster Container Storage. We would welcome suggestions for a better name. So for now it's GCS, I'll be calling it as GCS. Okay, so what is GCS? GCS is a new effort from the Gluster community to provide persistent storage for containers. So we try to provide the two normal demands of container persistent storage, the shared storage and what do you say, exclusive storage, the RWX and read write many and read write one's volumes. We try to use the standard container interfaces as much as possible right now, so GCS implements one of the projects in GCS is to implement the container storage interface on CSI and using that we provide volumes when requested. For now we are Kubernetes only but maybe later on if we get help we'll move on to other container orchestrators. And GCS is being designed for a hyperconverged deployment right now. As in your storage runs along with your apps on the same servers that your apps, parts are running right. So yeah, so while the storage parts run along with your app parts on the same nodes, these are not your normal app parts, these need privileges because we do deal with underlying devices and underlying file systems and all. So we need to run in a privileged mode right. Okay, so GCS right it's not just a single, it's not a single project, it brings to the a lot of Gluster related projects and under a single umbrella. So I'll go into detail about what each of the project does for GCS specifically. So we bring in Gluster FS that we, everyone I hope everyone already knows about right. So the file system, there's Gluster E2 that's the new management tool that we have been building. There's the CSI driver specifically, there's Ant Hill, Ant Hill is a Kubernetes operator that will deploy and manage Gluster on Kubernetes and there are other projects like Gluster Prometheus, which helps monitor Gluster inside a Kubernetes deployment. So basically can plug in Gluster into your Kubernetes dashboard and you can drill down into Gluster volumes. Gluster Mixing provides the dashboard configurations, provides the alerts and rules, etc. Right, okay, so now why would you want to use GCS over what we have currently with Hecti, right? So the main theme of GCS is to simplify the whole experience as much as possible right. So we simplify how the containers get deployed, how Gluster gets deployed on Kubernetes. We simplify the general overall management workflow intended for the admins or users, right. We also go ahead and we are trying to simplify the Gluster FS stack itself. So and try to simplify the stack, the file system stack and ensure that they work good well for the container ecosystem. So what we do is that we try to make opinionated defaults, right. So right now the normal Gluster FS installation doesn't do any opinionated defaults. So it just leaves it open to the users to what you want. Everything is enabled, everything is available. So GCS is in that. So there are, we choose features that need to be enabled. We choose options that need to be defaulted and yeah, that's how GCS goes. Yeah, and all of this is going to be automated. So hopefully the most admin needs to do is to write a manifest for Anthill, deploy Anthill, write a cluster manifest on Kubernetes and done. So Anthill takes care of deploying everything that is required. So all the projects that I mentioned earlier, so Anthill takes care of deploying all of them, managing all of them. In the future, they'll also do data operations like replacing a node, upgrades and other stuff, right. And all of this together, right, allows us to improve our scale in the number of volumes at least. So Gluster FS originally wasn't designed for the container sort of environment. So Gluster FS was designed as a NAS replacement wherein you have one single large volume or a few single large volumes. So, but in containers, you generally deal with lots of very small volumes. So Gluster FS isn't particularly designed for that, but we are doing optimizations. We are trying to figure out ways so that we can scale better. Our goal is to on it, I guess this is goal for a three node cluster that is like the minimum for, if I believe, open shift that is required, that we can provide 2000 read write, what do you say, read write many volumes and up to 5000 read write volumes. That's the goal. These are like small, small volumes. These are not really huge. So something that can run in a three node cluster, right. So this is what GCC is going to provide. That is going to be different from the original stack, right. So I'm not going to talk too much about the current stack. As I mentioned, we have a Hikitty Bay stack. So the Hikitty Bay stack looks somewhat like this. So we have a component within Kubernetes itself that so we are pretty much like tied into the communities release cycle. So it's not so independent. There's a Gluster provisioner within Kubernetes. We have a separate Hikitty part, right. Hikitty manages the Gluster parts. And the thing, one big problem with Hikitty, the best problem is that we have a Hikitty DB separate with a Gluster configuration separate. So there are two separate ways of the Gluster pool. Hikitty has its own view. Gluster has its own view. And the problems that we see there are mainly about these two going out of sync and recovery from that is hard. But they have a lot of work has been done on this. And right now we are a lot of it has been solved. But still there is a problem with this. And in general, this stack relies on the normal Gluster deployment as the normal Gluster installation as in without the opinionated defaults that the new GCS stack is going to provide. Right. So yeah, there are details available below in the slide. The link is available in the schedule. If you want to, we can you can look at this. So looking at it, it looks relatively simple, but deployment and managing this cluster is challenging. It's not easy. Right. So let's get into GCS itself. Right. So GCS looks like this right now. So there are few components. We have Kubernetes. We don't need to worry about that. So we have until until is a Kubernetes operator. As I mentioned, that is going to deploy and manage the rest of these things. Right. In addition to until we have the CSI driver that is going to be the point of interaction with Kubernetes for us. So Kubernetes talks the Gluster CSI drivers Gluster CSI driver will then talk to the Gluster parts in the Gluster parts. We have the new management tool running. Plus ready to we have Gluster FS we have the Prometheus exporter inside. And we also have an XCD cluster that is a single source of truth right now. So the new management tool Gluster E2 stores the cluster view in it. So we have we GCS runs its own it's a declutter and doesn't make use of the Kubernetes at CD cluster. Right. So yeah. Okay. I talked about all of them anyway. So let's go into until first right until is a Kubernetes operator. Right. Which defines custom Gluster content GCS resource definition. So communities provides base for what do you say other projects to define their own resource types right. Object types. So until defines the GCS resource types in communities and it automates deployment upgrades and day to day management of GCS. Right. At the current state and Hill is very early. There isn't actually anything other than the CRDs. There isn't actually anything present yet. But things are working on stuff. Things will come out soon. Hill is deployed developed under the Gluster project in GitHub at GitHub.com slash cluster slash and help. Next up we have the CSI driver. So the CSI driver consists of. Okay. So as I mentioned CSI driver translates the Kubernetes CSI request into the GD2 rest request the Gluster E2 rest request. That's the main goal of these drivers. Right. So in addition to doing the translation of the request to create and delete volumes right. It also will take care of doing the mounts unlike in Hecti right. What would happen is the Gluster provisioner in the Kubernetes within Kubernetes would take care of reaching out to Gluster or reaching out to Hecti. That provisioner would take care of doing the mounts. But in this case we do everything by the CSI driver. So it gives us more freedom to do things the way we want. Right. So the CSI driver consists of three different types of parts. There's a note plugin. There's a provisioner and an attacher. Right. I'll go through what each of them does. Attach your provisioner and note plugin. Right. The attacher is for us at least in GCS world. It's a no-op. It's useless for us, but we just need to provide it because the Kubernetes CSI spec expects it. There is work going on to ensure to allow that allow CSI drivers without attaches. So not all CSI drivers actually require attaches work. So we promised it. Right. This is the main part for us right now. So this takes care of translate. This is the one part that listens for Kubernetes persistent volume requests or persistent volume claims, forwards them to Gluster. And yeah, that does it basically. It listens for the create and delete volume requests and forwards them to Gluster, translating them to the GD2 request. So in this part, we have sorry. Oh yeah, in this part, we have four different things running. So there is a Kubernetes provisioner sidecar. So these a few of them are provided by Kubernetes themselves. These are not developed by us, but yeah, so there's a Kubernetes provisioner. The Kubernetes provisioner's job is to listen for Kubernetes persistent volume object requests. And then the provisioner would talk via the CSI spec to the Gluster CSI driver, which would then relay that forward to the Gluster layer. So yeah, so there's a provisioner. We have a snapshot or sidecar which takes request handles Kubernetes requests for volume snapshots, persistent volume snapshots. And there is a cluster driver registrar which registers Gluster fscsi as a cluster level driver. So it makes the Gluster storage available as a storage class. So that's what the provisioner does. This is deployed as a right now, if I am right, a state full set so that it's always available running. And yeah, then we have the node plugin. The node plugin basically runs on every Kubernetes node that you have. Its main function is to mount the volumes when requested or attach the volumes to the app parts when requested. So yeah, that's all it does. So there's a driver registrar again, it makes the Gluster driver available on that node for mounting. So that's there are two containers, the driver registrar makes sure that the driver is registered for. So the Kubernetes knows to reach out to the driver to perform on. So if the node plugin parts also contain the Gluster fsclind bits, so you don't need to have the Gluster fsclind bits installed on the Kubernetes hosts. With the previous deployment with Hikki, you needed to you needed the client bits installed on your Kubernetes hosts. So that was one more issue with that. So yeah. So then now let's go into the Gluster part itself right. So the Gluster part has Gluster e2 Gluster fsclind itself and the Gluster fsclind exporter. So first I'll talk about Gluster e2. So Gluster e2 is the newer management framework for Gluster fsclind. It's intended to provide automated management operations for Gluster fsclind as an automatically create volumes based on request from the user. Right. So the user can request for a volume of a particular size of particular capabilities and Gluster fsclind gd2 will automatically decide upon the layout of the volume and as the user requested and create the volume. Right. In addition to that, it also tries to provide intelligent or automated volume management operations about scaling a volume or shrinking a volume replacing nodes, stuff like that, replacing bricks, stuff like that. Right. So basically what it is doing is it is converging the functionality of different tools earlier that was there outside the cluster. Right. So it was running outside the Gluster cluster, Gluster pool. Sorry. There is there was a project called Gluster block, which is which is used to provide the RWA volumes or block volumes. Right. These were running outside the Gluster pool. Right. They were not directly managed by the Gluster layer. All of this is converged into one tool right now. So Gluster e2. Gluster e2 does the work of hekt right automated volume management. And it also provides a wall also provides block volumes in a different sort of way compared to Gluster block. We'll see how it does it in a little while. Okay. So yeah. So gd2 provides a rest API like hekt right against which you can program. So this allowed us to write our CSI driver. We have an API that can be reached out to over the network to the operation. So CSI driver works on that. Gd2 also has a very good orchestration engine built in right. So it's more resilient to failures than the orchestration engine in the current cluster management layer. Right. So we can do undoes. We can do rollbacks if some operation failed. We can do an operation and say a note comes back up later. It can read rerun the operation if required stuff like that. So it brings in features to do more resilient orchestration of operations. So basically the management framework what it does at the end is that it needs to go to each node and prepare the break start processes and do stuff like that. So the gd2 brings in a much better framework to do that. In addition as I mentioned earlier there is just a single source of truth for everything. There is no split in cluster information between hekt and cluster. So all the information is with Gd2. So Gd2 knows what all volumes we have, how the volumes are laid out, what all devices we have access to stuff like that. So how many devices we have used everything right. So yeah there is less possibility of failures because of hekt and the cluster going out of sync. Right. In addition Gd2 also brings in support for Prometheus and open sensors so that you can trace or monitor Gd2 with the standard tools that are used elsewhere in the container ecosystem. Right. So Prometheus is widely used for monitoring within Kubernetes. Open sensors implements the open tracing API. So again we provide open tracing API endpoint so you can trace operations across the cluster if required. Right. Yep. So as I mentioned Gd2 provides normal volumes as well as block, cluster block. So the normal volumes are used to satisfy the Kubernetes, ad-abliax volumes or rewrite many volumes. The way how Gloucester D2 helps there is that for normal volumes there is automatic provisioning of bricks. So we go ahead and we create the LVMs, we create the thin poles, we create the file system on top of that, we do the brick mounts. We do all of that Gd2 takes care of doing that. And I mentioned one special thing that we bought in was support for volume templates. Right. So the normal Gloucester installation would contain a volume template with everything. Right. All the features that we provide. But for the GCS use case, for the continuity use case, you wouldn't need all of them. So if you had all of the features enabled in the graph, cluster graph, it would be heavy. It would be resource heavy. Right. So you don't need that. We can provide a custom simple template for ad-abliax volumes for containers. Cool. In addition, we also provide ad-ablia volumes as in we also provide block devices. Right. So block devices are meant for single users. Right. Single client access. This is still a work in progress. Bits and pieces of app it have been merged in. It's not complete yet. But yeah. What this does differently from the Gloucester block project is that this tries, does block exports in a very simple way using loop pack devices. Right. So what it basically does is that you have a cluster of this mount on your Kubernetes node and from that mount, on that mount, we create a image, I mean, device file or image file and then do a loop pack mount so that it's available as a block device. Right. The Gloucester block project did its exports via ISCSI and there was a lot of involved processing getting that set up. So that's more complex. Scaling was a problematic there. This one allows us to scale really easily, much well. So okay. So let's see the current status. Yep. So as of this moment, we can scale to again, all of these I'm talking about in a three node cluster, right? Can scale to about a thousand volumes, IWX volumes and hundreds of requests parallel as in hundreds of volume creation requests or volume decision requests happening parallel. So it doesn't bother you. We at this scale, we have begun hitting the limits of LVM scaling itself. Right. So LVM again, like Gloucester previously LVM itself wasn't designed for this sort of a thing. So Gloucester uses LVM thin pools to provide snapshot capabilities. So we use LVM. So we need LVM, but right now we are hitting on LVM scale issues. We are trying to figure out optimizations in the way we perform LVM operations itself that can help scale a bit further. And we are also looking at other options if possible for IWO volumes. Yep. Yep. In addition to the LVM scaling, we also sort of hit HCD scaling when all of these requests come together. Again, we are tuning the HCD config itself, how we deploy HCD and we are tuning how we use HCD itself. So as in how we store data in HCD, how we access HCD, right? So we are doing that. And Gloucester D2 is also, again, developed as a separate project under the Gloucester community. So it's developed under GitHub, Gloucester D2. So now let's look at what we have been doing in Gloucester FS itself to help with GCS, right? So the main focus has been optimizing resource usage. So this has been in progress from the last few releases. So Gloucester 5, one of the main topics was optimizing resource usage. Gloucester 6, the upcoming release is also, there is a lot of focus, again, on optimizing resource usage. So basically we are trying to reduce the memory that we use. So we are fixing a lot of the memory leaks that we have and everything there. We are trying to also optimize how we use threads, how we deploy it. And yeah. So we are optimizing resource usage. And again, there is this, I don't know if you have heard of multiplexing before. This multiplexing is a feature in Gloucester where we run multiple bricks in a single process, right? So again, before GCS, it wasn't such a priority. It was okay, we wouldn't have thousands of volumes running with GCS, with containers. We can't have thousands of processes running inside the pod, right? So we need to consolidate that. So yeah, we have multiplexing already for that. Multiplexing works for bricks, but there isn't a similar process for self-healing. So the volumes that we provide are redundant as in replica Gloucester volumes, Gloucester replica volumes, right? So they need to have self-healed demons running. Self-healed demons right now are individual per volume. So again, there would be as many self-healed demons as volumes that we have. So we need to multiplex the self-healed demons as well. So that's work in progress, okay? And there are other features like fencing. What is fencing? Yeah, so fencing is like a feature where we can ensure that only one client can access a file at a given time. So this is required for block volumes so that we don't have clients or say, right, say now a pod was rescheduled on some other node. We don't want them to like begin accessing the image file before some cleanup has happened or some release of blocks or whatever has happened. So things like that. So again, Gloucester FS is developed under GitHub Gloucester FS, right? Okay. Next on monitoring. Yep, so monitoring is developed under two projects. Again under Gloucester, under the Gloucester community, there's Gloucester Prometheus and there's Gloucester Mixins, right? The Gloucester Prometheus project provides us with the Gloucester FS exporter demon. This runs along with GD2 and exports the Gloucester metrics available, right? So it's not just the cluster level metrics that GD2 could provide. It's like per volume metrics, per big metrics, process metrics as in how much CPU is being used, how much disk space is being used and all of that. So the Gloucester Prometheus project provides this. And there's a Gloucester Mixins project which provides rules and alerts, right, for Prometheus and for Grafana so that you can build dashboards out of it. So right now, the Mixins projects provides a dashboard configuration for Gloucester FS volume. There are others being built. Okay. So I'm going to skip this. I was just going to talk about how the flow of operations was going to happen for a volume creation, but in simple terms, it's just straightforward, right? So it's Kubernetes to CSI driver, CSI driver to Gloucester, again the same flow. Create a volume, mount a volume. It's like one flow, single flow. It's the same for both sorts of volumes because everything is in one place. There's no split. There's no HAKET so that the request needs to go to HAKET or 10 to plus, right? Okay. So, okay, if you want to try out GCS, just keep in mind that it's still in the very heavy development. We are planning on doing a one-dot release sometime soon where, again, it's not fully featured but at least you can make use of it and just see how it works, right? So yeah, at the moment, we have an Ansible-based deployment tool written. It's not a deployment poll, it's just a playbook that deploys GCS, right? As Anthill isn't complete yet, so we are planning on Ansible right now and this playbook deploy GCS.yaml. This performs the full deployment right now, but as Anthill grows and gains more features, we'll be cutting down what the playbook does. So in the end, the hope is that the playbook would only deploy Anthill and we are done with it, right? So for more information on how to use the playbook to deploy GCS on your Kubernetes cluster, you can read the readme. So the readme gives you information on how to prepare the Ansible inventory files and how to run the playbook so that you can deploy GCS, all right? But if you want to just do a quick test on your local laptop, so we have a vagrant file available which does this. This vagrant is only for now because we do some additional adding additional tests and all that we haven't yet figured out on how to do for vagrant with virtual box. Contributions are welcome to help to that. This vagrant based tool will deploy Kubernetes and GCS for you on a three node cluster, right? So Kubernetes deployment is handled using kubespray. We have a playbook that calls kubespray finally, but this playbook has some configurations that we need to enable for GCS to run, right? So we need to enable certain feature gates to allow the CSI driver to work. So CSI driver is still a work in progress in the Kubernetes community, so not all the features are enabled out of the box, so we need to specially enable some features for the CSI drivers. Then more information on how to use this is provided in the readme, right? So this deployment tool is available under the GCS project, so there is a github.com slash blaster slash GCS. Under this we have a deployed directory where all of this is contained. So yep, and that's basically it. And as I mentioned, GCS is being developed under github plus GCS. So under this project we mainly have the overall issues tracking the full GCS, what do you say, worldview sort of a thing. And there is a waffle board which tracks the GCS related issues across the multiple different projects that I just described. So that's it. Please try out GCS and let us give us your feedback. And I'm open for questions now. Hi, do you support extending volumes? Shrink and extending volumes. So do we support extending volumes? GD2 itself supports it. The CSI driver doesn't yet, I don't know if the Kubernetes CSI spec supports it yet. So if that support comes in, we can build it. Our the cluster management layer already has support for automatic expansion. So we just need to do an extra call when that is available. And in case of three nodes deployment, is it possible that, so I suppose both three nodes should have equal storage, equal hardware. It would be preferable. It would be preferable, but okay, sorry. The question was in a three node cluster. For example, I have one server with big storage and two others with smaller one. And on that big, I store some bricks and I provide one revenue for that. And for some other volumes, I ask to have for example three replicas. Is it possible? Okay, so I'll repeat the question. The second part was, is it possible to have non-homogeneous hosts? We prefer if you have homogeneous hosts, but GD2 should be able to work with non-homogeneous hosts. The second part was if you are able to choose the, what do you say, redundancy count, the epic account. Right now, we aren't targeting that because we want to simplify the number of options that users have for us to build confidence in what we are doing first. That might come in later on. So right now, the CSI driver is hardcoded to create three replica volumes. It's hardcoded. So those three replica volumes can be created on a heterogeneous cluster as in, but the thing is, since it expects always three replicas to be present, if we can't individually provision bricks on each of the three nodes, the request will fail. So that's going to happen. Any further questions? Yes? How stable is GCS at the moment and do you support upgrades between GCS and GD2? Not really stable. It's, okay. The question is how stable is GCS right now? GCS isn't really stable. We haven't reached something that we had asked users to deploy in production yet. So the upgrade stories and all should be pretty simple, but we haven't thought about that yet. We have been just working on getting what we want to deliver in the first release fixed. So the upgrades for GCS would be simple enough because it could be just deploying the new version of the cluster parts. All our configuration right now stays in HCD. HCD itself has its own upgrade mechanisms already defined for Kubernetes, so that's taken care of. The GCS parts themselves always rely on the information in HCD. So once we reach that stage, yes, it should be simple enough to do upgrades. Yeah? How do you tend to handle doing upgrades if you have parts of mounting volumes when you have to like move mount points or something? Yep, we need to do so. Okay. The question is how do we intend to do the upgrades, right? So if parts are mounting volumes and whatnot, so as I understand, there is some sort of, I don't know, gating sort of capabilities in the Kubernetes operator world. I don't really understand the operator part of it. So the operator would take care of doing that, right? So what the operator would do would be to like bring down one particular part, right? Then get a new version of the part up and running. Do whatever mounts is required inside that till that part is up and running. It would block the upgrades of the rest of the parts, do stuff like that. So it's not exactly clear at the moment, but I believe there is some way to get operations to allow this to be handled in a smooth way. Any further questions? Nope? Thank you, everyone. Thanks, guys. Thank you.
 All right, we'll stay on the topic of containers.  This time with Gluster, give a hand for Kaushal.  Thanks. Thank you everyone.  Okay, so today I'll be talking about Gluster container storage.  This is that is storage for containers running in your containers,  similar to what Rook is doing with Ceph. So this is what Gluster does for containers.  Right, so I'm Kaushal. I'm a senior engineer at Red Hat and I work on the Gluster project.  I'm a maintainer and I work mainly on the Gluster management frameworks  and right now I'm contributing to the container story around Gluster.  Right, so in today's talk, I'll talk about what GCS is.  We'll go into a little bit more of a detail about what GCS looks like,  how it performs as in how it does what it does,  and I'll let you know how you can try out GCS.  Okay, so just a quick question first up.  So has anyone already heard of Gluster in containers with something called Hiketi?  Right, so yeah, this is GCS is built on our experiences that we gained with the Hiketi project  and running Gluster in containers with Hiketi.  So I'll touch a little bit about Hiketi and try to compare what GCS does differently  from the Hiketi-based stack in the stock, but yeah, so let's get started.  So Gluster container storage, I'll be calling it GCS for short.  Please don't get confused with GCS because in Kubernetes there is already GCS  which stands for Google Cloud Storage, it's not Google Cloud Storage,  it's Gluster Container Storage.  We would welcome suggestions for a better name.  So for now it's GCS, I'll be calling it as GCS.  Okay, so what is GCS?  GCS is a new effort from the Gluster community to provide persistent storage for containers.  So we try to provide the two normal demands of container persistent storage,  the shared storage and what do you say, exclusive storage, the RWX  and read write many and read write one's volumes.  We try to use the standard container interfaces as much as possible right now,  so GCS implements one of the projects in GCS is to implement the container storage interface on CSI  and using that we provide volumes when requested.  For now we are Kubernetes only but maybe later on if we get help  we'll move on to other container orchestrators.  And GCS is being designed for a hyperconverged deployment right now.  As in your storage runs along with your apps on the same servers that your apps,  parts are running right.  So yeah, so while the storage parts run along with your app parts on the same nodes,  these are not your normal app parts, these need privileges because we do deal with  underlying devices and underlying file systems and all.  So we need to run in a privileged mode right.  Okay, so GCS right it's not just a single, it's not a single project,  it brings to the a lot of Gluster related projects and under a single umbrella.  So I'll go into detail about what each of the project does for GCS specifically.  So we bring in Gluster FS that we, everyone I hope everyone already knows about right.  So the file system, there's Gluster E2 that's the new management tool that we have been building.  There's the CSI driver specifically, there's Ant Hill, Ant Hill is a Kubernetes operator that will deploy  and manage Gluster on Kubernetes and there are other projects like Gluster Prometheus,  which helps monitor Gluster inside a Kubernetes deployment.  So basically can plug in Gluster into your Kubernetes dashboard and you can drill down into Gluster volumes.  Gluster Mixing provides the dashboard configurations, provides the alerts and rules, etc.  Right, okay, so now why would you want to use GCS over what we have currently with Hecti, right?  So the main theme of GCS is to simplify the whole experience as much as possible right.  So we simplify how the containers get deployed, how Gluster gets deployed on Kubernetes.  We simplify the general overall management workflow intended for the admins or users, right.  We also go ahead and we are trying to simplify the Gluster FS stack itself.  So and try to simplify the stack, the file system stack and ensure that they work good well for the container ecosystem.  So what we do is that we try to make opinionated defaults, right.  So right now the normal Gluster FS installation doesn't do any opinionated defaults.  So it just leaves it open to the users to what you want.  Everything is enabled, everything is available.  So GCS is in that.  So there are, we choose features that need to be enabled.  We choose options that need to be defaulted and yeah, that's how GCS goes.  Yeah, and all of this is going to be automated.  So hopefully the most admin needs to do is to write a manifest for Anthill,  deploy Anthill, write a cluster manifest on Kubernetes and done.  So Anthill takes care of deploying everything that is required.  So all the projects that I mentioned earlier, so Anthill takes care of deploying all of them, managing all of them.  In the future, they'll also do data operations like replacing a node, upgrades and other stuff, right.  And all of this together, right, allows us to improve our scale in the number of volumes at least.  So Gluster FS originally wasn't designed for the container sort of environment.  So Gluster FS was designed as a NAS replacement wherein you have one single large volume or a few single large volumes.  So, but in containers, you generally deal with lots of very small volumes.  So Gluster FS isn't particularly designed for that, but we are doing optimizations.  We are trying to figure out ways so that we can scale better.  Our goal is to on it, I guess this is goal for a three node cluster that is like the minimum for, if I believe, open shift that is required,  that we can provide 2000 read write, what do you say, read write many volumes and up to 5000 read write volumes.  That's the goal. These are like small, small volumes. These are not really huge.  So something that can run in a three node cluster, right.  So this is what GCC is going to provide. That is going to be different from the original stack, right.  So I'm not going to talk too much about the current stack.  As I mentioned, we have a Hikitty Bay stack.  So the Hikitty Bay stack looks somewhat like this.  So we have a component within Kubernetes itself that so we are pretty much like tied into the communities release cycle.  So it's not so independent. There's a Gluster provisioner within Kubernetes.  We have a separate Hikitty part, right. Hikitty manages the Gluster parts.  And the thing, one big problem with Hikitty, the best problem is that we have a Hikitty DB separate with a Gluster configuration separate.  So there are two separate ways of the Gluster pool. Hikitty has its own view.  Gluster has its own view. And the problems that we see there are mainly about these two going out of sync and recovery from that is hard.  But they have a lot of work has been done on this.  And right now we are a lot of it has been solved. But still there is a problem with this.  And in general, this stack relies on the normal Gluster deployment as the normal Gluster installation as in without the opinionated defaults that the new GCS stack is going to provide.  Right. So yeah, there are details available below in the slide.  The link is available in the schedule. If you want to, we can you can look at this.  So looking at it, it looks relatively simple, but deployment and managing this cluster is challenging. It's not easy.  Right. So let's get into GCS itself. Right. So GCS looks like this right now.  So there are few components. We have Kubernetes. We don't need to worry about that.  So we have until until is a Kubernetes operator. As I mentioned, that is going to deploy and manage the rest of these things. Right.  In addition to until we have the CSI driver that is going to be the point of interaction with Kubernetes for us.  So Kubernetes talks the Gluster CSI drivers Gluster CSI driver will then talk to the Gluster parts in the Gluster parts.  We have the new management tool running.  Plus ready to we have Gluster FS we have the Prometheus exporter inside.  And we also have an XCD cluster that is a single source of truth right now.  So the new management tool Gluster E2 stores the cluster view in it.  So we have we GCS runs its own it's a declutter and doesn't make use of the Kubernetes at CD cluster.  Right. So yeah.  Okay. I talked about all of them anyway. So let's go into until first right until is a Kubernetes operator.  Right. Which defines custom Gluster content GCS resource definition.  So communities provides base for what do you say other projects to define their own resource types right.  Object types. So until defines the GCS resource types in communities and it automates deployment upgrades and day to day management of GCS.  Right. At the current state and Hill is very early.  There isn't actually anything other than the CRDs.  There isn't actually anything present yet.  But things are working on stuff.  Things will come out soon.  Hill is deployed developed under the Gluster project in GitHub at GitHub.com slash cluster slash and help.  Next up we have the CSI driver.  So the CSI driver consists of.  Okay. So as I mentioned CSI driver translates the Kubernetes CSI request into the GD2 rest request the Gluster E2 rest request.  That's the main goal of these drivers. Right.  So in addition to doing the translation of the request to create and delete volumes right.  It also will take care of doing the mounts unlike in Hecti right.  What would happen is the Gluster provisioner in the Kubernetes within Kubernetes would take care of reaching out to Gluster or reaching out to Hecti.  That provisioner would take care of doing the mounts.  But in this case we do everything by the CSI driver.  So it gives us more freedom to do things the way we want.  Right. So the CSI driver consists of three different types of parts.  There's a note plugin. There's a provisioner and an attacher. Right.  I'll go through what each of them does.  Attach your provisioner and note plugin. Right.  The attacher is for us at least in GCS world. It's a no-op.  It's useless for us, but we just need to provide it because the Kubernetes CSI spec expects it.  There is work going on to ensure to allow that allow CSI drivers without attaches.  So not all CSI drivers actually require attaches work.  So we promised it. Right.  This is the main part for us right now.  So this takes care of translate. This is the one part that listens for Kubernetes persistent volume requests or persistent volume claims,  forwards them to Gluster.  And yeah, that does it basically.  It listens for the create and delete volume requests and forwards them to Gluster,  translating them to the GD2 request.  So in this part, we have sorry. Oh yeah, in this part, we have four different things running.  So there is a Kubernetes provisioner sidecar.  So these a few of them are provided by Kubernetes themselves.  These are not developed by us, but yeah, so there's a Kubernetes provisioner.  The Kubernetes provisioner's job is to listen for Kubernetes persistent volume object requests.  And then the provisioner would talk via the CSI spec to the Gluster CSI driver,  which would then relay that forward to the Gluster layer.  So yeah, so there's a provisioner.  We have a snapshot or sidecar which takes request handles Kubernetes requests for volume snapshots, persistent volume snapshots.  And there is a cluster driver registrar which registers Gluster fscsi as a cluster level driver.  So it makes the Gluster storage available as a storage class.  So that's what the provisioner does.  This is deployed as a right now, if I am right, a state full set so that it's always available running.  And yeah, then we have the node plugin.  The node plugin basically runs on every Kubernetes node that you have.  Its main function is to mount the volumes when requested or attach the volumes to the app parts when requested.  So yeah, that's all it does.  So there's a driver registrar again, it makes the Gluster driver available on that node for mounting.  So that's there are two containers, the driver registrar makes sure that the driver is registered for.  So the Kubernetes knows to reach out to the driver to perform on.  So if the node plugin parts also contain the Gluster fsclind bits,  so you don't need to have the Gluster fsclind bits installed on the Kubernetes hosts.  With the previous deployment with Hikki, you needed to you needed the client bits installed on your Kubernetes hosts.  So that was one more issue with that.  So yeah.  So then now let's go into the Gluster part itself right.  So the Gluster part has Gluster e2 Gluster fsclind itself and the Gluster fsclind exporter.  So first I'll talk about Gluster e2.  So Gluster e2 is the newer management framework for Gluster fsclind.  It's intended to provide automated management operations for Gluster fsclind as an automatically create volumes based on request from the user.  Right.  So the user can request for a volume of a particular size of particular capabilities and Gluster fsclind gd2 will automatically decide upon the layout of the volume and as the user requested and create the volume.  Right.  In addition to that, it also tries to provide intelligent or automated volume management operations about scaling a volume or shrinking a volume replacing nodes, stuff like that, replacing bricks, stuff like that.  Right.  So basically what it is doing is it is converging the functionality of different tools earlier that was there outside the cluster.  Right.  So it was running outside the Gluster cluster, Gluster pool.  Sorry.  There is there was a project called Gluster block, which is which is used to provide the RWA volumes or block volumes.  Right.  These were running outside the Gluster pool.  Right.  They were not directly managed by the Gluster layer.  All of this is converged into one tool right now.  So Gluster e2.  Gluster e2 does the work of hekt right automated volume management.  And it also provides a wall also provides block volumes in a different sort of way compared to Gluster block.  We'll see how it does it in a little while.  Okay.  So yeah.  So gd2 provides a rest API like hekt right against which you can program.  So this allowed us to write our CSI driver.  We have an API that can be reached out to over the network to the operation.  So CSI driver works on that.  Gd2 also has a very good orchestration engine built in right.  So it's more resilient to failures than the orchestration engine in the current cluster management layer.  Right.  So we can do undoes.  We can do rollbacks if some operation failed.  We can do an operation and say a note comes back up later.  It can read rerun the operation if required stuff like that.  So it brings in features to do more resilient orchestration of operations.  So basically the management framework what it does at the end is that it needs to go to each node and prepare the break start processes and do stuff like that.  So the gd2 brings in a much better framework to do that.  In addition as I mentioned earlier there is just a single source of truth for everything.  There is no split in cluster information between hekt and cluster.  So all the information is with Gd2.  So Gd2 knows what all volumes we have, how the volumes are laid out, what all devices we have access to stuff like that.  So how many devices we have used everything right.  So yeah there is less possibility of failures because of hekt and the cluster going out of sync.  Right.  In addition Gd2 also brings in support for Prometheus and open sensors so that you can trace or monitor Gd2 with the standard tools that are used elsewhere in the container ecosystem.  Right.  So Prometheus is widely used for monitoring within Kubernetes.  Open sensors implements the open tracing API.  So again we provide open tracing API endpoint so you can trace operations across the cluster if required.  Right.  Yep.  So as I mentioned Gd2 provides normal volumes as well as block, cluster block.  So the normal volumes are used to satisfy the Kubernetes, ad-abliax volumes or rewrite many volumes.  The way how Gloucester D2 helps there is that for normal volumes there is automatic provisioning of bricks.  So we go ahead and we create the LVMs, we create the thin poles, we create the file system on top of that, we do the brick mounts.  We do all of that Gd2 takes care of doing that.  And I mentioned one special thing that we bought in was support for volume templates.  Right.  So the normal Gloucester installation would contain a volume template with everything.  Right.  All the features that we provide.  But for the GCS use case, for the continuity use case, you wouldn't need all of them.  So if you had all of the features enabled in the graph, cluster graph, it would be heavy.  It would be resource heavy.  Right.  So you don't need that.  We can provide a custom simple template for ad-abliax volumes for containers.  Cool.  In addition, we also provide ad-ablia volumes as in we also provide block devices.  Right.  So block devices are meant for single users.  Right.  Single client access.  This is still a work in progress.  Bits and pieces of app it have been merged in.  It's not complete yet.  But yeah.  What this does differently from the Gloucester block project is that this tries, does block  exports in a very simple way using loop pack devices.  Right.  So what it basically does is that you have a cluster of this mount on your Kubernetes  node and from that mount, on that mount, we create a image, I mean, device file or image  file and then do a loop pack mount so that it's available as a block device.  Right.  The Gloucester block project did its exports via ISCSI and there was a lot of involved  processing getting that set up.  So that's more complex.  Scaling was a problematic there.  This one allows us to scale really easily, much well.  So okay.  So let's see the current status.  Yep.  So as of this moment, we can scale to again, all of these I'm talking about in a three  node cluster, right?  Can scale to about a thousand volumes, IWX volumes and hundreds of requests parallel  as in hundreds of volume creation requests or volume decision requests happening parallel.  So it doesn't bother you.  We at this scale, we have begun hitting the limits of LVM scaling itself.  Right.  So LVM again, like Gloucester previously LVM itself wasn't designed for this sort of  a thing.  So Gloucester uses LVM thin pools to provide snapshot capabilities.  So we use LVM.  So we need LVM, but right now we are hitting on LVM scale issues.  We are trying to figure out optimizations in the way we perform LVM operations itself  that can help scale a bit further.  And we are also looking at other options if possible for IWO volumes.  Yep.  Yep.  In addition to the LVM scaling, we also sort of hit HCD scaling when all of these requests  come together.  Again, we are tuning the HCD config itself, how we deploy HCD and we are tuning how we  use HCD itself.  So as in how we store data in HCD, how we access HCD, right?  So we are doing that.  And Gloucester D2 is also, again, developed as a separate project under the Gloucester  community.  So it's developed under GitHub, Gloucester D2.  So now let's look at what we have been doing in Gloucester FS itself to help with GCS,  right?  So the main focus has been optimizing resource usage.  So this has been in progress from the last few releases.  So Gloucester 5, one of the main topics was optimizing resource usage.  Gloucester 6, the upcoming release is also, there is a lot of focus, again, on optimizing  resource usage.  So basically we are trying to reduce the memory that we use.  So we are fixing a lot of the memory leaks that we have and everything there.  We are trying to also optimize how we use threads, how we deploy it.  And yeah.  So we are optimizing resource usage.  And again, there is this, I don't know if you have heard of multiplexing before.  This multiplexing is a feature in Gloucester where we run multiple bricks in a single process,  right?  So again, before GCS, it wasn't such a priority.  It was okay, we wouldn't have thousands of volumes running with GCS, with containers.  We can't have thousands of processes running inside the pod, right?  So we need to consolidate that.  So yeah, we have multiplexing already for that.  Multiplexing works for bricks, but there isn't a similar process for self-healing.  So the volumes that we provide are redundant as in replica Gloucester volumes, Gloucester  replica volumes, right?  So they need to have self-healed demons running.  Self-healed demons right now are individual per volume.  So again, there would be as many self-healed demons as volumes that we have.  So we need to multiplex the self-healed demons as well.  So that's work in progress, okay?  And there are other features like fencing.  What is fencing?  Yeah, so fencing is like a feature where we can ensure that only one client can access  a file at a given time.  So this is required for block volumes so that we don't have clients or say, right, say now  a pod was rescheduled on some other node.  We don't want them to like begin accessing the image file before some cleanup has happened  or some release of blocks or whatever has happened.  So things like that.  So again, Gloucester FS is developed under GitHub Gloucester FS, right?  Okay.  Next on monitoring.  Yep, so monitoring is developed under two projects.  Again under Gloucester, under the Gloucester community, there's Gloucester Prometheus and  there's Gloucester Mixins, right?  The Gloucester Prometheus project provides us with the Gloucester FS exporter demon.  This runs along with GD2 and exports the Gloucester metrics available, right?  So it's not just the cluster level metrics that GD2 could provide.  It's like per volume metrics, per big metrics, process metrics as in how much CPU is being  used, how much disk space is being used and all of that.  So the Gloucester Prometheus project provides this.  And there's a Gloucester Mixins project which provides rules and alerts, right, for Prometheus  and for Grafana so that you can build dashboards out of it.  So right now, the Mixins projects provides a dashboard configuration for Gloucester FS  volume.  There are others being built.  Okay.  So I'm going to skip this.  I was just going to talk about how the flow of operations was going to happen for a volume  creation, but in simple terms, it's just straightforward, right?  So it's Kubernetes to CSI driver, CSI driver to Gloucester, again the same flow.  Create a volume, mount a volume.  It's like one flow, single flow.  It's the same for both sorts of volumes because everything is in one place.  There's no split.  There's no HAKET so that the request needs to go to HAKET or 10 to plus, right?  Okay.  So, okay, if you want to try out GCS, just keep in mind that it's still in the very heavy  development.  We are planning on doing a one-dot release sometime soon where, again, it's not fully  featured but at least you can make use of it and just see how it works, right?  So yeah, at the moment, we have an Ansible-based deployment tool written.  It's not a deployment poll, it's just a playbook that deploys GCS, right?  As Anthill isn't complete yet, so we are planning on Ansible right now and this playbook  deploy GCS.yaml.  This performs the full deployment right now, but as Anthill grows and gains more features,  we'll be cutting down what the playbook does.  So in the end, the hope is that the playbook would only deploy Anthill and we are done  with it, right?  So for more information on how to use the playbook to deploy GCS on your Kubernetes  cluster, you can read the readme.  So the readme gives you information on how to prepare the Ansible inventory files and  how to run the playbook so that you can deploy GCS, all right?  But if you want to just do a quick test on your local laptop, so we have a vagrant file  available which does this.  This vagrant is only for now because we do some additional adding additional tests and  all that we haven't yet figured out on how to do for vagrant with virtual box.  Contributions are welcome to help to that.  This vagrant based tool will deploy Kubernetes and GCS for you on a three node cluster, right?  So Kubernetes deployment is handled using kubespray.  We have a playbook that calls kubespray finally, but this playbook has some configurations  that we need to enable for GCS to run, right?  So we need to enable certain feature gates to allow the CSI driver to work.  So CSI driver is still a work in progress in the Kubernetes community, so not all the  features are enabled out of the box, so we need to specially enable some features for  the CSI drivers.  Then more information on how to use this is provided in the readme, right?  So this deployment tool is available under the GCS project, so there is a github.com  slash blaster slash GCS.  Under this we have a deployed directory where all of this is contained.  So yep, and that's basically it.  And as I mentioned, GCS is being developed under github plus GCS.  So under this project we mainly have the overall issues tracking the full GCS, what  do you say, worldview sort of a thing.  And there is a waffle board which tracks the GCS related issues across the multiple different  projects that I just described.  So that's it.  Please try out GCS and let us give us your feedback.  And I'm open for questions now.  Hi, do you support extending volumes?  Shrink and extending volumes.  So do we support extending volumes?  GD2 itself supports it.  The CSI driver doesn't yet, I don't know if the Kubernetes CSI spec supports it yet.  So if that support comes in, we can build it.  Our the cluster management layer already has support for automatic expansion.  So we just need to do an extra call when that is available.  And in case of three nodes deployment, is it possible that, so I suppose both three  nodes should have equal storage, equal hardware.  It would be preferable.  It would be preferable, but okay, sorry.  The question was in a three node cluster.  For example, I have one server with big storage and two others with smaller one.  And on that big, I store some bricks and I provide one revenue for that.  And for some other volumes, I ask to have for example three replicas.  Is it possible?  Okay, so I'll repeat the question.  The second part was, is it possible to have non-homogeneous hosts?  We prefer if you have homogeneous hosts, but GD2 should be able to work with non-homogeneous  hosts.  The second part was if you are able to choose the, what do you say, redundancy count, the  epic account.  Right now, we aren't targeting that because we want to simplify the number of options  that users have for us to build confidence in what we are doing first.  That might come in later on.  So right now, the CSI driver is hardcoded to create three replica volumes.  It's hardcoded.  So those three replica volumes can be created on a heterogeneous cluster as in, but the  thing is, since it expects always three replicas to be present, if we can't individually  provision bricks on each of the three nodes, the request will fail.  So that's going to happen.  Any further questions?  Yes?  How stable is GCS at the moment and do you support upgrades between GCS and GD2?  Not really stable.  It's, okay.  The question is how stable is GCS right now?  GCS isn't really stable.  We haven't reached something that we had asked users to deploy in production yet.  So the upgrade stories and all should be pretty simple, but we haven't thought about that  yet.  We have been just working on getting what we want to deliver in the first release fixed.  So the upgrades for GCS would be simple enough because it could be just deploying the new  version of the cluster parts.  All our configuration right now stays in HCD.  HCD itself has its own upgrade mechanisms already defined for Kubernetes, so that's taken care  of.  The GCS parts themselves always rely on the information in HCD.  So once we reach that stage, yes, it should be simple enough to do upgrades.  Yeah?  How do you tend to handle doing upgrades if you have parts of mounting volumes when you  have to like move mount points or something?  Yep, we need to do so.  Okay.  The question is how do we intend to do the upgrades, right?  So if parts are mounting volumes and whatnot, so as I understand, there is some sort of,  I don't know, gating sort of capabilities in the Kubernetes operator world.  I don't really understand the operator part of it.  So the operator would take care of doing that, right?  So what the operator would do would be to like bring down one particular part, right?  Then get a new version of the part up and running.  Do whatever mounts is required inside that till that part is up and running.  It would block the upgrades of the rest of the parts, do stuff like that.  So it's not exactly clear at the moment, but I believe there is some way to get operations  to allow this to be handled in a smooth way.  Any further questions?  Nope?  Thank you, everyone.  Thanks, guys.  Thank you.
Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_0_179.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_180_1151.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_1152_1307.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_1308_1427.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_1428_1559.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_1560_1679.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_1680_2327.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_2328_2399.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_2400_2459.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_2460_2579.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_2580_2699.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_2700_2819.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_2820_3011.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_3012_5027.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_5028_6611.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_6612_10811.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_10812_10883.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_10884_13211.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_13212_13283.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_13284_14975.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_14976_15767.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_15768_16391.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_16392_16547.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_16548_17963.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_17964_21347.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_21348_22895.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_22896_23243.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_23244_25631.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_25632_28271.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_28272_29087.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_29088_32519.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_32520_35039.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_35040_35207.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_35208_38579.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_38580_38699.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_38700_38891.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_38892_39635.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_39636_40151.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_40152_40367.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_40368_40919.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_40920_40979.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_40980_41075.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_41076_41351.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_41352_41435.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_41436_42779.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_42780_44471.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_44472_44867.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_44868_52775.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_52776_52835.png Gluster_Container_Storage_Storage_for_containers__in_containers_S6JGOrnh3MA_52836_52988.png
All right, we'll stay on the topic of containers. This time with Gluster, give a hand for Kaushal. Thanks. Thank you everyone. Okay, so today I'll be talking about Gluster container storage. This is that is storage for containers running in your containers, similar to what Rook is doing with Ceph. So this is what Gluster does for containers. Right, so I'm Kaushal. I'm a senior engineer at Red Hat and I work on the Gluster project. I'm a maintainer and I work mainly on the Gluster management frameworks and right now I'm contributing to the container story around Gluster. Right, so in today's talk, I'll talk about what GCS is. We'll go into a little bit more of a detail about what GCS looks like, how it performs as in how it does what it does, and I'll let you know how you can try out GCS. Okay, so just a quick question first up. So has anyone already heard of Gluster in containers with something called Hiketi? Right, so yeah, this is GCS is built on our experiences that we gained with the Hiketi project and running Gluster in containers with Hiketi. So I'll touch a little bit about Hiketi and try to compare what GCS does differently from the Hiketi-based stack in the stock, but yeah, so let's get started. So Gluster container storage, I'll be calling it GCS for short. Please don't get confused with GCS because in Kubernetes there is already GCS which stands for Google Cloud Storage, it's not Google Cloud Storage, it's Gluster Container Storage. We would welcome suggestions for a better name. So for now it's GCS, I'll be calling it as GCS. Okay, so what is GCS? GCS is a new effort from the Gluster community to provide persistent storage for containers. So we try to provide the two normal demands of container persistent storage, the shared storage and what do you say, exclusive storage, the RWX and read write many and read write one's volumes. We try to use the standard container interfaces as much as possible right now, so GCS implements one of the projects in GCS is to implement the container storage interface on CSI and using that we provide volumes when requested. For now we are Kubernetes only but maybe later on if we get help we'll move on to other container orchestrators. And GCS is being designed for a hyperconverged deployment right now. As in your storage runs along with your apps on the same servers that your apps, parts are running right. So yeah, so while the storage parts run along with your app parts on the same nodes, these are not your normal app parts, these need privileges because we do deal with underlying devices and underlying file systems and all. So we need to run in a privileged mode right. Okay, so GCS right it's not just a single, it's not a single project, it brings to the a lot of Gluster related projects and under a single umbrella. So I'll go into detail about what each of the project does for GCS specifically. So we bring in Gluster FS that we, everyone I hope everyone already knows about right. So the file system, there's Gluster E2 that's the new management tool that we have been building. There's the CSI driver specifically, there's Ant Hill, Ant Hill is a Kubernetes operator that will deploy and manage Gluster on Kubernetes and there are other projects like Gluster Prometheus, which helps monitor Gluster inside a Kubernetes deployment. So basically can plug in Gluster into your Kubernetes dashboard and you can drill down into Gluster volumes. Gluster Mixing provides the dashboard configurations, provides the alerts and rules, etc. Right, okay, so now why would you want to use GCS over what we have currently with Hecti, right? So the main theme of GCS is to simplify the whole experience as much as possible right. So we simplify how the containers get deployed, how Gluster gets deployed on Kubernetes. We simplify the general overall management workflow intended for the admins or users, right. We also go ahead and we are trying to simplify the Gluster FS stack itself. So and try to simplify the stack, the file system stack and ensure that they work good well for the container ecosystem. So what we do is that we try to make opinionated defaults, right. So right now the normal Gluster FS installation doesn't do any opinionated defaults. So it just leaves it open to the users to what you want. Everything is enabled, everything is available. So GCS is in that. So there are, we choose features that need to be enabled. We choose options that need to be defaulted and yeah, that's how GCS goes. Yeah, and all of this is going to be automated. So hopefully the most admin needs to do is to write a manifest for Anthill, deploy Anthill, write a cluster manifest on Kubernetes and done. So Anthill takes care of deploying everything that is required. So all the projects that I mentioned earlier, so Anthill takes care of deploying all of them, managing all of them. In the future, they'll also do data operations like replacing a node, upgrades and other stuff, right. And all of this together, right, allows us to improve our scale in the number of volumes at least. So Gluster FS originally wasn't designed for the container sort of environment. So Gluster FS was designed as a NAS replacement wherein you have one single large volume or a few single large volumes. So, but in containers, you generally deal with lots of very small volumes. So Gluster FS isn't particularly designed for that, but we are doing optimizations. We are trying to figure out ways so that we can scale better. Our goal is to on it, I guess this is goal for a three node cluster that is like the minimum for, if I believe, open shift that is required, that we can provide 2000 read write, what do you say, read write many volumes and up to 5000 read write volumes. That's the goal. These are like small, small volumes. These are not really huge. So something that can run in a three node cluster, right. So this is what GCC is going to provide. That is going to be different from the original stack, right. So I'm not going to talk too much about the current stack. As I mentioned, we have a Hikitty Bay stack. So the Hikitty Bay stack looks somewhat like this. So we have a component within Kubernetes itself that so we are pretty much like tied into the communities release cycle. So it's not so independent. There's a Gluster provisioner within Kubernetes. We have a separate Hikitty part, right. Hikitty manages the Gluster parts. And the thing, one big problem with Hikitty, the best problem is that we have a Hikitty DB separate with a Gluster configuration separate. So there are two separate ways of the Gluster pool. Hikitty has its own view. Gluster has its own view. And the problems that we see there are mainly about these two going out of sync and recovery from that is hard. But they have a lot of work has been done on this. And right now we are a lot of it has been solved. But still there is a problem with this. And in general, this stack relies on the normal Gluster deployment as the normal Gluster installation as in without the opinionated defaults that the new GCS stack is going to provide. Right. So yeah, there are details available below in the slide. The link is available in the schedule. If you want to, we can you can look at this. So looking at it, it looks relatively simple, but deployment and managing this cluster is challenging. It's not easy. Right. So let's get into GCS itself. Right. So GCS looks like this right now. So there are few components. We have Kubernetes. We don't need to worry about that. So we have until until is a Kubernetes operator. As I mentioned, that is going to deploy and manage the rest of these things. Right. In addition to until we have the CSI driver that is going to be the point of interaction with Kubernetes for us. So Kubernetes talks the Gluster CSI drivers Gluster CSI driver will then talk to the Gluster parts in the Gluster parts. We have the new management tool running. Plus ready to we have Gluster FS we have the Prometheus exporter inside. And we also have an XCD cluster that is a single source of truth right now. So the new management tool Gluster E2 stores the cluster view in it. So we have we GCS runs its own it's a declutter and doesn't make use of the Kubernetes at CD cluster. Right. So yeah. Okay. I talked about all of them anyway. So let's go into until first right until is a Kubernetes operator. Right. Which defines custom Gluster content GCS resource definition. So communities provides base for what do you say other projects to define their own resource types right. Object types. So until defines the GCS resource types in communities and it automates deployment upgrades and day to day management of GCS. Right. At the current state and Hill is very early. There isn't actually anything other than the CRDs. There isn't actually anything present yet. But things are working on stuff. Things will come out soon. Hill is deployed developed under the Gluster project in GitHub at GitHub.com slash cluster slash and help. Next up we have the CSI driver. So the CSI driver consists of. Okay. So as I mentioned CSI driver translates the Kubernetes CSI request into the GD2 rest request the Gluster E2 rest request. That's the main goal of these drivers. Right. So in addition to doing the translation of the request to create and delete volumes right. It also will take care of doing the mounts unlike in Hecti right. What would happen is the Gluster provisioner in the Kubernetes within Kubernetes would take care of reaching out to Gluster or reaching out to Hecti. That provisioner would take care of doing the mounts. But in this case we do everything by the CSI driver. So it gives us more freedom to do things the way we want. Right. So the CSI driver consists of three different types of parts. There's a note plugin. There's a provisioner and an attacher. Right. I'll go through what each of them does. Attach your provisioner and note plugin. Right. The attacher is for us at least in GCS world. It's a no-op. It's useless for us, but we just need to provide it because the Kubernetes CSI spec expects it. There is work going on to ensure to allow that allow CSI drivers without attaches. So not all CSI drivers actually require attaches work. So we promised it. Right. This is the main part for us right now. So this takes care of translate. This is the one part that listens for Kubernetes persistent volume requests or persistent volume claims, forwards them to Gluster. And yeah, that does it basically. It listens for the create and delete volume requests and forwards them to Gluster, translating them to the GD2 request. So in this part, we have sorry. Oh yeah, in this part, we have four different things running. So there is a Kubernetes provisioner sidecar. So these a few of them are provided by Kubernetes themselves. These are not developed by us, but yeah, so there's a Kubernetes provisioner. The Kubernetes provisioner's job is to listen for Kubernetes persistent volume object requests. And then the provisioner would talk via the CSI spec to the Gluster CSI driver, which would then relay that forward to the Gluster layer. So yeah, so there's a provisioner. We have a snapshot or sidecar which takes request handles Kubernetes requests for volume snapshots, persistent volume snapshots. And there is a cluster driver registrar which registers Gluster fscsi as a cluster level driver. So it makes the Gluster storage available as a storage class. So that's what the provisioner does. This is deployed as a right now, if I am right, a state full set so that it's always available running. And yeah, then we have the node plugin. The node plugin basically runs on every Kubernetes node that you have. Its main function is to mount the volumes when requested or attach the volumes to the app parts when requested. So yeah, that's all it does. So there's a driver registrar again, it makes the Gluster driver available on that node for mounting. So that's there are two containers, the driver registrar makes sure that the driver is registered for. So the Kubernetes knows to reach out to the driver to perform on. So if the node plugin parts also contain the Gluster fsclind bits, so you don't need to have the Gluster fsclind bits installed on the Kubernetes hosts. With the previous deployment with Hikki, you needed to you needed the client bits installed on your Kubernetes hosts. So that was one more issue with that. So yeah. So then now let's go into the Gluster part itself right. So the Gluster part has Gluster e2 Gluster fsclind itself and the Gluster fsclind exporter. So first I'll talk about Gluster e2. So Gluster e2 is the newer management framework for Gluster fsclind. It's intended to provide automated management operations for Gluster fsclind as an automatically create volumes based on request from the user. Right. So the user can request for a volume of a particular size of particular capabilities and Gluster fsclind gd2 will automatically decide upon the layout of the volume and as the user requested and create the volume. Right. In addition to that, it also tries to provide intelligent or automated volume management operations about scaling a volume or shrinking a volume replacing nodes, stuff like that, replacing bricks, stuff like that. Right. So basically what it is doing is it is converging the functionality of different tools earlier that was there outside the cluster. Right. So it was running outside the Gluster cluster, Gluster pool. Sorry. There is there was a project called Gluster block, which is which is used to provide the RWA volumes or block volumes. Right. These were running outside the Gluster pool. Right. They were not directly managed by the Gluster layer. All of this is converged into one tool right now. So Gluster e2. Gluster e2 does the work of hekt right automated volume management. And it also provides a wall also provides block volumes in a different sort of way compared to Gluster block. We'll see how it does it in a little while. Okay. So yeah. So gd2 provides a rest API like hekt right against which you can program. So this allowed us to write our CSI driver. We have an API that can be reached out to over the network to the operation. So CSI driver works on that. Gd2 also has a very good orchestration engine built in right. So it's more resilient to failures than the orchestration engine in the current cluster management layer. Right. So we can do undoes. We can do rollbacks if some operation failed. We can do an operation and say a note comes back up later. It can read rerun the operation if required stuff like that. So it brings in features to do more resilient orchestration of operations. So basically the management framework what it does at the end is that it needs to go to each node and prepare the break start processes and do stuff like that. So the gd2 brings in a much better framework to do that. In addition as I mentioned earlier there is just a single source of truth for everything. There is no split in cluster information between hekt and cluster. So all the information is with Gd2. So Gd2 knows what all volumes we have, how the volumes are laid out, what all devices we have access to stuff like that. So how many devices we have used everything right. So yeah there is less possibility of failures because of hekt and the cluster going out of sync. Right. In addition Gd2 also brings in support for Prometheus and open sensors so that you can trace or monitor Gd2 with the standard tools that are used elsewhere in the container ecosystem. Right. So Prometheus is widely used for monitoring within Kubernetes. Open sensors implements the open tracing API. So again we provide open tracing API endpoint so you can trace operations across the cluster if required. Right. Yep. So as I mentioned Gd2 provides normal volumes as well as block, cluster block. So the normal volumes are used to satisfy the Kubernetes, ad-abliax volumes or rewrite many volumes. The way how Gloucester D2 helps there is that for normal volumes there is automatic provisioning of bricks. So we go ahead and we create the LVMs, we create the thin poles, we create the file system on top of that, we do the brick mounts. We do all of that Gd2 takes care of doing that. And I mentioned one special thing that we bought in was support for volume templates. Right. So the normal Gloucester installation would contain a volume template with everything. Right. All the features that we provide. But for the GCS use case, for the continuity use case, you wouldn't need all of them. So if you had all of the features enabled in the graph, cluster graph, it would be heavy. It would be resource heavy. Right. So you don't need that. We can provide a custom simple template for ad-abliax volumes for containers. Cool. In addition, we also provide ad-ablia volumes as in we also provide block devices. Right. So block devices are meant for single users. Right. Single client access. This is still a work in progress. Bits and pieces of app it have been merged in. It's not complete yet. But yeah. What this does differently from the Gloucester block project is that this tries, does block exports in a very simple way using loop pack devices. Right. So what it basically does is that you have a cluster of this mount on your Kubernetes node and from that mount, on that mount, we create a image, I mean, device file or image file and then do a loop pack mount so that it's available as a block device. Right. The Gloucester block project did its exports via ISCSI and there was a lot of involved processing getting that set up. So that's more complex. Scaling was a problematic there. This one allows us to scale really easily, much well. So okay. So let's see the current status. Yep. So as of this moment, we can scale to again, all of these I'm talking about in a three node cluster, right? Can scale to about a thousand volumes, IWX volumes and hundreds of requests parallel as in hundreds of volume creation requests or volume decision requests happening parallel. So it doesn't bother you. We at this scale, we have begun hitting the limits of LVM scaling itself. Right. So LVM again, like Gloucester previously LVM itself wasn't designed for this sort of a thing. So Gloucester uses LVM thin pools to provide snapshot capabilities. So we use LVM. So we need LVM, but right now we are hitting on LVM scale issues. We are trying to figure out optimizations in the way we perform LVM operations itself that can help scale a bit further. And we are also looking at other options if possible for IWO volumes. Yep. Yep. In addition to the LVM scaling, we also sort of hit HCD scaling when all of these requests come together. Again, we are tuning the HCD config itself, how we deploy HCD and we are tuning how we use HCD itself. So as in how we store data in HCD, how we access HCD, right? So we are doing that. And Gloucester D2 is also, again, developed as a separate project under the Gloucester community. So it's developed under GitHub, Gloucester D2. So now let's look at what we have been doing in Gloucester FS itself to help with GCS, right? So the main focus has been optimizing resource usage. So this has been in progress from the last few releases. So Gloucester 5, one of the main topics was optimizing resource usage. Gloucester 6, the upcoming release is also, there is a lot of focus, again, on optimizing resource usage. So basically we are trying to reduce the memory that we use. So we are fixing a lot of the memory leaks that we have and everything there. We are trying to also optimize how we use threads, how we deploy it. And yeah. So we are optimizing resource usage. And again, there is this, I don't know if you have heard of multiplexing before. This multiplexing is a feature in Gloucester where we run multiple bricks in a single process, right? So again, before GCS, it wasn't such a priority. It was okay, we wouldn't have thousands of volumes running with GCS, with containers. We can't have thousands of processes running inside the pod, right? So we need to consolidate that. So yeah, we have multiplexing already for that. Multiplexing works for bricks, but there isn't a similar process for self-healing. So the volumes that we provide are redundant as in replica Gloucester volumes, Gloucester replica volumes, right? So they need to have self-healed demons running. Self-healed demons right now are individual per volume. So again, there would be as many self-healed demons as volumes that we have. So we need to multiplex the self-healed demons as well. So that's work in progress, okay? And there are other features like fencing. What is fencing? Yeah, so fencing is like a feature where we can ensure that only one client can access a file at a given time. So this is required for block volumes so that we don't have clients or say, right, say now a pod was rescheduled on some other node. We don't want them to like begin accessing the image file before some cleanup has happened or some release of blocks or whatever has happened. So things like that. So again, Gloucester FS is developed under GitHub Gloucester FS, right? Okay. Next on monitoring. Yep, so monitoring is developed under two projects. Again under Gloucester, under the Gloucester community, there's Gloucester Prometheus and there's Gloucester Mixins, right? The Gloucester Prometheus project provides us with the Gloucester FS exporter demon. This runs along with GD2 and exports the Gloucester metrics available, right? So it's not just the cluster level metrics that GD2 could provide. It's like per volume metrics, per big metrics, process metrics as in how much CPU is being used, how much disk space is being used and all of that. So the Gloucester Prometheus project provides this. And there's a Gloucester Mixins project which provides rules and alerts, right, for Prometheus and for Grafana so that you can build dashboards out of it. So right now, the Mixins projects provides a dashboard configuration for Gloucester FS volume. There are others being built. Okay. So I'm going to skip this. I was just going to talk about how the flow of operations was going to happen for a volume creation, but in simple terms, it's just straightforward, right? So it's Kubernetes to CSI driver, CSI driver to Gloucester, again the same flow. Create a volume, mount a volume. It's like one flow, single flow. It's the same for both sorts of volumes because everything is in one place. There's no split. There's no HAKET so that the request needs to go to HAKET or 10 to plus, right? Okay. So, okay, if you want to try out GCS, just keep in mind that it's still in the very heavy development. We are planning on doing a one-dot release sometime soon where, again, it's not fully featured but at least you can make use of it and just see how it works, right? So yeah, at the moment, we have an Ansible-based deployment tool written. It's not a deployment poll, it's just a playbook that deploys GCS, right? As Anthill isn't complete yet, so we are planning on Ansible right now and this playbook deploy GCS.yaml. This performs the full deployment right now, but as Anthill grows and gains more features, we'll be cutting down what the playbook does. So in the end, the hope is that the playbook would only deploy Anthill and we are done with it, right? So for more information on how to use the playbook to deploy GCS on your Kubernetes cluster, you can read the readme. So the readme gives you information on how to prepare the Ansible inventory files and how to run the playbook so that you can deploy GCS, all right? But if you want to just do a quick test on your local laptop, so we have a vagrant file available which does this. This vagrant is only for now because we do some additional adding additional tests and all that we haven't yet figured out on how to do for vagrant with virtual box. Contributions are welcome to help to that. This vagrant based tool will deploy Kubernetes and GCS for you on a three node cluster, right? So Kubernetes deployment is handled using kubespray. We have a playbook that calls kubespray finally, but this playbook has some configurations that we need to enable for GCS to run, right? So we need to enable certain feature gates to allow the CSI driver to work. So CSI driver is still a work in progress in the Kubernetes community, so not all the features are enabled out of the box, so we need to specially enable some features for the CSI drivers. Then more information on how to use this is provided in the readme, right? So this deployment tool is available under the GCS project, so there is a github.com slash blaster slash GCS. Under this we have a deployed directory where all of this is contained. So yep, and that's basically it. And as I mentioned, GCS is being developed under github plus GCS. So under this project we mainly have the overall issues tracking the full GCS, what do you say, worldview sort of a thing. And there is a waffle board which tracks the GCS related issues across the multiple different projects that I just described. So that's it. Please try out GCS and let us give us your feedback. And I'm open for questions now. Hi, do you support extending volumes? Shrink and extending volumes. So do we support extending volumes? GD2 itself supports it. The CSI driver doesn't yet, I don't know if the Kubernetes CSI spec supports it yet. So if that support comes in, we can build it. Our the cluster management layer already has support for automatic expansion. So we just need to do an extra call when that is available. And in case of three nodes deployment, is it possible that, so I suppose both three nodes should have equal storage, equal hardware. It would be preferable. It would be preferable, but okay, sorry. The question was in a three node cluster. For example, I have one server with big storage and two others with smaller one. And on that big, I store some bricks and I provide one revenue for that. And for some other volumes, I ask to have for example three replicas. Is it possible? Okay, so I'll repeat the question. The second part was, is it possible to have non-homogeneous hosts? We prefer if you have homogeneous hosts, but GD2 should be able to work with non-homogeneous hosts. The second part was if you are able to choose the, what do you say, redundancy count, the epic account. Right now, we aren't targeting that because we want to simplify the number of options that users have for us to build confidence in what we are doing first. That might come in later on. So right now, the CSI driver is hardcoded to create three replica volumes. It's hardcoded. So those three replica volumes can be created on a heterogeneous cluster as in, but the thing is, since it expects always three replicas to be present, if we can't individually provision bricks on each of the three nodes, the request will fail. So that's going to happen. Any further questions? Yes? How stable is GCS at the moment and do you support upgrades between GCS and GD2? Not really stable. It's, okay. The question is how stable is GCS right now? GCS isn't really stable. We haven't reached something that we had asked users to deploy in production yet. So the upgrade stories and all should be pretty simple, but we haven't thought about that yet. We have been just working on getting what we want to deliver in the first release fixed. So the upgrades for GCS would be simple enough because it could be just deploying the new version of the cluster parts. All our configuration right now stays in HCD. HCD itself has its own upgrade mechanisms already defined for Kubernetes, so that's taken care of. The GCS parts themselves always rely on the information in HCD. So once we reach that stage, yes, it should be simple enough to do upgrades. Yeah? How do you tend to handle doing upgrades if you have parts of mounting volumes when you have to like move mount points or something? Yep, we need to do so. Okay. The question is how do we intend to do the upgrades, right? So if parts are mounting volumes and whatnot, so as I understand, there is some sort of, I don't know, gating sort of capabilities in the Kubernetes operator world. I don't really understand the operator part of it. So the operator would take care of doing that, right? So what the operator would do would be to like bring down one particular part, right? Then get a new version of the part up and running. Do whatever mounts is required inside that till that part is up and running. It would block the upgrades of the rest of the parts, do stuff like that. So it's not exactly clear at the moment, but I believe there is some way to get operations to allow this to be handled in a smooth way. Any further questions? Nope? Thank you, everyone. Thanks, guys. Thank you.
is that is storage for containers running in your GCS is built on our experiences that we gained Storage, it's Gluster Container Storage. We would what is GCS? GCS is a new effort from the Gluster community to provide persistent storage for now, so GCS implements one of the projects in GCS deploy and manage Gluster on Kubernetes and there helps monitor Gluster inside a Kubernetes this together, right, allows us to improve our with Hikitty, the best problem is that we have a going to deploy and manage the rest of these Kubernetes provisioner. The Kubernetes the functionality of different tools earlier that hekt and the cluster going out of sync. Right. In I mentioned Gd2 provides normal volumes as well on top of that, we do the brick mounts. We do all custom simple template for ad-abliax volumes for deploys GCS, right? As Anthill isn't complete options that users have for us to build how stable is GCS right now? GCS isn't really the Kubernetes operator world. I don't really