Configure Once, Run Everywhere
Edvald, Jon
Many of the challenges developers encounter with CI pipelines stem from the fact that CI is siloed from the rest of the development process: • Because development environments are configured separately from CI (and are often pared down and simplified), devs run into hard-to-predict CI errors caused by discrepancies in environments. • Integration and end-to-end tests are commonly available only in CI, meaning that troubleshooting these test failures requires long and inefficient feedback loops—every fix requires a developer to push and wait for another CI run. In this talk, we will describe and demonstrate how to use the open source project Garden to ensure that environments are consistent and predictable at every step of the pre-production pipeline—from development, to testing, to CI. We’ll also show how to give developers the ability to run integration and end-to-end tests during the development process, making it easier to catch and fix issues when it’s most efficient.
 Hello, thanks for tuning in. My name is Dionne, I am the CEO and one of the founders of Garden. Wanted to talk today about how and why you might set about using common configuration and tooling between your development and testing in CI environments. In terms of how we will be talking about Garden, the open source project that we maintain, but even if that's not what you're here for, I still think the why can be interesting to you and also just generally how we approach it at Garden, whether Garden is useful for you or not. Just a quick few words about Garden. So we started beginning of 2018 for engineers, all pretty frustrated with the developer experience around microservices, whether it was containers or Kubernetes or something else, based here in Berlin, sitting here at the Berlin office, sadly mostly empty these days. What Garden is, we provide what we call an automation platform for cloud native development and testing. I say cloud native, it's actually used for a few other things, but we focus a lot on containers and Kubernetes, at least for the moment. It's open source, free to use. There is also a commercial product, but I won't get into that so much here. And I mentioned we were frustrated with the developer experience around microservices and what do I mean by that? I'll start the talk with some of my top dislikes because why wouldn't I? So what I really don't enjoy is having to frequently switch contexts, hop between different things that I'm working on. I really don't enjoy repeating the same thing over and over again or solving solved problems for that matter, and I also have a distaste for vagueness, both in software systems and also humans, I guess. And you might notice that a lot of these things kind of apply when you're in the process of debugging something in CI CD. So that's kind of, you know, a big part of the motivation for starting Garden in the first place. What I do like, however, is a flow state where I can just keep working on what I'm working on, focus on something that I'm actually meant to be doing without interruption, without distraction. I love smooth automation, which is really what we're all about here at Garden, and automation that just, you know, gets out of my way, does what it's supposed to do, just works. I really enjoy, again, both with systems and humans, clarity and sincerity, something that I value highly, and I've been getting into 80s Japanese synth pop, which, you know, I think you'll find actually combines all of the three other things. Anyway. So what is it that makes the current experience around like the development CI CD, development delivery cycle less than ideal? And I think it's helpful just to kind of frame this a little bit to look at. So what's a typical sort of a flow as we work on things? We typically have our, you know, the, sorry, the inner loop where we edit our code, we work on our ID, we run our unit tests as we go. We hopefully, you know, not always the case, but we hopefully have some kind of a local environment where we can actually deploy what we're working on. It could be Docker compose, it could be some mini cube Docker desktop, something like that. And then, you know, when we feel satisfied, we've done what we set out to do. We push it to get that kicks off some pipeline. And that's where, you know, we build, build the application properly. We run often more tests than we're able to do locally. Integration tests may actually happen in the kind of general PR pipeline, but we often find that it happens a little later in the process. So I kind of put it like that, but maybe this is all part of the same workflow for you. And then ultimately, you know, after we've reviewed and merged, we promote it to production. Some variant of this setup is pretty typical. And you know, obviously it can be quite, quite different based on your context, but, you know, just to have some frame to talk about here. And so what are the problems? So one problem that we see frequently is these local dev preview testing environments are either difficult to maintain. It's a little bit of extra tooling that you configure separately from everything else. Sometimes it's simply not feasible to run stuff locally. And that can be the case if you're, you know, some restricted IT environment, you're not allowed to run Docker or Kubernetes. Your project is simply too heavy. And you know, sometimes it's not just a matter of whether your laptop likes it or not. Sometimes it's just too much code to run. And but yeah, certainly it's not implemented the same way as your production deployment. Another problem is that integration and end to end tests and, you know, other types of tests as well, like, you know, security validation, like, you know, compliance checks, policy checks, things of that sort, they tend to happen quite late in the process. Rarely do they happen in the inner loop as you're working on things. Sometimes they, you know, happen in the PR process or subset of them do. But just, you know, going by our user base, our customers, we find that a lot of the time people have been doing it quite late in the process. They have like a separate staging environment that's maybe shared between a bunch of different users. And that's where integration tests happen. And in any case, the feedback from those tends to come quite late in the process, which is obviously not ideal. You know, certainly lose your flow state waiting for those. And it's, you know, extra bad if you've already merged your code before you get the results from those. But, you know, another way to deal with it is just not have integration or end to end tests, which is pretty common as well. But that means you're left with either less test coverage, or you have mocks and stubs that you also need to maintain. And they also make assumptions on your code. So it's in any case not quite ideal. As I mentioned, and you know, I will mention again, debugging failures, like, you know, in this process in CI can be incredibly tedious. And, you know, if you have a bunch of things that only happen in CI and you're not able to replicate locally, you often find yourself in this super tedious feedback loop again and again, try and get pushed, you know, until you get it fixed. And that's something that grinds my gears at least. And it's very common as well for the tooling to be all kind of separately configured across these stages. So, you know, most commonly you have at least two, two different setups. You have one for the inner loop and maybe one for the rest. But it's also common for, you know, you have one thing that's going on in CI and another thing that's going on, you know, in your staging and production environments in terms of tooling and configuration. So you have to configure everything multiple times. There's going to be friction and discrepancies between the different phases. And those can be annoying and tedious to debug. So, I know, yeah, these problems obviously just get exacerbated the more individual pipelines that you have. And you may have one per service or you may, you know, have some, usually we have multiple pipelines to deal with, hopefully not like a full local setup for each and every one of them, although we have also seen that. But the more you have to maintain of these and the more they differ in their implementation, the, you know, the bigger the problem. So how do we approach this at Garden and with Garden? What we set out to do when we designed Garden was make a tool that's sort of a hybrid between a development tool and an automation tool so that we could configure our project once and run it, run it anywhere for development, testing, CI and even production as well. So how do we go about that? Central to it is something that we call the stack graph. And so the way you configure Garden, you configure every component in your project and the graph basically, so Garden will aggregate all of that into the graph. So with Garden, you're able to codify not just how every individual component is built, deployed and tested, importantly. You're also able to define what depends on what. And this basically means that, and you know, that was the objective that you can run from a single place Garden deploy and it will deploy the whole thing or you can run Garden test and it will deploy and test the whole thing from start to finish from a bunch of Git repositories or just the one without any further interaction from the user. And the dependency graph not only gives us that but also because Garden will map, so for every same module in your system which could map to a container, for example, we know which source files map to each of them and the configuration and we will hash those together to get a version, the current version of every node in the graph. And we use that so that when you make a change, pull request or commit, whether it's a single file or multiple, we will figure out what actually that impact. So which individual node it impacts and then what's below that in the graph. So we can figure out which precise parts of the graph need to be acted upon so that you don't have to run the full battery of builds, deploys and tests for every small thing that you do, which is another common problem with, especially if you're doing integration tests in a CI pipeline. How you actually configure this, so another thing that we decided early in the process was that we wanted our configuration to be distributed much like our system. So generally every part of your project describes itself and this is how I'm deployed, I'm built, I'm deployed, I'm tested and what each of those steps depend on and you co-locate that with every part of your system, whether it's in multiple repositories or in the single one. There's a bit of YAML. I don't mind the YAML personally as long as it doesn't have like 20 levels of indentation or something like in the Kubernetes manifest oftentimes. So you can, it's a little bit more tractable. You don't have these huge files that, monolithic configuration for a distributed system, which can be quite tedious and to make that feasible, we have quite powerful templating syntax. I will show you that in a demo later and you can also define your own abstractions. You can define templates for individual or sets of modules and that's super handy like if you have skilled SREs or platform team to provide that to other developers who may not care so much about how you write a Kubernetes manifest or something like that. Or you can bake in some, this is how it should always be done for every web service, API service, so on and so forth. I'll show you more of this a little bit later. And so Garden is a CLI tool and the idea is that you can run the same commands both during development and from your CI pipelines. And it doesn't matter which CI system you use. The idea is that you can always just run GardenDeploy or GardenTest with just different environments which will trigger some changes in your templating or deploy to a different cluster. You can use the same cluster for both development and testing. But the nice thing is that if you're sharing the infrastructure, whether it's for all of Dev and CI or just for the developers separately and then the CI separately, you get a cache that is shared between all the different instances of a project. So when you run GardenDeploy, generally a developer will get his or her own namespace. And the same goes for individual tests and pipeline runs. So the idea is that when one developer has built something or tested something, another developer doesn't have to do the same. So basically shortens the time for every individual run of GardenTest or GardenDeploy or any other command for that matter. And the idea, basically, you waste less time waiting for CI, debugging CI, both because you use the same tools and same configuration and also you get the benefits of caching across the two. So here's this all kind of visualized together. So Garden being a hybrid between the development tool that you use for the interloop and the automation tool that you use in your CI CD pipelines, stretches across both. I mentioned that some people also use it to deploy to production, your mileage may vary. And when I'm developing, I can easily run GardenDeploy, get my own preview environment. That will work the same way as one if you do the same from CI, create a preview environment that other developers or QA can review and preview the changes. I can run integration and test anything else which actually run inside the environment. So those tests can address services in the environment. Those can be run before I push to get. And even if I don't run them before I push to get, if something goes wrong in CI, the tests that are run here are the same ones that I would otherwise run in my local development preview environment. So whichever individual thing goes wrong, I can more surgically run that specific thing locally and debug it much more rapidly than having to do a git commit push, save commit, push again, look at the logs in CI over and over again. So that's kind of the basic idea. I'm going to show you how it works. So we are going to use a simple example project for the demo. Lots of example projects here in the Garden Core repo. And I will take a look at a couple more as part of the demo. But we'll start with this one. It's a simple application. It's a spin off the Docker compose voting example that you might have seen at some point. We have an API backend service, Postgres database, Redis message queue, a couple of web UI's and a backend worker. It won't dig too much into the actual application, but it's a fairly basic constellation of things you might have in a project. Before we dig into the configuration, just to visualize it a little bit better, I'm going to start the Garden dashboard process, which will spin up a simple web UI that's handy for development, but certainly for demo purposes because it helps us visualize. So we can see all the different components within their descriptions, different modules that contain services and may define tasks or tests. And here you can see it in the stack graph. So the stack graph shows us all the individual steps required to go from a bunch of code and configuration to a running system, whether it's in a single repository or multiple repositories. Garden will compile this based on all the individual configs. So generally something needs to be built before it's deployed. Here you can see an example of a task in these run notes. So before we start deploying or testing services, we want to, in this case, we create a couple of database tables. This could be running a schema migration framework, loading some test data, whatever the case may be in your case. And then as you go down, you see more and more of the test notes. Unit test notes may just depend on something being built. The integration tests here depend on something being running. So the vote integration test needs the vote service to be running and so on and so forth. So hopping back to the terminal here, what I'm going to do is deploy the project. So Garden, the CLI has lots of different commands. There's GardenDeploy, GardenTest, GardenBuild. You can deploy individual services or test individual modules. You can see it here traversing the graph. And this is also visualized here in the dashboard. And you may notice here many of the services are already deployed. So Garden will check the current status. It needed to redeploy the vote service. So you can see kind of the Kubernetes status is being reported, waiting for the health check to pass. You see the individual ingress URLs that we define as part of the services. When you run this in watch mode, it also creates port forwards for you automatically, which can be handy. So now the stack is deployed. How does Garden know what all of this is? So let's go and look at the config. So starting with the project configuration. So this generally is what you put in the project route, often in the repo route as well. And this is kind of the entry point into the application. So we can see here kind project. We define the environments. So we have here one local environment for development, another remote one also for development, and then stage and prod. You can name these whatever. Basically you can define as many or a few of these as you like. Just for the sake of example, obviously this is not a production application, but define this here. And for environments, production environments or other environments that you want to protect, you can define this flag. It will change some defaults in terms of how Garden generates, like Kubernetes manifests, if you're having Garden do that. It also has confirmation steps. So if you do Garden deploy and prod, it will, unless you put another command line argument in there, it will ask you if you're sure. So that's kind of handy if just to make sure that you don't accidentally ruin something if you have access to it to begin with. So those are the environments. And we'll get back to the variables in a moment. A project also defines providers. So for the local environment, we have the local Kubernetes provider, which is made to work with like Minicube, Docker desktop, Kind, et cetera. And then we have the normal Kubernetes provider, which just connects to any Kubernetes cluster basically. And that's used for the remote stage and prod environments. The context we define here is a variable. You can see templated reference to that. And we're also directing it to use build kit to build in the cluster. There are other ways to build in cluster as well, like Kaneko, but that's kind of out of scope for this particular talk. And yeah, you can see here how we define the remote context variable. This is just pointing, yeah, all these environments are just pointing out Dev cluster. We define a user ID. You can see an example of how you can use conditionals here. Garden supports pretty complex conditionals. You can do like ternary statements and you can do all kinds of different things. You can check for equality and pick a value based on that. I can maybe find some examples of that later. And then this variable is used in a few different places here. For example, we're defining what's the base host name that we use for the ingresses for the services a little bit differently based on which environment you're in. So these variables are defined globally for all environments. And these are the overrides that you can apply for individual environments. You can also use template strings to pick a value based on which environment you're in, like environment.name equals this. And you use this value, otherwise that value. So you have a few different options there. The variables can also be placed in separate files so you don't end up with one of those massive configuration files that we talked about. So you can have one var file we call them for each environment and one for the project. You have several options there to kind of keep your configuration looking nice and clean and sane. So that's the project configuration. Now let's look at the modules. So in Garden Parliance we have the project and then we have modules. And modules can contain one of our services or tests and or tests or tasks. So here, for example, we have a pretty simple container module. There's no Docker file so we're not building it. We're just fetching based on this image field here. So it's kind of similar to Docker Compose in that regard. And it also defines this particular service. You can have multiple if you like. And this obviously is fine for just development. And I suppose it's kind of analogous to doing something in Docker Compose. And I will later show you how you can, somewhat more realistic example, which uses a Helm chart in another example project, which is basically the same project except we're using Helm. And that's maybe something that would be more realistic if you're actually creating production-like environments. Maybe a slightly more complex example here in the API service. So here we have a Docker file next to it. So we're going to build this. Again, a container module type. There's a different module types in Garden. And I will show you a couple more. Another service is defined. A few more variables here. We define a health check. So you can see some kind of Kubernetes terminology, familiar syntax if that's what you're using. You can see an example of how we interpolate a template variable into a string, which is prepending on the base host name here. There's a runtime dependency on the red as message queue. And here's perhaps the interesting part. So we define a couple of different tests. The unit tests is just a dummy test that always passes. But the integration test is an actual test that is defined here. So we just pass different arguments to run in the same container. And this test depends on the API being running. So this tells Garden not just that we need to deploy the API service first, but also if the API service changes, Garden will know to retest it. So now let's hop back to the terminal. So I've already deployed the stack. Now I'm going to run Garden test. Actually this will complete pretty quickly because I just ran this a little while ago and those tests have already passed. So there you can see how Garden will test, sorry, cache the test results. But what now if I break the test? So the actual test just expects a particular status code coming from this super basic Flask Python service here. So if I have it return some, I don't know what that status code would mean, but, and then run Garden test again. Then because it's aware, like, you know, which source files belong to which module, it will recompute this hash here that you can see in a few different places. And it knows that it needs to rebuild the API and then anything that depends on that, including the tests and other services that depend on the API service, it needs to walk through these again. So here you can see the test broke. It will also do the same for the vote integration test because it effectively tests the same thing. And you can see the log here. And you can also navigate to that here in the dashboard. So this is a pretty, of course, a kind of a basic example, but you can see this is basically how the stack graph part of Garden works. It will generate a new hash based on the actual state of the source files and the configuration attached to it. So I'm going to unbreak this. And we can move on to a couple more examples. So all the templating in this one, for example, is super straightforward. But if we go to the helm version, here you can see we've split up each of these services. So we have, for this API service, we have the API image, which notice doesn't have a services key. It just takes care of the build. And then which tests we are to run in this container. You could also define these tests in the helm chart that references this image. So a few things to cover here. So this is a helm module type. We have something called a base chart. So we're using the same helm chart that we define here for multiple services. You could also define this in line. You could just have the helm chart files here for the API service if it's not meant to be the same as the other ones. You can see here we define the runtime dependency. And here we're passing values to the helm chart. And here's a good example of how you reference across different modules. So the API image module defines some outputs. And that's defined at the module type level. And one of them is the deployment image name. So basically so that you get the correct image identifier. And this will be, it's important to reference this in a template string because this will be different based on like, are you deploying locally? Are you deploying? Is there a container registry in the mix that you need to address somehow? And this is a clean way to do that. And then the tag will be the version of the module. So that's slightly more elaborate templating. And yeah, I invite you to actually go and look at this to see how it works. Another example here, referencing an upstream helm chart, just provide the repo, the chart name version. And you can do the same with the values here. You can template in whatever makes sense to template in. I'd mentioned in the previous slide the notion of module templates. So this is super handy, especially if you have a subset of your team or organization that knows Kubernetes and the other ones, maybe the application developers are not as well-versed. You can define a template and then use the templated module type. So here, this is pretty easy to understand. It's a Kubernetes container. And this could be whatever you define. And it expects these inputs. And the way you define this template is like this. You, it's a different kind, module template. You feed it a schema, which is just a standard JSON schema. You can define that however you like. And it will create these two modules. And this could be however many you like. It will generate some files. You can even do here, we're generating Kubernetes manifests. This could also be, you know, you can generate a Docker file. You can generate whatever file. And garden will do that when this, basically when you use the templated module and reference this template, garden will generate the file. So here, for example, we have a set of Kubernetes manifests and you can see how we're referencing inputs and doing some, here's like, here's an example of how we can do nested template references. And yeah, so there's lots of different things you can do. If you want to abstract things away from developers or kind of centralize how you define Kubernetes manifests, this is one way to do it. And yeah, so lots of lots more examples here. I think it'll leave it at that for now. But by all means, if it's interesting, go and look at the examples in the repo and the documentation. Right. So just to summarize, what you get out of this is you're able to run garden deploy for any different environment and get like a production like environment. And we think that's very important because you get much higher quality feedback early in the process. You're able to run tests within the project namespace so you can actually feasibly run integration and end to end tests. And you can actually, yeah, you can feasibly write them too because if your only means of running integration tests are through your CI CD pipeline, that makes them super tedious to implement to begin with. And that's why a lot of people just don't, which I totally understand. I wouldn't want to have like a 10 minute feedback loop for every little change I make to a test. But this now makes that feasible. You get out of this super tedious loop of, you know, get commit, ASTF, ASTF and push, hope for the best, get that X in your CI pipeline, look at the logs, figure out what you think might be wrong. Hopefully you have good error logs. Maybe you don't. And then rinse repeats over and over again, never managing to stay in flow, always just kind of context switching, going for more cups of coffee and slowly losing your sanity in the process. Or at least I do. A side benefit is that your build, test and deploy configuration, actually that reminds me, wanted to share one more thing. So here is the CI pipeline configuration for this. So this is CircleCI, could be whatever. And you know, there's a little bit of boilerplate to, you know, connect CircleCI authenticated with the cluster. But the pipeline itself, the workflow is really just this. We check out the branch, we configure the cluster and then run garden test. And just point it at a different environment. And that's about it. You know, here we also run cleanup. Whenever the test is done, whether it failed or not, you might also want to keep it around if it fails so that you can debug it, whatever makes sense for your workflow. Garden Enterprise has some nice features for this. Just a tiny little plug there. But you can also use get of actions. You have a lot of control over the different events and what does what. And this command, you could be running from whichever CI runner, just pointed at a different environment and it worked just the same as when you're developing yourself locally. So again, back to the slides. And yeah, once more, like you, because you're not just building everything locally or running stuff locally, if you're running things remotely, you get the benefit of having shared build caches both through the container registry and also in the in cluster builders. And you share test caches as well because of the way garden does a versioning and hashes of all the source files and convicts. Ultimately, this means shorter wait times on CI pipelines, even though you can be doing much more in the pipelines, fewer hours spent debugging and in the tedious kind of low value plumbing work that you might be doing that isn't really what you started your project or your company to do. And you can stay focused, stay in flow and just be generally a more pleasant and happy person, at least I am. Yeah, so that's about it. To dig into garden, you can just look at our docs or our GitHub docs.garden.io is a good place to start or our website garden.io. We just launched a new community forum and we'd love to hear any thoughts, any questions whether it's before trying garden or after trying it. Anything goes, super happy to chat there. We are like anybody else of course hiring. In particular, we're looking for an SRE with some Kubernetes experience. And there you have also my email and my Twitter handle, super happy to hear from you. Just talk about the general problem space or garden or whatever else don't hesitate to ping. And that is about it. I'll hop over to the chat for the Q&A. And thank you. Okay, I think we will be live very shortly. I'll post the link. And we are perfect. Thanks folks for viewing the talk and thanks John for giving the presentation about garden and the open source project. I think that's definitely a very sweet spot of not only of Kubernetes but in total the developer experience of about any platform is really something which often lacks, especially with one of those very low level building blocks like Kubernetes, which is very kind of from a developer point is very hard to get right. So that's a really cool project you are working on. And let's see if we let a couple of minutes for people to join the separate deaf room. But maybe we can already start with like one of the questions. And I think what would be very interesting to know would be like, how do you manage the major difference between environments like cloud infrastructure that isn't used during development, for example? Well, yeah, so I guess we didn't really get into that in the demo. So one thing that you can do that I didn't mention is you can have like some modules in your project are only in one environment and not in another. So and same with like individual services, so you can toggle them on or off besides being able to configure them differently. So a good example. And I guess another thing that I did not mention, I think garden supports both Kubernetes and Terraform and potentially other things in the future. So one typical example is that something that we do ourselves for our own development as well is like having like a managed like a post like an RDS instance for Postgres or something like that. Some cloud vendor specific thing for staging and production and for like, you know, like the smaller dev test environments you have like a basic like Postgres like a femoral container instance or just use a health chart. So that's pretty common and basically being because we support Terraform, you can do almost anything like, you know, outside of Kubernetes. And that's of course not even considered like I think two projects that allow you to deploy in from Kubernetes, of course, but so you have lots of flexibility there. And yeah, the progression is usually like you have kind of simplistic environments for like interloop dev and then more realistic as you get to staging and production. But yeah, that's kind of that's how we approach it generally. And how do you deal with like one of the shared dependencies? For example, if I guess not, I believe not all environments are like self contained, for example, I believe sometimes there are like shared dependencies, which you can like only use. But instead of spinning up, is there any idea inside garden to leverage those? So what we see people do is having separate like a separate garden project for shared infrastructure. And we have designs on like more natively supporting project dependencies. So you would have like, you know, anything shared for like dev and testing and staging would be deployed in one project. And then that kind of template it into another project. People are doing it a little bit more manually now, often using workflows, which is another feature kind of where you can define different steps. Like, and that can include like setting up everything related to a project. That's kind of generally how we approach it. And you know, it's just so specific to individual applications, like, you know, are, you know, are the shared services stateless? Are you able to like run tests against the same database? Or do you have to have it in a separate database? You know, we think a lot about these things. And ideally, you have kind of tenants in your application so you can share infrastructure without tests kind of colliding in terms of the state. But sometimes you do need to deploy it just like a separate data store state and Kubernetes. That's the hardest part really. Yeah, I believe it gets to the data. It gets really complicated, right? Yeah. You see data that's that's a hard thing. Right. And isn't it kind of expensive to let to run all this kind of different environments at the same time or spend enough? Isn't that something? Yeah, I mean, I think so, like what you just mentioned, like having some shared set of services that you aren't really expected to be working on and then don't have to be redeployed for every individual environment. That's one way to tackle that. Another is so obviously you would template, you know, the different environments or like your Dev and test environments have, you know, ways smaller resource requirements usually because they're not actually not exactly dealing with a lot of traffic. They're also ephemeral. So you know, you just run garden delete environment like for anything that you haven't been running. And you know, I'm not really here to plug the commercial product, but that does have features for automatically spinning down environments. But there are also open source projects that help with like, you know, spinning down namespaces on a schedule and things like that. So they're not left lingering and, you know, being able to easily maneuver them that way is certainly helps. And lastly, so if for a pipeline or like, you know, whether it's a pipeline one or just you working on something, I just showed in the demo like garden deploy, garden test, and it will run the whole thing. But if you say like referencing the example, if I just do like garden deploy API, it will only deploy the API service and what that depends on and just ignore the rest of the graph. So you can be more specific in like which parts of the stack are deployed when you're working on it.
 Hello, thanks for tuning in.  My name is Dionne, I am the CEO and one of the founders of Garden.  Wanted to talk today about how and why you might set about using common configuration  and tooling between your development and testing in CI environments.  In terms of how we will be talking about Garden, the open source project that we maintain,  but even if that's not what you're here for, I still think the why can be interesting to  you and also just generally how we approach it at Garden, whether Garden is useful for  you or not.  Just a quick few words about Garden.  So we started beginning of 2018 for engineers, all pretty frustrated with the developer experience  around microservices, whether it was containers or Kubernetes or something else, based here  in Berlin, sitting here at the Berlin office, sadly mostly empty these days.  What Garden is, we provide what we call an automation platform for cloud native development  and testing.  I say cloud native, it's actually used for a few other things, but we focus a lot on  containers and Kubernetes, at least for the moment.  It's open source, free to use.  There is also a commercial product, but I won't get into that so much here.  And I mentioned we were frustrated with the developer experience around microservices and  what do I mean by that?  I'll start the talk with some of my top dislikes because why wouldn't I?  So what I really don't enjoy is having to frequently switch contexts, hop between different  things that I'm working on.  I really don't enjoy repeating the same thing over and over again or solving solved problems  for that matter, and I also have a distaste for vagueness, both in software systems and  also humans, I guess.  And you might notice that a lot of these things kind of apply when you're in the process of  debugging something in CI CD.  So that's kind of, you know, a big part of the motivation for starting Garden in the  first place.  What I do like, however, is a flow state where I can just keep working on what I'm working  on, focus on something that I'm actually meant to be doing without interruption, without  distraction.  I love smooth automation, which is really what we're all about here at Garden, and automation  that just, you know, gets out of my way, does what it's supposed to do, just works.  I really enjoy, again, both with systems and humans, clarity and sincerity, something that  I value highly, and I've been getting into 80s Japanese synth pop, which, you know, I  think you'll find actually combines all of the three other things.  Anyway.  So what is it that makes the current experience around like the development CI CD, development  delivery cycle less than ideal?  And I think it's helpful just to kind of frame this a little bit to look at.  So what's a typical sort of a flow as we work on things?  We typically have our, you know, the, sorry, the inner loop where we edit our code, we  work on our ID, we run our unit tests as we go.  We hopefully, you know, not always the case, but we hopefully have some kind of a local  environment where we can actually deploy what we're working on.  It could be Docker compose, it could be some mini cube Docker desktop, something like that.  And then, you know, when we feel satisfied, we've done what we set out to do.  We push it to get that kicks off some pipeline.  And that's where, you know, we build, build the application properly.  We run often more tests than we're able to do locally.  Integration tests may actually happen in the kind of general PR pipeline, but we often  find that it happens a little later in the process.  So I kind of put it like that, but maybe this is all part of the same workflow for you.  And then ultimately, you know, after we've reviewed and merged, we promote it to production.  Some variant of this setup is pretty typical.  And you know, obviously it can be quite, quite different based on your context, but, you know,  just to have some frame to talk about here.  And so what are the problems?  So one problem that we see frequently is these local dev preview testing environments are  either difficult to maintain.  It's a little bit of extra tooling that you configure separately from everything else.  Sometimes it's simply not feasible to run stuff locally.  And that can be the case if you're, you know, some restricted IT environment, you're not  allowed to run Docker or Kubernetes.  Your project is simply too heavy.  And you know, sometimes it's not just a matter of whether your laptop likes it or not.  Sometimes it's just too much code to run.  And but yeah, certainly it's not implemented the same way as your production deployment.  Another problem is that integration and end to end tests and, you know, other types of  tests as well, like, you know, security validation, like, you know, compliance checks, policy checks,  things of that sort, they tend to happen quite late in the process.  Rarely do they happen in the inner loop as you're working on things.  Sometimes they, you know, happen in the PR process or subset of them do.  But just, you know, going by our user base, our customers, we find that a lot of the time  people have been doing it quite late in the process.  They have like a separate staging environment that's maybe shared between a bunch of different  users.  And that's where integration tests happen.  And in any case, the feedback from those tends to come quite late in the process, which is  obviously not ideal.  You know, certainly lose your flow state waiting for those.  And it's, you know, extra bad if you've already merged your code before you get the results  from those.  But, you know, another way to deal with it is just not have integration or end to end  tests, which is pretty common as well.  But that means you're left with either less test coverage, or you have mocks and stubs  that you also need to maintain.  And they also make assumptions on your code.  So it's in any case not quite ideal.  As I mentioned, and you know, I will mention again, debugging failures, like, you know,  in this process in CI can be incredibly tedious.  And, you know, if you have a bunch of things that only happen in CI and you're not able  to replicate locally, you often find yourself in this super tedious feedback loop again  and again, try and get pushed, you know, until you get it fixed.  And that's something that grinds my gears at least.  And it's very common as well for the tooling to be all kind of separately configured across  these stages.  So, you know, most commonly you have at least two, two different setups.  You have one for the inner loop and maybe one for the rest.  But it's also common for, you know, you have one thing that's going on in CI and another  thing that's going on, you know, in your staging and production environments in terms  of tooling and configuration.  So you have to configure everything multiple times.  There's going to be friction and discrepancies between the different phases.  And those can be annoying and tedious to debug.  So, I know, yeah, these problems obviously just get exacerbated the more individual pipelines  that you have.  And you may have one per service or you may, you know, have some, usually we have multiple  pipelines to deal with, hopefully not like a full local setup for each and every one  of them, although we have also seen that.  But the more you have to maintain of these and the more they differ in their implementation,  the, you know, the bigger the problem.  So how do we approach this at Garden and with Garden?  What we set out to do when we designed Garden was make a tool that's sort of a hybrid between  a development tool and an automation tool so that we could configure our project once  and run it, run it anywhere for development, testing, CI and even production as well.  So how do we go about that?  Central to it is something that we call the stack graph.  And so the way you configure Garden, you configure every component in your project and the graph  basically, so Garden will aggregate all of that into the graph.  So with Garden, you're able to codify not just how every individual component is built,  deployed and tested, importantly.  You're also able to define what depends on what.  And this basically means that, and you know, that was the objective that you can run from  a single place Garden deploy and it will deploy the whole thing or you can run Garden test  and it will deploy and test the whole thing from start to finish from a bunch of Git repositories  or just the one without any further interaction from the user.  And the dependency graph not only gives us that but also because Garden will map, so  for every same module in your system which could map to a container, for example, we  know which source files map to each of them and the configuration and we will hash those  together to get a version, the current version of every node in the graph.  And we use that so that when you make a change, pull request or commit, whether it's a single  file or multiple, we will figure out what actually that impact.  So which individual node it impacts and then what's below that in the graph.  So we can figure out which precise parts of the graph need to be acted upon so that you  don't have to run the full battery of builds, deploys and tests for every small thing that  you do, which is another common problem with, especially if you're doing integration tests  in a CI pipeline.  How you actually configure this, so another thing that we decided early in the process  was that we wanted our configuration to be distributed much like our system.  So generally every part of your project describes itself and this is how I'm deployed, I'm built,  I'm deployed, I'm tested and what each of those steps depend on and you co-locate that  with every part of your system, whether it's in multiple repositories or in the single one.  There's a bit of YAML.  I don't mind the YAML personally as long as it doesn't have like 20 levels of indentation  or something like in the Kubernetes manifest oftentimes.  So you can, it's a little bit more tractable.  You don't have these huge files that, monolithic configuration for a distributed system, which  can be quite tedious and to make that feasible, we have quite powerful templating syntax.  I will show you that in a demo later and you can also define your own abstractions.  You can define templates for individual or sets of modules and that's super handy like  if you have skilled SREs or platform team to provide that to other developers who may  not care so much about how you write a Kubernetes manifest or something like that.  Or you can bake in some, this is how it should always be done for every web service, API  service, so on and so forth.  I'll show you more of this a little bit later.  And so Garden is a CLI tool and the idea is that you can run the same commands both during  development and from your CI pipelines.  And it doesn't matter which CI system you use.  The idea is that you can always just run GardenDeploy or GardenTest with just different environments  which will trigger some changes in your templating or deploy to a different cluster.  You can use the same cluster for both development and testing.  But the nice thing is that if you're sharing the infrastructure, whether it's for all of  Dev and CI or just for the developers separately and then the CI separately, you get a cache  that is shared between all the different instances of a project.  So when you run GardenDeploy, generally a developer will get his or her own namespace.  And the same goes for individual tests and pipeline runs.  So the idea is that when one developer has built something or tested something, another  developer doesn't have to do the same.  So basically shortens the time for every individual run of GardenTest or GardenDeploy or any other  command for that matter.  And the idea, basically, you waste less time waiting for CI, debugging CI, both because  you use the same tools and same configuration and also you get the benefits of caching across  the two.  So here's this all kind of visualized together.  So Garden being a hybrid between the development tool that you use for the interloop and the  automation tool that you use in your CI CD pipelines, stretches across both.  I mentioned that some people also use it to deploy to production, your mileage may vary.  And when I'm developing, I can easily run GardenDeploy, get my own preview environment.  That will work the same way as one if you do the same from CI, create a preview environment  that other developers or QA can review and preview the changes.  I can run integration and test anything else which actually run inside the environment.  So those tests can address services in the environment.  Those can be run before I push to get.  And even if I don't run them before I push to get, if something goes wrong in CI, the  tests that are run here are the same ones that I would otherwise run in my local development  preview environment.  So whichever individual thing goes wrong, I can more surgically run that specific thing  locally and debug it much more rapidly than having to do a git commit push, save commit,  push again, look at the logs in CI over and over again.  So that's kind of the basic idea.  I'm going to show you how it works.  So we are going to use a simple example project for the demo.  Lots of example projects here in the Garden Core repo.  And I will take a look at a couple more as part of the demo.  But we'll start with this one.  It's a simple application.  It's a spin off the Docker compose voting example that you might have seen at some point.  We have an API backend service, Postgres database, Redis message queue, a couple of web UI's  and a backend worker.  It won't dig too much into the actual application, but it's a fairly basic constellation of  things you might have in a project.  Before we dig into the configuration, just to visualize it a little bit better, I'm going  to start the Garden dashboard process, which will spin up a simple web UI that's handy  for development, but certainly for demo purposes because it helps us visualize.  So we can see all the different components within their descriptions, different modules  that contain services and may define tasks or tests.  And here you can see it in the stack graph.  So the stack graph shows us all the individual steps required to go from a bunch of code and  configuration to a running system, whether it's in a single repository or multiple repositories.  Garden will compile this based on all the individual configs.  So generally something needs to be built before it's deployed.  Here you can see an example of a task in these run notes.  So before we start deploying or testing services, we want to, in this case, we create a couple  of database tables.  This could be running a schema migration framework, loading some test data, whatever the case  may be in your case.  And then as you go down, you see more and more of the test notes.  Unit test notes may just depend on something being built.  The integration tests here depend on something being running.  So the vote integration test needs the vote service to be running and so on and so forth.  So hopping back to the terminal here, what I'm going to do is deploy the project.  So Garden, the CLI has lots of different commands.  There's GardenDeploy, GardenTest, GardenBuild.  You can deploy individual services or test individual modules.  You can see it here traversing the graph.  And this is also visualized here in the dashboard.  And you may notice here many of the services are already deployed.  So Garden will check the current status.  It needed to redeploy the vote service.  So you can see kind of the Kubernetes status is being reported, waiting for the health  check to pass.  You see the individual ingress URLs that we define as part of the services.  When you run this in watch mode, it also creates port forwards for you automatically, which  can be handy.  So now the stack is deployed.  How does Garden know what all of this is?  So let's go and look at the config.  So starting with the project configuration.  So this generally is what you put in the project route, often in the repo route as well.  And this is kind of the entry point into the application.  So we can see here kind project.  We define the environments.  So we have here one local environment for development, another remote one also for development,  and then stage and prod.  You can name these whatever.  Basically you can define as many or a few of these as you like.  Just for the sake of example, obviously this is not a production application, but define  this here.  And for environments, production environments or other environments that you want to protect,  you can define this flag.  It will change some defaults in terms of how Garden generates, like Kubernetes manifests,  if you're having Garden do that.  It also has confirmation steps.  So if you do Garden deploy and prod, it will, unless you put another command line argument  in there, it will ask you if you're sure.  So that's kind of handy if just to make sure that you don't accidentally ruin something  if you have access to it to begin with.  So those are the environments.  And we'll get back to the variables in a moment.  A project also defines providers.  So for the local environment, we have the local Kubernetes provider, which is made to  work with like Minicube, Docker desktop, Kind, et cetera.  And then we have the normal Kubernetes provider, which just connects to any Kubernetes cluster  basically.  And that's used for the remote stage and prod environments.  The context we define here is a variable.  You can see templated reference to that.  And we're also directing it to use build kit to build in the cluster.  There are other ways to build in cluster as well, like Kaneko, but that's kind of out  of scope for this particular talk.  And yeah, you can see here how we define the remote context variable.  This is just pointing, yeah, all these environments are just pointing out Dev cluster.  We define a user ID.  You can see an example of how you can use conditionals here.  Garden supports pretty complex conditionals.  You can do like ternary statements and you can do all kinds of different things.  You can check for equality and pick a value based on that.  I can maybe find some examples of that later.  And then this variable is used in a few different places here.  For example, we're defining what's the base host name that we use for the ingresses for  the services a little bit differently based on which environment you're in.  So these variables are defined globally for all environments.  And these are the overrides that you can apply for individual environments.  You can also use template strings to pick a value based on which environment you're in,  like environment.name equals this.  And you use this value, otherwise that value.  So you have a few different options there.  The variables can also be placed in separate files so you don't end up with one of those  massive configuration files that we talked about.  So you can have one var file we call them for each environment and one for the project.  You have several options there to kind of keep your configuration looking nice and clean  and sane.  So that's the project configuration.  Now let's look at the modules.  So in Garden Parliance we have the project and then we have modules.  And modules can contain one of our services or tests and or tests or tasks.  So here, for example, we have a pretty simple container module.  There's no Docker file so we're not building it.  We're just fetching based on this image field here.  So it's kind of similar to Docker Compose in that regard.  And it also defines this particular service.  You can have multiple if you like.  And this obviously is fine for just development.  And I suppose it's kind of analogous to doing something in Docker Compose.  And I will later show you how you can, somewhat more realistic example, which uses a Helm  chart in another example project, which is basically the same project except we're using  Helm.  And that's maybe something that would be more realistic if you're actually creating production-like  environments.  Maybe a slightly more complex example here in the API service.  So here we have a Docker file next to it.  So we're going to build this.  Again, a container module type.  There's a different module types in Garden.  And I will show you a couple more.  Another service is defined.  A few more variables here.  We define a health check.  So you can see some kind of Kubernetes terminology, familiar syntax if that's what you're using.  You can see an example of how we interpolate a template variable into a string, which is  prepending on the base host name here.  There's a runtime dependency on the red as message queue.  And here's perhaps the interesting part.  So we define a couple of different tests.  The unit tests is just a dummy test that always passes.  But the integration test is an actual test that is defined here.  So we just pass different arguments to run in the same container.  And this test depends on the API being running.  So this tells Garden not just that we need to deploy the API service first, but also  if the API service changes, Garden will know to retest it.  So now let's hop back to the terminal.  So I've already deployed the stack.  Now I'm going to run Garden test.  Actually this will complete pretty quickly because I just ran this a little while ago  and those tests have already passed.  So there you can see how Garden will test, sorry, cache the test results.  But what now if I break the test?  So the actual test just expects a particular status code coming from this super basic Flask  Python service here.  So if I have it return some, I don't know what that status code would mean, but, and  then run Garden test again.  Then because it's aware, like, you know, which source files belong to which module, it will  recompute this hash here that you can see in a few different places.  And it knows that it needs to rebuild the API and then anything that depends on that,  including the tests and other services that depend on the API service, it needs to walk  through these again.  So here you can see the test broke.  It will also do the same for the vote integration test because it effectively tests the same  thing.  And you can see the log here.  And you can also navigate to that here in the dashboard.  So this is a pretty, of course, a kind of a basic example, but you can see this is basically  how the stack graph part of Garden works.  It will generate a new hash based on the actual state of the source files and the configuration  attached to it.  So I'm going to unbreak this.  And we can move on to a couple more examples.  So all the templating in this one, for example, is super straightforward.  But if we go to the helm version, here you can see we've split up each of these services.  So we have, for this API service, we have the API image, which notice doesn't have a  services key.  It just takes care of the build.  And then which tests we are to run in this container.  You could also define these tests in the helm chart that references this image.  So a few things to cover here.  So this is a helm module type.  We have something called a base chart.  So we're using the same helm chart that we define here for multiple services.  You could also define this in line.  You could just have the helm chart files here for the API service if it's not meant to be  the same as the other ones.  You can see here we define the runtime dependency.  And here we're passing values to the helm chart.  And here's a good example of how you reference across different modules.  So the API image module defines some outputs.  And that's defined at the module type level.  And one of them is the deployment image name.  So basically so that you get the correct image identifier.  And this will be, it's important to reference this in a template string because this will  be different based on like, are you deploying locally?  Are you deploying?  Is there a container registry in the mix that you need to address somehow?  And this is a clean way to do that.  And then the tag will be the version of the module.  So that's slightly more elaborate templating.  And yeah, I invite you to actually go and look at this to see how it works.  Another example here, referencing an upstream helm chart, just provide the repo, the chart  name version.  And you can do the same with the values here.  You can template in whatever makes sense to template in.  I'd mentioned in the previous slide the notion of module templates.  So this is super handy, especially if you have a subset of your team or organization  that knows Kubernetes and the other ones, maybe the application developers are not as  well-versed.  You can define a template and then use the templated module type.  So here, this is pretty easy to understand.  It's a Kubernetes container.  And this could be whatever you define.  And it expects these inputs.  And the way you define this template is like this.  You, it's a different kind, module template.  You feed it a schema, which is just a standard JSON schema.  You can define that however you like.  And it will create these two modules.  And this could be however many you like.  It will generate some files.  You can even do here, we're generating Kubernetes manifests.  This could also be, you know, you can generate a Docker file.  You can generate whatever file.  And garden will do that when this, basically when you use the templated module and reference  this template, garden will generate the file.  So here, for example, we have a set of Kubernetes manifests and you can see how we're referencing  inputs and doing some, here's like, here's an example of how we can do nested template  references.  And yeah, so there's lots of different things you can do.  If you want to abstract things away from developers or kind of centralize how you define Kubernetes  manifests, this is one way to do it.  And yeah, so lots of lots more examples here.  I think it'll leave it at that for now.  But by all means, if it's interesting, go and look at the examples in the repo and the  documentation.  Right.  So just to summarize, what you get out of this is you're able to run garden deploy for  any different environment and get like a production like environment.  And we think that's very important because you get much higher quality feedback early  in the process.  You're able to run tests within the project namespace so you can actually feasibly run  integration and end to end tests.  And you can actually, yeah, you can feasibly write them too because if your only means  of running integration tests are through your CI CD pipeline, that makes them super tedious  to implement to begin with.  And that's why a lot of people just don't, which I totally understand.  I wouldn't want to have like a 10 minute feedback loop for every little change I make  to a test.  But this now makes that feasible.  You get out of this super tedious loop of, you know, get commit, ASTF, ASTF and push,  hope for the best, get that X in your CI pipeline, look at the logs, figure out what you think  might be wrong.  Hopefully you have good error logs.  Maybe you don't.  And then rinse repeats over and over again, never managing to stay in flow, always just  kind of context switching, going for more cups of coffee and slowly losing your sanity  in the process.  Or at least I do.  A side benefit is that your build, test and deploy configuration, actually that reminds  me, wanted to share one more thing.  So here is the CI pipeline configuration for this.  So this is CircleCI, could be whatever.  And you know, there's a little bit of boilerplate to, you know, connect CircleCI authenticated  with the cluster.  But the pipeline itself, the workflow is really just this.  We check out the branch, we configure the cluster and then run garden test.  And just point it at a different environment.  And that's about it.  You know, here we also run cleanup.  Whenever the test is done, whether it failed or not, you might also want to keep it around  if it fails so that you can debug it, whatever makes sense for your workflow.  Garden Enterprise has some nice features for this.  Just a tiny little plug there.  But you can also use get of actions.  You have a lot of control over the different events and what does what.  And this command, you could be running from whichever CI runner, just pointed at a different  environment and it worked just the same as when you're developing yourself locally.  So again, back to the slides.  And yeah, once more, like you, because you're not just building everything locally or running  stuff locally, if you're running things remotely, you get the benefit of having shared build  caches both through the container registry and also in the in cluster builders.  And you share test caches as well because of the way garden does a versioning and hashes  of all the source files and convicts.  Ultimately, this means shorter wait times on CI pipelines, even though you can be doing  much more in the pipelines, fewer hours spent debugging and in the tedious kind of low value  plumbing work that you might be doing that isn't really what you started your project  or your company to do.  And you can stay focused, stay in flow and just be generally a more pleasant and happy  person, at least I am.  Yeah, so that's about it.  To dig into garden, you can just look at our docs or our GitHub docs.garden.io is a good  place to start or our website garden.io.  We just launched a new community forum and we'd love to hear any thoughts, any questions  whether it's before trying garden or after trying it.  Anything goes, super happy to chat there.  We are like anybody else of course hiring.  In particular, we're looking for an SRE with some Kubernetes experience.  And there you have also my email and my Twitter handle, super happy to hear from you.  Just talk about the general problem space or garden or whatever else don't hesitate  to ping.  And that is about it.  I'll hop over to the chat for the Q&A.  And thank you.  Okay, I think we will be live very shortly.  I'll post the link.  And we are perfect.  Thanks folks for viewing the talk and thanks John for giving the presentation about garden  and the open source project.  I think that's definitely a very sweet spot of not only of Kubernetes but in total the  developer experience of about any platform is really something which often lacks, especially  with one of those very low level building blocks like Kubernetes, which is very kind  of from a developer point is very hard to get right.  So that's a really cool project you are working on.  And let's see if we let a couple of minutes for people to join the separate deaf room.  But maybe we can already start with like one of the questions.  And I think what would be very interesting to know would be like, how do you manage the  major difference between environments like cloud infrastructure that isn't used during  development, for example?  Well, yeah, so I guess we didn't really get into that in the demo.  So one thing that you can do that I didn't mention is you can have like some modules  in your project are only in one environment and not in another.  So and same with like individual services, so you can toggle them on or off besides being  able to configure them differently.  So a good example.  And I guess another thing that I did not mention, I think garden supports both Kubernetes and  Terraform and potentially other things in the future.  So one typical example is that something that we do ourselves for our own development as  well is like having like a managed like a post like an RDS instance for Postgres or  something like that.  Some cloud vendor specific thing for staging and production and for like, you know, like  the smaller dev test environments you have like a basic like Postgres like a femoral  container instance or just use a health chart.  So that's pretty common and basically being because we support Terraform, you can do almost  anything like, you know, outside of Kubernetes.  And that's of course not even considered like I think two projects that allow you to deploy  in from Kubernetes, of course, but so you have lots of flexibility there.  And yeah, the progression is usually like you have kind of simplistic environments for  like interloop dev and then more realistic as you get to staging and production.  But yeah, that's kind of that's how we approach it generally.  And how do you deal with like one of the shared dependencies?  For example, if I guess not, I believe not all environments are like self contained,  for example, I believe sometimes there are like shared dependencies, which you can like  only use.  But instead of spinning up, is there any idea inside garden to leverage those?  So what we see people do is having separate like a separate garden project for shared  infrastructure.  And we have designs on like more natively supporting project dependencies.  So you would have like, you know, anything shared for like dev and testing and staging  would be deployed in one project.  And then that kind of template it into another project.  People are doing it a little bit more manually now, often using workflows, which is another  feature kind of where you can define different steps.  Like, and that can include like setting up everything related to a project.  That's kind of generally how we approach it.  And you know, it's just so specific to individual applications, like, you know, are, you know,  are the shared services stateless?  Are you able to like run tests against the same database?  Or do you have to have it in a separate database?  You know, we think a lot about these things.  And ideally, you have kind of tenants in your application so you can share infrastructure  without tests kind of colliding in terms of the state.  But sometimes you do need to deploy it just like a separate data store state and Kubernetes.  That's the hardest part really.  Yeah, I believe it gets to the data.  It gets really complicated, right?  Yeah.  You see data that's that's a hard thing.  Right.  And isn't it kind of expensive to let to run all this kind of different environments at  the same time or spend enough?  Isn't that something?  Yeah, I mean, I think so, like what you just mentioned, like having some shared set of  services that you aren't really expected to be working on and then don't have to be redeployed  for every individual environment.  That's one way to tackle that.  Another is so obviously you would template, you know, the different environments or like  your Dev and test environments have, you know, ways smaller resource requirements usually  because they're not actually not exactly dealing with a lot of traffic.  They're also ephemeral.  So you know, you just run garden delete environment like for anything that you haven't been running.  And you know, I'm not really here to plug the commercial product, but that does have  features for automatically spinning down environments.  But there are also open source projects that help with like, you know, spinning down namespaces  on a schedule and things like that.  So they're not left lingering and, you know, being able to easily maneuver them that way  is certainly helps.  And lastly, so if for a pipeline or like, you know, whether it's a pipeline one or just  you working on something, I just showed in the demo like garden deploy, garden test,  and it will run the whole thing.  But if you say like referencing the example, if I just do like garden deploy API, it will  only deploy the API service and what that depends on and just ignore the rest of the  graph.  So you can be more specific in like which parts of the stack are deployed when you're  working on it.
FOSDEM2021Configure_Once_Run_Everywhere_0_179.png FOSDEM2021Configure_Once_Run_Everywhere_180_14159.png FOSDEM2021Configure_Once_Run_Everywhere_14160_19247.png FOSDEM2021Configure_Once_Run_Everywhere_19248_24047.png FOSDEM2021Configure_Once_Run_Everywhere_24048_24299.png FOSDEM2021Configure_Once_Run_Everywhere_24300_25319.png FOSDEM2021Configure_Once_Run_Everywhere_25320_25967.png FOSDEM2021Configure_Once_Run_Everywhere_25968_27647.png FOSDEM2021Configure_Once_Run_Everywhere_27648_28115.png FOSDEM2021Configure_Once_Run_Everywhere_28116_29231.png FOSDEM2021Configure_Once_Run_Everywhere_29232_31571.png FOSDEM2021Configure_Once_Run_Everywhere_31572_32147.png FOSDEM2021Configure_Once_Run_Everywhere_32148_32687.png FOSDEM2021Configure_Once_Run_Everywhere_32688_33167.png FOSDEM2021Configure_Once_Run_Everywhere_33168_33347.png FOSDEM2021Configure_Once_Run_Everywhere_33348_33695.png FOSDEM2021Configure_Once_Run_Everywhere_33696_34295.png FOSDEM2021Configure_Once_Run_Everywhere_34296_35399.png FOSDEM2021Configure_Once_Run_Everywhere_35400_35975.png FOSDEM2021Configure_Once_Run_Everywhere_35976_36215.png FOSDEM2021Configure_Once_Run_Everywhere_36216_36287.png FOSDEM2021Configure_Once_Run_Everywhere_36288_36491.png FOSDEM2021Configure_Once_Run_Everywhere_36492_36827.png FOSDEM2021Configure_Once_Run_Everywhere_36828_37403.png FOSDEM2021Configure_Once_Run_Everywhere_37404_38399.png FOSDEM2021Configure_Once_Run_Everywhere_38400_39179.png FOSDEM2021Configure_Once_Run_Everywhere_39180_39779.png FOSDEM2021Configure_Once_Run_Everywhere_39780_40007.png FOSDEM2021Configure_Once_Run_Everywhere_40008_40415.png FOSDEM2021Configure_Once_Run_Everywhere_40416_40523.png FOSDEM2021Configure_Once_Run_Everywhere_40524_40679.png FOSDEM2021Configure_Once_Run_Everywhere_40680_40847.png FOSDEM2021Configure_Once_Run_Everywhere_40848_40907.png FOSDEM2021Configure_Once_Run_Everywhere_40908_41159.png FOSDEM2021Configure_Once_Run_Everywhere_41160_44123.png FOSDEM2021Configure_Once_Run_Everywhere_44124_45719.png FOSDEM2021Configure_Once_Run_Everywhere_45720_47135.png FOSDEM2021Configure_Once_Run_Everywhere_47136_47711.png FOSDEM2021Configure_Once_Run_Everywhere_47712_50363.png FOSDEM2021Configure_Once_Run_Everywhere_50364_50435.png FOSDEM2021Configure_Once_Run_Everywhere_50436_50639.png FOSDEM2021Configure_Once_Run_Everywhere_50640_50711.png FOSDEM2021Configure_Once_Run_Everywhere_50712_52235.png FOSDEM2021Configure_Once_Run_Everywhere_52236_55199.png FOSDEM2021Configure_Once_Run_Everywhere_55200_61223.png FOSDEM2021Configure_Once_Run_Everywhere_61224_65994.png
Hello, thanks for tuning in. My name is Dionne, I am the CEO and one of the founders of Garden. Wanted to talk today about how and why you might set about using common configuration and tooling between your development and testing in CI environments. In terms of how we will be talking about Garden, the open source project that we maintain, but even if that's not what you're here for, I still think the why can be interesting to you and also just generally how we approach it at Garden, whether Garden is useful for you or not. Just a quick few words about Garden. So we started beginning of 2018 for engineers, all pretty frustrated with the developer experience around microservices, whether it was containers or Kubernetes or something else, based here in Berlin, sitting here at the Berlin office, sadly mostly empty these days. What Garden is, we provide what we call an automation platform for cloud native development and testing. I say cloud native, it's actually used for a few other things, but we focus a lot on containers and Kubernetes, at least for the moment. It's open source, free to use. There is also a commercial product, but I won't get into that so much here. And I mentioned we were frustrated with the developer experience around microservices and what do I mean by that? I'll start the talk with some of my top dislikes because why wouldn't I? So what I really don't enjoy is having to frequently switch contexts, hop between different things that I'm working on. I really don't enjoy repeating the same thing over and over again or solving solved problems for that matter, and I also have a distaste for vagueness, both in software systems and also humans, I guess. And you might notice that a lot of these things kind of apply when you're in the process of debugging something in CI CD. So that's kind of, you know, a big part of the motivation for starting Garden in the first place. What I do like, however, is a flow state where I can just keep working on what I'm working on, focus on something that I'm actually meant to be doing without interruption, without distraction. I love smooth automation, which is really what we're all about here at Garden, and automation that just, you know, gets out of my way, does what it's supposed to do, just works. I really enjoy, again, both with systems and humans, clarity and sincerity, something that I value highly, and I've been getting into 80s Japanese synth pop, which, you know, I think you'll find actually combines all of the three other things. Anyway. So what is it that makes the current experience around like the development CI CD, development delivery cycle less than ideal? And I think it's helpful just to kind of frame this a little bit to look at. So what's a typical sort of a flow as we work on things? We typically have our, you know, the, sorry, the inner loop where we edit our code, we work on our ID, we run our unit tests as we go. We hopefully, you know, not always the case, but we hopefully have some kind of a local environment where we can actually deploy what we're working on. It could be Docker compose, it could be some mini cube Docker desktop, something like that. And then, you know, when we feel satisfied, we've done what we set out to do. We push it to get that kicks off some pipeline. And that's where, you know, we build, build the application properly. We run often more tests than we're able to do locally. Integration tests may actually happen in the kind of general PR pipeline, but we often find that it happens a little later in the process. So I kind of put it like that, but maybe this is all part of the same workflow for you. And then ultimately, you know, after we've reviewed and merged, we promote it to production. Some variant of this setup is pretty typical. And you know, obviously it can be quite, quite different based on your context, but, you know, just to have some frame to talk about here. And so what are the problems? So one problem that we see frequently is these local dev preview testing environments are either difficult to maintain. It's a little bit of extra tooling that you configure separately from everything else. Sometimes it's simply not feasible to run stuff locally. And that can be the case if you're, you know, some restricted IT environment, you're not allowed to run Docker or Kubernetes. Your project is simply too heavy. And you know, sometimes it's not just a matter of whether your laptop likes it or not. Sometimes it's just too much code to run. And but yeah, certainly it's not implemented the same way as your production deployment. Another problem is that integration and end to end tests and, you know, other types of tests as well, like, you know, security validation, like, you know, compliance checks, policy checks, things of that sort, they tend to happen quite late in the process. Rarely do they happen in the inner loop as you're working on things. Sometimes they, you know, happen in the PR process or subset of them do. But just, you know, going by our user base, our customers, we find that a lot of the time people have been doing it quite late in the process. They have like a separate staging environment that's maybe shared between a bunch of different users. And that's where integration tests happen. And in any case, the feedback from those tends to come quite late in the process, which is obviously not ideal. You know, certainly lose your flow state waiting for those. And it's, you know, extra bad if you've already merged your code before you get the results from those. But, you know, another way to deal with it is just not have integration or end to end tests, which is pretty common as well. But that means you're left with either less test coverage, or you have mocks and stubs that you also need to maintain. And they also make assumptions on your code. So it's in any case not quite ideal. As I mentioned, and you know, I will mention again, debugging failures, like, you know, in this process in CI can be incredibly tedious. And, you know, if you have a bunch of things that only happen in CI and you're not able to replicate locally, you often find yourself in this super tedious feedback loop again and again, try and get pushed, you know, until you get it fixed. And that's something that grinds my gears at least. And it's very common as well for the tooling to be all kind of separately configured across these stages. So, you know, most commonly you have at least two, two different setups. You have one for the inner loop and maybe one for the rest. But it's also common for, you know, you have one thing that's going on in CI and another thing that's going on, you know, in your staging and production environments in terms of tooling and configuration. So you have to configure everything multiple times. There's going to be friction and discrepancies between the different phases. And those can be annoying and tedious to debug. So, I know, yeah, these problems obviously just get exacerbated the more individual pipelines that you have. And you may have one per service or you may, you know, have some, usually we have multiple pipelines to deal with, hopefully not like a full local setup for each and every one of them, although we have also seen that. But the more you have to maintain of these and the more they differ in their implementation, the, you know, the bigger the problem. So how do we approach this at Garden and with Garden? What we set out to do when we designed Garden was make a tool that's sort of a hybrid between a development tool and an automation tool so that we could configure our project once and run it, run it anywhere for development, testing, CI and even production as well. So how do we go about that? Central to it is something that we call the stack graph. And so the way you configure Garden, you configure every component in your project and the graph basically, so Garden will aggregate all of that into the graph. So with Garden, you're able to codify not just how every individual component is built, deployed and tested, importantly. You're also able to define what depends on what. And this basically means that, and you know, that was the objective that you can run from a single place Garden deploy and it will deploy the whole thing or you can run Garden test and it will deploy and test the whole thing from start to finish from a bunch of Git repositories or just the one without any further interaction from the user. And the dependency graph not only gives us that but also because Garden will map, so for every same module in your system which could map to a container, for example, we know which source files map to each of them and the configuration and we will hash those together to get a version, the current version of every node in the graph. And we use that so that when you make a change, pull request or commit, whether it's a single file or multiple, we will figure out what actually that impact. So which individual node it impacts and then what's below that in the graph. So we can figure out which precise parts of the graph need to be acted upon so that you don't have to run the full battery of builds, deploys and tests for every small thing that you do, which is another common problem with, especially if you're doing integration tests in a CI pipeline. How you actually configure this, so another thing that we decided early in the process was that we wanted our configuration to be distributed much like our system. So generally every part of your project describes itself and this is how I'm deployed, I'm built, I'm deployed, I'm tested and what each of those steps depend on and you co-locate that with every part of your system, whether it's in multiple repositories or in the single one. There's a bit of YAML. I don't mind the YAML personally as long as it doesn't have like 20 levels of indentation or something like in the Kubernetes manifest oftentimes. So you can, it's a little bit more tractable. You don't have these huge files that, monolithic configuration for a distributed system, which can be quite tedious and to make that feasible, we have quite powerful templating syntax. I will show you that in a demo later and you can also define your own abstractions. You can define templates for individual or sets of modules and that's super handy like if you have skilled SREs or platform team to provide that to other developers who may not care so much about how you write a Kubernetes manifest or something like that. Or you can bake in some, this is how it should always be done for every web service, API service, so on and so forth. I'll show you more of this a little bit later. And so Garden is a CLI tool and the idea is that you can run the same commands both during development and from your CI pipelines. And it doesn't matter which CI system you use. The idea is that you can always just run GardenDeploy or GardenTest with just different environments which will trigger some changes in your templating or deploy to a different cluster. You can use the same cluster for both development and testing. But the nice thing is that if you're sharing the infrastructure, whether it's for all of Dev and CI or just for the developers separately and then the CI separately, you get a cache that is shared between all the different instances of a project. So when you run GardenDeploy, generally a developer will get his or her own namespace. And the same goes for individual tests and pipeline runs. So the idea is that when one developer has built something or tested something, another developer doesn't have to do the same. So basically shortens the time for every individual run of GardenTest or GardenDeploy or any other command for that matter. And the idea, basically, you waste less time waiting for CI, debugging CI, both because you use the same tools and same configuration and also you get the benefits of caching across the two. So here's this all kind of visualized together. So Garden being a hybrid between the development tool that you use for the interloop and the automation tool that you use in your CI CD pipelines, stretches across both. I mentioned that some people also use it to deploy to production, your mileage may vary. And when I'm developing, I can easily run GardenDeploy, get my own preview environment. That will work the same way as one if you do the same from CI, create a preview environment that other developers or QA can review and preview the changes. I can run integration and test anything else which actually run inside the environment. So those tests can address services in the environment. Those can be run before I push to get. And even if I don't run them before I push to get, if something goes wrong in CI, the tests that are run here are the same ones that I would otherwise run in my local development preview environment. So whichever individual thing goes wrong, I can more surgically run that specific thing locally and debug it much more rapidly than having to do a git commit push, save commit, push again, look at the logs in CI over and over again. So that's kind of the basic idea. I'm going to show you how it works. So we are going to use a simple example project for the demo. Lots of example projects here in the Garden Core repo. And I will take a look at a couple more as part of the demo. But we'll start with this one. It's a simple application. It's a spin off the Docker compose voting example that you might have seen at some point. We have an API backend service, Postgres database, Redis message queue, a couple of web UI's and a backend worker. It won't dig too much into the actual application, but it's a fairly basic constellation of things you might have in a project. Before we dig into the configuration, just to visualize it a little bit better, I'm going to start the Garden dashboard process, which will spin up a simple web UI that's handy for development, but certainly for demo purposes because it helps us visualize. So we can see all the different components within their descriptions, different modules that contain services and may define tasks or tests. And here you can see it in the stack graph. So the stack graph shows us all the individual steps required to go from a bunch of code and configuration to a running system, whether it's in a single repository or multiple repositories. Garden will compile this based on all the individual configs. So generally something needs to be built before it's deployed. Here you can see an example of a task in these run notes. So before we start deploying or testing services, we want to, in this case, we create a couple of database tables. This could be running a schema migration framework, loading some test data, whatever the case may be in your case. And then as you go down, you see more and more of the test notes. Unit test notes may just depend on something being built. The integration tests here depend on something being running. So the vote integration test needs the vote service to be running and so on and so forth. So hopping back to the terminal here, what I'm going to do is deploy the project. So Garden, the CLI has lots of different commands. There's GardenDeploy, GardenTest, GardenBuild. You can deploy individual services or test individual modules. You can see it here traversing the graph. And this is also visualized here in the dashboard. And you may notice here many of the services are already deployed. So Garden will check the current status. It needed to redeploy the vote service. So you can see kind of the Kubernetes status is being reported, waiting for the health check to pass. You see the individual ingress URLs that we define as part of the services. When you run this in watch mode, it also creates port forwards for you automatically, which can be handy. So now the stack is deployed. How does Garden know what all of this is? So let's go and look at the config. So starting with the project configuration. So this generally is what you put in the project route, often in the repo route as well. And this is kind of the entry point into the application. So we can see here kind project. We define the environments. So we have here one local environment for development, another remote one also for development, and then stage and prod. You can name these whatever. Basically you can define as many or a few of these as you like. Just for the sake of example, obviously this is not a production application, but define this here. And for environments, production environments or other environments that you want to protect, you can define this flag. It will change some defaults in terms of how Garden generates, like Kubernetes manifests, if you're having Garden do that. It also has confirmation steps. So if you do Garden deploy and prod, it will, unless you put another command line argument in there, it will ask you if you're sure. So that's kind of handy if just to make sure that you don't accidentally ruin something if you have access to it to begin with. So those are the environments. And we'll get back to the variables in a moment. A project also defines providers. So for the local environment, we have the local Kubernetes provider, which is made to work with like Minicube, Docker desktop, Kind, et cetera. And then we have the normal Kubernetes provider, which just connects to any Kubernetes cluster basically. And that's used for the remote stage and prod environments. The context we define here is a variable. You can see templated reference to that. And we're also directing it to use build kit to build in the cluster. There are other ways to build in cluster as well, like Kaneko, but that's kind of out of scope for this particular talk. And yeah, you can see here how we define the remote context variable. This is just pointing, yeah, all these environments are just pointing out Dev cluster. We define a user ID. You can see an example of how you can use conditionals here. Garden supports pretty complex conditionals. You can do like ternary statements and you can do all kinds of different things. You can check for equality and pick a value based on that. I can maybe find some examples of that later. And then this variable is used in a few different places here. For example, we're defining what's the base host name that we use for the ingresses for the services a little bit differently based on which environment you're in. So these variables are defined globally for all environments. And these are the overrides that you can apply for individual environments. You can also use template strings to pick a value based on which environment you're in, like environment.name equals this. And you use this value, otherwise that value. So you have a few different options there. The variables can also be placed in separate files so you don't end up with one of those massive configuration files that we talked about. So you can have one var file we call them for each environment and one for the project. You have several options there to kind of keep your configuration looking nice and clean and sane. So that's the project configuration. Now let's look at the modules. So in Garden Parliance we have the project and then we have modules. And modules can contain one of our services or tests and or tests or tasks. So here, for example, we have a pretty simple container module. There's no Docker file so we're not building it. We're just fetching based on this image field here. So it's kind of similar to Docker Compose in that regard. And it also defines this particular service. You can have multiple if you like. And this obviously is fine for just development. And I suppose it's kind of analogous to doing something in Docker Compose. And I will later show you how you can, somewhat more realistic example, which uses a Helm chart in another example project, which is basically the same project except we're using Helm. And that's maybe something that would be more realistic if you're actually creating production-like environments. Maybe a slightly more complex example here in the API service. So here we have a Docker file next to it. So we're going to build this. Again, a container module type. There's a different module types in Garden. And I will show you a couple more. Another service is defined. A few more variables here. We define a health check. So you can see some kind of Kubernetes terminology, familiar syntax if that's what you're using. You can see an example of how we interpolate a template variable into a string, which is prepending on the base host name here. There's a runtime dependency on the red as message queue. And here's perhaps the interesting part. So we define a couple of different tests. The unit tests is just a dummy test that always passes. But the integration test is an actual test that is defined here. So we just pass different arguments to run in the same container. And this test depends on the API being running. So this tells Garden not just that we need to deploy the API service first, but also if the API service changes, Garden will know to retest it. So now let's hop back to the terminal. So I've already deployed the stack. Now I'm going to run Garden test. Actually this will complete pretty quickly because I just ran this a little while ago and those tests have already passed. So there you can see how Garden will test, sorry, cache the test results. But what now if I break the test? So the actual test just expects a particular status code coming from this super basic Flask Python service here. So if I have it return some, I don't know what that status code would mean, but, and then run Garden test again. Then because it's aware, like, you know, which source files belong to which module, it will recompute this hash here that you can see in a few different places. And it knows that it needs to rebuild the API and then anything that depends on that, including the tests and other services that depend on the API service, it needs to walk through these again. So here you can see the test broke. It will also do the same for the vote integration test because it effectively tests the same thing. And you can see the log here. And you can also navigate to that here in the dashboard. So this is a pretty, of course, a kind of a basic example, but you can see this is basically how the stack graph part of Garden works. It will generate a new hash based on the actual state of the source files and the configuration attached to it. So I'm going to unbreak this. And we can move on to a couple more examples. So all the templating in this one, for example, is super straightforward. But if we go to the helm version, here you can see we've split up each of these services. So we have, for this API service, we have the API image, which notice doesn't have a services key. It just takes care of the build. And then which tests we are to run in this container. You could also define these tests in the helm chart that references this image. So a few things to cover here. So this is a helm module type. We have something called a base chart. So we're using the same helm chart that we define here for multiple services. You could also define this in line. You could just have the helm chart files here for the API service if it's not meant to be the same as the other ones. You can see here we define the runtime dependency. And here we're passing values to the helm chart. And here's a good example of how you reference across different modules. So the API image module defines some outputs. And that's defined at the module type level. And one of them is the deployment image name. So basically so that you get the correct image identifier. And this will be, it's important to reference this in a template string because this will be different based on like, are you deploying locally? Are you deploying? Is there a container registry in the mix that you need to address somehow? And this is a clean way to do that. And then the tag will be the version of the module. So that's slightly more elaborate templating. And yeah, I invite you to actually go and look at this to see how it works. Another example here, referencing an upstream helm chart, just provide the repo, the chart name version. And you can do the same with the values here. You can template in whatever makes sense to template in. I'd mentioned in the previous slide the notion of module templates. So this is super handy, especially if you have a subset of your team or organization that knows Kubernetes and the other ones, maybe the application developers are not as well-versed. You can define a template and then use the templated module type. So here, this is pretty easy to understand. It's a Kubernetes container. And this could be whatever you define. And it expects these inputs. And the way you define this template is like this. You, it's a different kind, module template. You feed it a schema, which is just a standard JSON schema. You can define that however you like. And it will create these two modules. And this could be however many you like. It will generate some files. You can even do here, we're generating Kubernetes manifests. This could also be, you know, you can generate a Docker file. You can generate whatever file. And garden will do that when this, basically when you use the templated module and reference this template, garden will generate the file. So here, for example, we have a set of Kubernetes manifests and you can see how we're referencing inputs and doing some, here's like, here's an example of how we can do nested template references. And yeah, so there's lots of different things you can do. If you want to abstract things away from developers or kind of centralize how you define Kubernetes manifests, this is one way to do it. And yeah, so lots of lots more examples here. I think it'll leave it at that for now. But by all means, if it's interesting, go and look at the examples in the repo and the documentation. Right. So just to summarize, what you get out of this is you're able to run garden deploy for any different environment and get like a production like environment. And we think that's very important because you get much higher quality feedback early in the process. You're able to run tests within the project namespace so you can actually feasibly run integration and end to end tests. And you can actually, yeah, you can feasibly write them too because if your only means of running integration tests are through your CI CD pipeline, that makes them super tedious to implement to begin with. And that's why a lot of people just don't, which I totally understand. I wouldn't want to have like a 10 minute feedback loop for every little change I make to a test. But this now makes that feasible. You get out of this super tedious loop of, you know, get commit, ASTF, ASTF and push, hope for the best, get that X in your CI pipeline, look at the logs, figure out what you think might be wrong. Hopefully you have good error logs. Maybe you don't. And then rinse repeats over and over again, never managing to stay in flow, always just kind of context switching, going for more cups of coffee and slowly losing your sanity in the process. Or at least I do. A side benefit is that your build, test and deploy configuration, actually that reminds me, wanted to share one more thing. So here is the CI pipeline configuration for this. So this is CircleCI, could be whatever. And you know, there's a little bit of boilerplate to, you know, connect CircleCI authenticated with the cluster. But the pipeline itself, the workflow is really just this. We check out the branch, we configure the cluster and then run garden test. And just point it at a different environment. And that's about it. You know, here we also run cleanup. Whenever the test is done, whether it failed or not, you might also want to keep it around if it fails so that you can debug it, whatever makes sense for your workflow. Garden Enterprise has some nice features for this. Just a tiny little plug there. But you can also use get of actions. You have a lot of control over the different events and what does what. And this command, you could be running from whichever CI runner, just pointed at a different environment and it worked just the same as when you're developing yourself locally. So again, back to the slides. And yeah, once more, like you, because you're not just building everything locally or running stuff locally, if you're running things remotely, you get the benefit of having shared build caches both through the container registry and also in the in cluster builders. And you share test caches as well because of the way garden does a versioning and hashes of all the source files and convicts. Ultimately, this means shorter wait times on CI pipelines, even though you can be doing much more in the pipelines, fewer hours spent debugging and in the tedious kind of low value plumbing work that you might be doing that isn't really what you started your project or your company to do. And you can stay focused, stay in flow and just be generally a more pleasant and happy person, at least I am. Yeah, so that's about it. To dig into garden, you can just look at our docs or our GitHub docs.garden.io is a good place to start or our website garden.io. We just launched a new community forum and we'd love to hear any thoughts, any questions whether it's before trying garden or after trying it. Anything goes, super happy to chat there. We are like anybody else of course hiring. In particular, we're looking for an SRE with some Kubernetes experience. And there you have also my email and my Twitter handle, super happy to hear from you. Just talk about the general problem space or garden or whatever else don't hesitate to ping. And that is about it. I'll hop over to the chat for the Q&A. And thank you. Okay, I think we will be live very shortly. I'll post the link. And we are perfect. Thanks folks for viewing the talk and thanks John for giving the presentation about garden and the open source project. I think that's definitely a very sweet spot of not only of Kubernetes but in total the developer experience of about any platform is really something which often lacks, especially with one of those very low level building blocks like Kubernetes, which is very kind of from a developer point is very hard to get right. So that's a really cool project you are working on. And let's see if we let a couple of minutes for people to join the separate deaf room. But maybe we can already start with like one of the questions. And I think what would be very interesting to know would be like, how do you manage the major difference between environments like cloud infrastructure that isn't used during development, for example? Well, yeah, so I guess we didn't really get into that in the demo. So one thing that you can do that I didn't mention is you can have like some modules in your project are only in one environment and not in another. So and same with like individual services, so you can toggle them on or off besides being able to configure them differently. So a good example. And I guess another thing that I did not mention, I think garden supports both Kubernetes and Terraform and potentially other things in the future. So one typical example is that something that we do ourselves for our own development as well is like having like a managed like a post like an RDS instance for Postgres or something like that. Some cloud vendor specific thing for staging and production and for like, you know, like the smaller dev test environments you have like a basic like Postgres like a femoral container instance or just use a health chart. So that's pretty common and basically being because we support Terraform, you can do almost anything like, you know, outside of Kubernetes. And that's of course not even considered like I think two projects that allow you to deploy in from Kubernetes, of course, but so you have lots of flexibility there. And yeah, the progression is usually like you have kind of simplistic environments for like interloop dev and then more realistic as you get to staging and production. But yeah, that's kind of that's how we approach it generally. And how do you deal with like one of the shared dependencies? For example, if I guess not, I believe not all environments are like self contained, for example, I believe sometimes there are like shared dependencies, which you can like only use. But instead of spinning up, is there any idea inside garden to leverage those? So what we see people do is having separate like a separate garden project for shared infrastructure. And we have designs on like more natively supporting project dependencies. So you would have like, you know, anything shared for like dev and testing and staging would be deployed in one project. And then that kind of template it into another project. People are doing it a little bit more manually now, often using workflows, which is another feature kind of where you can define different steps. Like, and that can include like setting up everything related to a project. That's kind of generally how we approach it. And you know, it's just so specific to individual applications, like, you know, are, you know, are the shared services stateless? Are you able to like run tests against the same database? Or do you have to have it in a separate database? You know, we think a lot about these things. And ideally, you have kind of tenants in your application so you can share infrastructure without tests kind of colliding in terms of the state. But sometimes you do need to deploy it just like a separate data store state and Kubernetes. That's the hardest part really. Yeah, I believe it gets to the data. It gets really complicated, right? Yeah. You see data that's that's a hard thing. Right. And isn't it kind of expensive to let to run all this kind of different environments at the same time or spend enough? Isn't that something? Yeah, I mean, I think so, like what you just mentioned, like having some shared set of services that you aren't really expected to be working on and then don't have to be redeployed for every individual environment. That's one way to tackle that. Another is so obviously you would template, you know, the different environments or like your Dev and test environments have, you know, ways smaller resource requirements usually because they're not actually not exactly dealing with a lot of traffic. They're also ephemeral. So you know, you just run garden delete environment like for anything that you haven't been running. And you know, I'm not really here to plug the commercial product, but that does have features for automatically spinning down environments. But there are also open source projects that help with like, you know, spinning down namespaces on a schedule and things like that. So they're not left lingering and, you know, being able to easily maneuver them that way is certainly helps. And lastly, so if for a pipeline or like, you know, whether it's a pipeline one or just you working on something, I just showed in the demo like garden deploy, garden test, and it will run the whole thing. But if you say like referencing the example, if I just do like garden deploy API, it will only deploy the API service and what that depends on and just ignore the rest of the graph. So you can be more specific in like which parts of the stack are deployed when you're working on it.
environments. In terms of how we will be talking about Garden, the open source project that we source, free to use. There is also a commercial testing environments are either difficult to you configure separately from everything else. Sometimes it's simply not feasible to run stuff failures, like, you know, in this process in CI configured across these stages. So, you know, most commonly you have at least two, two the way you configure Garden, you configure every between the development tool that you use for the basically when you use the templated module and feasibly run integration and end to end tests. are only in one environment and not in another. and test environments have, you know, ways with like, you know, spinning down namespaces on depends on and just ignore the rest of the graph. it.