How We Modified an Agent-based Automation System to Become an Agentless One
Wilhite, Megan
Salt initially was designed to be an agent-base automation system. Salt SSH was added later on to allow an alternative that does not require installing a agent, and instead communicate over SSH. We will discuss how we modified the initial agent-based solution and the architectural decisions behind this work. We will talk through some of the challenges, lessons learned and what worked well for the project.
 Yeah, so today I'm going to be talking about how at Salt we moved from an agent-based solution to being able to use Salt without installing an agent. So yeah, my name is Megan Wilhite. I'm known as Shell on the Interwebs. Bonus points to ever understand what Shell is. And I'm currently a software engineer at Saltstack, but I am working mostly on getting releases out right now. So that's me. So what is Salt? For anybody that doesn't know what Salt is, I think it's important to understand what Salt is before we go on and talking about Salt as a state, which is the agent-less solution. So Salt is a lot of things, but I would describe it as being able to do things such as config management, remote execution, automation, and orchestration. And it's very event-driven. So talking about how Salt is currently set up with the agent solution, it is a server agent communication model. So we refer to the server as the saltmaster, and we refer to the agent as the salt minion. And the way that you would set this up is you would install the master, and then you would install the minion. And there are a couple of network services that provide the transport layer within Salt, and we refer to them as the publisher and the request server. And these are using a Xeron Q's library, and the publisher essentially is sending out all the commands from the master to the minions, and the minions then determine from there whether they should actually run the command, and then we have the request server, and this is where we get a lot of the bi-directional communication between the master and the minion. So when talking about converting over to an agent-less solution, obviously SSH is unidirectional, so we had to be careful in terms of how we architected that so that we could use the power of Salt, but also use SSH. So I'm just going to show you a quick command within Salt that you can run. It's just kind of like our version of FlowWorld. It's just Salt, which is the name of the CLI tool that you would use, targeting whatever minion you want to target, and then TestStopPing is just what we refer to as an execution module. Execution modules are not meant to be item potent. They're more meant to be run as one-off commands. And so, yeah, it would go through that job flow, as I mentioned before. The master would publish that command of the minion. The minion would then run it and then return it through the request server. So as you can see through the CLI, maybe you can't see that super well, but essentially all it does is it runs TestStopPing on the minion and then returns true if it's able to communicate with the minion. Okay, another important thing when talking about how we architected Salt SSH is you need to understand the difference between Salt and SaltCall. So Salt, as we show here, we're targeting all minions with the ashras there and TestStopPing. That is the CLI tool you use on the master. And SaltCall gives you the same power on the minion. And specifically here, you can use either SaltCall or SaltCall-Local. Dash-Local will not try to communicate back to the master. It will keep all communications within the minion. And we use this within Salt SSH because that is how we create that unidirectional traffic so that it can all just run on the minion. All right, so now that we understand a little bit about what Salt is, let's talk about how we were able to convert the Salt to being able to use SSH as well. So first of all, why? Why do we provide an avenue for people to run Salt? Obviously, with SSH it is a bit slower, but there's a lot of positives to it as well. It gives our users a lot of variety. They can use both the agent and the agentless option. You don't have to choose one or the other. In fact, I know a lot of people have used Salt SSH to bootstrap their minions as well. So a lot of people use both of those. Obviously, you don't have to install an agent, and it's pretty simple to set up and get running. So how do you get it up and running? There's a couple things. There's Python needs to be on the targeting system unless you're using the dash-r argument, and that just runs a raw SSH command. So you don't get the power of Salt, but you need to also install Salt SSH, and then you need to set up a roster file. And this is an example of what a roster file is, just one of our default, what we refer to as a flat roster file. And as you can see, it's pretty simple. Web 1 is just an arbitrary name. Obviously, you want to name it something useful. Then the host is either the IP address or the DNS of the host name. Your password to be able to properly authenticate to that host. And just in this example, we're using pseudo, which just the user needs pseudo privileges. Obviously, there's more options that you can add within the roster, but that's just a simple example that you can get up and quickly get running. There are other rosters available. One great thing about Salt is it's very pluggable in a lot of aspects, including the roster so you can actually write your own roster. But what's available now is it can work with Ansible. The cache, cache is the master cache. So if you've already got minions you're targeting with your master, you can use those that are in the cache already. Cloud, which is our Salt Cloud offering, interacts with AWS. If you create a VM in the cloud, you can then use Salt SSH directly to interact with that. Cluster shell, flat is the default range scan. That's where you can just scan and define a network range. And also you can use your SSH config. But yeah, like I said, you can also create your own. Okay, so let's show the same example that we showed with the Salt CLI tool. The only thing that's different is you now have to use the Salt SSH CLI command. We target what we want to target, whichever minion that is, and we run just the simple example test.p. And as you would expect, we get the same thing. We get the response of true. So what is going on in the background? Obviously, you can see that we're getting the same result, but a lot of different things are happening to make this possible. So we're just using SSH as the transport layer. So what we're first going to do, and I've provided where in the code this is. That's just more for the curious minds. But we're going to set up what our target and our roster is. We're going to initialize that in our SSH class. And then we're going to generate what we refer to as the thin. So at this point, what it's going to be is we're going to determine what are all the dependencies that we're going to need in this thin to be able to run this on the minion. And it's also going to add the Salt call binary to the thin, and then it's going to zip that up. And then we're going to generate a new process, and we just do that using the multiprocess library. And then in that new process, we're going to generate the file that does a lot of the logic within Salt SSH. It's ssh underscore pi shin. And what we'll do is we have to take all the ops. Where's the directory? Where's the Salt call binary? All of those things that you need to run the command. And we're going to add that to that module. And then that module, when it's run, it's going to determine if it even needs to copy over that tar. So we use a checksum and a versioning. And if any of those have changed, then we would re-copy over the tar. But if we don't need to copy that, obviously the consecutive commands will be faster. So obviously that's a great part of Salt SSH and how we've tried to make it at least a bit faster, even though we're going over SSH. So yeah, once we've got the tar copied over, we're going to run that shim file. And that shim file actually runs the Salt call command. So if we dive into the code a bit, we can kind of see what's going on here like we discussed in that flow chart. So we can see here we're initializing the targets and the roster. And then there's a lot more things that are going on here, but this is just the key point. And then we're going to generate that thin. If we then look at this code, we can see that it's adding the Salt call binary to that thin. It's adding what we refer to as talks, what that command does essentially is just go in and determine what are all of the library or modules that we need to include. And also this extra mods, a user can define additional modules that they want to add to that. We also add a version of the Salt code, a Python version, and then we're going to use either GZIP or ZIP, and we're going to tar that up. So then as I discussed, we spin up a new process, and this is just kind of showing what we do there. We're just using the multiprocessing library, and we're just queuing all of that up. And then a lot of the initiation of the shim starts here. It creates that command string. It's trying to determine what is going to be run in that shim command and adds all the arguments that we would need to it. And then it's going to run that shim command. And that shim command, for example, if the thin is not already copied over, it will determine that and return deploy, or if the checksome is different or things like that. And then we see that deploy was returned, and we rerun that command, and then it would copy it over and run. The shim command just does a couple of things. It determines you can actually specify whether you need a TTY or not when using Salt SSH in the roster config. If it needs a TTY, it just runs our virtual terminal within the Salt code. If you are using a TTY, sorry, if not. And then if you are using a TTY, it needs to actually copy over that shim because you need to use that TTY. And it runs it from there. Okay, so this is all the magic that's going on in that shim file. We're going to determine where the thin path would be, then we're going to do a bunch of checks. There's more checks than I have shown here, but we do check the checksome, the version, and all those things. And then, obviously, you can see if it's different, we're going to say we need a deployment, and then we're going to untar that new thin. And then, as you can see here, we've got this Salt call command, and that is the actual command that we're going to be running alongside that test stop ping that we passed to it. And then we just use the sub-process module to run that. So all of that discussed is just talking and referring to Salt's remote execution capabilities. Obviously, there's a lot more you can do with Salt. So one of the really other popular aspects of Salt is the state system, and this is where we get a lot of our item potent behavior. So we refer to these SLS files as where you would define everything that you would want your machine to have to be in that certain state. And this is just a simple example. This one's using YAML. There are multiple other renderers that you can use within Salt, but YAML is one of the defaults, and this is a pretty simple example. We're just managing a file, and we want to state, or we want to put it worked into that file. So as you can see, we run it with Salt SSH. We run state.SLS, which is where we're going to run that SLS file, and it adds that new file and shows that a new change has occurred. Now if you were to run this again, as shown here, because it's item potent behavior, it doesn't try to change anything, and it just says it's in the correct state. So we have the execution modules, but we now have the state system, and the state system requires a little bit more because we have those SLS files, so we need to have some sort of deterministic behavior to figure out what files we also need to include and copy over to be able to run the state command. And what happens is it's the same as the execution module, except for when it comes to that command block. Command function, we run this one, runWfunc, and it essentially stands for wrapper functions. We have a bunch of, or a couple of wrapper modules within Salt SSH, and they're all displayed right there. And one of the things that makes it a bit tricky with using Salt SSH is because we've had to create these wrapper modules, and you'll see what we've done differently in the state.py example. It gets a little bit hard to manage both of those files because within Salt's code, we have a state.py, and Salt SSH, we have a state.py. So sometimes it becomes a little bit difficult to manage both of those files and make sure if the feature is added to one that we have to add it to the other. So that is one problem that we've run into with this current setup. So this function will set up all the pillar and grains, and within Salt, pillar and grains is just a lot of data. Pillar being, usually people use that for more of the sensitive information, and it's stored on the master. And grains being information about the minion stored on the minion. And any other ops that we need to be able to add to run these state files. Then we call all of those wrapper functions, and in this example, we're specifically going to call the state wrapper function. So as you can see that we're doing here is we're grabbing all the data that we're going to need for the call. We're initializing the function wrapper, and then we eventually we're just going to run it and return the JSON of the return. And this is how we're doing it in the state.py file. So in a normal SLS file, we do not have all these extra steps, because what these extra steps are doing for Salt SSH is, as you can see, we're preparing the tar. So within the tar, we determined what SLS files need to be added. One of the things that is a bit tricky is when people use includes in their file, you actually have to specify those separately in a configuration. So that is one thing I think could be improved upon as well. But yeah, so that prep trans tar, that's going to do all that magic there. And then we're going to sum up that and get it a hash. And then we're going to do the same thing we did before, initialize a single class, send over that tar, and then we'll run that command block again, and run essentially that same logic. So now that we've shown you what's going on underneath, you'll probably not remember all of that code. So what are some of the key architectural decisions to summarize all of that? One thing that's important to know is Salt was not modified to add Salt SSH. Salt SSH was added on top of it. And that's where you can see things like the wrapper functions and things like that where we didn't want to edit Salt. We use the tar ball where we tar everything up on the master. We determine what needs to be included on that, and then we send it over and run it on the Minion. As shown before, we use that Salt call. So Salt call, dash dash local, that's what provides us with the unidirectional communication. And we won't need to call out back to the master while the Minion is running. It's able to run everything it needs right there. It has all the information it needs. And then we use these wrapper functions where we're trying to provide the same capabilities that Salt has, but add it to Salt SSH. So what were some of the challenges when doing this? I've mentioned a lot of them already. Obviously, we lose some of the bi-directional where with SSH, we only have unidirectional, so that was a challenge. The wrapper challenges or functions, as I spoke before, is a bit difficult in terms of managing the code. And a lot of times when we see bugs within Salt SSH is because we don't have all the correct ops that we need. And ops is just a lot of the configs that we would need to run a Salt SSH command. So that is one of the challenges that I think we could do a bit better on and kind of make sure we have everything we need to be able to run within Salt and Salt SSH so there is more parity between the two. Okay, so what are our plans for the future? We actually are initially, we're initially talking right now. We don't have any set plans, but we do want to rearchitect Salt SSH and fix some of these issues that we have presented. Some of the ideas that we've been thinking on is trying to remove that Python dependency and how you do that. We have had some ideas of just including the minimal Python executable within the tar ball. Another reason I think this will be beneficial is eventually Python 2 is going to go away. And obviously users are still going to be using machines that have Python 2 on it, so if we remove the Python dependency and we include it in there, that won't be an issue. If we do go Python 3, we can use some of the capabilities of AsyncIO and hopefully we can resolve some of the issues that we stated such as the wrapper functions and so forth. All right. Does anybody have questions? Yeah, I don't know how to turn the light on. Now that you can see me, does anybody have any questions? Yeah. So what would be the benefits of using its agent versus using the agent? Yeah, essentially you don't have that agent. That is the benefit that I can see over Salt. It is slower because you don't, you know, obviously using SSH, but that is the benefit. You don't have to install the agent. Huh? Sorry. Okay. Go ahead. You can go again. Because I've been using Ansible a lot. Okay. So I'm kind of curious how do you, so have you done any testing on like performance-wise if you compare this solution with Ansible, for example? I haven't personally done. Bo, do you, have you done, I know you've done a lot of work with that. Have you done any performance testing? Yeah. Between the two? That could, that could be a problem. Okay. It's cashed. So Salt is copying itself to the temporary model. They are, they are, so next time we don't want to install it anymore, it's just for her monthly. So, so like faster than Ansible because it's already all the time. So not all friends are basically, yeah, not friends, yeah. As you mentioned, you copied the Salt binary, okay? There's just the Salt call. And then, yeah, all the other Salt dependencies you would need. Okay. So if you have a system where no binary exists, what happens next? The system doesn't have Salt on it. That's what it does. It copies over whatever Salt binaries that it would need and all the modules within it. Yeah. It's the same function as Ansible in that case. Okay. I don't, I don't know. Ansible initiate the connection. After that it copies some binaries. Okay. After that executes commands. Okay. Yeah. A little question here. Yeah. Colleague is asking about, what about Windows? Windows, we do have a Windows solution is enterprise only though. WinRM? Yeah. Yep. So I'm also, I'm thinking some of the, let's take up a little bit so that you can hear. Oh yeah, I should probably repeat the question too. No. No, it's not. It's been a long day, okay? The answer is no. So I'm just reading this just about having, as Salt has the same functionality as Ansible, unless you can explain to me how I can talk to her on, let's say a dump device, which I can do with Ansible, which is much, much like using expect. Sorry, I didn't quite understand that because the speakers are back there. Can you repeat the question? So I'm, I'm interested in functionality of, I'm currently also an Ansible user, but I'm interested in functionality in talking to, let's say dump devices, which have nothing on it where I need something like expect, which I can do with Ansible, but I don't know how to do it with Salt. With expect, is that what you're asking? How would you, sorry, I'm having a hard time understanding. With switch or something. Oh, like network switches, things like that. Oh yeah, so we, we have what's called Salt proxy, and that's how we, that's our solution for not even having to have Python. We actually have a solution with all salt, salt as a stage and it's salt proxy. I don't know what's going on underneath, but a lot of our network modules and all of that use heavily use salt proxy. So you'd want to look at that. Yeah. Hold on. Get some eggs. Thanks. Thank you. I don't use salt, so maybe the question is not exactly appropriate, but I was, I, I, I was thinking about the, where do you store the configuration about the machines when you don't have a model like minions and servers? Where do you store the, your assets, the information about the machine you will connect to? They are, they are described on, on client. The information such as. Let me say if you use a password or a certificate or which is the user you use to connect to the host, do you have a different YAML or configuration files, one for each client which uses SS and SSH? I think what you're asking is where is the information stored about the minion? Yeah. Yeah. Because the, the, the, so just to complete the question, because I was thinking about when you have a server, the information about the host you're managing with salt is in a way centralized on, on the server. Yeah. When you use this kind of approach, those disinformation are spread over the clients. Okay. Yeah. So I think what you're asking is, and what you're getting at is we have a concept of what we call pillar and grains. And pillar is what you would specify on the master. And a lot of people use these for like passwords. And yeah, it is just like a YAML file. You can use whatever render. The grains is initialized on, normally in the salt master salt minion, it's, it's initialized on the minion and it automatically determines all that information like operating system, host name, things like that. So it's automatically determining that. And that's what grains are. You also have the concept of the salt mine. And the salt mine will give you the ability to query all these minions and cache that information on the master. This is the pattern where you have the minions on your, on your, for example, for example. Yes. Oh, okay. So where you're specifying where your, what your hosts are and what, yeah, it's just the roster is what you would use. Because I was thinking about the adoption of this kind of pattern when you have, for example, your operation department, they are 10, 20 single clients. How do you, how do you ensure that the configuration they have is up to date and, and so on? The, like the Ansible inventory. So inventory files, for example, the list of hosts like this, to connect to this, I mean, salt has the same concept. Okay. Actually, it was, I believe you can use Ansible inventory as a source cluster. Yeah. Okay. And you had, you need to distribute and sync this, this information. Distribute and see the roster file. I mean, it's just a flat file or you can use a different sort of roster inquiry that information differently. But by default, it's just a flat file. Yeah. It could be, can it be remote to centralized or just a flat file? Yeah. It's just a flat file by default. But like I was showing you the other rosters, like you mentioned, you can use the target Ansible and you can do things like scan the network and things like that. Or you can create your own roster if you want to query that information for something else, like a database or something like that. You can create your own module and do that as well. Okay. Thank you. Sorry, hopefully I answered that well enough. Ansible does what? I before fear, I before sex, for sex. Yes, general information about the host host is distributed. So IPv4, IPv6, interface, information. Like how do you, does it use IPv4 and IPv6? Is that what you're asking? No, the host information about this variables, how they, how do you query like the idea? Yes, facts about the host. Yes, so it would be great. So you could just query the minion and use that and you can run grains.items and that would grab all the information about all the menus you'd need. Or like I was referring to, we have this idea of the salt mine and it can kind of do the same thing. You can clarify what information you want and it would cash it on the master. And that does work with salt SSH as well. Yeah. I'll take it. Yeah, yeah. So from what I understand, now you would have two different commands you want to invoke depending on what kind of setup you have or what kind of minior, your targeting. Is there any plans to kind of unify them so that you would define in your roster, I guess. I don't know enough about it, but if you define in the roster, okay, use classic agent base management with this minion and with this other minion, use salt SSH. So I know in our last release or it's coming up in this next release, one of our users added the functionality so that you can actually use the salt CLI and target both SSH and minions. And the same. Was that your question? Yeah. So that is actually added just recently. There is some sort of argument you'd have to use. I don't know it off the top of my head. Any other questions? I was asked, I just told the school of reamper as a wrapper that you can use with salt SSH. I believe. Last of five minutes. Yeah, so I'm almost positive you can because I didn't mention this, but before it actually submits that command to the minion, it actually is initializing a connection with the master event bus. And then when it gets that return, it actually returns that to the master event bus. So you could react off of that event. Yeah, so. All right. Thanks. Thank you very much. Thank you very much.
 Yeah, so today I'm going to be talking about how at Salt we moved from an agent-based  solution to being able to use Salt without installing an agent.  So yeah, my name is Megan Wilhite.  I'm known as Shell on the Interwebs.  Bonus points to ever understand what Shell is.  And I'm currently a software engineer at Saltstack, but I am working mostly on getting releases  out right now.  So that's me.  So what is Salt?  For anybody that doesn't know what Salt is, I think it's important to understand what  Salt is before we go on and talking about Salt as a state, which is the agent-less solution.  So Salt is a lot of things, but I would describe it as being able to do things such as config  management, remote execution, automation, and orchestration.  And it's very event-driven.  So talking about how Salt is currently set up with the agent solution, it is a server  agent communication model.  So we refer to the server as the saltmaster, and we refer to the agent as the salt minion.  And the way that you would set this up is you would install the master, and then you  would install the minion.  And there are a couple of network services that provide the transport layer within Salt,  and we refer to them as the publisher and the request server.  And these are using a Xeron Q's library, and the publisher essentially is sending out all  the commands from the master to the minions, and the minions then determine from there  whether they should actually run the command, and then we have the request server, and this  is where we get a lot of the bi-directional communication between the master and the minion.  So when talking about converting over to an agent-less solution, obviously SSH is unidirectional,  so we had to be careful in terms of how we architected that so that we could use the  power of Salt, but also use SSH.  So I'm just going to show you a quick command within Salt that you can run.  It's just kind of like our version of FlowWorld.  It's just Salt, which is the name of the CLI tool that you would use, targeting whatever  minion you want to target, and then TestStopPing is just what we refer to as an execution  module.  Execution modules are not meant to be item potent.  They're more meant to be run as one-off commands.  And so, yeah, it would go through that job flow, as I mentioned before.  The master would publish that command of the minion.  The minion would then run it and then return it through the request server.  So as you can see through the CLI, maybe you can't see that super well, but essentially  all it does is it runs TestStopPing on the minion and then returns true if it's able  to communicate with the minion.  Okay, another important thing when talking about how we architected Salt SSH is you need  to understand the difference between Salt and SaltCall.  So Salt, as we show here, we're targeting all minions with the ashras there and TestStopPing.  That is the CLI tool you use on the master.  And SaltCall gives you the same power on the minion.  And specifically here, you can use either SaltCall or SaltCall-Local.  Dash-Local will not try to communicate back to the master.  It will keep all communications within the minion.  And we use this within Salt SSH because that is how we create that unidirectional traffic  so that it can all just run on the minion.  All right, so now that we understand a little bit about what Salt is, let's talk about how  we were able to convert the Salt to being able to use SSH as well.  So first of all, why?  Why do we provide an avenue for people to run Salt?  Obviously, with SSH it is a bit slower, but there's a lot of positives to it as well.  It gives our users a lot of variety.  They can use both the agent and the agentless option.  You don't have to choose one or the other.  In fact, I know a lot of people have used Salt SSH to bootstrap their minions as well.  So a lot of people use both of those.  Obviously, you don't have to install an agent, and it's pretty simple to set up and get  running.  So how do you get it up and running?  There's a couple things.  There's Python needs to be on the targeting system unless you're using the dash-r argument,  and that just runs a raw SSH command.  So you don't get the power of Salt, but you need to also install Salt SSH, and then you  need to set up a roster file.  And this is an example of what a roster file is, just one of our default, what we refer  to as a flat roster file.  And as you can see, it's pretty simple.  Web 1 is just an arbitrary name.  Obviously, you want to name it something useful.  Then the host is either the IP address or the DNS of the host name.  Your password to be able to properly authenticate to that host.  And just in this example, we're using pseudo, which just the user needs pseudo privileges.  Obviously, there's more options that you can add within the roster, but that's just  a simple example that you can get up and quickly get running.  There are other rosters available.  One great thing about Salt is it's very pluggable in a lot of aspects, including the roster  so you can actually write your own roster.  But what's available now is it can work with Ansible.  The cache, cache is the master cache.  So if you've already got minions you're targeting with your master, you can use those that are  in the cache already.  Cloud, which is our Salt Cloud offering, interacts with AWS.  If you create a VM in the cloud, you can then use Salt SSH directly to interact with that.  Cluster shell, flat is the default range scan.  That's where you can just scan and define a network range.  And also you can use your SSH config.  But yeah, like I said, you can also create your own.  Okay, so let's show the same example that we showed with the Salt CLI tool.  The only thing that's different is you now have to use the Salt SSH CLI command.  We target what we want to target, whichever minion that is, and we run just the simple  example test.p.  And as you would expect, we get the same thing.  We get the response of true.  So what is going on in the background?  Obviously, you can see that we're getting the same result, but a lot of different things  are happening to make this possible.  So we're just using SSH as the transport layer.  So what we're first going to do, and I've provided where in the code this is.  That's just more for the curious minds.  But we're going to set up what our target and our roster is.  We're going to initialize that in our SSH class.  And then we're going to generate what we refer to as the thin.  So at this point, what it's going to be is we're going to determine what are all the  dependencies that we're going to need in this thin to be able to run this on the minion.  And it's also going to add the Salt call binary to the thin, and then it's going to zip that  up.  And then we're going to generate a new process, and we just do that using the multiprocess  library.  And then in that new process, we're going to generate the file that does a lot of the  logic within Salt SSH.  It's ssh underscore pi shin.  And what we'll do is we have to take all the ops.  Where's the directory?  Where's the Salt call binary?  All of those things that you need to run the command.  And we're going to add that to that module.  And then that module, when it's run, it's going to determine if it even needs to copy  over that tar.  So we use a checksum and a versioning.  And if any of those have changed, then we would re-copy over the tar.  But if we don't need to copy that, obviously the consecutive commands will be faster.  So obviously that's a great part of Salt SSH and how we've tried to make it at least  a bit faster, even though we're going over SSH.  So yeah, once we've got the tar copied over, we're going to run that shim file.  And that shim file actually runs the Salt call command.  So if we dive into the code a bit, we can kind of see what's going on here like we  discussed in that flow chart.  So we can see here we're initializing the targets and the roster.  And then there's a lot more things that are going on here, but this is just the key point.  And then we're going to generate that thin.  If we then look at this code, we can see that it's adding the Salt call binary to that thin.  It's adding what we refer to as talks, what that command does essentially is just go in  and determine what are all of the library or modules that we need to include.  And also this extra mods, a user can define additional modules that they want to add to  that.  We also add a version of the Salt code, a Python version, and then we're going to use  either GZIP or ZIP, and we're going to tar that up.  So then as I discussed, we spin up a new process, and this is just kind of showing what we  do there.  We're just using the multiprocessing library, and we're just queuing all of that up.  And then a lot of the initiation of the shim starts here.  It creates that command string.  It's trying to determine what is going to be run in that shim command and adds all the  arguments that we would need to it.  And then it's going to run that shim command.  And that shim command, for example, if the thin is not already copied over, it will determine  that and return deploy, or if the checksome is different or things like that.  And then we see that deploy was returned, and we rerun that command, and then it would  copy it over and run.  The shim command just does a couple of things.  It determines you can actually specify whether you need a TTY or not when using Salt SSH  in the roster config.  If it needs a TTY, it just runs our virtual terminal within the Salt code.  If you are using a TTY, sorry, if not.  And then if you are using a TTY, it needs to actually copy over that shim because you  need to use that TTY.  And it runs it from there.  Okay, so this is all the magic that's going on in that shim file.  We're going to determine where the thin path would be, then we're going to do a bunch of  checks.  There's more checks than I have shown here, but we do check the checksome, the version,  and all those things.  And then, obviously, you can see if it's different, we're going to say we need a deployment, and  then we're going to untar that new thin.  And then, as you can see here, we've got this Salt call command, and that is the actual  command that we're going to be running alongside that test stop ping that we passed to it.  And then we just use the sub-process module to run that.  So all of that discussed is just talking and referring to Salt's remote execution capabilities.  Obviously, there's a lot more you can do with Salt.  So one of the really other popular aspects of Salt is the state system, and this is where  we get a lot of our item potent behavior.  So we refer to these SLS files as where you would define everything that you would want  your machine to have to be in that certain state.  And this is just a simple example.  This one's using YAML.  There are multiple other renderers that you can use within Salt, but YAML is one of the  defaults, and this is a pretty simple example.  We're just managing a file, and we want to state, or we want to put it worked into that  file.  So as you can see, we run it with Salt SSH.  We run state.SLS, which is where we're going to run that SLS file, and it adds that new  file and shows that a new change has occurred.  Now if you were to run this again, as shown here, because it's item potent behavior, it  doesn't try to change anything, and it just says it's in the correct state.  So we have the execution modules, but we now have the state system, and the state system  requires a little bit more because we have those SLS files, so we need to have some sort  of deterministic behavior to figure out what files we also need to include and copy over  to be able to run the state command.  And what happens is it's the same as the execution module, except for when it comes to that command  block.  Command function, we run this one, runWfunc, and it essentially stands for wrapper functions.  We have a bunch of, or a couple of wrapper modules within Salt SSH, and they're all displayed  right there.  And one of the things that makes it a bit tricky with using Salt SSH is because we've  had to create these wrapper modules, and you'll see what we've done differently in the state.py  example.  It gets a little bit hard to manage both of those files because within Salt's code, we  have a state.py, and Salt SSH, we have a state.py.  So sometimes it becomes a little bit difficult to manage both of those files and make sure  if the feature is added to one that we have to add it to the other.  So that is one problem that we've run into with this current setup.  So this function will set up all the pillar and grains, and within Salt, pillar and grains  is just a lot of data.  Pillar being, usually people use that for more of the sensitive information, and it's  stored on the master.  And grains being information about the minion stored on the minion.  And any other ops that we need to be able to add to run these state files.  Then we call all of those wrapper functions, and in this example, we're specifically going  to call the state wrapper function.  So as you can see that we're doing here is we're grabbing all the data that we're going  to need for the call.  We're initializing the function wrapper, and then we eventually we're just going to run  it and return the JSON of the return.  And this is how we're doing it in the state.py file.  So in a normal SLS file, we do not have all these extra steps, because what these extra  steps are doing for Salt SSH is, as you can see, we're preparing the tar.  So within the tar, we determined what SLS files need to be added.  One of the things that is a bit tricky is when people use includes in their file, you  actually have to specify those separately in a configuration.  So that is one thing I think could be improved upon as well.  But yeah, so that prep trans tar, that's going to do all that magic there.  And then we're going to sum up that and get it a hash.  And then we're going to do the same thing we did before, initialize a single class,  send over that tar, and then we'll run that command block again, and run essentially that  same logic.  So now that we've shown you what's going on underneath, you'll probably not remember  all of that code.  So what are some of the key architectural decisions to summarize all of that?  One thing that's important to know is Salt was not modified to add Salt SSH.  Salt SSH was added on top of it.  And that's where you can see things like the wrapper functions and things like that where  we didn't want to edit Salt.  We use the tar ball where we tar everything up on the master.  We determine what needs to be included on that, and then we send it over and run it  on the Minion.  As shown before, we use that Salt call.  So Salt call, dash dash local, that's what provides us with the unidirectional communication.  And we won't need to call out back to the master while the Minion is running.  It's able to run everything it needs right there.  It has all the information it needs.  And then we use these wrapper functions where we're trying to provide the same capabilities  that Salt has, but add it to Salt SSH.  So what were some of the challenges when doing this?  I've mentioned a lot of them already.  Obviously, we lose some of the bi-directional where with SSH, we only have unidirectional,  so that was a challenge.  The wrapper challenges or functions, as I spoke before, is a bit difficult in terms  of managing the code.  And a lot of times when we see bugs within Salt SSH is because we don't have all the  correct ops that we need.  And ops is just a lot of the configs that we would need to run a Salt SSH command.  So that is one of the challenges that I think we could do a bit better on and kind of make  sure we have everything we need to be able to run within Salt and Salt SSH so there is  more parity between the two.  Okay, so what are our plans for the future?  We actually are initially, we're initially talking right now.  We don't have any set plans, but we do want to rearchitect Salt SSH and fix some of these  issues that we have presented.  Some of the ideas that we've been thinking on is trying to remove that Python dependency  and how you do that.  We have had some ideas of just including the minimal Python executable within the tar  ball.  Another reason I think this will be beneficial is eventually Python 2 is going to go away.  And obviously users are still going to be using machines that have Python 2 on it, so  if we remove the Python dependency and we include it in there, that won't be an issue.  If we do go Python 3, we can use some of the capabilities of AsyncIO and hopefully we can  resolve some of the issues that we stated such as the wrapper functions and so forth.  All right.  Does anybody have questions?  Yeah, I don't know how to turn the light on.  Now that you can see me, does anybody have any questions?  Yeah.  So what would be the benefits of using its agent versus using the agent?  Yeah, essentially you don't have that agent.  That is the benefit that I can see over Salt.  It is slower because you don't, you know, obviously using SSH, but that is the benefit.  You don't have to install the agent.  Huh?  Sorry.  Okay.  Go ahead.  You can go again.  Because I've been using Ansible a lot.  Okay.  So I'm kind of curious how do you, so have you done any testing on like performance-wise  if you compare this solution with Ansible, for example?  I haven't personally done.  Bo, do you, have you done, I know you've done a lot of work with that.  Have you done any performance testing?  Yeah.  Between the two?  That could, that could be a problem.  Okay.  It's cashed.  So Salt is copying itself to the temporary model.  They are, they are, so next time we don't want to install it anymore, it's just for  her monthly.  So, so like faster than Ansible because it's already all the time.  So not all friends are basically, yeah, not friends, yeah.  As you mentioned, you copied the Salt binary, okay?  There's just the Salt call.  And then, yeah, all the other Salt dependencies you would need.  Okay.  So if you have a system where no binary exists, what happens next?  The system doesn't have Salt on it.  That's what it does.  It copies over whatever Salt binaries that it would need and all the modules within it.  Yeah.  It's the same function as Ansible in that case.  Okay.  I don't, I don't know.  Ansible initiate the connection.  After that it copies some binaries.  Okay.  After that executes commands.  Okay.  Yeah.  A little question here.  Yeah.  Colleague is asking about, what about Windows?  Windows, we do have a Windows solution is enterprise only though.  WinRM?  Yeah.  Yep.  So I'm also, I'm thinking some of the, let's take up a little bit so that you can hear.  Oh yeah, I should probably repeat the question too.  No.  No, it's not.  It's been a long day, okay?  The answer is no.  So I'm just reading this just about having, as Salt has the same functionality as Ansible,  unless you can explain to me how I can talk to her on, let's say a dump device, which  I can do with Ansible, which is much, much like using expect.  Sorry, I didn't quite understand that because the speakers are back there.  Can you repeat the question?  So I'm, I'm interested in functionality of, I'm currently also an Ansible user, but I'm  interested in functionality in talking to, let's say dump devices, which have nothing  on it where I need something like expect, which I can do with Ansible, but I don't know how  to do it with Salt.  With expect, is that what you're asking?  How would you, sorry, I'm having a hard time understanding.  With switch or something.  Oh, like network switches, things like that.  Oh yeah, so we, we have what's called Salt proxy, and that's how we, that's our solution  for not even having to have Python.  We actually have a solution with all salt, salt as a stage and it's salt proxy.  I don't know what's going on underneath, but a lot of our network modules and all of that  use heavily use salt proxy.  So you'd want to look at that.  Yeah.  Hold on.  Get some eggs.  Thanks.  Thank you.  I don't use salt, so maybe the question is not exactly appropriate, but I was, I, I,  I was thinking about the, where do you store the configuration about the machines when  you don't have a model like minions and servers?  Where do you store the, your assets, the information about the machine you will connect to?  They are, they are described on, on client.  The information such as.  Let me say if you use a password or a certificate or which is the user you use to connect to  the host, do you have a different YAML or configuration files, one for each client which  uses SS and SSH?  I think what you're asking is where is the information stored about the minion?  Yeah.  Yeah.  Because the, the, the, so just to complete the question, because I was thinking about  when you have a server, the information about the host you're managing with salt is in a  way centralized on, on the server.  Yeah.  When you use this kind of approach, those disinformation are spread over the clients.  Okay.  Yeah.  So I think what you're asking is, and what you're getting at is we have a concept of  what we call pillar and grains.  And pillar is what you would specify on the master.  And a lot of people use these for like passwords.  And yeah, it is just like a YAML file.  You can use whatever render.  The grains is initialized on, normally in the salt master salt minion, it's, it's initialized  on the minion and it automatically determines all that information like operating system,  host name, things like that.  So it's automatically determining that.  And that's what grains are.  You also have the concept of the salt mine.  And the salt mine will give you the ability to query all these minions and cache that  information on the master.  This is the pattern where you have the minions on your, on your, for example, for example.  Yes.  Oh, okay.  So where you're specifying where your, what your hosts are and what, yeah, it's just the  roster is what you would use.  Because I was thinking about the adoption of this kind of pattern when you have, for example,  your operation department, they are 10, 20 single clients.  How do you, how do you ensure that the configuration they have is up to date and, and so on?  The, like the Ansible inventory.  So inventory files, for example, the list of hosts like this, to connect to this, I mean,  salt has the same concept.  Okay.  Actually, it was, I believe you can use Ansible inventory as a source cluster.  Yeah.  Okay.  And you had, you need to distribute and sync this, this information.  Distribute and see the roster file.  I mean, it's just a flat file or you can use a different sort of roster inquiry that information  differently.  But by default, it's just a flat file.  Yeah.  It could be, can it be remote to centralized or just a flat file?  Yeah.  It's just a flat file by default.  But like I was showing you the other rosters, like you mentioned, you can use the target  Ansible and you can do things like scan the network and things like that.  Or you can create your own roster if you want to query that information for something else,  like a database or something like that.  You can create your own module and do that as well.  Okay.  Thank you.  Sorry, hopefully I answered that well enough.  Ansible does what?  I before fear, I before sex, for sex.  Yes, general information about the host host is distributed.  So IPv4, IPv6, interface, information.  Like how do you, does it use IPv4 and IPv6?  Is that what you're asking?  No, the host information about this variables, how they, how do you query like the idea?  Yes, facts about the host.  Yes, so it would be great.  So you could just query the minion and use that and you can run grains.items and that  would grab all the information about all the menus you'd need.  Or like I was referring to, we have this idea of the salt mine and it can kind of do the  same thing.  You can clarify what information you want and it would cash it on the master.  And that does work with salt SSH as well.  Yeah.  I'll take it.  Yeah, yeah.  So from what I understand, now you would have two different commands you want to invoke  depending on what kind of setup you have or what kind of minior, your targeting.  Is there any plans to kind of unify them so that you would define in your roster, I guess.  I don't know enough about it, but if you define in the roster, okay, use classic agent base  management with this minion and with this other minion, use salt SSH.  So I know in our last release or it's coming up in this next release, one of our users  added the functionality so that you can actually use the salt CLI and target both SSH and  minions.  And the same.  Was that your question?  Yeah.  So that is actually added just recently.  There is some sort of argument you'd have to use.  I don't know it off the top of my head.  Any other questions?  I was asked, I just told the school of reamper as a wrapper that you can use with salt SSH.  I believe.  Last of five minutes.  Yeah, so I'm almost positive you can because I didn't mention this, but before it actually  submits that command to the minion, it actually is initializing a connection with the master  event bus.  And then when it gets that return, it actually returns that to the master event bus.  So you could react off of that event.  Yeah, so.  All right.  Thanks.  Thank you very much.  Thank you very much.
How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_0_143.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_144_419.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_420_875.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_876_1535.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_1536_3119.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_3120_3335.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_3336_4103.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_4104_4259.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_4260_4883.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_4884_5831.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_5832_7103.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_7104_7451.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_7452_8591.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_8592_8723.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_8724_9839.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_9840_10571.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_10572_12083.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_12084_12167.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_12168_13655.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_13656_15407.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_15408_18239.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_18240_19631.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_19632_20231.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_20232_21743.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_21744_21803.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_21804_22991.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_22992_23435.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_23436_24947.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_24948_24995.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_24996_26519.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_26520_26735.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_26736_29747.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_29748_31259.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_31260_31367.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_31368_31511.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_31512_31763.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_31764_32267.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_32268_32555.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_32556_32675.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_32676_33431.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_33432_34379.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_34380_35579.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_35580_41603.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_41604_45203.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_45204_46055.png How_We_Modified_an_Agent-based_Automation_System_to_Become_an_Agentless_One_Lessons_and_Architecture_46056_46177.png
Yeah, so today I'm going to be talking about how at Salt we moved from an agent-based solution to being able to use Salt without installing an agent. So yeah, my name is Megan Wilhite. I'm known as Shell on the Interwebs. Bonus points to ever understand what Shell is. And I'm currently a software engineer at Saltstack, but I am working mostly on getting releases out right now. So that's me. So what is Salt? For anybody that doesn't know what Salt is, I think it's important to understand what Salt is before we go on and talking about Salt as a state, which is the agent-less solution. So Salt is a lot of things, but I would describe it as being able to do things such as config management, remote execution, automation, and orchestration. And it's very event-driven. So talking about how Salt is currently set up with the agent solution, it is a server agent communication model. So we refer to the server as the saltmaster, and we refer to the agent as the salt minion. And the way that you would set this up is you would install the master, and then you would install the minion. And there are a couple of network services that provide the transport layer within Salt, and we refer to them as the publisher and the request server. And these are using a Xeron Q's library, and the publisher essentially is sending out all the commands from the master to the minions, and the minions then determine from there whether they should actually run the command, and then we have the request server, and this is where we get a lot of the bi-directional communication between the master and the minion. So when talking about converting over to an agent-less solution, obviously SSH is unidirectional, so we had to be careful in terms of how we architected that so that we could use the power of Salt, but also use SSH. So I'm just going to show you a quick command within Salt that you can run. It's just kind of like our version of FlowWorld. It's just Salt, which is the name of the CLI tool that you would use, targeting whatever minion you want to target, and then TestStopPing is just what we refer to as an execution module. Execution modules are not meant to be item potent. They're more meant to be run as one-off commands. And so, yeah, it would go through that job flow, as I mentioned before. The master would publish that command of the minion. The minion would then run it and then return it through the request server. So as you can see through the CLI, maybe you can't see that super well, but essentially all it does is it runs TestStopPing on the minion and then returns true if it's able to communicate with the minion. Okay, another important thing when talking about how we architected Salt SSH is you need to understand the difference between Salt and SaltCall. So Salt, as we show here, we're targeting all minions with the ashras there and TestStopPing. That is the CLI tool you use on the master. And SaltCall gives you the same power on the minion. And specifically here, you can use either SaltCall or SaltCall-Local. Dash-Local will not try to communicate back to the master. It will keep all communications within the minion. And we use this within Salt SSH because that is how we create that unidirectional traffic so that it can all just run on the minion. All right, so now that we understand a little bit about what Salt is, let's talk about how we were able to convert the Salt to being able to use SSH as well. So first of all, why? Why do we provide an avenue for people to run Salt? Obviously, with SSH it is a bit slower, but there's a lot of positives to it as well. It gives our users a lot of variety. They can use both the agent and the agentless option. You don't have to choose one or the other. In fact, I know a lot of people have used Salt SSH to bootstrap their minions as well. So a lot of people use both of those. Obviously, you don't have to install an agent, and it's pretty simple to set up and get running. So how do you get it up and running? There's a couple things. There's Python needs to be on the targeting system unless you're using the dash-r argument, and that just runs a raw SSH command. So you don't get the power of Salt, but you need to also install Salt SSH, and then you need to set up a roster file. And this is an example of what a roster file is, just one of our default, what we refer to as a flat roster file. And as you can see, it's pretty simple. Web 1 is just an arbitrary name. Obviously, you want to name it something useful. Then the host is either the IP address or the DNS of the host name. Your password to be able to properly authenticate to that host. And just in this example, we're using pseudo, which just the user needs pseudo privileges. Obviously, there's more options that you can add within the roster, but that's just a simple example that you can get up and quickly get running. There are other rosters available. One great thing about Salt is it's very pluggable in a lot of aspects, including the roster so you can actually write your own roster. But what's available now is it can work with Ansible. The cache, cache is the master cache. So if you've already got minions you're targeting with your master, you can use those that are in the cache already. Cloud, which is our Salt Cloud offering, interacts with AWS. If you create a VM in the cloud, you can then use Salt SSH directly to interact with that. Cluster shell, flat is the default range scan. That's where you can just scan and define a network range. And also you can use your SSH config. But yeah, like I said, you can also create your own. Okay, so let's show the same example that we showed with the Salt CLI tool. The only thing that's different is you now have to use the Salt SSH CLI command. We target what we want to target, whichever minion that is, and we run just the simple example test.p. And as you would expect, we get the same thing. We get the response of true. So what is going on in the background? Obviously, you can see that we're getting the same result, but a lot of different things are happening to make this possible. So we're just using SSH as the transport layer. So what we're first going to do, and I've provided where in the code this is. That's just more for the curious minds. But we're going to set up what our target and our roster is. We're going to initialize that in our SSH class. And then we're going to generate what we refer to as the thin. So at this point, what it's going to be is we're going to determine what are all the dependencies that we're going to need in this thin to be able to run this on the minion. And it's also going to add the Salt call binary to the thin, and then it's going to zip that up. And then we're going to generate a new process, and we just do that using the multiprocess library. And then in that new process, we're going to generate the file that does a lot of the logic within Salt SSH. It's ssh underscore pi shin. And what we'll do is we have to take all the ops. Where's the directory? Where's the Salt call binary? All of those things that you need to run the command. And we're going to add that to that module. And then that module, when it's run, it's going to determine if it even needs to copy over that tar. So we use a checksum and a versioning. And if any of those have changed, then we would re-copy over the tar. But if we don't need to copy that, obviously the consecutive commands will be faster. So obviously that's a great part of Salt SSH and how we've tried to make it at least a bit faster, even though we're going over SSH. So yeah, once we've got the tar copied over, we're going to run that shim file. And that shim file actually runs the Salt call command. So if we dive into the code a bit, we can kind of see what's going on here like we discussed in that flow chart. So we can see here we're initializing the targets and the roster. And then there's a lot more things that are going on here, but this is just the key point. And then we're going to generate that thin. If we then look at this code, we can see that it's adding the Salt call binary to that thin. It's adding what we refer to as talks, what that command does essentially is just go in and determine what are all of the library or modules that we need to include. And also this extra mods, a user can define additional modules that they want to add to that. We also add a version of the Salt code, a Python version, and then we're going to use either GZIP or ZIP, and we're going to tar that up. So then as I discussed, we spin up a new process, and this is just kind of showing what we do there. We're just using the multiprocessing library, and we're just queuing all of that up. And then a lot of the initiation of the shim starts here. It creates that command string. It's trying to determine what is going to be run in that shim command and adds all the arguments that we would need to it. And then it's going to run that shim command. And that shim command, for example, if the thin is not already copied over, it will determine that and return deploy, or if the checksome is different or things like that. And then we see that deploy was returned, and we rerun that command, and then it would copy it over and run. The shim command just does a couple of things. It determines you can actually specify whether you need a TTY or not when using Salt SSH in the roster config. If it needs a TTY, it just runs our virtual terminal within the Salt code. If you are using a TTY, sorry, if not. And then if you are using a TTY, it needs to actually copy over that shim because you need to use that TTY. And it runs it from there. Okay, so this is all the magic that's going on in that shim file. We're going to determine where the thin path would be, then we're going to do a bunch of checks. There's more checks than I have shown here, but we do check the checksome, the version, and all those things. And then, obviously, you can see if it's different, we're going to say we need a deployment, and then we're going to untar that new thin. And then, as you can see here, we've got this Salt call command, and that is the actual command that we're going to be running alongside that test stop ping that we passed to it. And then we just use the sub-process module to run that. So all of that discussed is just talking and referring to Salt's remote execution capabilities. Obviously, there's a lot more you can do with Salt. So one of the really other popular aspects of Salt is the state system, and this is where we get a lot of our item potent behavior. So we refer to these SLS files as where you would define everything that you would want your machine to have to be in that certain state. And this is just a simple example. This one's using YAML. There are multiple other renderers that you can use within Salt, but YAML is one of the defaults, and this is a pretty simple example. We're just managing a file, and we want to state, or we want to put it worked into that file. So as you can see, we run it with Salt SSH. We run state.SLS, which is where we're going to run that SLS file, and it adds that new file and shows that a new change has occurred. Now if you were to run this again, as shown here, because it's item potent behavior, it doesn't try to change anything, and it just says it's in the correct state. So we have the execution modules, but we now have the state system, and the state system requires a little bit more because we have those SLS files, so we need to have some sort of deterministic behavior to figure out what files we also need to include and copy over to be able to run the state command. And what happens is it's the same as the execution module, except for when it comes to that command block. Command function, we run this one, runWfunc, and it essentially stands for wrapper functions. We have a bunch of, or a couple of wrapper modules within Salt SSH, and they're all displayed right there. And one of the things that makes it a bit tricky with using Salt SSH is because we've had to create these wrapper modules, and you'll see what we've done differently in the state.py example. It gets a little bit hard to manage both of those files because within Salt's code, we have a state.py, and Salt SSH, we have a state.py. So sometimes it becomes a little bit difficult to manage both of those files and make sure if the feature is added to one that we have to add it to the other. So that is one problem that we've run into with this current setup. So this function will set up all the pillar and grains, and within Salt, pillar and grains is just a lot of data. Pillar being, usually people use that for more of the sensitive information, and it's stored on the master. And grains being information about the minion stored on the minion. And any other ops that we need to be able to add to run these state files. Then we call all of those wrapper functions, and in this example, we're specifically going to call the state wrapper function. So as you can see that we're doing here is we're grabbing all the data that we're going to need for the call. We're initializing the function wrapper, and then we eventually we're just going to run it and return the JSON of the return. And this is how we're doing it in the state.py file. So in a normal SLS file, we do not have all these extra steps, because what these extra steps are doing for Salt SSH is, as you can see, we're preparing the tar. So within the tar, we determined what SLS files need to be added. One of the things that is a bit tricky is when people use includes in their file, you actually have to specify those separately in a configuration. So that is one thing I think could be improved upon as well. But yeah, so that prep trans tar, that's going to do all that magic there. And then we're going to sum up that and get it a hash. And then we're going to do the same thing we did before, initialize a single class, send over that tar, and then we'll run that command block again, and run essentially that same logic. So now that we've shown you what's going on underneath, you'll probably not remember all of that code. So what are some of the key architectural decisions to summarize all of that? One thing that's important to know is Salt was not modified to add Salt SSH. Salt SSH was added on top of it. And that's where you can see things like the wrapper functions and things like that where we didn't want to edit Salt. We use the tar ball where we tar everything up on the master. We determine what needs to be included on that, and then we send it over and run it on the Minion. As shown before, we use that Salt call. So Salt call, dash dash local, that's what provides us with the unidirectional communication. And we won't need to call out back to the master while the Minion is running. It's able to run everything it needs right there. It has all the information it needs. And then we use these wrapper functions where we're trying to provide the same capabilities that Salt has, but add it to Salt SSH. So what were some of the challenges when doing this? I've mentioned a lot of them already. Obviously, we lose some of the bi-directional where with SSH, we only have unidirectional, so that was a challenge. The wrapper challenges or functions, as I spoke before, is a bit difficult in terms of managing the code. And a lot of times when we see bugs within Salt SSH is because we don't have all the correct ops that we need. And ops is just a lot of the configs that we would need to run a Salt SSH command. So that is one of the challenges that I think we could do a bit better on and kind of make sure we have everything we need to be able to run within Salt and Salt SSH so there is more parity between the two. Okay, so what are our plans for the future? We actually are initially, we're initially talking right now. We don't have any set plans, but we do want to rearchitect Salt SSH and fix some of these issues that we have presented. Some of the ideas that we've been thinking on is trying to remove that Python dependency and how you do that. We have had some ideas of just including the minimal Python executable within the tar ball. Another reason I think this will be beneficial is eventually Python 2 is going to go away. And obviously users are still going to be using machines that have Python 2 on it, so if we remove the Python dependency and we include it in there, that won't be an issue. If we do go Python 3, we can use some of the capabilities of AsyncIO and hopefully we can resolve some of the issues that we stated such as the wrapper functions and so forth. All right. Does anybody have questions? Yeah, I don't know how to turn the light on. Now that you can see me, does anybody have any questions? Yeah. So what would be the benefits of using its agent versus using the agent? Yeah, essentially you don't have that agent. That is the benefit that I can see over Salt. It is slower because you don't, you know, obviously using SSH, but that is the benefit. You don't have to install the agent. Huh? Sorry. Okay. Go ahead. You can go again. Because I've been using Ansible a lot. Okay. So I'm kind of curious how do you, so have you done any testing on like performance-wise if you compare this solution with Ansible, for example? I haven't personally done. Bo, do you, have you done, I know you've done a lot of work with that. Have you done any performance testing? Yeah. Between the two? That could, that could be a problem. Okay. It's cashed. So Salt is copying itself to the temporary model. They are, they are, so next time we don't want to install it anymore, it's just for her monthly. So, so like faster than Ansible because it's already all the time. So not all friends are basically, yeah, not friends, yeah. As you mentioned, you copied the Salt binary, okay? There's just the Salt call. And then, yeah, all the other Salt dependencies you would need. Okay. So if you have a system where no binary exists, what happens next? The system doesn't have Salt on it. That's what it does. It copies over whatever Salt binaries that it would need and all the modules within it. Yeah. It's the same function as Ansible in that case. Okay. I don't, I don't know. Ansible initiate the connection. After that it copies some binaries. Okay. After that executes commands. Okay. Yeah. A little question here. Yeah. Colleague is asking about, what about Windows? Windows, we do have a Windows solution is enterprise only though. WinRM? Yeah. Yep. So I'm also, I'm thinking some of the, let's take up a little bit so that you can hear. Oh yeah, I should probably repeat the question too. No. No, it's not. It's been a long day, okay? The answer is no. So I'm just reading this just about having, as Salt has the same functionality as Ansible, unless you can explain to me how I can talk to her on, let's say a dump device, which I can do with Ansible, which is much, much like using expect. Sorry, I didn't quite understand that because the speakers are back there. Can you repeat the question? So I'm, I'm interested in functionality of, I'm currently also an Ansible user, but I'm interested in functionality in talking to, let's say dump devices, which have nothing on it where I need something like expect, which I can do with Ansible, but I don't know how to do it with Salt. With expect, is that what you're asking? How would you, sorry, I'm having a hard time understanding. With switch or something. Oh, like network switches, things like that. Oh yeah, so we, we have what's called Salt proxy, and that's how we, that's our solution for not even having to have Python. We actually have a solution with all salt, salt as a stage and it's salt proxy. I don't know what's going on underneath, but a lot of our network modules and all of that use heavily use salt proxy. So you'd want to look at that. Yeah. Hold on. Get some eggs. Thanks. Thank you. I don't use salt, so maybe the question is not exactly appropriate, but I was, I, I, I was thinking about the, where do you store the configuration about the machines when you don't have a model like minions and servers? Where do you store the, your assets, the information about the machine you will connect to? They are, they are described on, on client. The information such as. Let me say if you use a password or a certificate or which is the user you use to connect to the host, do you have a different YAML or configuration files, one for each client which uses SS and SSH? I think what you're asking is where is the information stored about the minion? Yeah. Yeah. Because the, the, the, so just to complete the question, because I was thinking about when you have a server, the information about the host you're managing with salt is in a way centralized on, on the server. Yeah. When you use this kind of approach, those disinformation are spread over the clients. Okay. Yeah. So I think what you're asking is, and what you're getting at is we have a concept of what we call pillar and grains. And pillar is what you would specify on the master. And a lot of people use these for like passwords. And yeah, it is just like a YAML file. You can use whatever render. The grains is initialized on, normally in the salt master salt minion, it's, it's initialized on the minion and it automatically determines all that information like operating system, host name, things like that. So it's automatically determining that. And that's what grains are. You also have the concept of the salt mine. And the salt mine will give you the ability to query all these minions and cache that information on the master. This is the pattern where you have the minions on your, on your, for example, for example. Yes. Oh, okay. So where you're specifying where your, what your hosts are and what, yeah, it's just the roster is what you would use. Because I was thinking about the adoption of this kind of pattern when you have, for example, your operation department, they are 10, 20 single clients. How do you, how do you ensure that the configuration they have is up to date and, and so on? The, like the Ansible inventory. So inventory files, for example, the list of hosts like this, to connect to this, I mean, salt has the same concept. Okay. Actually, it was, I believe you can use Ansible inventory as a source cluster. Yeah. Okay. And you had, you need to distribute and sync this, this information. Distribute and see the roster file. I mean, it's just a flat file or you can use a different sort of roster inquiry that information differently. But by default, it's just a flat file. Yeah. It could be, can it be remote to centralized or just a flat file? Yeah. It's just a flat file by default. But like I was showing you the other rosters, like you mentioned, you can use the target Ansible and you can do things like scan the network and things like that. Or you can create your own roster if you want to query that information for something else, like a database or something like that. You can create your own module and do that as well. Okay. Thank you. Sorry, hopefully I answered that well enough. Ansible does what? I before fear, I before sex, for sex. Yes, general information about the host host is distributed. So IPv4, IPv6, interface, information. Like how do you, does it use IPv4 and IPv6? Is that what you're asking? No, the host information about this variables, how they, how do you query like the idea? Yes, facts about the host. Yes, so it would be great. So you could just query the minion and use that and you can run grains.items and that would grab all the information about all the menus you'd need. Or like I was referring to, we have this idea of the salt mine and it can kind of do the same thing. You can clarify what information you want and it would cash it on the master. And that does work with salt SSH as well. Yeah. I'll take it. Yeah, yeah. So from what I understand, now you would have two different commands you want to invoke depending on what kind of setup you have or what kind of minior, your targeting. Is there any plans to kind of unify them so that you would define in your roster, I guess. I don't know enough about it, but if you define in the roster, okay, use classic agent base management with this minion and with this other minion, use salt SSH. So I know in our last release or it's coming up in this next release, one of our users added the functionality so that you can actually use the salt CLI and target both SSH and minions. And the same. Was that your question? Yeah. So that is actually added just recently. There is some sort of argument you'd have to use. I don't know it off the top of my head. Any other questions? I was asked, I just told the school of reamper as a wrapper that you can use with salt SSH. I believe. Last of five minutes. Yeah, so I'm almost positive you can because I didn't mention this, but before it actually submits that command to the minion, it actually is initializing a connection with the master event bus. And then when it gets that return, it actually returns that to the master event bus. So you could react off of that event. Yeah, so. All right. Thanks. Thank you very much. Thank you very much.
at Salt we moved from an agent-based solution to will not try to communicate back to the master. of variety. They can use both the agent and the that does a lot of the logic within Salt SSH. the key architectural decisions to summarize all was added on top of it. And that's where you can the master. We determine what needs to be 3, we can use some of the capabilities of AsyncIO how we, that's our solution for not even having