Host your own on-premise Ansible Galaxy
Bouterse, Brian
This talk will demo the setup and features of an on-premise software for storing, mirroring, and distributing Ansible Collection and Role content. This is analogous to an on-premise version of galaxy.ansible. To get up and running quickly, we’ll be using a pre-built container with pulp_ansible and galaxy_ng. A single container is all that is needed for setup. Once setup, I will demonstrate: - Creating one or more repositories to store Collections and Roles - Installing Role and Collection content using the ansible-galaxy CLI client from these repositories - Synchronizing Collections and Roles from galaxy.ansible - Uploading Collection content via ansible-galaxy CLI - Copying Collections and Roles between multiple repositories, simulating dev -> staging -> production environments - Perform these operations using a great UI, whenever possible, and APIs otherwise
 Okay, welcome to the host your own on-premise Ansible Galaxy talk. My name is Brian Vauters and you can get all of this content, these commands that I'm using today at the slides which are hosted here. This is what we'll be doing today. We're going to set up an on-premise Galaxy. We're going to do that with a user interface and without a user interface. We're going to sync in collections and roles from galaxy.ansible.com. We're going to install collections and roles using the Ansible Galaxy CLI. We'll upload collections using the Ansible Galaxy CLI. The Ansible Galaxy CLI does not support uploading roles. Also we're going to organize roles and collections in various repositories. Let's look at the software that we're going to use today. The first is Galaxy NG. That stands for Next Generation. It's a fork and rewrite of the original Galaxy code. It has a user interface. It's developed by the same team that runs galaxy.ansible.com. It only supports collections. It does not support roles and it's GPLv2. We're also going to look at pulp Ansible. Pulp Ansible is the back end for Galaxy NG. You use it there as well. It has an API and a CLI. This is a management CLI for performing various actions with pulp Ansible. It has no user interface. It has collections and role support. It also supports roles. It is GPLv2. Let's go ahead and deploy Galaxy NG. This is an all container based situation. You can use either Docker or Podman. All of my commands are with Podman, but it all works with Docker as well. This is the container name that contains Galaxy NG. I've already pulled that on my system. You can run this on your system. The first thing you'll do after you have the container before you start it is to create a settings file and a couple of folders that the container will need. Let's go ahead and do that. This is where we'll be doing that. I'm going to first make the folders. This is what the container will use. Now I'm going to make that settings file in there. You can see the settings file. There's nothing really complicated about it. It's got just three settings in there. There are more settings you can set. You can see the documentation for them. This is what you need to just get running. Next we'll start the container. You can do this with or without SE Linux. It runs on port 8080. Right now it's HTTP only. We're going to work on fixing that. You can do this with Docker or Podman. The container name is Galaxy NG. Again, it's using this particular image. I'm going to use the SE Linux command option. This is the same command only without SE Linux. I'm going to use this one with SE Linux. Let's go ahead and start the container. The container started. You can see it running here. We're going to load some initial data. Galaxy NG requires you to load this initial data. There's also an Ansible installer to install this with if you don't want to use containers. That takes care of this step for you automatically. This is just some database fixture data that they want that their application expects. Let's go ahead and run these commands. This is going to install the fixture data into the container. It's loading that data into the database. These containers use Postgres. It's contained inside of the container. You can also have Postgres running outside of the container if that's what you want to do. This installed the fixture data into the database. That's just a required step to get going. The next thing that we're going to want to do is to set an admin password. To do this, we'll use this Podman and ZEC command. We're going to run this reset admin password command. The admin user is named admin. What we're going to do right now is give it a password. I'm just going to give it the password, password because it's a password. At this point, the container is running and set up. We can see it if we access this site here. Again, it's running on localhost port 8080. This is your login screen. Your users will use this. This is running from the container. I just gave it the password. This is what the site looks like when you first log into it without any data inside of it. I also, before we go too much further, I want to just point out you can use the Podman logs or Docker logs command. The container is called GalaxyNG. This is a good way to look at the logs. Let's go ahead and just get those up on the terminal here. I'm going to use this terminal to do so. I'm going to use this terminal to do so. It's just a running list of logs. Let's look around the user interface to check out this project, GalaxyNG. The first thing to know is that this fits your data populates database with some repositories. There is a community repository. Also, a repository is a set of collections or roles. It's like a bucket where you can keep some number of collections or roles in. The community repository is where content that is synced from galaxy.ansible.com lands. We're going to look at that. Published repository is content that you have uploaded. This might be private content. When I say content, I mean collections in this case because GalaxyNG only supports collections. I should say collections that you've uploaded either from the ansible galaxy CLI or through the web UI, which we'll show both of those. Then there's also the RH certified content. This is for content that you sync from cloud.redhat.com. Also there is this rejected and staging repositories. This is where collections can be staged for an approval and review process. I'll show that as well. Also here in the repo management area is this remote section. In here, you can configure what will be synced from either cloud.redhat from galaxy.ansible.com for the community content or cloud.redhat.com. We're only showing off the community content here, but the other one works roughly the same. Let's look at how we're going to specify what we want to sync. We'll do the repositories and the remotes. Let's configure a remote. Here's a simple text file with a requirements file. This is the same kind of a file you would hand to the ansible galaxy CLI to sync collections. You can hand that same data to this website and it will perform the syncing for you. It can sync over time, like every day or something like that, to constantly bring in new versions if they're published out there. We're going to take this. We're going to install the pulp installer, which is an ansible collection itself. It also needs the ansible.puzzix installer. This is going to bring in two collections here. Let's take that copy data and let's go over here to the edit screen. Here it's going to say, I need a requirements YAML. To do that, it will... This is contain basic.yaml. It's got that same text that I just showed you. We'll go ahead and say, hand it the requirements file there. We'll save that and then we'll go ahead and hit sync. This is the project running a background task, which is interacting with galaxy.ansible.com and bringing down collection content into galaxy ng. That task finished. What we have to do is we have to select... This pull down bar has the three different repositories that can contain content, kind of like buckets. This is going to be in the community bucket. You can see that it brought down these two pieces of content, just like we specified in the requirements file. Next, we can also use... I showed that there's this other... Let's say that we want to bring down the Amazon AWS collection. Well we can go to repo management and you could add it in there and just make your requirements file bigger to sync more and more. Or I can just give it the AWS one, which only has AWS in it and save that, sync that. That will run and that'll take some time. We'll come back to that. This does get to the gotchas. It does not provide dependency resolution, so your requirements file has to have everything in it that you need. Also every time you sync, it overwrites the previous collections that were there. Let's go and look at the logs, see how this is drawing. You can see the logs are reading lots of data from galaxy.ansible.com here. This should finish in a moment. We'll see this gotcha because now, so this completed, so now if we go to community area again, oh look, the other things are gone and now we only have the AWS collection. This is one of the gotchas that it doesn't, it overwrites the content that was there previously. You want to make sure to just have whatever you want synced in there. You can see various things about it. There isn't much that's readable in here, but other questions do have meaningful things. It did sync a lot of versions though. You can see that there's a lot of versions of this collection that were brought down. Then next, we're going to, oh, also do not have a netRC file because the galaxy CLI looks at that and that might cause you some problems. If you're wondering why stuff isn't working, do not use a netRC file when you're using Galaxy Engine. Let's go configure the CLI. The way that this works is you want to get an API token because you need a token to interact with Ansible Galaxy. We're going to take this token here. What we're going to do is we're going to make an ansible.cfg file here. I'll just put that token in there. Then we'll go back to Galaxy and G and we'll go to the repo management area. What we need to do is connect the CLI to a particular bucket. This content is in the community content. Let's go and we're going to take the CLI configuration area and we'll copy this to our cookboard. Then we will go put this here. You can see it says token, put your token here. We'll just put the token there. You can see it's pointing the CLI at the container and it's pointing it at a particular repository inside that container. That's configured. I am going to go back here and just sync down the other content. For the demo purposes, I actually wanted the other content here. I'm going to bring back that other content from the other requirements file. Here at the content is back. What we'll do now is after we configured the CLI, we have the token and the config in there. I will install it with a command like ansible galaxy collection install. I'm going to install it to the local directory. That's this dash p part. I'm going to tell it to install this collection. This is the collection that we sync down from Galaxy. Here you can see, if I install this, oh, actually I have a net RC file. This is why you shouldn't have a net RC file. That's why you don't have a net RC file. Cool. Now, you can see that it did receive the content, the collection, and it installed it right from my local system and not from the internet. You can see it installed here in the ansible collections. Next, we want to cover installing, let's see, uploading from the user interface. To do this, you'll want to have a collection. I'm going to use this collection. This is from actuallygalaxy.ansible.com. I've downloaded it as a tar ball. This is the collection I'm going to upload. This is private content that you would want to upload. You need to create a namespace for it in the UI. The namespace for this is NewswinderD. Let's go create that namespace in user interface. My namespace is, you can create this and say NewswinderD. After creating that namespace, you can then upload a collection. You can do that by just going up here to the upload collection button. You can, here's my tar ball with the collection. I'll upload this. You can see it goes through this import process, which actually validates and lends it and stuff like that. You'll see the logs here. This collection has some quality issues, but the point is that it imported correctly. Now, but you'll notice, this is in the published area, because it's stuff you've uploaded, but oh look, it's not here. That's because by default, Galaxy NG has an approval process. In the approval area, you will see collections that have been uploaded, but they need review. You can reject or approve. You can view the import logs. You can download it and evaluate it. This provides a review process for your organization to make sure that only quality or trusted content is being uploaded. I'm going to go ahead and approve this. There are no more approvals now, and you'll see that collection is now available in the published repository. If I hook the CLI up to the published repository, I would be able to now install to that. You can also turn off the approval workflow, and you can do this by putting this setting in your settings file and restarting the container. The docs are there. There are users groups and permissions in Galaxy NG. We're not going to go over, like, demo them that much, but you can create users, multiple users, and you can then organize them into groups which have various permissions, which is really useful. You can say things like, oh, edit. I'm going to add namespaces, change uploads namespaces, or collections. I'm going to be able to actually modify them, or I can change remotes, or I can't. This allows you to have different users who are able to add content and other users who are able to only consume content. That's very useful. Next we're going to demo uploading content from the CLI. This is the same as the install command, only use the publish command to do so. I'm not going to demo it because we already configured the CLI, and so you just use that same command to provide the publish. Now we're going to move on to pulp ansible. I'm going to stop the Galaxy NG container because they're using the same port. You could have them running together, but I just don't. You're going to pull down this container. This container is called pulp-fidora31. This has pulp ansible in it. It also has a bunch of other content types, like a Python registry and RPM support. It doesn't have Python registry. It has a Python API, like PyPy, and it has a container registry. It also contains pulp ansible, so I'm just conveniently using this. There's also a Sentos 8 version that's available. If you just wanted pulp ansible, you could build a container that just has it, or we can publish one for you if that's helpful. You would run podman or drpull and pull this container down. I already have done that, so let's do basically the same thing. This gets one extra setting, but let's go ahead and create the settings for this. I'm going to just remove the Galaxy NG folder because I don't need it anymore. Let's go ahead and create those settings. I'm going to make the directory. You can see it made similar folders there. I'm going to populate the settings file. There are the settings there. Now we're going to go ahead and start the container. Same thing. There's an SCLinux option. There's a non-SCLinux option. I'm going to use the SCLinux option here. Let's go ahead and run this. You can see I'm calling this container pulp because pulp ansible is in it, and it's using this tag name here, a container name there. Let's actually start the command for the container. The container started. That's great. Let's go look back here. We similarly need to assign it an admin password. Again, very similar process here. Let's assign that. I'm going to assign. There are some warnings. This has to do with compatibility for things for the future, but it's inspected. I'm going to give it password again. I think I didn't type the same password in, so it rejected it. The second time it set it correctly. I am actually going to, well, let's turn it on here. For this, we're going to use a CLI. This is going to let us configure remotes and repositories and do the things that you were doing through the user interface, only you're going to use the pulp CLI. First you'll pip install pulp CLI. I've already done that. It's over here. This terminal has the pulp CLI. You can see it's in a virtual amp here. Then you're going to want to point it at your container. This is in the config pulp settings.toml. There should be an L at the end of there. You can see that I already have that. I am pointing it out the same container. That configures the CLI. Also, you want a net RC here. This is the admin and password that I set because the pulp CLI uses your net RC. I just set the net RC there. That will allow it to authenticate. Let's do some basics. Podman logs pulp will show you the logs. That's here. Then we're going to use pulp status. This is going to show off just the normal health status. You can see there are online workers. You can see all the software that's installed in here. One of which is pulp ansible 061. It's ready to serve content. It's connected to Redis and Postgres. That's good. It also has a browsable API documentation. You can see this here. This is hosted from the container. It's at this pulp API v3 docs link. This has a lot of software into it. There's a lot of APIs here. This is a great way to see what all APIs are being offered here. For instance, we just looked at the status API. This is the API that drove that command that we just used. Next, those are just some basics. We're going to create an ansible repository. This is using the pulp CLI that we were just talking about. We're going to create an ansible repository. I just created a repository named my repo. Then we're going to create an ansible remote. This is a collection type remote because it's going to sync collection content from gallupc.ansible.com, which I give in the URL. I'm just naming it basic. It has that same requirements file that we had from the previous part of the demo. I made the remote. Now I'm going to make a distribution. A distribution is a pulp term for taking a repository and making it accessible. It's accessible at this base path argument. I'm going to put foo in here. You're going to see foo in the URL. This lets you have content and choose where in the web server to make it available. I'm going to make a distribution that's going to take the repository and make it able to be interacted with from the CLI, for example. Now we have a distribution. We can go and view it through the Galaxy API. This is a web server. This is a container API. This is basically what the CLI uses. Let's go ahead and look at it because this is a great way to look at content in pulp. You can see there's no content in here because it's available, but we haven't done anything with it. The next thing we're going to do is we're going to sync the content. That's with this pulp ansible repository sync command. This is actually performing a sync similar to what Galaxy NG does. Only it's done through this API call. That is done. Because that sync completed, if we look back, if we look back, where previously there was zero content, now there's two content. You can see it brought down pulp installer and posted on the POSIX collection. Just like what you saw on Galaxy NG, only this is using pulp ansible. Next you would want to install from the CLI. To do that, you want to look at the distribution list that you have. You can do this with the pulp ansible distribution list command. Sorry, I'm getting my command pasted here. The distribution has this base path of foo and actually it has this client URL here. This is the URL that you'll end up handing to the ansible Galaxy CLI. You will end up installing content with a command that looks like this one. You can actually specify it right on the command line. This is that same URL with foo in it. I'm going to tell it to install the pulp installer. Let's go ahead and do that. Let me remove the previous installation of it. This will do the install. You can see that it brought in that content hosted from pulp ansible there. That is all I'm going to show for that. At this point, there's more demo content in here, but I'm going to stop here because I'm out of time. I'll just take you through the slide parts of this. There's uploading CLI. You can upload content directly from the CLI. It's pretty similar. You just end up using the publish command ansible Galaxy collection publish. You end up giving it that same URL. There is no approval workflow with pulp ansible. You end up just once you publish it, it's immediately available. You can then do role things. This is a very similar process. Only the remote that you create is called role. You end up giving it a URL like this one, which specifies role content by virtue of the filtering that you tell on the Galaxy API. If you left this part off and you created a remote, you would literally be mirroring all of galaxy.ansible.com roles. You perform the sync. It's the same process as collections. You install roles, the same thing. You end up handing it this dash S to point it at the container. You give it the name of the role that you want to install. You can configure the CLI permanently by putting a command like this into your ansible.cfg. I'm not going to demo copying, but you can see that there's a whole copying workflow for both role and collection content. This lets you organize custom repositories with mixtures of content. Maybe your custom content layered on top of content from ansible.gallaxy.com. These are some places that you can get help. There's mailing lists and stuff on Free Note and also bug filing docs. If you have any feedback from me, please email me at this address. Thank you for your time. I also want to say thank you to everyone who helped create this project.
 Okay, welcome to the host your own on-premise Ansible Galaxy talk.  My name is Brian Vauters and you can get all of this content, these commands that I'm using  today at the slides which are hosted here.  This is what we'll be doing today.  We're going to set up an on-premise Galaxy.  We're going to do that with a user interface and without a user interface.  We're going to sync in collections and roles from galaxy.ansible.com.  We're going to install collections and roles using the Ansible Galaxy CLI.  We'll upload collections using the Ansible Galaxy CLI.  The Ansible Galaxy CLI does not support uploading roles.  Also we're going to organize roles and collections in various repositories.  Let's look at the software that we're going to use today.  The first is Galaxy NG.  That stands for Next Generation.  It's a fork and rewrite of the original Galaxy code.  It has a user interface.  It's developed by the same team that runs galaxy.ansible.com.  It only supports collections.  It does not support roles and it's GPLv2.  We're also going to look at pulp Ansible.  Pulp Ansible is the back end for Galaxy NG.  You use it there as well.  It has an API and a CLI.  This is a management CLI for performing various actions with pulp Ansible.  It has no user interface.  It has collections and role support.  It also supports roles.  It is GPLv2.  Let's go ahead and deploy Galaxy NG.  This is an all container based situation.  You can use either Docker or Podman.  All of my commands are with Podman, but it all works with Docker as well.  This is the container name that contains Galaxy NG.  I've already pulled that on my system.  You can run this on your system.  The first thing you'll do after you have the container before you start it is to create  a settings file and a couple of folders that the container will need.  Let's go ahead and do that.  This is where we'll be doing that.  I'm going to first make the folders.  This is what the container will use.  Now I'm going to make that settings file in there.  You can see the settings file.  There's nothing really complicated about it.  It's got just three settings in there.  There are more settings you can set.  You can see the documentation for them.  This is what you need to just get running.  Next we'll start the container.  You can do this with or without SE Linux.  It runs on port 8080.  Right now it's HTTP only.  We're going to work on fixing that.  You can do this with Docker or Podman.  The container name is Galaxy NG.  Again, it's using this particular image.  I'm going to use the SE Linux command option.  This is the same command only without SE Linux.  I'm going to use this one with SE Linux.  Let's go ahead and start the container.  The container started.  You can see it running here.  We're going to load some initial data.  Galaxy NG requires you to load this initial data.  There's also an Ansible installer to install this with if you don't want to use containers.  That takes care of this step for you automatically.  This is just some database fixture data that they want that their application expects.  Let's go ahead and run these commands.  This is going to install the fixture data into the container.  It's loading that data into the database.  These containers use Postgres.  It's contained inside of the container.  You can also have Postgres running outside of the container if that's what you want to  do.  This installed the fixture data into the database.  That's just a required step to get going.  The next thing that we're going to want to do is to set an admin password.  To do this, we'll use this Podman and ZEC command.  We're going to run this reset admin password command.  The admin user is named admin.  What we're going to do right now is give it a password.  I'm just going to give it the password, password because it's a password.  At this point, the container is running and set up.  We can see it if we access this site here.  Again, it's running on localhost port 8080.  This is your login screen.  Your users will use this.  This is running from the container.  I just gave it the password.  This is what the site looks like when you first log into it without any data inside of it.  I also, before we go too much further, I want to just point out you can use the Podman  logs or Docker logs command.  The container is called GalaxyNG.  This is a good way to look at the logs.  Let's go ahead and just get those up on the terminal here.  I'm going to use this terminal to do so.  I'm going to use this terminal to do so.  It's just a running list of logs.  Let's look around the user interface to check out this project, GalaxyNG.  The first thing to know is that this fits your data populates database with some repositories.  There is a community repository.  Also, a repository is a set of collections or roles.  It's like a bucket where you can keep some number of collections or roles in.  The community repository is where content that is synced from galaxy.ansible.com lands.  We're going to look at that.  Published repository is content that you have uploaded.  This might be private content.  When I say content, I mean collections in this case because GalaxyNG only supports collections.  I should say collections that you've uploaded either from the ansible galaxy CLI or through  the web UI, which we'll show both of those.  Then there's also the RH certified content.  This is for content that you sync from cloud.redhat.com.  Also there is this rejected and staging repositories.  This is where collections can be staged for an approval and review process.  I'll show that as well.  Also here in the repo management area is this remote section.  In here, you can configure what will be synced from either cloud.redhat from galaxy.ansible.com  for the community content or cloud.redhat.com.  We're only showing off the community content here, but the other one works roughly the  same.  Let's look at how we're going to specify what we want to sync.  We'll do the repositories and the remotes.  Let's configure a remote.  Here's a simple text file with a requirements file.  This is the same kind of a file you would hand to the ansible galaxy CLI to sync collections.  You can hand that same data to this website and it will perform the syncing for you.  It can sync over time, like every day or something like that, to constantly bring in new versions  if they're published out there.  We're going to take this.  We're going to install the pulp installer, which is an ansible collection itself.  It also needs the ansible.puzzix installer.  This is going to bring in two collections here.  Let's take that copy data and let's go over here to the edit screen.  Here it's going to say, I need a requirements YAML.  To do that, it will... This is contain basic.yaml.  It's got that same text that I just showed you.  We'll go ahead and say, hand it the requirements file there.  We'll save that and then we'll go ahead and hit sync.  This is the project running a background task, which is interacting with galaxy.ansible.com  and bringing down collection content into galaxy ng.  That task finished.  What we have to do is we have to select... This pull down bar has the three different repositories  that can contain content, kind of like buckets.  This is going to be in the community bucket.  You can see that it brought down these two pieces of content, just like we specified in  the requirements file.  Next, we can also use... I showed that there's this other... Let's say that we want to bring  down the Amazon AWS collection.  Well we can go to repo management and you could add it in there and just make your requirements  file bigger to sync more and more.  Or I can just give it the AWS one, which only has AWS in it and save that, sync that.  That will run and that'll take some time.  We'll come back to that.  This does get to the gotchas.  It does not provide dependency resolution, so your requirements file has to have everything  in it that you need.  Also every time you sync, it overwrites the previous collections that were there.  Let's go and look at the logs, see how this is drawing.  You can see the logs are reading lots of data from galaxy.ansible.com here.  This should finish in a moment.  We'll see this gotcha because now, so this completed, so now if we go to community area  again, oh look, the other things are gone and now we only have the AWS collection.  This is one of the gotchas that it doesn't, it overwrites the content that was there previously.  You want to make sure to just have whatever you want synced in there.  You can see various things about it.  There isn't much that's readable in here, but other questions do have meaningful things.  It did sync a lot of versions though.  You can see that there's a lot of versions of this collection that were brought down.  Then next, we're going to, oh, also do not have a netRC file because the galaxy CLI looks  at that and that might cause you some problems.  If you're wondering why stuff isn't working, do not use a netRC file when you're using  Galaxy Engine.  Let's go configure the CLI.  The way that this works is you want to get an API token because you need a token to interact  with Ansible Galaxy.  We're going to take this token here.  What we're going to do is we're going to make an ansible.cfg file here.  I'll just put that token in there.  Then we'll go back to Galaxy and G and we'll go to the repo management area.  What we need to do is connect the CLI to a particular bucket.  This content is in the community content.  Let's go and we're going to take the CLI configuration area and we'll copy this to our cookboard.  Then we will go put this here.  You can see it says token, put your token here.  We'll just put the token there.  You can see it's pointing the CLI at the container and it's pointing it at a particular repository  inside that container.  That's configured.  I am going to go back here and just sync down the other content.  For the demo purposes, I actually wanted the other content here.  I'm going to bring back that other content from the other requirements file.  Here at the content is back.  What we'll do now is after we configured the CLI, we have the token and the config in there.  I will install it with a command like ansible galaxy collection install.  I'm going to install it to the local directory.  That's this dash p part.  I'm going to tell it to install this collection.  This is the collection that we sync down from Galaxy.  Here you can see, if I install this, oh, actually I have a net RC file.  This is why you shouldn't have a net RC file.  That's why you don't have a net RC file.  Cool.  Now, you can see that it did receive the content, the collection, and it installed it right  from my local system and not from the internet.  You can see it installed here in the ansible collections.  Next, we want to cover installing, let's see, uploading from the user interface.  To do this, you'll want to have a collection.  I'm going to use this collection.  This is from actuallygalaxy.ansible.com.  I've downloaded it as a tar ball.  This is the collection I'm going to upload.  This is private content that you would want to upload.  You need to create a namespace for it in the UI.  The namespace for this is NewswinderD.  Let's go create that namespace in user interface.  My namespace is, you can create this and say NewswinderD.  After creating that namespace, you can then upload a collection.  You can do that by just going up here to the upload collection button.  You can, here's my tar ball with the collection.  I'll upload this.  You can see it goes through this import process, which actually validates and lends it and  stuff like that.  You'll see the logs here.  This collection has some quality issues, but the point is that it imported correctly.  Now, but you'll notice, this is in the published area, because it's stuff you've uploaded,  but oh look, it's not here.  That's because by default, Galaxy NG has an approval process.  In the approval area, you will see collections that have been uploaded, but they need review.  You can reject or approve.  You can view the import logs.  You can download it and evaluate it.  This provides a review process for your organization to make sure that only quality or trusted  content is being uploaded.  I'm going to go ahead and approve this.  There are no more approvals now, and you'll see that collection is now available in the  published repository.  If I hook the CLI up to the published repository, I would be able to now install to that.  You can also turn off the approval workflow, and you can do this by putting this setting  in your settings file and restarting the container.  The docs are there.  There are users groups and permissions in Galaxy NG.  We're not going to go over, like, demo them that much, but you can create users, multiple  users, and you can then organize them into groups which have various permissions, which  is really useful.  You can say things like, oh, edit.  I'm going to add namespaces, change uploads namespaces, or collections.  I'm going to be able to actually modify them, or I can change remotes, or I can't.  This allows you to have different users who are able to add content and other users who  are able to only consume content.  That's very useful.  Next we're going to demo uploading content from the CLI.  This is the same as the install command, only use the publish command to do so.  I'm not going to demo it because we already configured the CLI, and so you just use that  same command to provide the publish.  Now we're going to move on to pulp ansible.  I'm going to stop the Galaxy NG container because they're using the same port.  You could have them running together, but I just don't.  You're going to pull down this container.  This container is called pulp-fidora31.  This has pulp ansible in it.  It also has a bunch of other content types, like a Python registry and RPM support.  It doesn't have Python registry.  It has a Python API, like PyPy, and it has a container registry.  It also contains pulp ansible, so I'm just conveniently using this.  There's also a Sentos 8 version that's available.  If you just wanted pulp ansible, you could build a container that just has it, or we  can publish one for you if that's helpful.  You would run podman or drpull and pull this container down.  I already have done that, so let's do basically the same thing.  This gets one extra setting, but let's go ahead and create the settings for this.  I'm going to just remove the Galaxy NG folder because I don't need it anymore.  Let's go ahead and create those settings.  I'm going to make the directory.  You can see it made similar folders there.  I'm going to populate the settings file.  There are the settings there.  Now we're going to go ahead and start the container.  Same thing.  There's an SCLinux option.  There's a non-SCLinux option.  I'm going to use the SCLinux option here.  Let's go ahead and run this.  You can see I'm calling this container pulp because pulp ansible is in it, and it's using  this tag name here, a container name there.  Let's actually start the command for the container.  The container started.  That's great.  Let's go look back here.  We similarly need to assign it an admin password.  Again, very similar process here.  Let's assign that.  I'm going to assign.  There are some warnings.  This has to do with compatibility for things for the future, but it's inspected.  I'm going to give it password again.  I think I didn't type the same password in, so it rejected it.  The second time it set it correctly.  I am actually going to, well, let's turn it on here.  For this, we're going to use a CLI.  This is going to let us configure remotes and repositories and do the things that you  were doing through the user interface, only you're going to use the pulp CLI.  First you'll pip install pulp CLI.  I've already done that.  It's over here.  This terminal has the pulp CLI.  You can see it's in a virtual amp here.  Then you're going to want to point it at your container.  This is in the config pulp settings.toml.  There should be an L at the end of there.  You can see that I already have that.  I am pointing it out the same container.  That configures the CLI.  Also, you want a net RC here.  This is the admin and password that I set because the pulp CLI uses your net RC.  I just set the net RC there.  That will allow it to authenticate.  Let's do some basics.  Podman logs pulp will show you the logs.  That's here.  Then we're going to use pulp status.  This is going to show off just the normal health status.  You can see there are online workers.  You can see all the software that's installed in here.  One of which is pulp ansible 061.  It's ready to serve content.  It's connected to Redis and Postgres.  That's good.  It also has a browsable API documentation.  You can see this here.  This is hosted from the container.  It's at this pulp API v3 docs link.  This has a lot of software into it.  There's a lot of APIs here.  This is a great way to see what all APIs are being offered here.  For instance, we just looked at the status API.  This is the API that drove that command that we just used.  Next, those are just some basics.  We're going to create an ansible repository.  This is using the pulp CLI that we were just talking about.  We're going to create an ansible repository.  I just created a repository named my repo.  Then we're going to create an ansible remote.  This is a collection type remote because it's going to sync collection content from gallupc.ansible.com,  which I give in the URL.  I'm just naming it basic.  It has that same requirements file that we had from the previous part of the demo.  I made the remote.  Now I'm going to make a distribution.  A distribution is a pulp term for taking a repository and making it accessible.  It's accessible at this base path argument.  I'm going to put foo in here.  You're going to see foo in the URL.  This lets you have content and choose where in the web server to make it available.  I'm going to make a distribution that's going to take the repository and make it able to  be interacted with from the CLI, for example.  Now we have a distribution.  We can go and view it through the Galaxy API.  This is a web server.  This is a container API.  This is basically what the CLI uses.  Let's go ahead and look at it because this is a great way to look at content in pulp.  You can see there's no content in here because it's available, but we haven't done anything  with it.  The next thing we're going to do is we're going to sync the content.  That's with this pulp ansible repository sync command.  This is actually performing a sync similar to what Galaxy NG does.  Only it's done through this API call.  That is done.  Because that sync completed, if we look back, if we look back, where previously there was  zero content, now there's two content.  You can see it brought down pulp installer and posted on the POSIX collection.  Just like what you saw on Galaxy NG, only this is using pulp ansible.  Next you would want to install from the CLI.  To do that, you want to look at the distribution list that you have.  You can do this with the pulp ansible distribution list command.  Sorry, I'm getting my command pasted here.  The distribution has this base path of foo and actually it has this client URL here.  This is the URL that you'll end up handing to the ansible Galaxy CLI.  You will end up installing content with a command that looks like this one.  You can actually specify it right on the command line.  This is that same URL with foo in it.  I'm going to tell it to install the pulp installer.  Let's go ahead and do that.  Let me remove the previous installation of it.  This will do the install.  You can see that it brought in that content hosted from pulp ansible there.  That is all I'm going to show for that.  At this point, there's more demo content in here, but I'm going to stop here because  I'm out of time.  I'll just take you through the slide parts of this.  There's uploading CLI.  You can upload content directly from the CLI.  It's pretty similar.  You just end up using the publish command ansible Galaxy collection publish.  You end up giving it that same URL.  There is no approval workflow with pulp ansible.  You end up just once you publish it, it's immediately available.  You can then do role things.  This is a very similar process.  Only the remote that you create is called role.  You end up giving it a URL like this one, which specifies role content by virtue of  the filtering that you tell on the Galaxy API.  If you left this part off and you created a remote, you would literally be mirroring  all of galaxy.ansible.com roles.  You perform the sync.  It's the same process as collections.  You install roles, the same thing.  You end up handing it this dash S to point it at the container.  You give it the name of the role that you want to install.  You can configure the CLI permanently by putting a command like this into your ansible.cfg.  I'm not going to demo copying, but you can see that there's a whole copying workflow  for both role and collection content.  This lets you organize custom repositories with mixtures of content.  Maybe your custom content layered on top of content from ansible.gallaxy.com.  These are some places that you can get help.  There's mailing lists and stuff on Free Note and also bug filing docs.  If you have any feedback from me, please email me at this address.  Thank you for your time.  I also want to say thank you to everyone who helped create this project.
hostyourownansiblegalaxy_0_3491.png hostyourownansiblegalaxy_3492_5111.png hostyourownansiblegalaxy_5112_5435.png hostyourownansiblegalaxy_5436_5723.png hostyourownansiblegalaxy_5724_8975.png hostyourownansiblegalaxy_8976_16211.png hostyourownansiblegalaxy_16212_16739.png hostyourownansiblegalaxy_16740_17735.png hostyourownansiblegalaxy_17736_17843.png hostyourownansiblegalaxy_17844_18035.png hostyourownansiblegalaxy_18036_19043.png hostyourownansiblegalaxy_19044_25103.png hostyourownansiblegalaxy_25104_27671.png hostyourownansiblegalaxy_27672_30071.png hostyourownansiblegalaxy_30072_30335.png hostyourownansiblegalaxy_30336_33515.png hostyourownansiblegalaxy_33516_33731.png hostyourownansiblegalaxy_33732_33779.png hostyourownansiblegalaxy_33780_34079.png hostyourownansiblegalaxy_34080_34187.png hostyourownansiblegalaxy_34188_34835.png hostyourownansiblegalaxy_34836_35243.png hostyourownansiblegalaxy_35244_36143.png hostyourownansiblegalaxy_36144_37319.png hostyourownansiblegalaxy_37320_37367.png hostyourownansiblegalaxy_37368_37451.png hostyourownansiblegalaxy_37452_37691.png hostyourownansiblegalaxy_37692_37751.png hostyourownansiblegalaxy_37752_38795.png hostyourownansiblegalaxy_38796_40079.png hostyourownansiblegalaxy_40080_43583.png hostyourownansiblegalaxy_43584_43895.png hostyourownansiblegalaxy_43896_44463.png
Okay, welcome to the host your own on-premise Ansible Galaxy talk. My name is Brian Vauters and you can get all of this content, these commands that I'm using today at the slides which are hosted here. This is what we'll be doing today. We're going to set up an on-premise Galaxy. We're going to do that with a user interface and without a user interface. We're going to sync in collections and roles from galaxy.ansible.com. We're going to install collections and roles using the Ansible Galaxy CLI. We'll upload collections using the Ansible Galaxy CLI. The Ansible Galaxy CLI does not support uploading roles. Also we're going to organize roles and collections in various repositories. Let's look at the software that we're going to use today. The first is Galaxy NG. That stands for Next Generation. It's a fork and rewrite of the original Galaxy code. It has a user interface. It's developed by the same team that runs galaxy.ansible.com. It only supports collections. It does not support roles and it's GPLv2. We're also going to look at pulp Ansible. Pulp Ansible is the back end for Galaxy NG. You use it there as well. It has an API and a CLI. This is a management CLI for performing various actions with pulp Ansible. It has no user interface. It has collections and role support. It also supports roles. It is GPLv2. Let's go ahead and deploy Galaxy NG. This is an all container based situation. You can use either Docker or Podman. All of my commands are with Podman, but it all works with Docker as well. This is the container name that contains Galaxy NG. I've already pulled that on my system. You can run this on your system. The first thing you'll do after you have the container before you start it is to create a settings file and a couple of folders that the container will need. Let's go ahead and do that. This is where we'll be doing that. I'm going to first make the folders. This is what the container will use. Now I'm going to make that settings file in there. You can see the settings file. There's nothing really complicated about it. It's got just three settings in there. There are more settings you can set. You can see the documentation for them. This is what you need to just get running. Next we'll start the container. You can do this with or without SE Linux. It runs on port 8080. Right now it's HTTP only. We're going to work on fixing that. You can do this with Docker or Podman. The container name is Galaxy NG. Again, it's using this particular image. I'm going to use the SE Linux command option. This is the same command only without SE Linux. I'm going to use this one with SE Linux. Let's go ahead and start the container. The container started. You can see it running here. We're going to load some initial data. Galaxy NG requires you to load this initial data. There's also an Ansible installer to install this with if you don't want to use containers. That takes care of this step for you automatically. This is just some database fixture data that they want that their application expects. Let's go ahead and run these commands. This is going to install the fixture data into the container. It's loading that data into the database. These containers use Postgres. It's contained inside of the container. You can also have Postgres running outside of the container if that's what you want to do. This installed the fixture data into the database. That's just a required step to get going. The next thing that we're going to want to do is to set an admin password. To do this, we'll use this Podman and ZEC command. We're going to run this reset admin password command. The admin user is named admin. What we're going to do right now is give it a password. I'm just going to give it the password, password because it's a password. At this point, the container is running and set up. We can see it if we access this site here. Again, it's running on localhost port 8080. This is your login screen. Your users will use this. This is running from the container. I just gave it the password. This is what the site looks like when you first log into it without any data inside of it. I also, before we go too much further, I want to just point out you can use the Podman logs or Docker logs command. The container is called GalaxyNG. This is a good way to look at the logs. Let's go ahead and just get those up on the terminal here. I'm going to use this terminal to do so. I'm going to use this terminal to do so. It's just a running list of logs. Let's look around the user interface to check out this project, GalaxyNG. The first thing to know is that this fits your data populates database with some repositories. There is a community repository. Also, a repository is a set of collections or roles. It's like a bucket where you can keep some number of collections or roles in. The community repository is where content that is synced from galaxy.ansible.com lands. We're going to look at that. Published repository is content that you have uploaded. This might be private content. When I say content, I mean collections in this case because GalaxyNG only supports collections. I should say collections that you've uploaded either from the ansible galaxy CLI or through the web UI, which we'll show both of those. Then there's also the RH certified content. This is for content that you sync from cloud.redhat.com. Also there is this rejected and staging repositories. This is where collections can be staged for an approval and review process. I'll show that as well. Also here in the repo management area is this remote section. In here, you can configure what will be synced from either cloud.redhat from galaxy.ansible.com for the community content or cloud.redhat.com. We're only showing off the community content here, but the other one works roughly the same. Let's look at how we're going to specify what we want to sync. We'll do the repositories and the remotes. Let's configure a remote. Here's a simple text file with a requirements file. This is the same kind of a file you would hand to the ansible galaxy CLI to sync collections. You can hand that same data to this website and it will perform the syncing for you. It can sync over time, like every day or something like that, to constantly bring in new versions if they're published out there. We're going to take this. We're going to install the pulp installer, which is an ansible collection itself. It also needs the ansible.puzzix installer. This is going to bring in two collections here. Let's take that copy data and let's go over here to the edit screen. Here it's going to say, I need a requirements YAML. To do that, it will... This is contain basic.yaml. It's got that same text that I just showed you. We'll go ahead and say, hand it the requirements file there. We'll save that and then we'll go ahead and hit sync. This is the project running a background task, which is interacting with galaxy.ansible.com and bringing down collection content into galaxy ng. That task finished. What we have to do is we have to select... This pull down bar has the three different repositories that can contain content, kind of like buckets. This is going to be in the community bucket. You can see that it brought down these two pieces of content, just like we specified in the requirements file. Next, we can also use... I showed that there's this other... Let's say that we want to bring down the Amazon AWS collection. Well we can go to repo management and you could add it in there and just make your requirements file bigger to sync more and more. Or I can just give it the AWS one, which only has AWS in it and save that, sync that. That will run and that'll take some time. We'll come back to that. This does get to the gotchas. It does not provide dependency resolution, so your requirements file has to have everything in it that you need. Also every time you sync, it overwrites the previous collections that were there. Let's go and look at the logs, see how this is drawing. You can see the logs are reading lots of data from galaxy.ansible.com here. This should finish in a moment. We'll see this gotcha because now, so this completed, so now if we go to community area again, oh look, the other things are gone and now we only have the AWS collection. This is one of the gotchas that it doesn't, it overwrites the content that was there previously. You want to make sure to just have whatever you want synced in there. You can see various things about it. There isn't much that's readable in here, but other questions do have meaningful things. It did sync a lot of versions though. You can see that there's a lot of versions of this collection that were brought down. Then next, we're going to, oh, also do not have a netRC file because the galaxy CLI looks at that and that might cause you some problems. If you're wondering why stuff isn't working, do not use a netRC file when you're using Galaxy Engine. Let's go configure the CLI. The way that this works is you want to get an API token because you need a token to interact with Ansible Galaxy. We're going to take this token here. What we're going to do is we're going to make an ansible.cfg file here. I'll just put that token in there. Then we'll go back to Galaxy and G and we'll go to the repo management area. What we need to do is connect the CLI to a particular bucket. This content is in the community content. Let's go and we're going to take the CLI configuration area and we'll copy this to our cookboard. Then we will go put this here. You can see it says token, put your token here. We'll just put the token there. You can see it's pointing the CLI at the container and it's pointing it at a particular repository inside that container. That's configured. I am going to go back here and just sync down the other content. For the demo purposes, I actually wanted the other content here. I'm going to bring back that other content from the other requirements file. Here at the content is back. What we'll do now is after we configured the CLI, we have the token and the config in there. I will install it with a command like ansible galaxy collection install. I'm going to install it to the local directory. That's this dash p part. I'm going to tell it to install this collection. This is the collection that we sync down from Galaxy. Here you can see, if I install this, oh, actually I have a net RC file. This is why you shouldn't have a net RC file. That's why you don't have a net RC file. Cool. Now, you can see that it did receive the content, the collection, and it installed it right from my local system and not from the internet. You can see it installed here in the ansible collections. Next, we want to cover installing, let's see, uploading from the user interface. To do this, you'll want to have a collection. I'm going to use this collection. This is from actuallygalaxy.ansible.com. I've downloaded it as a tar ball. This is the collection I'm going to upload. This is private content that you would want to upload. You need to create a namespace for it in the UI. The namespace for this is NewswinderD. Let's go create that namespace in user interface. My namespace is, you can create this and say NewswinderD. After creating that namespace, you can then upload a collection. You can do that by just going up here to the upload collection button. You can, here's my tar ball with the collection. I'll upload this. You can see it goes through this import process, which actually validates and lends it and stuff like that. You'll see the logs here. This collection has some quality issues, but the point is that it imported correctly. Now, but you'll notice, this is in the published area, because it's stuff you've uploaded, but oh look, it's not here. That's because by default, Galaxy NG has an approval process. In the approval area, you will see collections that have been uploaded, but they need review. You can reject or approve. You can view the import logs. You can download it and evaluate it. This provides a review process for your organization to make sure that only quality or trusted content is being uploaded. I'm going to go ahead and approve this. There are no more approvals now, and you'll see that collection is now available in the published repository. If I hook the CLI up to the published repository, I would be able to now install to that. You can also turn off the approval workflow, and you can do this by putting this setting in your settings file and restarting the container. The docs are there. There are users groups and permissions in Galaxy NG. We're not going to go over, like, demo them that much, but you can create users, multiple users, and you can then organize them into groups which have various permissions, which is really useful. You can say things like, oh, edit. I'm going to add namespaces, change uploads namespaces, or collections. I'm going to be able to actually modify them, or I can change remotes, or I can't. This allows you to have different users who are able to add content and other users who are able to only consume content. That's very useful. Next we're going to demo uploading content from the CLI. This is the same as the install command, only use the publish command to do so. I'm not going to demo it because we already configured the CLI, and so you just use that same command to provide the publish. Now we're going to move on to pulp ansible. I'm going to stop the Galaxy NG container because they're using the same port. You could have them running together, but I just don't. You're going to pull down this container. This container is called pulp-fidora31. This has pulp ansible in it. It also has a bunch of other content types, like a Python registry and RPM support. It doesn't have Python registry. It has a Python API, like PyPy, and it has a container registry. It also contains pulp ansible, so I'm just conveniently using this. There's also a Sentos 8 version that's available. If you just wanted pulp ansible, you could build a container that just has it, or we can publish one for you if that's helpful. You would run podman or drpull and pull this container down. I already have done that, so let's do basically the same thing. This gets one extra setting, but let's go ahead and create the settings for this. I'm going to just remove the Galaxy NG folder because I don't need it anymore. Let's go ahead and create those settings. I'm going to make the directory. You can see it made similar folders there. I'm going to populate the settings file. There are the settings there. Now we're going to go ahead and start the container. Same thing. There's an SCLinux option. There's a non-SCLinux option. I'm going to use the SCLinux option here. Let's go ahead and run this. You can see I'm calling this container pulp because pulp ansible is in it, and it's using this tag name here, a container name there. Let's actually start the command for the container. The container started. That's great. Let's go look back here. We similarly need to assign it an admin password. Again, very similar process here. Let's assign that. I'm going to assign. There are some warnings. This has to do with compatibility for things for the future, but it's inspected. I'm going to give it password again. I think I didn't type the same password in, so it rejected it. The second time it set it correctly. I am actually going to, well, let's turn it on here. For this, we're going to use a CLI. This is going to let us configure remotes and repositories and do the things that you were doing through the user interface, only you're going to use the pulp CLI. First you'll pip install pulp CLI. I've already done that. It's over here. This terminal has the pulp CLI. You can see it's in a virtual amp here. Then you're going to want to point it at your container. This is in the config pulp settings.toml. There should be an L at the end of there. You can see that I already have that. I am pointing it out the same container. That configures the CLI. Also, you want a net RC here. This is the admin and password that I set because the pulp CLI uses your net RC. I just set the net RC there. That will allow it to authenticate. Let's do some basics. Podman logs pulp will show you the logs. That's here. Then we're going to use pulp status. This is going to show off just the normal health status. You can see there are online workers. You can see all the software that's installed in here. One of which is pulp ansible 061. It's ready to serve content. It's connected to Redis and Postgres. That's good. It also has a browsable API documentation. You can see this here. This is hosted from the container. It's at this pulp API v3 docs link. This has a lot of software into it. There's a lot of APIs here. This is a great way to see what all APIs are being offered here. For instance, we just looked at the status API. This is the API that drove that command that we just used. Next, those are just some basics. We're going to create an ansible repository. This is using the pulp CLI that we were just talking about. We're going to create an ansible repository. I just created a repository named my repo. Then we're going to create an ansible remote. This is a collection type remote because it's going to sync collection content from gallupc.ansible.com, which I give in the URL. I'm just naming it basic. It has that same requirements file that we had from the previous part of the demo. I made the remote. Now I'm going to make a distribution. A distribution is a pulp term for taking a repository and making it accessible. It's accessible at this base path argument. I'm going to put foo in here. You're going to see foo in the URL. This lets you have content and choose where in the web server to make it available. I'm going to make a distribution that's going to take the repository and make it able to be interacted with from the CLI, for example. Now we have a distribution. We can go and view it through the Galaxy API. This is a web server. This is a container API. This is basically what the CLI uses. Let's go ahead and look at it because this is a great way to look at content in pulp. You can see there's no content in here because it's available, but we haven't done anything with it. The next thing we're going to do is we're going to sync the content. That's with this pulp ansible repository sync command. This is actually performing a sync similar to what Galaxy NG does. Only it's done through this API call. That is done. Because that sync completed, if we look back, if we look back, where previously there was zero content, now there's two content. You can see it brought down pulp installer and posted on the POSIX collection. Just like what you saw on Galaxy NG, only this is using pulp ansible. Next you would want to install from the CLI. To do that, you want to look at the distribution list that you have. You can do this with the pulp ansible distribution list command. Sorry, I'm getting my command pasted here. The distribution has this base path of foo and actually it has this client URL here. This is the URL that you'll end up handing to the ansible Galaxy CLI. You will end up installing content with a command that looks like this one. You can actually specify it right on the command line. This is that same URL with foo in it. I'm going to tell it to install the pulp installer. Let's go ahead and do that. Let me remove the previous installation of it. This will do the install. You can see that it brought in that content hosted from pulp ansible there. That is all I'm going to show for that. At this point, there's more demo content in here, but I'm going to stop here because I'm out of time. I'll just take you through the slide parts of this. There's uploading CLI. You can upload content directly from the CLI. It's pretty similar. You just end up using the publish command ansible Galaxy collection publish. You end up giving it that same URL. There is no approval workflow with pulp ansible. You end up just once you publish it, it's immediately available. You can then do role things. This is a very similar process. Only the remote that you create is called role. You end up giving it a URL like this one, which specifies role content by virtue of the filtering that you tell on the Galaxy API. If you left this part off and you created a remote, you would literally be mirroring all of galaxy.ansible.com roles. You perform the sync. It's the same process as collections. You install roles, the same thing. You end up handing it this dash S to point it at the container. You give it the name of the role that you want to install. You can configure the CLI permanently by putting a command like this into your ansible.cfg. I'm not going to demo copying, but you can see that there's a whole copying workflow for both role and collection content. This lets you organize custom repositories with mixtures of content. Maybe your custom content layered on top of content from ansible.gallaxy.com. These are some places that you can get help. There's mailing lists and stuff on Free Note and also bug filing docs. If you have any feedback from me, please email me at this address. Thank you for your time. I also want to say thank you to everyone who helped create this project.
you can get all of this content, these commands We're going to set up an on-premise Galaxy. We're collections using the Ansible Galaxy CLI. The This is where we'll be doing that. I'm going to Linux. I'm going to use this one with SE Linux. this point, the container is running and set up. that is synced from galaxy.ansible.com lands. certified content. This is for content that you rejected and staging repositories. This is where galaxy.ansible.com for the community content or token and the config in there. I will install it to create a namespace for it in the UI. The here. This is a great way to see what all APIs I'm going to make a distribution. A distribution want to install from the CLI. To do that, you point, there's more demo content in here, but I'm literally be mirroring all of galaxy.ansible.com