Habitat 101: An Introduction to Habitat
Timberman, Joshua
Habitat is an open-source framework that gives modern application teams an application-centric automation platform. Build, deploy, and manage modern and legacy applications with Habitat. In this talk we will explore: Introduction to Habitat The problems Habitat solves Getting started with Habitat Plans Using Habitat in your Chef workflow.
 Hi. Welcome to Habitat by Chef. I'm going to not use this remote. Habitat 101. So I'm going to tell you about Habitat today. So if you don't know who I am, I'm Joshua Timberman. I probably wrote cookbooks that you use for Chef that were rewritten by other people so that they're good. I'm a member of the Chef Board of Governance, which is part of our community guiding board. I've taught Chef and I am now on the Habitat core team. So I work full time with Habitat and making it do cool things. And I probably overheard something you said and posted on Twitter. So this is the abstract that is on the Chef Conf site. This is probably what you saw in the schedule. I'm kind of going to cover these, but really I have some goals for today that look more like this. I want you to understand why Habitat exists, if you don't already know. I want you to know what Habitat itself is and I want you to be able to get started using it. Now I don't have enough time to go into an in-depth tutorial about the various pieces and parts. Habitat is actually a really featureful and big software program. So there's a lot of things to know about it. It's application automation and of course for running any kind of applications we have to have infrastructure. So why do we run infrastructure? Right? Is it because it's fun? I mean I think it's fun. I'm a systems administrator. So I have a lot of fun with running infrastructure and being frustrated with computers because then you have that challenge that you get to solve. But do we run infrastructure because the business requires it? We don't, of course. We need to run applications, right? Well we don't actually need to run applications. We want to ship our ideas, right? We want to get those ideas out there. Of course those ideas are usually wrapped up in some kind of application that provides customer value in some way. So why do we have these applications? Is it because the business requires it? Software is eating the world, right? Everybody has applications. They're the foundation of any nutritious breakfast, you know, like breakfast tacos. My massage therapist has a website and application that she has to run so that people can schedule massages with her. You know, Etsy and Facebook and Google and your company all have applications. So that's great. We have applications. But how do we run them? Everyone runs their applications a little different. Some people run their applications using Java as the language that it's written in and then run it in the JVM. Some people write.NET applications and do it that way on Windows. And we can argue all day about which one is better, about which way to paint the, you know, which color to paint the bike shed of how we run our applications. At the end of the day, we end up with industry best practices that look like this. Glad Hadoop is in there. But these days, it's all about containers, right? And Adam talked about the, about using Docker and having that container, you know, that production difficulty and it gets more difficult as you go closer to production and then you have learning cliff and then you're like, okay, I've mastered Docker and then all of a sudden you're like ready to go to prod and you're like, oh, yeah, we have all these other things we have to do. And, you know, containers are our best practice. But, you know, not everybody's running containers. Some of us run our servers on bare metal. You can actually buy this t-shirt. The artist, the link is up there and I'll make the slides available later. But I'm definitely going to get that t-shirt because that's amazing. But at the end of the day, most of the time, what we have when it comes to running our applications and running our infrastructure is the configuration management trifecta, right? You have a package, you have a config file or 17 config files and then you have a service, right? Your application runs as a demonized service listening on port 80 or port 8000 or whatever it is and then people can use it. And we solve this problem with Chef by installing the package for your application and then having a separate config template that lives in a cookbook instead of your application repository. And then, you know, you manage the service of the application which is going to be system D or run it or, you know, all that. But like, Chef abstracts that away for you and, you know, we totally solve that problem, right? I mean, obviously not because I'm totally leaving you on. We have operating system considerations, all right? This is another area of bike shedding and, you know, arguing on the Internet. Do we use Red Hat? Do we use Debian? Are we just going to drop off a jar file because our application is Java? Are we just going to use TAR-GZs because we're on Gentoon or Slackware or just because we think that's the best way to package things? We need to run our service. Is it going to be system D or are we going to use upstart? It depends on which distribution we're using, right? Like, the operating, we've tied our infrastructure to these operating systems that have all these preconceived notions about how they think that people are going to want to run their applications. But they don't know your application. You know your application and you're like, putting it in, run it is ridiculous. Why would you do that? Or, I don't like system D because they have taken over the entire operating system. I don't want to do that either. And aside from that, where do you put your config files? Do you put them in slash Etsy? Do you put them in the applications directory? Are those config files dot comf? Like standard Unix config files that are not standard at all? Are they YAML where white space is pretty cool? Or a JSON where you have no comments? So, you know, we have all these different standards because of the way that we all run our applications, right? Because everybody's infrastructures all look the same, no? But the wonderful thing about standards is that none of them matter. Like I said, what the distributions are doing, the way that they think that you're going to run your app, they have no idea because they don't know your app. And the applications themselves don't care. They just need to be able to run somewhere. And what happens is at the end of the day, most organizations write these various layers between the application and the operating system. You've got things like Chef and Console and EtsyD and you have glue scripts that have been hobbled together over the various years. And then we also have language runtimes that ignore what the operating system is doing entirely, right? You have, you know, the languages are going to install in user local or they're going to install in varopt or some weird location that you never know what's going to happen with them because you don't necessarily know what their assumptions were about it. And it doesn't, you know, you just have to make your application work with it. So, the way that we solve this, of course, is with Habitat. And what Habitat is, it's application automation that enables modern application teams to build, deploy, and run any application in any environment. From traditional data centers to containerized microservices. That sounds like an awful lot of buzzwords. So, let's break that down a little bit and, you know, put it into terms that are maybe a little more meaningful. We're talking about the applications. So, we treat the applications as the first class citizen. So, the application comes first, its concerns are what we want to worry about. We don't want to worry about the distribution and how it's packaged things and all that. So, put the application first. Then we defer the decisions to runtime. Habitat itself exposes an interface that allows you to change the behavior and change the configuration of your applications at the runtime and handle reconfiguration dynamically. So, what that means is that you can set environment variables at runtime when you start the application to affect its behavior, rather than having to lay out an entire file system full of config files before you can actually start it. Habitat is automation that travels with the application. You package everything that your application needs to run with Habitat. You package up the application code. You package up any of the runtime dependencies. If you're using like a Rails app, you would bundle your dependencies into the package and you have all your configuration, lives in the applications repository, your startup scripts, all that stays with the application in a package. And the way we do these packages, we build them in a clean room called a studio. So, the studio is just a charooted environment that doesn't have anything in it. So, there's nothing from a distribution that's going to interfere with the way that you build your application. So, if you have a Java application, it's completely isolated away from any distribution-specific versions of the underlying libraries. So, you need to have all of your dependencies in there and you package those as Habitat as well. So, we'll get to that here pretty soon. So, when we enter a studio, we use the Hab command and has a studio sub-command and it gives us some output. It says it's creating a studio directory and then it implore imports the delivery secret origin key and that is because that's called delivery. It could be called core, it could be called Acme Corp, it could be called J Timberman. Well, you couldn't use J Timberman because that's my key, so, you know, you make your own. And then it imports the origin secret key and then it enters the studio. And now I'm in a 64-bit Linux environment. That's what Habitat runs on currently. And in here, I can perform the build of my package. And I can exit the studio and then upload the package and do things with it from there. And I can go back into the studio and have a repeatable build. So, if I need to release a new version of the application, I go into the studio, I do a new build of the application and it gives me the application artifact again for the new version. So Habitat itself is a packaging system. You build a packaged app for your application as an artifact and then you install that wherever it needs to run. So let's look at the anatomy of a Habitat package. Every Habitat package has a fully qualified package identifier that consists of the origin, in this case the origin is core, then the package name, in this case, this is the Hab package. It could be Apache, it could be MySQL, it could be Redis, whatever your cool application is named. Then you have a version. This version is going to be whatever the version of your app is. If I'm using Apache 2.2, then this would be 2.2. If I'm packaging up a dependency like Ruby and I want Ruby 2.3.0, that would be 2.3.0. And then the last part is the release. So every time you build an artifact with Habitat, the package gives a specific time stamp release so that you can always use the latest one or you can use a specific release of a package. Then the Habitat artifact itself uses the same kind of naming. This is the artifact for the HabSoup package. So it has the origin, the package name, the version, the release. And then also in this file it has the target which is the architecture and the operating system that that package is built for. And then we use a heart extension because we love you. So inside of a Habitat package, we extract those out to slash Hab, slash packages, slash the origin name. So in this case it's core, but it could be Acme or Microsoft or Chef or whoever packaged that thing, it would be that origin. And then the name of the package again, the version, the release. So that's a directory and it has a bunch of files in it. And the files in here that are in all caps are the metadata files that Habitat created when the package was built. There's these, there's some others. I'm going to talk about some of them. So the other thing is that in this package I have a bin directory with a Hab binary. Hab is a statically compiled binary. So it doesn't have any runtime libraries or anything else in there. But your application might have a lib directory or an include directory with headers. Or if you're using dependencies like libpng or libxml, those packages have those directories because that's part of that software. But if you have like a Rails application that has a config directory and your lib directory and a vendor directory where you put your bundle, like all that stuff would be in this directory here. Some of the metadata files are related to the dependencies of the package. You have a build depth, which is the dependencies for building this package. So if I'm in a studio and I'm going to build the package, those are the ones that will be installed on the system in that true to the environment so that the package can actually get built. Sometimes you need things like make and gcc for build time, but you don't need them for run time, right? So the run time dependencies are in the depths file. And that is just the things that that package needs to run. Now, this is an excerpt from the build depths for the hab package. Like I said, it's statically compiled and we use the musil c library so that it's smaller than using glibc. But these are all build dependencies. The hab package doesn't have any depths. There isn't a depths file because it has no run time dependencies. Now, if I had a package that had run time dependency, like a Java app would depend on a JDK, so I'd have a JDK package and it would be listed in here with core slash open JDK slash 1.8 dot whatever slash and then a release. So all of these dependencies are all the specific release of each one of those packages. So this way you know that when you install this package, it has these dependencies listed with these specific releases. So you can always get the same thing every time when you install it. When we think about the principles of immutable infrastructures where you've got, you know, you don't want things to change, you can verify that things haven't changed because we know that all these are the same dependencies. We also have a t-debs file and a build t-debs file. That's for transit dependencies. So those are things that a dependency that I have depends on, right? So I might need open SSL and open SSL needs Zlib, but I don't need Zlib directly. I know I'm just going to get it with open SSL. We also have a file called files and this is a list of all of the files in the package when the package was built. And we have the Blake2B checksum for, it's a hash of the content for every file. Now, the slide isn't wide enough because those checksums are really long, but just imagine that those are all one line each. And then this files file is signed so that you know that it hasn't been tampered with. So you can check this file on disk and verify that everything in here is exactly what you needed. For certain organizations that have compliance requirements or security requirements that say that they need to, you know, you need to verify everything, that is easy to do with this. Then we have an ident file and this is the fully qualified identifier for the package. And then one of the other metadata files that is important is the manifest. The manifest is a read, is like a markdown file that we can put on the Habitat web UI and display it nicely. So you can see manifest down there kind of just below the middle. That manifest is the markdown file that would be the same thing as if I looked at it on disk, but it just being markdown. Every package that you have that has binaries that you want to be able to run, well, you define where the binaries directory is going to be when the package is built. Usually it's bin and it will automatically create a path file that when Habitat runs it can load those for the application. All the dependencies paths files are added to the path when a user logs in. So you'll have access to all those locations because all the Habitat packages are all installed in slash hab. So we don't care if you have slash user bin bash on your system because we have a bash package, so if you need bash for Habitat you use Habitat's bash and then you know that it's there. And it would be added to your path. We also have an exposes file. So a lot of applications need to listen on some port somewhere, so you define what ports you want in your plan and then this exposes file gets written out with those ports. There's some other files that are useful if you're building like C software that has C flags or LD RunPath and LD flags. So after you've built your package on your studio, you have the heart file, then you upload that package or your packages to the depot. I mean the depot. Habitat also has built in service discovery that is over a gossip protocol. We use Swim for that and every system that starts up for running your application like you need a database, you need your app server, you might have a proxy load balancer in front of it, those all start up and they can discover each other through this gossip layer. And if you listen to Adam's keynote, he talked about how we wrote our own gossip layer we used to use at CD and that is based on the Swim protocol. So when we start up a service like the Core Redis package, we can see that there's some information about the gossip listener and the distributor and we also have the supervisor is running on this IP address. So when we start other services or other applications and services that need to use this, we can tell it to talk to that IP and then that will join the gossip ring. The Hab supervisor also has a configuration service. So that allows us to ask questions through a RESTful API. It runs on port 9631 and we can hit slash config and it will dump out the Habitat Toml configuration about that particular service. There's a couple of other endpoints that we can get to as well. These are all documented in the Habitat documentation. Because we work with a lot of companies that have security and compliance concerns and we want to be responsible grown-up operators, we want to have encryption and signing so that we can have verifiable packages and content that we know is not been tampered with, we use Libsodium and Blake2B for signing artifacts and hashing file contents for their integrity. And then for running the depot and the website, those both use HTTPS and we support HTTPS. If you wanted to run your own depot, then you'd be able to do that with HTTPS in your own wildcard cert or whatever. The signed packages, one of the cool things about the heart artifact files, the Habitat artifact, is that you can just use head to read the header that has the signing information. So in this section, I'm going to cover a number of Habitat basics. I'm going to go to my terminal here in a moment and I'm going to run some commands and I'm not going to go super in depth because of time constraints but I want to give everybody a feel for what it's like to, what the various parts of Habitat are and what it's like to work with it. The website, Habitat.sh, has tutorials and a demo that you can work through on your own and then there's a download link, you can download the binary and start using it right away. So I'm going to go to my terminal. So Habitat is a get-reported repository on GitHub and I have it here. I'm going to go back over here. So for example purposes, we have a Ruby on Rails application. Now I know not everybody runs Ruby on Rails and that's fine. Whatever your application you run, you'd go to whatever its repository is and then you would, if you're going to start using Habitat, you'd create a Habitat directory because that's the default location that Habitat's going to look for when it's working with your packet or when it's working with your application to package it and then we have a plan.sh. So the plan.sh is a bash shell script. It's all Apache open source, Apache license open sourced and when we write a plan, it's a shell script that is going to be loaded up by a build script that just sources it. So the build script sources my plan.sh and it's going to pick up a bunch of variables and some callback functions to tell it how it's supposed to build this package. So all these package underscore variables are all the things that inform the build script about various parts of it so that it can build this application. So the package name is Ruby Rails sample. So remember we saw slash core slash hab. We would see core slash Ruby dash Rails dash sample. We have the package version is 0.0.1. I could change that and make it a 0.0.2. I can make it a 0.2.2. Whatever my versioning scheme for my application is going to be, I can change that. I was working with this in Chef delivery which is now the workflow part of Chef Automate. So I have the package origin as delivery. The maintainer is the Habitat maintainers. This application itself is MIT licensed. I forked it from a sample that Heroku uses. I download the source from GitHub and I get the package Shasam. I know that that file is going to have that Shasam and I get downloads. I define all my dependencies as a bash array. So it needs bundler and CA search and it also needs, I don't know why you need node to run a Rails app but like I'm not going to judge, it's cool. This application uses Postgres so it needs Postgres in order to have the Postgres client libraries and of course it needs Ruby. The build depths, it needs core utils, GCC and make. Those won't be installed on the resulting container or resulting, sorry, they won't be in the dependencies when I install it on a container or a VM or something. And we have the lib directories, the include directories and we expose port 3000. Now these callbacks or these functions are callbacks that are, they override functions that are defined in the build script. So the build script has a prepare function that does a thing. It has a build, a do build function that does something depending on what the software is. But I'm going to override those and make it do what I need for my application in order to be built. So I'm setting some local variables. I export some environment variables. I handle Nukogiri, note that we're using the system libraries. This would use the hab system libraries. Has anybody compiled Nukogiri for a Rails app? Hands up? Yeah. So we solved this problem for you. You're welcome. This sets the bundle config. Like this is all the stuff that I need in order to have this Rails app be something that I can deploy. We bundle install into the vendor bundle directory. And so when I run this on my system, when I go into a studio and I do a build, it's going to load this plan and get to work. The do install is similar to like if you have a, if you know like GNU C times, you do like configure make make install. That would be the do build. And then the do install would be make install, right? So that's, you know, those are, those functions are broken up like that. This is how we're doing the install for this package. So this may not finish before, before I'm out of time. So I just wanted to talk about some of the output here real quick. So remember I said we enter a studio to do our build. And I'm entering the studio from the Ruby Rails sample application repository. Then when I go in there, like we saw on that slide, I run build. And this is build as a script that loads the SRC Habitat plan.sh. It looks in Habitat for the plan.sh file. You can also specify that if you want. And now it is loaded. So before we have nothing here, before this colon, but now we have Ruby Rails sample because it's loaded that package name and it's going to continue doing the rest of the build steps. It automatically went and installed core utils and GCC and make and their dependencies, right? So we've got, we've got some libmpc and so on. So it installed all the things that it needs to build the software and it sets some, some variables and here's where it's running bundler. So we're building the package. It's taking a little while because bundle installed can take some time. So the other things that you work with in Habitat, when you're going to run your application, you need to run it in some way, right? So we have a, the notion of hooks and you have a, sorry, we have the run script which does whatever you need to do to initialize and run your application. So in this case, we need to set some environment variables so that the application knows where the gem home is or the gem path is, what the, what the path where commands are going to be located are, what the Rails environment is and then at the end here, we actually start the application and the run script should run something that runs in the foreground because it's going to supervise it and handle it. So you don't, you don't want something to background because then it'll think, you know, it'll, it'll have lost what, what, what it thought it was supervising. So at the end here, I love it when a plan.sh comes together. At the end here, we have a, where's the line? We have a tar ball that was created and then it gets compressed with XZ. So we have a tar dot XZ and then we sign it and create the heart file. So this is the artifact that gets created. So when Habitat runs, it creates this HabCache artifacts directory and it then writes in the artifacts that it builds out to the heart files there. We also create a slash src slash results because this is a clean room environment. It's a, it's actually, this is actually running in a Docker container so that it's completely isolated and also because I can't run Linux directly on OSX, right? Because, you know, you have to have a VM of some sort running. So I just use the native Docker for OSX. So if I exit this, then try and go back in the studio, all the stuff in slash Hab is gone. But we have a results directory because the Ruby Rails sample, the application directory that I was in, when I did the Hab Studio Enter, it bind mounts that directory as slash src. So that's how I have all the, all the Rails sample stuff. And then the results directory has my artifact as well. So when I exit, I still have access to it. We also have a last build.m which is just a key values of information about the build that you can use inside Jenkins or Chef Automate or whatever your CI, CD system might be. You can, you can use this elsewhere. Like if you needed to publish a list of package check sums on a website for your downloads, and you can get that out of this file. All right. So getting low on time, so I'm going to continue with the rest of the presentation as soon as I figure out where my turn off mirroring is. Okay. So let's talk a little bit about Habitat and Chef workflows. So we love Chef recipes. And in order to use Habitat and Chef recipes, right now, you want, you, like the main thing would be you'd want to get Habitat installed on the systems where you're going to run your applications that are packaged with Habitat. So we can use a remote file resource to download the core Hab package, and then we can extract it using tail and xzcat and tar. Now this, of course, isn't convergent. It's item potent because it will run that every time, and it won't change anything because it's going to get the same version each time unless you change those attributes. But it shouldn't, like, run that every time, so you'd want to have, like, a guard in there. But for this, it's fine. This is just the basic Chef recipe to get Habitat installed and then install the package with Hab install. We do some of this in our build cookbook. So we have a build cookbook for using Habitat inside a Chef workflow if you're using Chef Automate. Now that build cookbook has a detailed readme, so if you're using Chef Automate and you have a workflow, which is called delivery two days ago, you would use that build cookbook in your own build cookbook. You can depend on it and make sure you get it by updating the Birx file and the metadata in your build cookbook. We also have a Ruby client library for Habitat, and this is used in the build cookbook for a few things, like being able to get information about a package from the repository. So just a little bit of sample code. We can create a new instance of the Habitat client, and we can also point it at a different depot. So if we set up our own depot on local host or on some internal system instead of using the public one, we can do that. Then we can show packages, and we also have the notion in the depot to promote packages, which puts them into a different view, so you could have like a production, a testing, a staging. With delivery or with Chef workflows, you'd have views that you'd promote through the various union rehearsal and delivered phases inside of that Chef workflow. It's available as a Ruby gem from rubygems.org, so you just use the new cookbook metadata feature where you can say a gem to install and it will handle it, or you can just put Chef gem Habitat client in your recipe. We don't yet have Habitat package and Habitat service resources, but we'll have those soon so that we can have those that have install, not be an execute block. That'd be pretty cool. And that's about it for the Chef workflow section. So I have a few, as we wrap up, I have a few frequently anticipated questions, so hopefully get ahead of some of the questions since we won't have a whole lot of time for that today, but the first one, does Habitat replace Chef? No, because you still have infrastructure you need to manage. You still have users that need permissions and you need to set up your monitoring and logging and all that stuff for the operating system and for, you know, the infrastructure that your application is run on. Habitat supplements Chef by giving you a first class application-centric way to package and manage your applications. And then you can use that within Chef. How does Habitat work with containers? Because Docker, Docker, Docker, Docker, Docker, mushroom, mushroom. Docker is a big buzzword lately. It's really easy to use and it's delightful for people and they want to be able to have containers run their applications as microservices, right? Like, we've got to do that. That's what all the Internet says. Habitat works with containers in that, in a couple of ways. We do all of our development of our packages currently in Docker containers because we know that we can use a minimal Docker container that has nothing else in it. We can use like an Alpine Linux and build our packages on that. And those packages, by the way, will work on CentOS or Ubuntu because they use all of Habitat's dependencies, not whatever the underlying OS' dependencies are. So we do clean room builds inside a Docker container. So that's one way we use it. And the other way is that you can, after you've built the package, so I did that build, I could then run Hab package export Docker core slash Ruby slash Ruby Rails sample and then I would get a Docker container out of it. Right now we can export packages as Docker containers but in the future we probably want to be able to do CoreOS' Rocket, EC2, AMIs and that sort of thing. Those are like future things that we might be building at some point. Next question is, does Habitat work on Windows? No? Sorry. But the good news is that we're going to work on porting Habitat to Windows. And we want the Windows user experience to be as easy and delightful and as intuitive as Windows admins would expect that to be. Because you... We love our Windows administrative brethren and sisters and, you know, they don't necessarily have an easy time with the tools that they're given to do automation. And Microsoft is really coming around on this and we're working with them to make this a really, truly delightful experience. So it's... I don't know what the, like the roadmap terms of that are but I know that it's something that we're going to work on very soon and having Habitat work on Windows is a high priority for us. Yeah. Like all kinds of good stuff will be coming out of that. Does Habitat work with the OS or distro packages? Nope. It doesn't use them. It doesn't care about them. It doesn't matter if you're using Ubuntu or CentOS because you're doing all of your builds in this clean environment that don't have any of that. So it doesn't need any of that. So I can take that Ruby Rails sample heart file that I just built on an OSX system that was running a Linux container and I can take that package and install it on Ubuntu 16.04 or on Red Hat 7 or on Gen 2 and as long as I have a 64-bit kernel greater than 2.6.32, it will work because it's all using Habitat and Habitat's binaries and the way that it was built. So it gives us a lot of freedom to be able to move our applications to any kind of any other Linux platform that we want. Maybe the question, would it work on FreeBSD? Probably with the Linux compatibility layer of FreeBSD. Probably. I don't know I haven't tested it. We also have a whole lot of core packages. So we have most all the dependencies you would need for common application runtime frameworks like Rails and Erlang and Python. We're working on PHP. We're working on Java. We have all these are in the core repository that the Habitat team maintains. The astute may wonder, didn't you just create a new distribution? I mean, sort of. But what we did is we created a way for people to be able to build and manage their applications no matter what their distribution is. Is it better than Hadoop? Of course it's better than Hadoop. All right. We have the Habitat zone on the opposite side of the building around the corner. Like, I don't know. I don't even know where it is from here. But it's over to Room 404. So if you can't find it, just look for the Habitat banners and stuff. And unfortunately, you just missed Fletcher Nickel talking about Habitat Internals. Sorry. And we'll also have Nathan Harvey talking about building the Habitat community over there. I think I just left. Then why can't you say from scratch by example? So not a Docker question, but what's still needed that you actually needed a base OS in the container? Okay. Besides the base OS in the container, you need the Hab binary. And when we use Hab Studio Enter, it automatically installs what it needs to do that. So one of the things that we learned while we were building this is what Adam was talking about with having humane software. And in order to have humane software, you need to have software that behaves pretty much like you'd expect. And so there's a little bit of magic, but there's a whole lot of delight. We've worked really hard to make it easy to use. And it actually, once you start using it, it ends up being more simple than you thought it was going to be. You'd be like, okay, so how do I get all the things I'm going to need to install this? It's like, it gets all that for you. It handles it automatically. So there's a question I have about upgradability of the packages afterwards. Right? We quite often end up deploying a package out. And then we have no control necessarily of when a customer is going to upgrade a package. How does it handle that? Or does it handle that? Or is that just like left up to us to figure it out? So yes, it can really automatically upgrade the packages. It can detect from the people that the package has been able to do that. We're working on fleshing out what all those update strategies might be. But the main things are that we have the capability to automatically upgrade. And we have the capability to not automatically upgrade. And doing like rolling kinds of things is still something that's kind of in the air about that. For a complex, for example, a Python dependency chain, for example, we would need to pin many, many upstream Python pit modules. Is that possible in the plan? Can we set different sources? Do they all need to come from the, I guess, the depot? So I don't know a lot about Python. So one of the things we do with Ruby, you have a gem file. And in that gem file, you define your dependencies. I imagine Python has a similar thing. And then when you do a bundle, you get a gem file.lock. And then you can commit that lock file to your version control system so that you can always get those specific dependencies. And imagine Python has a similar thing. Yeah. So you can always get the same thing. You would always get those dependencies from doing the build. You'd have the build step that would install those for you inside of your app. Now, if you wanted, you could have, you could package up all those various Python packages and make those available. But what I would probably recommend is to go the route where you vendor that all into your application so that it's all in lockstep. And that way we don't have, like, version sprawl of all these various Python packages out in the depot. Josh, you said you just said to package up all of the dependencies, be it bundler or pip or what have you. So are habitat containers or are they composable? Can I start at some sort of base layer with a set of things that I typically would use? Or will I have to come from the clean room as it were every time? I would recommend going from the clean room every time. Habitat doesn't do like a layering kind of thing like Docker does. It doesn't have that as a thing. Like when I was going through the deps files, the build depths, the depth, the depth, those are all the versions of all the packages that it depends on. So it wouldn't be built up over like, you wouldn't have like multiple versions of OpenSSL as a dependency. You'd depend on whatever the latest one was or if you needed a specific release, something like that. Does that answer your question? Yes. Okay. And then I saw in the plan file that you exposed ports for the app. Would multiple hab installs need or, I'm sorry, would they conflict in any possible way other than those ports if you were to, like we were talking about attempt to compose different hab installs on the same machine? Can you repeat it, Josh? No, if you didn't hear the question, you exposed ports, would you have possible conflicts of those ports between multiple different packages? Yeah, I mean, you could. No. Because we isolate all the dependencies. So like every time you start up an EC2 instance or VMware image or Brackenstack, a system that's going to run hab on bare metal, you know, each of those would be their own isolated system and they would have like whatever service that they're running would, it would get its dependencies for that. Now, if you had multiple ports running, I mean, they're on different systems, they're on different IPs, right? So if you had multiple packages running on the same system, there's a way to do that with Habitat. It's called the director and it can run like you could run Postgres and Ruby app and put engine X in front of it and run that all on one single system and they wouldn't have any, assume they wouldn't have any conflicts because you'd want to expose them on different ports.
 Hi. Welcome to Habitat by Chef. I'm going to not use this remote. Habitat 101. So I'm  going to tell you about Habitat today. So if you don't know who I am, I'm Joshua Timberman.  I probably wrote cookbooks that you use for Chef that were rewritten by other people so  that they're good. I'm a member of the Chef Board of Governance, which is part of our community  guiding board. I've taught Chef and I am now on the Habitat core team. So I work full time with  Habitat and making it do cool things. And I probably overheard something you said and posted on  Twitter. So this is the abstract that is on the Chef Conf site. This is probably what you saw in  the schedule. I'm kind of going to cover these, but really I have some goals for today that look  more like this. I want you to understand why Habitat exists, if you don't already know. I want  you to know what Habitat itself is and I want you to be able to get started using it. Now I don't  have enough time to go into an in-depth tutorial about the various pieces and parts. Habitat is  actually a really featureful and big software program. So there's a lot of things to know about  it. It's application automation and of course for running any kind of applications we have to have  infrastructure. So why do we run infrastructure? Right? Is it because it's fun? I mean I think it's  fun. I'm a systems administrator. So I have a lot of fun with running infrastructure and being  frustrated with computers because then you have that challenge that you get to solve. But do we run  infrastructure because the business requires it? We don't, of course. We need to run applications,  right? Well we don't actually need to run applications. We want to ship our ideas, right? We want to  get those ideas out there. Of course those ideas are usually wrapped up in some kind of application  that provides customer value in some way. So why do we have these applications? Is it because the  business requires it? Software is eating the world, right? Everybody has applications. They're the  foundation of any nutritious breakfast, you know, like breakfast tacos. My massage therapist has a  website and application that she has to run so that people can schedule massages with her. You  know, Etsy and Facebook and Google and your company all have applications. So that's great. We have  applications. But how do we run them? Everyone runs their applications a little different. Some  people run their applications using Java as the language that it's written in and then run it in  the JVM. Some people write.NET applications and do it that way on Windows. And we can argue all day  about which one is better, about which way to paint the, you know, which color to paint the bike shed  of how we run our applications. At the end of the day, we end up with industry best practices that  look like this. Glad Hadoop is in there. But these days, it's all about containers, right? And Adam  talked about the, about using Docker and having that container, you know, that production difficulty  and it gets more difficult as you go closer to production and then you have learning cliff and  then you're like, okay, I've mastered Docker and then all of a sudden you're like ready to go to  prod and you're like, oh, yeah, we have all these other things we have to do. And, you know,  containers are our best practice. But, you know, not everybody's running containers. Some of us  run our servers on bare metal. You can actually buy this t-shirt. The artist, the link is up there  and I'll make the slides available later. But I'm definitely going to get that t-shirt because  that's amazing. But at the end of the day, most of the time, what we have when it comes to running  our applications and running our infrastructure is the configuration management trifecta, right?  You have a package, you have a config file or 17 config files and then you have a service, right?  Your application runs as a demonized service listening on port 80 or port 8000 or whatever it  is and then people can use it. And we solve this problem with Chef by installing the package for  your application and then having a separate config template that lives in a cookbook instead of your  application repository. And then, you know, you manage the service of the application which is  going to be system D or run it or, you know, all that. But like, Chef abstracts that away for you  and, you know, we totally solve that problem, right? I mean, obviously not because I'm totally  leaving you on. We have operating system considerations, all right? This is another area  of bike shedding and, you know, arguing on the Internet. Do we use Red Hat? Do we use Debian?  Are we just going to drop off a jar file because our application is Java? Are we just going to use  TAR-GZs because we're on Gentoon or Slackware or just because we think that's the best way to package  things? We need to run our service. Is it going to be system D or are we going to use upstart? It  depends on which distribution we're using, right? Like, the operating, we've tied our infrastructure  to these operating systems that have all these preconceived notions about how they think that  people are going to want to run their applications. But they don't know your application. You know  your application and you're like, putting it in, run it is ridiculous. Why would you do that? Or,  I don't like system D because they have taken over the entire operating system. I don't want to do  that either. And aside from that, where do you put your config files? Do you put them in slash  Etsy? Do you put them in the applications directory? Are those config files dot comf? Like  standard Unix config files that are not standard at all? Are they YAML where white space is pretty  cool? Or a JSON where you have no comments? So, you know, we have all these different standards  because of the way that we all run our applications, right? Because everybody's infrastructures all  look the same, no? But the wonderful thing about standards is that none of them matter. Like I  said, what the distributions are doing, the way that they think that you're going to run your  app, they have no idea because they don't know your app. And the applications themselves don't  care. They just need to be able to run somewhere. And what happens is at the end of the day, most  organizations write these various layers between the application and the operating system. You've  got things like Chef and Console and EtsyD and you have glue scripts that have been hobbled  together over the various years. And then we also have language runtimes that ignore what the  operating system is doing entirely, right? You have, you know, the languages are going to install  in user local or they're going to install in varopt or some weird location that you never know  what's going to happen with them because you don't necessarily know what their assumptions were  about it. And it doesn't, you know, you just have to make your application work with it. So, the way  that we solve this, of course, is with Habitat. And what Habitat is, it's application automation  that enables modern application teams to build, deploy, and run any application in any environment.  From traditional data centers to containerized microservices. That sounds like an awful lot of  buzzwords. So, let's break that down a little bit and, you know, put it into terms that are maybe  a little more meaningful. We're talking about the applications. So, we treat the applications as  the first class citizen. So, the application comes first, its concerns are what we want to worry  about. We don't want to worry about the distribution and how it's packaged things and all that. So,  put the application first. Then we defer the decisions to runtime. Habitat itself exposes an  interface that allows you to change the behavior and change the configuration of your applications  at the runtime and handle reconfiguration dynamically. So, what that means is that you can  set environment variables at runtime when you start the application to affect its behavior,  rather than having to lay out an entire file system full of config files before you can actually  start it. Habitat is automation that travels with the application. You package everything that your  application needs to run with Habitat. You package up the application code. You package up any of  the runtime dependencies. If you're using like a Rails app, you would bundle your dependencies  into the package and you have all your configuration, lives in the applications repository, your  startup scripts, all that stays with the application in a package. And the way we do these packages,  we build them in a clean room called a studio. So, the studio is just a charooted environment that  doesn't have anything in it. So, there's nothing from a distribution that's going to interfere with  the way that you build your application. So, if you have a Java application, it's completely  isolated away from any distribution-specific versions of the underlying libraries. So, you  need to have all of your dependencies in there and you package those as Habitat as well. So,  we'll get to that here pretty soon. So, when we enter a studio, we use the Hab command and has a  studio sub-command and it gives us some output. It says it's creating a studio directory and then  it implore imports the delivery secret origin key and that is because that's called delivery. It  could be called core, it could be called Acme Corp, it could be called J Timberman. Well,  you couldn't use J Timberman because that's my key, so, you know, you make your own. And then it  imports the origin secret key and then it enters the studio. And now I'm in a 64-bit Linux  environment. That's what Habitat runs on currently. And in here, I can perform the build of my  package. And I can exit the studio and then upload the package and do things with it from there.  And I can go back into the studio and have a repeatable build. So, if I need to release a new  version of the application, I go into the studio, I do a new build of the application and it gives  me the application artifact again for the new version. So Habitat itself is a packaging system.  You build a packaged app for your application as an artifact and then you install that wherever it  needs to run. So let's look at the anatomy of a Habitat package. Every Habitat package has a fully  qualified package identifier that consists of the origin, in this case the origin is core, then the  package name, in this case, this is the Hab package. It could be Apache, it could be MySQL, it could  be Redis, whatever your cool application is named. Then you have a version. This version is going to  be whatever the version of your app is. If I'm using Apache 2.2, then this would be 2.2. If I'm  packaging up a dependency like Ruby and I want Ruby 2.3.0, that would be 2.3.0. And then the last  part is the release. So every time you build an artifact with Habitat, the package gives a specific  time stamp release so that you can always use the latest one or you can use a specific release of a  package. Then the Habitat artifact itself uses the same kind of naming. This is the artifact for the  HabSoup package. So it has the origin, the package name, the version, the release. And then also in  this file it has the target which is the architecture and the operating system that that  package is built for. And then we use a heart extension because we love you. So inside of a  Habitat package, we extract those out to slash Hab, slash packages, slash the origin name. So in this  case it's core, but it could be Acme or Microsoft or Chef or whoever packaged that thing, it would be  that origin. And then the name of the package again, the version, the release. So that's a  directory and it has a bunch of files in it. And the files in here that are in all caps are the  metadata files that Habitat created when the package was built. There's these, there's some others.  I'm going to talk about some of them. So the other thing is that in this package I have a bin  directory with a Hab binary. Hab is a statically compiled binary. So it doesn't have any runtime  libraries or anything else in there. But your application might have a lib directory or an  include directory with headers. Or if you're using dependencies like libpng or libxml, those  packages have those directories because that's part of that software. But if you have like a  Rails application that has a config directory and your lib directory and a vendor directory where  you put your bundle, like all that stuff would be in this directory here. Some of the metadata  files are related to the dependencies of the package. You have a build depth, which is the  dependencies for building this package. So if I'm in a studio and I'm going to build the package,  those are the ones that will be installed on the system in that true to the environment so that  the package can actually get built. Sometimes you need things like make and gcc for build time,  but you don't need them for run time, right? So the run time dependencies are in the depths file.  And that is just the things that that package needs to run. Now, this is an excerpt from the  build depths for the hab package. Like I said, it's statically compiled and we use the musil c  library so that it's smaller than using glibc. But these are all build dependencies. The hab  package doesn't have any depths. There isn't a depths file because it has no run time dependencies.  Now, if I had a package that had run time dependency, like a Java app would depend on a JDK,  so I'd have a JDK package and it would be listed in here with core slash open JDK slash 1.8 dot  whatever slash and then a release. So all of these dependencies are all the specific release of each  one of those packages. So this way you know that when you install this package, it has these  dependencies listed with these specific releases. So you can always get the same thing every time  when you install it. When we think about the principles of immutable infrastructures where  you've got, you know, you don't want things to change, you can verify that things haven't changed  because we know that all these are the same dependencies. We also have a t-debs file and a  build t-debs file. That's for transit dependencies. So those are things that a dependency that I have  depends on, right? So I might need open SSL and open SSL needs Zlib, but I don't need Zlib directly.  I know I'm just going to get it with open SSL. We also have a file called files and this is a  list of all of the files in the package when the package was built. And we have the Blake2B  checksum for, it's a hash of the content for every file. Now, the slide isn't wide enough  because those checksums are really long, but just imagine that those are all one line each.  And then this files file is signed so that you know that it hasn't been tampered with.  So you can check this file on disk and verify that everything in here is exactly what you  needed. For certain organizations that have compliance requirements or security requirements  that say that they need to, you know, you need to verify everything, that is easy to do with this.  Then we have an ident file and this is the fully qualified identifier for the package.  And then one of the other metadata files that is important is the manifest. The manifest is a read,  is like a markdown file that we can put on the Habitat web UI and display it nicely.  So you can see manifest down there kind of just below the middle. That manifest is the  markdown file that would be the same thing as if I looked at it on disk, but it just  being markdown. Every package that you have that has binaries that you want to be able  to run, well, you define where the binaries directory is going to be when the package  is built. Usually it's bin and it will automatically create a path file that when Habitat runs  it can load those for the application. All the dependencies paths files are added to  the path when a user logs in. So you'll have access to all those locations because all  the Habitat packages are all installed in slash hab. So we don't care if you have slash  user bin bash on your system because we have a bash package, so if you need bash for Habitat  you use Habitat's bash and then you know that it's there. And it would be added to your  path. We also have an exposes file. So a lot of applications need to listen on some port  somewhere, so you define what ports you want in your plan and then this exposes file gets  written out with those ports. There's some other files that are useful if you're building  like C software that has C flags or LD RunPath and LD flags. So after you've built your package  on your studio, you have the heart file, then you upload that package or your packages to  the depot. I mean the depot. Habitat also has built in service discovery that is over  a gossip protocol. We use Swim for that and every system that starts up for running your  application like you need a database, you need your app server, you might have a proxy  load balancer in front of it, those all start up and they can discover each other through  this gossip layer. And if you listen to Adam's keynote, he talked about how we wrote our  own gossip layer we used to use at CD and that is based on the Swim protocol. So when  we start up a service like the Core Redis package, we can see that there's some information  about the gossip listener and the distributor and we also have the supervisor is running  on this IP address. So when we start other services or other applications and services  that need to use this, we can tell it to talk to that IP and then that will join the gossip  ring. The Hab supervisor also has a configuration service. So that allows us to ask questions  through a RESTful API. It runs on port 9631 and we can hit slash config and it will dump  out the Habitat Toml configuration about that particular service. There's a couple  of other endpoints that we can get to as well. These are all documented in the Habitat documentation.  Because we work with a lot of companies that have security and compliance concerns and  we want to be responsible grown-up operators, we want to have encryption and signing so  that we can have verifiable packages and content that we know is not been tampered with, we  use Libsodium and Blake2B for signing artifacts and hashing file contents for their integrity.  And then for running the depot and the website, those both use HTTPS and we support HTTPS.  If you wanted to run your own depot, then you'd be able to do that with HTTPS in your  own wildcard cert or whatever. The signed packages, one of the cool things  about the heart artifact files, the Habitat artifact, is that you can just use head to  read the header that has the signing information. So in this section, I'm going to cover a number  of Habitat basics. I'm going to go to my terminal here in a moment and I'm going to  run some commands and I'm not going to go super in depth because of time constraints  but I want to give everybody a feel for what it's like to, what the various parts of Habitat  are and what it's like to work with it. The website, Habitat.sh, has tutorials and  a demo that you can work through on your own and then there's a download link, you can  download the binary and start using it right away. So I'm going to go to my terminal. So Habitat is a get-reported  repository on GitHub and I have it here. I'm going to go back over here. So for example  purposes, we have a Ruby on Rails application. Now I know not everybody runs Ruby on Rails  and that's fine. Whatever your application you run, you'd go to whatever its repository  is and then you would, if you're going to start using Habitat, you'd create a Habitat  directory because that's the default location that Habitat's going to look for when it's  working with your packet or when it's working with your application to package it and then  we have a plan.sh. So the plan.sh is a bash shell script. It's all Apache open source,  Apache license open sourced and when we write a plan, it's a shell script that is going  to be loaded up by a build script that just sources it. So the build script sources my  plan.sh and it's going to pick up a bunch of variables and some callback functions to  tell it how it's supposed to build this package. So all these package underscore variables  are all the things that inform the build script about various parts of it so that it can build  this application. So the package name is Ruby Rails sample. So remember we saw slash core  slash hab. We would see core slash Ruby dash Rails dash sample. We have the package version  is 0.0.1. I could change that and make it a 0.0.2. I can make it a 0.2.2. Whatever my  versioning scheme for my application is going to be, I can change that. I was working with  this in Chef delivery which is now the workflow part of Chef Automate. So I have the package  origin as delivery. The maintainer is the Habitat maintainers. This application itself  is MIT licensed. I forked it from a sample that Heroku uses. I download the source from  GitHub and I get the package Shasam. I know that that file is going to have that Shasam  and I get downloads. I define all my dependencies as a bash array. So it needs bundler and CA  search and it also needs, I don't know why you need node to run a Rails app but like  I'm not going to judge, it's cool. This application uses Postgres so it needs Postgres in order  to have the Postgres client libraries and of course it needs Ruby. The build depths,  it needs core utils, GCC and make. Those won't be installed on the resulting container or  resulting, sorry, they won't be in the dependencies when I install it on a container or a VM  or something. And we have the lib directories, the include directories and we expose port  3000. Now these callbacks or these functions are callbacks that are, they override functions  that are defined in the build script. So the build script has a prepare function that does  a thing. It has a build, a do build function that does something depending on what the  software is. But I'm going to override those and make it do what I need for my application  in order to be built. So I'm setting some local variables. I export some environment  variables. I handle Nukogiri, note that we're using the system libraries. This would use  the hab system libraries. Has anybody compiled Nukogiri for a Rails app? Hands up? Yeah.  So we solved this problem for you. You're welcome. This sets the bundle config. Like  this is all the stuff that I need in order to have this Rails app be something that I  can deploy. We bundle install into the vendor bundle directory. And so when I run this on  my system, when I go into a studio and I do a build, it's going to load this plan and  get to work. The do install is similar to like if you have a, if you know like GNU C  times, you do like configure make make install. That would be the do build. And then the do  install would be make install, right? So that's, you know, those are, those functions are broken  up like that. This is how we're doing the install for this package.  So this may not finish before, before I'm out of time. So I just wanted to talk about  some of the output here real quick. So remember I said we enter a studio to do our build.  And I'm entering the studio from the Ruby Rails sample application repository. Then  when I go in there, like we saw on that slide, I run build. And this is build as a script  that loads the SRC Habitat plan.sh. It looks in Habitat for the plan.sh file. You can  also specify that if you want. And now it is loaded. So before we have nothing here,  before this colon, but now we have Ruby Rails sample because it's loaded that package name  and it's going to continue doing the rest of the build steps. It automatically went and  installed core utils and GCC and make and their dependencies, right? So we've got, we've  got some libmpc and so on. So it installed all the things that it needs to build the  software and it sets some, some variables and here's where it's running bundler. So  we're building the package. It's taking a little while because bundle installed can  take some time. So the other things that you work with in Habitat, when you're going to  run your application, you need to run it in some way, right? So we have a, the notion  of hooks and you have a, sorry, we have the run script which does whatever you need to  do to initialize and run your application. So in this case, we need to set some environment  variables so that the application knows where the gem home is or the gem path is, what the,  what the path where commands are going to be located are, what the Rails environment  is and then at the end here, we actually start the application and the run script should  run something that runs in the foreground because it's going to supervise it and handle  it. So you don't, you don't want something to background because then it'll think, you  know, it'll, it'll have lost what, what, what it thought it was supervising.  So at the end here, I love it when a plan.sh comes together. At the end here, we have a,  where's the line? We have a tar ball that was created and then it gets compressed with  XZ. So we have a tar dot XZ and then we sign it and create the heart file. So this is the  artifact that gets created. So when Habitat runs, it creates this HabCache artifacts directory  and it then writes in the artifacts that it builds out to the heart files there. We also  create a slash src slash results because this is a clean room environment. It's a, it's  actually, this is actually running in a Docker container so that it's completely isolated  and also because I can't run Linux directly on OSX, right? Because, you know, you have  to have a VM of some sort running. So I just use the native Docker for OSX. So if I exit  this, then try and go back in the studio, all the stuff in slash Hab is gone. But we  have a results directory because the Ruby Rails sample, the application directory that  I was in, when I did the Hab Studio Enter, it bind mounts that directory as slash src.  So that's how I have all the, all the Rails sample stuff. And then the results directory  has my artifact as well. So when I exit, I still have access to it. We also have a last  build.m which is just a key values of information about the build that you can use inside Jenkins  or Chef Automate or whatever your CI, CD system might be. You can, you can use this elsewhere.  Like if you needed to publish a list of package check sums on a website for your downloads,  and you can get that out of this file. All right. So getting low on time, so I'm going  to continue with the rest of the presentation as soon as I figure out where my turn off  mirroring is. Okay. So let's talk a little bit about Habitat and Chef workflows. So  we love Chef recipes. And in order to use Habitat and Chef recipes, right now, you want,  you, like the main thing would be you'd want to get Habitat installed on the systems where  you're going to run your applications that are packaged with Habitat. So we can use a  remote file resource to download the core Hab package, and then we can extract it using  tail and xzcat and tar. Now this, of course, isn't convergent. It's item potent because  it will run that every time, and it won't change anything because it's going to get the same  version each time unless you change those attributes. But it shouldn't, like, run that  every time, so you'd want to have, like, a guard in there. But for this, it's fine. This  is just the basic Chef recipe to get Habitat installed and then install the package with  Hab install. We do some of this in our build cookbook. So we have a build cookbook for using  Habitat inside a Chef workflow if you're using Chef Automate. Now that build cookbook has  a detailed readme, so if you're using Chef Automate and you have a workflow, which is  called delivery two days ago, you would use that build cookbook in your own build cookbook.  You can depend on it and make sure you get it by updating the Birx file and the metadata  in your build cookbook. We also have a Ruby client library for Habitat, and this is used  in the build cookbook for a few things, like being able to get information about a package  from the repository. So just a little bit of sample code. We can create a new instance  of the Habitat client, and we can also point it at a different depot. So if we set up our  own depot on local host or on some internal system instead of using the public one, we  can do that. Then we can show packages, and we also have the notion in the depot to promote  packages, which puts them into a different view, so you could have like a production,  a testing, a staging. With delivery or with Chef workflows, you'd have views that you'd  promote through the various union rehearsal and delivered phases inside of that Chef workflow.  It's available as a Ruby gem from rubygems.org, so you just use the new cookbook metadata  feature where you can say a gem to install and it will handle it, or you can just put  Chef gem Habitat client in your recipe. We don't yet have Habitat package and Habitat  service resources, but we'll have those soon so that we can have those that have install,  not be an execute block. That'd be pretty cool. And that's about it for the Chef workflow  section. So I have a few, as we wrap up, I have a few frequently anticipated questions,  so hopefully get ahead of some of the questions since we won't have a whole lot of time for  that today, but the first one, does Habitat replace Chef? No, because you still have infrastructure  you need to manage. You still have users that need permissions and you need to set up your  monitoring and logging and all that stuff for the operating system and for, you know,  the infrastructure that your application is run on. Habitat supplements Chef by giving  you a first class application-centric way to package and manage your applications. And  then you can use that within Chef. How does Habitat work with containers? Because Docker,  Docker, Docker, Docker, Docker, mushroom, mushroom. Docker is a big buzzword lately. It's really  easy to use and it's delightful for people and they want to be able to have containers  run their applications as microservices, right? Like, we've got to do that. That's what all  the Internet says. Habitat works with containers in that, in a couple of ways. We do all of  our development of our packages currently in Docker containers because we know that we  can use a minimal Docker container that has nothing else in it. We can use like an Alpine  Linux and build our packages on that. And those packages, by the way, will work on CentOS  or Ubuntu because they use all of Habitat's dependencies, not whatever the underlying  OS' dependencies are. So we do clean room builds inside a Docker container. So that's  one way we use it. And the other way is that you can, after you've built the package, so  I did that build, I could then run Hab package export Docker core slash Ruby slash Ruby Rails  sample and then I would get a Docker container out of it. Right now we can export packages  as Docker containers but in the future we probably want to be able to do CoreOS' Rocket, EC2,  AMIs and that sort of thing. Those are like future things that we might be building at  some point. Next question is, does Habitat work on Windows? No? Sorry. But the good news  is that we're going to work on porting Habitat to Windows. And we want the Windows user experience  to be as easy and delightful and as intuitive as Windows admins would expect that to be.  Because you... We love our Windows administrative brethren and sisters and, you know, they don't  necessarily have an easy time with the tools that they're given to do automation. And Microsoft  is really coming around on this and we're working with them to make this a really, truly  delightful experience. So it's... I don't know what the, like the roadmap terms of that  are but I know that it's something that we're going to work on very soon and having Habitat  work on Windows is a high priority for us. Yeah. Like all kinds of good stuff will be  coming out of that. Does Habitat work with the OS or distro packages? Nope. It doesn't  use them. It doesn't care about them. It doesn't matter if you're using Ubuntu or CentOS because  you're doing all of your builds in this clean environment that don't have any of that. So  it doesn't need any of that. So I can take that Ruby Rails sample heart file that I just  built on an OSX system that was running a Linux container and I can take that package  and install it on Ubuntu 16.04 or on Red Hat 7 or on Gen 2 and as long as I have a 64-bit  kernel greater than 2.6.32, it will work because it's all using Habitat and Habitat's  binaries and the way that it was built. So it gives us a lot of freedom to be able to  move our applications to any kind of any other Linux platform that we want. Maybe the question,  would it work on FreeBSD? Probably with the Linux compatibility layer of FreeBSD. Probably.  I don't know I haven't tested it. We also have a whole lot of core packages. So we have  most all the dependencies you would need for common application runtime frameworks like  Rails and Erlang and Python. We're working on PHP. We're working on Java. We have all  these are in the core repository that the Habitat team maintains. The astute may wonder,  didn't you just create a new distribution? I mean, sort of. But what we did is we created  a way for people to be able to build and manage their applications no matter what their distribution  is. Is it better than Hadoop? Of course it's better than Hadoop. All right. We have the  Habitat zone on the opposite side of the building around the corner. Like, I don't know. I don't  even know where it is from here. But it's over to Room 404. So if you can't find it,  just look for the Habitat banners and stuff. And unfortunately, you just missed Fletcher  Nickel talking about Habitat Internals. Sorry. And we'll also have Nathan Harvey talking  about building the Habitat community over there.  I think I just left. Then why can't you say from scratch by example? So not a Docker question,  but what's still needed that you actually needed a base OS in the container?  Okay. Besides the base OS in the container, you need the Hab binary. And when we use Hab  Studio Enter, it automatically installs what it needs to do that. So one of the things  that we learned while we were building this is what Adam was talking about with having  humane software. And in order to have humane software, you need to have software that behaves  pretty much like you'd expect. And so there's a little bit of magic, but there's a whole  lot of delight. We've worked really hard to make it easy to use. And it actually, once  you start using it, it ends up being more simple than you thought it was going to be.  You'd be like, okay, so how do I get all the things I'm going to need to install this?  It's like, it gets all that for you. It handles it automatically.  So there's a question I have about upgradability of the packages afterwards. Right? We quite  often end up deploying a package out. And then we have no control necessarily of when  a customer is going to upgrade a package. How does it handle that? Or does it handle  that? Or is that just like left up to us to figure it out?  So yes, it can really automatically upgrade the packages. It can detect from the people  that the package has been  able to do that. We're working on fleshing out what all those update strategies might  be. But the main things are that we have the capability to automatically upgrade. And we  have the capability to not automatically upgrade. And doing like rolling kinds of things is  still something that's kind of in the air about that.  For a complex, for example, a Python dependency chain, for example, we would need to pin many,  many upstream Python pit modules. Is that possible in the plan? Can we set different  sources? Do they all need to come from the, I guess, the depot?  So I don't know a lot about Python. So one of the things we do with Ruby, you have a  gem file. And in that gem file, you define your dependencies. I imagine Python has a  similar thing. And then when you do a bundle, you get a gem file.lock. And then you can  commit that lock file to your version control system so that you can always get those specific  dependencies. And imagine Python has a similar thing. Yeah. So you can always get the same  thing. You would always get those dependencies from doing the build. You'd have the build  step that would install those for you inside of your app. Now, if you wanted, you could  have, you could package up all those various Python packages and make those available. But  what I would probably recommend is to go the route where you vendor that all into your  application so that it's all in lockstep. And that way we don't have, like, version  sprawl of all these various Python packages out in the depot.  Josh, you said you just said to package up all of the dependencies, be it bundler or  pip or what have you. So are habitat containers or are they composable? Can I start at some  sort of base layer with a set of things that I typically would use? Or will I have to come  from the clean room as it were every time?  I would recommend going from the clean room every time. Habitat doesn't do like a layering  kind of thing like Docker does. It doesn't have that as a thing. Like when I was going  through the deps files, the build depths, the depth, the depth, those are all the versions  of all the packages that it depends on. So it wouldn't be built up over like, you wouldn't  have like multiple versions of OpenSSL as a dependency. You'd depend on whatever the  latest one was or if you needed a specific release, something like that. Does that answer  your question?  Yes.  Okay. And then I saw in the plan file that you exposed ports for the app. Would multiple  hab installs need or, I'm sorry, would they conflict in any possible way other than those  ports if you were to, like we were talking about attempt to compose different hab installs  on the same machine?  Can you repeat it, Josh?  No, if you didn't hear the question, you exposed ports, would you have possible conflicts of  those ports between multiple different packages? Yeah, I mean, you could. No. Because we isolate  all the dependencies. So like every time you start up an EC2 instance or VMware image or  Brackenstack, a system that's going to run hab on bare metal, you know, each of those  would be their own isolated system and they would have like whatever service that they're  running would, it would get its dependencies for that. Now, if you had multiple ports running,  I mean, they're on different systems, they're on different IPs, right? So if you had multiple  packages running on the same system, there's a way to do that with Habitat. It's called  the director and it can run like you could run Postgres and Ruby app and put engine X  in front of it and run that all on one single system and they wouldn't have any, assume  they wouldn't have any conflicts because you'd want to expose them on different ports.
Habitat_101_0_131.png Habitat_101_132_2183.png Habitat_101_2184_2267.png Habitat_101_2268_4391.png Habitat_101_4392_4955.png Habitat_101_4956_5639.png Habitat_101_5640_7655.png Habitat_101_7656_11195.png Habitat_101_11196_12323.png Habitat_101_12324_23951.png Habitat_101_23952_27347.png Habitat_101_27348_28391.png Habitat_101_28392_30599.png Habitat_101_30600_31631.png Habitat_101_31632_32999.png Habitat_101_33000_33107.png Habitat_101_33108_33299.png Habitat_101_33300_34619.png Habitat_101_34620_34739.png Habitat_101_34740_37439.png Habitat_101_37440_37643.png Habitat_101_37644_37691.png Habitat_101_37692_37751.png Habitat_101_37752_38063.png Habitat_101_38064_38231.png Habitat_101_38232_38807.png Habitat_101_38808_39035.png Habitat_101_39036_40727.png Habitat_101_40728_42155.png Habitat_101_42156_42383.png Habitat_101_42384_42443.png Habitat_101_42444_44567.png Habitat_101_44568_44951.png Habitat_101_44952_47867.png Habitat_101_47868_47927.png Habitat_101_47928_48503.png Habitat_101_48504_50315.png Habitat_101_50316_56195.png Habitat_101_56196_60383.png Habitat_101_60384_61871.png Habitat_101_61872_62891.png Habitat_101_62892_63347.png Habitat_101_63348_63479.png Habitat_101_63480_63527.png Habitat_101_63528_63635.png Habitat_101_63636_63779.png Habitat_101_63780_63935.png Habitat_101_63936_64727.png Habitat_101_64728_65159.png Habitat_101_65160_65243.png Habitat_101_65244_65675.png Habitat_101_65676_65867.png Habitat_101_65868_65927.png Habitat_101_65928_66107.png Habitat_101_66108_66503.png Habitat_101_66504_66791.png Habitat_101_66792_67307.png Habitat_101_67308_67391.png Habitat_101_67392_68147.png Habitat_101_68148_68615.png Habitat_101_68616_68759.png Habitat_101_68760_68819.png Habitat_101_68820_68867.png Habitat_101_68868_68915.png Habitat_101_68916_68987.png Habitat_101_68988_69035.png Habitat_101_69036_70163.png Habitat_101_70164_70457.png
Hi. Welcome to Habitat by Chef. I'm going to not use this remote. Habitat 101. So I'm going to tell you about Habitat today. So if you don't know who I am, I'm Joshua Timberman. I probably wrote cookbooks that you use for Chef that were rewritten by other people so that they're good. I'm a member of the Chef Board of Governance, which is part of our community guiding board. I've taught Chef and I am now on the Habitat core team. So I work full time with Habitat and making it do cool things. And I probably overheard something you said and posted on Twitter. So this is the abstract that is on the Chef Conf site. This is probably what you saw in the schedule. I'm kind of going to cover these, but really I have some goals for today that look more like this. I want you to understand why Habitat exists, if you don't already know. I want you to know what Habitat itself is and I want you to be able to get started using it. Now I don't have enough time to go into an in-depth tutorial about the various pieces and parts. Habitat is actually a really featureful and big software program. So there's a lot of things to know about it. It's application automation and of course for running any kind of applications we have to have infrastructure. So why do we run infrastructure? Right? Is it because it's fun? I mean I think it's fun. I'm a systems administrator. So I have a lot of fun with running infrastructure and being frustrated with computers because then you have that challenge that you get to solve. But do we run infrastructure because the business requires it? We don't, of course. We need to run applications, right? Well we don't actually need to run applications. We want to ship our ideas, right? We want to get those ideas out there. Of course those ideas are usually wrapped up in some kind of application that provides customer value in some way. So why do we have these applications? Is it because the business requires it? Software is eating the world, right? Everybody has applications. They're the foundation of any nutritious breakfast, you know, like breakfast tacos. My massage therapist has a website and application that she has to run so that people can schedule massages with her. You know, Etsy and Facebook and Google and your company all have applications. So that's great. We have applications. But how do we run them? Everyone runs their applications a little different. Some people run their applications using Java as the language that it's written in and then run it in the JVM. Some people write.NET applications and do it that way on Windows. And we can argue all day about which one is better, about which way to paint the, you know, which color to paint the bike shed of how we run our applications. At the end of the day, we end up with industry best practices that look like this. Glad Hadoop is in there. But these days, it's all about containers, right? And Adam talked about the, about using Docker and having that container, you know, that production difficulty and it gets more difficult as you go closer to production and then you have learning cliff and then you're like, okay, I've mastered Docker and then all of a sudden you're like ready to go to prod and you're like, oh, yeah, we have all these other things we have to do. And, you know, containers are our best practice. But, you know, not everybody's running containers. Some of us run our servers on bare metal. You can actually buy this t-shirt. The artist, the link is up there and I'll make the slides available later. But I'm definitely going to get that t-shirt because that's amazing. But at the end of the day, most of the time, what we have when it comes to running our applications and running our infrastructure is the configuration management trifecta, right? You have a package, you have a config file or 17 config files and then you have a service, right? Your application runs as a demonized service listening on port 80 or port 8000 or whatever it is and then people can use it. And we solve this problem with Chef by installing the package for your application and then having a separate config template that lives in a cookbook instead of your application repository. And then, you know, you manage the service of the application which is going to be system D or run it or, you know, all that. But like, Chef abstracts that away for you and, you know, we totally solve that problem, right? I mean, obviously not because I'm totally leaving you on. We have operating system considerations, all right? This is another area of bike shedding and, you know, arguing on the Internet. Do we use Red Hat? Do we use Debian? Are we just going to drop off a jar file because our application is Java? Are we just going to use TAR-GZs because we're on Gentoon or Slackware or just because we think that's the best way to package things? We need to run our service. Is it going to be system D or are we going to use upstart? It depends on which distribution we're using, right? Like, the operating, we've tied our infrastructure to these operating systems that have all these preconceived notions about how they think that people are going to want to run their applications. But they don't know your application. You know your application and you're like, putting it in, run it is ridiculous. Why would you do that? Or, I don't like system D because they have taken over the entire operating system. I don't want to do that either. And aside from that, where do you put your config files? Do you put them in slash Etsy? Do you put them in the applications directory? Are those config files dot comf? Like standard Unix config files that are not standard at all? Are they YAML where white space is pretty cool? Or a JSON where you have no comments? So, you know, we have all these different standards because of the way that we all run our applications, right? Because everybody's infrastructures all look the same, no? But the wonderful thing about standards is that none of them matter. Like I said, what the distributions are doing, the way that they think that you're going to run your app, they have no idea because they don't know your app. And the applications themselves don't care. They just need to be able to run somewhere. And what happens is at the end of the day, most organizations write these various layers between the application and the operating system. You've got things like Chef and Console and EtsyD and you have glue scripts that have been hobbled together over the various years. And then we also have language runtimes that ignore what the operating system is doing entirely, right? You have, you know, the languages are going to install in user local or they're going to install in varopt or some weird location that you never know what's going to happen with them because you don't necessarily know what their assumptions were about it. And it doesn't, you know, you just have to make your application work with it. So, the way that we solve this, of course, is with Habitat. And what Habitat is, it's application automation that enables modern application teams to build, deploy, and run any application in any environment. From traditional data centers to containerized microservices. That sounds like an awful lot of buzzwords. So, let's break that down a little bit and, you know, put it into terms that are maybe a little more meaningful. We're talking about the applications. So, we treat the applications as the first class citizen. So, the application comes first, its concerns are what we want to worry about. We don't want to worry about the distribution and how it's packaged things and all that. So, put the application first. Then we defer the decisions to runtime. Habitat itself exposes an interface that allows you to change the behavior and change the configuration of your applications at the runtime and handle reconfiguration dynamically. So, what that means is that you can set environment variables at runtime when you start the application to affect its behavior, rather than having to lay out an entire file system full of config files before you can actually start it. Habitat is automation that travels with the application. You package everything that your application needs to run with Habitat. You package up the application code. You package up any of the runtime dependencies. If you're using like a Rails app, you would bundle your dependencies into the package and you have all your configuration, lives in the applications repository, your startup scripts, all that stays with the application in a package. And the way we do these packages, we build them in a clean room called a studio. So, the studio is just a charooted environment that doesn't have anything in it. So, there's nothing from a distribution that's going to interfere with the way that you build your application. So, if you have a Java application, it's completely isolated away from any distribution-specific versions of the underlying libraries. So, you need to have all of your dependencies in there and you package those as Habitat as well. So, we'll get to that here pretty soon. So, when we enter a studio, we use the Hab command and has a studio sub-command and it gives us some output. It says it's creating a studio directory and then it implore imports the delivery secret origin key and that is because that's called delivery. It could be called core, it could be called Acme Corp, it could be called J Timberman. Well, you couldn't use J Timberman because that's my key, so, you know, you make your own. And then it imports the origin secret key and then it enters the studio. And now I'm in a 64-bit Linux environment. That's what Habitat runs on currently. And in here, I can perform the build of my package. And I can exit the studio and then upload the package and do things with it from there. And I can go back into the studio and have a repeatable build. So, if I need to release a new version of the application, I go into the studio, I do a new build of the application and it gives me the application artifact again for the new version. So Habitat itself is a packaging system. You build a packaged app for your application as an artifact and then you install that wherever it needs to run. So let's look at the anatomy of a Habitat package. Every Habitat package has a fully qualified package identifier that consists of the origin, in this case the origin is core, then the package name, in this case, this is the Hab package. It could be Apache, it could be MySQL, it could be Redis, whatever your cool application is named. Then you have a version. This version is going to be whatever the version of your app is. If I'm using Apache 2.2, then this would be 2.2. If I'm packaging up a dependency like Ruby and I want Ruby 2.3.0, that would be 2.3.0. And then the last part is the release. So every time you build an artifact with Habitat, the package gives a specific time stamp release so that you can always use the latest one or you can use a specific release of a package. Then the Habitat artifact itself uses the same kind of naming. This is the artifact for the HabSoup package. So it has the origin, the package name, the version, the release. And then also in this file it has the target which is the architecture and the operating system that that package is built for. And then we use a heart extension because we love you. So inside of a Habitat package, we extract those out to slash Hab, slash packages, slash the origin name. So in this case it's core, but it could be Acme or Microsoft or Chef or whoever packaged that thing, it would be that origin. And then the name of the package again, the version, the release. So that's a directory and it has a bunch of files in it. And the files in here that are in all caps are the metadata files that Habitat created when the package was built. There's these, there's some others. I'm going to talk about some of them. So the other thing is that in this package I have a bin directory with a Hab binary. Hab is a statically compiled binary. So it doesn't have any runtime libraries or anything else in there. But your application might have a lib directory or an include directory with headers. Or if you're using dependencies like libpng or libxml, those packages have those directories because that's part of that software. But if you have like a Rails application that has a config directory and your lib directory and a vendor directory where you put your bundle, like all that stuff would be in this directory here. Some of the metadata files are related to the dependencies of the package. You have a build depth, which is the dependencies for building this package. So if I'm in a studio and I'm going to build the package, those are the ones that will be installed on the system in that true to the environment so that the package can actually get built. Sometimes you need things like make and gcc for build time, but you don't need them for run time, right? So the run time dependencies are in the depths file. And that is just the things that that package needs to run. Now, this is an excerpt from the build depths for the hab package. Like I said, it's statically compiled and we use the musil c library so that it's smaller than using glibc. But these are all build dependencies. The hab package doesn't have any depths. There isn't a depths file because it has no run time dependencies. Now, if I had a package that had run time dependency, like a Java app would depend on a JDK, so I'd have a JDK package and it would be listed in here with core slash open JDK slash 1.8 dot whatever slash and then a release. So all of these dependencies are all the specific release of each one of those packages. So this way you know that when you install this package, it has these dependencies listed with these specific releases. So you can always get the same thing every time when you install it. When we think about the principles of immutable infrastructures where you've got, you know, you don't want things to change, you can verify that things haven't changed because we know that all these are the same dependencies. We also have a t-debs file and a build t-debs file. That's for transit dependencies. So those are things that a dependency that I have depends on, right? So I might need open SSL and open SSL needs Zlib, but I don't need Zlib directly. I know I'm just going to get it with open SSL. We also have a file called files and this is a list of all of the files in the package when the package was built. And we have the Blake2B checksum for, it's a hash of the content for every file. Now, the slide isn't wide enough because those checksums are really long, but just imagine that those are all one line each. And then this files file is signed so that you know that it hasn't been tampered with. So you can check this file on disk and verify that everything in here is exactly what you needed. For certain organizations that have compliance requirements or security requirements that say that they need to, you know, you need to verify everything, that is easy to do with this. Then we have an ident file and this is the fully qualified identifier for the package. And then one of the other metadata files that is important is the manifest. The manifest is a read, is like a markdown file that we can put on the Habitat web UI and display it nicely. So you can see manifest down there kind of just below the middle. That manifest is the markdown file that would be the same thing as if I looked at it on disk, but it just being markdown. Every package that you have that has binaries that you want to be able to run, well, you define where the binaries directory is going to be when the package is built. Usually it's bin and it will automatically create a path file that when Habitat runs it can load those for the application. All the dependencies paths files are added to the path when a user logs in. So you'll have access to all those locations because all the Habitat packages are all installed in slash hab. So we don't care if you have slash user bin bash on your system because we have a bash package, so if you need bash for Habitat you use Habitat's bash and then you know that it's there. And it would be added to your path. We also have an exposes file. So a lot of applications need to listen on some port somewhere, so you define what ports you want in your plan and then this exposes file gets written out with those ports. There's some other files that are useful if you're building like C software that has C flags or LD RunPath and LD flags. So after you've built your package on your studio, you have the heart file, then you upload that package or your packages to the depot. I mean the depot. Habitat also has built in service discovery that is over a gossip protocol. We use Swim for that and every system that starts up for running your application like you need a database, you need your app server, you might have a proxy load balancer in front of it, those all start up and they can discover each other through this gossip layer. And if you listen to Adam's keynote, he talked about how we wrote our own gossip layer we used to use at CD and that is based on the Swim protocol. So when we start up a service like the Core Redis package, we can see that there's some information about the gossip listener and the distributor and we also have the supervisor is running on this IP address. So when we start other services or other applications and services that need to use this, we can tell it to talk to that IP and then that will join the gossip ring. The Hab supervisor also has a configuration service. So that allows us to ask questions through a RESTful API. It runs on port 9631 and we can hit slash config and it will dump out the Habitat Toml configuration about that particular service. There's a couple of other endpoints that we can get to as well. These are all documented in the Habitat documentation. Because we work with a lot of companies that have security and compliance concerns and we want to be responsible grown-up operators, we want to have encryption and signing so that we can have verifiable packages and content that we know is not been tampered with, we use Libsodium and Blake2B for signing artifacts and hashing file contents for their integrity. And then for running the depot and the website, those both use HTTPS and we support HTTPS. If you wanted to run your own depot, then you'd be able to do that with HTTPS in your own wildcard cert or whatever. The signed packages, one of the cool things about the heart artifact files, the Habitat artifact, is that you can just use head to read the header that has the signing information. So in this section, I'm going to cover a number of Habitat basics. I'm going to go to my terminal here in a moment and I'm going to run some commands and I'm not going to go super in depth because of time constraints but I want to give everybody a feel for what it's like to, what the various parts of Habitat are and what it's like to work with it. The website, Habitat.sh, has tutorials and a demo that you can work through on your own and then there's a download link, you can download the binary and start using it right away. So I'm going to go to my terminal. So Habitat is a get-reported repository on GitHub and I have it here. I'm going to go back over here. So for example purposes, we have a Ruby on Rails application. Now I know not everybody runs Ruby on Rails and that's fine. Whatever your application you run, you'd go to whatever its repository is and then you would, if you're going to start using Habitat, you'd create a Habitat directory because that's the default location that Habitat's going to look for when it's working with your packet or when it's working with your application to package it and then we have a plan.sh. So the plan.sh is a bash shell script. It's all Apache open source, Apache license open sourced and when we write a plan, it's a shell script that is going to be loaded up by a build script that just sources it. So the build script sources my plan.sh and it's going to pick up a bunch of variables and some callback functions to tell it how it's supposed to build this package. So all these package underscore variables are all the things that inform the build script about various parts of it so that it can build this application. So the package name is Ruby Rails sample. So remember we saw slash core slash hab. We would see core slash Ruby dash Rails dash sample. We have the package version is 0.0.1. I could change that and make it a 0.0.2. I can make it a 0.2.2. Whatever my versioning scheme for my application is going to be, I can change that. I was working with this in Chef delivery which is now the workflow part of Chef Automate. So I have the package origin as delivery. The maintainer is the Habitat maintainers. This application itself is MIT licensed. I forked it from a sample that Heroku uses. I download the source from GitHub and I get the package Shasam. I know that that file is going to have that Shasam and I get downloads. I define all my dependencies as a bash array. So it needs bundler and CA search and it also needs, I don't know why you need node to run a Rails app but like I'm not going to judge, it's cool. This application uses Postgres so it needs Postgres in order to have the Postgres client libraries and of course it needs Ruby. The build depths, it needs core utils, GCC and make. Those won't be installed on the resulting container or resulting, sorry, they won't be in the dependencies when I install it on a container or a VM or something. And we have the lib directories, the include directories and we expose port 3000. Now these callbacks or these functions are callbacks that are, they override functions that are defined in the build script. So the build script has a prepare function that does a thing. It has a build, a do build function that does something depending on what the software is. But I'm going to override those and make it do what I need for my application in order to be built. So I'm setting some local variables. I export some environment variables. I handle Nukogiri, note that we're using the system libraries. This would use the hab system libraries. Has anybody compiled Nukogiri for a Rails app? Hands up? Yeah. So we solved this problem for you. You're welcome. This sets the bundle config. Like this is all the stuff that I need in order to have this Rails app be something that I can deploy. We bundle install into the vendor bundle directory. And so when I run this on my system, when I go into a studio and I do a build, it's going to load this plan and get to work. The do install is similar to like if you have a, if you know like GNU C times, you do like configure make make install. That would be the do build. And then the do install would be make install, right? So that's, you know, those are, those functions are broken up like that. This is how we're doing the install for this package. So this may not finish before, before I'm out of time. So I just wanted to talk about some of the output here real quick. So remember I said we enter a studio to do our build. And I'm entering the studio from the Ruby Rails sample application repository. Then when I go in there, like we saw on that slide, I run build. And this is build as a script that loads the SRC Habitat plan.sh. It looks in Habitat for the plan.sh file. You can also specify that if you want. And now it is loaded. So before we have nothing here, before this colon, but now we have Ruby Rails sample because it's loaded that package name and it's going to continue doing the rest of the build steps. It automatically went and installed core utils and GCC and make and their dependencies, right? So we've got, we've got some libmpc and so on. So it installed all the things that it needs to build the software and it sets some, some variables and here's where it's running bundler. So we're building the package. It's taking a little while because bundle installed can take some time. So the other things that you work with in Habitat, when you're going to run your application, you need to run it in some way, right? So we have a, the notion of hooks and you have a, sorry, we have the run script which does whatever you need to do to initialize and run your application. So in this case, we need to set some environment variables so that the application knows where the gem home is or the gem path is, what the, what the path where commands are going to be located are, what the Rails environment is and then at the end here, we actually start the application and the run script should run something that runs in the foreground because it's going to supervise it and handle it. So you don't, you don't want something to background because then it'll think, you know, it'll, it'll have lost what, what, what it thought it was supervising. So at the end here, I love it when a plan.sh comes together. At the end here, we have a, where's the line? We have a tar ball that was created and then it gets compressed with XZ. So we have a tar dot XZ and then we sign it and create the heart file. So this is the artifact that gets created. So when Habitat runs, it creates this HabCache artifacts directory and it then writes in the artifacts that it builds out to the heart files there. We also create a slash src slash results because this is a clean room environment. It's a, it's actually, this is actually running in a Docker container so that it's completely isolated and also because I can't run Linux directly on OSX, right? Because, you know, you have to have a VM of some sort running. So I just use the native Docker for OSX. So if I exit this, then try and go back in the studio, all the stuff in slash Hab is gone. But we have a results directory because the Ruby Rails sample, the application directory that I was in, when I did the Hab Studio Enter, it bind mounts that directory as slash src. So that's how I have all the, all the Rails sample stuff. And then the results directory has my artifact as well. So when I exit, I still have access to it. We also have a last build.m which is just a key values of information about the build that you can use inside Jenkins or Chef Automate or whatever your CI, CD system might be. You can, you can use this elsewhere. Like if you needed to publish a list of package check sums on a website for your downloads, and you can get that out of this file. All right. So getting low on time, so I'm going to continue with the rest of the presentation as soon as I figure out where my turn off mirroring is. Okay. So let's talk a little bit about Habitat and Chef workflows. So we love Chef recipes. And in order to use Habitat and Chef recipes, right now, you want, you, like the main thing would be you'd want to get Habitat installed on the systems where you're going to run your applications that are packaged with Habitat. So we can use a remote file resource to download the core Hab package, and then we can extract it using tail and xzcat and tar. Now this, of course, isn't convergent. It's item potent because it will run that every time, and it won't change anything because it's going to get the same version each time unless you change those attributes. But it shouldn't, like, run that every time, so you'd want to have, like, a guard in there. But for this, it's fine. This is just the basic Chef recipe to get Habitat installed and then install the package with Hab install. We do some of this in our build cookbook. So we have a build cookbook for using Habitat inside a Chef workflow if you're using Chef Automate. Now that build cookbook has a detailed readme, so if you're using Chef Automate and you have a workflow, which is called delivery two days ago, you would use that build cookbook in your own build cookbook. You can depend on it and make sure you get it by updating the Birx file and the metadata in your build cookbook. We also have a Ruby client library for Habitat, and this is used in the build cookbook for a few things, like being able to get information about a package from the repository. So just a little bit of sample code. We can create a new instance of the Habitat client, and we can also point it at a different depot. So if we set up our own depot on local host or on some internal system instead of using the public one, we can do that. Then we can show packages, and we also have the notion in the depot to promote packages, which puts them into a different view, so you could have like a production, a testing, a staging. With delivery or with Chef workflows, you'd have views that you'd promote through the various union rehearsal and delivered phases inside of that Chef workflow. It's available as a Ruby gem from rubygems.org, so you just use the new cookbook metadata feature where you can say a gem to install and it will handle it, or you can just put Chef gem Habitat client in your recipe. We don't yet have Habitat package and Habitat service resources, but we'll have those soon so that we can have those that have install, not be an execute block. That'd be pretty cool. And that's about it for the Chef workflow section. So I have a few, as we wrap up, I have a few frequently anticipated questions, so hopefully get ahead of some of the questions since we won't have a whole lot of time for that today, but the first one, does Habitat replace Chef? No, because you still have infrastructure you need to manage. You still have users that need permissions and you need to set up your monitoring and logging and all that stuff for the operating system and for, you know, the infrastructure that your application is run on. Habitat supplements Chef by giving you a first class application-centric way to package and manage your applications. And then you can use that within Chef. How does Habitat work with containers? Because Docker, Docker, Docker, Docker, Docker, mushroom, mushroom. Docker is a big buzzword lately. It's really easy to use and it's delightful for people and they want to be able to have containers run their applications as microservices, right? Like, we've got to do that. That's what all the Internet says. Habitat works with containers in that, in a couple of ways. We do all of our development of our packages currently in Docker containers because we know that we can use a minimal Docker container that has nothing else in it. We can use like an Alpine Linux and build our packages on that. And those packages, by the way, will work on CentOS or Ubuntu because they use all of Habitat's dependencies, not whatever the underlying OS' dependencies are. So we do clean room builds inside a Docker container. So that's one way we use it. And the other way is that you can, after you've built the package, so I did that build, I could then run Hab package export Docker core slash Ruby slash Ruby Rails sample and then I would get a Docker container out of it. Right now we can export packages as Docker containers but in the future we probably want to be able to do CoreOS' Rocket, EC2, AMIs and that sort of thing. Those are like future things that we might be building at some point. Next question is, does Habitat work on Windows? No? Sorry. But the good news is that we're going to work on porting Habitat to Windows. And we want the Windows user experience to be as easy and delightful and as intuitive as Windows admins would expect that to be. Because you... We love our Windows administrative brethren and sisters and, you know, they don't necessarily have an easy time with the tools that they're given to do automation. And Microsoft is really coming around on this and we're working with them to make this a really, truly delightful experience. So it's... I don't know what the, like the roadmap terms of that are but I know that it's something that we're going to work on very soon and having Habitat work on Windows is a high priority for us. Yeah. Like all kinds of good stuff will be coming out of that. Does Habitat work with the OS or distro packages? Nope. It doesn't use them. It doesn't care about them. It doesn't matter if you're using Ubuntu or CentOS because you're doing all of your builds in this clean environment that don't have any of that. So it doesn't need any of that. So I can take that Ruby Rails sample heart file that I just built on an OSX system that was running a Linux container and I can take that package and install it on Ubuntu 16.04 or on Red Hat 7 or on Gen 2 and as long as I have a 64-bit kernel greater than 2.6.32, it will work because it's all using Habitat and Habitat's binaries and the way that it was built. So it gives us a lot of freedom to be able to move our applications to any kind of any other Linux platform that we want. Maybe the question, would it work on FreeBSD? Probably with the Linux compatibility layer of FreeBSD. Probably. I don't know I haven't tested it. We also have a whole lot of core packages. So we have most all the dependencies you would need for common application runtime frameworks like Rails and Erlang and Python. We're working on PHP. We're working on Java. We have all these are in the core repository that the Habitat team maintains. The astute may wonder, didn't you just create a new distribution? I mean, sort of. But what we did is we created a way for people to be able to build and manage their applications no matter what their distribution is. Is it better than Hadoop? Of course it's better than Hadoop. All right. We have the Habitat zone on the opposite side of the building around the corner. Like, I don't know. I don't even know where it is from here. But it's over to Room 404. So if you can't find it, just look for the Habitat banners and stuff. And unfortunately, you just missed Fletcher Nickel talking about Habitat Internals. Sorry. And we'll also have Nathan Harvey talking about building the Habitat community over there. I think I just left. Then why can't you say from scratch by example? So not a Docker question, but what's still needed that you actually needed a base OS in the container? Okay. Besides the base OS in the container, you need the Hab binary. And when we use Hab Studio Enter, it automatically installs what it needs to do that. So one of the things that we learned while we were building this is what Adam was talking about with having humane software. And in order to have humane software, you need to have software that behaves pretty much like you'd expect. And so there's a little bit of magic, but there's a whole lot of delight. We've worked really hard to make it easy to use. And it actually, once you start using it, it ends up being more simple than you thought it was going to be. You'd be like, okay, so how do I get all the things I'm going to need to install this? It's like, it gets all that for you. It handles it automatically. So there's a question I have about upgradability of the packages afterwards. Right? We quite often end up deploying a package out. And then we have no control necessarily of when a customer is going to upgrade a package. How does it handle that? Or does it handle that? Or is that just like left up to us to figure it out? So yes, it can really automatically upgrade the packages. It can detect from the people that the package has been able to do that. We're working on fleshing out what all those update strategies might be. But the main things are that we have the capability to automatically upgrade. And we have the capability to not automatically upgrade. And doing like rolling kinds of things is still something that's kind of in the air about that. For a complex, for example, a Python dependency chain, for example, we would need to pin many, many upstream Python pit modules. Is that possible in the plan? Can we set different sources? Do they all need to come from the, I guess, the depot? So I don't know a lot about Python. So one of the things we do with Ruby, you have a gem file. And in that gem file, you define your dependencies. I imagine Python has a similar thing. And then when you do a bundle, you get a gem file.lock. And then you can commit that lock file to your version control system so that you can always get those specific dependencies. And imagine Python has a similar thing. Yeah. So you can always get the same thing. You would always get those dependencies from doing the build. You'd have the build step that would install those for you inside of your app. Now, if you wanted, you could have, you could package up all those various Python packages and make those available. But what I would probably recommend is to go the route where you vendor that all into your application so that it's all in lockstep. And that way we don't have, like, version sprawl of all these various Python packages out in the depot. Josh, you said you just said to package up all of the dependencies, be it bundler or pip or what have you. So are habitat containers or are they composable? Can I start at some sort of base layer with a set of things that I typically would use? Or will I have to come from the clean room as it were every time? I would recommend going from the clean room every time. Habitat doesn't do like a layering kind of thing like Docker does. It doesn't have that as a thing. Like when I was going through the deps files, the build depths, the depth, the depth, those are all the versions of all the packages that it depends on. So it wouldn't be built up over like, you wouldn't have like multiple versions of OpenSSL as a dependency. You'd depend on whatever the latest one was or if you needed a specific release, something like that. Does that answer your question? Yes. Okay. And then I saw in the plan file that you exposed ports for the app. Would multiple hab installs need or, I'm sorry, would they conflict in any possible way other than those ports if you were to, like we were talking about attempt to compose different hab installs on the same machine? Can you repeat it, Josh? No, if you didn't hear the question, you exposed ports, would you have possible conflicts of those ports between multiple different packages? Yeah, I mean, you could. No. Because we isolate all the dependencies. So like every time you start up an EC2 instance or VMware image or Brackenstack, a system that's going to run hab on bare metal, you know, each of those would be their own isolated system and they would have like whatever service that they're running would, it would get its dependencies for that. Now, if you had multiple ports running, I mean, they're on different systems, they're on different IPs, right? So if you had multiple packages running on the same system, there's a way to do that with Habitat. It's called the director and it can run like you could run Postgres and Ruby app and put engine X in front of it and run that all on one single system and they wouldn't have any, assume they wouldn't have any conflicts because you'd want to expose them on different ports.
automation that enables modern application teams to build, deploy, and run any application in any an artifact with Habitat, the package gives a to run. Now, this is an excerpt from the build Habitat inside a Chef workflow if you're using way to package and manage your applications. And