Raindrops: CentOS in the clouds
Singh, Karanbir
The present status of Cloud and Virtualised Machine instances is very chaotic; different Vendors doing their own thing, every hypervisor needing its own magic and various cloud instances imposing their own site requirements. The RainDrops project aims to reduce the impact this has on users, giving them a unified interface ( web, git and REST ) that they can define their requirements against. And let the system handle the various requirements. In this presentation we aim to define the scope of the problem and demonstrate our solution. Our demo will consist of defining a single kickstart file, and then using that to build multiple instances for different vendors and technologies, with very little effort and no vendor specific configs needed. The raindrops project has been in development and closed testing upto this point, we hope to use the Fosdem talk to launch it publicly.
 Some of these slides have been recycled from talks and some of you may have heard the talk last year except the project has been evolving. So hopefully when you go through the whole thing what you see here now, a lot of maybes and a lot of coming soon have actually been replaced by functionality that exists right now. So just as a show of hands, how many people here use the cloud or a cloud or the cloud? Okay, so most people, excellent. So we don't need to go into the basic stuff. The raindrops interface is something that, okay, a little bit about me. I'm a sad man. I'm based out of London. I do sent our stuff. I like bullet points and I like the rain. And Belgian bears, yes, Belgian bears are good. Although I've lived in the UK now for 12 years, I still prefer Belgian bears. I'm probably going to get assassinated on the way back or something. I'm saying that. So the thing is just looking at the problem definition, what we were trying to solve. Sentos is everywhere in terms of the cloud, right? It's a lot of people, a lot of people who are selling services on the cloud. You sent us a lot of people who are consuming the cloud in various ways. You sent us a lot of vendors, a lot of hosting companies, retail and the cell sent us images and instances. But the sent us distribution itself is built for traditional use cases. It wasn't built with the cloud in mind. I mean, the way the installer works is meant to be sort of automatable in a conventional traditional kind of an infrastructure environment. The way the management tools work is, again, it's kind of built for traditional enterprise users or traditional kind of setups. And that kind of caused a bit of a problem. Because even a delivery for what we actually push out is we deliver ISOs on an HTTP interface that people can download, which are not usable directly. But what they deliver to you as a user is finite, is fixed. When you install sent us, the kernel you get is predetermined. You know exactly what kernel you're going to get. The version of bash you're going to get, the version of PHP or HTTP or Apache or whatever you're going to get is predetermined. But the mechanism that we shift with and the mechanism that we should deliver with is tuned for traditional architectures as it were. And traditional media, we do ISOs or whatever. But how different are the cloud guys? They're not really very different in that you still have to do a provision. You still have to build an image. You still need to have something that you can run. You still need to manage that instance. You still need to patch manage within that instance for security issues, for bug fixes. So from that perspective, they're not very different from traditional use cases. But they're different enough to merit specific consideration. And to us, the biggest deal was that we need to be able to talk to them about delivery mechanisms and how they consume what we ship and how they deliver to users, what they further on ship to consume. So the issue was that we found a lot of people were coming back to us and saying, hey, I bought a VPS from hosting company A and they gave me this, this, this in it. Then I wanted to move to hosting company B and I bought a CentOS 6 VPS and that had something completely different. What is that all about? And then I wanted to move to AWS and that was a completely different thing. And then I wanted to set up a developer VM on my laptop and that looked like it was completely different. And basically what was happening was that every vendor takes what we push out and salts it to taste. And when I mean taste, that usually depends on what distribution, what best practices, what issues, what history, what has been the life-term experience of the guy who's actually creating the images. In many times, it doesn't reflect company policy. In many times, it doesn't reflect what they're actually expecting to shift. It reflects what the guy building the image thought was what his users want. And that's a dangerous situation to be in. So we started talking to vendors. We started talking to people who would come back and complain to us. And we found that there were issues like people were shipping 5.3 in 2011. People were shipping 6.0 in 2012. And we were like, do you know that those images have security issues? Do you know that you should not be doing this kind of thing? And a lot of them were, you know, we put in the effort. We went through the pain of setting up a VM. It took us like three weeks to get it going. I don't have the resources to do it again. And that was pretty much uniform from all hosting companies from HPC places across the board. So we thought, okay, this is a problem that we can solve. So we spoke to some of the users that the vendors gave us access to. So the vendor said, hey, I've got 100 users. Here's 10 you can talk to and see what the actual problems are. And we did that. We spoke to people who consume liquid web services. We spoke to people who host with Pair One. We spoke to the big AWS users. We spoke to some people Vee knew who are running private clouds as well to see what we could actually do. And the problems like we were talking about delayed access. People want, vendors don't have resources to make that happen. So basically there was a massive gap between what the vendors were doing and what the users really wanted. So the problem we can really define down to users want persistence for images. They want something that is reliable that looks like CentOS irrespective of where they are going to. What vendors they're working with. What sort of environment they're on. If they're on a local box or if they're not a remote box. But they still want to retain third party support. So it's like if you want a VM from a hosting company, let's say Pair One, you want to be able to get Pair One to still support your image for you. You still want to have that phone number that you can call. Because if Pair One does something like here, here's the official CentOS image, all the best. Let me know if you have a problem and just make sure your credit card is valid and we'll keep charging you. You don't want to be in a situation where you can't get support anymore. Which means we need a consistent platform from the vendors to CentOS and from CentOS why are those vendors across to the users as well. Does that kind of make sense? Yeah? Okay. So we found that on cloud and off cloud are not, they don't really look the same. You can still have VMs. But you could have a VM on your machine, you could have virtualized infrastructure or you could have a cloud. To me, those three things look very different. Virtualized infrastructure is classic sort of VMware kind of perspective. Cloud is classic AWS kind of a thing where HA may not be that big a deal. You HA your app rather than HA in your instance. And all of them are kind of, they look just different enough that they merit sort of consideration specifically. And of course, kernel and app support, somebody's going to run Oracle, somebody's going to run DB2 or whatever, somebody's going to run MySQL, somebody's going to run SQLite. And then people like me still think that if you only have a thousand or two objects, then flat files are fine. It's true. It's true. I mean, you know, it's the, it's the, it's the want to pain that people go through to set up MySQL with, you know, all sorts of schemas and everything and then they run a blog which has five posts, right? Have you tried running flat files with five lines in it these days? Do you know what kind of performance you can get there? So basically what we did was we got all of that stuff together and we came up with an instance called raindrops.centOS.org. We're actually launching this as project raindrops.net, but it's also available from, from the raindrops.centOS.org interface. What it is is it's a web, email and a Git interface. You create a kickstart file and this looks like a conventional kickstart file. You've got install, because that's what you want to do. Things that are different. I'll point this out here. The URL that you're installing from has to have rep a host as the place is going to pick up content from because the builds happen privately. They're not available on the internet. So you have to have that and you have to have whatever the centOS mirror URL is. It can't be, you can't use variables. You can't use base arch or you can't use release version. It has to be fixed to six and the architecture that you want or five in the architecture. Everything else is about what you'd have in a normal kickstart file. You've got root password which should, you should probably keep fairly secure. You have a network device. Okay. So the other thing which is probably worth mentioning over here is that you have to have partitioning information and you can set it up whichever way you want. And you have to have a kickstart which completes without user interaction. So if you remove, for example, the network config line or you remove the root password file, it will get stuck. Anaconda cannot handle that. It's going to ask for input. So you have to have a kickstart file that's complete. Is that, is that kind of clear with everybody? Is there anything? Okay. Now what you also need to do is you need to create a config file that tells you what, that tells the rain drops what to do with that kickstart file. Right? So in this case we've got an image target which is the architecture that we want to build on. Now this sounds a bit silly because if you look back, your URL has that in it. But what happens is that we actually deploy host environment that we're going to build the image into and you can control that. So you can say, I want, and this is me being completely fictitious here. You can say, hey, I want a MIPS build on an x8664 sherooted environment which you can achieve with rain drops. We don't have a MIPS build, but if you did, it'll be the best MIPS build in the world. You have to have a UID which is actually injected in if you don't put it in. So we recommend you don't use a UID here. And then things, life is a lot easier because you won't have to remember three sets of UIDs. Just get one. You can have a PostScript which is external to the kickstart. If you want something to run, if you want kickstart to deploy an image, instantiate the image and then run a script, you can inject that here. That'll get run in. Schedule is something which is coming soon. It's not there yet. But what you can say is that I want to schedule and give it 30 days. And what will happen is this particular config file will get reprocessed every 30 days. So you get a new image every 30 days. This is not functional yet because we want to see, we want to do a little bit of testing around how we can handle load and how we can handle scale and then how we can actually get a scheduler which kind of does the right thing. The updates, yes, again, is something like if you're doing an install, we just need to tell what OS we're installing. Having an updates is equal to yes will mean it'll actually run a yum updates post install. It's just you can have that in your kickstart if you want or you can have it in here as well. This is handy when we do installations for certain hypervisors which don't allow us to have multiple repositories in the kickstart. The kickstart file. The kickstart file has to match the name that you're going to use here. So whatever kickstart you've called it, you can call it that. You have an image tag if you want to add something in. You need to use what hypervisor target you want and who you want to notify. So this could be an email address. It could also be a URL if you want to call back. So if you're automating your scripts and you're automating your kickstarts, you can have a URL over here. And when the image is done, it'll actually do a call back and it'll give you some details. Okay, nobody can read that without my email address. If you were using the git interface, this is what you would do. You would do git add my image.ks. You would git add your config file. You'd git commit with build colon space, whatever you want, and you do a git push. And that's it. And that would trigger the build. It would process the config file, get the kickstart into place. It'll trigger the image and do whatever you're told it to do. This is literally all you need to do to get your image out. I'd asked for an email notification. So this is kind of like the notification you'll get. Your job is complete. You get the download and the log. Now, we have finite storage because we've had to kind of get the hat around and pass it around to see who would actually donate the machines and infrastructure. I'm very grateful to peer one for forking out six machines. And the storage boxes have about two and a half terabytes of space. But this is going public. Two and a half terabytes does not last very long. So if you guys do start using it and you do ask for email notification, we highly recommend you get that image within a few hours or maximum 24 hours. Because what's going to start happening is when we hit the 80% high watermark, we start deleting the oldest images. Right? Now, we don't know if that's going to scale or not. We're hoping it does. We think we should be able to retain images for two to three days. But if this becomes really popular, it goes viral and hundreds of people start hating it, we may not be able to retain images for very long. So keep that in mind. How are we doing for time? Okay. We have 15 minutes in. We have literally all the rest of it. You get your image, download it, and you can instantiate it on whatever you want. But not quite because vendors being vendors still want to retain their ability of doing a little bit of salting, you know, in whatever way to make it make it work for what you want to do. So we extend that a little bit. This is the base config file, right? What you were looking at earlier. And it's still the same. Now, what you can do is you can add an extra stanza to your config file and you can add a bit more information in it. So you can have a vendor which at the moment is only sent to us. We're hoping we can add some other people in there as well. You can ask for a specific kind of disk type. Now, at the moment, it's sparse or raw because of the way it's set up to do a Zen build. But if you were doing a KVM build, you could have QCOW up there. You could have whatever other format you want. Essentially there's a QEMU IMG and command that gets run. So any five format that QEMU can handle, the delivery agent within Rain Drops will process it for you. And I think there's like, I don't know, correct me if I'm wrong, but there's about 10 or 15 different formats that you can use with QEMU. If you want a different format, if you can write a bash script which can take a standard in for an image and output whatever format you want, we can plumb that in. You can specify what disk size you want. Ideally keep this to less than 50 gigs, please, because provisioning 50 gigs in like, for example, a raw file takes up a lot of space and time. But there is no actual check on this. So if you go and say, give me two terabytes worth of a disk. The builder machines only have about 200 gigs. So it'll try and allocate that disk. It'll fail and that'll come up in your log file that, hey, I couldn't, the disk process didn't complete. If you want the image compressed, you can ask for a few things. You can ask for different kind of formats. And this is the coolest bit. This is the thing that took me the most amount of time to write is the delivery stanza. So you can either have this as pickup, in which case you will get an email with the URL saying, hey, this is the URL that you can download from. Or you could say aws, colon, colon, your aws authorization tag, the string, the author tag in here. And what raindrops will do is it'll actually log into your account on aws, push that image into an S3 backed, an instance backed image, and send you an email with the AMI, the AKI, and the image ID that is used. So that's pretty cool. And then you can also have an FTP delivery. If you have an FTP server at your end, you can say give it to me at ftp colon with the whole URL. Hopefully add a user ID and password into it. It'll try and use LFTP to push it. It can also do rsync. But rsync is dangerous because it requires a lot of network resources at our end, so we don't really want to use rsync. We're happier with FTP. So if you have an rsync target at your end, you say rsync colon, whatever, it'll actually rsync the image to you. And you get the email notification once the delivery is complete. So if you've asked for AWS delivery, you'll get the email once the image is in AWS. And then you can add extra stuff. Like you want no IPv6. You want a specific mac address to be used or whatever you can add those in. By default, we only set up one network interface, but the system supports up to three. And this is something which is, I guess, more relevant to vendors who may want to have every image coming up with a public interface, a private interface, and a management interface. And this works. It handles up to three images at the moment. If you want any of those to be set up in a specific kind of way, you can specify a mac address or you can say DHCP and it won't give you, and it won't set up an HWA DDR line. How many people have used Open Nebula in the past? Right, a couple. So you guys know what I mean, right? So Open Nebula does not need contextualization to set up IPs for VMs. It converts the mac address into an IP. And the IP is the quad notification of the mac address. So over here, you can literally, what you're doing is you're specifying the IP address for the VM. Does that make sense? With the extra options? Is there any questions anybody has so far? Hi, De. So the question is, how would you include software which doesn't come with the mac address? So the question is, how would you include software which isn't distributed by CentOS? Whatever mechanism it may come from, right? The script that you can run here would be the script that you would influence those kind of things. For example, if you wanted to include the Apple repository and you wanted to install something from Apple, what does Apple provide? Let's say you wanted to run IRCD. So what you could do is you could have that in a script. And what you put in over here would be my script. And then when you do your get add, you need to have that script added in as well. The only thing is you have to have changes in all three files because the builder process only checks out the last particular change set. Does that make sense? Right. So the point at which the image gets created, that happens privately. But the image is then pushed out to a machine which does have internet access, which is when the script gets run. So the reason why we have the script up there, rather than telling people to use it in the post script for the kickstart, is that difference. At kickstart time, there is no public interface. There's no public access. But at this point, there is. Because that's how even the callbacks and stuff like that would happen. Of course, it's probably for the sake of completeness, it's worth pointing out that you can only do that with the raw file. Because if you ask for a VMDK file, we can't really loop-mount it to do these things. And because we don't have VMWare running on the builders, we can't actually create an instance out of it to run that. So there are some caveats. A lot of these things are not being tested for at the moment. So for example, you can say, hey, give me a Vbox image and deliver it to AWS. That's not going to work. Because firstly, there's no cloud in it. There's no metadata in it. Secondly, Vbox would be doing HVM, not Zen. So if you're looking for a CentOS 5 install where the Zen kernel is different from the stock kernel, it's not going to boot. And what we will do is we will very happily do everything. We will put it into S3 for you and give you an email saying done. But obviously, it's not going to work. So we're kind of hoping that as we release this stuff out for beta today tomorrow, a lot of you guys are going to go out there and break the system in as many ways as possible and then submit patches on how to fix it, hopefully. It is on Gitaurus at the moment in Components. It's also held, but the actual complete stuff is held on Naza.current.org. It's a public Git repository. So I'll give you the URL for that. And the way that things are implemented is everything is run as, again, of Beanstalk workers. And there's a concept of we use standard and standard out to cascade through drivers. So for example, I'll give you an example of how the delivery agent works. There's a message that comes through the delivery queue saying, hey, this job is ready for delivery. And then the worker will work out, how am I delivering this? I need to put it into an HTTP interface. Or do I need to deliver it to AWS? Or do I need to send it by FTP? So let's assume that there's an FTP request. It'll actually fire up the drv underscore delivery underscore ftp.sh with the UID. And then that script will do whatever it needs to do to make that delivery happen. And everything is monitored for standard and standard out. So if that script exits with a non-zero value, we treat that as an error, transfer the logover for the user to see. Which essentially means that if you've got arcane crazy stuff that you want to do, or let's say the HP Cloud guys want to have an interface that allows us to push images there, they can just write a delivery agent for it, which has all of that stuff in it. We do have some things which are available, like Python Boto is available anyway. We have the JCloud stack that is available, so you can consume that. So we've gone through this stuff. So this is stuff that is kind of functional now. We've got KVM, QVM, you stuff. LXE works, Zen works, VMware works, Vbox works with caveats. And I'll kind of come to that in a second. Open stack HP Cloud is consumable because we have Cloud in it available within an extra repository, so you can use that. That kind of works for eucalyptus as well. Cloud stack as well, I believe. You can do AWS, EBS works in a really weird kind of a way in that we inject stuff into an S3 instance. We request using your credentials a block device, and then we rsync the OS over to the block device, remove the SSH keys, remove the metadata which is instance specific and reboot the machine for you. So this has never failed for me, but a lot of people tell me there's a better way of doing it. But hey, rsync is the gift of God, right? It works. And Open Nebula works because you don't need contextualization. The mac addresses are converted in, and that kind of works. This is how it's all set up. We have a scheduler with the Git repositories, which is not really there at the moment. We're hoping most people use the web interface. We have the repositories, the builder machines, storage nodes, all hosted at pair one at no cost to us. We're not going to do that, guys. This is kind of how it works. We do PXC images from the distro to actually bootstrap a real install, except for VMware and VBox, where we use the distribution ISO and then boot it from that ISO to do the install. And a couple of things. We don't have EFI support. So if you need EFI, you're most welcome to engineer that. We don't have that support there. And CloudNet is possible. It's available over there. So big challenges. Hidling handling passwords from Kickstart files. Don't make your Git repositories public by putting them on GitHub, please. You're going to be cloning from here, but do not put them there because your Kickstart files have to have a root password in there. So even if it's encrypted with MD5 or SHA, given enough resources, people can still go backwards from it. So do not do that. Images are delivered pre-boot run. So you need to handle first boot if that's an issue for you. If you're asking for images and if your infrastructure cannot handle first boot, then you need to kind of do something in the Kickstart file so first boot does not kick in. Scripts are injected via loop mount. This is what we were talking about. If you specify a post-run script, it will be run after the VM is delivered to the head node. So you can do stuff like, you know, you can WGAT, you can Git clone. You can do things like if you want your puppet manifest to be injected into the instance before it's delivered to you. That's a good point to do it from there. And the entire log is available. Very soon we're going to have stock images for CentOS available. So if you just want a basic CentOS image, you don't really have to go through the process. You can just use public images that we'll make available through the process. And we'll update it every week, every month. It's not happening yet. We want to kind of see how scale works and we might need to get a dedicated builder for this. But this should happen soon. Also schedules are going to happen very soon where you can specify schedules that you want an updated image delivered to you every month, every week, every five hours, whatever. And that just effectively uses cron at the moment. It's not public yet. Don't try to use it. Big challenges. Will it scale? It takes between 44 minutes to 22 minutes per build for an image at the moment. If you've asked for compression, we do all compression for all images on a very small machine, because we don't really want to waste CPU cycles doing compression, which means it could take up to 30 minutes to compress the image before it's available for download. We don't know how good or bad that is. But everything else we think is going to scale quite well. It's secure, so we trust the images that are coming through. Third-party ACL handling is tricky, but we're kind of hoping people do that on their own. We're doing AWS. We might do the Rackspace image as well, the Rackspace cloud stuff. But we're hoping that Windows do their own thing for the other stuff. And big challenges. We're kind of running out of time. We've got like three minutes. So I just want to quickly do one thing. Let me quickly do a demo of what we have up there so you guys can take a look at it. And then we'll see if we can come back. Let me log out. Can everybody see that up there? So we see log onto the web interface. This is what you end up at. Raindrops.net slash login. It is available right now. Please do not send hundreds of build instances to it yet. This is what I hate doing is looking at stuff. So you can sign in with any of the other, any of the options here. We're going to try and get a few more options going. We don't want to know who you are. We don't want to have to verify who you are. We kind of trust that GitHub and friends have done this already. So I'm going to try and log in with GitHub. It should hopefully work. There we go. I'm actually logged into GitHub at the back so that you don't see me typing my password and acoustically decoded. This is what the interface looks like at the moment. You've got jobs that we've set up, none yet. We've got configuration files, none yet. Kick starts, none yet. So let's start with the new kickstart file. We have a template set up already. So you can change a few things in there if you want, whatever. It should kind of work. I need to give it a name. So let's say test one. So there's nothing specific in here in the kickstart. Because it's a template, it's actually rendered the URL bit there. Let's see what happens here. This web interface is really quick, really fast, and it's a network over here which is really slow. So if any actually go and see it like away from here, you'll find that it is actually quite a responsive thing. We keep version tracking for you with the aim that this is going to actually interface with the Git repository at some point. So you've got the kickstart in. Let's try and get a configuration file going as well. What am I doing? I'm hitting the wrong button. So you've got the kickstart file. Let's just, the config file, let's just call this for the sake of interest, the same thing. Let's set a notify by email. Most of this stuff should be, it's pretty standard. I want it picked up and I have compression turned on. I want a 10 gig size disk with a QCao 2 format and no script or anything like that. And this should then, it would help if I, did I go back? Okay. Sorry. What am I hitting? I'm hitting the wrong button here every second. Oh, come on. Which way am I going? Let me just see if I can do that. Right. So you've got a config file. You should now be able to go and create a job. And this should have a dropdown with our existing stuff and submit. So what actually happens is you've got, you can open the job again, you can delete it, or you can see the results, which is, and you should see results as things are building. So what's also interesting here is that you can actually see the status of the back end workers, which are all sleeping at the moment. Let's see if that picks up something. It's actually waiting for the next schedule, next build schedule to come up. Hopefully that'll happen in the next second or so. Let me see here if I can influence that in any way. I'd actually pulled up the worker screen for the build that is going to run. So once the prep happens, this is effectively what's running at the back, you've got the delivery agent running. So the one that I was talking about, the compression system. That's kind of like how it works. It should also be a builder instance here. So these are the builder threads that are waiting for the job to come through. Are we out of time? We are out of time. Does anybody want to talk about anything in the minute or two that it takes to get the builder going? Because the builds will only run once every five minutes for the schedule to come around. Sure. Inside? Yes. So what we're actually hoping to do is actually deliver the entire raindrop stack as a VM within raindrops itself. So what you could say is, hey, give me a raindrops instance for my cloud stack deployment. And then you have your own private set up which is already there. And the software is all free. I think we've been talking about Open Nebula. Open Nebula is going to try and ship a VM which runs raindrops from within the Open Nebula market space. We're kind of hoping more people do that as well. Does that answer your question? Is there anything else you want to talk about? We probably won't have time to see this build running. We've got another two minutes to go for the next schedule. No? Well, thank you for coming along. Highly recommend everybody goes to the interface, check it, break it, and then submit patches to fix it, please. Submit patches is what you need to focus on. Everything else will happen automatically, I think. Thank you. Thanks, guys. Thanks for being there. Bye.
 Some of these slides have been recycled from talks and some of you may have heard the talk  last year except the project has been evolving.  So hopefully when you go through the whole thing what you see here now, a lot of maybes  and a lot of coming soon have actually been replaced by functionality that exists right  now.  So just as a show of hands, how many people here use the cloud or a cloud or the cloud?  Okay, so most people, excellent.  So we don't need to go into the basic stuff.  The raindrops interface is something that, okay, a little bit about me.  I'm a sad man.  I'm based out of London.  I do sent our stuff.  I like bullet points and I like the rain.  And Belgian bears, yes, Belgian bears are good.  Although I've lived in the UK now for 12 years, I still prefer Belgian bears.  I'm probably going to get assassinated on the way back or something.  I'm saying that.  So the thing is just looking at the problem definition, what we were trying to solve.  Sentos is everywhere in terms of the cloud, right?  It's a lot of people, a lot of people who are selling services on the cloud.  You sent us a lot of people who are consuming the cloud in various ways.  You sent us a lot of vendors, a lot of hosting companies, retail and the cell sent us images  and instances.  But the sent us distribution itself is built for traditional use cases.  It wasn't built with the cloud in mind.  I mean, the way the installer works is meant to be sort of automatable in a conventional  traditional kind of an infrastructure environment.  The way the management tools work is, again, it's kind of built for traditional enterprise  users or traditional kind of setups.  And that kind of caused a bit of a problem.  Because even a delivery for what we actually push out is we deliver ISOs on an HTTP interface  that people can download, which are not usable directly.  But what they deliver to you as a user is finite, is fixed.  When you install sent us, the kernel you get is predetermined.  You know exactly what kernel you're going to get.  The version of bash you're going to get, the version of PHP or HTTP or Apache or whatever  you're going to get is predetermined.  But the mechanism that we shift with and the mechanism that we should deliver with is tuned  for traditional architectures as it were.  And traditional media, we do ISOs or whatever.  But how different are the cloud guys?  They're not really very different in that you still have to do a provision.  You still have to build an image.  You still need to have something that you can run.  You still need to manage that instance.  You still need to patch manage within that instance for security issues, for bug fixes.  So from that perspective, they're not very different from traditional use cases.  But they're different enough to merit specific consideration.  And to us, the biggest deal was that we need to be able to talk to them about delivery  mechanisms and how they consume what we ship and how they deliver to users, what they further  on ship to consume.  So the issue was that we found a lot of people were coming back to us and saying, hey, I  bought a VPS from hosting company A and they gave me this, this, this in it.  Then I wanted to move to hosting company B and I bought a CentOS 6 VPS and that had something  completely different.  What is that all about?  And then I wanted to move to AWS and that was a completely different thing.  And then I wanted to set up a developer VM on my laptop and that looked like it was completely  different.  And basically what was happening was that every vendor takes what we push out and salts it  to taste.  And when I mean taste, that usually depends on what distribution, what best practices,  what issues, what history, what has been the life-term experience of the guy who's actually  creating the images.  In many times, it doesn't reflect company policy.  In many times, it doesn't reflect what they're actually expecting to shift.  It reflects what the guy building the image thought was what his users want.  And that's a dangerous situation to be in.  So we started talking to vendors.  We started talking to people who would come back and complain to us.  And we found that there were issues like people were shipping 5.3 in 2011.  People were shipping 6.0 in 2012.  And we were like, do you know that those images have security issues?  Do you know that you should not be doing this kind of thing?  And a lot of them were, you know, we put in the effort.  We went through the pain of setting up a VM.  It took us like three weeks to get it going.  I don't have the resources to do it again.  And that was pretty much uniform from all hosting companies from HPC places across the  board.  So we thought, okay, this is a problem that we can solve.  So we spoke to some of the users that the vendors gave us access to.  So the vendor said, hey, I've got 100 users.  Here's 10 you can talk to and see what the actual problems are.  And we did that.  We spoke to people who consume liquid web services.  We spoke to people who host with Pair One.  We spoke to the big AWS users.  We spoke to some people Vee knew who are running private clouds as well to see what we could  actually do.  And the problems like we were talking about delayed access.  People want, vendors don't have resources to make that happen.  So basically there was a massive gap between what the vendors were doing and what the users  really wanted.  So the problem we can really define down to users want persistence for images.  They want something that is reliable that looks like CentOS irrespective of where they  are going to.  What vendors they're working with.  What sort of environment they're on.  If they're on a local box or if they're not a remote box.  But they still want to retain third party support.  So it's like if you want a VM from a hosting company, let's say Pair One, you want to be  able to get Pair One to still support your image for you.  You still want to have that phone number that you can call.  Because if Pair One does something like here, here's the official CentOS image, all the  best.  Let me know if you have a problem and just make sure your credit card is valid and we'll  keep charging you.  You don't want to be in a situation where you can't get support anymore.  Which means we need a consistent platform from the vendors to CentOS and from CentOS  why are those vendors across to the users as well.  Does that kind of make sense?  Yeah?  Okay.  So we found that on cloud and off cloud are not, they don't really look the same.  You can still have VMs.  But you could have a VM on your machine, you could have virtualized infrastructure or  you could have a cloud.  To me, those three things look very different.  Virtualized infrastructure is classic sort of VMware kind of perspective.  Cloud is classic AWS kind of a thing where HA may not be that big a deal.  You HA your app rather than HA in your instance.  And all of them are kind of, they look just different enough that they merit sort of  consideration specifically.  And of course, kernel and app support, somebody's going to run Oracle, somebody's going to  run DB2 or whatever, somebody's going to run MySQL, somebody's going to run SQLite.  And then people like me still think that if you only have a thousand or two objects,  then flat files are fine.  It's true.  It's true.  I mean, you know, it's the, it's the, it's the want to pain that people go through to  set up MySQL with, you know, all sorts of schemas and everything and then they run a  blog which has five posts, right?  Have you tried running flat files with five lines in it these days?  Do you know what kind of performance you can get there?  So basically what we did was we got all of that stuff together and we came up with an  instance called raindrops.centOS.org.  We're actually launching this as project raindrops.net, but it's also available from, from the  raindrops.centOS.org interface.  What it is is it's a web, email and a Git interface.  You create a kickstart file and this looks like a conventional kickstart file.  You've got install, because that's what you want to do.  Things that are different.  I'll point this out here.  The URL that you're installing from has to have rep a host as the place is going to  pick up content from because the builds happen privately.  They're not available on the internet.  So you have to have that and you have to have whatever the centOS mirror URL is.  It can't be, you can't use variables.  You can't use base arch or you can't use release version.  It has to be fixed to six and the architecture that you want or five in the architecture.  Everything else is about what you'd have in a normal kickstart file.  You've got root password which should, you should probably keep fairly secure.  You have a network device.  Okay.  So the other thing which is probably worth mentioning over here is that you have to have  partitioning information and you can set it up whichever way you want.  And you have to have a kickstart which completes without user interaction.  So if you remove, for example, the network config line or you remove the root password  file, it will get stuck.  Anaconda cannot handle that.  It's going to ask for input.  So you have to have a kickstart file that's complete.  Is that, is that kind of clear with everybody?  Is there anything?  Okay.  Now what you also need to do is you need to create a config file that tells you what,  that tells the rain drops what to do with that kickstart file.  Right?  So in this case we've got an image target which is the architecture that we want to  build on.  Now this sounds a bit silly because if you look back, your URL has that in it.  But what happens is that we actually deploy host environment that we're going to build  the image into and you can control that.  So you can say, I want, and this is me being completely fictitious here.  You can say, hey, I want a MIPS build on an x8664 sherooted environment which you can  achieve with rain drops.  We don't have a MIPS build, but if you did, it'll be the best MIPS build in the world.  You have to have a UID which is actually injected in if you don't put it in.  So we recommend you don't use a UID here.  And then things, life is a lot easier because you won't have to remember three sets of UIDs.  Just get one.  You can have a PostScript which is external to the kickstart.  If you want something to run, if you want kickstart to deploy an image, instantiate the  image and then run a script, you can inject that here.  That'll get run in.  Schedule is something which is coming soon.  It's not there yet.  But what you can say is that I want to schedule and give it 30 days.  And what will happen is this particular config file will get reprocessed every 30 days.  So you get a new image every 30 days.  This is not functional yet because we want to see, we want to do a little bit of testing  around how we can handle load and how we can handle scale and then how we can actually  get a scheduler which kind of does the right thing.  The updates, yes, again, is something like if you're doing an install, we just need to  tell what OS we're installing.  Having an updates is equal to yes will mean it'll actually run a yum updates post install.  It's just you can have that in your kickstart if you want or you can have it in here as  well.  This is handy when we do installations for certain hypervisors which don't allow us to  have multiple repositories in the kickstart.  The kickstart file.  The kickstart file has to match the name that you're going to use here.  So whatever kickstart you've called it, you can call it that.  You have an image tag if you want to add something in.  You need to use what hypervisor target you want and who you want to notify.  So this could be an email address.  It could also be a URL if you want to call back.  So if you're automating your scripts and you're automating your kickstarts, you can have a  URL over here.  And when the image is done, it'll actually do a call back and it'll give you some details.  Okay, nobody can read that without my email address.  If you were using the git interface, this is what you would do.  You would do git add my image.ks.  You would git add your config file.  You'd git commit with build colon space, whatever you want, and you do a git push.  And that's it.  And that would trigger the build.  It would process the config file, get the kickstart into place.  It'll trigger the image and do whatever you're told it to do.  This is literally all you need to do to get your image out.  I'd asked for an email notification.  So this is kind of like the notification you'll get.  Your job is complete.  You get the download and the log.  Now, we have finite storage because we've had to kind of get the hat around and pass  it around to see who would actually donate the machines and infrastructure.  I'm very grateful to peer one for forking out six machines.  And the storage boxes have about two and a half terabytes of space.  But this is going public.  Two and a half terabytes does not last very long.  So if you guys do start using it and you do ask for email notification, we highly recommend  you get that image within a few hours or maximum 24 hours.  Because what's going to start happening is when we hit the 80% high watermark, we start  deleting the oldest images.  Right?  Now, we don't know if that's going to scale or not.  We're hoping it does.  We think we should be able to retain images for two to three days.  But if this becomes really popular, it goes viral and hundreds of people start hating  it, we may not be able to retain images for very long.  So keep that in mind.  How are we doing for time?  Okay.  We have 15 minutes in.  We have literally all the rest of it.  You get your image, download it, and you can instantiate it on whatever you want.  But not quite because vendors being vendors still want to retain their ability of doing  a little bit of salting, you know, in whatever way to make it make it work for what you want  to do.  So we extend that a little bit.  This is the base config file, right?  What you were looking at earlier.  And it's still the same.  Now, what you can do is you can add an extra stanza to your config file and you can add  a bit more information in it.  So you can have a vendor which at the moment is only sent to us.  We're hoping we can add some other people in there as well.  You can ask for a specific kind of disk type.  Now, at the moment, it's sparse or raw because of the way it's set up to do a Zen build.  But if you were doing a KVM build, you could have QCOW up there.  You could have whatever other format you want.  Essentially there's a QEMU IMG and command that gets run.  So any five format that QEMU can handle, the delivery agent within Rain Drops will  process it for you.  And I think there's like, I don't know, correct me if I'm wrong, but there's about 10 or  15 different formats that you can use with QEMU.  If you want a different format, if you can write a bash script which can take a standard  in for an image and output whatever format you want, we can plumb that in.  You can specify what disk size you want.  Ideally keep this to less than 50 gigs, please, because provisioning 50 gigs in like, for  example, a raw file takes up a lot of space and time.  But there is no actual check on this.  So if you go and say, give me two terabytes worth of a disk.  The builder machines only have about 200 gigs.  So it'll try and allocate that disk.  It'll fail and that'll come up in your log file that, hey, I couldn't, the disk process  didn't complete.  If you want the image compressed, you can ask for a few things.  You can ask for different kind of formats.  And this is the coolest bit.  This is the thing that took me the most amount of time to write is the delivery stanza.  So you can either have this as pickup, in which case you will get an email with the URL  saying, hey, this is the URL that you can download from.  Or you could say aws, colon, colon, your aws authorization tag, the string, the author  tag in here.  And what raindrops will do is it'll actually log into your account on aws, push that image  into an S3 backed, an instance backed image, and send you an email with the AMI, the AKI,  and the image ID that is used.  So that's pretty cool.  And then you can also have an FTP delivery.  If you have an FTP server at your end, you can say give it to me at ftp colon with the  whole URL.  Hopefully add a user ID and password into it.  It'll try and use LFTP to push it.  It can also do rsync.  But rsync is dangerous because it requires a lot of network resources at our end, so  we don't really want to use rsync.  We're happier with FTP.  So if you have an rsync target at your end, you say rsync colon, whatever, it'll actually  rsync the image to you.  And you get the email notification once the delivery is complete.  So if you've asked for AWS delivery, you'll get the email once the image is in AWS.  And then you can add extra stuff.  Like you want no IPv6.  You want a specific mac address to be used or whatever you can add those in.  By default, we only set up one network interface, but the system supports up to three.  And this is something which is, I guess, more relevant to vendors who may want to have every  image coming up with a public interface, a private interface, and a management interface.  And this works.  It handles up to three images at the moment.  If you want any of those to be set up in a specific kind of way, you can specify a mac  address or you can say DHCP and it won't give you, and it won't set up an HWA DDR line.  How many people have used Open Nebula in the past?  Right, a couple.  So you guys know what I mean, right?  So Open Nebula does not need contextualization to set up IPs for VMs.  It converts the mac address into an IP.  And the IP is the quad notification of the mac address.  So over here, you can literally, what you're doing is you're specifying the IP address  for the VM.  Does that make sense?  With the extra options?  Is there any questions anybody has so far?  Hi, De.  So the question is, how would you include software which doesn't come with the mac address?  So the question is, how would you include software which isn't distributed by CentOS?  Whatever mechanism it may come from, right?  The script that you can run here would be the script that you would influence those  kind of things.  For example, if you wanted to include the Apple repository and you wanted to install  something from Apple, what does Apple provide?  Let's say you wanted to run IRCD.  So what you could do is you could have that in a script.  And what you put in over here would be my script.  And then when you do your get add, you need to have that script added in as well.  The only thing is you have to have changes in all three files because the builder process  only checks out the last particular change set.  Does that make sense?  Right.  So the point at which the image gets created, that happens privately.  But the image is then pushed out to a machine which does have internet access, which is  when the script gets run.  So the reason why we have the script up there, rather than telling people to use it in the  post script for the kickstart, is that difference.  At kickstart time, there is no public interface.  There's no public access.  But at this point, there is.  Because that's how even the callbacks and stuff like that would happen.  Of course, it's probably for the sake of completeness, it's worth pointing out that you can only  do that with the raw file.  Because if you ask for a VMDK file, we can't really loop-mount it to do these things.  And because we don't have VMWare running on the builders, we can't actually create an  instance out of it to run that.  So there are some caveats.  A lot of these things are not being tested for at the moment.  So for example, you can say, hey, give me a Vbox image and deliver it to AWS.  That's not going to work.  Because firstly, there's no cloud in it.  There's no metadata in it.  Secondly, Vbox would be doing HVM, not Zen.  So if you're looking for a CentOS 5 install where the Zen kernel is different from the  stock kernel, it's not going to boot.  And what we will do is we will very happily do everything.  We will put it into S3 for you and give you an email saying done.  But obviously, it's not going to work.  So we're kind of hoping that as we release this stuff out for beta today tomorrow, a lot  of you guys are going to go out there and break the system in as many ways as possible  and then submit patches on how to fix it, hopefully.  It is on Gitaurus at the moment in Components.  It's also held, but the actual complete stuff is held on Naza.current.org.  It's a public Git repository.  So I'll give you the URL for that.  And the way that things are implemented is everything is run as, again, of Beanstalk  workers.  And there's a concept of we use standard and standard out to cascade through drivers.  So for example, I'll give you an example of how the delivery agent works.  There's a message that comes through the delivery queue saying, hey, this job is ready for delivery.  And then the worker will work out, how am I delivering this?  I need to put it into an HTTP interface.  Or do I need to deliver it to AWS?  Or do I need to send it by FTP?  So let's assume that there's an FTP request.  It'll actually fire up the drv underscore delivery underscore ftp.sh with the UID.  And then that script will do whatever it needs to do to make that delivery happen.  And everything is monitored for standard and standard out.  So if that script exits with a non-zero value, we treat that as an error, transfer the logover  for the user to see.  Which essentially means that if you've got arcane crazy stuff that you want to do, or  let's say the HP Cloud guys want to have an interface that allows us to push images there,  they can just write a delivery agent for it, which has all of that stuff in it.  We do have some things which are available, like Python Boto is available anyway.  We have the JCloud stack that is available, so you can consume that.  So we've gone through this stuff.  So this is stuff that is kind of functional now.  We've got KVM, QVM, you stuff.  LXE works, Zen works, VMware works, Vbox works with caveats.  And I'll kind of come to that in a second.  Open stack HP Cloud is consumable because we have Cloud in it available within an extra  repository, so you can use that.  That kind of works for eucalyptus as well.  Cloud stack as well, I believe.  You can do AWS, EBS works in a really weird kind of a way in that we inject stuff into  an S3 instance.  We request using your credentials a block device, and then we rsync the OS over to the  block device, remove the SSH keys, remove the metadata which is instance specific and  reboot the machine for you.  So this has never failed for me, but a lot of people tell me there's a better way of  doing it.  But hey, rsync is the gift of God, right?  It works.  And Open Nebula works because you don't need contextualization.  The mac addresses are converted in, and that kind of works.  This is how it's all set up.  We have a scheduler with the Git repositories, which is not really there at the moment.  We're hoping most people use the web interface.  We have the repositories, the builder machines, storage nodes, all hosted at pair one at no  cost to us.  We're not going to do that, guys.  This is kind of how it works.  We do PXC images from the distro to actually bootstrap a real install, except for VMware  and VBox, where we use the distribution ISO and then boot it from that ISO to do the install.  And a couple of things.  We don't have EFI support.  So if you need EFI, you're most welcome to engineer that.  We don't have that support there.  And CloudNet is possible.  It's available over there.  So big challenges.  Hidling handling passwords from Kickstart files.  Don't make your Git repositories public by putting them on GitHub, please.  You're going to be cloning from here, but do not put them there because your Kickstart  files have to have a root password in there.  So even if it's encrypted with MD5 or SHA, given enough resources, people can still go  backwards from it.  So do not do that.  Images are delivered pre-boot run.  So you need to handle first boot if that's an issue for you.  If you're asking for images and if your infrastructure cannot handle first boot, then you need to  kind of do something in the Kickstart file so first boot does not kick in.  Scripts are injected via loop mount.  This is what we were talking about.  If you specify a post-run script, it will be run after the VM is delivered to the head  node.  So you can do stuff like, you know, you can WGAT, you can Git clone.  You can do things like if you want your puppet manifest to be injected into the instance before  it's delivered to you.  That's a good point to do it from there.  And the entire log is available.  Very soon we're going to have stock images for CentOS available.  So if you just want a basic CentOS image, you don't really have to go through the process.  You can just use public images that we'll make available through the process.  And we'll update it every week, every month.  It's not happening yet.  We want to kind of see how scale works and we might need to get a dedicated builder for  this.  But this should happen soon.  Also schedules are going to happen very soon where you can specify schedules that you want  an updated image delivered to you every month, every week, every five hours, whatever.  And that just effectively uses cron at the moment.  It's not public yet.  Don't try to use it.  Big challenges.  Will it scale?  It takes between 44 minutes to 22 minutes per build for an image at the moment.  If you've asked for compression, we do all compression for all images on a very small  machine, because we don't really want to waste CPU cycles doing compression, which means  it could take up to 30 minutes to compress the image before it's available for download.  We don't know how good or bad that is.  But everything else we think is going to scale quite well.  It's secure, so we trust the images that are coming through.  Third-party ACL handling is tricky, but we're kind of hoping people do that on their own.  We're doing AWS.  We might do the Rackspace image as well, the Rackspace cloud stuff.  But we're hoping that Windows do their own thing for the other stuff.  And big challenges.  We're kind of running out of time.  We've got like three minutes.  So I just want to quickly do one thing.  Let me quickly do a demo of what we have up there so you guys can take a look at it.  And then we'll see if we can come back.  Let me log out.  Can everybody see that up there?  So we see log onto the web interface.  This is what you end up at.  Raindrops.net slash login.  It is available right now.  Please do not send hundreds of build instances to it yet.  This is what I hate doing is looking at stuff.  So you can sign in with any of the other, any of the options here.  We're going to try and get a few more options going.  We don't want to know who you are.  We don't want to have to verify who you are.  We kind of trust that GitHub and friends have done this already.  So I'm going to try and log in with GitHub.  It should hopefully work.  There we go.  I'm actually logged into GitHub at the back so that you don't see me typing my password  and acoustically decoded.  This is what the interface looks like at the moment.  You've got jobs that we've set up, none yet.  We've got configuration files, none yet.  Kick starts, none yet.  So let's start with the new kickstart file.  We have a template set up already.  So you can change a few things in there if you want, whatever.  It should kind of work.  I need to give it a name.  So let's say test one.  So there's nothing specific in here in the kickstart.  Because it's a template, it's actually rendered the URL bit there.  Let's see what happens here.  This web interface is really quick, really fast, and it's a network over here which is  really slow.  So if any actually go and see it like away from here, you'll find that it is actually  quite a responsive thing.  We keep version tracking for you with the aim that this is going to actually interface  with the Git repository at some point.  So you've got the kickstart in.  Let's try and get a configuration file going as well.  What am I doing?  I'm hitting the wrong button.  So you've got the kickstart file.  Let's just, the config file, let's just call this for the sake of interest, the same thing.  Let's set a notify by email.  Most of this stuff should be, it's pretty standard.  I want it picked up and I have compression turned on.  I want a 10 gig size disk with a QCao 2 format and no script or anything like that.  And this should then, it would help if I, did I go back?  Okay.  Sorry.  What am I hitting?  I'm hitting the wrong button here every second.  Oh, come on.  Which way am I going?  Let me just see if I can do that.  Right.  So you've got a config file.  You should now be able to go and create a job.  And this should have a dropdown with our existing stuff and submit.  So what actually happens is you've got, you can open the job again, you can delete it,  or you can see the results, which is, and you should see results as things are building.  So what's also interesting here is that you can actually see the status of the back end  workers, which are all sleeping at the moment.  Let's see if that picks up something.  It's actually waiting for the next schedule, next build schedule to come up.  Hopefully that'll happen in the next second or so.  Let me see here if I can influence that in any way.  I'd actually pulled up the worker screen for the build that is going to run.  So once the prep happens, this is effectively what's running at the back, you've got the  delivery agent running.  So the one that I was talking about, the compression system.  That's kind of like how it works.  It should also be a builder instance here.  So these are the builder threads that are waiting for the job to come through.  Are we out of time?  We are out of time.  Does anybody want to talk about anything in the minute or two that it takes to get the  builder going?  Because the builds will only run once every five minutes for the schedule to come around.  Sure.  Inside?  Yes.  So what we're actually hoping to do is actually deliver the entire raindrop stack as a VM  within raindrops itself.  So what you could say is, hey, give me a raindrops instance for my cloud stack deployment.  And then you have your own private set up which is already there.  And the software is all free.  I think we've been talking about Open Nebula.  Open Nebula is going to try and ship a VM which runs raindrops from within the Open  Nebula market space.  We're kind of hoping more people do that as well.  Does that answer your question?  Is there anything else you want to talk about?  We probably won't have time to see this build running.  We've got another two minutes to go for the next schedule.  No?  Well, thank you for coming along.  Highly recommend everybody goes to the interface, check it, break it, and then submit patches  to fix it, please.  Submit patches is what you need to focus on.  Everything else will happen automatically, I think.  Thank you.  Thanks, guys.  Thanks for being there.  Bye.
Raindrops_XmjYttyQf_U_2_0_5651.png Raindrops_XmjYttyQf_U_2_5652_6107.png Raindrops_XmjYttyQf_U_2_6108_8483.png Raindrops_XmjYttyQf_U_2_8484_9563.png Raindrops_XmjYttyQf_U_2_9564_9659.png Raindrops_XmjYttyQf_U_2_9660_9707.png Raindrops_XmjYttyQf_U_2_9708_12551.png Raindrops_XmjYttyQf_U_2_12552_13583.png Raindrops_XmjYttyQf_U_2_13584_15455.png Raindrops_XmjYttyQf_U_2_15456_15683.png Raindrops_XmjYttyQf_U_2_15684_16583.png Raindrops_XmjYttyQf_U_2_16584_17927.png Raindrops_XmjYttyQf_U_2_17928_20327.png Raindrops_XmjYttyQf_U_2_20328_22571.png Raindrops_XmjYttyQf_U_2_22572_24227.png Raindrops_XmjYttyQf_U_2_24228_24395.png Raindrops_XmjYttyQf_U_2_24396_25115.png Raindrops_XmjYttyQf_U_2_25116_25367.png Raindrops_XmjYttyQf_U_2_25368_25547.png Raindrops_XmjYttyQf_U_2_25548_26075.png Raindrops_XmjYttyQf_U_2_26076_28187.png Raindrops_XmjYttyQf_U_2_28188_28535.png Raindrops_XmjYttyQf_U_2_28536_32183.png Raindrops_XmjYttyQf_U_2_32184_34559.png Raindrops_XmjYttyQf_U_2_34560_38555.png Raindrops_XmjYttyQf_U_2_38556_39623.png Raindrops_XmjYttyQf_U_2_39624_42431.png Raindrops_XmjYttyQf_U_2_42432_43187.png Raindrops_XmjYttyQf_U_2_43188_44447.png Raindrops_XmjYttyQf_U_2_44448_44579.png Raindrops_XmjYttyQf_U_2_44580_44586.png
Some of these slides have been recycled from talks and some of you may have heard the talk last year except the project has been evolving. So hopefully when you go through the whole thing what you see here now, a lot of maybes and a lot of coming soon have actually been replaced by functionality that exists right now. So just as a show of hands, how many people here use the cloud or a cloud or the cloud? Okay, so most people, excellent. So we don't need to go into the basic stuff. The raindrops interface is something that, okay, a little bit about me. I'm a sad man. I'm based out of London. I do sent our stuff. I like bullet points and I like the rain. And Belgian bears, yes, Belgian bears are good. Although I've lived in the UK now for 12 years, I still prefer Belgian bears. I'm probably going to get assassinated on the way back or something. I'm saying that. So the thing is just looking at the problem definition, what we were trying to solve. Sentos is everywhere in terms of the cloud, right? It's a lot of people, a lot of people who are selling services on the cloud. You sent us a lot of people who are consuming the cloud in various ways. You sent us a lot of vendors, a lot of hosting companies, retail and the cell sent us images and instances. But the sent us distribution itself is built for traditional use cases. It wasn't built with the cloud in mind. I mean, the way the installer works is meant to be sort of automatable in a conventional traditional kind of an infrastructure environment. The way the management tools work is, again, it's kind of built for traditional enterprise users or traditional kind of setups. And that kind of caused a bit of a problem. Because even a delivery for what we actually push out is we deliver ISOs on an HTTP interface that people can download, which are not usable directly. But what they deliver to you as a user is finite, is fixed. When you install sent us, the kernel you get is predetermined. You know exactly what kernel you're going to get. The version of bash you're going to get, the version of PHP or HTTP or Apache or whatever you're going to get is predetermined. But the mechanism that we shift with and the mechanism that we should deliver with is tuned for traditional architectures as it were. And traditional media, we do ISOs or whatever. But how different are the cloud guys? They're not really very different in that you still have to do a provision. You still have to build an image. You still need to have something that you can run. You still need to manage that instance. You still need to patch manage within that instance for security issues, for bug fixes. So from that perspective, they're not very different from traditional use cases. But they're different enough to merit specific consideration. And to us, the biggest deal was that we need to be able to talk to them about delivery mechanisms and how they consume what we ship and how they deliver to users, what they further on ship to consume. So the issue was that we found a lot of people were coming back to us and saying, hey, I bought a VPS from hosting company A and they gave me this, this, this in it. Then I wanted to move to hosting company B and I bought a CentOS 6 VPS and that had something completely different. What is that all about? And then I wanted to move to AWS and that was a completely different thing. And then I wanted to set up a developer VM on my laptop and that looked like it was completely different. And basically what was happening was that every vendor takes what we push out and salts it to taste. And when I mean taste, that usually depends on what distribution, what best practices, what issues, what history, what has been the life-term experience of the guy who's actually creating the images. In many times, it doesn't reflect company policy. In many times, it doesn't reflect what they're actually expecting to shift. It reflects what the guy building the image thought was what his users want. And that's a dangerous situation to be in. So we started talking to vendors. We started talking to people who would come back and complain to us. And we found that there were issues like people were shipping 5.3 in 2011. People were shipping 6.0 in 2012. And we were like, do you know that those images have security issues? Do you know that you should not be doing this kind of thing? And a lot of them were, you know, we put in the effort. We went through the pain of setting up a VM. It took us like three weeks to get it going. I don't have the resources to do it again. And that was pretty much uniform from all hosting companies from HPC places across the board. So we thought, okay, this is a problem that we can solve. So we spoke to some of the users that the vendors gave us access to. So the vendor said, hey, I've got 100 users. Here's 10 you can talk to and see what the actual problems are. And we did that. We spoke to people who consume liquid web services. We spoke to people who host with Pair One. We spoke to the big AWS users. We spoke to some people Vee knew who are running private clouds as well to see what we could actually do. And the problems like we were talking about delayed access. People want, vendors don't have resources to make that happen. So basically there was a massive gap between what the vendors were doing and what the users really wanted. So the problem we can really define down to users want persistence for images. They want something that is reliable that looks like CentOS irrespective of where they are going to. What vendors they're working with. What sort of environment they're on. If they're on a local box or if they're not a remote box. But they still want to retain third party support. So it's like if you want a VM from a hosting company, let's say Pair One, you want to be able to get Pair One to still support your image for you. You still want to have that phone number that you can call. Because if Pair One does something like here, here's the official CentOS image, all the best. Let me know if you have a problem and just make sure your credit card is valid and we'll keep charging you. You don't want to be in a situation where you can't get support anymore. Which means we need a consistent platform from the vendors to CentOS and from CentOS why are those vendors across to the users as well. Does that kind of make sense? Yeah? Okay. So we found that on cloud and off cloud are not, they don't really look the same. You can still have VMs. But you could have a VM on your machine, you could have virtualized infrastructure or you could have a cloud. To me, those three things look very different. Virtualized infrastructure is classic sort of VMware kind of perspective. Cloud is classic AWS kind of a thing where HA may not be that big a deal. You HA your app rather than HA in your instance. And all of them are kind of, they look just different enough that they merit sort of consideration specifically. And of course, kernel and app support, somebody's going to run Oracle, somebody's going to run DB2 or whatever, somebody's going to run MySQL, somebody's going to run SQLite. And then people like me still think that if you only have a thousand or two objects, then flat files are fine. It's true. It's true. I mean, you know, it's the, it's the, it's the want to pain that people go through to set up MySQL with, you know, all sorts of schemas and everything and then they run a blog which has five posts, right? Have you tried running flat files with five lines in it these days? Do you know what kind of performance you can get there? So basically what we did was we got all of that stuff together and we came up with an instance called raindrops.centOS.org. We're actually launching this as project raindrops.net, but it's also available from, from the raindrops.centOS.org interface. What it is is it's a web, email and a Git interface. You create a kickstart file and this looks like a conventional kickstart file. You've got install, because that's what you want to do. Things that are different. I'll point this out here. The URL that you're installing from has to have rep a host as the place is going to pick up content from because the builds happen privately. They're not available on the internet. So you have to have that and you have to have whatever the centOS mirror URL is. It can't be, you can't use variables. You can't use base arch or you can't use release version. It has to be fixed to six and the architecture that you want or five in the architecture. Everything else is about what you'd have in a normal kickstart file. You've got root password which should, you should probably keep fairly secure. You have a network device. Okay. So the other thing which is probably worth mentioning over here is that you have to have partitioning information and you can set it up whichever way you want. And you have to have a kickstart which completes without user interaction. So if you remove, for example, the network config line or you remove the root password file, it will get stuck. Anaconda cannot handle that. It's going to ask for input. So you have to have a kickstart file that's complete. Is that, is that kind of clear with everybody? Is there anything? Okay. Now what you also need to do is you need to create a config file that tells you what, that tells the rain drops what to do with that kickstart file. Right? So in this case we've got an image target which is the architecture that we want to build on. Now this sounds a bit silly because if you look back, your URL has that in it. But what happens is that we actually deploy host environment that we're going to build the image into and you can control that. So you can say, I want, and this is me being completely fictitious here. You can say, hey, I want a MIPS build on an x8664 sherooted environment which you can achieve with rain drops. We don't have a MIPS build, but if you did, it'll be the best MIPS build in the world. You have to have a UID which is actually injected in if you don't put it in. So we recommend you don't use a UID here. And then things, life is a lot easier because you won't have to remember three sets of UIDs. Just get one. You can have a PostScript which is external to the kickstart. If you want something to run, if you want kickstart to deploy an image, instantiate the image and then run a script, you can inject that here. That'll get run in. Schedule is something which is coming soon. It's not there yet. But what you can say is that I want to schedule and give it 30 days. And what will happen is this particular config file will get reprocessed every 30 days. So you get a new image every 30 days. This is not functional yet because we want to see, we want to do a little bit of testing around how we can handle load and how we can handle scale and then how we can actually get a scheduler which kind of does the right thing. The updates, yes, again, is something like if you're doing an install, we just need to tell what OS we're installing. Having an updates is equal to yes will mean it'll actually run a yum updates post install. It's just you can have that in your kickstart if you want or you can have it in here as well. This is handy when we do installations for certain hypervisors which don't allow us to have multiple repositories in the kickstart. The kickstart file. The kickstart file has to match the name that you're going to use here. So whatever kickstart you've called it, you can call it that. You have an image tag if you want to add something in. You need to use what hypervisor target you want and who you want to notify. So this could be an email address. It could also be a URL if you want to call back. So if you're automating your scripts and you're automating your kickstarts, you can have a URL over here. And when the image is done, it'll actually do a call back and it'll give you some details. Okay, nobody can read that without my email address. If you were using the git interface, this is what you would do. You would do git add my image.ks. You would git add your config file. You'd git commit with build colon space, whatever you want, and you do a git push. And that's it. And that would trigger the build. It would process the config file, get the kickstart into place. It'll trigger the image and do whatever you're told it to do. This is literally all you need to do to get your image out. I'd asked for an email notification. So this is kind of like the notification you'll get. Your job is complete. You get the download and the log. Now, we have finite storage because we've had to kind of get the hat around and pass it around to see who would actually donate the machines and infrastructure. I'm very grateful to peer one for forking out six machines. And the storage boxes have about two and a half terabytes of space. But this is going public. Two and a half terabytes does not last very long. So if you guys do start using it and you do ask for email notification, we highly recommend you get that image within a few hours or maximum 24 hours. Because what's going to start happening is when we hit the 80% high watermark, we start deleting the oldest images. Right? Now, we don't know if that's going to scale or not. We're hoping it does. We think we should be able to retain images for two to three days. But if this becomes really popular, it goes viral and hundreds of people start hating it, we may not be able to retain images for very long. So keep that in mind. How are we doing for time? Okay. We have 15 minutes in. We have literally all the rest of it. You get your image, download it, and you can instantiate it on whatever you want. But not quite because vendors being vendors still want to retain their ability of doing a little bit of salting, you know, in whatever way to make it make it work for what you want to do. So we extend that a little bit. This is the base config file, right? What you were looking at earlier. And it's still the same. Now, what you can do is you can add an extra stanza to your config file and you can add a bit more information in it. So you can have a vendor which at the moment is only sent to us. We're hoping we can add some other people in there as well. You can ask for a specific kind of disk type. Now, at the moment, it's sparse or raw because of the way it's set up to do a Zen build. But if you were doing a KVM build, you could have QCOW up there. You could have whatever other format you want. Essentially there's a QEMU IMG and command that gets run. So any five format that QEMU can handle, the delivery agent within Rain Drops will process it for you. And I think there's like, I don't know, correct me if I'm wrong, but there's about 10 or 15 different formats that you can use with QEMU. If you want a different format, if you can write a bash script which can take a standard in for an image and output whatever format you want, we can plumb that in. You can specify what disk size you want. Ideally keep this to less than 50 gigs, please, because provisioning 50 gigs in like, for example, a raw file takes up a lot of space and time. But there is no actual check on this. So if you go and say, give me two terabytes worth of a disk. The builder machines only have about 200 gigs. So it'll try and allocate that disk. It'll fail and that'll come up in your log file that, hey, I couldn't, the disk process didn't complete. If you want the image compressed, you can ask for a few things. You can ask for different kind of formats. And this is the coolest bit. This is the thing that took me the most amount of time to write is the delivery stanza. So you can either have this as pickup, in which case you will get an email with the URL saying, hey, this is the URL that you can download from. Or you could say aws, colon, colon, your aws authorization tag, the string, the author tag in here. And what raindrops will do is it'll actually log into your account on aws, push that image into an S3 backed, an instance backed image, and send you an email with the AMI, the AKI, and the image ID that is used. So that's pretty cool. And then you can also have an FTP delivery. If you have an FTP server at your end, you can say give it to me at ftp colon with the whole URL. Hopefully add a user ID and password into it. It'll try and use LFTP to push it. It can also do rsync. But rsync is dangerous because it requires a lot of network resources at our end, so we don't really want to use rsync. We're happier with FTP. So if you have an rsync target at your end, you say rsync colon, whatever, it'll actually rsync the image to you. And you get the email notification once the delivery is complete. So if you've asked for AWS delivery, you'll get the email once the image is in AWS. And then you can add extra stuff. Like you want no IPv6. You want a specific mac address to be used or whatever you can add those in. By default, we only set up one network interface, but the system supports up to three. And this is something which is, I guess, more relevant to vendors who may want to have every image coming up with a public interface, a private interface, and a management interface. And this works. It handles up to three images at the moment. If you want any of those to be set up in a specific kind of way, you can specify a mac address or you can say DHCP and it won't give you, and it won't set up an HWA DDR line. How many people have used Open Nebula in the past? Right, a couple. So you guys know what I mean, right? So Open Nebula does not need contextualization to set up IPs for VMs. It converts the mac address into an IP. And the IP is the quad notification of the mac address. So over here, you can literally, what you're doing is you're specifying the IP address for the VM. Does that make sense? With the extra options? Is there any questions anybody has so far? Hi, De. So the question is, how would you include software which doesn't come with the mac address? So the question is, how would you include software which isn't distributed by CentOS? Whatever mechanism it may come from, right? The script that you can run here would be the script that you would influence those kind of things. For example, if you wanted to include the Apple repository and you wanted to install something from Apple, what does Apple provide? Let's say you wanted to run IRCD. So what you could do is you could have that in a script. And what you put in over here would be my script. And then when you do your get add, you need to have that script added in as well. The only thing is you have to have changes in all three files because the builder process only checks out the last particular change set. Does that make sense? Right. So the point at which the image gets created, that happens privately. But the image is then pushed out to a machine which does have internet access, which is when the script gets run. So the reason why we have the script up there, rather than telling people to use it in the post script for the kickstart, is that difference. At kickstart time, there is no public interface. There's no public access. But at this point, there is. Because that's how even the callbacks and stuff like that would happen. Of course, it's probably for the sake of completeness, it's worth pointing out that you can only do that with the raw file. Because if you ask for a VMDK file, we can't really loop-mount it to do these things. And because we don't have VMWare running on the builders, we can't actually create an instance out of it to run that. So there are some caveats. A lot of these things are not being tested for at the moment. So for example, you can say, hey, give me a Vbox image and deliver it to AWS. That's not going to work. Because firstly, there's no cloud in it. There's no metadata in it. Secondly, Vbox would be doing HVM, not Zen. So if you're looking for a CentOS 5 install where the Zen kernel is different from the stock kernel, it's not going to boot. And what we will do is we will very happily do everything. We will put it into S3 for you and give you an email saying done. But obviously, it's not going to work. So we're kind of hoping that as we release this stuff out for beta today tomorrow, a lot of you guys are going to go out there and break the system in as many ways as possible and then submit patches on how to fix it, hopefully. It is on Gitaurus at the moment in Components. It's also held, but the actual complete stuff is held on Naza.current.org. It's a public Git repository. So I'll give you the URL for that. And the way that things are implemented is everything is run as, again, of Beanstalk workers. And there's a concept of we use standard and standard out to cascade through drivers. So for example, I'll give you an example of how the delivery agent works. There's a message that comes through the delivery queue saying, hey, this job is ready for delivery. And then the worker will work out, how am I delivering this? I need to put it into an HTTP interface. Or do I need to deliver it to AWS? Or do I need to send it by FTP? So let's assume that there's an FTP request. It'll actually fire up the drv underscore delivery underscore ftp.sh with the UID. And then that script will do whatever it needs to do to make that delivery happen. And everything is monitored for standard and standard out. So if that script exits with a non-zero value, we treat that as an error, transfer the logover for the user to see. Which essentially means that if you've got arcane crazy stuff that you want to do, or let's say the HP Cloud guys want to have an interface that allows us to push images there, they can just write a delivery agent for it, which has all of that stuff in it. We do have some things which are available, like Python Boto is available anyway. We have the JCloud stack that is available, so you can consume that. So we've gone through this stuff. So this is stuff that is kind of functional now. We've got KVM, QVM, you stuff. LXE works, Zen works, VMware works, Vbox works with caveats. And I'll kind of come to that in a second. Open stack HP Cloud is consumable because we have Cloud in it available within an extra repository, so you can use that. That kind of works for eucalyptus as well. Cloud stack as well, I believe. You can do AWS, EBS works in a really weird kind of a way in that we inject stuff into an S3 instance. We request using your credentials a block device, and then we rsync the OS over to the block device, remove the SSH keys, remove the metadata which is instance specific and reboot the machine for you. So this has never failed for me, but a lot of people tell me there's a better way of doing it. But hey, rsync is the gift of God, right? It works. And Open Nebula works because you don't need contextualization. The mac addresses are converted in, and that kind of works. This is how it's all set up. We have a scheduler with the Git repositories, which is not really there at the moment. We're hoping most people use the web interface. We have the repositories, the builder machines, storage nodes, all hosted at pair one at no cost to us. We're not going to do that, guys. This is kind of how it works. We do PXC images from the distro to actually bootstrap a real install, except for VMware and VBox, where we use the distribution ISO and then boot it from that ISO to do the install. And a couple of things. We don't have EFI support. So if you need EFI, you're most welcome to engineer that. We don't have that support there. And CloudNet is possible. It's available over there. So big challenges. Hidling handling passwords from Kickstart files. Don't make your Git repositories public by putting them on GitHub, please. You're going to be cloning from here, but do not put them there because your Kickstart files have to have a root password in there. So even if it's encrypted with MD5 or SHA, given enough resources, people can still go backwards from it. So do not do that. Images are delivered pre-boot run. So you need to handle first boot if that's an issue for you. If you're asking for images and if your infrastructure cannot handle first boot, then you need to kind of do something in the Kickstart file so first boot does not kick in. Scripts are injected via loop mount. This is what we were talking about. If you specify a post-run script, it will be run after the VM is delivered to the head node. So you can do stuff like, you know, you can WGAT, you can Git clone. You can do things like if you want your puppet manifest to be injected into the instance before it's delivered to you. That's a good point to do it from there. And the entire log is available. Very soon we're going to have stock images for CentOS available. So if you just want a basic CentOS image, you don't really have to go through the process. You can just use public images that we'll make available through the process. And we'll update it every week, every month. It's not happening yet. We want to kind of see how scale works and we might need to get a dedicated builder for this. But this should happen soon. Also schedules are going to happen very soon where you can specify schedules that you want an updated image delivered to you every month, every week, every five hours, whatever. And that just effectively uses cron at the moment. It's not public yet. Don't try to use it. Big challenges. Will it scale? It takes between 44 minutes to 22 minutes per build for an image at the moment. If you've asked for compression, we do all compression for all images on a very small machine, because we don't really want to waste CPU cycles doing compression, which means it could take up to 30 minutes to compress the image before it's available for download. We don't know how good or bad that is. But everything else we think is going to scale quite well. It's secure, so we trust the images that are coming through. Third-party ACL handling is tricky, but we're kind of hoping people do that on their own. We're doing AWS. We might do the Rackspace image as well, the Rackspace cloud stuff. But we're hoping that Windows do their own thing for the other stuff. And big challenges. We're kind of running out of time. We've got like three minutes. So I just want to quickly do one thing. Let me quickly do a demo of what we have up there so you guys can take a look at it. And then we'll see if we can come back. Let me log out. Can everybody see that up there? So we see log onto the web interface. This is what you end up at. Raindrops.net slash login. It is available right now. Please do not send hundreds of build instances to it yet. This is what I hate doing is looking at stuff. So you can sign in with any of the other, any of the options here. We're going to try and get a few more options going. We don't want to know who you are. We don't want to have to verify who you are. We kind of trust that GitHub and friends have done this already. So I'm going to try and log in with GitHub. It should hopefully work. There we go. I'm actually logged into GitHub at the back so that you don't see me typing my password and acoustically decoded. This is what the interface looks like at the moment. You've got jobs that we've set up, none yet. We've got configuration files, none yet. Kick starts, none yet. So let's start with the new kickstart file. We have a template set up already. So you can change a few things in there if you want, whatever. It should kind of work. I need to give it a name. So let's say test one. So there's nothing specific in here in the kickstart. Because it's a template, it's actually rendered the URL bit there. Let's see what happens here. This web interface is really quick, really fast, and it's a network over here which is really slow. So if any actually go and see it like away from here, you'll find that it is actually quite a responsive thing. We keep version tracking for you with the aim that this is going to actually interface with the Git repository at some point. So you've got the kickstart in. Let's try and get a configuration file going as well. What am I doing? I'm hitting the wrong button. So you've got the kickstart file. Let's just, the config file, let's just call this for the sake of interest, the same thing. Let's set a notify by email. Most of this stuff should be, it's pretty standard. I want it picked up and I have compression turned on. I want a 10 gig size disk with a QCao 2 format and no script or anything like that. And this should then, it would help if I, did I go back? Okay. Sorry. What am I hitting? I'm hitting the wrong button here every second. Oh, come on. Which way am I going? Let me just see if I can do that. Right. So you've got a config file. You should now be able to go and create a job. And this should have a dropdown with our existing stuff and submit. So what actually happens is you've got, you can open the job again, you can delete it, or you can see the results, which is, and you should see results as things are building. So what's also interesting here is that you can actually see the status of the back end workers, which are all sleeping at the moment. Let's see if that picks up something. It's actually waiting for the next schedule, next build schedule to come up. Hopefully that'll happen in the next second or so. Let me see here if I can influence that in any way. I'd actually pulled up the worker screen for the build that is going to run. So once the prep happens, this is effectively what's running at the back, you've got the delivery agent running. So the one that I was talking about, the compression system. That's kind of like how it works. It should also be a builder instance here. So these are the builder threads that are waiting for the job to come through. Are we out of time? We are out of time. Does anybody want to talk about anything in the minute or two that it takes to get the builder going? Because the builds will only run once every five minutes for the schedule to come around. Sure. Inside? Yes. So what we're actually hoping to do is actually deliver the entire raindrop stack as a VM within raindrops itself. So what you could say is, hey, give me a raindrops instance for my cloud stack deployment. And then you have your own private set up which is already there. And the software is all free. I think we've been talking about Open Nebula. Open Nebula is going to try and ship a VM which runs raindrops from within the Open Nebula market space. We're kind of hoping more people do that as well. Does that answer your question? Is there anything else you want to talk about? We probably won't have time to see this build running. We've got another two minutes to go for the next schedule. No? Well, thank you for coming along. Highly recommend everybody goes to the interface, check it, break it, and then submit patches to fix it, please. Submit patches is what you need to focus on. Everything else will happen automatically, I think. Thank you. Thanks, guys. Thanks for being there. Bye.
last year except the project has been evolving. stuff. The raindrops interface is something that, that every vendor takes what we push out and of them were, you know, we put in the effort. We users. Here's 10 you can talk to and see what the between what the vendors were doing and what the users really wanted. So the problem we can really they look just different enough that they merit it's a web, email and a Git interface. You create a kickstart file and this looks like a architecture that we want to build on. Now this do a little bit of testing around how we can something in. You need to use what hypervisor you were using the git interface, this is what you want no IPv6. You want a specific mac address to boot. And what we will do is we will very we use the distribution ISO and then boot it from that Windows do their own thing for the other version tracking for you with the aim that this actually see the status of the back end workers,