Architecture of a cloud hosting service using python technologies: django, ansible and celery
Martin, Abraham
Abraham Martin - Architecture of a cloud hosting service using python technologies: django, ansible and celery The talk will show the architecture and inners of a cloud hosting service we are developing in the University of Cambridge based on python technologies, mainly django, ansible, and celery. The users manage their hosts using a web panel, developed in django, with common options: ability to create a vhost, associate domain names to vhosts, install packages, recover from backups, make snapshots, etc. Interaction between the panel and the hosts are made using ansible playbooks launched asynchronously by celery tasks. The VM architecture has been designed to be VM platform agnostic and to provide disk replication and high availability. The University of Cambridge central IT services also provides other services to the rest of the university like domain name registration, authentication, authorisation, TLS certificates, etc. We link all these other services with the hosting service by using APIs while keeping a microservices architecture approach. Thus, enabling the use/link of other services within the same hosting service web application.
 Hello everyone, I'm Abraham Martin. I work for the University of Cambridge, as you may have guessed from the huge logo on the screen. I was thinking about making a little introduction and I thought everyone knows what the University of Cambridge is, so I think I have little to say about that. And I thought I can show you some pretty pictures so you can invite me where I live and this preplace and where I work. This nice architecture, this nice rivers you can pond with your friends during summer, well the two days of summers we have a year. The classic math bridge, as you may know, we have a lot of clever people there, some noble prices, people that usually work around the city dressed like this, which are gongs. It seems weird but you really see, when you go there, you really see these people working around academics. This seems pretty classic, pretty old, but we also have pretty nice new buildings like this one, which is the new University data center, which is one of the top data centers in the UK. It's pretty big, it's green, we have a lot of innovation inside it. We even have HPC service, which has, in 2003 was the second most green computer in the top of 500, so it's not just classic building and architecture, we also have some cool things. This is the computer lab, what I used to work. I did my relationship with the university, started with the computer lab, I was doing my PhD and then I worked there as a postdoc. This building is called William Gates Building because Bill Gates paid for the half of the building and I work on this building now, which is the university computing service, where we provide IT services for the rest of the university. Both buildings share a common history. They used to be the mathematical laboratory where this machine, brownie points if you know what it is, it's the exact computer, one of the first computers in the world based on the von Neumann architecture. Those built there, we still have some pieces in the computer lab as a museum. But also other things were built there and in the part that I'm working now, which is the university computing service, like Exim, which you probably know about because still 50% of mail servers use it. We have pretty cool people working there, I'm not of any of them, but we have some pretty cool people that also work in a lot of open source projects. What I want to explain to you today is one service that was proposed a lot of years ago, which is the managed web service, and was born to solve a problem. We have a lot of researchers in the university, as you may know, and a lot of them usually hold a conference, they do research, they want to do some simple website with the statistics, show statistics, show results from the research, or either do questionaries, et cetera, et cetera. They end up using their own web servers under the desk. It was a cheap computer, it was running under the desk. It was not maintained usually because the academics usually use that computer for the conference and then they left this computer under the desk, the software was not updated, and then we get security problems. We get servers hacked, et cetera, et cetera, you know. The proposal that the AT services and the university did for solving that problem was centralizing these web services. The solution was to provide a service where you don't have to worry about maintaining the OS or the software, you only have to worry about maintaining the web application. We maintain the OS, we give basic web hosting capabilities like external services does, you don't have to worry about backups, and you have some dedicated resources to your web app. That's very, very old, and when I say gold, it's like 15 years ago. The first version of the managed web service was using a Solaris 7 running in the Solaris machine, so you can see that it was using a very old version of Apache, PHP, and it was using a true system to maintain the separation between these different web pages. The second version came sooner than the other one, provided the new software, XLRS 10, Apache 2, more new software, and started to use Solaris zones, which is a kind of virtualization inside the Solaris machine. It's kind of a container, so we were using containers before it was cool, but it's still pretty old. It also had more enhanced features like database, driven script, so you could do a script based on some information in database, so it's centralized, it's very to manage, some needs, and NFS server, very classic, set-afes file system, so it provides also snapshots, which is good, and the users were able to create Vhost, Ali-Assist, et cetera, but the problem was everything is manual, so they had the users send us an email saying, I want this, and we make the changes, we execute the scripts, the scripts make the changes, but everything is manual, so we need a lot of human intervention. So when it started to grow, we currently now have more than 200 users and more than 400 websites, it started to become a little bit difficult to manage, because it requires a lot of time. So before we ended up making a new version of the managed web service, another solution was in parallel, which is the Falcon service, which is a clone-based, you only get a clone instance, you don't get access to a server or anything, it's just a CMS as a service, and we also have like 200 websites there. So if you go to any university website, you probably will end up in either Falcon service or a managed web service service, so for example, one of the most visited websites inside the service is the Stephen Hawkins website. So we decided to make from scratch a new service, so restart what we have done, because we don't have more Solaris machines, Solaris machines are routing, they're pretty old, we don't have a replacement for that, so we also thought, let's do more automation, we can do more automation, so it requires less time from us. So we decided to go to the classic dedicated VMs, but still maintain the same things that was proposed by the previous ones, like no root access for users, and everything is maintained by us. I want to say by us, I mean by Ansible, because we don't touch anything, but we will see that later. And so to end up these emails that come to our inbox saying, can you please install these packages, can you please install these, we created a web panel using Django, where we delegate some power to the users, so the users can do things without having root access or anything. So architecture is basically a devian aid machine, we installed the basic packages that we have been installing up to now, like the Apache, which is the most common feature of the month, but we also support other Apache available, like if you want to install Python, Django, et cetera. We have a list of system packages that you can install, and they are pre-approved, so you don't end up with a machine with a lot of packages that you don't need or that are strange to need. And we give them all the power to do authorization to the sites, create the host, apply for domain names, install TLS certificates in the machines, do the back-ups from them, password reset, power management, et cetera, et cetera. So we give them the power to do a lot of things that we were doing before. So they have a panel, don't blame me for the design, it's an in-house design, if you visit any Cambridge website, you will see that all of them look exactly the same. So it's just a panel with some options to manage your site, so when you create a site you get this web panel based on Django, you have some options to create Vhost, ask for domain names, et cetera, and you get an extra VM, which is a test server, so you can clone your production server to a test server, you can test things there without having to compromise your production server, so that's good, especially for people that have Drupal installing the manage web servers when they Drupal, really bad things happen. So it's very, you can test it before, and if it goes right, you clone it back to the production server. So architecture looks like that, we will go one by one, see how we build it. Be aware that this is not a talk about OpenStack, neither about talker, so don't expect any of that. But we use all the, most of them are Python technologies, and we did that project in a few months, it's still not finished, we are still working on it, but most of it we have done it by using one and two, 1.2, 1.3 FT, so it's not much people, so the amount of resources that you require for doing it, although it seems like a huge service, it's not that big. So we have here the VM architecture, here, the VM is separated, the VM service is separated from the rest of the stack, so we start describing the VM architecture. The VM architecture is just a VMware solution, you may know these VMware solutions, it's just ESXCI servers, and you can manage these XCI servers using VSphere control panel and some APIs, and we have a standard backup server where we do the backups, but it's not replicated, so if something happens, then we rebuild the VM and recover the things from the backup server. So the flow is easy, a user enters to the Django web panel, authenticates, so we know who he is, and then he asks for a new managed web server. A host name and an IP64 are allocated to this site, the VM API creates a new VM, the VM API installs the OS, and when the OS is ready, Ansible is executed, and Ansible is the one that configures the whole machine, so we are using Ansible as a configuration management and it does everything we need. So there's no Ansible, it's just a bunch of things together, they are easy to script, so it's very easy to understand what they are doing, they separate into folders, which is very good, and you can find the file that you are looking for, and there is separation of things in the different files that you can see. So it's pretty good to use, it also has inventory, so you can define all your servers based on dynamically or static, so you can have a file with all your servers, or you can inject the output from another API as a list of servers you have, or even the database, et cetera. So it's pretty nice, it works really well, and it based on playbooks, playbooks is just a bunch of roles linked to a bunch of targets, so you have a role and definition of role is things you want to install in this role, in these machines that have this role, and then you have targets, and then you say this target, these machines, I want to install this role, for example, a web server, web server can be a role, and the web server role has a lot of tasks, then install Apache, configure Apache, et cetera, et cetera. This is an Ansible playbook, as I said before, you define the host where you want to install things, and then you define the roles, does these machines in this list will have. For each role you have tasks, templates, which are changed to templates, scripts, handles, and variables, you can have also global variables or variables entering into the script, and this is how a role looks like, just a bunch of tasks inside the role, you can see that here we're installing packages, it's a Jamel file, as you can see, and it's pretty easy to understand what you are doing, so if you're working with more people, it's easy to modify the file, change other configuration, et cetera. You can see here that the templates can be used with variables, are Ginge to templates, we use variables there, and therefore we use the same templates for all the configurations, all machines. You also have Handlers, which are basically callbacks, when some function in Ansible is executed, then you have a callback later, and you can, for example, if you have updated the Apache configuration or your Django app, you can restart Apache, and the callback is cool. So this is from the VM part, we use this VM for infrastructure, we use the APIs, we launch it, we create the VM, everything is good, after that, Ansible configures the machine, and then we can offer the service to the user. So if we start from the top of the stack, we see authentication part, we have our own authentication, we use Raven, Raven is our authentication service, so you can see that we have a lot of services interconnected using a lot of APIs. It is based on a web of API, and we have to build a custom Django web again, but this could be substituted by any authentication that you could use. You can use the Django one if you want, you can link with your own enterprise if you want, et cetera. So the second layer is authorization, we have a kind of a Nellabish service, it is called lookup, and then what we have there is just a list of users and a list of groups, we can see these users if they belong to each, which institutions they belong to, which groups they belong to, so the end user can configure their MW server based on this list, they can search for other users, authorize them as administrators, authorize by groups, et cetera. So it is just a basic list. We use that instead of using the Django groups, because it is more useful for us, because people use in the university to this L-Lab service that we have, so they create the groups there and they are automatically updated if someone leaves, et cetera. So, when the user has authorize the user, another user to enter the machine or use the service, we need to still install the user into the machine. So we have another service over there called Django, which provides more information about users, it is like user identity management. So we get from Django a unique UID from the user, we need that, because if we install the same user in different machines, we still need to identify the files that belong to him or to her in different machines, so they have the same unique UID and we use this unique UID in all the user installs. Users install using Ansible as well, it is installed on all the VMs where it is authorized, and we have periodic refreshes to refresh the lookup groups we have authorized, so if the groups change, people different change, and we allow people to upload their own SSH keys and the SSH keys is also installed in the user configuration, so they can enter either using the password, which is checked with this L-Lab server or either the SSH that they have installed in the panel. So once they can access to the Django panel and they have the user installed in the VM, they can already access the machine, everything is configured for them, so they can start using it. For previously, we had also another communication with the IP register API, which is on the bottom there. This allows us, this is another external service, so as you can see, we have the main service and then we have a lot of other services that we communicate with, which provides the university registration for KAMACAC domains, so if you want to register a new KAMACAC domain, for example, important studies.KAMACAC, you can launch an API, we launch an API request using the same Django panel, Django panel sends the request, and then we get the main name alias for that site, so everything is configured automatically, the user doesn't have to worry about the internal processes, this API tells us if the user is authorized for that domain name or if this domain name is already in use, et cetera, and the same API also provides us from IP addresses. We have to pre-allocate some IP addresses together with the host name, so when the user requests a new site, he can access directly using the domain name without having to wait for a DNS refresh, so what we do is just pre-allocate some addresses, and then when the user gets their site, they can access really using the host name without waiting for a DNS update. We use two IP addresses, we have one address, one host name as well, as a host address, so the communications for the host and another for the service, so if we want to move the service to another machine, we can do it without having to modify the host name or the host of the machine, so we can separate what is the service and what is the host, and we can move the service IP, and you will see this is useful later. Additionally, we have SSHSP records and DNSSEC, anyone knows what SSHSP records stands for? No one, good. This is to forget about that, I'm sure pretty much you all have seen this screen like that, and what it does is you can upload SSHSP record into the DNS with your public host key, and then if you have DNSSEC activated, you don't have to check the host fingerprint manually because the DNS server does it for you, it gives you the fingerprint, and when you check that the fingerprint is the one that is in the DNS server, which is secure communicating with you, and then you don't have to check manually if the fingerprint or the machine you are connecting is the one that it's claiming to be. So that's pretty useful, as I said, we have a lot of services in the same architecture, we have an inventory there, which is using another API, it's based on JSON API, on pool, consume, so we give to this service called best the data of all our services, so we can use it as an external database as well, when we know where all VMs are, we know where they are located, the IP address they have, etc., etc., so it can be used even as an inventory for Ansible, or it can be used for other purposes. So as you saw, we have a lot of APIs, different ways of accessing APIs, we use SSHAPIs, REST, non-REST, HTTPS, using JSON, non-JSON, but we have to deal with a lot of them in an asking way, because we don't want Django, the main thread of Django, to be stopped by that, so what we do is executing as a background process, using a Chrome jobs, which is the easy way if you don't need the API executed after just after the user has launched the petition, or if you want the execution scheduled, you can use salary and credit, which is what we use. Sellery is pretty good for us, because it works pretty well with Django, it's very easy to configure, you just have to add on the top of the function, you just have to declare that it's a shared task, you can use different templates, like this task with failure, you can define the number of retries, you define the template, so you can define if it fails to log something or send you an email, et cetera, et cetera, so it's pretty easy to configure, and it works pretty well, and you can also execute Chrome jobs from the same salary, it's called salary beat, so it's different, but the jobs are just configured as it was a Chrome job, so it's pretty useful for us, it works well, and these architect, these salary, these APIs and these services are supporting all of these, all Ansible driven, so the changes are then in Django, Django stores these changes in the database, and then Ansible is executed, takes these changes from the database, and then it executes these changes on the VMs, so we have this service, we went to the community in the university, we made a workshop, and we said, we have this for you, we thought that you will like it, and they said, hmm, we will like it, but what about if the service fails, and then we thought, well, we have a backup, you can recover for the backup, it won't take too much, we create the VM, et cetera, and then they said, hmm, but I need an SLA if I want to switch to you, but we didn't have an SLA because we had a backup, we have a plan, but we didn't have, we didn't thought about what happens if 300 VMs fails at the same time, which requires a lot of time to recreate and a lot of time to take from backups, so some of the people were saying, hmm, we are thinking to change into MWS3, but only if you provide high availability, so lucky for us, we designed the application so it can cope with different VM architectures, which is good because you don't have to worry about the VM architecture that you are using because you are creating the VM using an API, which maybe this one that we are providing or could be an Amazon EC2 server, and then we execute everything through Ansible, Ansible only needs an SSH connection, so it's pretty easy, so we just need to replace this component, which is the VM architecture. So we thought, okay, let's update VMware with high availability, and then we saw that we need replicated vSphere, replicated the storage, which we didn't have, and replicated the storage for a lot of servers, it's very expensive to maintain because you need to do huge file storage that is shared between all the VMs, so we had a lot of things pending from this architecture, so we thought that's pretty risky for the low time we have better take another architecture because it's still, it's also expensive to acquire all the hardware and software that we need, so we decided, okay, we don't want to maintain a huge share of the file system, so what we do is replicate each one of the VMs file system to another one, so we thought we can use the VMware, still use the VMware infrastructure, we had a pacemaker cursing, which is basically a cluster that checks that all the VMs are in contact with each other and then can change the service network configuration, this is why it's useful to have service network configuration to any of these two production VMs, so we have a replicated VM, the second one, the second column is just a VM that is waiting in something fails to be changed and start acting as a reactive VM of the cluster, and then we replicated the storage individually for each one of the VMs using the RVD, which is basically a driver that sends all the rides of a machine into the other VM, so the storage is replicated to the second one and the pacemaker takes care that if some of the components fail, the switchover is made automatically, so we don't have to worry about it, but then we thought, oh, this is maintaining a lot of clusters, we may end up with one cluster for each one of the VMs that we're going to have, because for each one of the VMs we will have to have a pacemaker cluster, and this is very expensive to contain, and it may fail if we need to execute an C-ball, we need to execute an C-ball in both sides in the two VMs, so they are synchronized, so that's a lot of work, and it's going to break quite easily, so we thought, okay, let's start from scratch. We move away from VMWare, and we decided to use Shen, Zen can be configured very similarly as the VMWare, so you can see there are two Zen servers there, they are also executed, pacemaker and coursing, but the difference is we don't do clustering for each one of the VMs, we do clustering for each one of the Zen servers, the Zen servers have a lot of VMs inside, so if something happens with one of the servers, the whole cluster, the whole server and all the VMs that are inside one of the Zen servers that you can see on the top, all the Zen VMs that have a Zen server are automatically migrated, live migration, to the second one, and you don't notice anything, even the sockets are kept open, you don't notice that the switchover has happened, with the VMWare solutions you will have to wait until we restart the VM, for example, and that kind of stuff, with Zen, you don't notice anything, you just don't see that nothing has happened, so inside you have changed your VM from one Zen server to the other, but it's completely transparent to you, so for doing that it's a bit more complicated, because this is the file system that we had to use, this is a very complex file system where you have all of these, these disks are tied together in a physical volume, and then you have the left one is the DOM zero, which is basically the operating system that is running in the Zen server, and then each one of these are the V, each one of the individual Zen hosts, so you have all the storage replicated to the other Zen server, which provides this live migration, so we had a happy transition, it's working well now, but the architecture is the same, because we designed the architecture, so we could change the VM provided, but just changing the API, the API is a middle API, so we only had to write an API for the Zen server, and then execute everything was exactly the same, so we are happy with that, seems people may be happy with that as well, and we changed it from the VMware solution to the Zen solution, which is three node clusters, and all clusters are in different locations, we can do live migration so the users don't know anything, it's still using Ansible, and we also use Ansible to deploy more clusters, so this is an example of the Zen server clusters, we can deploy many of them, it is easy to deploy because it's Ansible, so if we want to create more Zen server clusters, it's just as easy as create, well, get the machine, the physical machine, start it, and launch it, so it's pretty easy, let's talk one minute about security, because we like security, well, I am not an expert in security, we like to enforce security to our users, so we then end up with problems, we decided to not use root passwords when we create the Zen host, so we don't have to manage root passwords, which is difficult to secure a lot of root passwords in a database, we don't want to manage that way, so we only use keys, we connect to the machines using keys, Ansible connects to the machines using keys, etc. We have a separation of privilege, for example, we need to regenerate the host keys of the Zen host, the host needs to be generated previously, because we need to upload the SSHFP records before even the machine is created, so we need to have a pool of host keys that we can use in the future, and we can install in the machines. We use user service, which provides a useful interface, so users can execute commands from root to users, or other more privileged users based on some filtering and some templating. We also provide a TLS certificate service, this one, an additional one, because we want to follow one of the new initiatives from the EFF and the Mozilla, which is HTTPS everywhere, or let's encrypt, it's an open source CA, which provides you with free certificates for your web page, HTTPS everywhere, EFF trying to force everyone to have HTTPS web servers, and even the HTTP2 specification doesn't include to enforce HTTPS, but a lot of people are saying, when we move to HTTP2, everyone is going to be HTTPS. This sounds really true, but because the specification doesn't say it, but all the implementations from Microsoft, Internet Explorer, Mozilla, and Google, and Chrome only implemented HTTP2 if it uses HTTPS. So we like to test our servers. I encourage you to do the same if you enter to the SSL labs, you can get a qualification of how secure your web server is, which is pretty good, because it gives you some hints of if you have any open back, all the specification, all the version of open SSL, et cetera, et cetera. Then apart from security, we use also some metrics and logging systems, so we can give users some information about how their host works. For example, we use a metrics service, which is basically a statsD and collectsD in each one of the machines installed, also using Ansible. We have a cluster of metrics brokers that get this information from all the hosts, and then we have a cluster of carbon graphite, which is stored, this information gathered from all the machines. So the user can see these graphs in their panel, the web panel, and they can see how the machine is behaving, et cetera, et cetera. And we are now trying to implement LogStash, Elasticsearch, and Kivana, which also provides information about the hosting, the web server, and how it behaves, whether you have the VCs, how it behaves during different periods of time, et cetera. You can have a lot of logs gathered by LogStash stored in Elasticsearch and then showed in a Kivana. So that's pretty much all I want to talk about. I hope you like it. Thank you. Thank you, Abraham, and we have a few minutes for questions. Any questions? Yes? Why did you choose Shen instead of KVM, for instance? What made you choose one thing and not the other? This was a long discussion we had between the developers, which is basically three. We didn't have any strong reason to choose one or the other. We saw doing a little bit of research that Shen worked a little bit better with the IPD, which is one of the main companies we wanted to use because we wanted to do the replication of storage from one server to the other. So we thought that it was integrated inside the same server. We decided to go that way. But we could have chosen KVM. It was in the list of bugs that we had to research and decide. Hi. Thank you for the talk. It was really fun. But to balance, I didn't really understand the subtleties in the last architecture because you had several hard drives and then you have several Zen servers that actually overlap on several drives. So were there virtual servers? Yes. So this is the picture of the VM architecture on the top. This is more the view of the file storage. This is a single machine with a rate of disks. And this is a file storage for a single machine. So you have the physical volume and then you have the first column is where it used to be called DOM zero which is the operating system that manage all the VMs when you access to the Zen server. You access to this. It's also a VM but you access to this and it has direct access to the hardware instead of the VM which have access to the hypervisor. And all the other columns, each one of them are a Zen host which are one of these VMs or DOM view as it's called in Zen. And each one of them has the DRBD device which is basically a virtual block device which is replicated from here to another DRBD server. So you have each one of these DR devices are inside this list of virtual block devices and they are replicated through the network in real time when they are written or read or only written because it's a sync to the secondary Zen server. So it's done automatically through the network when it's executed but the difference is each one of them has a DR device. Okay. Thank you. Okay. We have maybe one minute if anyone has a really quick question. Okay. Please join me in thanking Abraham once again. Thank you.
 Hello everyone, I'm Abraham Martin.  I work for the University of Cambridge, as you may have guessed from the huge logo on  the screen.  I was thinking about making a little introduction and I thought everyone knows what the University  of Cambridge is, so I think I have little to say about that.  And I thought I can show you some pretty pictures so you can invite me where I live  and this preplace and where I work.  This nice architecture, this nice rivers you can pond with your friends during summer,  well the two days of summers we have a year.  The classic math bridge, as you may know, we have a lot of clever people there, some  noble prices, people that usually work around the city dressed like this, which are gongs.  It seems weird but you really see, when you go there, you really see these people working  around academics.  This seems pretty classic, pretty old, but we also have pretty nice new buildings like  this one, which is the new University data center, which is one of the top data centers  in the UK.  It's pretty big, it's green, we have a lot of innovation inside it.  We even have HPC service, which has, in 2003 was the second most green computer in the top  of 500, so it's not just classic building and architecture, we also have some cool things.  This is the computer lab, what I used to work.  I did my relationship with the university, started with the computer lab, I was doing  my PhD and then I worked there as a postdoc.  This building is called William Gates Building because Bill Gates paid for the half of the  building and I work on this building now, which is the university computing service,  where we provide IT services for the rest of the university.  Both buildings share a common history.  They used to be the mathematical laboratory where this machine, brownie points if you  know what it is, it's the exact computer, one of the first computers in the world based  on the von Neumann architecture.  Those built there, we still have some pieces in the computer lab as a museum.  But also other things were built there and in the part that I'm working now, which is  the university computing service, like Exim, which you probably know about because still  50% of mail servers use it.  We have pretty cool people working there, I'm not of any of them, but we have some pretty  cool people that also work in a lot of open source projects.  What I want to explain to you today is one service that was proposed a lot of years ago,  which is the managed web service, and was born to solve a problem.  We have a lot of researchers in the university, as you may know, and a lot of them usually  hold a conference, they do research, they want to do some simple website with the  statistics, show statistics, show results from the research, or either do questionaries,  et cetera, et cetera.  They end up using their own web servers under the desk.  It was a cheap computer, it was running under the desk.  It was not maintained usually because the academics usually use that computer for the  conference and then they left this computer under the desk, the software was not updated,  and then we get security problems.  We get servers hacked, et cetera, et cetera, you know.  The proposal that the AT services and the university did for solving that problem was  centralizing these web services.  The solution was to provide a service where you don't have to worry about maintaining the  OS or the software, you only have to worry about maintaining the web application.  We maintain the OS, we give basic web hosting capabilities like external services does,  you don't have to worry about backups, and you have some dedicated resources to your  web app.  That's very, very old, and when I say gold, it's like 15 years ago.  The first version of the managed web service was using a Solaris 7 running in the Solaris  machine, so you can see that it was using a very old version of Apache, PHP, and it  was using a true system to maintain the separation between these different web pages.  The second version came sooner than the other one, provided the new software, XLRS 10, Apache  2, more new software, and started to use Solaris zones, which is a kind of virtualization  inside the Solaris machine.  It's kind of a container, so we were using containers before it was cool, but it's still  pretty old.  It also had more enhanced features like database, driven script, so you could do a script based  on some information in database, so it's centralized, it's very to manage, some needs,  and NFS server, very classic, set-afes file system, so it provides also snapshots, which  is good, and the users were able to create Vhost, Ali-Assist, et cetera, but the problem  was everything is manual, so they had the users send us an email saying, I want this,  and we make the changes, we execute the scripts, the scripts make the changes, but everything  is manual, so we need a lot of human intervention.  So when it started to grow, we currently now have more than 200 users and more than 400  websites, it started to become a little bit difficult to manage, because it requires  a lot of time.  So before we ended up making a new version of the managed web service, another solution  was in parallel, which is the Falcon service, which is a clone-based, you only get a clone  instance, you don't get access to a server or anything, it's just a CMS as a service,  and we also have like 200 websites there.  So if you go to any university website, you probably will end up in either Falcon service  or a managed web service service, so for example, one of the most visited websites inside the  service is the Stephen Hawkins website.  So we decided to make from scratch a new service, so restart what we have done, because we don't  have more Solaris machines, Solaris machines are routing, they're pretty old, we don't  have a replacement for that, so we also thought, let's do more automation, we can do more automation,  so it requires less time from us.  So we decided to go to the classic dedicated VMs, but still maintain the same things that  was proposed by the previous ones, like no root access for users, and everything is maintained  by us.  I want to say by us, I mean by Ansible, because we don't touch anything, but we will see that  later.  And so to end up these emails that come to our inbox saying, can you please install these  packages, can you please install these, we created a web panel using Django, where we  delegate some power to the users, so the users can do things without having root access or  anything.  So architecture is basically a devian aid machine, we installed the basic packages that  we have been installing up to now, like the Apache, which is the most common feature of  the month, but we also support other Apache available, like if you want to install Python,  Django, et cetera.  We have a list of system packages that you can install, and they are pre-approved, so  you don't end up with a machine with a lot of packages that you don't need or that are  strange to need.  And we give them all the power to do authorization to the sites, create the host, apply for domain  names, install TLS certificates in the machines, do the back-ups from them, password reset,  power management, et cetera, et cetera.  So we give them the power to do a lot of things that we were doing before.  So they have a panel, don't blame me for the design, it's an in-house design, if you visit  any Cambridge website, you will see that all of them look exactly the same.  So it's just a panel with some options to manage your site, so when you create a site  you get this web panel based on Django, you have some options to create Vhost, ask for  domain names, et cetera, and you get an extra VM, which is a test server, so you can clone  your production server to a test server, you can test things there without having to compromise  your production server, so that's good, especially for people that have Drupal installing the  manage web servers when they Drupal, really bad things happen.  So it's very, you can test it before, and if it goes right, you clone it back to the  production server.  So architecture looks like that, we will go one by one, see how we build it.  Be aware that this is not a talk about OpenStack, neither about talker, so don't expect any  of that.  But we use all the, most of them are Python technologies, and we did that project in a  few months, it's still not finished, we are still working on it, but most of it we have  done it by using one and two, 1.2, 1.3 FT, so it's not much people, so the amount of resources  that you require for doing it, although it seems like a huge service, it's not that big.  So we have here the VM architecture, here, the VM is separated, the VM service is separated  from the rest of the stack, so we start describing the VM architecture.  The VM architecture is just a VMware solution, you may know these VMware solutions, it's  just ESXCI servers, and you can manage these XCI servers using VSphere control panel and  some APIs, and we have a standard backup server where we do the backups, but it's not replicated,  so if something happens, then we rebuild the VM and recover the things from the backup  server.  So the flow is easy, a user enters to the Django web panel, authenticates, so we know  who he is, and then he asks for a new managed web server.  A host name and an IP64 are allocated to this site, the VM API creates a new VM, the VM  API installs the OS, and when the OS is ready, Ansible is executed, and Ansible is the one  that configures the whole machine, so we are using Ansible as a configuration management  and it does everything we need.  So there's no Ansible, it's just a bunch of things together, they are easy to script,  so it's very easy to understand what they are doing, they separate into folders, which  is very good, and you can find the file that you are looking for, and there is separation  of things in the different files that you can see.  So it's pretty good to use, it also has inventory, so you can define all your servers based on  dynamically or static, so you can have a file with all your servers, or you can inject  the output from another API as a list of servers you have, or even the database, et cetera.  So it's pretty nice, it works really well, and it based on playbooks, playbooks is just  a bunch of roles linked to a bunch of targets, so you have a role and definition of role  is things you want to install in this role, in these machines that have this role, and  then you have targets, and then you say this target, these machines, I want to install this  role, for example, a web server, web server can be a role, and the web server role has  a lot of tasks, then install Apache, configure Apache, et cetera, et cetera.  This is an Ansible playbook, as I said before, you define the host where you want to install  things, and then you define the roles, does these machines in this list will have.  For each role you have tasks, templates, which are changed to templates, scripts, handles,  and variables, you can have also global variables or variables entering into the script, and  this is how a role looks like, just a bunch of tasks inside the role, you can see that  here we're installing packages, it's a Jamel file, as you can see, and it's pretty easy  to understand what you are doing, so if you're working with more people, it's easy to modify  the file, change other configuration, et cetera.  You can see here that the templates can be used with variables, are Ginge to templates,  we use variables there, and therefore we use the same templates for all the configurations,  all machines.  You also have Handlers, which are basically callbacks, when some function in Ansible is  executed, then you have a callback later, and you can, for example, if you have updated  the Apache configuration or your Django app, you can restart Apache, and the callback is  cool.  So this is from the VM part, we use this VM for infrastructure, we use the APIs, we  launch it, we create the VM, everything is good, after that, Ansible configures the machine,  and then we can offer the service to the user.  So if we start from the top of the stack, we see authentication part, we have our own  authentication, we use Raven, Raven is our authentication service, so you can see that  we have a lot of services interconnected using a lot of APIs.  It is based on a web of API, and we have to build a custom Django web again, but this  could be substituted by any authentication that you could use.  You can use the Django one if you want, you can link with your own enterprise if you want,  et cetera.  So the second layer is authorization, we have a kind of a Nellabish service, it is called  lookup, and then what we have there is just a list of users and a list of groups, we can  see these users if they belong to each, which institutions they belong to, which groups they  belong to, so the end user can configure their MW server based on this list, they can search  for other users, authorize them as administrators, authorize by groups, et cetera.  So it is just a basic list.  We use that instead of using the Django groups, because it is more useful for us, because  people use in the university to this L-Lab service that we have, so they create the groups  there and they are automatically updated if someone leaves, et cetera.  So, when the user has authorize the user, another user to enter the machine or use the  service, we need to still install the user into the machine.  So we have another service over there called Django, which provides more information about  users, it is like user identity management.  So we get from Django a unique UID from the user, we need that, because if we install the  same user in different machines, we still need to identify the files that belong to him  or to her in different machines, so they have the same unique UID and we use this unique  UID in all the user installs.  Users install using Ansible as well, it is installed on all the VMs where it is authorized,  and we have periodic refreshes to refresh the lookup groups we have authorized, so if the  groups change, people different change, and we allow people to upload their own SSH keys  and the SSH keys is also installed in the user configuration, so they can enter either  using the password, which is checked with this L-Lab server or either the SSH that they have  installed in the panel.  So once they can access to the Django panel and they have the user installed in the VM,  they can already access the machine, everything is configured for them, so they can start  using it.  For previously, we had also another communication with the IP register API, which is on the  bottom there.  This allows us, this is another external service, so as you can see, we have the main service  and then we have a lot of other services that we communicate with, which provides the university  registration for KAMACAC domains, so if you want to register a new KAMACAC domain, for  example, important studies.KAMACAC, you can launch an API, we launch an API request using  the same Django panel, Django panel sends the request, and then we get the main name alias  for that site, so everything is configured automatically, the user doesn't have to worry  about the internal processes, this API tells us if the user is authorized for that domain  name or if this domain name is already in use, et cetera, and the same API also provides  us from IP addresses.  We have to pre-allocate some IP addresses together with the host name, so when the user  requests a new site, he can access directly using the domain name without having to wait  for a DNS refresh, so what we do is just pre-allocate some addresses, and then when the user gets  their site, they can access really using the host name without waiting for a DNS update.  We use two IP addresses, we have one address, one host name as well, as a host address,  so the communications for the host and another for the service, so if we want to move the  service to another machine, we can do it without having to modify the host name or the host  of the machine, so we can separate what is the service and what is the host, and we can  move the service IP, and you will see this is useful later.  Additionally, we have SSHSP records and DNSSEC, anyone knows what SSHSP records stands for?  No one, good.  This is to forget about that, I'm sure pretty much you all have seen this screen like that,  and what it does is you can upload SSHSP record into the DNS with your public host key, and  then if you have DNSSEC activated, you don't have to check the host fingerprint manually  because the DNS server does it for you, it gives you the fingerprint, and when you check  that the fingerprint is the one that is in the DNS server, which is secure communicating  with you, and then you don't have to check manually if the fingerprint or the machine  you are connecting is the one that it's claiming to be.  So that's pretty useful, as I said, we have a lot of services in the same architecture,  we have an inventory there, which is using another API, it's based on JSON API, on pool,  consume, so we give to this service called best the data of all our services, so we can  use it as an external database as well, when we know where all VMs are, we know where they  are located, the IP address they have, etc., etc., so it can be used even as an inventory  for Ansible, or it can be used for other purposes.  So as you saw, we have a lot of APIs, different ways of accessing APIs, we use SSHAPIs, REST,  non-REST, HTTPS, using JSON, non-JSON, but we have to deal with a lot of them in an  asking way, because we don't want Django, the main thread of Django, to be stopped by  that, so what we do is executing as a background process, using a Chrome jobs, which is the  easy way if you don't need the API executed after just after the user has launched the  petition, or if you want the execution scheduled, you can use salary and credit, which is what  we use.  Sellery is pretty good for us, because it works pretty well with Django, it's very easy  to configure, you just have to add on the top of the function, you just have to declare  that it's a shared task, you can use different templates, like this task with failure, you  can define the number of retries, you define the template, so you can define if it fails  to log something or send you an email, et cetera, et cetera, so it's pretty easy to configure,  and it works pretty well, and you can also execute Chrome jobs from the same salary,  it's called salary beat, so it's different, but the jobs are just configured as it was  a Chrome job, so it's pretty useful for us, it works well, and these architect, these  salary, these APIs and these services are supporting all of these, all Ansible driven,  so the changes are then in Django, Django stores these changes in the database, and  then Ansible is executed, takes these changes from the database, and then it executes these  changes on the VMs, so we have this service, we went to the community in the university,  we made a workshop, and we said, we have this for you, we thought that you will like it,  and they said, hmm, we will like it, but what about if the service fails, and then we thought,  well, we have a backup, you can recover for the backup, it won't take too much, we create  the VM, et cetera, and then they said, hmm, but I need an SLA if I want to switch to you,  but we didn't have an SLA because we had a backup, we have a plan, but we didn't have,  we didn't thought about what happens if 300 VMs fails at the same time, which requires  a lot of time to recreate and a lot of time to take from backups, so some of the people  were saying, hmm, we are thinking to change into MWS3, but only if you provide high availability,  so lucky for us, we designed the application so it can cope with different VM architectures,  which is good because you don't have to worry about the VM architecture that you are using  because you are creating the VM using an API, which maybe this one that we are providing  or could be an Amazon EC2 server, and then we execute everything through Ansible, Ansible  only needs an SSH connection, so it's pretty easy, so we just need to replace this component,  which is the VM architecture.  So we thought, okay, let's update VMware with high availability, and then we saw that we  need replicated vSphere, replicated the storage, which we didn't have, and replicated the storage  for a lot of servers, it's very expensive to maintain because you need to do huge file  storage that is shared between all the VMs, so we had a lot of things pending from this  architecture, so we thought that's pretty risky for the low time we have better take  another architecture because it's still, it's also expensive to acquire all the hardware  and software that we need, so we decided, okay, we don't want to maintain a huge share  of the file system, so what we do is replicate each one of the VMs file system to another  one, so we thought we can use the VMware, still use the VMware infrastructure, we had  a pacemaker cursing, which is basically a cluster that checks that all the VMs are in  contact with each other and then can change the service network configuration, this is  why it's useful to have service network configuration to any of these two production VMs, so we  have a replicated VM, the second one, the second column is just a VM that is waiting  in something fails to be changed and start acting as a reactive VM of the cluster, and  then we replicated the storage individually for each one of the VMs using the RVD, which  is basically a driver that sends all the rides of a machine into the other VM, so the storage  is replicated to the second one and the pacemaker takes care that if some of the components fail,  the switchover is made automatically, so we don't have to worry about it, but then we  thought, oh, this is maintaining a lot of clusters, we may end up with one cluster for  each one of the VMs that we're going to have, because for each one of the VMs we will have  to have a pacemaker cluster, and this is very expensive to contain, and it may fail if we  need to execute an C-ball, we need to execute an C-ball in both sides in the two VMs, so  they are synchronized, so that's a lot of work, and it's going to break quite easily,  so we thought, okay, let's start from scratch.  We move away from VMWare, and we decided to use Shen, Zen can be configured very similarly  as the VMWare, so you can see there are two Zen servers there, they are also executed,  pacemaker and coursing, but the difference is we don't do clustering for each one of  the VMs, we do clustering for each one of the Zen servers, the Zen servers have a lot  of VMs inside, so if something happens with one of the servers, the whole cluster, the  whole server and all the VMs that are inside one of the Zen servers that you can see on  the top, all the Zen VMs that have a Zen server are automatically migrated, live migration,  to the second one, and you don't notice anything, even the sockets are kept open, you don't  notice that the switchover has happened, with the VMWare solutions you will have to wait  until we restart the VM, for example, and that kind of stuff, with Zen, you don't notice  anything, you just don't see that nothing has happened, so inside you have changed your  VM from one Zen server to the other, but it's completely transparent to you, so for doing  that it's a bit more complicated, because this is the file system that we had to use,  this is a very complex file system where you have all of these, these disks are tied together  in a physical volume, and then you have the left one is the DOM zero, which is basically  the operating system that is running in the Zen server, and then each one of these are  the V, each one of the individual Zen hosts, so you have all the storage replicated to  the other Zen server, which provides this live migration, so we had a happy transition,  it's working well now, but the architecture is the same, because we designed the architecture,  so we could change the VM provided, but just changing the API, the API is a middle API,  so we only had to write an API for the Zen server, and then execute everything was exactly  the same, so we are happy with that, seems people may be happy with that as well, and  we changed it from the VMware solution to the Zen solution, which is three node clusters,  and all clusters are in different locations, we can do live migration so the users don't  know anything, it's still using Ansible, and we also use Ansible to deploy more clusters,  so this is an example of the Zen server clusters, we can deploy many of them, it is easy to  deploy because it's Ansible, so if we want to create more Zen server clusters, it's just  as easy as create, well, get the machine, the physical machine, start it, and launch  it, so it's pretty easy, let's talk one minute about security, because we like security,  well, I am not an expert in security, we like to enforce security to our users, so we then  end up with problems, we decided to not use root passwords when we create the Zen host,  so we don't have to manage root passwords, which is difficult to secure a lot of root  passwords in a database, we don't want to manage that way, so we only use keys, we connect  to the machines using keys, Ansible connects to the machines using keys, etc.  We have a separation of privilege, for example, we need to regenerate the host keys of the  Zen host, the host needs to be generated previously, because we need to upload the SSHFP records  before even the machine is created, so we need to have a pool of host keys that we can use  in the future, and we can install in the machines.  We use user service, which provides a useful interface, so users can execute commands from  root to users, or other more privileged users based on some filtering and some templating.  We also provide a TLS certificate service, this one, an additional one, because we want  to follow one of the new initiatives from the EFF and the Mozilla, which is HTTPS everywhere,  or let's encrypt, it's an open source CA, which provides you with free certificates  for your web page, HTTPS everywhere, EFF trying to force everyone to have HTTPS web servers,  and even the HTTP2 specification doesn't include to enforce HTTPS, but a lot of people are  saying, when we move to HTTP2, everyone is going to be HTTPS.  This sounds really true, but because the specification doesn't say it, but all the implementations  from Microsoft, Internet Explorer, Mozilla, and Google, and Chrome only implemented HTTP2  if it uses HTTPS.  So we like to test our servers.  I encourage you to do the same if you enter to the SSL labs, you can get a qualification  of how secure your web server is, which is pretty good, because it gives you some hints  of if you have any open back, all the specification, all the version of open SSL, et cetera, et  cetera.  Then apart from security, we use also some metrics and logging systems, so we can give  users some information about how their host works.  For example, we use a metrics service, which is basically a statsD and collectsD in each  one of the machines installed, also using Ansible.  We have a cluster of metrics brokers that get this information from all the hosts, and then  we have a cluster of carbon graphite, which is stored, this information gathered from  all the machines.  So the user can see these graphs in their panel, the web panel, and they can see how  the machine is behaving, et cetera, et cetera.  And we are now trying to implement LogStash, Elasticsearch, and Kivana, which also provides  information about the hosting, the web server, and how it behaves, whether you have the  VCs, how it behaves during different periods of time, et cetera.  You can have a lot of logs gathered by LogStash stored in Elasticsearch and then showed in  a Kivana.  So that's pretty much all I want to talk about.  I hope you like it.  Thank you.  Thank you, Abraham, and we have a few minutes for questions.  Any questions?  Yes?  Why did you choose Shen instead of KVM, for instance?  What made you choose one thing and not the other?  This was a long discussion we had between the developers, which is basically three.  We didn't have any strong reason to choose one or the other.  We saw doing a little bit of research that Shen worked a little bit better with the IPD,  which is one of the main companies we wanted to use because we wanted to do the replication  of storage from one server to the other.  So we thought that it was integrated inside the same server.  We decided to go that way.  But we could have chosen KVM.  It was in the list of bugs that we had to research and decide.  Hi.  Thank you for the talk.  It was really fun.  But to balance, I didn't really understand the subtleties in the last architecture because  you had several hard drives and then you have several Zen servers that actually overlap  on several drives.  So were there virtual servers?  Yes.  So this is the picture of the VM architecture on the top.  This is more the view of the file storage.  This is a single machine with a rate of disks.  And this is a file storage for a single machine.  So you have the physical volume and then you have the first column is where it used to  be called DOM zero which is the operating system that manage all the VMs when you access  to the Zen server.  You access to this.  It's also a VM but you access to this and it has direct access to the hardware instead  of the VM which have access to the hypervisor.  And all the other columns, each one of them are a Zen host which are one of these VMs  or DOM view as it's called in Zen.  And each one of them has the DRBD device which is basically a virtual block device which  is replicated from here to another DRBD server.  So you have each one of these DR devices are inside this list of virtual block devices  and they are replicated through the network in real time when they are written or read  or only written because it's a sync to the secondary Zen server.  So it's done automatically through the network when it's executed but the difference is each  one of them has a DR device.  Okay.  Thank you.  Okay.  We have maybe one minute if anyone has a really quick question.  Okay.  Please join me in thanking Abraham once again.  Thank you.
Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_0_635.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_636_1055.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_1056_1367.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_1368_1907.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_1908_2135.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_2136_2387.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_2388_3635.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_3636_4859.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_4860_4979.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_4980_7247.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_7248_8711.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_8712_9815.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_9816_14255.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_14256_15755.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_15756_17111.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_17112_18803.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_18804_22943.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_22944_23531.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_23532_24251.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_24252_33023.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_33024_35483.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_35484_35615.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_35616_36095.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_36096_36863.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_36864_36959.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_36960_37067.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_37068_37175.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_37176_37295.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_37296_37679.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_37680_37967.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_37968_38243.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_38244_38387.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_38388_38663.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_38664_39695.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_39696_46415.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_46416_46895.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_46896_48935.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_48936_49655.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_49656_51119.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_51120_51287.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_51288_52067.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_52068_52283.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_52284_53039.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_53040_53159.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_53160_53507.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_53508_53615.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_53616_55283.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_55284_55799.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_55800_55847.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_55848_55931.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_55932_56051.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_56052_56363.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_56364_56531.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_56532_56795.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_56796_57011.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_57012_57431.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_57432_57551.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_57552_57695.png Architecture_of_a_cloud_hosting_service_using_python_technologies__django__ansible_and_celery_57696_58895.png
Hello everyone, I'm Abraham Martin. I work for the University of Cambridge, as you may have guessed from the huge logo on the screen. I was thinking about making a little introduction and I thought everyone knows what the University of Cambridge is, so I think I have little to say about that. And I thought I can show you some pretty pictures so you can invite me where I live and this preplace and where I work. This nice architecture, this nice rivers you can pond with your friends during summer, well the two days of summers we have a year. The classic math bridge, as you may know, we have a lot of clever people there, some noble prices, people that usually work around the city dressed like this, which are gongs. It seems weird but you really see, when you go there, you really see these people working around academics. This seems pretty classic, pretty old, but we also have pretty nice new buildings like this one, which is the new University data center, which is one of the top data centers in the UK. It's pretty big, it's green, we have a lot of innovation inside it. We even have HPC service, which has, in 2003 was the second most green computer in the top of 500, so it's not just classic building and architecture, we also have some cool things. This is the computer lab, what I used to work. I did my relationship with the university, started with the computer lab, I was doing my PhD and then I worked there as a postdoc. This building is called William Gates Building because Bill Gates paid for the half of the building and I work on this building now, which is the university computing service, where we provide IT services for the rest of the university. Both buildings share a common history. They used to be the mathematical laboratory where this machine, brownie points if you know what it is, it's the exact computer, one of the first computers in the world based on the von Neumann architecture. Those built there, we still have some pieces in the computer lab as a museum. But also other things were built there and in the part that I'm working now, which is the university computing service, like Exim, which you probably know about because still 50% of mail servers use it. We have pretty cool people working there, I'm not of any of them, but we have some pretty cool people that also work in a lot of open source projects. What I want to explain to you today is one service that was proposed a lot of years ago, which is the managed web service, and was born to solve a problem. We have a lot of researchers in the university, as you may know, and a lot of them usually hold a conference, they do research, they want to do some simple website with the statistics, show statistics, show results from the research, or either do questionaries, et cetera, et cetera. They end up using their own web servers under the desk. It was a cheap computer, it was running under the desk. It was not maintained usually because the academics usually use that computer for the conference and then they left this computer under the desk, the software was not updated, and then we get security problems. We get servers hacked, et cetera, et cetera, you know. The proposal that the AT services and the university did for solving that problem was centralizing these web services. The solution was to provide a service where you don't have to worry about maintaining the OS or the software, you only have to worry about maintaining the web application. We maintain the OS, we give basic web hosting capabilities like external services does, you don't have to worry about backups, and you have some dedicated resources to your web app. That's very, very old, and when I say gold, it's like 15 years ago. The first version of the managed web service was using a Solaris 7 running in the Solaris machine, so you can see that it was using a very old version of Apache, PHP, and it was using a true system to maintain the separation between these different web pages. The second version came sooner than the other one, provided the new software, XLRS 10, Apache 2, more new software, and started to use Solaris zones, which is a kind of virtualization inside the Solaris machine. It's kind of a container, so we were using containers before it was cool, but it's still pretty old. It also had more enhanced features like database, driven script, so you could do a script based on some information in database, so it's centralized, it's very to manage, some needs, and NFS server, very classic, set-afes file system, so it provides also snapshots, which is good, and the users were able to create Vhost, Ali-Assist, et cetera, but the problem was everything is manual, so they had the users send us an email saying, I want this, and we make the changes, we execute the scripts, the scripts make the changes, but everything is manual, so we need a lot of human intervention. So when it started to grow, we currently now have more than 200 users and more than 400 websites, it started to become a little bit difficult to manage, because it requires a lot of time. So before we ended up making a new version of the managed web service, another solution was in parallel, which is the Falcon service, which is a clone-based, you only get a clone instance, you don't get access to a server or anything, it's just a CMS as a service, and we also have like 200 websites there. So if you go to any university website, you probably will end up in either Falcon service or a managed web service service, so for example, one of the most visited websites inside the service is the Stephen Hawkins website. So we decided to make from scratch a new service, so restart what we have done, because we don't have more Solaris machines, Solaris machines are routing, they're pretty old, we don't have a replacement for that, so we also thought, let's do more automation, we can do more automation, so it requires less time from us. So we decided to go to the classic dedicated VMs, but still maintain the same things that was proposed by the previous ones, like no root access for users, and everything is maintained by us. I want to say by us, I mean by Ansible, because we don't touch anything, but we will see that later. And so to end up these emails that come to our inbox saying, can you please install these packages, can you please install these, we created a web panel using Django, where we delegate some power to the users, so the users can do things without having root access or anything. So architecture is basically a devian aid machine, we installed the basic packages that we have been installing up to now, like the Apache, which is the most common feature of the month, but we also support other Apache available, like if you want to install Python, Django, et cetera. We have a list of system packages that you can install, and they are pre-approved, so you don't end up with a machine with a lot of packages that you don't need or that are strange to need. And we give them all the power to do authorization to the sites, create the host, apply for domain names, install TLS certificates in the machines, do the back-ups from them, password reset, power management, et cetera, et cetera. So we give them the power to do a lot of things that we were doing before. So they have a panel, don't blame me for the design, it's an in-house design, if you visit any Cambridge website, you will see that all of them look exactly the same. So it's just a panel with some options to manage your site, so when you create a site you get this web panel based on Django, you have some options to create Vhost, ask for domain names, et cetera, and you get an extra VM, which is a test server, so you can clone your production server to a test server, you can test things there without having to compromise your production server, so that's good, especially for people that have Drupal installing the manage web servers when they Drupal, really bad things happen. So it's very, you can test it before, and if it goes right, you clone it back to the production server. So architecture looks like that, we will go one by one, see how we build it. Be aware that this is not a talk about OpenStack, neither about talker, so don't expect any of that. But we use all the, most of them are Python technologies, and we did that project in a few months, it's still not finished, we are still working on it, but most of it we have done it by using one and two, 1.2, 1.3 FT, so it's not much people, so the amount of resources that you require for doing it, although it seems like a huge service, it's not that big. So we have here the VM architecture, here, the VM is separated, the VM service is separated from the rest of the stack, so we start describing the VM architecture. The VM architecture is just a VMware solution, you may know these VMware solutions, it's just ESXCI servers, and you can manage these XCI servers using VSphere control panel and some APIs, and we have a standard backup server where we do the backups, but it's not replicated, so if something happens, then we rebuild the VM and recover the things from the backup server. So the flow is easy, a user enters to the Django web panel, authenticates, so we know who he is, and then he asks for a new managed web server. A host name and an IP64 are allocated to this site, the VM API creates a new VM, the VM API installs the OS, and when the OS is ready, Ansible is executed, and Ansible is the one that configures the whole machine, so we are using Ansible as a configuration management and it does everything we need. So there's no Ansible, it's just a bunch of things together, they are easy to script, so it's very easy to understand what they are doing, they separate into folders, which is very good, and you can find the file that you are looking for, and there is separation of things in the different files that you can see. So it's pretty good to use, it also has inventory, so you can define all your servers based on dynamically or static, so you can have a file with all your servers, or you can inject the output from another API as a list of servers you have, or even the database, et cetera. So it's pretty nice, it works really well, and it based on playbooks, playbooks is just a bunch of roles linked to a bunch of targets, so you have a role and definition of role is things you want to install in this role, in these machines that have this role, and then you have targets, and then you say this target, these machines, I want to install this role, for example, a web server, web server can be a role, and the web server role has a lot of tasks, then install Apache, configure Apache, et cetera, et cetera. This is an Ansible playbook, as I said before, you define the host where you want to install things, and then you define the roles, does these machines in this list will have. For each role you have tasks, templates, which are changed to templates, scripts, handles, and variables, you can have also global variables or variables entering into the script, and this is how a role looks like, just a bunch of tasks inside the role, you can see that here we're installing packages, it's a Jamel file, as you can see, and it's pretty easy to understand what you are doing, so if you're working with more people, it's easy to modify the file, change other configuration, et cetera. You can see here that the templates can be used with variables, are Ginge to templates, we use variables there, and therefore we use the same templates for all the configurations, all machines. You also have Handlers, which are basically callbacks, when some function in Ansible is executed, then you have a callback later, and you can, for example, if you have updated the Apache configuration or your Django app, you can restart Apache, and the callback is cool. So this is from the VM part, we use this VM for infrastructure, we use the APIs, we launch it, we create the VM, everything is good, after that, Ansible configures the machine, and then we can offer the service to the user. So if we start from the top of the stack, we see authentication part, we have our own authentication, we use Raven, Raven is our authentication service, so you can see that we have a lot of services interconnected using a lot of APIs. It is based on a web of API, and we have to build a custom Django web again, but this could be substituted by any authentication that you could use. You can use the Django one if you want, you can link with your own enterprise if you want, et cetera. So the second layer is authorization, we have a kind of a Nellabish service, it is called lookup, and then what we have there is just a list of users and a list of groups, we can see these users if they belong to each, which institutions they belong to, which groups they belong to, so the end user can configure their MW server based on this list, they can search for other users, authorize them as administrators, authorize by groups, et cetera. So it is just a basic list. We use that instead of using the Django groups, because it is more useful for us, because people use in the university to this L-Lab service that we have, so they create the groups there and they are automatically updated if someone leaves, et cetera. So, when the user has authorize the user, another user to enter the machine or use the service, we need to still install the user into the machine. So we have another service over there called Django, which provides more information about users, it is like user identity management. So we get from Django a unique UID from the user, we need that, because if we install the same user in different machines, we still need to identify the files that belong to him or to her in different machines, so they have the same unique UID and we use this unique UID in all the user installs. Users install using Ansible as well, it is installed on all the VMs where it is authorized, and we have periodic refreshes to refresh the lookup groups we have authorized, so if the groups change, people different change, and we allow people to upload their own SSH keys and the SSH keys is also installed in the user configuration, so they can enter either using the password, which is checked with this L-Lab server or either the SSH that they have installed in the panel. So once they can access to the Django panel and they have the user installed in the VM, they can already access the machine, everything is configured for them, so they can start using it. For previously, we had also another communication with the IP register API, which is on the bottom there. This allows us, this is another external service, so as you can see, we have the main service and then we have a lot of other services that we communicate with, which provides the university registration for KAMACAC domains, so if you want to register a new KAMACAC domain, for example, important studies.KAMACAC, you can launch an API, we launch an API request using the same Django panel, Django panel sends the request, and then we get the main name alias for that site, so everything is configured automatically, the user doesn't have to worry about the internal processes, this API tells us if the user is authorized for that domain name or if this domain name is already in use, et cetera, and the same API also provides us from IP addresses. We have to pre-allocate some IP addresses together with the host name, so when the user requests a new site, he can access directly using the domain name without having to wait for a DNS refresh, so what we do is just pre-allocate some addresses, and then when the user gets their site, they can access really using the host name without waiting for a DNS update. We use two IP addresses, we have one address, one host name as well, as a host address, so the communications for the host and another for the service, so if we want to move the service to another machine, we can do it without having to modify the host name or the host of the machine, so we can separate what is the service and what is the host, and we can move the service IP, and you will see this is useful later. Additionally, we have SSHSP records and DNSSEC, anyone knows what SSHSP records stands for? No one, good. This is to forget about that, I'm sure pretty much you all have seen this screen like that, and what it does is you can upload SSHSP record into the DNS with your public host key, and then if you have DNSSEC activated, you don't have to check the host fingerprint manually because the DNS server does it for you, it gives you the fingerprint, and when you check that the fingerprint is the one that is in the DNS server, which is secure communicating with you, and then you don't have to check manually if the fingerprint or the machine you are connecting is the one that it's claiming to be. So that's pretty useful, as I said, we have a lot of services in the same architecture, we have an inventory there, which is using another API, it's based on JSON API, on pool, consume, so we give to this service called best the data of all our services, so we can use it as an external database as well, when we know where all VMs are, we know where they are located, the IP address they have, etc., etc., so it can be used even as an inventory for Ansible, or it can be used for other purposes. So as you saw, we have a lot of APIs, different ways of accessing APIs, we use SSHAPIs, REST, non-REST, HTTPS, using JSON, non-JSON, but we have to deal with a lot of them in an asking way, because we don't want Django, the main thread of Django, to be stopped by that, so what we do is executing as a background process, using a Chrome jobs, which is the easy way if you don't need the API executed after just after the user has launched the petition, or if you want the execution scheduled, you can use salary and credit, which is what we use. Sellery is pretty good for us, because it works pretty well with Django, it's very easy to configure, you just have to add on the top of the function, you just have to declare that it's a shared task, you can use different templates, like this task with failure, you can define the number of retries, you define the template, so you can define if it fails to log something or send you an email, et cetera, et cetera, so it's pretty easy to configure, and it works pretty well, and you can also execute Chrome jobs from the same salary, it's called salary beat, so it's different, but the jobs are just configured as it was a Chrome job, so it's pretty useful for us, it works well, and these architect, these salary, these APIs and these services are supporting all of these, all Ansible driven, so the changes are then in Django, Django stores these changes in the database, and then Ansible is executed, takes these changes from the database, and then it executes these changes on the VMs, so we have this service, we went to the community in the university, we made a workshop, and we said, we have this for you, we thought that you will like it, and they said, hmm, we will like it, but what about if the service fails, and then we thought, well, we have a backup, you can recover for the backup, it won't take too much, we create the VM, et cetera, and then they said, hmm, but I need an SLA if I want to switch to you, but we didn't have an SLA because we had a backup, we have a plan, but we didn't have, we didn't thought about what happens if 300 VMs fails at the same time, which requires a lot of time to recreate and a lot of time to take from backups, so some of the people were saying, hmm, we are thinking to change into MWS3, but only if you provide high availability, so lucky for us, we designed the application so it can cope with different VM architectures, which is good because you don't have to worry about the VM architecture that you are using because you are creating the VM using an API, which maybe this one that we are providing or could be an Amazon EC2 server, and then we execute everything through Ansible, Ansible only needs an SSH connection, so it's pretty easy, so we just need to replace this component, which is the VM architecture. So we thought, okay, let's update VMware with high availability, and then we saw that we need replicated vSphere, replicated the storage, which we didn't have, and replicated the storage for a lot of servers, it's very expensive to maintain because you need to do huge file storage that is shared between all the VMs, so we had a lot of things pending from this architecture, so we thought that's pretty risky for the low time we have better take another architecture because it's still, it's also expensive to acquire all the hardware and software that we need, so we decided, okay, we don't want to maintain a huge share of the file system, so what we do is replicate each one of the VMs file system to another one, so we thought we can use the VMware, still use the VMware infrastructure, we had a pacemaker cursing, which is basically a cluster that checks that all the VMs are in contact with each other and then can change the service network configuration, this is why it's useful to have service network configuration to any of these two production VMs, so we have a replicated VM, the second one, the second column is just a VM that is waiting in something fails to be changed and start acting as a reactive VM of the cluster, and then we replicated the storage individually for each one of the VMs using the RVD, which is basically a driver that sends all the rides of a machine into the other VM, so the storage is replicated to the second one and the pacemaker takes care that if some of the components fail, the switchover is made automatically, so we don't have to worry about it, but then we thought, oh, this is maintaining a lot of clusters, we may end up with one cluster for each one of the VMs that we're going to have, because for each one of the VMs we will have to have a pacemaker cluster, and this is very expensive to contain, and it may fail if we need to execute an C-ball, we need to execute an C-ball in both sides in the two VMs, so they are synchronized, so that's a lot of work, and it's going to break quite easily, so we thought, okay, let's start from scratch. We move away from VMWare, and we decided to use Shen, Zen can be configured very similarly as the VMWare, so you can see there are two Zen servers there, they are also executed, pacemaker and coursing, but the difference is we don't do clustering for each one of the VMs, we do clustering for each one of the Zen servers, the Zen servers have a lot of VMs inside, so if something happens with one of the servers, the whole cluster, the whole server and all the VMs that are inside one of the Zen servers that you can see on the top, all the Zen VMs that have a Zen server are automatically migrated, live migration, to the second one, and you don't notice anything, even the sockets are kept open, you don't notice that the switchover has happened, with the VMWare solutions you will have to wait until we restart the VM, for example, and that kind of stuff, with Zen, you don't notice anything, you just don't see that nothing has happened, so inside you have changed your VM from one Zen server to the other, but it's completely transparent to you, so for doing that it's a bit more complicated, because this is the file system that we had to use, this is a very complex file system where you have all of these, these disks are tied together in a physical volume, and then you have the left one is the DOM zero, which is basically the operating system that is running in the Zen server, and then each one of these are the V, each one of the individual Zen hosts, so you have all the storage replicated to the other Zen server, which provides this live migration, so we had a happy transition, it's working well now, but the architecture is the same, because we designed the architecture, so we could change the VM provided, but just changing the API, the API is a middle API, so we only had to write an API for the Zen server, and then execute everything was exactly the same, so we are happy with that, seems people may be happy with that as well, and we changed it from the VMware solution to the Zen solution, which is three node clusters, and all clusters are in different locations, we can do live migration so the users don't know anything, it's still using Ansible, and we also use Ansible to deploy more clusters, so this is an example of the Zen server clusters, we can deploy many of them, it is easy to deploy because it's Ansible, so if we want to create more Zen server clusters, it's just as easy as create, well, get the machine, the physical machine, start it, and launch it, so it's pretty easy, let's talk one minute about security, because we like security, well, I am not an expert in security, we like to enforce security to our users, so we then end up with problems, we decided to not use root passwords when we create the Zen host, so we don't have to manage root passwords, which is difficult to secure a lot of root passwords in a database, we don't want to manage that way, so we only use keys, we connect to the machines using keys, Ansible connects to the machines using keys, etc. We have a separation of privilege, for example, we need to regenerate the host keys of the Zen host, the host needs to be generated previously, because we need to upload the SSHFP records before even the machine is created, so we need to have a pool of host keys that we can use in the future, and we can install in the machines. We use user service, which provides a useful interface, so users can execute commands from root to users, or other more privileged users based on some filtering and some templating. We also provide a TLS certificate service, this one, an additional one, because we want to follow one of the new initiatives from the EFF and the Mozilla, which is HTTPS everywhere, or let's encrypt, it's an open source CA, which provides you with free certificates for your web page, HTTPS everywhere, EFF trying to force everyone to have HTTPS web servers, and even the HTTP2 specification doesn't include to enforce HTTPS, but a lot of people are saying, when we move to HTTP2, everyone is going to be HTTPS. This sounds really true, but because the specification doesn't say it, but all the implementations from Microsoft, Internet Explorer, Mozilla, and Google, and Chrome only implemented HTTP2 if it uses HTTPS. So we like to test our servers. I encourage you to do the same if you enter to the SSL labs, you can get a qualification of how secure your web server is, which is pretty good, because it gives you some hints of if you have any open back, all the specification, all the version of open SSL, et cetera, et cetera. Then apart from security, we use also some metrics and logging systems, so we can give users some information about how their host works. For example, we use a metrics service, which is basically a statsD and collectsD in each one of the machines installed, also using Ansible. We have a cluster of metrics brokers that get this information from all the hosts, and then we have a cluster of carbon graphite, which is stored, this information gathered from all the machines. So the user can see these graphs in their panel, the web panel, and they can see how the machine is behaving, et cetera, et cetera. And we are now trying to implement LogStash, Elasticsearch, and Kivana, which also provides information about the hosting, the web server, and how it behaves, whether you have the VCs, how it behaves during different periods of time, et cetera. You can have a lot of logs gathered by LogStash stored in Elasticsearch and then showed in a Kivana. So that's pretty much all I want to talk about. I hope you like it. Thank you. Thank you, Abraham, and we have a few minutes for questions. Any questions? Yes? Why did you choose Shen instead of KVM, for instance? What made you choose one thing and not the other? This was a long discussion we had between the developers, which is basically three. We didn't have any strong reason to choose one or the other. We saw doing a little bit of research that Shen worked a little bit better with the IPD, which is one of the main companies we wanted to use because we wanted to do the replication of storage from one server to the other. So we thought that it was integrated inside the same server. We decided to go that way. But we could have chosen KVM. It was in the list of bugs that we had to research and decide. Hi. Thank you for the talk. It was really fun. But to balance, I didn't really understand the subtleties in the last architecture because you had several hard drives and then you have several Zen servers that actually overlap on several drives. So were there virtual servers? Yes. So this is the picture of the VM architecture on the top. This is more the view of the file storage. This is a single machine with a rate of disks. And this is a file storage for a single machine. So you have the physical volume and then you have the first column is where it used to be called DOM zero which is the operating system that manage all the VMs when you access to the Zen server. You access to this. It's also a VM but you access to this and it has direct access to the hardware instead of the VM which have access to the hypervisor. And all the other columns, each one of them are a Zen host which are one of these VMs or DOM view as it's called in Zen. And each one of them has the DRBD device which is basically a virtual block device which is replicated from here to another DRBD server. So you have each one of these DR devices are inside this list of virtual block devices and they are replicated through the network in real time when they are written or read or only written because it's a sync to the secondary Zen server. So it's done automatically through the network when it's executed but the difference is each one of them has a DR device. Okay. Thank you. Okay. We have maybe one minute if anyone has a really quick question. Okay. Please join me in thanking Abraham once again. Thank you.
Hello everyone, I'm Abraham Martin. I work for the University of Cambridge, as you may have computing service, where we provide IT services for the rest of the university. Both buildings about maintaining the web application. We install these, we created a web panel using basic packages that we have been installing up to now, like the Apache, which is the most common install TLS certificates in the machines, do the just a panel with some options to manage your create Vhost, ask for domain names, et cetera, VM architecture. The VM architecture is just a panel and some APIs, and we have a standard have a lot of services interconnected using a lot could be substituted by any authentication that we have a lot of other services that we name is already in use, et cetera, and the same API also provides us from IP addresses. We have these APIs and these services are supporting all this service, we went to the community in the lot of time to take from backups, so some of the availability, so lucky for us, we designed the has happened, with the VMWare solutions you will keys, etc. We have a separation of privilege, for users based on some filtering and some discussion we had between the developers, which