An HTTP request's journey through a platform-as-a-service
Thomas, Giles
Giles Thomas - An HTTP request's journey through a platform-as-a-service PythonAnywhere hosts tens of thousands of Python web applications, with traffic ranging from a couple of hits a week to dozens of hits a second. Hosting this many sites reliably at a reasonable cost requires a well-designed infrastructure, but it uses the same standard components as many other Python-based websites. We've built our stack on GNU/Linux, nginx, uWSGI, Redis, and Lua - all managed with Python. In this talk we'll give a high-level overview of how it all works, by tracing how a request goes from the browser to the Python application and its response goes back again. As well as showing how a fairly large deployment works, we'll give tips on scaling and share a few insights that may help people running smaller sites discover how they can speed things up.
 The first of our speakers is going to be Giles Thomas. Giles started programming Python when founding a business. He wanted to revolutionize the spreadsheet world by making spreadsheets programmable. And then he tried to sell them to financial companies. That didn't work. Then his team moved over to producing the Python system that they wanted for people like themselves. And that sold a lot better. Giles is also playing the guitar. Today, however, he is going to take you on a journey. The journey of an HTTP request through a platform as a service. Please welcome with a hot applause Giles Thomas. Well, thanks for the excellent introduction. Thanks everyone for coming. So yeah, the thing that we wrote, the thing that we thought that we were going to build the Python system that we wanted is called Python Anywhere. It's a platform as a service. It does lots of things. One thing that it does is that it hosts a fair number of websites. So just wanted to get a general feeling about how much people in the room know about running websites. How many people here are responsible for the continued operation of a website, maybe a personal blog or company pages? Okay, so a fair number. Yeah, that's maybe 50%. Let's bring that up a little bit. How many are responsible for several websites? Okay, more than 10? Still a few? More than 100? Still one? More than 1,000? Okay, I'm not going to keep going up because it'll be really embarrassing. It'll turn out that you actually run more websites than we do. We have 24,241 websites running on Python Anywhere as of this morning, actually probably a few more by now. And we've got infrastructure to run this. It's a simple platform as a service infrastructure. I'm going to go through a description of it pretty quickly, touching on a few of the details. But what I'd like to do is leave quite a lot of time for questions because I think which bits are interesting to drill down into probably come more from you guys than from me, guessing at what you're going to be interested in. The websites we have, they range from very basic things where somebody has basically started using a particular framework. So somebody here has been trying out Web2Py. And maybe they're going to build something. It'll get a visitor a day. Maybe they're just a hobbyist experimenting. The next stage is maybe sites which get a couple of hundred visitors a day. This guy is learning Mandarin Chinese in a particular way. He's sharing his lessons with a few other people. So a couple of hundred visitors a day, we want to spend almost no resources on the first kind. Enough resources to keep this kind of site responsible. That's very responsive even. Other people are running popular technical blogs. This is my colleague Harry's, a beta testing go. It's a companion site for a book he's written, which is awesome. And you should totally buy it if you're interested in test driven development with Python and Django. But he gets maybe 2,000 visitors a day. So the site needs to be responsive. It needs to be up all the time. But it doesn't need that much more than that. It's not a high volume site. This is one of our most fun customers. This guy is running a site and it's insanely popular. It gets dozens of hits every second pouring through there. It's actually quite a good selection of music, even if you don't like getting out of your head and various things. But it's, again, quite a popular website. It's not Amazon. It's not Google. But it's got to be there. It's responsive and it's got to be maintained at an affordable price. So how do we do this? Well, here's a very basic list of what it is, a set of logos. These are the tools that we use. We use GNU Linux, obviously. We use Nginx for our load balancing for all of our HTTP needs. We use Uwizgi, which is an absolutely awesome product which manages Python processes for you so they can serve up web applications. It can serve basically any web application that uses the Uwizgi protocol. So that's going to be Django. That's Web2Py. That's Bottle. That's Flask. That's all of the big ones, possibly except for most tornado installations, which doesn't play so well with Uwizgi. We do use Redis. I'm not going to go into much detail on that today. We also use Lua for a certain amount of scripting. Now, I know I'm in a bit of danger here for talking about the ones of Lua at a Python conference, but we do use it. It's awesome for what it does for the specific use case we have. Now you'll notice that I've got Django and I've got Python there. I didn't mention them. Well, all of our infrastructure uses the tools that I've described so far. All of the configuration is managed by Python. It's managed by a number of Django applications, which basically spit out the configuration files that all the other stuff needs to run and keeps the cluster live and doing what it's meant to do. So I promised a description of a HTTP request journey through this platform service. Here are the machines that are involved in that. So you can see what I have here is up in the top left, let's use the mouse pointer, you can see each of these blue boxes is a separate physical machine or a separate instance running on Amazon, AWS or whatever. Here's the user's laptop. He's running Chrome. Down here we have a load balancer and a bunch of back end servers. Anything apart from this machine up here in the top left is part of Python Anywhere's infrastructure. We run on Amazon AWS, but that's kind of, that's not particularly relevant to the context of this talk. Let's say that the person who's running the browser up here wants to view my friend Harry's website. They want to go to visit www.abaythetestingote.com. Well, their browser makes a DNS request. The DNS request comes back with the IP address of abaythetestingote.com, which is the IP address of this load balancer down here. So it opens up a TCP IP connection down to the load balancer. It sends the request to the load balancer. Now in order to route it through to the web application, let's say that a Bay of the Testing Goat is this web application here, this Python process that's running over on this particular physical machine, the middle one on the right hand side if you can't see the mouse pointer. So the load balancer needs to have the intelligence to be able to know that a Bay of the Testing Goat.com is running on back end server 2. So we'll just say that's magic for the moment. Let's say it magically knows back end server 2, it makes a connection. And now we have two TCP IP connections, one from the client, the load balancer, one from the load balancer, the back end. The back end now needs to identify that the process where about 4 is the Python process that's running a Bay of the Testing Goat.com. It does that, again, we'll say magically, and makes the connection. The web application code does its calculations, it renders templates, it talks to the database, it does whatever magic it does to generate a page, it sends it back to the back end server, back end server sends it back to the load balancer, load balancer sends it back to the client. Now if we're used to running normal kinds of websites, the kind of system where I personally, for example, I have a VPS where I used to host my personal blog, then you might be thinking what's the point of the load balancer in there? Because normally you'd simply have a server that looks rather like the back end server here. It's running a front end web server like Nginx or Apache, and it's got a number of web applications running, one or more web applications running as Python processes underneath it. Why do we have this extra set for the load balancer? That's kind of where the magic comes in, because that's the load balancer that allows us to scale up, to scale down, to add in resilience and failover and all those other good things that people expect when they outsource running their web applications to a third party like us, rather than renting a VPS. So I said the load balancer knows by magic which back end it's got to send that request to. Our load balancer is running Nginx. It's running a specific flavor of Nginx called OpenResty. Now Nginx is an awesome web server. It's extremely fast. It's very good at proxying connections through it like we do through the load balancer, and it has a lot of great plugins. OpenResty is basically Nginx with batteries included. One of the batteries that's included is Lua scripting. The kind of Lua scripting you can do is actually insanely powerful. You can do any amount of Lua processing inside every single request. It works extremely fast. Lua, I think, is a nice language. It's not as nice as Python, but some of the design decisions they made that make it a less pleasant language to look at and work with are actually very good for speed and efficiency. I think they chose it for the majority of Nginx scripting. What we do inside our load balancer code is actually really very simple. What I've got here is the Nginx configuration file. Hopefully that's reasonably readable. At the top here, we're saying, in it by Lua file, when Nginx starts up, it's going to load, it's going to run that script in it backends. In it backends basically just specifies some global context which is available to any Lua script inside Nginx saying, here is a list of all of the back end servers. That's all it does. As we go down here, we come into our server block. So we're listing on ports 80 and 443. This location slash block is basically something that's going to be executed. Think of it as code that's executed for every request. What we do is we extract the host that this request is asking for, www.abadethetestingote.com. We extract it from the HTTP host, header from the HTTP request, and it's actually in a variable called root host. We then set a back end IP variable to empty string. This is basically a function call here. We're calling the Lua function that's contained in get back end IP. Now you can guess what get back end IP does. It returns the IP in this back end IP variable. Then we go into this little bit of Nginx magic which is proxy pass. That says, just hand off the processing of this request to that server over there identified by this IP. Nginx is off the rest for us. Let's take a look at that Lua file. This is an interesting bit of code because it was something we put in for the first cut of our load balance, which we thought we were going to get rid of in a week or so. It seemed too simple. It seemed too, what's the word? It didn't seem complicated enough to work. All it does is hash the host name that comes in. So that's literally the code that Python uses when you hash a string, convert it into Lua. If it hashes that, so you've got a number from the host name, we then take that modulo, the number of back ends, and use that to index into the list of back ends. So that means that every single web server we're running, every single website we're running is assigned essentially randomly to one of the different back ends. It's stably assigned to the same back end. If we add new back end to the cluster, the modulo number we're using increases, and so everything automatically spreads itself out over the cluster again. That's the load balancer. I said that the back end server also needs to identify which process is running a particular web application. So this is some really basic Nginx configuration that any of you who've done Uiski stuff on Nginx will recognize. All we're saying here is, again, extract the domain name from the request that's being made from the request we're processing, different way of doing it, but the same effect. And what we say Nginx to do is delegate all requests for www.bay, the testinggo.com, to a particular socket. This is all dynamic stuff. This is what the config actually looks like. It's not a sample there. So any request that comes into this Nginx, it will immediately look for a socket in that particular location and expect there to be a Uiski process sitting on the other end of it running the website that should be on that particular domain. How does Uiski know that it needs to have a web application running on that socket? Well, Uiski has a directory called, it contains what they call Vassal files. A Vassal is Uiski's terminology for running Python process or set of processes that's responsible for a particular web application. It's configured by a Vassal file, and a Vassal file basically has various things saying where the code is, what kind of sandboxing you want to apply, how many worker processes you want. Importantly, it also has this line at the top here, excuse me, which is the socket that it needs to listen on. Uiski is very clever. If a Vassal file is created, configuring a Uiski Vassal like this, it will immediately detect the creation of that file if it's in the right directory, and it will fire up all the processes immediately. And that means that obviously the web application has started. So what we need to do is start a web application when requests come in. This is where things get a little bit more complicated. What happens if a request comes into one of our backends and there is no process running for that particular web application? Well, I told you the engine X I showed you earlier was simplified. Here's something a bit closer to the truth. When engine X tries to connect to a Uiski back end that's not there, maybe there's no socket, maybe Uiski itself hasn't started the processes, maybe it's killed them because they timed out after a certain amount of inactivity, engine X will internally generate a 502 error. Normally that just goes back to the browser and obviously things look bad. What we have here is error page handler. If there is a 502 error, we essentially do a go to to this other block here at fallback. Error page 502, if there's an error, we wind up inside this code here. And all we do here is we check whether there is a Vassal file for that particular domain. So let's say we're looking for www.baythetestingote.com. The process isn't running. The first thing we do is jump to this fallback. We see whether there's a Vassal file for that domain. If there's a Vassal file for that domain, we can safely assume that there are processes running. So actually this was a real 502. Maybe something went wrong inside the web application so we generate a real 502 error. But if there isn't a Vassal file for that particular web app, we know we need to start it. Now you remember that proxy pass from the load balancer where essentially we're saying delegate or work for this request to this IP over there. This is another proxy pass here which is delegating to a little microservice running locally. The microservice running locally is actually a very small Django application. It has access to the database that configures all of the websites we run. When it receives a call on its initialized web app view, then it says, OK, I need to start up that particular web application. It goes to the database. It gathers all the information about the user. It works out whether we have a virtual container for this particular user running on this particular machine. It starts that up if necessary. It then creates the whiskey configuration. It generates a whiskey.ne file, the Vassal file, passes that off to you whiskey. You whiskey starts the process up and running and suddenly we can start delegating all the work to that. So why is this interesting? Well, what it means is that we can actually scale pretty much transparently. Let's say we've had a busy day and let's imagine we've only had until now, say, three web servers in our cluster, and then suddenly things hot up. Maybe the web applications we've got are getting more busy or a bunch of new people are signed up and we've got to serve more websites. All we do is we create a new back-end server, which is very easy with Amazon. We just fire up a new instance. And then we tell the load balancer about it. Immediately on telling EngineX to reload its configuration, it will start distributing requests differently across the back-ends and any back-ends that need to start web apps will automatically start them. The ones that are running web applications that they no longer need to run will all start timing out and killing themselves. So dynamically reconfigure the cluster very, very simply. Now let's say that something goes wrong. Last night, one of our web servers started showing problems. And so we got pinged by Pingdom saying LiveWeb1 was going down. LiveWeb1 is a particular server that we have on Amazon. And every year about this time, hardware starts failing on AWS. I think what happens is that all the regular engineers go on holiday and the interns haven't been given enough information on how to manage their systems. LiveWeb1 started failing. And so all we did was log into the load balancer, remove it from our list of back-ends. Everything, and then immediately reconfigured itself automatically just through the use of this hashing function to run on the remaining servers. That meant that, of course, all of the web apps were running a little bit more slowly because our machines were closer to their load limit. But that's fine. We have a fair amount of capacity for that. We could bounce the web server. Sorry, could bounce fix the broken server, bring it back into rotation with a new IP, and suddenly everything worked again. That was a very, very rapid tour through how the whole system works. And now I'd really like to hand over to you guys for any questions that we can drill down on anything that was interesting. Thank you. So we got a lot of time for questions. And here at the front. So the question is about the initialization of our script. Could you just use your whiskey to automatically start the final process instead? Why would you not use that much? The real problem is actually in the amount of work that needs to be done to start the final process because all of our users run inside sandbox environments, which we have to have control over the code to actually start them up. And you whiskey at the time when we started using it and I think still doesn't have the capability to do all of the setup work required to do it. It can start processes quite happily. It can run certain kinds of pre-init scripts. I don't record precisely what, but there's something that it could not do to do with the virtualization essentially. First question, how do you do the sandboxing? The sandboxing. Okay. It's kind of a roll your own thing. These days, if we're starting Python anywhere today, we might think of, we'd probably use Linux containers or we'd potentially use Docker. We're using Docker for some stuff we're working on now and think you're rolling some changes back in. But the good thing about Linux containers is that it was built out of reusable components. It uses Troutes. It uses process namespaces, network namespaces. And that meant that all of these have been becoming available for a number of years. We've essentially rolled our own kind of Linux containers lights by plugging those things together. Okay. And what happens if a worker is down and let's see free requests come in at the same time? Is there some kind of a risk condition here, right? Sorry. I don't think I understood that. So if you need to start a whiskey process for a website that was down, but free requests come in at the same time, is there some kind of locking to make sure you only started once and you don't lose any requests while it's starting up? Yes. Yes. There's locking inside our initialized web apps which handles that. YouWiskey does queue things to a certain degree as well, so we kind of got belt embraces there. We use the code that starts up our sandboxes in various places including these. We do in browser consoles and things like that. So we've kind of got locking on our site to protect us from that. And youWiskey does a certain amount of queuing. Okay. Thank you. Do you currently support web sockets or persistent connections? No, we don't. And that's something we really do want support. All of our infrastructure does support it to one extent or another. I think I'm not actually, I'm not sure what youWiskey support for web sockets is like at the moment. But I mean, the problem is that the Whiskey protocol doesn't really support web sockets. If youWiskey does support it, it'll be in some kind of extension on top of that. I think that even what we do so well, we definitely will support it. When we support it, we'll either wind up rolling something of our own to be able to manage long running, say, tornado processes and use the same engine X infrastructure to route through to appropriate places. Or maybe youWiskey will by that time do something that we can use. You never said how you deal with persistent states or with databases. Sorry, could you say that again? Well, what happens to the databases of the web apps? Oh, I see, okay, that's managed separately. We have MySQL instances and we're working on supporting Postgres instances just separate the kind of behind those back end servers. Do you have some code that manages that? Sorry, do we have code that? Could we repeat the question, please? Do you have some services that manage databases for users? On the MySQL side, it's all a little bit messy and was built ad hoc for, we're adding Postgres support right now and we're basically building that as a, we have a Flask microservice which runs on one of a set of Postgres servers which fires up Docker containers, each of which runs one Postgres instance. So the Flask microservice does the provisioning. We're hard at work on that at the moment, it's working well enough to pass our functional test which probably means it's a month or so away from deployment. Last question, please. There's one. Hi, you said that when one of your instances, you got an alert on it, then you manually removed him from the load balancer? Is the reason why you don't set up auto, like automatically remove or and also if you have auto upscale because you said that you have some sort of limit on your instances? Yeah, that's an excellent question. It's really been a matter of development time. One of the features we do need to add is automatic instance killing. When we first created it, it made sense to do it manually because each instance failure was rare enough, occurrence and was kind of unique enough in the way that it failed, that it was better to have a human in the loop. Whereas now, I think we've managed to get a list of the different ways in which instances can fail and we can probably start building in more automated responses. But yeah, that's just a case of we haven't had time to do it yet. So thank you very much, Giles, speaking here.
 The first of our speakers is going to be Giles Thomas.  Giles started programming Python when founding a business.  He wanted to revolutionize the spreadsheet world by making spreadsheets programmable.  And then he tried to sell them to financial companies.  That didn't work.  Then his team moved over to producing the Python system that they wanted for people  like themselves.  And that sold a lot better.  Giles is also playing the guitar.  Today, however, he is going to take you on a journey.  The journey of an HTTP request through a platform as a service.  Please welcome with a hot applause Giles Thomas.  Well, thanks for the excellent introduction.  Thanks everyone for coming.  So yeah, the thing that we wrote, the thing that we thought that we were going to build  the Python system that we wanted is called Python Anywhere.  It's a platform as a service.  It does lots of things.  One thing that it does is that it hosts a fair number of websites.  So just wanted to get a general feeling about how much people in the room know about running  websites.  How many people here are responsible for the continued operation of a website, maybe a  personal blog or company pages?  Okay, so a fair number.  Yeah, that's maybe 50%.  Let's bring that up a little bit.  How many are responsible for several websites?  Okay, more than 10?  Still a few?  More than 100?  Still one?  More than 1,000?  Okay, I'm not going to keep going up because it'll be really embarrassing.  It'll turn out that you actually run more websites than we do.  We have 24,241 websites running on Python Anywhere as of this morning, actually probably  a few more by now.  And we've got infrastructure to run this.  It's a simple platform as a service infrastructure.  I'm going to go through a description of it pretty quickly, touching on a few of the details.  But what I'd like to do is leave quite a lot of time for questions because I think which  bits are interesting to drill down into probably come more from you guys than from me, guessing  at what you're going to be interested in.  The websites we have, they range from very basic things where somebody has basically  started using a particular framework.  So somebody here has been trying out Web2Py.  And maybe they're going to build something.  It'll get a visitor a day.  Maybe they're just a hobbyist experimenting.  The next stage is maybe sites which get a couple of hundred visitors a day.  This guy is learning Mandarin Chinese in a particular way.  He's sharing his lessons with a few other people.  So a couple of hundred visitors a day, we want to spend almost no resources on the first  kind.  Enough resources to keep this kind of site responsible.  That's very responsive even.  Other people are running popular technical blogs.  This is my colleague Harry's, a beta testing go.  It's a companion site for a book he's written, which is awesome.  And you should totally buy it if you're interested in test driven development with Python and  Django.  But he gets maybe 2,000 visitors a day.  So the site needs to be responsive.  It needs to be up all the time.  But it doesn't need that much more than that.  It's not a high volume site.  This is one of our most fun customers.  This guy is running a site and it's insanely popular.  It gets dozens of hits every second pouring through there.  It's actually quite a good selection of music, even if you don't like getting out of your  head and various things.  But it's, again, quite a popular website.  It's not Amazon.  It's not Google.  But it's got to be there.  It's responsive and it's got to be maintained at an affordable price.  So how do we do this?  Well, here's a very basic list of what it is, a set of logos.  These are the tools that we use.  We use GNU Linux, obviously.  We use Nginx for our load balancing for all of our HTTP needs.  We use Uwizgi, which is an absolutely awesome product which manages Python processes for  you so they can serve up web applications.  It can serve basically any web application that uses the Uwizgi protocol.  So that's going to be Django.  That's Web2Py.  That's Bottle.  That's Flask.  That's all of the big ones, possibly except for most tornado installations, which doesn't  play so well with Uwizgi.  We do use Redis.  I'm not going to go into much detail on that today.  We also use Lua for a certain amount of scripting.  Now, I know I'm in a bit of danger here for talking about the ones of Lua at a Python conference,  but we do use it.  It's awesome for what it does for the specific use case we have.  Now you'll notice that I've got Django and I've got Python there.  I didn't mention them.  Well, all of our infrastructure uses the tools that I've described so far.  All of the configuration is managed by Python.  It's managed by a number of Django applications, which basically spit out the configuration  files that all the other stuff needs to run and keeps the cluster live and doing what  it's meant to do.  So I promised a description of a HTTP request journey through this platform service.  Here are the machines that are involved in that.  So you can see what I have here is up in the top left, let's use the mouse pointer, you  can see each of these blue boxes is a separate physical machine or a separate instance running  on Amazon, AWS or whatever.  Here's the user's laptop.  He's running Chrome.  Down here we have a load balancer and a bunch of back end servers.  Anything apart from this machine up here in the top left is part of Python Anywhere's  infrastructure.  We run on Amazon AWS, but that's kind of, that's not particularly relevant to the context  of this talk.  Let's say that the person who's running the browser up here wants to view my friend Harry's  website.  They want to go to visit www.abaythetestingote.com.  Well, their browser makes a DNS request.  The DNS request comes back with the IP address of abaythetestingote.com, which is the IP  address of this load balancer down here.  So it opens up a TCP IP connection down to the load balancer.  It sends the request to the load balancer.  Now in order to route it through to the web application, let's say that a Bay of the  Testing Goat is this web application here, this Python process that's running over on  this particular physical machine, the middle one on the right hand side if you can't see  the mouse pointer.  So the load balancer needs to have the intelligence to be able to know that a Bay of the Testing  Goat.com is running on back end server 2.  So we'll just say that's magic for the moment.  Let's say it magically knows back end server 2, it makes a connection.  And now we have two TCP IP connections, one from the client, the load balancer, one from  the load balancer, the back end.  The back end now needs to identify that the process where about 4 is the Python process  that's running a Bay of the Testing Goat.com.  It does that, again, we'll say magically, and makes the connection.  The web application code does its calculations, it renders templates, it talks to the database,  it does whatever magic it does to generate a page, it sends it back to the back end server,  back end server sends it back to the load balancer, load balancer sends it back to the  client.  Now if we're used to running normal kinds of websites, the kind of system where I personally,  for example, I have a VPS where I used to host my personal blog, then you might be thinking  what's the point of the load balancer in there?  Because normally you'd simply have a server that looks rather like the back end server  here.  It's running a front end web server like Nginx or Apache, and it's got a number of web applications  running, one or more web applications running as Python processes underneath it.  Why do we have this extra set for the load balancer?  That's kind of where the magic comes in, because that's the load balancer that allows us to  scale up, to scale down, to add in resilience and failover and all those other good things  that people expect when they outsource running their web applications to a third party like  us, rather than renting a VPS.  So I said the load balancer knows by magic which back end it's got to send that request  to.  Our load balancer is running Nginx.  It's running a specific flavor of Nginx called OpenResty.  Now Nginx is an awesome web server.  It's extremely fast.  It's very good at proxying connections through it like we do through the load balancer, and  it has a lot of great plugins.  OpenResty is basically Nginx with batteries included.  One of the batteries that's included is Lua scripting.  The kind of Lua scripting you can do is actually insanely powerful.  You can do any amount of Lua processing inside every single request.  It works extremely fast.  Lua, I think, is a nice language.  It's not as nice as Python, but some of the design decisions they made that make it a  less pleasant language to look at and work with are actually very good for speed and  efficiency.  I think they chose it for the majority of Nginx scripting.  What we do inside our load balancer code is actually really very simple.  What I've got here is the Nginx configuration file.  Hopefully that's reasonably readable.  At the top here, we're saying, in it by Lua file, when Nginx starts up, it's going to  load, it's going to run that script in it backends.  In it backends basically just specifies some global context which is available to any Lua  script inside Nginx saying, here is a list of all of the back end servers.  That's all it does.  As we go down here, we come into our server block.  So we're listing on ports 80 and 443.  This location slash block is basically something that's going to be executed.  Think of it as code that's executed for every request.  What we do is we extract the host that this request is asking for, www.abadethetestingote.com.  We extract it from the HTTP host, header from the HTTP request, and it's actually in  a variable called root host.  We then set a back end IP variable to empty string.  This is basically a function call here.  We're calling the Lua function that's contained in get back end IP.  Now you can guess what get back end IP does.  It returns the IP in this back end IP variable.  Then we go into this little bit of Nginx magic which is proxy pass.  That says, just hand off the processing of this request to that server over there identified  by this IP.  Nginx is off the rest for us.  Let's take a look at that Lua file.  This is an interesting bit of code because it was something we put in for the first  cut of our load balance, which we thought we were going to get rid of in a week or so.  It seemed too simple.  It seemed too, what's the word?  It didn't seem complicated enough to work.  All it does is hash the host name that comes in.  So that's literally the code that Python uses when you hash a string, convert it into Lua.  If it hashes that, so you've got a number from the host name, we then take that modulo,  the number of back ends, and use that to index into the list of back ends.  So that means that every single web server we're running, every single website we're  running is assigned essentially randomly to one of the different back ends.  It's stably assigned to the same back end.  If we add new back end to the cluster, the modulo number we're using increases, and so  everything automatically spreads itself out over the cluster again.  That's the load balancer.  I said that the back end server also needs to identify which process is running a particular  web application.  So this is some really basic Nginx configuration that any of you who've done Uiski stuff on  Nginx will recognize.  All we're saying here is, again, extract the domain name from the request that's being  made from the request we're processing, different way of doing it, but the same effect.  And what we say Nginx to do is delegate all requests for www.bay, the testinggo.com, to  a particular socket.  This is all dynamic stuff.  This is what the config actually looks like.  It's not a sample there.  So any request that comes into this Nginx, it will immediately look for a socket in that  particular location and expect there to be a Uiski process sitting on the other end of  it running the website that should be on that particular domain.  How does Uiski know that it needs to have a web application running on that socket?  Well, Uiski has a directory called, it contains what they call Vassal files.  A Vassal is Uiski's terminology for running Python process or set of processes that's  responsible for a particular web application.  It's configured by a Vassal file, and a Vassal file basically has various things saying where  the code is, what kind of sandboxing you want to apply, how many worker processes you want.  Importantly, it also has this line at the top here, excuse me, which is the socket that  it needs to listen on.  Uiski is very clever.  If a Vassal file is created, configuring a Uiski Vassal like this, it will immediately  detect the creation of that file if it's in the right directory, and it will fire up all  the processes immediately.  And that means that obviously the web application has started.  So what we need to do is start a web application when requests come in.  This is where things get a little bit more complicated.  What happens if a request comes into one of our backends and there is no process running  for that particular web application?  Well, I told you the engine X I showed you earlier was simplified.  Here's something a bit closer to the truth.  When engine X tries to connect to a Uiski back end that's not there, maybe there's  no socket, maybe Uiski itself hasn't started the processes, maybe it's killed them because  they timed out after a certain amount of inactivity, engine X will internally generate a 502  error.  Normally that just goes back to the browser and obviously things look bad.  What we have here is error page handler.  If there is a 502 error, we essentially do a go to to this other block here at fallback.  Error page 502, if there's an error, we wind up inside this code here.  And all we do here is we check whether there is a Vassal file for that particular domain.  So let's say we're looking for www.baythetestingote.com.  The process isn't running.  The first thing we do is jump to this fallback.  We see whether there's a Vassal file for that domain.  If there's a Vassal file for that domain, we can safely assume that there are processes  running.  So actually this was a real 502.  Maybe something went wrong inside the web application so we generate a real 502 error.  But if there isn't a Vassal file for that particular web app, we know we need to start  it.  Now you remember that proxy pass from the load balancer where essentially we're saying delegate  or work for this request to this IP over there.  This is another proxy pass here which is delegating to a little microservice running locally.  The microservice running locally is actually a very small Django application.  It has access to the database that configures all of the websites we run.  When it receives a call on its initialized web app view, then it says, OK, I need to  start up that particular web application.  It goes to the database.  It gathers all the information about the user.  It works out whether we have a virtual container for this particular user running on this  particular machine.  It starts that up if necessary.  It then creates the whiskey configuration.  It generates a whiskey.ne file, the Vassal file, passes that off to you whiskey.  You whiskey starts the process up and running and suddenly we can start delegating all the  work to that.  So why is this interesting?  Well, what it means is that we can actually scale pretty much transparently.  Let's say we've had a busy day and let's imagine we've only had until now, say, three  web servers in our cluster, and then suddenly things hot up.  Maybe the web applications we've got are getting more busy or a bunch of new people  are signed up and we've got to serve more websites.  All we do is we create a new back-end server, which is very easy with Amazon.  We just fire up a new instance.  And then we tell the load balancer about it.  Immediately on telling EngineX to reload its configuration, it will start distributing  requests differently across the back-ends and any back-ends that need to start web  apps will automatically start them.  The ones that are running web applications that they no longer need to run will all start  timing out and killing themselves.  So dynamically reconfigure the cluster very, very simply.  Now let's say that something goes wrong.  Last night, one of our web servers started showing problems.  And so we got pinged by Pingdom saying LiveWeb1 was going down.  LiveWeb1 is a particular server that we have on Amazon.  And every year about this time, hardware starts failing on AWS.  I think what happens is that all the regular engineers go on holiday and the interns haven't  been given enough information on how to manage their systems.  LiveWeb1 started failing.  And so all we did was log into the load balancer, remove it from our list of back-ends.  Everything, and then immediately reconfigured itself automatically just through the use  of this hashing function to run on the remaining servers.  That meant that, of course, all of the web apps were running a little bit more slowly  because our machines were closer to their load limit.  But that's fine.  We have a fair amount of capacity for that.  We could bounce the web server.  Sorry, could bounce fix the broken server, bring it back into rotation with a new IP,  and suddenly everything worked again.  That was a very, very rapid tour through how the whole system works.  And now I'd really like to hand over to you guys for any questions that we can drill down  on anything that was interesting.  Thank you.  So we got a lot of time for questions.  And here at the front.  So the question is about the initialization of our script.  Could you just use your whiskey to automatically start the final process instead?  Why would you not use that much?  The real problem is actually in the amount of work that needs to be done to start the  final process because all of our users run inside sandbox environments, which we have  to have control over the code to actually start them up.  And you whiskey at the time when we started using it and I think still doesn't have the  capability to do all of the setup work required to do it.  It can start processes quite happily.  It can run certain kinds of pre-init scripts.  I don't record precisely what, but there's something that it could not do to do with  the virtualization essentially.  First question, how do you do the sandboxing?  The sandboxing.  Okay.  It's kind of a roll your own thing.  These days, if we're starting Python anywhere today, we might think of, we'd probably use  Linux containers or we'd potentially use Docker.  We're using Docker for some stuff we're working on now and think you're rolling some changes  back in.  But the good thing about Linux containers is that it was built out of reusable components.  It uses Troutes.  It uses process namespaces, network namespaces.  And that meant that all of these have been becoming available for a number of years.  We've essentially rolled our own kind of Linux containers lights by plugging those things  together.  Okay.  And what happens if a worker is down and let's see free requests come in at the same time?  Is there some kind of a risk condition here, right?  Sorry.  I don't think I understood that.  So if you need to start a whiskey process for a website that was down, but free requests  come in at the same time, is there some kind of locking to make sure you only started once  and you don't lose any requests while it's starting up?  Yes.  Yes.  There's locking inside our initialized web apps which handles that.  YouWiskey does queue things to a certain degree as well, so we kind of got belt embraces there.  We use the code that starts up our sandboxes in various places including these.  We do in browser consoles and things like that.  So we've kind of got locking on our site to protect us from that.  And youWiskey does a certain amount of queuing.  Okay.  Thank you.  Do you currently support web sockets or persistent connections?  No, we don't.  And that's something we really do want support.  All of our infrastructure does support it to one extent or another.  I think I'm not actually, I'm not sure what youWiskey support for web sockets is like  at the moment.  But I mean, the problem is that the Whiskey protocol doesn't really support web sockets.  If youWiskey does support it, it'll be in some kind of extension on top of that.  I think that even what we do so well, we definitely will support it.  When we support it, we'll either wind up rolling something of our own to be able to manage  long running, say, tornado processes and use the same engine X infrastructure to route  through to appropriate places.  Or maybe youWiskey will by that time do something that we can use.  You never said how you deal with persistent states or with databases.  Sorry, could you say that again?  Well, what happens to the databases of the web apps?  Oh, I see, okay, that's managed separately.  We have MySQL instances and we're working on supporting Postgres instances just separate  the kind of behind those back end servers.  Do you have some code that manages that?  Sorry, do we have code that?  Could we repeat the question, please?  Do you have some services that manage databases for users?  On the MySQL side, it's all a little bit messy and was built ad hoc for, we're adding Postgres  support right now and we're basically building that as a, we have a Flask microservice which  runs on one of a set of Postgres servers which fires up Docker containers, each of which  runs one Postgres instance.  So the Flask microservice does the provisioning.  We're hard at work on that at the moment, it's working well enough to pass our functional  test which probably means it's a month or so away from deployment.  Last question, please.  There's one.  Hi, you said that when one of your instances, you got an alert on it, then you manually removed  him from the load balancer?  Is the reason why you don't set up auto, like automatically remove or and also if you have  auto upscale because you said that you have some sort of limit on your instances?  Yeah, that's an excellent question.  It's really been a matter of development time.  One of the features we do need to add is automatic instance killing.  When we first created it, it made sense to do it manually because each instance failure  was rare enough, occurrence and was kind of unique enough in the way that it failed, that  it was better to have a human in the loop.  Whereas now, I think we've managed to get a list of the different ways in which instances  can fail and we can probably start building in more automated responses.  But yeah, that's just a case of we haven't had time to do it yet.  So thank you very much, Giles, speaking here.
Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_0_383.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_384_1379.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_1380_1811.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_1812_1895.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_1896_2015.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_2016_2123.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_2124_2243.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_2244_3851.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_3852_4211.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_4212_4715.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_4716_5663.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_5664_6323.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_6324_7835.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_7836_7955.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_7956_8219.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_8220_8363.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_8364_16271.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_16272_17675.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_17676_20591.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_20592_25091.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_25092_26603.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_26604_27155.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_27156_27203.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_27204_27275.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_27276_27419.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_27420_28799.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_28800_32879.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_32880_36179.png Giles_Thomas_-_An_HTTP_request_s_journey_through_a_platform-as-a-service_36180_36298.png
The first of our speakers is going to be Giles Thomas. Giles started programming Python when founding a business. He wanted to revolutionize the spreadsheet world by making spreadsheets programmable. And then he tried to sell them to financial companies. That didn't work. Then his team moved over to producing the Python system that they wanted for people like themselves. And that sold a lot better. Giles is also playing the guitar. Today, however, he is going to take you on a journey. The journey of an HTTP request through a platform as a service. Please welcome with a hot applause Giles Thomas. Well, thanks for the excellent introduction. Thanks everyone for coming. So yeah, the thing that we wrote, the thing that we thought that we were going to build the Python system that we wanted is called Python Anywhere. It's a platform as a service. It does lots of things. One thing that it does is that it hosts a fair number of websites. So just wanted to get a general feeling about how much people in the room know about running websites. How many people here are responsible for the continued operation of a website, maybe a personal blog or company pages? Okay, so a fair number. Yeah, that's maybe 50%. Let's bring that up a little bit. How many are responsible for several websites? Okay, more than 10? Still a few? More than 100? Still one? More than 1,000? Okay, I'm not going to keep going up because it'll be really embarrassing. It'll turn out that you actually run more websites than we do. We have 24,241 websites running on Python Anywhere as of this morning, actually probably a few more by now. And we've got infrastructure to run this. It's a simple platform as a service infrastructure. I'm going to go through a description of it pretty quickly, touching on a few of the details. But what I'd like to do is leave quite a lot of time for questions because I think which bits are interesting to drill down into probably come more from you guys than from me, guessing at what you're going to be interested in. The websites we have, they range from very basic things where somebody has basically started using a particular framework. So somebody here has been trying out Web2Py. And maybe they're going to build something. It'll get a visitor a day. Maybe they're just a hobbyist experimenting. The next stage is maybe sites which get a couple of hundred visitors a day. This guy is learning Mandarin Chinese in a particular way. He's sharing his lessons with a few other people. So a couple of hundred visitors a day, we want to spend almost no resources on the first kind. Enough resources to keep this kind of site responsible. That's very responsive even. Other people are running popular technical blogs. This is my colleague Harry's, a beta testing go. It's a companion site for a book he's written, which is awesome. And you should totally buy it if you're interested in test driven development with Python and Django. But he gets maybe 2,000 visitors a day. So the site needs to be responsive. It needs to be up all the time. But it doesn't need that much more than that. It's not a high volume site. This is one of our most fun customers. This guy is running a site and it's insanely popular. It gets dozens of hits every second pouring through there. It's actually quite a good selection of music, even if you don't like getting out of your head and various things. But it's, again, quite a popular website. It's not Amazon. It's not Google. But it's got to be there. It's responsive and it's got to be maintained at an affordable price. So how do we do this? Well, here's a very basic list of what it is, a set of logos. These are the tools that we use. We use GNU Linux, obviously. We use Nginx for our load balancing for all of our HTTP needs. We use Uwizgi, which is an absolutely awesome product which manages Python processes for you so they can serve up web applications. It can serve basically any web application that uses the Uwizgi protocol. So that's going to be Django. That's Web2Py. That's Bottle. That's Flask. That's all of the big ones, possibly except for most tornado installations, which doesn't play so well with Uwizgi. We do use Redis. I'm not going to go into much detail on that today. We also use Lua for a certain amount of scripting. Now, I know I'm in a bit of danger here for talking about the ones of Lua at a Python conference, but we do use it. It's awesome for what it does for the specific use case we have. Now you'll notice that I've got Django and I've got Python there. I didn't mention them. Well, all of our infrastructure uses the tools that I've described so far. All of the configuration is managed by Python. It's managed by a number of Django applications, which basically spit out the configuration files that all the other stuff needs to run and keeps the cluster live and doing what it's meant to do. So I promised a description of a HTTP request journey through this platform service. Here are the machines that are involved in that. So you can see what I have here is up in the top left, let's use the mouse pointer, you can see each of these blue boxes is a separate physical machine or a separate instance running on Amazon, AWS or whatever. Here's the user's laptop. He's running Chrome. Down here we have a load balancer and a bunch of back end servers. Anything apart from this machine up here in the top left is part of Python Anywhere's infrastructure. We run on Amazon AWS, but that's kind of, that's not particularly relevant to the context of this talk. Let's say that the person who's running the browser up here wants to view my friend Harry's website. They want to go to visit www.abaythetestingote.com. Well, their browser makes a DNS request. The DNS request comes back with the IP address of abaythetestingote.com, which is the IP address of this load balancer down here. So it opens up a TCP IP connection down to the load balancer. It sends the request to the load balancer. Now in order to route it through to the web application, let's say that a Bay of the Testing Goat is this web application here, this Python process that's running over on this particular physical machine, the middle one on the right hand side if you can't see the mouse pointer. So the load balancer needs to have the intelligence to be able to know that a Bay of the Testing Goat.com is running on back end server 2. So we'll just say that's magic for the moment. Let's say it magically knows back end server 2, it makes a connection. And now we have two TCP IP connections, one from the client, the load balancer, one from the load balancer, the back end. The back end now needs to identify that the process where about 4 is the Python process that's running a Bay of the Testing Goat.com. It does that, again, we'll say magically, and makes the connection. The web application code does its calculations, it renders templates, it talks to the database, it does whatever magic it does to generate a page, it sends it back to the back end server, back end server sends it back to the load balancer, load balancer sends it back to the client. Now if we're used to running normal kinds of websites, the kind of system where I personally, for example, I have a VPS where I used to host my personal blog, then you might be thinking what's the point of the load balancer in there? Because normally you'd simply have a server that looks rather like the back end server here. It's running a front end web server like Nginx or Apache, and it's got a number of web applications running, one or more web applications running as Python processes underneath it. Why do we have this extra set for the load balancer? That's kind of where the magic comes in, because that's the load balancer that allows us to scale up, to scale down, to add in resilience and failover and all those other good things that people expect when they outsource running their web applications to a third party like us, rather than renting a VPS. So I said the load balancer knows by magic which back end it's got to send that request to. Our load balancer is running Nginx. It's running a specific flavor of Nginx called OpenResty. Now Nginx is an awesome web server. It's extremely fast. It's very good at proxying connections through it like we do through the load balancer, and it has a lot of great plugins. OpenResty is basically Nginx with batteries included. One of the batteries that's included is Lua scripting. The kind of Lua scripting you can do is actually insanely powerful. You can do any amount of Lua processing inside every single request. It works extremely fast. Lua, I think, is a nice language. It's not as nice as Python, but some of the design decisions they made that make it a less pleasant language to look at and work with are actually very good for speed and efficiency. I think they chose it for the majority of Nginx scripting. What we do inside our load balancer code is actually really very simple. What I've got here is the Nginx configuration file. Hopefully that's reasonably readable. At the top here, we're saying, in it by Lua file, when Nginx starts up, it's going to load, it's going to run that script in it backends. In it backends basically just specifies some global context which is available to any Lua script inside Nginx saying, here is a list of all of the back end servers. That's all it does. As we go down here, we come into our server block. So we're listing on ports 80 and 443. This location slash block is basically something that's going to be executed. Think of it as code that's executed for every request. What we do is we extract the host that this request is asking for, www.abadethetestingote.com. We extract it from the HTTP host, header from the HTTP request, and it's actually in a variable called root host. We then set a back end IP variable to empty string. This is basically a function call here. We're calling the Lua function that's contained in get back end IP. Now you can guess what get back end IP does. It returns the IP in this back end IP variable. Then we go into this little bit of Nginx magic which is proxy pass. That says, just hand off the processing of this request to that server over there identified by this IP. Nginx is off the rest for us. Let's take a look at that Lua file. This is an interesting bit of code because it was something we put in for the first cut of our load balance, which we thought we were going to get rid of in a week or so. It seemed too simple. It seemed too, what's the word? It didn't seem complicated enough to work. All it does is hash the host name that comes in. So that's literally the code that Python uses when you hash a string, convert it into Lua. If it hashes that, so you've got a number from the host name, we then take that modulo, the number of back ends, and use that to index into the list of back ends. So that means that every single web server we're running, every single website we're running is assigned essentially randomly to one of the different back ends. It's stably assigned to the same back end. If we add new back end to the cluster, the modulo number we're using increases, and so everything automatically spreads itself out over the cluster again. That's the load balancer. I said that the back end server also needs to identify which process is running a particular web application. So this is some really basic Nginx configuration that any of you who've done Uiski stuff on Nginx will recognize. All we're saying here is, again, extract the domain name from the request that's being made from the request we're processing, different way of doing it, but the same effect. And what we say Nginx to do is delegate all requests for www.bay, the testinggo.com, to a particular socket. This is all dynamic stuff. This is what the config actually looks like. It's not a sample there. So any request that comes into this Nginx, it will immediately look for a socket in that particular location and expect there to be a Uiski process sitting on the other end of it running the website that should be on that particular domain. How does Uiski know that it needs to have a web application running on that socket? Well, Uiski has a directory called, it contains what they call Vassal files. A Vassal is Uiski's terminology for running Python process or set of processes that's responsible for a particular web application. It's configured by a Vassal file, and a Vassal file basically has various things saying where the code is, what kind of sandboxing you want to apply, how many worker processes you want. Importantly, it also has this line at the top here, excuse me, which is the socket that it needs to listen on. Uiski is very clever. If a Vassal file is created, configuring a Uiski Vassal like this, it will immediately detect the creation of that file if it's in the right directory, and it will fire up all the processes immediately. And that means that obviously the web application has started. So what we need to do is start a web application when requests come in. This is where things get a little bit more complicated. What happens if a request comes into one of our backends and there is no process running for that particular web application? Well, I told you the engine X I showed you earlier was simplified. Here's something a bit closer to the truth. When engine X tries to connect to a Uiski back end that's not there, maybe there's no socket, maybe Uiski itself hasn't started the processes, maybe it's killed them because they timed out after a certain amount of inactivity, engine X will internally generate a 502 error. Normally that just goes back to the browser and obviously things look bad. What we have here is error page handler. If there is a 502 error, we essentially do a go to to this other block here at fallback. Error page 502, if there's an error, we wind up inside this code here. And all we do here is we check whether there is a Vassal file for that particular domain. So let's say we're looking for www.baythetestingote.com. The process isn't running. The first thing we do is jump to this fallback. We see whether there's a Vassal file for that domain. If there's a Vassal file for that domain, we can safely assume that there are processes running. So actually this was a real 502. Maybe something went wrong inside the web application so we generate a real 502 error. But if there isn't a Vassal file for that particular web app, we know we need to start it. Now you remember that proxy pass from the load balancer where essentially we're saying delegate or work for this request to this IP over there. This is another proxy pass here which is delegating to a little microservice running locally. The microservice running locally is actually a very small Django application. It has access to the database that configures all of the websites we run. When it receives a call on its initialized web app view, then it says, OK, I need to start up that particular web application. It goes to the database. It gathers all the information about the user. It works out whether we have a virtual container for this particular user running on this particular machine. It starts that up if necessary. It then creates the whiskey configuration. It generates a whiskey.ne file, the Vassal file, passes that off to you whiskey. You whiskey starts the process up and running and suddenly we can start delegating all the work to that. So why is this interesting? Well, what it means is that we can actually scale pretty much transparently. Let's say we've had a busy day and let's imagine we've only had until now, say, three web servers in our cluster, and then suddenly things hot up. Maybe the web applications we've got are getting more busy or a bunch of new people are signed up and we've got to serve more websites. All we do is we create a new back-end server, which is very easy with Amazon. We just fire up a new instance. And then we tell the load balancer about it. Immediately on telling EngineX to reload its configuration, it will start distributing requests differently across the back-ends and any back-ends that need to start web apps will automatically start them. The ones that are running web applications that they no longer need to run will all start timing out and killing themselves. So dynamically reconfigure the cluster very, very simply. Now let's say that something goes wrong. Last night, one of our web servers started showing problems. And so we got pinged by Pingdom saying LiveWeb1 was going down. LiveWeb1 is a particular server that we have on Amazon. And every year about this time, hardware starts failing on AWS. I think what happens is that all the regular engineers go on holiday and the interns haven't been given enough information on how to manage their systems. LiveWeb1 started failing. And so all we did was log into the load balancer, remove it from our list of back-ends. Everything, and then immediately reconfigured itself automatically just through the use of this hashing function to run on the remaining servers. That meant that, of course, all of the web apps were running a little bit more slowly because our machines were closer to their load limit. But that's fine. We have a fair amount of capacity for that. We could bounce the web server. Sorry, could bounce fix the broken server, bring it back into rotation with a new IP, and suddenly everything worked again. That was a very, very rapid tour through how the whole system works. And now I'd really like to hand over to you guys for any questions that we can drill down on anything that was interesting. Thank you. So we got a lot of time for questions. And here at the front. So the question is about the initialization of our script. Could you just use your whiskey to automatically start the final process instead? Why would you not use that much? The real problem is actually in the amount of work that needs to be done to start the final process because all of our users run inside sandbox environments, which we have to have control over the code to actually start them up. And you whiskey at the time when we started using it and I think still doesn't have the capability to do all of the setup work required to do it. It can start processes quite happily. It can run certain kinds of pre-init scripts. I don't record precisely what, but there's something that it could not do to do with the virtualization essentially. First question, how do you do the sandboxing? The sandboxing. Okay. It's kind of a roll your own thing. These days, if we're starting Python anywhere today, we might think of, we'd probably use Linux containers or we'd potentially use Docker. We're using Docker for some stuff we're working on now and think you're rolling some changes back in. But the good thing about Linux containers is that it was built out of reusable components. It uses Troutes. It uses process namespaces, network namespaces. And that meant that all of these have been becoming available for a number of years. We've essentially rolled our own kind of Linux containers lights by plugging those things together. Okay. And what happens if a worker is down and let's see free requests come in at the same time? Is there some kind of a risk condition here, right? Sorry. I don't think I understood that. So if you need to start a whiskey process for a website that was down, but free requests come in at the same time, is there some kind of locking to make sure you only started once and you don't lose any requests while it's starting up? Yes. Yes. There's locking inside our initialized web apps which handles that. YouWiskey does queue things to a certain degree as well, so we kind of got belt embraces there. We use the code that starts up our sandboxes in various places including these. We do in browser consoles and things like that. So we've kind of got locking on our site to protect us from that. And youWiskey does a certain amount of queuing. Okay. Thank you. Do you currently support web sockets or persistent connections? No, we don't. And that's something we really do want support. All of our infrastructure does support it to one extent or another. I think I'm not actually, I'm not sure what youWiskey support for web sockets is like at the moment. But I mean, the problem is that the Whiskey protocol doesn't really support web sockets. If youWiskey does support it, it'll be in some kind of extension on top of that. I think that even what we do so well, we definitely will support it. When we support it, we'll either wind up rolling something of our own to be able to manage long running, say, tornado processes and use the same engine X infrastructure to route through to appropriate places. Or maybe youWiskey will by that time do something that we can use. You never said how you deal with persistent states or with databases. Sorry, could you say that again? Well, what happens to the databases of the web apps? Oh, I see, okay, that's managed separately. We have MySQL instances and we're working on supporting Postgres instances just separate the kind of behind those back end servers. Do you have some code that manages that? Sorry, do we have code that? Could we repeat the question, please? Do you have some services that manage databases for users? On the MySQL side, it's all a little bit messy and was built ad hoc for, we're adding Postgres support right now and we're basically building that as a, we have a Flask microservice which runs on one of a set of Postgres servers which fires up Docker containers, each of which runs one Postgres instance. So the Flask microservice does the provisioning. We're hard at work on that at the moment, it's working well enough to pass our functional test which probably means it's a month or so away from deployment. Last question, please. There's one. Hi, you said that when one of your instances, you got an alert on it, then you manually removed him from the load balancer? Is the reason why you don't set up auto, like automatically remove or and also if you have auto upscale because you said that you have some sort of limit on your instances? Yeah, that's an excellent question. It's really been a matter of development time. One of the features we do need to add is automatic instance killing. When we first created it, it made sense to do it manually because each instance failure was rare enough, occurrence and was kind of unique enough in the way that it failed, that it was better to have a human in the loop. Whereas now, I think we've managed to get a list of the different ways in which instances can fail and we can probably start building in more automated responses. But yeah, that's just a case of we haven't had time to do it yet. So thank you very much, Giles, speaking here.
with a hot applause Giles Thomas. Well, thanks 24,241 websites running on Python Anywhere as of few other people. So a couple of hundred visitors and it's insanely popular. It gets dozens of hits they can serve up web applications. It can serve basically any web application that uses the about the ones of Lua at a Python conference, but Python. It's managed by a number of Django description of a HTTP request journey through relevant to the context of this talk. Let's say the load balancer, one from the load balancer, Goat.com. It does that, again, we'll say all those other good things that people expect in it backends. In it backends basically just just goes back to the browser and obviously one of our web servers started showing problems. and use the same engine X infrastructure to route