Drive-by SSL certificate creation with nginx
Lüchinger, Dominic
When working with webservers it's sometimes necessary to introduce some dynamic parts in the life cycle of an HTTP process. nginx provides this possibilities with an integration of Lua. On the example of dynamic SSL certificate creation we explore the capabilities of Lua within the core of nginx.
 Welcome you to my talk, Drive by SSL Certificate Creation with Nginx. During this talk you will see what I mean by this, what I have in mind about an IDE, what Nginx is capable of. First of all about me, I'm Dominic, working for a hosting company back in Switzerland. We are compared to German market, quite small, but we are in the top five of Switzerland, top four. Everything is factor 10 smaller in Switzerland. I'm mostly developer there doing a lot of backend security, that's why I sell them, have to pick something on the website, that's part of my team, but not mine. You find me on Twitter, it's also my nickname. If you have questions, please ask right away and don't hesitate to wait at the end of the talk. I came across a capability of Nginx when Cloudflare in 2004 or 2014 announced that they give away free SSL certificate for all. They explain what they are doing and I followed along the Nginx community. We used the Nginx and the scriptable capabilities and I thought it would be nice having such a feature. Recently there was a patch contributed by Cloudflare and this talk is about this patch and the possibility. This year is 2015 and if Frostconn has a SSL certificate that is only for this specific domain, it has no SAN or Y-Cast certificate, they have reissue certificate every year. The same goes for if you are a pass provider, good example, Slack. Maybe some of you know Slack as a chat program. Each company has its own subdomain and I guess what they provide is a Y-Cast certificate. What would be really nice is having a certificate on the fly generation per subdomain. That is the whole idea behind it and I will introduce you to steps to build such a system. If it is reasonable, I don't know. It is more a technical challenge than a real use case but it could be. In the example of Cloudflare, it is a good example. About Nginx, written and spelled, it is a different thing. Nginx itself, if you are familiar, is more on the developer side. Who is responsible for server stuff and deploying? It is like the majority. I will skip this quickly. You know the second most used web server. His properties are high performance, low memory, footprint. He is a web server, he is also proxy, load balancer, you can do caching. Since the new version 1.9, you can also do TCP proxy. That is a new feature that they introduced. They came along with H8 proxy. The whole system is event driven and has a sort of business design. It started in 2002, went public in 2004 and now in 2014, Nginx.com, the company was built to have a professional support. The open source domain is Nginx.org, the commercial one is Nginx.com. Think of Nginx. It is like Batman, he can do a lot of things and he is great and he is established. He has all the good benefits and features. What makes Batman really, really a superhero is that he has a lot of accessories that he can use. The same applies to Nginx. Nginx has quite a small core. Nginx is capable of doing HTTP stuff, SSL stuff, but not much with a little bit of logging and a little bit of load balancing. In the ecosystem around Nginx are the accessories. People build in modules written in C to extend the capabilities or the belt of Nginx. A few examples, it is really wild west. If you look at the third-party modules, you see abandoned stuff, you see stuff like video thumbnail extractor or upload process. That is actually quite often used or a circle gift that is generating gifts on demand and resizing them. Drizzle is also a thing that is often used. It is a myoscal possibility to talk to myoscal within the Nginx scope and talk to Rettis as well. Another thing that I can... Here is a little example. That is actually two different third-party modules. This one is the echo module. It is not in the core and returns the value. The other one is the Rettis extension. The first one was shitty, they built another one. With this example, you can set different values directly to Rettis or read it and then do some dynamic low-balancing or configuration stuff. Another thing is Nginx because it is event-driven. It has different phases or different stages during the start of an HTTP request and handling and finishing it off. There are some steps in between, I left them out because they are mostly pre-imposed stuff, configuration, read and write. The important one are the rewrite phase. In this phase, you could apply different rewrites, how Nginx is behaving. The access phase is normal, what you know about HTTP authentication, try files, it is looking at the file system content, it is delivering content out of different file system database, different backend. Log is after the process is done, it is locked. Then you can apply filters, if the request is at the stage where you can push back to the client, you can do filtering of the header and the body. An example you can see here is the echo example. It is in the content phase. The set is actually in the rewrite phase. If you write a C extension, you have to decide in which event you hook into the Nginx core and what are the changes you are applying. Because there are different modules, you can sometimes mess up stuff from other people. Then there would be nothing without Robin. In my talk, it is Lua. Lua is a programming language. Lua is fast by design, not by design, because it has a very, very lightweight design. A good example is learn Lua in 15 minutes. There is a web page. If you are familiar with programming, within 15 minutes, I can assure you are capable. The syntax is so easy. Invitable is one of the features that Lua is famous for. For example, Far Cry and a lot of game engines have the possibility to write modifications in Lua. They expose different modifications possibilities. You see embeddability also in the automotive industry, where you have simulations and then you hook in. Lua is used in network switches as well to packet filtering and on-the-fly giving the scriptable language possibility to hardware boxes. That is actually from the 15 minutes. It is not the whole part, but you see the syntax is quite easy. You have no types. Numbers are always doubled. You have double and single quotes. You have multi-line. If you need something, it goes to garbage collection. You have loops and if-else, and there are some other constructs. What do you know from different programming language hashes or arrays? They have tables. They have no class support, like object-orientated. You can do stuff, but no rich functionality. It is mostly procedural programming. In the UND, Lua has a lot of accessories. Lua comes with a rich variety of different modules that can be written in Lua or in C itself. What I mean by this is a good example for this is an MVC. It is a web framework that is entirely written in Lua that you can use. In the Lua file system, it has some C extension to hook into the file system itself. It is like a C-json implementation exposed to Lua. What you also have in the Lua space is LuaRocks. LuaRocks is a different package manager in every language. It is quite limited compared to other tools that I have seen. Most of the time it works because people are not depending much on each other's modules. The JavaScript community has two lines of code, and everybody inherits these two lines. Then you need circle dependency resolutions and conflict resolutions. Lua is capable of compiling C-based code. If someone ships C-embedded Lua functionality with LuaRocks, if you have the right dependencies on your OS operating system, then you can install this module. As always, there is an awesome language GitHub account where they have a lot of summaries about different tools and stuff about this ecosystem. Then there is LuaChit because LuaChit was started as a project because the core or the language itself is so small. A guy, I can't remember his name, started the LuaChit project. Now you have the possibility to either use Lua or LuaChit as the name for testing time compilers. If you run this code in a continuous way, it optimizes on the fly a lot of code. This is very useful in a web server where you would like to apply web application firewalls. These rules have to be run over and over and over again. They get hot and they are optimized and compiled to machine code. They say 10 to 100 times faster than the normal Lua implementation itself. One of the main drivers behind it is Cloudflare as well because now they ship around 10% of the traffic worldwide. If they could optimize LuaChit by 5 or 10%, they gain a lot of capacity. Think of LuaChit as robbing on steroids. The nice part about LuaChit is that it compiles on nearly all systems and all process architectures. It is binary compatible. It means that you can switch from Lua to LuaChit. It has the same syntax and it works. It works on a PS3 or PS4 which is quite nice. One thing to remember or take away is that Robbin and Batman are two heroes. They are both hands with a community and tools behind it. That is not enough. You have NGINX and Lua. Now we are at NGINX Lua or HTTP Lua module. That is the package name on the documentation and the GitHub name. It is confusing if you are searching for the thing. It means embedding the power of Lua into NGINX. Here is a small example. You would like to get the content. Content by Lua means within these brackets you write normal Lua code. It gets interpreted and it is printing out HALO work. That is the integration. You have a lot of these. You have init by Lua, init worker. In it it runs once when the server starts when a new master comes up. In init worker is when the master process forks a new worker process. You have set by Lua. In the previous example you have seen set as a directive of NGINX. Now you can dynamically set things in the scope of NGINX. Rewrite access content, log header filter and body filter. These are the phases where you can hook into NGINX and modify it by yourself. It gains some attraction because you saw this wild west of C modules. People started migrating their code into Lua because the burden of maintaining a C extension in an ever-changing NGINX code was too difficult or too time-consuming compared to Lua. That is always the same or more or less the same. That is the reason why nowadays a lot of modules and stuff are moving to the Lua world. All stuff are ripped out or is dying off. Another thing that is possible is Lua file. That means in the difference here you write this directly in your NGINX configuration. Here you can specify a specific Lua file. I always would recommend this way because you can see the burden that you have with double and single quotes within the NGINX configuration. There is a fix on the way but it needs some time. There is also another benefit. The Lua code normally if it is embedded by Lua file then it is compiled. Meaning the Lua cheat compiles it into an optimized way. Then it is always interpreted from this optimized way. During development every single character that you change in the Lua file you have to reload NGINX to get updates into your NGINX. That is one of the tricks that you can do. It is that you during development disable the cache and then you can edit right away. You always see the new updates if a new request hits this Lua specific directory. Another thing is these are the parts where the Lua NGINX Lua API comes in. Within the scope of Lua you have to talk somehow to the web server. Meaning that you need some API to talk to the outside world of NGINX. The first thing is NGINX ARC. It is something like you have on the console, on the bash script or in every other language. It is like parameters passed in. There is some situation where it is happening. For example set by Lua. You can pass in parameters or in the filters. But that is rarely the case that you work with. Most of the time you work with NGINX VARs. A good example is outside. VARs always has to be declared outside of the Lua scope. Meaning that during the lifecycle of a request you could change the values. This is very handy if you have, for example, at this point, going to a proxy or to a proxy upstream and define the variable of the proxy. In the Lua code you could do health checks or you could do load balancing or intelligently load balancing algorithm. If you set the variable in the right phase, then NGINX is performing proxy stuff with this modified variable. Another thing is NGINX header. It is used to read and modify headers depending on the stage. If you are passed the lock phase, you are not able to modify them anymore. NGINX status is the HTTP status that is sent back to the client. NGINX context is also a handy thing if you have to pass information between phases. Meaning you have a rewrite phase that writes some information, extracted some information. You need this on a content phase. You can do context. If you are familiar with Android, it is like on an Android system, if you switch the view, you have to assign all the information on a context and then switch the view. Otherwise, these information are lost. That is quite handy if you are passing states or information along with different phases. Is that clear? Other things in the Lua API, NGINX Lua API are the output stuff. We saw NGINX say send headers and so on and so forth. You could do redirects, exit, subrequest. I will come to this later. And co-sockets. I have some examples about these two. NGINX capture. It is a nice thing. What you could do is you could do advanced cookie checking or spam checking. In this example, what we are doing is we are capturing during a process. We do a subrequest within NGINX and waiting for the response. And then we interpret. This is like an HTTP request within an HTTP request. And you can do multiple captures. There is a capability of doing parallel capturing and aggregating the information and sending them back. Another example for this would be if you have a central login server, you could always use NGINX as a proxy. And he is checking against this login server. Are these credentials sent by these clients are valid? For this reason, I let him pass. Otherwise, I deny the access. An example that I used to implement was like MailChimp newsletter. I wanted to have a custom newsletter subscription form. And normally what I have done is like posting on my own website, capturing the stuff with NGINX. Send it to MailChimp server with the API credentials attached. They are actually within NGINX. And then I modify the response that I can differentiate between different errors on the front and side. These are the possibilities that you could do with capture. Another thing is the code socket IP. A thing about NGINX within these phases, everything has to be non-blocking. And in the Lua sphere, normally you have these Lua modules. For example, simple HTTP or Redis implementation, and they use a blocking socket underneath. Meaning if you use this library within NGINX, it's not going to work because if a request comes in and you're blocking while waiting for Redis to give you an answer, every other request has to wait until Redis gives you an answer. And for that reason NGINX request soccer was introduced, meaning giving the user the possibility to do non-blocking socket connection, TCP connection. There are UDP connection as well, but not much space on the slide. But what you have to do is implement a protocol on top of the socket. A nice feature about this is you can use keep alive, meaning that if you're not closing the connection, if a new request comes in, it reuses the keep alive connection. That's a handy thing to get more performance out of it. A good example are the Redis memcache, MySQL, Postgre integration. DNS is actually a UDP implementation. What they look like, it's like you include the Lua Redis clients that's crafted for NGINX, or work within NGINX, and then you do your normal manipulation, reading, writing. Then there are lots of more stuff that you could do with NGINX and Lua. For example, you have LightFret, you can run some work offloads. I've seen people doing a job queue within NGINX. That's a funny thing because you have in it by worker, you could do things with this part. It's a strange thing in LightFret. It actually worked, but it's somehow scary. Shared memory access, meaning that because you have one master process and you have a lot of for worker processes, you could share a caching layer over all these worker processes, meaning that, yeah, and that's a reason why there are some basic caching algorithms within there to help with the cache. Also, Ragex is directly in the core for modifications and stuff. That's basically it, and there's some, yeah, some tiny little things that I never counted, and yeah, they are there, but it's not cluttered. It's quite small, but you have with this capture and co-socket thing, you could do a lot. Meaning that if you combine these two, you get a Batmobile. It's kind of cool. It's like you have NGINX and then you have Lua. You could do a lot of programming, but the problem is you have to build it. It's an own module. You have some people wrote on GitHub, Redis integration. Some people wrote the MySQL integration, and that's where OpenResty kicks in. OpenResty, it's actually a combination of NGINX, Lua. They could also use Lua as well. NGINX Lua, integration Resty, Lua, modules, and C-Lips, meaning aggregating a lot of stuff from the community and packing in the right space that you have a rich environment that you can start with and a normal Makefile. That's very handy. You download OpenResty, Make, Make, Install. What it's doing in the background, it's more or less, first of all, it downloads Lua.chit, compiles it and bets it in NGINX, compiles it with NGINX and NGINX with all the different C-modules, NGINX C-modules and Lua modules, and then place it on the correct system. I think that's not happening at the moment. There's no distro packages. What I normally do, I build distro packages myself, they're not that complicated, and I'm in touch with the creator of the OpenResty project, but he's so busy at Cloudflare. I guess I have to contribute stuff. There's also Docker image existing, but about Resty modules, the Resty modules, as you saw, you have modules coming from NGINX extending the NGINX score, you have modules coming from Lua extending the Lua modules with Lua or Lua C, and the Resty modules, it's always a mix between, I don't know, it's also a little bit Wild World West. The OpenResty project tried to only include modules that are well maintained, that work, that are tested. Actually, the testing of OpenResty is quite impressive. If you ever go on the OpenResty webpage, go to qa.openresty.org. They have heavy testing. That's the reason I never had any issues with OpenResty in our infrastructure. Another thing is, if you have to install things that are not bundled with OpenResty, please use Lua ROX and install it on one specific path. Within NGINX, you then can include this specific path of ROX into your current environment, and then use the installed... It's not very easy, you have to get used to it, but once you use it, you get it going. If you need a BMW, take Kong. Kong is actually an open source project from Mashable Ape. They do a lot of API, localizing, authentication and stuff, and what they build, they call it microservice and API management layer. What they actually took is... because OpenResty has the things that you can build your application on top of it, what they actually done is going a step further. For example, what they included is a central login layer, or JVT, decryption layer, and so on, the central login server, meaning that in an ever-changing environment, you have an API where you can manage your NGINX configuration. That's kind of a neat thing. You can add SSL hosts, you can add hosts, you can add authentication information, you can localize and shut things down. They use a Cassandra cluster on each node, and the information is shared in this. If you need something, and they have a plugable system, meaning that every one or two months, they bring out a new plugin that you can enable or disable a little bit of configuration, and then you have a new API endpoint where you can manage XYZ. That's kind of a cool and neat thing, and gains some attraction in the microservice docker community, because that's a big deal to dynamically low-balance things without changing NGINX configuration. One thing to take away is use ready-built test solutions, looking at OpenResty, if you'd like to be more on the basic side, or Kong, if you'd like more or less production-ready stuff. Take the fast lane in association with Batman and dropping, take the fast lane of crime prevention. Now, to the thing of my talk. What I was thinking, or I often, I'm active in the OpenResty community. We've been in our company for different services, and recently there was a patch about SSL by Lua, meaning that you could secure your SSL connection controlled by Lua. Batman and Robin, they have the same problem. Recently they were tapped, and now they have to introduce new mechanisms to make them safe again on their phone. This is actually a patch. It's not right now in OpenResty. It will be, as I mentioned, the creator of OpenResty is quite busy at Cloudflare. What they newly introduced is SSL certificate by Lua, and SSL certificate by Lua file. What it's actually doing is it's a step before the process even happens, that you could do SSL generation, SSL delivery, OSTP checking, stamp-ling. There's a heavy discussion on issue 331 on the OpenResty project. At the moment there's a branch. You can compile it. What I've done is I took the branch, and I took the OpenResty project. I was doing all the compilation and stuff, and swapped out the OpenResty... Because OpenResty is downloading Lua Engine X module, I swapped out this download with the download of this branch. At the moment you have to patch the engine core as well, because OpenResty downloads the Engine X core, and also the Lua Engine X module. What I had to apply was changing the branch in the download and patching the Engine X core before it's built. That's the thing I had to do. It's not that tricky. The thing is powered by Cloudflare, meaning that they run this since 2013 or 2014. That's definitely battle-testing. That's the thing that's driving their SSL-free for their customer. What they're actually doing is they partner with a CA, I'm not sure if it's a global sign. What they are doing is combining a lot of domains, generating sound certificates, and what they have to do internally if a domain or an SNI only supports SNI. If a SNI request comes in, they look up in which certificate belongs to this list of sounds, and they have to fetch their certificate from the backend. Because they have around 30 to 40 data centers, they have to do this on a distributed way. They do it on the fly, otherwise they have to send a lot of SSL certificates across the globe. Meaning if I hit now here a Cloudflare instance, that's not having my SSL certificate, it gets fetched on the fly. It's kind of a neat thing. I was inspired by this, and I can show you a demo about what I've done. I'll switch to the other screen. My setup is the following. I have a vagrant box running. I have a vagrant box running with NGINX and my Lua code. What I can do is if I request... Wait a second. Now, what I'm actually doing is the following. If the scroll is working, I open an SSL client connection to localhost, giving a server name, meaning a SNI connection, test.driveby.tld, giving a CA file, meaning that certificate authority file for validation, meaning that checking the certificate that I get back, if this CA signed this certificate. What we see here, it's verified, verification chain. What I add dynamically generated by Lua at Saturday 8.40, because of time zones on my machine and in the vagrant box, that's actually the two hours difference. If I hit it again, meaning I could get the same, but a little bit later, now it's 44, and the CN, the common name. If I change it, I take the short, then the information are here. If I pass in another subdomain, subdomain, Glyph 2015, for example, then I get a new SNName, meaning the certificate is valid, but the NGINX core generated on the fly SSL certificate, and then in this returning. The same, I can show you with a fresh Firefox, if you work with Firefox and install a lot of demo drive by TLD. Not working. Ah, now it's okay. The thing is that I, ah, the stupid, if I connected to the Internet, then the thing is I have a DNS mask entry within the DNS mask set up within the network manager. If I'm not connected to the Internet, then DNS mask within network manager, it's not responding, that kind of a bummer. But what I wanted to show is what you could do on the fresh installation of Firefox, you could add the certificate authority, adding a new certificate, and then I can go to every subdomain and I get a valid browser certificate or no warning. Um, is that example clear what I wanted to show? Yeah. Yeah, never mind. The actual code. No. Sorry about that. Not starting at the point. The code in Lua or in NGINX, it looks like this. You, first of all, I include the module REST-SSL. That's an extension coming from this SSL patch. Then I need open SSL for the whole certificate generation, key generation. P-key means like there's a public key cryptography. Then I require a CSR, certificate sign request. Afterwards, I load in the CA, meaning the certificate authority certificate. Load the thing from the file into an object that's actually the certificate. Then I could do manipulation on the thing. The same happens to the key of the authority because with the key of the authority, I have to sign the newly generated key certificate. What I then do, if it's not a SSL server name, meaning if it's not an SNI, if someone connects not with an SNI name, we drop out because of this limitation. Then I start with a new certificate request, meaning the subject. That means the region generated by it. You see here that's actually the date. It's always included. From the SNI name, the SNI name will be the so-called common name. What I then do is generate a new public and private key pair for the new certificate. Afterwards, I assign the key to the certificate and generate a certificate request. The certificate request then from the certificate request, a new certificate is issued. Actually, I had to change this because Firefox is blocking. This is the serial number, meaning that the certificate authority have to increase the number of the serial for each certificate they issue. Firefox is actually checking this. I had to do this morning a little bit of hacking to overcome this issue. Now it works. It's not actually the code on the server. It's not the same. A little bit modified, but it's more or less in the same thing. Then I set the validation time. What I then do is taking this, the naming is wrong, meaning that the client certificate or the certificate is signed with the key and the certificate of the authority. Here, that's actually where the magic happens. I guess in Cloudflare, it looks like this. The thing on the top is the generation and signing is somewhere offloaded. Meaning that the only thing that they do on the fly is capturing the response and certificate on the fly and setting it and caching it on the local server. What we then do is clear the certificate that we had in the engine configuration, set the certificate in the format of the DER, meaning a binary string, converting the binary string to the certificate into a binary, the same goes for the private key of the new generated certificate. That's all the thing that you need to do. At the moment, it's a little bit of tricky to get everything right, but I think in the future you see a lot of application. The first idea was at the moment, if you heard about Let's Encrypt, that's an initiative from Mozilla, Akamai, Cisco, they build an automated system to get you free SSL certificate. My initial idea was going to an upstream server to the Mozilla server, so I was writing a valid public certificate reissuing on the fly, meaning that every time someone hits the server, I get an instantly valid certificate. I never have to think about reissuing or stuff. Everything I can write down in Lua. If someone hits the first time, he has to wait, I don't know, two seconds off or something, or I don't know, a minute, then the request responds, but the request afterwards, it's okay. Or I can do a cron job that is doing that every night. That was the basic idea. At the moment, we are implementing this for our web hosting customers, and during this work, we saw that the API and the specification is changing too radically that I could adopt these changes all the time before the talk. I started with the adoption and I ran into the problem that actually the OpenSSL project had a lot of issues. First of all, I had to fix a lot of C code issues within this project before I could go on, and that's why we have only a simplified version of that. Nina's say, securing in an ever-changing system is hard, but it's also possible that you can introduce security layers or dynamic stuff on the web server. At least you can try, but don't produce snakegoys. One of the pitfalls that I learned during my years and also preparing for this presentation, if you do a lot of engineering configuration modification, please always run service engineering conflict test. I had sometimes the issue that I thought I had changed something, made a reload, but actually if you do a reload and you have conflict issues, engineering is doing nothing and says nothing about it. That's the reason why I always do service engineering conflict test and then a reload. The same happens if you do a lot of conflict changes. I have a when change, which is a little small Python script. If the file changes, I access H into the server and do a conflict test and reload the server. I also fell into this trap, like the escape characters in Lua. It's not very intuitive, but better than Ruby. As I mentioned, always use Lua by file. It's easier to handle with and no escaping. If you'd like to read post-words in an access phase, meaning like one of the rewrite or access phase, then you have to do an additional request head body, meaning that in the face of rewrite and access, NGINX has not yet loaded the body or the post information. Tips and trick debugging in an NGINX environment, especially in a dynamic NGINX environment with Lua. It's not easy or actually hard. I was burned many times, like little pieces that, yeah. One of the things that I often do is like adding debug information to the header information, meaning aggregating information. You could log all the information, but the thing is on the log sometimes on a business, it happens so much thing. What you could do, I have special tags and if I specify this hidden tag, it gets me like debug output on a live server. So now that's just kind of cool neat trick. What you actually also could do is like generating a tracer, tracing ID, and then piping, you know, or giving this tracer ID as a header parameter parameter to the web server and then going through your logs. That's also a possibility. But I found the NGINX header dump the easier. And if you're looking for code or useful example, actually the OpenRESI project has very good tests. And for this SSL talk, I had to look many times in the code because it's at the moment is not yet documented. And there are some conditions where the documentation lag. Yeah. And I'm at the end of my talk. Thank you for listening. Are there any questions, regards, or concern about the stupidity about this project? Yeah. What was the future ratio on the server that you have in terms of the problem? No, what I do now, it's kind of a silly example. I generate for each request a new certificate. It's kind of stupid. It's not, it's not. But what you normally could do is offloading money. As I mentioned, people doing worker demons within NGINX, you could do the same. It's like if the first request hits the server, you could offload this to worker. Then it's in its own worker and it's not blocking the other requests. Yeah. But you could do, or... Okay. No, they run under the NGINX user. Meaning that I had to give well permission or at least for the CEA, for the CEA certificate I had to give permission to the NGINX user. Yeah. But normally you do this in a secure backend and then you get, yeah. So I'm thinking that in principle, we also for other users, just for the sake of the NGINX user, we don't see a certificate of other clients. Yeah. Where the case is attached or back into the client, how much the NGINX certificate. Yeah. So, for the reason I need the private key to sign, otherwise... No, no, no. My use, yes, this use case, yes. But the use case of LatinGrip, for example, I go into an HGT backend and they give me an SSS certificate. I could also program that I log into the complicated website and do web scraping of global sign or something else that's not limited to this because we flew out of the capabilities. Yeah. But for this purpose, because I'm not a CEA, I have to trick the browser to accept me as a CEA. Yeah. The initial idea was going to an official CEA where they have a REST API where I can issue. Yeah. Okay. So far, Bova Jaskity. And thanks to agent CH, that's the guy behind all the RESTy stuff and CloudFare for the contribution. And hopefully we'll see some of you in the OpenRESTy and GNX Lua community playing around. Thank you so much. Thank you.
 Welcome you to my talk, Drive by SSL Certificate Creation with Nginx.  During this talk you will see what I mean by this, what I have in mind about an IDE,  what Nginx is capable of.  First of all about me, I'm Dominic, working for a hosting company back in Switzerland.  We are compared to German market, quite small, but we are in the top five of Switzerland, top four.  Everything is factor 10 smaller in Switzerland.  I'm mostly developer there doing a lot of backend security, that's why I sell them,  have to pick something on the website, that's part of my team, but not mine.  You find me on Twitter, it's also my nickname.  If you have questions, please ask right away and don't hesitate to wait at the end of the talk.  I came across a capability of Nginx when Cloudflare in 2004 or 2014 announced that they give away free SSL certificate for all.  They explain what they are doing and I followed along the Nginx community.  We used the Nginx and the scriptable capabilities and I thought it would be nice having such a feature.  Recently there was a patch contributed by Cloudflare and this talk is about this patch and the possibility.  This year is 2015 and if Frostconn has a SSL certificate that is only for this specific domain,  it has no SAN or Y-Cast certificate, they have reissue certificate every year.  The same goes for if you are a pass provider, good example, Slack.  Maybe some of you know Slack as a chat program.  Each company has its own subdomain and I guess what they provide is a Y-Cast certificate.  What would be really nice is having a certificate on the fly generation per subdomain.  That is the whole idea behind it and I will introduce you to steps to build such a system.  If it is reasonable, I don't know.  It is more a technical challenge than a real use case but it could be.  In the example of Cloudflare, it is a good example.  About Nginx, written and spelled, it is a different thing.  Nginx itself, if you are familiar, is more on the developer side.  Who is responsible for server stuff and deploying?  It is like the majority.  I will skip this quickly.  You know the second most used web server.  His properties are high performance, low memory, footprint.  He is a web server, he is also proxy, load balancer, you can do caching.  Since the new version 1.9, you can also do TCP proxy.  That is a new feature that they introduced.  They came along with H8 proxy.  The whole system is event driven and has a sort of business design.  It started in 2002, went public in 2004 and now in 2014, Nginx.com,  the company was built to have a professional support.  The open source domain is Nginx.org, the commercial one is Nginx.com.  Think of Nginx.  It is like Batman, he can do a lot of things and he is great and he is established.  He has all the good benefits and features.  What makes Batman really, really a superhero is that he has a lot of accessories that he can use.  The same applies to Nginx.  Nginx has quite a small core.  Nginx is capable of doing HTTP stuff, SSL stuff,  but not much with a little bit of logging and a little bit of load balancing.  In the ecosystem around Nginx are the accessories.  People build in modules written in C to extend the capabilities or the belt of Nginx.  A few examples, it is really wild west.  If you look at the third-party modules, you see abandoned stuff,  you see stuff like video thumbnail extractor or upload process.  That is actually quite often used or a circle gift that is generating gifts on demand and resizing them.  Drizzle is also a thing that is often used.  It is a myoscal possibility to talk to myoscal within the Nginx scope and talk to Rettis as well.  Another thing that I can...  Here is a little example.  That is actually two different third-party modules.  This one is the echo module.  It is not in the core and returns the value.  The other one is the Rettis extension.  The first one was shitty, they built another one.  With this example, you can set different values directly to Rettis or read it  and then do some dynamic low-balancing or configuration stuff.  Another thing is Nginx because it is event-driven.  It has different phases or different stages during the start of an HTTP request  and handling and finishing it off.  There are some steps in between, I left them out because they are mostly pre-imposed stuff, configuration, read and write.  The important one are the rewrite phase.  In this phase, you could apply different rewrites, how Nginx is behaving.  The access phase is normal, what you know about HTTP authentication, try files,  it is looking at the file system content,  it is delivering content out of different file system database, different backend.  Log is after the process is done, it is locked.  Then you can apply filters, if the request is at the stage where you can push back to the client,  you can do filtering of the header and the body.  An example you can see here is the echo example.  It is in the content phase.  The set is actually in the rewrite phase.  If you write a C extension, you have to decide in which event you hook into the Nginx core  and what are the changes you are applying.  Because there are different modules, you can sometimes mess up stuff from other people.  Then there would be nothing without Robin.  In my talk, it is Lua.  Lua is a programming language.  Lua is fast by design, not by design, because it has a very, very lightweight design.  A good example is learn Lua in 15 minutes.  There is a web page.  If you are familiar with programming, within 15 minutes, I can assure you are capable.  The syntax is so easy.  Invitable is one of the features that Lua is famous for.  For example, Far Cry and a lot of game engines have the possibility to write modifications in Lua.  They expose different modifications possibilities.  You see embeddability also in the automotive industry, where you have simulations and then you hook in.  Lua is used in network switches as well to packet filtering and on-the-fly giving the scriptable language possibility to hardware boxes.  That is actually from the 15 minutes.  It is not the whole part, but you see the syntax is quite easy.  You have no types.  Numbers are always doubled.  You have double and single quotes.  You have multi-line.  If you need something, it goes to garbage collection.  You have loops and if-else, and there are some other constructs.  What do you know from different programming language hashes or arrays?  They have tables.  They have no class support, like object-orientated.  You can do stuff, but no rich functionality.  It is mostly procedural programming.  In the UND, Lua has a lot of accessories.  Lua comes with a rich variety of different modules that can be written in Lua or in C itself.  What I mean by this is a good example for this is an MVC.  It is a web framework that is entirely written in Lua that you can use.  In the Lua file system, it has some C extension to hook into the file system itself.  It is like a C-json implementation exposed to Lua.  What you also have in the Lua space is LuaRocks.  LuaRocks is a different package manager in every language.  It is quite limited compared to other tools that I have seen.  Most of the time it works because people are not depending much on each other's modules.  The JavaScript community has two lines of code, and everybody inherits these two lines.  Then you need circle dependency resolutions and conflict resolutions.  Lua is capable of compiling C-based code.  If someone ships C-embedded Lua functionality with LuaRocks,  if you have the right dependencies on your OS operating system, then you can install this module.  As always, there is an awesome language GitHub account where they have a lot of summaries about different tools and stuff about this ecosystem.  Then there is LuaChit because LuaChit was started as a project because the core or the language itself is so small.  A guy, I can't remember his name, started the LuaChit project.  Now you have the possibility to either use Lua or LuaChit as the name for testing time compilers.  If you run this code in a continuous way, it optimizes on the fly a lot of code.  This is very useful in a web server where you would like to apply web application firewalls.  These rules have to be run over and over and over again.  They get hot and they are optimized and compiled to machine code.  They say 10 to 100 times faster than the normal Lua implementation itself.  One of the main drivers behind it is Cloudflare as well because now they ship around 10% of the traffic worldwide.  If they could optimize LuaChit by 5 or 10%, they gain a lot of capacity.  Think of LuaChit as robbing on steroids.  The nice part about LuaChit is that it compiles on nearly all systems and all process architectures.  It is binary compatible. It means that you can switch from Lua to LuaChit.  It has the same syntax and it works.  It works on a PS3 or PS4 which is quite nice.  One thing to remember or take away is that Robbin and Batman are two heroes.  They are both hands with a community and tools behind it.  That is not enough. You have NGINX and Lua.  Now we are at NGINX Lua or HTTP Lua module.  That is the package name on the documentation and the GitHub name.  It is confusing if you are searching for the thing.  It means embedding the power of Lua into NGINX.  Here is a small example. You would like to get the content.  Content by Lua means within these brackets you write normal Lua code.  It gets interpreted and it is printing out HALO work.  That is the integration.  You have a lot of these. You have init by Lua, init worker.  In it it runs once when the server starts when a new master comes up.  In init worker is when the master process forks a new worker process.  You have set by Lua. In the previous example you have seen set as a directive of NGINX.  Now you can dynamically set things in the scope of NGINX.  Rewrite access content, log header filter and body filter.  These are the phases where you can hook into NGINX and modify it by yourself.  It gains some attraction because you saw this wild west of C modules.  People started migrating their code into Lua because the burden of maintaining a C extension  in an ever-changing NGINX code was too difficult or too time-consuming compared to Lua.  That is always the same or more or less the same.  That is the reason why nowadays a lot of modules and stuff are moving to the Lua world.  All stuff are ripped out or is dying off.  Another thing that is possible is Lua file.  That means in the difference here you write this directly in your NGINX configuration.  Here you can specify a specific Lua file.  I always would recommend this way because you can see the burden that you have with double and single quotes within the NGINX configuration.  There is a fix on the way but it needs some time.  There is also another benefit.  The Lua code normally if it is embedded by Lua file then it is compiled.  Meaning the Lua cheat compiles it into an optimized way.  Then it is always interpreted from this optimized way.  During development every single character that you change in the Lua file you have to reload NGINX to get updates into your NGINX.  That is one of the tricks that you can do.  It is that you during development disable the cache and then you can edit right away.  You always see the new updates if a new request hits this Lua specific directory.  Another thing is these are the parts where the Lua NGINX Lua API comes in.  Within the scope of Lua you have to talk somehow to the web server.  Meaning that you need some API to talk to the outside world of NGINX.  The first thing is NGINX ARC.  It is something like you have on the console, on the bash script or in every other language.  It is like parameters passed in.  There is some situation where it is happening.  For example set by Lua.  You can pass in parameters or in the filters.  But that is rarely the case that you work with.  Most of the time you work with NGINX VARs.  A good example is outside.  VARs always has to be declared outside of the Lua scope.  Meaning that during the lifecycle of a request you could change the values.  This is very handy if you have, for example, at this point, going to a proxy or to a proxy upstream  and define the variable of the proxy.  In the Lua code you could do health checks or you could do load balancing or intelligently load balancing algorithm.  If you set the variable in the right phase, then NGINX is performing proxy stuff with this modified variable.  Another thing is NGINX header.  It is used to read and modify headers depending on the stage.  If you are passed the lock phase, you are not able to modify them anymore.  NGINX status is the HTTP status that is sent back to the client.  NGINX context is also a handy thing if you have to pass information between phases.  Meaning you have a rewrite phase that writes some information, extracted some information.  You need this on a content phase.  You can do context.  If you are familiar with Android, it is like on an Android system, if you switch the view,  you have to assign all the information on a context and then switch the view.  Otherwise, these information are lost.  That is quite handy if you are passing states or information along with different phases.  Is that clear?  Other things in the Lua API, NGINX Lua API are the output stuff.  We saw NGINX say send headers and so on and so forth.  You could do redirects, exit, subrequest.  I will come to this later.  And co-sockets.  I have some examples about these two.  NGINX capture.  It is a nice thing.  What you could do is you could do advanced cookie checking or spam checking.  In this example, what we are doing is we are capturing during a process.  We do a subrequest within NGINX and waiting for the response.  And then we interpret.  This is like an HTTP request within an HTTP request.  And you can do multiple captures.  There is a capability of doing parallel capturing and aggregating the information and sending them back.  Another example for this would be if you have a central login server, you could always use NGINX as a proxy.  And he is checking against this login server.  Are these credentials sent by these clients are valid?  For this reason, I let him pass.  Otherwise, I deny the access.  An example that I used to implement was like MailChimp newsletter.  I wanted to have a custom newsletter subscription form.  And normally what I have done is like posting on my own website, capturing the stuff with NGINX.  Send it to MailChimp server with the API credentials attached.  They are actually within NGINX.  And then I modify the response that I can differentiate between different errors on the front and side.  These are the possibilities that you could do with capture.  Another thing is the code socket IP.  A thing about NGINX within these phases, everything has to be non-blocking.  And in the Lua sphere, normally you have these Lua modules.  For example, simple HTTP or Redis implementation, and they use a blocking socket underneath.  Meaning if you use this library within NGINX, it's not going to work because if a request comes in  and you're blocking while waiting for Redis to give you an answer,  every other request has to wait until Redis gives you an answer.  And for that reason NGINX request soccer was introduced,  meaning giving the user the possibility to do non-blocking socket connection, TCP connection.  There are UDP connection as well, but not much space on the slide.  But what you have to do is implement a protocol on top of the socket.  A nice feature about this is you can use keep alive, meaning that if you're not closing the connection,  if a new request comes in, it reuses the keep alive connection.  That's a handy thing to get more performance out of it.  A good example are the Redis memcache, MySQL, Postgre integration.  DNS is actually a UDP implementation.  What they look like, it's like you include the Lua Redis clients that's crafted for NGINX,  or work within NGINX, and then you do your normal manipulation, reading, writing.  Then there are lots of more stuff that you could do with NGINX and Lua.  For example, you have LightFret, you can run some work offloads.  I've seen people doing a job queue within NGINX.  That's a funny thing because you have in it by worker, you could do things with this part.  It's a strange thing in LightFret.  It actually worked, but it's somehow scary.  Shared memory access, meaning that because you have one master process  and you have a lot of for worker processes, you could share a caching layer  over all these worker processes, meaning that, yeah,  and that's a reason why there are some basic caching algorithms within there to help with the cache.  Also, Ragex is directly in the core for modifications and stuff.  That's basically it, and there's some, yeah, some tiny little things that I never counted,  and yeah, they are there, but it's not cluttered.  It's quite small, but you have with this capture and co-socket thing, you could do a lot.  Meaning that if you combine these two, you get a Batmobile.  It's kind of cool.  It's like you have NGINX and then you have Lua.  You could do a lot of programming, but the problem is you have to build it.  It's an own module.  You have some people wrote on GitHub, Redis integration.  Some people wrote the MySQL integration, and that's where OpenResty kicks in.  OpenResty, it's actually a combination of NGINX, Lua.  They could also use Lua as well.  NGINX Lua, integration Resty, Lua, modules, and C-Lips,  meaning aggregating a lot of stuff from the community and packing in the right space  that you have a rich environment that you can start with and a normal Makefile.  That's very handy.  You download OpenResty, Make, Make, Install.  What it's doing in the background, it's more or less, first of all, it downloads Lua.chit,  compiles it and bets it in NGINX, compiles it with NGINX and NGINX with all the different C-modules,  NGINX C-modules and Lua modules, and then place it on the correct system.  I think that's not happening at the moment.  There's no distro packages.  What I normally do, I build distro packages myself, they're not that complicated,  and I'm in touch with the creator of the OpenResty project, but he's so busy at Cloudflare.  I guess I have to contribute stuff.  There's also Docker image existing, but about Resty modules,  the Resty modules, as you saw, you have modules coming from NGINX extending the NGINX score,  you have modules coming from Lua extending the Lua modules with Lua or Lua C,  and the Resty modules, it's always a mix between, I don't know, it's also a little bit Wild World West.  The OpenResty project tried to only include modules that are well maintained, that work, that are tested.  Actually, the testing of OpenResty is quite impressive.  If you ever go on the OpenResty webpage, go to qa.openresty.org.  They have heavy testing.  That's the reason I never had any issues with OpenResty in our infrastructure.  Another thing is, if you have to install things that are not bundled with OpenResty,  please use Lua ROX and install it on one specific path.  Within NGINX, you then can include this specific path of ROX into your current environment,  and then use the installed...  It's not very easy, you have to get used to it, but once you use it, you get it going.  If you need a BMW, take Kong.  Kong is actually an open source project from Mashable Ape.  They do a lot of API, localizing, authentication and stuff,  and what they build, they call it microservice and API management layer.  What they actually took is... because OpenResty has the things that you can build your application on top of it,  what they actually done is going a step further.  For example, what they included is a central login layer, or JVT, decryption layer,  and so on, the central login server, meaning that in an ever-changing environment,  you have an API where you can manage your NGINX configuration.  That's kind of a neat thing.  You can add SSL hosts, you can add hosts, you can add authentication information,  you can localize and shut things down.  They use a Cassandra cluster on each node, and the information is shared in this.  If you need something, and they have a plugable system,  meaning that every one or two months, they bring out a new plugin that you can enable or disable  a little bit of configuration, and then you have a new API endpoint where you can manage XYZ.  That's kind of a cool and neat thing, and gains some attraction in the microservice docker community,  because that's a big deal to dynamically low-balance things without changing NGINX configuration.  One thing to take away is use ready-built test solutions, looking at OpenResty,  if you'd like to be more on the basic side, or Kong, if you'd like more or less production-ready stuff.  Take the fast lane in association with Batman and dropping, take the fast lane of crime prevention.  Now, to the thing of my talk.  What I was thinking, or I often, I'm active in the OpenResty community.  We've been in our company for different services, and recently there was a patch about SSL by Lua,  meaning that you could secure your SSL connection controlled by Lua.  Batman and Robin, they have the same problem.  Recently they were tapped, and now they have to introduce new mechanisms to make them safe again on their phone.  This is actually a patch. It's not right now in OpenResty.  It will be, as I mentioned, the creator of OpenResty is quite busy at Cloudflare.  What they newly introduced is SSL certificate by Lua, and SSL certificate by Lua file.  What it's actually doing is it's a step before the process even happens,  that you could do SSL generation, SSL delivery, OSTP checking, stamp-ling.  There's a heavy discussion on issue 331 on the OpenResty project.  At the moment there's a branch. You can compile it.  What I've done is I took the branch, and I took the OpenResty project.  I was doing all the compilation and stuff, and swapped out the OpenResty...  Because OpenResty is downloading Lua Engine X module, I swapped out this download with the download of this branch.  At the moment you have to patch the engine core as well, because OpenResty downloads the Engine X core,  and also the Lua Engine X module.  What I had to apply was changing the branch in the download and patching the Engine X core before it's built.  That's the thing I had to do. It's not that tricky.  The thing is powered by Cloudflare, meaning that they run this since 2013 or 2014.  That's definitely battle-testing. That's the thing that's driving their SSL-free for their customer.  What they're actually doing is they partner with a CA, I'm not sure if it's a global sign.  What they are doing is combining a lot of domains, generating sound certificates,  and what they have to do internally if a domain or an SNI only supports SNI.  If a SNI request comes in, they look up in which certificate belongs to this list of sounds,  and they have to fetch their certificate from the backend.  Because they have around 30 to 40 data centers, they have to do this on a distributed way.  They do it on the fly, otherwise they have to send a lot of SSL certificates across the globe.  Meaning if I hit now here a Cloudflare instance, that's not having my SSL certificate,  it gets fetched on the fly. It's kind of a neat thing.  I was inspired by this, and I can show you a demo about what I've done.  I'll switch to the other screen. My setup is the following. I have a vagrant box running.  I have a vagrant box running with NGINX and my Lua code.  What I can do is if I request... Wait a second.  Now, what I'm actually doing is the following.  If the scroll is working, I open an SSL client connection to localhost,  giving a server name, meaning a SNI connection, test.driveby.tld,  giving a CA file, meaning that certificate authority file for validation,  meaning that checking the certificate that I get back,  if this CA signed this certificate.  What we see here, it's verified, verification chain.  What I add dynamically generated by Lua at Saturday 8.40,  because of time zones on my machine and in the vagrant box,  that's actually the two hours difference.  If I hit it again, meaning I could get the same, but a little bit later,  now it's 44, and the CN, the common name. If I change it, I take the short,  then the information are here.  If I pass in another subdomain, subdomain, Glyph 2015, for example,  then I get a new SNName, meaning the certificate is valid,  but the NGINX core generated on the fly SSL certificate, and then in this returning.  The same, I can show you with a fresh Firefox,  if you work with Firefox and install a lot of demo drive by TLD.  Not working.  Ah, now it's okay.  The thing is that I, ah, the stupid, if I connected to the Internet,  then the thing is I have a DNS mask entry within the DNS mask set up within the network manager.  If I'm not connected to the Internet, then DNS mask within network manager,  it's not responding, that kind of a bummer.  But what I wanted to show is what you could do on the fresh installation of Firefox,  you could add the certificate authority, adding a new certificate,  and then I can go to every subdomain and I get a valid browser certificate or no warning.  Um, is that example clear what I wanted to show?  Yeah.  Yeah, never mind.  The actual code.  No.  Sorry about that.  Not starting at the point.  The code in Lua or in NGINX, it looks like this.  You, first of all, I include the module REST-SSL.  That's an extension coming from this SSL patch.  Then I need open SSL for the whole certificate generation, key generation.  P-key means like there's a public key cryptography.  Then I require a CSR, certificate sign request.  Afterwards, I load in the CA, meaning the certificate authority certificate.  Load the thing from the file into an object that's actually the certificate.  Then I could do manipulation on the thing.  The same happens to the key of the authority because with the key of the authority,  I have to sign the newly generated key certificate.  What I then do, if it's not a SSL server name, meaning if it's not an SNI,  if someone connects not with an SNI name, we drop out because of this limitation.  Then I start with a new certificate request, meaning the subject.  That means the region generated by it.  You see here that's actually the date. It's always included.  From the SNI name, the SNI name will be the so-called common name.  What I then do is generate a new public and private key pair for the new certificate.  Afterwards, I assign the key to the certificate and generate a certificate request.  The certificate request then from the certificate request, a new certificate is issued.  Actually, I had to change this because Firefox is blocking.  This is the serial number, meaning that the certificate authority have to increase  the number of the serial for each certificate they issue.  Firefox is actually checking this.  I had to do this morning a little bit of hacking to overcome this issue.  Now it works.  It's not actually the code on the server.  It's not the same.  A little bit modified, but it's more or less in the same thing.  Then I set the validation time.  What I then do is taking this, the naming is wrong, meaning that the client certificate  or the certificate is signed with the key and the certificate of the authority.  Here, that's actually where the magic happens.  I guess in Cloudflare, it looks like this.  The thing on the top is the generation and signing is somewhere offloaded.  Meaning that the only thing that they do on the fly is capturing the response and certificate on the fly  and setting it and caching it on the local server.  What we then do is clear the certificate that we had in the engine configuration,  set the certificate in the format of the DER, meaning a binary string,  converting the binary string to the certificate into a binary,  the same goes for the private key of the new generated certificate.  That's all the thing that you need to do.  At the moment, it's a little bit of tricky to get everything right,  but I think in the future you see a lot of application.  The first idea was at the moment, if you heard about Let's Encrypt,  that's an initiative from Mozilla, Akamai, Cisco,  they build an automated system to get you free SSL certificate.  My initial idea was going to an upstream server to the Mozilla server,  so I was writing a valid public certificate reissuing on the fly,  meaning that every time someone hits the server, I get an instantly valid certificate.  I never have to think about reissuing or stuff.  Everything I can write down in Lua.  If someone hits the first time, he has to wait, I don't know, two seconds off or something,  or I don't know, a minute, then the request responds,  but the request afterwards, it's okay.  Or I can do a cron job that is doing that every night.  That was the basic idea.  At the moment, we are implementing this for our web hosting customers,  and during this work, we saw that the API and the specification is changing too radically  that I could adopt these changes all the time before the talk.  I started with the adoption and I ran into the problem that actually the OpenSSL project  had a lot of issues.  First of all, I had to fix a lot of C code issues within this project before I could go on,  and that's why we have only a simplified version of that.  Nina's say, securing in an ever-changing system is hard,  but it's also possible that you can introduce security layers or dynamic stuff on the web server.  At least you can try, but don't produce snakegoys.  One of the pitfalls that I learned during my years and also preparing for this presentation,  if you do a lot of engineering configuration modification, please always run service engineering conflict test.  I had sometimes the issue that I thought I had changed something, made a reload,  but actually if you do a reload and you have conflict issues, engineering is doing nothing  and says nothing about it.  That's the reason why I always do service engineering conflict test and then a reload.  The same happens if you do a lot of conflict changes.  I have a when change, which is a little small Python script.  If the file changes, I access H into the server and do a conflict test and reload the server.  I also fell into this trap, like the escape characters in Lua.  It's not very intuitive, but better than Ruby.  As I mentioned, always use Lua by file.  It's easier to handle with and no escaping.  If you'd like to read post-words in an access phase, meaning like one of the rewrite or access phase,  then you have to do an additional request head body, meaning that in the face of rewrite and access,  NGINX has not yet loaded the body or the post information.  Tips and trick debugging in an NGINX environment, especially in a dynamic NGINX environment with Lua.  It's not easy or actually hard.  I was burned many times, like little pieces that, yeah.  One of the things that I often do is like adding debug information to the header information,  meaning aggregating information.  You could log all the information, but the thing is on the log sometimes on a business,  it happens so much thing.  What you could do, I have special tags and if I specify this hidden tag,  it gets me like debug output on a live server.  So now that's just kind of cool neat trick.  What you actually also could do is like generating a tracer, tracing ID,  and then piping, you know, or giving this tracer ID as a header parameter parameter to the web server  and then going through your logs.  That's also a possibility.  But I found the NGINX header dump the easier.  And if you're looking for code or useful example,  actually the OpenRESI project has very good tests.  And for this SSL talk, I had to look many times in the code  because it's at the moment is not yet documented.  And there are some conditions where the documentation lag.  Yeah.  And I'm at the end of my talk.  Thank you for listening.  Are there any questions, regards, or concern about the stupidity about this project?  Yeah.  What was the future ratio on the server that you have in terms of the problem?  No, what I do now, it's kind of a silly example.  I generate for each request a new certificate.  It's kind of stupid.  It's not, it's not.  But what you normally could do is offloading money.  As I mentioned, people doing worker demons within NGINX, you could do the same.  It's like if the first request hits the server, you could offload this to worker.  Then it's in its own worker and it's not blocking the other requests.  Yeah.  But you could do, or...  Okay.  No, they run under the NGINX user.  Meaning that I had to give well permission or at least for the CEA,  for the CEA certificate I had to give permission to the NGINX user.  Yeah.  But normally you do this in a secure backend and then you get, yeah.  So I'm thinking that in principle, we also for other users,  just for the sake of the NGINX user, we don't see a certificate of other clients.  Yeah.  Where the case is attached or back into the client, how much the NGINX certificate.  Yeah.  So, for the reason I need the private key to sign, otherwise...  No, no, no.  My use, yes, this use case, yes.  But the use case of LatinGrip, for example, I go into an HGT backend  and they give me an SSS certificate.  I could also program that I log into the complicated website and do web scraping of global sign  or something else that's not limited to this because we flew out of the capabilities.  Yeah.  But for this purpose, because I'm not a CEA, I have to trick the browser to accept me as a CEA.  Yeah.  The initial idea was going to an official CEA where they have a REST API where I can issue.  Yeah.  Okay.  So far, Bova Jaskity.  And thanks to agent CH, that's the guy behind all the RESTy stuff and CloudFare for the contribution.  And hopefully we'll see some of you in the OpenRESTy and GNX Lua community playing around.  Thank you so much.  Thank you.
Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_0_71.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_72_515.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_516_599.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_600_2243.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_2244_2855.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_2856_4163.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_4164_4427.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_4428_7079.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_7080_7679.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_7680_7775.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_7776_8819.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_8820_11195.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_11196_11459.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_11460_11531.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_11532_11603.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_11604_11651.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_11652_11843.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_11844_11999.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12000_12203.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12204_12263.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12264_12455.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12456_12539.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12540_12587.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12588_12659.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12660_12743.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12744_12803.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_12804_13055.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_13056_13979.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_13980_14183.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_14184_17051.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_17052_25259.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_25260_26819.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_26820_27719.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_27720_29399.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_29400_29591.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_29592_29867.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_29868_30155.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_30156_30239.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_30240_30383.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_30384_30443.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_30444_31715.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_31716_31859.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_31860_33155.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_33156_37211.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_37212_38087.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_38088_40823.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_40824_43811.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_43812_46499.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_46500_47651.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_47652_51899.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_51900_55031.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_55032_55271.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_55272_55883.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_55884_57131.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_57132_57335.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_57336_57623.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_57624_58367.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_58368_59039.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_59040_62111.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_62112_62603.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_62604_63227.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_63228_63575.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_63576_65327.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_65328_65735.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_65736_66743.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_66744_66851.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_66852_67379.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_67380_69563.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_69564_70223.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_70224_70775.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_70776_70859.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_70860_70919.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_70920_71051.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_71052_71171.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_71172_71447.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_71448_71555.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_71556_71927.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_71928_72695.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_72696_73847.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_73848_73979.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_73980_74027.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_74028_75299.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_75300_75971.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_75972_76343.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_76344_80039.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_80040_80759.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_80760_81527.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_81528_91127.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_91128_91247.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_91248_91451.png Dominic_Luchinger__Drive-by_SSL_certificate_creation_with_nginx_91452_91692.png
Welcome you to my talk, Drive by SSL Certificate Creation with Nginx. During this talk you will see what I mean by this, what I have in mind about an IDE, what Nginx is capable of. First of all about me, I'm Dominic, working for a hosting company back in Switzerland. We are compared to German market, quite small, but we are in the top five of Switzerland, top four. Everything is factor 10 smaller in Switzerland. I'm mostly developer there doing a lot of backend security, that's why I sell them, have to pick something on the website, that's part of my team, but not mine. You find me on Twitter, it's also my nickname. If you have questions, please ask right away and don't hesitate to wait at the end of the talk. I came across a capability of Nginx when Cloudflare in 2004 or 2014 announced that they give away free SSL certificate for all. They explain what they are doing and I followed along the Nginx community. We used the Nginx and the scriptable capabilities and I thought it would be nice having such a feature. Recently there was a patch contributed by Cloudflare and this talk is about this patch and the possibility. This year is 2015 and if Frostconn has a SSL certificate that is only for this specific domain, it has no SAN or Y-Cast certificate, they have reissue certificate every year. The same goes for if you are a pass provider, good example, Slack. Maybe some of you know Slack as a chat program. Each company has its own subdomain and I guess what they provide is a Y-Cast certificate. What would be really nice is having a certificate on the fly generation per subdomain. That is the whole idea behind it and I will introduce you to steps to build such a system. If it is reasonable, I don't know. It is more a technical challenge than a real use case but it could be. In the example of Cloudflare, it is a good example. About Nginx, written and spelled, it is a different thing. Nginx itself, if you are familiar, is more on the developer side. Who is responsible for server stuff and deploying? It is like the majority. I will skip this quickly. You know the second most used web server. His properties are high performance, low memory, footprint. He is a web server, he is also proxy, load balancer, you can do caching. Since the new version 1.9, you can also do TCP proxy. That is a new feature that they introduced. They came along with H8 proxy. The whole system is event driven and has a sort of business design. It started in 2002, went public in 2004 and now in 2014, Nginx.com, the company was built to have a professional support. The open source domain is Nginx.org, the commercial one is Nginx.com. Think of Nginx. It is like Batman, he can do a lot of things and he is great and he is established. He has all the good benefits and features. What makes Batman really, really a superhero is that he has a lot of accessories that he can use. The same applies to Nginx. Nginx has quite a small core. Nginx is capable of doing HTTP stuff, SSL stuff, but not much with a little bit of logging and a little bit of load balancing. In the ecosystem around Nginx are the accessories. People build in modules written in C to extend the capabilities or the belt of Nginx. A few examples, it is really wild west. If you look at the third-party modules, you see abandoned stuff, you see stuff like video thumbnail extractor or upload process. That is actually quite often used or a circle gift that is generating gifts on demand and resizing them. Drizzle is also a thing that is often used. It is a myoscal possibility to talk to myoscal within the Nginx scope and talk to Rettis as well. Another thing that I can... Here is a little example. That is actually two different third-party modules. This one is the echo module. It is not in the core and returns the value. The other one is the Rettis extension. The first one was shitty, they built another one. With this example, you can set different values directly to Rettis or read it and then do some dynamic low-balancing or configuration stuff. Another thing is Nginx because it is event-driven. It has different phases or different stages during the start of an HTTP request and handling and finishing it off. There are some steps in between, I left them out because they are mostly pre-imposed stuff, configuration, read and write. The important one are the rewrite phase. In this phase, you could apply different rewrites, how Nginx is behaving. The access phase is normal, what you know about HTTP authentication, try files, it is looking at the file system content, it is delivering content out of different file system database, different backend. Log is after the process is done, it is locked. Then you can apply filters, if the request is at the stage where you can push back to the client, you can do filtering of the header and the body. An example you can see here is the echo example. It is in the content phase. The set is actually in the rewrite phase. If you write a C extension, you have to decide in which event you hook into the Nginx core and what are the changes you are applying. Because there are different modules, you can sometimes mess up stuff from other people. Then there would be nothing without Robin. In my talk, it is Lua. Lua is a programming language. Lua is fast by design, not by design, because it has a very, very lightweight design. A good example is learn Lua in 15 minutes. There is a web page. If you are familiar with programming, within 15 minutes, I can assure you are capable. The syntax is so easy. Invitable is one of the features that Lua is famous for. For example, Far Cry and a lot of game engines have the possibility to write modifications in Lua. They expose different modifications possibilities. You see embeddability also in the automotive industry, where you have simulations and then you hook in. Lua is used in network switches as well to packet filtering and on-the-fly giving the scriptable language possibility to hardware boxes. That is actually from the 15 minutes. It is not the whole part, but you see the syntax is quite easy. You have no types. Numbers are always doubled. You have double and single quotes. You have multi-line. If you need something, it goes to garbage collection. You have loops and if-else, and there are some other constructs. What do you know from different programming language hashes or arrays? They have tables. They have no class support, like object-orientated. You can do stuff, but no rich functionality. It is mostly procedural programming. In the UND, Lua has a lot of accessories. Lua comes with a rich variety of different modules that can be written in Lua or in C itself. What I mean by this is a good example for this is an MVC. It is a web framework that is entirely written in Lua that you can use. In the Lua file system, it has some C extension to hook into the file system itself. It is like a C-json implementation exposed to Lua. What you also have in the Lua space is LuaRocks. LuaRocks is a different package manager in every language. It is quite limited compared to other tools that I have seen. Most of the time it works because people are not depending much on each other's modules. The JavaScript community has two lines of code, and everybody inherits these two lines. Then you need circle dependency resolutions and conflict resolutions. Lua is capable of compiling C-based code. If someone ships C-embedded Lua functionality with LuaRocks, if you have the right dependencies on your OS operating system, then you can install this module. As always, there is an awesome language GitHub account where they have a lot of summaries about different tools and stuff about this ecosystem. Then there is LuaChit because LuaChit was started as a project because the core or the language itself is so small. A guy, I can't remember his name, started the LuaChit project. Now you have the possibility to either use Lua or LuaChit as the name for testing time compilers. If you run this code in a continuous way, it optimizes on the fly a lot of code. This is very useful in a web server where you would like to apply web application firewalls. These rules have to be run over and over and over again. They get hot and they are optimized and compiled to machine code. They say 10 to 100 times faster than the normal Lua implementation itself. One of the main drivers behind it is Cloudflare as well because now they ship around 10% of the traffic worldwide. If they could optimize LuaChit by 5 or 10%, they gain a lot of capacity. Think of LuaChit as robbing on steroids. The nice part about LuaChit is that it compiles on nearly all systems and all process architectures. It is binary compatible. It means that you can switch from Lua to LuaChit. It has the same syntax and it works. It works on a PS3 or PS4 which is quite nice. One thing to remember or take away is that Robbin and Batman are two heroes. They are both hands with a community and tools behind it. That is not enough. You have NGINX and Lua. Now we are at NGINX Lua or HTTP Lua module. That is the package name on the documentation and the GitHub name. It is confusing if you are searching for the thing. It means embedding the power of Lua into NGINX. Here is a small example. You would like to get the content. Content by Lua means within these brackets you write normal Lua code. It gets interpreted and it is printing out HALO work. That is the integration. You have a lot of these. You have init by Lua, init worker. In it it runs once when the server starts when a new master comes up. In init worker is when the master process forks a new worker process. You have set by Lua. In the previous example you have seen set as a directive of NGINX. Now you can dynamically set things in the scope of NGINX. Rewrite access content, log header filter and body filter. These are the phases where you can hook into NGINX and modify it by yourself. It gains some attraction because you saw this wild west of C modules. People started migrating their code into Lua because the burden of maintaining a C extension in an ever-changing NGINX code was too difficult or too time-consuming compared to Lua. That is always the same or more or less the same. That is the reason why nowadays a lot of modules and stuff are moving to the Lua world. All stuff are ripped out or is dying off. Another thing that is possible is Lua file. That means in the difference here you write this directly in your NGINX configuration. Here you can specify a specific Lua file. I always would recommend this way because you can see the burden that you have with double and single quotes within the NGINX configuration. There is a fix on the way but it needs some time. There is also another benefit. The Lua code normally if it is embedded by Lua file then it is compiled. Meaning the Lua cheat compiles it into an optimized way. Then it is always interpreted from this optimized way. During development every single character that you change in the Lua file you have to reload NGINX to get updates into your NGINX. That is one of the tricks that you can do. It is that you during development disable the cache and then you can edit right away. You always see the new updates if a new request hits this Lua specific directory. Another thing is these are the parts where the Lua NGINX Lua API comes in. Within the scope of Lua you have to talk somehow to the web server. Meaning that you need some API to talk to the outside world of NGINX. The first thing is NGINX ARC. It is something like you have on the console, on the bash script or in every other language. It is like parameters passed in. There is some situation where it is happening. For example set by Lua. You can pass in parameters or in the filters. But that is rarely the case that you work with. Most of the time you work with NGINX VARs. A good example is outside. VARs always has to be declared outside of the Lua scope. Meaning that during the lifecycle of a request you could change the values. This is very handy if you have, for example, at this point, going to a proxy or to a proxy upstream and define the variable of the proxy. In the Lua code you could do health checks or you could do load balancing or intelligently load balancing algorithm. If you set the variable in the right phase, then NGINX is performing proxy stuff with this modified variable. Another thing is NGINX header. It is used to read and modify headers depending on the stage. If you are passed the lock phase, you are not able to modify them anymore. NGINX status is the HTTP status that is sent back to the client. NGINX context is also a handy thing if you have to pass information between phases. Meaning you have a rewrite phase that writes some information, extracted some information. You need this on a content phase. You can do context. If you are familiar with Android, it is like on an Android system, if you switch the view, you have to assign all the information on a context and then switch the view. Otherwise, these information are lost. That is quite handy if you are passing states or information along with different phases. Is that clear? Other things in the Lua API, NGINX Lua API are the output stuff. We saw NGINX say send headers and so on and so forth. You could do redirects, exit, subrequest. I will come to this later. And co-sockets. I have some examples about these two. NGINX capture. It is a nice thing. What you could do is you could do advanced cookie checking or spam checking. In this example, what we are doing is we are capturing during a process. We do a subrequest within NGINX and waiting for the response. And then we interpret. This is like an HTTP request within an HTTP request. And you can do multiple captures. There is a capability of doing parallel capturing and aggregating the information and sending them back. Another example for this would be if you have a central login server, you could always use NGINX as a proxy. And he is checking against this login server. Are these credentials sent by these clients are valid? For this reason, I let him pass. Otherwise, I deny the access. An example that I used to implement was like MailChimp newsletter. I wanted to have a custom newsletter subscription form. And normally what I have done is like posting on my own website, capturing the stuff with NGINX. Send it to MailChimp server with the API credentials attached. They are actually within NGINX. And then I modify the response that I can differentiate between different errors on the front and side. These are the possibilities that you could do with capture. Another thing is the code socket IP. A thing about NGINX within these phases, everything has to be non-blocking. And in the Lua sphere, normally you have these Lua modules. For example, simple HTTP or Redis implementation, and they use a blocking socket underneath. Meaning if you use this library within NGINX, it's not going to work because if a request comes in and you're blocking while waiting for Redis to give you an answer, every other request has to wait until Redis gives you an answer. And for that reason NGINX request soccer was introduced, meaning giving the user the possibility to do non-blocking socket connection, TCP connection. There are UDP connection as well, but not much space on the slide. But what you have to do is implement a protocol on top of the socket. A nice feature about this is you can use keep alive, meaning that if you're not closing the connection, if a new request comes in, it reuses the keep alive connection. That's a handy thing to get more performance out of it. A good example are the Redis memcache, MySQL, Postgre integration. DNS is actually a UDP implementation. What they look like, it's like you include the Lua Redis clients that's crafted for NGINX, or work within NGINX, and then you do your normal manipulation, reading, writing. Then there are lots of more stuff that you could do with NGINX and Lua. For example, you have LightFret, you can run some work offloads. I've seen people doing a job queue within NGINX. That's a funny thing because you have in it by worker, you could do things with this part. It's a strange thing in LightFret. It actually worked, but it's somehow scary. Shared memory access, meaning that because you have one master process and you have a lot of for worker processes, you could share a caching layer over all these worker processes, meaning that, yeah, and that's a reason why there are some basic caching algorithms within there to help with the cache. Also, Ragex is directly in the core for modifications and stuff. That's basically it, and there's some, yeah, some tiny little things that I never counted, and yeah, they are there, but it's not cluttered. It's quite small, but you have with this capture and co-socket thing, you could do a lot. Meaning that if you combine these two, you get a Batmobile. It's kind of cool. It's like you have NGINX and then you have Lua. You could do a lot of programming, but the problem is you have to build it. It's an own module. You have some people wrote on GitHub, Redis integration. Some people wrote the MySQL integration, and that's where OpenResty kicks in. OpenResty, it's actually a combination of NGINX, Lua. They could also use Lua as well. NGINX Lua, integration Resty, Lua, modules, and C-Lips, meaning aggregating a lot of stuff from the community and packing in the right space that you have a rich environment that you can start with and a normal Makefile. That's very handy. You download OpenResty, Make, Make, Install. What it's doing in the background, it's more or less, first of all, it downloads Lua.chit, compiles it and bets it in NGINX, compiles it with NGINX and NGINX with all the different C-modules, NGINX C-modules and Lua modules, and then place it on the correct system. I think that's not happening at the moment. There's no distro packages. What I normally do, I build distro packages myself, they're not that complicated, and I'm in touch with the creator of the OpenResty project, but he's so busy at Cloudflare. I guess I have to contribute stuff. There's also Docker image existing, but about Resty modules, the Resty modules, as you saw, you have modules coming from NGINX extending the NGINX score, you have modules coming from Lua extending the Lua modules with Lua or Lua C, and the Resty modules, it's always a mix between, I don't know, it's also a little bit Wild World West. The OpenResty project tried to only include modules that are well maintained, that work, that are tested. Actually, the testing of OpenResty is quite impressive. If you ever go on the OpenResty webpage, go to qa.openresty.org. They have heavy testing. That's the reason I never had any issues with OpenResty in our infrastructure. Another thing is, if you have to install things that are not bundled with OpenResty, please use Lua ROX and install it on one specific path. Within NGINX, you then can include this specific path of ROX into your current environment, and then use the installed... It's not very easy, you have to get used to it, but once you use it, you get it going. If you need a BMW, take Kong. Kong is actually an open source project from Mashable Ape. They do a lot of API, localizing, authentication and stuff, and what they build, they call it microservice and API management layer. What they actually took is... because OpenResty has the things that you can build your application on top of it, what they actually done is going a step further. For example, what they included is a central login layer, or JVT, decryption layer, and so on, the central login server, meaning that in an ever-changing environment, you have an API where you can manage your NGINX configuration. That's kind of a neat thing. You can add SSL hosts, you can add hosts, you can add authentication information, you can localize and shut things down. They use a Cassandra cluster on each node, and the information is shared in this. If you need something, and they have a plugable system, meaning that every one or two months, they bring out a new plugin that you can enable or disable a little bit of configuration, and then you have a new API endpoint where you can manage XYZ. That's kind of a cool and neat thing, and gains some attraction in the microservice docker community, because that's a big deal to dynamically low-balance things without changing NGINX configuration. One thing to take away is use ready-built test solutions, looking at OpenResty, if you'd like to be more on the basic side, or Kong, if you'd like more or less production-ready stuff. Take the fast lane in association with Batman and dropping, take the fast lane of crime prevention. Now, to the thing of my talk. What I was thinking, or I often, I'm active in the OpenResty community. We've been in our company for different services, and recently there was a patch about SSL by Lua, meaning that you could secure your SSL connection controlled by Lua. Batman and Robin, they have the same problem. Recently they were tapped, and now they have to introduce new mechanisms to make them safe again on their phone. This is actually a patch. It's not right now in OpenResty. It will be, as I mentioned, the creator of OpenResty is quite busy at Cloudflare. What they newly introduced is SSL certificate by Lua, and SSL certificate by Lua file. What it's actually doing is it's a step before the process even happens, that you could do SSL generation, SSL delivery, OSTP checking, stamp-ling. There's a heavy discussion on issue 331 on the OpenResty project. At the moment there's a branch. You can compile it. What I've done is I took the branch, and I took the OpenResty project. I was doing all the compilation and stuff, and swapped out the OpenResty... Because OpenResty is downloading Lua Engine X module, I swapped out this download with the download of this branch. At the moment you have to patch the engine core as well, because OpenResty downloads the Engine X core, and also the Lua Engine X module. What I had to apply was changing the branch in the download and patching the Engine X core before it's built. That's the thing I had to do. It's not that tricky. The thing is powered by Cloudflare, meaning that they run this since 2013 or 2014. That's definitely battle-testing. That's the thing that's driving their SSL-free for their customer. What they're actually doing is they partner with a CA, I'm not sure if it's a global sign. What they are doing is combining a lot of domains, generating sound certificates, and what they have to do internally if a domain or an SNI only supports SNI. If a SNI request comes in, they look up in which certificate belongs to this list of sounds, and they have to fetch their certificate from the backend. Because they have around 30 to 40 data centers, they have to do this on a distributed way. They do it on the fly, otherwise they have to send a lot of SSL certificates across the globe. Meaning if I hit now here a Cloudflare instance, that's not having my SSL certificate, it gets fetched on the fly. It's kind of a neat thing. I was inspired by this, and I can show you a demo about what I've done. I'll switch to the other screen. My setup is the following. I have a vagrant box running. I have a vagrant box running with NGINX and my Lua code. What I can do is if I request... Wait a second. Now, what I'm actually doing is the following. If the scroll is working, I open an SSL client connection to localhost, giving a server name, meaning a SNI connection, test.driveby.tld, giving a CA file, meaning that certificate authority file for validation, meaning that checking the certificate that I get back, if this CA signed this certificate. What we see here, it's verified, verification chain. What I add dynamically generated by Lua at Saturday 8.40, because of time zones on my machine and in the vagrant box, that's actually the two hours difference. If I hit it again, meaning I could get the same, but a little bit later, now it's 44, and the CN, the common name. If I change it, I take the short, then the information are here. If I pass in another subdomain, subdomain, Glyph 2015, for example, then I get a new SNName, meaning the certificate is valid, but the NGINX core generated on the fly SSL certificate, and then in this returning. The same, I can show you with a fresh Firefox, if you work with Firefox and install a lot of demo drive by TLD. Not working. Ah, now it's okay. The thing is that I, ah, the stupid, if I connected to the Internet, then the thing is I have a DNS mask entry within the DNS mask set up within the network manager. If I'm not connected to the Internet, then DNS mask within network manager, it's not responding, that kind of a bummer. But what I wanted to show is what you could do on the fresh installation of Firefox, you could add the certificate authority, adding a new certificate, and then I can go to every subdomain and I get a valid browser certificate or no warning. Um, is that example clear what I wanted to show? Yeah. Yeah, never mind. The actual code. No. Sorry about that. Not starting at the point. The code in Lua or in NGINX, it looks like this. You, first of all, I include the module REST-SSL. That's an extension coming from this SSL patch. Then I need open SSL for the whole certificate generation, key generation. P-key means like there's a public key cryptography. Then I require a CSR, certificate sign request. Afterwards, I load in the CA, meaning the certificate authority certificate. Load the thing from the file into an object that's actually the certificate. Then I could do manipulation on the thing. The same happens to the key of the authority because with the key of the authority, I have to sign the newly generated key certificate. What I then do, if it's not a SSL server name, meaning if it's not an SNI, if someone connects not with an SNI name, we drop out because of this limitation. Then I start with a new certificate request, meaning the subject. That means the region generated by it. You see here that's actually the date. It's always included. From the SNI name, the SNI name will be the so-called common name. What I then do is generate a new public and private key pair for the new certificate. Afterwards, I assign the key to the certificate and generate a certificate request. The certificate request then from the certificate request, a new certificate is issued. Actually, I had to change this because Firefox is blocking. This is the serial number, meaning that the certificate authority have to increase the number of the serial for each certificate they issue. Firefox is actually checking this. I had to do this morning a little bit of hacking to overcome this issue. Now it works. It's not actually the code on the server. It's not the same. A little bit modified, but it's more or less in the same thing. Then I set the validation time. What I then do is taking this, the naming is wrong, meaning that the client certificate or the certificate is signed with the key and the certificate of the authority. Here, that's actually where the magic happens. I guess in Cloudflare, it looks like this. The thing on the top is the generation and signing is somewhere offloaded. Meaning that the only thing that they do on the fly is capturing the response and certificate on the fly and setting it and caching it on the local server. What we then do is clear the certificate that we had in the engine configuration, set the certificate in the format of the DER, meaning a binary string, converting the binary string to the certificate into a binary, the same goes for the private key of the new generated certificate. That's all the thing that you need to do. At the moment, it's a little bit of tricky to get everything right, but I think in the future you see a lot of application. The first idea was at the moment, if you heard about Let's Encrypt, that's an initiative from Mozilla, Akamai, Cisco, they build an automated system to get you free SSL certificate. My initial idea was going to an upstream server to the Mozilla server, so I was writing a valid public certificate reissuing on the fly, meaning that every time someone hits the server, I get an instantly valid certificate. I never have to think about reissuing or stuff. Everything I can write down in Lua. If someone hits the first time, he has to wait, I don't know, two seconds off or something, or I don't know, a minute, then the request responds, but the request afterwards, it's okay. Or I can do a cron job that is doing that every night. That was the basic idea. At the moment, we are implementing this for our web hosting customers, and during this work, we saw that the API and the specification is changing too radically that I could adopt these changes all the time before the talk. I started with the adoption and I ran into the problem that actually the OpenSSL project had a lot of issues. First of all, I had to fix a lot of C code issues within this project before I could go on, and that's why we have only a simplified version of that. Nina's say, securing in an ever-changing system is hard, but it's also possible that you can introduce security layers or dynamic stuff on the web server. At least you can try, but don't produce snakegoys. One of the pitfalls that I learned during my years and also preparing for this presentation, if you do a lot of engineering configuration modification, please always run service engineering conflict test. I had sometimes the issue that I thought I had changed something, made a reload, but actually if you do a reload and you have conflict issues, engineering is doing nothing and says nothing about it. That's the reason why I always do service engineering conflict test and then a reload. The same happens if you do a lot of conflict changes. I have a when change, which is a little small Python script. If the file changes, I access H into the server and do a conflict test and reload the server. I also fell into this trap, like the escape characters in Lua. It's not very intuitive, but better than Ruby. As I mentioned, always use Lua by file. It's easier to handle with and no escaping. If you'd like to read post-words in an access phase, meaning like one of the rewrite or access phase, then you have to do an additional request head body, meaning that in the face of rewrite and access, NGINX has not yet loaded the body or the post information. Tips and trick debugging in an NGINX environment, especially in a dynamic NGINX environment with Lua. It's not easy or actually hard. I was burned many times, like little pieces that, yeah. One of the things that I often do is like adding debug information to the header information, meaning aggregating information. You could log all the information, but the thing is on the log sometimes on a business, it happens so much thing. What you could do, I have special tags and if I specify this hidden tag, it gets me like debug output on a live server. So now that's just kind of cool neat trick. What you actually also could do is like generating a tracer, tracing ID, and then piping, you know, or giving this tracer ID as a header parameter parameter to the web server and then going through your logs. That's also a possibility. But I found the NGINX header dump the easier. And if you're looking for code or useful example, actually the OpenRESI project has very good tests. And for this SSL talk, I had to look many times in the code because it's at the moment is not yet documented. And there are some conditions where the documentation lag. Yeah. And I'm at the end of my talk. Thank you for listening. Are there any questions, regards, or concern about the stupidity about this project? Yeah. What was the future ratio on the server that you have in terms of the problem? No, what I do now, it's kind of a silly example. I generate for each request a new certificate. It's kind of stupid. It's not, it's not. But what you normally could do is offloading money. As I mentioned, people doing worker demons within NGINX, you could do the same. It's like if the first request hits the server, you could offload this to worker. Then it's in its own worker and it's not blocking the other requests. Yeah. But you could do, or... Okay. No, they run under the NGINX user. Meaning that I had to give well permission or at least for the CEA, for the CEA certificate I had to give permission to the NGINX user. Yeah. But normally you do this in a secure backend and then you get, yeah. So I'm thinking that in principle, we also for other users, just for the sake of the NGINX user, we don't see a certificate of other clients. Yeah. Where the case is attached or back into the client, how much the NGINX certificate. Yeah. So, for the reason I need the private key to sign, otherwise... No, no, no. My use, yes, this use case, yes. But the use case of LatinGrip, for example, I go into an HGT backend and they give me an SSS certificate. I could also program that I log into the complicated website and do web scraping of global sign or something else that's not limited to this because we flew out of the capabilities. Yeah. But for this purpose, because I'm not a CEA, I have to trick the browser to accept me as a CEA. Yeah. The initial idea was going to an official CEA where they have a REST API where I can issue. Yeah. Okay. So far, Bova Jaskity. And thanks to agent CH, that's the guy behind all the RESTy stuff and CloudFare for the contribution. And hopefully we'll see some of you in the OpenRESTy and GNX Lua community playing around. Thank you so much. Thank you.
real use case but it could be. In the example of modules written in C to extend the capabilities echo module. It is not in the core and returns different stages during the start of an HTTP this library within NGINX, it's not going to work certificate by Lua, and SSL certificate by Lua someone connects not with an SNI name, we drop