http://exploration
Buchek, Craig
We're web developers. But how well do we know the web's core protocol, HTTP? In this lab, we'll explore the protocol to see exactly what's going on between the browser and the web server. We'll cover: HTTP basics HTTP methods (GET, POST, PUT, etc.) HTTPS Troubleshooting tools Proxies Caching HTTP/2 We'll investigate how we can take advantage of HTTP features to troubleshoot problems, and to improve the performance of our Rails apps.
 Alright, folks, I'm going to get started here. Thanks for coming out. USB keys have two files you'll need, the Vagrant file and the HTTP exploration box file. If you don't have Vagrant and VirtualBox, there are executables on there for Windows, Linux, a couple of variants of Linux and Mac. You'll need to have those installed. I'm going to do a half hour lecture, hopefully a little shorter than that, and then we'll have about an hour to run through the exercises. The exercises will walk you through starting up Vagrant and everything you need to do. So I have Charlie Sanders helping me out. Raise your hand if you need a USB key to get the files from. So basically you've got half an hour to get all that working. If you don't, don't worry too much, you can pair up with the neighbor. I would appreciate feedback, negative or positive. You can tweet me at Craig Buchek. Email address is up there. My presentations are on GitHub. I haven't put the latest versions of this presentation yet, but it should be there tomorrow. If you guys want to follow along, TinyCC HV exploration with an underscore. To take you to this presentation, actually I think I need to update that. I'll do that when we get to the exercises. The exercises will begin there too, so I'll update them when we start the exercises. The reason I started doing this is I actually have a previous life as an assistant in a network admin. I've actually done a lot of troubleshooting of HTTP, going through networks, going through firewalls, and now I'm a Rails developer. So I've seen both sides. So we're going to talk about HTTP basics, request, responses, talk about proxies, some troubleshooting, and HTTP2. And then we'll get into the exercises that touch on all of those. So HTTP has been around since 1991. The first version didn't actually have a version number, but retroactively we call it 0.9. It was standardized in 96. The version that we typically use now is standardized in 2007. That RSC is very handy if you have any questions about how HTTP works. I was just updated a few months ago. It was broken up into multiple pieces, basically in preparation for HTTP2. HTTP2 has been ratified, standardized, but it hasn't been published yet. But that's a good URL to check out if you're interested. So HTTP is stateless, which means that each time you connect to the server, it doesn't really remember what you did the last time you connected to the server. And the only way around that is through cookies, pretty much. It is text-based. That's kind of the whole point of the presentation. We're going to look at the text that's going across the wire. And it's a request-response cycle. Your client, your browser, will make a request to the server. The server will respond. So I want to talk real quick about URLs. So pretty much every piece of that URL on the screen will go from the client to the server except for the fragment. The fragment just tells the browser to go to this specific spot on the page. The scheme is going to be HTTP. The host is obvious. You can specify username and password in the URL, recommend against that, but it's possible. And then the path is actually the piece that goes that we'll be working with for the most part. All right. So the HTTP request looks like this. That thing in green is called the method. So we'll talk about that in a bit. The next thing is past as the URL. Usually it's not the full URL. Usually it's just slash. It starts with a slash and it's relative to the top of the site. And then you have the version of HTTP specified there. In blue are the headers. So the headers are sort of metadata about what you want to get or what you want to do. In this case, there's a host header and a content length header. And the body, not all requests have a body. If you're just getting something, it won't have a body. But if you're posting something or putting something, it will have a body. In this case, I tried to put something to Google's front page. Probably not going to work. I actually tried that. And it gives a 405 error, which is hard to find. So that's the body text, basically the content that you want to upload. So the methods I talked about, normally when you go to a little website, you're going to do a get request. They get this page. You go to Google's front page, you do a get. A post is when you want to update something or when you submit a form. Although some forms can actually do get, depending on if you're changing something or just doing a query. Like when you go to Google's front page, that form, when you type in the query, that's a get because you're actually just asking for information you're not asking to change anything. The put is actually, it's an update, or actually it's more of an upsert. You're saying, I'm giving you the whole thing. Replace whatever you got if you have anything. Otherwise, create it. Rails is a little weird on that. It doesn't actually do the create part of that. So it uses put a little bit oddly. Delete does what it says. If you have permission, you actually can delete pages or resources through the web. Get is basically a get request but without the body. You're saying, show me the headers that I would get if I did the get request for this URL. There's a concept of safe methods. So safe methods don't have any effect on information. When I talked about the post, it changes something. When you submit a form, it changes something on the server. A get is not that case. So the takeaway here is don't have your app change things when you do a get because when Google comes scraping your site, it's going to do gets on your site. If your site changes something on the back end, you're going to have a bad day. So just keep safety in mind. There's also something called item potency. That means that you can call the same thing multiple times. You can do a get to the Google front page multiple times. You'll get the same thing. Same thing with the head. Same thing with if you delete resource, the resource is going to be gone if it was there or not. A put, you're actually saying replace something. So if it's there, replace it. If it's not there, allow it. So multiple times doesn't matter. The one that's missing from this list, again, is post. So if you post something multiple times, you'll end up with multiple copies of that thing. So request headers. We saw that we can provide those headers. These are some common headers. The host header is basically required. It says the name of the website we're accessing. And the reason that that was added is so you could host multiple websites on a single server. The accept header says list the types that the browser would like to receive. So the browser may say that it likes to have HTTP, it likes to have JSON, but not list XML. So if the server knows how to do HTML and JSON, it will pick one of those, but it wouldn't pick XML unless it didn't know anything else. The weird thing about the accept headers, you can actually specify relative quality. Sometimes you'll see a queue number if you look at the accept header. And that says I prefer these types of things over the other things. The default quality setting is one, which means top priority. If you prefer JSON over XML, you could set its quality to 0.9 and XML is to 0.8. The content length on a request header is the content of the body of the request that you're sending. And if you don't specify that, it actually just doesn't let you specify anything. Once you've uploaded the content length number of bytes in the body, the server will close the connection. The content type tells what you're uploading. Am I uploading some HTML? Am I uploading some XML? Am I uploading some JSON? The referrer is if you are a browser, it's the page you were on when you clicked the link. That is spelled incorrectly, but it's in the standard document, so that's what we use. The user agent is another name for a browser or a web client. It could be a web crawler, it could be various things. It's a string containing the information about the browser. If you take a look at those, those get to be pretty crazy long, especially in Chrome and Firefox. They try to pretend. Back in the day, there was something called browser sniffing. The server would look at that string and try to determine, oh, I'm Internet Explorer, I'm going to give you this copy of the page, and your Firefox, I'm going to give you this copy of the page, which could be completely different. Google didn't like that because Google wants there to be one canonical version of every page that it indexes. It doesn't want you to fool it, so it sees one thing and then the user sees something else. Browser sniffing is not used too much anymore at all. You end up with these really long strings that tell you all the backwards compatibility that the browser tries to do. We'll take a look at those headers in the exercises. Yeah? Yeah. It was you who requested better and accept. Sorry, I'm sorry. Yeah. Comment like accept. Accept is what you, you said what you want to expect back. Yes, yes. And then content type is only for a post or a put, and it's what type the body that you're sending up to the server is. Yeah. So authorization is for, if you ever go to a website and you get the pop-up box, authorization is basically that information you type into the username and password box. It's not encrypted, it is hashed and sent. And we're actually going to, there's an exercise that we're actually going to find that going across the wire. Content encoding, you can actually gzip things. The server can gzip the content coming back to you. This basically says that's okay. You can say content encoding gzip, commit deflate. That tells the server, hey, I can gzip this and save some bandwidth across the wire. Connection usually is for keep alive. It is the client saying, hey, I've made this request, but when I'm done, I'm going to make another request. I'm going to make another turn down the TCP connection. I want to reuse it for the next request. So if I'm getting a web page and I know it's probably going to have some JavaScript, it's probably going to have some CSS, it lets me get those all in one without dropping the connection. Cookie. I talked about the, a cookie is, the server has sent us this cookie and we are supposed to return that cookie back to it. And that maintains sessions with the server, between the server and the client. And that's the only thing that maintains a session between the client and the server. And that header is limited to about 4K of information. So if we do our cookie-based sessions in Rails, we do have to worry about that limit. You don't want to store a whole lot in that cookie. All right. So we've made the request and the server response with the HP response. That first line is called the status line. It's got the HP version. Interesting thing, you can actually make a request for one version and get a different version back. I find that a little odd. The 201 there is our status code and then there's the description of the status code which is created in this case. And then like the request, the response actually also has headers. And then it has a body. Not every response has a body. There's a few that don't have to have a body. Created is actually one of those. So if you are doing a put or a post, it could actually just say created and then don't give you anything back. But you probably want to do a redirect. So you probably want some extra information in there maybe. So status codes. So the status codes have different meanings and they are actually standardized. Although occasionally you'll see some non-standard ones. The 100s are informational. You'll rarely see those. The 200s are what you'll see most of the time. Usually for a get, you're going to get a 200. For a post or a put, you're probably going to see a 201. Redirection, if you go to google.com, it'll actually redirect you to www.google.com. That's a redirect. I believe that would be a 301. If you send headers that are involved with caching, which we'll do in an exercise, you'll get a sometimes you'll get a not modified. That says, hey, you've told me you already have a copy of this in your cache. So I'm just going to give you the metadata and the headers. I'm not going to give you the body. You should use the body that I gave you last time. So then we've got the error response codes. So 400s are client errors where the client made some sort of mistake. First is if you have not sent an authenticate header and the web server requires you to authenticate, the client is supposed to retry with the authenticate header. And so before it does that, it pops the box up, has you type in your username and password, resends with that header. Forbidden means you've probably either you can't authenticate or you've authenticated, but you still don't have permissions because maybe you're not an administrator in the box. 404, we've all seen that one before, page not found. 407 is proxy authentication required. It's like a 401, except it's the non-transparent proxy that is asking for a username and password. That's pretty rare to see. We will do a little bit of proxies, but we're not going to do any proxy authentication. 422 is unprocessable entity, which is a weird way of saying I don't understand what you're trying to ask me. That is recommended. That's what I would recommend, either 422 or 409 if you are making an API request and it doesn't have the right information that it needs in the JSON or XML or whatever that you sent. Server errors. If your server crashes really badly, you will see a 500 error. I believe you see that in Rails a lot of times when you're debugging. 502 and 504 are gate-wears. If you've got a reverse proxy sitting in front of your servers and your servers have gone down or take too long to respond, you'll see a 502 or a 504. There are plenty of others. I ran into a 405 earlier. It's one of the April 1st RFCs called I'm a teapot response, 418 code. Not sure when you'll need that. I'm sure someone has implemented it, though. Response headers. So like a request, the response has headers. The content length will almost always be there because you'll have a response that has content in the body. The content type, it's the mine type. So it's going to be like text slash HTML, text slash plain, application slash JSON, or image slash JPEG, image slash PNG. That tells the client what type of file this is and then it can handle that however it expects to. The content encoding is, I talked about accept encoding where you can Gzip it. The content encoding says that the body has been GZipped. Notice that the body is GZipped, but the headers are actually still in plain text. Content disposition is a little trick that you use when you want to, when the person clicks on the link, you want them to download the file instead of have the file display in the browser. In that case, you would use content disposition header and you can also provide a default file name that the browser will try to save it as. Location. Location is used for redirecting and we'll have an exercise on that. Usually you want to provide a response code that has, that says redirect and then you provide the URL to redirect to in the location header. Set cookie, we talked about cookies for maintaining state, maintaining sessions. That cookie tells the browser to remember this token and it's just roughly a random string of characters and when it gets sent back by the browser, the server knows to associate it with a session. It looks it up in the database. www.authenticate is basically telling the browser that to pop up the box or if they've already typed their username and password to provide that information to the server. So we'll run into proxies. We'll have some exercises on proxies. So proxy is something that acts in place of another. In the case of HTTP proxy, a web proxy, it intercepts our HTTP requests. So it can modify that request. So what it does is it intercepts a request, modifies it probably, does some caching perhaps and sends it onto the server, gets the response back from the server, can modify it then again and then sends it back to the client. So it sits in between the client and the server and it can modify pretty much anything that it sees in there. So proxies are good for caching. You can add security. So actually our exercises, we will add some SSL to our Rails app. So you do this to simplify so you don't have to have the Rails app understand SSL. It also can save some CPU time. Also can be used for load balancing. It can be used for authentication. I've had it where I had Apache in front of an application server and we added the pop-up authentication for it with Apache acting as a reverse proxy which I'll talk about in a second. So there are transparent proxies. Basically you don't have to set anything up. It's inputted itself into the stream of the network and there's non-transparent proxies which a lot of times if you're at work and at a big company, you'll have to configure a browser to point at the proxy. That's a non-transparent proxy. And we've got an exercise on that. Reverse and forward proxies. So the proxy you have at work that sits sort of right next to the firewalls would be a forward proxy. A reverse proxy sits next to the server. So right near the servers, anything that's, any proxy that's added on the server end is a reverse proxy. And we will actually have exercises on both of those. A CDN is a content delivery network. It's basically a paid service that does proxying for caching purposes. So you can cache all your static content. Technical things in here I won't get too much into. One of the nice things that can provide you, protect you from distributed denial of services. If you've got a big site, you should probably be looking into these. Troubleshooting. So any network problem you have, you have to think about the OSI model. And I wish I had a picture of that. I forgot to put that on there. So you've basically got the physical layer. You've got the network layer. You've got the transport layer, which is TCP. And then you've got your applications sitting on that. So there's a lot of things that could go wrong. You could have a network cable pulled out. You could have, you know, your routers down. Or the server's not running. So all those different layers you have to think about when you have a problem. Troubleshooting is trying to figure out which layer it is. And then narrowing down on that. So one of the first tools is ping. Can I connect to the IP address that the server is on? The problem with that is sometimes the firewalls will prevent that, either firewall in your company or the firewall on the other end. Tracerout is similar, but it shows you all the hops in between. So maybe the network is down between your internet provider and Google. Tracerout might be able to help you find that out. Tell that we're going to use in our exercises. It's a good way to tell if the port is listening. And if I've got connectivity to the IP address, tell that will tell me if the service is listening. And that's actually kind of where I start. I kind of start in the middle. And then if tell that doesn't work, I'll try the lower layers. And if it does work, I'll try the upper layers. If you're on your server and you want to see if your service is listening, you can do a net stat on Linux. That's dash PLA and T. It's easy, mnemonic for me to remember. On Mac, it doesn't have all those options. You'd use dash NA and then grab for listen. And then so that's going to list over on the left side. It's going to list all the IP addresses, which is usually going to always be your IP address. And then colon in the port number. So if you're looking to see if Rails is running, look on the left side for something colon 3,000, assuming you use the default port for Rails. Telnet doesn't work with HTVS because it's encrypted. So you have to use this tool that OpenSSL provides called sClient. And we've got an exercise on that. TCP dump, if you really want to see all the details on what's going across the wire, it will tell you everything. We've got a short exercise on that. There's so much information, we could probably spend a couple hours on that. Wireshark is a good way to visually see what's going on. You can actually take the output of TCP dump, save it to a file, pull it into Wireshark. Wireshark has a nice feature. TCP dump shows you each individual packet and their disjoint. When you're having communication between a client and a server, your packets can only be so big. So the communication will be broken up into pieces. Wireshark has a nice feature to put all those back together, which is kind of cool. All right. So as I said, HTTP2 was recently approved and ratified. It came out of a project at Google called Speedy, SPDY. Apparently, they wanted to make it Speedy to say, to write the word Speedy. So as I said, we can compress, we can Gzip the body, but we can't, in HTTP 1.1, we can't compress the headers. HPEC is a part of HTTP2 that allows header compression. Another thing, the standard doesn't require it, but every HTTP2 implementation out there requires TLS. And that's probably because it's got TLS or SSL has a protocol and negotiation built into it. And that protocol and negotiation can say, hey, do you have HTTP2? I'd like to use that. Do you have Speedy 3? I'd like to use that, but if not, fall back to HTTP1. So I said that HTTP is all text. HTTP2, that turns out not to be the case because we're compressing the headers. It has the same semantics. You won't be able to use Telnet. You won't be able to use TCP dump, but you will be able to use some of the other tools that we'll be looking at today. I did get an example to work with HTTP2 that we can do as an exercise. When you're making a lot of small requests, like let's say you've got a lot of icons on your page, ideally you would just get each icon individually, but those icons are only 25K or something. At that point, the headers at a couple K are starting to become large overhead. If we could compress those down to a couple dozen bytes, it would save a lot of bandwidth. There's some other features in HB2 we'll get to in a sec. Basically just to save bandwidth. The other thing about HTTP2 is it multiplexes the connections. You can actually have multiple files coming across at the same time with a single TCP connection. Right now your browser has to make multiple TCP connections. I don't know what the browsers are up to. They started at four at a time. They got up to eight at a time. That means you've got... the TCP connection takes up resources and it takes time to set up. If you could do just one TCP connection, that would save some time. That's one thing they've done. You can grab your HTML file and you can actually start processing and see, hey, I've got some JavaScript, I've got some images, I've got some CSS, I need to grab all those too. I can grab all those simultaneously. That's going to save a lot of time. Server push. The server can actually know, hey, he's getting this HTML file. He needs a CSS file too. I'm going to start giving it to him even if he didn't ask for it. The client can say, well, I'll take that. Hey, I finished with the HTML file. I didn't need that CSS file. You can stop giving that to me. Which is kind of weird with caching. I don't know how it knows what's been cached and what doesn't. The possibility is there to save some time. When we do web design and web development, we've got the asset pipeline. That combines all our JavaScript files into one big file. It combines the CSS. We probably combined that into one file. The images, we probably do something called spriting where we put a bunch of images in one file. Then the CSS has to go grab each piece to put each icon on there. If you look at the, I know I've looked at the homepage for Yahoo. It's got all the icons on the left. Those are all in one file. The CSS gymnastics we have to do is a pain in the butt. HTTP2 will hopefully allow us to stop doing that. We can just write it the way that it should have been written in the first place. Not worry about performance. HTTP2 should do that for us. HTTP2 is kind of weird that it starts with a 1.1 connection and then it upgrades. The semantics of that are pretty crazy. You probably don't want to get involved in that. We'll see a little bit about that in the verbosity of the tools we use. We'll show you a little bit of that. Here is HTTP2 working. Chrome 41 has it. Firefox 36. IE 11 but only in Windows 10. Nobody has that yet except in beta. Engine X says they're going to support it by the end of 2015. They already support speedy 3.1, which is pretty darn close. I could not get Curl to work with Engine X using speedy or HTTP2. Call for. Right now you have to specify the HTTP2 flag. For this I had to manually compile that feature in. Wireshark 2 will have it. They're currently in the beta series 1.99. Patches doesn't seem to have plans, which seems a little weird, but it does have mod speedy available. All right. So it's time for the exercises. I need to update that URL real quick. But Charlie will be walking around helping people get started. The exercises are basically starting on page 27 of this. And so the first step is basically add the vagrant box with that command there. Do a vagrant up and then a vagrant SSH. You'll be in the box. And then you can move on to slide 28. Will Rails support HTTP2? Probably not for a long time. So Rails really doesn't even support HTTP. Your Unicorn or your Puma are what supports that. So when those support HTTP2, we will have it built into Rails. Rails itself sort of sits right behind that. Right now if you want it, you would put a proxy in front. You would put InginX, is my recommendation. In fact, InginX said they have 95% of the, well, it was speedy at the time, servers on the internet. So you would use a reverse proxy. Well, Rails also doesn't support HTVS. So you probably want HTVS on your site, so you probably need that proxy anyway. Another question? Other link? You should see this version without the vagrant init. If you did the vagrant init, you'll need to get the vagrant file back. And I think you'll need to do vagrant box remove. But raise your hand if you did the init and it will help you out. So once you guys get into vagrant with the vagrant SSH, just work through the slides. Raise your hands if you have any questions and we'll come around to help. Ok. You see the space again. It returns. Well, it does. I think it does. You said that it was long. I think it's done. It's been pretty fascinating to look at. Oh, no, I didn't. Yeah, it's done. Yeah, it's done. It's done. It's done. I'll just paste it in. Yeah, you can put the save text once. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. To have everything worked. Okay. That is a thing that's doing better. Okay. It's in the virtual box. Oh, sorry. I was for a little bit of a we for the interview we'll be back in a minute. We'll be back in a minute. We'll be back in a minute. We'll be back in a minute. We'll be back in a minute. We will answer her question... There is one asking...... that will pick up and you can think it in that President, or Some ignore me They might not agree. In question, I think the next thing we're gonna do is start with the end. I don't know why I had to. Hey folks, for the HV1.0 and 1.1, make sure you enter the blank line when you're using Telnet. If you don't use a blank line, I'm just sitting there waiting for you. I'm gonna be doing this now. I thought it was odd to make it so he's got a big spot. I think you're all right there. He's going to start. I just want to try to make a different spot. From the right, the third, the rail. Oh, yeah. You've got a response from some of the other people. Yeah. They should be able to do something like get. To get in the... I don't remember if you tell them that. Well, if you can propose that you should get in. Oh, I think it was. I think just a... Oh, yeah. Yeah. Good. Yeah. I'm going to try to get some. I actually don't know how. I just want to get some. Do you have some running background? Um... Yeah. Oh, I have old tips for my guys. I'm going to try to get some. Yeah. You want me to tell them to go? So I'll call them on a business side. Send the kit part. First one. Yeah. Do I need to just press enter now? And don't hit enter or something like that. No. I'm going to go back to it. We'll be able to tell them what to come to the table. There you go. We did. If people have a speaker, we can do a few more things like this. We can do a few more things like this. We'll be able to still get more. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay.
 Alright, folks, I'm going to get started here.  Thanks for coming out.  USB keys have two files you'll need, the Vagrant file and the HTTP exploration box  file.  If you don't have Vagrant and VirtualBox, there are executables on there for Windows,  Linux, a couple of variants of Linux and Mac.  You'll need to have those installed.  I'm going to do a half hour lecture, hopefully a little shorter than that, and then we'll  have about an hour to run through the exercises.  The exercises will walk you through starting up Vagrant and everything you need to do.  So I have Charlie Sanders helping me out.  Raise your hand if you need a USB key to get the files from.  So basically you've got half an hour to get all that working.  If you don't, don't worry too much, you can pair up with the neighbor.  I would appreciate feedback, negative or positive.  You can tweet me at Craig Buchek.  Email address is up there.  My presentations are on GitHub.  I haven't put the latest versions of this presentation yet, but it should be there  tomorrow.  If you guys want to follow along, TinyCC HV exploration with an underscore.  To take you to this presentation, actually I think I need to update that.  I'll do that when we get to the exercises.  The exercises will begin there too, so I'll update them when we start the exercises.  The reason I started doing this is I actually have a previous life as an assistant in a  network admin.  I've actually done a lot of troubleshooting of HTTP, going through networks, going through  firewalls, and now I'm a Rails developer.  So I've seen both sides.  So we're going to talk about HTTP basics, request, responses, talk about proxies, some  troubleshooting, and HTTP2.  And then we'll get into the exercises that touch on all of those.  So HTTP has been around since 1991.  The first version didn't actually have a version number, but retroactively we call it 0.9.  It was standardized in 96.  The version that we typically use now is standardized in 2007.  That RSC is very handy if you have any questions about how HTTP works.  I was just updated a few months ago.  It was broken up into multiple pieces, basically in preparation for HTTP2.  HTTP2 has been ratified, standardized, but it hasn't been published yet.  But that's a good URL to check out if you're interested.  So HTTP is stateless, which means that each time you connect to the server, it doesn't  really remember what you did the last time you connected to the server.  And the only way around that is through cookies, pretty much.  It is text-based.  That's kind of the whole point of the presentation.  We're going to look at the text that's going across the wire.  And it's a request-response cycle.  Your client, your browser, will make a request to the server.  The server will respond.  So I want to talk real quick about URLs.  So pretty much every piece of that URL on the screen will go from the client to the  server except for the fragment.  The fragment just tells the browser to go to this specific spot on the page.  The scheme is going to be HTTP.  The host is obvious.  You can specify username and password in the URL, recommend against that, but it's possible.  And then the path is actually the piece that goes that we'll be working with for the most  part.  All right.  So the HTTP request looks like this.  That thing in green is called the method.  So we'll talk about that in a bit.  The next thing is past as the URL.  Usually it's not the full URL.  Usually it's just slash.  It starts with a slash and it's relative to the top of the site.  And then you have the version of HTTP specified there.  In blue are the headers.  So the headers are sort of metadata about what you want to get or what you want to do.  In this case, there's a host header and a content length header.  And the body, not all requests have a body.  If you're just getting something, it won't have a body.  But if you're posting something or putting something, it will have a body.  In this case, I tried to put something to Google's front page.  Probably not going to work.  I actually tried that.  And it gives a 405 error, which is hard to find.  So that's the body text, basically the content that you want to upload.  So the methods I talked about, normally when you go to a little website, you're going to  do a get request.  They get this page.  You go to Google's front page, you do a get.  A post is when you want to update something or when you submit a form.  Although some forms can actually do get, depending on if you're changing something or just doing  a query.  Like when you go to Google's front page, that form, when you type in the query, that's  a get because you're actually just asking for information you're not asking to change  anything.  The put is actually, it's an update, or actually it's more of an upsert.  You're saying, I'm giving you the whole thing.  Replace whatever you got if you have anything.  Otherwise, create it.  Rails is a little weird on that.  It doesn't actually do the create part of that.  So it uses put a little bit oddly.  Delete does what it says.  If you have permission, you actually can delete pages or resources through the web.  Get is basically a get request but without the body.  You're saying, show me the headers that I would get if I did the get request for this  URL.  There's a concept of safe methods.  So safe methods don't have any effect on information.  When I talked about the post, it changes something.  When you submit a form, it changes something on the server.  A get is not that case.  So the takeaway here is don't have your app change things when you do a get because when  Google comes scraping your site, it's going to do gets on your site.  If your site changes something on the back end, you're going to have a bad day.  So just keep safety in mind.  There's also something called item potency.  That means that you can call the same thing multiple times.  You can do a get to the Google front page multiple times.  You'll get the same thing.  Same thing with the head.  Same thing with if you delete resource, the resource is going to be gone if it was there  or not.  A put, you're actually saying replace something.  So if it's there, replace it.  If it's not there, allow it.  So multiple times doesn't matter.  The one that's missing from this list, again, is post.  So if you post something multiple times, you'll end up with multiple copies of that thing.  So request headers.  We saw that we can provide those headers.  These are some common headers.  The host header is basically required.  It says the name of the website we're accessing.  And the reason that that was added is so you could host multiple websites on a single server.  The accept header says list the types that the browser would like to receive.  So the browser may say that it likes to have HTTP, it likes to have JSON, but not list  XML.  So if the server knows how to do HTML and JSON, it will pick one of those, but it wouldn't  pick XML unless it didn't know anything else.  The weird thing about the accept headers, you can actually specify relative quality.  Sometimes you'll see a queue number if you look at the accept header.  And that says I prefer these types of things over the other things.  The default quality setting is one, which means top priority.  If you prefer JSON over XML, you could set its quality to 0.9 and XML is to 0.8.  The content length on a request header is the content of the body of the request that  you're sending.  And if you don't specify that, it actually just doesn't let you specify anything.  Once you've uploaded the content length number of bytes in the body, the server will close  the connection.  The content type tells what you're uploading.  Am I uploading some HTML?  Am I uploading some XML?  Am I uploading some JSON?  The referrer is if you are a browser, it's the page you were on when you clicked the  link.  That is spelled incorrectly, but it's in the standard document, so that's what we use.  The user agent is another name for a browser or a web client.  It could be a web crawler, it could be various things.  It's a string containing the information about the browser.  If you take a look at those, those get to be pretty crazy long, especially in Chrome  and Firefox.  They try to pretend.  Back in the day, there was something called browser sniffing.  The server would look at that string and try to determine, oh, I'm Internet Explorer,  I'm going to give you this copy of the page, and your Firefox, I'm going to give you this  copy of the page, which could be completely different.  Google didn't like that because Google wants there to be one canonical version of every  page that it indexes.  It doesn't want you to fool it, so it sees one thing and then the user sees something  else.  Browser sniffing is not used too much anymore at all.  You end up with these really long strings that tell you all the backwards compatibility  that the browser tries to do.  We'll take a look at those headers in the exercises.  Yeah?  Yeah.  It was you who requested better and accept.  Sorry, I'm sorry.  Yeah.  Comment like accept.  Accept is what you, you said what you want to expect back.  Yes, yes.  And then content type is only for a post or a put, and it's what type the body that you're  sending up to the server is.  Yeah.  So authorization is for, if you ever go to a website and you get the pop-up box, authorization  is basically that information you type into the username and password box.  It's not encrypted, it is hashed and sent.  And we're actually going to, there's an exercise that we're actually going to find that going  across the wire.  Content encoding, you can actually gzip things.  The server can gzip the content coming back to you.  This basically says that's okay.  You can say content encoding gzip, commit deflate.  That tells the server, hey, I can gzip this and save some bandwidth across the wire.  Connection usually is for keep alive.  It is the client saying, hey, I've made this request, but when I'm done, I'm going to make  another request.  I'm going to make another turn down the TCP connection.  I want to reuse it for the next request.  So if I'm getting a web page and I know it's probably going to have some JavaScript, it's  probably going to have some CSS, it lets me get those all in one without dropping the  connection.  Cookie.  I talked about the, a cookie is, the server has sent us this cookie and we are supposed  to return that cookie back to it.  And that maintains sessions with the server, between the server and the client.  And that's the only thing that maintains a session between the client and the server.  And that header is limited to about 4K of information.  So if we do our cookie-based sessions in Rails, we do have to worry about that limit.  You don't want to store a whole lot in that cookie.  All right.  So we've made the request and the server response with the HP response.  That first line is called the status line.  It's got the HP version.  Interesting thing, you can actually make a request for one version and get a different  version back.  I find that a little odd.  The 201 there is our status code and then there's the description of the status code  which is created in this case.  And then like the request, the response actually also has headers.  And then it has a body.  Not every response has a body.  There's a few that don't have to have a body.  Created is actually one of those.  So if you are doing a put or a post, it could actually just say created and then don't give  you anything back.  But you probably want to do a redirect.  So you probably want some extra information in there maybe.  So status codes.  So the status codes have different meanings and they are actually standardized.  Although occasionally you'll see some non-standard ones.  The 100s are informational.  You'll rarely see those.  The 200s are what you'll see most of the time.  Usually for a get, you're going to get a 200.  For a post or a put, you're probably going to see a 201.  Redirection, if you go to google.com, it'll actually redirect you to www.google.com.  That's a redirect.  I believe that would be a 301.  If you send headers that are involved with caching, which we'll do in an exercise, you'll  get a sometimes you'll get a not modified.  That says, hey, you've told me you already have a copy of this in your cache.  So I'm just going to give you the metadata and the headers.  I'm not going to give you the body.  You should use the body that I gave you last time.  So then we've got the error response codes.  So 400s are client errors where the client made some sort of mistake.  First is if you have not sent an authenticate header and the web server requires you to  authenticate, the client is supposed to retry with the authenticate header.  And so before it does that, it pops the box up, has you type in your username and password,  resends with that header.  Forbidden means you've probably either you can't authenticate or you've authenticated,  but you still don't have permissions because maybe you're not an administrator in the box.  404, we've all seen that one before, page not found.  407 is proxy authentication required.  It's like a 401, except it's the non-transparent proxy that is asking for a username and password.  That's pretty rare to see.  We will do a little bit of proxies, but we're not going to do any proxy authentication.  422 is unprocessable entity, which is a weird way of saying I don't understand what you're  trying to ask me.  That is recommended.  That's what I would recommend, either 422 or 409 if you are making an API request and  it doesn't have the right information that it needs in the JSON or XML or whatever that  you sent.  Server errors.  If your server crashes really badly, you will see a 500 error.  I believe you see that in Rails a lot of times when you're debugging.  502 and 504 are gate-wears.  If you've got a reverse proxy sitting in front of your servers and your servers have  gone down or take too long to respond, you'll see a 502 or a 504.  There are plenty of others.  I ran into a 405 earlier.  It's one of the April 1st RFCs called I'm a teapot response, 418 code.  Not sure when you'll need that.  I'm sure someone has implemented it, though.  Response headers.  So like a request, the response has headers.  The content length will almost always be there because you'll have a response that has content  in the body.  The content type, it's the mine type.  So it's going to be like text slash HTML, text slash plain, application slash JSON,  or image slash JPEG, image slash PNG.  That tells the client what type of file this is and then it can handle that however it  expects to.  The content encoding is, I talked about accept encoding where you can Gzip it.  The content encoding says that the body has been GZipped.  Notice that the body is GZipped, but the headers are actually still in plain text.  Content disposition is a little trick that you use when you want to, when the person  clicks on the link, you want them to download the file instead of have the file display  in the browser.  In that case, you would use content disposition header and you can also provide a default  file name that the browser will try to save it as.  Location.  Location is used for redirecting and we'll have an exercise on that.  Usually you want to provide a response code that has, that says redirect and then you  provide the URL to redirect to in the location header.  Set cookie, we talked about cookies for maintaining state, maintaining sessions.  That cookie tells the browser to remember this token and it's just roughly a random  string of characters and when it gets sent back by the browser, the server knows to associate  it with a session.  It looks it up in the database.  www.authenticate is basically telling the browser that to pop up the box or if they've  already typed their username and password to provide that information to the server.  So we'll run into proxies.  We'll have some exercises on proxies.  So proxy is something that acts in place of another.  In the case of HTTP proxy, a web proxy, it intercepts our HTTP requests.  So it can modify that request.  So what it does is it intercepts a request, modifies it probably, does some caching perhaps  and sends it onto the server, gets the response back from the server, can modify it then again  and then sends it back to the client.  So it sits in between the client and the server and it can modify pretty much anything that  it sees in there.  So proxies are good for caching.  You can add security.  So actually our exercises, we will add some SSL to our Rails app.  So you do this to simplify so you don't have to have the Rails app understand SSL.  It also can save some CPU time.  Also can be used for load balancing.  It can be used for authentication.  I've had it where I had Apache in front of an application server and we added the pop-up  authentication for it with Apache acting as a reverse proxy which I'll talk about in  a second.  So there are transparent proxies.  Basically you don't have to set anything up.  It's inputted itself into the stream of the network and there's non-transparent proxies  which a lot of times if you're at work and at a big company, you'll have to configure  a browser to point at the proxy.  That's a non-transparent proxy.  And we've got an exercise on that.  Reverse and forward proxies.  So the proxy you have at work that sits sort of right next to the firewalls would be a  forward proxy.  A reverse proxy sits next to the server.  So right near the servers, anything that's, any proxy that's added on the server end  is a reverse proxy.  And we will actually have exercises on both of those.  A CDN is a content delivery network.  It's basically a paid service that does proxying for caching purposes.  So you can cache all your static content.  Technical things in here I won't get too much into.  One of the nice things that can provide you, protect you from distributed denial of services.  If you've got a big site, you should probably be looking into these.  Troubleshooting.  So any network problem you have, you have to think about the OSI model.  And I wish I had a picture of that.  I forgot to put that on there.  So you've basically got the physical layer.  You've got the network layer.  You've got the transport layer, which is TCP.  And then you've got your applications sitting on that.  So there's a lot of things that could go wrong.  You could have a network cable pulled out.  You could have, you know, your routers down.  Or the server's not running.  So all those different layers you have to think about when you have a problem.  Troubleshooting is trying to figure out which layer it is.  And then narrowing down on that.  So one of the first tools is ping.  Can I connect to the IP address that the server is on?  The problem with that is sometimes the firewalls will prevent that, either firewall in your  company or the firewall on the other end.  Tracerout is similar, but it shows you all the hops in between.  So maybe the network is down between your internet provider and Google.  Tracerout might be able to help you find that out.  Tell that we're going to use in our exercises.  It's a good way to tell if the port is listening.  And if I've got connectivity to the IP address, tell that will tell me if the service is listening.  And that's actually kind of where I start.  I kind of start in the middle.  And then if tell that doesn't work, I'll try the lower layers.  And if it does work, I'll try the upper layers.  If you're on your server and you want to see if your service is listening, you can do a  net stat on Linux.  That's dash PLA and T. It's easy, mnemonic for me to remember.  On Mac, it doesn't have all those options.  You'd use dash NA and then grab for listen.  And then so that's going to list over on the left side.  It's going to list all the IP addresses, which is usually going to always be your IP address.  And then colon in the port number.  So if you're looking to see if Rails is running, look on the left side for something colon  3,000, assuming you use the default port for Rails.  Telnet doesn't work with HTVS because it's encrypted.  So you have to use this tool that OpenSSL provides called sClient.  And we've got an exercise on that.  TCP dump, if you really want to see all the details on what's going across the wire, it  will tell you everything.  We've got a short exercise on that.  There's so much information, we could probably spend a couple hours on that.  Wireshark is a good way to visually see what's going on.  You can actually take the output of TCP dump, save it to a file, pull it into Wireshark.  Wireshark has a nice feature.  TCP dump shows you each individual packet and their disjoint.  When you're having communication between a client and a server, your packets can only  be so big.  So the communication will be broken up into pieces.  Wireshark has a nice feature to put all those back together, which is kind of cool.  All right.  So as I said, HTTP2 was recently approved and ratified.  It came out of a project at Google called Speedy, SPDY.  Apparently, they wanted to make it Speedy to say, to write the word Speedy.  So as I said, we can compress, we can Gzip the body, but we can't, in HTTP 1.1, we can't  compress the headers.  HPEC is a part of HTTP2 that allows header compression.  Another thing, the standard doesn't require it, but every HTTP2 implementation out there  requires TLS.  And that's probably because it's got TLS or SSL has a protocol and negotiation built  into it.  And that protocol and negotiation can say, hey, do you have HTTP2?  I'd like to use that.  Do you have Speedy 3?  I'd like to use that, but if not, fall back to HTTP1.  So I said that HTTP is all text.  HTTP2, that turns out not to be the case because we're compressing the headers.  It has the same semantics.  You won't be able to use Telnet.  You won't be able to use TCP dump, but you will be able to use some of the other tools  that we'll be looking at today.  I did get an example to work with HTTP2 that we can do as an exercise.  When you're making a lot of small requests, like let's say you've got a lot of icons on  your page, ideally you would just get each icon individually, but those icons are only  25K or something.  At that point, the headers at a couple K are starting to become large overhead.  If we could compress those down to a couple dozen bytes, it would save a lot of bandwidth.  There's some other features in HB2 we'll get to in a sec.  Basically just to save bandwidth.  The other thing about HTTP2 is it multiplexes the connections.  You can actually have multiple files coming across at the same time with a single TCP connection.  Right now your browser has to make multiple TCP connections.  I don't know what the browsers are up to.  They started at four at a time.  They got up to eight at a time.  That means you've got... the TCP connection takes up resources and it takes time to set  up.  If you could do just one TCP connection, that would save some time.  That's one thing they've done.  You can grab your HTML file and you can actually start processing and see, hey, I've got some  JavaScript, I've got some images, I've got some CSS, I need to grab all those too.  I can grab all those simultaneously.  That's going to save a lot of time.  Server push.  The server can actually know, hey, he's getting this HTML file.  He needs a CSS file too.  I'm going to start giving it to him even if he didn't ask for it.  The client can say, well, I'll take that.  Hey, I finished with the HTML file.  I didn't need that CSS file.  You can stop giving that to me.  Which is kind of weird with caching.  I don't know how it knows what's been cached and what doesn't.  The possibility is there to save some time.  When we do web design and web development, we've got the asset pipeline.  That combines all our JavaScript files into one big file.  It combines the CSS.  We probably combined that into one file.  The images, we probably do something called spriting where we put a bunch of images in  one file.  Then the CSS has to go grab each piece to put each icon on there.  If you look at the, I know I've looked at the homepage for Yahoo.  It's got all the icons on the left.  Those are all in one file.  The CSS gymnastics we have to do is a pain in the butt.  HTTP2 will hopefully allow us to stop doing that.  We can just write it the way that it should have been written in the first place.  Not worry about performance.  HTTP2 should do that for us.  HTTP2 is kind of weird that it starts with a 1.1 connection and then it upgrades.  The semantics of that are pretty crazy.  You probably don't want to get involved in that.  We'll see a little bit about that in the verbosity of the tools we use.  We'll show you a little bit of that.  Here is HTTP2 working.  Chrome 41 has it.  Firefox 36.  IE 11 but only in Windows 10.  Nobody has that yet except in beta.  Engine X says they're going to support it by the end of 2015.  They already support speedy 3.1, which is pretty darn close.  I could not get Curl to work with Engine X using speedy or HTTP2.  Call for.  Right now you have to specify the HTTP2 flag.  For this I had to manually compile that feature in.  Wireshark 2 will have it.  They're currently in the beta series 1.99.  Patches doesn't seem to have plans, which seems a little weird, but it does have mod  speedy available.  All right.  So it's time for the exercises.  I need to update that URL real quick.  But Charlie will be walking around helping people get started.  The exercises are basically starting on page 27 of this.  And so the first step is basically add the vagrant box with that command there.  Do a vagrant up and then a vagrant SSH.  You'll be in the box.  And then you can move on to slide 28.  Will Rails support HTTP2?  Probably not for a long time.  So Rails really doesn't even support HTTP.  Your Unicorn or your Puma are what supports that.  So when those support HTTP2, we will have it built into Rails.  Rails itself sort of sits right behind that.  Right now if you want it, you would put a proxy in front.  You would put InginX, is my recommendation.  In fact, InginX said they have 95% of the, well, it was speedy at the time, servers on  the internet.  So you would use a reverse proxy.  Well, Rails also doesn't support HTVS.  So you probably want HTVS on your site, so you probably need that proxy anyway.  Another question?  Other link?  You should see this version without the vagrant init.  If you did the vagrant init, you'll need to get the vagrant file back.  And I think you'll need to do vagrant box remove.  But raise your hand if you did the init and it will help you out.  So once you guys get into vagrant with the vagrant SSH, just work through the slides.  Raise your hands if you have any questions and we'll come around to help.  Ok.  You see the space again.  It returns.  Well, it does.  I think it does.  You said that it was long.  I think it's done.  It's been pretty fascinating to look at.  Oh, no, I didn't.  Yeah, it's done.  Yeah, it's done.  It's done.  It's done.  I'll just paste it in.  Yeah, you can put the save text once.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  To have everything worked.  Okay.  That is a thing that's doing better.  Okay.  It's in the virtual box.  Oh, sorry.  I was for a little bit of a  we for the interview  we'll be back in a minute.  We'll be back in a minute.  We'll be back in a minute.  We'll be back in a minute.  We'll be back in a minute.  We will answer her question...  There is one asking... ...  that will pick up and you can think  it in that President,  or Some ignore me  They might not agree.  In question,  I think the next thing we're gonna do is start with the end.  I don't know why I had to.  Hey folks, for the HV1.0 and 1.1, make sure you enter the blank line when you're using  Telnet.  If you don't use a blank line, I'm just sitting there waiting for you.  I'm gonna be doing this now.  I thought it was odd to make it so he's got a big spot.  I think you're all right there.  He's going to start.  I just want to try to make a different spot.  From the right, the third, the rail.  Oh, yeah.  You've got a response from some of the other people.  Yeah.  They should be able to do something like get.  To get in the...  I don't remember if you tell them that.  Well, if you can propose that you should get in.  Oh, I think it was.  I think just a...  Oh, yeah.  Yeah.  Good.  Yeah.  I'm going to try to get some.  I actually don't know how.  I just want to get some.  Do you have some running background?  Um...  Yeah.  Oh, I have old tips for my guys.  I'm going to try to get some.  Yeah.  You want me to tell them to go?  So I'll call them on a business side.  Send the kit part.  First one.  Yeah.  Do I need to just press enter now?  And don't hit enter or something like that.  No.  I'm going to go back to it.  We'll be able to tell them what to come to the table.  There you go.  We did.  If people have a speaker, we can do a few more things like this.  We can do a few more things like this.  We'll be able to still get more.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  Okay.  We'll be able to tell them what to come to the table.  Okay.  We'll be able to tell them what to come to the table.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.
exploration_0_47.png exploration_48_155.png exploration_156_287.png exploration_288_335.png exploration_336_1427.png exploration_1428_2039.png exploration_2040_2435.png exploration_2436_2555.png exploration_2556_3011.png exploration_3012_3347.png exploration_3348_4307.png exploration_4308_5051.png exploration_5052_6011.png exploration_6012_6479.png exploration_6480_6647.png exploration_6648_6743.png exploration_6744_6923.png exploration_6924_7427.png exploration_7428_7571.png exploration_7572_9431.png exploration_9432_10223.png exploration_10224_11231.png exploration_11232_11579.png exploration_11580_15239.png exploration_15240_15623.png exploration_15624_15779.png exploration_15780_15851.png exploration_15852_16283.png exploration_16284_19007.png exploration_19008_19571.png exploration_19572_20423.png exploration_20424_21911.png exploration_21912_21995.png exploration_21996_22055.png exploration_22056_24491.png exploration_24492_25247.png exploration_25248_27947.png exploration_27948_28355.png exploration_28356_29351.png exploration_29352_31547.png exploration_31548_31859.png exploration_31860_32243.png exploration_32244_33755.png exploration_33756_34847.png exploration_34848_36671.png exploration_36672_39215.png exploration_39216_39683.png exploration_39684_43463.png exploration_43464_44099.png exploration_44100_44855.png exploration_44856_45299.png exploration_45300_45443.png exploration_45444_46991.png exploration_46992_47063.png exploration_47064_47159.png exploration_47160_47531.png exploration_47532_47579.png exploration_47580_47675.png exploration_47676_47759.png exploration_47760_47939.png exploration_47940_50579.png exploration_50580_53207.png exploration_53208_55835.png exploration_55836_58463.png exploration_58464_61091.png exploration_61092_63719.png exploration_63720_64403.png exploration_64404_66335.png exploration_66336_68963.png exploration_68964_71591.png exploration_71592_74219.png exploration_74220_75455.png exploration_75456_76847.png exploration_76848_79475.png exploration_79476_82091.png exploration_82092_84719.png exploration_84720_85379.png exploration_85380_87347.png exploration_87348_89975.png exploration_89976_92603.png exploration_92604_95231.png exploration_95232_97847.png exploration_97848_100475.png exploration_100476_103103.png exploration_103104_103967.png exploration_103968_104291.png exploration_104292_105731.png exploration_105732_108359.png exploration_108360_109895.png exploration_109896_110507.png exploration_110508_110579.png exploration_110580_110627.png exploration_110628_110675.png exploration_110676_110723.png exploration_110724_110783.png exploration_110784_110891.png exploration_110892_110904.png
Alright, folks, I'm going to get started here. Thanks for coming out. USB keys have two files you'll need, the Vagrant file and the HTTP exploration box file. If you don't have Vagrant and VirtualBox, there are executables on there for Windows, Linux, a couple of variants of Linux and Mac. You'll need to have those installed. I'm going to do a half hour lecture, hopefully a little shorter than that, and then we'll have about an hour to run through the exercises. The exercises will walk you through starting up Vagrant and everything you need to do. So I have Charlie Sanders helping me out. Raise your hand if you need a USB key to get the files from. So basically you've got half an hour to get all that working. If you don't, don't worry too much, you can pair up with the neighbor. I would appreciate feedback, negative or positive. You can tweet me at Craig Buchek. Email address is up there. My presentations are on GitHub. I haven't put the latest versions of this presentation yet, but it should be there tomorrow. If you guys want to follow along, TinyCC HV exploration with an underscore. To take you to this presentation, actually I think I need to update that. I'll do that when we get to the exercises. The exercises will begin there too, so I'll update them when we start the exercises. The reason I started doing this is I actually have a previous life as an assistant in a network admin. I've actually done a lot of troubleshooting of HTTP, going through networks, going through firewalls, and now I'm a Rails developer. So I've seen both sides. So we're going to talk about HTTP basics, request, responses, talk about proxies, some troubleshooting, and HTTP2. And then we'll get into the exercises that touch on all of those. So HTTP has been around since 1991. The first version didn't actually have a version number, but retroactively we call it 0.9. It was standardized in 96. The version that we typically use now is standardized in 2007. That RSC is very handy if you have any questions about how HTTP works. I was just updated a few months ago. It was broken up into multiple pieces, basically in preparation for HTTP2. HTTP2 has been ratified, standardized, but it hasn't been published yet. But that's a good URL to check out if you're interested. So HTTP is stateless, which means that each time you connect to the server, it doesn't really remember what you did the last time you connected to the server. And the only way around that is through cookies, pretty much. It is text-based. That's kind of the whole point of the presentation. We're going to look at the text that's going across the wire. And it's a request-response cycle. Your client, your browser, will make a request to the server. The server will respond. So I want to talk real quick about URLs. So pretty much every piece of that URL on the screen will go from the client to the server except for the fragment. The fragment just tells the browser to go to this specific spot on the page. The scheme is going to be HTTP. The host is obvious. You can specify username and password in the URL, recommend against that, but it's possible. And then the path is actually the piece that goes that we'll be working with for the most part. All right. So the HTTP request looks like this. That thing in green is called the method. So we'll talk about that in a bit. The next thing is past as the URL. Usually it's not the full URL. Usually it's just slash. It starts with a slash and it's relative to the top of the site. And then you have the version of HTTP specified there. In blue are the headers. So the headers are sort of metadata about what you want to get or what you want to do. In this case, there's a host header and a content length header. And the body, not all requests have a body. If you're just getting something, it won't have a body. But if you're posting something or putting something, it will have a body. In this case, I tried to put something to Google's front page. Probably not going to work. I actually tried that. And it gives a 405 error, which is hard to find. So that's the body text, basically the content that you want to upload. So the methods I talked about, normally when you go to a little website, you're going to do a get request. They get this page. You go to Google's front page, you do a get. A post is when you want to update something or when you submit a form. Although some forms can actually do get, depending on if you're changing something or just doing a query. Like when you go to Google's front page, that form, when you type in the query, that's a get because you're actually just asking for information you're not asking to change anything. The put is actually, it's an update, or actually it's more of an upsert. You're saying, I'm giving you the whole thing. Replace whatever you got if you have anything. Otherwise, create it. Rails is a little weird on that. It doesn't actually do the create part of that. So it uses put a little bit oddly. Delete does what it says. If you have permission, you actually can delete pages or resources through the web. Get is basically a get request but without the body. You're saying, show me the headers that I would get if I did the get request for this URL. There's a concept of safe methods. So safe methods don't have any effect on information. When I talked about the post, it changes something. When you submit a form, it changes something on the server. A get is not that case. So the takeaway here is don't have your app change things when you do a get because when Google comes scraping your site, it's going to do gets on your site. If your site changes something on the back end, you're going to have a bad day. So just keep safety in mind. There's also something called item potency. That means that you can call the same thing multiple times. You can do a get to the Google front page multiple times. You'll get the same thing. Same thing with the head. Same thing with if you delete resource, the resource is going to be gone if it was there or not. A put, you're actually saying replace something. So if it's there, replace it. If it's not there, allow it. So multiple times doesn't matter. The one that's missing from this list, again, is post. So if you post something multiple times, you'll end up with multiple copies of that thing. So request headers. We saw that we can provide those headers. These are some common headers. The host header is basically required. It says the name of the website we're accessing. And the reason that that was added is so you could host multiple websites on a single server. The accept header says list the types that the browser would like to receive. So the browser may say that it likes to have HTTP, it likes to have JSON, but not list XML. So if the server knows how to do HTML and JSON, it will pick one of those, but it wouldn't pick XML unless it didn't know anything else. The weird thing about the accept headers, you can actually specify relative quality. Sometimes you'll see a queue number if you look at the accept header. And that says I prefer these types of things over the other things. The default quality setting is one, which means top priority. If you prefer JSON over XML, you could set its quality to 0.9 and XML is to 0.8. The content length on a request header is the content of the body of the request that you're sending. And if you don't specify that, it actually just doesn't let you specify anything. Once you've uploaded the content length number of bytes in the body, the server will close the connection. The content type tells what you're uploading. Am I uploading some HTML? Am I uploading some XML? Am I uploading some JSON? The referrer is if you are a browser, it's the page you were on when you clicked the link. That is spelled incorrectly, but it's in the standard document, so that's what we use. The user agent is another name for a browser or a web client. It could be a web crawler, it could be various things. It's a string containing the information about the browser. If you take a look at those, those get to be pretty crazy long, especially in Chrome and Firefox. They try to pretend. Back in the day, there was something called browser sniffing. The server would look at that string and try to determine, oh, I'm Internet Explorer, I'm going to give you this copy of the page, and your Firefox, I'm going to give you this copy of the page, which could be completely different. Google didn't like that because Google wants there to be one canonical version of every page that it indexes. It doesn't want you to fool it, so it sees one thing and then the user sees something else. Browser sniffing is not used too much anymore at all. You end up with these really long strings that tell you all the backwards compatibility that the browser tries to do. We'll take a look at those headers in the exercises. Yeah? Yeah. It was you who requested better and accept. Sorry, I'm sorry. Yeah. Comment like accept. Accept is what you, you said what you want to expect back. Yes, yes. And then content type is only for a post or a put, and it's what type the body that you're sending up to the server is. Yeah. So authorization is for, if you ever go to a website and you get the pop-up box, authorization is basically that information you type into the username and password box. It's not encrypted, it is hashed and sent. And we're actually going to, there's an exercise that we're actually going to find that going across the wire. Content encoding, you can actually gzip things. The server can gzip the content coming back to you. This basically says that's okay. You can say content encoding gzip, commit deflate. That tells the server, hey, I can gzip this and save some bandwidth across the wire. Connection usually is for keep alive. It is the client saying, hey, I've made this request, but when I'm done, I'm going to make another request. I'm going to make another turn down the TCP connection. I want to reuse it for the next request. So if I'm getting a web page and I know it's probably going to have some JavaScript, it's probably going to have some CSS, it lets me get those all in one without dropping the connection. Cookie. I talked about the, a cookie is, the server has sent us this cookie and we are supposed to return that cookie back to it. And that maintains sessions with the server, between the server and the client. And that's the only thing that maintains a session between the client and the server. And that header is limited to about 4K of information. So if we do our cookie-based sessions in Rails, we do have to worry about that limit. You don't want to store a whole lot in that cookie. All right. So we've made the request and the server response with the HP response. That first line is called the status line. It's got the HP version. Interesting thing, you can actually make a request for one version and get a different version back. I find that a little odd. The 201 there is our status code and then there's the description of the status code which is created in this case. And then like the request, the response actually also has headers. And then it has a body. Not every response has a body. There's a few that don't have to have a body. Created is actually one of those. So if you are doing a put or a post, it could actually just say created and then don't give you anything back. But you probably want to do a redirect. So you probably want some extra information in there maybe. So status codes. So the status codes have different meanings and they are actually standardized. Although occasionally you'll see some non-standard ones. The 100s are informational. You'll rarely see those. The 200s are what you'll see most of the time. Usually for a get, you're going to get a 200. For a post or a put, you're probably going to see a 201. Redirection, if you go to google.com, it'll actually redirect you to www.google.com. That's a redirect. I believe that would be a 301. If you send headers that are involved with caching, which we'll do in an exercise, you'll get a sometimes you'll get a not modified. That says, hey, you've told me you already have a copy of this in your cache. So I'm just going to give you the metadata and the headers. I'm not going to give you the body. You should use the body that I gave you last time. So then we've got the error response codes. So 400s are client errors where the client made some sort of mistake. First is if you have not sent an authenticate header and the web server requires you to authenticate, the client is supposed to retry with the authenticate header. And so before it does that, it pops the box up, has you type in your username and password, resends with that header. Forbidden means you've probably either you can't authenticate or you've authenticated, but you still don't have permissions because maybe you're not an administrator in the box. 404, we've all seen that one before, page not found. 407 is proxy authentication required. It's like a 401, except it's the non-transparent proxy that is asking for a username and password. That's pretty rare to see. We will do a little bit of proxies, but we're not going to do any proxy authentication. 422 is unprocessable entity, which is a weird way of saying I don't understand what you're trying to ask me. That is recommended. That's what I would recommend, either 422 or 409 if you are making an API request and it doesn't have the right information that it needs in the JSON or XML or whatever that you sent. Server errors. If your server crashes really badly, you will see a 500 error. I believe you see that in Rails a lot of times when you're debugging. 502 and 504 are gate-wears. If you've got a reverse proxy sitting in front of your servers and your servers have gone down or take too long to respond, you'll see a 502 or a 504. There are plenty of others. I ran into a 405 earlier. It's one of the April 1st RFCs called I'm a teapot response, 418 code. Not sure when you'll need that. I'm sure someone has implemented it, though. Response headers. So like a request, the response has headers. The content length will almost always be there because you'll have a response that has content in the body. The content type, it's the mine type. So it's going to be like text slash HTML, text slash plain, application slash JSON, or image slash JPEG, image slash PNG. That tells the client what type of file this is and then it can handle that however it expects to. The content encoding is, I talked about accept encoding where you can Gzip it. The content encoding says that the body has been GZipped. Notice that the body is GZipped, but the headers are actually still in plain text. Content disposition is a little trick that you use when you want to, when the person clicks on the link, you want them to download the file instead of have the file display in the browser. In that case, you would use content disposition header and you can also provide a default file name that the browser will try to save it as. Location. Location is used for redirecting and we'll have an exercise on that. Usually you want to provide a response code that has, that says redirect and then you provide the URL to redirect to in the location header. Set cookie, we talked about cookies for maintaining state, maintaining sessions. That cookie tells the browser to remember this token and it's just roughly a random string of characters and when it gets sent back by the browser, the server knows to associate it with a session. It looks it up in the database. www.authenticate is basically telling the browser that to pop up the box or if they've already typed their username and password to provide that information to the server. So we'll run into proxies. We'll have some exercises on proxies. So proxy is something that acts in place of another. In the case of HTTP proxy, a web proxy, it intercepts our HTTP requests. So it can modify that request. So what it does is it intercepts a request, modifies it probably, does some caching perhaps and sends it onto the server, gets the response back from the server, can modify it then again and then sends it back to the client. So it sits in between the client and the server and it can modify pretty much anything that it sees in there. So proxies are good for caching. You can add security. So actually our exercises, we will add some SSL to our Rails app. So you do this to simplify so you don't have to have the Rails app understand SSL. It also can save some CPU time. Also can be used for load balancing. It can be used for authentication. I've had it where I had Apache in front of an application server and we added the pop-up authentication for it with Apache acting as a reverse proxy which I'll talk about in a second. So there are transparent proxies. Basically you don't have to set anything up. It's inputted itself into the stream of the network and there's non-transparent proxies which a lot of times if you're at work and at a big company, you'll have to configure a browser to point at the proxy. That's a non-transparent proxy. And we've got an exercise on that. Reverse and forward proxies. So the proxy you have at work that sits sort of right next to the firewalls would be a forward proxy. A reverse proxy sits next to the server. So right near the servers, anything that's, any proxy that's added on the server end is a reverse proxy. And we will actually have exercises on both of those. A CDN is a content delivery network. It's basically a paid service that does proxying for caching purposes. So you can cache all your static content. Technical things in here I won't get too much into. One of the nice things that can provide you, protect you from distributed denial of services. If you've got a big site, you should probably be looking into these. Troubleshooting. So any network problem you have, you have to think about the OSI model. And I wish I had a picture of that. I forgot to put that on there. So you've basically got the physical layer. You've got the network layer. You've got the transport layer, which is TCP. And then you've got your applications sitting on that. So there's a lot of things that could go wrong. You could have a network cable pulled out. You could have, you know, your routers down. Or the server's not running. So all those different layers you have to think about when you have a problem. Troubleshooting is trying to figure out which layer it is. And then narrowing down on that. So one of the first tools is ping. Can I connect to the IP address that the server is on? The problem with that is sometimes the firewalls will prevent that, either firewall in your company or the firewall on the other end. Tracerout is similar, but it shows you all the hops in between. So maybe the network is down between your internet provider and Google. Tracerout might be able to help you find that out. Tell that we're going to use in our exercises. It's a good way to tell if the port is listening. And if I've got connectivity to the IP address, tell that will tell me if the service is listening. And that's actually kind of where I start. I kind of start in the middle. And then if tell that doesn't work, I'll try the lower layers. And if it does work, I'll try the upper layers. If you're on your server and you want to see if your service is listening, you can do a net stat on Linux. That's dash PLA and T. It's easy, mnemonic for me to remember. On Mac, it doesn't have all those options. You'd use dash NA and then grab for listen. And then so that's going to list over on the left side. It's going to list all the IP addresses, which is usually going to always be your IP address. And then colon in the port number. So if you're looking to see if Rails is running, look on the left side for something colon 3,000, assuming you use the default port for Rails. Telnet doesn't work with HTVS because it's encrypted. So you have to use this tool that OpenSSL provides called sClient. And we've got an exercise on that. TCP dump, if you really want to see all the details on what's going across the wire, it will tell you everything. We've got a short exercise on that. There's so much information, we could probably spend a couple hours on that. Wireshark is a good way to visually see what's going on. You can actually take the output of TCP dump, save it to a file, pull it into Wireshark. Wireshark has a nice feature. TCP dump shows you each individual packet and their disjoint. When you're having communication between a client and a server, your packets can only be so big. So the communication will be broken up into pieces. Wireshark has a nice feature to put all those back together, which is kind of cool. All right. So as I said, HTTP2 was recently approved and ratified. It came out of a project at Google called Speedy, SPDY. Apparently, they wanted to make it Speedy to say, to write the word Speedy. So as I said, we can compress, we can Gzip the body, but we can't, in HTTP 1.1, we can't compress the headers. HPEC is a part of HTTP2 that allows header compression. Another thing, the standard doesn't require it, but every HTTP2 implementation out there requires TLS. And that's probably because it's got TLS or SSL has a protocol and negotiation built into it. And that protocol and negotiation can say, hey, do you have HTTP2? I'd like to use that. Do you have Speedy 3? I'd like to use that, but if not, fall back to HTTP1. So I said that HTTP is all text. HTTP2, that turns out not to be the case because we're compressing the headers. It has the same semantics. You won't be able to use Telnet. You won't be able to use TCP dump, but you will be able to use some of the other tools that we'll be looking at today. I did get an example to work with HTTP2 that we can do as an exercise. When you're making a lot of small requests, like let's say you've got a lot of icons on your page, ideally you would just get each icon individually, but those icons are only 25K or something. At that point, the headers at a couple K are starting to become large overhead. If we could compress those down to a couple dozen bytes, it would save a lot of bandwidth. There's some other features in HB2 we'll get to in a sec. Basically just to save bandwidth. The other thing about HTTP2 is it multiplexes the connections. You can actually have multiple files coming across at the same time with a single TCP connection. Right now your browser has to make multiple TCP connections. I don't know what the browsers are up to. They started at four at a time. They got up to eight at a time. That means you've got... the TCP connection takes up resources and it takes time to set up. If you could do just one TCP connection, that would save some time. That's one thing they've done. You can grab your HTML file and you can actually start processing and see, hey, I've got some JavaScript, I've got some images, I've got some CSS, I need to grab all those too. I can grab all those simultaneously. That's going to save a lot of time. Server push. The server can actually know, hey, he's getting this HTML file. He needs a CSS file too. I'm going to start giving it to him even if he didn't ask for it. The client can say, well, I'll take that. Hey, I finished with the HTML file. I didn't need that CSS file. You can stop giving that to me. Which is kind of weird with caching. I don't know how it knows what's been cached and what doesn't. The possibility is there to save some time. When we do web design and web development, we've got the asset pipeline. That combines all our JavaScript files into one big file. It combines the CSS. We probably combined that into one file. The images, we probably do something called spriting where we put a bunch of images in one file. Then the CSS has to go grab each piece to put each icon on there. If you look at the, I know I've looked at the homepage for Yahoo. It's got all the icons on the left. Those are all in one file. The CSS gymnastics we have to do is a pain in the butt. HTTP2 will hopefully allow us to stop doing that. We can just write it the way that it should have been written in the first place. Not worry about performance. HTTP2 should do that for us. HTTP2 is kind of weird that it starts with a 1.1 connection and then it upgrades. The semantics of that are pretty crazy. You probably don't want to get involved in that. We'll see a little bit about that in the verbosity of the tools we use. We'll show you a little bit of that. Here is HTTP2 working. Chrome 41 has it. Firefox 36. IE 11 but only in Windows 10. Nobody has that yet except in beta. Engine X says they're going to support it by the end of 2015. They already support speedy 3.1, which is pretty darn close. I could not get Curl to work with Engine X using speedy or HTTP2. Call for. Right now you have to specify the HTTP2 flag. For this I had to manually compile that feature in. Wireshark 2 will have it. They're currently in the beta series 1.99. Patches doesn't seem to have plans, which seems a little weird, but it does have mod speedy available. All right. So it's time for the exercises. I need to update that URL real quick. But Charlie will be walking around helping people get started. The exercises are basically starting on page 27 of this. And so the first step is basically add the vagrant box with that command there. Do a vagrant up and then a vagrant SSH. You'll be in the box. And then you can move on to slide 28. Will Rails support HTTP2? Probably not for a long time. So Rails really doesn't even support HTTP. Your Unicorn or your Puma are what supports that. So when those support HTTP2, we will have it built into Rails. Rails itself sort of sits right behind that. Right now if you want it, you would put a proxy in front. You would put InginX, is my recommendation. In fact, InginX said they have 95% of the, well, it was speedy at the time, servers on the internet. So you would use a reverse proxy. Well, Rails also doesn't support HTVS. So you probably want HTVS on your site, so you probably need that proxy anyway. Another question? Other link? You should see this version without the vagrant init. If you did the vagrant init, you'll need to get the vagrant file back. And I think you'll need to do vagrant box remove. But raise your hand if you did the init and it will help you out. So once you guys get into vagrant with the vagrant SSH, just work through the slides. Raise your hands if you have any questions and we'll come around to help. Ok. You see the space again. It returns. Well, it does. I think it does. You said that it was long. I think it's done. It's been pretty fascinating to look at. Oh, no, I didn't. Yeah, it's done. Yeah, it's done. It's done. It's done. I'll just paste it in. Yeah, you can put the save text once. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. To have everything worked. Okay. That is a thing that's doing better. Okay. It's in the virtual box. Oh, sorry. I was for a little bit of a we for the interview we'll be back in a minute. We'll be back in a minute. We'll be back in a minute. We'll be back in a minute. We'll be back in a minute. We will answer her question... There is one asking...... that will pick up and you can think it in that President, or Some ignore me They might not agree. In question, I think the next thing we're gonna do is start with the end. I don't know why I had to. Hey folks, for the HV1.0 and 1.1, make sure you enter the blank line when you're using Telnet. If you don't use a blank line, I'm just sitting there waiting for you. I'm gonna be doing this now. I thought it was odd to make it so he's got a big spot. I think you're all right there. He's going to start. I just want to try to make a different spot. From the right, the third, the rail. Oh, yeah. You've got a response from some of the other people. Yeah. They should be able to do something like get. To get in the... I don't remember if you tell them that. Well, if you can propose that you should get in. Oh, I think it was. I think just a... Oh, yeah. Yeah. Good. Yeah. I'm going to try to get some. I actually don't know how. I just want to get some. Do you have some running background? Um... Yeah. Oh, I have old tips for my guys. I'm going to try to get some. Yeah. You want me to tell them to go? So I'll call them on a business side. Send the kit part. First one. Yeah. Do I need to just press enter now? And don't hit enter or something like that. No. I'm going to go back to it. We'll be able to tell them what to come to the table. There you go. We did. If people have a speaker, we can do a few more things like this. We can do a few more things like this. We'll be able to still get more. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. Okay. We'll be able to tell them what to come to the table. Okay. We'll be able to tell them what to come to the table. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay.
we're going to talk about HTTP basics, request, troubleshooting, and HTTP2. And then we'll get want to get or what you want to do. In this case, compatibility that the browser tries to do. We'll web server requires you to authenticate, the to the client. So it sits in between the client our Rails app. So you do this to simplify so you visually see what's going on. You can actually about that in the verbosity of the tools we use. don't know how. I just want to get some. Do you