Check Yourself Before You Wreck Yourself
Jansma, Nic
Boomerang is an open-source Real User Monitoring (RUM) JavaScript library used by thousands of websites to measure their visitor's experiences. The developers behind Boomerang take pride in building a reliable and performant third-party library that everyone can use without being concerned about its measurements affecting their site. We recently performed and shared an audit of Boomerang's performance, to help communicate its "cost of doing business", and in doing so we found several areas of code that we wanted to improve. We'll discuss how we performed the audit, some of the improvements we've made, how we're testing and validating our changes, and the real-time telemetry we capture for our library to ensure we're having as little of an impact as possible on the sites we're included on. Boomerang is an open-source Real User Monitoring (RUM) JavaScript library used by thousands of websites to measure their visitor's experiences. Boomerang runs on billions of page loads a day, either via the open-source library or as part of Akamai's mPulse RUM service. The developers behind Boomerang take pride in building a reliable and performant third-party library that everyone can use without being concerned about its measurements affecting their site. Recently, we performed and shared an audit of Boomerang's performance, to help communicate the "cost of doing business" of including Boomerang on a page while it takes its measurements. In doing the audit, we found several areas of code that we wanted to improve and have been making continuous improvements ever since. We've taken ideas and contributions from the OSS community, and have built a Performance Lab that helps "lock in" our improvements by continuously measuring the metrics that are important to us. We'll discuss how we performed the audit, some of the improvements we've made, how we're testing and validating our changes, and the real-time telemetry we capture on our library to ensure we're having as little of an impact as possible on the sites we're included on.
 Hello everybody. First of all, I'm honored here to kick off this web performance dev room here at FASDEM. Definitely want to thank the Wikimedia performance team for organizing this and for you all to be here today. Thanks for coming to listen to me and the others. My name is Nick Jansma. I work at Akamai. I work on our impulse product. And specifically, I work on an open source library called Boomerang. And today I'm going to talk about the performance audit that we gave on Boomerang and what the findings that we found, the things that we improved and kind of how we've made it better. So why are we here today? So Boomerang is an open source real user monitoring library. It monitors performance. When you visit a website, if Boomerang is on it, it's going to capture all the performance data that we can, how fast it took, characteristics of the page itself. We at Akamai are the primary contributors to Boomerang. Here's the GitHub repository if you care about it, but it is an open source product. You are welcome to use Boomerang if you have your own back end that you want to send the data to. You can send it to there. If you want to use our product impulse, we will be the real time dashboards for you to show all this data. So we are a third party, if you will. We provide a script that other people can include on their websites to capture this data. In building a third party, especially real user monitoring script, we think it's very important to try to not affect the page too much, to minimize our costs. It'd be quite a shame if we were a performance monitoring product that also had an effect on the performance of the pages that we're on. In fact, our customers, people that pay for impulse, have been increasingly sophisticated in making sure and asking us whether we have a cost when they're including us on their website. So two years ago, what we did was we decided to take a step back and look at from a holistic perspective, what is the cost of including Boomerang on your page? What is the cost of including the script? We wanted to not only understand it better for ourselves, we wanted to look to see if there were places that we could improve, and we wanted to be able to share this with our customers. We think it's very important to be transparent. I think it builds a lot of trust with our customers to show the good and the bad with what you have, and to help the customer understand the benefit that you're bringing along with the overhead that it takes. So for the past two years, we've been working on performance as one of our main features for the product. We're really trying to improve Boomerang consistently over time. And this is the kind of stuff that makes me passionate, makes our team passionate. We have a team of five developers, a couple of them are here today. We love performance, right? And so this is what I'm going to talk about today. So why should you care? Why are you here? Well, for one, sorry, you're just stuck in the room with me, so you got to stick with it. We locked the doors. You can't escape. Just kidding. Maybe you're like me. Maybe you develop a third-party library. Does anybody else actually develop a library that other people use at all? For any reasons? A couple of hands in them. A lot of people, web developers, et cetera, use third-party libraries. They use other scripts to help build their websites. So maybe you've had this happen if you've been working at a website or a big company. Your boss comes by and says, hey, I really want you to include this one new library that we need for our website. It's just this one simple line of code, just a script tag. Please include it. Nothing can go wrong, right? This is very simple. Unfortunately, that one little line means a lot. Including that script on your page means it could stop your page from loading entirely. It could slow down your website. It could create incompatibilities with other libraries on the page, too. It could change from underneath you, especially if you're loading it from a CDN or from another place, that those JavaScript bytes could change without you even asking it to. And at the end of the day, it really has complete control over your web page. A JavaScript is running on your web page. It can do anything, intentionally or not. So I wanted to put this into some sort of context for us. As I said, we provide boomerang for our customers at Akamai. We're on over 14,000 of our customers' websites. We track and measure over a billion page loads a day, which is quite a lot. According to the open source statistics that I could find, we're on somewhere between 75,000 and almost a half a million websites, depending on what data sources you're looking at. So a lot of people are including the open source version of boomerang on their pages as well. And if you take all the different combinations of all the different pages that boomerang's on, all the different browsers that can load it, all the different people that can load it, and all the different geolocations that can load it, all the different other scripts that could be on the page, too, it's kind of scary. There's a really, really big matrix here of things that can go wrong. You know, you could have an incompatibility with another script. You could have edge cases for your performance, et cetera. So what I hope to do today is to share some of the things that we've learned when we've evaluated ourselves so that you might be able to do this as well for any library that you're taking a look at. So how would you evaluate the cost of a third-party script? How do people that want to include boomerang on their page, what questions should they be asking about what is its cost? And how can we, as a provider of a third-party script, convince our customers that the benefits outweigh those costs? So every script has a lot of different aspects to it, right? One of those aspects would be how big is it? What's the byte size? So taking an example here, modernizer.js, it's 97, or sorry, 92 kilobytes on a minified, 32 kilobytes GZipped. Is that good? Bad? I don't know. Honestly, it's just a number at the end of the day. You probably want to minimize this as much as possible. And size can be an important factor of a JavaScript library, but it's definitely not the only thing. It doesn't tell the whole picture. Resource weight is the concept of the total byte size of your page. So this includes all your JavaScript, all your images, everything else that's required to build your website, right? And it's very important. Every single byte matters when you're trying to improve the performance of your page. And lowering the byte size of different libraries, you know, choosing thinner libraries and stuff like that can really help for your performance budget. And while it's probably one of the easiest ways to judge a third party, just based on that size, it's definitely only just one aspect of it. So what we decided to do with our audit of our own code was kind of break up the life cycle of our script into five main phases. So first, you have your loader snippet or how the script gets on the page. Maybe you include this in your JavaScript bundle. Maybe you load it via tag manager. But the browser needs to know to load your script in the first place. The second thing, the browser goes and fetches the script itself. And obviously, the less bytes you have, the better here. The browser then needs to parse it and compile it, get it ready to be run. Finally, it runs it for the first time. So there's probably a lot of initialization that happens. Maybe registering global variables or event handlers. And then run time. So the reason that it's there in the first place, why is this library on your website? And all of these are important from the entire life cycle of the script. So what we did for boomerang is we decided to jump into the developer tools, profilers from Chrome and other, really kind of show you at a really high level what's going on and then give you details for all the specifics. So this is what boomerang looks like if you're loading it on an empty page. We start with the loader snippet, which is the way that we get boomerang onto the page in the first place. Our customers, Akamai load it from our CDN. Open source customers could bundle it into your application bundle or however you want to load it. Then we download it. Boomerang gets downloaded. The browser parses the JavaScript and executes it. We initialize. We set up some global variables. We register event handlers, et cetera. In our case for impulse, we actually go fetch some more configuration data via a JSON request. And then we initialize that. And then finally, the main reason that we're there is we're collecting all this performance information. We're gathering data on how long it took. And that's what we call our onload event handler. We package all that data up and we beacon it out. We send it back to the mothership for processing. So these are kind of the main five stages of our script loading. So keeping those phases in mind, let's take a look at how you might audit a new third-party script or how we audited boomerang when looking at these different phases. So the first question is, how does a new script get on a website? Some libraries suggest a very simple script tag, right? A script async tag or something like that. This tag itself has no intrinsic cost. The browser just knows to go fetch that data. Other libraries like Google Analytics, for example, have a small snippet. This usually is meant to load the script asynchronously. You can see in very small fonts right there that's kind of Google Analytics loader snippet. And this has a very minor cost, a couple milliseconds usually to execute it and then to trigger that download of the JavaScript. We at boomerang like to do things a little bit differently. We actually have a more complex loader snippet that we give to our customers. We don't use just a script async tag because while you can load JavaScript asynchronously, the browser will still block on that content before it fires the onload event. In other words, if you have even asynchronous script tag that's loading your analytics, if that analytics package takes 10 seconds to load, the browser is still going to be in its loading state. The loading indicator is still going to be there. The onload event won't fire, et cetera. What we want to do as a performance monitoring script is make sure that we're not blocking onload. We're not affecting the performance of the page that we're on. So we have this 40 lines of JavaScript essentially that tells the browser to load boomerang in an asynchronous and non-blocking manner. We use an iframe to do this or we used an iframe to do this. It's a little more complex. It actually costs a little bit more than the previous ways I mentioned. So in our case, it could be up to 40 milliseconds. And a lot of this is because the iframe itself creating an iframe in the browser actually has a cost to it. So it's a little more expensive. And we actually wanted to focus on this when we were doing our audit as one of the main things that we could try to see if we can improve on. Can we get down the cost while still maintaining the non-blocking nature of what we're giving to our customers? And if you want any more details on exactly how we do everything that we do, it is in the boomerang documentation on GitHub. You can read all about it. We've explained it quite a bit more there. Okay. So now the browser knows that you want to load your JavaScript, right? It's going to go fetch it from the network. This is when the download begins. Every byte that is downloaded affects the overall page weight. You know, every byte matters here. Depending on how the script is loaded, it could affect other things on the page too. If it's part of your main application bundle, for example, it will block all the stuff afterward. I know that a lot of people are using more exotic ways of loading JavaScript these days, like through modules and stuff. But it's important to keep in mind that everything that you are loading has a cost to it. And it's also really important to know that a lot of libraries that you will load, analytics libraries or other widgets, social widgets, et cetera, they'll often load additional data after they load the JavaScript. So they may load other CSS or images, get JSON from various places, for example, right? In fact, some of you may know this pretty cool tool from Simon Hearns called RequestMap. And it kind of lets you see all the different things that get fetched and what triggers other things to get fetched, et cetera. It's a good way of knowing kind of like the whole cost of a library when you load it. So it's on RequestMap.byperf.tools. Check it out. You could also use just things like web page test or looking at your network tab to see all the other things that are downloaded. So if we take a look at a couple of sample popular JavaScript libraries, we kind of just picked a sample of ones that I knew of, put them in order of size. We can actually see that boomerangs kind of near the high end of cost here. So simple things like underscore could just be a couple kilobytes. Big things like the D3 charting libraries, 70 kilobytes. Boomerang ends up at around 47 kilobytes, so a little less than 50 kilobytes. It's big, honestly. This is one of the things that we thought we could improve on. We are doing a lot of things in the library, but maybe there's ways for us to have it not be such a big library. Also, I did want to point out that the build that I was talking about is the impulse-specific version of Boomerang, which is a pretty big build for all of our customers. If you're using the open-source version of Boomerang, you can choose to build it smaller. We have a plugin architecture, and you can choose which plugins to include or not. So if you don't need single-page app support and JavaScript error tracking that we include for impulse, you could trim it down to about half the size, even. Okay, so now the browser is downloaded to your JavaScript bytes. What does it have to do? It actually has to parse and compile it. It can do this before it runs any of it. The main idea here is the more bytes it is, the more complex it is to parse, to compile, etc. So again, you want to minimize byte costs, because this does have a cost. For us, for Boomerang, on a modern browser, modern device, it's 10 milliseconds or so, so not a lot. But some of the bigger libraries, like Angular, can be quite a bit more 20, 25 milliseconds to just get it ready before it even runs. So again, building a smaller library can really help with a lot of this. Okay, second and last phase is initialization. So this is when the browser has everything ready and it hits run on the script. For every script, this is different, right? This is what they're doing on the page. They might be registering global variables or hooking in two different events, etc. They might be fetching more resources. Maybe it's a social widget. Hopefully it's only doing what you're asking it to do. In our case, for Boomerang, again, we register a couple variables and listen for the onload event and some other events depending on the features that are enabled. We don't do a lot of work here, only 10 milliseconds or so, not too bad, I would say. We did find a couple ideas for improvements here. We might defer some of the work that we're doing. We might break up some of the work that we're doing, but not too bad overall. And then finally, the runtime of the script itself. So again, depends exactly on what you're using the script for. If it's a utility script, maybe you're calling into it quite a bit. If it's a social widget, maybe it's loading all the likes for something. Maybe you have a Bitcoin miner, it's mining Bitcoin for whatever reason. For us, Boomerang, this is where Boomerang does the majority of its work. So after the page load has happened, we look at all the performance data for that page. We package it up, we compress it, we put it on a beacon, and we send it out. So we're capturing things like all the resources that were fetched on the page, how long it took to navigate, what the DNS time was, and a lot of other things like that. If there were any JavaScript errors, we package those up and send those as well. In a lot of the cases on most sites, this wasn't taking too long, say less than 50 milliseconds. But we did find in some examples, especially on lower end devices, on more complex web pages, that we were taking 300 milliseconds or more. And that's starting to get into the territory of people would notice it, a visitor would notice it, that we would be affecting actually the performance of the page. So this is one of the areas that we flagged that we really wanted to get into and figure out if we can improve. And I'll talk about some of the things that we found later. So one thing I did want to point out really quick too is all of the bold phases up there are part of the critical path of the browser using your JavaScript. All of the things up there are generally done serially on the main thread, depending on the browser. But any of these things that you're doing here are affecting the rest of the site being built. And if the user is trying to interact with the page, you're potentially affecting their interaction. So you're delaying the user input, et cetera. So what you really want to do when you have a third party library, a script like this, minimize the work that you're doing, bake it up. If you know you need to do a really big calculation, try to bake it up into pieces. This avoids things called long tasks, which are tasks that are running on the main browser thread that would potentially block user input if they're trying to click or scroll. It would make for a non-responsive user experience. So after everything that I said, these are kind of the numbers that we came up with at the end. We did a lot of investigations on a lot of different sites. We took a lot of profiles. We wrote about it. We published it in a blog post. And these are kind of the TLDR costs of boomerang. So the loader snippet, generally only a few milliseconds on some browsers. It can be up to 40 milliseconds. Downloading, we're about 50 kilobytes or less. Parsing, again, it's related to the size of boomerang. And this is kind of like the low end to high end devices when we were looking at it. Anywhere from 6 to 47 milliseconds. Realization, again, less than 15 milliseconds. Onload, depending on the work on the page, we could be doing a lot of work here. We could be spending upwards of 300 milliseconds, which is quite a bit. And then we package all the data. I didn't really talk about the beacon much, but we have to send the data somewhere, right? This could be anywhere from 2 to 20 kilobytes or more, depending on the complexity of the page itself that we're looking at. So an important output of this, too, was we filed bugs. Everything that we found, every little trace that we looked at, idea big or small, we filed bugs within our GitHub repo. You can actually check them all out there. It's right on GitHub. And since then, we've been trying to make steady progress on fixing these. So I'll go over a sample of some of those in a little bit. But one thing I did want to chat about really briefly is just the tooling, some of the tools that we use to do some of this evaluation. So we really heavily relied on browser developer tools. All the browsers today have really good developer tools. We use profilers and all the different profilers because all browsers behave slightly differently. I know not many of you, maybe some of you, are not super comfortable with things like profilers, but they can really give you the insight that you would need to really look at a third-party script and to evaluate it. So profilers can show opportunities. My advice is if you're new to profilers, if you want to look at it, but you just don't know where to start, take your time, look at the big picture, try to load like a third-party script on a blank page and see what it does, and look for the extremes. Look for the longest amount of time of something being run or the largest call stack. Those can really point to different opportunities or places that are not performing the best. There's a lot of other free open source tools for evaluating different libraries. Lighthouse from Chrome obviously is a fantastic resource request map that I talked about earlier. Web page test, the list goes on and on. I also have a tool that I've made called third-party IEO that helps you audit third-party scripts, and I'll show you more about that later. So as I said, we performed this audit, but one of the main takeaways out of it was we wanted to improve, right? So along the way, we filed a lot of different ideas for improvements. I'm going to go over some of those. I don't think they're going to be appropriate for everybody. You may not care for some of these. I think a lot of them are somewhat interesting, and maybe it'll trigger a little bit of thoughts if you're working on a library or a script. So for some ideas of ways that you can improve yourselves, but it's also just kind of good to understand some of the ways that you can improve. So one of the things I talked about initially was our loader snippet. Again, the snippet that we give our customers helps make sure that boomerang is loaded in a non-blocking manner, but it's expensive. It takes a lot of time to run. We actually looked at some of the features of modern browsers, and we found something that allows us to load boomerang in a non-blocking manner that's much cheaper. So if the browser supports the preload feature, we actually can rely in preload to load boomerang in a way that's a lot better. In this case, instead of 40 milliseconds, it only takes a millisecond. It's basically a no-op. You can barely see it in the profiler anymore. So a lot of research was done into looking into this methodology. We tested it on a ton of websites. We're still kind of deploying it out to some of our customers and stuff like that. We documented all this in our boomerang documentation, but it was a big win for us to be able to point to our new methodology that is still allowing you to load the script in a non-blocking way, but be way cheaper. The other big area that I pointed out was we were doing a lot of work during the Onload Event. This is when we're capturing all the performance data on the page. We're compressing it, putting it on the beacon, and sending it back out to our back end. Unfortunately, this is also one of the most expensive things that we did. So one of the things that we found is we actually captured all of the resources that were fetched on the page. We look at all of the other timings. We save all the URLs. We compress them and then put them on the beacon. This allows our customers and open source users to look at the full waterfall for a page load for every single page that is measured. We compressed it down because it's otherwise really big, a lot of data, but this was taking a long time to do. So what we looked at, we took a look at the algorithm. We tried to figure out ways that we could improve it, and we actually decided that, or found a way that we could decrease the efficiency of it slightly. So it gets about maybe 2% larger payloads than before. So a little bit less efficient in compressing, but it was four times faster to run. So just this little tweak of the algorithm, we thought that trade-off, obviously, of 4x speed up is worth it for the minimal cost in the size of the bytes. So on a lot of the sites, the sites that were taken 300 plus milliseconds are now under 100 milliseconds, 75 milliseconds, et cetera. So that was a big win for us. Another big area of thing, or a big thing that we wanted to focus on was just reducing the size of boomerang in general. We actually had quite a few improvements in this area. For one example, we actually had an open source community member notice that we include a lot of debug messages even in our production builds. These are things that we use in our debug builds to understand what's going on better, but the messages, even though they would never be used, were put into our production builds. So we stripped all those out, saved 6% of the bytes. We did other things like change our minifier from Aglify 2 to 3, saved a couple bytes. We weren't using Brotly before. We were just gzipping our content. Luckily, the Akamai CDN knows how to do Brotly, so we enabled that. So that was a really easy, cheap win for us to do, saved 11% of the bytes. And then we did things like refactoring our plugins. So we have things like our SPA and our MDI, MD5 plugin. We were able to shrink those down tremendously and save a few more bytes there. And I'll show you some of the details about that in a bit. One other thing I didn't talk about too much, or I didn't talk about it all yet, is our cookie. So we set a first-party cookie on all of the pages that were on to help measure sessions. So this is how many pages the person visited on your website, for example. Our customers want the cookie to be as small as possible. And in some cases, we're over a kilobyte big. We found some ways to reduce the size of the cookie, for example, instead of storing the full URL of the page, we hash it and just store the hash, because we do comparisons of URLs in various places. Anyway, at the end of the day, we ended up making it 41% smaller, which a lot of our customers were happy about. And then another thing that we found while profiling it was we were reading and writing the cookie a lot. We were doing 21 reads of the cookie on a traditional, on a regular page load. We reduced that to two, and we're doing eight writes, and we reduced that down to four. So we just, you know, made it a lot more efficient. And then the final big area improvement that we did was just simplifying our plugins. One of the things here was we actually were using MD5 to do hashes of things like URLs. One of our developers found a replacement for MD5 that is about 5% of the size of MD5 and three times faster. And for our needs, it provided just as good hashes as MD5 was. So that gave us a really big byte savings and speed up. And then we did another kind of refactoring of our single page application plugin that just made it smaller as well. So with all that, we actually have some improvements from the numbers that showed up earlier. So the loader snippet in modern browsers is generally only a millisecond or less. I talked about a lot of different ways that we reduced the size of boomerang from all those changes. If you can see the number here, it's actually the same size right now. We also worked on new features in the meantime, and those new features ended up eating a lot of the byte savings that we did over the same period. So it's definitely something you've got to keep an eye out for is the needs of performance versus the needs of new features, et cetera, and often take away all the improvements that you're making. So it's still something that we're focusing on, and we have other ideas for reducing the size of it. But I guess another way to look at it is it would be a lot bigger now with all these new features had we not spent the time to also do other improvements. And then the other big section was the onload event. We made it a lot faster. There's still a lot of opportunities here. If you take a look at this list, these are things that we still want to make better. We want to make faster if possible. And we're hoping to focus in 2020 and beyond on further reducing the size of boomerang, further reducing the overhead, making it more efficient, making it much less likely to have an observer effect for any website that we're on. Again, we're tracking all these improvements in GitHub. And one thing that we kind of focused on for, and I thought was a really good quote from Rico Mariani from Microsoft is, in a mature product with a healthy process, you're much more likely to see a 50% gain come in the form of many 5% gains compounding. In other words, you're not always going to find the really big low hanging fruit that's going to make it 50% faster, right? It's going to take constant, constant iteration, finding little things, doing little tweaks here and there, and making improvements over time and continuing to make those improvements over time. So another important thing to us is now that we've made all these improvements, how can we make sure that we don't accidentally regress later, right? It's very easy to undo your hard work with a few couple mistakes. One of the things that we worked on was what we call our boomerang performance lab, which is just a suite of tests that we run in our CI environment. It's a bunch of scenarios, simple scenarios and metrics that we capture with a headless browser. We do things like looking at the CPU time of the profiles for all the work that we're doing. We have various counts and durations that we're measuring just using user timing marks and measures. And we just look at simple things like the code size that we have, and we can plot this over time. Another way of protecting the line is looking at real time telemetry. So you could, if you are already sending data from your library, you could look at things like runtime stats for different events that you're doing, whether you're triggering any long tasks. For us, for example, we actually capture all of the errors that we throw within our own code. We have a big try catcher on everything, essentially. We package that onto the beacon and send them into our real time dashboards. So we can see in real time, as new versions of boomerang come out, as new customers come online, as new browsers come out and break things, what's happening with the health of our library in real time. So these are all the different JavaScript error messages that we're seeing recently. So if you have all this data, it can really provide an answer into the, provide a look into the health of your system. And finally, there is a really cool new API out there called the JavaScript self-profiling API. You could use this for your website. It's an origin trial right now. But it lets you actually get sampled profile traces in the wild from a subset of your visitors. And so if you have a way of looking at these sample profiles in aggregate, you can kind of get a sense of how expensive your code is running in real time, not just on your developer machine. So that's what we did. What can you do? Let's say again, your boss comes to you and wants you to add a new script to your page. What should you do? Well, you can perform a lightweight performance audit. You don't have to do all the stuff that we did. You don't have to go into that depth of it. But it can be useful just to see exactly what that script is doing when you're putting it on your page. Put it in an empty page and take a profile trace, see what it does, see what it loads, make sure that its benefits are outweighing its costs. I think you should also try to ask the vendor, have you performed a performance audit? How does your library perform? I think it's good for us as a community to all share this information to be transparent. I think it'll help everybody here. I'm a little opinionated on that. But I think it'd be great if other people did this as well. And then, you know, every library that you include on your page, try to have an owner, the internal champion that knows about that script, why it's there, what it's doing, and when can you remove it. If you have somebody that really knows this or if you've documented this somewhere, it can really help with the overall life of, like, a web application for all the different apps that are being included. Besides just all the performance stuff that you could see in a profiler, there's a lot of other things that third-party scripts should be doing. There's pretty much like a checklist of best practices, right? So try to load the script from a CDN or your own CDN. Make sure it's compressed, make sure it's minified, make sure it has the right caching headers, make sure it's minimizing the amount of work that it's doing from CPU and from the network, make sure it's minimally touching the DOM, et cetera, and then make sure it's not doing a bunch of things, too. So make sure it's not triggering JavaScript errors. Make sure it doesn't have the debugger keyword in it. Make sure it's not throwing alerts. These are all things that scripts, ideally, should not be doing if you're including them on your page. One of the tools that I made to help you know about this is called third-party I.O. So if you take any JavaScript URL and paste it into the text box, it runs that JavaScript in a headless browser. It goes through that checklist that I just told you about and more and kind of gives you a list of best practices. Is this script loading too much? How does this compare to all the other scripts that we've looked at, for example? So it's a free tool. Please use it. Let me know if you like it. A bunch of links. I'm sure we're going to share out these slides, but it's just pretty much everything I've been talking about from the audit that we published. We just recently published an update with a lot of the stuff I talked about today in the Perf Planet calendar and third-party I.O. as well. Thank you. That's all I have for the slides. Do we have a few minutes for, okay. And we do have a few minutes for questions, three or four minutes if anybody does have one. So, yeah. So the question was we said we cut down the number of reads and writes to the cookie. Could you, is there a performance implication to doing that? We actually saw these cookie reads and writes on the profiler quite a bit. My guess is because often cookies actually get pushed to disk or to more solid state storage. You may see them pop up a little bit more just because of that. So after we did this optimization, we often don't see cookies read writes showing up as much. What we were doing was excessive, you know, 21 reads of the cookie doesn't seem necessary. A lot of that was probably cache in memory, but so we just figured we could do it better. Question? Yeah. You're doing some computation with the beacon data. You do this on the main thread? Yes. The question was we're doing all this work on the main thread to compress the data into the beacon and stuff. We have thought about that a little bit. It would be quite a complex thing for us to get a web worker working on all of our customer sites as well. So we played with the idea, but ultimately we decided just to do everything that we're doing on just in the library itself as opposed to a web worker. Yeah. Yeah. Yeah. Can you say the first part of that again? Yeah. I mean, we're using Canvas. I would like to know how you would help us to profile what Canvas do behind the scenes. So the question is they're using Canvas a lot and what can you do to profile it behind the scenes? I don't have a ton of experience in Canvas. I would assume that the profiler would be able to tell you a lot about what it's doing. I mean, there's still all these interactions with the APIs that you're taking, et cetera. Profilers like Chrome also show you the frames when they're rendered, et cetera, so you can see all the work that it took getting up to that frame and stuff like that. But I would just say profiler itself, if you're using a library to draw the Canvas like using D3 or something like that, you can certainly put it through its paces as well in the profiler. You can make sure it's being loaded in all the right ways, but those libraries are generally pretty optimized if they're doing drawing and stuff. Yeah. We still have another minute or two. One more question over there. And I will take more questions afterward if anybody else wants to. Please. Question is how is your product compared to century? Compared to... Century? What? Prox, sorry. Century. Century. Oh, JavaScript, you're reporting stuff? Century I.O. Yeah. Yeah. So the question was how does Boomerang itself compare to something like Century? I haven't used it very much. Boomerang does capture JavaScript errors on the page, and we do report those in our real-time dashboards providing a similar-ish service, but I'm not sure all the comparison details between the two. But yeah, it's certainly worth checking out if you're interested. We hope.
 Hello everybody. First of all, I'm honored here to kick off this web performance dev room  here at FASDEM. Definitely want to thank the Wikimedia performance team for organizing  this and for you all to be here today. Thanks for coming to listen to me and the others.  My name is Nick Jansma. I work at Akamai. I work on our impulse product. And specifically,  I work on an open source library called Boomerang. And today I'm going to talk about the performance  audit that we gave on Boomerang and what the findings that we found, the things that we  improved and kind of how we've made it better. So why are we here today? So Boomerang is  an open source real user monitoring library. It monitors performance. When you visit a  website, if Boomerang is on it, it's going to capture all the performance data that we  can, how fast it took, characteristics of the page itself. We at Akamai are the primary  contributors to Boomerang. Here's the GitHub repository if you care about it, but it is  an open source product. You are welcome to use Boomerang if you have your own back end  that you want to send the data to. You can send it to there. If you want to use our product  impulse, we will be the real time dashboards for you to show all this data. So we are a  third party, if you will. We provide a script that other people can include on their websites  to capture this data. In building a third party, especially real user monitoring script,  we think it's very important to try to not affect the page too much, to minimize our  costs. It'd be quite a shame if we were a performance monitoring product that also had  an effect on the performance of the pages that we're on. In fact, our customers, people  that pay for impulse, have been increasingly sophisticated in making sure and asking us  whether we have a cost when they're including us on their website. So two years ago, what  we did was we decided to take a step back and look at from a holistic perspective, what  is the cost of including Boomerang on your page? What is the cost of including the script?  We wanted to not only understand it better for ourselves, we wanted to look to see if  there were places that we could improve, and we wanted to be able to share this with our  customers. We think it's very important to be transparent. I think it builds a lot of  trust with our customers to show the good and the bad with what you have, and to help  the customer understand the benefit that you're bringing along with the overhead that it takes.  So for the past two years, we've been working on performance as one of our main features  for the product. We're really trying to improve Boomerang consistently over time. And this  is the kind of stuff that makes me passionate, makes our team passionate. We have a team  of five developers, a couple of them are here today. We love performance, right? And  so this is what I'm going to talk about today. So why should you care? Why are you here?  Well, for one, sorry, you're just stuck in the room with me, so you got to stick with  it. We locked the doors. You can't escape. Just kidding. Maybe you're like me. Maybe  you develop a third-party library. Does anybody else actually develop a library that other  people use at all? For any reasons? A couple of hands in them. A lot of people, web developers,  et cetera, use third-party libraries. They use other scripts to help build their websites.  So maybe you've had this happen if you've been working at a website or a big company.  Your boss comes by and says, hey, I really want you to include this one new library that  we need for our website. It's just this one simple line of code, just a script tag. Please  include it. Nothing can go wrong, right? This is very simple. Unfortunately, that one little  line means a lot. Including that script on your page means it could stop your page from  loading entirely. It could slow down your website. It could create incompatibilities  with other libraries on the page, too. It could change from underneath you, especially  if you're loading it from a CDN or from another place, that those JavaScript bytes could change  without you even asking it to. And at the end of the day, it really has complete control  over your web page. A JavaScript is running on your web page. It can do anything, intentionally  or not. So I wanted to put this into some sort of context for us. As I said, we provide  boomerang for our customers at Akamai. We're on over 14,000 of our customers' websites.  We track and measure over a billion page loads a day, which is quite a lot. According to  the open source statistics that I could find, we're on somewhere between 75,000 and almost  a half a million websites, depending on what data sources you're looking at. So a lot of  people are including the open source version of boomerang on their pages as well. And if  you take all the different combinations of all the different pages that boomerang's on,  all the different browsers that can load it, all the different people that can load it,  and all the different geolocations that can load it, all the different other scripts that  could be on the page, too, it's kind of scary. There's a really, really big matrix here of  things that can go wrong. You know, you could have an incompatibility with another script.  You could have edge cases for your performance, et cetera. So what I hope to do today is to  share some of the things that we've learned when we've evaluated ourselves so that you  might be able to do this as well for any library that you're taking a look at.  So how would you evaluate the cost of a third-party script? How do people that want to include  boomerang on their page, what questions should they be asking about what is its cost? And  how can we, as a provider of a third-party script, convince our customers that the benefits  outweigh those costs? So every script has a lot of different aspects to it, right? One  of those aspects would be how big is it? What's the byte size? So taking an example here,  modernizer.js, it's 97, or sorry, 92 kilobytes on a minified, 32 kilobytes GZipped. Is that  good? Bad? I don't know. Honestly, it's just a number at the end of the day. You probably  want to minimize this as much as possible. And size can be an important factor of a JavaScript  library, but it's definitely not the only thing. It doesn't tell the whole picture.  Resource weight is the concept of the total byte size of your page. So this includes all  your JavaScript, all your images, everything else that's required to build your website,  right? And it's very important. Every single byte matters when you're trying to improve  the performance of your page. And lowering the byte size of different libraries, you  know, choosing thinner libraries and stuff like that can really help for your performance  budget. And while it's probably one of the easiest ways to judge a third party, just  based on that size, it's definitely only just one aspect of it. So what we decided to do  with our audit of our own code was kind of break up the life cycle of our script into  five main phases. So first, you have your loader snippet or how the script gets on the page.  Maybe you include this in your JavaScript bundle. Maybe you load it via tag manager.  But the browser needs to know to load your script in the first place. The second thing,  the browser goes and fetches the script itself. And obviously, the less bytes you have, the  better here. The browser then needs to parse it and compile it, get it ready to be run.  Finally, it runs it for the first time. So there's probably a lot of initialization that  happens. Maybe registering global variables or event handlers. And then run time. So the  reason that it's there in the first place, why is this library on your website? And all  of these are important from the entire life cycle of the script. So what we did for boomerang  is we decided to jump into the developer tools, profilers from Chrome and other, really kind  of show you at a really high level what's going on and then give you details for all  the specifics. So this is what boomerang looks like if you're loading it on an empty page.  We start with the loader snippet, which is the way that we get boomerang onto the page  in the first place. Our customers, Akamai load it from our CDN. Open source customers  could bundle it into your application bundle or however you want to load it. Then we download  it. Boomerang gets downloaded. The browser parses the JavaScript and executes it. We  initialize. We set up some global variables. We register event handlers, et cetera. In our  case for impulse, we actually go fetch some more configuration data via a JSON request.  And then we initialize that. And then finally, the main reason that we're there is we're  collecting all this performance information. We're gathering data on how long it took.  And that's what we call our onload event handler. We package all that data up and we  beacon it out. We send it back to the mothership for processing. So these are kind of the main  five stages of our script loading. So keeping those phases in mind, let's take a look at  how you might audit a new third-party script or how we audited boomerang when looking at  these different phases. So the first question is, how does a new script get on a website?  Some libraries suggest a very simple script tag, right? A script async tag or something  like that. This tag itself has no intrinsic cost. The browser just knows to go fetch that  data. Other libraries like Google Analytics, for example, have a small snippet. This usually  is meant to load the script asynchronously. You can see in very small fonts right there  that's kind of Google Analytics loader snippet. And this has a very minor cost, a couple milliseconds  usually to execute it and then to trigger that download of the JavaScript. We at boomerang  like to do things a little bit differently. We actually have a more complex loader snippet  that we give to our customers. We don't use just a script async tag because while you  can load JavaScript asynchronously, the browser will still block on that content before it  fires the onload event. In other words, if you have even asynchronous script tag that's  loading your analytics, if that analytics package takes 10 seconds to load, the browser  is still going to be in its loading state. The loading indicator is still going to be  there. The onload event won't fire, et cetera. What we want to do as a performance monitoring  script is make sure that we're not blocking onload. We're not affecting the performance  of the page that we're on. So we have this 40 lines of JavaScript essentially that tells  the browser to load boomerang in an asynchronous and non-blocking manner. We use an iframe  to do this or we used an iframe to do this. It's a little more complex. It actually costs  a little bit more than the previous ways I mentioned. So in our case, it could be up  to 40 milliseconds. And a lot of this is because the iframe itself creating an iframe in the  browser actually has a cost to it. So it's a little more expensive. And we actually wanted  to focus on this when we were doing our audit as one of the main things that we could try  to see if we can improve on. Can we get down the cost while still maintaining the non-blocking  nature of what we're giving to our customers? And if you want any more details on exactly  how we do everything that we do, it is in the boomerang documentation on GitHub. You  can read all about it. We've explained it quite a bit more there.  Okay. So now the browser knows that you want to load your JavaScript, right? It's going  to go fetch it from the network. This is when the download begins. Every byte that is downloaded  affects the overall page weight. You know, every byte matters here. Depending on how  the script is loaded, it could affect other things on the page too. If it's part of your  main application bundle, for example, it will block all the stuff afterward. I know that  a lot of people are using more exotic ways of loading JavaScript these days, like through  modules and stuff. But it's important to keep in mind that everything that you are loading  has a cost to it. And it's also really important to know that a lot of libraries that you will  load, analytics libraries or other widgets, social widgets, et cetera, they'll often load  additional data after they load the JavaScript. So they may load other CSS or images, get  JSON from various places, for example, right? In fact, some of you may know this pretty  cool tool from Simon Hearns called RequestMap. And it kind of lets you see all the different  things that get fetched and what triggers other things to get fetched, et cetera. It's  a good way of knowing kind of like the whole cost of a library when you load it. So it's  on RequestMap.byperf.tools. Check it out. You could also use just things like web page  test or looking at your network tab to see all the other things that are downloaded.  So if we take a look at a couple of sample popular JavaScript libraries, we kind of just  picked a sample of ones that I knew of, put them in order of size. We can actually see  that boomerangs kind of near the high end of cost here. So simple things like underscore  could just be a couple kilobytes. Big things like the D3 charting libraries, 70 kilobytes.  Boomerang ends up at around 47 kilobytes, so a little less than 50 kilobytes. It's big,  honestly. This is one of the things that we thought we could improve on. We are doing  a lot of things in the library, but maybe there's ways for us to have it not be such  a big library.  Also, I did want to point out that the build that I was talking about is the impulse-specific  version of Boomerang, which is a pretty big build for all of our customers. If you're  using the open-source version of Boomerang, you can choose to build it smaller. We have  a plugin architecture, and you can choose which plugins to include or not. So if you  don't need single-page app support and JavaScript error tracking that we include for impulse,  you could trim it down to about half the size, even.  Okay, so now the browser is downloaded to your JavaScript bytes. What does it have to  do? It actually has to parse and compile it. It can do this before it runs any of it. The  main idea here is the more bytes it is, the more complex it is to parse, to compile, etc.  So again, you want to minimize byte costs, because this does have a cost. For us, for  Boomerang, on a modern browser, modern device, it's 10 milliseconds or so, so not a lot.  But some of the bigger libraries, like Angular, can be quite a bit more 20, 25 milliseconds  to just get it ready before it even runs. So again, building a smaller library can really  help with a lot of this.  Okay, second and last phase is initialization. So this is when the browser has everything  ready and it hits run on the script. For every script, this is different, right? This is  what they're doing on the page. They might be registering global variables or hooking  in two different events, etc. They might be fetching more resources. Maybe it's a social  widget. Hopefully it's only doing what you're asking it to do. In our case, for Boomerang,  again, we register a couple variables and listen for the onload event and some other  events depending on the features that are enabled. We don't do a lot of work here, only  10 milliseconds or so, not too bad, I would say. We did find a couple ideas for improvements  here. We might defer some of the work that we're doing. We might break up some of the  work that we're doing, but not too bad overall.  And then finally, the runtime of the script itself. So again, depends exactly on what  you're using the script for. If it's a utility script, maybe you're calling into it quite  a bit. If it's a social widget, maybe it's loading all the likes for something. Maybe  you have a Bitcoin miner, it's mining Bitcoin for whatever reason. For us, Boomerang, this  is where Boomerang does the majority of its work. So after the page load has happened,  we look at all the performance data for that page. We package it up, we compress it, we  put it on a beacon, and we send it out. So we're capturing things like all the resources  that were fetched on the page, how long it took to navigate, what the DNS time was, and  a lot of other things like that. If there were any JavaScript errors, we package those  up and send those as well. In a lot of the cases on most sites, this wasn't taking too  long, say less than 50 milliseconds. But we did find in some examples, especially on  lower end devices, on more complex web pages, that we were taking 300 milliseconds or more.  And that's starting to get into the territory of people would notice it, a visitor would  notice it, that we would be affecting actually the performance of the page. So this is one  of the areas that we flagged that we really wanted to get into and figure out if we can  improve. And I'll talk about some of the things that we found later.  So one thing I did want to point out really quick too is all of the bold phases up there  are part of the critical path of the browser using your JavaScript. All of the things up  there are generally done serially on the main thread, depending on the browser. But any  of these things that you're doing here are affecting the rest of the site being built.  And if the user is trying to interact with the page, you're potentially affecting their  interaction. So you're delaying the user input, et cetera. So what you really want to do when  you have a third party library, a script like this, minimize the work that you're doing,  bake it up. If you know you need to do a really big calculation, try to bake it up into pieces.  This avoids things called long tasks, which are tasks that are running on the main browser  thread that would potentially block user input if they're trying to click or scroll.  It would make for a non-responsive user experience.  So after everything that I said, these are kind of the numbers that we came up with at  the end. We did a lot of investigations on a lot of different sites. We took a lot of  profiles. We wrote about it. We published it in a blog post. And these are kind of the  TLDR costs of boomerang. So the loader snippet, generally only a few milliseconds on some  browsers. It can be up to 40 milliseconds. Downloading, we're about 50 kilobytes or less.  Parsing, again, it's related to the size of boomerang. And this is kind of like the low  end to high end devices when we were looking at it. Anywhere from 6 to 47 milliseconds.  Realization, again, less than 15 milliseconds. Onload, depending on the work on the page,  we could be doing a lot of work here. We could be spending upwards of 300 milliseconds, which  is quite a bit. And then we package all the data. I didn't really talk about the beacon  much, but we have to send the data somewhere, right? This could be anywhere from 2 to 20  kilobytes or more, depending on the complexity of the page itself that we're looking at.  So an important output of this, too, was we filed bugs. Everything that we found, every  little trace that we looked at, idea big or small, we filed bugs within our GitHub repo.  You can actually check them all out there. It's right on GitHub. And since then, we've  been trying to make steady progress on fixing these. So I'll go over a sample of some of  those in a little bit. But one thing I did want to chat about really briefly is just  the tooling, some of the tools that we use to do some of this evaluation. So we really  heavily relied on browser developer tools. All the browsers today have really good developer  tools. We use profilers and all the different profilers because all browsers behave slightly  differently. I know not many of you, maybe some of you, are not super comfortable with  things like profilers, but they can really give you the insight that you would need to  really look at a third-party script and to evaluate it. So profilers can show opportunities.  My advice is if you're new to profilers, if you want to look at it, but you just don't  know where to start, take your time, look at the big picture, try to load like a third-party  script on a blank page and see what it does, and look for the extremes. Look for the longest  amount of time of something being run or the largest call stack. Those can really point  to different opportunities or places that are not performing the best.  There's a lot of other free open source tools for evaluating different libraries. Lighthouse  from Chrome obviously is a fantastic resource request map that I talked about earlier. Web  page test, the list goes on and on. I also have a tool that I've made called third-party  IEO that helps you audit third-party scripts, and I'll show you more about that later.  So as I said, we performed this audit, but one of the main takeaways out of it was we  wanted to improve, right? So along the way, we filed a lot of different ideas for improvements.  I'm going to go over some of those. I don't think they're going to be appropriate for  everybody. You may not care for some of these. I think a lot of them are somewhat interesting,  and maybe it'll trigger a little bit of thoughts if you're working on a library or a script.  So for some ideas of ways that you can improve yourselves, but it's also just kind of good  to understand some of the ways that you can improve.  So one of the things I talked about initially was our loader snippet. Again, the snippet  that we give our customers helps make sure that boomerang is loaded in a non-blocking  manner, but it's expensive. It takes a lot of time to run. We actually looked at some  of the features of modern browsers, and we found something that allows us to load boomerang  in a non-blocking manner that's much cheaper. So if the browser supports the preload feature,  we actually can rely in preload to load boomerang in a way that's a lot better. In this case,  instead of 40 milliseconds, it only takes a millisecond. It's basically a no-op. You  can barely see it in the profiler anymore. So a lot of research was done into looking  into this methodology. We tested it on a ton of websites. We're still kind of deploying  it out to some of our customers and stuff like that. We documented all this in our boomerang  documentation, but it was a big win for us to be able to point to our new methodology  that is still allowing you to load the script in a non-blocking way, but be way cheaper.  The other big area that I pointed out was we were doing a lot of work during the Onload  Event. This is when we're capturing all the performance data on the page. We're compressing  it, putting it on the beacon, and sending it back out to our back end. Unfortunately,  this is also one of the most expensive things that we did. So one of the things that we  found is we actually captured all of the resources that were fetched on the page. We look at all  of the other timings. We save all the URLs. We compress them and then put them on the  beacon. This allows our customers and open source users to look at the full waterfall  for a page load for every single page that is measured. We compressed it down because  it's otherwise really big, a lot of data, but this was taking a long time to do. So  what we looked at, we took a look at the algorithm. We tried to figure out ways that we could  improve it, and we actually decided that, or found a way that we could decrease the efficiency  of it slightly. So it gets about maybe 2% larger payloads than before. So a little bit  less efficient in compressing, but it was four times faster to run. So just this little  tweak of the algorithm, we thought that trade-off, obviously, of 4x speed up is worth it for the  minimal cost in the size of the bytes. So on a lot of the sites, the sites that were taken  300 plus milliseconds are now under 100 milliseconds, 75 milliseconds, et cetera. So that was a  big win for us. Another big area of thing, or a big thing that we wanted to focus on  was just reducing the size of boomerang in general. We actually had quite a few improvements  in this area. For one example, we actually had an open source community member notice  that we include a lot of debug messages even in our production builds. These are things  that we use in our debug builds to understand what's going on better, but the messages,  even though they would never be used, were put into our production builds. So we stripped  all those out, saved 6% of the bytes. We did other things like change our minifier from  Aglify 2 to 3, saved a couple bytes. We weren't using Brotly before. We were just gzipping  our content. Luckily, the Akamai CDN knows how to do Brotly, so we enabled that. So that  was a really easy, cheap win for us to do, saved 11% of the bytes. And then we did things  like refactoring our plugins. So we have things like our SPA and our MDI, MD5 plugin. We were  able to shrink those down tremendously and save a few more bytes there. And I'll show  you some of the details about that in a bit. One other thing I didn't talk about too much,  or I didn't talk about it all yet, is our cookie. So we set a first-party cookie on  all of the pages that were on to help measure sessions. So this is how many pages the person  visited on your website, for example. Our customers want the cookie to be as small as  possible. And in some cases, we're over a kilobyte big. We found some ways to reduce  the size of the cookie, for example, instead of storing the full URL of the page, we hash  it and just store the hash, because we do comparisons of URLs in various places. Anyway,  at the end of the day, we ended up making it 41% smaller, which a lot of our customers  were happy about. And then another thing that we found while profiling it was we were reading  and writing the cookie a lot. We were doing 21 reads of the cookie on a traditional, on  a regular page load. We reduced that to two, and we're doing eight writes, and we reduced  that down to four. So we just, you know, made it a lot more efficient.  And then the final big area improvement that we did was just simplifying our plugins. One  of the things here was we actually were using MD5 to do hashes of things like URLs. One  of our developers found a replacement for MD5 that is about 5% of the size of MD5 and  three times faster. And for our needs, it provided just as good hashes as MD5 was. So  that gave us a really big byte savings and speed up. And then we did another kind of  refactoring of our single page application plugin that just made it smaller as well.  So with all that, we actually have some improvements from the numbers that showed up earlier. So  the loader snippet in modern browsers is generally only a millisecond or less. I talked about  a lot of different ways that we reduced the size of boomerang from all those changes.  If you can see the number here, it's actually the same size right now. We also worked on  new features in the meantime, and those new features ended up eating a lot of the byte  savings that we did over the same period. So it's definitely something you've got to  keep an eye out for is the needs of performance versus the needs of new features, et cetera,  and often take away all the improvements that you're making. So it's still something that  we're focusing on, and we have other ideas for reducing the size of it. But I guess another  way to look at it is it would be a lot bigger now with all these new features had we not  spent the time to also do other improvements. And then the other big section was the onload  event. We made it a lot faster. There's still a lot of opportunities here. If you take a  look at this list, these are things that we still want to make better. We want to make  faster if possible. And we're hoping to focus in 2020 and beyond on further reducing the  size of boomerang, further reducing the overhead, making it more efficient, making it much  less likely to have an observer effect for any website that we're on. Again, we're tracking  all these improvements in GitHub. And one thing that we kind of focused on for, and  I thought was a really good quote from Rico Mariani from Microsoft is, in a mature product  with a healthy process, you're much more likely to see a 50% gain come in the form of many  5% gains compounding. In other words, you're not always going to find the really big low  hanging fruit that's going to make it 50% faster, right? It's going to take constant,  constant iteration, finding little things, doing little tweaks here and there, and making  improvements over time and continuing to make those improvements over time. So another  important thing to us is now that we've made all these improvements, how can we make sure  that we don't accidentally regress later, right? It's very easy to undo your hard work  with a few couple mistakes. One of the things that we worked on was what we call our boomerang  performance lab, which is just a suite of tests that we run in our CI environment. It's  a bunch of scenarios, simple scenarios and metrics that we capture with a headless browser.  We do things like looking at the CPU time of the profiles for all the work that we're  doing. We have various counts and durations that we're measuring just using user timing  marks and measures. And we just look at simple things like the code size that we have, and  we can plot this over time. Another way of protecting the line is looking at real time  telemetry. So you could, if you are already sending data from your library, you could  look at things like runtime stats for different events that you're doing, whether you're  triggering any long tasks. For us, for example, we actually capture all of the errors that  we throw within our own code. We have a big try catcher on everything, essentially. We  package that onto the beacon and send them into our real time dashboards. So we can see  in real time, as new versions of boomerang come out, as new customers come online, as  new browsers come out and break things, what's happening with the health of our library in  real time. So these are all the different JavaScript error messages that we're seeing  recently. So if you have all this data, it can really provide an answer into the, provide  a look into the health of your system. And finally, there is a really cool new API out  there called the JavaScript self-profiling API. You could use this for your website. It's  an origin trial right now. But it lets you actually get sampled profile traces in the  wild from a subset of your visitors. And so if you have a way of looking at these sample  profiles in aggregate, you can kind of get a sense of how expensive your code is running  in real time, not just on your developer machine. So that's what we did. What can you do? Let's  say again, your boss comes to you and wants you to add a new script to your page. What  should you do? Well, you can perform a lightweight performance audit. You don't have to do all  the stuff that we did. You don't have to go into that depth of it. But it can be useful  just to see exactly what that script is doing when you're putting it on your page. Put it  in an empty page and take a profile trace, see what it does, see what it loads, make  sure that its benefits are outweighing its costs. I think you should also try to ask  the vendor, have you performed a performance audit? How does your library perform? I think  it's good for us as a community to all share this information to be transparent. I think  it'll help everybody here. I'm a little opinionated on that. But I think it'd be great if other  people did this as well. And then, you know, every library that you include on your page,  try to have an owner, the internal champion that knows about that script, why it's there,  what it's doing, and when can you remove it. If you have somebody that really knows this  or if you've documented this somewhere, it can really help with the overall life of,  like, a web application for all the different apps that are being included. Besides just  all the performance stuff that you could see in a profiler, there's a lot of other things  that third-party scripts should be doing. There's pretty much like a checklist of best  practices, right? So try to load the script from a CDN or your own CDN. Make sure it's  compressed, make sure it's minified, make sure it has the right caching headers, make  sure it's minimizing the amount of work that it's doing from CPU and from the network,  make sure it's minimally touching the DOM, et cetera, and then make sure it's not doing  a bunch of things, too. So make sure it's not triggering JavaScript errors. Make sure  it doesn't have the debugger keyword in it. Make sure it's not throwing alerts. These  are all things that scripts, ideally, should not be doing if you're including them on your  page. One of the tools that I made to help you know about this is called third-party  I.O. So if you take any JavaScript URL and paste it into the text box, it runs that JavaScript  in a headless browser. It goes through that checklist that I just told you about and more  and kind of gives you a list of best practices. Is this script loading too much? How does  this compare to all the other scripts that we've looked at, for example? So it's a free  tool. Please use it. Let me know if you like it. A bunch of links. I'm sure we're going  to share out these slides, but it's just pretty much everything I've been talking about from  the audit that we published. We just recently published an update with a lot of the stuff  I talked about today in the Perf Planet calendar and third-party I.O. as well. Thank you. That's  all I have for the slides. Do we have a few minutes for, okay. And we do have a few minutes  for questions, three or four minutes if anybody does have one. So, yeah. So the question was  we said we cut down the number of reads and writes to the cookie. Could you, is there  a performance implication to doing that? We actually saw these cookie reads and writes  on the profiler quite a bit. My guess is because often cookies actually get pushed to disk  or to more solid state storage. You may see them pop up a little bit more just because  of that. So after we did this optimization, we often don't see cookies read writes showing  up as much. What we were doing was excessive, you know, 21 reads of the cookie doesn't seem  necessary. A lot of that was probably cache in memory, but so we just figured we could  do it better. Question? Yeah. You're doing some computation with the beacon data. You  do this on the main thread? Yes. The question was we're doing all this work on the main  thread to compress the data into the beacon and stuff. We have thought about that a little  bit. It would be quite a complex thing for us to get a web worker working on all of our  customer sites as well. So we played with the idea, but ultimately we decided just to do  everything that we're doing on just in the library itself as opposed to a web worker.  Yeah. Yeah. Yeah. Can you say the first part of that again? Yeah. I mean, we're using  Canvas. I would like to know how you would help us to profile what Canvas do behind the  scenes. So the question is they're using Canvas a lot and what can you do to profile  it behind the scenes? I don't have a ton of experience in Canvas. I would assume that  the profiler would be able to tell you a lot about what it's doing. I mean, there's still  all these interactions with the APIs that you're taking, et cetera. Profilers like Chrome  also show you the frames when they're rendered, et cetera, so you can see all the work that  it took getting up to that frame and stuff like that. But I would just say profiler itself,  if you're using a library to draw the Canvas like using D3 or something like that, you  can certainly put it through its paces as well in the profiler. You can make sure it's  being loaded in all the right ways, but those libraries are generally pretty optimized if  they're doing drawing and stuff. Yeah. We still have another minute or two. One more  question over there. And I will take more questions afterward if anybody else wants  to. Please.  Question is how is your product compared to century?  Compared to...  Century?  What? Prox, sorry.  Century.  Century.  Oh, JavaScript, you're reporting stuff? Century I.O. Yeah. Yeah. So the question was how does  Boomerang itself compare to something like Century? I haven't used it very much. Boomerang  does capture JavaScript errors on the page, and we do report those in our real-time dashboards  providing a similar-ish service, but I'm not sure all the comparison details between the  two. But yeah, it's certainly worth checking out if you're interested.  We hope.
webperf_boomerang_optimisation_0_167.png webperf_boomerang_optimisation_168_407.png webperf_boomerang_optimisation_408_707.png webperf_boomerang_optimisation_708_959.png webperf_boomerang_optimisation_960_1703.png webperf_boomerang_optimisation_1704_2279.png webperf_boomerang_optimisation_2280_3119.png webperf_boomerang_optimisation_3120_3443.png webperf_boomerang_optimisation_3444_5171.png webperf_boomerang_optimisation_5172_5459.png webperf_boomerang_optimisation_5460_6719.png webperf_boomerang_optimisation_6720_7115.png webperf_boomerang_optimisation_7116_7511.png webperf_boomerang_optimisation_7512_8411.png webperf_boomerang_optimisation_8412_9371.png webperf_boomerang_optimisation_9372_9995.png webperf_boomerang_optimisation_9996_11039.png webperf_boomerang_optimisation_11040_12635.png webperf_boomerang_optimisation_12636_12983.png webperf_boomerang_optimisation_12984_15683.png webperf_boomerang_optimisation_15684_15863.png webperf_boomerang_optimisation_15864_16199.png webperf_boomerang_optimisation_16200_17603.png webperf_boomerang_optimisation_17604_17867.png webperf_boomerang_optimisation_17868_18851.png webperf_boomerang_optimisation_18852_19055.png webperf_boomerang_optimisation_19056_19487.png webperf_boomerang_optimisation_19488_23411.png webperf_boomerang_optimisation_23412_25019.png webperf_boomerang_optimisation_25020_25271.png webperf_boomerang_optimisation_25272_25499.png webperf_boomerang_optimisation_25500_27683.png webperf_boomerang_optimisation_27684_29315.png webperf_boomerang_optimisation_29316_30611.png webperf_boomerang_optimisation_30612_36827.png webperf_boomerang_optimisation_36828_39059.png webperf_boomerang_optimisation_39060_41267.png webperf_boomerang_optimisation_41268_42767.png webperf_boomerang_optimisation_42768_45527.png webperf_boomerang_optimisation_45528_48671.png webperf_boomerang_optimisation_48672_49631.png webperf_boomerang_optimisation_49632_51947.png webperf_boomerang_optimisation_51948_52072.png
Hello everybody. First of all, I'm honored here to kick off this web performance dev room here at FASDEM. Definitely want to thank the Wikimedia performance team for organizing this and for you all to be here today. Thanks for coming to listen to me and the others. My name is Nick Jansma. I work at Akamai. I work on our impulse product. And specifically, I work on an open source library called Boomerang. And today I'm going to talk about the performance audit that we gave on Boomerang and what the findings that we found, the things that we improved and kind of how we've made it better. So why are we here today? So Boomerang is an open source real user monitoring library. It monitors performance. When you visit a website, if Boomerang is on it, it's going to capture all the performance data that we can, how fast it took, characteristics of the page itself. We at Akamai are the primary contributors to Boomerang. Here's the GitHub repository if you care about it, but it is an open source product. You are welcome to use Boomerang if you have your own back end that you want to send the data to. You can send it to there. If you want to use our product impulse, we will be the real time dashboards for you to show all this data. So we are a third party, if you will. We provide a script that other people can include on their websites to capture this data. In building a third party, especially real user monitoring script, we think it's very important to try to not affect the page too much, to minimize our costs. It'd be quite a shame if we were a performance monitoring product that also had an effect on the performance of the pages that we're on. In fact, our customers, people that pay for impulse, have been increasingly sophisticated in making sure and asking us whether we have a cost when they're including us on their website. So two years ago, what we did was we decided to take a step back and look at from a holistic perspective, what is the cost of including Boomerang on your page? What is the cost of including the script? We wanted to not only understand it better for ourselves, we wanted to look to see if there were places that we could improve, and we wanted to be able to share this with our customers. We think it's very important to be transparent. I think it builds a lot of trust with our customers to show the good and the bad with what you have, and to help the customer understand the benefit that you're bringing along with the overhead that it takes. So for the past two years, we've been working on performance as one of our main features for the product. We're really trying to improve Boomerang consistently over time. And this is the kind of stuff that makes me passionate, makes our team passionate. We have a team of five developers, a couple of them are here today. We love performance, right? And so this is what I'm going to talk about today. So why should you care? Why are you here? Well, for one, sorry, you're just stuck in the room with me, so you got to stick with it. We locked the doors. You can't escape. Just kidding. Maybe you're like me. Maybe you develop a third-party library. Does anybody else actually develop a library that other people use at all? For any reasons? A couple of hands in them. A lot of people, web developers, et cetera, use third-party libraries. They use other scripts to help build their websites. So maybe you've had this happen if you've been working at a website or a big company. Your boss comes by and says, hey, I really want you to include this one new library that we need for our website. It's just this one simple line of code, just a script tag. Please include it. Nothing can go wrong, right? This is very simple. Unfortunately, that one little line means a lot. Including that script on your page means it could stop your page from loading entirely. It could slow down your website. It could create incompatibilities with other libraries on the page, too. It could change from underneath you, especially if you're loading it from a CDN or from another place, that those JavaScript bytes could change without you even asking it to. And at the end of the day, it really has complete control over your web page. A JavaScript is running on your web page. It can do anything, intentionally or not. So I wanted to put this into some sort of context for us. As I said, we provide boomerang for our customers at Akamai. We're on over 14,000 of our customers' websites. We track and measure over a billion page loads a day, which is quite a lot. According to the open source statistics that I could find, we're on somewhere between 75,000 and almost a half a million websites, depending on what data sources you're looking at. So a lot of people are including the open source version of boomerang on their pages as well. And if you take all the different combinations of all the different pages that boomerang's on, all the different browsers that can load it, all the different people that can load it, and all the different geolocations that can load it, all the different other scripts that could be on the page, too, it's kind of scary. There's a really, really big matrix here of things that can go wrong. You know, you could have an incompatibility with another script. You could have edge cases for your performance, et cetera. So what I hope to do today is to share some of the things that we've learned when we've evaluated ourselves so that you might be able to do this as well for any library that you're taking a look at. So how would you evaluate the cost of a third-party script? How do people that want to include boomerang on their page, what questions should they be asking about what is its cost? And how can we, as a provider of a third-party script, convince our customers that the benefits outweigh those costs? So every script has a lot of different aspects to it, right? One of those aspects would be how big is it? What's the byte size? So taking an example here, modernizer.js, it's 97, or sorry, 92 kilobytes on a minified, 32 kilobytes GZipped. Is that good? Bad? I don't know. Honestly, it's just a number at the end of the day. You probably want to minimize this as much as possible. And size can be an important factor of a JavaScript library, but it's definitely not the only thing. It doesn't tell the whole picture. Resource weight is the concept of the total byte size of your page. So this includes all your JavaScript, all your images, everything else that's required to build your website, right? And it's very important. Every single byte matters when you're trying to improve the performance of your page. And lowering the byte size of different libraries, you know, choosing thinner libraries and stuff like that can really help for your performance budget. And while it's probably one of the easiest ways to judge a third party, just based on that size, it's definitely only just one aspect of it. So what we decided to do with our audit of our own code was kind of break up the life cycle of our script into five main phases. So first, you have your loader snippet or how the script gets on the page. Maybe you include this in your JavaScript bundle. Maybe you load it via tag manager. But the browser needs to know to load your script in the first place. The second thing, the browser goes and fetches the script itself. And obviously, the less bytes you have, the better here. The browser then needs to parse it and compile it, get it ready to be run. Finally, it runs it for the first time. So there's probably a lot of initialization that happens. Maybe registering global variables or event handlers. And then run time. So the reason that it's there in the first place, why is this library on your website? And all of these are important from the entire life cycle of the script. So what we did for boomerang is we decided to jump into the developer tools, profilers from Chrome and other, really kind of show you at a really high level what's going on and then give you details for all the specifics. So this is what boomerang looks like if you're loading it on an empty page. We start with the loader snippet, which is the way that we get boomerang onto the page in the first place. Our customers, Akamai load it from our CDN. Open source customers could bundle it into your application bundle or however you want to load it. Then we download it. Boomerang gets downloaded. The browser parses the JavaScript and executes it. We initialize. We set up some global variables. We register event handlers, et cetera. In our case for impulse, we actually go fetch some more configuration data via a JSON request. And then we initialize that. And then finally, the main reason that we're there is we're collecting all this performance information. We're gathering data on how long it took. And that's what we call our onload event handler. We package all that data up and we beacon it out. We send it back to the mothership for processing. So these are kind of the main five stages of our script loading. So keeping those phases in mind, let's take a look at how you might audit a new third-party script or how we audited boomerang when looking at these different phases. So the first question is, how does a new script get on a website? Some libraries suggest a very simple script tag, right? A script async tag or something like that. This tag itself has no intrinsic cost. The browser just knows to go fetch that data. Other libraries like Google Analytics, for example, have a small snippet. This usually is meant to load the script asynchronously. You can see in very small fonts right there that's kind of Google Analytics loader snippet. And this has a very minor cost, a couple milliseconds usually to execute it and then to trigger that download of the JavaScript. We at boomerang like to do things a little bit differently. We actually have a more complex loader snippet that we give to our customers. We don't use just a script async tag because while you can load JavaScript asynchronously, the browser will still block on that content before it fires the onload event. In other words, if you have even asynchronous script tag that's loading your analytics, if that analytics package takes 10 seconds to load, the browser is still going to be in its loading state. The loading indicator is still going to be there. The onload event won't fire, et cetera. What we want to do as a performance monitoring script is make sure that we're not blocking onload. We're not affecting the performance of the page that we're on. So we have this 40 lines of JavaScript essentially that tells the browser to load boomerang in an asynchronous and non-blocking manner. We use an iframe to do this or we used an iframe to do this. It's a little more complex. It actually costs a little bit more than the previous ways I mentioned. So in our case, it could be up to 40 milliseconds. And a lot of this is because the iframe itself creating an iframe in the browser actually has a cost to it. So it's a little more expensive. And we actually wanted to focus on this when we were doing our audit as one of the main things that we could try to see if we can improve on. Can we get down the cost while still maintaining the non-blocking nature of what we're giving to our customers? And if you want any more details on exactly how we do everything that we do, it is in the boomerang documentation on GitHub. You can read all about it. We've explained it quite a bit more there. Okay. So now the browser knows that you want to load your JavaScript, right? It's going to go fetch it from the network. This is when the download begins. Every byte that is downloaded affects the overall page weight. You know, every byte matters here. Depending on how the script is loaded, it could affect other things on the page too. If it's part of your main application bundle, for example, it will block all the stuff afterward. I know that a lot of people are using more exotic ways of loading JavaScript these days, like through modules and stuff. But it's important to keep in mind that everything that you are loading has a cost to it. And it's also really important to know that a lot of libraries that you will load, analytics libraries or other widgets, social widgets, et cetera, they'll often load additional data after they load the JavaScript. So they may load other CSS or images, get JSON from various places, for example, right? In fact, some of you may know this pretty cool tool from Simon Hearns called RequestMap. And it kind of lets you see all the different things that get fetched and what triggers other things to get fetched, et cetera. It's a good way of knowing kind of like the whole cost of a library when you load it. So it's on RequestMap.byperf.tools. Check it out. You could also use just things like web page test or looking at your network tab to see all the other things that are downloaded. So if we take a look at a couple of sample popular JavaScript libraries, we kind of just picked a sample of ones that I knew of, put them in order of size. We can actually see that boomerangs kind of near the high end of cost here. So simple things like underscore could just be a couple kilobytes. Big things like the D3 charting libraries, 70 kilobytes. Boomerang ends up at around 47 kilobytes, so a little less than 50 kilobytes. It's big, honestly. This is one of the things that we thought we could improve on. We are doing a lot of things in the library, but maybe there's ways for us to have it not be such a big library. Also, I did want to point out that the build that I was talking about is the impulse-specific version of Boomerang, which is a pretty big build for all of our customers. If you're using the open-source version of Boomerang, you can choose to build it smaller. We have a plugin architecture, and you can choose which plugins to include or not. So if you don't need single-page app support and JavaScript error tracking that we include for impulse, you could trim it down to about half the size, even. Okay, so now the browser is downloaded to your JavaScript bytes. What does it have to do? It actually has to parse and compile it. It can do this before it runs any of it. The main idea here is the more bytes it is, the more complex it is to parse, to compile, etc. So again, you want to minimize byte costs, because this does have a cost. For us, for Boomerang, on a modern browser, modern device, it's 10 milliseconds or so, so not a lot. But some of the bigger libraries, like Angular, can be quite a bit more 20, 25 milliseconds to just get it ready before it even runs. So again, building a smaller library can really help with a lot of this. Okay, second and last phase is initialization. So this is when the browser has everything ready and it hits run on the script. For every script, this is different, right? This is what they're doing on the page. They might be registering global variables or hooking in two different events, etc. They might be fetching more resources. Maybe it's a social widget. Hopefully it's only doing what you're asking it to do. In our case, for Boomerang, again, we register a couple variables and listen for the onload event and some other events depending on the features that are enabled. We don't do a lot of work here, only 10 milliseconds or so, not too bad, I would say. We did find a couple ideas for improvements here. We might defer some of the work that we're doing. We might break up some of the work that we're doing, but not too bad overall. And then finally, the runtime of the script itself. So again, depends exactly on what you're using the script for. If it's a utility script, maybe you're calling into it quite a bit. If it's a social widget, maybe it's loading all the likes for something. Maybe you have a Bitcoin miner, it's mining Bitcoin for whatever reason. For us, Boomerang, this is where Boomerang does the majority of its work. So after the page load has happened, we look at all the performance data for that page. We package it up, we compress it, we put it on a beacon, and we send it out. So we're capturing things like all the resources that were fetched on the page, how long it took to navigate, what the DNS time was, and a lot of other things like that. If there were any JavaScript errors, we package those up and send those as well. In a lot of the cases on most sites, this wasn't taking too long, say less than 50 milliseconds. But we did find in some examples, especially on lower end devices, on more complex web pages, that we were taking 300 milliseconds or more. And that's starting to get into the territory of people would notice it, a visitor would notice it, that we would be affecting actually the performance of the page. So this is one of the areas that we flagged that we really wanted to get into and figure out if we can improve. And I'll talk about some of the things that we found later. So one thing I did want to point out really quick too is all of the bold phases up there are part of the critical path of the browser using your JavaScript. All of the things up there are generally done serially on the main thread, depending on the browser. But any of these things that you're doing here are affecting the rest of the site being built. And if the user is trying to interact with the page, you're potentially affecting their interaction. So you're delaying the user input, et cetera. So what you really want to do when you have a third party library, a script like this, minimize the work that you're doing, bake it up. If you know you need to do a really big calculation, try to bake it up into pieces. This avoids things called long tasks, which are tasks that are running on the main browser thread that would potentially block user input if they're trying to click or scroll. It would make for a non-responsive user experience. So after everything that I said, these are kind of the numbers that we came up with at the end. We did a lot of investigations on a lot of different sites. We took a lot of profiles. We wrote about it. We published it in a blog post. And these are kind of the TLDR costs of boomerang. So the loader snippet, generally only a few milliseconds on some browsers. It can be up to 40 milliseconds. Downloading, we're about 50 kilobytes or less. Parsing, again, it's related to the size of boomerang. And this is kind of like the low end to high end devices when we were looking at it. Anywhere from 6 to 47 milliseconds. Realization, again, less than 15 milliseconds. Onload, depending on the work on the page, we could be doing a lot of work here. We could be spending upwards of 300 milliseconds, which is quite a bit. And then we package all the data. I didn't really talk about the beacon much, but we have to send the data somewhere, right? This could be anywhere from 2 to 20 kilobytes or more, depending on the complexity of the page itself that we're looking at. So an important output of this, too, was we filed bugs. Everything that we found, every little trace that we looked at, idea big or small, we filed bugs within our GitHub repo. You can actually check them all out there. It's right on GitHub. And since then, we've been trying to make steady progress on fixing these. So I'll go over a sample of some of those in a little bit. But one thing I did want to chat about really briefly is just the tooling, some of the tools that we use to do some of this evaluation. So we really heavily relied on browser developer tools. All the browsers today have really good developer tools. We use profilers and all the different profilers because all browsers behave slightly differently. I know not many of you, maybe some of you, are not super comfortable with things like profilers, but they can really give you the insight that you would need to really look at a third-party script and to evaluate it. So profilers can show opportunities. My advice is if you're new to profilers, if you want to look at it, but you just don't know where to start, take your time, look at the big picture, try to load like a third-party script on a blank page and see what it does, and look for the extremes. Look for the longest amount of time of something being run or the largest call stack. Those can really point to different opportunities or places that are not performing the best. There's a lot of other free open source tools for evaluating different libraries. Lighthouse from Chrome obviously is a fantastic resource request map that I talked about earlier. Web page test, the list goes on and on. I also have a tool that I've made called third-party IEO that helps you audit third-party scripts, and I'll show you more about that later. So as I said, we performed this audit, but one of the main takeaways out of it was we wanted to improve, right? So along the way, we filed a lot of different ideas for improvements. I'm going to go over some of those. I don't think they're going to be appropriate for everybody. You may not care for some of these. I think a lot of them are somewhat interesting, and maybe it'll trigger a little bit of thoughts if you're working on a library or a script. So for some ideas of ways that you can improve yourselves, but it's also just kind of good to understand some of the ways that you can improve. So one of the things I talked about initially was our loader snippet. Again, the snippet that we give our customers helps make sure that boomerang is loaded in a non-blocking manner, but it's expensive. It takes a lot of time to run. We actually looked at some of the features of modern browsers, and we found something that allows us to load boomerang in a non-blocking manner that's much cheaper. So if the browser supports the preload feature, we actually can rely in preload to load boomerang in a way that's a lot better. In this case, instead of 40 milliseconds, it only takes a millisecond. It's basically a no-op. You can barely see it in the profiler anymore. So a lot of research was done into looking into this methodology. We tested it on a ton of websites. We're still kind of deploying it out to some of our customers and stuff like that. We documented all this in our boomerang documentation, but it was a big win for us to be able to point to our new methodology that is still allowing you to load the script in a non-blocking way, but be way cheaper. The other big area that I pointed out was we were doing a lot of work during the Onload Event. This is when we're capturing all the performance data on the page. We're compressing it, putting it on the beacon, and sending it back out to our back end. Unfortunately, this is also one of the most expensive things that we did. So one of the things that we found is we actually captured all of the resources that were fetched on the page. We look at all of the other timings. We save all the URLs. We compress them and then put them on the beacon. This allows our customers and open source users to look at the full waterfall for a page load for every single page that is measured. We compressed it down because it's otherwise really big, a lot of data, but this was taking a long time to do. So what we looked at, we took a look at the algorithm. We tried to figure out ways that we could improve it, and we actually decided that, or found a way that we could decrease the efficiency of it slightly. So it gets about maybe 2% larger payloads than before. So a little bit less efficient in compressing, but it was four times faster to run. So just this little tweak of the algorithm, we thought that trade-off, obviously, of 4x speed up is worth it for the minimal cost in the size of the bytes. So on a lot of the sites, the sites that were taken 300 plus milliseconds are now under 100 milliseconds, 75 milliseconds, et cetera. So that was a big win for us. Another big area of thing, or a big thing that we wanted to focus on was just reducing the size of boomerang in general. We actually had quite a few improvements in this area. For one example, we actually had an open source community member notice that we include a lot of debug messages even in our production builds. These are things that we use in our debug builds to understand what's going on better, but the messages, even though they would never be used, were put into our production builds. So we stripped all those out, saved 6% of the bytes. We did other things like change our minifier from Aglify 2 to 3, saved a couple bytes. We weren't using Brotly before. We were just gzipping our content. Luckily, the Akamai CDN knows how to do Brotly, so we enabled that. So that was a really easy, cheap win for us to do, saved 11% of the bytes. And then we did things like refactoring our plugins. So we have things like our SPA and our MDI, MD5 plugin. We were able to shrink those down tremendously and save a few more bytes there. And I'll show you some of the details about that in a bit. One other thing I didn't talk about too much, or I didn't talk about it all yet, is our cookie. So we set a first-party cookie on all of the pages that were on to help measure sessions. So this is how many pages the person visited on your website, for example. Our customers want the cookie to be as small as possible. And in some cases, we're over a kilobyte big. We found some ways to reduce the size of the cookie, for example, instead of storing the full URL of the page, we hash it and just store the hash, because we do comparisons of URLs in various places. Anyway, at the end of the day, we ended up making it 41% smaller, which a lot of our customers were happy about. And then another thing that we found while profiling it was we were reading and writing the cookie a lot. We were doing 21 reads of the cookie on a traditional, on a regular page load. We reduced that to two, and we're doing eight writes, and we reduced that down to four. So we just, you know, made it a lot more efficient. And then the final big area improvement that we did was just simplifying our plugins. One of the things here was we actually were using MD5 to do hashes of things like URLs. One of our developers found a replacement for MD5 that is about 5% of the size of MD5 and three times faster. And for our needs, it provided just as good hashes as MD5 was. So that gave us a really big byte savings and speed up. And then we did another kind of refactoring of our single page application plugin that just made it smaller as well. So with all that, we actually have some improvements from the numbers that showed up earlier. So the loader snippet in modern browsers is generally only a millisecond or less. I talked about a lot of different ways that we reduced the size of boomerang from all those changes. If you can see the number here, it's actually the same size right now. We also worked on new features in the meantime, and those new features ended up eating a lot of the byte savings that we did over the same period. So it's definitely something you've got to keep an eye out for is the needs of performance versus the needs of new features, et cetera, and often take away all the improvements that you're making. So it's still something that we're focusing on, and we have other ideas for reducing the size of it. But I guess another way to look at it is it would be a lot bigger now with all these new features had we not spent the time to also do other improvements. And then the other big section was the onload event. We made it a lot faster. There's still a lot of opportunities here. If you take a look at this list, these are things that we still want to make better. We want to make faster if possible. And we're hoping to focus in 2020 and beyond on further reducing the size of boomerang, further reducing the overhead, making it more efficient, making it much less likely to have an observer effect for any website that we're on. Again, we're tracking all these improvements in GitHub. And one thing that we kind of focused on for, and I thought was a really good quote from Rico Mariani from Microsoft is, in a mature product with a healthy process, you're much more likely to see a 50% gain come in the form of many 5% gains compounding. In other words, you're not always going to find the really big low hanging fruit that's going to make it 50% faster, right? It's going to take constant, constant iteration, finding little things, doing little tweaks here and there, and making improvements over time and continuing to make those improvements over time. So another important thing to us is now that we've made all these improvements, how can we make sure that we don't accidentally regress later, right? It's very easy to undo your hard work with a few couple mistakes. One of the things that we worked on was what we call our boomerang performance lab, which is just a suite of tests that we run in our CI environment. It's a bunch of scenarios, simple scenarios and metrics that we capture with a headless browser. We do things like looking at the CPU time of the profiles for all the work that we're doing. We have various counts and durations that we're measuring just using user timing marks and measures. And we just look at simple things like the code size that we have, and we can plot this over time. Another way of protecting the line is looking at real time telemetry. So you could, if you are already sending data from your library, you could look at things like runtime stats for different events that you're doing, whether you're triggering any long tasks. For us, for example, we actually capture all of the errors that we throw within our own code. We have a big try catcher on everything, essentially. We package that onto the beacon and send them into our real time dashboards. So we can see in real time, as new versions of boomerang come out, as new customers come online, as new browsers come out and break things, what's happening with the health of our library in real time. So these are all the different JavaScript error messages that we're seeing recently. So if you have all this data, it can really provide an answer into the, provide a look into the health of your system. And finally, there is a really cool new API out there called the JavaScript self-profiling API. You could use this for your website. It's an origin trial right now. But it lets you actually get sampled profile traces in the wild from a subset of your visitors. And so if you have a way of looking at these sample profiles in aggregate, you can kind of get a sense of how expensive your code is running in real time, not just on your developer machine. So that's what we did. What can you do? Let's say again, your boss comes to you and wants you to add a new script to your page. What should you do? Well, you can perform a lightweight performance audit. You don't have to do all the stuff that we did. You don't have to go into that depth of it. But it can be useful just to see exactly what that script is doing when you're putting it on your page. Put it in an empty page and take a profile trace, see what it does, see what it loads, make sure that its benefits are outweighing its costs. I think you should also try to ask the vendor, have you performed a performance audit? How does your library perform? I think it's good for us as a community to all share this information to be transparent. I think it'll help everybody here. I'm a little opinionated on that. But I think it'd be great if other people did this as well. And then, you know, every library that you include on your page, try to have an owner, the internal champion that knows about that script, why it's there, what it's doing, and when can you remove it. If you have somebody that really knows this or if you've documented this somewhere, it can really help with the overall life of, like, a web application for all the different apps that are being included. Besides just all the performance stuff that you could see in a profiler, there's a lot of other things that third-party scripts should be doing. There's pretty much like a checklist of best practices, right? So try to load the script from a CDN or your own CDN. Make sure it's compressed, make sure it's minified, make sure it has the right caching headers, make sure it's minimizing the amount of work that it's doing from CPU and from the network, make sure it's minimally touching the DOM, et cetera, and then make sure it's not doing a bunch of things, too. So make sure it's not triggering JavaScript errors. Make sure it doesn't have the debugger keyword in it. Make sure it's not throwing alerts. These are all things that scripts, ideally, should not be doing if you're including them on your page. One of the tools that I made to help you know about this is called third-party I.O. So if you take any JavaScript URL and paste it into the text box, it runs that JavaScript in a headless browser. It goes through that checklist that I just told you about and more and kind of gives you a list of best practices. Is this script loading too much? How does this compare to all the other scripts that we've looked at, for example? So it's a free tool. Please use it. Let me know if you like it. A bunch of links. I'm sure we're going to share out these slides, but it's just pretty much everything I've been talking about from the audit that we published. We just recently published an update with a lot of the stuff I talked about today in the Perf Planet calendar and third-party I.O. as well. Thank you. That's all I have for the slides. Do we have a few minutes for, okay. And we do have a few minutes for questions, three or four minutes if anybody does have one. So, yeah. So the question was we said we cut down the number of reads and writes to the cookie. Could you, is there a performance implication to doing that? We actually saw these cookie reads and writes on the profiler quite a bit. My guess is because often cookies actually get pushed to disk or to more solid state storage. You may see them pop up a little bit more just because of that. So after we did this optimization, we often don't see cookies read writes showing up as much. What we were doing was excessive, you know, 21 reads of the cookie doesn't seem necessary. A lot of that was probably cache in memory, but so we just figured we could do it better. Question? Yeah. You're doing some computation with the beacon data. You do this on the main thread? Yes. The question was we're doing all this work on the main thread to compress the data into the beacon and stuff. We have thought about that a little bit. It would be quite a complex thing for us to get a web worker working on all of our customer sites as well. So we played with the idea, but ultimately we decided just to do everything that we're doing on just in the library itself as opposed to a web worker. Yeah. Yeah. Yeah. Can you say the first part of that again? Yeah. I mean, we're using Canvas. I would like to know how you would help us to profile what Canvas do behind the scenes. So the question is they're using Canvas a lot and what can you do to profile it behind the scenes? I don't have a ton of experience in Canvas. I would assume that the profiler would be able to tell you a lot about what it's doing. I mean, there's still all these interactions with the APIs that you're taking, et cetera. Profilers like Chrome also show you the frames when they're rendered, et cetera, so you can see all the work that it took getting up to that frame and stuff like that. But I would just say profiler itself, if you're using a library to draw the Canvas like using D3 or something like that, you can certainly put it through its paces as well in the profiler. You can make sure it's being loaded in all the right ways, but those libraries are generally pretty optimized if they're doing drawing and stuff. Yeah. We still have another minute or two. One more question over there. And I will take more questions afterward if anybody else wants to. Please. Question is how is your product compared to century? Compared to... Century? What? Prox, sorry. Century. Century. Oh, JavaScript, you're reporting stuff? Century I.O. Yeah. Yeah. So the question was how does Boomerang itself compare to something like Century? I haven't used it very much. Boomerang does capture JavaScript errors on the page, and we do report those in our real-time dashboards providing a similar-ish service, but I'm not sure all the comparison details between the two. But yeah, it's certainly worth checking out if you're interested. We hope.
a website, if Boomerang is on it, it's going to websites to capture this data. In building a perspective, what is the cost of including Boomerang on your page? What is the cost of improve, and we wanted to be able to share this bad with what you have, and to help the customer with the overhead that it takes. So for the past library that we need for our website. It's just page loads a day, which is quite a lot. According to minimize this as much as possible. And size trying to improve the performance of your page. downloaded. The browser parses the JavaScript and other things that are downloaded. So if we take a affecting the rest of the site being built. And you're potentially affecting their interaction. third-party IEO that helps you audit third-party library or a script. So for some ideas of ways we're capturing all the performance data on the on a lot of the sites, the sites that were taken builds. These are things that we use in our debug another thing that we found while profiling it away all the improvements that you're making. So finding little things, doing little tweaks here So another important thing to us is now that we've made all these improvements, how can we metrics that we capture with a headless browser. the code size that we have, and we can plot this subset of your visitors. And so if you have a way community to all share this information to be are being included. Besides just all the runs that JavaScript in a headless browser. It everything I've been talking about from the audit that we published. We just recently published an