Scaling Facebook with OpenSource tools
Recordon, David
MacVicar, Scott
This talk will give you a better idea of what it takes to scale Facebook. From the day that Mark Zuckerberg started building Facebook in his Harvard dorm room in 2004 to today, the site has been built on common open source software such as Linux, Apache, MySQL, and PHP. Today Facebook reaches over 350 million people per month, is the largest PHP site in the World, and has released major pieces of our infrastructure as open source. It's not possible to scale a site like Facebook simply by sharding your databases, rather we've developed and contributed to a series of open source infrastructure technologies. Some of these projects include Cassandra, Hive, Haystack, memcached, and Scribe, where each focuses on solving a specific problem with Thrift allowing them to communicate across languages. This talk will give you a better idea of what it takes to scale Facebook, a look into the infrastructure we use to do so, and dive into performance work we're focused on in order to scale PHP to over 350 billion page views per month.
 I'm David Rekordon. I spend my time at Facebook focused on open source and web standards. My name is Scott Mavicker and I work on anything open source that's PHP related within Facebook. So the recently released hip hop is my little baby at the moment. So first we should really talk about what do people do on Facebook so you can sort of get the idea of the scaling challenges that we sort of have there. For example, 8 billion minutes every day are spent on Facebook. And it's over 3.5 billion pieces of content is shared every week, be it photos, writing on the walls, talking. And that's 2.5 billion photos every month are added. It actually makes us bigger than the biggest photo sharing website out there. And it's more than just the actual website, there's also the API and the platform that people build with Facebook. There's a million users of that. And so that's sort of an idea of some of the amount of content that's being shared on Facebook, the amount of developers that are going and building with Facebook. But Facebook is also incredibly international. Over 70% of our traffic and our users come from outside of the United States. We've gone and built a translation system which is focused on going and engaging the user base on translating the site. I think we translated Facebook into French in just under 24 hours. And so you can sort of see how we've gone and spread around the world, which also presented a set of scaling challenges in terms of how do we go and serve all this information quickly with only having a small number of data centers. So when you think about scaling a traditional website, you're able to go and sort of break up that information. So if you have an email application, that user is generally interacting with their own email. They're not interacting with email from lots of different places. So you're able to go and shard your databases. You're able to go and break this apart by your users of saying, oh, Bob's interacting with data here, Felicia's interacting with data there, and sort of continue just to break that apart. As you have more users, you add more databases and you scale from that perspective. And so when Facebook started, Mark Zuckerberg created it in his Harvard dorm room and it was sort of pretty similar in terms of all of the users were on a Harvard server. And then as more colleges were added, you were able to go and shard your databases based on each college. So students at Harvard would interact with other students at Harvard at Stanford that would interact with other students at Stanford. But then in 2006, Facebook sort of opened this up so you could go and have friend relationships across these schools. And pretty soon you got to the point where it didn't matter whether you were on the Harvard server or you were on the Stanford server or you were on the University of Washington server, but you were going and interacting with people from college campuses all over the country. And so you might think, oh, well, you could just go and scale Facebook from a country perspective, put all of the users in the United States in one place, put all of the users in the United Kingdom in another, put another server for Belgium. But pretty quickly, Facebook got to the point where you couldn't even do that. So this is a visualization showing live friend requests around the world from a few years ago, really going and illustrating that point that on Facebook, everyone is connecting to people all over the world. It's not a site where you can go and scale broken up based on where your users are or just what information they're interacting with. And so this is really that idea of that to go and render any page on Facebook, we're pulling data from many different places. On average, a user has about 150 different friends. And so to go and render a page like that, we're going and talking to all sorts of different pieces of our infrastructure without being able to go and sort of separate it apart by user. So if we take like a standard page on Facebook, say the news feed, and on average, a person will have about 150 friends. So to construct that, we need to go grab data from 150 different friends, and that's split across multiple servers. So we have to do that in less than milliseconds it has to be done in. So we go do that, grab all the data. But it's not just your direct friends. If their friends have commented on their photos or the comments or any of their status updates, that has to be pulled as well. So we're pulling from tens of thousands of different sources, all just to render that one single page. And it's also that every single page that we go and render is different, not just per person, but at what time they saw it. But the graph has also been evolving. I mean, Facebook has always been based on this idea that you're connected to other people and to other things, that you have relationships, and so from a math perspective, we think about it as a graph. So we have nodes, which are the people or the things, and then we have edges, which represent the relationships between them. But we've also seen Facebook grow to no longer just be relationships between people, but you might become a fan of a company or a person. And it's a very different scaling challenge of going and connecting a person to a few hundred other people, versus going and serving Michael Jackson's fan page, which has over 10 million people connected to it. So that's also presented some different scaling challenges as the sites continue to grow. So this is Facebook's architecture at a high level. And it more or less looks like this. There's a load balancer on top and requests come in there, and the requests are spread amongst a pool of web servers. And then these web servers will either use one of our services to fetch data, or it will fall into Memcached, which is our in-memory fast database access, before eventually falling back to the database. And I mean, this looks like pretty much any website that you're going and building today. In terms of you have your web servers, you have your PHP stack, you have your services. So at a high level, Facebook isn't architected very differently than any other site that you would go and build. So first, we're going to look at the web server stack, or essentially what PHP is built on. And that's Facebook. What Facebook is built on, which is PHP. And we like PHP for a number of reasons. The first being, it's simple to learn. There's not very much barrier to entry to learn PHP, so if we hire new developers, then we can teach them PHP relatively quickly because the syntax is very similar to that of C. And it's an interpreted language, so if you want to make changes to it, you make your changes, you deploy them or save them, and you can see them live. You're not compiling, you're not waiting five, 10, 15 minutes for things to compile, so you can see it. And one of the nice things about PHP is the Hello World example. The Hello World example in PHP doesn't actually require any opening tags or closing tags because PHP itself is a templating language. So this is the sort of the nice things that make PHP good for Facebook. Because we have to develop fast and sort of, we move fast, so we can't spend time waiting for other languages. So there's a couple of things that are PHP problematic for Facebook. The first being high CPU usage. Because we do a lot of data assembly, pulling from multiple sources and actually building it within the application server, then we actually spend a lot of time doing that with CPU. We use a lot of CPU for that, as well as memory usage. Memory usage is quite high because of the way PHP works. It's sort of loosely typed, which is one of its benefits. It also uses a lot of memory for each variable because it has to represent a collection of six types. So every variable gets 140 bytes of memory allocated for it. We also potentially like to use logic between multiple services. We have things written in C++ and Python and Erlang, as well as PHP. So if it was just written in solid in PHP, we're going to have to duplicate that logic across multiple services. And finally, if you want to sort of move this into PHP extension to get the speed benefit, then it's sort of hard to write because the macros aren't well documented for the Zend engine. So to solve these issues, especially the first two, we come up with this thing called HipHop. And HipHop for PHP was this thing that we sort of announced last Tuesday and tend to open source quite soon. And what it does is it you point HipHop at your application framework, and what it will do is it will translate your PHP into highly optimized C++ code. And then from that, we can just pass it into the normal G++ compiler and it will just self-contained binary with the web server and your application and all the library logic built in. And HipHop is really designed to give us sort of the best of both worlds in terms of the speed that you can go and move with an interpreted language so that we can go and make changes really quickly from a development perspective, but then also the speed from a performance perspective of a language like C++. And so one of the ways that we think about the different types of PHP code you'll find is sort of this breakdown between the common stuff of like if else or the magic in terms of going and doing like evaluation and dynamic variable and function assignment. And so when you look at a PHP code base, you actually generally find much more of the more mundane things. It's very rare, at least inside of our code base, that we use things like a val or some of those other functionality. And so this is some of what makes HipHop work really well, is that we can go and take the majority of our code base and greatly speed it up in terms of going and looking at stack function calls and stack variables and then taking advantage of a lot of the optimizations that already exist inside of G++. The stuff that's magic is a little bit harder. It's not going to give us as much of a performance optimization. It won't give you as much of a performance optimization. But there's definitely some benefit there as well. So roughly how HipHop works is it performs a static analysis on your entire code base, looking to see what function, what class is declared where and what variables are declared. And from that, I think it's in our type inference engine, which looks at the data types for each of the variables. So if something was an integer and it was always assigned as an integer and it was always acted like it was an integer, then at that point we can just say, this is always going to be an integer, so let's generate just a straight integer rather than our variant type. So we require support. This is for all the variables. So integers map to actual integers in C++, strings map to strings, arrays map to the STL, like the standard array types, and classes map to actual C++ classes. And it's only when we have a bit of the magic stuff, or when types are into a type of variables in integer, then it becomes a string then backed in integer again, that we'll use the variant type, and then eventually go to the code generation. And the code generation is what outputs actual C++ files. And the transformation process for this looks a little like this. So it goes through a parser that's very similar to the standard PHP parser, and then a former static analysis on it, and then a pre-optimizer. And the pre-optimizer tries to remove things like dead code branches that won't be used or change some of the semantics when it knows that the execution could produce the exact same code regardless of the order. So, say you're doing a post-optimization, it's actually easier to do a pre-increment, because then you don't have to use this third variable to store the result. So it does like little optimizations like that, forks into the type inference engine, and eventually there's some more post-optimization to remove any more dead codes that you could find. And it's just code generation and then compiling is normal. So we have some plans for hip-hop over the next year, and at the moment it only supports PHP 5.2. So we intend to get that back up to speed with the latest 5.2 release before eventually adding some of the new features from PHP 5.3. And there are a few things that we're thinking about that aren't currently planned for PHP, but we want to sort of make sure it gets in there as well as in hip-hop. So one of the things we're sort of dedicated to is making sure that the features in PHP and the features in hip-hop don't differentiate too much, so you can actually use either. We plan to add multi-threading support through asynchronous function execution, so you can execute multiple functions at the same time, and eventually support for Apache will come in, probably via FastCGI. And hopefully from that point on, we start to encourage people other outside of Facebook to sort of use hip-hop. We're sort of eager to create a community around that. Yeah, and so we announced hip-hop a little bit earlier this week. We've been working on continuing to release it even a few hours before we got here, so definitely look for that on Facebook's GitHub account as well. So we want to take a look at our services layer a little bit and give you a better idea of a few of the different backend services that we both use and have created to scale different aspects of Facebook. And so from a philosophical perspective, we sort of think about this question of when do we need a service? So is it something that needs to be really quick? Is it something that currently has a big overhead in terms of our application layer around deployment and maintenance? Do we want yet another failure point from our infrastructure, sort of balancing some of those trade-offs? And we try to go and take advantage of as many bits of common functionality as we can between all these different services. And that's another reason why hip-hop is really interesting to us in terms of being able to go and reuse logic we've already written in PHP, compiled into C++, and be able to pull that into our backend services, which are written in a variety of languages. And so this is another piece from a philosophy perspective, is that we don't go and choose a single language when we're going and building out Facebook infrastructure. Our entire web server infrastructure is PHP, but we use many different languages from a backend infrastructure perspective, depending on what we're building, depending on who's involved, depending on when, and some of the different characteristics of these languages for what we're doing. So to communicate between all the different services that we have, we use this technology called Thrift. And Thrift is something that we open-sourced. And it's essentially an RPC server that will generate... It's an RPC server, and you provide it with a damage-agnostic RPC server. So you provide it with a file of your Thrift service, and it's able to generate code for you in C++, PHP, Rang, Python, Java, Ruby. So it generates all of these for you, and it also deals with the communication between each of the services, as well as the actual structure, so the structure of the communication. So it'll do the serializing and the unsealizing of the data to pass it along the wire, so to speak. And the wire protocols that it supports is like Socket, the file transport, and then just standard in-memory buffers. So you can write something in C++, serialize it with Thrift, and then talk to PHP Thrift server, and then it'll unsealize it for you. So it's the sort of how we pass data amongst our different services at Facebook. And Thrift is actually an Apache incubator, so anyone can go download it and use it, and that's one of the things that we gave back to the open-source community. I'm not exactly sure when to put it. Well, and Thrift is also using a variety of different projects, I think companies like ReCapture are making use of it, as well as other projects. So Thrift is used by some bit of Hadoop as well. So one of the other services that we want to talk about is logging. How many people here have used Syslog? Yeah, we did two. So I think this is one of those themes of, we always start with what is looking at outside of Facebook, what is either a really great open-source solution, or what is a good commercial solution, and then ultimately going and building on top of it. And so we were using Syslog for logging all of our data, and as you saw behind me, the logging server sort of exploded, because we were going and logging so much information from all these page views. At this point, I think we do about 400 billion page views per month on Facebook. And we log about 25 terabytes a day. So we went and created Scribe, and the basic idea behind Scribe is that we're able to go and sort of break up this funnel from a logging perspective. So you'll have this information coming from our web servers, or coming from other backend services. That will get routed into Scribe servers, which sort of work at funneling it down and condensing it. And then we ultimately go and store this information inside of our Hadoop and Hive cluster, so we can go and look at it from an analysis perspective later on. And so Scribe is one of those technologies where logging is a common problem. Everyone goes and has to have logging inside of any site that they build. And we got to the point where we just weren't able to work within the scale of some of the other solutions that were there. So we also open source Scribe. It's also being used by Twitter right now, and they're contributing back to the project. And I think it's actually one of our projects that's sort of the least known about, but compared to sort of what it does and how much of a common problem it is. So one of the more popular things to do on Facebook is actually to upload photos. And at the moment, there's 40 billion photos that have been actually uploaded to Facebook. And we store that in four different resolutions for each of the areas. So there's the thumbnail for the wall, then there's the thumbnail within the actual gallery overview, then there's the actual photo that you look at, and there's a fourth one as well because original larger size is kept. So we store four different photos, four different variants for every single photo. And we have to, and we serve that. That's 1.2 million photos or seconds are served. So that's how many photos we're serving every second that people browse around the site, because you've seen the Facebook homepage, it's pretty photo intensive. Okay. And one of the problems that are how you would normally try and scale photos, on a site so large would be sort of like an NFS share. So you set up a central NFS share and then multiple servers would then read and write to that NFS share. And you would use HTTP just to serve the photos directly from NFS. And the first thing we actually did was a commercial solution, because you have to choose the battles that you're going to fight with and what will take more time to do. So we went for the easiest solution, which was to get a commercial, vendors come in, get them to set up NFS and use their technology. But unfortunately, it just didn't scale. And the reason it didn't scale wasn't because the commercial technology was bad. We just had so much IO, we were just trying to read so much and write so much that it just simply didn't work. So we had to look for another solution. And so then we started optimizing. So I mean, we went and broke up in terms of where we're caching images. So make sure that images that are more recent, that are smaller, that we're being able to serve those out of a cache versus having to go back to disk every time, starting to use content delivery networks to go and spread across this load, even pull some of that caching outside of our own infrastructure, as well as then did some actual caching around the NFS infrastructure as well, trying to remove some more of that metadata. But fundamentally, part of the challenge that we were running into was just looking at how an operating system stores information on disk. So in order to serve one of these files, we were having to look up about three pieces of metadata for every file. So you had to go and look at the directory I know, the data inside of that, then the file I know, and then actually get at the data itself. And so what we wanted to try to do was go and do this in one disk seek. And so we developed a system called Haystack. And what it does is it allows us, for any photo that we're going and looking up, to go and serve it in terms of one physical read on the disk. So it doesn't matter from a random data access perspective, it's always a consistent time of one physical read in order to serve a file. Overall, it takes about 300 megabytes of RAM to store this index for every terabyte of photo information that we store. And so you're able to sort of see how this compares to some of the other solutions. We started at a system which was about 10 physical disk seeks to serve a photo. We got down to three. And then Haystack ultimately got us to one. And I think one of the other really interesting parts of a technology like Haystack, and this is true for a lot of our infrastructure, is that it was built by three people. So it's really one of those which is interesting for us in terms of being able to go and have an engineering culture where small teams are really able to have a large impact, are able to go and ship things. And so Haystack is a technology which isn't currently open source, but we're working on open sourcing it, because it's one of those that we really think is useful to all sorts of different sites from a variety of sizes, and not just for Facebook as well. So let me talk another bit about one part of our infrastructure, and that's Hive, which builds on top of Hadoop. So I'll talk a little bit about Hadoop first. This is some of the Hadoop. Hadoop is map reduce. So the first thing here, this example runs at the top is the map query, which looks for any key that's greater than 100 and just outputs whatever that key was. And then next example is the actual reducer, which looks for the count of unique items that were in the output from the map script, and then actually prints the value as the key and then the count. Now the problem with Hadoop is it's just not simple to use. It's very hard to understand, and we come up with another solution for that, which would be Hive. And Hive is very much an SQL syntax, and what it does is it builds on top of... Oh, you can't quite see that there. The people here can't on the left. Okay, you can stand up. You can see that Hive is very much an SQL syntax, so you can just select key, count from val, we're key greater than 100, grouped by the key, and in the background, that actually translates it into Hadoop query and run it on your map reduce request and then run it on your behalf. So Hive is actually in the Apache... It's actually an Apache project now. We open source that again, put it into the incubator, and Apache picked it up, and the support and development is actually happening outside of Facebook now. We're still one of the main contributors to it. So, yeah. And the interesting thing about Hadoop is, Hadoop actually came from Yahoo, and that actually came from papers that Google had put out. So the interesting thing here is that lots of different corporations are building on top of each other's open source technology, and that's something we're kind of big on. We want to make sure we can open source the things that we are finding useful within Facebook. And I think we really love Hadoop in terms of what it's able to let us do, being able, as Scott said, to go and build infrastructure on top of it. But I think some of it, when you go back to that syntax bit, is just looking at who's using it. And so one of the things that we were really trying to do was make Hadoop much more accessible inside of the company so that more people were able to go and work with it and sort of use Hive as a layer on top of the underlying Hadoop infrastructure as well. And so this is sort of what our data flow architecture looks like. This relates back to what we were talking about with Scribe a bit in terms of we have our web servers going and sending data into Scribe, going and figuring out where to file that, putting that into our Hive and Hadoop cluster, replicating that, and allowing people to go and run jobs on top of this. And so you also saw a picture in the previous slide which might look familiar. And so this is what our Hive and Hadoop cluster looks like for that. But, I mean, overall, we have a lot of information going into this cluster. We couldn't have done it without Hadoop coming before us. And I think that's really one of these important points from an open source perspective is being able to go and build on top of software that others have released, going and changing and being able to make it solve your use cases a little bit better, going and releasing it again for others to continue to build on to make better to innovate together. And really with the idea with Hive was going and simplifying Hadoop because we wanted people other than engineers to start working with it. At this point, about 250 people throughout the company use Hive every month. We're running about 7,500 jobs every single day on that. And a lot of people who aren't engineers are actually doing this. So we have data analysts all throughout the company in different parts, which are going and taking advantage of Hive syntax as well as some other web tools on top of it to really access this information, which Hadoop is going and crunching for us. So I think this is one of those really great cases of you have a technology like Hadoop, which is awesome. It's a great piece of infrastructure. And then being able to go and use open source to continue to build another layer on top of that, which makes it more accessible to more people. So one of the other pieces of the infrastructure that we want to talk about is Memcache. I'm guessing if I ask how many people here have used Memcache, they will be about the same as Syslog if not, yep, exactly. So like Memcache is awesome. We love Memcache. It was one of those technologies where when you go back and start thinking about how was Facebook built from the beginning? I mean, it was built on top of Linux and Apache and MySQL and PHP. And then if we could, we would add like one more M after that. So it would be like the Lampum stack with Memcache in there. In terms of it's one of those technologies that's really robust, it's scalable. It lets you get a lot more out of your site if you're going and using it in a smart way. So overall, I mean Memcache was developed by Brad Fitzpatrick at Danga, originally for LiveJournal. It's being used by just about any dynamic site that you can think about. And it's something that gives us a lot of performance benefit, but it's also up to our engineers in terms of making sure that they use it and that they use it in a smart manner. So we'll talk about that a little bit more. So this is a slide which I think was originally created to try to describe caches to a non-technical audience. And so it's sort of interesting from that perspective. But I think it's a little bit more interesting. We currently serve about 120 million Memcache requests every second. So we're incredibly reliant on Memcache in order to make Facebook work. And so if you go and think about that from how these different pieces of the stack come together, which Scott will talk about in a little bit, Memcache is really a critical piece of it. We've done some performance work around it as well. One of the good examples of how we used Memcache and how we do use it was when we launched usernames. I think this was a little bit over a year ago. I think there were about 200 million active users at the time on Facebook. And we were trying to figure out what would be a fair way to go and give everyone a username. And so we thought about a few different options. And ultimately we decided that the most fair way to do this would be just whoever got there first. So we went and asked 200 million people to go and access Facebook at exactly the same time. So I think a lot of people probably are thinking of this as like a denial of service attack that you bring upon yourself. But for us it was really a product launch. And so the team got together, they tweaked things, they tuned things. And we served and gave away over 200,000 usernames in the first three minutes. And a million usernames were assigned in that first hour. And so really that graph before was going and showing our memcache hits in terms of leading up to this release and then at that point as well. So memcache is an incredibly important piece of technology for us and something that has a huge impact in terms of how we go and scale. Memcached itself is generally quite robust. It's filled with really nice features. But we wanted to make it better. So a couple of things that we did with the memcache at Facebook is we made it a 64-bit port because most of our memcache machines have about 16 gigs of RAM. And before we used to have to actually run three, four memcache processes so we could actually allocate all of the memory because it was only a 32-bit port. So what we did is we ported to 64-bit and we open sourced that. And that's now in the main memcache that you can get from most distributions. We added a multi-threading so we could actually utilize all of the cores that were on the processors. It's all the cores and processors that were in the machine. So we could fetch data and get it out there quicker. And one of the more interesting things was actually adding UDP support. Now when you add UDP support, it means we actually have to do packet assembly on the memcache client rather than relying on the network stack to do it. And that was mainly because there was so much memory getting allocated to the buffers within the kernel that it was using more memory on the machine than we'd like to. So we changed it to UDP and then we adjusted the memcache client so it could assemble UDP packets itself. At the moment, we haven't got around. We released about a year ago, maybe a bit more. We released all the changes we had up until that point. And some of them have made it back into the actual memcache server. And sometime this year we're going to do another release of the memcache server. And hopefully they'll get merged back into the upstream version again. Yeah, and I mean memcache is one of those examples where it works really great in terms of core memcache for most site that you go and build. And there are actually some of these things that we've gone and done which won't be as useful at a smaller scale. And we're going and making some of those trade-offs between performance and features as well. But overall, like memcache is, I think, another one of those examples of incredible open source technology that came before Facebook that we were able to go and really take advantage of as well as then contribute back to and evolve. So next I'm going to look a little bit at the database part of Facebook. And here at Facebook we use MySQL. MySQL is our database of choice. And mainly because it's the ShareNothing architecture. We have lots of master-master databases that replicate themselves around. And you can sort of think of this as like if one, an army, and if one sort of soldier loses its head, like we have an example here at the top of the photo, then it doesn't matter because it will keep going forward. And at Facebook, it doesn't really matter if a single server disappears. We have other servers that will back it up. And that's really one of those choices that you make when you're thinking about how you scale is do you go and invest in technologies where you sort of like grow one piece of your infrastructure really large, or do you go and find a way where you're able to sort of break that up? And I think that's really what we've done with our databases with MySQL in terms of being able to go and have a lot of more independent database clusters that were able to continue growing that entire pool rather than having one really large database cluster by itself. And overall, I mean, we really like MySQL. We found it to be simple and fast and reliable. And normally when we go and break our databases, it's because one of us did something stupid, not because the software wasn't working right. And so we really think about MySQL as like that freight train, which is really solid, it keeps on running. And a lot of that is also based on how we go and use our databases in terms of we don't take advantage of a lot of the relational aspects, we don't go and use joins inside of our database. But in general, we found MySQL to be an incredibly reliable piece of storage infrastructure. And definitely is a good example of what you can go and build from an open source perspective. I see. The database that we actually use, the N-O-D-B engine at Engine, because it's the only one that's got full transactional support and we can make sure everything makes its way to disk. But this is overall, okay, so this is sort of the high level look of how the architecture is at Facebook and we'll look at databases again. So most people see databases as sort of a collection of secondary indices. That's where you get all your relational features, that's where you do your joins. But here, we more or less use it as a way to keep persistent data. It's where we store the data that writes it to disk. That's our persistence layer. And if we want to do any sort of joins, then the joins are actually done within the web servers. And that's why HIP-HOP is important because we do a lot of processing there and reassembling the data from its different sources. And then once we have that sort of put into Memcache. And Memcache can be seen as sort of our secondary index. That's where our data is stored. So really it looks a little more like this. So that's probably a more true representation of our stack where the web server looks in our distributed index to see if the data is there. And if that fails, it will look in their persistent storage. But it's more than just the web servers that look within both Memcache and MySQL. We also have other search services that themselves will look in the various data locations. Yeah, and so hopefully we gave you sort of a pretty good overview of a few of the different services and technologies that go into scaling Facebook. Both in terms of open source software, which we use, some of the things that we've built on top of that as well. And so that was really our goal in terms of going and looking at a few of these different places of our architecture and really how important open source software is to go and scale Facebook since the beginning of the site. I think one of the other things, and we talked about this a little bit with Haystack, was that a lot of this actually starts as a hack project. So it's not necessarily that we're going and set out trying to solve this specific problem and have a lot of people thinking about it and how to do it, but rather a few people will just sit down or a single person. I mean Haystack was started by one engineer, Hayping, sorry, that was Hip Hop. Haystack was started by Jason Sobel. And it's really getting to the point where we prototype something. And we're also from an infrastructure perspective, when we're trying to solve a problem, we might go and try two or three different things at the same time, prototype them out for a few months, and then throw two of them away and move forward with the third one. And I think that's something that's really important, and it's sort of a piece of our culture as well from an engineering and company perspective, in terms of really how these technologies help us go and do things like move fast. And so that's a really important piece of Hip Hop, in terms of it allows us from an engineering perspective to go and make changes with the development speed of PHP, being able to go and sort of save something and refresh that immediately on your development server, but then going and having a lot of the operational performance of a compiled language like C++, being able to go and develop technologies in a small team and actually have them scale out to the entire site, release this open source, and really that idea of continuing to be bold and innovating, goes into our engineering culture as well. So everything that we talked about, you can go and find at facebook.com. We also have a variety of developer tools there, so libraries such as 320, which are a lot of the UI aspects and some of the data aspects behind our iPhone app. So it lets you go and do things such as like have a list which you scroll through, which is actually going and pulling images from a server, instead of having all that information being stored locally. As well as tools like Tornado, which is another pre-faceted infrastructure, which powers FriendFeed's web server. It's designed to be a really high speed web server, because they have a connection open for every user that's actually on the site at the time. As well as other development tools, things like PHP shell for going and doing autocomplete and stuff like that. So overall, we're really excited to be here and have a little bit of time for questions, as well as a bit of a pile of shwag down at the front. We want to come and grab any. So thank you. APPLAUSE Yeah, my question is about Hey Hope. Why are you translating PHP to C++ instead of directly developing C++? So what was the last bit? Sorry. Yeah, why aren't you not developing their website in C++, instead of developing PHP then translating to C++, optimizing and doing a lot of stuff? OK, the question was why aren't we just developing in straight C++? Why aren't we? OK. So the current code base at Facebook is about four million lines of code. So the first problem, four million lines of PHP codes. So the first problem would be how do we translate all of that into C++, as well as allowing people to keep developing with the site. We can't exactly hold up development while that's being done. And it comes back to how long does it take a C++ project to compile? We like the fact that we can move fast, people can make little changes, save it and then look onto the web server and the changes right there. Yeah, so a lot of it is just that development speed of being able to, like, more people know PHP, it's easier to work inside of PHP, so we really want to keep it as a language from a development perspective. Hi. Thanks for the presentation. What eventual consistency do you tolerate? So how much consistency do we tolerate? So, I mean, we tried... Oh. Hmm. I don't know the exact answer to that question. I mean, so, like, we really try it for, I mean, anything that you post, any comment that you comment on, like, as soon as it comes in, we record it, we make sure that we go and serve it again. I think with services like Newsfeed, you have the two views now. So you have one which is going and showing that most recent view, everything that's come in, but then you also have a view which goes and sort of looks at what do we think is the most relevant thing for you to see right now. So I actually think it's probably a little bit less from a data storage perspective and more from a presentation perspective in terms of what specific feature on the side are you going and looking at and how are we going and optimizing that at the time. You guys were talking about, you chose to put the joins and all the filters in the web server level. I suspect you guys did a lot of benchmarking and checks at what point that would be an advantage by doing that because for smaller size it probably won't be. Do you have figures for that? Can you distribute them? Because those would be really interesting to see and at what point it would finally be an advantage to do it. Yeah, so in terms of, like, at what point do different pieces of these technologies make sense? Yeah, I don't think we necessarily have a lot of benchmarks in terms of something like do you do joins in your database or do you go and just pull data together and join it together at your application level, at your application layer. Some of that tolerance is also probably depending on how large do you want to scale your database cluster and where are you wanting to spend some of that CPU time. Another question probably, data is very crucial for Facebook. So how do you prevent the loss of the data and probably you use the mechanism like duplicate and et cetera. And the second question here about the data, did you have already the problems with it and how do you solve them if you had? So I think your question, your first question was about replication and then what was it second? The second was if you had some data loss in the past and if you had, how did you solve it and to not have it in the future? Yeah, I mean, I think from a data loss perspective, a lot of those things that you learn about how do you go and run database clusters apply the same for us as well. In terms of going and making sure that you're having backups that your replication is working, that you're keeping your bin logs so that you can go and can replace some of that information. So a lot of, I think the scaling lessons that you learn, at least in terms of how you go and run databases, are pretty similar, whether you're at Facebook scale or smaller than that as well. It's really a lot of those best practices applied at a much larger scale. You've talked about scaling at technology level. How do you scale internally with your developers and you've got fairly high velocity releases? How do you ensure all your developers on a consistent frame of reference when they're developing new features? Yeah, so really scaling is more than technology, as you said. It's also about scaling in organizations, scaling in engineering team. And so we've developed, I mean, we've developed a variety of internal tools which also help us do that. So we've gone and layered things on top of our version control repositories, so have tools to go and do things like code reviews, to be able to go and trace a code review all the way back from that release that we're doing to when was the code committed, who reviewed it, what was happening inside of that. Every single commit at Facebook is reviewed by someone else. And so it's definitely one of those challenges that we focus on as well in terms of how do we continue growing and moving at this pace. Hello. Yeah, over here. Excellent presentation, by the way. Love it. Really interesting in Haystack. So I guess the average size of a photo, how big is that? A few hundred K or less? Yeah, I mean, I think an average size is certainly under 500 K. Yeah. Generally, the larger size that we store is 640 by 480. And what we actually do is our photo uploader, so when it's occurring on your computer, is going and resizing that photo before it's even sent over the network. But my question is then, how will the, could this scale for, I don't know, say, org-vorbis standard 4 megabyte file or even movies? Or how do, how many discreeds per megabyte sort of? So I think one of the things that, to keep in mind there, is that percentage in terms of the amount of time you spend going, reading metadata compared to the file size is really important. And so when you have a smaller file, then the amount of time that you spend going and looking for that metadata is actually much larger. If you go and start to get into something like videos, then it becomes a much smaller percentage of the time that you spend overall. So we don't actually use Haystack to store videos, but only for photos right now. Yeah? Yes? Up here, up here. Okay, I was wondering, can you give us some numbers on the breakdown of how many servers are web servers, how many servers are memcache, and how many servers are the database? No, I actually can't. But I can tell you we have tens of thousands of servers overall. Just on there. Sorry. What language is hip hop written in and what's your plans for growing a community around it? What was the second the written in C++ and what was the second question? What's your plans for growing a community around hip-hop? What's our plans for creating a community around it? At the moment we've open sourced it. We haven't released the code yet That's gonna be happening soon a couple hours before this I was finishing the last of that with a CMake and Making sure it could build on non Facebook platforms We've created a mailing list and we've had a couple of thousand people joined the mailing list already Is that right? Yeah, there's a couple of thousand people joined the mailing list already and we're hoping as people start using it The community will just naturally develop around it You have to remember that hip-hop isn't designed for everyone to use It's they can if they want but it doesn't really make sense for the small person running their own shared WordPress blog It's more for those larger sites So we're hoping that they'll realize performance benefits that come from it and they'll start adopting it and from there Oh, just hopefully just snowball Yeah, hi. Yeah, thanks for the talk. What about HTTP servers? Do you work with the patch engines or? multiple So we've generally been using Apache and PHP But hip-hop actually has its own embedded web server, which is sort of really simple web server built on top of lib event And so we're now been moving to using that How hard it is to port a zend extension to hip-hop? Hi To change an extension to hip-hop you basically just write a C++ class as normal And then it's got the similar parts of the PHP extensions initialization function There's a request initialization request destroy and request sorry module unload and again It's some more glue code with macros to expose your C++ class back into hip-hop So it's similar to the PHP extensions, but less random macros they might not understand Hi Thank you for the presentation first of all and my question was Where exactly do you use Erlang and do you make use of Indonesia and the concurrency that the Erlang provides? So we use Erlang for our chat service So for Facebook chat It's your generated C++ code using a boost library and what libraries do you use in the generated C++ code? Yeah, we missed that completely. Sorry. We couldn't understand we couldn't hear you Is your generate C++ code using the boost library? Yes, our generating C++ code uses the best libraries it uses file system system and a couple of the mutexing and the concurrent execution the threading ones If you have more questions come down and ask afterwards Hello, thank you for the presentation, but when will we get a dislike button? What when will we get what a dislike button? Um, we let's talk about that later. Let me know if there's specific information you want off of them Okay, I have another question about hip-hop Is it currently possible to cross compile it for example to get a windows binary? And if that's not possible would that be easy to implement that function? At the moment it only compiles on Linux Specifically last week only compiled on CentOS We've now got it to the point that should compile on all Linux distributions It uses CMake and CMake has support for cross compilation At the moment there's no support on OS X or Windows, but we're hoping with something from Microsoft is contacted us and hopefully they'll contribute the Windows part and Hopefully something the community will either contribute the Mac port or I myself will do Hi, my understanding is that you only have data centers in the US, right? No What are you asking where servers are now? My question is how do you deal with the high latency from the US for users that are outside of the US? So dealing with latency So I mean in the US we have data centers on both the West Coast and the East Coast We also use CDNs to help distribute some of the like things like images and CSS and JavaScript around the world I think that's been something that our operations team continues to work on as this site continues to grow internationally as well So there's still five minutes left are there any more questions? And we'll stick around as well if you want to come ask down here. We might be able to hear you better too Yeah, hi, you mentioned that you use CDNs. Do you use other people's CDNs or do you have your own CDN deployed around the world? So we use Akamai for a lot of our CDNs Hi, I was wondering what kind of open-source software you guys use for Monitoring capacity planning and trending and if you experienced issues scaling with those as well as the Facebook application Yeah, we know a little bit less about the monitoring technologies that we use I know that we use ganglia. I think we also use cacti We also have some tools that we've developed internally in terms of managing our overall infrastructure But I honestly know a little bit less about the scaling challenges that we've run into those You talked that you're using revision control systems for Fastly developing your site, but what kind of software are you using? Are you using currently single revision control system or different teams with different revision control systems? So I think you were asking about like what software our developers use I mean So yeah, I mean We have a variety of like we use then for a lot of our development environments Engineers then use I mean emacs or vi or whatever insert favorite text editor here And then we'll have some other tools built over on top of subversion and get our main repositories or subversion Over half of our engineering team now uses get on top of that and then we have some tools built on top around that in terms of things like code review and and documentation and things like that Hi there Can you say a little bit about Cassandra and what using Oracle for please? Thank you. So Cassandra Yeah, so we do a team at Facebook developed Cassandra a few years ago. It's also in the patchy incubator right now we use it for inbox search on Facebook and We haven't really been developing it actively in the past year But at this point I think dig and Twitter and Rackspace are really the largest contributors to Cassandra Inside of the patchy project and so I mean I think from our perspective It's really great to continue to see it develop and have a community around it that's going and continuing to push it forward Even though we're not actively developing it as well Hello, how do you manage the deployment of software? So software deployments. I mean we have a variety of tools built around that as well In terms of being able to go and push out information to different hosts going and running jobs on those hosts Distributing information and I mean I think that's something that we'll talk about a little bit more this summer Hopefully at the velocity conference Are there any more questions Okay, I think not so thank you very much for your talk
 I'm David Rekordon. I spend my time at Facebook focused on open source and web standards.  My name is Scott Mavicker and I work on anything open source that's PHP related within Facebook.  So the recently released hip hop is my little baby at the moment.  So first we should really talk about what do people do on Facebook so you can sort of get the idea of the scaling challenges that we sort of have there.  For example, 8 billion minutes every day are spent on Facebook.  And it's over 3.5 billion pieces of content is shared every week, be it photos, writing on the walls, talking.  And that's 2.5 billion photos every month are added.  It actually makes us bigger than the biggest photo sharing website out there.  And it's more than just the actual website, there's also the API and the platform that people build with Facebook.  There's a million users of that.  And so that's sort of an idea of some of the amount of content that's being shared on Facebook, the amount of developers that are going and building with Facebook.  But Facebook is also incredibly international.  Over 70% of our traffic and our users come from outside of the United States.  We've gone and built a translation system which is focused on going and engaging the user base on translating the site.  I think we translated Facebook into French in just under 24 hours.  And so you can sort of see how we've gone and spread around the world, which also presented a set of scaling challenges in terms of how do we go and serve all this information quickly with only having a small number of data centers.  So when you think about scaling a traditional website, you're able to go and sort of break up that information.  So if you have an email application, that user is generally interacting with their own email.  They're not interacting with email from lots of different places.  So you're able to go and shard your databases.  You're able to go and break this apart by your users of saying, oh, Bob's interacting with data here, Felicia's interacting with data there, and sort of continue just to break that apart.  As you have more users, you add more databases and you scale from that perspective.  And so when Facebook started, Mark Zuckerberg created it in his Harvard dorm room and it was sort of pretty similar in terms of all of the users were on a Harvard server.  And then as more colleges were added, you were able to go and shard your databases based on each college.  So students at Harvard would interact with other students at Harvard at Stanford that would interact with other students at Stanford.  But then in 2006, Facebook sort of opened this up so you could go and have friend relationships across these schools.  And pretty soon you got to the point where it didn't matter whether you were on the Harvard server or you were on the Stanford server or you were on the University of Washington server,  but you were going and interacting with people from college campuses all over the country.  And so you might think, oh, well, you could just go and scale Facebook from a country perspective, put all of the users in the United States in one place,  put all of the users in the United Kingdom in another, put another server for Belgium.  But pretty quickly, Facebook got to the point where you couldn't even do that.  So this is a visualization showing live friend requests around the world from a few years ago, really going and illustrating that point  that on Facebook, everyone is connecting to people all over the world.  It's not a site where you can go and scale broken up based on where your users are or just what information they're interacting with.  And so this is really that idea of that to go and render any page on Facebook, we're pulling data from many different places.  On average, a user has about 150 different friends. And so to go and render a page like that,  we're going and talking to all sorts of different pieces of our infrastructure without being able to go and sort of separate it apart by user.  So if we take like a standard page on Facebook, say the news feed, and on average, a person will have about 150 friends.  So to construct that, we need to go grab data from 150 different friends, and that's split across multiple servers.  So we have to do that in less than milliseconds it has to be done in.  So we go do that, grab all the data. But it's not just your direct friends.  If their friends have commented on their photos or the comments or any of their status updates, that has to be pulled as well.  So we're pulling from tens of thousands of different sources, all just to render that one single page.  And it's also that every single page that we go and render is different, not just per person, but at what time they saw it.  But the graph has also been evolving. I mean, Facebook has always been based on this idea that you're connected to other people and to other things,  that you have relationships, and so from a math perspective, we think about it as a graph.  So we have nodes, which are the people or the things, and then we have edges, which represent the relationships between them.  But we've also seen Facebook grow to no longer just be relationships between people, but you might become a fan of a company or a person.  And it's a very different scaling challenge of going and connecting a person to a few hundred other people,  versus going and serving Michael Jackson's fan page, which has over 10 million people connected to it.  So that's also presented some different scaling challenges as the sites continue to grow.  So this is Facebook's architecture at a high level.  And it more or less looks like this. There's a load balancer on top and requests come in there,  and the requests are spread amongst a pool of web servers.  And then these web servers will either use one of our services to fetch data,  or it will fall into Memcached, which is our in-memory fast database access,  before eventually falling back to the database.  And I mean, this looks like pretty much any website that you're going and building today.  In terms of you have your web servers, you have your PHP stack, you have your services.  So at a high level, Facebook isn't architected very differently than any other site that you would go and build.  So first, we're going to look at the web server stack, or essentially what PHP is built on.  And that's Facebook. What Facebook is built on, which is PHP.  And we like PHP for a number of reasons. The first being, it's simple to learn.  There's not very much barrier to entry to learn PHP, so if we hire new developers,  then we can teach them PHP relatively quickly because the syntax is very similar to that of C.  And it's an interpreted language, so if you want to make changes to it, you make your changes,  you deploy them or save them, and you can see them live.  You're not compiling, you're not waiting five, 10, 15 minutes for things to compile, so you can see it.  And one of the nice things about PHP is the Hello World example.  The Hello World example in PHP doesn't actually require any opening tags or closing tags  because PHP itself is a templating language.  So this is the sort of the nice things that make PHP good for Facebook.  Because we have to develop fast and sort of, we move fast, so we can't spend time waiting for other languages.  So there's a couple of things that are PHP problematic for Facebook.  The first being high CPU usage.  Because we do a lot of data assembly, pulling from multiple sources and actually building it within the application server,  then we actually spend a lot of time doing that with CPU.  We use a lot of CPU for that, as well as memory usage.  Memory usage is quite high because of the way PHP works.  It's sort of loosely typed, which is one of its benefits.  It also uses a lot of memory for each variable because it has to represent a collection of six types.  So every variable gets 140 bytes of memory allocated for it.  We also potentially like to use logic between multiple services.  We have things written in C++ and Python and Erlang, as well as PHP.  So if it was just written in solid in PHP, we're going to have to duplicate that logic across multiple services.  And finally, if you want to sort of move this into PHP extension to get the speed benefit,  then it's sort of hard to write because the macros aren't well documented for the Zend engine.  So to solve these issues, especially the first two, we come up with this thing called HipHop.  And HipHop for PHP was this thing that we sort of announced last Tuesday and tend to open source quite soon.  And what it does is it you point HipHop at your application framework,  and what it will do is it will translate your PHP into highly optimized C++ code.  And then from that, we can just pass it into the normal G++ compiler  and it will just self-contained binary with the web server and your application  and all the library logic built in.  And HipHop is really designed to give us sort of the best of both worlds  in terms of the speed that you can go and move with an interpreted language  so that we can go and make changes really quickly from a development perspective,  but then also the speed from a performance perspective of a language like C++.  And so one of the ways that we think about the different types of PHP code you'll find  is sort of this breakdown between the common stuff of like if else or the magic  in terms of going and doing like evaluation and dynamic variable and function assignment.  And so when you look at a PHP code base, you actually generally find much more of the more mundane things.  It's very rare, at least inside of our code base, that we use things like a val  or some of those other functionality.  And so this is some of what makes HipHop work really well,  is that we can go and take the majority of our code base and greatly speed it up  in terms of going and looking at stack function calls and stack variables  and then taking advantage of a lot of the optimizations that already exist inside of G++.  The stuff that's magic is a little bit harder.  It's not going to give us as much of a performance optimization.  It won't give you as much of a performance optimization.  But there's definitely some benefit there as well.  So roughly how HipHop works is it performs a static analysis on your entire code base,  looking to see what function, what class is declared where and what variables are declared.  And from that, I think it's in our type inference engine, which looks at the data types for each of the variables.  So if something was an integer and it was always assigned as an integer  and it was always acted like it was an integer, then at that point we can just say,  this is always going to be an integer, so let's generate just a straight integer rather than our variant type.  So we require support.  This is for all the variables.  So integers map to actual integers in C++, strings map to strings,  arrays map to the STL, like the standard array types,  and classes map to actual C++ classes.  And it's only when we have a bit of the magic stuff,  or when types are into a type of variables in integer,  then it becomes a string then backed in integer again,  that we'll use the variant type,  and then eventually go to the code generation.  And the code generation is what outputs actual C++ files.  And the transformation process for this looks a little like this.  So it goes through a parser that's very similar to the standard PHP parser,  and then a former static analysis on it,  and then a pre-optimizer.  And the pre-optimizer tries to remove things like dead code branches that won't be used  or change some of the semantics when it knows that the execution  could produce the exact same code regardless of the order.  So, say you're doing a post-optimization,  it's actually easier to do a pre-increment,  because then you don't have to use this third variable to store the result.  So it does like little optimizations like that,  forks into the type inference engine,  and eventually there's some more post-optimization  to remove any more dead codes that you could find.  And it's just code generation and then compiling is normal.  So we have some plans for hip-hop over the next year,  and at the moment it only supports PHP 5.2.  So we intend to get that back up to speed with the latest 5.2 release  before eventually adding some of the new features from PHP 5.3.  And there are a few things that we're thinking about  that aren't currently planned for PHP,  but we want to sort of make sure it gets in there as well as in hip-hop.  So one of the things we're sort of dedicated to  is making sure that the features in PHP and the features in hip-hop  don't differentiate too much, so you can actually use either.  We plan to add multi-threading support  through asynchronous function execution,  so you can execute multiple functions at the same time,  and eventually support for Apache will come in,  probably via FastCGI.  And hopefully from that point on,  we start to encourage people other outside of Facebook  to sort of use hip-hop.  We're sort of eager to create a community around that.  Yeah, and so we announced hip-hop a little bit earlier this week.  We've been working on continuing to release it even a few hours  before we got here, so definitely look for that  on Facebook's GitHub account as well.  So we want to take a look at our services layer a little bit  and give you a better idea of a few of the different backend services  that we both use and have created  to scale different aspects of Facebook.  And so from a philosophical perspective,  we sort of think about this question of when do we need a service?  So is it something that needs to be really quick?  Is it something that currently has a big overhead  in terms of our application layer around deployment and maintenance?  Do we want yet another failure point from our infrastructure,  sort of balancing some of those trade-offs?  And we try to go and take advantage of as many bits of common functionality  as we can between all these different services.  And that's another reason why hip-hop is really interesting to us  in terms of being able to go and reuse logic we've already written in PHP,  compiled into C++, and be able to pull that into our backend services,  which are written in a variety of languages.  And so this is another piece from a philosophy perspective,  is that we don't go and choose a single language  when we're going and building out Facebook infrastructure.  Our entire web server infrastructure is PHP,  but we use many different languages from a backend infrastructure perspective,  depending on what we're building, depending on who's involved,  depending on when, and some of the different characteristics  of these languages for what we're doing.  So to communicate between all the different services that we have,  we use this technology called Thrift.  And Thrift is something that we open-sourced.  And it's essentially an RPC server that will generate...  It's an RPC server, and you provide it with a damage-agnostic RPC server.  So you provide it with a file of your Thrift service,  and it's able to generate code for you in C++, PHP, Rang, Python, Java, Ruby.  So it generates all of these for you,  and it also deals with the communication between each of the services,  as well as the actual structure,  so the structure of the communication.  So it'll do the serializing and the unsealizing of the data  to pass it along the wire, so to speak.  And the wire protocols that it supports is like Socket,  the file transport, and then just standard in-memory buffers.  So you can write something in C++, serialize it with Thrift,  and then talk to PHP Thrift server, and then it'll unsealize it for you.  So it's the sort of how we pass data amongst our different services at Facebook.  And Thrift is actually an Apache incubator,  so anyone can go download it and use it,  and that's one of the things that we gave back to the open-source community.  I'm not exactly sure when to put it.  Well, and Thrift is also using a variety of different projects,  I think companies like ReCapture are making use of it,  as well as other projects.  So Thrift is used by some bit of Hadoop as well.  So one of the other services that we want to talk about is logging.  How many people here have used Syslog?  Yeah, we did two.  So I think this is one of those themes of,  we always start with what is looking at outside of Facebook,  what is either a really great open-source solution,  or what is a good commercial solution,  and then ultimately going and building on top of it.  And so we were using Syslog for logging all of our data,  and as you saw behind me, the logging server sort of exploded,  because we were going and logging so much information from all these page views.  At this point, I think we do about 400 billion page views per month on Facebook.  And we log about 25 terabytes a day.  So we went and created Scribe,  and the basic idea behind Scribe is that we're able to go and sort of break up this funnel  from a logging perspective.  So you'll have this information coming from our web servers,  or coming from other backend services.  That will get routed into Scribe servers,  which sort of work at funneling it down and condensing it.  And then we ultimately go and store this information  inside of our Hadoop and Hive cluster,  so we can go and look at it from an analysis perspective later on.  And so Scribe is one of those technologies where logging is a common problem.  Everyone goes and has to have logging inside of any site that they build.  And we got to the point where we just weren't able to work within the scale  of some of the other solutions that were there.  So we also open source Scribe.  It's also being used by Twitter right now,  and they're contributing back to the project.  And I think it's actually one of our projects that's sort of the least known about,  but compared to sort of what it does and how much of a common problem it is.  So one of the more popular things to do on Facebook is actually to upload photos.  And at the moment, there's 40 billion photos that have been actually uploaded to Facebook.  And we store that in four different resolutions for each of the areas.  So there's the thumbnail for the wall,  then there's the thumbnail within the actual gallery overview,  then there's the actual photo that you look at,  and there's a fourth one as well because original larger size is kept.  So we store four different photos, four different variants for every single photo.  And we have to, and we serve that.  That's 1.2 million photos or seconds are served.  So that's how many photos we're serving every second that people browse around the site,  because you've seen the Facebook homepage, it's pretty photo intensive.  Okay.  And one of the problems that are how you would normally try and scale photos,  on a site so large would be sort of like an NFS share.  So you set up a central NFS share and then multiple servers would then read and write to that NFS share.  And you would use HTTP just to serve the photos directly from NFS.  And the first thing we actually did was a commercial solution,  because you have to choose the battles that you're going to fight with  and what will take more time to do.  So we went for the easiest solution, which was to get a commercial,  vendors come in, get them to set up NFS and use their technology.  But unfortunately, it just didn't scale.  And the reason it didn't scale wasn't because the commercial technology was bad.  We just had so much IO, we were just trying to read so much and write so much that it just simply didn't work.  So we had to look for another solution.  And so then we started optimizing.  So I mean, we went and broke up in terms of where we're caching images.  So make sure that images that are more recent, that are smaller,  that we're being able to serve those out of a cache versus having to go back to disk every time,  starting to use content delivery networks to go and spread across this load,  even pull some of that caching outside of our own infrastructure,  as well as then did some actual caching around the NFS infrastructure as well,  trying to remove some more of that metadata.  But fundamentally, part of the challenge that we were running into  was just looking at how an operating system stores information on disk.  So in order to serve one of these files,  we were having to look up about three pieces of metadata for every file.  So you had to go and look at the directory I know, the data inside of that,  then the file I know, and then actually get at the data itself.  And so what we wanted to try to do was go and do this in one disk seek.  And so we developed a system called Haystack.  And what it does is it allows us, for any photo that we're going and looking up,  to go and serve it in terms of one physical read on the disk.  So it doesn't matter from a random data access perspective,  it's always a consistent time of one physical read in order to serve a file.  Overall, it takes about 300 megabytes of RAM to store this index  for every terabyte of photo information that we store.  And so you're able to sort of see how this compares to some of the other solutions.  We started at a system which was about 10 physical disk seeks to serve a photo.  We got down to three.  And then Haystack ultimately got us to one.  And I think one of the other really interesting parts of a technology like Haystack,  and this is true for a lot of our infrastructure,  is that it was built by three people.  So it's really one of those which is interesting for us in terms of being able to go  and have an engineering culture where small teams are really able to have a large impact,  are able to go and ship things.  And so Haystack is a technology which isn't currently open source,  but we're working on open sourcing it,  because it's one of those that we really think is useful to all sorts of different sites  from a variety of sizes, and not just for Facebook as well.  So let me talk another bit about one part of our infrastructure, and that's Hive,  which builds on top of Hadoop.  So I'll talk a little bit about Hadoop first.  This is some of the Hadoop.  Hadoop is map reduce.  So the first thing here, this example runs at the top is the map query,  which looks for any key that's greater than 100 and just outputs whatever that key was.  And then next example is the actual reducer,  which looks for the count of unique items that were in the output from the map script,  and then actually prints the value as the key and then the count.  Now the problem with Hadoop is it's just not simple to use.  It's very hard to understand, and we come up with another solution for that,  which would be Hive.  And Hive is very much an SQL syntax,  and what it does is it builds on top of...  Oh, you can't quite see that there.  The people here can't on the left.  Okay, you can stand up.  You can see that Hive is very much an SQL syntax,  so you can just select key, count from val,  we're key greater than 100, grouped by the key,  and in the background, that actually translates it into Hadoop query  and run it on your map reduce request and then run it on your behalf.  So Hive is actually in the Apache...  It's actually an Apache project now.  We open source that again, put it into the incubator,  and Apache picked it up,  and the support and development is actually happening outside of Facebook now.  We're still one of the main contributors to it.  So, yeah.  And the interesting thing about Hadoop is,  Hadoop actually came from Yahoo,  and that actually came from papers that Google had put out.  So the interesting thing here is that lots of different corporations  are building on top of each other's open source technology,  and that's something we're kind of big on.  We want to make sure we can open source the things  that we are finding useful within Facebook.  And I think we really love Hadoop in terms of what it's able to let us do,  being able, as Scott said, to go and build infrastructure on top of it.  But I think some of it, when you go back to that syntax bit,  is just looking at who's using it.  And so one of the things that we were really trying to do  was make Hadoop much more accessible inside of the company  so that more people were able to go and work with it  and sort of use Hive as a layer on top of the underlying Hadoop infrastructure as well.  And so this is sort of what our data flow architecture looks like.  This relates back to what we were talking about with Scribe a bit  in terms of we have our web servers going and sending data into Scribe,  going and figuring out where to file that, putting that into our Hive and Hadoop cluster,  replicating that, and allowing people to go and run jobs on top of this.  And so you also saw a picture in the previous slide which might look familiar.  And so this is what our Hive and Hadoop cluster looks like for that.  But, I mean, overall, we have a lot of information going into this cluster.  We couldn't have done it without Hadoop coming before us.  And I think that's really one of these important points from an open source perspective  is being able to go and build on top of software that others have released,  going and changing and being able to make it solve your use cases a little bit better,  going and releasing it again for others to continue to build on  to make better to innovate together.  And really with the idea with Hive was going and simplifying Hadoop  because we wanted people other than engineers to start working with it.  At this point, about 250 people throughout the company use Hive every month.  We're running about 7,500 jobs every single day on that.  And a lot of people who aren't engineers are actually doing this.  So we have data analysts all throughout the company in different parts,  which are going and taking advantage of Hive syntax as well as some other web tools on top of it  to really access this information, which Hadoop is going and crunching for us.  So I think this is one of those really great cases of you have a technology like Hadoop,  which is awesome. It's a great piece of infrastructure.  And then being able to go and use open source to continue to build another layer on top of that,  which makes it more accessible to more people.  So one of the other pieces of the infrastructure that we want to talk about is Memcache.  I'm guessing if I ask how many people here have used Memcache,  they will be about the same as Syslog if not, yep, exactly.  So like Memcache is awesome. We love Memcache.  It was one of those technologies where when you go back and start thinking about  how was Facebook built from the beginning?  I mean, it was built on top of Linux and Apache and MySQL and PHP.  And then if we could, we would add like one more M after that.  So it would be like the Lampum stack with Memcache in there.  In terms of it's one of those technologies that's really robust, it's scalable.  It lets you get a lot more out of your site if you're going and using it in a smart way.  So overall, I mean Memcache was developed by Brad Fitzpatrick at Danga,  originally for LiveJournal. It's being used by just about any dynamic site that you can think about.  And it's something that gives us a lot of performance benefit,  but it's also up to our engineers in terms of making sure that they use it  and that they use it in a smart manner.  So we'll talk about that a little bit more.  So this is a slide which I think was originally created to try to describe caches  to a non-technical audience.  And so it's sort of interesting from that perspective.  But I think it's a little bit more interesting.  We currently serve about 120 million Memcache requests every second.  So we're incredibly reliant on Memcache in order to make Facebook work.  And so if you go and think about that from how these different pieces of the stack come together,  which Scott will talk about in a little bit, Memcache is really a critical piece of it.  We've done some performance work around it as well.  One of the good examples of how we used Memcache and how we do use it  was when we launched usernames.  I think this was a little bit over a year ago.  I think there were about 200 million active users at the time on Facebook.  And we were trying to figure out what would be a fair way to go and give everyone a username.  And so we thought about a few different options.  And ultimately we decided that the most fair way to do this would be just whoever got there first.  So we went and asked 200 million people to go and access Facebook at exactly the same time.  So I think a lot of people probably are thinking of this as like a denial of service attack  that you bring upon yourself.  But for us it was really a product launch.  And so the team got together, they tweaked things, they tuned things.  And we served and gave away over 200,000 usernames in the first three minutes.  And a million usernames were assigned in that first hour.  And so really that graph before was going and showing our memcache hits  in terms of leading up to this release and then at that point as well.  So memcache is an incredibly important piece of technology for us  and something that has a huge impact in terms of how we go and scale.  Memcached itself is generally quite robust.  It's filled with really nice features.  But we wanted to make it better.  So a couple of things that we did with the memcache at Facebook is we made it a 64-bit port  because most of our memcache machines have about 16 gigs of RAM.  And before we used to have to actually run three, four memcache processes  so we could actually allocate all of the memory because it was only a 32-bit port.  So what we did is we ported to 64-bit and we open sourced that.  And that's now in the main memcache that you can get from most distributions.  We added a multi-threading so we could actually utilize all of the cores that were on the processors.  It's all the cores and processors that were in the machine.  So we could fetch data and get it out there quicker.  And one of the more interesting things was actually adding UDP support.  Now when you add UDP support, it means we actually have to do packet assembly  on the memcache client rather than relying on the network stack to do it.  And that was mainly because there was so much memory getting allocated to the buffers within the kernel  that it was using more memory on the machine than we'd like to.  So we changed it to UDP and then we adjusted the memcache client  so it could assemble UDP packets itself.  At the moment, we haven't got around.  We released about a year ago, maybe a bit more.  We released all the changes we had up until that point.  And some of them have made it back into the actual memcache server.  And sometime this year we're going to do another release of the memcache server.  And hopefully they'll get merged back into the upstream version again.  Yeah, and I mean memcache is one of those examples where it works really great  in terms of core memcache for most site that you go and build.  And there are actually some of these things that we've gone and done  which won't be as useful at a smaller scale.  And we're going and making some of those trade-offs between performance and features as well.  But overall, like memcache is, I think, another one of those examples of incredible open source technology  that came before Facebook that we were able to go and really take advantage of  as well as then contribute back to and evolve.  So next I'm going to look a little bit at the database part of Facebook.  And here at Facebook we use MySQL.  MySQL is our database of choice.  And mainly because it's the ShareNothing architecture.  We have lots of master-master databases that replicate themselves around.  And you can sort of think of this as like if one, an army,  and if one sort of soldier loses its head, like we have an example here at the top of the photo,  then it doesn't matter because it will keep going forward.  And at Facebook, it doesn't really matter if a single server disappears.  We have other servers that will back it up.  And that's really one of those choices that you make when you're thinking about how you scale  is do you go and invest in technologies where you sort of like grow one piece of your infrastructure  really large, or do you go and find a way where you're able to sort of break that up?  And I think that's really what we've done with our databases with MySQL  in terms of being able to go and have a lot of more independent database clusters  that were able to continue growing that entire pool  rather than having one really large database cluster by itself.  And overall, I mean, we really like MySQL.  We found it to be simple and fast and reliable.  And normally when we go and break our databases, it's because one of us did something stupid,  not because the software wasn't working right.  And so we really think about MySQL as like that freight train,  which is really solid, it keeps on running.  And a lot of that is also based on how we go and use our databases  in terms of we don't take advantage of a lot of the relational aspects,  we don't go and use joins inside of our database.  But in general, we found MySQL to be an incredibly reliable piece of storage infrastructure.  And definitely is a good example of what you can go and build from an open source perspective.  I see.  The database that we actually use, the N-O-D-B engine at Engine,  because it's the only one that's got full transactional support  and we can make sure everything makes its way to disk.  But this is overall, okay, so this is sort of the high level look  of how the architecture is at Facebook and we'll look at databases again.  So most people see databases as sort of a collection of secondary indices.  That's where you get all your relational features, that's where you do your joins.  But here, we more or less use it as a way to keep persistent data.  It's where we store the data that writes it to disk.  That's our persistence layer.  And if we want to do any sort of joins, then the joins are actually done within the web servers.  And that's why HIP-HOP is important because we do a lot of processing there  and reassembling the data from its different sources.  And then once we have that sort of put into Memcache.  And Memcache can be seen as sort of our secondary index.  That's where our data is stored.  So really it looks a little more like this.  So that's probably a more true representation of our stack  where the web server looks in our distributed index to see if the data is there.  And if that fails, it will look in their persistent storage.  But it's more than just the web servers that look within both Memcache and MySQL.  We also have other search services that themselves will look in the various data locations.  Yeah, and so hopefully we gave you sort of a pretty good overview  of a few of the different services and technologies that go into scaling Facebook.  Both in terms of open source software, which we use,  some of the things that we've built on top of that as well.  And so that was really our goal in terms of going and looking at a few of these different places  of our architecture and really how important open source software is  to go and scale Facebook since the beginning of the site.  I think one of the other things, and we talked about this a little bit with Haystack,  was that a lot of this actually starts as a hack project.  So it's not necessarily that we're going and set out trying to solve this specific problem  and have a lot of people thinking about it and how to do it,  but rather a few people will just sit down or a single person.  I mean Haystack was started by one engineer, Hayping,  sorry, that was Hip Hop. Haystack was started by Jason Sobel.  And it's really getting to the point where we prototype something.  And we're also from an infrastructure perspective, when we're trying to solve a problem,  we might go and try two or three different things at the same time,  prototype them out for a few months, and then throw two of them away  and move forward with the third one.  And I think that's something that's really important, and it's sort of a piece of our culture as well  from an engineering and company perspective,  in terms of really how these technologies help us go and do things like move fast.  And so that's a really important piece of Hip Hop,  in terms of it allows us from an engineering perspective to go and make changes  with the development speed of PHP,  being able to go and sort of save something and refresh that immediately on your development server,  but then going and having a lot of the operational performance of a compiled language like C++,  being able to go and develop technologies in a small team  and actually have them scale out to the entire site, release this open source,  and really that idea of continuing to be bold and innovating,  goes into our engineering culture as well.  So everything that we talked about, you can go and find at facebook.com.  We also have a variety of developer tools there,  so libraries such as 320, which are a lot of the UI aspects  and some of the data aspects behind our iPhone app.  So it lets you go and do things such as like have a list which you scroll through,  which is actually going and pulling images from a server,  instead of having all that information being stored locally.  As well as tools like Tornado, which is another pre-faceted infrastructure,  which powers FriendFeed's web server.  It's designed to be a really high speed web server,  because they have a connection open for every user that's actually on the site at the time.  As well as other development tools,  things like PHP shell for going and doing autocomplete and stuff like that.  So overall, we're really excited to be here  and have a little bit of time for questions,  as well as a bit of a pile of shwag down at the front.  We want to come and grab any. So thank you.  APPLAUSE  Yeah, my question is about Hey Hope.  Why are you translating PHP to C++ instead of directly developing C++?  So what was the last bit? Sorry.  Yeah, why aren't you not developing their website in C++,  instead of developing PHP then translating to C++,  optimizing and doing a lot of stuff?  OK, the question was why aren't we just developing in straight C++?  Why aren't we? OK.  So the current code base at Facebook is about four million lines of code.  So the first problem, four million lines of PHP codes.  So the first problem would be how do we translate all of that into C++,  as well as allowing people to keep developing with the site.  We can't exactly hold up development while that's being done.  And it comes back to how long does it take a C++ project to compile?  We like the fact that we can move fast, people can make little changes,  save it and then look onto the web server and the changes right there.  Yeah, so a lot of it is just that development speed of being able to,  like, more people know PHP, it's easier to work inside of PHP,  so we really want to keep it as a language from a development perspective.  Hi. Thanks for the presentation.  What eventual consistency do you tolerate?  So how much consistency do we tolerate?  So, I mean, we tried...  Oh.  Hmm.  I don't know the exact answer to that question.  I mean, so, like, we really try it for, I mean, anything that you post,  any comment that you comment on, like, as soon as it comes in,  we record it, we make sure that we go and serve it again.  I think with services like Newsfeed, you have the two views now.  So you have one which is going and showing that most recent view,  everything that's come in, but then you also have a view which goes  and sort of looks at what do we think is the most relevant thing  for you to see right now.  So I actually think it's probably a little bit less from a data storage  perspective and more from a presentation perspective in terms of  what specific feature on the side are you going and looking at  and how are we going and optimizing that at the time.  You guys were talking about, you chose to put the joins  and all the filters in the web server level.  I suspect you guys did a lot of benchmarking and checks at what point  that would be an advantage by doing that because for smaller size  it probably won't be.  Do you have figures for that? Can you distribute them?  Because those would be really interesting to see and at what point  it would finally be an advantage to do it.  Yeah, so in terms of, like, at what point do different  pieces of these technologies make sense?  Yeah, I don't think we necessarily have a lot of benchmarks  in terms of something like do you do joins in your database  or do you go and just pull data together and join it together  at your application level, at your application layer.  Some of that tolerance is also probably depending on how large  do you want to scale your database cluster and where are you wanting  to spend some of that CPU time.  Another question probably,  data is very crucial for Facebook.  So how do you prevent the loss of the data and probably you use  the mechanism like duplicate and et cetera.  And the second question here about the data,  did you have already the problems with it and how do you solve them  if you had?  So I think your question, your first question was about replication  and then what was it second?  The second was if you had some data loss in the past  and if you had, how did you solve it and to not have it in the future?  Yeah, I mean, I think from a data loss perspective,  a lot of those things that you learn about how do you go and run  database clusters apply the same for us as well.  In terms of going and making sure that you're having backups  that your replication is working, that you're keeping your bin logs  so that you can go and can replace some of that information.  So a lot of, I think the scaling lessons that you learn,  at least in terms of how you go and run databases,  are pretty similar, whether you're at Facebook scale  or smaller than that as well.  It's really a lot of those best practices applied at a much larger scale.  You've talked about scaling at technology level.  How do you scale internally with your developers  and you've got fairly high velocity releases?  How do you ensure all your developers on a consistent frame of reference  when they're developing new features?  Yeah, so really scaling is more than technology, as you said.  It's also about scaling in organizations, scaling in engineering team.  And so we've developed, I mean, we've developed a variety of internal tools  which also help us do that.  So we've gone and layered things on top of our version control repositories,  so have tools to go and do things like code reviews,  to be able to go and trace a code review all the way back  from that release that we're doing to when was the code committed,  who reviewed it, what was happening inside of that.  Every single commit at Facebook is reviewed by someone else.  And so it's definitely one of those challenges that we focus on as well  in terms of how do we continue growing and moving at this pace.  Hello.  Yeah, over here.  Excellent presentation, by the way. Love it.  Really interesting in Haystack.  So I guess the average size of a photo, how big is that?  A few hundred K or less?  Yeah, I mean, I think an average size is certainly under 500 K.  Yeah.  Generally, the larger size that we store is 640 by 480.  And what we actually do is our photo uploader,  so when it's occurring on your computer,  is going and resizing that photo before it's even sent over the network.  But my question is then, how will the, could this scale for, I don't know,  say, org-vorbis standard 4 megabyte file or even movies?  Or how do, how many discreeds per megabyte sort of?  So I think one of the things that, to keep in mind there,  is that percentage in terms of the amount of time you spend  going, reading metadata compared to the file size is really important.  And so when you have a smaller file, then the amount of time  that you spend going and looking for that metadata is actually much larger.  If you go and start to get into something like videos,  then it becomes a much smaller percentage of the time that you spend overall.  So we don't actually use Haystack to store videos,  but only for photos right now.  Yeah?  Yes?  Up here, up here.  Okay, I was wondering, can you give us some numbers on the breakdown  of how many servers are web servers, how many servers are memcache,  and how many servers are the database?  No, I actually can't.  But I can tell you we have tens of thousands of servers overall.  Just on there.  Sorry.  What language is hip hop written in  and what's your plans for growing a community around it?  What was the second the written in C++ and what was the second question?  What's your plans for growing a community around hip-hop?  What's our plans for creating a community around it?  At the moment we've open sourced it. We haven't released the code yet  That's gonna be happening soon a couple hours before this I was finishing the last of that with a CMake and  Making sure it could build on non Facebook platforms  We've created a mailing list and we've had a couple of thousand people joined the mailing list already  Is that right?  Yeah, there's a couple of thousand people joined the mailing list already and we're hoping as people start using it  The community will just naturally develop around it  You have to remember that hip-hop isn't designed for everyone to use  It's they can if they want but it doesn't really make sense for the small person running their own shared WordPress blog  It's more for those larger sites  So we're hoping that they'll realize performance benefits that come from it and they'll start adopting it and from there  Oh, just hopefully just snowball  Yeah, hi. Yeah, thanks for the talk. What about HTTP servers? Do you work with the patch engines or?  multiple  So we've generally been using Apache and PHP  But hip-hop actually has its own embedded web server, which is sort of really simple web server built on top of lib event  And so we're now been moving to using that  How hard it is to port a zend extension to hip-hop?  Hi  To change an extension to hip-hop you basically just write a C++ class as normal  And then it's got the similar parts of the PHP extensions initialization function  There's a request initialization request destroy and request sorry module unload and again  It's some more glue code with macros to expose your C++ class back into hip-hop  So it's similar to the PHP extensions, but less random macros they might not understand  Hi  Thank you for the presentation first of all and my question was  Where exactly do you use Erlang and do you make use of Indonesia and the concurrency that the Erlang provides?  So we use Erlang for our chat service  So for Facebook chat  It's your generated C++ code using a boost library and what libraries do you use in the generated C++ code?  Yeah, we missed that completely. Sorry. We couldn't understand we couldn't hear you  Is your generate C++ code using the boost library?  Yes, our generating C++ code uses the best libraries it uses file system system and  a couple of the mutexing and the concurrent execution the threading ones  If you have more questions come down and ask afterwards  Hello, thank you for the presentation, but when will we get a dislike button?  What when will we get what a dislike button?  Um, we let's talk about that later. Let me know if there's specific information you want off of them  Okay, I have another question about hip-hop  Is it currently possible to cross compile it for example to get a windows binary?  And if that's not possible would that be easy to implement that function?  At the moment it only compiles on Linux  Specifically last week only compiled on CentOS  We've now got it to the point that should compile on all Linux distributions  It uses CMake and CMake has support for cross compilation  At the moment there's no support on OS X or Windows, but we're hoping with something from Microsoft is contacted us and hopefully they'll contribute the Windows part and  Hopefully something the community will either contribute the Mac port or I myself will do  Hi, my understanding is that you only have data centers in the US, right?  No  What are you asking where servers are now? My question is how do you deal with the high latency from the US for users that are outside of the US?  So dealing with latency  So I mean in the US we have data centers on both the West Coast and the East Coast  We also use CDNs to help distribute some of the  like things like  images and  CSS and JavaScript around the world  I think that's been something that our operations team continues to work on as this site continues to grow internationally as well  So there's still five minutes left are there any more questions?  And we'll stick around as well if you want to come ask down here. We might be able to hear you better too  Yeah, hi, you mentioned that you use CDNs. Do you use other people's CDNs or do you have your own CDN deployed around the world?  So we use Akamai for a lot of our CDNs  Hi, I was wondering what kind of open-source software you guys use for  Monitoring capacity planning and trending and if you experienced issues scaling with those as well as the Facebook application  Yeah, we know a little bit less about the monitoring technologies that we use  I know that we use ganglia. I think we also use cacti  We also have some tools that we've developed internally in terms of managing our overall infrastructure  But I honestly know a little bit less about the scaling challenges that we've run into those  You talked that you're using revision control systems for  Fastly developing your site, but what kind of software are you using?  Are you using currently single revision control system or different teams with different revision control systems?  So I think you were asking about like what software our developers use  I mean  So yeah, I mean  We have a variety of like we use then for a lot of our development environments  Engineers then use I mean emacs or vi or whatever insert favorite text editor here  And then we'll have some other tools built over on top of subversion and get our main repositories or subversion  Over half of our engineering team now uses get on top of that  and then we have some tools built on top around that in terms of things like code review and  and  documentation and things like that  Hi there  Can you say a little bit about Cassandra and what using Oracle for please? Thank you. So Cassandra  Yeah, so we do a team at Facebook developed Cassandra a few years ago. It's also in the patchy incubator right now  we use it for inbox search on Facebook and  We haven't really been developing it actively in the past year  But at this point I think dig and Twitter and Rackspace are really the largest contributors to Cassandra  Inside of the patchy project and so I mean I think from our perspective  It's really great to continue to see it develop and have a community around it that's going and continuing to push it forward  Even though we're not actively developing it as well  Hello, how do you manage the deployment of software?  So software deployments. I mean we have a variety of tools built around that as well  In terms of being able to go and push out information to different hosts going and running jobs on those hosts  Distributing information and I mean I think that's something that we'll talk about a little bit more this summer  Hopefully at the velocity conference  Are there any more questions  Okay, I think not so thank you very much for your talk
Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_0_131.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_132_179.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_180_1283.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_1284_2147.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_2148_3083.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_3084_3407.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_3408_3767.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_3768_5087.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_5088_5135.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_5136_5567.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_5568_5651.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_5652_13403.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_13404_15143.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_15144_16751.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_16752_20867.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_20868_23795.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_23796_26231.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_26232_27455.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_27456_35759.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_35760_41147.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_41148_44783.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_44784_52355.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_52356_59147.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_59148_60959.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_60960_61187.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61188_61283.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61284_61619.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61620_61667.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61668_61751.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61752_61835.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61836_61883.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61884_61991.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_61992_62087.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62088_62363.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62364_62435.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62436_62483.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62484_62531.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62532_62687.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62688_62759.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62760_62843.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62844_62963.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_62964_63107.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_63108_63239.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_63240_63659.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_63660_63707.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_63708_63983.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_63984_64043.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_64044_64379.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_64380_64571.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_64572_64895.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_64896_68915.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_68916_69047.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_69048_69503.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_69504_69707.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_69708_69755.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_69756_69875.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_69876_70295.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_70296_70607.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_70608_70667.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_70668_71507.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_71508_72431.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_72432_72479.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_72480_74087.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_74088_74135.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_74136_75239.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_75240_75299.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_75300_75911.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_75912_76871.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_76872_76919.png Scaling_Facebook_with_OpenSource_tools_HAzeJTgFwDc_1_76920_77071.png
I'm David Rekordon. I spend my time at Facebook focused on open source and web standards. My name is Scott Mavicker and I work on anything open source that's PHP related within Facebook. So the recently released hip hop is my little baby at the moment. So first we should really talk about what do people do on Facebook so you can sort of get the idea of the scaling challenges that we sort of have there. For example, 8 billion minutes every day are spent on Facebook. And it's over 3.5 billion pieces of content is shared every week, be it photos, writing on the walls, talking. And that's 2.5 billion photos every month are added. It actually makes us bigger than the biggest photo sharing website out there. And it's more than just the actual website, there's also the API and the platform that people build with Facebook. There's a million users of that. And so that's sort of an idea of some of the amount of content that's being shared on Facebook, the amount of developers that are going and building with Facebook. But Facebook is also incredibly international. Over 70% of our traffic and our users come from outside of the United States. We've gone and built a translation system which is focused on going and engaging the user base on translating the site. I think we translated Facebook into French in just under 24 hours. And so you can sort of see how we've gone and spread around the world, which also presented a set of scaling challenges in terms of how do we go and serve all this information quickly with only having a small number of data centers. So when you think about scaling a traditional website, you're able to go and sort of break up that information. So if you have an email application, that user is generally interacting with their own email. They're not interacting with email from lots of different places. So you're able to go and shard your databases. You're able to go and break this apart by your users of saying, oh, Bob's interacting with data here, Felicia's interacting with data there, and sort of continue just to break that apart. As you have more users, you add more databases and you scale from that perspective. And so when Facebook started, Mark Zuckerberg created it in his Harvard dorm room and it was sort of pretty similar in terms of all of the users were on a Harvard server. And then as more colleges were added, you were able to go and shard your databases based on each college. So students at Harvard would interact with other students at Harvard at Stanford that would interact with other students at Stanford. But then in 2006, Facebook sort of opened this up so you could go and have friend relationships across these schools. And pretty soon you got to the point where it didn't matter whether you were on the Harvard server or you were on the Stanford server or you were on the University of Washington server, but you were going and interacting with people from college campuses all over the country. And so you might think, oh, well, you could just go and scale Facebook from a country perspective, put all of the users in the United States in one place, put all of the users in the United Kingdom in another, put another server for Belgium. But pretty quickly, Facebook got to the point where you couldn't even do that. So this is a visualization showing live friend requests around the world from a few years ago, really going and illustrating that point that on Facebook, everyone is connecting to people all over the world. It's not a site where you can go and scale broken up based on where your users are or just what information they're interacting with. And so this is really that idea of that to go and render any page on Facebook, we're pulling data from many different places. On average, a user has about 150 different friends. And so to go and render a page like that, we're going and talking to all sorts of different pieces of our infrastructure without being able to go and sort of separate it apart by user. So if we take like a standard page on Facebook, say the news feed, and on average, a person will have about 150 friends. So to construct that, we need to go grab data from 150 different friends, and that's split across multiple servers. So we have to do that in less than milliseconds it has to be done in. So we go do that, grab all the data. But it's not just your direct friends. If their friends have commented on their photos or the comments or any of their status updates, that has to be pulled as well. So we're pulling from tens of thousands of different sources, all just to render that one single page. And it's also that every single page that we go and render is different, not just per person, but at what time they saw it. But the graph has also been evolving. I mean, Facebook has always been based on this idea that you're connected to other people and to other things, that you have relationships, and so from a math perspective, we think about it as a graph. So we have nodes, which are the people or the things, and then we have edges, which represent the relationships between them. But we've also seen Facebook grow to no longer just be relationships between people, but you might become a fan of a company or a person. And it's a very different scaling challenge of going and connecting a person to a few hundred other people, versus going and serving Michael Jackson's fan page, which has over 10 million people connected to it. So that's also presented some different scaling challenges as the sites continue to grow. So this is Facebook's architecture at a high level. And it more or less looks like this. There's a load balancer on top and requests come in there, and the requests are spread amongst a pool of web servers. And then these web servers will either use one of our services to fetch data, or it will fall into Memcached, which is our in-memory fast database access, before eventually falling back to the database. And I mean, this looks like pretty much any website that you're going and building today. In terms of you have your web servers, you have your PHP stack, you have your services. So at a high level, Facebook isn't architected very differently than any other site that you would go and build. So first, we're going to look at the web server stack, or essentially what PHP is built on. And that's Facebook. What Facebook is built on, which is PHP. And we like PHP for a number of reasons. The first being, it's simple to learn. There's not very much barrier to entry to learn PHP, so if we hire new developers, then we can teach them PHP relatively quickly because the syntax is very similar to that of C. And it's an interpreted language, so if you want to make changes to it, you make your changes, you deploy them or save them, and you can see them live. You're not compiling, you're not waiting five, 10, 15 minutes for things to compile, so you can see it. And one of the nice things about PHP is the Hello World example. The Hello World example in PHP doesn't actually require any opening tags or closing tags because PHP itself is a templating language. So this is the sort of the nice things that make PHP good for Facebook. Because we have to develop fast and sort of, we move fast, so we can't spend time waiting for other languages. So there's a couple of things that are PHP problematic for Facebook. The first being high CPU usage. Because we do a lot of data assembly, pulling from multiple sources and actually building it within the application server, then we actually spend a lot of time doing that with CPU. We use a lot of CPU for that, as well as memory usage. Memory usage is quite high because of the way PHP works. It's sort of loosely typed, which is one of its benefits. It also uses a lot of memory for each variable because it has to represent a collection of six types. So every variable gets 140 bytes of memory allocated for it. We also potentially like to use logic between multiple services. We have things written in C++ and Python and Erlang, as well as PHP. So if it was just written in solid in PHP, we're going to have to duplicate that logic across multiple services. And finally, if you want to sort of move this into PHP extension to get the speed benefit, then it's sort of hard to write because the macros aren't well documented for the Zend engine. So to solve these issues, especially the first two, we come up with this thing called HipHop. And HipHop for PHP was this thing that we sort of announced last Tuesday and tend to open source quite soon. And what it does is it you point HipHop at your application framework, and what it will do is it will translate your PHP into highly optimized C++ code. And then from that, we can just pass it into the normal G++ compiler and it will just self-contained binary with the web server and your application and all the library logic built in. And HipHop is really designed to give us sort of the best of both worlds in terms of the speed that you can go and move with an interpreted language so that we can go and make changes really quickly from a development perspective, but then also the speed from a performance perspective of a language like C++. And so one of the ways that we think about the different types of PHP code you'll find is sort of this breakdown between the common stuff of like if else or the magic in terms of going and doing like evaluation and dynamic variable and function assignment. And so when you look at a PHP code base, you actually generally find much more of the more mundane things. It's very rare, at least inside of our code base, that we use things like a val or some of those other functionality. And so this is some of what makes HipHop work really well, is that we can go and take the majority of our code base and greatly speed it up in terms of going and looking at stack function calls and stack variables and then taking advantage of a lot of the optimizations that already exist inside of G++. The stuff that's magic is a little bit harder. It's not going to give us as much of a performance optimization. It won't give you as much of a performance optimization. But there's definitely some benefit there as well. So roughly how HipHop works is it performs a static analysis on your entire code base, looking to see what function, what class is declared where and what variables are declared. And from that, I think it's in our type inference engine, which looks at the data types for each of the variables. So if something was an integer and it was always assigned as an integer and it was always acted like it was an integer, then at that point we can just say, this is always going to be an integer, so let's generate just a straight integer rather than our variant type. So we require support. This is for all the variables. So integers map to actual integers in C++, strings map to strings, arrays map to the STL, like the standard array types, and classes map to actual C++ classes. And it's only when we have a bit of the magic stuff, or when types are into a type of variables in integer, then it becomes a string then backed in integer again, that we'll use the variant type, and then eventually go to the code generation. And the code generation is what outputs actual C++ files. And the transformation process for this looks a little like this. So it goes through a parser that's very similar to the standard PHP parser, and then a former static analysis on it, and then a pre-optimizer. And the pre-optimizer tries to remove things like dead code branches that won't be used or change some of the semantics when it knows that the execution could produce the exact same code regardless of the order. So, say you're doing a post-optimization, it's actually easier to do a pre-increment, because then you don't have to use this third variable to store the result. So it does like little optimizations like that, forks into the type inference engine, and eventually there's some more post-optimization to remove any more dead codes that you could find. And it's just code generation and then compiling is normal. So we have some plans for hip-hop over the next year, and at the moment it only supports PHP 5.2. So we intend to get that back up to speed with the latest 5.2 release before eventually adding some of the new features from PHP 5.3. And there are a few things that we're thinking about that aren't currently planned for PHP, but we want to sort of make sure it gets in there as well as in hip-hop. So one of the things we're sort of dedicated to is making sure that the features in PHP and the features in hip-hop don't differentiate too much, so you can actually use either. We plan to add multi-threading support through asynchronous function execution, so you can execute multiple functions at the same time, and eventually support for Apache will come in, probably via FastCGI. And hopefully from that point on, we start to encourage people other outside of Facebook to sort of use hip-hop. We're sort of eager to create a community around that. Yeah, and so we announced hip-hop a little bit earlier this week. We've been working on continuing to release it even a few hours before we got here, so definitely look for that on Facebook's GitHub account as well. So we want to take a look at our services layer a little bit and give you a better idea of a few of the different backend services that we both use and have created to scale different aspects of Facebook. And so from a philosophical perspective, we sort of think about this question of when do we need a service? So is it something that needs to be really quick? Is it something that currently has a big overhead in terms of our application layer around deployment and maintenance? Do we want yet another failure point from our infrastructure, sort of balancing some of those trade-offs? And we try to go and take advantage of as many bits of common functionality as we can between all these different services. And that's another reason why hip-hop is really interesting to us in terms of being able to go and reuse logic we've already written in PHP, compiled into C++, and be able to pull that into our backend services, which are written in a variety of languages. And so this is another piece from a philosophy perspective, is that we don't go and choose a single language when we're going and building out Facebook infrastructure. Our entire web server infrastructure is PHP, but we use many different languages from a backend infrastructure perspective, depending on what we're building, depending on who's involved, depending on when, and some of the different characteristics of these languages for what we're doing. So to communicate between all the different services that we have, we use this technology called Thrift. And Thrift is something that we open-sourced. And it's essentially an RPC server that will generate... It's an RPC server, and you provide it with a damage-agnostic RPC server. So you provide it with a file of your Thrift service, and it's able to generate code for you in C++, PHP, Rang, Python, Java, Ruby. So it generates all of these for you, and it also deals with the communication between each of the services, as well as the actual structure, so the structure of the communication. So it'll do the serializing and the unsealizing of the data to pass it along the wire, so to speak. And the wire protocols that it supports is like Socket, the file transport, and then just standard in-memory buffers. So you can write something in C++, serialize it with Thrift, and then talk to PHP Thrift server, and then it'll unsealize it for you. So it's the sort of how we pass data amongst our different services at Facebook. And Thrift is actually an Apache incubator, so anyone can go download it and use it, and that's one of the things that we gave back to the open-source community. I'm not exactly sure when to put it. Well, and Thrift is also using a variety of different projects, I think companies like ReCapture are making use of it, as well as other projects. So Thrift is used by some bit of Hadoop as well. So one of the other services that we want to talk about is logging. How many people here have used Syslog? Yeah, we did two. So I think this is one of those themes of, we always start with what is looking at outside of Facebook, what is either a really great open-source solution, or what is a good commercial solution, and then ultimately going and building on top of it. And so we were using Syslog for logging all of our data, and as you saw behind me, the logging server sort of exploded, because we were going and logging so much information from all these page views. At this point, I think we do about 400 billion page views per month on Facebook. And we log about 25 terabytes a day. So we went and created Scribe, and the basic idea behind Scribe is that we're able to go and sort of break up this funnel from a logging perspective. So you'll have this information coming from our web servers, or coming from other backend services. That will get routed into Scribe servers, which sort of work at funneling it down and condensing it. And then we ultimately go and store this information inside of our Hadoop and Hive cluster, so we can go and look at it from an analysis perspective later on. And so Scribe is one of those technologies where logging is a common problem. Everyone goes and has to have logging inside of any site that they build. And we got to the point where we just weren't able to work within the scale of some of the other solutions that were there. So we also open source Scribe. It's also being used by Twitter right now, and they're contributing back to the project. And I think it's actually one of our projects that's sort of the least known about, but compared to sort of what it does and how much of a common problem it is. So one of the more popular things to do on Facebook is actually to upload photos. And at the moment, there's 40 billion photos that have been actually uploaded to Facebook. And we store that in four different resolutions for each of the areas. So there's the thumbnail for the wall, then there's the thumbnail within the actual gallery overview, then there's the actual photo that you look at, and there's a fourth one as well because original larger size is kept. So we store four different photos, four different variants for every single photo. And we have to, and we serve that. That's 1.2 million photos or seconds are served. So that's how many photos we're serving every second that people browse around the site, because you've seen the Facebook homepage, it's pretty photo intensive. Okay. And one of the problems that are how you would normally try and scale photos, on a site so large would be sort of like an NFS share. So you set up a central NFS share and then multiple servers would then read and write to that NFS share. And you would use HTTP just to serve the photos directly from NFS. And the first thing we actually did was a commercial solution, because you have to choose the battles that you're going to fight with and what will take more time to do. So we went for the easiest solution, which was to get a commercial, vendors come in, get them to set up NFS and use their technology. But unfortunately, it just didn't scale. And the reason it didn't scale wasn't because the commercial technology was bad. We just had so much IO, we were just trying to read so much and write so much that it just simply didn't work. So we had to look for another solution. And so then we started optimizing. So I mean, we went and broke up in terms of where we're caching images. So make sure that images that are more recent, that are smaller, that we're being able to serve those out of a cache versus having to go back to disk every time, starting to use content delivery networks to go and spread across this load, even pull some of that caching outside of our own infrastructure, as well as then did some actual caching around the NFS infrastructure as well, trying to remove some more of that metadata. But fundamentally, part of the challenge that we were running into was just looking at how an operating system stores information on disk. So in order to serve one of these files, we were having to look up about three pieces of metadata for every file. So you had to go and look at the directory I know, the data inside of that, then the file I know, and then actually get at the data itself. And so what we wanted to try to do was go and do this in one disk seek. And so we developed a system called Haystack. And what it does is it allows us, for any photo that we're going and looking up, to go and serve it in terms of one physical read on the disk. So it doesn't matter from a random data access perspective, it's always a consistent time of one physical read in order to serve a file. Overall, it takes about 300 megabytes of RAM to store this index for every terabyte of photo information that we store. And so you're able to sort of see how this compares to some of the other solutions. We started at a system which was about 10 physical disk seeks to serve a photo. We got down to three. And then Haystack ultimately got us to one. And I think one of the other really interesting parts of a technology like Haystack, and this is true for a lot of our infrastructure, is that it was built by three people. So it's really one of those which is interesting for us in terms of being able to go and have an engineering culture where small teams are really able to have a large impact, are able to go and ship things. And so Haystack is a technology which isn't currently open source, but we're working on open sourcing it, because it's one of those that we really think is useful to all sorts of different sites from a variety of sizes, and not just for Facebook as well. So let me talk another bit about one part of our infrastructure, and that's Hive, which builds on top of Hadoop. So I'll talk a little bit about Hadoop first. This is some of the Hadoop. Hadoop is map reduce. So the first thing here, this example runs at the top is the map query, which looks for any key that's greater than 100 and just outputs whatever that key was. And then next example is the actual reducer, which looks for the count of unique items that were in the output from the map script, and then actually prints the value as the key and then the count. Now the problem with Hadoop is it's just not simple to use. It's very hard to understand, and we come up with another solution for that, which would be Hive. And Hive is very much an SQL syntax, and what it does is it builds on top of... Oh, you can't quite see that there. The people here can't on the left. Okay, you can stand up. You can see that Hive is very much an SQL syntax, so you can just select key, count from val, we're key greater than 100, grouped by the key, and in the background, that actually translates it into Hadoop query and run it on your map reduce request and then run it on your behalf. So Hive is actually in the Apache... It's actually an Apache project now. We open source that again, put it into the incubator, and Apache picked it up, and the support and development is actually happening outside of Facebook now. We're still one of the main contributors to it. So, yeah. And the interesting thing about Hadoop is, Hadoop actually came from Yahoo, and that actually came from papers that Google had put out. So the interesting thing here is that lots of different corporations are building on top of each other's open source technology, and that's something we're kind of big on. We want to make sure we can open source the things that we are finding useful within Facebook. And I think we really love Hadoop in terms of what it's able to let us do, being able, as Scott said, to go and build infrastructure on top of it. But I think some of it, when you go back to that syntax bit, is just looking at who's using it. And so one of the things that we were really trying to do was make Hadoop much more accessible inside of the company so that more people were able to go and work with it and sort of use Hive as a layer on top of the underlying Hadoop infrastructure as well. And so this is sort of what our data flow architecture looks like. This relates back to what we were talking about with Scribe a bit in terms of we have our web servers going and sending data into Scribe, going and figuring out where to file that, putting that into our Hive and Hadoop cluster, replicating that, and allowing people to go and run jobs on top of this. And so you also saw a picture in the previous slide which might look familiar. And so this is what our Hive and Hadoop cluster looks like for that. But, I mean, overall, we have a lot of information going into this cluster. We couldn't have done it without Hadoop coming before us. And I think that's really one of these important points from an open source perspective is being able to go and build on top of software that others have released, going and changing and being able to make it solve your use cases a little bit better, going and releasing it again for others to continue to build on to make better to innovate together. And really with the idea with Hive was going and simplifying Hadoop because we wanted people other than engineers to start working with it. At this point, about 250 people throughout the company use Hive every month. We're running about 7,500 jobs every single day on that. And a lot of people who aren't engineers are actually doing this. So we have data analysts all throughout the company in different parts, which are going and taking advantage of Hive syntax as well as some other web tools on top of it to really access this information, which Hadoop is going and crunching for us. So I think this is one of those really great cases of you have a technology like Hadoop, which is awesome. It's a great piece of infrastructure. And then being able to go and use open source to continue to build another layer on top of that, which makes it more accessible to more people. So one of the other pieces of the infrastructure that we want to talk about is Memcache. I'm guessing if I ask how many people here have used Memcache, they will be about the same as Syslog if not, yep, exactly. So like Memcache is awesome. We love Memcache. It was one of those technologies where when you go back and start thinking about how was Facebook built from the beginning? I mean, it was built on top of Linux and Apache and MySQL and PHP. And then if we could, we would add like one more M after that. So it would be like the Lampum stack with Memcache in there. In terms of it's one of those technologies that's really robust, it's scalable. It lets you get a lot more out of your site if you're going and using it in a smart way. So overall, I mean Memcache was developed by Brad Fitzpatrick at Danga, originally for LiveJournal. It's being used by just about any dynamic site that you can think about. And it's something that gives us a lot of performance benefit, but it's also up to our engineers in terms of making sure that they use it and that they use it in a smart manner. So we'll talk about that a little bit more. So this is a slide which I think was originally created to try to describe caches to a non-technical audience. And so it's sort of interesting from that perspective. But I think it's a little bit more interesting. We currently serve about 120 million Memcache requests every second. So we're incredibly reliant on Memcache in order to make Facebook work. And so if you go and think about that from how these different pieces of the stack come together, which Scott will talk about in a little bit, Memcache is really a critical piece of it. We've done some performance work around it as well. One of the good examples of how we used Memcache and how we do use it was when we launched usernames. I think this was a little bit over a year ago. I think there were about 200 million active users at the time on Facebook. And we were trying to figure out what would be a fair way to go and give everyone a username. And so we thought about a few different options. And ultimately we decided that the most fair way to do this would be just whoever got there first. So we went and asked 200 million people to go and access Facebook at exactly the same time. So I think a lot of people probably are thinking of this as like a denial of service attack that you bring upon yourself. But for us it was really a product launch. And so the team got together, they tweaked things, they tuned things. And we served and gave away over 200,000 usernames in the first three minutes. And a million usernames were assigned in that first hour. And so really that graph before was going and showing our memcache hits in terms of leading up to this release and then at that point as well. So memcache is an incredibly important piece of technology for us and something that has a huge impact in terms of how we go and scale. Memcached itself is generally quite robust. It's filled with really nice features. But we wanted to make it better. So a couple of things that we did with the memcache at Facebook is we made it a 64-bit port because most of our memcache machines have about 16 gigs of RAM. And before we used to have to actually run three, four memcache processes so we could actually allocate all of the memory because it was only a 32-bit port. So what we did is we ported to 64-bit and we open sourced that. And that's now in the main memcache that you can get from most distributions. We added a multi-threading so we could actually utilize all of the cores that were on the processors. It's all the cores and processors that were in the machine. So we could fetch data and get it out there quicker. And one of the more interesting things was actually adding UDP support. Now when you add UDP support, it means we actually have to do packet assembly on the memcache client rather than relying on the network stack to do it. And that was mainly because there was so much memory getting allocated to the buffers within the kernel that it was using more memory on the machine than we'd like to. So we changed it to UDP and then we adjusted the memcache client so it could assemble UDP packets itself. At the moment, we haven't got around. We released about a year ago, maybe a bit more. We released all the changes we had up until that point. And some of them have made it back into the actual memcache server. And sometime this year we're going to do another release of the memcache server. And hopefully they'll get merged back into the upstream version again. Yeah, and I mean memcache is one of those examples where it works really great in terms of core memcache for most site that you go and build. And there are actually some of these things that we've gone and done which won't be as useful at a smaller scale. And we're going and making some of those trade-offs between performance and features as well. But overall, like memcache is, I think, another one of those examples of incredible open source technology that came before Facebook that we were able to go and really take advantage of as well as then contribute back to and evolve. So next I'm going to look a little bit at the database part of Facebook. And here at Facebook we use MySQL. MySQL is our database of choice. And mainly because it's the ShareNothing architecture. We have lots of master-master databases that replicate themselves around. And you can sort of think of this as like if one, an army, and if one sort of soldier loses its head, like we have an example here at the top of the photo, then it doesn't matter because it will keep going forward. And at Facebook, it doesn't really matter if a single server disappears. We have other servers that will back it up. And that's really one of those choices that you make when you're thinking about how you scale is do you go and invest in technologies where you sort of like grow one piece of your infrastructure really large, or do you go and find a way where you're able to sort of break that up? And I think that's really what we've done with our databases with MySQL in terms of being able to go and have a lot of more independent database clusters that were able to continue growing that entire pool rather than having one really large database cluster by itself. And overall, I mean, we really like MySQL. We found it to be simple and fast and reliable. And normally when we go and break our databases, it's because one of us did something stupid, not because the software wasn't working right. And so we really think about MySQL as like that freight train, which is really solid, it keeps on running. And a lot of that is also based on how we go and use our databases in terms of we don't take advantage of a lot of the relational aspects, we don't go and use joins inside of our database. But in general, we found MySQL to be an incredibly reliable piece of storage infrastructure. And definitely is a good example of what you can go and build from an open source perspective. I see. The database that we actually use, the N-O-D-B engine at Engine, because it's the only one that's got full transactional support and we can make sure everything makes its way to disk. But this is overall, okay, so this is sort of the high level look of how the architecture is at Facebook and we'll look at databases again. So most people see databases as sort of a collection of secondary indices. That's where you get all your relational features, that's where you do your joins. But here, we more or less use it as a way to keep persistent data. It's where we store the data that writes it to disk. That's our persistence layer. And if we want to do any sort of joins, then the joins are actually done within the web servers. And that's why HIP-HOP is important because we do a lot of processing there and reassembling the data from its different sources. And then once we have that sort of put into Memcache. And Memcache can be seen as sort of our secondary index. That's where our data is stored. So really it looks a little more like this. So that's probably a more true representation of our stack where the web server looks in our distributed index to see if the data is there. And if that fails, it will look in their persistent storage. But it's more than just the web servers that look within both Memcache and MySQL. We also have other search services that themselves will look in the various data locations. Yeah, and so hopefully we gave you sort of a pretty good overview of a few of the different services and technologies that go into scaling Facebook. Both in terms of open source software, which we use, some of the things that we've built on top of that as well. And so that was really our goal in terms of going and looking at a few of these different places of our architecture and really how important open source software is to go and scale Facebook since the beginning of the site. I think one of the other things, and we talked about this a little bit with Haystack, was that a lot of this actually starts as a hack project. So it's not necessarily that we're going and set out trying to solve this specific problem and have a lot of people thinking about it and how to do it, but rather a few people will just sit down or a single person. I mean Haystack was started by one engineer, Hayping, sorry, that was Hip Hop. Haystack was started by Jason Sobel. And it's really getting to the point where we prototype something. And we're also from an infrastructure perspective, when we're trying to solve a problem, we might go and try two or three different things at the same time, prototype them out for a few months, and then throw two of them away and move forward with the third one. And I think that's something that's really important, and it's sort of a piece of our culture as well from an engineering and company perspective, in terms of really how these technologies help us go and do things like move fast. And so that's a really important piece of Hip Hop, in terms of it allows us from an engineering perspective to go and make changes with the development speed of PHP, being able to go and sort of save something and refresh that immediately on your development server, but then going and having a lot of the operational performance of a compiled language like C++, being able to go and develop technologies in a small team and actually have them scale out to the entire site, release this open source, and really that idea of continuing to be bold and innovating, goes into our engineering culture as well. So everything that we talked about, you can go and find at facebook.com. We also have a variety of developer tools there, so libraries such as 320, which are a lot of the UI aspects and some of the data aspects behind our iPhone app. So it lets you go and do things such as like have a list which you scroll through, which is actually going and pulling images from a server, instead of having all that information being stored locally. As well as tools like Tornado, which is another pre-faceted infrastructure, which powers FriendFeed's web server. It's designed to be a really high speed web server, because they have a connection open for every user that's actually on the site at the time. As well as other development tools, things like PHP shell for going and doing autocomplete and stuff like that. So overall, we're really excited to be here and have a little bit of time for questions, as well as a bit of a pile of shwag down at the front. We want to come and grab any. So thank you. APPLAUSE Yeah, my question is about Hey Hope. Why are you translating PHP to C++ instead of directly developing C++? So what was the last bit? Sorry. Yeah, why aren't you not developing their website in C++, instead of developing PHP then translating to C++, optimizing and doing a lot of stuff? OK, the question was why aren't we just developing in straight C++? Why aren't we? OK. So the current code base at Facebook is about four million lines of code. So the first problem, four million lines of PHP codes. So the first problem would be how do we translate all of that into C++, as well as allowing people to keep developing with the site. We can't exactly hold up development while that's being done. And it comes back to how long does it take a C++ project to compile? We like the fact that we can move fast, people can make little changes, save it and then look onto the web server and the changes right there. Yeah, so a lot of it is just that development speed of being able to, like, more people know PHP, it's easier to work inside of PHP, so we really want to keep it as a language from a development perspective. Hi. Thanks for the presentation. What eventual consistency do you tolerate? So how much consistency do we tolerate? So, I mean, we tried... Oh. Hmm. I don't know the exact answer to that question. I mean, so, like, we really try it for, I mean, anything that you post, any comment that you comment on, like, as soon as it comes in, we record it, we make sure that we go and serve it again. I think with services like Newsfeed, you have the two views now. So you have one which is going and showing that most recent view, everything that's come in, but then you also have a view which goes and sort of looks at what do we think is the most relevant thing for you to see right now. So I actually think it's probably a little bit less from a data storage perspective and more from a presentation perspective in terms of what specific feature on the side are you going and looking at and how are we going and optimizing that at the time. You guys were talking about, you chose to put the joins and all the filters in the web server level. I suspect you guys did a lot of benchmarking and checks at what point that would be an advantage by doing that because for smaller size it probably won't be. Do you have figures for that? Can you distribute them? Because those would be really interesting to see and at what point it would finally be an advantage to do it. Yeah, so in terms of, like, at what point do different pieces of these technologies make sense? Yeah, I don't think we necessarily have a lot of benchmarks in terms of something like do you do joins in your database or do you go and just pull data together and join it together at your application level, at your application layer. Some of that tolerance is also probably depending on how large do you want to scale your database cluster and where are you wanting to spend some of that CPU time. Another question probably, data is very crucial for Facebook. So how do you prevent the loss of the data and probably you use the mechanism like duplicate and et cetera. And the second question here about the data, did you have already the problems with it and how do you solve them if you had? So I think your question, your first question was about replication and then what was it second? The second was if you had some data loss in the past and if you had, how did you solve it and to not have it in the future? Yeah, I mean, I think from a data loss perspective, a lot of those things that you learn about how do you go and run database clusters apply the same for us as well. In terms of going and making sure that you're having backups that your replication is working, that you're keeping your bin logs so that you can go and can replace some of that information. So a lot of, I think the scaling lessons that you learn, at least in terms of how you go and run databases, are pretty similar, whether you're at Facebook scale or smaller than that as well. It's really a lot of those best practices applied at a much larger scale. You've talked about scaling at technology level. How do you scale internally with your developers and you've got fairly high velocity releases? How do you ensure all your developers on a consistent frame of reference when they're developing new features? Yeah, so really scaling is more than technology, as you said. It's also about scaling in organizations, scaling in engineering team. And so we've developed, I mean, we've developed a variety of internal tools which also help us do that. So we've gone and layered things on top of our version control repositories, so have tools to go and do things like code reviews, to be able to go and trace a code review all the way back from that release that we're doing to when was the code committed, who reviewed it, what was happening inside of that. Every single commit at Facebook is reviewed by someone else. And so it's definitely one of those challenges that we focus on as well in terms of how do we continue growing and moving at this pace. Hello. Yeah, over here. Excellent presentation, by the way. Love it. Really interesting in Haystack. So I guess the average size of a photo, how big is that? A few hundred K or less? Yeah, I mean, I think an average size is certainly under 500 K. Yeah. Generally, the larger size that we store is 640 by 480. And what we actually do is our photo uploader, so when it's occurring on your computer, is going and resizing that photo before it's even sent over the network. But my question is then, how will the, could this scale for, I don't know, say, org-vorbis standard 4 megabyte file or even movies? Or how do, how many discreeds per megabyte sort of? So I think one of the things that, to keep in mind there, is that percentage in terms of the amount of time you spend going, reading metadata compared to the file size is really important. And so when you have a smaller file, then the amount of time that you spend going and looking for that metadata is actually much larger. If you go and start to get into something like videos, then it becomes a much smaller percentage of the time that you spend overall. So we don't actually use Haystack to store videos, but only for photos right now. Yeah? Yes? Up here, up here. Okay, I was wondering, can you give us some numbers on the breakdown of how many servers are web servers, how many servers are memcache, and how many servers are the database? No, I actually can't. But I can tell you we have tens of thousands of servers overall. Just on there. Sorry. What language is hip hop written in and what's your plans for growing a community around it? What was the second the written in C++ and what was the second question? What's your plans for growing a community around hip-hop? What's our plans for creating a community around it? At the moment we've open sourced it. We haven't released the code yet That's gonna be happening soon a couple hours before this I was finishing the last of that with a CMake and Making sure it could build on non Facebook platforms We've created a mailing list and we've had a couple of thousand people joined the mailing list already Is that right? Yeah, there's a couple of thousand people joined the mailing list already and we're hoping as people start using it The community will just naturally develop around it You have to remember that hip-hop isn't designed for everyone to use It's they can if they want but it doesn't really make sense for the small person running their own shared WordPress blog It's more for those larger sites So we're hoping that they'll realize performance benefits that come from it and they'll start adopting it and from there Oh, just hopefully just snowball Yeah, hi. Yeah, thanks for the talk. What about HTTP servers? Do you work with the patch engines or? multiple So we've generally been using Apache and PHP But hip-hop actually has its own embedded web server, which is sort of really simple web server built on top of lib event And so we're now been moving to using that How hard it is to port a zend extension to hip-hop? Hi To change an extension to hip-hop you basically just write a C++ class as normal And then it's got the similar parts of the PHP extensions initialization function There's a request initialization request destroy and request sorry module unload and again It's some more glue code with macros to expose your C++ class back into hip-hop So it's similar to the PHP extensions, but less random macros they might not understand Hi Thank you for the presentation first of all and my question was Where exactly do you use Erlang and do you make use of Indonesia and the concurrency that the Erlang provides? So we use Erlang for our chat service So for Facebook chat It's your generated C++ code using a boost library and what libraries do you use in the generated C++ code? Yeah, we missed that completely. Sorry. We couldn't understand we couldn't hear you Is your generate C++ code using the boost library? Yes, our generating C++ code uses the best libraries it uses file system system and a couple of the mutexing and the concurrent execution the threading ones If you have more questions come down and ask afterwards Hello, thank you for the presentation, but when will we get a dislike button? What when will we get what a dislike button? Um, we let's talk about that later. Let me know if there's specific information you want off of them Okay, I have another question about hip-hop Is it currently possible to cross compile it for example to get a windows binary? And if that's not possible would that be easy to implement that function? At the moment it only compiles on Linux Specifically last week only compiled on CentOS We've now got it to the point that should compile on all Linux distributions It uses CMake and CMake has support for cross compilation At the moment there's no support on OS X or Windows, but we're hoping with something from Microsoft is contacted us and hopefully they'll contribute the Windows part and Hopefully something the community will either contribute the Mac port or I myself will do Hi, my understanding is that you only have data centers in the US, right? No What are you asking where servers are now? My question is how do you deal with the high latency from the US for users that are outside of the US? So dealing with latency So I mean in the US we have data centers on both the West Coast and the East Coast We also use CDNs to help distribute some of the like things like images and CSS and JavaScript around the world I think that's been something that our operations team continues to work on as this site continues to grow internationally as well So there's still five minutes left are there any more questions? And we'll stick around as well if you want to come ask down here. We might be able to hear you better too Yeah, hi, you mentioned that you use CDNs. Do you use other people's CDNs or do you have your own CDN deployed around the world? So we use Akamai for a lot of our CDNs Hi, I was wondering what kind of open-source software you guys use for Monitoring capacity planning and trending and if you experienced issues scaling with those as well as the Facebook application Yeah, we know a little bit less about the monitoring technologies that we use I know that we use ganglia. I think we also use cacti We also have some tools that we've developed internally in terms of managing our overall infrastructure But I honestly know a little bit less about the scaling challenges that we've run into those You talked that you're using revision control systems for Fastly developing your site, but what kind of software are you using? Are you using currently single revision control system or different teams with different revision control systems? So I think you were asking about like what software our developers use I mean So yeah, I mean We have a variety of like we use then for a lot of our development environments Engineers then use I mean emacs or vi or whatever insert favorite text editor here And then we'll have some other tools built over on top of subversion and get our main repositories or subversion Over half of our engineering team now uses get on top of that and then we have some tools built on top around that in terms of things like code review and and documentation and things like that Hi there Can you say a little bit about Cassandra and what using Oracle for please? Thank you. So Cassandra Yeah, so we do a team at Facebook developed Cassandra a few years ago. It's also in the patchy incubator right now we use it for inbox search on Facebook and We haven't really been developing it actively in the past year But at this point I think dig and Twitter and Rackspace are really the largest contributors to Cassandra Inside of the patchy project and so I mean I think from our perspective It's really great to continue to see it develop and have a community around it that's going and continuing to push it forward Even though we're not actively developing it as well Hello, how do you manage the deployment of software? So software deployments. I mean we have a variety of tools built around that as well In terms of being able to go and push out information to different hosts going and running jobs on those hosts Distributing information and I mean I think that's something that we'll talk about a little bit more this summer Hopefully at the velocity conference Are there any more questions Okay, I think not so thank you very much for your talk
over 3.5 billion pieces of content is shared you're able to go and shard your databases. started, Mark Zuckerberg created it in his Harvard dorm room and it was sort of pretty over the world. It's not a site where you can go graph has also been evolving. I mean, Facebook what it will do is it will translate your PHP little optimizations like that, forks into the and give you a better idea of a few of the characteristics of these languages for what we're serialize it with Thrift, and then talk to PHP views per month on Facebook. And we log about 25 open source Scribe. It's also being used by lot of our infrastructure, is that it was built Linux and Apache and MySQL and PHP. And then if in order to make Facebook work. And so if you go performance work around it as well. One of the million people to go and access Facebook at software is to go and scale Facebook since the specific problem and have a lot of people And so we've developed, I mean, we've developed a monitoring technologies that we use I know that are really the largest contributors to Cassandra