Benchmarking FreeBSD
Voras, Ivan
System optimization and tuning is tricky business. Benchmarking such systems is even more complicated as the number of things which can go wrong at doubles at the least. FreeBSD is best known for its killer features, but it also includes over 2500 sysctls, most of which can be tuned to do something interesting. This talk aims to give an overview of some of the more interesting things which can be tuned in FreeBSD, and advice on how to avoid the most common errors in benchmarking FreeBSD. Tuning a system heavily depends on hardware present in the system, so all tuning advice will necessary contain system-specific parts, but overall there is much to discuss when talking about optimizing specific systems: networking, storage, file systems, even the CPU scheduler. Networking is still very much dependant on the quality of the NIC and its driver, but large parts of the systems such as ZFS are pure software and can benefit from tweaks and tuning which slightly alter the behaviour of algorithms. Things get very complicated when comparing different hardware configurations, and even more when comparing different operating systems. Doing a good benchmark of two unrelated operating systems is tricky because it requires similar tune-ups to both systems. This talk will try to explain where the pitfalls are, and also present some field results.
 Thank you everyone for coming. I'm Ivan Ronoz and I'm here to talk about benchmarking, which is a hot topic always and you know everybody is excited when new benchmarks come out and everybody is trying to either repeat them or prove them wrong or something. This presentation will be mostly about how to do certain aspects of benchmarking. More advanced users will probably know them already. So this is how to do and what not to do when doing a benchmark. It was operating system. And the thing is I've also tried to do some new benchmarks of previous D which would be kind of extensive and useful for the years to come. The last one, last ones were done by Chris Canaway in 2008. This is the time frame of the 7.x release. It's a bit old. And also there are some benchmarks appearing on the mailing list and on various blogs which measured certain aspects of previous D performance and which are not really professionally done and which paint not so good picture for previous D. And I tried to either repeat them or prove them to be done in a less than satisfactory way. So when I say mostly developers, I mean developers who are interested in measuring performance of certain aspects of their systems or subsystems they are working on. System administrations, system administrators will also find some useful material here. And lastly I'm going to talk for a few slides or a few sentences about how and when to avoid benchmarking because sometimes you don't really need it. Generally, when you're doing a benchmark, that means you have some purpose in mind. Either you want to compare your system to some other system, you want to compare your hardware to some other hardware, you want to compare your application to some other application. It is mostly useless to perform a benchmark on its own. So if you have a number that says that many megabytes per second or that many operations per second and you don't have anything to compare it to, it's just mostly useless. I mean you can compare it to a previous version of a system that's also valid. It doesn't have to be a completely new system or complete new hardware. But generally benchmarking means you have to compare it to something. And the goal of a good benchmark is to make it repeatable, to make it useful for other people to make the description of the benchmark as elaborate or as complete as possible so that other people can repeat it and see for themselves if what you did is good or are their own measurements comparable to yours. So I'm also going to talk extensively of how the benchmarks I did were set up and what it is to actually arrive at the results. Repeatability is always a good thing. Even if you're doing benchmark only for yourself, it means that next week or next month or next year you will be able to create the same environment and repeat your benchmark to check if something says change, for example, is the new version of an application or operating system different or more or less performant. I hope to get some high-end hardware for this, but unfortunately I didn't. So this is all done on to identically configured servers, IBM 1U servers and only a four core CPU without hyper-trading. So four cores is everything you get in this presentation. There's two gigabytes of RAM and four satellites set in RAID 0 because they wanted to assess performance and not really redundancy and reliability. And networking was done on a gigabit embedded NIC which has two ports, one of which is connected to the Internet and one of which is used to connect the back sides of the servers with a really simple wire, so without a switch in the mean in between. Some notes about software versions. I did both previous D9.1, which is the release version, and the previous D10 current, which is the development, the future version 10. This version 10 was approximately two weeks ago, so it is really recent. Of course, the debugging was turned off, witness invariance and such things. For comparison with Linux, I used CentOS 6.3 because it is really probably the most used so-called enterprise version of Linux. I had PostgreSQL 9.2, Blockbench, Bony++, Filebench, and my own bullet cache server. I will describe it later. And as preliminaries also, I'd like to talk about what and how to interpret, to post-process your benchmark results. And this is a picture which is in every college textbook about doing any kind of measurements. So I will also repeat it here. When we talk about measurement results, we can talk about how accurate they are or how precise they are. If they are accurate, they measure the real thing, measure the real aspect of the system you are trying to measure. And if they are precise, they are all clustered around basically the same value, which should be the one true value you are trying to measure, but it usually isn't. So it will be more clear in examples. But it is also related to introducing measurement errors because of domestic errors, those which are influencing our measurement in a way that we measure actually something else. There will be some interesting examples here. So basically, you are thinking you are measuring one aspect of the system, and in actuality, you are measuring something different. And also, random errors, which are basically noise, measurement noise, which can come from all parts of the system. You can basically either reduce it or just leave it and use statistics to establish some kind of medium or average value. Lots of benchmarks also exaggerate in their precision. So you can see some benchmarks on the Internet that says a hard drive has 412 SSD in this case, 412.567 megabytes per second. This is usually nonsense because you cannot actually measure hard drive or SSD performance to this precision. In most cases, the best you can do is measure maybe the first two significant digits. So this will probably be better expressed like 410 megabytes per second. In theory, you could achieve really, really, really good precision like in this case, but in practice, you won't ever. It's also useful to introduce the error bars in the form of either on graphs, doing this small extensions on the graph below and about the real value or express it numerically or something. Basically, the error bars are an expression of the precision of your measurements, the confidence that the measurements are really actually precise or really repeatable in a way that is useful. My favorite example here is measuring hard drive performance because you get a lot of file system benchmarks on the Internet. You just create a RAID array or you just use a single drive and create a file system on it and just run some kind of benchmark on it. This is not exactly the way it should be done because especially on mechanical hard drives, so-called spinning grass and such, you have huge differences in performance, linear sequential read performance or sequential write performance between the inner part of the drive and the outer part of the drive. If you use the disk info utility, which is part of every FeeBSD, it's a part of the base system, you can get a really nice illustration of how performance differs when you go from the outside of the drive platters to the inside of the drive platters. For example, if you just create a file system on such drive and measure file system performance in any kind of utility, I mean really any kind of benchmark utility, it's very much influenced on which part of the drive your file system code places the data. If it places the data on the outside, you get one kind of performance curve and if it places in the inside, you get another type of performance curve. This is a difference of approximately, I think, 100 megabytes per second. Really, it is significant. So don't do file system benchmarking on whole drives, especially not on mechanical drives. SSDs have a different kind of problem. They have a lot of internal structure. The flesh sensation layer basically says that your data, which you're thinking you're writing a particular place on a drive, can actually end up in some other place on the drive and it can be internally fragmented and it can be compressed and it can be transferred in a way you have no influence over. So benchmarking with flesh is a whole other topic. This is also an illustration of a few runs of file system of this info benchmark, which is shown in the previous slide. It basically shows how there's quite a lot of noise benchmarking hard drives. Even though this is this info, so it doesn't pass through a file system. It talks directly to the hard drive via dev slash da0 or some other device area specific path. Basically, it says that there's still significant noise here. This is noise between, for example, 300 megabytes per second and 350 megabytes per second. It's significant. The noise is also present in all cases. If the measurement is on the outside of the platters or on the inside of the platters, it is still present. It is not really something you can avoid. So you use statistics to get an average value. How many people have seen this video? Yeah. Basically, the guy shouts at an drive area. There's a drive area. It's on a data center. The guy shouts at it and his acoustic vibrations are transferred to the drive heads and he sees unnoticeable spikes in latency. So really, mechanical drives can be really, really sensitive. You get visible, visible spikes. This means that if you really want to get scientific about it, you will really need to go to really much trouble to do it. Fortunately, you can always aim for ballpark value. For example, you always want to get values like 300 megabytes per second, not exactly 335.456. You really have to be realistic about your precision, so you can expect on it. So what do you want to do with your matchpad? A hard drive or file system. Basically, you want to create a partition which is somewhere close to the outside of the drive and which is small enough so that the difference in performance is not noticeable between its beginning and its end. This partition has to be of large enough size so you can run meaningful benchmarks on it. For example, it has to be at least two times larger than your available memory because if you run a sequential read or sequential write test, you need to ensure that it's not cached. But since a lot of data comes to file system metadata and you have the reserved portion of file systems which is in UFS approximately 8%, usually you have to do three times or four times the size of your RAM for this kind of partition. And as an illustration, this on the left side, this is my partition. And on the right side, this is again the graph showing the whole drive performance. So when you look at just this partition, the inner, middle and, the outer, middle and inner parts of this partition basically have the same performance, plus minus some noise. And of course, the whole drive is completely different story. So these are the results. I think the blue channel is not working on this projector. This should be a blue bar. Blue bars or in this case, black bars are Linux performance. And the red one is 3BSD9 and the yellow one is 3BSD10. This is a result from Bonnie++. I'm using the file system bandwidth, actually the sequential read and write rates and also the rewrite rate. You can see that the results are interesting and it's hard to tell if you can set much emphasis to the Linux performance which is worse on writing and better on reading or the other way around. What is interesting and which will be confirmed in other benchmarks is that actually there's happened some development between 9.5 and 1.5 release of 3BSD and 10.0 release of 3BSD in such a way that basically rewrite performance is much better. Also, the read performance is a little bit better. It means that some kind of new development has probably happened in the area of concurrency on maybe more fine-grained locking in some part of the system. We need to take off file systems. Basically file systems are kind of a database. You get records which are files stored on a medium. These records are simple but they have associated metadata. They have associated security permissions. They have attributes. They also need to be concurrently accessed whether for reading or for writing. This database really can get quite complex. ZFS is notorious for having a really, really complex layout. It's more compared or more often compared to a database than of traditional file systems. The file system needs to support other features like they need to be reliable. They need to support three operations. Network file systems are also complex in a completely different way. Next I'd like to talk about the blockbench benchmark. It basically is a very small utility but very cleverly written. Its goal is to create a tree of smallish files. It has two types of files. One is a sort of text files of two kilobytes in size or four kilobytes in size. In the other ways like multimedia files or in this case image files which are 64K in size. Generally the sizes are not always the same. They are spread across this spectrum between two kilobytes and 64 kilobytes. The interesting thing is the blockbench is multi-threaded. It creates hundreds of threads. It can create hundreds of threads. The default configuration starts hundreds and tens of threads. All of these threads are divided into several groups. One group creates new files. The other group modifies files. The third group reads files. In this case the third group, the one that reads files, is the largest one. We have, for example, 80 threads reading files and the rest are writing or creating them. It also uses atomic renames for some writes. For example, it modifies a blog file and then it renames it to overwrite the old content. This is important because in FreeBSD much emphasis is given to write looking. If you've seen any Linux benchmarks, you've seen pictures like this. Basically Linux performance in the blockbench is all the way over there, almost a million. FreeBSD performance is way over here, 700,000. It's puzzling. Why would this be? This is all done on UFS. I have no ZFS numbers here. Basically it comes down to how the operating system schedules reading and writing. What is interesting is that there is some improvement between nine and ten versions. To prove it, you can use the very easy, very convenient iterability present in FreeBSD, which is Ministat. It is also part of the base system. We already have two benchmarks. One is this KINFO and one is, or two utilities. One is this KINFO, one is Ministat. It can be used to create quality benchmarks. I basically was wondering what's the difference between nine dot one and ten releases. It says that approximately 90% better performance is achieved in ten, but in the read case, but approximately 17% lower performance is achieved for the write case. It needs more investigation. Blockbench is a multilateral benchmark. It uses a lot of parallel operations on the same tree of files. There's something called the write bias of FreeBSD, which basically means that writes on a single file block all other access. I think it even blocks other reads. I'm not sure. I think it is such. It is useful to remember that the directories themselves are also files. So write operations on the directories themselves are also blocked. I did some system call tracing. Basically, you get a long list of CIS calls. It clearly shows that there are multitude of threads during various operations, opening, grading, writing, closing, and such. From all this data, we can create a pretty picture, which kind of characterizes the number and the frequency of operations. What isn't shown on this axis is these zeros should be operation duration times. They are really very small in the order of microseconds. It got all rounded up to zero. We see from this graph that Blockbench does a lot of what we knew before. Read operations are dominant. Surprisingly, there are a lot of closed operations, which is kind of normal when you say, when you know that Blockbench opens files, modifies them, or reads them, and then closes them again. There's a really small number of write operations, basically this curve over here. It is interesting to note that read operations also take the longest, comparatively, to everything else, which I think indicates that they are actually blocked by other types of operations, but this means a closer investigation. The other benchmark which is often used in 3BSE world is PostgreSQL benchmark. It's a database. In this case, I've used the PG benchmark, which is a part of PostgreSQL source base. I've initialized the benchmark database with the scale factor of 1,000, which means 100 million records. The database is approximately 16 gigabytes in size. It fits in RAM. This is deliberate because I'd like to test, for example, from currency in database access without really benchmarking the hard drive at the same time. The configuration is, of course, stable during the whole test. These two machines, I was talking about at the beginning, are configured so that they dual boot Linux and 3BSD. Both Linux and 3BSD have access to the same partition. The same file system is used for both Linux benchmarks and 3BSD benchmarks. This is the partitions I was talking about at the beginning, so that one with small deviations between access performance on outside, middle, and inner tracks. PostgreSQL is a fairly modern database, which means it has lots going on in the back end. The most important thing is the right logging. Basically, if you do a right operation in PostgreSQL, the data doesn't go directly to the database store itself. It is written in the right-hand log first. Periodically, this data is then transferred from this right-hand log to the database storage itself. The problem is that this can happen at unexpected times. For example, if you run a short benchmark, you can run the whole benchmark with the data residing in the right-hand log and not in the database storage itself. You can run another benchmark, which is started, and then right in the middle of benchmarks, this transfer occurs between the right-hand logs and the proper storage. This introduces a large amount of noise in your benchmark results. The benchmark runs need to be longish at least five minutes, 10 minutes. There are recommendations for even longer times, like half an hour. It is crucial that the PostgreSQL server itself, the daemon itself, be restarted between the benchmarks because a restart of PostgreSQL causes all data to be transferred from the logs, from the right-hand logs, to the data storage itself. You get more consistent, more precise measurements. Also, it is necessary to turn on auto-vehicle. The vacuum in PostgreSQL is the part of the system which basically cleans up data in case of updated records and deleted records and calculates some statistics information, which can also influence your results so they can be scattered or very noisy. On this particular machine, which is the quad-core Xeon I was talking about, I run benchmarks which are read-only, which are read-write, and also I run benchmarks which are on the file system, meaning on the drives, on the drives themselves, on the disk array, and on the memory file system, which is a ramfs in case of Linux and tmpfs in case of gpsd. I've also done some remote benchmarking, meaning remote connection to the database from the other server. I will talk about some of the results in more detail. So, on Linux, I discovered that there is almost no difference between fully cached database on X4 in read-only benchmarks and its ram file system or memory file system. On 3BSD, using tmpfs is faster than using UFS, which is fully loaded or fully pre-warmed database content. The blue lines are basically Linux and the red lines are 3BSD. This dip here is very surprising. It is also very repeatable. I get it in all configurations in all benchmark runs. I have a theory why is it happening, I'll try to explain it later. What most people are interested in right now is that on this particular machine, I get lower performance of 3BSD than of Linux, except for this dip here. I don't know if you can see it, but there are two lines here, the dark blue one and the light blue one. The dark blue one is disk file system, and the light blue one is a memory file system. On 3BSD, the dark red one is the disk file system, UFS, and the light red one is tmpfs, the memory file system. There is a noticeable difference here. These are the right results. Again, the Linux curve is the blue one and the 3BSD curve is the red one. Even though this particular benchmark should really depend on the disk performance, apparently it doesn't. The opening system has a huge influence on the total performance. This graph shows benchmarks of remote access performance. What I did was use the same servers. One was obviously running the database, and the other was running the benchmark client. The database was always on memory file system. This is the distinction between local access, which is the full lines here, the dark blue one and the dark red one, and the remote access over here, which is the light blue one and the yellow one. We can see that remote access helps a little bit, because the benchmark client itself is a CPU intensive process. When separating the benchmark client and the other machine, you'll see some resources on the server itself. This is offset almost entirely by the fact that there is a TCP connection between those machines, even though it's just a wire between the two metro cards. The same general conclusion can be drawn that 3BSD still has some work to do in performance improvement. Because we can now separate the client on another machine, I think that this database is actually caused by some kind of scheduler issue in 3BSD. In the original case, you have both the client and the server processors on the same machine. Somehow, if approximately 12 concurrent clients, Linux has some kind of a problem, I think. My best guess is that this is a scheduler issue in Linux. However, to get some more optimistic news, these are the results done by Florian Smith and I also think Jeff Robertson on a hugely different machine. This machine has 40 cores. It has four CPUs, 10 cores each, and 80 threads in total on the system. The results are much better on such a configuration. Basically, in this configuration, 3BSD is consistently better than Linux, so there's some hope yet. The algorithms used in achieving large scalabilities probably favor systems with a large number of CPUs, so something needs to be done to maybe close that gap. Linux, which is blue in this graph, lacks in performance up until approximately 32 clients or 32 connections to the database server. This is also PostgreSQL 9.2. This is also a benchmark of local benchmark client and the local server on the same machine. Next benchmark I would like to talk about is File Bench. This is a benchmark done, I think, by some microsystems back in the day, and it has a large number of profiles. One profile, for example, is a file server which basically creates a large file, and the other is the proxy profile, which creates smaller files. The different benchmark operations being done on these profiles. Unfortunately, I think that this benchmark has some problems running on 3BSD. I have some strange results, I'll talk about them later, and I'm not so sure that it's really a correct benchmark. I'm not sure that these results I will show you are really correct on the 3BSD side. I've done local draft measurements on UFS, on this partition I was talking about, and also NFS 3 and NFS 4, because there was some talk on the mailing list about the huge performance difference between NFS 3 and 4 in 3BSD, so I wanted to find out what's happening. This is the file server profile. The results are strange. On the one side, we have Linux, which is this huge dark bar here. The result is larger than 800 megabytes per second, which is obviously due to caching, which is all right, or actually it would be all right if it were not for 3BSD results, which is this red bar here, which is almost 100 megabytes per second. The drives themselves can pull 450 megabytes per second at least, so I think that the huge difference between the Linux results and 3BSD results are due to benchmark problems. This is an example of maybe a systemic error or systemic problem in benchmarks. I've run NFS 3 and NFS 4. These are the two bars right here, the light blue one and the dark blue one. The light blue one is NFS 3 and the dark blue one is NFS 4, and even on Linux, NFS 4 is significantly slower for some reason. I was hoping to talk to Rik, but I think he's not on the conference right now. The 3BSD results are lower than Linux results. I really cannot explain why local disk performance should be so low, because this is much lower than the hardware itself can do on this machine. The NFS result is again significantly lower than the NFS 3 result. The NFS 4 was mounted over TCP. On the other hand, we have the web proxy profile, which again shows really quirky results, but a bit different. This time Linux has lower performance on the local hard drive measurements, and 3BSD has huge better performance on the same conditions. Again, it looks to me like the benchmark itself is wrong. The benchmark itself seems to be either it has a bug or is Linux specific and there's some problem with porting it to 3BSD or something. This is possible, but not really probable that such results would happen. This just goes to say that you need to choose your benchmarks carefully. As an interesting side result, this is also the same file benchmark on the file server profile. I've used 3BSD as an NFS 3 server and Linux as the NFS 3 client, and shows that the results are basically the same. Which is encouraging. Last year I talked about my bullet cache server. It's a memory cache server similar to memcached. It creates a TCP traffic of very small transactions, like 22 byte messages going back and forth between the client and the server. It is heavily multithreaded and also uses non-blocking IO. If you benchmark our unique sockets, meaning both the server and the client are around the same machine and talk about, talk to you in the main sockets, you get easily 2 million transactions per second on mid-range hardware. The results are in favor of 3BSD on this particular benchmark, but the benchmark itself was developed on 3BSD by me, so I used a lot of tuning for this to actually happen. I also would not advertise this as a truly good benchmark with Linux and 3BSD. It's more of a curiosity. Between measurement errors, the same performance is between 9.1 and 10 releases. The benchmark is interesting because it illustrates that 3BSD's TCP stack is fairly multithreaded and fairly finally locked. TCP transactions, TCP streams do not influence each other. Multiple concurrent streams can be used on 3BSD without performance problems. It also shows that I got, for example, nearly 500,000 backers per second per direction in this configuration. This is a lot of really small TCP messages. I'm satisfied with the performance shown. To elicit the difference between TCP over the plain internet via connecting the servers and Unix Domain's sockets, Unix Domain's sockets performance crosses 1 million transactions per second. This is also very naturally expected between on the other one. On one hand, you have the whole internet channel. You have the whole internet actual networking happening. On the other hand, you have only memory passing between processes. One thing that a lot of people are trying to figure out is what to tune on 3BSD to get acceptable performance. I would argue that especially with release 10 and even the now current release 9.1, there is not much you can do or actually the system auto-configures itself pretty well. We had an increase in really head tunable recently. Also buff space, high buff space and low buff space. They are now auto-tuned. You don't have to touch them. Some tutorials advertise increasing the network transmit and receive descriptor rings or increasing their size. I haven't actually noticed some significant difference on my previous benchmark. There is some difference. I'm not so sure if this is because of measurement noise. I need to do some more benchmarking to actually find out. But the point is the default values are actually pretty good. I also try to increase the number of interrupts per second allowed on the EM driver, on the CPUs. If I increase them, I get also very, very small improvement, maybe less than 1% of performance. Unless you are really trying to squeeze out some percent performance from your machines, you don't have to touch the, for example, network driver configuration. Current.max users governs basically the size and number of internal network, internal kernel structures. It's been auto-tuned for years now. For the last example, this tunable, which is maybe familiar to everyone who had run Apache or PostgreSQL or something else, which has a lot of processes, networking is also gone from 9.x releases. My point is you don't have to rely on manual tuning as much as you used to on 3BSD. I didn't give any CPU-specific benchmarks. By that I mean I didn't do any encryption, I didn't do any kind of compression benchmarks and such, because for the most part, they are trivial. Unless you have a broken compiler or unless you have a different tuning in your compression libraries or different algorithms in your libc, you're not going to find any significant or any useful difference in such benchmarks. Most of the benchmarks which are now available on the Internet show a noticeable difference in compression performance. For example, they use Gzip or some other libc utility to compress a huge file on Linux and they do it again on 3BSD and then show, for example, that Linux performance is a little bit better or the 3BSD performance is a little bit better. This, for example, relies on how the libc was compiled itself. For one time, 3BSD didn't use an assembler optimized part of the compression algorithm because there was a security bug in it. So libc was compiled with C only code, which had a lower performance than the assembler optimized version. So this is the cause of a drop in such benchmarks. If you want to, you can test the quality of libc and lib implementation, for example, you can test the quality of the IO, SDIO set of calls or something like this, but then be sure, you know, what are you benchmarking? You're not benchmarking, for example, the kernel in this case. You're benchmarking the algorithms in your libc. You can also test your compiler. 3BSD has a sort of old-age compiler by default. This will change in 9.0. And just as an illustration, you can find, you can find benchmarks like this floating around the internet where you have one side GCC and recent versions of GCC as well. And on the other hand, you have LVM, C-Lang compiler. And so, and for example, this particular benchmark shows huge difference between C-Lang and GCC. What this picture doesn't show is that there are other benchmarks showing completely different values. So in this case, it is mostly, you can run benchmarks like this. There's no problem in running benchmarks like this. But you have to know that you are running a benchmark of a compiler. So if you compile, for example, a compression library or an encryption library with different compilers on different operating system, it's all right. You can do this. But just realize that you are not exactly benchmarking, for example, the operating system kernel, you are benchmarking the quality of a compiler. And sometimes, benchmarking is done in circumstances which really don't require it. There are a lot of reasons why you can do benchmarks. But there are also reasons why benchmarking really is not as useful or the numbers are not as useful as you think. You can choose an operating system based on its features, based on its community, based on its large number of software, or supported software. You can also say that it's cheaper to buy another machine and run a slightly slower operating system than to reconfigure everything. And one conclusion that is also good to keep in mind is that actually, pre-BSD gets better through the years and it consistently gets more and more scalable. During the Chris Kenneway benchmarks of 7.x area, pre-BSD has tried to achieve scalability on 8 CPU hardware. And it succeeded, but that's all news now. And during the 10.x era, realistically, pre-BSD probably aimed for being scalable or being better than Linux on hardware containing 32 CPUs. So it's consistently improving. It is very much a modern system going forward. You need to benchmark, maybe, out of curiosity. You need to benchmark if you really want to know what's going on in the system when you're planning or budgeting for a new project, when your boss tells it to. And finally, we get advocacy. And this is perhaps the most commonly used reason to do, or actually, publish benchmarks on the internet. So, yeah. During the developer summit, I was talking to some of the other developers and some of the class administrators, so pre-BSD. And we will try to get up some continual benchmarking going on. So maybe the next BSD Kenne or something like that, I will have some updated numbers and I will have some nice performance curve improvements over the time, over the year of pre-BSD development. I think this is it. The point of this presentation was mainly to point out some interesting facts about benchmarking itself, about what to do and what not to do when benchmarking a non-primary system. So I hope it was reasonably interesting presentation to hear and thank you for staying.
 Thank you everyone for coming. I'm Ivan Ronoz and I'm here to talk about benchmarking,  which is a hot topic always and you know everybody is excited when new benchmarks come out and  everybody is trying to either repeat them or prove them wrong or something. This presentation  will be mostly about how to do certain aspects of benchmarking. More advanced users will  probably know them already. So this is how to do and what not to do when doing a benchmark.  It was operating system. And the thing is I've also tried to do some new benchmarks of  previous D which would be kind of extensive and useful for the years to come. The last one,  last ones were done by Chris Canaway in 2008. This is the time frame of the 7.x release.  It's a bit old. And also there are some benchmarks appearing on the mailing list and on various  blogs which measured certain aspects of previous D performance and which are not really professionally  done and which paint not so good picture for previous D. And I tried to either repeat them  or prove them to be done in a less than satisfactory way. So when I say mostly developers, I mean  developers who are interested in measuring performance of certain aspects of their systems  or subsystems they are working on. System administrations, system administrators will  also find some useful material here. And lastly I'm going to talk for a few slides or a few  sentences about how and when to avoid benchmarking because sometimes you don't really need it.  Generally, when you're doing a benchmark, that means you have some purpose in mind. Either  you want to compare your system to some other system, you want to compare your hardware  to some other hardware, you want to compare your application to some other application.  It is mostly useless to perform a benchmark on its own. So if you have a number that  says that many megabytes per second or that many operations per second and you don't have  anything to compare it to, it's just mostly useless. I mean you can compare it to a previous  version of a system that's also valid. It doesn't have to be a completely new system or complete  new hardware. But generally benchmarking means you have to compare it to something.  And the goal of a good benchmark is to make it repeatable, to make it useful for other  people to make the description of the benchmark as elaborate or as complete as possible so  that other people can repeat it and see for themselves if what you did is good or are  their own measurements comparable to yours. So I'm also going to talk extensively of  how the benchmarks I did were set up and what it is to actually arrive at the results.  Repeatability is always a good thing. Even if you're doing benchmark only for yourself,  it means that next week or next month or next year you will be able to create the same environment  and repeat your benchmark to check if something says change, for example, is the new version  of an application or operating system different or more or less performant. I hope to get  some high-end hardware for this, but unfortunately I didn't. So this is all done on to identically  configured servers, IBM 1U servers and only a four core CPU without hyper-trading. So  four cores is everything you get in this presentation. There's two gigabytes of RAM and four satellites  set in RAID 0 because they wanted to assess performance and not really redundancy and  reliability. And networking was done on a gigabit embedded NIC which has two ports, one of which  is connected to the Internet and one of which is used to connect the back sides of the servers  with a really simple wire, so without a switch in the mean in between. Some notes about software  versions. I did both previous D9.1, which is the release version, and the previous D10  current, which is the development, the future version 10. This version 10 was approximately  two weeks ago, so it is really recent. Of course, the debugging was turned off, witness  invariance and such things. For comparison with Linux, I used CentOS 6.3 because it is  really probably the most used so-called enterprise version of Linux. I had PostgreSQL 9.2, Blockbench,  Bony++, Filebench, and my own bullet cache server. I will describe it later. And as preliminaries  also, I'd like to talk about what and how to interpret, to post-process your benchmark  results. And this is a picture which is in every college textbook about doing any kind  of measurements. So I will also repeat it here. When we talk about measurement results,  we can talk about how accurate they are or how precise they are. If they are accurate,  they measure the real thing, measure the real aspect of the system you are trying to measure.  And if they are precise, they are all clustered around basically the same value, which should  be the one true value you are trying to measure, but it usually isn't. So it will be more clear  in examples. But it is also related to introducing measurement errors because of domestic errors,  those which are influencing our measurement in a way that we measure actually something  else. There will be some interesting examples here. So basically, you are thinking you are  measuring one aspect of the system, and in actuality, you are measuring something different.  And also, random errors, which are basically noise, measurement noise, which can come from  all parts of the system. You can basically either reduce it or just leave it and use  statistics to establish some kind of medium or average value. Lots of benchmarks also  exaggerate in their precision. So you can see some benchmarks on the Internet that says  a hard drive has 412 SSD in this case, 412.567 megabytes per second. This is usually nonsense  because you cannot actually measure hard drive or SSD performance to this precision. In most  cases, the best you can do is measure maybe the first two significant digits. So this  will probably be better expressed like 410 megabytes per second. In theory, you could  achieve really, really, really good precision like in this case, but in practice, you won't  ever. It's also useful to introduce the error bars in the form of either on graphs, doing  this small extensions on the graph below and about the real value or express it numerically  or something. Basically, the error bars are an expression of the precision of your measurements,  the confidence that the measurements are really actually precise or really repeatable in a  way that is useful. My favorite example here is measuring hard drive performance because  you get a lot of file system benchmarks on the Internet. You just create a RAID array  or you just use a single drive and create a file system on it and just run some kind  of benchmark on it. This is not exactly the way it should be done because especially on  mechanical hard drives, so-called spinning grass and such, you have huge differences  in performance, linear sequential read performance or sequential write performance between the  inner part of the drive and the outer part of the drive. If you use the disk info utility,  which is part of every FeeBSD, it's a part of the base system, you can get a really  nice illustration of how performance differs when you go from the outside of the drive  platters to the inside of the drive platters. For example, if you just create a file system  on such drive and measure file system performance in any kind of utility, I mean really any  kind of benchmark utility, it's very much influenced on which part of the drive your  file system code places the data. If it places the data on the outside, you get one kind  of performance curve and if it places in the inside, you get another type of performance  curve. This is a difference of approximately, I think, 100 megabytes per second. Really,  it is significant. So don't do file system benchmarking on whole drives, especially not  on mechanical drives. SSDs have a different kind of problem. They have a lot of internal  structure. The flesh sensation layer basically says that your data, which you're thinking  you're writing a particular place on a drive, can actually end up in some other place on  the drive and it can be internally fragmented and it can be compressed and it can be transferred  in a way you have no influence over. So benchmarking with flesh is a whole other topic. This is  also an illustration of a few runs of file system of this info benchmark, which is shown  in the previous slide. It basically shows how there's quite a lot of noise benchmarking  hard drives. Even though this is this info, so it doesn't pass through a file system.  It talks directly to the hard drive via dev slash da0 or some other device area specific  path. Basically, it says that there's still significant noise here. This is noise between,  for example, 300 megabytes per second and 350 megabytes per second. It's significant.  The noise is also present in all cases. If the measurement is on the outside of the platters  or on the inside of the platters, it is still present. It is not really something you can  avoid. So you use statistics to get an average value. How many people have seen this video?  Yeah. Basically, the guy shouts at an drive area. There's a drive area. It's on a data  center. The guy shouts at it and his acoustic vibrations are transferred to the drive heads  and he sees unnoticeable spikes in latency. So really, mechanical drives can be really,  really sensitive. You get visible, visible spikes. This means that if you really want  to get scientific about it, you will really need to go to really much trouble to do it.  Fortunately, you can always aim for ballpark value. For example, you always want to get  values like 300 megabytes per second, not exactly 335.456. You really have to be realistic  about your precision, so you can expect on it.  So what do you want to do with your matchpad? A hard drive or file system. Basically, you  want to create a partition which is somewhere close to the outside of the drive and which  is small enough so that the difference in performance is not noticeable between its  beginning and its end. This partition has to be of large enough size so you can run  meaningful benchmarks on it. For example, it has to be at least two times larger than  your available memory because if you run a sequential read or sequential write test, you  need to ensure that it's not cached. But since a lot of data comes to file system  metadata and you have the reserved portion of file systems which is in UFS approximately  8%, usually you have to do three times or four times the size of your RAM for this kind  of partition. And as an illustration, this on the left side, this is my partition. And  on the right side, this is again the graph showing the whole drive performance. So when  you look at just this partition, the inner, middle and, the outer, middle and inner parts  of this partition basically have the same performance, plus minus some noise. And of  course, the whole drive is completely different story. So these are the results. I think the  blue channel is not working on this projector. This should be a blue bar. Blue bars or in  this case, black bars are Linux performance. And the red one is 3BSD9 and the yellow one  is 3BSD10. This is a result from Bonnie++. I'm using the file system bandwidth, actually  the sequential read and write rates and also the rewrite rate. You can see that the results  are interesting and it's hard to tell if you can set much emphasis to the Linux performance  which is worse on writing and better on reading or the other way around. What is interesting  and which will be confirmed in other benchmarks is that actually there's happened some development  between 9.5 and 1.5 release of 3BSD and 10.0 release of 3BSD in such a way that basically  rewrite performance is much better. Also, the read performance is a little bit better.  It means that some kind of new development has probably happened in the area of concurrency  on maybe more fine-grained locking in some part of the system.  We need to take off file systems. Basically file systems are kind of a database. You get  records which are files stored on a medium. These records are simple but they have associated  metadata. They have associated security permissions. They have attributes. They also need to be  concurrently accessed whether for reading or for writing. This database really can get  quite complex. ZFS is notorious for having a really, really complex layout. It's more  compared or more often compared to a database than of traditional file systems. The file  system needs to support other features like they need to be reliable. They need to support  three operations. Network file systems are also complex in a completely different way.  Next I'd like to talk about the blockbench benchmark. It basically is a very small utility  but very cleverly written. Its goal is to create a tree of smallish files. It has two  types of files. One is a sort of text files of two kilobytes in size or four kilobytes  in size. In the other ways like multimedia files or in this case image files which are  64K in size. Generally the sizes are not always the same. They are spread across this spectrum  between two kilobytes and 64 kilobytes. The interesting thing is the blockbench is multi-threaded.  It creates hundreds of threads. It can create hundreds of threads. The default configuration  starts hundreds and tens of threads. All of these threads are divided into several groups.  One group creates new files. The other group modifies files. The third group reads files.  In this case the third group, the one that reads files, is the largest one. We have,  for example, 80 threads reading files and the rest are writing or creating them. It  also uses atomic renames for some writes. For example, it modifies a blog file and then  it renames it to overwrite the old content. This is important because in FreeBSD much  emphasis is given to write looking. If you've seen any Linux benchmarks, you've seen pictures  like this. Basically Linux performance in the blockbench is all the way over there, almost  a million. FreeBSD performance is way over here, 700,000. It's puzzling. Why would this  be? This is all done on UFS. I have no ZFS numbers here. Basically it comes down to how  the operating system schedules reading and writing. What is interesting is that there  is some improvement between nine and ten versions. To prove it, you can use the very easy, very  convenient iterability present in FreeBSD, which is Ministat. It is also part of the  base system. We already have two benchmarks. One is this KINFO and one is, or two utilities.  One is this KINFO, one is Ministat. It can be used to create quality benchmarks. I basically  was wondering what's the difference between nine dot one and ten releases. It says that  approximately 90% better performance is achieved in ten, but in the read case, but approximately  17% lower performance is achieved for the write case. It needs more investigation.  Blockbench is a multilateral benchmark. It uses a lot of parallel operations on the same  tree of files. There's something called the write bias of FreeBSD, which basically means  that writes on a single file block all other access. I think it even blocks other reads.  I'm not sure. I think it is such. It is useful to remember that the directories themselves  are also files. So write operations on the directories themselves are also blocked. I  did some system call tracing. Basically, you get a long list of CIS calls. It clearly  shows that there are multitude of threads during various operations, opening, grading,  writing, closing, and such. From all this data, we can create a pretty picture, which  kind of characterizes the number and the frequency of operations. What isn't shown  on this axis is these zeros should be operation duration times. They are really very small  in the order of microseconds. It got all rounded up to zero. We see from this graph that Blockbench  does a lot of what we knew before. Read operations are dominant. Surprisingly, there are a lot  of closed operations, which is kind of normal when you say, when you know that Blockbench  opens files, modifies them, or reads them, and then closes them again. There's a really  small number of write operations, basically this curve over here. It is interesting to  note that read operations also take the longest, comparatively, to everything else, which I  think indicates that they are actually blocked by other types of operations, but this means  a closer investigation. The other benchmark which is often used in 3BSE world is PostgreSQL  benchmark. It's a database. In this case, I've used the PG benchmark, which is a part  of PostgreSQL source base. I've initialized the benchmark database with the scale factor  of 1,000, which means 100 million records. The database is approximately 16 gigabytes  in size. It fits in RAM. This is deliberate because I'd like to test, for example, from  currency in database access without really benchmarking the hard drive at the same time.  The configuration is, of course, stable during the whole test. These two machines, I was  talking about at the beginning, are configured so that they dual boot Linux and 3BSD. Both  Linux and 3BSD have access to the same partition. The same file system is used for both Linux  benchmarks and 3BSD benchmarks. This is the partitions I was talking about at the beginning,  so that one with small deviations between access performance on outside, middle, and  inner tracks. PostgreSQL is a fairly modern database, which means it has lots going on  in the back end. The most important thing is the right logging. Basically, if you do  a right operation in PostgreSQL, the data doesn't go directly to the database store  itself. It is written in the right-hand log first. Periodically, this data is then transferred  from this right-hand log to the database storage itself. The problem is that this can happen  at unexpected times. For example, if you run a short benchmark, you can run the whole benchmark  with the data residing in the right-hand log and not in the database storage itself. You  can run another benchmark, which is started, and then right in the middle of benchmarks,  this transfer occurs between the right-hand logs and the proper storage. This introduces  a large amount of noise in your benchmark results. The benchmark runs need to be longish  at least five minutes, 10 minutes. There are recommendations for even longer times, like  half an hour. It is crucial that the PostgreSQL server itself, the daemon itself, be restarted  between the benchmarks because a restart of PostgreSQL causes all data to be transferred  from the logs, from the right-hand logs, to the data storage itself. You get more consistent,  more precise measurements. Also, it is necessary to turn on auto-vehicle. The vacuum in PostgreSQL  is the part of the system which basically cleans up data in case of updated records and deleted  records and calculates some statistics information, which can also influence your results so they  can be scattered or very noisy. On this particular machine, which is the quad-core  Xeon I was talking about, I run benchmarks which are read-only, which are read-write,  and also I run benchmarks which are on the file system, meaning on the drives, on the  drives themselves, on the disk array, and on the memory file system, which is a ramfs  in case of Linux and tmpfs in case of gpsd. I've also done some remote benchmarking,  meaning remote connection to the database from the other server. I will talk about some  of the results in more detail. So, on Linux, I discovered that there is almost no difference  between fully cached database on X4 in read-only benchmarks and its ram file system or memory  file system. On 3BSD, using tmpfs is faster than using UFS, which is fully loaded or fully  pre-warmed database content. The blue lines are basically Linux and the red lines are  3BSD. This dip here is very surprising. It is also very repeatable. I get it in all configurations  in all benchmark runs. I have a theory why is it happening, I'll try to explain it later.  What most people are interested in right now is that on this particular machine, I get  lower performance of 3BSD than of Linux, except for this dip here. I don't know if you can  see it, but there are two lines here, the dark blue one and the light blue one. The  dark blue one is disk file system, and the light blue one is a memory file system. On  3BSD, the dark red one is the disk file system, UFS, and the light red one is tmpfs, the  memory file system. There is a noticeable difference here. These are the right results.  Again, the Linux curve is the blue one and the 3BSD curve is the red one. Even though  this particular benchmark should really depend on the disk performance, apparently it doesn't.  The opening system has a huge influence on the total performance. This graph shows benchmarks  of remote access performance. What I did was use the same servers. One was obviously running  the database, and the other was running the benchmark client. The database was always  on memory file system. This is the distinction between local access, which is the full lines  here, the dark blue one and the dark red one, and the remote access over here, which is  the light blue one and the yellow one. We can see that remote access helps a little bit,  because the benchmark client itself is a CPU intensive process. When separating the benchmark  client and the other machine, you'll see some resources on the server itself. This is offset  almost entirely by the fact that there is a TCP connection between those machines, even  though it's just a wire between the two metro cards. The same general conclusion can be  drawn that 3BSD still has some work to do in performance improvement. Because we can  now separate the client on another machine, I think that this database is actually caused  by some kind of scheduler issue in 3BSD. In the original case, you have both the client  and the server processors on the same machine. Somehow, if approximately 12 concurrent clients,  Linux has some kind of a problem, I think. My best guess is that this is a scheduler  issue in Linux. However, to get some more optimistic news, these are the results done  by Florian Smith and I also think Jeff Robertson on a hugely different machine. This machine  has 40 cores. It has four CPUs, 10 cores each, and 80 threads in total on the system. The  results are much better on such a configuration. Basically, in this configuration, 3BSD is  consistently better than Linux, so there's some hope yet. The algorithms used in achieving  large scalabilities probably favor systems with a large number of CPUs, so something  needs to be done to maybe close that gap. Linux, which is blue in this graph, lacks  in performance up until approximately 32 clients or 32 connections to the database server.  This is also PostgreSQL 9.2. This is also a benchmark of local benchmark client and  the local server on the same machine. Next benchmark I would like to talk about is File  Bench. This is a benchmark done, I think, by some microsystems back in the day, and  it has a large number of profiles. One profile, for example, is a file server which basically  creates a large file, and the other is the proxy profile, which creates smaller files.  The different benchmark operations being done on these profiles. Unfortunately, I think  that this benchmark has some problems running on 3BSD. I have some strange results, I'll  talk about them later, and I'm not so sure that it's really a correct benchmark. I'm  not sure that these results I will show you are really correct on the 3BSD side. I've  done local draft measurements on UFS, on this partition I was talking about, and also NFS  3 and NFS 4, because there was some talk on the mailing list about the huge performance  difference between NFS 3 and 4 in 3BSD, so I wanted to find out what's happening. This  is the file server profile. The results are strange. On the one side, we have Linux, which  is this huge dark bar here. The result is larger than 800 megabytes per second, which  is obviously due to caching, which is all right, or actually it would be all right if  it were not for 3BSD results, which is this red bar here, which is almost 100 megabytes  per second. The drives themselves can pull 450 megabytes per second at least, so I think  that the huge difference between the Linux results and 3BSD results are due to benchmark  problems. This is an example of maybe a systemic error or systemic problem in benchmarks.  I've run NFS 3 and NFS 4. These are the two bars right here, the light blue one and the  dark blue one. The light blue one is NFS 3 and the dark blue one is NFS 4, and even on  Linux, NFS 4 is significantly slower for some reason. I was hoping to talk to Rik, but I  think he's not on the conference right now. The 3BSD results are lower than Linux results.  I really cannot explain why local disk performance should be so low, because this is much lower  than the hardware itself can do on this machine. The NFS result is again significantly lower  than the NFS 3 result. The NFS 4 was mounted over TCP. On the other hand, we have the web  proxy profile, which again shows really quirky results, but a bit different. This time Linux  has lower performance on the local hard drive measurements, and 3BSD has huge better performance  on the same conditions. Again, it looks to me like the benchmark itself is wrong. The  benchmark itself seems to be either it has a bug or is Linux specific and there's some  problem with porting it to 3BSD or something. This is possible, but not really probable  that such results would happen. This just goes to say that you need to choose your benchmarks  carefully. As an interesting side result, this is also the same file benchmark on the  file server profile. I've used 3BSD as an NFS 3 server and Linux as the NFS 3 client,  and shows that the results are basically the same. Which is encouraging. Last year I talked  about my bullet cache server. It's a memory cache server similar to memcached. It creates  a TCP traffic of very small transactions, like 22 byte messages going back and forth  between the client and the server. It is heavily multithreaded and also uses non-blocking IO.  If you benchmark our unique sockets, meaning both the server and the client are around  the same machine and talk about, talk to you in the main sockets, you get easily 2 million  transactions per second on mid-range hardware. The results are in favor of 3BSD on this particular  benchmark, but the benchmark itself was developed on 3BSD by me, so I used a lot of tuning for  this to actually happen. I also would not advertise this as a truly good benchmark with  Linux and 3BSD. It's more of a curiosity. Between measurement errors, the same performance  is between 9.1 and 10 releases. The benchmark is interesting because it illustrates that  3BSD's TCP stack is fairly multithreaded and fairly finally locked. TCP transactions,  TCP streams do not influence each other. Multiple concurrent streams can be used on  3BSD without performance problems. It also shows that I got, for example, nearly 500,000  backers per second per direction in this configuration. This is a lot of really small TCP messages.  I'm satisfied with the performance shown. To elicit the difference between TCP over  the plain internet via connecting the servers and Unix Domain's sockets, Unix Domain's  sockets performance crosses 1 million transactions per second. This is also very naturally expected  between on the other one. On one hand, you have the whole internet channel. You have the  whole internet actual networking happening. On the other hand, you have only memory passing  between processes. One thing that a lot of people are trying to figure out is what to  tune on 3BSD to get acceptable performance. I would argue that especially with release  10 and even the now current release 9.1, there is not much you can do or actually the system  auto-configures itself pretty well. We had an increase in really head tunable recently.  Also buff space, high buff space and low buff space. They are now auto-tuned. You don't  have to touch them. Some tutorials advertise increasing the network transmit and receive  descriptor rings or increasing their size. I haven't actually noticed some significant  difference on my previous benchmark. There is some difference. I'm not so sure if this  is because of measurement noise. I need to do some more benchmarking to actually find  out. But the point is the default values are actually pretty good. I also try to increase  the number of interrupts per second allowed on the EM driver, on the CPUs. If I increase  them, I get also very, very small improvement, maybe less than 1% of performance. Unless  you are really trying to squeeze out some percent performance from your machines, you  don't have to touch the, for example, network driver configuration. Current.max users governs  basically the size and number of internal network, internal kernel structures. It's  been auto-tuned for years now. For the last example, this tunable, which is maybe familiar  to everyone who had run Apache or PostgreSQL or something else, which has a lot of processes,  networking is also gone from 9.x releases. My point is you don't have to rely on manual  tuning as much as you used to on 3BSD. I didn't give any CPU-specific benchmarks. By  that I mean I didn't do any encryption, I didn't do any kind of compression benchmarks  and such, because for the most part, they are trivial. Unless you have a broken compiler  or unless you have a different tuning in your compression libraries or different algorithms  in your libc, you're not going to find any significant or any useful difference in such  benchmarks. Most of the benchmarks which are now available on the Internet show a noticeable  difference in compression performance. For example, they use Gzip or some other libc  utility to compress a huge file on Linux and they do it again on 3BSD and then show, for  example, that Linux performance is a little bit better or the 3BSD performance is a little  bit better. This, for example, relies on how the libc was compiled itself. For one time,  3BSD didn't use an assembler optimized part of the compression algorithm because there  was a security bug in it. So libc was compiled with C only code, which had a lower performance  than the assembler optimized version. So this is the cause of a drop in such benchmarks.  If you want to, you can test the quality of libc and lib implementation, for example,  you can test the quality of the IO, SDIO set of calls or something like this, but then  be sure, you know, what are you benchmarking? You're not benchmarking, for example, the  kernel in this case. You're benchmarking the algorithms in your libc. You can also test  your compiler. 3BSD has a sort of old-age compiler by default. This will change in 9.0.  And just as an illustration, you can find, you can find benchmarks like this floating  around the internet where you have one side GCC and recent versions of GCC as well. And  on the other hand, you have LVM, C-Lang compiler. And so, and for example, this particular benchmark  shows huge difference between C-Lang and GCC. What this picture doesn't show is that  there are other benchmarks showing completely different values. So in this case, it is mostly,  you can run benchmarks like this. There's no problem in running benchmarks like this.  But you have to know that you are running a benchmark of a compiler. So if you compile,  for example, a compression library or an encryption library with different compilers  on different operating system, it's all right. You can do this. But just realize that you  are not exactly benchmarking, for example, the operating system kernel, you are benchmarking  the quality of a compiler. And sometimes, benchmarking is done in circumstances which  really don't require it. There are a lot of reasons why you can do benchmarks. But there  are also reasons why benchmarking really is not as useful or the numbers are not as useful  as you think. You can choose an operating system based on its features, based on its  community, based on its large number of software, or supported software. You can also say that  it's cheaper to buy another machine and run a slightly slower operating system than to  reconfigure everything. And one conclusion that is also good to keep in mind is that  actually, pre-BSD gets better through the years and it consistently gets more and more  scalable. During the Chris Kenneway benchmarks of 7.x area, pre-BSD has tried to achieve  scalability on 8 CPU hardware. And it succeeded, but that's all news now. And during the 10.x  era, realistically, pre-BSD probably aimed for being scalable or being better than Linux  on hardware containing 32 CPUs. So it's consistently improving. It is very much a modern system  going forward. You need to benchmark, maybe, out of curiosity. You need to benchmark if  you really want to know what's going on in the system when you're planning or budgeting  for a new project, when your boss tells it to. And finally, we get advocacy. And this  is perhaps the most commonly used reason to do, or actually, publish benchmarks on the  internet. So, yeah. During the developer summit, I was talking to some of the other developers  and some of the class administrators, so pre-BSD. And we will try to get up some continual  benchmarking going on. So maybe the next BSD Kenne or something like that, I will have  some updated numbers and I will have some nice performance curve improvements over the  time, over the year of pre-BSD development. I think this is it. The point of this presentation  was mainly to point out some interesting facts about benchmarking itself, about what to do  and what not to do when benchmarking a non-primary system. So I hope it was reasonably interesting  presentation to hear and thank you for staying.
Benchmarking_FreeBSD_0_311.png Benchmarking_FreeBSD_312_455.png Benchmarking_FreeBSD_456_551.png Benchmarking_FreeBSD_552_2231.png Benchmarking_FreeBSD_2232_5567.png Benchmarking_FreeBSD_5568_10847.png Benchmarking_FreeBSD_10848_13091.png Benchmarking_FreeBSD_13092_16475.png Benchmarking_FreeBSD_16476_18251.png Benchmarking_FreeBSD_18252_18323.png Benchmarking_FreeBSD_18324_18395.png Benchmarking_FreeBSD_18396_18443.png Benchmarking_FreeBSD_18444_18491.png Benchmarking_FreeBSD_18492_18587.png Benchmarking_FreeBSD_18588_18635.png Benchmarking_FreeBSD_18636_18683.png Benchmarking_FreeBSD_18684_18743.png Benchmarking_FreeBSD_18744_18827.png Benchmarking_FreeBSD_18828_18887.png Benchmarking_FreeBSD_18888_18935.png Benchmarking_FreeBSD_18936_19199.png Benchmarking_FreeBSD_19200_19463.png Benchmarking_FreeBSD_19464_19655.png Benchmarking_FreeBSD_19656_19931.png Benchmarking_FreeBSD_19932_24527.png Benchmarking_FreeBSD_24528_28307.png Benchmarking_FreeBSD_28308_29171.png Benchmarking_FreeBSD_29172_29279.png Benchmarking_FreeBSD_29280_30671.png Benchmarking_FreeBSD_30672_31619.png Benchmarking_FreeBSD_31620_32099.png Benchmarking_FreeBSD_32100_34223.png Benchmarking_FreeBSD_34224_36575.png Benchmarking_FreeBSD_36576_39743.png Benchmarking_FreeBSD_39744_40811.png Benchmarking_FreeBSD_40812_43355.png Benchmarking_FreeBSD_43356_44123.png Benchmarking_FreeBSD_44124_49271.png Benchmarking_FreeBSD_49272_51023.png Benchmarking_FreeBSD_51024_54143.png Benchmarking_FreeBSD_54144_57599.png Benchmarking_FreeBSD_57600_58559.png Benchmarking_FreeBSD_58560_64595.png Benchmarking_FreeBSD_64596_67667.png Benchmarking_FreeBSD_67668_69431.png Benchmarking_FreeBSD_69432_71819.png Benchmarking_FreeBSD_71820_71903.png Benchmarking_FreeBSD_71904_72191.png Benchmarking_FreeBSD_72192_72383.png Benchmarking_FreeBSD_72384_72539.png Benchmarking_FreeBSD_72540_73511.png Benchmarking_FreeBSD_73512_73559.png Benchmarking_FreeBSD_73560_73727.png Benchmarking_FreeBSD_73728_73823.png Benchmarking_FreeBSD_73824_73883.png Benchmarking_FreeBSD_73884_74080.png
Thank you everyone for coming. I'm Ivan Ronoz and I'm here to talk about benchmarking, which is a hot topic always and you know everybody is excited when new benchmarks come out and everybody is trying to either repeat them or prove them wrong or something. This presentation will be mostly about how to do certain aspects of benchmarking. More advanced users will probably know them already. So this is how to do and what not to do when doing a benchmark. It was operating system. And the thing is I've also tried to do some new benchmarks of previous D which would be kind of extensive and useful for the years to come. The last one, last ones were done by Chris Canaway in 2008. This is the time frame of the 7.x release. It's a bit old. And also there are some benchmarks appearing on the mailing list and on various blogs which measured certain aspects of previous D performance and which are not really professionally done and which paint not so good picture for previous D. And I tried to either repeat them or prove them to be done in a less than satisfactory way. So when I say mostly developers, I mean developers who are interested in measuring performance of certain aspects of their systems or subsystems they are working on. System administrations, system administrators will also find some useful material here. And lastly I'm going to talk for a few slides or a few sentences about how and when to avoid benchmarking because sometimes you don't really need it. Generally, when you're doing a benchmark, that means you have some purpose in mind. Either you want to compare your system to some other system, you want to compare your hardware to some other hardware, you want to compare your application to some other application. It is mostly useless to perform a benchmark on its own. So if you have a number that says that many megabytes per second or that many operations per second and you don't have anything to compare it to, it's just mostly useless. I mean you can compare it to a previous version of a system that's also valid. It doesn't have to be a completely new system or complete new hardware. But generally benchmarking means you have to compare it to something. And the goal of a good benchmark is to make it repeatable, to make it useful for other people to make the description of the benchmark as elaborate or as complete as possible so that other people can repeat it and see for themselves if what you did is good or are their own measurements comparable to yours. So I'm also going to talk extensively of how the benchmarks I did were set up and what it is to actually arrive at the results. Repeatability is always a good thing. Even if you're doing benchmark only for yourself, it means that next week or next month or next year you will be able to create the same environment and repeat your benchmark to check if something says change, for example, is the new version of an application or operating system different or more or less performant. I hope to get some high-end hardware for this, but unfortunately I didn't. So this is all done on to identically configured servers, IBM 1U servers and only a four core CPU without hyper-trading. So four cores is everything you get in this presentation. There's two gigabytes of RAM and four satellites set in RAID 0 because they wanted to assess performance and not really redundancy and reliability. And networking was done on a gigabit embedded NIC which has two ports, one of which is connected to the Internet and one of which is used to connect the back sides of the servers with a really simple wire, so without a switch in the mean in between. Some notes about software versions. I did both previous D9.1, which is the release version, and the previous D10 current, which is the development, the future version 10. This version 10 was approximately two weeks ago, so it is really recent. Of course, the debugging was turned off, witness invariance and such things. For comparison with Linux, I used CentOS 6.3 because it is really probably the most used so-called enterprise version of Linux. I had PostgreSQL 9.2, Blockbench, Bony++, Filebench, and my own bullet cache server. I will describe it later. And as preliminaries also, I'd like to talk about what and how to interpret, to post-process your benchmark results. And this is a picture which is in every college textbook about doing any kind of measurements. So I will also repeat it here. When we talk about measurement results, we can talk about how accurate they are or how precise they are. If they are accurate, they measure the real thing, measure the real aspect of the system you are trying to measure. And if they are precise, they are all clustered around basically the same value, which should be the one true value you are trying to measure, but it usually isn't. So it will be more clear in examples. But it is also related to introducing measurement errors because of domestic errors, those which are influencing our measurement in a way that we measure actually something else. There will be some interesting examples here. So basically, you are thinking you are measuring one aspect of the system, and in actuality, you are measuring something different. And also, random errors, which are basically noise, measurement noise, which can come from all parts of the system. You can basically either reduce it or just leave it and use statistics to establish some kind of medium or average value. Lots of benchmarks also exaggerate in their precision. So you can see some benchmarks on the Internet that says a hard drive has 412 SSD in this case, 412.567 megabytes per second. This is usually nonsense because you cannot actually measure hard drive or SSD performance to this precision. In most cases, the best you can do is measure maybe the first two significant digits. So this will probably be better expressed like 410 megabytes per second. In theory, you could achieve really, really, really good precision like in this case, but in practice, you won't ever. It's also useful to introduce the error bars in the form of either on graphs, doing this small extensions on the graph below and about the real value or express it numerically or something. Basically, the error bars are an expression of the precision of your measurements, the confidence that the measurements are really actually precise or really repeatable in a way that is useful. My favorite example here is measuring hard drive performance because you get a lot of file system benchmarks on the Internet. You just create a RAID array or you just use a single drive and create a file system on it and just run some kind of benchmark on it. This is not exactly the way it should be done because especially on mechanical hard drives, so-called spinning grass and such, you have huge differences in performance, linear sequential read performance or sequential write performance between the inner part of the drive and the outer part of the drive. If you use the disk info utility, which is part of every FeeBSD, it's a part of the base system, you can get a really nice illustration of how performance differs when you go from the outside of the drive platters to the inside of the drive platters. For example, if you just create a file system on such drive and measure file system performance in any kind of utility, I mean really any kind of benchmark utility, it's very much influenced on which part of the drive your file system code places the data. If it places the data on the outside, you get one kind of performance curve and if it places in the inside, you get another type of performance curve. This is a difference of approximately, I think, 100 megabytes per second. Really, it is significant. So don't do file system benchmarking on whole drives, especially not on mechanical drives. SSDs have a different kind of problem. They have a lot of internal structure. The flesh sensation layer basically says that your data, which you're thinking you're writing a particular place on a drive, can actually end up in some other place on the drive and it can be internally fragmented and it can be compressed and it can be transferred in a way you have no influence over. So benchmarking with flesh is a whole other topic. This is also an illustration of a few runs of file system of this info benchmark, which is shown in the previous slide. It basically shows how there's quite a lot of noise benchmarking hard drives. Even though this is this info, so it doesn't pass through a file system. It talks directly to the hard drive via dev slash da0 or some other device area specific path. Basically, it says that there's still significant noise here. This is noise between, for example, 300 megabytes per second and 350 megabytes per second. It's significant. The noise is also present in all cases. If the measurement is on the outside of the platters or on the inside of the platters, it is still present. It is not really something you can avoid. So you use statistics to get an average value. How many people have seen this video? Yeah. Basically, the guy shouts at an drive area. There's a drive area. It's on a data center. The guy shouts at it and his acoustic vibrations are transferred to the drive heads and he sees unnoticeable spikes in latency. So really, mechanical drives can be really, really sensitive. You get visible, visible spikes. This means that if you really want to get scientific about it, you will really need to go to really much trouble to do it. Fortunately, you can always aim for ballpark value. For example, you always want to get values like 300 megabytes per second, not exactly 335.456. You really have to be realistic about your precision, so you can expect on it. So what do you want to do with your matchpad? A hard drive or file system. Basically, you want to create a partition which is somewhere close to the outside of the drive and which is small enough so that the difference in performance is not noticeable between its beginning and its end. This partition has to be of large enough size so you can run meaningful benchmarks on it. For example, it has to be at least two times larger than your available memory because if you run a sequential read or sequential write test, you need to ensure that it's not cached. But since a lot of data comes to file system metadata and you have the reserved portion of file systems which is in UFS approximately 8%, usually you have to do three times or four times the size of your RAM for this kind of partition. And as an illustration, this on the left side, this is my partition. And on the right side, this is again the graph showing the whole drive performance. So when you look at just this partition, the inner, middle and, the outer, middle and inner parts of this partition basically have the same performance, plus minus some noise. And of course, the whole drive is completely different story. So these are the results. I think the blue channel is not working on this projector. This should be a blue bar. Blue bars or in this case, black bars are Linux performance. And the red one is 3BSD9 and the yellow one is 3BSD10. This is a result from Bonnie++. I'm using the file system bandwidth, actually the sequential read and write rates and also the rewrite rate. You can see that the results are interesting and it's hard to tell if you can set much emphasis to the Linux performance which is worse on writing and better on reading or the other way around. What is interesting and which will be confirmed in other benchmarks is that actually there's happened some development between 9.5 and 1.5 release of 3BSD and 10.0 release of 3BSD in such a way that basically rewrite performance is much better. Also, the read performance is a little bit better. It means that some kind of new development has probably happened in the area of concurrency on maybe more fine-grained locking in some part of the system. We need to take off file systems. Basically file systems are kind of a database. You get records which are files stored on a medium. These records are simple but they have associated metadata. They have associated security permissions. They have attributes. They also need to be concurrently accessed whether for reading or for writing. This database really can get quite complex. ZFS is notorious for having a really, really complex layout. It's more compared or more often compared to a database than of traditional file systems. The file system needs to support other features like they need to be reliable. They need to support three operations. Network file systems are also complex in a completely different way. Next I'd like to talk about the blockbench benchmark. It basically is a very small utility but very cleverly written. Its goal is to create a tree of smallish files. It has two types of files. One is a sort of text files of two kilobytes in size or four kilobytes in size. In the other ways like multimedia files or in this case image files which are 64K in size. Generally the sizes are not always the same. They are spread across this spectrum between two kilobytes and 64 kilobytes. The interesting thing is the blockbench is multi-threaded. It creates hundreds of threads. It can create hundreds of threads. The default configuration starts hundreds and tens of threads. All of these threads are divided into several groups. One group creates new files. The other group modifies files. The third group reads files. In this case the third group, the one that reads files, is the largest one. We have, for example, 80 threads reading files and the rest are writing or creating them. It also uses atomic renames for some writes. For example, it modifies a blog file and then it renames it to overwrite the old content. This is important because in FreeBSD much emphasis is given to write looking. If you've seen any Linux benchmarks, you've seen pictures like this. Basically Linux performance in the blockbench is all the way over there, almost a million. FreeBSD performance is way over here, 700,000. It's puzzling. Why would this be? This is all done on UFS. I have no ZFS numbers here. Basically it comes down to how the operating system schedules reading and writing. What is interesting is that there is some improvement between nine and ten versions. To prove it, you can use the very easy, very convenient iterability present in FreeBSD, which is Ministat. It is also part of the base system. We already have two benchmarks. One is this KINFO and one is, or two utilities. One is this KINFO, one is Ministat. It can be used to create quality benchmarks. I basically was wondering what's the difference between nine dot one and ten releases. It says that approximately 90% better performance is achieved in ten, but in the read case, but approximately 17% lower performance is achieved for the write case. It needs more investigation. Blockbench is a multilateral benchmark. It uses a lot of parallel operations on the same tree of files. There's something called the write bias of FreeBSD, which basically means that writes on a single file block all other access. I think it even blocks other reads. I'm not sure. I think it is such. It is useful to remember that the directories themselves are also files. So write operations on the directories themselves are also blocked. I did some system call tracing. Basically, you get a long list of CIS calls. It clearly shows that there are multitude of threads during various operations, opening, grading, writing, closing, and such. From all this data, we can create a pretty picture, which kind of characterizes the number and the frequency of operations. What isn't shown on this axis is these zeros should be operation duration times. They are really very small in the order of microseconds. It got all rounded up to zero. We see from this graph that Blockbench does a lot of what we knew before. Read operations are dominant. Surprisingly, there are a lot of closed operations, which is kind of normal when you say, when you know that Blockbench opens files, modifies them, or reads them, and then closes them again. There's a really small number of write operations, basically this curve over here. It is interesting to note that read operations also take the longest, comparatively, to everything else, which I think indicates that they are actually blocked by other types of operations, but this means a closer investigation. The other benchmark which is often used in 3BSE world is PostgreSQL benchmark. It's a database. In this case, I've used the PG benchmark, which is a part of PostgreSQL source base. I've initialized the benchmark database with the scale factor of 1,000, which means 100 million records. The database is approximately 16 gigabytes in size. It fits in RAM. This is deliberate because I'd like to test, for example, from currency in database access without really benchmarking the hard drive at the same time. The configuration is, of course, stable during the whole test. These two machines, I was talking about at the beginning, are configured so that they dual boot Linux and 3BSD. Both Linux and 3BSD have access to the same partition. The same file system is used for both Linux benchmarks and 3BSD benchmarks. This is the partitions I was talking about at the beginning, so that one with small deviations between access performance on outside, middle, and inner tracks. PostgreSQL is a fairly modern database, which means it has lots going on in the back end. The most important thing is the right logging. Basically, if you do a right operation in PostgreSQL, the data doesn't go directly to the database store itself. It is written in the right-hand log first. Periodically, this data is then transferred from this right-hand log to the database storage itself. The problem is that this can happen at unexpected times. For example, if you run a short benchmark, you can run the whole benchmark with the data residing in the right-hand log and not in the database storage itself. You can run another benchmark, which is started, and then right in the middle of benchmarks, this transfer occurs between the right-hand logs and the proper storage. This introduces a large amount of noise in your benchmark results. The benchmark runs need to be longish at least five minutes, 10 minutes. There are recommendations for even longer times, like half an hour. It is crucial that the PostgreSQL server itself, the daemon itself, be restarted between the benchmarks because a restart of PostgreSQL causes all data to be transferred from the logs, from the right-hand logs, to the data storage itself. You get more consistent, more precise measurements. Also, it is necessary to turn on auto-vehicle. The vacuum in PostgreSQL is the part of the system which basically cleans up data in case of updated records and deleted records and calculates some statistics information, which can also influence your results so they can be scattered or very noisy. On this particular machine, which is the quad-core Xeon I was talking about, I run benchmarks which are read-only, which are read-write, and also I run benchmarks which are on the file system, meaning on the drives, on the drives themselves, on the disk array, and on the memory file system, which is a ramfs in case of Linux and tmpfs in case of gpsd. I've also done some remote benchmarking, meaning remote connection to the database from the other server. I will talk about some of the results in more detail. So, on Linux, I discovered that there is almost no difference between fully cached database on X4 in read-only benchmarks and its ram file system or memory file system. On 3BSD, using tmpfs is faster than using UFS, which is fully loaded or fully pre-warmed database content. The blue lines are basically Linux and the red lines are 3BSD. This dip here is very surprising. It is also very repeatable. I get it in all configurations in all benchmark runs. I have a theory why is it happening, I'll try to explain it later. What most people are interested in right now is that on this particular machine, I get lower performance of 3BSD than of Linux, except for this dip here. I don't know if you can see it, but there are two lines here, the dark blue one and the light blue one. The dark blue one is disk file system, and the light blue one is a memory file system. On 3BSD, the dark red one is the disk file system, UFS, and the light red one is tmpfs, the memory file system. There is a noticeable difference here. These are the right results. Again, the Linux curve is the blue one and the 3BSD curve is the red one. Even though this particular benchmark should really depend on the disk performance, apparently it doesn't. The opening system has a huge influence on the total performance. This graph shows benchmarks of remote access performance. What I did was use the same servers. One was obviously running the database, and the other was running the benchmark client. The database was always on memory file system. This is the distinction between local access, which is the full lines here, the dark blue one and the dark red one, and the remote access over here, which is the light blue one and the yellow one. We can see that remote access helps a little bit, because the benchmark client itself is a CPU intensive process. When separating the benchmark client and the other machine, you'll see some resources on the server itself. This is offset almost entirely by the fact that there is a TCP connection between those machines, even though it's just a wire between the two metro cards. The same general conclusion can be drawn that 3BSD still has some work to do in performance improvement. Because we can now separate the client on another machine, I think that this database is actually caused by some kind of scheduler issue in 3BSD. In the original case, you have both the client and the server processors on the same machine. Somehow, if approximately 12 concurrent clients, Linux has some kind of a problem, I think. My best guess is that this is a scheduler issue in Linux. However, to get some more optimistic news, these are the results done by Florian Smith and I also think Jeff Robertson on a hugely different machine. This machine has 40 cores. It has four CPUs, 10 cores each, and 80 threads in total on the system. The results are much better on such a configuration. Basically, in this configuration, 3BSD is consistently better than Linux, so there's some hope yet. The algorithms used in achieving large scalabilities probably favor systems with a large number of CPUs, so something needs to be done to maybe close that gap. Linux, which is blue in this graph, lacks in performance up until approximately 32 clients or 32 connections to the database server. This is also PostgreSQL 9.2. This is also a benchmark of local benchmark client and the local server on the same machine. Next benchmark I would like to talk about is File Bench. This is a benchmark done, I think, by some microsystems back in the day, and it has a large number of profiles. One profile, for example, is a file server which basically creates a large file, and the other is the proxy profile, which creates smaller files. The different benchmark operations being done on these profiles. Unfortunately, I think that this benchmark has some problems running on 3BSD. I have some strange results, I'll talk about them later, and I'm not so sure that it's really a correct benchmark. I'm not sure that these results I will show you are really correct on the 3BSD side. I've done local draft measurements on UFS, on this partition I was talking about, and also NFS 3 and NFS 4, because there was some talk on the mailing list about the huge performance difference between NFS 3 and 4 in 3BSD, so I wanted to find out what's happening. This is the file server profile. The results are strange. On the one side, we have Linux, which is this huge dark bar here. The result is larger than 800 megabytes per second, which is obviously due to caching, which is all right, or actually it would be all right if it were not for 3BSD results, which is this red bar here, which is almost 100 megabytes per second. The drives themselves can pull 450 megabytes per second at least, so I think that the huge difference between the Linux results and 3BSD results are due to benchmark problems. This is an example of maybe a systemic error or systemic problem in benchmarks. I've run NFS 3 and NFS 4. These are the two bars right here, the light blue one and the dark blue one. The light blue one is NFS 3 and the dark blue one is NFS 4, and even on Linux, NFS 4 is significantly slower for some reason. I was hoping to talk to Rik, but I think he's not on the conference right now. The 3BSD results are lower than Linux results. I really cannot explain why local disk performance should be so low, because this is much lower than the hardware itself can do on this machine. The NFS result is again significantly lower than the NFS 3 result. The NFS 4 was mounted over TCP. On the other hand, we have the web proxy profile, which again shows really quirky results, but a bit different. This time Linux has lower performance on the local hard drive measurements, and 3BSD has huge better performance on the same conditions. Again, it looks to me like the benchmark itself is wrong. The benchmark itself seems to be either it has a bug or is Linux specific and there's some problem with porting it to 3BSD or something. This is possible, but not really probable that such results would happen. This just goes to say that you need to choose your benchmarks carefully. As an interesting side result, this is also the same file benchmark on the file server profile. I've used 3BSD as an NFS 3 server and Linux as the NFS 3 client, and shows that the results are basically the same. Which is encouraging. Last year I talked about my bullet cache server. It's a memory cache server similar to memcached. It creates a TCP traffic of very small transactions, like 22 byte messages going back and forth between the client and the server. It is heavily multithreaded and also uses non-blocking IO. If you benchmark our unique sockets, meaning both the server and the client are around the same machine and talk about, talk to you in the main sockets, you get easily 2 million transactions per second on mid-range hardware. The results are in favor of 3BSD on this particular benchmark, but the benchmark itself was developed on 3BSD by me, so I used a lot of tuning for this to actually happen. I also would not advertise this as a truly good benchmark with Linux and 3BSD. It's more of a curiosity. Between measurement errors, the same performance is between 9.1 and 10 releases. The benchmark is interesting because it illustrates that 3BSD's TCP stack is fairly multithreaded and fairly finally locked. TCP transactions, TCP streams do not influence each other. Multiple concurrent streams can be used on 3BSD without performance problems. It also shows that I got, for example, nearly 500,000 backers per second per direction in this configuration. This is a lot of really small TCP messages. I'm satisfied with the performance shown. To elicit the difference between TCP over the plain internet via connecting the servers and Unix Domain's sockets, Unix Domain's sockets performance crosses 1 million transactions per second. This is also very naturally expected between on the other one. On one hand, you have the whole internet channel. You have the whole internet actual networking happening. On the other hand, you have only memory passing between processes. One thing that a lot of people are trying to figure out is what to tune on 3BSD to get acceptable performance. I would argue that especially with release 10 and even the now current release 9.1, there is not much you can do or actually the system auto-configures itself pretty well. We had an increase in really head tunable recently. Also buff space, high buff space and low buff space. They are now auto-tuned. You don't have to touch them. Some tutorials advertise increasing the network transmit and receive descriptor rings or increasing their size. I haven't actually noticed some significant difference on my previous benchmark. There is some difference. I'm not so sure if this is because of measurement noise. I need to do some more benchmarking to actually find out. But the point is the default values are actually pretty good. I also try to increase the number of interrupts per second allowed on the EM driver, on the CPUs. If I increase them, I get also very, very small improvement, maybe less than 1% of performance. Unless you are really trying to squeeze out some percent performance from your machines, you don't have to touch the, for example, network driver configuration. Current.max users governs basically the size and number of internal network, internal kernel structures. It's been auto-tuned for years now. For the last example, this tunable, which is maybe familiar to everyone who had run Apache or PostgreSQL or something else, which has a lot of processes, networking is also gone from 9.x releases. My point is you don't have to rely on manual tuning as much as you used to on 3BSD. I didn't give any CPU-specific benchmarks. By that I mean I didn't do any encryption, I didn't do any kind of compression benchmarks and such, because for the most part, they are trivial. Unless you have a broken compiler or unless you have a different tuning in your compression libraries or different algorithms in your libc, you're not going to find any significant or any useful difference in such benchmarks. Most of the benchmarks which are now available on the Internet show a noticeable difference in compression performance. For example, they use Gzip or some other libc utility to compress a huge file on Linux and they do it again on 3BSD and then show, for example, that Linux performance is a little bit better or the 3BSD performance is a little bit better. This, for example, relies on how the libc was compiled itself. For one time, 3BSD didn't use an assembler optimized part of the compression algorithm because there was a security bug in it. So libc was compiled with C only code, which had a lower performance than the assembler optimized version. So this is the cause of a drop in such benchmarks. If you want to, you can test the quality of libc and lib implementation, for example, you can test the quality of the IO, SDIO set of calls or something like this, but then be sure, you know, what are you benchmarking? You're not benchmarking, for example, the kernel in this case. You're benchmarking the algorithms in your libc. You can also test your compiler. 3BSD has a sort of old-age compiler by default. This will change in 9.0. And just as an illustration, you can find, you can find benchmarks like this floating around the internet where you have one side GCC and recent versions of GCC as well. And on the other hand, you have LVM, C-Lang compiler. And so, and for example, this particular benchmark shows huge difference between C-Lang and GCC. What this picture doesn't show is that there are other benchmarks showing completely different values. So in this case, it is mostly, you can run benchmarks like this. There's no problem in running benchmarks like this. But you have to know that you are running a benchmark of a compiler. So if you compile, for example, a compression library or an encryption library with different compilers on different operating system, it's all right. You can do this. But just realize that you are not exactly benchmarking, for example, the operating system kernel, you are benchmarking the quality of a compiler. And sometimes, benchmarking is done in circumstances which really don't require it. There are a lot of reasons why you can do benchmarks. But there are also reasons why benchmarking really is not as useful or the numbers are not as useful as you think. You can choose an operating system based on its features, based on its community, based on its large number of software, or supported software. You can also say that it's cheaper to buy another machine and run a slightly slower operating system than to reconfigure everything. And one conclusion that is also good to keep in mind is that actually, pre-BSD gets better through the years and it consistently gets more and more scalable. During the Chris Kenneway benchmarks of 7.x area, pre-BSD has tried to achieve scalability on 8 CPU hardware. And it succeeded, but that's all news now. And during the 10.x era, realistically, pre-BSD probably aimed for being scalable or being better than Linux on hardware containing 32 CPUs. So it's consistently improving. It is very much a modern system going forward. You need to benchmark, maybe, out of curiosity. You need to benchmark if you really want to know what's going on in the system when you're planning or budgeting for a new project, when your boss tells it to. And finally, we get advocacy. And this is perhaps the most commonly used reason to do, or actually, publish benchmarks on the internet. So, yeah. During the developer summit, I was talking to some of the other developers and some of the class administrators, so pre-BSD. And we will try to get up some continual benchmarking going on. So maybe the next BSD Kenne or something like that, I will have some updated numbers and I will have some nice performance curve improvements over the time, over the year of pre-BSD development. I think this is it. The point of this presentation was mainly to point out some interesting facts about benchmarking itself, about what to do and what not to do when benchmarking a non-primary system. So I hope it was reasonably interesting presentation to hear and thank you for staying.
will be mostly about how to do certain aspects of also there are some benchmarks appearing on the to avoid benchmarking because sometimes you don't of a good benchmark is to make it repeatable, to embedded NIC which has two ports, one of which is measurement results, we can talk about how And also, random errors, which are basically noise, measurement noise, which can come from all parts of the system. You can basically either utility, it's very much influenced on which part on the inside of the platters, it is still kind of partition. And as an illustration, this systems. Basically file systems are kind of a in the blockbench is all the way over there, What is interesting is that there is some convenient iterability present in FreeBSD, which file system is used for both Linux benchmarks and talking about at the beginning, so that one with storage. This introduces a large amount of noise get it in all configurations in all benchmark try to explain it later. What most people are systems with a large number of CPUs, so something because this is much lower than the hardware It is heavily multithreaded and also uses this particular benchmark, but the benchmark benchmark is interesting because it illustrates especially with release 10 and even the now different compilers on different operating benchmarking the quality of a compiler. And