Static analysis, test coverage, and other 12+ letter words
Eisentraut, Peter
It is hard to maintain complex C code free of bugs. Numerous tools have been developed to help, and we have thrown many of them at the PostgreSQL source code. We will take a look at tools such as Coverity Clang scan-build AddressSanitizer Valgrind Gcov How do they work? How should their results be interpreted? What have they and have they not delivered for us in practice? How can we integrate them into our workflow going forward? How can we structure our code better to work with tools like these?
 OK, is the recording on? Microphone? Microphone? OK, hello, good morning. I'm Peter. I'm PostgreSQL developer, and this is the story of my life. So I want to talk about various tools that I and others have thrown at the PostgreSQL to find bugs and other issues. I'm glad with this room, because the other one was so dark that I was almost falling asleep. So let's start with something simple. Have you ever written a C program, like a one-off, to just do a test case, and you compiled it? And it seemed to compile fine, and you ran it and crashed immediately. And then you noticed you did everything wrong, because the compiler didn't warn you about any of the problems. So if you have done that, then you will know that without compiler warnings, a C compiler is worthless. So that is why we in PostgreSQL have put a lot of effort into both cleaning up existing warnings to the point of 1 or 0, depending on how you look at the flex issue. And continue to add new warnings. So this is the current set we're using by default. And if you just look through, the WL is obviously the big one, but if you look through all the other ones and you imagine how your life would be if you didn't have those, it would be quite sad. So that is the set we're using, and we're pretty comfortable with it. This is, I'll be on the screen here. This is actually what I tend to use in my own builds. So there are a couple groups here. The first two, or the first one is not really a warning option, but what it does, it's on glibc slash Linux. It changes the definition of several library functions. So they declare themselves that you shouldn't ignore their return values and things like that. And that, in combination with the other warning options, then gives you warnings if you missed checking the return code of a function, for example, which could lead to security problems. Yeah. How will we play in that? How will we play in that? Well, how much courage do you get? Well, we are currently not getting any garbage because of two reasons. One is I always have this on in my own build, so I would find it pretty soon. The other is that on Ubuntu, they have somehow hard coded this into their compiler, so they're always building this. So the original reason why we actually started cleaning up is that we got these reports from Ubuntu people. So I must be the subject of the calls that the covariate complaints are about. Yeah. We'll get to that. We'll get to the covariate in a moment. So logical op is also something I'm using. That finds issues like if you have an if statement life. If x is greater than 0 and x is less than 5, that would be a type of it. It would find that. So that's actually pretty useful. It actually has found at least one bug in Postgres a few years ago. I actually submitted patches to make both of these a default awhile ago, if you recall. But the problem is that in very old GCC compilers that people are still using, they produce a lot of garbage. Because I guess the older compilers were not as smart to produce useful. Is this just Tom's bug again? Is it what? Is this just Tom's, you know? No, this has nothing to do with that. Those were reasonable. I don't remember the details, but those were reasonable platch ones. They were just a few years older. And this is the story that is a problem with all of these. You want to add these other ones. The second group there is things that I sometimes also use. They haven't really found much. So you could use them. The third group there is something that I use with Clang specifically. And the ones below there are new in Clang 3.4. So they haven't really found anything yet. But they are also sort of on the horizon, on the road map, so to speak. But the problem is really you can't make these the defaults for the reasons I mentioned. Because different compiler generations produce different nonsense for that. So if you want to maintain what we have, you know, the zero count of the warnings, it's really hard to move forward. That's a kind of annoying issue. So it's going to, you know, people who are interested in this will basically have to do this manually going forward also. So you have to do a lot of work to make that. I think a better plan would be to have a built-in member or something like that. I don't want to spend too much time on the warnings. But yeah, there's obviously a lot of fiddling we could do to make this better. But the real problem is the built-in member doesn't tell you how to do it. Yeah, well, that's another issue that many issues here. That's good. All good notes. So one thing I always do is compile with a W error, which turns all warnings into errors. And I have done this successfully, so to speak, for almost five, eight years. So if I compile code and it flies by, I don't see the warnings. I need to have a boolean answer from the compiler. That's just my workflow. The problem is if you pass W error into configure, configure fails because it generates warnings internally as part of its testing. And the workaround I use, so to speak, is I have a shell alias that does this sort of thing. So if you want to do this, you can take a photo or look at the thing later. This works for me because actually the shell alias I use is much longer because it also picks different options for different branches and so on. But that, for me, is useful. And I think everyone else should look into that because it's really helpful. The problem right now is I'm always the first one to find the warnings if someone accidentally commits one. But I'm happy if I've chosen this path. Here is a chart I computed over the weekend, which caused this computer to heat up my office for a while as it was building all this. And this kind of shows the progress of compilers becoming more and more either smaller or annoying, depending on how you interpret it, and also us catching up with them. And you can sort of see a diagonal here as time progresses. The problem with this is going back to the W error issue is if you need to make a change, let's say you need to back patch something into 9.0 right now, you need to either very carefully look at the warnings or not that your compile generates, or you need to pull up this chart and then actually try to install GCC 4.4 and build that because that was current at the time. That is very annoying and pretty much an unsolved issue for working in back branches. So I don't know how to fix that, other than keeping a lot of old compilers around, which I essentially do. But not all platforms have the option to install all that. There's all things. It's kind of hard. In case you're curious what this is, a lot of the options in 8.3 was that the compiler was complaining that inlining failed, and that generates hundreds of warnings almost, and then we later turned that off. In GCC 4.6, they added a variable was set but not used, which apparently, it was good. Now the issue with this is a lot of developers built with assertions enabled. But if you turn the assertions off, you get a different set of warnings because sometimes the warning variable is used. Sometimes those sort of problems don't get detected for a long time until someone builds without assertions. Very difficult. Yeah, we mentioned already, Buildform doesn't do anything about warnings, so this doesn't help. But when the other talk, Rob Pike was mentioning, who is also one of the creators of Go, the programming language, which you can have different opinions about. But one interesting thing they did, they don't have any compiler warnings. It's either an error or not, which makes this sort of thing much easier, which I think is a good decision. So now we were talking. Yes? How soon I have to know what's the 36? Yeah, OK. So the 36, this is on Mac OS 10, version 10.9 specifically. Those 36, if you're on a Linux platform, you might not get those. This is a, I argue it's a bug. I filed a bug with GCC about it. Complains if the right-hand side of a comma operator does not have a side effect. OK, OK. And this is a new warning they introduced. And this happens in the system house of the operating system for various macro expansions. And they're not very smart about that. So you might not see this, but if you're me and you develop on this 50% of the time and you run with W error, then this is GCC 4.9 is broken for me at the moment. So I actually use 4.9. Yes, of course. I've never been able to get some of your other work. So you've always come up with various errors and some serious errors. So I would kind of grab the errors and then just go to the end of the line. And then print all the error messages. I know a lot of people have. I've seen in discussions that people have macros or things that do that. In my experience, it always works. Or if not, I fix it. But that's the other thing you can do also. I think that's a good point. Well, I do wonder. I mean, we could turn on the error for both forms. It is kind of, yeah. Obviously, there are obscure platforms where it is not zero at the moment. Because we're actually going to be able to limit GCC more on so much better. Yeah. Well, we're also pretty good on Clang. But of course, if you use, for example, the Solaris, whatever it's called, Sun, former Sun Studio, you get a bunch of different things. And it's not easy to turn them off. They don't really have these options. So on those kind of platforms, you pretty much screwed at the moment. Let's talk about severity. So the way the compiler options are designed is so they don't slow down the compilation so much. So the next level of intensity here, so to speak, is the static analysis tools that are specifically designed to produce essentially as many warnings as possible at the expense of being super slowly built. And the severity was kind of the first that at least came to our attention on a broad scale. So the severity came out of a research project, but it's a commercial company. They sell this product for a lot of money I hear to do statically analyze your C or C++ code and started doing that. In around 2005, they started this open source program. I think that was also somehow government sponsored and built a bunch of open source projects and made this publicly available. And they have expanded this platform over time. And now it's almost to the degree that anyone can sign up and submit builds there. But the results are still under some kind of license. So I can't really go into too much details without not knowing what I'm getting myself into. Are you associated with that company? We'll have another example, which is the Clang Scan Build, which is open source. So if you can drill into deeper, they're very similar. Just a clarity goes a little bit deeper. So first of all, we have kind of made the decision that the only people who should have access to this is the people who are also on the security team, because there could be security issues in there. So if you're not on that team, what you can do is submit your own project to Corrarian. Just download the Postgres code. And if you figure that out, you have the results yourself. That's homework. But if you're one of the circles, so to speak, and you want to have access to this, you can talk to Steven Frost, primarily. But I'm also an admin on this, and Magnus is also an admin on this, so we could help you out. This is the URL. You can go. I think we'd be, we want to get to a point where we actually have a whole thing cleaned up. I think we'd be more willing to go. As you can see, there's a lot of things. So I found in the commit log 71 issues mentioned. Well, that's the debate of, well, it's hard to have that argument about a 1,500, right? It would be more easy to have that argument about a 150, right? So we don't know. We have been through, yes, between you and myself and others, we have been through a lot of those. But so, yeah, I'll show an example in a moment of what that looks like. I found 71 commits, which doesn't mean there was 71 issues. There were probably more in the older of 200 or 300, because a lot of these commits bunched a bunch of things together. So a lot of those are in the vision of memory leaks or memory access problems. So as it says here, there are 1,501 issues outstanding. Again, we have disqualified most of those, but who really knows? Again, if you're one of the developers, you can get access to that if you want. Coerity put out a notification that they detect a heart bleed bug, which was a recent security vulnerability. So they added that a week after it came out. So that didn't help. But that's the sort of level they work on. So they do taint checks of where data comes from and warn you about if you're using data that you get from external sources into a good way. Problem with that in Postgres, we actually do that a lot. If you think about what Postgres does a lot is read the block from disk and trust it implicitly that this has the right structure. And then just assumes, oh, if there's a length right here of that, we assume there's that much. And if that doesn't work out, it's just bad things happen. So essentially, we trust whatever's in the data directory implicitly. So all these taint checks don't really work for us. Which there's probably a lot of taint checks amongst those 1,500, but who really knows if there's really one that does the problem? Another example of this is if, for example, PG-based backup, there was an issue out of that store there. If PG-based backup pulls down incorrect data from the server, you can make it crash potentially by having a division by 0 somewhere, as possible. It's not a huge issue if your goal is not to make PG-based backup crash. The opposite of that would obviously be a much bigger problem if you send incorrect data up to the server and the server crashes because you send 0 somewhere. But they can't really tell what is more important than what. So that's hard. Let's see if I can pull up a web browser here on this. Where did it go? Oh, there it is. Cursor is there. Now it's here. This is very exciting. So this is the dashboard you get here. It's a little bit smaller. The back branches don't seem to really work because they only find one defective. Well, the actual interesting thing is this one here, where you can go through all the problems here. And the screen is a little bit funny now. This is interesting. I just looked at this the other day. So this is, you have op here compared to this. And then down here, you compare to that. And there's nothing in here that changed that. I just saw that last night and it looked kind of funny. So that doesn't. I would pick so. I didn't. I seriously didn't pick that. I just looked for an easy example. So arguably a compiler could detect that also because it's pretty straightforward, but obviously it needs more time to analyze that. So this is the level of issues you got here. So we will get him an account. It is pretty cool. You can waste a lot of time in there. So the open source alternative to that is Scanville, which came out of the. I'll show you later. Which came out of the LLVO. Stephen, you want to explain how queries built in your basement and then it's uploaded. No, no. OK. Hey, it's not the same physical box that I could pick from. So I'm going to go on and talk to you. No, it's not. I'm just saying. So it is basically someone in the infrastructure team regularly builds Postgres and then you upload some binary blob to them and they analyze it. So it is. Now we actually have more resources in terms of VM space. So Scanville is kind of an open source alternative that came out of the LLVM project. This is the URL. So if you want to use that, here's a recipe. So you can try that yourself. Most reasonably up-to-date platforms should have this inner package that might be called LLVM or Clang. Yeah, I've heard that the recent build is from the recent version of the stream build that introduced the number of the two-lane fault-pot. It is, yes. It is. It seems like recent versions of the bugs. It is an aggressive development. So you have to kind of, they do stuff and you do stuff in parallel. So yes. That's how I use it. It didn't have the built-in effect that was also a lot of error. And then you put an abort on that next thing. And you also don't figure out the effect. So we'll kind of do that. But it's not like that. OK, let's talk about it in a second. Yes. So if you want to build that, you download that tool. The O is basically where it stores its output. You configure it, you ought to build with enable assert. And then you just build it. It takes between half an hour and an hour or so. Then it tells you how many bugs it found. And then what is it? Oh, I'll explain in a moment. You don't have to, but it will not be. It won't work too well. And then there is this tool called ScanView, which pulls up the result in a HTML report, similar to what CorelD just showed. So let's look at it. So here's the result. As you can see, here are the bugs. Let's look at fancy JavaScript here. Let's look at 1. Division by 0. So here's all the places where you have division by 0. So you can see all the Q sort will fail if you pass the size of 0. That's by design. So you can't really do anything about that. The print routines will fail, for example, if you print the result set with 0 columns, which you wouldn't normally get, but as you know, we can fake that with tables with 0 columns. But the reason why this is not a problem is that tables with 0 columns also have 0 rows, so no row is actually printed. So if you could fake up a result set, you could crash this. But in practice, it's not a problem. The fix in this whole situation is that you put an assert somewhere in there that will just say, assert column count is greater than 0. And then you'll still crash, but in a more controlled way. And this tool will know about it and those things go away. That's why you need to build the assertions. Problem is, in the back end, you can turn the assertions off at runtime. So none of that will work, because the code path analysis always has to analyze the path. Well, what if assertions are turned off? Then I have to look at this path here. So that just doesn't work. So this tool doesn't know about that. Corverty has somehow, they said they had sort of manually tweaked something so they know about that. No, we have that problem. Maybe that used to be a while ago. They have the same class of problems in any case. So, sign values go up a bit on the front. So, Bison generated code apparently has lots of problems that you don't know. Hard to argue about that. Here is something in parser. And see, it gives you kind of this path here. You can go look, see how this comes about. So this is this function, which gets in a list of arguments. And then, no, this is all. Well, and then, you know, so this goes, loops around here. And assigns the arguments of the list to this array, which is this fixed size. So if you pass in the list as longer than that, you will have problems down here. This will not actually fail in practice, because like three levels up, there's a check that you can't, you know, on a more semantic level that you can't pass in anything longer than, you know, from MaxArk. So, right. So the fix again here would be to pass in an assertion, to put in an assertion. But the problem is, this will fix that. But as you know, this is not actually the problem it's complaining about, right? This problem is down here, where it loops through that same thing again, and assigns it back into a list. But it has no knowledge about lists, so it doesn't know that by exercising this loop, you do a number of steps of something. It doesn't know that at all. So it would have, even if you put in an assertion at list length or something, it has no knowledge of how these structures work. Because all the lists are in the user space, so to speak. I don't think there's anything you can do here to. It doesn't know that that's a list that has properties of any kind, so that's hard. Let's do no one. Do you assert it on the list? Do you specifically know what the assertion is? No. If you just assert, it will know. It knows what assertion does. Basically, this code path is not relevant. So Unix API, I had something in the Swift logger. This is a common problem, passing a null pointer to a function that shouldn't get one. And if you trace this all the way back, this value is assigned as a result of a PL lock. But this tool doesn't know that PL lock never returns null. So that might be something we can instrument somewhere. But I haven't figured out how to do that. But that would possibly help in a lot of cases. And there are GCC also has some attributes you can put on functions that tell you that this argument shouldn't be null, but I don't think they have an attribute that tells that this function never returns null. So that's actually what we use in a lot more cases. Assert would help here again. And there was a question about what about the E-log issue. And we added, she also had no return attributes to functions that also prune a lot of potential code paths and reduce a bunch of issues. So is there anything that you found that's like the thing and then gets a nine? But anything that actually understands what we do when we do the rare error signaling? And how we do the send up and whatnot? Or is it just null? No, it's not. It's not null. That's the only way to get a six-prone. No, I think you just do the null return. Because what I actually have to trace through and then see that is a long jump in some cases. But it potentially could do that. But I don't think it digs that deep. That's I think the problem. Because you have to have a trade-off of how long this thing is going to run, I think. I haven't got any tools to do it. I'm just wondering. Is that one of the big down-plates of all these tools that none of them understand that? So it's kind of a tender for them to do it. So this is the number of reported bugs we've had over time. And at some point, in 9.3, I think, is when we put in a bunch of null returns. And as you can see, that helps a lot. Pre-requisite for this is that you get clang to build it out warnings. Because it's kind of the same foundation. So a lot of the old releases have just basic clang bugs that we didn't attend to at the time. The there's no particular reason why 9.4 has more. I was kind of surprised by that. But there doesn't seem to be any systematic change there. It's just certain unrelated changes have potential issues. But it doesn't seem to be serious. The star column there is kind of a branch I have made myself where I have made assertions that you can't turn them off. So it prunes certain code paths and did a few other possibly controversial tweaks to kind of see how it can get that number. Ideally, we'd like to get that to zero. Because again, this comes back to the warning versus error issue. If you always have to compare numbers, you're never going to get really a bullying answer. But as was mentioned, scan build is still in development. And things come and go. So it's kind of hard. But this is a useful tool that everyone can use, open source. And you can play around with it. So Richard, you're using that apparently as well? Yes. The older versions of the new ones. Yeah. They're a lot easier. So yeah, I should have, if I had more, if I would have wanted to stress this laptop more, I could have done this for N releases. And it would probably take many days to build. So another static tool that came out of LVM is the address sanitizer, which actually came out of Google for their Chrome browser, I believe. But it's now part of Clang Compilo and GCC 4.8. You can also use it in GCC. So the way this works, you give it this option sanitize address, this one is the key one. And then these are recommended to get better backtraces, but they're not the key here. And then you build it, you run whatever check test suite you might have. And if there's a problem with whatever this checks for, which is mainly buffer overruns, then it will crash at this point. And then it will print to standard error a cryptic backtrace. And then there's this script that you can run this backtrace through, and you will get a backtrace. I can show an example. What is it? I don't know what these do. They're just recommended to get better backtraces. I've never actually seen that. This is a Clang option, so I don't think you might not have seen this in GCC. This is more well-known. Now, I don't know what this does. It's in the recommended set of options. So one issue we found, this might look familiar to some, is this issue in a non-issue. That's why this was ignored for a long time. If you run the name, it will do eventually a stir copy of these two things. And there is a certain case where these are the same. So you would think that's fine, because it just copies it over the top of itself. And for the last 20 years or so, that did work fine. But this thing was founded for strictness reasons. And it was ignored for a long time. But then some new compiler in MacOS 10.9 actually complained about this or crashed here, just as part of the regular compilation, so at that point, it needed to be fixed. And then every scientist was happy. And once you fix that, this is a real issue that we found. Here, you get the user like rule name, which is a string constant or a C string. And you cast it to a datum, pass it to this prepared plan, which expects a name datum there. And in most cases, that's fine, because the name is just a 64 byte string. But at some point later down here, this string gets copied into another copy of itself. And that will then copy the full 64 bytes, whereas this is not 64 bytes long. So that's a confusion between C string and name. And so this could have been a real problem when someone is actually at the end of memory. So this was useful, and this was fixed. So the problem with address sanitizers is just whenever it finds the first problem, it crashes. So you need to fix that, or at least work around it before you can get to the next point. So how do you do that? I don't remember the name. OK, I didn't find. Well, obviously, you can just comment that out, and you can pass by, right? So it's just for analysis. But it's a. I'm not familiar with that. But. I mean, what would anyone want to add? This is the. But I know you have a lot of crates to do it already. So this is the current issue. This is as of now. Which is actually kind of weird, because it will work fine on 32-bit builds, but it will do this crash on 64-bit builds. And this was discussed on the main list a few months ago, and it was actually argued that this is not a real problem. So you don't have to worry about it too much, but it will crash this way if you do it right now. So it would be worth fixing at some point. Are you similar to what I've seen in Valgrind? So yeah, so Valgrind has received a lot of attention in the community lately, and it's a much more general tool. Actually, on the website, they now described themselves as it's a just-in-time compiler, a pluggable just-in-time compiler. So it can do a lot of things now. But obviously, it's more known as a memory-checking tool. And yes, you can also use Valgrind. They, all of these tools overlap in some degree. The covariate can check find memory leaks. If it can reason about it statically, Valgrind will find memory leaks just because it keeps track of it. And it's more guaranteed to have a good answer. So the important thing, I would like to do more with Valgrind. And I think a lot of people here are of the same opinion. You can do a lot of things. So here's a couple ways you can use it. This is, I think, something that Hakey posted just the other day as an example to do back-end work. Something I've looked at. So here's a wiki page that was put up recently to also cover some of that. One thing I'm kind of interested in is also memory leaks and front-end programs. For example, in PGDump, memory leaks could be kind of a problem, but are not well tracked at the moment. If you actually do this, you will get a memory leak. But as you will see, it's kind of a stupid one. This is what you get from PGDump right now. But as you can see, the memory leaks are in undisputed vacuum. We just had the same problem. So yeah, here it is. So here, this is every program in Postgres, or at least client programs, this problem that allocates a few strings for local services, but you know, the program then doesn't free them. So any time you do this, you will get this sort of trace. And as we know, we can add suppression files to get rid of those, which I would like to do. Basically, you get all of these things down to zero. And then be able to run test suites again with a Boolean answer at the end, whether there are any problems or not. But we got to fix some of those first. So this was specifically a leak check. A lot of the other work that has gone in in the back end is access to memory that you shouldn't access of various kinds, which is what this down here would find. There's been a lot of work on that lately. I only found actually 19 commits mentioning ball grind. So we haven't really done a whole lot, but there was a lot of activity just lately. So one of my goals is to basically be able to run this automatically and just give you an exit status at the end. We look very concerned, Greg. Last time, no, last time it was ball grind. It's a old, curvy runtime. It wasn't static. Yes, this is, yes, I should have made the segue better. So the, uh, Is that so? I haven't had that problem, really. You would know a lot about running things on smartphones. So I'll believe you, yes. Yeah, if you're interested, this is, this wiki page is all we have at the moment. It doesn't even go into this much detail. So a lot of documentation work can go into this. Trace children is important in the back end because of the forking. I don't actually remember right now what Trace Orange is us and the read the bar info is just so you get a better back trace. Do you know what the track Orange is this about? It tells you where a member is allocated. It tells you the same structure as you. Okay. Does that have to do with the Palog instrumentation we added recently? Well, the puzzle of instrumentation is, you know, usually to, you know, what you get from the pool. But yeah, it will tell you where you access the instrumentation number, for example, like, you know, where it's located. So like that's very useful. You say this runs a lot slower. Yeah, it's like, I think it's like, why do you use a lot of the methods and then I'll just make the slides a little bit different? Right. So I think some more documentation here would be useful. And I think also if you do run and report problems, you've got to report on how you ran it because there's different things you can check with all of them. But so I see this sort of as a progression of tools that get more sort of intensive. So you can, you know, start with compiler warning, static analysis finds a lot of memory issues as well. But they can only reason why it's static. These are runtime tools and they have runtime knowledge of what's actually happening. So that's great. So in order to actually get anything useful out of this specifically, but also this, you need to have tests that exercise this, right? So, and we talked about some of that yesterday. So there was a company called Sun a long time ago that in the PostgreSQL community, and they actually submitted this, that they contributed this feature. And it wasn't me, it was someone else. So this has been there for a long time. I just want to remind people how this works. If you want to find out whether something, what the test coverage is of any given test or anything you run yourself, use this, enable coverage, make, then coverage clean just deletes the HTML output from the previous run. After a long, many puzzling nights, I discovered it's best to run with max connections equals one so you don't have any concurrent stuff happening because the output files override each other at runtime. And then it just produces garbage. So that might be useful knowledge. And then it produces HTML output and you can look at it in your browser. Let's look at that. I have a Jenkins job that does that. Looks like this. Yesterday, Hickey was complaining that his speech is vacuum is not exercise, which you can see here. So that's a lot of arguments about how much coverage you need and all that. So that's a different problem. But red is probably not good. So I think we can agree with that. So. You're a good speaker. SPG. Yeah, right. Well. You're a good speaker. I'm working. I think coverage has a low. Yeah, it's sort of a coverage. Yeah, so we know we don't really have a lot of knowledge about that. But we're knowing that we have a problem with not testing the state-based problem. Right. Yeah, and you know all I'm. I think it would be one of the devs to have a slave follow up with the problem of getting the slave into the state. Yeah. So. You had this thing that was like an AM, Jack saying. So obviously, this is not about how to design tests and stuff that's a bigger problem. I think this is more like a 10% to 20% solution to issues that we could possibly have. Actually, no, design tests is obviously a much harder question. So let's talk about go to fail. We talk about hard bleed. Let's talk about go to fail. So for those who don't know, go to fail was an issue in the Apple SSI libraries where they had accidentally duplicated a go to statement. And then it skipped a bunch of. What is it? Right. So I tried to replicate it, and I made this patch in Postgres just locally. So don't worry about it. And see what would happen. So obviously, pg-indend would be kind of helpful in the situation. Just on a side note, that's good. Also, if you do this, tests will fail because the protocol is all messed up. So you will see this. But this could be in the actual SSL code in Postgres as well, which also has goto's in it and is not exercised in the test suites that people usually run. So this could be a big problem. And there was no compiler warning. Scan build didn't say anything about it. I couldn't check severity. I have more confidence that they might find that. But I couldn't submit my own build. But this is not a trivial problem to adjust. So this could happen. Could you pass the right option to c-1? If you know that option, I'd be very interested because I tried a lot of them. If they remove it, then you get it. They remove it. Yes, there was a long time ago in GCC on that code analysis, which they removed because it didn't work. And you need it. Yeah, it's a hard problem. Again, in languages like Go, for example, they have made a strict decision in the language design that that code is not allowed. So they will give you an error. In Java, they have made a compromise, but it's a well defined compromise of when it's allowed or not. In C, there's a lot of things that are possible with long jumps and if-dabs. And so it's not possible to design an option that works for everyone. You're claiming that there is now a brand new option that might catch that. I'd be very interested. Yeah, again, if you run a tool, I'm not familiar with lint, but it will then give you possibly thousands of unreachable code in the PostgreSQL base. That's the problem. So yeah, this is an open question. I don't know if anyone has a good idea. I just tried the usual tools that we have available. If anyone knows how to fix that, it would be a great addition to this suite. I think so, right? Yes. But we'll see. What's Chris? Well, it's splint is on my list of other things to try. I should have had lint here. I've played with lint on and off. Again, if it gives you 1,000 reports, it's not helpful. It needs to be really- and splint isn't the same category. It has promise, and I've heard good things about it, but it needs to be tuned to hack out of it to understand everything that's going on. And it has a bunch of options like are you using C this or C that or Pro 6 or Unix, what are we using? We have a really custom combination of all these things. How do you tell that? It's very hard. Lang has a bunch of other sanitizers that are worth trying out, but they haven't really done anything for me. Sanitized undefined would find undefined behavior. I've never found anything with that, but that's probably good. The memory sanitizer only works on a limited amount of platforms, so it's not really a good solution. But there's a lot of overlap with Allgrind and Adresinitizer and any of these static analysis tools. Obviously, we don't use threats, but there's another thing out there. Mention splint, this Viva PBS Studio. Some people might remember that. That came out on the Hackers mailing list around New Year. This is a commercial tool where they threw that at Postgres and reported some of the issues back. And someone were fixed, and it's kind of an odd mix of things that they reported. But someone were good. But I downloaded the full list of things they reported, and the entire list contained 51,572 issues. So that's not really that helpful. And then someone said, well, this shouldn't actually be checked, because that's kind of a bogus check. So once I filtered that out, we're back down to 12,460. Still not helpful. And there, at that point, it's not useful to look at nobody wants to go through that. And they're very picky, for example, about using, for example, the long type. Every time you use a long type, it tells you, well, this type could have a different size, depending on architecture. So this tool might have come out around the time where 64 bit architectures were coming out, and people need to report to it. It's a commercial tool. There's a link here if you get those slides a little bit. I don't think that's going to help us much. I'll be on time. We're way over on tree. Here's the thing to worry about a little bit. So here's our lines of code. See how many committers actually contributed to that. So that's not going in the right direction, right? Then month since release and commit since release. And then how many commits per month? That's also not going in the right direction. And the last is hours per day. That's not changing. So that's a problem. We know we're all getting busy or so. I hope if you throw all these tools at it, we can at least make up the difference there a little bit. Obviously, it's not going to help find more complex issues that we know about. So I think this is more of a 10% to 20% solution if we put a lot of work in it. But I think it could be helpful. And I plan to do more work of that if someone wants to join in. It'd be great. So I think good goal for next release is to do a lot of work with get the ball grind and add the sanitizer. Working really well and add more tests to exercise some of the stuff that's not exercised yet. And if people have issues with building with WRR, let's help fix these issues if we can. Let me know if you need help with that. That is the last slide. So I think we've got to go. Any questions? OK, thank you. I'll be here all day. Thank you. Thank you.
 OK, is the recording on?  Microphone?  Microphone?  OK, hello, good morning.  I'm Peter.  I'm PostgreSQL developer, and this is the story of my life.  So I want to talk about various tools that I and others have  thrown at the PostgreSQL to find bugs and other issues.  I'm glad with this room, because the other one was so dark  that I was almost falling asleep.  So let's start with something simple.  Have you ever written a C program, like a one-off, to  just do a test case, and you compiled it?  And it seemed to compile fine, and you  ran it and crashed immediately.  And then you noticed you did everything wrong, because the  compiler didn't warn you about any of the problems.  So if you have done that, then you will know that without  compiler warnings, a C compiler is worthless.  So that is why we in PostgreSQL have put a lot of effort  into both cleaning up existing warnings to the point of 1  or 0, depending on how you look at the flex issue.  And continue to add new warnings.  So this is the current set we're using by default.  And if you just look through, the WL is obviously the big  one, but if you look through all the other ones and you  imagine how your life would be if you didn't have those, it  would be quite sad.  So that is the set we're using, and we're pretty  comfortable with it.  This is, I'll be on the screen here.  This is actually what I tend to use in my own builds.  So there are a couple groups here.  The first two, or the first one is not really a warning  option, but what it does, it's on glibc slash Linux.  It changes the definition of several library functions.  So they declare themselves that you shouldn't ignore  their return values and things like that.  And that, in combination with the other warning options,  then gives you warnings if you missed checking the return  code of a function, for example, which could lead to  security problems.  Yeah.  How will we play in that?  How will we play in that?  Well, how much courage do you get?  Well, we are currently not getting any garbage because  of two reasons.  One is I always have this on in my own build, so I would  find it pretty soon.  The other is that on Ubuntu, they have somehow hard  coded this into their compiler, so they're always  building this.  So the original reason why we actually started cleaning up  is that we got these reports from Ubuntu people.  So I must be the subject of the calls that the  covariate complaints are about.  Yeah.  We'll get to that.  We'll get to the covariate in a moment.  So logical op is also something I'm using.  That finds issues like if you have an if statement life.  If x is greater than 0 and x is less than 5, that would be  a type of it.  It would find that.  So that's actually pretty useful.  It actually has found at least one bug in Postgres a few  years ago.  I actually submitted patches to make both of these a default  awhile ago, if you recall.  But the problem is that in very old GCC compilers that  people are still using, they produce a lot of garbage.  Because I guess the older compilers were not as smart  to produce useful.  Is this just Tom's bug again?  Is it what?  Is this just Tom's, you know?  No, this has nothing to do with that.  Those were reasonable.  I don't remember the details, but those were reasonable  platch ones.  They were just a few years older.  And this is the story that is a problem with all of these.  You want to add these other ones.  The second group there is things that I sometimes also  use.  They haven't really found much.  So you could use them.  The third group there is something that I use with  Clang specifically.  And the ones below there are new in Clang 3.4.  So they haven't really found anything yet.  But they are also sort of on the horizon, on the road map,  so to speak.  But the problem is really you can't make these the defaults  for the reasons I mentioned.  Because different compiler generations produce different  nonsense for that.  So if you want to maintain what we have, you know, the  zero count of the warnings, it's really hard to move  forward.  That's a kind of annoying issue.  So it's going to, you know, people who are interested in  this will basically have to do this manually going forward  also.  So you have to do a lot of work to make that.  I think a better plan would be to have a built-in  member or something like that.  I don't want to spend too much time on the warnings.  But yeah, there's obviously a lot of fiddling we could do  to make this better.  But the real problem is the built-in member doesn't  tell you how to do it.  Yeah, well, that's another issue that many issues here.  That's good.  All good notes.  So one thing I always do is compile with a W error, which  turns all warnings into errors.  And I have done this successfully, so to speak,  for almost five, eight years.  So if I compile code and it flies by, I don't see the  warnings.  I need to have a boolean answer from the compiler.  That's just my workflow.  The problem is if you pass W error into configure,  configure fails because it generates warnings internally  as part of its testing.  And the workaround I use, so to speak, is I have a shell  alias that does this sort of thing.  So if you want to do this, you can take a photo or look at  the thing later.  This works for me because actually the shell alias I  use is much longer because it also picks different options  for different branches and so on.  But that, for me, is useful.  And I think everyone else should look into that because  it's really helpful.  The problem right now is I'm always the first one to find  the warnings if someone accidentally commits one.  But I'm happy if I've chosen this path.  Here is a chart I computed over the weekend, which caused  this computer to heat up my office for a while as it was  building all this.  And this kind of shows the progress of compilers becoming  more and more either smaller or annoying, depending on how  you interpret it, and also us catching up with them.  And you can sort of see a diagonal here as time  progresses.  The problem with this is going back to the W error issue is  if you need to make a change, let's say you need to back  patch something into 9.0 right now, you need to either very  carefully look at the warnings or not that your  compile generates, or you need to pull up this chart and  then actually try to install GCC 4.4 and build that because  that was current at the time.  That is very annoying and pretty much an unsolved issue  for working in back branches.  So I don't know how to fix that, other than keeping a lot  of old compilers around, which I essentially do.  But not all platforms have the option to install all that.  There's all things.  It's kind of hard.  In case you're curious what this is, a lot of the options  in 8.3 was that the compiler was complaining that inlining  failed, and that generates hundreds of warnings almost,  and then we later turned that off.  In GCC 4.6, they added a variable was set but not used,  which apparently, it was good.  Now the issue with this is a lot of developers built with  assertions enabled.  But if you turn the assertions off, you get a different set  of warnings because sometimes the warning variable is used.  Sometimes those sort of problems don't get detected for a  long time until someone builds without assertions.  Very difficult.  Yeah, we mentioned already, Buildform doesn't do anything  about warnings, so this doesn't help.  But when the other talk, Rob Pike was mentioning, who is  also one of the creators of Go, the programming language,  which you can have different opinions about.  But one interesting thing they did, they don't have any  compiler warnings.  It's either an error or not, which makes this sort of  thing much easier, which I think is a good decision.  So now we were talking.  Yes?  How soon I have to know what's the 36?  Yeah, OK.  So the 36, this is on Mac OS 10, version 10.9 specifically.  Those 36, if you're on a Linux platform, you  might not get those.  This is a, I argue it's a bug.  I filed a bug with GCC about it.  Complains if the right-hand side of a comma operator does  not have a side effect.  OK, OK.  And this is a new warning they introduced.  And this happens in the system house of the operating  system for various macro expansions.  And they're not very smart about that.  So you might not see this, but if you're me and you  develop on this 50% of the time and you run with W error,  then this is GCC 4.9 is broken for me at the moment.  So I actually use 4.9.  Yes, of course.  I've never been able to get some of your other work.  So you've always come up with various errors and  some serious errors.  So I would kind of grab the errors and then  just go to the end of the line.  And then print all the error messages.  I know a lot of people have.  I've seen in discussions that people have macros or things  that do that.  In my experience, it always works.  Or if not, I fix it.  But that's the other thing you can do also.  I think that's a good point.  Well, I do wonder.  I mean, we could turn on the error for both forms.  It is kind of, yeah.  Obviously, there are obscure platforms where it is not  zero at the moment.  Because we're actually going to be able to limit GCC  more on so much better.  Yeah.  Well, we're also pretty good on Clang.  But of course, if you use, for example, the Solaris,  whatever it's called, Sun, former Sun Studio, you get a  bunch of different things.  And it's not easy to turn them off.  They don't really have these options.  So on those kind of platforms, you pretty much  screwed at the moment.  Let's talk about severity.  So the way the compiler options are designed is so they  don't slow down the compilation so much.  So the next level of intensity here, so to speak, is the  static analysis tools that are specifically designed to  produce essentially as many warnings as possible at the  expense of being super slowly built.  And the severity was kind of the first that at least came to  our attention on a broad scale.  So the severity came out of a research project, but it's a  commercial company.  They sell this product for a lot of money I hear to do  statically analyze your C or C++ code and started doing that.  In around 2005, they started this open source program.  I think that was also somehow government sponsored and  built a bunch of open source projects and made this  publicly available.  And they have expanded this platform over time.  And now it's almost to the degree that anyone can sign  up and submit builds there.  But the results are still under some kind of license.  So I can't really go into too much details without not  knowing what I'm getting myself into.  Are you associated with that company?  We'll have another example, which is the Clang Scan Build,  which is open source.  So if you can drill into deeper, they're very similar.  Just a clarity goes a little bit deeper.  So first of all, we have kind of made the decision that the  only people who should have access to this is the people  who are also on the security team, because there could be  security issues in there.  So if you're not on that team, what you can do is submit your  own project to Corrarian.  Just download the Postgres code.  And if you figure that out, you have the results yourself.  That's homework.  But if you're one of the circles, so to speak, and  you want to have access to this, you can talk to Steven  Frost, primarily.  But I'm also an admin on this, and Magnus is also an admin  on this, so we could help you out.  This is the URL.  You can go.  I think we'd be, we want to get to a point where we actually  have a whole thing cleaned up.  I think we'd be more willing to go.  As you can see, there's a lot of things.  So I found in the commit log 71 issues mentioned.  Well, that's the debate of, well, it's hard to have that  argument about a 1,500, right?  It would be more easy to have that argument about a  150, right?  So we don't know.  We have been through, yes, between you and myself and  others, we have been through a lot of those.  But so, yeah, I'll show an example in a moment of what  that looks like.  I found 71 commits, which doesn't mean there was 71  issues.  There were probably more in the older of 200 or 300,  because a lot of these commits bunched a bunch of things  together.  So a lot of those are in the vision of memory leaks or  memory access problems.  So as it says here, there are 1,501 issues  outstanding.  Again, we have disqualified most of those, but who really  knows?  Again, if you're one of the developers, you can get  access to that if you want.  Coerity put out a notification that they  detect a heart bleed bug, which was a recent security  vulnerability.  So they added that a week after it came out.  So that didn't help.  But that's the sort of level they work on.  So they do taint checks of where data comes from and warn  you about if you're using data that you get from external  sources into a good way.  Problem with that in Postgres, we actually do that a lot.  If you think about what Postgres does a lot is read the  block from disk and trust it implicitly that this has the  right structure.  And then just assumes, oh, if there's a length right here  of that, we assume there's that much.  And if that doesn't work out, it's just bad things happen.  So essentially, we trust whatever's in the data  directory implicitly.  So all these taint checks don't really work for us.  Which there's probably a lot of taint checks amongst those  1,500, but who really knows if there's really one that  does the problem?  Another example of this is if, for example, PG-based  backup, there was an issue out of that store there.  If PG-based backup pulls down incorrect data from the  server, you can make it crash potentially by having a  division by 0 somewhere, as possible.  It's not a huge issue if your goal is not to make PG-based  backup crash.  The opposite of that would obviously be a much bigger  problem if you send incorrect data up to the server and the  server crashes because you send 0 somewhere.  But they can't really tell what is more important than what.  So that's hard.  Let's see if I can pull up a web browser here on this.  Where did it go?  Oh, there it is.  Cursor is there.  Now it's here.  This is very exciting.  So this is the dashboard you get here.  It's a little bit smaller.  The back branches don't seem to really work because they  only find one defective.  Well, the actual interesting thing is this one here, where  you can go through all the problems here.  And the screen is a little bit funny now.  This is interesting.  I just looked at this the other day.  So this is, you have op here compared to this.  And then down here, you compare to that.  And there's nothing in here that changed that.  I just saw that last night and it looked kind of funny.  So that doesn't.  I would pick so.  I didn't.  I seriously didn't pick that.  I just looked for an easy example.  So arguably a compiler could detect that also because it's  pretty straightforward, but obviously it needs more time  to analyze that.  So this is the level of issues you got here.  So we will get him an account.  It is pretty cool.  You can waste a lot of time in there.  So the open source alternative to that is Scanville, which  came out of the.  I'll show you later.  Which came out of the LLVO.  Stephen, you want to explain how queries built in your  basement and then it's uploaded.  No, no.  OK.  Hey, it's not the same physical box that I could  pick from.  So I'm going to go on and talk to you.  No, it's not.  I'm just saying.  So it is basically someone in the infrastructure team  regularly builds Postgres and then you upload some binary  blob to them and they analyze it.  So it is.  Now we actually have more resources in terms of VM space.  So Scanville is kind of an open source alternative that  came out of the LLVM project.  This is the URL.  So if you want to use that, here's a recipe.  So you can try that yourself.  Most reasonably up-to-date platforms should have this  inner package that might be called LLVM or Clang.  Yeah, I've heard that the recent build is from the recent  version of the stream build that introduced the number of  the two-lane fault-pot.  It is, yes.  It is.  It seems like recent versions of the bugs.  It is an aggressive development.  So you have to kind of, they do stuff and you do stuff in  parallel.  So yes.  That's how I use it.  It didn't have the built-in effect that was also a  lot of error.  And then you put an abort on that next thing.  And you also don't figure out the effect.  So we'll kind of do that.  But it's not like that.  OK, let's talk about it in a second.  Yes.  So if you want to build that, you download that tool.  The O is basically where it stores its output.  You configure it, you ought to build with enable assert.  And then you just build it.  It takes between half an hour and an hour or so.  Then it tells you how many bugs it found.  And then what is it?  Oh, I'll explain in a moment.  You don't have to, but it will not be.  It won't work too well.  And then there is this tool called ScanView, which pulls  up the result in a HTML report, similar to what  CorelD just showed.  So let's look at it.  So here's the result.  As you can see, here are the bugs.  Let's look at fancy JavaScript here.  Let's look at 1.  Division by 0.  So here's all the places where you have division by 0.  So you can see all the Q sort will fail if you pass the  size of 0.  That's by design.  So you can't really do anything about that.  The print routines will fail, for example, if you print the  result set with 0 columns, which you wouldn't normally get,  but as you know, we can fake that with tables with 0  columns.  But the reason why this is not a problem is that tables  with 0 columns also have 0 rows, so no row is actually  printed.  So if you could fake up a result set, you could crash  this.  But in practice, it's not a problem.  The fix in this whole situation is that you put an  assert somewhere in there that will just say, assert  column count is greater than 0.  And then you'll still crash, but in a more controlled way.  And this tool will know about it and those things go away.  That's why you need to build the assertions.  Problem is, in the back end, you can turn the assertions off  at runtime.  So none of that will work, because the code path analysis  always has to analyze the path.  Well, what if assertions are turned off?  Then I have to look at this path here.  So that just doesn't work.  So this tool doesn't know about that.  Corverty has somehow, they said they had sort of manually  tweaked something so they know about that.  No, we have that problem.  Maybe that used to be a while ago.  They have the same class of problems in any case.  So, sign values go up a bit on the front.  So, Bison generated code apparently has lots of  problems that you don't know.  Hard to argue about that.  Here is something in parser.  And see, it gives you kind of this path here.  You can go look, see how this comes about.  So this is this function, which gets in a list of arguments.  And then, no, this is all.  Well, and then, you know, so this goes, loops around here.  And assigns the arguments of the list to this array, which  is this fixed size.  So if you pass in the list as longer than that, you will have  problems down here.  This will not actually fail in practice, because like three  levels up, there's a check that you can't, you know, on a  more semantic level that you can't pass in anything longer  than, you know, from MaxArk.  So, right.  So the fix again here would be to pass in an assertion, to  put in an assertion.  But the problem is, this will fix that.  But as you know, this is not actually the problem it's  complaining about, right?  This problem is down here, where it loops through that  same thing again, and assigns it back into a list.  But it has no knowledge about lists, so it doesn't know  that by exercising this loop, you do a number of steps of  something.  It doesn't know that at all.  So it would have, even if you put in an assertion at list  length or something, it has no knowledge of how these  structures work.  Because all the lists are in the user space, so to speak.  I don't think there's anything you can do here to.  It doesn't know that that's a list that has properties of  any kind, so that's hard.  Let's do no one.  Do you assert it on the list?  Do you specifically know what the assertion is?  No.  If you just assert, it will know.  It knows what assertion does.  Basically, this code path is not relevant.  So Unix API, I had something in the Swift logger.  This is a common problem, passing a null pointer to a  function that shouldn't get one.  And if you trace this all the way back, this value is  assigned as a result of a PL lock.  But this tool doesn't know that PL lock never returns null.  So that might be something we can instrument somewhere.  But I haven't figured out how to do that.  But that would possibly help in a lot of cases.  And there are GCC also has some attributes you can put on  functions that tell you that this argument shouldn't be  null, but I don't think they have an attribute that tells  that this function never returns null.  So that's actually what we use in a lot more cases.  Assert would help here again.  And there was a question about what about the E-log issue.  And we added, she also had no return attributes to  functions that also prune a lot of potential code paths and  reduce a bunch of issues.  So is there anything that you found that's like the thing  and then gets a nine?  But anything that actually understands what we do when  we do the rare error signaling?  And how we do the send up and whatnot?  Or is it just null?  No, it's not.  It's not null.  That's the only way to get a six-prone.  No, I think you just do the null return.  Because what I actually have to trace through and then see  that is a long jump in some cases.  But it potentially could do that.  But I don't think it digs that deep.  That's I think the problem.  Because you have to have a trade-off of how long this  thing is going to run, I think.  I haven't got any tools to do it.  I'm just wondering.  Is that one of the big down-plates of all these  tools that none of them understand that?  So it's kind of a tender for them to do it.  So this is the number of reported bugs we've had over time.  And at some point, in 9.3, I think, is when we put in a  bunch of null returns.  And as you can see, that helps a lot.  Pre-requisite for this is that you get clang to build it  out warnings.  Because it's kind of the same foundation.  So a lot of the old releases have just basic clang  bugs that we didn't attend to at the time.  The there's no particular reason why 9.4 has more.  I was kind of surprised by that.  But there doesn't seem to be any systematic change there.  It's just certain unrelated changes have potential issues.  But it doesn't seem to be serious.  The star column there is kind of a branch I have made  myself where I have made assertions that you can't  turn them off.  So it prunes certain code paths and did a few other  possibly controversial tweaks to kind of see how it can get  that number.  Ideally, we'd like to get that to zero.  Because again, this comes back to the warning versus error  issue.  If you always have to compare numbers, you're never going  to get really a bullying answer.  But as was mentioned, scan build is still in development.  And things come and go.  So it's kind of hard.  But this is a useful tool that everyone can use, open source.  And you can play around with it.  So Richard, you're using that apparently as well?  Yes.  The older versions of the new ones.  Yeah.  They're a lot easier.  So yeah, I should have, if I had more, if I would have  wanted to stress this laptop more, I could have done this  for N releases.  And it would probably take many days to build.  So another static tool that came out of LVM is the  address sanitizer, which actually came out of Google  for their Chrome browser, I believe.  But it's now part of Clang Compilo and GCC 4.8.  You can also use it in GCC.  So the way this works, you give it this option  sanitize address, this one is the key one.  And then these are recommended to get better backtraces, but  they're not the key here.  And then you build it, you run whatever check test suite you  might have.  And if there's a problem with whatever this checks for,  which is mainly buffer overruns, then it will crash at  this point.  And then it will print to standard error a cryptic  backtrace.  And then there's this script that you can run this  backtrace through, and you will get a backtrace.  I can show an example.  What is it?  I don't know what these do.  They're just recommended to get better backtraces.  I've never actually seen that.  This is a Clang option, so I don't think you might not have  seen this in GCC.  This is more well-known.  Now, I don't know what this does.  It's in the recommended set of options.  So one issue we found, this might look familiar to some, is  this issue in a non-issue.  That's why this was ignored for a long time.  If you run the name, it will do eventually a stir copy of  these two things.  And there is a certain case where these are the same.  So you would think that's fine, because it just copies it  over the top of itself.  And for the last 20 years or so, that did work fine.  But this thing was founded for strictness reasons.  And it was ignored for a long time.  But then some new compiler in MacOS 10.9 actually complained  about this or crashed here, just as part of the regular  compilation, so at that point, it needed to be fixed.  And then every scientist was happy.  And once you fix that, this is a real issue that we found.  Here, you get the user like rule name, which is a string  constant or a C string.  And you cast it to a datum, pass it to this prepared  plan, which expects a name datum there.  And in most cases, that's fine, because the name is just  a 64 byte string.  But at some point later down here, this string gets  copied into another copy of itself.  And that will then copy the full 64 bytes, whereas this  is not 64 bytes long.  So that's a confusion between C string and name.  And so this could have been a real problem when someone is  actually at the end of memory.  So this was useful, and this was fixed.  So the problem with address sanitizers is just whenever  it finds the first problem, it crashes.  So you need to fix that, or at least work around it before  you can get to the next point.  So how do you do that?  I don't remember the name.  OK, I didn't find.  Well, obviously, you can just comment that out, and you can  pass by, right?  So it's just for analysis.  But it's a.  I'm not familiar with that.  But.  I mean, what would anyone want to add?  This is the.  But I know you have a lot of crates to do it already.  So this is the current issue.  This is as of now.  Which is actually kind of weird, because it will work fine  on 32-bit builds, but it will do this crash on 64-bit  builds.  And this was discussed on the main list a few months ago,  and it was actually argued that this is not a real problem.  So you don't have to worry about it too much, but it will  crash this way if you do it right now.  So it would be worth fixing at some point.  Are you similar to what I've seen in Valgrind?  So yeah, so Valgrind has received a lot of attention  in the community lately, and it's a much more general tool.  Actually, on the website, they now described themselves as  it's a just-in-time compiler, a pluggable just-in-time  compiler.  So it can do a lot of things now.  But obviously, it's more known as a memory-checking tool.  And yes, you can also use Valgrind.  They, all of these tools overlap in some degree.  The covariate can check find memory leaks.  If it can reason about it statically, Valgrind will  find memory leaks just because it keeps track of it.  And it's more guaranteed to have a good answer.  So the important thing, I would like to do more with  Valgrind.  And I think a lot of people here are of the same opinion.  You can do a lot of things.  So here's a couple ways you can use it.  This is, I think, something that Hakey posted just the  other day as an example to do back-end work.  Something I've looked at.  So here's a wiki page that was put up recently to also cover  some of that.  One thing I'm kind of interested in is also memory  leaks and front-end programs.  For example, in PGDump, memory leaks could be kind of a  problem, but are not well tracked at the moment.  If you actually do this, you will get a memory leak.  But as you will see, it's kind of a stupid one.  This is what you get from PGDump right now.  But as you can see, the memory leaks are in  undisputed vacuum.  We just had the same problem.  So yeah, here it is.  So here, this is every program in Postgres, or at least  client programs, this problem that allocates a few strings  for local services, but you know, the program then  doesn't free them.  So any time you do this, you will get this sort of trace.  And as we know, we can add suppression files to get rid  of those, which I would like to do.  Basically, you get all of these things down to zero.  And then be able to run test suites again with a Boolean  answer at the end, whether there are any problems or not.  But we got to fix some of those first.  So this was specifically a leak check.  A lot of the other work that has gone in in the back end is  access to memory that you shouldn't access of various  kinds, which is what this down here would find.  There's been a lot of work on that lately.  I only found actually 19 commits mentioning ball grind.  So we haven't really done a whole lot, but there was a lot  of activity just lately.  So one of my goals is to basically be able to run  this automatically and just give you an exit status at the  end.  We look very concerned, Greg.  Last time, no, last time it was ball grind.  It's a old, curvy runtime.  It wasn't static.  Yes, this is, yes, I should have made the segue better.  So the, uh,  Is that so?  I haven't had that problem, really.  You would know a lot about running things on smartphones.  So I'll believe you, yes.  Yeah, if you're interested, this is, this wiki page is all  we have at the moment.  It doesn't even go into this much detail.  So a lot of documentation work can go into this.  Trace children is important in the back end because of the  forking.  I don't actually remember right now what Trace Orange is  us and the read the bar info is just so you get a better  back trace.  Do you know what the track Orange is this about?  It tells you where a member is allocated.  It tells you the same structure as you.  Okay.  Does that have to do with the Palog instrumentation we added  recently?  Well, the puzzle of instrumentation is, you know,  usually to, you know, what you get from the pool.  But yeah, it will tell you where you access the  instrumentation number, for example, like, you know,  where it's located.  So like that's very useful.  You say this runs a lot slower.  Yeah, it's like, I think it's like, why do you use a lot  of the methods and then I'll just make the  slides a little bit different?  Right.  So I think some more documentation here would be  useful.  And I think also if you do run and report problems, you've  got to report on how you ran it because there's different  things you can check with all of them.  But so I see this sort of as a progression of tools that  get more sort of intensive.  So you can, you know, start with compiler warning,  static analysis finds a lot of memory issues as well.  But they can only reason why it's static.  These are runtime tools and they have runtime knowledge of  what's actually happening.  So that's great.  So in order to actually get anything useful out of this  specifically, but also this, you need to have tests that  exercise this, right?  So, and we talked about some of that yesterday.  So there was a company called Sun a long time ago that in  the PostgreSQL community, and they actually submitted this,  that they contributed this feature.  And it wasn't me, it was someone else.  So this has been there for a long time.  I just want to remind people how this works.  If you want to find out whether something, what the test  coverage is of any given test or anything you run yourself,  use this, enable coverage, make, then coverage clean just  deletes the HTML output from the previous run.  After a long, many puzzling nights, I discovered it's  best to run with max connections equals one so you  don't have any concurrent stuff happening because the  output files override each other at runtime.  And then it just produces garbage.  So that might be useful knowledge.  And then it produces HTML output and you can look at it in  your browser.  Let's look at that.  I have a Jenkins job that does that.  Looks like this.  Yesterday, Hickey was complaining that his speech is  vacuum is not exercise, which you can see here.  So that's a lot of arguments about how much coverage you  need and all that.  So that's a different problem.  But red is probably not good.  So I think we can agree with that.  So.  You're a good speaker.  SPG.  Yeah, right.  Well.  You're a good speaker.  I'm working.  I think coverage has a low.  Yeah, it's sort of a coverage.  Yeah, so we know we don't really have a lot of knowledge  about that.  But we're knowing that we have a problem with not  testing the state-based problem.  Right.  Yeah, and you know all I'm.  I think it would be one of the devs to have a slave  follow up with the problem of getting the slave into the  state.  Yeah.  So.  You had this thing that was like an AM, Jack saying.  So obviously, this is not about how to design tests and  stuff that's a bigger problem.  I think this is more like a 10% to 20% solution to issues  that we could possibly have.  Actually, no, design tests is obviously a much  harder question.  So let's talk about go to fail.  We talk about hard bleed.  Let's talk about go to fail.  So for those who don't know, go to fail was an issue in the  Apple SSI libraries where they had accidentally  duplicated a go to statement.  And then it skipped a bunch of.  What is it?  Right.  So I tried to replicate it, and I made this patch in  Postgres just locally.  So don't worry about it.  And see what would happen.  So obviously, pg-indend would be kind of helpful in the  situation.  Just on a side note, that's good.  Also, if you do this, tests will fail because the  protocol is all messed up.  So you will see this.  But this could be in the actual SSL code in  Postgres as well, which also has goto's in it and is not  exercised in the test suites that people usually run.  So this could be a big problem.  And there was no compiler warning.  Scan build didn't say anything about it.  I couldn't check severity.  I have more confidence that they might find that.  But I couldn't submit my own build.  But this is not a trivial problem to adjust.  So this could happen.  Could you pass the right option to c-1?  If you know that option, I'd be very interested because I  tried a lot of them.  If they remove it, then you get it.  They remove it.  Yes, there was a long time ago in GCC on that code  analysis, which they removed because it didn't work.  And you need it.  Yeah, it's a hard problem.  Again, in languages like Go, for example, they have made a  strict decision in the language design that that code is  not allowed.  So they will give you an error.  In Java, they have made a compromise, but it's a well  defined compromise of when it's allowed or not.  In C, there's a lot of things that are possible with long  jumps and if-dabs.  And so it's not possible to design an option that works  for everyone.  You're claiming that there is now a brand new option that  might catch that.  I'd be very interested.  Yeah, again, if you run a tool, I'm not familiar with  lint, but it will then give you possibly thousands of  unreachable code in the PostgreSQL base.  That's the problem.  So yeah, this is an open question.  I don't know if anyone has a good idea.  I just tried the usual tools that we have available.  If anyone knows how to fix that, it would be a great  addition to this suite.  I think so, right?  Yes.  But we'll see.  What's Chris?  Well, it's splint is on my list of other things to try.  I should have had lint here.  I've played with lint on and off.  Again, if it gives you 1,000 reports, it's not helpful.  It needs to be really- and splint isn't the same  category.  It has promise, and I've heard good things about it, but  it needs to be tuned to hack out of it to understand  everything that's going on.  And it has a bunch of options like are you using C this or  C that or Pro 6 or Unix, what are we using?  We have a really custom combination of all these  things.  How do you tell that?  It's very hard.  Lang has a bunch of other sanitizers that are worth  trying out, but they haven't really done anything for me.  Sanitized undefined would find undefined behavior.  I've never found anything with that, but that's probably  good.  The memory sanitizer only works on a limited amount of  platforms, so it's not really a good solution.  But there's a lot of overlap with Allgrind and Adresinitizer  and any of these static analysis tools.  Obviously, we don't use threats, but there's another  thing out there.  Mention splint, this Viva PBS Studio.  Some people might remember that.  That came out on the Hackers mailing list around New Year.  This is a commercial tool where they threw that at  Postgres and reported some of the issues back.  And someone were fixed, and it's kind of an odd mix of  things that they reported.  But someone were good.  But I downloaded the full list of things they reported, and  the entire list contained 51,572 issues.  So that's not really that helpful.  And then someone said, well, this shouldn't actually be  checked, because that's kind of a bogus check.  So once I filtered that out, we're back down to 12,460.  Still not helpful.  And there, at that point, it's not useful to look at  nobody wants to go through that.  And they're very picky, for example, about using, for  example, the long type.  Every time you use a long type, it tells you, well, this  type could have a different size,  depending on architecture.  So this tool might have come out around the time where 64  bit architectures were coming out, and people need to  report to it.  It's a commercial tool.  There's a link here if you get those slides a little bit.  I don't think that's going to help us much.  I'll be on time.  We're way over on tree.  Here's the thing to worry about a little bit.  So here's our lines of code.  See how many committers actually contributed to that.  So that's not going in the right direction, right?  Then month since release and commit since release.  And then how many commits per month?  That's also not going in the right direction.  And the last is hours per day.  That's not changing.  So that's a problem.  We know we're all getting busy or so.  I hope if you throw all these tools at it, we can at least  make up the difference there a little bit.  Obviously, it's not going to help find more complex issues  that we know about.  So I think this is more of a 10% to 20% solution if we put a  lot of work in it.  But I think it could be helpful.  And I plan to do more work of that if someone wants to join  in.  It'd be great.  So I think good goal for next release is to do a lot of  work with get the ball grind and add the sanitizer.  Working really well and add more tests to exercise some of  the stuff that's not exercised yet.  And if people have issues with building with WRR, let's  help fix these issues if we can.  Let me know if you need help with that.  That is the last slide.  So I think we've got to go.  Any questions?  OK, thank you.  I'll be here all day.  Thank you.  Thank you.
Static_analysis__test_coverage__and_other_12__letter_words_2_0_2123.png Static_analysis__test_coverage__and_other_12__letter_words_2_2124_2207.png Static_analysis__test_coverage__and_other_12__letter_words_2_2208_3083.png Static_analysis__test_coverage__and_other_12__letter_words_2_3084_11243.png Static_analysis__test_coverage__and_other_12__letter_words_2_11244_18503.png Static_analysis__test_coverage__and_other_12__letter_words_2_18504_19247.png Static_analysis__test_coverage__and_other_12__letter_words_2_19248_19523.png Static_analysis__test_coverage__and_other_12__letter_words_2_19524_20771.png Static_analysis__test_coverage__and_other_12__letter_words_2_20772_21635.png Static_analysis__test_coverage__and_other_12__letter_words_2_21636_23327.png Static_analysis__test_coverage__and_other_12__letter_words_2_23328_24647.png Static_analysis__test_coverage__and_other_12__letter_words_2_24648_25379.png Static_analysis__test_coverage__and_other_12__letter_words_2_25380_26651.png Static_analysis__test_coverage__and_other_12__letter_words_2_26652_26771.png Static_analysis__test_coverage__and_other_12__letter_words_2_26772_26891.png Static_analysis__test_coverage__and_other_12__letter_words_2_26892_26999.png Static_analysis__test_coverage__and_other_12__letter_words_2_27000_27263.png Static_analysis__test_coverage__and_other_12__letter_words_2_27264_27383.png Static_analysis__test_coverage__and_other_12__letter_words_2_27384_27515.png Static_analysis__test_coverage__and_other_12__letter_words_2_27516_29171.png Static_analysis__test_coverage__and_other_12__letter_words_2_29172_30647.png Static_analysis__test_coverage__and_other_12__letter_words_2_30648_33911.png Static_analysis__test_coverage__and_other_12__letter_words_2_33912_37079.png Static_analysis__test_coverage__and_other_12__letter_words_2_37080_37583.png Static_analysis__test_coverage__and_other_12__letter_words_2_37584_37679.png Static_analysis__test_coverage__and_other_12__letter_words_2_37680_37727.png Static_analysis__test_coverage__and_other_12__letter_words_2_37728_37871.png Static_analysis__test_coverage__and_other_12__letter_words_2_37872_38039.png Static_analysis__test_coverage__and_other_12__letter_words_2_38040_38159.png Static_analysis__test_coverage__and_other_12__letter_words_2_38160_40475.png Static_analysis__test_coverage__and_other_12__letter_words_2_40476_40967.png Static_analysis__test_coverage__and_other_12__letter_words_2_40968_41099.png Static_analysis__test_coverage__and_other_12__letter_words_2_41100_43571.png Static_analysis__test_coverage__and_other_12__letter_words_2_43572_44135.png Static_analysis__test_coverage__and_other_12__letter_words_2_44136_45203.png Static_analysis__test_coverage__and_other_12__letter_words_2_45204_47627.png Static_analysis__test_coverage__and_other_12__letter_words_2_47628_55703.png Static_analysis__test_coverage__and_other_12__letter_words_2_55704_63155.png Static_analysis__test_coverage__and_other_12__letter_words_2_63156_63443.png Static_analysis__test_coverage__and_other_12__letter_words_2_63444_65183.png Static_analysis__test_coverage__and_other_12__letter_words_2_65184_72959.png Static_analysis__test_coverage__and_other_12__letter_words_2_72960_76325.png
OK, is the recording on? Microphone? Microphone? OK, hello, good morning. I'm Peter. I'm PostgreSQL developer, and this is the story of my life. So I want to talk about various tools that I and others have thrown at the PostgreSQL to find bugs and other issues. I'm glad with this room, because the other one was so dark that I was almost falling asleep. So let's start with something simple. Have you ever written a C program, like a one-off, to just do a test case, and you compiled it? And it seemed to compile fine, and you ran it and crashed immediately. And then you noticed you did everything wrong, because the compiler didn't warn you about any of the problems. So if you have done that, then you will know that without compiler warnings, a C compiler is worthless. So that is why we in PostgreSQL have put a lot of effort into both cleaning up existing warnings to the point of 1 or 0, depending on how you look at the flex issue. And continue to add new warnings. So this is the current set we're using by default. And if you just look through, the WL is obviously the big one, but if you look through all the other ones and you imagine how your life would be if you didn't have those, it would be quite sad. So that is the set we're using, and we're pretty comfortable with it. This is, I'll be on the screen here. This is actually what I tend to use in my own builds. So there are a couple groups here. The first two, or the first one is not really a warning option, but what it does, it's on glibc slash Linux. It changes the definition of several library functions. So they declare themselves that you shouldn't ignore their return values and things like that. And that, in combination with the other warning options, then gives you warnings if you missed checking the return code of a function, for example, which could lead to security problems. Yeah. How will we play in that? How will we play in that? Well, how much courage do you get? Well, we are currently not getting any garbage because of two reasons. One is I always have this on in my own build, so I would find it pretty soon. The other is that on Ubuntu, they have somehow hard coded this into their compiler, so they're always building this. So the original reason why we actually started cleaning up is that we got these reports from Ubuntu people. So I must be the subject of the calls that the covariate complaints are about. Yeah. We'll get to that. We'll get to the covariate in a moment. So logical op is also something I'm using. That finds issues like if you have an if statement life. If x is greater than 0 and x is less than 5, that would be a type of it. It would find that. So that's actually pretty useful. It actually has found at least one bug in Postgres a few years ago. I actually submitted patches to make both of these a default awhile ago, if you recall. But the problem is that in very old GCC compilers that people are still using, they produce a lot of garbage. Because I guess the older compilers were not as smart to produce useful. Is this just Tom's bug again? Is it what? Is this just Tom's, you know? No, this has nothing to do with that. Those were reasonable. I don't remember the details, but those were reasonable platch ones. They were just a few years older. And this is the story that is a problem with all of these. You want to add these other ones. The second group there is things that I sometimes also use. They haven't really found much. So you could use them. The third group there is something that I use with Clang specifically. And the ones below there are new in Clang 3.4. So they haven't really found anything yet. But they are also sort of on the horizon, on the road map, so to speak. But the problem is really you can't make these the defaults for the reasons I mentioned. Because different compiler generations produce different nonsense for that. So if you want to maintain what we have, you know, the zero count of the warnings, it's really hard to move forward. That's a kind of annoying issue. So it's going to, you know, people who are interested in this will basically have to do this manually going forward also. So you have to do a lot of work to make that. I think a better plan would be to have a built-in member or something like that. I don't want to spend too much time on the warnings. But yeah, there's obviously a lot of fiddling we could do to make this better. But the real problem is the built-in member doesn't tell you how to do it. Yeah, well, that's another issue that many issues here. That's good. All good notes. So one thing I always do is compile with a W error, which turns all warnings into errors. And I have done this successfully, so to speak, for almost five, eight years. So if I compile code and it flies by, I don't see the warnings. I need to have a boolean answer from the compiler. That's just my workflow. The problem is if you pass W error into configure, configure fails because it generates warnings internally as part of its testing. And the workaround I use, so to speak, is I have a shell alias that does this sort of thing. So if you want to do this, you can take a photo or look at the thing later. This works for me because actually the shell alias I use is much longer because it also picks different options for different branches and so on. But that, for me, is useful. And I think everyone else should look into that because it's really helpful. The problem right now is I'm always the first one to find the warnings if someone accidentally commits one. But I'm happy if I've chosen this path. Here is a chart I computed over the weekend, which caused this computer to heat up my office for a while as it was building all this. And this kind of shows the progress of compilers becoming more and more either smaller or annoying, depending on how you interpret it, and also us catching up with them. And you can sort of see a diagonal here as time progresses. The problem with this is going back to the W error issue is if you need to make a change, let's say you need to back patch something into 9.0 right now, you need to either very carefully look at the warnings or not that your compile generates, or you need to pull up this chart and then actually try to install GCC 4.4 and build that because that was current at the time. That is very annoying and pretty much an unsolved issue for working in back branches. So I don't know how to fix that, other than keeping a lot of old compilers around, which I essentially do. But not all platforms have the option to install all that. There's all things. It's kind of hard. In case you're curious what this is, a lot of the options in 8.3 was that the compiler was complaining that inlining failed, and that generates hundreds of warnings almost, and then we later turned that off. In GCC 4.6, they added a variable was set but not used, which apparently, it was good. Now the issue with this is a lot of developers built with assertions enabled. But if you turn the assertions off, you get a different set of warnings because sometimes the warning variable is used. Sometimes those sort of problems don't get detected for a long time until someone builds without assertions. Very difficult. Yeah, we mentioned already, Buildform doesn't do anything about warnings, so this doesn't help. But when the other talk, Rob Pike was mentioning, who is also one of the creators of Go, the programming language, which you can have different opinions about. But one interesting thing they did, they don't have any compiler warnings. It's either an error or not, which makes this sort of thing much easier, which I think is a good decision. So now we were talking. Yes? How soon I have to know what's the 36? Yeah, OK. So the 36, this is on Mac OS 10, version 10.9 specifically. Those 36, if you're on a Linux platform, you might not get those. This is a, I argue it's a bug. I filed a bug with GCC about it. Complains if the right-hand side of a comma operator does not have a side effect. OK, OK. And this is a new warning they introduced. And this happens in the system house of the operating system for various macro expansions. And they're not very smart about that. So you might not see this, but if you're me and you develop on this 50% of the time and you run with W error, then this is GCC 4.9 is broken for me at the moment. So I actually use 4.9. Yes, of course. I've never been able to get some of your other work. So you've always come up with various errors and some serious errors. So I would kind of grab the errors and then just go to the end of the line. And then print all the error messages. I know a lot of people have. I've seen in discussions that people have macros or things that do that. In my experience, it always works. Or if not, I fix it. But that's the other thing you can do also. I think that's a good point. Well, I do wonder. I mean, we could turn on the error for both forms. It is kind of, yeah. Obviously, there are obscure platforms where it is not zero at the moment. Because we're actually going to be able to limit GCC more on so much better. Yeah. Well, we're also pretty good on Clang. But of course, if you use, for example, the Solaris, whatever it's called, Sun, former Sun Studio, you get a bunch of different things. And it's not easy to turn them off. They don't really have these options. So on those kind of platforms, you pretty much screwed at the moment. Let's talk about severity. So the way the compiler options are designed is so they don't slow down the compilation so much. So the next level of intensity here, so to speak, is the static analysis tools that are specifically designed to produce essentially as many warnings as possible at the expense of being super slowly built. And the severity was kind of the first that at least came to our attention on a broad scale. So the severity came out of a research project, but it's a commercial company. They sell this product for a lot of money I hear to do statically analyze your C or C++ code and started doing that. In around 2005, they started this open source program. I think that was also somehow government sponsored and built a bunch of open source projects and made this publicly available. And they have expanded this platform over time. And now it's almost to the degree that anyone can sign up and submit builds there. But the results are still under some kind of license. So I can't really go into too much details without not knowing what I'm getting myself into. Are you associated with that company? We'll have another example, which is the Clang Scan Build, which is open source. So if you can drill into deeper, they're very similar. Just a clarity goes a little bit deeper. So first of all, we have kind of made the decision that the only people who should have access to this is the people who are also on the security team, because there could be security issues in there. So if you're not on that team, what you can do is submit your own project to Corrarian. Just download the Postgres code. And if you figure that out, you have the results yourself. That's homework. But if you're one of the circles, so to speak, and you want to have access to this, you can talk to Steven Frost, primarily. But I'm also an admin on this, and Magnus is also an admin on this, so we could help you out. This is the URL. You can go. I think we'd be, we want to get to a point where we actually have a whole thing cleaned up. I think we'd be more willing to go. As you can see, there's a lot of things. So I found in the commit log 71 issues mentioned. Well, that's the debate of, well, it's hard to have that argument about a 1,500, right? It would be more easy to have that argument about a 150, right? So we don't know. We have been through, yes, between you and myself and others, we have been through a lot of those. But so, yeah, I'll show an example in a moment of what that looks like. I found 71 commits, which doesn't mean there was 71 issues. There were probably more in the older of 200 or 300, because a lot of these commits bunched a bunch of things together. So a lot of those are in the vision of memory leaks or memory access problems. So as it says here, there are 1,501 issues outstanding. Again, we have disqualified most of those, but who really knows? Again, if you're one of the developers, you can get access to that if you want. Coerity put out a notification that they detect a heart bleed bug, which was a recent security vulnerability. So they added that a week after it came out. So that didn't help. But that's the sort of level they work on. So they do taint checks of where data comes from and warn you about if you're using data that you get from external sources into a good way. Problem with that in Postgres, we actually do that a lot. If you think about what Postgres does a lot is read the block from disk and trust it implicitly that this has the right structure. And then just assumes, oh, if there's a length right here of that, we assume there's that much. And if that doesn't work out, it's just bad things happen. So essentially, we trust whatever's in the data directory implicitly. So all these taint checks don't really work for us. Which there's probably a lot of taint checks amongst those 1,500, but who really knows if there's really one that does the problem? Another example of this is if, for example, PG-based backup, there was an issue out of that store there. If PG-based backup pulls down incorrect data from the server, you can make it crash potentially by having a division by 0 somewhere, as possible. It's not a huge issue if your goal is not to make PG-based backup crash. The opposite of that would obviously be a much bigger problem if you send incorrect data up to the server and the server crashes because you send 0 somewhere. But they can't really tell what is more important than what. So that's hard. Let's see if I can pull up a web browser here on this. Where did it go? Oh, there it is. Cursor is there. Now it's here. This is very exciting. So this is the dashboard you get here. It's a little bit smaller. The back branches don't seem to really work because they only find one defective. Well, the actual interesting thing is this one here, where you can go through all the problems here. And the screen is a little bit funny now. This is interesting. I just looked at this the other day. So this is, you have op here compared to this. And then down here, you compare to that. And there's nothing in here that changed that. I just saw that last night and it looked kind of funny. So that doesn't. I would pick so. I didn't. I seriously didn't pick that. I just looked for an easy example. So arguably a compiler could detect that also because it's pretty straightforward, but obviously it needs more time to analyze that. So this is the level of issues you got here. So we will get him an account. It is pretty cool. You can waste a lot of time in there. So the open source alternative to that is Scanville, which came out of the. I'll show you later. Which came out of the LLVO. Stephen, you want to explain how queries built in your basement and then it's uploaded. No, no. OK. Hey, it's not the same physical box that I could pick from. So I'm going to go on and talk to you. No, it's not. I'm just saying. So it is basically someone in the infrastructure team regularly builds Postgres and then you upload some binary blob to them and they analyze it. So it is. Now we actually have more resources in terms of VM space. So Scanville is kind of an open source alternative that came out of the LLVM project. This is the URL. So if you want to use that, here's a recipe. So you can try that yourself. Most reasonably up-to-date platforms should have this inner package that might be called LLVM or Clang. Yeah, I've heard that the recent build is from the recent version of the stream build that introduced the number of the two-lane fault-pot. It is, yes. It is. It seems like recent versions of the bugs. It is an aggressive development. So you have to kind of, they do stuff and you do stuff in parallel. So yes. That's how I use it. It didn't have the built-in effect that was also a lot of error. And then you put an abort on that next thing. And you also don't figure out the effect. So we'll kind of do that. But it's not like that. OK, let's talk about it in a second. Yes. So if you want to build that, you download that tool. The O is basically where it stores its output. You configure it, you ought to build with enable assert. And then you just build it. It takes between half an hour and an hour or so. Then it tells you how many bugs it found. And then what is it? Oh, I'll explain in a moment. You don't have to, but it will not be. It won't work too well. And then there is this tool called ScanView, which pulls up the result in a HTML report, similar to what CorelD just showed. So let's look at it. So here's the result. As you can see, here are the bugs. Let's look at fancy JavaScript here. Let's look at 1. Division by 0. So here's all the places where you have division by 0. So you can see all the Q sort will fail if you pass the size of 0. That's by design. So you can't really do anything about that. The print routines will fail, for example, if you print the result set with 0 columns, which you wouldn't normally get, but as you know, we can fake that with tables with 0 columns. But the reason why this is not a problem is that tables with 0 columns also have 0 rows, so no row is actually printed. So if you could fake up a result set, you could crash this. But in practice, it's not a problem. The fix in this whole situation is that you put an assert somewhere in there that will just say, assert column count is greater than 0. And then you'll still crash, but in a more controlled way. And this tool will know about it and those things go away. That's why you need to build the assertions. Problem is, in the back end, you can turn the assertions off at runtime. So none of that will work, because the code path analysis always has to analyze the path. Well, what if assertions are turned off? Then I have to look at this path here. So that just doesn't work. So this tool doesn't know about that. Corverty has somehow, they said they had sort of manually tweaked something so they know about that. No, we have that problem. Maybe that used to be a while ago. They have the same class of problems in any case. So, sign values go up a bit on the front. So, Bison generated code apparently has lots of problems that you don't know. Hard to argue about that. Here is something in parser. And see, it gives you kind of this path here. You can go look, see how this comes about. So this is this function, which gets in a list of arguments. And then, no, this is all. Well, and then, you know, so this goes, loops around here. And assigns the arguments of the list to this array, which is this fixed size. So if you pass in the list as longer than that, you will have problems down here. This will not actually fail in practice, because like three levels up, there's a check that you can't, you know, on a more semantic level that you can't pass in anything longer than, you know, from MaxArk. So, right. So the fix again here would be to pass in an assertion, to put in an assertion. But the problem is, this will fix that. But as you know, this is not actually the problem it's complaining about, right? This problem is down here, where it loops through that same thing again, and assigns it back into a list. But it has no knowledge about lists, so it doesn't know that by exercising this loop, you do a number of steps of something. It doesn't know that at all. So it would have, even if you put in an assertion at list length or something, it has no knowledge of how these structures work. Because all the lists are in the user space, so to speak. I don't think there's anything you can do here to. It doesn't know that that's a list that has properties of any kind, so that's hard. Let's do no one. Do you assert it on the list? Do you specifically know what the assertion is? No. If you just assert, it will know. It knows what assertion does. Basically, this code path is not relevant. So Unix API, I had something in the Swift logger. This is a common problem, passing a null pointer to a function that shouldn't get one. And if you trace this all the way back, this value is assigned as a result of a PL lock. But this tool doesn't know that PL lock never returns null. So that might be something we can instrument somewhere. But I haven't figured out how to do that. But that would possibly help in a lot of cases. And there are GCC also has some attributes you can put on functions that tell you that this argument shouldn't be null, but I don't think they have an attribute that tells that this function never returns null. So that's actually what we use in a lot more cases. Assert would help here again. And there was a question about what about the E-log issue. And we added, she also had no return attributes to functions that also prune a lot of potential code paths and reduce a bunch of issues. So is there anything that you found that's like the thing and then gets a nine? But anything that actually understands what we do when we do the rare error signaling? And how we do the send up and whatnot? Or is it just null? No, it's not. It's not null. That's the only way to get a six-prone. No, I think you just do the null return. Because what I actually have to trace through and then see that is a long jump in some cases. But it potentially could do that. But I don't think it digs that deep. That's I think the problem. Because you have to have a trade-off of how long this thing is going to run, I think. I haven't got any tools to do it. I'm just wondering. Is that one of the big down-plates of all these tools that none of them understand that? So it's kind of a tender for them to do it. So this is the number of reported bugs we've had over time. And at some point, in 9.3, I think, is when we put in a bunch of null returns. And as you can see, that helps a lot. Pre-requisite for this is that you get clang to build it out warnings. Because it's kind of the same foundation. So a lot of the old releases have just basic clang bugs that we didn't attend to at the time. The there's no particular reason why 9.4 has more. I was kind of surprised by that. But there doesn't seem to be any systematic change there. It's just certain unrelated changes have potential issues. But it doesn't seem to be serious. The star column there is kind of a branch I have made myself where I have made assertions that you can't turn them off. So it prunes certain code paths and did a few other possibly controversial tweaks to kind of see how it can get that number. Ideally, we'd like to get that to zero. Because again, this comes back to the warning versus error issue. If you always have to compare numbers, you're never going to get really a bullying answer. But as was mentioned, scan build is still in development. And things come and go. So it's kind of hard. But this is a useful tool that everyone can use, open source. And you can play around with it. So Richard, you're using that apparently as well? Yes. The older versions of the new ones. Yeah. They're a lot easier. So yeah, I should have, if I had more, if I would have wanted to stress this laptop more, I could have done this for N releases. And it would probably take many days to build. So another static tool that came out of LVM is the address sanitizer, which actually came out of Google for their Chrome browser, I believe. But it's now part of Clang Compilo and GCC 4.8. You can also use it in GCC. So the way this works, you give it this option sanitize address, this one is the key one. And then these are recommended to get better backtraces, but they're not the key here. And then you build it, you run whatever check test suite you might have. And if there's a problem with whatever this checks for, which is mainly buffer overruns, then it will crash at this point. And then it will print to standard error a cryptic backtrace. And then there's this script that you can run this backtrace through, and you will get a backtrace. I can show an example. What is it? I don't know what these do. They're just recommended to get better backtraces. I've never actually seen that. This is a Clang option, so I don't think you might not have seen this in GCC. This is more well-known. Now, I don't know what this does. It's in the recommended set of options. So one issue we found, this might look familiar to some, is this issue in a non-issue. That's why this was ignored for a long time. If you run the name, it will do eventually a stir copy of these two things. And there is a certain case where these are the same. So you would think that's fine, because it just copies it over the top of itself. And for the last 20 years or so, that did work fine. But this thing was founded for strictness reasons. And it was ignored for a long time. But then some new compiler in MacOS 10.9 actually complained about this or crashed here, just as part of the regular compilation, so at that point, it needed to be fixed. And then every scientist was happy. And once you fix that, this is a real issue that we found. Here, you get the user like rule name, which is a string constant or a C string. And you cast it to a datum, pass it to this prepared plan, which expects a name datum there. And in most cases, that's fine, because the name is just a 64 byte string. But at some point later down here, this string gets copied into another copy of itself. And that will then copy the full 64 bytes, whereas this is not 64 bytes long. So that's a confusion between C string and name. And so this could have been a real problem when someone is actually at the end of memory. So this was useful, and this was fixed. So the problem with address sanitizers is just whenever it finds the first problem, it crashes. So you need to fix that, or at least work around it before you can get to the next point. So how do you do that? I don't remember the name. OK, I didn't find. Well, obviously, you can just comment that out, and you can pass by, right? So it's just for analysis. But it's a. I'm not familiar with that. But. I mean, what would anyone want to add? This is the. But I know you have a lot of crates to do it already. So this is the current issue. This is as of now. Which is actually kind of weird, because it will work fine on 32-bit builds, but it will do this crash on 64-bit builds. And this was discussed on the main list a few months ago, and it was actually argued that this is not a real problem. So you don't have to worry about it too much, but it will crash this way if you do it right now. So it would be worth fixing at some point. Are you similar to what I've seen in Valgrind? So yeah, so Valgrind has received a lot of attention in the community lately, and it's a much more general tool. Actually, on the website, they now described themselves as it's a just-in-time compiler, a pluggable just-in-time compiler. So it can do a lot of things now. But obviously, it's more known as a memory-checking tool. And yes, you can also use Valgrind. They, all of these tools overlap in some degree. The covariate can check find memory leaks. If it can reason about it statically, Valgrind will find memory leaks just because it keeps track of it. And it's more guaranteed to have a good answer. So the important thing, I would like to do more with Valgrind. And I think a lot of people here are of the same opinion. You can do a lot of things. So here's a couple ways you can use it. This is, I think, something that Hakey posted just the other day as an example to do back-end work. Something I've looked at. So here's a wiki page that was put up recently to also cover some of that. One thing I'm kind of interested in is also memory leaks and front-end programs. For example, in PGDump, memory leaks could be kind of a problem, but are not well tracked at the moment. If you actually do this, you will get a memory leak. But as you will see, it's kind of a stupid one. This is what you get from PGDump right now. But as you can see, the memory leaks are in undisputed vacuum. We just had the same problem. So yeah, here it is. So here, this is every program in Postgres, or at least client programs, this problem that allocates a few strings for local services, but you know, the program then doesn't free them. So any time you do this, you will get this sort of trace. And as we know, we can add suppression files to get rid of those, which I would like to do. Basically, you get all of these things down to zero. And then be able to run test suites again with a Boolean answer at the end, whether there are any problems or not. But we got to fix some of those first. So this was specifically a leak check. A lot of the other work that has gone in in the back end is access to memory that you shouldn't access of various kinds, which is what this down here would find. There's been a lot of work on that lately. I only found actually 19 commits mentioning ball grind. So we haven't really done a whole lot, but there was a lot of activity just lately. So one of my goals is to basically be able to run this automatically and just give you an exit status at the end. We look very concerned, Greg. Last time, no, last time it was ball grind. It's a old, curvy runtime. It wasn't static. Yes, this is, yes, I should have made the segue better. So the, uh, Is that so? I haven't had that problem, really. You would know a lot about running things on smartphones. So I'll believe you, yes. Yeah, if you're interested, this is, this wiki page is all we have at the moment. It doesn't even go into this much detail. So a lot of documentation work can go into this. Trace children is important in the back end because of the forking. I don't actually remember right now what Trace Orange is us and the read the bar info is just so you get a better back trace. Do you know what the track Orange is this about? It tells you where a member is allocated. It tells you the same structure as you. Okay. Does that have to do with the Palog instrumentation we added recently? Well, the puzzle of instrumentation is, you know, usually to, you know, what you get from the pool. But yeah, it will tell you where you access the instrumentation number, for example, like, you know, where it's located. So like that's very useful. You say this runs a lot slower. Yeah, it's like, I think it's like, why do you use a lot of the methods and then I'll just make the slides a little bit different? Right. So I think some more documentation here would be useful. And I think also if you do run and report problems, you've got to report on how you ran it because there's different things you can check with all of them. But so I see this sort of as a progression of tools that get more sort of intensive. So you can, you know, start with compiler warning, static analysis finds a lot of memory issues as well. But they can only reason why it's static. These are runtime tools and they have runtime knowledge of what's actually happening. So that's great. So in order to actually get anything useful out of this specifically, but also this, you need to have tests that exercise this, right? So, and we talked about some of that yesterday. So there was a company called Sun a long time ago that in the PostgreSQL community, and they actually submitted this, that they contributed this feature. And it wasn't me, it was someone else. So this has been there for a long time. I just want to remind people how this works. If you want to find out whether something, what the test coverage is of any given test or anything you run yourself, use this, enable coverage, make, then coverage clean just deletes the HTML output from the previous run. After a long, many puzzling nights, I discovered it's best to run with max connections equals one so you don't have any concurrent stuff happening because the output files override each other at runtime. And then it just produces garbage. So that might be useful knowledge. And then it produces HTML output and you can look at it in your browser. Let's look at that. I have a Jenkins job that does that. Looks like this. Yesterday, Hickey was complaining that his speech is vacuum is not exercise, which you can see here. So that's a lot of arguments about how much coverage you need and all that. So that's a different problem. But red is probably not good. So I think we can agree with that. So. You're a good speaker. SPG. Yeah, right. Well. You're a good speaker. I'm working. I think coverage has a low. Yeah, it's sort of a coverage. Yeah, so we know we don't really have a lot of knowledge about that. But we're knowing that we have a problem with not testing the state-based problem. Right. Yeah, and you know all I'm. I think it would be one of the devs to have a slave follow up with the problem of getting the slave into the state. Yeah. So. You had this thing that was like an AM, Jack saying. So obviously, this is not about how to design tests and stuff that's a bigger problem. I think this is more like a 10% to 20% solution to issues that we could possibly have. Actually, no, design tests is obviously a much harder question. So let's talk about go to fail. We talk about hard bleed. Let's talk about go to fail. So for those who don't know, go to fail was an issue in the Apple SSI libraries where they had accidentally duplicated a go to statement. And then it skipped a bunch of. What is it? Right. So I tried to replicate it, and I made this patch in Postgres just locally. So don't worry about it. And see what would happen. So obviously, pg-indend would be kind of helpful in the situation. Just on a side note, that's good. Also, if you do this, tests will fail because the protocol is all messed up. So you will see this. But this could be in the actual SSL code in Postgres as well, which also has goto's in it and is not exercised in the test suites that people usually run. So this could be a big problem. And there was no compiler warning. Scan build didn't say anything about it. I couldn't check severity. I have more confidence that they might find that. But I couldn't submit my own build. But this is not a trivial problem to adjust. So this could happen. Could you pass the right option to c-1? If you know that option, I'd be very interested because I tried a lot of them. If they remove it, then you get it. They remove it. Yes, there was a long time ago in GCC on that code analysis, which they removed because it didn't work. And you need it. Yeah, it's a hard problem. Again, in languages like Go, for example, they have made a strict decision in the language design that that code is not allowed. So they will give you an error. In Java, they have made a compromise, but it's a well defined compromise of when it's allowed or not. In C, there's a lot of things that are possible with long jumps and if-dabs. And so it's not possible to design an option that works for everyone. You're claiming that there is now a brand new option that might catch that. I'd be very interested. Yeah, again, if you run a tool, I'm not familiar with lint, but it will then give you possibly thousands of unreachable code in the PostgreSQL base. That's the problem. So yeah, this is an open question. I don't know if anyone has a good idea. I just tried the usual tools that we have available. If anyone knows how to fix that, it would be a great addition to this suite. I think so, right? Yes. But we'll see. What's Chris? Well, it's splint is on my list of other things to try. I should have had lint here. I've played with lint on and off. Again, if it gives you 1,000 reports, it's not helpful. It needs to be really- and splint isn't the same category. It has promise, and I've heard good things about it, but it needs to be tuned to hack out of it to understand everything that's going on. And it has a bunch of options like are you using C this or C that or Pro 6 or Unix, what are we using? We have a really custom combination of all these things. How do you tell that? It's very hard. Lang has a bunch of other sanitizers that are worth trying out, but they haven't really done anything for me. Sanitized undefined would find undefined behavior. I've never found anything with that, but that's probably good. The memory sanitizer only works on a limited amount of platforms, so it's not really a good solution. But there's a lot of overlap with Allgrind and Adresinitizer and any of these static analysis tools. Obviously, we don't use threats, but there's another thing out there. Mention splint, this Viva PBS Studio. Some people might remember that. That came out on the Hackers mailing list around New Year. This is a commercial tool where they threw that at Postgres and reported some of the issues back. And someone were fixed, and it's kind of an odd mix of things that they reported. But someone were good. But I downloaded the full list of things they reported, and the entire list contained 51,572 issues. So that's not really that helpful. And then someone said, well, this shouldn't actually be checked, because that's kind of a bogus check. So once I filtered that out, we're back down to 12,460. Still not helpful. And there, at that point, it's not useful to look at nobody wants to go through that. And they're very picky, for example, about using, for example, the long type. Every time you use a long type, it tells you, well, this type could have a different size, depending on architecture. So this tool might have come out around the time where 64 bit architectures were coming out, and people need to report to it. It's a commercial tool. There's a link here if you get those slides a little bit. I don't think that's going to help us much. I'll be on time. We're way over on tree. Here's the thing to worry about a little bit. So here's our lines of code. See how many committers actually contributed to that. So that's not going in the right direction, right? Then month since release and commit since release. And then how many commits per month? That's also not going in the right direction. And the last is hours per day. That's not changing. So that's a problem. We know we're all getting busy or so. I hope if you throw all these tools at it, we can at least make up the difference there a little bit. Obviously, it's not going to help find more complex issues that we know about. So I think this is more of a 10% to 20% solution if we put a lot of work in it. But I think it could be helpful. And I plan to do more work of that if someone wants to join in. It'd be great. So I think good goal for next release is to do a lot of work with get the ball grind and add the sanitizer. Working really well and add more tests to exercise some of the stuff that's not exercised yet. And if people have issues with building with WRR, let's help fix these issues if we can. Let me know if you need help with that. That is the last slide. So I think we've got to go. Any questions? OK, thank you. I'll be here all day. Thank you. Thank you.
I and others have thrown at the PostgreSQL to So if you want to maintain what we have, you want to do this, you can take a photo or look at statically analyze your C or C++ code and started that's the sort of level they work on. So they do of the bugs. It is an aggressive development. So you could crash this. But in practice, it's not a free them. So any time you do this, you will get structure as you. Okay. Does that have to do with hard problem. Again, in languages like Go, for going to help us much. I'll be on time. We're way