3.1 Regression Testing
McLennan, Michael
 Online simulation and more for Nanotechnology. In this next section I want to talk about regression testing. Now that you've built a tool, you can set it up so that you can test the tool. And more importantly, when you publish the tool, you can test it again and again and again. So, we've spent a few days now. You guys are pretty good now at building Rapture tools. You got the tool built. You're running it in your workspace. It works. In a minute I'll show you how you can get it published on Nanohub. So congratulations. You've got a tool. We'll go through all of this business about how you install and launch and test and finally publish the tool. But pretty soon you'll know how to do this part too. Congratulations. You've got a tool and it's published. So, what's next? Well, what's next is once your tool is published, people on Nanohub may put in wishes. They say, hey, I love your tool, but I wish it simulated gallium arsenide. Hey, I love your tool, but I wish it had a few more options for plotting. They'll put in different ideas of things that they want to see and there's wish lists on Nanohub to do that. They'll also find bugs in your tool. If you write software, you're writing bugs at the same time. Always. So, people will find problems and then they'll put in tickets on Nanohub. And guess what? If there's a problem with your tool, you get the ticket. Congratulations. Isn't that great? So, when you write software, there are bugs and somebody's got to fix it. And if you wrote the tool, you get to fix it. So, I'm sure you'll be making changes because nothing is ever perfect. The other thing that's going to happen is that your advisor will come along and say, hey, that's great. I can't believe you finished that tool so quickly. I've got some new ideas. Let's try adding this equation. Let's try adding that equation. So, your advisor will say, there's new physics that I want you to try and add this into the model and add that into the model and you'll have to add a few more inputs. So, you'll be making changes. No matter what, you're going to be making changes to your tool. I'm pretty sure. So, let me tell you a story about back in the day. Remember I told you about that sequel program that I wrote back in the day when I was a grad student? I got to this point. I was adding some new physics in. I had a whole section of parameters here in my physics. You know, eb to j and pi and all these different constants that I had fixed up. And I added just a little bit of new physics into my tool. I wanted to republish, right? So, at some point when you add that new physics, you'll get in and you'll click the link that says, please install the latest code for testing and approval and you'll want to republish your tool. Okay. Congratulations. You published your tool all over again. I'll tell you the rest of the story in a minute. But you've got this new tool. You've got all these different versions. Turns out on NanoHub you can keep multiple versions online. You can have like a version 3.01 that's the very latest and that's online. And then some people may say, well, hey, I want to keep this old version 2.1 online because a lot of people use it. It's got a different model, a simpler model. People also use it, but they want to keep the old version and the new version online. So, NanoHub lets you have multiple versions of the tool online. But getting back to my story, this is where I thought I was headed. My SQL program, way back in the day when it was 1985 and I was watching Back to the Future and playing the Tetris game. So I had this program and I wanted to make a change, a simple change, relatively simple change to the physics. It turns out there was a little bit of a code, part of the code where I had to do like a sorting routine. And I don't know, you guys may have seen the standard bubble sort code before, but this is kind of what it looks like. I needed to go through a list of values and I needed to compare two values. If this value is greater than that value, then swap them. That's how you do a bubble sort. And when you're swapping two variables, what you typically do is you set the temp to the first and first to the second and the second to the temp variable. You introduce the temp variable, right? So you can swap values around. So I made a two line fix, or a seven line fix to my program to sort the values like this. And then I started running again. All of a sudden my program got this bizarre physics. It's like started doing all this weird stuff. I got like this funny little curve in the thing and everything. And my major professor was very excited because we discovered some brand new quantum physics, right? But we looked more carefully into the code and it turns out, no, it was actually bizarre. The behavior of everything was completely bizarre because it turned out when I was doing my swapping here with my temporary variable, there was another global variable up in my state declaration. It was also called temp. Actually, I was using that for the temperature of the simulation. So as I was doing my bubble sort, my temp variable was overwriting the temperature of the simulation. And instead of being at 4 degrees Kelvin, it was going all over the place. It was like 193 degrees Kelvin and then 7,000 degrees Kelvin and then 0 degrees Kelvin. It was going all over the place. Because of this simple one line fix, the one line fix, I thought all I was doing is adding a few lines of code into the program. But I forgot about my global variable and all hell broke loose. So the point is, when you go to republish your program, you get everything all perfect, right? You'll test it, you'll check it, everything will be all perfect. You publish it on Nino. And then a bug comes in and you'll make a five line fix. And then you say, whatever, put it back up. And it turns out like the five line fix could be completely horrible to the rest of your program. Every time you go to publish, you really should check your program again. As if you were publishing it for the first time. Go through everything, check it all, right? And you really should. You really should. Nobody does that because it's a lot of work, right? You fix a bug and you put it out. You fix a bug, put it out. And you just want to be done with it because you're just fixing bugs. But really, you should be checking everything very carefully each time. So Rapture has a solution for that. Rapture can solve your problem because it has a testing tool. And the way it works is you build up a suite of test cases. That first time through when you're building your program. Take a few extra minutes and build up a bunch of test cases and save them. And then you might not ever need them again. But then the bug report comes in or the wish comes in or whatever. You go back to your tool and you add something. Just before you go to publish again, you can run back through these test cases. But instead of you doing all the work and taking all the time, you just push a button. And Rapture will do it automatically. Rapture will go through all the test cases that you started with and check to see if they're still working. They may not. They may be broken. And when they're broken, you can look at each of the ones that's broken and figure out why it's broken. In my case, the temperature was varying wildly and I was getting strange physics out. And I wish I had known that because if I had a series of test cases, I would have seen right away that these simple test cases are not working anymore. And I would have caught the error right away. So let me show you how it works. Maybe the best way right now, let me flip and show you kind of a demo of what it looks like. All right. So if I've got this program here, my Fermi function calculator thing, and I can run it. All right. Working great, right? Go ahead and republish the tool, right? Fantastic. Everything's working. If I run Rapture-Tester, it'll bring up all these test cases that I created before. All these different test cases that at one point worked perfectly for me. Now what I can do is say, select all and run. And this little regression tester will run back through all those test cases. First one worked great. Oh, wait a minute. That one failed. Oh, wait a minute. That one failed. It walks through each of the test cases and tries to replay the old results. And wherever it fails, it puts like a red X. The ones that are fine give me check marks. And the ones that are failing give me red Xs. So you can run through all of these tests. And again, this is much easier to do than it is to run them all by hand and look at them. And for any particular test, if it's working fine, great. But when it's not working, it'll show you what's wrong. So this one, in this case, it says the output status is bad. The test run failure was not expected. So if I look, it's giving me child process exited abnormally. In other words, that test case cordoned for whatever reason. And so that wasn't expected. It was supposed to give me a real result there. So, okay, something's wrong with that case. I should really look carefully at the one EV case. Zero EV gave me different things. Here it tells me that there's a result missing. It expected to get a certain string in the output. It expected ASDF, but for some reason it isn't there. And then here it's telling me the details have changed. If I click view, something changed. Oh, in this case, it's really innocuous. What happened was it used to be called fairy direct statistics. And now it's called fairy direct factor. You may look at that and say, oh yeah, I changed that. That's supposed to be different now, right? And in that case, that's okay. Don't worry about changes like that, because what we can do is, if I like that particular test case, I can say, this test case is good. It's a new golden standard for me. Everything's working fine. It was not supposed to have that string ASDF, and it is supposed to have the different label fairy direct factors. So if you look at a test case like this and you say that's working correctly, then all you have to do is click on new golden standard like this, and it says, are you sure that this is right? All right, this is perfectly right. And I click yes, and so now I basically told the system, don't worry, that test case is fine. So now when I run, that test case won't fail anymore. It's using the new output as the golden standard. So sometimes there are changes in your tests that really, things really did change. They're supposed to change. And other times there are problems like this where it really wasn't supposed to core dump, right? So you want to go through all of these tests that fail one by one and see what's happening. For some reason, 1EV is killing me with that test case. And so I can bring up my tool and I can set it to 1EV and run it. That ran there, but maybe it doesn't match the, oh, maybe it's a different, it may be a different temperature too. I should look at the test case. By the way, it leaves around all these run.xml files so I can check them out. And there's a series of test cases here that I can look at. I think it was test three that was failing. So I can look in test three and see what was the default value and the current value and all of that and kind of nail down what was going wrong. Let me get back to the lecture and show you now sort of the rest of it. Now you got a sense of how this works. Let me go back and explain how this is working. So from the normal Rapture runtime environment, we can run the tool. You remember it generates a driver, it runs your program, it does the run.xml, and we load up the result. And then normally that run.xml gets whisked away into your data results directory. It's like we've seen up to now. So normally that goes there. It'd be nice if that kind of stuck around because I'd like to use, I'd like to capture that and use it as a test case. And there are a couple of different ways of doing that. One way of stopping it from storing that result is to set the, to unset the session-dure variable. What normally what Rapture does is it looks for an environment variable called session-dure. And it tries to put that result in that session-dure. And if you can't find that environment variable, then it won't put the result there. So if you want to keep those run files sitting around, one way to do it is, is just by unsetting session-dure. Another way to do it is to create a driver file and run by hand. We've seen that too, right? When you run your program by hand with a driver file, it leaves the run file around. Because in that case you're running the program, not Rapture. But somehow one way or another you want to get your hands on a run file. And then you can move the run file into a test directory. So if I've got a run case like this one, if I move it into a test directory, I can keep it around for later as a test case. So imagine I'm running half a dozen different cases, grabbing all the run files, and moving them into the test directory. For each one of those test cases, if I just add a little label up to the top of it, then Rapture will recognize it now as a test case. So I can edit that run.xml file, and I can add a little test, label, and a description, and all of that. And this suddenly, that's how basically it makes it a test case for the tester. So all I need is a run.xml with a little bit of label stuff, and I have a new test for the tester. In the label, you can put a little pipe symbol, and that's how you create folders. This is a folder room temp, and then a value called 0EV. So if I have room temp and 0EV, that will show up now in a series of folders in the tester. And the description here is a little note to yourself to remember why was I testing this case. Maybe you should say I should always core-jump when you pass in 0EV. And maybe that's something you test for, because you can do negative testing with this too. You can expect failures when certain things happen. Or you can say I'm trying to make sure that it does work in 0EV, because that's a corner case in the problem. So just by taking your run file and adding a little bit of test stuff into it, you can create yourself a test case. And then, just like we saw before, you can run through all these test cases and see what's going wrong for each particular one. So we can run through and look at the various outputs and see what's failing. Oh, we should also make sure my laptop is plugged in. There we go. One other thing that I didn't show you before... We run through all these test cases looking for the failures. Remember, that's the one that Regold and I, so it's passing now. All right, so we have a complete set of test results. Sometimes you'll get outputs that are slightly different. This is a case where the output curve it says differs from expected value. And if I view it, it will actually show me subtle differences in the output. What looked okay to my eye is actually wrong. My old case, it expected to get... Let me see, the black is the expected result. So it expected to get this black curve and the red is the test result. And this is what it actually got. So here's a case where if I just looked at it with my eyes, I would think everything's fine. But yet, it's actually wrong, completely wrong compared to what I got when I first published the tool. So that's exactly the kind of subtle error that you want to be able to check. You notice it gives you two ways. You can see the output here as a string, then you can also see the output as a plot. And you can scroll through. Turns out, I guess there's one value in here that's correct. So at the value 0.5, we actually match the same value. And that's right in the middle when they cross, right? But everything else is wrong on this curve. So the tester catches errors like that too. So you can go through case by case and examine all the results and try to figure out what's wrong. Here it says there's an extra number in the output. So here somehow there's this extra number that the tool is reporting that isn't supposed to be there. And you should say, oh yeah, that's good. That's actually supposed to be there now. Or you should say, don't, I forgot. I deleted that out of the tool.xml and I didn't mean to. So one way or the other, you go through all of these different things. You notice that it'll tell you the subtle differences in labeling and the run status and so forth. And it's up to you to look at each one of these and go through point by point and try to convince yourself whether or not the difference is real and it should be there or that the difference is completely wrong and you need to fix your tool. And you can go back into the tool then and edit it. If you do some debugging on this particular Fermi one, you can see how it's working. And you might notice, ah, well here it was 4.2 Kelvin. So if it's 4.2 Kelvin and one EV, it's actually dying. I put in an explicit kill kind of thing. So if I simulate, oops, not that one. If I simulate 4.2 EV and one EV. Ah, child process exited abnormally. So that's the problem that I ran into in my test case. And I can fix the code just by saying, oh yeah, now that's fixed, doesn't do that anymore. So if I put in 4.2 and one EV, then now it works. Good, that works the way it should. And if I run my tester and select, well, that particular case, let's say, and run it. Ah, that one's working now. So little by little, you can work your way through all of the failures that you have and convince yourself, yes, this is working now, working correctly. And you want to make sure that you get all green check marks down that column before you republish your tool. So the last thing you do is kind of double check. At some point, as we get more and more people using the tester, we may start using it as a standard for all the tools. And we'll be able to, on Nanohub, say, this is a tool to test sweep and pass. And this is a tool that doesn't have a test sweep or didn't pass the test the last time it ran. So we'll be using that as kind of a quality metric. In the field of science, they call this verification and validation. Verification is making sure that your code runs correctly like this. And validation is making sure that the model actually explains reality. So scientists do this very carefully when they're building codes like this. And in the future, we may start to, like I said, require that or use that for Nanohub. Right now, we don't because the tester is not quite feature complete. Like the builder, the tester is about halfway there, well, even a maybe less quarter or a third of the way there. It does some simple things. It handles most of the simple inputs like the builder. On the output side, it only handles a couple of outputs. It'll handle curves. But for example, may or may not handle images. It doesn't handle Unirec 2D and fields for sure. We haven't implemented that yet. So there's still some things in the tester that are not quite there. But for simple tools, it'll work great for you. So give it a try and see what you think. These are the kinds of errors that you'll be able to catch. The fact that an output value is changed, that an output value is missing or extra, and also that the input value has changed. Maybe the label change, the units changed, whether it's missing or it's extra. So you can track down all of those cases. And again, if you convince yourself that should be like that. I really did remove that input. It's different now. Then you can click New Golden Standard and force that test case to pass. From then on, that'll be used. So here's the assignment for you guys. You have a nice tool working now. Yeah, actually we might want to back up and use the very oldest version of this tool that you guys created. Again, because the tester is not feature complete. I'm not sure it'll work very well with all the fancy stuff. The notes and the groups and all the enable and disable and all that stuff that you put into your latest Spirograph. I'm not sure that works in the tester. So if we back up to the plain Jane version of your Spirograph that just has three numbers and produces a Spirograph plot. You can create a test suite for your Spirograph plot. So what you do that again is you make a test directory. You run your test cases a few times by hand. You take the run.xml files and copy them in. I want you to test those three cases. I want you to run the tool as a fancy cross, as a flower, as a palm branch. Create the run.xml, put it in the test directory, and then try to run the Rapture regression tester on it. At that point, if you run it, everything should run cleanly in pass. And then you can edit one of your tests and delete some of the numbers in the test case and then run it again. And at that point, your test will be weird compared to the current result. And you should at least be able to catch the error. And you'll say, no, no, my tool is working properly now. I know I corrupted the test, so then you can re-goldenize and make it all properly again. So give that a try. Back up to your early version of Spirograph. If you don't have a version, let me know because we can put one up on the bootcamp site that you can start from. And then create those three tests and mess around with the tester. Okay, let me show you my solution for the lab assignment here with the regression tester. So here I am in my directory. I can run Rapture and simulate different cases. And I want to build the set of regression tests here for various test cases with different values. And normally when I run Rapture, it'll give me results. But when I close Rapture and look around, I don't see the run files. Geez, if I could get my hands on those run files, I'd have test cases that I could use. So there are two ways to get your hands on those run files. One way is to copy them out of your data results directory. Under data results, there's all these other directories that you can see that all depend. They're all different workspace sessions that I've fired up. And you might wonder, well, which one's the current one? Well, there are two ways to find that. I can echo $session and it shows me that my current session is 3264L. So I can go into data results, 3264L, and there are a bunch of run files. So those are all the run files. Usually the run file with the highest number is the one that I just ran. So 78, 7, 3, 9, 3, 9. Geez, I'm blind. 3, 9, 2. Well, I probably wanted 3, 9, 4. That's the latest one. All right, so there's a run file. In fact, if I want to double check and see what it looks like, remember you can do rapture-load and it'll pop it up. Okay, so that's that run file. And it's an interesting one because it's 17, 10, and minus 3, and it gives me that funny-looking shape. So I'm going to actually grab that one as a new test case. I'll add another test case. So I'm going to copy that file from that directory back into my test directory. And I'll give it a name, I could leave it the same or I can give it a name funny-shape.xml. All right, so now in my test directory, I've got four test cases. Actually, I need to make that into a proper test case. And I do that by adding a test in here and slash test with a label and a description. Okay, description should probably remind you later when it breaks, like what was important about this test case. Now, another way to grab test cases for the run.xml is to, if you do the unset your session dir, and then if you run Rapture, all the cases that I run from here on, one and two. Okay, so that's another test case. And now if I look, that run file is sitting right there. So I don't have to go over and hunt it down in my data results directory. If I break that link and get rid of that session dir, the run file is sitting right there. And now I can take that run file and I can move it into my test directory and I'll give it a different name again, another.xml. Let me edit that one and give it a test description. With the label. And I should set the description. I'll just skip it. All right, so that's how I generate test cases. And some people were asking me, oh boy, how do I set my session dir back? Well, one way is just by starting a new X term. Because if I start fresh with the new X term, it'll have the session dir back. Or I can sort of copy and paste that value and do the export session dir equals. And can I copy and paste our middle mouse button? There we go. So now I've got my session dir back. All right, so now I've got all these different test cases. And if I run rapture dash tester, it'll bring it up and it finds them all. If it doesn't find some of your test cases, it's probably because you forgot to put test with the label in there and all that. So anything that it finds in the test directory that has a label on it, it can pull up as a test case. And you notice there's no description for that test case, but everything else has a description. So I can grab one test case or I can grab all of the test cases and run through them now. And oh, it's telling me there's something wrong with that test case. Let's take a look. So if I look at that test case and it shows me just the output is different, the result differs. If I click on it, it'll show me it expected these values and it got a bunch of other new values over here. And you can even see the difference on the plot here. The black line versus the red line right there is sort of the difference between this and this. And if you look at it very carefully, whenever you see a difference like that, you should always ask yourself, what the heck is going on? You look in your program, you convince yourself whether or not your program is correct. You should ask yourself, is my program right? In this case, my program seems to be doing the right thing. The latest result that I got, the red result, the test result, is smooth and correct. And the old result is actually weird and choppy. And you might ask yourself at this point, how did that test ever get in there? But maybe you just never noticed it. The last time you ran the test case, it looked okay to your eyeballs, but it's wrong. So at that point, you might say, well, all right, the test is wrong. My program is correct, the test is wrong. And in that case, I want to use the current output for my program as the new golden standard. Now, Rapture says, are you sure? Are you sure that the latest results are completely correct? And if so, then yes. So now if I run that test, now it's checking the output against my latest standard, right? So Rapture will look at all kinds of things. Let's say I run the Rapture builder and I'm going to open my tool.xml. And suppose I add a Boolean control in now to my tool.xml, and I'm going to save that. Uh-oh. You set a default value and save that. All right. So I'll save that tool.xml. Now when I run Rapture, you see I've got the extra control in there, right? And now when I run my Rapture-tester and I run all these tests, they're all failing. Why did they fail? Well, it's telling me that the input's changed. This test case doesn't specify that new Boolean input value. And it didn't, of course, because back in the day when I created these tests, I didn't have that control. But now I do. So the question is, is that okay? And again, if that's okay, then I can say, all right, keep that as my new golden standard. And unfortunately, you have to do it for each one. Yes, yes, yes, yes. Maybe someday there'll be some kind of batch thing where you can select them all. So now I've re-goldenized my tests again, so now they expect that control. And if I run them all, they'll pass cleanly because I changed the interface. So as you're building your tool and checking everything, you always want to ask yourself, is it something that I did in the, did I make a mistake in terms of my simulator, or did I make a mistake? Did I add something to Rapture or what? You remember, it's real easy to mess up your code. It's not very hard at all to make what looks like an innocuous change in your code, like 2.1, and save your program. Oops. Not that much of a change. And run your tests, and they're all failing again. And this time, I'm like, yeah, yeah, yeah, uh-oh. The output is actually completely different now from what it's supposed to be. Completely different. And it was like, oh yeah, I was debugging something and I added that stupid 2.1 in there. I didn't mean to. So you eventually track down the problem and you say, oh man, this is wrong. And you fix the problem and you run it again. Oops. With the tester. And now you can convince yourself that everything's running cleanly. So that's the whole point of the regression tester is to say, hey, I got all these test cases, and I want to make sure that when the temperature is zero or the temperature is 500 million degrees Kelvin or whatever, the electron volts is negative. I want to make sure my simulator doesn't crash. I want to make sure that it gives the proper value. And I didn't make any mistakes by accident because it's so easy, right, to change one character in your code and it screws everything up in the program. Oh, I got another one for you. Oh, I should have done this example case. Oh man. The classic example is I have a loop variable up here. I set I equals one, right, or I equals 10 for my loop counter. But wait, wait, I is also the imaginary number in MATLAB, right? So now, if I'm running my test case, select all and run, see a simple innocuous statement like I equals one that you wouldn't think twice about because you were putting in a loop counter, all of a sudden you completely broke your program. And you can look at each case and convince yourself, like, what did I do? Oh my gosh. Oh my gosh, what did I do, right? Is that even putting out a spirograph anymore? Let alone the wrong one. So when you do stuff like that, be careful. Check your code carefully and make sure when you go to publish, oops, I always forget the tester part, make sure when you're about to republish your tool, you always go back through your regression test suite and have it run cleanly. And again, my biggest apology is that right now the Rapture regression tester doesn't work for really complicated tools. We need to fix it and finish it so that it works with all the complicated output types because then it would really work for you guys and you can use it to check everything. But at least you know it's there and at least you know you should be very scared of any one-line change that you make to your code because the simplest one-line change sometimes can break everything.
 Online simulation and more for Nanotechnology.  In this next section I want to talk about regression testing.  Now that you've built a tool, you can set it up so that you can test the tool.  And more importantly, when you publish the tool, you can test it again and again and again.  So, we've spent a few days now.  You guys are pretty good now at building Rapture tools.  You got the tool built. You're running it in your workspace.  It works.  In a minute I'll show you how you can get it published on Nanohub.  So congratulations. You've got a tool.  We'll go through all of this business about how you install and launch and test and finally publish the tool.  But pretty soon you'll know how to do this part too.  Congratulations. You've got a tool and it's published.  So, what's next?  Well, what's next is once your tool is published, people on Nanohub may put in wishes.  They say, hey, I love your tool, but I wish it simulated gallium arsenide.  Hey, I love your tool, but I wish it had a few more options for plotting.  They'll put in different ideas of things that they want to see and there's wish lists on Nanohub to do that.  They'll also find bugs in your tool.  If you write software, you're writing bugs at the same time. Always.  So, people will find problems and then they'll put in tickets on Nanohub.  And guess what? If there's a problem with your tool, you get the ticket.  Congratulations. Isn't that great?  So, when you write software, there are bugs and somebody's got to fix it.  And if you wrote the tool, you get to fix it.  So, I'm sure you'll be making changes because nothing is ever perfect.  The other thing that's going to happen is that your advisor will come along and say, hey, that's great.  I can't believe you finished that tool so quickly. I've got some new ideas.  Let's try adding this equation. Let's try adding that equation.  So, your advisor will say, there's new physics that I want you to try and add this into the model  and add that into the model and you'll have to add a few more inputs.  So, you'll be making changes. No matter what, you're going to be making changes to your tool.  I'm pretty sure.  So, let me tell you a story about back in the day.  Remember I told you about that sequel program that I wrote back in the day when I was a grad student?  I got to this point. I was adding some new physics in.  I had a whole section of parameters here in my physics.  You know, eb to j and pi and all these different constants that I had fixed up.  And I added just a little bit of new physics into my tool.  I wanted to republish, right?  So, at some point when you add that new physics, you'll get in and you'll click the link that says,  please install the latest code for testing and approval and you'll want to republish your tool.  Okay. Congratulations. You published your tool all over again.  I'll tell you the rest of the story in a minute.  But you've got this new tool. You've got all these different versions.  Turns out on NanoHub you can keep multiple versions online.  You can have like a version 3.01 that's the very latest and that's online.  And then some people may say, well, hey, I want to keep this old version 2.1 online because a lot of people use it.  It's got a different model, a simpler model. People also use it, but they want to keep the old version and the new version online.  So, NanoHub lets you have multiple versions of the tool online.  But getting back to my story, this is where I thought I was headed.  My SQL program, way back in the day when it was 1985 and I was watching Back to the Future and playing the Tetris game.  So I had this program and I wanted to make a change, a simple change, relatively simple change to the physics.  It turns out there was a little bit of a code, part of the code where I had to do like a sorting routine.  And I don't know, you guys may have seen the standard bubble sort code before, but this is kind of what it looks like.  I needed to go through a list of values and I needed to compare two values.  If this value is greater than that value, then swap them. That's how you do a bubble sort.  And when you're swapping two variables, what you typically do is you set the temp to the first and first to the second and the second to the temp variable.  You introduce the temp variable, right? So you can swap values around.  So I made a two line fix, or a seven line fix to my program to sort the values like this.  And then I started running again.  All of a sudden my program got this bizarre physics.  It's like started doing all this weird stuff. I got like this funny little curve in the thing and everything.  And my major professor was very excited because we discovered some brand new quantum physics, right?  But we looked more carefully into the code and it turns out, no, it was actually bizarre.  The behavior of everything was completely bizarre because it turned out when I was doing my swapping here with my temporary variable,  there was another global variable up in my state declaration. It was also called temp.  Actually, I was using that for the temperature of the simulation.  So as I was doing my bubble sort, my temp variable was overwriting the temperature of the simulation.  And instead of being at 4 degrees Kelvin, it was going all over the place.  It was like 193 degrees Kelvin and then 7,000 degrees Kelvin and then 0 degrees Kelvin.  It was going all over the place.  Because of this simple one line fix, the one line fix, I thought all I was doing is adding a few lines of code into the program.  But I forgot about my global variable and all hell broke loose.  So the point is, when you go to republish your program, you get everything all perfect, right?  You'll test it, you'll check it, everything will be all perfect.  You publish it on Nino. And then a bug comes in and you'll make a five line fix.  And then you say, whatever, put it back up.  And it turns out like the five line fix could be completely horrible to the rest of your program.  Every time you go to publish, you really should check your program again.  As if you were publishing it for the first time. Go through everything, check it all, right?  And you really should. You really should. Nobody does that because it's a lot of work, right?  You fix a bug and you put it out. You fix a bug, put it out.  And you just want to be done with it because you're just fixing bugs.  But really, you should be checking everything very carefully each time.  So Rapture has a solution for that.  Rapture can solve your problem because it has a testing tool.  And the way it works is you build up a suite of test cases.  That first time through when you're building your program.  Take a few extra minutes and build up a bunch of test cases and save them.  And then you might not ever need them again.  But then the bug report comes in or the wish comes in or whatever.  You go back to your tool and you add something.  Just before you go to publish again, you can run back through these test cases.  But instead of you doing all the work and taking all the time, you just push a button.  And Rapture will do it automatically.  Rapture will go through all the test cases that you started with  and check to see if they're still working.  They may not. They may be broken.  And when they're broken, you can look at each of the ones that's broken  and figure out why it's broken.  In my case, the temperature was varying wildly and I was getting strange physics out.  And I wish I had known that because if I had a series of test cases,  I would have seen right away that these simple test cases are not working anymore.  And I would have caught the error right away.  So let me show you how it works.  Maybe the best way right now, let me flip and show you kind of a demo of what it looks like.  All right. So if I've got this program here, my Fermi function calculator thing,  and I can run it.  All right. Working great, right?  Go ahead and republish the tool, right? Fantastic. Everything's working.  If I run Rapture-Tester, it'll bring up all these test cases that I created before.  All these different test cases that at one point worked perfectly for me.  Now what I can do is say, select all and run.  And this little regression tester will run back through all those test cases.  First one worked great.  Oh, wait a minute. That one failed.  Oh, wait a minute. That one failed.  It walks through each of the test cases and tries to replay the old results.  And wherever it fails, it puts like a red X.  The ones that are fine give me check marks.  And the ones that are failing give me red Xs.  So you can run through all of these tests.  And again, this is much easier to do than it is to run them all by hand and look at them.  And for any particular test, if it's working fine, great.  But when it's not working, it'll show you what's wrong.  So this one, in this case, it says the output status is bad.  The test run failure was not expected.  So if I look, it's giving me child process exited abnormally.  In other words, that test case cordoned for whatever reason.  And so that wasn't expected.  It was supposed to give me a real result there.  So, okay, something's wrong with that case.  I should really look carefully at the one EV case.  Zero EV gave me different things.  Here it tells me that there's a result missing.  It expected to get a certain string in the output.  It expected ASDF, but for some reason it isn't there.  And then here it's telling me the details have changed.  If I click view, something changed.  Oh, in this case, it's really innocuous.  What happened was it used to be called fairy direct statistics.  And now it's called fairy direct factor.  You may look at that and say, oh yeah, I changed that.  That's supposed to be different now, right?  And in that case, that's okay.  Don't worry about changes like that, because what we can do is,  if I like that particular test case, I can say, this test case is good.  It's a new golden standard for me.  Everything's working fine.  It was not supposed to have that string ASDF,  and it is supposed to have the different label fairy direct factors.  So if you look at a test case like this and you say that's working correctly,  then all you have to do is click on new golden standard like this,  and it says, are you sure that this is right?  All right, this is perfectly right.  And I click yes, and so now I basically told the system,  don't worry, that test case is fine.  So now when I run, that test case won't fail anymore.  It's using the new output as the golden standard.  So sometimes there are changes in your tests that really,  things really did change.  They're supposed to change.  And other times there are problems like this where it really wasn't supposed to core dump, right?  So you want to go through all of these tests that fail one by one and see what's happening.  For some reason, 1EV is killing me with that test case.  And so I can bring up my tool and I can set it to 1EV and run it.  That ran there, but maybe it doesn't match the, oh, maybe it's a different,  it may be a different temperature too.  I should look at the test case.  By the way, it leaves around all these run.xml files so I can check them out.  And there's a series of test cases here that I can look at.  I think it was test three that was failing.  So I can look in test three and see what was the default value and the current value and all of that  and kind of nail down what was going wrong.  Let me get back to the lecture and show you now sort of the rest of it.  Now you got a sense of how this works.  Let me go back and explain how this is working.  So from the normal Rapture runtime environment, we can run the tool.  You remember it generates a driver, it runs your program, it does the run.xml,  and we load up the result.  And then normally that run.xml gets whisked away into your data results directory.  It's like we've seen up to now.  So normally that goes there.  It'd be nice if that kind of stuck around because I'd like to use,  I'd like to capture that and use it as a test case.  And there are a couple of different ways of doing that.  One way of stopping it from storing that result is to set the, to unset the session-dure variable.  What normally what Rapture does is it looks for an environment variable called session-dure.  And it tries to put that result in that session-dure.  And if you can't find that environment variable, then it won't put the result there.  So if you want to keep those run files sitting around, one way to do it is,  is just by unsetting session-dure.  Another way to do it is to create a driver file and run by hand.  We've seen that too, right?  When you run your program by hand with a driver file, it leaves the run file around.  Because in that case you're running the program, not Rapture.  But somehow one way or another you want to get your hands on a run file.  And then you can move the run file into a test directory.  So if I've got a run case like this one, if I move it into a test directory,  I can keep it around for later as a test case.  So imagine I'm running half a dozen different cases, grabbing all the run files,  and moving them into the test directory.  For each one of those test cases, if I just add a little label up to the top of it,  then Rapture will recognize it now as a test case.  So I can edit that run.xml file, and I can add a little test, label, and a description,  and all of that.  And this suddenly, that's how basically it makes it a test case for the tester.  So all I need is a run.xml with a little bit of label stuff,  and I have a new test for the tester.  In the label, you can put a little pipe symbol, and that's how you create folders.  This is a folder room temp, and then a value called 0EV.  So if I have room temp and 0EV, that will show up now in a series of folders in the tester.  And the description here is a little note to yourself to remember why was I testing this case.  Maybe you should say I should always core-jump when you pass in 0EV.  And maybe that's something you test for, because you can do negative testing with this too.  You can expect failures when certain things happen.  Or you can say I'm trying to make sure that it does work in 0EV, because that's a corner case in the problem.  So just by taking your run file and adding a little bit of test stuff into it,  you can create yourself a test case.  And then, just like we saw before, you can run through all these test cases  and see what's going wrong for each particular one.  So we can run through and look at the various outputs and see what's failing.  Oh, we should also make sure my laptop is plugged in.  There we go.  One other thing that I didn't show you before...  We run through all these test cases looking for the failures.  Remember, that's the one that Regold and I, so it's passing now.  All right, so we have a complete set of test results.  Sometimes you'll get outputs that are slightly different.  This is a case where the output curve it says differs from expected value.  And if I view it, it will actually show me subtle differences in the output.  What looked okay to my eye is actually wrong.  My old case, it expected to get...  Let me see, the black is the expected result.  So it expected to get this black curve and the red is the test result.  And this is what it actually got.  So here's a case where if I just looked at it with my eyes, I would think everything's fine.  But yet, it's actually wrong, completely wrong compared to what I got when I first published the tool.  So that's exactly the kind of subtle error that you want to be able to check.  You notice it gives you two ways.  You can see the output here as a string, then you can also see the output as a plot.  And you can scroll through. Turns out, I guess there's one value in here that's correct.  So at the value 0.5, we actually match the same value.  And that's right in the middle when they cross, right?  But everything else is wrong on this curve.  So the tester catches errors like that too.  So you can go through case by case and examine all the results and try to figure out what's wrong.  Here it says there's an extra number in the output.  So here somehow there's this extra number that the tool is reporting that isn't supposed to be there.  And you should say, oh yeah, that's good. That's actually supposed to be there now.  Or you should say, don't, I forgot. I deleted that out of the tool.xml and I didn't mean to.  So one way or the other, you go through all of these different things.  You notice that it'll tell you the subtle differences in labeling and the run status and so forth.  And it's up to you to look at each one of these and go through point by point and try to convince yourself  whether or not the difference is real and it should be there or that the difference is completely wrong  and you need to fix your tool.  And you can go back into the tool then and edit it.  If you do some debugging on this particular Fermi one, you can see how it's working.  And you might notice, ah, well here it was 4.2 Kelvin.  So if it's 4.2 Kelvin and one EV, it's actually dying.  I put in an explicit kill kind of thing.  So if I simulate, oops, not that one.  If I simulate 4.2 EV and one EV.  Ah, child process exited abnormally.  So that's the problem that I ran into in my test case.  And I can fix the code just by saying, oh yeah, now that's fixed, doesn't do that anymore.  So if I put in 4.2 and one EV, then now it works.  Good, that works the way it should.  And if I run my tester and select, well, that particular case, let's say, and run it.  Ah, that one's working now.  So little by little, you can work your way through all of the failures that you have  and convince yourself, yes, this is working now, working correctly.  And you want to make sure that you get all green check marks down that column  before you republish your tool.  So the last thing you do is kind of double check.  At some point, as we get more and more people using the tester, we may start using it as a standard for all the tools.  And we'll be able to, on Nanohub, say, this is a tool to test sweep and pass.  And this is a tool that doesn't have a test sweep or didn't pass the test the last time it ran.  So we'll be using that as kind of a quality metric.  In the field of science, they call this verification and validation.  Verification is making sure that your code runs correctly like this.  And validation is making sure that the model actually explains reality.  So scientists do this very carefully when they're building codes like this.  And in the future, we may start to, like I said, require that or use that for Nanohub.  Right now, we don't because the tester is not quite feature complete.  Like the builder, the tester is about halfway there, well, even a maybe less quarter or a third of the way there.  It does some simple things. It handles most of the simple inputs like the builder.  On the output side, it only handles a couple of outputs. It'll handle curves.  But for example, may or may not handle images. It doesn't handle Unirec 2D and fields for sure.  We haven't implemented that yet.  So there's still some things in the tester that are not quite there.  But for simple tools, it'll work great for you.  So give it a try and see what you think.  These are the kinds of errors that you'll be able to catch. The fact that an output value is changed,  that an output value is missing or extra, and also that the input value has changed.  Maybe the label change, the units changed, whether it's missing or it's extra.  So you can track down all of those cases.  And again, if you convince yourself that should be like that.  I really did remove that input. It's different now.  Then you can click New Golden Standard and force that test case to pass.  From then on, that'll be used.  So here's the assignment for you guys.  You have a nice tool working now.  Yeah, actually we might want to back up and use the very oldest version of this tool that you guys created.  Again, because the tester is not feature complete.  I'm not sure it'll work very well with all the fancy stuff.  The notes and the groups and all the enable and disable and all that stuff that you put into your latest Spirograph.  I'm not sure that works in the tester.  So if we back up to the plain Jane version of your Spirograph that just has three numbers and produces a Spirograph plot.  You can create a test suite for your Spirograph plot.  So what you do that again is you make a test directory.  You run your test cases a few times by hand.  You take the run.xml files and copy them in.  I want you to test those three cases.  I want you to run the tool as a fancy cross, as a flower, as a palm branch.  Create the run.xml, put it in the test directory, and then try to run the Rapture regression tester on it.  At that point, if you run it, everything should run cleanly in pass.  And then you can edit one of your tests and delete some of the numbers in the test case and then run it again.  And at that point, your test will be weird compared to the current result.  And you should at least be able to catch the error.  And you'll say, no, no, my tool is working properly now.  I know I corrupted the test, so then you can re-goldenize and make it all properly again.  So give that a try.  Back up to your early version of Spirograph.  If you don't have a version, let me know because we can put one up on the bootcamp site that you can start from.  And then create those three tests and mess around with the tester.  Okay, let me show you my solution for the lab assignment here with the regression tester.  So here I am in my directory.  I can run Rapture and simulate different cases.  And I want to build the set of regression tests here for various test cases with different values.  And normally when I run Rapture, it'll give me results.  But when I close Rapture and look around, I don't see the run files.  Geez, if I could get my hands on those run files, I'd have test cases that I could use.  So there are two ways to get your hands on those run files.  One way is to copy them out of your data results directory.  Under data results, there's all these other directories that you can see that all depend.  They're all different workspace sessions that I've fired up.  And you might wonder, well, which one's the current one?  Well, there are two ways to find that.  I can echo $session and it shows me that my current session is 3264L.  So I can go into data results, 3264L, and there are a bunch of run files.  So those are all the run files.  Usually the run file with the highest number is the one that I just ran.  So 78, 7, 3, 9, 3, 9.  Geez, I'm blind.  3, 9, 2.  Well, I probably wanted 3, 9, 4.  That's the latest one.  All right, so there's a run file.  In fact, if I want to double check and see what it looks like,  remember you can do rapture-load and it'll pop it up.  Okay, so that's that run file.  And it's an interesting one because it's 17, 10, and minus 3, and it gives me that funny-looking shape.  So I'm going to actually grab that one as a new test case.  I'll add another test case.  So I'm going to copy that file from that directory back into my test directory.  And I'll give it a name, I could leave it the same or I can give it a name funny-shape.xml.  All right, so now in my test directory, I've got four test cases.  Actually, I need to make that into a proper test case.  And I do that by adding a test in here and slash test with a label and a description.  Okay, description should probably remind you later when it breaks,  like what was important about this test case.  Now, another way to grab test cases for the run.xml is to,  if you do the unset your session dir, and then if you run Rapture,  all the cases that I run from here on, one and two.  Okay, so that's another test case.  And now if I look, that run file is sitting right there.  So I don't have to go over and hunt it down in my data results directory.  If I break that link and get rid of that session dir, the run file is sitting right there.  And now I can take that run file and I can move it into my test directory  and I'll give it a different name again, another.xml.  Let me edit that one and give it a test description.  With the label.  And I should set the description.  I'll just skip it.  All right, so that's how I generate test cases.  And some people were asking me, oh boy, how do I set my session dir back?  Well, one way is just by starting a new X term.  Because if I start fresh with the new X term, it'll have the session dir back.  Or I can sort of copy and paste that value and do the export session dir equals.  And can I copy and paste our middle mouse button?  There we go.  So now I've got my session dir back.  All right, so now I've got all these different test cases.  And if I run rapture dash tester, it'll bring it up and it finds them all.  If it doesn't find some of your test cases, it's probably because you forgot to put test with the label in there and all that.  So anything that it finds in the test directory that has a label on it, it can pull up as a test case.  And you notice there's no description for that test case, but everything else has a description.  So I can grab one test case or I can grab all of the test cases and run through them now.  And oh, it's telling me there's something wrong with that test case.  Let's take a look.  So if I look at that test case and it shows me just the output is different, the result differs.  If I click on it, it'll show me it expected these values and it got a bunch of other new values over here.  And you can even see the difference on the plot here.  The black line versus the red line right there is sort of the difference between this and this.  And if you look at it very carefully, whenever you see a difference like that, you should always ask yourself, what the heck is going on?  You look in your program, you convince yourself whether or not your program is correct.  You should ask yourself, is my program right?  In this case, my program seems to be doing the right thing.  The latest result that I got, the red result, the test result, is smooth and correct.  And the old result is actually weird and choppy.  And you might ask yourself at this point, how did that test ever get in there?  But maybe you just never noticed it.  The last time you ran the test case, it looked okay to your eyeballs, but it's wrong.  So at that point, you might say, well, all right, the test is wrong.  My program is correct, the test is wrong.  And in that case, I want to use the current output for my program as the new golden standard.  Now, Rapture says, are you sure?  Are you sure that the latest results are completely correct?  And if so, then yes.  So now if I run that test, now it's checking the output against my latest standard, right?  So Rapture will look at all kinds of things.  Let's say I run the Rapture builder and I'm going to open my tool.xml.  And suppose I add a Boolean control in now to my tool.xml, and I'm going to save that.  Uh-oh.  You set a default value and save that.  All right.  So I'll save that tool.xml.  Now when I run Rapture, you see I've got the extra control in there, right?  And now when I run my Rapture-tester and I run all these tests, they're all failing.  Why did they fail?  Well, it's telling me that the input's changed.  This test case doesn't specify that new Boolean input value.  And it didn't, of course, because back in the day when I created these tests, I didn't have that control.  But now I do.  So the question is, is that okay?  And again, if that's okay, then I can say, all right, keep that as my new golden standard.  And unfortunately, you have to do it for each one.  Yes, yes, yes, yes.  Maybe someday there'll be some kind of batch thing where you can select them all.  So now I've re-goldenized my tests again, so now they expect that control.  And if I run them all, they'll pass cleanly because I changed the interface.  So as you're building your tool and checking everything, you always want to ask yourself,  is it something that I did in the, did I make a mistake in terms of my simulator,  or did I make a mistake?  Did I add something to Rapture or what?  You remember, it's real easy to mess up your code.  It's not very hard at all to make what looks like an innocuous change in your code, like 2.1,  and save your program.  Oops.  Not that much of a change.  And run your tests, and they're all failing again.  And this time, I'm like, yeah, yeah, yeah, uh-oh.  The output is actually completely different now from what it's supposed to be.  Completely different.  And it was like, oh yeah, I was debugging something and I added that stupid 2.1 in there.  I didn't mean to.  So you eventually track down the problem and you say, oh man, this is wrong.  And you fix the problem and you run it again.  Oops.  With the tester.  And now you can convince yourself that everything's running cleanly.  So that's the whole point of the regression tester is to say, hey, I got all these test cases,  and I want to make sure that when the temperature is zero or the temperature is 500 million degrees Kelvin  or whatever, the electron volts is negative.  I want to make sure my simulator doesn't crash.  I want to make sure that it gives the proper value.  And I didn't make any mistakes by accident because it's so easy, right,  to change one character in your code and it screws everything up in the program.  Oh, I got another one for you.  Oh, I should have done this example case.  Oh man.  The classic example is I have a loop variable up here.  I set I equals one, right, or I equals 10 for my loop counter.  But wait, wait, I is also the imaginary number in MATLAB, right?  So now, if I'm running my test case, select all and run,  see a simple innocuous statement like I equals one that you wouldn't think twice about  because you were putting in a loop counter, all of a sudden you completely broke your program.  And you can look at each case and convince yourself, like, what did I do?  Oh my gosh.  Oh my gosh, what did I do, right?  Is that even putting out a spirograph anymore?  Let alone the wrong one.  So when you do stuff like that, be careful.  Check your code carefully and make sure when you go to publish,  oops, I always forget the tester part,  make sure when you're about to republish your tool,  you always go back through your regression test suite and have it run cleanly.  And again, my biggest apology is that right now the Rapture regression tester doesn't work for really complicated tools.  We need to fix it and finish it so that it works with all the complicated output types  because then it would really work for you guys and you can use it to check everything.  But at least you know it's there and at least you know you should be very scared of any one-line change  that you make to your code because the simplest one-line change sometimes can break everything.
Rappture_Bootcamp_3_1__Regression_Testing_0_215.png Rappture_Bootcamp_3_1__Regression_Testing_216_455.png Rappture_Bootcamp_3_1__Regression_Testing_456_515.png Rappture_Bootcamp_3_1__Regression_Testing_516_1079.png Rappture_Bootcamp_3_1__Regression_Testing_1080_1271.png Rappture_Bootcamp_3_1__Regression_Testing_1272_3131.png Rappture_Bootcamp_3_1__Regression_Testing_3132_3887.png Rappture_Bootcamp_3_1__Regression_Testing_3888_4355.png Rappture_Bootcamp_3_1__Regression_Testing_4356_4487.png Rappture_Bootcamp_3_1__Regression_Testing_4488_5255.png Rappture_Bootcamp_3_1__Regression_Testing_5256_5735.png Rappture_Bootcamp_3_1__Regression_Testing_5736_6719.png Rappture_Bootcamp_3_1__Regression_Testing_6720_9491.png Rappture_Bootcamp_3_1__Regression_Testing_9492_11435.png Rappture_Bootcamp_3_1__Regression_Testing_11436_17927.png Rappture_Bootcamp_3_1__Regression_Testing_17928_17999.png Rappture_Bootcamp_3_1__Regression_Testing_18000_18107.png Rappture_Bootcamp_3_1__Regression_Testing_18108_18359.png Rappture_Bootcamp_3_1__Regression_Testing_18360_18503.png Rappture_Bootcamp_3_1__Regression_Testing_18504_18743.png Rappture_Bootcamp_3_1__Regression_Testing_18744_19055.png Rappture_Bootcamp_3_1__Regression_Testing_19056_19115.png Rappture_Bootcamp_3_1__Regression_Testing_19116_20471.png Rappture_Bootcamp_3_1__Regression_Testing_20472_20939.png Rappture_Bootcamp_3_1__Regression_Testing_20940_22763.png Rappture_Bootcamp_3_1__Regression_Testing_22764_23723.png Rappture_Bootcamp_3_1__Regression_Testing_23724_23903.png Rappture_Bootcamp_3_1__Regression_Testing_23904_28007.png Rappture_Bootcamp_3_1__Regression_Testing_28008_29603.png Rappture_Bootcamp_3_1__Regression_Testing_29604_29951.png Rappture_Bootcamp_3_1__Regression_Testing_29952_31823.png Rappture_Bootcamp_3_1__Regression_Testing_31824_32651.png Rappture_Bootcamp_3_1__Regression_Testing_32652_33419.png Rappture_Bootcamp_3_1__Regression_Testing_33420_36119.png Rappture_Bootcamp_3_1__Regression_Testing_36120_36335.png Rappture_Bootcamp_3_1__Regression_Testing_36336_38027.png Rappture_Bootcamp_3_1__Regression_Testing_38028_40055.png Rappture_Bootcamp_3_1__Regression_Testing_40056_40691.png Rappture_Bootcamp_3_1__Regression_Testing_40692_42239.png Rappture_Bootcamp_3_1__Regression_Testing_42240_42491.png Rappture_Bootcamp_3_1__Regression_Testing_42492_46247.png Rappture_Bootcamp_3_1__Regression_Testing_46248_46439.png Rappture_Bootcamp_3_1__Regression_Testing_46440_46979.png Rappture_Bootcamp_3_1__Regression_Testing_46980_48707.png Rappture_Bootcamp_3_1__Regression_Testing_48708_48983.png Rappture_Bootcamp_3_1__Regression_Testing_48984_49535.png Rappture_Bootcamp_3_1__Regression_Testing_49536_50039.png Rappture_Bootcamp_3_1__Regression_Testing_50040_50351.png Rappture_Bootcamp_3_1__Regression_Testing_50352_51287.png Rappture_Bootcamp_3_1__Regression_Testing_51288_51755.png Rappture_Bootcamp_3_1__Regression_Testing_51756_52463.png Rappture_Bootcamp_3_1__Regression_Testing_52464_52799.png Rappture_Bootcamp_3_1__Regression_Testing_52800_53546.png
Online simulation and more for Nanotechnology. In this next section I want to talk about regression testing. Now that you've built a tool, you can set it up so that you can test the tool. And more importantly, when you publish the tool, you can test it again and again and again. So, we've spent a few days now. You guys are pretty good now at building Rapture tools. You got the tool built. You're running it in your workspace. It works. In a minute I'll show you how you can get it published on Nanohub. So congratulations. You've got a tool. We'll go through all of this business about how you install and launch and test and finally publish the tool. But pretty soon you'll know how to do this part too. Congratulations. You've got a tool and it's published. So, what's next? Well, what's next is once your tool is published, people on Nanohub may put in wishes. They say, hey, I love your tool, but I wish it simulated gallium arsenide. Hey, I love your tool, but I wish it had a few more options for plotting. They'll put in different ideas of things that they want to see and there's wish lists on Nanohub to do that. They'll also find bugs in your tool. If you write software, you're writing bugs at the same time. Always. So, people will find problems and then they'll put in tickets on Nanohub. And guess what? If there's a problem with your tool, you get the ticket. Congratulations. Isn't that great? So, when you write software, there are bugs and somebody's got to fix it. And if you wrote the tool, you get to fix it. So, I'm sure you'll be making changes because nothing is ever perfect. The other thing that's going to happen is that your advisor will come along and say, hey, that's great. I can't believe you finished that tool so quickly. I've got some new ideas. Let's try adding this equation. Let's try adding that equation. So, your advisor will say, there's new physics that I want you to try and add this into the model and add that into the model and you'll have to add a few more inputs. So, you'll be making changes. No matter what, you're going to be making changes to your tool. I'm pretty sure. So, let me tell you a story about back in the day. Remember I told you about that sequel program that I wrote back in the day when I was a grad student? I got to this point. I was adding some new physics in. I had a whole section of parameters here in my physics. You know, eb to j and pi and all these different constants that I had fixed up. And I added just a little bit of new physics into my tool. I wanted to republish, right? So, at some point when you add that new physics, you'll get in and you'll click the link that says, please install the latest code for testing and approval and you'll want to republish your tool. Okay. Congratulations. You published your tool all over again. I'll tell you the rest of the story in a minute. But you've got this new tool. You've got all these different versions. Turns out on NanoHub you can keep multiple versions online. You can have like a version 3.01 that's the very latest and that's online. And then some people may say, well, hey, I want to keep this old version 2.1 online because a lot of people use it. It's got a different model, a simpler model. People also use it, but they want to keep the old version and the new version online. So, NanoHub lets you have multiple versions of the tool online. But getting back to my story, this is where I thought I was headed. My SQL program, way back in the day when it was 1985 and I was watching Back to the Future and playing the Tetris game. So I had this program and I wanted to make a change, a simple change, relatively simple change to the physics. It turns out there was a little bit of a code, part of the code where I had to do like a sorting routine. And I don't know, you guys may have seen the standard bubble sort code before, but this is kind of what it looks like. I needed to go through a list of values and I needed to compare two values. If this value is greater than that value, then swap them. That's how you do a bubble sort. And when you're swapping two variables, what you typically do is you set the temp to the first and first to the second and the second to the temp variable. You introduce the temp variable, right? So you can swap values around. So I made a two line fix, or a seven line fix to my program to sort the values like this. And then I started running again. All of a sudden my program got this bizarre physics. It's like started doing all this weird stuff. I got like this funny little curve in the thing and everything. And my major professor was very excited because we discovered some brand new quantum physics, right? But we looked more carefully into the code and it turns out, no, it was actually bizarre. The behavior of everything was completely bizarre because it turned out when I was doing my swapping here with my temporary variable, there was another global variable up in my state declaration. It was also called temp. Actually, I was using that for the temperature of the simulation. So as I was doing my bubble sort, my temp variable was overwriting the temperature of the simulation. And instead of being at 4 degrees Kelvin, it was going all over the place. It was like 193 degrees Kelvin and then 7,000 degrees Kelvin and then 0 degrees Kelvin. It was going all over the place. Because of this simple one line fix, the one line fix, I thought all I was doing is adding a few lines of code into the program. But I forgot about my global variable and all hell broke loose. So the point is, when you go to republish your program, you get everything all perfect, right? You'll test it, you'll check it, everything will be all perfect. You publish it on Nino. And then a bug comes in and you'll make a five line fix. And then you say, whatever, put it back up. And it turns out like the five line fix could be completely horrible to the rest of your program. Every time you go to publish, you really should check your program again. As if you were publishing it for the first time. Go through everything, check it all, right? And you really should. You really should. Nobody does that because it's a lot of work, right? You fix a bug and you put it out. You fix a bug, put it out. And you just want to be done with it because you're just fixing bugs. But really, you should be checking everything very carefully each time. So Rapture has a solution for that. Rapture can solve your problem because it has a testing tool. And the way it works is you build up a suite of test cases. That first time through when you're building your program. Take a few extra minutes and build up a bunch of test cases and save them. And then you might not ever need them again. But then the bug report comes in or the wish comes in or whatever. You go back to your tool and you add something. Just before you go to publish again, you can run back through these test cases. But instead of you doing all the work and taking all the time, you just push a button. And Rapture will do it automatically. Rapture will go through all the test cases that you started with and check to see if they're still working. They may not. They may be broken. And when they're broken, you can look at each of the ones that's broken and figure out why it's broken. In my case, the temperature was varying wildly and I was getting strange physics out. And I wish I had known that because if I had a series of test cases, I would have seen right away that these simple test cases are not working anymore. And I would have caught the error right away. So let me show you how it works. Maybe the best way right now, let me flip and show you kind of a demo of what it looks like. All right. So if I've got this program here, my Fermi function calculator thing, and I can run it. All right. Working great, right? Go ahead and republish the tool, right? Fantastic. Everything's working. If I run Rapture-Tester, it'll bring up all these test cases that I created before. All these different test cases that at one point worked perfectly for me. Now what I can do is say, select all and run. And this little regression tester will run back through all those test cases. First one worked great. Oh, wait a minute. That one failed. Oh, wait a minute. That one failed. It walks through each of the test cases and tries to replay the old results. And wherever it fails, it puts like a red X. The ones that are fine give me check marks. And the ones that are failing give me red Xs. So you can run through all of these tests. And again, this is much easier to do than it is to run them all by hand and look at them. And for any particular test, if it's working fine, great. But when it's not working, it'll show you what's wrong. So this one, in this case, it says the output status is bad. The test run failure was not expected. So if I look, it's giving me child process exited abnormally. In other words, that test case cordoned for whatever reason. And so that wasn't expected. It was supposed to give me a real result there. So, okay, something's wrong with that case. I should really look carefully at the one EV case. Zero EV gave me different things. Here it tells me that there's a result missing. It expected to get a certain string in the output. It expected ASDF, but for some reason it isn't there. And then here it's telling me the details have changed. If I click view, something changed. Oh, in this case, it's really innocuous. What happened was it used to be called fairy direct statistics. And now it's called fairy direct factor. You may look at that and say, oh yeah, I changed that. That's supposed to be different now, right? And in that case, that's okay. Don't worry about changes like that, because what we can do is, if I like that particular test case, I can say, this test case is good. It's a new golden standard for me. Everything's working fine. It was not supposed to have that string ASDF, and it is supposed to have the different label fairy direct factors. So if you look at a test case like this and you say that's working correctly, then all you have to do is click on new golden standard like this, and it says, are you sure that this is right? All right, this is perfectly right. And I click yes, and so now I basically told the system, don't worry, that test case is fine. So now when I run, that test case won't fail anymore. It's using the new output as the golden standard. So sometimes there are changes in your tests that really, things really did change. They're supposed to change. And other times there are problems like this where it really wasn't supposed to core dump, right? So you want to go through all of these tests that fail one by one and see what's happening. For some reason, 1EV is killing me with that test case. And so I can bring up my tool and I can set it to 1EV and run it. That ran there, but maybe it doesn't match the, oh, maybe it's a different, it may be a different temperature too. I should look at the test case. By the way, it leaves around all these run.xml files so I can check them out. And there's a series of test cases here that I can look at. I think it was test three that was failing. So I can look in test three and see what was the default value and the current value and all of that and kind of nail down what was going wrong. Let me get back to the lecture and show you now sort of the rest of it. Now you got a sense of how this works. Let me go back and explain how this is working. So from the normal Rapture runtime environment, we can run the tool. You remember it generates a driver, it runs your program, it does the run.xml, and we load up the result. And then normally that run.xml gets whisked away into your data results directory. It's like we've seen up to now. So normally that goes there. It'd be nice if that kind of stuck around because I'd like to use, I'd like to capture that and use it as a test case. And there are a couple of different ways of doing that. One way of stopping it from storing that result is to set the, to unset the session-dure variable. What normally what Rapture does is it looks for an environment variable called session-dure. And it tries to put that result in that session-dure. And if you can't find that environment variable, then it won't put the result there. So if you want to keep those run files sitting around, one way to do it is, is just by unsetting session-dure. Another way to do it is to create a driver file and run by hand. We've seen that too, right? When you run your program by hand with a driver file, it leaves the run file around. Because in that case you're running the program, not Rapture. But somehow one way or another you want to get your hands on a run file. And then you can move the run file into a test directory. So if I've got a run case like this one, if I move it into a test directory, I can keep it around for later as a test case. So imagine I'm running half a dozen different cases, grabbing all the run files, and moving them into the test directory. For each one of those test cases, if I just add a little label up to the top of it, then Rapture will recognize it now as a test case. So I can edit that run.xml file, and I can add a little test, label, and a description, and all of that. And this suddenly, that's how basically it makes it a test case for the tester. So all I need is a run.xml with a little bit of label stuff, and I have a new test for the tester. In the label, you can put a little pipe symbol, and that's how you create folders. This is a folder room temp, and then a value called 0EV. So if I have room temp and 0EV, that will show up now in a series of folders in the tester. And the description here is a little note to yourself to remember why was I testing this case. Maybe you should say I should always core-jump when you pass in 0EV. And maybe that's something you test for, because you can do negative testing with this too. You can expect failures when certain things happen. Or you can say I'm trying to make sure that it does work in 0EV, because that's a corner case in the problem. So just by taking your run file and adding a little bit of test stuff into it, you can create yourself a test case. And then, just like we saw before, you can run through all these test cases and see what's going wrong for each particular one. So we can run through and look at the various outputs and see what's failing. Oh, we should also make sure my laptop is plugged in. There we go. One other thing that I didn't show you before... We run through all these test cases looking for the failures. Remember, that's the one that Regold and I, so it's passing now. All right, so we have a complete set of test results. Sometimes you'll get outputs that are slightly different. This is a case where the output curve it says differs from expected value. And if I view it, it will actually show me subtle differences in the output. What looked okay to my eye is actually wrong. My old case, it expected to get... Let me see, the black is the expected result. So it expected to get this black curve and the red is the test result. And this is what it actually got. So here's a case where if I just looked at it with my eyes, I would think everything's fine. But yet, it's actually wrong, completely wrong compared to what I got when I first published the tool. So that's exactly the kind of subtle error that you want to be able to check. You notice it gives you two ways. You can see the output here as a string, then you can also see the output as a plot. And you can scroll through. Turns out, I guess there's one value in here that's correct. So at the value 0.5, we actually match the same value. And that's right in the middle when they cross, right? But everything else is wrong on this curve. So the tester catches errors like that too. So you can go through case by case and examine all the results and try to figure out what's wrong. Here it says there's an extra number in the output. So here somehow there's this extra number that the tool is reporting that isn't supposed to be there. And you should say, oh yeah, that's good. That's actually supposed to be there now. Or you should say, don't, I forgot. I deleted that out of the tool.xml and I didn't mean to. So one way or the other, you go through all of these different things. You notice that it'll tell you the subtle differences in labeling and the run status and so forth. And it's up to you to look at each one of these and go through point by point and try to convince yourself whether or not the difference is real and it should be there or that the difference is completely wrong and you need to fix your tool. And you can go back into the tool then and edit it. If you do some debugging on this particular Fermi one, you can see how it's working. And you might notice, ah, well here it was 4.2 Kelvin. So if it's 4.2 Kelvin and one EV, it's actually dying. I put in an explicit kill kind of thing. So if I simulate, oops, not that one. If I simulate 4.2 EV and one EV. Ah, child process exited abnormally. So that's the problem that I ran into in my test case. And I can fix the code just by saying, oh yeah, now that's fixed, doesn't do that anymore. So if I put in 4.2 and one EV, then now it works. Good, that works the way it should. And if I run my tester and select, well, that particular case, let's say, and run it. Ah, that one's working now. So little by little, you can work your way through all of the failures that you have and convince yourself, yes, this is working now, working correctly. And you want to make sure that you get all green check marks down that column before you republish your tool. So the last thing you do is kind of double check. At some point, as we get more and more people using the tester, we may start using it as a standard for all the tools. And we'll be able to, on Nanohub, say, this is a tool to test sweep and pass. And this is a tool that doesn't have a test sweep or didn't pass the test the last time it ran. So we'll be using that as kind of a quality metric. In the field of science, they call this verification and validation. Verification is making sure that your code runs correctly like this. And validation is making sure that the model actually explains reality. So scientists do this very carefully when they're building codes like this. And in the future, we may start to, like I said, require that or use that for Nanohub. Right now, we don't because the tester is not quite feature complete. Like the builder, the tester is about halfway there, well, even a maybe less quarter or a third of the way there. It does some simple things. It handles most of the simple inputs like the builder. On the output side, it only handles a couple of outputs. It'll handle curves. But for example, may or may not handle images. It doesn't handle Unirec 2D and fields for sure. We haven't implemented that yet. So there's still some things in the tester that are not quite there. But for simple tools, it'll work great for you. So give it a try and see what you think. These are the kinds of errors that you'll be able to catch. The fact that an output value is changed, that an output value is missing or extra, and also that the input value has changed. Maybe the label change, the units changed, whether it's missing or it's extra. So you can track down all of those cases. And again, if you convince yourself that should be like that. I really did remove that input. It's different now. Then you can click New Golden Standard and force that test case to pass. From then on, that'll be used. So here's the assignment for you guys. You have a nice tool working now. Yeah, actually we might want to back up and use the very oldest version of this tool that you guys created. Again, because the tester is not feature complete. I'm not sure it'll work very well with all the fancy stuff. The notes and the groups and all the enable and disable and all that stuff that you put into your latest Spirograph. I'm not sure that works in the tester. So if we back up to the plain Jane version of your Spirograph that just has three numbers and produces a Spirograph plot. You can create a test suite for your Spirograph plot. So what you do that again is you make a test directory. You run your test cases a few times by hand. You take the run.xml files and copy them in. I want you to test those three cases. I want you to run the tool as a fancy cross, as a flower, as a palm branch. Create the run.xml, put it in the test directory, and then try to run the Rapture regression tester on it. At that point, if you run it, everything should run cleanly in pass. And then you can edit one of your tests and delete some of the numbers in the test case and then run it again. And at that point, your test will be weird compared to the current result. And you should at least be able to catch the error. And you'll say, no, no, my tool is working properly now. I know I corrupted the test, so then you can re-goldenize and make it all properly again. So give that a try. Back up to your early version of Spirograph. If you don't have a version, let me know because we can put one up on the bootcamp site that you can start from. And then create those three tests and mess around with the tester. Okay, let me show you my solution for the lab assignment here with the regression tester. So here I am in my directory. I can run Rapture and simulate different cases. And I want to build the set of regression tests here for various test cases with different values. And normally when I run Rapture, it'll give me results. But when I close Rapture and look around, I don't see the run files. Geez, if I could get my hands on those run files, I'd have test cases that I could use. So there are two ways to get your hands on those run files. One way is to copy them out of your data results directory. Under data results, there's all these other directories that you can see that all depend. They're all different workspace sessions that I've fired up. And you might wonder, well, which one's the current one? Well, there are two ways to find that. I can echo $session and it shows me that my current session is 3264L. So I can go into data results, 3264L, and there are a bunch of run files. So those are all the run files. Usually the run file with the highest number is the one that I just ran. So 78, 7, 3, 9, 3, 9. Geez, I'm blind. 3, 9, 2. Well, I probably wanted 3, 9, 4. That's the latest one. All right, so there's a run file. In fact, if I want to double check and see what it looks like, remember you can do rapture-load and it'll pop it up. Okay, so that's that run file. And it's an interesting one because it's 17, 10, and minus 3, and it gives me that funny-looking shape. So I'm going to actually grab that one as a new test case. I'll add another test case. So I'm going to copy that file from that directory back into my test directory. And I'll give it a name, I could leave it the same or I can give it a name funny-shape.xml. All right, so now in my test directory, I've got four test cases. Actually, I need to make that into a proper test case. And I do that by adding a test in here and slash test with a label and a description. Okay, description should probably remind you later when it breaks, like what was important about this test case. Now, another way to grab test cases for the run.xml is to, if you do the unset your session dir, and then if you run Rapture, all the cases that I run from here on, one and two. Okay, so that's another test case. And now if I look, that run file is sitting right there. So I don't have to go over and hunt it down in my data results directory. If I break that link and get rid of that session dir, the run file is sitting right there. And now I can take that run file and I can move it into my test directory and I'll give it a different name again, another.xml. Let me edit that one and give it a test description. With the label. And I should set the description. I'll just skip it. All right, so that's how I generate test cases. And some people were asking me, oh boy, how do I set my session dir back? Well, one way is just by starting a new X term. Because if I start fresh with the new X term, it'll have the session dir back. Or I can sort of copy and paste that value and do the export session dir equals. And can I copy and paste our middle mouse button? There we go. So now I've got my session dir back. All right, so now I've got all these different test cases. And if I run rapture dash tester, it'll bring it up and it finds them all. If it doesn't find some of your test cases, it's probably because you forgot to put test with the label in there and all that. So anything that it finds in the test directory that has a label on it, it can pull up as a test case. And you notice there's no description for that test case, but everything else has a description. So I can grab one test case or I can grab all of the test cases and run through them now. And oh, it's telling me there's something wrong with that test case. Let's take a look. So if I look at that test case and it shows me just the output is different, the result differs. If I click on it, it'll show me it expected these values and it got a bunch of other new values over here. And you can even see the difference on the plot here. The black line versus the red line right there is sort of the difference between this and this. And if you look at it very carefully, whenever you see a difference like that, you should always ask yourself, what the heck is going on? You look in your program, you convince yourself whether or not your program is correct. You should ask yourself, is my program right? In this case, my program seems to be doing the right thing. The latest result that I got, the red result, the test result, is smooth and correct. And the old result is actually weird and choppy. And you might ask yourself at this point, how did that test ever get in there? But maybe you just never noticed it. The last time you ran the test case, it looked okay to your eyeballs, but it's wrong. So at that point, you might say, well, all right, the test is wrong. My program is correct, the test is wrong. And in that case, I want to use the current output for my program as the new golden standard. Now, Rapture says, are you sure? Are you sure that the latest results are completely correct? And if so, then yes. So now if I run that test, now it's checking the output against my latest standard, right? So Rapture will look at all kinds of things. Let's say I run the Rapture builder and I'm going to open my tool.xml. And suppose I add a Boolean control in now to my tool.xml, and I'm going to save that. Uh-oh. You set a default value and save that. All right. So I'll save that tool.xml. Now when I run Rapture, you see I've got the extra control in there, right? And now when I run my Rapture-tester and I run all these tests, they're all failing. Why did they fail? Well, it's telling me that the input's changed. This test case doesn't specify that new Boolean input value. And it didn't, of course, because back in the day when I created these tests, I didn't have that control. But now I do. So the question is, is that okay? And again, if that's okay, then I can say, all right, keep that as my new golden standard. And unfortunately, you have to do it for each one. Yes, yes, yes, yes. Maybe someday there'll be some kind of batch thing where you can select them all. So now I've re-goldenized my tests again, so now they expect that control. And if I run them all, they'll pass cleanly because I changed the interface. So as you're building your tool and checking everything, you always want to ask yourself, is it something that I did in the, did I make a mistake in terms of my simulator, or did I make a mistake? Did I add something to Rapture or what? You remember, it's real easy to mess up your code. It's not very hard at all to make what looks like an innocuous change in your code, like 2.1, and save your program. Oops. Not that much of a change. And run your tests, and they're all failing again. And this time, I'm like, yeah, yeah, yeah, uh-oh. The output is actually completely different now from what it's supposed to be. Completely different. And it was like, oh yeah, I was debugging something and I added that stupid 2.1 in there. I didn't mean to. So you eventually track down the problem and you say, oh man, this is wrong. And you fix the problem and you run it again. Oops. With the tester. And now you can convince yourself that everything's running cleanly. So that's the whole point of the regression tester is to say, hey, I got all these test cases, and I want to make sure that when the temperature is zero or the temperature is 500 million degrees Kelvin or whatever, the electron volts is negative. I want to make sure my simulator doesn't crash. I want to make sure that it gives the proper value. And I didn't make any mistakes by accident because it's so easy, right, to change one character in your code and it screws everything up in the program. Oh, I got another one for you. Oh, I should have done this example case. Oh man. The classic example is I have a loop variable up here. I set I equals one, right, or I equals 10 for my loop counter. But wait, wait, I is also the imaginary number in MATLAB, right? So now, if I'm running my test case, select all and run, see a simple innocuous statement like I equals one that you wouldn't think twice about because you were putting in a loop counter, all of a sudden you completely broke your program. And you can look at each case and convince yourself, like, what did I do? Oh my gosh. Oh my gosh, what did I do, right? Is that even putting out a spirograph anymore? Let alone the wrong one. So when you do stuff like that, be careful. Check your code carefully and make sure when you go to publish, oops, I always forget the tester part, make sure when you're about to republish your tool, you always go back through your regression test suite and have it run cleanly. And again, my biggest apology is that right now the Rapture regression tester doesn't work for really complicated tools. We need to fix it and finish it so that it works with all the complicated output types because then it would really work for you guys and you can use it to check everything. But at least you know it's there and at least you know you should be very scared of any one-line change that you make to your code because the simplest one-line change sometimes can break everything.