Keynote: Sustainable Research Software – as Code, as Paper, as Book
Zeller, Andreas
How can I make research software sustainable? Five basic rules from Software Engineering and Open Source development have the biggest effect: Use a repository, build automatically, test extensively, allow for extension, use examples. The biggest breakthrough, however, comes with modern environments such as Jupyter Notebooks, which integrate code, description, examples, and results in one single document. In this talk, I present the basic rules for sustainable software and show using our textbook "Generating Software Tests" (www.fuzzingbook.org), how software, documentation, textbook and slides fit in a single comprehensive package.
 Okay, good morning everyone. I hope you can hear me well. It's a great honor to be among you I'm very happy to see this community of research software engineers forming such that those of you always been We've all been working for other people in some shady rooms all isolated now can find nothing Can I can finally come together and get the appreciation that you deserve? so I'm here as your as your as one of your keynote speakers today and as a keynote speaker and Also, actually as being one of the eldest one of the oldest guys in the room Okay, okay, we Okay, we're going we're going to debate that As you as a keynote speaker your typical situation is to come in very much like very much like this person here on this picture And to bring over some insides to you Which you may otherwise which you may otherwise not have so my role here is the Super duper big software engineering professor who's now meant to come in and tell you how to do things By the way, this is you on the other side. No worries. We'll have a gender neutral version of that slide later on So what am I what am I what am I actually what am I actually doing here? Well, sorry simple I'm passing on knowledge to you and this knowledge well traditionally comes in the form say of books for instance Here's such a book which I'm now handing over to you is actually not my book But this would be say a traditional book on software engineering When people ask me dear professor Zeller, we have these and these problems. What is it that we should do? Well, the first thing I told them well get a book on software engineering This is actually an 800 page book and you can read the most important things in about two hours or so Just by skimming through it's a software engineering is typically a very fast read It takes more time to implement this stuff, but getting the most important things done is actually not very difficult So here I am I'm done. Thank you very much read the book And okay good. That's not what I want to do good and thing is a book slow and software engineering typically Typically tell you well what software engineering is meant for that is to build some Super duper huge software system say you want to build whatever a travel reservation system for the world or a worldwide Internet worldwide information framework or whatsoever. Yes, and if you want to build this, it's good to have a professional software engineer on board plus reading a book But when we're working on when we're talking about research software, we typically have much more constrained budgets in time in money and resources and Therefore the super duper big software engineering Processes are typically not the best ones But for these I would rather recommend to take a look not at software engineering books But rather look but rather take a look into open source software developed by volunteers in their spare time with little money little resources and yet having plenty of impact That is question is how can you build software that works with a minimum of effort and a maximum of impact? Well, this is exactly what we want to do in our research projects just as well So and we can learn quite a number of things from open source software I'll point out the five most important the five most important Features to you the five things which I consider most important and these are the ones that are sort of built in this Build a necessary foundation for creating open source for creating successful open source projects The first one in here Is to have a repository? That's not well The question of the question here is when I'm telling you this half of you are going to know Yeah, we know that thank you very much and the other half and the other half of you will oh wow That's super cool. This is the typical reaction I get so I'm going to spend a few minutes on this just to but I'm going to gloss over it Such that those of you who don't know it. Yes, we can look this up The other ones will find out but this is a recent but this is a necessity You have a repository have version control you have a place where it can we put up all your software and where it will be available for others Good thing is to make this Accessible to the public in the first place such that it's easier for others to join your project make this available make this open source Adhere to standard conventions you go to one of the standard services such as github and you get everything in one place You get a version repository you can file bugs you can have more collaborators and it's all set up for you It's very easy to use don't typically there's very little reason to go for something that you have homegrown Second thing is Everybody should be able and everybody I mean Everybody in the world should be able to download and build your system from scratch That is you should be able to press a button and then everything will everything will build automatically This for a very simple reason the more people you have to join your project the more people will eventually have to build your project and If every time you have to resort to a number of phone calls or may or email exchanges and everything This is going to be time-consuming so automate as much as you can. This is necessary for growth The third thing is the third thing is have tests Yeah, I already see some people already see some people frowning here. This is this This is also a necessity because people not only want to build your system. They also want to rely on it and even more important When your system grows and when you have more and more users when it's actually in use people will come and will want to We want to change things when you change things You want to make sure that after a change everything works everything everything still works as intended So you should have a big set of automatic tests that you can pretty again that you can that you can run Typically, this will be unit tests or set of validation tests that you can run at the touch of a button Or even if you're if you're so inclined you can even make them run automatically all day long with continuous integration That's the keyword here But the general idea is you should have tests that you can run that automatically check things because then it's much easier for you to Have others contribute to your project such that you can validate whether their changes and fixes don't break anything new Just for the record and software engineering project. So those that we analyze one out of three fixes introduces another bug So this is why such tests are a necessity And I'm not saying one of three changes. I'm saying one of three fixes, right? So just big just people want to fix one bug and then they introduce another bug and then they freeze it another bug Over time there's a damping effect that's that eventually the number of bugs goes down But still you need to make sure that you want to make sure that you don't get accidental new bugs Number four on my list is to structure your software such as a software software Is to structure your software such that is open for extension if your software is successful It's going to grow and if your software grows over time Then you can run into trouble. This is also something that software engineers have realized long ago Here's a simple example. This is a piece of software which has been Expanded again and again So we have a connection type here and initially it was modem 56k and now people have added ethernet and then people have added Wireless and then people have added UMTS and next we're going to have 5g in there And every time there's a new piece of code being added and over time this piece of software becomes entirely becomes huge and Unmaintainable if you have some fun go look at the window source code Which is exactly written this way Where you have say Where you have say some standard function. I want to allocate memory This is the standard way and then come all that then come all sorts of special cases which replicate bugs for Which replicate bugs that have been there in previous version of windows? So there's actually checks in there would say if this is sim city To K in the version for windows 95 then replicate this very specific bug That also was there in windows 95 such that Sim city 2k still runs on your whatever windows on your current windows 10 system This is what happens if you insert a CD of Wind to K. Yes, there's this code been executed and needless to say this is a huge maintenance problem for windows That is on the other hand They may be still a few dozen people in the world who still play sim city 2000 and they're very happy that it still runs Okay, really it's fun looking at this looking at the source code you find all this stuff all this stuff. It's just horrible Everywhere else say if this will index and open source system people would simply say well Just just play an open source game and fix it. No, no, no chance. This is windows. It is binaries binaries are never going to change better option here is to design your system in A way that is actually extensible So object oriented programming tells you here that it's a good idea to deconnect the application from the actual implementation And then you have an abstract class that says okay This is my interface for connections and here I can have an I can have a modem connection Wi-Fi connection ethernet connection every time somebody comes up with some new kind of connection Simply create a new superclass and everything that is in my app over here Can stay the same because it uses the same interfaces and Last but not least this is frequently overlooked It doesn't suffice simply to put a big blob of software on your son Make them to make a big blob of software available for others. You have to tell people how to actually use it So it's a good thing to invest into it's a very good thing to invest into tutorials Not simply documentation documentation is one thing but the other thing is simply having you read me Where do I start? How do I get this to run? How do I get this to use? This is something that open source developers over or often overlook But this is the number one thing that actually gets people to use your system Please provide some tutorials some help some examples Whatever you can do in order to make it easier for others to adopt your system so all of these are pretty all of these are simple simple rules that are are Not necessarily easy to follow the design thing for instance Yeah, you can spend you can certainly spend weeks and weeks on that discussing these but overall these are all these are all things That an open source software have proved to be have proven to be Successful and that have proven to have lots of impact with relatively little effort so so So this of course is part of The knowledge that we can pass along as software engineers. Oh and by the way If you find that there's too many males on that picture here's a here's another variant. This is from Ruth Schreiber That is a bit more that is a bit more distributed Actually, it would be a good thing to have a female keynote speaker next year such that we can pass on that slide and the But one thing I also found out is that when People talk to us We as software engineers, how is that when we build our own systems? How is life going to be because we know right we know everything is it so this is going for so people sometimes Assume that when we build software and this is some kind of paradise where we have unicorns and they fart rainbows and whatever And of course every single piece of every single line of code I'm right. I'm writing works from the very beginning actually I'm writing 1000 lines in a row without ever removing the cursor anyway, and then I press a button and oops automatically everything unfolds Yeah, sure well, I Would be perfectly happy to keep the reality to keep that illusion as it is But the reality is actually a bit different the reality is more like this So this is me and this is me making great talks about however How everything is just fine and there's surroundings here. This is say my my institute my software my computer whatsoever and The reason for that is The reason for that is actually pretty simple. Let's get back to this picture one more time But now let's focus not so much on the interaction between me and customers or whoever But red slather focus on This part of the picture. This is actually the people producing this knowledge and this is the people producing that software Actually, actually I'd like to imagine myself me being the beardy guy in the middle and all of these all of these are obviously my PhD students and others Well in this case obviously supporting me without their support look at this guy out out here must be a system administrator with You don't even see the face. This is how unimportant is see Michelangelo Pa, okay? With without these guys obviously I would just fall down from the sky and then who knows what happens So actually actually it's all these you know, they're struggling is horrible without all these guys and nothing would ever happen guys and girls and the thing is however that This that it's actually there. It's actually there in the interaction of many people producing software Where the trouble begins and we are not exempt from this trouble at all. No, I'm sorry not at all because well Let's face it So we are producing software as we are producing the very same software that we eventually pass on over to you So that's nice. We building this stuff. We're writing this printing it out, too But the standard way we are doing this Actually works as follows. So Some of us has a great idea. Hey, we could build a whatever we could build a perfect test system We could come up with full-fledged verification whatsoever. Wow, whatever good thing. Wow. Oh, yes And the idea is super typically very simple if only people would do it right then we build it and since we are and Since all of us are real matches. We want to build this not for some toy language. We built this for see or for Assembler machine language whatsoever process a microcode because just it's so hard and then we find out it doesn't really work Then we refine the whole thing and sometimes it works and then we have to come up with more ideas And then we reject the original ideas and we keep on working on this stuff and Then it's if then this code evolves evolves I eventually gets into a direction that was entirely different from the original idea But yet the original data structures came from the original idea. So we have to revise these as well in other words eventually what we get is a big pile of code which sometimes works sometimes is dead Sometimes was in which in places was meant to work generally which in other places has a couple of ugly very Copy a couple of very ugly hacks But in the end we managed to get all of all of this pile to run on A set of subjects and we get a paper out of it We're happy and there we are and then we can put this and then in principle we could put this big pile of code on a server and Well, let the rest of the world deal with it as they can Of course, there is no documentation or little documentation. There's no tutorials. Well, there sometimes are tests Yeah, there's some of our software actually looks like this. So forgot everything So so all of these five rules which I told you beforehand. This is something I tell to my PhD students to and sometimes they listen to it The interesting thing is once we have now we are in this situation and Then say a student comes along say whatever and you some the student who wants to write the thesis a customer whatsoever and asks, how can I how can you use that and Then we have the then the student gets a look into the code and Well, if he's not faint of heart some of the students just run away when they see our code But then typically they think it's their fault not ours So they think okay, I'm just going to dig deep into that and they and and and they and they talk to the PhD students and the PhD Students guide them into it into the code that they're written this takes a week or so and then the student learns more and more and more and Say suppose this is a bachelor's thesis, right? So this is three months So a student spends about a one month just figuring out what all of this is about Spends another month figuring out how to use it and then all of a sudden Oh, there's only one month left for running all the experiments and for writing a thesis about it So typically the student has one week to do the experiments Which then are incomplete and then finally one week to do the final write-up Which is very which is not very helpful because the final write-up is just as incomplete as the documentation was in the first place So we're running into a big timing problem because we're spending lots and lots of human time Just figuring out how things just just just spending just spending time with Fox and guiding them into individual parts And this is has been a problem for bar for as long as I can remember So since I'm in academia, this has been this has been pretty much the case Well, there's a couple of pieces of software which we have built the official way, but well, it's research So we're building stuff. We don't know where it's going to end if only we knew in the very beginning life would be much easier So and by the way, I have to point out when the project runs out and the PhD students Who have created all this leave? Well, then the code is effectively dead The paper persists paper lives forever. That's great. The idea still lives on it's in the paper, but for everything else try to Write mean email dear professor Zella. We've read your paper out of say of five years ago. We'd like to work on the code Sure, I have the code somewhere. Oh, it requires eclipse 2.1. Do you have that somewhere? No, we don't maintain eclipse. Can you adapt this to a new version of eclipse? No, no, no We can't do that. No, no chance. Oh, too bad. Okay, good We are going to implement it ourselves and then other people do this go through the very same cycle until their PhD student goes away So what are we doing now to to get over this problem? Which actually is my problem just as well as the problem of every software engineer with research software one thing is We are now very much embracing prototyping What does prototyping mean? Well, it's actually the same cycle as before So we have an idea except that in this step We are building we are building something that is not necessarily meant to be the real thing But we do something in the simplest and fastest way as we can where fast means We are fast not the computer is fast we don't care about the computer speed at this point because CPU cycles are cheap our cycles are expensive So and then we keep on refining this prototype and with for this way is prototype languages We are scripting languages these days So we don't implement things in C or Java and the first thing is Python instead or It's Python instead or Ruby the simplest thing that can that we can use to validate our idea is just fine And then we again write a paper about it Turns out to that turns out that in our case We've always thought the community is not going to accept anything that doesn't work on the real language like see it turns out This was a total mature attitude all of Netflix runs on Python Google has Python all the way down Facebook has millions of code lines of Python. So paper Python it is and ups turns out that by building such prototypes We are about five to ten times faster than we used to be and The neat thing when a student comes along turns out these prototypes are much easier they're much easier to understand because they don't contain all Sorts of optimizations. It's much easier to say analyze Python programs for instance So students can interact with these with this parchment with these parts much faster And also they can quicker they are much faster in writing papers and fun part is Since these prototypes serve as blueprints for how to do things because they focus on the algorithm on the algorithms other than Implementation details we have students write our implementations trying out what this thing that we already worked out how it works in a prototype After we have after we have understood how the algorithm should be how the approach should be then we let students Port all this into other languages like C or other operating systems whatsoever Just to find out what the performance is this performance thing can then go into the paper and but if this implementation dies Down here if this dies when the student goes along that's not so much of a problem because the prototype and the idea still persists so This is a one rule we have learned and which we have which we have adopted And this is very much boosted our research The second thing we're doing a lot these days is literate programming Literate programming was invented by Donald Knuth If you don't know Donald Knuth, he's one of the biggest Most influential people in computer science and in science overall Donald Knuth has written one of the most influential books on Programming art of computer programming and in order to wrote in order to write these books He invented his own type setting system. This is tech and latex as many of you may know this is the language with the backslashes And so he has a lot has a lot of influence in computer in computer type setting too And it's incredible When I sometimes come to a conference, there's a couple of there's a couple of students typically from China want to take pictures with me That's all cute Recently I met Donald Knuth and I absolutely wanted to have a picture taken with me. So that was my situation Didn't wash my hand afterwards. Well, that was a year ago since then Finally washed it. This is my relationship to don Knuth. So so so I may so so He's if I'm a god, he's a super god and well actually to him I'm somewhere very down there in the mud and One thing you know one thing invented Uh is called literate programming and the idea here is that you do have one single document Which contains both code as well as the introduction to it This is how uh, this is how you write a literate literate program. You have pieces of code That explain individual parts of the program and then you have documentation for it And there is actual very much as in latex There's actual characters that change the meaning of things and from the single document from one single document And that contains both That contains both an introduction to the code as well as the code itself You can derive well the code you can extract the code and this is something you directly feed into a compiler You don't even look into it The second thing however is you can pipe this into a type setting system. Well, it's tech again and Oh, sorry wrong button And then this automatically gives you a very nicely printed version of your code Which then comes with all of the Introduction so it tells you here. This is how we do it. We since we give a couple of introductions We don't do this we don't do that and you can actually read through this entire book that is that is being produced there From one part to the next Which by the way gives you a very nice introduction on how the code works. So you understand So you understand how it's going how it's going to be used Which then again can be very useful for others who want to work with this very code so this is something That we also have adopted these days in order to create in order to create code That is well documented and that can be and that also Um, that also is all self contained in one single document. That is you don't have separate documents for whatever. Here's the tutorial Here's the code. Here's another help text. Here's the here's the here's the comments in the code But we wanted to have one single representation Now we have these two ideas. We'd had prototyping and literate programming Now we bring in now we went and brought these together coming up with a new term which we call literate prototyping This is actually the first time ever i'm using this i'm using this word combination. I made this up just yesterday in order to illustrate what we're doing Um, literate prototyping Sounds like a super fancy idea, right? And in particular you don't want to do the opposite But you may you may want to do not prototyping, but you definitely not want to be illiterate here. So This is why this is the nice thing about this word literate And the question is how do we do this? Well, we actually we actually use plenty of jupyter notebooks these days Which actually bring these features together which allow us to If you haven't worked with jupyter notebooks yet now is a good chance These are tools that allow you to bring together documentation that is text elements as well as code elements And you have them right in your browser. You can execute the code right away You can toy with the code and while you and all the results of any experiments can be included too Having examples having visualizations. This is actually far superior to what we can do what we can do in literate programming As invented by Knuth. Well, this was because literate programming was invented at the time When we didn't have these large interactive systems And just to give you an idea How we do this I'm going to give you a very short demo here So let's see whether Things still work as they should So we're switching over How cool is that wonderful so you don't see a thing great Connection failed. It was to be expected Okay, good So This is a bit small, right? So let's try to Get this a bit bigger So this is a this is a book Well, this is a book on automated testing which I have just completed A couple of weeks ago. This is a book on Generating software tests which allows you to Which allows you to explore quite a number actually to Eight chapters of various ways of testing your program automatically And this is one of the chapters in here. So you see we have a nice. We have a nice Book in the browser which comes with all sorts of extra features quotes and whatnot And here we do have actually here. We have actual code in here Which you can read and this is the same as the output of a little of a little programming process Where you have code in here a way of code and way to do it In here a way of code and we do have actual text that describes the code And we also have examples of how this code is being used This is a fuzzer for instance a fuzzer produces a string Of random arguments which you can then pipe into a program of your choice in order to get it to break And in the next chapters we're going to find out how to come up with all sorts of different fuzzers The neat thing is that this is actually the output of a jupiter notebook And this jupiter notebook is something that let me see you can actually go Resources edit as notebook if I press on this then you will automatically get into this notebook This is going to be started In the background and so depending on the current network speed we'll have this in a minute or so In the meanwhile, this is this is what this looks like when you have it in your browser And this is what we give out to students So you have cells here with code which you can directly edit as you like So you can add new code to it you can change the code and you can also see And you can also find out how any of the changes you make Is finally going to is finally going to affect the outcome So students can directly work with this and when I say students This involves students. I'm having in my class or regular Or the bachelor's and master's students, but this also involves any PhD students who want to work with this code This also includes any researcher out there in the world who wants to extend this code Because people can directly see okay. Here's the introduction. This is how it works. Here's the code This is how this is this is how you can use it and here's examples on how to how to and how to put this to use jupiter notebook is very versatile here you can You can go and also create you can create various formats right out of the notebook So you can also create slides out of this. So this is the This is the these are the slides that we get that we get Right out of the book. So if I wanted to give a presentation on the material of the book I could use this slide presentation and then show okay. This is how this works There's also interactive versions of these slides. So you can actually go and edit right into them And finally for those of you who are who want to do this in a more standard way You can also create a standard PDF out of it and then you're going to get a document in this case for the chapter 36 34 pages which then again contains all the material now in a way that can be printed or passed on to or passed on to a publisher and All of this is available without any extra effort is available right out of your computer right out of the browser And it creates an infrastructure Which is helpful for us for teaching For research and for and actually for dissemination of our results so Let me just go back here here we go This has number of this has a number of benefits So again, so For one thing We again have our prototyping we're going to have a prototyping cycle here But now we call these prototypes chapters and we actually have our students again This is bachelor students master students as well as phd students actually creating new notebooks Which eventually end up in the contribution section of the book So if a student wants to write say a test system For mp3 files, we just had that so we automatically generate mp3 files And then we try to fuzz try to test mp3 players with it This becomes a chapter and this eventually and this eventually becomes this chapter gets included in the book others can work on this And if we find a bug this also ends up in the paper And well again The student who previously has spent months Figuring out how this big implementation works Now spends a day or two Figuring out how the prototype works because there's a chapter which documents everything And then the student has far more time to actually work on the actual subject of his to actually work on the subject of his thesis Thing is For me it takes about Two days to implement such a chapter. Well, if it's one of the easier ones, that's the code It takes one extra day to document it all such that I write down what I thought And this extra day is saved again and again and again for every new student for every user who comes along Because they don't have to ask me questions how everything works This is the this is the big benefit of Literate programming or generally of good documentation So what do we have here? We have one single source for codes books slides everything we can develop new ideas very very quickly Including their evaluation Jupiter notebooks great for data science. You want to do any kind of evaluation you get graphs you get interactive You get interactive elements. You also have presentations on that subject here You get you can hand out this material to students only the stuff that works actually goes into production Because many of the ideas we have do not work So it's good if you have a prototype that fails rather than the actual thing And the chapters the prototypes Will have a much longer re ability to have a much longer life than previously our implementations We can have bachelor students master students work on these notebooks when I'm teaching this I'm actually giving out this material to the students first So we're having an inverted classroom principle and then um and then I typically do live coding So I leave out a couple of code pieces and I'm implementing them Which is always fun for the students to see me make the same mistakes They do and then to try and to debug code. Fortunately, this happens not too often Um, then I'm explaining how things work. So I'm thinking loud all of this works Well, and when students come along and say, okay, professor's only what happens? Why is this parameter like this? What happens if you change that couldn't it be that sure we can do this live right right away? It's like um It's like pair programming except that you have 40 people in your room who all come up with ideas Students like that a lot and students then become later recruits for later thesis with me So last year we had 40 students in the lecture The 10 students ended up in an advanced seminar where they're actually writing chapters and out of these 10 students I'm expecting two to three PhD students coming out Which is quite an outcome. We are in a very competitive environment in subrican where there's plenty of people hunting for good students And yes, this is a great way to attract people also These notebooks are lots of fun to work with interactive reproducible extensible They can be reused and build upon if they are well designed that is I'm come to that in a second And for me the question is Why do we why do we why at our conferences do we require whatever two page pdf formatted as if it were as if it were 1970 Why do we insist on printed books people come to me? Andreas you have finished your book. When can I finally buy this and put this into up to my bookshelf? And I tell you know a printed book is not the exact best way of interacting with all this But well eventually if people insist we're going to put a big 1000 page pdf somewhere in the server and then they can print it out um However, I'd also like to point out that If you work with notebooks, this does not necessarily equate equate quality. Simply you do that um, we have uh done We have started to explore so since since we have so much fun with notebook We wanted to know what what the general status of notebooks is and so we took Nearly 2000 open source notebooks from the web and did a couple of checks on them. Yeah, we could because we can because we're software engineers um, there's a There's a tool that checks python code for its adherence to A python conventions whether it's essentially whether your code is written according to python conventions and Well, it turned out that one in three lines violates a python convention in notebooks One in three most of this is the lines being too long or not having the exact spacing or anything But generally this means that people will not have fun reading that code and no we don't uh, there's actually far larger than for regular python code Where so it's about three times as many three times as many troublesome spots This is not just stylistic stuff 40 percent of all notebooks we found had unused variables. That is there's something there's something being computed and then stored But it's actually never being used Which in regular code actually flags as a flags as a flags as a potential mistake because this typically happens when you have some When you have some if you have an unused variable This simply means well you have computed something this should be used and the reason that it is not used may point to an error later on in the code We have just started scratching the surface here, but We actually actually have a good feeling that there are plenty of mistakes in published notebooks That call for that call for being discovered Our our biggest hope is finally to find one error in one notebook that has that then Eventually leads to the retraction of some nature or science article. That would be real cool if we could do that But of course we adhere to the principles of responsible disclosure here very much like vulnerability So we would tell the authors first and then they can fix things That would be real cool. Well, we're just have started this a couple of weeks ago This is first findings. You've seen them. You see they're here first Also, we have deprecated functions that is notebooks are not going to be We will not we will no longer work in a couple in in a couple of months. So There's still lots and lots of work to be done And this is consequence for myself and my community We need much better tools and we need better languages Better language integration to enable such prototyping The program analysis community so far has very much Uh ignored languages like python because there are toy languages because they are too simple whatsoever But we need much much better tools for dealing with these we need tools for analyzing our programs for analyzing python programs for helping developers in those languages creating better creating better programs better notebooks and This again calls for well, this is then a call for my community to create such tools And to hand hand them over to a community like yours such that you can use them and in the long run What we want to achieve here is a community where we have a large number of people Creating pieces of code that interact with each other that can build on each other that can that can work together to create science both in a both speaking from a community perspective as well as from a technical perspective A community that is really large that is strives where all of research engineers software engineers come together and work together to build great scientific work With this i'd like to close. So what have you seen a bit of five rules for sustainable research software bit of experiences With prototyping creating prototypes that eventually after having been refined end up in an implementation a couple of insights on how to Bring together literate prototyping in notebooks and other new tools And finally also hint to the simple fact that simply using notebooks is not the solution to everything but we're going to need better tools and better languages to also Come up with better production in notebooks With this i'd like to close. Thank you for your attention and i'm happy to take questions Thank you very much indeed. I think we have time for two or three questions. So I can see one two three And that's that yes, please Three decades of open source software development has generated its own social model Do you want to make some comments about the social context of some of this? These ideas that you presented here The social context for us is we are very much influenced by the open source model because also as researchers for us It's the most economic way to have a maximum of impact. This also this also extends to our concept of notebooks We can pass them along others can work on this and what always has made open source very successful Is the fact that people that other people actually read the source code other people extend the source code Because this makes it easier for others to spot errors It also is it also is helpful that the code is available for other sources It also is helpful that the code is available for others to run the tools on them so if we want to try out new bug finding tools for instance, we run them on open source and By and by bringing together these best open source practices together with literature programming That is we introduce how things work. We allow others to extend all of this I think we are We are pushing forward the best practices of open source and bringing a couple of new best practices in there Hello here. Thanks a lot for this talk. It was I think the first one I heard to bring solid principles into research software engineering. So thank you One question about the notebook. So how do you solve this problem that as you interact with the notebook? A lot of yeah stuff accumulates in the jason and they are difficult to version control in git for example Ah That's the good thing. That's the good thing of being a being a season programmer First thing I did is I built a couple of my little bit of tools myself I can do that. So There are so having said that there are established there are there are nice Packages around that allow you to do very nice version control together with together with notebooks Um, and we have adopted them, but it turns out that for quite a number of things we need to we need to write our own tools so simple simple things such as Violating python conventions for instance I have built my own. I have built my own plugin for that which I can simply interact in jupiter with I can press a button and then it tells me these things So you're going to find in my code. You're going to find less of these violations because of this every reason But I think that um, but I think that we as a community That's partially also research software engineers. Well, generally software engineers We need to invest more in such tools that help everybody whether this is in whatever eclipse intelligente But also jupiter notebooks To create to to come up with better code and to spot errors more easily and we have and I think I think this can be a great investment There is a million of jupiter notebooks open source out there around There are at this point. I don't know how many tens of thousands data scientists working with jupiter notebooks every day coming up with all sorts of results all day long and well Somebody at some point we'll have to quality check what's actually going on in data science We already have the science of a big reproducible reproducibility crisis Uh, at least people push out their notebooks. That's a good thing. That's the start But I think that there is much much more to be done and that we can do in terms of quality control. I'm also happy to hear Matias Kattabur from DFG to join a new initiative Soon on that focuses on quality of research software. So there's plenty of things to be done Okay, I think we have time for one last question uh, sorry, um Okay, um, thank you very much for the talk. Um, I was wondering That um, so you said we can learn from open source software for research software I would even argue that we could learn from this for for just general uh analysis for example or like um papers in general you talked about the reproducibility crisis. This is usually Rather Simple analysis scripts that people write. Um, but that are often not reproducible and um I would even argue we should go a step further and make it as easy as possible for for the end users like the applied people To make it make it easy to make their work reproducible from from my standpoint from my standpoint The scientific paper as we know it these days should actually be should actually be obsolete at least for the fields in which in which data analysis data science plays any role And I think we should go towards towards mechanisms that allow you to that allow you to recreate the entire process from the raw source data to the finally published chart with all the intermediate steps in between with all the algorithms you used This is how science should be but I am But I have to say this. I'm a really good researcher I am but I have to say this. I'm a reviewer for papers and And typically well we have this big source of data Then we apply the super magical algorithm this goes on the following five pages Which uses the following seven seven parameters Which we have chosen to be minus seven dot four minus 55 and six six point four and the sign of square root of two whatever And this works see see see here's the charts. It works. Thank you very much. No our data is not available. No our algorithm is not available No, there is no that justification for any of these choices. There's probably there's probably two There's probably some more 30 40 design choices somewhere in there, too This is not the way it should be If we can make it if you can make this an open science If you can make this an open science process Where is this here's the raw data? This is what we found out. This is great Where people can actually replicate this where people can reproduce this where people can talk about this This also involves making these incisible, which is another problem But that's another thing if you build something that's used all over the place This is something that will end up in your CV and then you can then you can see then later you can say yes I built this particular piece of code and everybody and Everybody knows this thing and these are all this is our culture that we have to introduce culture that we have to change But and it will be us software engineers in research to help with that Thank you. Uh due to circumstances we do have time for another question. So I think there was one over there. Yeah Andreas, I think um, it's it's good. I I I completely support your arguments and underwriters um, uh, but I believe many here are also in a situation that The question of scalability comes in and there's a point where you have might have to leave this way of working in order to come up with this multimillion dollar information system and um, how To find I mean that you have to change your environment and set up in another professional software engineering environment to come up with a So a decent way of working in that area I think that's that's for sure. Everybody can do this and can read it as for example reading the book of our great colleague but What are the signs? When to switch yes From my perspective. So my perspective is I have to say this my perspective is to a great extent the perspective of a researcher Where software is the subject of research and where subject is the instrument of research So and where it is not necessarily clear from the very beginning how things should be built and what things we Would think how things should be done and this is we are prototyping simply we're prototyping simply absolutely excels Because this allows you to try out many alternatives before actually going for the real thing if you are if however Uh as also as also is often the case in projects if however you have to do something if you have if you have to do if you have to Build something that hasn't been built Thousand times already and where the path is pretty clear how to do this you have to build a database system for storing research data for Replicating research data. You have to build a website a web page whatsoever This is of course material where such a prototyping approach Uh very prototyping approach such as I sketched in here may also make sense say for web pages For instance, you want to do user testing you want to come up with another design and another design This is how it works if you want to if you choose a database you can do experiments with different databases before settling on one particular thing but here but here Um It's but but here it's uh, but here it's much more important to follow best practices if they exist That is this is how I choose a database. This is how I build things and I'm not going to event things on my own I'm simply going to do this as a few others as a few hundred others have done before me So for those domains where it's well understood how to build things go and familiarize yourself with the best practices in the field If you're out there in the unknown, however Then it's a good idea. Then it's definitely a good idea to try out number of alternatives fast quickly Prototypes stick with simple and simple and simple solutions Before you finally settle on one and then dive into the unknown
 Okay, good morning everyone. I hope you can hear me well. It's a great honor to be among you  I'm very happy to see this community of research software engineers forming such that those of you always been  We've all been working for other people in some shady rooms all isolated now can find nothing  Can I can finally come together and get the appreciation that you deserve?  so I'm here as your as your as one of your keynote speakers today and  as a keynote speaker and  Also, actually as being one of the eldest one of the oldest guys in the room  Okay, okay, we  Okay, we're going we're going to debate that  As you as a keynote speaker your typical situation is to come in very much like very much like this person here on this picture  And to bring over some insides to you  Which you may otherwise which you may otherwise not have so my role here is the  Super duper big software engineering professor who's now meant to come in and tell you how to do things  By the way, this is you on the other side. No worries. We'll have a gender neutral version of that slide later on  So what am I what am I what am I actually what am I actually doing here? Well, sorry simple  I'm passing on knowledge to you and this knowledge well traditionally comes in the form say of books for instance  Here's such a book which I'm now handing over to you is actually not my book  But this would be say a traditional book on software engineering  When people ask me dear professor Zeller, we have these and these problems. What is it that we should do?  Well, the first thing I told them well get a book on software engineering  This is actually an 800 page book and you can read the most important things in about two hours or so  Just by skimming through it's a software engineering is typically a very fast read  It takes more time to implement this stuff, but getting the most important things done is actually not very difficult  So here I am I'm done. Thank you very much read the book  And okay good. That's not what I want to do good and thing is a book slow and software engineering typically  Typically tell you well what software engineering is meant for that is to build some  Super duper huge software system say you want to build whatever a travel reservation system for the world or a worldwide  Internet worldwide information framework or whatsoever. Yes, and if you want to build this, it's good to have a professional software engineer on board  plus reading a book  But when we're working on when we're talking about research software, we typically have much more constrained  budgets in time in money and resources and  Therefore the super duper big software engineering  Processes are typically not the best ones  But for these I would rather recommend to take a look not at software engineering books  But rather look but rather take a look into open source software  developed by volunteers in their spare time with little money little resources and yet having plenty of impact  That is question is how can you build software that works with a minimum of effort and a maximum of impact?  Well, this is exactly what we want to do in our research projects just as well  So and we can learn quite a number of things from open source software  I'll point out the five most important the five most important  Features to you the five things which I consider most important and these are the ones that are sort of built in this  Build a necessary foundation for creating open source for creating successful open source projects  The first one in here  Is to have a repository?  That's not well  The question of the question here is when I'm telling you this half of you are going to know  Yeah, we know that thank you very much and the other half and the other half of you will oh wow  That's super cool. This is the typical reaction I get so I'm going to spend a few minutes on this just to but I'm going to gloss over it  Such that those of you who don't know it. Yes, we can look this up  The other ones will find out but this is a recent but this is a necessity  You have a repository have version control you have a place where it can we put up all your software and where it will be available for others  Good thing is to make this  Accessible to the public in the first place such that it's easier for others to join your project make this available make this open source  Adhere to standard conventions you go to one of the standard services such as github and you get everything in one place  You get a version repository you can file bugs you can have more collaborators and it's all set up for you  It's very easy to use don't typically there's very little reason to go for  something that you have homegrown  Second thing is  Everybody should be able and everybody I mean  Everybody in the world should be able to download and build your system from scratch  That is you should be able to press a button and then everything will everything will build automatically  This for a very simple reason the more people you have to join your project the more people will eventually have to build your project and  If every time you have to resort to a number of phone calls or may or email exchanges and everything  This is going to be time-consuming so automate as much as you can. This is necessary for growth  The third thing is the third thing is have tests  Yeah, I already see some people already see some people frowning here. This is this  This is also a necessity because people not only want to build your system. They also want to rely on it and even more important  When your system grows and when you have more and more users when it's actually in use people will come and will want to  We want to change things when you change things  You want to make sure that after a change everything works everything everything still works as intended  So you should have a big set of automatic tests that you can pretty again that you can that you can run  Typically, this will be unit tests or set of validation tests that you can run at the touch of a button  Or even if you're if you're so inclined you can even make them run automatically all day long with continuous integration  That's the keyword here  But the general idea is you should have tests that you can run that automatically check things because then it's much easier for you to  Have others contribute to your project such that you can validate whether their changes and fixes don't break anything new  Just for the record and software engineering project. So those that we analyze one out of three fixes introduces another bug  So this is why such tests are a necessity  And I'm not saying one of three changes. I'm saying one of three fixes, right?  So just big just people want to fix one bug and then they introduce another bug and then they freeze it another bug  Over time there's a damping effect that's that eventually the number of bugs goes down  But still you need to make sure that you want to make sure that you don't get accidental new bugs  Number four on my list is to structure your software such as a software software  Is to structure your software such that is open for extension if your software is successful  It's going to grow and if your software grows over time  Then you can run into trouble. This is also something that software engineers have realized long ago  Here's a simple example. This is a piece of software which has been  Expanded again and again  So we have a connection type here and initially it was modem 56k and now people have added ethernet and then people have added  Wireless and then people have added UMTS and next we're going to have 5g in there  And every time there's a new piece of code being added and over time this piece of software becomes entirely becomes huge and  Unmaintainable if you have some fun go look at the window source code  Which is exactly written this way  Where you have say  Where you have say some standard function. I want to allocate memory  This is the standard way and then come all that then come all sorts of special cases which replicate bugs for  Which replicate bugs that have been there in previous version of windows?  So there's actually checks in there would say if this is sim city  To K in the version for windows 95 then replicate this very specific bug  That also was there in windows 95 such that  Sim city 2k still runs on your whatever windows on your current windows 10 system  This is what happens if you insert a CD of  Wind to K. Yes, there's this code been executed and needless to say this is a huge maintenance problem for windows  That is on the other hand  They may be still a few dozen people in the world who still play sim city 2000 and they're very happy that it still runs  Okay, really it's fun looking at this looking at the source code you find all this stuff all this stuff. It's just horrible  Everywhere else say if this will index and open source system people would simply say well  Just just play an open source game and fix it. No, no, no chance. This is windows. It is binaries binaries are never going to change  better option here is to  design your system in  A way that is actually extensible  So object oriented programming tells you here that it's a good idea to deconnect the application from the actual implementation  And then you have an abstract class that says okay  This is my interface for connections and here I can have an I can have a modem connection  Wi-Fi connection ethernet connection every time somebody comes up with some new kind of connection  Simply create a new superclass and everything that is in my app over here  Can stay the same because it uses the same interfaces  and  Last but not least this is  frequently overlooked  It doesn't suffice simply to put a big blob of software on your son  Make them to make a big blob of software available for others. You have to tell people how to actually use it  So it's a good thing to invest into it's a very good thing to invest into tutorials  Not simply documentation documentation is one thing but the other thing is simply having you read me  Where do I start? How do I get this to run? How do I get this to use? This is something that open source developers over or often overlook  But this is the number one thing that actually gets people to use your system  Please provide some tutorials some help some examples  Whatever you can do in order to make it easier for others to adopt your system  so all of these are pretty all of these are  simple simple rules that are  are  Not necessarily easy to follow the design thing for instance  Yeah, you can spend you can certainly spend weeks and weeks on that discussing these but overall these are all these are all things  That an open source software have proved to be have proven to be  Successful and that have proven to have lots of impact with relatively little effort  so  so  So this of course is part of  The knowledge that we can pass along as software engineers. Oh and by the way  If you find that there's too many males on that picture here's a here's another variant. This is from Ruth Schreiber  That is a bit more that is a bit more distributed  Actually, it would be a good thing to have a female keynote speaker next year such that we can pass on that slide and  the  But one thing I also found out is that  when  People talk to us  We as software engineers, how is that when we build our own systems?  How is life going to be because we know right we know everything is it so this is going for so people sometimes  Assume that when we build software and this is some kind of paradise where we have unicorns and they fart rainbows and whatever  And of course every single piece of every single line of code  I'm right. I'm writing works from the very beginning actually  I'm writing 1000 lines in a row without ever removing the cursor anyway, and then I press a button and oops automatically everything unfolds  Yeah, sure  well, I  Would be perfectly happy to keep the reality to keep that illusion as it is  But the reality is actually a bit different the reality is more like this  So this is me and this is me making great talks about however  How everything is just fine and there's surroundings here. This is say my my institute my software my computer whatsoever and  The reason for that is  The reason for that is actually pretty simple. Let's get back to this picture one more time  But now let's focus not so much on the interaction between  me and  customers or whoever  But red slather focus on  This part of the picture. This is actually the people producing this knowledge and this is the people producing that software  Actually, actually I'd like to imagine myself me being the beardy guy in the middle and all of these all of these are obviously my PhD students and others  Well in this case obviously supporting me without their support look at this guy out out here must be a system administrator  with  You don't even see the face. This is how unimportant is see Michelangelo Pa, okay?  With without these guys obviously I would just fall down from the sky and then who knows what happens  So actually actually it's all these you know, they're struggling is horrible  without all these guys and nothing would ever happen guys and girls and  the thing is however that  This that it's actually there. It's actually there in the interaction of many people producing software  Where the trouble begins and we are not exempt from this trouble at all. No, I'm sorry not at all  because well  Let's face it  So we are producing software as we are producing the very same software that we eventually pass on over to you  So that's nice. We building this stuff. We're writing this printing it out, too  But the standard way we are doing this  Actually works as follows. So  Some of us has a great idea. Hey, we could build a whatever we could build a perfect test system  We could come up with full-fledged verification whatsoever. Wow, whatever good thing. Wow. Oh, yes  And the idea is super typically very simple if only people would do it right then we build it and  since we are and  Since all of us are real matches. We want to build this not for some toy language. We built this for see or for  Assembler machine language whatsoever process a microcode because just it's so hard and then we find out it doesn't really work  Then we refine the whole thing and sometimes it works and then we have to come up with more ideas  And then we reject the original ideas and we keep on working on this stuff and  Then it's if then this code evolves evolves  I eventually gets into a direction that was entirely different from the original idea  But yet the original data structures came from the original idea. So we have to revise these as well  in other words eventually what we get is a big pile of code which sometimes works sometimes is dead  Sometimes was in which in places was meant to work generally which in other places has a couple of ugly very  Copy a couple of very ugly hacks  But in the end we managed to get all of all of this pile to run on  A set of subjects and we get a paper out of it  We're happy and there we are and then we can put this and then in principle we could put this big pile of code on a server and  Well, let the rest of the world deal with it as they can  Of course, there is no documentation or little documentation. There's no tutorials. Well, there sometimes are tests  Yeah, there's some of our software actually looks like this. So forgot everything  So so all of these five rules which I told you beforehand. This is something I tell to my PhD students to and sometimes they listen to it  The interesting thing is once we have now we are in this situation and  Then say a student comes along say whatever and you some the student who wants to write the thesis a  customer whatsoever and  asks, how can I how can you use that and  Then we have the then the student  gets a look into the code and  Well, if he's not faint of heart some of the students just run away when they see our code  But then typically they think it's their fault not ours  So they think okay, I'm just going to dig deep into that and they and and and they and they talk to the PhD students and the PhD  Students guide them into it into the code that they're written this takes a week or so and then the student learns more and more and more and  Say suppose this is a bachelor's thesis, right? So this is three months  So a student spends about a one month just figuring out what all of this is about  Spends another month figuring out how to use it and then all of a sudden  Oh, there's only one month left for running all the experiments and for writing a thesis about it  So typically the student has one week to do the experiments  Which then are incomplete and then finally one week to do the final write-up  Which is very which is not very helpful because the final write-up is just as incomplete as the documentation was in the first place  So we're running into a big timing problem because we're spending lots and lots of human time  Just figuring out how things just just just spending just spending time with Fox and guiding them into individual parts  And this is has been a problem for bar for as long as I can remember  So since I'm in academia, this has been this has been pretty much the case  Well, there's a couple of pieces of software which we have built the official way, but well, it's research  So we're building stuff. We don't know where it's going to end if only we knew in the very beginning life would be much easier  So and by the way, I have to point out  when the project runs out and  the PhD students  Who have created all this leave?  Well, then the code is effectively dead  The paper persists paper lives forever. That's great. The idea still lives on it's in the paper, but for everything else try to  Write mean email dear professor Zella. We've read your paper out of say of five years ago. We'd like to work on the code  Sure, I have the code somewhere. Oh, it requires eclipse 2.1. Do you have that somewhere?  No, we don't maintain eclipse. Can you adapt this to a new version of eclipse? No, no, no  We can't do that. No, no chance. Oh, too bad. Okay, good  We are going to implement it ourselves and then other people do this go through the very same cycle until their PhD student goes away  So what are we doing now to to get over this problem?  Which actually is my problem just as well as the problem of every software engineer with research software  one thing is  We are now very much embracing prototyping  What does prototyping mean? Well, it's actually the same cycle as before  So we have an idea except that  in this step  We are building we are building something that is not necessarily meant to be the real thing  But we do something in the simplest and fastest way as we can where fast means  We are fast not the computer is fast we don't care about the computer speed at this point because CPU cycles are cheap our cycles are expensive  So and then we keep on refining this prototype and with for this way is prototype languages  We are scripting languages these days  So we don't implement things in C or Java and the first thing is Python instead or  It's Python instead or Ruby the simplest thing that can that we can use to validate our idea is just fine  And then we again write a paper about it  Turns out to that turns out that in our case  We've always thought the community is not going to accept anything that doesn't work on the real language like see it turns out  This was a total mature attitude all of Netflix runs on Python Google has Python all the way down  Facebook has millions of code lines of Python. So paper Python it is and ups turns out that by building such prototypes  We are about five to ten times faster than we used to be  and  The neat thing when a student comes along turns out these prototypes are much easier they're much easier to understand because they don't contain all  Sorts of optimizations. It's much easier to say analyze Python programs for instance  So students can interact with these with this parchment with these parts much faster  And also they can quicker they are much faster in writing papers  and fun part is  Since these prototypes serve as blueprints for how to do things because they focus on the algorithm on the algorithms other than  Implementation details we have students write our implementations trying out what this thing that we already worked out how it works in a prototype  After we have after we have understood how the algorithm should be how the approach should be then we let students  Port all this into other languages like C or other operating systems whatsoever  Just to find out what the performance is this performance thing can then go into the paper and but if this implementation dies  Down here if this dies when the student goes along that's not so much of a problem because the prototype and the idea still persists  so  This is a one rule we have learned and which we have which we have adopted  And this is very much boosted our research  The second thing we're doing a lot these days is literate programming  Literate programming was invented by Donald Knuth  If you don't know Donald Knuth, he's one of the biggest  Most influential people in computer science and in science overall  Donald Knuth has written one of the most influential books on  Programming art of computer programming and in order to wrote in order to write these books  He invented his own type setting system. This is tech and latex as many of you may know this is the language with the backslashes  And so he has a lot has a lot of influence in computer in computer type setting too  And it's incredible  When I sometimes come to a conference, there's a couple of there's a couple of students typically from China want to take pictures with me  That's all cute  Recently I met Donald Knuth and I absolutely wanted to have a picture taken with me. So that was my situation  Didn't wash my hand afterwards. Well, that was a year ago since then  Finally washed it. This is my relationship to don Knuth. So so so I may so so  He's if I'm a god, he's a super god and well actually to him  I'm somewhere very down there in the mud  and  One thing you know one thing invented  Uh is called literate programming and the idea here is that you do have one single document  Which contains both code as well as the introduction to it  This is how uh, this is how you write a literate literate program. You have pieces of code  That explain individual parts of the program and then you have documentation for it  And there is actual very much as in latex  There's actual characters that change the meaning of things and from the single document from one single document  And that contains both  That contains both an introduction to the code as well as the code itself  You can derive well the code you can extract the code and this is something you directly feed into a compiler  You don't even look into it  The second thing however is you can pipe this into a type setting system. Well, it's tech again and  Oh, sorry wrong button  And then this automatically gives you a very nicely printed version of your code  Which then comes with all of the  Introduction so it tells you here. This is how we do it. We since we give a couple of introductions  We don't do this we don't do that and you can actually read through this entire book that is that is being produced there  From one part to the next  Which by the way gives you a very nice introduction on how the code works. So you understand  So you understand how it's going how it's going to be used  Which then again can be very useful for others who want to work with this very code  so this is something  That we also have adopted these days in order to create in order to create code  That is well documented and that can be and that also  Um, that also is all self contained in one single document. That is you don't have separate documents for whatever. Here's the tutorial  Here's the code. Here's another help text. Here's the here's the here's the comments in the code  But we wanted to have one single representation  Now we have these two ideas. We'd had prototyping  and literate programming  Now we bring in now we went and brought these together coming up with a new term which we call literate prototyping  This is actually the first time ever i'm using this i'm using this word combination. I made this up just yesterday in order to illustrate what we're doing  Um, literate prototyping  Sounds like a super fancy idea, right? And in particular you don't want to do the opposite  But you may you may want to do not prototyping, but you definitely not want to be illiterate here. So  This is why this is the nice thing about this word literate  And the question is how do we do this? Well, we actually we actually use plenty of jupyter notebooks these days  Which actually bring these features together which allow us to  If you haven't worked with jupyter notebooks yet now is a good chance  These are tools that allow you to bring together  documentation that is text elements as well as code elements  And you have them right in your browser. You can execute the code right away  You can toy with the code and while you and all the results of any experiments can be included too  Having examples having visualizations. This is actually far superior to what we can do what we can do in literate programming  As invented by Knuth. Well, this was because literate programming was invented at the time  When we didn't have these large interactive systems  And just to give you an idea  How we do this I'm going to give you a very short demo here  So let's see whether  Things still work as they should  So we're switching over  How cool is that wonderful so you don't see a thing great  Connection failed. It was to be expected  Okay, good  So  This is a bit small, right? So let's try to  Get this a bit bigger  So this is a this is a book  Well, this is a book on automated testing which I have just completed  A couple of weeks ago. This is a book on  Generating software tests which allows you to  Which allows you to explore quite a number actually to  Eight chapters of various ways of testing your program automatically  And this is one of the chapters in here. So you see we have a nice. We have a nice  Book in the browser which comes with all sorts of extra features quotes and whatnot  And here we do have actually here. We have actual code in here  Which you can read and this is the same as the output of a little of a little programming process  Where you have code in here a way of code and way to do it  In here a way of code and we do have actual text that describes the code  And we also have examples of how this code is being used  This is a fuzzer for instance a fuzzer produces a string  Of random arguments which you can then pipe into a program of your choice in order to get it to break  And in the next chapters we're going to find out how to come up with all sorts of different fuzzers  The neat thing is that this is actually the output of a jupiter notebook  And this jupiter notebook is something that let me see you can actually go  Resources edit as notebook if I press on this then you will automatically get into this notebook  This is going to be started  In the background and so depending on the current network speed we'll have this in a minute or so  In the meanwhile, this is this is what this looks like when you have it in your browser  And this is what we give out to students  So you have cells here with code which you can directly edit as you like  So you can add new code to it you can change the code and you can also see  And you can also find out how any of the changes you make  Is finally going to is finally going to affect the outcome  So students can directly work with this and when I say students  This involves students. I'm having in my class or regular  Or the bachelor's and master's students, but this also involves any PhD students who want to work with this code  This also includes any researcher out there in the world who wants to extend this code  Because people can directly see okay. Here's the introduction. This is how it works. Here's the code  This is how this is this is how you can use it and here's examples on how to how to and how to put this to use  jupiter notebook is very versatile here you can  You can go and also create you can create various formats right out of the notebook  So you can also create slides out of this. So this is the  This is the these are the slides that we get that we get  Right out of the book. So if I wanted to give a presentation on the material of the book  I could use this slide presentation and then show okay. This is how this works  There's also interactive versions of these slides. So you can actually go and edit right into them  And finally for those of you who are who want to do this in a more standard way  You can also create a standard PDF out of it and then you're going to get a document in this case for the chapter  36 34 pages which then again contains all the material now in a way that can be printed or passed on to  or passed on to a publisher  and  All of this is available without any extra effort is available right out of your computer right out of the browser  And it creates an infrastructure  Which is helpful for us for teaching  For research and for and actually for dissemination of our results  so  Let me just go back here here we go  This has number of this has a number of benefits  So again, so  For one thing  We again have our prototyping we're going to have a prototyping cycle here  But now we call these prototypes chapters and we actually have our students again  This is bachelor students master students as well as phd students actually creating new notebooks  Which eventually end up in the contribution section of the book  So if a student wants to write say a test system  For mp3 files, we just had that so we automatically generate mp3 files  And then we try to fuzz try to test mp3 players with it  This becomes a chapter and this eventually and this eventually becomes this chapter gets included in the book others can work on this  And if we find a bug this also ends up in the paper  And well again  The student who previously  has spent months  Figuring out how this big implementation works  Now spends a day or two  Figuring out how the prototype works because there's a chapter which documents everything  And then the student has far more time to actually work on the actual subject of his to actually work on the subject of his thesis  Thing is  For me it takes about  Two days to implement such a chapter. Well, if it's one of the easier ones, that's the code  It takes one extra day to document it all such that I write down what I thought  And this extra day is saved again and again and again for every new student for every user who comes along  Because they don't have to ask me questions how everything works  This is the this is the big benefit of  Literate programming or generally of good documentation  So what do we have here?  We have one single source for codes books slides everything we can develop new ideas very very quickly  Including their evaluation  Jupiter notebooks great for data science. You want to do any kind of evaluation you get graphs you get interactive  You get interactive elements. You also have presentations on that subject here  You get you can hand out this material to students only the stuff that works actually goes into production  Because many of the ideas we have do not work  So it's good if you have a prototype that fails rather than the actual thing  And the chapters the prototypes  Will have a much longer re ability to have a much longer life than previously our implementations  We can have bachelor students master students work on these notebooks when I'm teaching this I'm actually giving out this material to the students first  So we're having an inverted classroom principle and then um and then I typically do live coding  So I leave out a couple of code pieces and I'm implementing them  Which is always fun for the students to see me make the same mistakes  They do and then to try and to debug code. Fortunately, this happens not too often  Um, then I'm explaining how things work. So I'm thinking loud all of this works  Well, and when students come along and say, okay, professor's only what happens?  Why is this parameter like this? What happens if you change that couldn't it be that sure we can do this live right right away?  It's like um  It's like pair programming except that you have 40 people in your room who all come up with ideas  Students like that a lot and students then become later recruits for later thesis with me  So last year we had 40 students in the lecture  The 10 students ended up in an advanced seminar where they're actually writing chapters and out of these 10 students  I'm expecting two to three PhD students coming out  Which is quite an outcome. We are in a very competitive environment in subrican where there's plenty of people hunting for good students  And yes, this is a great way to attract people  also  These notebooks are lots of fun to work with  interactive reproducible extensible  They can be reused and build upon if they are well designed that is I'm come to that in a second  And for me the question is  Why do we why do we why at our conferences do we require whatever two page pdf formatted as if it were as if it were 1970  Why do we insist on printed books people come to me?  Andreas you have finished your book. When can I finally buy this and put this into up to my bookshelf?  And I tell you know a printed book is not the exact best way of interacting with all this  But well eventually if people insist we're going to put a big  1000 page pdf somewhere in the server and then they can print it out  um  However, I'd also like to point out that  If you work with notebooks, this does not necessarily equate equate quality. Simply you do that  um, we have uh done  We have started to explore so since since we have so much fun with notebook  We wanted to know what what the general status of notebooks is  and so we took  Nearly 2000 open source notebooks from the web and did a couple of checks on them. Yeah, we could because we can because we're software engineers  um, there's a  There's a tool  that checks python code for its adherence to  A python conventions whether it's essentially whether your code is written according to python conventions and  Well, it turned out that one in three lines violates a python convention in notebooks  One in three most of this is the lines being too long or not having the exact spacing or anything  But generally this means that people will not have fun reading that code and no we don't uh, there's actually far larger than for regular python code  Where so it's about three times as many three times as many troublesome spots  This is not just stylistic stuff  40 percent of all notebooks we found had unused variables. That is there's something there's something being computed and then stored  But it's actually never being used  Which in regular code actually flags as a flags as a flags as a potential mistake because this typically happens when you have some  When you have some if you have an unused variable  This simply means well you have computed something this should be used and the reason that it is not used may point to an error later on in the code  We have just started scratching the surface here, but  We actually actually have a good feeling that there are plenty of mistakes in published notebooks  That call for that call for being discovered  Our our biggest hope is finally to find one error in one notebook that has that then  Eventually leads to the retraction of some nature or science article. That would be real cool if we could do that  But of course we adhere to the principles of responsible disclosure here very much like vulnerability  So we would tell the authors first and then they can fix things  That would be real cool. Well, we're just have started this a couple of weeks ago  This is first findings. You've seen them. You see they're here first  Also, we have deprecated functions that is notebooks are not going to be  We will not we will no longer work in a couple in in a couple of months. So  There's still lots and lots of work to be done  And this is consequence for myself and my community  We need much better tools and we need better languages  Better language integration to enable such prototyping  The program analysis community so far has very much  Uh ignored languages like python because there are toy languages because they are too simple whatsoever  But we need much much better tools for dealing with these we need tools for analyzing our programs for  analyzing python programs for helping  developers in those languages creating better creating better programs better notebooks  and  This again calls for well, this is then a call for my community to create such tools  And to hand hand them over to a community like yours such that you can use them  and in the long run  What we want to achieve here is a community  where we have a large number of people  Creating pieces of code that interact with each other that can build on each other that can that can work together to create science  both in a both speaking from a community perspective as well as from a technical perspective  A community that is really large that is strives where all of research engineers software engineers  come together and work together to build great scientific work  With this i'd like to close. So what have you seen a bit of five rules for sustainable research software  bit of experiences  With prototyping creating prototypes that eventually after having been refined end up in an implementation  a couple of insights on how to  Bring together literate prototyping in notebooks and other new tools  And finally also hint to the simple fact that simply using notebooks is not the solution to everything  but we're going to need better tools and better languages to also  Come up with better production in notebooks  With this i'd like to close. Thank you for your attention and i'm happy to take questions  Thank you very much indeed. I think we have time for two or three questions. So I can see one two three  And that's that yes, please  Three decades of open source software development has generated its own social model  Do you want to make some comments about the social context of some of this?  These ideas that you presented here  The social context for us is we are very much influenced by the open source model because also as researchers for us  It's the most economic way to have a maximum of impact. This also this also extends to our concept of notebooks  We can pass them along others can work on this and what always has made open source very successful  Is the fact that people that other people actually read the source code other people extend the source code  Because this makes it easier for others to spot errors  It also is it also is helpful that the code is available for other sources  It also is helpful that the code is available for others to run the tools on them  so if we want to try out new bug finding tools for instance, we run them on open source and  By and by bringing together these best open source practices together with literature programming  That is we introduce how things work. We allow others to extend all of this  I think we are  We are pushing forward the best practices of open source and bringing a couple of new best practices in there  Hello here. Thanks a lot for this talk. It was I think the first one  I heard to bring solid principles into research software engineering. So thank you  One question about the notebook. So how do you solve this problem that as you interact with the notebook?  A lot of yeah stuff accumulates in the jason and they are difficult to version control in git for example  Ah  That's the good thing. That's the good thing of being a being a season programmer  First thing I did is I built a couple of my little bit of tools myself  I can do that. So  There are so having said that there are established there are there are nice  Packages around that allow you to do very nice version control together with together with notebooks  Um, and we have adopted them, but it turns out that for quite a number of things we need to we need to write our own tools  so simple simple things such as  Violating python conventions for instance  I have built my own. I have built my own plugin for that which I can simply interact in jupiter with  I can press a button and then it tells me these things  So you're going to find in my code. You're going to find less of these violations because of this every reason  But I think that um, but I think that we as a community  That's partially also research software engineers. Well, generally software engineers  We need to invest more in such tools that help everybody whether this is in whatever eclipse intelligente  But also jupiter notebooks  To create to to come up with better code and to spot errors more easily and we have and I think I think this can be a great investment  There is a million of jupiter notebooks open source out there around  There are at this point. I don't know how many tens of thousands data scientists working with jupiter notebooks every day coming up with all sorts of results all day long  and  well  Somebody at some point we'll have to quality check what's actually going on in data science  We already have the science of a big reproducible reproducibility crisis  Uh, at least people push out their notebooks. That's a good thing. That's the start  But I think that there is much much more to be done and that we can do in terms of quality control. I'm also happy to hear  Matias Kattabur from DFG to join a new initiative  Soon on that focuses on quality of research software. So there's plenty of things to be done  Okay, I think we have time for one last question  uh, sorry, um  Okay, um, thank you very much for the talk. Um, I was wondering  That um, so you said we can learn from open source software for research software  I would even argue that we could learn from this for for just general  uh analysis for example or like um papers in general you talked about the reproducibility crisis. This is usually  Rather  Simple analysis scripts that people write. Um, but that are often not reproducible and um  I would even argue we should go a step further and make it as easy as possible for for the end users like the applied people  To make it make it easy to make their work reproducible from from my standpoint from my standpoint  The scientific paper as we know it these days should actually be should actually be  obsolete at least for the fields in which in which data analysis data science plays any role  And I think we should go towards towards mechanisms that allow you to that allow you to recreate the entire process from the raw source data to the finally published chart with all the intermediate steps in between with all the algorithms you used  This is how science should be  but  I am  But I have to say this. I'm a really good researcher  I am but I have to say this. I'm a reviewer for papers and  And typically well we have this big source of data  Then we apply the super magical algorithm this goes on the following five pages  Which uses the following seven seven parameters  Which we have chosen to be minus seven dot four minus 55 and six six point four and the sign of square root of two whatever  And this works see see see here's the charts. It works. Thank you very much. No our data is not available. No our algorithm is not available  No, there is no that justification for any of these choices. There's probably there's probably two  There's probably some more 30 40 design choices somewhere in there, too  This is not the way it should be  If we can make it if you can make this an open science  If you can make this an open science process  Where is this here's the raw data? This is what we found out. This is great  Where people can actually replicate this where people can reproduce this where people can talk about this  This also involves making these incisible, which is another problem  But that's another thing if you build something that's used all over the place  This is something that will end up in your CV and then you can then you can see then later you can say yes  I built this particular piece of code and everybody and  Everybody knows this thing and these are all this is our culture that we have to introduce culture that we have to change  But and it will be us software engineers in research to help with that  Thank you. Uh due to circumstances we do have time for another question. So I think there was one over there. Yeah  Andreas, I think um, it's it's good. I I I completely support your arguments and underwriters  um, uh, but I believe many here are also in a situation that  The question of scalability comes in and there's a point where you have  might have to leave this  way of working in order to come up with this multimillion  dollar information system and um, how  To find I mean that you have to change your environment and set up in  another professional software engineering environment to come up with a  So a decent way of working in that area  I think that's that's for sure. Everybody can do this and can read it as for example reading the book of our great colleague  but  What are the signs?  When to switch yes  From my perspective. So my perspective is I have to say this my perspective is to a great extent the perspective of a researcher  Where software is the subject of research and where subject is the instrument of research  So and where it is not necessarily clear from the very beginning how things should be built and what things we  Would think how things should be done and this is we are prototyping simply we're prototyping simply absolutely excels  Because this allows you to try out many alternatives before actually going for the real thing  if you are if however  Uh as also as also is often the case in projects if however you have to do something if you have if you have to do if you have to  Build something that hasn't been built  Thousand times already and where the path is pretty clear how to do this you have to build a database system for storing research data for  Replicating research data. You have to build a website  a web page whatsoever  This is of course material where such a prototyping approach  Uh very prototyping approach such as I sketched in here may also make sense say for web pages  For instance, you want to do user testing you want to come up with another design and another design  This is how it works if you want to if you choose a database you can do experiments with different databases before settling on one particular thing  but here but here  Um  It's but but here it's uh, but here it's much more important to follow best practices if they exist  That is this is how I choose a database. This is how I build things and I'm not going to event things on my own  I'm simply going to do this as a few others as a few hundred others have done before me  So for those domains where it's well understood how to build things go and familiarize yourself with the best practices in the field  If you're out there in the unknown, however  Then it's a good idea. Then it's definitely a good idea to try out number of alternatives fast quickly  Prototypes stick with simple and simple and simple solutions  Before you finally settle on one and then dive into the unknown
Andreas_Zeller_0_887.png Andreas_Zeller_888_3503.png Andreas_Zeller_3504_7511.png Andreas_Zeller_7512_17495.png Andreas_Zeller_17496_17867.png Andreas_Zeller_17868_18107.png Andreas_Zeller_18108_18311.png Andreas_Zeller_18312_18359.png Andreas_Zeller_18360_18959.png Andreas_Zeller_18960_19703.png Andreas_Zeller_19704_20891.png Andreas_Zeller_20892_22715.png Andreas_Zeller_22716_22883.png Andreas_Zeller_22884_27611.png Andreas_Zeller_27612_28043.png Andreas_Zeller_28044_28451.png Andreas_Zeller_28452_28535.png Andreas_Zeller_28536_29195.png Andreas_Zeller_29196_29423.png Andreas_Zeller_29424_30479.png Andreas_Zeller_30480_30731.png Andreas_Zeller_30732_30875.png Andreas_Zeller_30876_31319.png Andreas_Zeller_31320_32363.png Andreas_Zeller_32364_32747.png Andreas_Zeller_32748_32927.png Andreas_Zeller_32928_33131.png Andreas_Zeller_33132_38987.png Andreas_Zeller_38988_39227.png Andreas_Zeller_39228_39275.png Andreas_Zeller_39276_40067.png Andreas_Zeller_40068_40283.png Andreas_Zeller_40284_44375.png Andreas_Zeller_44376_46151.png Andreas_Zeller_46152_46295.png Andreas_Zeller_46296_46403.png Andreas_Zeller_46404_46463.png Andreas_Zeller_46464_53939.png Andreas_Zeller_53940_57539.png Andreas_Zeller_57540_58031.png Andreas_Zeller_58032_60383.png Andreas_Zeller_60384_67475.png Andreas_Zeller_67476_69083.png Andreas_Zeller_69084_69923.png Andreas_Zeller_69924_70355.png Andreas_Zeller_70356_70475.png Andreas_Zeller_70476_71015.png Andreas_Zeller_71016_71999.png Andreas_Zeller_72000_72383.png Andreas_Zeller_72384_72647.png Andreas_Zeller_72648_72707.png Andreas_Zeller_72708_73115.png Andreas_Zeller_73116_73235.png Andreas_Zeller_73236_73607.png Andreas_Zeller_73608_74183.png Andreas_Zeller_74184_74255.png Andreas_Zeller_74256_74447.png Andreas_Zeller_74448_74914.png
Okay, good morning everyone. I hope you can hear me well. It's a great honor to be among you I'm very happy to see this community of research software engineers forming such that those of you always been We've all been working for other people in some shady rooms all isolated now can find nothing Can I can finally come together and get the appreciation that you deserve? so I'm here as your as your as one of your keynote speakers today and as a keynote speaker and Also, actually as being one of the eldest one of the oldest guys in the room Okay, okay, we Okay, we're going we're going to debate that As you as a keynote speaker your typical situation is to come in very much like very much like this person here on this picture And to bring over some insides to you Which you may otherwise which you may otherwise not have so my role here is the Super duper big software engineering professor who's now meant to come in and tell you how to do things By the way, this is you on the other side. No worries. We'll have a gender neutral version of that slide later on So what am I what am I what am I actually what am I actually doing here? Well, sorry simple I'm passing on knowledge to you and this knowledge well traditionally comes in the form say of books for instance Here's such a book which I'm now handing over to you is actually not my book But this would be say a traditional book on software engineering When people ask me dear professor Zeller, we have these and these problems. What is it that we should do? Well, the first thing I told them well get a book on software engineering This is actually an 800 page book and you can read the most important things in about two hours or so Just by skimming through it's a software engineering is typically a very fast read It takes more time to implement this stuff, but getting the most important things done is actually not very difficult So here I am I'm done. Thank you very much read the book And okay good. That's not what I want to do good and thing is a book slow and software engineering typically Typically tell you well what software engineering is meant for that is to build some Super duper huge software system say you want to build whatever a travel reservation system for the world or a worldwide Internet worldwide information framework or whatsoever. Yes, and if you want to build this, it's good to have a professional software engineer on board plus reading a book But when we're working on when we're talking about research software, we typically have much more constrained budgets in time in money and resources and Therefore the super duper big software engineering Processes are typically not the best ones But for these I would rather recommend to take a look not at software engineering books But rather look but rather take a look into open source software developed by volunteers in their spare time with little money little resources and yet having plenty of impact That is question is how can you build software that works with a minimum of effort and a maximum of impact? Well, this is exactly what we want to do in our research projects just as well So and we can learn quite a number of things from open source software I'll point out the five most important the five most important Features to you the five things which I consider most important and these are the ones that are sort of built in this Build a necessary foundation for creating open source for creating successful open source projects The first one in here Is to have a repository? That's not well The question of the question here is when I'm telling you this half of you are going to know Yeah, we know that thank you very much and the other half and the other half of you will oh wow That's super cool. This is the typical reaction I get so I'm going to spend a few minutes on this just to but I'm going to gloss over it Such that those of you who don't know it. Yes, we can look this up The other ones will find out but this is a recent but this is a necessity You have a repository have version control you have a place where it can we put up all your software and where it will be available for others Good thing is to make this Accessible to the public in the first place such that it's easier for others to join your project make this available make this open source Adhere to standard conventions you go to one of the standard services such as github and you get everything in one place You get a version repository you can file bugs you can have more collaborators and it's all set up for you It's very easy to use don't typically there's very little reason to go for something that you have homegrown Second thing is Everybody should be able and everybody I mean Everybody in the world should be able to download and build your system from scratch That is you should be able to press a button and then everything will everything will build automatically This for a very simple reason the more people you have to join your project the more people will eventually have to build your project and If every time you have to resort to a number of phone calls or may or email exchanges and everything This is going to be time-consuming so automate as much as you can. This is necessary for growth The third thing is the third thing is have tests Yeah, I already see some people already see some people frowning here. This is this This is also a necessity because people not only want to build your system. They also want to rely on it and even more important When your system grows and when you have more and more users when it's actually in use people will come and will want to We want to change things when you change things You want to make sure that after a change everything works everything everything still works as intended So you should have a big set of automatic tests that you can pretty again that you can that you can run Typically, this will be unit tests or set of validation tests that you can run at the touch of a button Or even if you're if you're so inclined you can even make them run automatically all day long with continuous integration That's the keyword here But the general idea is you should have tests that you can run that automatically check things because then it's much easier for you to Have others contribute to your project such that you can validate whether their changes and fixes don't break anything new Just for the record and software engineering project. So those that we analyze one out of three fixes introduces another bug So this is why such tests are a necessity And I'm not saying one of three changes. I'm saying one of three fixes, right? So just big just people want to fix one bug and then they introduce another bug and then they freeze it another bug Over time there's a damping effect that's that eventually the number of bugs goes down But still you need to make sure that you want to make sure that you don't get accidental new bugs Number four on my list is to structure your software such as a software software Is to structure your software such that is open for extension if your software is successful It's going to grow and if your software grows over time Then you can run into trouble. This is also something that software engineers have realized long ago Here's a simple example. This is a piece of software which has been Expanded again and again So we have a connection type here and initially it was modem 56k and now people have added ethernet and then people have added Wireless and then people have added UMTS and next we're going to have 5g in there And every time there's a new piece of code being added and over time this piece of software becomes entirely becomes huge and Unmaintainable if you have some fun go look at the window source code Which is exactly written this way Where you have say Where you have say some standard function. I want to allocate memory This is the standard way and then come all that then come all sorts of special cases which replicate bugs for Which replicate bugs that have been there in previous version of windows? So there's actually checks in there would say if this is sim city To K in the version for windows 95 then replicate this very specific bug That also was there in windows 95 such that Sim city 2k still runs on your whatever windows on your current windows 10 system This is what happens if you insert a CD of Wind to K. Yes, there's this code been executed and needless to say this is a huge maintenance problem for windows That is on the other hand They may be still a few dozen people in the world who still play sim city 2000 and they're very happy that it still runs Okay, really it's fun looking at this looking at the source code you find all this stuff all this stuff. It's just horrible Everywhere else say if this will index and open source system people would simply say well Just just play an open source game and fix it. No, no, no chance. This is windows. It is binaries binaries are never going to change better option here is to design your system in A way that is actually extensible So object oriented programming tells you here that it's a good idea to deconnect the application from the actual implementation And then you have an abstract class that says okay This is my interface for connections and here I can have an I can have a modem connection Wi-Fi connection ethernet connection every time somebody comes up with some new kind of connection Simply create a new superclass and everything that is in my app over here Can stay the same because it uses the same interfaces and Last but not least this is frequently overlooked It doesn't suffice simply to put a big blob of software on your son Make them to make a big blob of software available for others. You have to tell people how to actually use it So it's a good thing to invest into it's a very good thing to invest into tutorials Not simply documentation documentation is one thing but the other thing is simply having you read me Where do I start? How do I get this to run? How do I get this to use? This is something that open source developers over or often overlook But this is the number one thing that actually gets people to use your system Please provide some tutorials some help some examples Whatever you can do in order to make it easier for others to adopt your system so all of these are pretty all of these are simple simple rules that are are Not necessarily easy to follow the design thing for instance Yeah, you can spend you can certainly spend weeks and weeks on that discussing these but overall these are all these are all things That an open source software have proved to be have proven to be Successful and that have proven to have lots of impact with relatively little effort so so So this of course is part of The knowledge that we can pass along as software engineers. Oh and by the way If you find that there's too many males on that picture here's a here's another variant. This is from Ruth Schreiber That is a bit more that is a bit more distributed Actually, it would be a good thing to have a female keynote speaker next year such that we can pass on that slide and the But one thing I also found out is that when People talk to us We as software engineers, how is that when we build our own systems? How is life going to be because we know right we know everything is it so this is going for so people sometimes Assume that when we build software and this is some kind of paradise where we have unicorns and they fart rainbows and whatever And of course every single piece of every single line of code I'm right. I'm writing works from the very beginning actually I'm writing 1000 lines in a row without ever removing the cursor anyway, and then I press a button and oops automatically everything unfolds Yeah, sure well, I Would be perfectly happy to keep the reality to keep that illusion as it is But the reality is actually a bit different the reality is more like this So this is me and this is me making great talks about however How everything is just fine and there's surroundings here. This is say my my institute my software my computer whatsoever and The reason for that is The reason for that is actually pretty simple. Let's get back to this picture one more time But now let's focus not so much on the interaction between me and customers or whoever But red slather focus on This part of the picture. This is actually the people producing this knowledge and this is the people producing that software Actually, actually I'd like to imagine myself me being the beardy guy in the middle and all of these all of these are obviously my PhD students and others Well in this case obviously supporting me without their support look at this guy out out here must be a system administrator with You don't even see the face. This is how unimportant is see Michelangelo Pa, okay? With without these guys obviously I would just fall down from the sky and then who knows what happens So actually actually it's all these you know, they're struggling is horrible without all these guys and nothing would ever happen guys and girls and the thing is however that This that it's actually there. It's actually there in the interaction of many people producing software Where the trouble begins and we are not exempt from this trouble at all. No, I'm sorry not at all because well Let's face it So we are producing software as we are producing the very same software that we eventually pass on over to you So that's nice. We building this stuff. We're writing this printing it out, too But the standard way we are doing this Actually works as follows. So Some of us has a great idea. Hey, we could build a whatever we could build a perfect test system We could come up with full-fledged verification whatsoever. Wow, whatever good thing. Wow. Oh, yes And the idea is super typically very simple if only people would do it right then we build it and since we are and Since all of us are real matches. We want to build this not for some toy language. We built this for see or for Assembler machine language whatsoever process a microcode because just it's so hard and then we find out it doesn't really work Then we refine the whole thing and sometimes it works and then we have to come up with more ideas And then we reject the original ideas and we keep on working on this stuff and Then it's if then this code evolves evolves I eventually gets into a direction that was entirely different from the original idea But yet the original data structures came from the original idea. So we have to revise these as well in other words eventually what we get is a big pile of code which sometimes works sometimes is dead Sometimes was in which in places was meant to work generally which in other places has a couple of ugly very Copy a couple of very ugly hacks But in the end we managed to get all of all of this pile to run on A set of subjects and we get a paper out of it We're happy and there we are and then we can put this and then in principle we could put this big pile of code on a server and Well, let the rest of the world deal with it as they can Of course, there is no documentation or little documentation. There's no tutorials. Well, there sometimes are tests Yeah, there's some of our software actually looks like this. So forgot everything So so all of these five rules which I told you beforehand. This is something I tell to my PhD students to and sometimes they listen to it The interesting thing is once we have now we are in this situation and Then say a student comes along say whatever and you some the student who wants to write the thesis a customer whatsoever and asks, how can I how can you use that and Then we have the then the student gets a look into the code and Well, if he's not faint of heart some of the students just run away when they see our code But then typically they think it's their fault not ours So they think okay, I'm just going to dig deep into that and they and and and they and they talk to the PhD students and the PhD Students guide them into it into the code that they're written this takes a week or so and then the student learns more and more and more and Say suppose this is a bachelor's thesis, right? So this is three months So a student spends about a one month just figuring out what all of this is about Spends another month figuring out how to use it and then all of a sudden Oh, there's only one month left for running all the experiments and for writing a thesis about it So typically the student has one week to do the experiments Which then are incomplete and then finally one week to do the final write-up Which is very which is not very helpful because the final write-up is just as incomplete as the documentation was in the first place So we're running into a big timing problem because we're spending lots and lots of human time Just figuring out how things just just just spending just spending time with Fox and guiding them into individual parts And this is has been a problem for bar for as long as I can remember So since I'm in academia, this has been this has been pretty much the case Well, there's a couple of pieces of software which we have built the official way, but well, it's research So we're building stuff. We don't know where it's going to end if only we knew in the very beginning life would be much easier So and by the way, I have to point out when the project runs out and the PhD students Who have created all this leave? Well, then the code is effectively dead The paper persists paper lives forever. That's great. The idea still lives on it's in the paper, but for everything else try to Write mean email dear professor Zella. We've read your paper out of say of five years ago. We'd like to work on the code Sure, I have the code somewhere. Oh, it requires eclipse 2.1. Do you have that somewhere? No, we don't maintain eclipse. Can you adapt this to a new version of eclipse? No, no, no We can't do that. No, no chance. Oh, too bad. Okay, good We are going to implement it ourselves and then other people do this go through the very same cycle until their PhD student goes away So what are we doing now to to get over this problem? Which actually is my problem just as well as the problem of every software engineer with research software one thing is We are now very much embracing prototyping What does prototyping mean? Well, it's actually the same cycle as before So we have an idea except that in this step We are building we are building something that is not necessarily meant to be the real thing But we do something in the simplest and fastest way as we can where fast means We are fast not the computer is fast we don't care about the computer speed at this point because CPU cycles are cheap our cycles are expensive So and then we keep on refining this prototype and with for this way is prototype languages We are scripting languages these days So we don't implement things in C or Java and the first thing is Python instead or It's Python instead or Ruby the simplest thing that can that we can use to validate our idea is just fine And then we again write a paper about it Turns out to that turns out that in our case We've always thought the community is not going to accept anything that doesn't work on the real language like see it turns out This was a total mature attitude all of Netflix runs on Python Google has Python all the way down Facebook has millions of code lines of Python. So paper Python it is and ups turns out that by building such prototypes We are about five to ten times faster than we used to be and The neat thing when a student comes along turns out these prototypes are much easier they're much easier to understand because they don't contain all Sorts of optimizations. It's much easier to say analyze Python programs for instance So students can interact with these with this parchment with these parts much faster And also they can quicker they are much faster in writing papers and fun part is Since these prototypes serve as blueprints for how to do things because they focus on the algorithm on the algorithms other than Implementation details we have students write our implementations trying out what this thing that we already worked out how it works in a prototype After we have after we have understood how the algorithm should be how the approach should be then we let students Port all this into other languages like C or other operating systems whatsoever Just to find out what the performance is this performance thing can then go into the paper and but if this implementation dies Down here if this dies when the student goes along that's not so much of a problem because the prototype and the idea still persists so This is a one rule we have learned and which we have which we have adopted And this is very much boosted our research The second thing we're doing a lot these days is literate programming Literate programming was invented by Donald Knuth If you don't know Donald Knuth, he's one of the biggest Most influential people in computer science and in science overall Donald Knuth has written one of the most influential books on Programming art of computer programming and in order to wrote in order to write these books He invented his own type setting system. This is tech and latex as many of you may know this is the language with the backslashes And so he has a lot has a lot of influence in computer in computer type setting too And it's incredible When I sometimes come to a conference, there's a couple of there's a couple of students typically from China want to take pictures with me That's all cute Recently I met Donald Knuth and I absolutely wanted to have a picture taken with me. So that was my situation Didn't wash my hand afterwards. Well, that was a year ago since then Finally washed it. This is my relationship to don Knuth. So so so I may so so He's if I'm a god, he's a super god and well actually to him I'm somewhere very down there in the mud and One thing you know one thing invented Uh is called literate programming and the idea here is that you do have one single document Which contains both code as well as the introduction to it This is how uh, this is how you write a literate literate program. You have pieces of code That explain individual parts of the program and then you have documentation for it And there is actual very much as in latex There's actual characters that change the meaning of things and from the single document from one single document And that contains both That contains both an introduction to the code as well as the code itself You can derive well the code you can extract the code and this is something you directly feed into a compiler You don't even look into it The second thing however is you can pipe this into a type setting system. Well, it's tech again and Oh, sorry wrong button And then this automatically gives you a very nicely printed version of your code Which then comes with all of the Introduction so it tells you here. This is how we do it. We since we give a couple of introductions We don't do this we don't do that and you can actually read through this entire book that is that is being produced there From one part to the next Which by the way gives you a very nice introduction on how the code works. So you understand So you understand how it's going how it's going to be used Which then again can be very useful for others who want to work with this very code so this is something That we also have adopted these days in order to create in order to create code That is well documented and that can be and that also Um, that also is all self contained in one single document. That is you don't have separate documents for whatever. Here's the tutorial Here's the code. Here's another help text. Here's the here's the here's the comments in the code But we wanted to have one single representation Now we have these two ideas. We'd had prototyping and literate programming Now we bring in now we went and brought these together coming up with a new term which we call literate prototyping This is actually the first time ever i'm using this i'm using this word combination. I made this up just yesterday in order to illustrate what we're doing Um, literate prototyping Sounds like a super fancy idea, right? And in particular you don't want to do the opposite But you may you may want to do not prototyping, but you definitely not want to be illiterate here. So This is why this is the nice thing about this word literate And the question is how do we do this? Well, we actually we actually use plenty of jupyter notebooks these days Which actually bring these features together which allow us to If you haven't worked with jupyter notebooks yet now is a good chance These are tools that allow you to bring together documentation that is text elements as well as code elements And you have them right in your browser. You can execute the code right away You can toy with the code and while you and all the results of any experiments can be included too Having examples having visualizations. This is actually far superior to what we can do what we can do in literate programming As invented by Knuth. Well, this was because literate programming was invented at the time When we didn't have these large interactive systems And just to give you an idea How we do this I'm going to give you a very short demo here So let's see whether Things still work as they should So we're switching over How cool is that wonderful so you don't see a thing great Connection failed. It was to be expected Okay, good So This is a bit small, right? So let's try to Get this a bit bigger So this is a this is a book Well, this is a book on automated testing which I have just completed A couple of weeks ago. This is a book on Generating software tests which allows you to Which allows you to explore quite a number actually to Eight chapters of various ways of testing your program automatically And this is one of the chapters in here. So you see we have a nice. We have a nice Book in the browser which comes with all sorts of extra features quotes and whatnot And here we do have actually here. We have actual code in here Which you can read and this is the same as the output of a little of a little programming process Where you have code in here a way of code and way to do it In here a way of code and we do have actual text that describes the code And we also have examples of how this code is being used This is a fuzzer for instance a fuzzer produces a string Of random arguments which you can then pipe into a program of your choice in order to get it to break And in the next chapters we're going to find out how to come up with all sorts of different fuzzers The neat thing is that this is actually the output of a jupiter notebook And this jupiter notebook is something that let me see you can actually go Resources edit as notebook if I press on this then you will automatically get into this notebook This is going to be started In the background and so depending on the current network speed we'll have this in a minute or so In the meanwhile, this is this is what this looks like when you have it in your browser And this is what we give out to students So you have cells here with code which you can directly edit as you like So you can add new code to it you can change the code and you can also see And you can also find out how any of the changes you make Is finally going to is finally going to affect the outcome So students can directly work with this and when I say students This involves students. I'm having in my class or regular Or the bachelor's and master's students, but this also involves any PhD students who want to work with this code This also includes any researcher out there in the world who wants to extend this code Because people can directly see okay. Here's the introduction. This is how it works. Here's the code This is how this is this is how you can use it and here's examples on how to how to and how to put this to use jupiter notebook is very versatile here you can You can go and also create you can create various formats right out of the notebook So you can also create slides out of this. So this is the This is the these are the slides that we get that we get Right out of the book. So if I wanted to give a presentation on the material of the book I could use this slide presentation and then show okay. This is how this works There's also interactive versions of these slides. So you can actually go and edit right into them And finally for those of you who are who want to do this in a more standard way You can also create a standard PDF out of it and then you're going to get a document in this case for the chapter 36 34 pages which then again contains all the material now in a way that can be printed or passed on to or passed on to a publisher and All of this is available without any extra effort is available right out of your computer right out of the browser And it creates an infrastructure Which is helpful for us for teaching For research and for and actually for dissemination of our results so Let me just go back here here we go This has number of this has a number of benefits So again, so For one thing We again have our prototyping we're going to have a prototyping cycle here But now we call these prototypes chapters and we actually have our students again This is bachelor students master students as well as phd students actually creating new notebooks Which eventually end up in the contribution section of the book So if a student wants to write say a test system For mp3 files, we just had that so we automatically generate mp3 files And then we try to fuzz try to test mp3 players with it This becomes a chapter and this eventually and this eventually becomes this chapter gets included in the book others can work on this And if we find a bug this also ends up in the paper And well again The student who previously has spent months Figuring out how this big implementation works Now spends a day or two Figuring out how the prototype works because there's a chapter which documents everything And then the student has far more time to actually work on the actual subject of his to actually work on the subject of his thesis Thing is For me it takes about Two days to implement such a chapter. Well, if it's one of the easier ones, that's the code It takes one extra day to document it all such that I write down what I thought And this extra day is saved again and again and again for every new student for every user who comes along Because they don't have to ask me questions how everything works This is the this is the big benefit of Literate programming or generally of good documentation So what do we have here? We have one single source for codes books slides everything we can develop new ideas very very quickly Including their evaluation Jupiter notebooks great for data science. You want to do any kind of evaluation you get graphs you get interactive You get interactive elements. You also have presentations on that subject here You get you can hand out this material to students only the stuff that works actually goes into production Because many of the ideas we have do not work So it's good if you have a prototype that fails rather than the actual thing And the chapters the prototypes Will have a much longer re ability to have a much longer life than previously our implementations We can have bachelor students master students work on these notebooks when I'm teaching this I'm actually giving out this material to the students first So we're having an inverted classroom principle and then um and then I typically do live coding So I leave out a couple of code pieces and I'm implementing them Which is always fun for the students to see me make the same mistakes They do and then to try and to debug code. Fortunately, this happens not too often Um, then I'm explaining how things work. So I'm thinking loud all of this works Well, and when students come along and say, okay, professor's only what happens? Why is this parameter like this? What happens if you change that couldn't it be that sure we can do this live right right away? It's like um It's like pair programming except that you have 40 people in your room who all come up with ideas Students like that a lot and students then become later recruits for later thesis with me So last year we had 40 students in the lecture The 10 students ended up in an advanced seminar where they're actually writing chapters and out of these 10 students I'm expecting two to three PhD students coming out Which is quite an outcome. We are in a very competitive environment in subrican where there's plenty of people hunting for good students And yes, this is a great way to attract people also These notebooks are lots of fun to work with interactive reproducible extensible They can be reused and build upon if they are well designed that is I'm come to that in a second And for me the question is Why do we why do we why at our conferences do we require whatever two page pdf formatted as if it were as if it were 1970 Why do we insist on printed books people come to me? Andreas you have finished your book. When can I finally buy this and put this into up to my bookshelf? And I tell you know a printed book is not the exact best way of interacting with all this But well eventually if people insist we're going to put a big 1000 page pdf somewhere in the server and then they can print it out um However, I'd also like to point out that If you work with notebooks, this does not necessarily equate equate quality. Simply you do that um, we have uh done We have started to explore so since since we have so much fun with notebook We wanted to know what what the general status of notebooks is and so we took Nearly 2000 open source notebooks from the web and did a couple of checks on them. Yeah, we could because we can because we're software engineers um, there's a There's a tool that checks python code for its adherence to A python conventions whether it's essentially whether your code is written according to python conventions and Well, it turned out that one in three lines violates a python convention in notebooks One in three most of this is the lines being too long or not having the exact spacing or anything But generally this means that people will not have fun reading that code and no we don't uh, there's actually far larger than for regular python code Where so it's about three times as many three times as many troublesome spots This is not just stylistic stuff 40 percent of all notebooks we found had unused variables. That is there's something there's something being computed and then stored But it's actually never being used Which in regular code actually flags as a flags as a flags as a potential mistake because this typically happens when you have some When you have some if you have an unused variable This simply means well you have computed something this should be used and the reason that it is not used may point to an error later on in the code We have just started scratching the surface here, but We actually actually have a good feeling that there are plenty of mistakes in published notebooks That call for that call for being discovered Our our biggest hope is finally to find one error in one notebook that has that then Eventually leads to the retraction of some nature or science article. That would be real cool if we could do that But of course we adhere to the principles of responsible disclosure here very much like vulnerability So we would tell the authors first and then they can fix things That would be real cool. Well, we're just have started this a couple of weeks ago This is first findings. You've seen them. You see they're here first Also, we have deprecated functions that is notebooks are not going to be We will not we will no longer work in a couple in in a couple of months. So There's still lots and lots of work to be done And this is consequence for myself and my community We need much better tools and we need better languages Better language integration to enable such prototyping The program analysis community so far has very much Uh ignored languages like python because there are toy languages because they are too simple whatsoever But we need much much better tools for dealing with these we need tools for analyzing our programs for analyzing python programs for helping developers in those languages creating better creating better programs better notebooks and This again calls for well, this is then a call for my community to create such tools And to hand hand them over to a community like yours such that you can use them and in the long run What we want to achieve here is a community where we have a large number of people Creating pieces of code that interact with each other that can build on each other that can that can work together to create science both in a both speaking from a community perspective as well as from a technical perspective A community that is really large that is strives where all of research engineers software engineers come together and work together to build great scientific work With this i'd like to close. So what have you seen a bit of five rules for sustainable research software bit of experiences With prototyping creating prototypes that eventually after having been refined end up in an implementation a couple of insights on how to Bring together literate prototyping in notebooks and other new tools And finally also hint to the simple fact that simply using notebooks is not the solution to everything but we're going to need better tools and better languages to also Come up with better production in notebooks With this i'd like to close. Thank you for your attention and i'm happy to take questions Thank you very much indeed. I think we have time for two or three questions. So I can see one two three And that's that yes, please Three decades of open source software development has generated its own social model Do you want to make some comments about the social context of some of this? These ideas that you presented here The social context for us is we are very much influenced by the open source model because also as researchers for us It's the most economic way to have a maximum of impact. This also this also extends to our concept of notebooks We can pass them along others can work on this and what always has made open source very successful Is the fact that people that other people actually read the source code other people extend the source code Because this makes it easier for others to spot errors It also is it also is helpful that the code is available for other sources It also is helpful that the code is available for others to run the tools on them so if we want to try out new bug finding tools for instance, we run them on open source and By and by bringing together these best open source practices together with literature programming That is we introduce how things work. We allow others to extend all of this I think we are We are pushing forward the best practices of open source and bringing a couple of new best practices in there Hello here. Thanks a lot for this talk. It was I think the first one I heard to bring solid principles into research software engineering. So thank you One question about the notebook. So how do you solve this problem that as you interact with the notebook? A lot of yeah stuff accumulates in the jason and they are difficult to version control in git for example Ah That's the good thing. That's the good thing of being a being a season programmer First thing I did is I built a couple of my little bit of tools myself I can do that. So There are so having said that there are established there are there are nice Packages around that allow you to do very nice version control together with together with notebooks Um, and we have adopted them, but it turns out that for quite a number of things we need to we need to write our own tools so simple simple things such as Violating python conventions for instance I have built my own. I have built my own plugin for that which I can simply interact in jupiter with I can press a button and then it tells me these things So you're going to find in my code. You're going to find less of these violations because of this every reason But I think that um, but I think that we as a community That's partially also research software engineers. Well, generally software engineers We need to invest more in such tools that help everybody whether this is in whatever eclipse intelligente But also jupiter notebooks To create to to come up with better code and to spot errors more easily and we have and I think I think this can be a great investment There is a million of jupiter notebooks open source out there around There are at this point. I don't know how many tens of thousands data scientists working with jupiter notebooks every day coming up with all sorts of results all day long and well Somebody at some point we'll have to quality check what's actually going on in data science We already have the science of a big reproducible reproducibility crisis Uh, at least people push out their notebooks. That's a good thing. That's the start But I think that there is much much more to be done and that we can do in terms of quality control. I'm also happy to hear Matias Kattabur from DFG to join a new initiative Soon on that focuses on quality of research software. So there's plenty of things to be done Okay, I think we have time for one last question uh, sorry, um Okay, um, thank you very much for the talk. Um, I was wondering That um, so you said we can learn from open source software for research software I would even argue that we could learn from this for for just general uh analysis for example or like um papers in general you talked about the reproducibility crisis. This is usually Rather Simple analysis scripts that people write. Um, but that are often not reproducible and um I would even argue we should go a step further and make it as easy as possible for for the end users like the applied people To make it make it easy to make their work reproducible from from my standpoint from my standpoint The scientific paper as we know it these days should actually be should actually be obsolete at least for the fields in which in which data analysis data science plays any role And I think we should go towards towards mechanisms that allow you to that allow you to recreate the entire process from the raw source data to the finally published chart with all the intermediate steps in between with all the algorithms you used This is how science should be but I am But I have to say this. I'm a really good researcher I am but I have to say this. I'm a reviewer for papers and And typically well we have this big source of data Then we apply the super magical algorithm this goes on the following five pages Which uses the following seven seven parameters Which we have chosen to be minus seven dot four minus 55 and six six point four and the sign of square root of two whatever And this works see see see here's the charts. It works. Thank you very much. No our data is not available. No our algorithm is not available No, there is no that justification for any of these choices. There's probably there's probably two There's probably some more 30 40 design choices somewhere in there, too This is not the way it should be If we can make it if you can make this an open science If you can make this an open science process Where is this here's the raw data? This is what we found out. This is great Where people can actually replicate this where people can reproduce this where people can talk about this This also involves making these incisible, which is another problem But that's another thing if you build something that's used all over the place This is something that will end up in your CV and then you can then you can see then later you can say yes I built this particular piece of code and everybody and Everybody knows this thing and these are all this is our culture that we have to introduce culture that we have to change But and it will be us software engineers in research to help with that Thank you. Uh due to circumstances we do have time for another question. So I think there was one over there. Yeah Andreas, I think um, it's it's good. I I I completely support your arguments and underwriters um, uh, but I believe many here are also in a situation that The question of scalability comes in and there's a point where you have might have to leave this way of working in order to come up with this multimillion dollar information system and um, how To find I mean that you have to change your environment and set up in another professional software engineering environment to come up with a So a decent way of working in that area I think that's that's for sure. Everybody can do this and can read it as for example reading the book of our great colleague but What are the signs? When to switch yes From my perspective. So my perspective is I have to say this my perspective is to a great extent the perspective of a researcher Where software is the subject of research and where subject is the instrument of research So and where it is not necessarily clear from the very beginning how things should be built and what things we Would think how things should be done and this is we are prototyping simply we're prototyping simply absolutely excels Because this allows you to try out many alternatives before actually going for the real thing if you are if however Uh as also as also is often the case in projects if however you have to do something if you have if you have to do if you have to Build something that hasn't been built Thousand times already and where the path is pretty clear how to do this you have to build a database system for storing research data for Replicating research data. You have to build a website a web page whatsoever This is of course material where such a prototyping approach Uh very prototyping approach such as I sketched in here may also make sense say for web pages For instance, you want to do user testing you want to come up with another design and another design This is how it works if you want to if you choose a database you can do experiments with different databases before settling on one particular thing but here but here Um It's but but here it's uh, but here it's much more important to follow best practices if they exist That is this is how I choose a database. This is how I build things and I'm not going to event things on my own I'm simply going to do this as a few others as a few hundred others have done before me So for those domains where it's well understood how to build things go and familiarize yourself with the best practices in the field If you're out there in the unknown, however Then it's a good idea. Then it's definitely a good idea to try out number of alternatives fast quickly Prototypes stick with simple and simple and simple solutions Before you finally settle on one and then dive into the unknown
here Is to have a repository? That's not well The build automatically This for a very simple reason your software such as a software software Is to whatsoever and asks, how can I how can you use one of the biggest Most influential people in code Which then comes with all of the contained in one single document. That is you features together which allow us to If you this in a minute or so In the meanwhile, this is for dissemination of our results so Let me just what have you seen a bit of five rules for sustainable research software bit of experiences of open source software development has generated from open source software for research software I