SDAPS
Berg, Benjamin
This talk will give an introduction into SDAPS how it works and how it can be used automate surveys or even examinations on paper. In the core SDAPS is an optical mark recognition software that is integrated with LaTeX and LibreOffice. It provides all the utilities to automate simple surveys, including questionnaire creation, manual error correction, and a simple report generation module. Also important is its flexibility to be used for data acquisition by either exporting the data or even interfacing with SDAPS python API directly for specialized applications. Benjamin Berg
 Okay, I'm going to talk a bit about Asstops today, which is a project that, well, basically it is about optical micro recognition mostly. We may have seen the pieces of paper in the back that you had, which is a small survey that they are doing here, and they are using a software which is non-free, which is actually used by a lot of universities throughout Germany. What we had, why we started Asstops, is that we needed to read out exactly these forms at the university. So, SDAPS, which is an acronym, I'm not going to spill it out, but it probably doesn't make much sense. It's really just an optical micro recognition. It's only about recognizing which checkboxes are marked by the user and then figuring that out and getting all the data out from the scans that you have done. However, to a large extent, the problem there is to the questionnaire design. You want to design the questionnaire in a way that is easy for the user. You want to quickly create something like this. You don't want to, I don't know, I've seen tools where you create the questionnaire somewhere else and then you import it, then you mark the regions that need to be recognized. And in my opinion, that's very, not very useful. So what SDAPS does is that you create the questionnaire in Lattech and all the metadata that is required is exported into a file, which is then picked up later on and can be analyzed by the program. In general, it's a Python program with just about 10K lines, so it's relatively small overall. And it's designed in a way that is extensible so that you could create own plugins and commands that build on top of the internal data model that it has. And yeah, mostly it's a command line utility. The only point where the user interface comes into play is for doing manual error prediction because that's where you need a user interface and the rest of this is very much designed to be scriptable. So we started the project back in 2008 actually when in Cod3 at the university, it was still the university of Cod3 at the time, today is the Cod3 Institute of Technology, it was decided that the evaluation would go into the centralized and we weren't quite happy with the bylaws that were in place at the time. So what we decided at the faculty of electrical engineering is that we would boycott the evaluation at the university. Now we had an issue because at the same time there was the change to the Bachelor of Master System here in Germany going on and we really didn't want to lose the data that was coming in especially for the lower, for semester's bachelor's if there were any changes. So we had to actually use the, do the data analysis ourselves. What we ended up doing is that we created a tool which read out these forms in a pretty quick hack like over the Christmas holidays and then ended up actually evaluating most of the, let's remove the cell phone, should help, most of the lectures at the faculty were using our tool and actually in cooperation with the faculty itself. So that worked very well and after that we realized that we could create such a tool but obviously we were using these original forms that time so we needed a way of also creating the format not only reading it in and we first decided to use OpenOffice for that and later LibreOffice and then even later created the Latin support which is now basically the main part of doing it. So since then people have tried, oh sorry, probably should have, let's do this first. Since then people have used it for a lot of different things or even tried to use it for different things. So this is a survey that is being used in the UK by a company called GPTools and they offer some services there for doctors who need to do a yearly evaluation of their office for the NHS and what they offer is a very simple form so the doctor or 20 patients or something the day and send it in and they will create a nice report for it. There is even an example report online. They basically take the normal output that SDAPS generates at somewhere statistics data themselves and that's then it, what they are offering and they are selling the service there. Another thing is examinations so both every TH often the physics department there, they are looking into using that and they have, sorry? Okay, sure, no worries. All right, that is a good point. That should be it. Okay, thanks for the, so yeah, examination is a point which is actually quite interesting. I'll come back to that a bit later. So they have designed a completely new way of asking, formulating the questions for the examination. This is an example for the tutorials that they were using but they have used it for the real examination too and this is an example from Princeton where they have tried to get it to work and you can actually, if you look closely you can see that SDAPS is not really designed to handle these cases. So what they did here is that there is a large box which is just taking out as an image and then they have placed something else inside to be able to get the data out from after the scan and then have a look at the image there. But I've seen, I think that they have also issues that students write outside the field and stuff like that which makes it a lot harder to handle the data later on. So yeah, this is different layouts, layout issues that they are having. They, the general me there, he did a pretty good job trying to hack out the class in a way, the Latif class, that you could get the layouts to work like this. I'm working on improving this. This is another example from Nation Builder. This is a sheet that was used during the 2014 elections in the US. This was for the, well, door to door, you go, walk from door to door, you ask questions and then you just check out the answers there. And they sold the service also to some people, to some organization and lecture parties I think that walked around there. In theory voting, I have had someone doing this ballot at some point. I'm not sure if they ever used it. I think that the pirates in Berlin have tried to use SDAPS for a relatively complicated voting system because it's a lot, it's a pain to count manually. But I've not, I don't know whether they're actually using it other than trying it out once. Just month ago I got the request for someone who wanted to do Autocross, German Autosladung or something. And there you need to count the cones that are hit by the cars. And for that you wanted a sheet like this. They are currently using some Excel and then manual things. So I actually played around trying to do this. And this is using some new code that is not in production yet. So as you can see there are a lot of different use cases there with SDAPS and you need to think about how to make it easier to use for the user. You need to think about the different requirements that you have. In some cases you want, let's say you're doing a survey, you want anonymous versus nonanonymous surveys. You might do a patient survey where a patient gets the same questionnaire every month and then it's not anonymous. You also need to know from which month it was later on. There are some technical aspects. You need to think about logistics. If you have only one piece of paper like this one, it's very simple. But consider an examination, for example, with a huge amount of papers, four pages, 10 pages, then if you staple it, it's hard to scan in again because you need to cut off edge or something which might destroy the edge markers that SDAPS is using. There's a lot of things to think about at that point so that everything runs smoothly. This is most to a large extent not even a software issue but an issue of getting all the integration on thinking about how to do it. For example, another thing is error handling, especially in examinations. You need to make sure that the students, if they mark some piece of paper that they can correctly correct their results and then you can go back and you know exactly what happened and you can correct it manually instead of just running it through the computer and then you end up with wrong data and the student will complain that the mark is wrong or you might not even get complained and then you're in trouble. So that's not too easy to solve in general. There's some consideration that you should think about before doing an examination or even a survey on paper. But at the same time, it's nice to... Paper surveys can make a lot of sense in some contexts. So here is the thing, some parts of it are solved by S-Tubs. So what S-Tubs does is for multi-page handling, we have a barcode at the bottom corner and if you look closely it says 0002 which is the second page. So on the first page there's no barcode but on the second page there's a barcode saying which page number is and there is a large random number identifying the survey which is reproducible. So if you use the same Latch-Input file, compile it again, it will result in the same number. But if you do some minor modifications you will get a new number because the software or the areas moved around. So that means that you are reproducible. We have to do it. Then you get user-defined barcodes. So with the example that... For example I'm sending a survey out to a patient every month or so, you need to know which month the piece of paper was from so you can just add this information to there or another example might be lectures where you have 10 lectures and you want to know which lecture is. You can print on the piece of paper directly to ensure that even if you mix up the papers in any way you want you know afterwards which lecture it was from. But for non-unanimous servers you want to be able to mark the paper with a custom barcode that identifies the specific user that you hand it out to. So that is what SSTEPPS provides internally while the center one, it's basically not even used by a SSTEPPS itself because it's not useful for it. It's just there for the user to be used later on. In general the process is that you have to create the questionnaire, you have to then pass it over to the main program which expects all the metadata out of it. Then you are able to create the printable document afterwards after some processing. Then you hand that out, you scan it back in, then obviously you want to hand it over to the main program again to do the optical recognition. The manual error correction depends on what your needs are and obviously data export afterwards. So these steps are different commands here. So we have SSTEPPS, we have a stamp command which does the printable document, we have an add command which simply adds the scan, we have the recognize command which does the recognition step, user interface, CVS export command. And because they change the aspect ratio of the slides is the report command which creates a PDF document. So the setup command that simply compiles the lattice document or takes the oddity file, parses it, reads in the PDF file, extracts all the metadata from there and after it has all the metadata it can recompile the document with the correct information or overlay the corner marks and stuff. So that is what happens and after that, yeah. So considering the metadata and stuff what you want is to create a nice looking document and for that we have the lattice class. This is just a simple example and when you compile the lattice class, the lattice class, this is the example document actually. So what we can see here is that it's relatively simple document. What we do is we create begin questionnaire and questionnaire. This is just a wrapper environment to be able to repeat it so that we can generate like a hundred unique questionnaires with different barcodes from lattice directly. And then it's a simple section and then single mark is just mark question with lower bound upper bound as you can see here. Then you can add a choice question with different choices and it's a metric. It's layouted as a table. The four there means that it is four columns and then we have two choices and another choice for text input. And text boxes are out of skating. So if you look at the page here, this is the bottom of the page and you can see the end of the text box because it scales over the whole page. And that's it. What it comes out is one, the PDF file which is kind of printed but on the other hand all the metadata with the exact position on page of all the check boxes and the questions that you were using so that we can create a report which contains the question text, which contains the information. And also the program has all the data required to analyze the paper. So the next step sometimes depends if you're doing a non-unknown survey is that you still don't have a printable document because you still need to add more barcodes which identify the user, which identify the single piece of paper. So you re-render, you pass that over to S-Tubs and then you can create, reproduce, use the stamp command to create the actual printable file. After that all the processing has to be done, obviously. You have the papers, you add, you scan them. The scan is right now you need to provide a monochromative file, multi-page tiff file, or there are some, I've got some work which means that you can import basically any photo. So it actually works, I've tried it, that you use the cell phone camera, just take a snapshot with your cell phone, the mobile device and pass it over to the import feature which converts it internally to the multi-page tiff, does the 3D transformation and everything. And then you could also use that. But if you're scanning and you don't want to use all this overhead, then you need to have a scanner which outputs a tiff file, monochrome tiff file. And basically the ad command just checks that it's a monochrome tiff file and that the page count is seen and then copies the data over into the project. And the next step is the actual data processing where you run the recognition and it loads the image, it calculates the correction matrix. So what we do here, it extracts the meter data, the barcodes are read, the rotation is done and optical macro recognition happens. So if you consider an image, this is a photo now, it's a black and white image otherwise, obviously. But what happens is that the corner marks are detected, then you can do the desqueuing of the page because actually it's surprising how bad scanners are in printers. The human eye doesn't see it, but it's not quite accurate. So even over the page, you don't only get rotation, you get ensqueues and stuff. We also get that the piece of paper didn't quite move in the same speed while you're scanning or printing. So that means that the checkboxes are actually not quite at the correct position. So what SSTAPS does is that there is an extra correlation there that analyzes the checkbox area just looking for the bounding box of the checkbox or if it's circular for the circle. And then goes back after that to analyze just the center of the checkboxes. There are a couple of characteristics that I've implemented. It's probably not the best. I'm not a computer science guy. I don't know that much about memory data processing. I guess that you could do a lot better. What SSTAPS does is a simple black level check. So it just comes with a number of black pixels in the area, center area. So you can see only the red areas analyzed. The white is just padding to make sure that we don't see the surrounding box. Then another thing is that what I've implemented is a half transformation. So it tries to detect lines and then removes these lines. And the idea is that if a user fills in the box, it's supposed to be considered not marked again. So if I remove the lines from the scan and pretty much nothing is left, then it's probably not filled out completely, which is, let's say, the other way around. If quite a lot is left, then it is filled in. And so it's just a big pin mark. And another thing is that I try to detect continuous white areas to be able to find whether it's filled in just sloppily and you get lots of small white areas. So the coverage is pretty low, but still it's not really a single mark or something. This kind of works. It's not great, but it works mostly so. In my experience, it's fine enough for most use cases in examination. I probably want to think about how exactly you're doing because there you can't really accept much errors. And then we have user interface. It's for menu. It's written in GTK. And it simply shows the document, the scan, and you can hit the checkbox to toggle it and to move it around. And that's all there is to it, really. And obviously, we have that export to CSV and the report. But the thing is with the report, I don't consider SSTUPS to be a statistics tool. However, it's nice to have a PDF report for quick service where you don't really need to be due to complex statistical stuff. So there is some support for statistics. There is some support for report generation. But I don't see a point in adding much more features to SSTUPS with regard to statistics because if someone is actually doing more complicated stuff, they probably want to export the data anyways and then process it further using R or whatever is their preferred way of doing statistics. But you can, for example, export just a subset of data using filters. So you can do some statistics. It works some analysis on the data. So there's a filtering support. It's just a neat feature, but it's also quite hard to explain actually to people. And then we have got some debugging commands, but that's not separate. So I was thinking of that might be actually quite complicated right now. So if we consider, this is just the example document that I have online for the field. So it's just a large PDF document with lots of questions and we chose all the different features. You have mark groups, you have mark line, which is just condensed version of marks. You have a choice questions that we already had, which is the number of choices. And the resulting document. What does it do now? That was correct. Oops. This is the result. So we can do like questions for range questions like this one. We can measure them up into different so that they're more condensed, ask multiple questions at the same time. The choice questions are just multiple choice questions with the possibility of having free form fields which are on the one hand a check box because if something is written it's checked, on the other hand you can get the image data later on if you want to. Then the same thing for choice questions. So this is basically all that I stopped support, but it is enough for most use cases. So it's fine usually. So this would be the... So this is the example document and I can now... What? No. Okay. I thought it's just... So what I can now do is that I'm running the S-Tubs from the development version now, but that doesn't really make much of a difference. So we create the test directory. This is the project directory S-Tubs keeps all the data in. We run the setup test command, which means that we want to test file as an input command and it will compile them. So what happens now is that S-Tubs compiles the document a couple of times, then reads the metadata in, it shows the metadata here, which is what it found, calculates the ID, the unique ID status, are required and then reruns the whole process again so that the printable document is produced in this case. Actually, I'm not quite sure. I think I've modified it in a way that I don't have the printable document now. Let me see. Questionary test? PDF. No, it's printable. It's fine. So what you can see now is that there's no draft. You might have seen before that there was a large draft text on the background so that you wouldn't print it by default. So right now, because it's not a minima survey, we don't identify the user. There's no barcode here. We could simply print this. I have an example scan here. All right. I need to go back here. Let's go. Example. If you have any questions for you to ask at any point, just, if this is an example scan that I have. Okay, right, it's on this screen here. Or actually mixed between the both screens. So I'm just going to use that as an example. So here you can see that it doesn't matter if it's on upside down the scan because as steps find the barcode, that just simply rotates the page. Rather simple to fix the issue. So if I hand this over to S-Taps, it will simply, okay, let's do this. We want the test project which is there. Then we want the example.tiff file. And I forgot the add command, obviously. That's it. It's very quick because it simply does pretty much only copies the file. And now we do the recognize. This takes a little bit. Now we have the data is analyzed and we can have a look at it. So this is it. You can see it. If I simply, I know, make sure that I can see the whole page. I can quickly check whether everything is correct. Oh yeah, that's good. Here it detected the area that the writing is in because the border is both thick. It means that there is text there. That's it. And I'm done. You have reached the first page. Okay, I just went backward, but whatever. And so basically now all I want that is left is a nice looking report, right? That's pretty boring. Oh, there we go. We got a little bit of statistics here. Pretty simple, but it does the job. The text, the free form text fields are simply included as images here so that you can read them later on. And that is all there is to it really. And this uses, this actually creates a lot of document again in this case, but there are different ways of doing it in S steps. So all right, we could also, let's try that. Export. This is actually pretty ugly right now. I think it's data. So it's just a large list of ones and zeros and the ones that, so at the top is the checkbooks. It's the question number encoded there. So one, one is question one, checkbooks one or something. I'm not quite sure I would have expected it to have more. I know the first question is a mark question so it has the integer in there which of the checkbox is checked. So one, one is the first question in the first section and its value is one, three, four depending on what was checked there. Okay, that's it. Right. Any questions or I guess? Probably not. Okay. So all right. So I was thinking of just a little bit about what they did in Aachen. So in examinations you have their different issues. What they decided on doing is making sure that they don't get, they don't have to staple the paper or anything by using a three-folded sheet. So they were limited to four pages. They designed the whole thing in a way that this would be possible and then created some scripts around the whole thing to slice up the images correctly after the scan and combine the four pages into printable document and stuff like that. And then we all, and they also thought quite a bit about the error handling and selection for manual inspection. So this is what their first design looked like which was used for a tutorial. I don't have a design of the later ones. And you can see that basically each question is just the checkboxes. They ask for the formula. They ask for the value that is the result and then they ask for the unit. So basically they just put quite a lot of thought into how to do the questions, how to change the design of the examination so that it would work properly. And they have used it at the beginning of this year and apparently were trying for them. The first, then, but error connection, what they first were trying is that they would go back, they would, sorry, what they first tried to do is that if a student corrected something they were supposed to raise their hand, then if someone would go in and would circle the point where the correction was made here and do the cross and then note it down which piece of paper had to be manually checked later on. They changed that around a bit and then came up with this, with a scheme where the students themselves just go ahead and do the correction. But are supposed to note it down on the first page just so that you know that there is something weird there and then you can explicitly go in. You can filter to just have a look at all the ones that were, this was filled in, the box in the user interface and then you just go through all the pages and look carefully but you can skip all of the ones that are fine right from the beginning so you don't even need to look at those and only look at those if the student seems things that there was a mistake done. And apparently they had two errors with the whole process, two check boxes that were not identified correctly overall where students complained. But then the other thing which was most apparent with the people and with the guy in Princeton, they tried to do very specific layouts and at some point I decided this is, there's no way of going forward with the current lattice code. So what I'm working on right now and I've been working on this off and on for quite a while is a completely rewrite of the lattice class and because the current code is really designed only for very, very simple surveys and only has pretty few layout options. You have basically two question types only. I don't care about single choice, multiple choice which also, which already makes a difference in question design because people do expect that a single choice question is a toggle around checkbox and multiple choice is a square checkbox and things like that because that is what people are learning from the computers. So you really want to mirror these kind of things and it's not designed to do that well. And there are some other aspects like you saw this ESB export. I had the 1.1.1.2 and I didn't know what it was about, right? So what I want is that you can actually change this around later on and you specify in the lattice file what it's called in the export so that you can better do that. So the new code, I've been experimenting a lot with lattice 3 which is really nice. To some extent, probably most of you won't care about that because it's all the background machinery where it's only relevant but I have separated out different support packages. I have a base package which implements a lot of infrastructure. I'm actually afraid that I've over-engineered a little bit but so that you can design the questionnaire properly, you can have sections which are designed slightly differently. You could overwrite some options locally. You could also for examinations if you want the standard answers. You might want to check some boxes and then print it out. And you can do things like that, you can overlay data on top of the checkbox. You can create a text box around normal lattice code. So I've done a lot of work there. So what the support package does is that I edit the context management thing. So what you could, for example, do here is you create a context, context begin, context end and then you enable PDF form creation and you get in this section, in this inside, in between those context commands you get a PDF form elements overlay and you can actually use it correctly in Acrobat Reader to mark things. What would still be needed to be able to post this data somehow to server or something, I guess. It's a variable name, I've tried to find a scheme where you hierarchically can name variables. So if you have a section, you could give the section a variable name then append underscore and then get the question and so forth. Or you could change the checkbox design a bit. And then you have ways of creating your objects, QObjectBegin, QObjectEnd, which is the more important internal name and then this is a choice question. This is going to be written into the metadata so that you have the metadata output and this also means that I separate the metadata output from the actual design, which is a problem when you get really special things like, let's say, graphics as a question. So if you think it wasn't in the examination, they had a graph, a plot, a diagram as the question basically and then different answers. And it doesn't make sense. You can't have a diagram in a metadata in a report. It doesn't make sense to export. So you really want to be able to overwrite the, by default, use the string that the user used, but also have a way of specifying a normal human readable string instead of a graphics, a PDF or whatever. And this is the, I've called it overwrites where you can actually pre-mark checkboxes that have a specific variable name so that, yeah, you could pre-mark them or you could do standard examination results with a feature like that. This is an example document that, well, a test document that I used. So this, as you can see, the blue ones have PDF form elements can be changed. Different ways of overlaying data. There's form data here and this box is simply around the box. And all of this basically works. So actually, S-Tubs can extract the metadata there and can work with the data. Another feature is that it was requested from people. I've talked to a number of people and those were, I think, was Nando. They suggested it. That if you create a matrix of questions, you want a more profile layout options. So what I've tried here, and this is, Henry Menker helped me a lot with this, is that you can actually, by simply changing this one flag in the code, change the layout entirely. And by that, you're very flexible with the questionnaire design because you can focus on what the questionnaire is about and don't have to focus on the layout right away at this point. I still have to do a lot of work with this because I'm not quite sure. This is a matrix layout, right? The table layout. But I'm thinking of optimizing things a lot so that the tables, if you have two tables above each other, for example, that the columns aren't differently sized, I'm not quite sure what kind of features I want to support there. And it's a lot of work to do, unfortunately. But I think it's worth the effort at this point because you want to, yeah, I want to have a nice looking questionnaire, obviously. Yeah, that's basically it for now. I'm done. So if you are interested, there's a homepage, there's a website, there is a mailing list if you can join or IRC. Well, talk to me. Yeah, that's basically it. Thank you. Any questions? Maybe. None? Okay. Oh, I was going to ask about knowledge and explaining what the rules are, or the patterns So the question, I'm not sure if I understood it quite correctly. The question is, could you import data? Or? It's a complex code. If you have an ABC, answer it with how many of the people are answering A? So the question is whether I could ask, ASTEPPS gives me a way of knowing that how many people answered A? If a user answered question A with a certain answer, what else did these people answer there? This is actually something that is possible with the filters because you can create a report where only the answers are included that are, which are, so only the sheets are included with it, which the questionnaires are included where the answer was done. So let's say, let's try this out actually. It should be simple, right? I hope. What do we want? I want projects installed and then this direct show. Let's do the user interface. So let's say we use question one, question one and see whether the question checkbox is checked. So we get this report with everything is good there. And okay, I'm just zooming right now. And this one won't be included. And probably the other ones neither. So I could actually use this. I can use the filter in different programs, right? I could use the filter here. So I could say here, one, one equals, equals one. There you go. We only have two pages and everything else is hidden away. And now if I create a report with that, I get a new file, which is just numbered through here by default, which just includes one in a questionnaire. And that's it. It's a very, very simple way. If you want anything more advanced, you'll need to do the statistics yourself imported somehow. But yeah, it works. Okay, you're welcome. Thank you. Thank you.
 Okay, I'm going to talk a bit about Asstops today, which is a project that, well, basically  it is about optical micro recognition mostly.  We may have seen the pieces of paper in the back that you had, which is a small survey  that they are doing here, and they are using a software which is non-free, which is actually  used by a lot of universities throughout Germany.  What we had, why we started Asstops, is that we needed to read out exactly these forms  at the university.  So, SDAPS, which is an acronym, I'm not going to spill it out, but it probably doesn't make  much sense.  It's really just an optical micro recognition.  It's only about recognizing which checkboxes are marked by the user and then figuring that  out and getting all the data out from the scans that you have done.  However, to a large extent, the problem there is to the questionnaire design.  You want to design the questionnaire in a way that is easy for the user.  You want to quickly create something like this.  You don't want to, I don't know, I've seen tools where you create the questionnaire somewhere  else and then you import it, then you mark the regions that need to be recognized.  And in my opinion, that's very, not very useful.  So what SDAPS does is that you create the questionnaire in Lattech and all the metadata  that is required is exported into a file, which is then picked up later on and can be  analyzed by the program.  In general, it's a Python program with just about 10K lines, so it's relatively small  overall.  And it's designed in a way that is extensible so that you could create own plugins and commands  that build on top of the internal data model that it has.  And yeah, mostly it's a command line utility.  The only point where the user interface comes into play is for doing manual error prediction  because that's where you need a user interface and the rest of this is very much designed  to be scriptable.  So we started the project back in 2008 actually when in Cod3 at the university, it was still  the university of Cod3 at the time, today is the Cod3 Institute of Technology, it was  decided that the evaluation would go into the centralized and we weren't quite happy  with the bylaws that were in place at the time.  So what we decided at the faculty of electrical engineering is that we would boycott the evaluation  at the university.  Now we had an issue because at the same time there was the change to the Bachelor of Master  System here in Germany going on and we really didn't want to lose the data that was coming  in especially for the lower, for semester's bachelor's if there were any changes.  So we had to actually use the, do the data analysis ourselves.  What we ended up doing is that we created a tool which read out these forms in a pretty  quick hack like over the Christmas holidays and then ended up actually evaluating most  of the, let's remove the cell phone, should help, most of the lectures at the faculty  were using our tool and actually in cooperation with the faculty itself.  So that worked very well and after that we realized that we could create such a tool  but obviously we were using these original forms that time so we needed a way of also  creating the format not only reading it in and we first decided to use OpenOffice for  that and later LibreOffice and then even later created the Latin support which is now basically  the main part of doing it.  So since then people have tried, oh sorry, probably should have, let's do this first.  Since then people have used it for a lot of different things or even tried to use it for  different things.  So this is a survey that is being used in the UK by a company called GPTools and they  offer some services there for doctors who need to do a yearly evaluation of their office  for the NHS and what they offer is a very simple form so the doctor or 20 patients or something  the day and send it in and they will create a nice report for it.  There is even an example report online.  They basically take the normal output that SDAPS generates at somewhere statistics data  themselves and that's then it, what they are offering and they are selling the service  there.  Another thing is examinations so both every TH often the physics department there, they  are looking into using that and they have, sorry?  Okay, sure, no worries.  All right, that is a good point.  That should be it.  Okay, thanks for the, so yeah, examination is a point which is actually quite interesting.  I'll come back to that a bit later.  So they have designed a completely new way of asking, formulating the questions for the  examination.  This is an example for the tutorials that they were using but they have used it for the real  examination too and this is an example from Princeton where they have tried to get it  to work and you can actually, if you look closely you can see that SDAPS is not really  designed to handle these cases.  So what they did here is that there is a large box which is just taking out as an image and  then they have placed something else inside to be able to get the data out from after  the scan and then have a look at the image there.  But I've seen, I think that they have also issues that students write outside the field  and stuff like that which makes it a lot harder to handle the data later on.  So yeah, this is different layouts, layout issues that they are having.  They, the general me there, he did a pretty good job trying to hack out the class in a  way, the Latif class, that you could get the layouts to work like this.  I'm working on improving this.  This is another example from Nation Builder.  This is a sheet that was used during the 2014 elections in the US.  This was for the, well, door to door, you go, walk from door to door, you ask questions  and then you just check out the answers there.  And they sold the service also to some people, to some organization and lecture parties I  think that walked around there.  In theory voting, I have had someone doing this ballot at some point.  I'm not sure if they ever used it.  I think that the pirates in Berlin have tried to use SDAPS for a relatively complicated  voting system because it's a lot, it's a pain to count manually.  But I've not, I don't know whether they're actually using it other than trying it out  once.  Just month ago I got the request for someone who wanted to do Autocross, German Autosladung  or something.  And there you need to count the cones that are hit by the cars.  And for that you wanted a sheet like this.  They are currently using some Excel and then manual things.  So I actually played around trying to do this.  And this is using some new code that is not in production yet.  So as you can see there are a lot of different use cases there with SDAPS and you need to  think about how to make it easier to use for the user.  You need to think about the different requirements that you have.  In some cases you want, let's say you're doing a survey, you want anonymous versus nonanonymous  surveys.  You might do a patient survey where a patient gets the same questionnaire every month and  then it's not anonymous.  You also need to know from which month it was later on.  There are some technical aspects.  You need to think about logistics.  If you have only one piece of paper like this one, it's very simple.  But consider an examination, for example, with a huge amount of papers, four pages,  10 pages, then if you staple it, it's hard to scan in again because you need to cut off  edge or something which might destroy the edge markers that SDAPS is using.  There's a lot of things to think about at that point so that everything runs smoothly.  This is most to a large extent not even a software issue but an issue of getting all  the integration on thinking about how to do it.  For example, another thing is error handling, especially in examinations.  You need to make sure that the students, if they mark some piece of paper that they can  correctly correct their results and then you can go back and you know exactly what happened  and you can correct it manually instead of just running it through the computer and then  you end up with wrong data and the student will complain that the mark is wrong or you  might not even get complained and then you're in trouble.  So that's not too easy to solve in general.  There's some consideration that you should think about before doing an examination or  even a survey on paper.  But at the same time, it's nice to...  Paper surveys can make a lot of sense in some contexts.  So here is the thing, some parts of it are solved by S-Tubs.  So what S-Tubs does is for multi-page handling, we have a barcode at the bottom corner and  if you look closely it says 0002 which is the second page.  So on the first page there's no barcode but on the second page there's a barcode saying  which page number is and there is a large random number identifying the survey which  is reproducible.  So if you use the same Latch-Input file, compile it again, it will result in the same number.  But if you do some minor modifications you will get a new number because the software  or the areas moved around.  So that means that you are reproducible.  We have to do it.  Then you get user-defined barcodes.  So with the example that...  For example I'm sending a survey out to a patient every month or so, you need to know  which month the piece of paper was from so you can just add this information to there  or another example might be lectures where you have 10 lectures and you want to know  which lecture is.  You can print on the piece of paper directly to ensure that even if you mix up the papers  in any way you want you know afterwards which lecture it was from.  But for non-unanimous servers you want to be able to mark the paper with a custom barcode  that identifies the specific user that you hand it out to.  So that is what SSTEPPS provides internally while the center one, it's basically not  even used by a SSTEPPS itself because it's not useful for it.  It's just there for the user to be used later on.  In general the process is that you have to create the questionnaire, you have to then  pass it over to the main program which expects all the metadata out of it.  Then you are able to create the printable document afterwards after some processing.  Then you hand that out, you scan it back in, then obviously you want to hand it over to  the main program again to do the optical recognition.  The manual error correction depends on what your needs are and obviously data export afterwards.  So these steps are different commands here.  So we have SSTEPPS, we have a stamp command which does the printable document, we have  an add command which simply adds the scan, we have the recognize command which does the  recognition step, user interface, CVS export command.  And because they change the aspect ratio of the slides is the report command which creates  a PDF document.  So the setup command that simply compiles the lattice document or takes the oddity  file, parses it, reads in the PDF file, extracts all the metadata from there and after it has  all the metadata it can recompile the document with the correct information or overlay the  corner marks and stuff.  So that is what happens and after that, yeah.  So considering the metadata and stuff what you want is to create a nice looking document  and for that we have the lattice class.  This is just a simple example and when you compile the lattice class, the lattice class,  this is the example document actually.  So what we can see here is that it's relatively simple document.  What we do is we create begin questionnaire and questionnaire.  This is just a wrapper environment to be able to repeat it so that we can generate like  a hundred unique questionnaires with different barcodes from lattice directly.  And then it's a simple section and then single mark is just mark question with lower bound  upper bound as you can see here.  Then you can add a choice question with different choices and it's a metric.  It's layouted as a table.  The four there means that it is four columns and then we have two choices and another choice  for text input.  And text boxes are out of skating.  So if you look at the page here, this is the bottom of the page and you can see the end  of the text box because it scales over the whole page.  And that's it.  What it comes out is one, the PDF file which is kind of printed but on the other hand all  the metadata with the exact position on page of all the check boxes and the questions that  you were using so that we can create a report which contains the question text, which contains  the information.  And also the program has all the data required to analyze the paper.  So the next step sometimes depends if you're doing a non-unknown survey is that you still  don't have a printable document because you still need to add more barcodes which identify  the user, which identify the single piece of paper.  So you re-render, you pass that over to S-Tubs and then you can create, reproduce, use the  stamp command to create the actual printable file.  After that all the processing has to be done, obviously.  You have the papers, you add, you scan them.  The scan is right now you need to provide a monochromative file, multi-page tiff file,  or there are some, I've got some work which means that you can import basically any photo.  So it actually works, I've tried it, that you use the cell phone camera, just take a  snapshot with your cell phone, the mobile device and pass it over to the import feature  which converts it internally to the multi-page tiff, does the 3D transformation and everything.  And then you could also use that.  But if you're scanning and you don't want to use all this overhead, then you need to  have a scanner which outputs a tiff file, monochrome tiff file.  And basically the ad command just checks that it's a monochrome tiff file and that the  page count is seen and then copies the data over into the project.  And the next step is the actual data processing where you run the recognition and it loads  the image, it calculates the correction matrix.  So what we do here, it extracts the meter data, the barcodes are read, the rotation  is done and optical macro recognition happens.  So if you consider an image, this is a photo now, it's a black and white image otherwise,  obviously.  But what happens is that the corner marks are detected, then you can do the desqueuing of  the page because actually it's surprising how bad scanners are in printers.  The human eye doesn't see it, but it's not quite accurate.  So even over the page, you don't only get rotation, you get ensqueues and stuff.  We also get that the piece of paper didn't quite move in the same speed while you're  scanning or printing.  So that means that the checkboxes are actually not quite at the correct position.  So what SSTAPS does is that there is an extra correlation there that analyzes the checkbox  area just looking for the bounding box of the checkbox or if it's circular for the  circle.  And then goes back after that to analyze just the center of the checkboxes.  There are a couple of characteristics that I've implemented.  It's probably not the best.  I'm not a computer science guy.  I don't know that much about memory data processing.  I guess that you could do a lot better.  What SSTAPS does is a simple black level check.  So it just comes with a number of black pixels in the area, center area.  So you can see only the red areas analyzed.  The white is just padding to make sure that we don't see the surrounding box.  Then another thing is that what I've implemented is a half transformation.  So it tries to detect lines and then removes these lines.  And the idea is that if a user fills in the box, it's supposed to be considered not marked  again.  So if I remove the lines from the scan and pretty much nothing is left, then it's probably  not filled out completely, which is, let's say, the other way around.  If quite a lot is left, then it is filled in.  And so it's just a big pin mark.  And another thing is that I try to detect continuous white areas to be able to find whether it's  filled in just sloppily and you get lots of small white areas.  So the coverage is pretty low, but still it's not really a single mark or something.  This kind of works.  It's not great, but it works mostly so.  In my experience, it's fine enough for most use cases in examination.  I probably want to think about how exactly you're doing because there you can't really  accept much errors.  And then we have user interface.  It's for menu.  It's written in GTK.  And it simply shows the document, the scan, and you can hit the checkbox to toggle it  and to move it around.  And that's all there is to it, really.  And obviously, we have that export to CSV and the report.  But the thing is with the report, I don't consider SSTUPS to be a statistics tool.  However, it's nice to have a PDF report for quick service where you don't really need  to be due to complex statistical stuff.  So there is some support for statistics.  There is some support for report generation.  But I don't see a point in adding much more features to SSTUPS with regard to statistics  because if someone is actually doing more complicated stuff, they probably want to export  the data anyways and then process it further using R or whatever is their preferred way  of doing statistics.  But you can, for example, export just a subset of data using filters.  So you can do some statistics.  It works some analysis on the data.  So there's a filtering support.  It's just a neat feature, but it's also quite hard to explain actually to people.  And then we have got some debugging commands, but that's not separate.  So I was thinking of that might be actually quite complicated right now.  So if we consider, this is just the example document that I have online for the field.  So it's just a large PDF document with lots of questions and we chose all the different  features.  You have mark groups, you have mark line, which is just condensed version of marks.  You have a choice questions that we already had, which is the number of choices.  And the resulting document.  What does it do now?  That was correct.  Oops.  This is the result.  So we can do like questions for range questions like this one.  We can measure them up into different so that they're more condensed, ask multiple questions  at the same time.  The choice questions are just multiple choice questions with the possibility of having free  form fields which are on the one hand a check box because if something is written it's  checked, on the other hand you can get the image data later on if you want to.  Then the same thing for choice questions.  So this is basically all that I stopped support, but it is enough for most use cases.  So it's fine usually.  So this would be the...  So this is the example document and I can now...  What?  No.  Okay.  I thought it's just...  So what I can now do is that I'm running the S-Tubs from the development version now,  but that doesn't really make much of a difference.  So we create the test directory.  This is the project directory S-Tubs keeps all the data in.  We run the setup test command, which means that we want to test file as an input command  and it will compile them.  So what happens now is that S-Tubs compiles the document a couple of times, then reads  the metadata in, it shows the metadata here, which is what it found, calculates the ID,  the unique ID status, are required and then reruns the whole process again so that the  printable document is produced in this case.  Actually, I'm not quite sure.  I think I've modified it in a way that I don't have the printable document now.  Let me see.  Questionary test?  PDF.  No, it's printable.  It's fine.  So what you can see now is that there's no draft.  You might have seen before that there was a large draft text on the background so that  you wouldn't print it by default.  So right now, because it's not a minima survey, we don't identify the user.  There's no barcode here.  We could simply print this.  I have an example scan here.  All right.  I need to go back here.  Let's go.  Example.  If you have any questions for you to ask at any point, just, if this is an example scan  that I have.  Okay, right, it's on this screen here.  Or actually mixed between the both screens.  So I'm just going to use that as an example.  So here you can see that it doesn't matter if it's on upside down the scan because as  steps find the barcode, that just simply rotates the page.  Rather simple to fix the issue.  So if I hand this over to S-Taps, it will simply, okay, let's do this.  We want the test project which is there.  Then we want the example.tiff file.  And I forgot the add command, obviously.  That's it.  It's very quick because it simply does pretty much only copies the file.  And now we do the recognize.  This takes a little bit.  Now we have the data is analyzed and we can have a look at it.  So this is it.  You can see it.  If I simply, I know, make sure that I can see the whole page.  I can quickly check whether everything is correct.  Oh yeah, that's good.  Here it detected the area that the writing is in because the border is both thick.  It means that there is text there.  That's it.  And I'm done.  You have reached the first page.  Okay, I just went backward, but whatever.  And so basically now all I want that is left is a nice looking report, right?  That's pretty boring.  Oh, there we go.  We got a little bit of statistics here.  Pretty simple, but it does the job.  The text, the free form text fields are simply included as images here so that you can read  them later on.  And that is all there is to it really.  And this uses, this actually creates a lot of document again in this case, but there  are different ways of doing it in S steps.  So all right, we could also, let's try that.  Export.  This is actually pretty ugly right now.  I think it's data.  So it's just a large list of ones and zeros and the ones that, so at the top is the checkbooks.  It's the question number encoded there.  So one, one is question one, checkbooks one or something.  I'm not quite sure I would have expected it to have more.  I know the first question is a mark question so it has the integer in there which of the  checkbox is checked.  So one, one is the first question in the first section and its value is one, three, four  depending on what was checked there.  Okay, that's it.  Right.  Any questions or I guess?  Probably not.  Okay.  So all right.  So I was thinking of just a little bit about what they did in Aachen.  So in examinations you have their different issues.  What they decided on doing is making sure that they don't get, they don't have to staple  the paper or anything by using a three-folded sheet.  So they were limited to four pages.  They designed the whole thing in a way that this would be possible and then created some  scripts around the whole thing to slice up the images correctly after the scan and combine  the four pages into printable document and stuff like that.  And then we all, and they also thought quite a bit about the error handling and selection  for manual inspection.  So this is what their first design looked like which was used for a tutorial.  I don't have a design of the later ones.  And you can see that basically each question is just the checkboxes.  They ask for the formula.  They ask for the value that is the result and then they ask for the unit.  So basically they just put quite a lot of thought into how to do the questions, how  to change the design of the examination so that it would work properly.  And they have used it at the beginning of this year and apparently were trying for them.  The first, then, but error connection, what they first were trying is that they would  go back, they would, sorry, what they first tried to do is that if a student corrected  something they were supposed to raise their hand, then if someone would go in and would  circle the point where the correction was made here and do the cross and then note it  down which piece of paper had to be manually checked later on.  They changed that around a bit and then came up with this, with a scheme where the students  themselves just go ahead and do the correction.  But are supposed to note it down on the first page just so that you know that there is something  weird there and then you can explicitly go in.  You can filter to just have a look at all the ones that were, this was filled in, the  box in the user interface and then you just go through all the pages and look carefully  but you can skip all of the ones that are fine right from the beginning so you don't  even need to look at those and only look at those if the student seems things that there  was a mistake done.  And apparently they had two errors with the whole process, two check boxes that were not  identified correctly overall where students complained.  But then the other thing which was most apparent with the people and with the guy in Princeton,  they tried to do very specific layouts and at some point I decided this is, there's no  way of going forward with the current lattice code.  So what I'm working on right now and I've been working on this off and on for quite  a while is a completely rewrite of the lattice class and because the current code is really  designed only for very, very simple surveys and only has pretty few layout options.  You have basically two question types only.  I don't care about single choice, multiple choice which also, which already makes a difference  in question design because people do expect that a single choice question is a toggle  around checkbox and multiple choice is a square checkbox and things like that because that  is what people are learning from the computers.  So you really want to mirror these kind of things and it's not designed to do that well.  And there are some other aspects like you saw this ESB export.  I had the 1.1.1.2 and I didn't know what it was about, right?  So what I want is that you can actually change this around later on and you specify in the  lattice file what it's called in the export so that you can better do that.  So the new code, I've been experimenting a lot with lattice 3 which is really nice.  To some extent, probably most of you won't care about that because it's all the background  machinery where it's only relevant but I have separated out different support packages.  I have a base package which implements a lot of infrastructure.  I'm actually afraid that I've over-engineered a little bit but so that you can design the  questionnaire properly, you can have sections which are designed slightly differently.  You could overwrite some options locally.  You could also for examinations if you want the standard answers.  You might want to check some boxes and then print it out.  And you can do things like that, you can overlay data on top of the checkbox.  You can create a text box around normal lattice code.  So I've done a lot of work there.  So what the support package does is that I edit the context management thing.  So what you could, for example, do here is you create a context, context begin, context  end and then you enable PDF form creation and you get in this section, in this inside,  in between those context commands you get a PDF form elements overlay and you can actually  use it correctly in Acrobat Reader to mark things.  What would still be needed to be able to post this data somehow to server or something,  I guess.  It's a variable name, I've tried to find a scheme where you hierarchically can name  variables.  So if you have a section, you could give the section a variable name then append underscore  and then get the question and so forth.  Or you could change the checkbox design a bit.  And then you have ways of creating your objects, QObjectBegin, QObjectEnd, which is the more  important internal name and then this is a choice question.  This is going to be written into the metadata so that you have the metadata output and this  also means that I separate the metadata output from the actual design, which is a problem  when you get really special things like, let's say, graphics as a question.  So if you think it wasn't in the examination, they had a graph, a plot, a diagram as the  question basically and then different answers.  And it doesn't make sense.  You can't have a diagram in a metadata in a report.  It doesn't make sense to export.  So you really want to be able to overwrite the, by default, use the string that the user  used, but also have a way of specifying a normal human readable string instead of a graphics,  a PDF or whatever.  And this is the, I've called it overwrites where you can actually pre-mark checkboxes  that have a specific variable name so that, yeah, you could pre-mark them or you could  do standard examination results with a feature like that.  This is an example document that, well, a test document that I used.  So this, as you can see, the blue ones have PDF form elements can be changed.  Different ways of overlaying data.  There's form data here and this box is simply around the box.  And all of this basically works.  So actually, S-Tubs can extract the metadata there and can work with the data.  Another feature is that it was requested from people.  I've talked to a number of people and those were, I think, was Nando.  They suggested it.  That if you create a matrix of questions, you want a more profile layout options.  So what I've tried here, and this is, Henry Menker helped me a lot with this, is that  you can actually, by simply changing this one flag in the code, change the layout entirely.  And by that, you're very flexible with the questionnaire design because you can focus  on what the questionnaire is about and don't have to focus on the layout right away at  this point.  I still have to do a lot of work with this because I'm not quite sure.  This is a matrix layout, right?  The table layout.  But I'm thinking of optimizing things a lot so that the tables, if you have two tables  above each other, for example, that the columns aren't differently sized, I'm not quite sure  what kind of features I want to support there.  And it's a lot of work to do, unfortunately.  But I think it's worth the effort at this point because you want to, yeah, I want to  have a nice looking questionnaire, obviously.  Yeah, that's basically it for now.  I'm done.  So if you are interested, there's a homepage, there's a website, there is a mailing list  if you can join or IRC.  Well, talk to me.  Yeah, that's basically it.  Thank you.  Any questions?  Maybe.  None?  Okay.  Oh, I was going to ask about knowledge and explaining what the rules are, or the patterns  So the question, I'm not sure if I understood it quite correctly.  The question is, could you import data?  Or?  It's a complex code.  If you have an ABC, answer it with how many of the people are answering A?  So the question is whether I could ask, ASTEPPS gives me a way of knowing that how many people  answered A?  If a user answered question A with a certain answer, what else did these people answer there?  This is actually something that is possible with the filters because you can create a  report where only the answers are included that are, which are, so only the sheets are  included with it, which the questionnaires are included where the answer was done.  So let's say, let's try this out actually.  It should be simple, right?  I hope.  What do we want?  I want projects installed and then this direct show.  Let's do the user interface.  So let's say we use question one, question one and see whether the question checkbox is  checked.  So we get this report with everything is good there.  And okay, I'm just zooming right now.  And this one won't be included.  And probably the other ones neither.  So I could actually use this.  I can use the filter in different programs, right?  I could use the filter here.  So I could say here, one, one equals, equals one.  There you go.  We only have two pages and everything else is hidden away.  And now if I create a report with that, I get a new file, which is just numbered through  here by default, which just includes one in a questionnaire.  And that's it.  It's a very, very simple way.  If you want anything more advanced, you'll need to do the statistics yourself imported  somehow.  But yeah, it works.  Okay, you're welcome.  Thank you.  Thank you.
Benjamin_Berg__SDAPS_0_59.png Benjamin_Berg__SDAPS_60_203.png Benjamin_Berg__SDAPS_204_1595.png Benjamin_Berg__SDAPS_1596_5927.png Benjamin_Berg__SDAPS_5928_6515.png Benjamin_Berg__SDAPS_6516_7619.png Benjamin_Berg__SDAPS_7620_10331.png Benjamin_Berg__SDAPS_10332_26579.png Benjamin_Berg__SDAPS_26580_26807.png Benjamin_Berg__SDAPS_26808_31547.png Benjamin_Berg__SDAPS_31548_32471.png Benjamin_Berg__SDAPS_32472_32567.png Benjamin_Berg__SDAPS_32568_32663.png Benjamin_Berg__SDAPS_32664_32759.png Benjamin_Berg__SDAPS_32760_32987.png Benjamin_Berg__SDAPS_32988_33383.png Benjamin_Berg__SDAPS_33384_34439.png Benjamin_Berg__SDAPS_34440_34679.png Benjamin_Berg__SDAPS_34680_35483.png Benjamin_Berg__SDAPS_35484_36011.png Benjamin_Berg__SDAPS_36012_36503.png Benjamin_Berg__SDAPS_36504_37415.png Benjamin_Berg__SDAPS_37416_37559.png Benjamin_Berg__SDAPS_37560_37667.png Benjamin_Berg__SDAPS_37668_37883.png Benjamin_Berg__SDAPS_37884_37955.png Benjamin_Berg__SDAPS_37956_38123.png Benjamin_Berg__SDAPS_38124_39431.png Benjamin_Berg__SDAPS_39432_39695.png Benjamin_Berg__SDAPS_39696_39743.png Benjamin_Berg__SDAPS_39744_39791.png Benjamin_Berg__SDAPS_39792_39839.png Benjamin_Berg__SDAPS_39840_39995.png Benjamin_Berg__SDAPS_39996_40079.png Benjamin_Berg__SDAPS_40080_40439.png Benjamin_Berg__SDAPS_40440_40871.png Benjamin_Berg__SDAPS_40872_41267.png Benjamin_Berg__SDAPS_41268_42755.png Benjamin_Berg__SDAPS_42756_43127.png Benjamin_Berg__SDAPS_43128_62423.png Benjamin_Berg__SDAPS_62424_62651.png Benjamin_Berg__SDAPS_62652_62951.png Benjamin_Berg__SDAPS_62952_63659.png Benjamin_Berg__SDAPS_63660_63755.png Benjamin_Berg__SDAPS_63756_63777.png
Okay, I'm going to talk a bit about Asstops today, which is a project that, well, basically it is about optical micro recognition mostly. We may have seen the pieces of paper in the back that you had, which is a small survey that they are doing here, and they are using a software which is non-free, which is actually used by a lot of universities throughout Germany. What we had, why we started Asstops, is that we needed to read out exactly these forms at the university. So, SDAPS, which is an acronym, I'm not going to spill it out, but it probably doesn't make much sense. It's really just an optical micro recognition. It's only about recognizing which checkboxes are marked by the user and then figuring that out and getting all the data out from the scans that you have done. However, to a large extent, the problem there is to the questionnaire design. You want to design the questionnaire in a way that is easy for the user. You want to quickly create something like this. You don't want to, I don't know, I've seen tools where you create the questionnaire somewhere else and then you import it, then you mark the regions that need to be recognized. And in my opinion, that's very, not very useful. So what SDAPS does is that you create the questionnaire in Lattech and all the metadata that is required is exported into a file, which is then picked up later on and can be analyzed by the program. In general, it's a Python program with just about 10K lines, so it's relatively small overall. And it's designed in a way that is extensible so that you could create own plugins and commands that build on top of the internal data model that it has. And yeah, mostly it's a command line utility. The only point where the user interface comes into play is for doing manual error prediction because that's where you need a user interface and the rest of this is very much designed to be scriptable. So we started the project back in 2008 actually when in Cod3 at the university, it was still the university of Cod3 at the time, today is the Cod3 Institute of Technology, it was decided that the evaluation would go into the centralized and we weren't quite happy with the bylaws that were in place at the time. So what we decided at the faculty of electrical engineering is that we would boycott the evaluation at the university. Now we had an issue because at the same time there was the change to the Bachelor of Master System here in Germany going on and we really didn't want to lose the data that was coming in especially for the lower, for semester's bachelor's if there were any changes. So we had to actually use the, do the data analysis ourselves. What we ended up doing is that we created a tool which read out these forms in a pretty quick hack like over the Christmas holidays and then ended up actually evaluating most of the, let's remove the cell phone, should help, most of the lectures at the faculty were using our tool and actually in cooperation with the faculty itself. So that worked very well and after that we realized that we could create such a tool but obviously we were using these original forms that time so we needed a way of also creating the format not only reading it in and we first decided to use OpenOffice for that and later LibreOffice and then even later created the Latin support which is now basically the main part of doing it. So since then people have tried, oh sorry, probably should have, let's do this first. Since then people have used it for a lot of different things or even tried to use it for different things. So this is a survey that is being used in the UK by a company called GPTools and they offer some services there for doctors who need to do a yearly evaluation of their office for the NHS and what they offer is a very simple form so the doctor or 20 patients or something the day and send it in and they will create a nice report for it. There is even an example report online. They basically take the normal output that SDAPS generates at somewhere statistics data themselves and that's then it, what they are offering and they are selling the service there. Another thing is examinations so both every TH often the physics department there, they are looking into using that and they have, sorry? Okay, sure, no worries. All right, that is a good point. That should be it. Okay, thanks for the, so yeah, examination is a point which is actually quite interesting. I'll come back to that a bit later. So they have designed a completely new way of asking, formulating the questions for the examination. This is an example for the tutorials that they were using but they have used it for the real examination too and this is an example from Princeton where they have tried to get it to work and you can actually, if you look closely you can see that SDAPS is not really designed to handle these cases. So what they did here is that there is a large box which is just taking out as an image and then they have placed something else inside to be able to get the data out from after the scan and then have a look at the image there. But I've seen, I think that they have also issues that students write outside the field and stuff like that which makes it a lot harder to handle the data later on. So yeah, this is different layouts, layout issues that they are having. They, the general me there, he did a pretty good job trying to hack out the class in a way, the Latif class, that you could get the layouts to work like this. I'm working on improving this. This is another example from Nation Builder. This is a sheet that was used during the 2014 elections in the US. This was for the, well, door to door, you go, walk from door to door, you ask questions and then you just check out the answers there. And they sold the service also to some people, to some organization and lecture parties I think that walked around there. In theory voting, I have had someone doing this ballot at some point. I'm not sure if they ever used it. I think that the pirates in Berlin have tried to use SDAPS for a relatively complicated voting system because it's a lot, it's a pain to count manually. But I've not, I don't know whether they're actually using it other than trying it out once. Just month ago I got the request for someone who wanted to do Autocross, German Autosladung or something. And there you need to count the cones that are hit by the cars. And for that you wanted a sheet like this. They are currently using some Excel and then manual things. So I actually played around trying to do this. And this is using some new code that is not in production yet. So as you can see there are a lot of different use cases there with SDAPS and you need to think about how to make it easier to use for the user. You need to think about the different requirements that you have. In some cases you want, let's say you're doing a survey, you want anonymous versus nonanonymous surveys. You might do a patient survey where a patient gets the same questionnaire every month and then it's not anonymous. You also need to know from which month it was later on. There are some technical aspects. You need to think about logistics. If you have only one piece of paper like this one, it's very simple. But consider an examination, for example, with a huge amount of papers, four pages, 10 pages, then if you staple it, it's hard to scan in again because you need to cut off edge or something which might destroy the edge markers that SDAPS is using. There's a lot of things to think about at that point so that everything runs smoothly. This is most to a large extent not even a software issue but an issue of getting all the integration on thinking about how to do it. For example, another thing is error handling, especially in examinations. You need to make sure that the students, if they mark some piece of paper that they can correctly correct their results and then you can go back and you know exactly what happened and you can correct it manually instead of just running it through the computer and then you end up with wrong data and the student will complain that the mark is wrong or you might not even get complained and then you're in trouble. So that's not too easy to solve in general. There's some consideration that you should think about before doing an examination or even a survey on paper. But at the same time, it's nice to... Paper surveys can make a lot of sense in some contexts. So here is the thing, some parts of it are solved by S-Tubs. So what S-Tubs does is for multi-page handling, we have a barcode at the bottom corner and if you look closely it says 0002 which is the second page. So on the first page there's no barcode but on the second page there's a barcode saying which page number is and there is a large random number identifying the survey which is reproducible. So if you use the same Latch-Input file, compile it again, it will result in the same number. But if you do some minor modifications you will get a new number because the software or the areas moved around. So that means that you are reproducible. We have to do it. Then you get user-defined barcodes. So with the example that... For example I'm sending a survey out to a patient every month or so, you need to know which month the piece of paper was from so you can just add this information to there or another example might be lectures where you have 10 lectures and you want to know which lecture is. You can print on the piece of paper directly to ensure that even if you mix up the papers in any way you want you know afterwards which lecture it was from. But for non-unanimous servers you want to be able to mark the paper with a custom barcode that identifies the specific user that you hand it out to. So that is what SSTEPPS provides internally while the center one, it's basically not even used by a SSTEPPS itself because it's not useful for it. It's just there for the user to be used later on. In general the process is that you have to create the questionnaire, you have to then pass it over to the main program which expects all the metadata out of it. Then you are able to create the printable document afterwards after some processing. Then you hand that out, you scan it back in, then obviously you want to hand it over to the main program again to do the optical recognition. The manual error correction depends on what your needs are and obviously data export afterwards. So these steps are different commands here. So we have SSTEPPS, we have a stamp command which does the printable document, we have an add command which simply adds the scan, we have the recognize command which does the recognition step, user interface, CVS export command. And because they change the aspect ratio of the slides is the report command which creates a PDF document. So the setup command that simply compiles the lattice document or takes the oddity file, parses it, reads in the PDF file, extracts all the metadata from there and after it has all the metadata it can recompile the document with the correct information or overlay the corner marks and stuff. So that is what happens and after that, yeah. So considering the metadata and stuff what you want is to create a nice looking document and for that we have the lattice class. This is just a simple example and when you compile the lattice class, the lattice class, this is the example document actually. So what we can see here is that it's relatively simple document. What we do is we create begin questionnaire and questionnaire. This is just a wrapper environment to be able to repeat it so that we can generate like a hundred unique questionnaires with different barcodes from lattice directly. And then it's a simple section and then single mark is just mark question with lower bound upper bound as you can see here. Then you can add a choice question with different choices and it's a metric. It's layouted as a table. The four there means that it is four columns and then we have two choices and another choice for text input. And text boxes are out of skating. So if you look at the page here, this is the bottom of the page and you can see the end of the text box because it scales over the whole page. And that's it. What it comes out is one, the PDF file which is kind of printed but on the other hand all the metadata with the exact position on page of all the check boxes and the questions that you were using so that we can create a report which contains the question text, which contains the information. And also the program has all the data required to analyze the paper. So the next step sometimes depends if you're doing a non-unknown survey is that you still don't have a printable document because you still need to add more barcodes which identify the user, which identify the single piece of paper. So you re-render, you pass that over to S-Tubs and then you can create, reproduce, use the stamp command to create the actual printable file. After that all the processing has to be done, obviously. You have the papers, you add, you scan them. The scan is right now you need to provide a monochromative file, multi-page tiff file, or there are some, I've got some work which means that you can import basically any photo. So it actually works, I've tried it, that you use the cell phone camera, just take a snapshot with your cell phone, the mobile device and pass it over to the import feature which converts it internally to the multi-page tiff, does the 3D transformation and everything. And then you could also use that. But if you're scanning and you don't want to use all this overhead, then you need to have a scanner which outputs a tiff file, monochrome tiff file. And basically the ad command just checks that it's a monochrome tiff file and that the page count is seen and then copies the data over into the project. And the next step is the actual data processing where you run the recognition and it loads the image, it calculates the correction matrix. So what we do here, it extracts the meter data, the barcodes are read, the rotation is done and optical macro recognition happens. So if you consider an image, this is a photo now, it's a black and white image otherwise, obviously. But what happens is that the corner marks are detected, then you can do the desqueuing of the page because actually it's surprising how bad scanners are in printers. The human eye doesn't see it, but it's not quite accurate. So even over the page, you don't only get rotation, you get ensqueues and stuff. We also get that the piece of paper didn't quite move in the same speed while you're scanning or printing. So that means that the checkboxes are actually not quite at the correct position. So what SSTAPS does is that there is an extra correlation there that analyzes the checkbox area just looking for the bounding box of the checkbox or if it's circular for the circle. And then goes back after that to analyze just the center of the checkboxes. There are a couple of characteristics that I've implemented. It's probably not the best. I'm not a computer science guy. I don't know that much about memory data processing. I guess that you could do a lot better. What SSTAPS does is a simple black level check. So it just comes with a number of black pixels in the area, center area. So you can see only the red areas analyzed. The white is just padding to make sure that we don't see the surrounding box. Then another thing is that what I've implemented is a half transformation. So it tries to detect lines and then removes these lines. And the idea is that if a user fills in the box, it's supposed to be considered not marked again. So if I remove the lines from the scan and pretty much nothing is left, then it's probably not filled out completely, which is, let's say, the other way around. If quite a lot is left, then it is filled in. And so it's just a big pin mark. And another thing is that I try to detect continuous white areas to be able to find whether it's filled in just sloppily and you get lots of small white areas. So the coverage is pretty low, but still it's not really a single mark or something. This kind of works. It's not great, but it works mostly so. In my experience, it's fine enough for most use cases in examination. I probably want to think about how exactly you're doing because there you can't really accept much errors. And then we have user interface. It's for menu. It's written in GTK. And it simply shows the document, the scan, and you can hit the checkbox to toggle it and to move it around. And that's all there is to it, really. And obviously, we have that export to CSV and the report. But the thing is with the report, I don't consider SSTUPS to be a statistics tool. However, it's nice to have a PDF report for quick service where you don't really need to be due to complex statistical stuff. So there is some support for statistics. There is some support for report generation. But I don't see a point in adding much more features to SSTUPS with regard to statistics because if someone is actually doing more complicated stuff, they probably want to export the data anyways and then process it further using R or whatever is their preferred way of doing statistics. But you can, for example, export just a subset of data using filters. So you can do some statistics. It works some analysis on the data. So there's a filtering support. It's just a neat feature, but it's also quite hard to explain actually to people. And then we have got some debugging commands, but that's not separate. So I was thinking of that might be actually quite complicated right now. So if we consider, this is just the example document that I have online for the field. So it's just a large PDF document with lots of questions and we chose all the different features. You have mark groups, you have mark line, which is just condensed version of marks. You have a choice questions that we already had, which is the number of choices. And the resulting document. What does it do now? That was correct. Oops. This is the result. So we can do like questions for range questions like this one. We can measure them up into different so that they're more condensed, ask multiple questions at the same time. The choice questions are just multiple choice questions with the possibility of having free form fields which are on the one hand a check box because if something is written it's checked, on the other hand you can get the image data later on if you want to. Then the same thing for choice questions. So this is basically all that I stopped support, but it is enough for most use cases. So it's fine usually. So this would be the... So this is the example document and I can now... What? No. Okay. I thought it's just... So what I can now do is that I'm running the S-Tubs from the development version now, but that doesn't really make much of a difference. So we create the test directory. This is the project directory S-Tubs keeps all the data in. We run the setup test command, which means that we want to test file as an input command and it will compile them. So what happens now is that S-Tubs compiles the document a couple of times, then reads the metadata in, it shows the metadata here, which is what it found, calculates the ID, the unique ID status, are required and then reruns the whole process again so that the printable document is produced in this case. Actually, I'm not quite sure. I think I've modified it in a way that I don't have the printable document now. Let me see. Questionary test? PDF. No, it's printable. It's fine. So what you can see now is that there's no draft. You might have seen before that there was a large draft text on the background so that you wouldn't print it by default. So right now, because it's not a minima survey, we don't identify the user. There's no barcode here. We could simply print this. I have an example scan here. All right. I need to go back here. Let's go. Example. If you have any questions for you to ask at any point, just, if this is an example scan that I have. Okay, right, it's on this screen here. Or actually mixed between the both screens. So I'm just going to use that as an example. So here you can see that it doesn't matter if it's on upside down the scan because as steps find the barcode, that just simply rotates the page. Rather simple to fix the issue. So if I hand this over to S-Taps, it will simply, okay, let's do this. We want the test project which is there. Then we want the example.tiff file. And I forgot the add command, obviously. That's it. It's very quick because it simply does pretty much only copies the file. And now we do the recognize. This takes a little bit. Now we have the data is analyzed and we can have a look at it. So this is it. You can see it. If I simply, I know, make sure that I can see the whole page. I can quickly check whether everything is correct. Oh yeah, that's good. Here it detected the area that the writing is in because the border is both thick. It means that there is text there. That's it. And I'm done. You have reached the first page. Okay, I just went backward, but whatever. And so basically now all I want that is left is a nice looking report, right? That's pretty boring. Oh, there we go. We got a little bit of statistics here. Pretty simple, but it does the job. The text, the free form text fields are simply included as images here so that you can read them later on. And that is all there is to it really. And this uses, this actually creates a lot of document again in this case, but there are different ways of doing it in S steps. So all right, we could also, let's try that. Export. This is actually pretty ugly right now. I think it's data. So it's just a large list of ones and zeros and the ones that, so at the top is the checkbooks. It's the question number encoded there. So one, one is question one, checkbooks one or something. I'm not quite sure I would have expected it to have more. I know the first question is a mark question so it has the integer in there which of the checkbox is checked. So one, one is the first question in the first section and its value is one, three, four depending on what was checked there. Okay, that's it. Right. Any questions or I guess? Probably not. Okay. So all right. So I was thinking of just a little bit about what they did in Aachen. So in examinations you have their different issues. What they decided on doing is making sure that they don't get, they don't have to staple the paper or anything by using a three-folded sheet. So they were limited to four pages. They designed the whole thing in a way that this would be possible and then created some scripts around the whole thing to slice up the images correctly after the scan and combine the four pages into printable document and stuff like that. And then we all, and they also thought quite a bit about the error handling and selection for manual inspection. So this is what their first design looked like which was used for a tutorial. I don't have a design of the later ones. And you can see that basically each question is just the checkboxes. They ask for the formula. They ask for the value that is the result and then they ask for the unit. So basically they just put quite a lot of thought into how to do the questions, how to change the design of the examination so that it would work properly. And they have used it at the beginning of this year and apparently were trying for them. The first, then, but error connection, what they first were trying is that they would go back, they would, sorry, what they first tried to do is that if a student corrected something they were supposed to raise their hand, then if someone would go in and would circle the point where the correction was made here and do the cross and then note it down which piece of paper had to be manually checked later on. They changed that around a bit and then came up with this, with a scheme where the students themselves just go ahead and do the correction. But are supposed to note it down on the first page just so that you know that there is something weird there and then you can explicitly go in. You can filter to just have a look at all the ones that were, this was filled in, the box in the user interface and then you just go through all the pages and look carefully but you can skip all of the ones that are fine right from the beginning so you don't even need to look at those and only look at those if the student seems things that there was a mistake done. And apparently they had two errors with the whole process, two check boxes that were not identified correctly overall where students complained. But then the other thing which was most apparent with the people and with the guy in Princeton, they tried to do very specific layouts and at some point I decided this is, there's no way of going forward with the current lattice code. So what I'm working on right now and I've been working on this off and on for quite a while is a completely rewrite of the lattice class and because the current code is really designed only for very, very simple surveys and only has pretty few layout options. You have basically two question types only. I don't care about single choice, multiple choice which also, which already makes a difference in question design because people do expect that a single choice question is a toggle around checkbox and multiple choice is a square checkbox and things like that because that is what people are learning from the computers. So you really want to mirror these kind of things and it's not designed to do that well. And there are some other aspects like you saw this ESB export. I had the 1.1.1.2 and I didn't know what it was about, right? So what I want is that you can actually change this around later on and you specify in the lattice file what it's called in the export so that you can better do that. So the new code, I've been experimenting a lot with lattice 3 which is really nice. To some extent, probably most of you won't care about that because it's all the background machinery where it's only relevant but I have separated out different support packages. I have a base package which implements a lot of infrastructure. I'm actually afraid that I've over-engineered a little bit but so that you can design the questionnaire properly, you can have sections which are designed slightly differently. You could overwrite some options locally. You could also for examinations if you want the standard answers. You might want to check some boxes and then print it out. And you can do things like that, you can overlay data on top of the checkbox. You can create a text box around normal lattice code. So I've done a lot of work there. So what the support package does is that I edit the context management thing. So what you could, for example, do here is you create a context, context begin, context end and then you enable PDF form creation and you get in this section, in this inside, in between those context commands you get a PDF form elements overlay and you can actually use it correctly in Acrobat Reader to mark things. What would still be needed to be able to post this data somehow to server or something, I guess. It's a variable name, I've tried to find a scheme where you hierarchically can name variables. So if you have a section, you could give the section a variable name then append underscore and then get the question and so forth. Or you could change the checkbox design a bit. And then you have ways of creating your objects, QObjectBegin, QObjectEnd, which is the more important internal name and then this is a choice question. This is going to be written into the metadata so that you have the metadata output and this also means that I separate the metadata output from the actual design, which is a problem when you get really special things like, let's say, graphics as a question. So if you think it wasn't in the examination, they had a graph, a plot, a diagram as the question basically and then different answers. And it doesn't make sense. You can't have a diagram in a metadata in a report. It doesn't make sense to export. So you really want to be able to overwrite the, by default, use the string that the user used, but also have a way of specifying a normal human readable string instead of a graphics, a PDF or whatever. And this is the, I've called it overwrites where you can actually pre-mark checkboxes that have a specific variable name so that, yeah, you could pre-mark them or you could do standard examination results with a feature like that. This is an example document that, well, a test document that I used. So this, as you can see, the blue ones have PDF form elements can be changed. Different ways of overlaying data. There's form data here and this box is simply around the box. And all of this basically works. So actually, S-Tubs can extract the metadata there and can work with the data. Another feature is that it was requested from people. I've talked to a number of people and those were, I think, was Nando. They suggested it. That if you create a matrix of questions, you want a more profile layout options. So what I've tried here, and this is, Henry Menker helped me a lot with this, is that you can actually, by simply changing this one flag in the code, change the layout entirely. And by that, you're very flexible with the questionnaire design because you can focus on what the questionnaire is about and don't have to focus on the layout right away at this point. I still have to do a lot of work with this because I'm not quite sure. This is a matrix layout, right? The table layout. But I'm thinking of optimizing things a lot so that the tables, if you have two tables above each other, for example, that the columns aren't differently sized, I'm not quite sure what kind of features I want to support there. And it's a lot of work to do, unfortunately. But I think it's worth the effort at this point because you want to, yeah, I want to have a nice looking questionnaire, obviously. Yeah, that's basically it for now. I'm done. So if you are interested, there's a homepage, there's a website, there is a mailing list if you can join or IRC. Well, talk to me. Yeah, that's basically it. Thank you. Any questions? Maybe. None? Okay. Oh, I was going to ask about knowledge and explaining what the rules are, or the patterns So the question, I'm not sure if I understood it quite correctly. The question is, could you import data? Or? It's a complex code. If you have an ABC, answer it with how many of the people are answering A? So the question is whether I could ask, ASTEPPS gives me a way of knowing that how many people answered A? If a user answered question A with a certain answer, what else did these people answer there? This is actually something that is possible with the filters because you can create a report where only the answers are included that are, which are, so only the sheets are included with it, which the questionnaires are included where the answer was done. So let's say, let's try this out actually. It should be simple, right? I hope. What do we want? I want projects installed and then this direct show. Let's do the user interface. So let's say we use question one, question one and see whether the question checkbox is checked. So we get this report with everything is good there. And okay, I'm just zooming right now. And this one won't be included. And probably the other ones neither. So I could actually use this. I can use the filter in different programs, right? I could use the filter here. So I could say here, one, one equals, equals one. There you go. We only have two pages and everything else is hidden away. And now if I create a report with that, I get a new file, which is just numbered through here by default, which just includes one in a questionnaire. And that's it. It's a very, very simple way. If you want anything more advanced, you'll need to do the statistics yourself imported somehow. But yeah, it works. Okay, you're welcome. Thank you. Thank you.
sense. It's really just an optical micro figuring that out and getting all the data out can be analyzed by the program. In general, it's questions for the examination. This is an example with SDAPS and you need to think about how to the student will complain that the mark is wrong examination or even a survey on paper. But at the for it. It's just there for the user to be used recognition. The manual error correction depends This is just a simple example and when you questionnaire and questionnaire. This is just a some support for report generation. But I don't imported somehow. But yeah, it works. Okay,