WPS Benchmarking Session
Fenoy, Gérald
With the interrest for OGC Web Processing Service growing, we need to know details about the Open Source Solutions available. Various projects implementing WPS designed individuals from their community to participate in this talk to introduce their project and summarize their key features, they are the following: 52¡North WPS, Constellation SDI, GeoServer, PyWPS, ZOO-Project. For being able to provide good quality results, the tests should be run locally, so a server dedicated to this WPS Benchmarking Session, hosting OSGeoLive environments, is accessible to each team for running test procedures defined through discussions. The results of the given test procedures will be presented during this session and will compare implementations from different aspects: capabilities, compliancy, ressources usage and performance.
 So, good afternoon everybody. I'm Gérald Fanouf from OSGO and I am here to present to you the effort led by OSGO during this year to try to present to you some kind of WPS benchmarking result. So, we, some kind of history about the WPS benchmarking. So it started in Barcelona. I don't know if you remember, but Theodor Forester made some talks in Barcelona and there were some strikes because we were not involved in the test and there were 100% failure for many open source projects. There were even one blog post on Clever Elephant about WPS and about Women Pro Soccer League in United States made by Paul Ramsey when we, he came back at home. Then in 2011, in Denver, there was a WPS shootout made by Jody Garnett. Then after last year in the first four G-nottingham, there was a WPS run-up led by Yakim Shepesky. So there was a past sheriff and this year, I am the sheriff in town. So I am running the WPS benchmarking. So, what our goals and objective is to make conformance testing and performance testing. They did already in the past the interoperability testing. So I am mainly focused on the first two points, so conformance and performance testing. So it's all about server here. So to be able to run this test, we think to ourselves that it can be better to provide an infrastructure for each project for being able to set up their WPS implementation, optimize the WPS implementation if required. So we bought Geolabs company, my company bought this server with this specification that we gave to OSGO. The server will be available until April 2015 because I don't think the WPS benchmarking should stop at any time. It should run all over the year. This way, all the WPS implementation can increase, can announce their implementation and fix some troubles, some issue they can face when looking at the slides. So we provide an OSGO live environment for each project to set up. So obviously, we have some limitation due to the OSGO live DVD. As you know, it's working only on 32-bit environment and we are using shrewted environment. As we don't have any fund to do this work, we simply created shrewted environment, which was far simple to install. So what was the implementation which we are willing to participate? We had a zoo project, obviously, which proposed to run the WPS benchmarking thanks to Yakim, which gave us a sheriff's staff. Then there were the 52 North WPS, PiWPS, which is an OSGO labs, where OSGO zoo project is an OSGO project in incubation still. Geo server and degree are project of OSGO, but degree didn't want to participate. We are really infortinate about this. And then, Constellation SDI from the Geomatis company said they will participate, but as you can see, there is no link to any server, so there is not that much to say about this. They were willing to participate, but at the end, they didn't participate. So for being able to run the test, we provided some shell script which is available on GitHub. As I told you, everything is about conformance and performance testing, so we called the project Cplit testing suite. We mainly use Libxml2 to validate using XML-land tools from command line, and we use Apache benchmark, which was probably a bad choice, but it was a choice we made at the beginning. All this story started in May, earlier this week, and all the slide you are seeing there are the output of the testing suite we published on GitHub. So we will start with the get capabilities test. We will have some graph, so don't take too much care about the graph, but take a look mainly about the number of requests per second. So we will have two kind of tests for the get method, HTTP method, using upper or lower case for parameters, and also the post request, then we will send various kind of requests with one missing parameter or with one wrong value parameter. So as you can see here, in the order of the slide, you will have always the project first. The first slide of result will be about the project, the second will be about 52 North, third will be piWPS, and at the end it will be GeoServer. But for GeoServer, the guy which made the setup yesterday during the lunch told me that he didn't do any optimization, because I asked him when I saw the number of requests per second, I was wondering if they were still willing to participate. So here is the result of Zoo project, so you can see 1,700 requests per second. You have the same result for 52 North WPS with 700 and something requests per second. Then you have for piWPS, 140 requests per second, and for GeoServer, we have six requests per second. So obviously, Andrea, as you told me, as you told earlier that we have to modify the configuration and to announce the number of threads, so they didn't do anything. They did only yesterday. So I had to run all the tests this morning, so I do apologize if some slide has a bit wrong. So then there is the other gate capability request, which is almost the same, okay? So you can see the similar kind of results. Here you have the time which passed and this is the response time. So if you want to send this graph, we didn't include the GeoServer stuff. So the cyan line and the black line are piWPS and you cannot see any more Zoo or 52 North which are downstairs. If you go to the post request, I was surprised to see that piWPS is working pretty faster using post than using, as you can see here, they can handle 1,400 requests per second by when you send the post request gate capabilities, but only 100 and something when you send the KVP request. Anyway, so for Zoo, it was always under 1,500 something requests per second. So you can see also that for the wrong capability request, we are checking both if the response code is valid regarding the WPS specification and we are also checking if the AWS exception report is valid against the specific schema and we are checking if the message contain the correct code and the correct locator even if we saw today thanks to the site test of OJC that we may be wrong saying that those value can be right. So anyway, you can see that we are under 30,000 something for when it failed, 7,000 something for 52 North, 1,007 something for piWPS, you can see that only the HTTP return code is not valid but who cares. Same for GeoServer strangely, I mean when the exception report is returned, the HTTP code should not be 200, it should be 400. If I remember well in the WPS1.0.0 specification, it is right in black and white that it should be returned 501 code when the request parameter were wrong. I'm pretty sure. Okay. Anyway. So here is the result for the gate capability. You can see obviously also the slide on the Internet. You can also have some details on the WPS benchmarking on the Wiki of OSGO because we are doing everything using the OSGO infrastructure and the Wiki. So then we have the describe process. So we first launch the describe process for all identifiers. So let's say we want to have all the describe process of all services available. Then we will ask with only one service identifier and then we will send the same using the post request. As before, we will send the wrong request with one missing parameter and then we need one wrong value. So for describe process, we have a similar result as before. So one thousand and something for zoo. But there is something strange because Geo52 North WPS implementation is not supporting the identifier named all. So that's why it is not, there is no slide here. So you cannot compare this describe process with the next one. You should compare this one with the next one. So this one is five thousand five hundred something and for 52 North it is four thousand seven hundred something. Pi WPS, we found some issue at any time when you run the test script automatically if something failed in the XML validation. You have a reference file where you can see what, where, on which line the issue occurs. This way WPS implementer can fix the issue on their own. So then you, this is for pi WPS. So pi WPS for describe process is only, only six requests per second. But one more time we just run the test today. So maybe it can, they can announce by modifying the configuration of the server or I don't know. Then here we are, we have GeoServer, so three hundred and something describe process request per second. One hundred and something for standard KVP with only one service identifier. So for pi WPS is ninety something. For 52 North it's four thousand and one more time for Zoo project is five thousand and five hundred. I do apologize because I represent the Zoo project and I would like to say that I tried to make the test as fair as possible. But unfortunately we have, by now at least we have the best result. But as I said, we should ask all the team to do a better setup of the WPS implementation. I mean at the beginning of the WPS benchmarking to be one hundred percent honest with you, we was the last. But then we announced the code. We announced the way we set up the server and then finally we finished first. But we need the same test again without any fake result or anything like this. It was not the topic at all. So what to say more? There is a five thousand, let's say six thousand describe process per second for Zoo. Three thousand five something, five hundred something for 52 North, one thousand five hundred for pi WPS and here strangely for Zoo server we have on five thousand request, two thousand six hundred request which failed. But I don't know. You have to take a look I think on the server specification. So maybe I can find something wrong with Zoo. Yeah, I found something wrong with Zoo. In fact, you ask, in fact we already fixed this but we didn't fix the slide just to find something wrong to say about Zoo project. So we return an invalid code and the AWS, there is no AWS exception report because when you send an identifier which is not existing Zoo is still returning describe process, empty describe process not but it's odd, it's wrong. So we have to fix this. It will be fixed in one dot four version hopefully. So you can, all implementers were invited to run the test on their own as it will automatically created the slide. Then we just have a aggregated script to put all the slides together and to build this kind of slides. Here is the result so maybe I should not put pi WPS on those slides, on those graph because it scales so high that we cannot see the other implementation which are here. Then we would like to have some kind of execute request analysis but except GeoServer which had provided services. Let's say the services I spoke about earlier. So I made a list in May about GeoXA CML function, let's say intersection, union, intersect, stuff like that, really simple implementation. And I ask all the participants to develop these services and we are fortunate to have only GeoServer and Zoo project which provided per default I think those services. But we cannot ask also other WPS server implementation to spend too much time as we are wasted so much time on our own already. So anyway, for the execute request you know in WPS that you can execute the request synchronously or asynchronously. So first, the first test was to run using synchronous request. So the first test will be to run buffer, union or intersects by sending gson value inside the post-xml. Then we will embed inside the XML the gml value directly. We in test three we will send the value, we will send the value by reference using an xlink-shref. Then in text five we will send the value as reference as gml using post request, internal post request inside the WPS post request. And then for text six we will do the same as test five but we will request the result as reference. So hopefully one more time I will have bad words to say about Zoo project even if it's already fixed in the SVN. So for executing, so here you will see how nonsense benchmarking can be. So first I told you it's about gson, geogson correct. So we can handle 1,400 and something. Then we have the same service using gml and we will have 64 requests per second. So what is the conclusion? Geogson is far better to read, far easier, faster to read than you reading gml. So it's quite easy to understand. Here we have some failed request, 100 requests on 500. At the beginning we planned to add 5,000 requests but some server failed so we prefer to stop as at 500. In fact, every time you are running the script you can define how much request you want to run and how much concurrent request you want to have. So here you have 64 requests per second and you always have also the result. So let me remember 1, 2, 3. So 3 is when you send the value as reference as gml file. Here you have 47 requests per second so you can understand that when you have to execute it's quite slow. Here for the 1, 2, 3, 4, 5, for test 5, value as reference as gml using post request we can handle 47, so same, 47 same. Here I have 1,800 requests per second. So let me 1, 2, 3, 4, 5, 6, 7. Okay. It depends on the service we called. So here we called union and prior we called buffer because indeed in using the test suite you can specify all the services you want to use and it will run the test all over. So let's say some bad words about zoo. So I will go to the last slide in this section. Here we have bad behavior when you are returning literal data and the user request this literal data to be stored on the server. The reference we return as a wrong rather than having MIME type text plane we are stupidly defining the data type equal the real data type of the literal data. So for the other servers there is no slide as you can see. I told you at the beginning that there were no service implementation available and I have one bad news for your server is that I was not able to fix my test script this morning because in the name scheme which is used by your server you are using gts two point and the name of the function and I am stupidly using said from command line using two point as separator so I used slash but I don't know why when I tried this morning he fell so I just stop. Then you have the same kind of execute using asynchronous request. So here you can notice that we are not using nj and x server anymore. We are using Apache because we have to admit that our fast cgi version is not working when you are asking for asynchronous request because we cannot fork cleanly. So here we have a 400 request per second for the first request 50 request per second 56 so quite slow anyway. Then during the lunch today we went at the table and we saw Luis Bermondez seated there speaking with Jeff about OSGO and trig RATD into OGC portal and so on and so forth. And finally we discussed with Luis and we spoke about this WPS benchmarking effort we were driving and he told us but do you know that there is already a site test available on our site infrastructure? So we should only gone there and we tried to run some compliant testing and sick test. So Zoo was able to so no no for your server and the other one I will have some small note, side note to give. So for Zoo we have 14 successful PASED tests, PIIPS have 18 PASED tests and then suddenly I try with 52 North and I made a bad mistake I use 52 and the degree sign in my session name so I try to log in again and I cannot do any test anymore. So the server and 52 North result are wrong because of the name I gave to my session so I do apologize for this. So if you have any question I'm open to any question as you know this server will be available for until next year so I hope we can have a good time together to work again on WPS benchmarking and we can present it all over the world in all the conference. So I want to thank namely Nicolas Bozon for his participation. Yacine Shepesky obviously George G. De Jesus which I think made the PIIPS setup. Not myself, Jody Garnet, Olivier Nougat, Benjamin Prost and Venkatesh Raghavan which was at the inception of the proposal. And I had the sheriff's star, I should bring it somewhere, it should be somewhere in the room but I cannot find it but I will give the sheriff's star for next year to Benjamin Prost for him to handling the next WPS benchmarking. Thank you very much. I know those tests are not really high quality tests but let's say we tried our best and we didn't know what we can offer better in one morning into half an hour of work this morning. I did the same presentation in Bremen. There were only two implementation available, Zoo and 52 North but we had similar results. But hopefully 52 North fixed many problems in their implementation thanks to this WPS benchmarking. So even if the result or my slide are not that great, the result coming out, this WPS benchmarking are great. So thank you very much. Any question? Just another question. Even if I normally work on Ubuntu and deploy in Ubuntu, I normally don't run Java and Tomcat running shipped from the distribution because they are crap. And we have to install Tomcat and Java from their respective original sites because something in the Debian packaging is wrong. And I was just talking with a user yesterday and he told me, well, I installed the G-server on Ubuntu and it was performing like crap using Java and Tomcat from the repository. And then I installed Java from Oracle and Tomcat from the Tomcat website and it was very fast. So that might be one thing. Can I ask a question to you? Can you do the setup again? I don't know. I'm always this busy but possibly. The other thing, this is instead of question, how hard it is to install your benchmarking on a random machine instead of your server? It's quite easy. It's simply shell script. You need a libxml2 tool and Apache benchmark, which is quite easy. Which I normally use anyways. So yeah. Good. But I would like to say that maybe we should hold an ounce next time because I don't know if you go to the Elidem talk in Bremen, but they used another testing suite rather than Apache benchmark and they can send 1,000 parallel requests. They can simulate 1,000 parallel requests so we can maybe have better results than with Apache benchmark which is limited to the number of parallel requests as it will open too much file. So it is limited by the kernel itself. Yeah. I would like to thank to Gerald actually for the work he did because it was actually the first time that the result of the blue PS benchmarking isn't the presentation itself. But the result of this benchmarking is the test suite itself and he really put lots of effort into it. So one more time. Thank you very much. Thank you. He was the best sheriff in town so far and I believe Benjamin will doing at least as good as Gerald did. One comment to Andreas. Thought so to say, yeah. As you might see, everybody is about that busy. So this might be one of the reasons why some servers don't perform correctly while the response time on the mailing list was so long and so on. But as I said, finally, we have something we can work on. We are, there is like no deadline in front of us. Maybe it was for Europe for the next year. I hope so. And we can continue and I hope there will be some better results for the next time. Yeah. Thank you. Thank you very much.
 So, good afternoon everybody. I'm Gérald Fanouf from OSGO and I am here to present  to you the effort led by OSGO during this year to try to present to you some kind of  WPS benchmarking result. So, we, some kind of history about the WPS benchmarking. So  it started in Barcelona. I don't know if you remember, but Theodor Forester made some  talks in Barcelona and there were some strikes because we were not involved in the test and  there were 100% failure for many open source projects. There were even one blog post on  Clever Elephant about WPS and about Women Pro Soccer League in United States made by  Paul Ramsey when we, he came back at home. Then in 2011, in Denver, there was a WPS shootout  made by Jody Garnett. Then after last year in the first four G-nottingham, there was  a WPS run-up led by Yakim Shepesky. So there was a past sheriff and this year, I am the  sheriff in town. So I am running the WPS benchmarking. So, what our goals and objective is to make  conformance testing and performance testing. They did already in the past the interoperability  testing. So I am mainly focused on the first two points, so conformance and performance  testing. So it's all about server here. So to be able to run this test, we think to ourselves  that it can be better to provide an infrastructure for each project for being able to set up  their WPS implementation, optimize the WPS implementation if required. So we bought  Geolabs company, my company bought this server with this specification that we gave to OSGO.  The server will be available until April 2015 because I don't think the WPS benchmarking  should stop at any time. It should run all over the year. This way, all the WPS implementation  can increase, can announce their implementation and fix some troubles, some issue they can  face when looking at the slides. So we provide an OSGO live environment for each project  to set up. So obviously, we have some limitation due to the OSGO live DVD. As you know, it's  working only on 32-bit environment and we are using shrewted environment. As we don't  have any fund to do this work, we simply created shrewted environment, which was far simple  to install. So what was the implementation which we are willing to participate? We had  a zoo project, obviously, which proposed to run the WPS benchmarking thanks to Yakim,  which gave us a sheriff's staff. Then there were the 52 North WPS, PiWPS, which is an  OSGO labs, where OSGO zoo project is an OSGO project in incubation still. Geo server and  degree are project of OSGO, but degree didn't want to participate. We are really infortinate  about this. And then, Constellation SDI from the Geomatis company said they will participate,  but as you can see, there is no link to any server, so there is not that much to say about  this. They were willing to participate, but at the end, they didn't participate.  So for being able to run the test, we provided some shell script which is available on GitHub.  As I told you, everything is about conformance and performance testing, so we called the  project Cplit testing suite. We mainly use Libxml2 to validate using XML-land tools from  command line, and we use Apache benchmark, which was probably a bad choice, but it was  a choice we made at the beginning. All this story started in May, earlier this week, and  all the slide you are seeing there are the output of the testing suite we published on  GitHub. So we will start with the get capabilities test. We will have some graph, so don't take  too much care about the graph, but take a look mainly about the number of requests per  second. So we will have two kind of tests for the get method, HTTP method, using upper  or lower case for parameters, and also the post request, then we will send various kind  of requests with one missing parameter or with one wrong value parameter. So as you can see  here, in the order of the slide, you will have always the project first. The first slide  of result will be about the project, the second will be about 52 North, third will be piWPS,  and at the end it will be GeoServer. But for GeoServer, the guy which made the setup yesterday  during the lunch told me that he didn't do any optimization, because I asked him when  I saw the number of requests per second, I was wondering if they were still willing  to participate. So here is the result of Zoo project, so you can see 1,700 requests per  second. You have the same result for 52 North WPS with 700 and something requests per second.  Then you have for piWPS, 140 requests per second, and for GeoServer, we have six requests  per second. So obviously, Andrea, as you told me, as you told earlier that we have to modify  the configuration and to announce the number of threads, so they didn't do anything. They  did only yesterday. So I had to run all the tests this morning, so I do apologize if some  slide has a bit wrong. So then there is the other gate capability request, which is almost  the same, okay? So you can see the similar kind of results. Here you have the time which  passed and this is the response time. So if you want to send this graph, we didn't include  the GeoServer stuff. So the cyan line and the black line are piWPS and you cannot see  any more Zoo or 52 North which are downstairs. If you go to the post request, I was surprised  to see that piWPS is working pretty faster using post than using, as you can see here,  they can handle 1,400 requests per second by when you send the post request gate capabilities,  but only 100 and something when you send the KVP request. Anyway, so for Zoo, it was always  under 1,500 something requests per second. So you can see also that for the wrong capability  request, we are checking both if the response code is valid regarding the WPS specification  and we are also checking if the AWS exception report is valid against the specific schema  and we are checking if the message contain the correct code and the correct locator even  if we saw today thanks to the site test of OJC that we may be wrong saying that those  value can be right. So anyway, you can see that we are under 30,000 something for when  it failed, 7,000 something for 52 North, 1,007 something for piWPS, you can see that only  the HTTP return code is not valid but who cares. Same for GeoServer strangely, I mean  when the exception report is returned, the HTTP code should not be 200, it should be  400. If I remember well in the WPS1.0.0 specification, it is right in black and white that it should  be returned 501 code when the request parameter were wrong. I'm pretty sure. Okay. Anyway.  So here is the result for the gate capability. You can see obviously also the slide on the  Internet. You can also have some details on the WPS benchmarking on the Wiki of OSGO because  we are doing everything using the OSGO infrastructure and the Wiki. So then we have the describe  process. So we first launch the describe process for all identifiers. So let's say we want  to have all the describe process of all services available. Then we will ask with only one  service identifier and then we will send the same using the post request. As before, we  will send the wrong request with one missing parameter and then we need one wrong value.  So for describe process, we have a similar result as before. So one thousand and something  for zoo. But there is something strange because Geo52 North WPS implementation is not supporting  the identifier named all. So that's why it is not, there is no slide here. So you cannot  compare this describe process with the next one. You should compare this one with the  next one. So this one is five thousand five hundred something and for 52 North it is four  thousand seven hundred something. Pi WPS, we found some issue at any time when you run  the test script automatically if something failed in the XML validation. You have a reference  file where you can see what, where, on which line the issue occurs. This way WPS implementer  can fix the issue on their own. So then you, this is for pi WPS. So pi WPS for describe  process is only, only six requests per second. But one more time we just run the test today.  So maybe it can, they can announce by modifying the configuration of the server or I don't  know. Then here we are, we have GeoServer, so three hundred and something describe process  request per second. One hundred and something for standard KVP with only one service identifier.  So for pi WPS is ninety something. For 52 North it's four thousand and one more time  for Zoo project is five thousand and five hundred. I do apologize because I represent  the Zoo project and I would like to say that I tried to make the test as fair as possible.  But unfortunately we have, by now at least we have the best result. But as I said, we  should ask all the team to do a better setup of the WPS implementation. I mean at the beginning  of the WPS benchmarking to be one hundred percent honest with you, we was the last.  But then we announced the code. We announced the way we set up the server and then finally  we finished first. But we need the same test again without any fake result or anything  like this. It was not the topic at all. So what to say more? There is a five thousand,  let's say six thousand describe process per second for Zoo. Three thousand five something,  five hundred something for 52 North, one thousand five hundred for pi WPS and here strangely  for Zoo server we have on five thousand request, two thousand six hundred request which failed.  But I don't know. You have to take a look I think on the server specification. So maybe  I can find something wrong with Zoo. Yeah, I found something wrong with Zoo. In fact,  you ask, in fact we already fixed this but we didn't fix the slide just to find something  wrong to say about Zoo project. So we return an invalid code and the AWS, there is no AWS  exception report because when you send an identifier which is not existing Zoo is still  returning describe process, empty describe process not but it's odd, it's wrong. So we  have to fix this. It will be fixed in one dot four version hopefully. So you can, all  implementers were invited to run the test on their own as it will automatically created  the slide. Then we just have a aggregated script to put all the slides together and  to build this kind of slides. Here is the result so maybe I should not put pi WPS on  those slides, on those graph because it scales so high that we cannot see the other implementation  which are here. Then we would like to have some kind of execute request analysis but  except GeoServer which had provided services. Let's say the services I spoke about earlier.  So I made a list in May about GeoXA CML function, let's say intersection, union, intersect,  stuff like that, really simple implementation. And I ask all the participants to develop  these services and we are fortunate to have only GeoServer and Zoo project which provided  per default I think those services. But we cannot ask also other WPS server implementation  to spend too much time as we are wasted so much time on our own already. So anyway, for  the execute request you know in WPS that you can execute the request synchronously or  asynchronously. So first, the first test was to run using synchronous request. So the first  test will be to run buffer, union or intersects by sending gson value inside the post-xml.  Then we will embed inside the XML the gml value directly. We in test three we will send the  value, we will send the value by reference using an xlink-shref. Then in text five we  will send the value as reference as gml using post request, internal post request inside  the WPS post request. And then for text six we will do the same as test five but we will  request the result as reference. So hopefully one more time I will have bad words to say  about Zoo project even if it's already fixed in the SVN. So for executing, so here you  will see how nonsense benchmarking can be. So first I told you it's about gson, geogson  correct. So we can handle 1,400 and something. Then we have the same service using gml and  we will have 64 requests per second. So what is the conclusion? Geogson is far better to  read, far easier, faster to read than you reading gml. So it's quite easy to understand.  Here we have some failed request, 100 requests on 500. At the beginning we planned to add  5,000 requests but some server failed so we prefer to stop as at 500. In fact, every time  you are running the script you can define how much request you want to run and how much  concurrent request you want to have. So here you have 64 requests per second and you always  have also the result. So let me remember 1, 2, 3. So 3 is when you send the value as  reference as gml file. Here you have 47 requests per second so you can understand that when  you have to execute it's quite slow. Here for the 1, 2, 3, 4, 5, for test 5, value as  reference as gml using post request we can handle 47, so same, 47 same. Here I have 1,800  requests per second. So let me 1, 2, 3, 4, 5, 6, 7. Okay. It depends on the service we  called. So here we called union and prior we called buffer because indeed in using the  test suite you can specify all the services you want to use and it will run the test all  over. So let's say some bad words about zoo. So I will go to the last slide in this section.  Here we have bad behavior when you are returning literal data and the user request this literal  data to be stored on the server. The reference we return as a wrong rather than having MIME  type text plane we are stupidly defining the data type equal the real data type of the  literal data. So for the other servers there is no slide as you can see. I told you at  the beginning that there were no service implementation available and I have one bad  news for your server is that I was not able to fix my test script this morning because  in the name scheme which is used by your server you are using gts two point and the name of  the function and I am stupidly using said from command line using two point as separator  so I used slash but I don't know why when I tried this morning he fell so I just stop.  Then you have the same kind of execute using asynchronous request. So here you can notice  that we are not using nj and x server anymore. We are using Apache because we have to admit  that our fast cgi version is not working when you are asking for asynchronous request because  we cannot fork cleanly. So here we have a 400 request per second for the first request  50 request per second 56 so quite slow anyway. Then during the lunch today we went at the  table and we saw Luis Bermondez seated there speaking with Jeff about OSGO and trig RATD  into OGC portal and so on and so forth. And finally we discussed with Luis and we spoke  about this WPS benchmarking effort we were driving and he told us but do you know that  there is already a site test available on our site infrastructure? So we should only  gone there and we tried to run some compliant testing and sick test. So Zoo was able to  so no no for your server and the other one I will have some small note, side note to  give. So for Zoo we have 14 successful PASED tests, PIIPS have 18 PASED tests and then  suddenly I try with 52 North and I made a bad mistake I use 52 and the degree sign in  my session name so I try to log in again and I cannot do any test anymore. So the server  and 52 North result are wrong because of the name I gave to my session so I do apologize  for this. So if you have any question I'm open to any question as you know this server  will be available for until next year so I hope we can have a good time together to work  again on WPS benchmarking and we can present it all over the world in all the conference.  So I want to thank namely Nicolas Bozon for his participation. Yacine Shepesky obviously  George G. De Jesus which I think made the PIIPS setup. Not myself, Jody Garnet, Olivier  Nougat, Benjamin Prost and Venkatesh Raghavan which was at the inception of the proposal.  And I had the sheriff's star, I should bring it somewhere, it should be somewhere in the  room but I cannot find it but I will give the sheriff's star for next year to Benjamin  Prost for him to handling the next WPS benchmarking. Thank you very much. I know those tests are  not really high quality tests but let's say we tried our best and we didn't know what  we can offer better in one morning into half an hour of work this morning. I did the same  presentation in Bremen. There were only two implementation available, Zoo and 52 North  but we had similar results. But hopefully 52 North fixed many problems in their implementation  thanks to this WPS benchmarking. So even if the result or my slide are not that great,  the result coming out, this WPS benchmarking are great. So thank you very much. Any question?  Just another question. Even if I normally work on Ubuntu and deploy in Ubuntu, I normally  don't run Java and Tomcat running shipped from the distribution because they are crap.  And we have to install Tomcat and Java from their respective original sites because something  in the Debian packaging is wrong. And I was just talking with a user yesterday and he  told me, well, I installed the G-server on Ubuntu and it was performing like crap using  Java and Tomcat from the repository. And then I installed Java from Oracle and Tomcat from  the Tomcat website and it was very fast. So that might be one thing.  Can I ask a question to you? Can you do the setup again?  I don't know. I'm always this busy but possibly. The other thing, this is instead of question,  how hard it is to install your benchmarking on a random machine instead of your server?  It's quite easy. It's simply shell script. You need a libxml2 tool and Apache benchmark,  which is quite easy. Which I normally use anyways. So yeah.  Good. But I would like to say that maybe we should  hold an ounce next time because I don't know if you go to the Elidem talk in Bremen, but  they used another testing suite rather than Apache benchmark and they can send 1,000 parallel  requests. They can simulate 1,000 parallel requests so we can maybe have better results  than with Apache benchmark which is limited to the number of parallel requests as it will  open too much file. So it is limited by the kernel itself.  Yeah. I would like to thank to Gerald actually for the work he did because it was actually  the first time that the result of the blue PS benchmarking isn't the presentation itself.  But the result of this benchmarking is the test suite itself and he really put lots of  effort into it. So one more time. Thank you very much.  Thank you. He was the best sheriff in town so far and  I believe Benjamin will doing at least as good as Gerald did.  One comment to Andreas. Thought so to say, yeah. As you might see, everybody is about  that busy. So this might be one of the reasons why some servers don't perform correctly while  the response time on the mailing list was so long and so on. But as I said, finally,  we have something we can work on. We are, there is like no deadline in front of us. Maybe  it was for Europe for the next year. I hope so. And we can continue and I hope there will  be some better results for the next time. Yeah. Thank you.  Thank you very much.
06_Gerald_Fenoy_720p_0_371.png 06_Gerald_Fenoy_720p_372_5291.png 06_Gerald_Fenoy_720p_5292_9887.png 06_Gerald_Fenoy_720p_9888_10103.png 06_Gerald_Fenoy_720p_10104_19547.png 06_Gerald_Fenoy_720p_19548_20471.png 06_Gerald_Fenoy_720p_20472_29915.png 06_Gerald_Fenoy_720p_29916_30827.png 06_Gerald_Fenoy_720p_30828_32879.png 06_Gerald_Fenoy_720p_32880_32927.png 06_Gerald_Fenoy_720p_32928_34115.png 06_Gerald_Fenoy_720p_34116_34487.png 06_Gerald_Fenoy_720p_34488_36251.png 06_Gerald_Fenoy_720p_36252_36467.png 06_Gerald_Fenoy_720p_36468_37631.png 06_Gerald_Fenoy_720p_37632_38291.png 06_Gerald_Fenoy_720p_38292_38351.png 06_Gerald_Fenoy_720p_38352_38411.png 06_Gerald_Fenoy_720p_38412_38459.png 06_Gerald_Fenoy_720p_38460_38507.png 06_Gerald_Fenoy_720p_38508_38843.png 06_Gerald_Fenoy_720p_38844_38891.png 06_Gerald_Fenoy_720p_38892_38975.png 06_Gerald_Fenoy_720p_38976_39047.png 06_Gerald_Fenoy_720p_39048_39155.png 06_Gerald_Fenoy_720p_39156_39215.png 06_Gerald_Fenoy_720p_39216_39443.png 06_Gerald_Fenoy_720p_39444_39451.png
So, good afternoon everybody. I'm Gérald Fanouf from OSGO and I am here to present to you the effort led by OSGO during this year to try to present to you some kind of WPS benchmarking result. So, we, some kind of history about the WPS benchmarking. So it started in Barcelona. I don't know if you remember, but Theodor Forester made some talks in Barcelona and there were some strikes because we were not involved in the test and there were 100% failure for many open source projects. There were even one blog post on Clever Elephant about WPS and about Women Pro Soccer League in United States made by Paul Ramsey when we, he came back at home. Then in 2011, in Denver, there was a WPS shootout made by Jody Garnett. Then after last year in the first four G-nottingham, there was a WPS run-up led by Yakim Shepesky. So there was a past sheriff and this year, I am the sheriff in town. So I am running the WPS benchmarking. So, what our goals and objective is to make conformance testing and performance testing. They did already in the past the interoperability testing. So I am mainly focused on the first two points, so conformance and performance testing. So it's all about server here. So to be able to run this test, we think to ourselves that it can be better to provide an infrastructure for each project for being able to set up their WPS implementation, optimize the WPS implementation if required. So we bought Geolabs company, my company bought this server with this specification that we gave to OSGO. The server will be available until April 2015 because I don't think the WPS benchmarking should stop at any time. It should run all over the year. This way, all the WPS implementation can increase, can announce their implementation and fix some troubles, some issue they can face when looking at the slides. So we provide an OSGO live environment for each project to set up. So obviously, we have some limitation due to the OSGO live DVD. As you know, it's working only on 32-bit environment and we are using shrewted environment. As we don't have any fund to do this work, we simply created shrewted environment, which was far simple to install. So what was the implementation which we are willing to participate? We had a zoo project, obviously, which proposed to run the WPS benchmarking thanks to Yakim, which gave us a sheriff's staff. Then there were the 52 North WPS, PiWPS, which is an OSGO labs, where OSGO zoo project is an OSGO project in incubation still. Geo server and degree are project of OSGO, but degree didn't want to participate. We are really infortinate about this. And then, Constellation SDI from the Geomatis company said they will participate, but as you can see, there is no link to any server, so there is not that much to say about this. They were willing to participate, but at the end, they didn't participate. So for being able to run the test, we provided some shell script which is available on GitHub. As I told you, everything is about conformance and performance testing, so we called the project Cplit testing suite. We mainly use Libxml2 to validate using XML-land tools from command line, and we use Apache benchmark, which was probably a bad choice, but it was a choice we made at the beginning. All this story started in May, earlier this week, and all the slide you are seeing there are the output of the testing suite we published on GitHub. So we will start with the get capabilities test. We will have some graph, so don't take too much care about the graph, but take a look mainly about the number of requests per second. So we will have two kind of tests for the get method, HTTP method, using upper or lower case for parameters, and also the post request, then we will send various kind of requests with one missing parameter or with one wrong value parameter. So as you can see here, in the order of the slide, you will have always the project first. The first slide of result will be about the project, the second will be about 52 North, third will be piWPS, and at the end it will be GeoServer. But for GeoServer, the guy which made the setup yesterday during the lunch told me that he didn't do any optimization, because I asked him when I saw the number of requests per second, I was wondering if they were still willing to participate. So here is the result of Zoo project, so you can see 1,700 requests per second. You have the same result for 52 North WPS with 700 and something requests per second. Then you have for piWPS, 140 requests per second, and for GeoServer, we have six requests per second. So obviously, Andrea, as you told me, as you told earlier that we have to modify the configuration and to announce the number of threads, so they didn't do anything. They did only yesterday. So I had to run all the tests this morning, so I do apologize if some slide has a bit wrong. So then there is the other gate capability request, which is almost the same, okay? So you can see the similar kind of results. Here you have the time which passed and this is the response time. So if you want to send this graph, we didn't include the GeoServer stuff. So the cyan line and the black line are piWPS and you cannot see any more Zoo or 52 North which are downstairs. If you go to the post request, I was surprised to see that piWPS is working pretty faster using post than using, as you can see here, they can handle 1,400 requests per second by when you send the post request gate capabilities, but only 100 and something when you send the KVP request. Anyway, so for Zoo, it was always under 1,500 something requests per second. So you can see also that for the wrong capability request, we are checking both if the response code is valid regarding the WPS specification and we are also checking if the AWS exception report is valid against the specific schema and we are checking if the message contain the correct code and the correct locator even if we saw today thanks to the site test of OJC that we may be wrong saying that those value can be right. So anyway, you can see that we are under 30,000 something for when it failed, 7,000 something for 52 North, 1,007 something for piWPS, you can see that only the HTTP return code is not valid but who cares. Same for GeoServer strangely, I mean when the exception report is returned, the HTTP code should not be 200, it should be 400. If I remember well in the WPS1.0.0 specification, it is right in black and white that it should be returned 501 code when the request parameter were wrong. I'm pretty sure. Okay. Anyway. So here is the result for the gate capability. You can see obviously also the slide on the Internet. You can also have some details on the WPS benchmarking on the Wiki of OSGO because we are doing everything using the OSGO infrastructure and the Wiki. So then we have the describe process. So we first launch the describe process for all identifiers. So let's say we want to have all the describe process of all services available. Then we will ask with only one service identifier and then we will send the same using the post request. As before, we will send the wrong request with one missing parameter and then we need one wrong value. So for describe process, we have a similar result as before. So one thousand and something for zoo. But there is something strange because Geo52 North WPS implementation is not supporting the identifier named all. So that's why it is not, there is no slide here. So you cannot compare this describe process with the next one. You should compare this one with the next one. So this one is five thousand five hundred something and for 52 North it is four thousand seven hundred something. Pi WPS, we found some issue at any time when you run the test script automatically if something failed in the XML validation. You have a reference file where you can see what, where, on which line the issue occurs. This way WPS implementer can fix the issue on their own. So then you, this is for pi WPS. So pi WPS for describe process is only, only six requests per second. But one more time we just run the test today. So maybe it can, they can announce by modifying the configuration of the server or I don't know. Then here we are, we have GeoServer, so three hundred and something describe process request per second. One hundred and something for standard KVP with only one service identifier. So for pi WPS is ninety something. For 52 North it's four thousand and one more time for Zoo project is five thousand and five hundred. I do apologize because I represent the Zoo project and I would like to say that I tried to make the test as fair as possible. But unfortunately we have, by now at least we have the best result. But as I said, we should ask all the team to do a better setup of the WPS implementation. I mean at the beginning of the WPS benchmarking to be one hundred percent honest with you, we was the last. But then we announced the code. We announced the way we set up the server and then finally we finished first. But we need the same test again without any fake result or anything like this. It was not the topic at all. So what to say more? There is a five thousand, let's say six thousand describe process per second for Zoo. Three thousand five something, five hundred something for 52 North, one thousand five hundred for pi WPS and here strangely for Zoo server we have on five thousand request, two thousand six hundred request which failed. But I don't know. You have to take a look I think on the server specification. So maybe I can find something wrong with Zoo. Yeah, I found something wrong with Zoo. In fact, you ask, in fact we already fixed this but we didn't fix the slide just to find something wrong to say about Zoo project. So we return an invalid code and the AWS, there is no AWS exception report because when you send an identifier which is not existing Zoo is still returning describe process, empty describe process not but it's odd, it's wrong. So we have to fix this. It will be fixed in one dot four version hopefully. So you can, all implementers were invited to run the test on their own as it will automatically created the slide. Then we just have a aggregated script to put all the slides together and to build this kind of slides. Here is the result so maybe I should not put pi WPS on those slides, on those graph because it scales so high that we cannot see the other implementation which are here. Then we would like to have some kind of execute request analysis but except GeoServer which had provided services. Let's say the services I spoke about earlier. So I made a list in May about GeoXA CML function, let's say intersection, union, intersect, stuff like that, really simple implementation. And I ask all the participants to develop these services and we are fortunate to have only GeoServer and Zoo project which provided per default I think those services. But we cannot ask also other WPS server implementation to spend too much time as we are wasted so much time on our own already. So anyway, for the execute request you know in WPS that you can execute the request synchronously or asynchronously. So first, the first test was to run using synchronous request. So the first test will be to run buffer, union or intersects by sending gson value inside the post-xml. Then we will embed inside the XML the gml value directly. We in test three we will send the value, we will send the value by reference using an xlink-shref. Then in text five we will send the value as reference as gml using post request, internal post request inside the WPS post request. And then for text six we will do the same as test five but we will request the result as reference. So hopefully one more time I will have bad words to say about Zoo project even if it's already fixed in the SVN. So for executing, so here you will see how nonsense benchmarking can be. So first I told you it's about gson, geogson correct. So we can handle 1,400 and something. Then we have the same service using gml and we will have 64 requests per second. So what is the conclusion? Geogson is far better to read, far easier, faster to read than you reading gml. So it's quite easy to understand. Here we have some failed request, 100 requests on 500. At the beginning we planned to add 5,000 requests but some server failed so we prefer to stop as at 500. In fact, every time you are running the script you can define how much request you want to run and how much concurrent request you want to have. So here you have 64 requests per second and you always have also the result. So let me remember 1, 2, 3. So 3 is when you send the value as reference as gml file. Here you have 47 requests per second so you can understand that when you have to execute it's quite slow. Here for the 1, 2, 3, 4, 5, for test 5, value as reference as gml using post request we can handle 47, so same, 47 same. Here I have 1,800 requests per second. So let me 1, 2, 3, 4, 5, 6, 7. Okay. It depends on the service we called. So here we called union and prior we called buffer because indeed in using the test suite you can specify all the services you want to use and it will run the test all over. So let's say some bad words about zoo. So I will go to the last slide in this section. Here we have bad behavior when you are returning literal data and the user request this literal data to be stored on the server. The reference we return as a wrong rather than having MIME type text plane we are stupidly defining the data type equal the real data type of the literal data. So for the other servers there is no slide as you can see. I told you at the beginning that there were no service implementation available and I have one bad news for your server is that I was not able to fix my test script this morning because in the name scheme which is used by your server you are using gts two point and the name of the function and I am stupidly using said from command line using two point as separator so I used slash but I don't know why when I tried this morning he fell so I just stop. Then you have the same kind of execute using asynchronous request. So here you can notice that we are not using nj and x server anymore. We are using Apache because we have to admit that our fast cgi version is not working when you are asking for asynchronous request because we cannot fork cleanly. So here we have a 400 request per second for the first request 50 request per second 56 so quite slow anyway. Then during the lunch today we went at the table and we saw Luis Bermondez seated there speaking with Jeff about OSGO and trig RATD into OGC portal and so on and so forth. And finally we discussed with Luis and we spoke about this WPS benchmarking effort we were driving and he told us but do you know that there is already a site test available on our site infrastructure? So we should only gone there and we tried to run some compliant testing and sick test. So Zoo was able to so no no for your server and the other one I will have some small note, side note to give. So for Zoo we have 14 successful PASED tests, PIIPS have 18 PASED tests and then suddenly I try with 52 North and I made a bad mistake I use 52 and the degree sign in my session name so I try to log in again and I cannot do any test anymore. So the server and 52 North result are wrong because of the name I gave to my session so I do apologize for this. So if you have any question I'm open to any question as you know this server will be available for until next year so I hope we can have a good time together to work again on WPS benchmarking and we can present it all over the world in all the conference. So I want to thank namely Nicolas Bozon for his participation. Yacine Shepesky obviously George G. De Jesus which I think made the PIIPS setup. Not myself, Jody Garnet, Olivier Nougat, Benjamin Prost and Venkatesh Raghavan which was at the inception of the proposal. And I had the sheriff's star, I should bring it somewhere, it should be somewhere in the room but I cannot find it but I will give the sheriff's star for next year to Benjamin Prost for him to handling the next WPS benchmarking. Thank you very much. I know those tests are not really high quality tests but let's say we tried our best and we didn't know what we can offer better in one morning into half an hour of work this morning. I did the same presentation in Bremen. There were only two implementation available, Zoo and 52 North but we had similar results. But hopefully 52 North fixed many problems in their implementation thanks to this WPS benchmarking. So even if the result or my slide are not that great, the result coming out, this WPS benchmarking are great. So thank you very much. Any question? Just another question. Even if I normally work on Ubuntu and deploy in Ubuntu, I normally don't run Java and Tomcat running shipped from the distribution because they are crap. And we have to install Tomcat and Java from their respective original sites because something in the Debian packaging is wrong. And I was just talking with a user yesterday and he told me, well, I installed the G-server on Ubuntu and it was performing like crap using Java and Tomcat from the repository. And then I installed Java from Oracle and Tomcat from the Tomcat website and it was very fast. So that might be one thing. Can I ask a question to you? Can you do the setup again? I don't know. I'm always this busy but possibly. The other thing, this is instead of question, how hard it is to install your benchmarking on a random machine instead of your server? It's quite easy. It's simply shell script. You need a libxml2 tool and Apache benchmark, which is quite easy. Which I normally use anyways. So yeah. Good. But I would like to say that maybe we should hold an ounce next time because I don't know if you go to the Elidem talk in Bremen, but they used another testing suite rather than Apache benchmark and they can send 1,000 parallel requests. They can simulate 1,000 parallel requests so we can maybe have better results than with Apache benchmark which is limited to the number of parallel requests as it will open too much file. So it is limited by the kernel itself. Yeah. I would like to thank to Gerald actually for the work he did because it was actually the first time that the result of the blue PS benchmarking isn't the presentation itself. But the result of this benchmarking is the test suite itself and he really put lots of effort into it. So one more time. Thank you very much. Thank you. He was the best sheriff in town so far and I believe Benjamin will doing at least as good as Gerald did. One comment to Andreas. Thought so to say, yeah. As you might see, everybody is about that busy. So this might be one of the reasons why some servers don't perform correctly while the response time on the mailing list was so long and so on. But as I said, finally, we have something we can work on. We are, there is like no deadline in front of us. Maybe it was for Europe for the next year. I hope so. And we can continue and I hope there will be some better results for the next time. Yeah. Thank you. Thank you very much.
result. So, we, some kind of history about the and performance testing. So it's all about server ourselves that it can be better to provide an infrastructure for each project for being able to want to participate. We are really infortinate about this. And then, Constellation SDI from the get capabilities test. We will have some graph, first. The first slide of result will be about had to run all the tests this morning, so I do is right in black and white that it should be also have some details on the WPS benchmarking on missing parameter and then we need one wrong should ask all the team to do a better setup of test available on our site infrastructure? So we of the name I gave to my session so I do problems in their implementation thanks to this they are crap. And we have to install Tomcat and