Test Software On Emulated Hardware In Containers... In The Cloud
Parker, Sean A.
Schroeder, Paul
Modernizing the traveler information systems of an international railway and transportation company, including the modernization and renewal of traveler facing devices at the train stations. For a variety of devices ranging from 20-year-old x86 PC104 based embedded systems up to modern 64bit multi-core systems, a Buildroot based Linux system, and a custom application stack is being developed. In this talk, we will show how we use a fully automated CI pipeline to build our custom application components resulting in deployable Linux disk images. These images are then containerized and deployed on our Kubernetes cluster. Using Qemu in our containers allows us to simulate external hardware normally connected through serial interfaces and is the basis for automated tests. - presentation of our technology stack. - multi-stage Linux disk image creation in the CI. - running embedded device images in a Kubernetes cluster. - automated testing of simulated embedded systems.
 Why not? This is surprisingly crowded. We were originally told that there might be like one, two, three people. Of course a little bit more. Okay. Hey, this is Sean Parker and I'm Paul Schridder and we're here and tell you something about the existing software on emulated hardware in containers, in the cloud. And I hope you all like that. We think it's quite fancy. Yeah. Will you tell you something about the project? Why we're doing this? It's high level architecture. It's high level architecture. The test permit where our stuff is located in the test permit. About the target devices and the challenges related to them. A little demo and in the end Q&A and recap if there's still time left for. So we're working for an international train company and it's not about cargo. It's about travel information like the thing below. You might all have experienced the delays and so like five minutes delay on the display and five minutes later it's 10 minutes delay and 10 minutes later it's 15 minutes delay. And the reason for that is the federated decentralized system of servers serving all these displays and if a train did not arrive in the visible area of this server, the operator doesn't know when it will arrive. And so it's just starting to tie five minutes and five minutes later if the train still not arrived, he's increasing that. So the goal of our project is to centralize that. We have a partner project in another city which is doing this consolidation of information. If you have conflicting sources and conflicting information which source we trust more and this is a tough problem on its own but it's thankfully not ours. And we're more focused on devices itself, the management of them and the rendering for the different display types. And this is where I hand over to Sean. Yeah, so I mean Paul already mentioned that like the old system was a federated system where depending on where the information display is or perhaps where the train is, the information might not propagate properly like to the end station. If you have a train starting, I mean in, I don't know, it's like eight hours early, like in a totally different part of the country, the information displays are currently with our client perhaps not timely updated because yeah, the flow of information is just not giving that information. So this now is basically a simplified abstract version of what we're doing. So basically Paul just mentioned the system that is basically at our other project consolidating the data and they're planning basically delays and updating the information continuously. So in the background we have something like a, well, a Reviton Hube broker where you can basically subscribe to a certain train track and so forth and then you can basically get the information and you continuously update it. Sounds easy, it's actually quite complex and I mean it's improving from year to year. What we are then doing, I mean the other project is basically the external domain up there. So there's single source of truth block of travel information, that's basically where all the information goes in. So train delays, when the train start, when the train arrived, scheduling and all that kind of stuff. And it's also basically interfacing in the background with information systems from other countries or other operators. So that's where all the information goes in. And our project is basically interfacing with that and you might see at train tracks different devices, basically voice devices. So basically if a voice announcement comes, that's from one of those devices. If you have this display, LCD of TFT, the old ones and the new ones, that's one of those devices. And they might still have a variety of sensors. So you could think of our system a little bit like Netflix for travel information. So basically we have a big Microsoft architecture in the background and depending on demand, I mean you can imagine, during the night there's not that much going on. I mean there are cargo trains, there are night trains, but really rather during the rush hour we have like most of the information going through our system. So it's growing. It's really pretty much like a sine wave over the day and falling down in the night. Yeah, indeed. So what we are focusing on, I mean the spec end block without getting too much in the nitty gritty parts is already complex enough. So that's the Netflix bit. And then still we have quite a lot of different devices we have to interface with. So those devices are quite often more paid by public money so you can't just replace them. So you have to basically retrofit them with your operating system. That's what we are doing because I mean there was no clean standard basically in the past. For instance, we had a lot of different device types with different interfaces and that of course doesn't make it that easy to integrate with them. So we are building a new operating system and hopefully it will work quite well in the future. So that's what this talk is about in its course. So this little OS box, we are building the operating system based on build route. Who knows build route? Quite a few, good. So it's basically a toolbox really in an abstract way. It's something you can use in order to build a Linux operating system and customize it to your needs. Yeah, we are doing that but that's of course quite difficult because testing software for hardware with different configurations it gets complex quite quickly due to commentatorics. Cool. Yeah, so who of you has seen a test permit so far? So probably you have experienced an enterprise domain somewhere else or you have made some nice PowerPoint slides about that to convince your manager that it might be reasonable to write a few tests. Yes, that's a good idea. So integration tests and unit tests that usually happens with software but at some point you have to integrate with the real world and that's usually where things blow up and you actually want to find bugs and problems as early as possible because I mean in the beginning if you have a unit test, right, you find a bug it's just like you fix something and recompile and it's done and it's good and you're happy but if the client finds the bug it's like you're getting called, operate, Ops is unhappy, you have to roll out everything, you might have to wait quite a long time in order to get your fix out and it's just not making me as a developer happy because basically my development flow is broken but yeah, long story short, you just want to find bugs as early as possible and that's what we're doing with those operating systems which we put in containers in order to run them in the cloud because then we can do that at scale more automated in the CI instead of flashing it or automatically flashing it to a limited set of devices and testing it manually because there we are basically limited to what we have hardware-wise and you have to be there physically. Cool. The stuff we want to present you fits in this column of the test pyramid and the target devices are pretty... there's a huge number of them, I guess 6,000 at least. A lot of different types, most of them are... the old ones like the LCDs over there are PC-1004s, the form factor but they vary in CPU, in RAM size, in CPU types in the interfaces you have these LCDs and the TFTs which are better TVs but they are powerful quad cores with gigabytes of RAM and the outputs are different, we have different boot mechanisms like some devices only support UEFI older ones only support MBR and our goal is to have one image that runs on every single device this is not possible, we have a variety but still we want to have it reduced and not extended to have it still maintainable and testable. There is different... this is just the devices itself, there's different external peripherals like it's called Zughalz-Senzone, train stop sensors let's call it like this the displays are sometimes controlled via serial protocols UARTs or RS485 some sensors are connected via I2S-E sometimes there's special translational hardware which presents itself to the system as a graphics card and the whole thing gets translated into pixels down to the LCDs so proprietary old legacy stuff, no source and there's sometimes a lot of reverse engineering involved and because some of the suppliers of these devices are not a bankrupt or they're being bought by a different company and they're not willing to cooperate basically helping us it's like a closed source software so why should we help you? because in the future you might do everything yourself sometimes they even don't have to source themselves anymore sometimes they go down in the basement and have a look in some file folders and real paper and stuff it's quite interesting, sometimes you log on to those old machines and you have a look at the history and people jumping from one machine to another and all of this is fixed but because they don't know yeah so that's something we really want to fix because that's something you can, well you probably know yourself that's something that is not maintainable so we really want to push a standardised interface so it will be maintainable and we'll make us, I'm a devops guy and use more developer guy but that will make me personally really happy if that works yeah, the standard workflow of the suppliers was have one deviant image, put their software stack on it put it in the device fiddle around till it works and then never touch it again and what we're trying to do is like automated updates in the field and on 6000 devices with thin network lines with sometimes only 10kb bandwidth you really don't want to push that often an image through it especially if it's like 200, 300 megabytes so we would like to test the stuff before we roll it out and make sure that it's working like we wanted to so the problem now arises that we have like a huge hardware variety and how can we simulate that hardware in combination with the software we want to run it on and we're trying to use QEMO the QEMO is pretty awesome because it can, you can really find details, specify how the hardware should behave you can select different CPUs, you can connect different serial devices in different options like is it connected by an ISA bus or a USB and you can change the details of the DMI table which is like a specific part of memory on the hardware motherboard itself and we heavily rely on that to identify the different devices so in order that we know what services we have to start and which not, so on a TFT you need, we use there an electron to display stuff and on an LCD we use like custom written controller service which is communicating with the LCD displays so yeah, we choose QEMO and it's open source so this is the container how it looks like it's not what we're using on the project but it basically reflects the state of the composition you have the outer container, it's an alpine whereas QEMO is installed you have an inner QEMO running which gets the disk image via an volume mount and some ports are exposed for example the SSH to remotely control it in the future we would like to get rid of that and do everything by AMQP commands but this is still far future Does everybody know what AMQP is? Basically the protocol used by RevitonQ which is our middleware basically, message broker so the interesting stuff for QEMO is the DEVTTYS0 which gets exposed on a socket 9000 and can interact with external hardware in this case with a mocked hardware and the control port of the mocked hardware is also exposed we have a noVNC which lets us see how the boot process is but really the yellow stuff is only optional it's not really necessary for testing the image itself So it's quite nice, so you know FarnC it's basically HTML FarnC, VNC, Applet basically so you can browse to port 6080 and basically see what is going on on that machine and you don't need one of those applications so it's tightly coupled in there so what we have been doing at our project we first did that actually with graphical applications with a noFarnC and a container so we were running up to 600 of those devices and you could just click a link with a Unic identifier of the simulated device and you could see what's going on on one of those devices so that's really helpful if you want to stress test your system with real devices especially the back end, not the devices itself to generate load on the back end yeah, cool okay, what you're going to see is something like this setup it's not much different but the mocked, you see like the mocking part is deleted the TTYS0 port is just like exposed by the Docker container so we can interact directly so we kind of doing the manual mocking yeah, so what we're going to show you basically how that thing works we're going to do that unfortunately locally because we can't interface with our client's CI CD system at this point because we didn't get approval but that's fine I mean, we didn't get an answer so far so that's why we're showing you it basically and then we're telling you how we're doing that currently in the CI CD context of our project and how we're deploying that okay, day more time um, I guess if I, I think if I didn't have a mirror it would be better okay Windows P Windows P ah, mirror yeah, give me a second ah, okay perfect so there's the Docker file which builds like the container you've seen so far there's this QEMO disk image I was trying something, not so interesting we have a make file which should make it easy to do stuff right now but we can make runCocker so we're still attached to the container which boots up right now so we have a little delay between the QEMO start and the NoVNC start because if you start them right after each other sometimes NoVNC says it can't connect to the VNC port of QEMO so detail is not so important um the next interesting thing is yeah we can have a look at the booting container which looks like this so it's, this is what you would see if you would start QEMO manually and you could interact but I guess here it's disabled um we can now connect to the container, to the running session inside um and the service running inside runs in a T-Mux so I can attach to the output again but on boot it starts as root so we have to sudo T-Mux we can ls first, we see a session running um we can attach to this, to the session with this name so there we see that the HTTP server started and the sensor handler started on the the FTTY as 0 with the given board rate um so now we can, now we have a look at the at the service we want to test which is on local port 8000 which looks like this, so far it hasn't received anything I'll make this a little smaller that you see okay and then we can send some data I have no time we're not waiting for a response Q0 um connect to local host and the exposed port and okay okay let's try this so you see the sensor received some raw bytes uh decoded them, stripped them, passed them to int and if we so thanks um if we plug something in which is like non-passable as an int, we see that an error happens and test scenario would we put garbage in and see if it's got parsed by the program itself so we see it didn't change so what we're doing with sensor data perhaps is we all our devices basically have an exporter endpoint basically so Prometheus is basically scraping matrix from that system and well it then basically and Prometheus digests all those matrix and we can do later on something without quite interesting to see in the morning where the sun is going up or how bright it is or whatever but it's also for diagnostics quite interesting so I just go back a little um too comfortable we so we push some data in here it got processed by the custom service and got exposed on the 888 port again so this is like a round trip through the software to test yeah so since it's Dockerized you can put it on Kubernetes and put it in the CI and run tests on it, do it automatically yeah so what we're doing we have our project still help charts which is the configuration layer you basically tie around one of those Docker containers and then you can basically mount the news field or OS image you want to put in there then basically run it and you're also through Kubernetes can tag it, put all the metadata into it and run it at scale so it's quite helpful for us at least because now we can basically show the client what we're doing and what kind of different configurations, it's retestable which is usually not a case with hardware, I mean probably you have to try replicating things with hardware it just takes a lot of time as you see it's not silver bullet because the special hardware is really hard to mock special graphic cards you can't really mock in QNU and this is not feasible but the standard cases are doable and it's not possible to have a sample of every hardware configuration in the lab, first we would need like a warehouse to store them all, on the other hand you would have to plug in like the images or like deploy them via updates but still and you're not really controlling the input on the test, you're just testing that it's doing something but you don't know if it's doing the right thing for example, if you can't control the temperature sensor and so this approach makes it kind of possible to get rid of all these problems and not all of them for sure but get rid of some of these problems and make the workflow a little more streamlined Cool, that was it basically so we're open for questions I guess we have plenty of time for questions Oh, go for it Do we have a microphone? I'll just ask a couple of questions You mentioned that you guys have a lot of powerful legacy machines so these are quad-core machines This to me sounds like a bad method of making Did you guys ever run into any problems with that? Did you know of any of these machines that were connected in the past? Please repeat the question The question was if these old legacy systems were hacked, if that's correct Yeah, yes That's a good question Was it even possible to install custom OSes on these machines? Most of these legacy systems typically don't allow you to do a lot of stuff with them So do you guys have like another part of hardware already installed on the stations that you can do anything with? We actually have a lot of those devices shipped to our office so we have a quite big laboratory where we can test quite a certain range of models but office is quite stuffed with it so I mean that's our bottleneck We can test a few of those devices live at the stations but from a security standpoint of view we try to limit the access to the network for once because of past happenings and security is also a reason why we're doing all of this so we can basically close down the operating systems The workflow how the suppliers deployed a new image was like a guy was going there opening the device, putting a CF card in there with the new image and then running it around and so if that was a question When you guys actually developed these mocks for like hardware you basically do it by experimenting with the lab and then coding the behavior that you see so you're actually just reverse engineering? Yes, the mocks are getting developed by reverse engineering sometimes there's a spec but usually it's wide as a door and every vendor made its own solution to that You first Have you considered LinuxKit as an option to build operating systems? There was a... if we have considered LinuxKit, I guess so, the decisions we made before I came but we have good architects I guess they did so but I can't tell you why the picked build route I think because you can choose pretty specific wards, driver wards, core utility you want to have in there and if you really need it and I guess that was the reason for it Back in the time build route basically seemed like a good option so when the project was started I was not quite sure what the variety of hardware is out there so with of course you test against what you know or you basically build your architecture against what you might expect and at that time we had a certain amount of knowledge and build route was basically fitting for that I mean it's a long running project so we haven't started like yesterday I was interested in the options If you are interested just drop us a note and we have the code basically in the public repository so we are probably going to publish that, is that possible? Cool, yeah, so you can have a look at it So you talked about using Helen to deploy things and you showed us the developer experience, it was really nice, do you also have some CI of the internet so I should push things up or is it very much aimed at the developer? It's like our project is completely built in the CI so we have a GitLab CI system and as soon as you push basically through the hook basically it then builds all the code and it deploys automatically So you were talking about exposing those ports and then just you'll get the thing that runs and you'll get a run event point at those ports or this GitLab, it does that thing so at the moment it actually just deploys it so the lifecycle of the deployment is there independent from the run time of the CI or deployment basically and we haven't integrated that, this one so far because the requirement was we just want to see what is running on there and if it's crashing then you can see over time quite well in Kubernetes through logs, metrics and so forth Another thing, compiling build route is like a huge deal it takes really long even in the CI even with heavy machines so we reduced it down to one and a half hours I guess, like with the whole toolchain but still you want to have it you can't wait, click a button and wait for it to compile there If we build separate images for the target hardware and the QEMO no, we don't want to do that we want to test the real image which goes into the field not a specific image for QEMO We are having some kind of variety for some in-house monitoring but this is not going ever in the field down there We are using GitLab CI for integration and you are building build route and you are trying to time the caching mechanism between builds or from scratch If we are using caching for the build route part between runs or stages Yes we are caching the toolchain so usually build route builds the toolchain as well, especially if it has a blank system but what things would really speed up is the cc cache but we didn't get it working till now but this would be a really boost I guess we could make it in half an hour if we do so Someone else? I guess then we are spot on Thank you very much for your time
 Why not?  This is surprisingly crowded.  We were originally told that there might be like one, two, three people.  Of course a little bit more.  Okay.  Hey, this is Sean Parker and I'm Paul Schridder and we're here and tell you something about  the existing software on emulated hardware in containers, in the cloud.  And I hope you all like that.  We think it's quite fancy.  Yeah.  Will you tell you something about the project?  Why we're doing this?  It's high level architecture.  It's high level architecture.  The test permit where our stuff is located in the test permit.  About the target devices and the challenges related to them.  A little demo and in the end Q&A and recap if there's still time left for.  So we're working for an international train company and it's not about cargo.  It's about travel information like the thing below.  You might all have experienced the delays and so like five minutes delay on the display  and five minutes later it's 10 minutes delay and 10 minutes later it's 15 minutes delay.  And the reason for that is the federated decentralized system of servers serving all these displays  and if a train did not arrive in the visible area of this server,  the operator doesn't know when it will arrive.  And so it's just starting to tie five minutes and five minutes later if the train still not arrived,  he's increasing that.  So the goal of our project is to centralize that.  We have a partner project in another city which is doing this consolidation of information.  If you have conflicting sources and conflicting information which source we trust more  and this is a tough problem on its own but it's thankfully not ours.  And we're more focused on devices itself, the management of them and the rendering for the different display types.  And this is where I hand over to Sean.  Yeah, so I mean Paul already mentioned that like the old system was a federated system  where depending on where the information display is or perhaps where the train is,  the information might not propagate properly like to the end station.  If you have a train starting, I mean in, I don't know, it's like eight hours early,  like in a totally different part of the country,  the information displays are currently with our client perhaps not timely updated  because yeah, the flow of information is just not giving that information.  So this now is basically a simplified abstract version of what we're doing.  So basically Paul just mentioned the system that is basically at our other project consolidating the data  and they're planning basically delays and updating the information continuously.  So in the background we have something like a, well, a Reviton Hube broker  where you can basically subscribe to a certain train track and so forth  and then you can basically get the information and you continuously update it.  Sounds easy, it's actually quite complex and I mean it's improving from year to year.  What we are then doing, I mean the other project is basically the external domain up there.  So there's single source of truth block of travel information,  that's basically where all the information goes in.  So train delays, when the train start, when the train arrived, scheduling and all that kind of stuff.  And it's also basically interfacing in the background with information systems  from other countries or other operators.  So that's where all the information goes in.  And our project is basically interfacing with that  and you might see at train tracks different devices, basically voice devices.  So basically if a voice announcement comes, that's from one of those devices.  If you have this display, LCD of TFT, the old ones and the new ones, that's one of those devices.  And they might still have a variety of sensors.  So you could think of our system a little bit like Netflix for travel information.  So basically we have a big Microsoft architecture in the background  and depending on demand, I mean you can imagine,  during the night there's not that much going on.  I mean there are cargo trains, there are night trains,  but really rather during the rush hour we have like most of the information going through our system.  So it's growing.  It's really pretty much like a sine wave over the day and falling down in the night.  Yeah, indeed.  So what we are focusing on, I mean the spec end block  without getting too much in the nitty gritty parts is already complex enough.  So that's the Netflix bit.  And then still we have quite a lot of different devices we have to interface with.  So those devices are quite often more paid by public money  so you can't just replace them.  So you have to basically retrofit them with your operating system.  That's what we are doing because I mean there was no clean standard basically in the past.  For instance, we had a lot of different device types with different interfaces  and that of course doesn't make it that easy to integrate with them.  So we are building a new operating system and hopefully it will work quite well in the future.  So that's what this talk is about in its course.  So this little OS box, we are building the operating system based on build route.  Who knows build route?  Quite a few, good.  So it's basically a toolbox really in an abstract way.  It's something you can use in order to build a Linux operating system  and customize it to your needs.  Yeah, we are doing that but that's of course quite difficult  because testing software for hardware with different configurations  it gets complex quite quickly due to commentatorics.  Cool.  Yeah, so who of you has seen a test permit so far?  So probably you have experienced an enterprise domain somewhere else  or you have made some nice PowerPoint slides about that to convince your manager  that it might be reasonable to write a few tests.  Yes, that's a good idea.  So integration tests and unit tests that usually happens with software  but at some point you have to integrate with the real world  and that's usually where things blow up  and you actually want to find bugs and problems as early as possible  because I mean in the beginning if you have a unit test, right,  you find a bug it's just like you fix something and recompile  and it's done and it's good and you're happy  but if the client finds the bug it's like you're getting called,  operate, Ops is unhappy, you have to roll out everything,  you might have to wait quite a long time in order to get your fix out  and it's just not making me as a developer happy  because basically my development flow is broken  but yeah, long story short, you just want to find bugs as early as possible  and that's what we're doing with those operating systems  which we put in containers in order to run them in the cloud  because then we can do that at scale more automated in the CI  instead of flashing it or automatically flashing it to a limited set of devices  and testing it manually  because there we are basically limited to what we have hardware-wise  and you have to be there physically.  Cool.  The stuff we want to present you  fits in this column of the test pyramid  and the target devices are pretty...  there's a huge number of them, I guess 6,000 at least.  A lot of different types, most of them are...  the old ones like the LCDs over there are PC-1004s, the form factor  but they vary in CPU, in RAM size, in CPU types  in the interfaces you have these LCDs and the TFTs  which are better TVs  but they are powerful quad cores with gigabytes of RAM  and the outputs are different, we have different boot mechanisms  like some devices only support UEFI  older ones only support MBR  and our goal is to have one image that runs on every single device  this is not possible, we have a variety  but still we want to have it reduced and not extended  to have it still maintainable and testable.  There is different...  this is just the devices itself, there's different external peripherals  like it's called Zughalz-Senzone, train stop sensors  let's call it like this  the displays are sometimes controlled via serial protocols  UARTs or RS485  some sensors are connected via I2S-E  sometimes there's special translational hardware  which presents itself to the system as a graphics card  and the whole thing gets translated into pixels down to the LCDs  so proprietary old legacy stuff, no source  and there's sometimes a lot of reverse engineering involved  and because some of the suppliers of these devices are not a bankrupt  or they're being bought by a different company  and they're not willing to cooperate  basically helping us  it's like a closed source software  so why should we help you?  because in the future you might do everything yourself  sometimes they even don't have to source themselves anymore  sometimes they go down in the basement  and have a look in some file folders  and real paper and stuff  it's quite interesting, sometimes you log on to those old machines  and you have a look at the history  and people jumping from one machine to another  and all of this is fixed but because they don't know  yeah  so that's something we really want to fix  because that's something you can, well you probably know yourself  that's something that is not maintainable  so we really want to push a standardised interface  so it will be maintainable  and we'll make us, I'm a devops guy and use more developer guy  but that will make me personally really happy if that works  yeah, the standard workflow of the suppliers was  have one deviant image, put their software stack on it  put it in the device  fiddle around till it works and then never touch it again  and what we're trying to do is like  automated updates in the field  and on 6000 devices with thin network lines  with sometimes only 10kb bandwidth  you really don't want to push that often an image through it  especially if it's like 200, 300 megabytes  so we would like to test the stuff before we roll it out  and make sure that it's working like we wanted to  so the problem now arises that we have like  a huge hardware variety and how can we  simulate that hardware in combination with the software we want to run it on  and we're trying to use QEMO  the QEMO is pretty awesome because it can, you can  really find details, specify how the hardware should behave  you can select different CPUs, you can connect  different serial devices in different options  like is it connected by an ISA bus or a USB  and you can change the details of the DMI table  which is like a specific part of memory on the hardware motherboard itself  and we heavily rely on that to identify the different devices  so in order that we know what services we have to start  and which not, so on a TFT you need, we use there  an electron to display stuff and on an LCD we use like custom written  controller service which is communicating with the LCD displays  so yeah, we choose QEMO and it's open source  so this is the container how it looks like  it's not what we're using on the project but it basically  reflects the state of the composition  you have the outer container, it's an alpine whereas QEMO is installed  you have an inner QEMO running which gets the disk image  via an volume mount and some ports are exposed  for example the SSH to remotely control it  in the future we would like to get rid of that and do everything by  AMQP commands but this is still far future  Does everybody know what AMQP is?  Basically the protocol used by RevitonQ which is our middleware  basically, message broker  so the interesting stuff for QEMO is the DEVTTYS0  which gets exposed on a socket 9000  and can interact with external hardware in this case with a mocked hardware  and the control port of the mocked hardware is also exposed  we have a noVNC which lets us see how the boot process is  but really the yellow stuff is only optional  it's not really necessary for testing the image itself  So it's quite nice, so you know FarnC  it's basically HTML FarnC, VNC, Applet basically  so you can browse to port 6080 and basically see what is going on  on that machine and you don't need one of those applications  so it's tightly coupled in there  so what we have been doing at our project  we first did that actually with graphical applications  with a noFarnC and a container  so we were running up to 600 of those devices  and you could just click a link with a Unic identifier of the simulated device  and you could see what's going on on one of those devices  so that's really helpful if you want to stress test your system with real devices  especially the back end, not the devices itself to generate load on the back end  yeah, cool  okay, what you're going to see is something like this setup  it's not much different but the mocked, you see like the mocking part is deleted  the TTYS0 port is just like exposed by the Docker container  so we can interact directly  so we kind of doing the manual mocking  yeah, so what we're going to show you basically how that thing works  we're going to do that unfortunately locally  because we can't interface with our client's CI CD system at this point  because we didn't get approval but that's fine  I mean, we didn't get an answer so far  so that's why we're showing you it basically  and then we're telling you how we're doing that currently in the CI CD context of our project  and how we're deploying that  okay, day more time  um, I guess if I, I think if I didn't have a mirror it would be better  okay  Windows P  Windows P  ah, mirror  yeah, give me a second  ah, okay  perfect  so there's the Docker file which builds like the container you've seen so far  there's this QEMO disk image  I was trying something, not so interesting  we have a make file which should make it easy to do stuff right now  but we can make runCocker  so we're still attached to the container which boots up right now  so we have a little delay between the QEMO start and the NoVNC start  because if you start them right after each other  sometimes NoVNC says it can't connect to the VNC port of QEMO  so detail is not so important  um  the next interesting thing is  yeah  we can have a look at the booting container  which looks like this  so it's, this is what you would see if you would start QEMO manually  and you could interact but I guess here it's disabled  um  we can now connect to the container, to the running session inside  um  and the service running inside runs in a T-Mux  so I can attach to the output again  but on boot it starts as root  so we have to sudo T-Mux  we can ls first, we see a session running  um we can attach to this, to the session with this name  so there we see that the HTTP server started  and the sensor handler started on the  the FTTY as 0 with the given board rate  um so now we can, now we have a look at the  at the service we want to test  which is on local port 8000  which looks like this, so far it hasn't received anything  I'll make this a little smaller that you see  okay and then we can send some data  I have no time  we're not waiting for a response  Q0  um connect to local host  and the exposed port  and  okay  okay  let's try this  so you see the sensor received some raw bytes  uh decoded them, stripped them, passed them to int  and if we  so  thanks  um  if we plug something in which is like  non-passable as an int, we see that an error happens  and test scenario would we put garbage in  and see if it's got parsed by the program itself  so we see it didn't change  so what we're doing with  sensor data perhaps is we all our devices basically have  an exporter endpoint basically  so Prometheus is basically scraping matrix from that system  and well it then basically  and Prometheus digests all those matrix and we can do later on something without  quite interesting to see in the morning where the sun is going up  or how bright it is or whatever  but it's also for diagnostics quite interesting  so I just go back a little  um  too comfortable  we  so we push some data in here  it got processed by the custom  service and got exposed on the 888 port again  so this is like a round trip  through the software to test  yeah so since it's Dockerized  you can put it on Kubernetes and put it in the  CI and run tests on it, do it automatically  yeah so what we're doing  we have our project still help charts  which is the configuration layer you basically tie around one of those  Docker containers and then you can basically mount the  news field or OS image you want to put in there  then basically run it and you're also through Kubernetes can  tag it, put all the metadata into it and run it at scale  so it's quite helpful for us at least  because now we can basically show the client what we're doing and what kind of different  configurations, it's retestable which is usually not  a case with hardware, I mean probably  you have to try replicating things with hardware  it just takes a lot of time  as you see it's not silver bullet because  the special hardware is really hard to mock  special graphic cards you can't really mock in QNU  and this is not feasible but the standard cases  are doable and  it's not possible to have a sample of every hardware configuration  in the lab, first we would need like a warehouse to store them  all, on the other hand you would have to plug in like the images  or like deploy them via updates but still  and you're not really controlling  the input on the test, you're just testing that it's doing something but  you don't know if it's doing the right thing for example, if you can't  control the temperature sensor  and so this approach makes it kind of possible  to get rid of all these problems and  not all of them for sure but get rid of some of these problems  and make the workflow a little more  streamlined  Cool, that was it basically so we're open for questions  I guess we have plenty of time for questions  Oh, go for it  Do we have a microphone?  I'll just ask a couple of questions  You mentioned that you guys have a lot of powerful legacy machines  so these are quad-core machines  This to me sounds like a bad method of making  Did you guys ever run into any problems with that?  Did you know of any of these machines that were connected in the past?  Please repeat the question  The question was if these old legacy systems  were hacked, if that's correct  Yeah, yes  That's a good question  Was it even possible to install custom OSes on these machines?  Most of these legacy systems typically don't allow you to do a lot of stuff with them  So do you guys have like another part of hardware  already installed on the stations that you can do anything with?  We actually have a lot of those  devices shipped to our office so we have a quite big laboratory where we can  test quite a certain range of models  but office is quite stuffed with it so I mean that's our bottleneck  We can test a few of those devices live at the  stations but from a security standpoint of view we try to limit  the access to the network  for once because of past happenings  and security is also a reason why we're doing all of this  so we can basically close down the operating systems  The workflow how the suppliers  deployed a new image was like a guy was going there  opening the device, putting a CF card in there with the new image  and then running it around and so if that was a question  When you guys actually developed these mocks for like  hardware you basically do it by experimenting with the  lab and then coding the behavior that you see  so you're actually just reverse engineering?  Yes, the mocks are getting developed by reverse engineering  sometimes there's a spec but usually it's wide as a door  and every vendor made its own solution to that  You first  Have you considered LinuxKit as an option to build  operating systems?  There was a... if we have considered  LinuxKit, I guess so, the decisions we made before  I came but we have good architects  I guess they did so but I can't tell you why the picked build route  I think because you can choose pretty specific wards, driver  wards, core utility you want to have in there  and if you really need it and I guess that was the reason for it  Back in the time build route basically seemed like a good option  so when the project was started I was not quite sure what the  variety of hardware is out there so with  of course you test against what you know or you basically build your architecture  against what you might expect and at that time we had  a certain amount of knowledge and build route was basically fitting for that  I mean it's a long running project so we  haven't started like yesterday  I was interested in the options  If you are interested just drop us a note  and we have the code basically in the public  repository so we are probably going to publish  that, is that possible? Cool, yeah, so you can have a look at it  So you talked about using Helen  to deploy things and you showed us the developer  experience, it was really nice, do you also have some  CI of the internet so I should push things up  or is it very much aimed at the developer?  It's like our project is completely built in the CI so we have  a GitLab CI system and as soon as you push basically  through the hook basically it then builds all the code  and it deploys automatically  So you were talking about exposing those ports and then just  you'll get the thing that runs and you'll get a run event point at those  ports or this GitLab, it does that thing  so at the moment it actually just deploys it  so the lifecycle of the deployment is there independent from  the run time of the CI  or deployment basically  and we haven't integrated that, this one so far because the requirement  was we just want to see what is running on there and if it's crashing  then you can see over time quite well in Kubernetes  through logs, metrics and so forth  Another thing, compiling build route is like a huge deal  it takes really long even in the CI even with heavy machines  so we reduced it down  to one and a half hours I guess, like with the whole toolchain  but still you want to have it  you can't wait, click a button and wait for  it to compile  there  If we build separate images for the target hardware and the QEMO  no, we don't want to do that  we want to test the real image which goes into the field  not a specific image for QEMO  We are having some kind of variety for  some in-house monitoring but this is not going ever in the field  down there  We are using GitLab CI for integration  and you are building build route and you are trying to time  the caching mechanism between builds or from scratch  If we are using caching for the build route part  between runs or stages  Yes  we are caching the toolchain  so usually build route builds the toolchain  as well, especially if it has a blank system  but what things would really speed up  is the cc cache but we didn't get it working  till now but this would be a really boost  I guess we could make it in half an hour if we do so  Someone else?  I guess then we are spot on  Thank you very much for your time
test_software_on_emulated_hardware_in_containers_in_the_cloud_0_143.png test_software_on_emulated_hardware_in_containers_in_the_cloud_144_947.png test_software_on_emulated_hardware_in_containers_in_the_cloud_948_1703.png test_software_on_emulated_hardware_in_containers_in_the_cloud_1704_10091.png test_software_on_emulated_hardware_in_containers_in_the_cloud_10092_19403.png test_software_on_emulated_hardware_in_containers_in_the_cloud_19404_21119.png test_software_on_emulated_hardware_in_containers_in_the_cloud_21120_26303.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26304_26627.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26628_26699.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26700_26759.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26760_26867.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26868_26927.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26928_26975.png test_software_on_emulated_hardware_in_containers_in_the_cloud_26976_27023.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27024_27071.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27072_27275.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27276_27383.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27384_27443.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27444_27599.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27600_27647.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27648_27767.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27768_27839.png test_software_on_emulated_hardware_in_containers_in_the_cloud_27840_28007.png test_software_on_emulated_hardware_in_containers_in_the_cloud_28008_28055.png test_software_on_emulated_hardware_in_containers_in_the_cloud_28056_28259.png test_software_on_emulated_hardware_in_containers_in_the_cloud_28260_28859.png test_software_on_emulated_hardware_in_containers_in_the_cloud_28860_28907.png test_software_on_emulated_hardware_in_containers_in_the_cloud_28908_29075.png test_software_on_emulated_hardware_in_containers_in_the_cloud_29076_29423.png test_software_on_emulated_hardware_in_containers_in_the_cloud_29424_29531.png test_software_on_emulated_hardware_in_containers_in_the_cloud_29532_29723.png test_software_on_emulated_hardware_in_containers_in_the_cloud_29724_30059.png test_software_on_emulated_hardware_in_containers_in_the_cloud_30060_30179.png test_software_on_emulated_hardware_in_containers_in_the_cloud_30180_31415.png test_software_on_emulated_hardware_in_containers_in_the_cloud_31416_31499.png test_software_on_emulated_hardware_in_containers_in_the_cloud_31500_31547.png test_software_on_emulated_hardware_in_containers_in_the_cloud_31548_31619.png test_software_on_emulated_hardware_in_containers_in_the_cloud_31620_31943.png test_software_on_emulated_hardware_in_containers_in_the_cloud_31944_32051.png test_software_on_emulated_hardware_in_containers_in_the_cloud_32052_32231.png test_software_on_emulated_hardware_in_containers_in_the_cloud_32232_32867.png test_software_on_emulated_hardware_in_containers_in_the_cloud_32868_33107.png test_software_on_emulated_hardware_in_containers_in_the_cloud_33108_34139.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34140_34235.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34236_34283.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34284_34355.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34356_34403.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34404_34475.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34476_34715.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34716_34943.png test_software_on_emulated_hardware_in_containers_in_the_cloud_34944_35759.png test_software_on_emulated_hardware_in_containers_in_the_cloud_35760_50984.png
Why not? This is surprisingly crowded. We were originally told that there might be like one, two, three people. Of course a little bit more. Okay. Hey, this is Sean Parker and I'm Paul Schridder and we're here and tell you something about the existing software on emulated hardware in containers, in the cloud. And I hope you all like that. We think it's quite fancy. Yeah. Will you tell you something about the project? Why we're doing this? It's high level architecture. It's high level architecture. The test permit where our stuff is located in the test permit. About the target devices and the challenges related to them. A little demo and in the end Q&A and recap if there's still time left for. So we're working for an international train company and it's not about cargo. It's about travel information like the thing below. You might all have experienced the delays and so like five minutes delay on the display and five minutes later it's 10 minutes delay and 10 minutes later it's 15 minutes delay. And the reason for that is the federated decentralized system of servers serving all these displays and if a train did not arrive in the visible area of this server, the operator doesn't know when it will arrive. And so it's just starting to tie five minutes and five minutes later if the train still not arrived, he's increasing that. So the goal of our project is to centralize that. We have a partner project in another city which is doing this consolidation of information. If you have conflicting sources and conflicting information which source we trust more and this is a tough problem on its own but it's thankfully not ours. And we're more focused on devices itself, the management of them and the rendering for the different display types. And this is where I hand over to Sean. Yeah, so I mean Paul already mentioned that like the old system was a federated system where depending on where the information display is or perhaps where the train is, the information might not propagate properly like to the end station. If you have a train starting, I mean in, I don't know, it's like eight hours early, like in a totally different part of the country, the information displays are currently with our client perhaps not timely updated because yeah, the flow of information is just not giving that information. So this now is basically a simplified abstract version of what we're doing. So basically Paul just mentioned the system that is basically at our other project consolidating the data and they're planning basically delays and updating the information continuously. So in the background we have something like a, well, a Reviton Hube broker where you can basically subscribe to a certain train track and so forth and then you can basically get the information and you continuously update it. Sounds easy, it's actually quite complex and I mean it's improving from year to year. What we are then doing, I mean the other project is basically the external domain up there. So there's single source of truth block of travel information, that's basically where all the information goes in. So train delays, when the train start, when the train arrived, scheduling and all that kind of stuff. And it's also basically interfacing in the background with information systems from other countries or other operators. So that's where all the information goes in. And our project is basically interfacing with that and you might see at train tracks different devices, basically voice devices. So basically if a voice announcement comes, that's from one of those devices. If you have this display, LCD of TFT, the old ones and the new ones, that's one of those devices. And they might still have a variety of sensors. So you could think of our system a little bit like Netflix for travel information. So basically we have a big Microsoft architecture in the background and depending on demand, I mean you can imagine, during the night there's not that much going on. I mean there are cargo trains, there are night trains, but really rather during the rush hour we have like most of the information going through our system. So it's growing. It's really pretty much like a sine wave over the day and falling down in the night. Yeah, indeed. So what we are focusing on, I mean the spec end block without getting too much in the nitty gritty parts is already complex enough. So that's the Netflix bit. And then still we have quite a lot of different devices we have to interface with. So those devices are quite often more paid by public money so you can't just replace them. So you have to basically retrofit them with your operating system. That's what we are doing because I mean there was no clean standard basically in the past. For instance, we had a lot of different device types with different interfaces and that of course doesn't make it that easy to integrate with them. So we are building a new operating system and hopefully it will work quite well in the future. So that's what this talk is about in its course. So this little OS box, we are building the operating system based on build route. Who knows build route? Quite a few, good. So it's basically a toolbox really in an abstract way. It's something you can use in order to build a Linux operating system and customize it to your needs. Yeah, we are doing that but that's of course quite difficult because testing software for hardware with different configurations it gets complex quite quickly due to commentatorics. Cool. Yeah, so who of you has seen a test permit so far? So probably you have experienced an enterprise domain somewhere else or you have made some nice PowerPoint slides about that to convince your manager that it might be reasonable to write a few tests. Yes, that's a good idea. So integration tests and unit tests that usually happens with software but at some point you have to integrate with the real world and that's usually where things blow up and you actually want to find bugs and problems as early as possible because I mean in the beginning if you have a unit test, right, you find a bug it's just like you fix something and recompile and it's done and it's good and you're happy but if the client finds the bug it's like you're getting called, operate, Ops is unhappy, you have to roll out everything, you might have to wait quite a long time in order to get your fix out and it's just not making me as a developer happy because basically my development flow is broken but yeah, long story short, you just want to find bugs as early as possible and that's what we're doing with those operating systems which we put in containers in order to run them in the cloud because then we can do that at scale more automated in the CI instead of flashing it or automatically flashing it to a limited set of devices and testing it manually because there we are basically limited to what we have hardware-wise and you have to be there physically. Cool. The stuff we want to present you fits in this column of the test pyramid and the target devices are pretty... there's a huge number of them, I guess 6,000 at least. A lot of different types, most of them are... the old ones like the LCDs over there are PC-1004s, the form factor but they vary in CPU, in RAM size, in CPU types in the interfaces you have these LCDs and the TFTs which are better TVs but they are powerful quad cores with gigabytes of RAM and the outputs are different, we have different boot mechanisms like some devices only support UEFI older ones only support MBR and our goal is to have one image that runs on every single device this is not possible, we have a variety but still we want to have it reduced and not extended to have it still maintainable and testable. There is different... this is just the devices itself, there's different external peripherals like it's called Zughalz-Senzone, train stop sensors let's call it like this the displays are sometimes controlled via serial protocols UARTs or RS485 some sensors are connected via I2S-E sometimes there's special translational hardware which presents itself to the system as a graphics card and the whole thing gets translated into pixels down to the LCDs so proprietary old legacy stuff, no source and there's sometimes a lot of reverse engineering involved and because some of the suppliers of these devices are not a bankrupt or they're being bought by a different company and they're not willing to cooperate basically helping us it's like a closed source software so why should we help you? because in the future you might do everything yourself sometimes they even don't have to source themselves anymore sometimes they go down in the basement and have a look in some file folders and real paper and stuff it's quite interesting, sometimes you log on to those old machines and you have a look at the history and people jumping from one machine to another and all of this is fixed but because they don't know yeah so that's something we really want to fix because that's something you can, well you probably know yourself that's something that is not maintainable so we really want to push a standardised interface so it will be maintainable and we'll make us, I'm a devops guy and use more developer guy but that will make me personally really happy if that works yeah, the standard workflow of the suppliers was have one deviant image, put their software stack on it put it in the device fiddle around till it works and then never touch it again and what we're trying to do is like automated updates in the field and on 6000 devices with thin network lines with sometimes only 10kb bandwidth you really don't want to push that often an image through it especially if it's like 200, 300 megabytes so we would like to test the stuff before we roll it out and make sure that it's working like we wanted to so the problem now arises that we have like a huge hardware variety and how can we simulate that hardware in combination with the software we want to run it on and we're trying to use QEMO the QEMO is pretty awesome because it can, you can really find details, specify how the hardware should behave you can select different CPUs, you can connect different serial devices in different options like is it connected by an ISA bus or a USB and you can change the details of the DMI table which is like a specific part of memory on the hardware motherboard itself and we heavily rely on that to identify the different devices so in order that we know what services we have to start and which not, so on a TFT you need, we use there an electron to display stuff and on an LCD we use like custom written controller service which is communicating with the LCD displays so yeah, we choose QEMO and it's open source so this is the container how it looks like it's not what we're using on the project but it basically reflects the state of the composition you have the outer container, it's an alpine whereas QEMO is installed you have an inner QEMO running which gets the disk image via an volume mount and some ports are exposed for example the SSH to remotely control it in the future we would like to get rid of that and do everything by AMQP commands but this is still far future Does everybody know what AMQP is? Basically the protocol used by RevitonQ which is our middleware basically, message broker so the interesting stuff for QEMO is the DEVTTYS0 which gets exposed on a socket 9000 and can interact with external hardware in this case with a mocked hardware and the control port of the mocked hardware is also exposed we have a noVNC which lets us see how the boot process is but really the yellow stuff is only optional it's not really necessary for testing the image itself So it's quite nice, so you know FarnC it's basically HTML FarnC, VNC, Applet basically so you can browse to port 6080 and basically see what is going on on that machine and you don't need one of those applications so it's tightly coupled in there so what we have been doing at our project we first did that actually with graphical applications with a noFarnC and a container so we were running up to 600 of those devices and you could just click a link with a Unic identifier of the simulated device and you could see what's going on on one of those devices so that's really helpful if you want to stress test your system with real devices especially the back end, not the devices itself to generate load on the back end yeah, cool okay, what you're going to see is something like this setup it's not much different but the mocked, you see like the mocking part is deleted the TTYS0 port is just like exposed by the Docker container so we can interact directly so we kind of doing the manual mocking yeah, so what we're going to show you basically how that thing works we're going to do that unfortunately locally because we can't interface with our client's CI CD system at this point because we didn't get approval but that's fine I mean, we didn't get an answer so far so that's why we're showing you it basically and then we're telling you how we're doing that currently in the CI CD context of our project and how we're deploying that okay, day more time um, I guess if I, I think if I didn't have a mirror it would be better okay Windows P Windows P ah, mirror yeah, give me a second ah, okay perfect so there's the Docker file which builds like the container you've seen so far there's this QEMO disk image I was trying something, not so interesting we have a make file which should make it easy to do stuff right now but we can make runCocker so we're still attached to the container which boots up right now so we have a little delay between the QEMO start and the NoVNC start because if you start them right after each other sometimes NoVNC says it can't connect to the VNC port of QEMO so detail is not so important um the next interesting thing is yeah we can have a look at the booting container which looks like this so it's, this is what you would see if you would start QEMO manually and you could interact but I guess here it's disabled um we can now connect to the container, to the running session inside um and the service running inside runs in a T-Mux so I can attach to the output again but on boot it starts as root so we have to sudo T-Mux we can ls first, we see a session running um we can attach to this, to the session with this name so there we see that the HTTP server started and the sensor handler started on the the FTTY as 0 with the given board rate um so now we can, now we have a look at the at the service we want to test which is on local port 8000 which looks like this, so far it hasn't received anything I'll make this a little smaller that you see okay and then we can send some data I have no time we're not waiting for a response Q0 um connect to local host and the exposed port and okay okay let's try this so you see the sensor received some raw bytes uh decoded them, stripped them, passed them to int and if we so thanks um if we plug something in which is like non-passable as an int, we see that an error happens and test scenario would we put garbage in and see if it's got parsed by the program itself so we see it didn't change so what we're doing with sensor data perhaps is we all our devices basically have an exporter endpoint basically so Prometheus is basically scraping matrix from that system and well it then basically and Prometheus digests all those matrix and we can do later on something without quite interesting to see in the morning where the sun is going up or how bright it is or whatever but it's also for diagnostics quite interesting so I just go back a little um too comfortable we so we push some data in here it got processed by the custom service and got exposed on the 888 port again so this is like a round trip through the software to test yeah so since it's Dockerized you can put it on Kubernetes and put it in the CI and run tests on it, do it automatically yeah so what we're doing we have our project still help charts which is the configuration layer you basically tie around one of those Docker containers and then you can basically mount the news field or OS image you want to put in there then basically run it and you're also through Kubernetes can tag it, put all the metadata into it and run it at scale so it's quite helpful for us at least because now we can basically show the client what we're doing and what kind of different configurations, it's retestable which is usually not a case with hardware, I mean probably you have to try replicating things with hardware it just takes a lot of time as you see it's not silver bullet because the special hardware is really hard to mock special graphic cards you can't really mock in QNU and this is not feasible but the standard cases are doable and it's not possible to have a sample of every hardware configuration in the lab, first we would need like a warehouse to store them all, on the other hand you would have to plug in like the images or like deploy them via updates but still and you're not really controlling the input on the test, you're just testing that it's doing something but you don't know if it's doing the right thing for example, if you can't control the temperature sensor and so this approach makes it kind of possible to get rid of all these problems and not all of them for sure but get rid of some of these problems and make the workflow a little more streamlined Cool, that was it basically so we're open for questions I guess we have plenty of time for questions Oh, go for it Do we have a microphone? I'll just ask a couple of questions You mentioned that you guys have a lot of powerful legacy machines so these are quad-core machines This to me sounds like a bad method of making Did you guys ever run into any problems with that? Did you know of any of these machines that were connected in the past? Please repeat the question The question was if these old legacy systems were hacked, if that's correct Yeah, yes That's a good question Was it even possible to install custom OSes on these machines? Most of these legacy systems typically don't allow you to do a lot of stuff with them So do you guys have like another part of hardware already installed on the stations that you can do anything with? We actually have a lot of those devices shipped to our office so we have a quite big laboratory where we can test quite a certain range of models but office is quite stuffed with it so I mean that's our bottleneck We can test a few of those devices live at the stations but from a security standpoint of view we try to limit the access to the network for once because of past happenings and security is also a reason why we're doing all of this so we can basically close down the operating systems The workflow how the suppliers deployed a new image was like a guy was going there opening the device, putting a CF card in there with the new image and then running it around and so if that was a question When you guys actually developed these mocks for like hardware you basically do it by experimenting with the lab and then coding the behavior that you see so you're actually just reverse engineering? Yes, the mocks are getting developed by reverse engineering sometimes there's a spec but usually it's wide as a door and every vendor made its own solution to that You first Have you considered LinuxKit as an option to build operating systems? There was a... if we have considered LinuxKit, I guess so, the decisions we made before I came but we have good architects I guess they did so but I can't tell you why the picked build route I think because you can choose pretty specific wards, driver wards, core utility you want to have in there and if you really need it and I guess that was the reason for it Back in the time build route basically seemed like a good option so when the project was started I was not quite sure what the variety of hardware is out there so with of course you test against what you know or you basically build your architecture against what you might expect and at that time we had a certain amount of knowledge and build route was basically fitting for that I mean it's a long running project so we haven't started like yesterday I was interested in the options If you are interested just drop us a note and we have the code basically in the public repository so we are probably going to publish that, is that possible? Cool, yeah, so you can have a look at it So you talked about using Helen to deploy things and you showed us the developer experience, it was really nice, do you also have some CI of the internet so I should push things up or is it very much aimed at the developer? It's like our project is completely built in the CI so we have a GitLab CI system and as soon as you push basically through the hook basically it then builds all the code and it deploys automatically So you were talking about exposing those ports and then just you'll get the thing that runs and you'll get a run event point at those ports or this GitLab, it does that thing so at the moment it actually just deploys it so the lifecycle of the deployment is there independent from the run time of the CI or deployment basically and we haven't integrated that, this one so far because the requirement was we just want to see what is running on there and if it's crashing then you can see over time quite well in Kubernetes through logs, metrics and so forth Another thing, compiling build route is like a huge deal it takes really long even in the CI even with heavy machines so we reduced it down to one and a half hours I guess, like with the whole toolchain but still you want to have it you can't wait, click a button and wait for it to compile there If we build separate images for the target hardware and the QEMO no, we don't want to do that we want to test the real image which goes into the field not a specific image for QEMO We are having some kind of variety for some in-house monitoring but this is not going ever in the field down there We are using GitLab CI for integration and you are building build route and you are trying to time the caching mechanism between builds or from scratch If we are using caching for the build route part between runs or stages Yes we are caching the toolchain so usually build route builds the toolchain as well, especially if it has a blank system but what things would really speed up is the cc cache but we didn't get it working till now but this would be a really boost I guess we could make it in half an hour if we do so Someone else? I guess then we are spot on Thank you very much for your time
we're working for an international train company the train is, the information might not propagate background with information systems from other variety of sensors. So you could think of our the information going through our system. So it's different interfaces and that of course doesn't are building a new operating system and hopefully what this talk is about in its course. So this you can use in order to build a Linux operating is like automated updates in the field and on simulate that hardware in combination with the CPUs, you can connect different serial devices in and on an LCD we use like custom written inner QEMO running which gets the disk image via with external hardware in this case with a mocked up to 600 of those devices and you could just simulated device and you could see what's going what we're going to show you basically how that Kubernetes and put it in the CI and run tests on one of those Docker containers and then you can few of those devices live at the stations but