Closing Keynote: Continuous Everything in Mobile
Ballinger, Keith
In today's world, you can't just build a mobile app and call it done. You need to build a high-quality mobile app and react to your user's needs before they know them. This constant iteration, continuous delivery, is difficult for mobile today. What are the best developers doing today to overcome these limitations? How are the world's best companies implementing continuous delivery for mobile? This closing keynote was sponsored by Microsoft.
 Thank you very much. So yes, my name is Keith Ballinger. I was the VP of product at Xamarin where we made various tooling and services for mobile developers. And as you know, very recently, we were purchased by Microsoft. And so now at Microsoft, I lead a team who's focusing on mobile DevOps and continuous delivery, continuous testing, these kinds of things for mobile developers. And so what I wanted to speak to you about today was how to be continuous, and even more so than how to be continuous, how to be not just with delivery, but with everything you do when it comes to mobile development. But first, I think I would like to tell you a story, which is about a mobile startup that I did called Ravid a few years ago. After I left Microsoft, me and a co-founder of mine back in Boston at the time, we decided to make a mobile video messaging app that would allow somebody to open up their phone, choose someone in their contact list, record a video, and send it to them. And we felt this was probably 2009, 2012, or sorry, 2010. We felt this was something that was kind of missing in the world. MMS wasn't really capable of doing this, and so we thought, you know, this is an idea. This is a startup idea. We can go strong with it. Unfortunately, or fortunately, I should say, we were very agile. There was two of us to begin with. We added some more later, and we were able to code really fast. We had a nice Kanban board. We did great customer development. So we had no trouble writing our features. And we started with Android. We were kind of an Android first startup, which was rare at the time. Most people were focusing on iOS, but we felt it was particularly useful to start with Android. So we were very proud of ourselves. We were very agile. We were writing lots of code, but we were also slow. And the reason we were slow is because of this pipeline of development. As you can see, we could code quite quickly. But after it was time to, after someone would check in something, then it would come time to make a build. And to begin with, I'm sure many of you have seen this before, me or my co-founder, we opened up our MacBook, and we would make a build. And then we would test it. And when we tested it, we didn't have automated tests. We had unit tests, but we really didn't have things that would test the UI. We would hold up the phone, and we would start playing with it. And then if we thought everything was good, we'd submit it to the store. And of course, that took quite a while. And with the Android, with the Play Store, it was fairly quick. But then we had to let people know about it. And ultimately, we would start seeing through analytics or through crash reporting what people thought of it. But this entire process was very error-prone, and it really took us a long time to do it. The constraint for testing, though, remained even after we implemented continuous integration. We stood up a box that would take pull requests and automatically make builds. It would mail us out those builds. But we still had to manually go and interact with our mobile devices to make sure that the functionality was good and that we weren't regressing anybody. Now, what even worse for us would be, we would hear bug reports from our users. And in the bug reports, it would be, hey, I have a Samsung S3, or hey, I have an HTC Evolve, and I'm having this crash. Or hey, the camera doesn't work. And we didn't have all of those devices. So what did we do? We would actually get up. We'd go to the mall. And in the United States, malls have these kiosks in the middle where you can buy phones. And we would go and say, hey, we'd like to buy a Samsung phone. Can I play with it for a second? Make, you know, see if I like it? And then the person would be like, yeah, sure. And then we'd secretly install our app, and we'd reproduce the bug. And then we'd be like, OK, now I know what the problem is. And then we'd go back home, and we'd go fix the bug. As you can imagine, that is not very quick. That's not very speedy. And so we ended up with this huge, huge constraint in our pipeline of testing. So we thought, let's speed it up. We just won't test anymore. That'll work. We'll just wait for the crash reports to go in. We'll just live at the mall. I think you can imagine how well this worked out. It worked out real well for us. We got lots of one-star reviews, lots of comments like, hey, I upgraded your app. Doesn't work. Uninstalling now. This was just terrible for us. We needed to test. But we didn't know how to do it quickly enough. This was really the result. This is an artistic rendition of how popular our app was and then how quickly it became very unpopular. Even worse, we weren't learning anything from our customers. And that's really at the heart of it. We had this mobile application that we could use to send videos. And you could send any size video. There were users who were sending two-hour long videos across multiple continents with our technology. At its core, it was brilliant technology, reliable messaging with great ubiquitous video encoding. But we didn't know how users wanted to use it. We didn't know if our login screen was good. We didn't know if our message composition screen was good. But our competition, they did. Our competition could release faster than us. They could release higher quality applications than us, and they were able to determine what users wanted much, much more quickly than we were. So how do we solve this problem? How does anybody solve the problem of releasing their software in a way that is high quality enough but in a way that is fast enough so that they can learn what their users need and then deliver that actual solution? Well, the answer isn't just agile development. The answer is actually DevOps. And that can be surprising to people because they think, what, developers doing operations? How does that work? Well, it's not at its core what DevOps is about. Now, I have a logical syllogism for you. Validated learning, which I will define later, is equal to continuous delivery. If you do continuous delivery, you can have validated learning. And if you want to do continuous deployment and continuous delivery, you need DevOps. That is how you do it. And so that means if you have DevOps, you can be learning from your customers. So let's dive into this a little bit more. I want to unpack in more detail what DevOps is. And at its core, DevOps is not just tooling. DevOps is not just using CI in an automated fashion. DevOps is actually a culture of how you compose your team, how you organize, and how you look at the problem of software engineering and delivery. But the core of this culture are three pillars that a very smart man named Gene Kim has come up with. And these pillars are very useful a useful way to think about any problem that you have in engineering in a DevOps-like manner. Now the first way of DevOps is what Gene Kim calls systems thinking. That is, you can't really treat your coding separately from the problem of how you're going to monitor your users, separate from how you're going to test your software, separate from how you're going to build. No, with DevOps, you need to have a systematic look at the entire thing. You need to understand where your actual bottlenecks are in delivering your software and solve those bottlenecks versus just looking for the easiest thing to make faster. Going back to my account earlier, we were very fast at coding, but it didn't matter how fast we were because we were slow at everything else. And the slowest thing we had was testing. You need to optimize the most important bottlenecks. Now the second way of DevOps is to have feedback loops. I'm not going to get a chance to go into this in much detail with today's talk, but the important thing to think about here is your pipeline, how you deliver software and the stages it goes through, that is a queue. And if you've ever used queues in programming and you've connected multiple queues together, one of the things that you'll soon discover is that if one queue is full, you don't want the next queue to be popping things off and trying to shove them into the next one. You need what's called back pressure. You need systematic feedback throughout your pipeline. Finally, the third way of DevOps is the one that I was talking about earlier. That is, at its core, DevOps helps you to learn about your technology and about your users and how they're going to use your application and system. What do I mean by validated learning? And how is that different from just plain old learning? Validated learning is really about shipping the right thing to the right customer at the right time and doing that through experiments. That's a very key thing to think about. It's shipping the right thing. It's the process of how you get to the place where you've shipped the right thing to the right customer. So what do I mean by the right thing? Well, there's three kind of main components there, main ideas in the right thing. One is that it has to function correctly. If you ship software to somebody that doesn't work, they start the app and it crashes, it just doesn't matter. Secondly, you have to actually be solving someone's problem. If you're not solving the user's problem, it's not the right thing because they're not going to use it. It doesn't matter how perfect your quality is, it just doesn't matter. And finally, even if you've made something that doesn't crash, that works the way you specced it out, and even if it actually solves someone's problem, if it's not usable, if it's not easy to use and responsive and quick, discoverable in your user interface, then, again, you haven't shipped the right thing. So how do you do this? Well, as I mentioned before, you do it with experiments. You have to be a scientist and not just an engineer. And so what do I mean by that? Well, validated learning is really a four-step process. First you're going to code something, and really you're coding something because you've made a hypothesis. So for instance, we had a hypothesis that customers wanted videos of any length. So how did we validate that hypothesis? Well, we wrote code, and then we gave it to users, and then we monitored those users. We looked at their behavior to understand if, given this opportunity to make videos of any length, how long a video would they make? And over and over again. This loop of learning, hypothesizing, creating an experiment, deploying it out there, learning from it. This is at the core of how you improve software. And this is at the core of how you end up shipping the right thing to the right customer. And as you can imagine, if you can't do this very quickly, you're not going to ship many great features. So let's imagine for a second, two companies. The A, that was my startup. We took weeks and weeks to form a single hypothesis, code it up, get it out there, and learn from it. Company B, that was our competition. They were able to, in a much smaller span of time, create smaller bets, but equally interesting ones, form that hypothesis, get it out there, learn from the customer. So in the time it took us to learn one thing, they learned three things or four things. Now many times what they learned is people don't want this or people don't want that. But that's just as important. We spent months and months finding out that people actually didn't want really long videos. Our competition found out very quickly, people love to send very short videos, 10 seconds, 30 seconds, 30 seconds, not one hour and two hours, and they were able to leap ahead of us in the marketplace over and over again. This was our kind of curve of learning. As you gain knowledge at compounds like interest in a bank account, and we were learning more and more, but they were learning even faster and faster, and when you plot our kind of trajectories together, you can see they were just able to outperform us because they were able to out learn what we were doing. So continuous integration, as I mentioned before, it leads to something, right? It leads to continuous quality. And continuous quality leads to continuous deployment, which then leads to this validated learning. And that is why they were able to do so much better than us. Because when you deliver continuously, you are actually learning continuously. And if you are learning continuously, you will be successful, much more successful than someone like me five, six years ago who was not. Let's take another step back to DevOps. I talked about culture, and again, culture at the end, it's not about what software they were using, right? It wasn't about what framework they used to build their app, what framework they used to test their app. My competition had a culture of that continuous learning, but we didn't. Now tooling, though, is important. Without the right tools, you can't have that culture. And then you need to use those tools correctly. So the tools you use can actually enable cultural change with your company, with your team. Without those tools, you end up having to write everything from scratch and build those tools. You can't make many tools out there today that you can use. So I'm just going to take one example, because I'm very familiar with it, and that is of quality. You want continuous quality. You want a world where you're checking in your code, it's automatically getting built. It's automatically getting tested. It's automatically getting deployed. And you're automatically monitoring your users. You don't want to have to have human intervention here. But with quality, that is, knowing that your software is functioning correctly, that can be very, very difficult. And it's difficult on Android because of all the form factors and all the various types of hardware. There's literally thousands of models. And while it is an utter joy, at least for me, to write software for Android, being able to make sure that everything is functioning correctly could be quite difficult. And back again in 2009, 2010, this was particularly an issue for me with the camera and with video playback and with certain kinds of rendering. So but you know, Android isn't the only place that actually is a problem. iOS, actually, is very fragmented as well. Oftentimes people don't think of it that way, but it really is. There's a variety of form factors. There's a variety of operating systems. There's a variety of ways people interact with your app, enter your app, input data, all of those kinds of things. So with all that fragmentation, you end up with this constraint. I've seen this with many, many development teams who are automated with their continuous integration system, who even have automated deployment and monitoring solutions. But it doesn't really matter that they have automated deployment solutions because they're still in this place, this constraint of testing, where if they actually want to validate that they've built something that functions correctly on their user's actual devices, well, they have to do that manually. Or like me, they have to, I guess, go to the mall. So functional testing that is automating that your software works on the types of devices that your users have, that is key to removing that constraint. Now again, if we want to remove that bottleneck, that does mean, though, that we need our results quickly. We need our results that we know, hey, this works on these HTC models. It works on these Samsung models. We need to know that as quickly as we can make a build with CI. And we need to be sure that we're actually validating the right things. So my one kind of product pitch to you today is I will tell you about Test Cloud, which is our solution, our tool to enable this piece of the culture that we built at Xamarin. And Test Cloud allows you to build automated tests, run them on actual real devices, and hook that up into your continuous integration system. And by doing that, all of a sudden, you can see I can code, I can build, I can test, and that constraint goes away. And then you can worry about the next thing, which is how am I going to deploy? How am I going to monitor? You want this entire chain to be as automatic as possible. Well that was the end of what I was here to discuss with you, so I'd like to actually take questions now. Thank you. Who of you still has the power to ask a question? Maybe, well, you don't let me down. I'm proud of you. You didn't go into any specific spot. Like what kind of test Xamarin actually supports, does it support only Xamarin apps or native apps and those details? Great question. Yeah, I didn't want to do too much of a product pitch. I wanted to kind of focus on the philosophy here. And also, I just had 20 minutes. But it is a good question. And just to repeat, what kind of test, what kind of scenarios does Test Cloud support? Basically any kind of application can be tested in Test Cloud. So for instance, and this is public, Slack, their mobile apps which are native, those are tested in Test Cloud. And obviously there are many applications built with Xamarin's platform, that is the C Sharp for iOS and Android technology that are also tested. You can use Ruby to write your test, you can use C Sharp to write your test. Within just a small period of time, the summer, you'll be able to use Java with Appium to write your test. So in terms of platforms and the languages you want to use, it's fairly broad and will continue to make it very broad. It's a very complex topic as to how you should test, for instance, what you should mock, what you should not mock, what scenarios are most important, which are entire topics of their own. But we do have various blogs and documentation on some of these things. Thank you for the question. One topic with Android is that version 4.4 is most popular for games and that even for this conference, the developers like to use version 5 and now version 6 and so on. And what is the testing strategy to find out that it doesn't work with a certain, that it, at least after the testing works, this all must have supported versions. So how to, okay, you can set up your emulators for 5, 6 and so on, but it doesn't help if you just test the OS and then the SDK level and all this stuff. So there must be a test matrix. This is suggested to implement such this test matrix, which is known from totally different testing qualities also from the engineering. This is a very good point. You can't just test on the latest operating system. You have to test on the devices that your users actually have. In fact, one of the things we do in Test Cloud is when you create a test to submit, you can select, okay, I want devices that map to certain market share. I want the top 20 most popular devices, but not just the devices. What we're actually going to give you is the top 20 model plus operating systems that are popular. Of course, if your audience, if your users are using something different, you can select those. So we don't stick to just the most recent operating systems. If you want to test on various popular Android models with Android 4.4, for instance, you can do that to ensure you're seeing that. And very quickly, in fact, what you'll see those differences. And I think it's a very common kind of testing scenario we've seen with people where they started by just trying the latest with emulators. And it's only after using real devices with real world operating systems that people actually use not just the latest, that they discover a lot of those bugs that they're giving them one-stars, right? Yeah, one more question over here. Thank you for a great talk. On the previous slide, you showed like this Xamarin Continuous, maybe one, or this one, whatever, like this Xamarin Continuous Integration Stack. So the question is, is it possible to replace one of the steps with some third party tool or platform, whatever? Like for instance, if I want to take this cloud test, is it possible to replace particular this module with Firebase Test Cloud, for instance, or it's going to be like isolated structure? That's a good question. So in fact, this diagram isn't supposed to represent the software and services that we have built at Xamarin. In fact, we do not have a continuous integration service with Xamarin. This is just the pipeline of how people build software. We have tooling for testing, and we have tooling for monitoring, but we actually don't have tooling for all of these things today. Microsoft certainly does things like Visual Studio Team Services let you do CI. But Test Cloud itself specifically works with any continuous integration service, Jenkins, Team City, Bitrise, CircleCI, all of those will work with Test Cloud. And conversely, if you're doing continuous integration with something like Visual Studio Team Services or Bitrise or whatever, you can just as easily use, like you said, Firebase Test Cloud instead of Xamarin Test Cloud. The only thing you really have to watch out for as you mix and match tooling is just to make sure that you have somewhat of a consistent experience. So for instance, if you decide to use Calabash and Ruby with Test Cloud, you'll want to make sure that another testing, if you want to move to another testing vendor, that they also support Calabash with Ruby so that you don't have to replace all of your, all the work that you've done. But in general, I think most of the top performing teams in the world, they choose the best in class service for each one of these pieces. And that doesn't necessarily mean they get everything from the same vendor. They get the thing that's best for them, and then they make them work together. And I think that interoperability, that's like a UNIX-like philosophy, right? Have a tool that does one thing really well. That's very important, I think, for high performing teams. And important to take care of. Thank you. Thank you, Keith. I don't see any more questions. One more warm hand for Keith, please. Thank you so much. Thank you for being with us.
 Thank you very much.  So yes, my name is Keith Ballinger.  I was the VP of product at Xamarin where we made various tooling and services for mobile  developers.  And as you know, very recently, we were purchased by Microsoft.  And so now at Microsoft, I lead a team who's focusing on mobile DevOps and continuous  delivery, continuous testing, these kinds of things for mobile developers.  And so what I wanted to speak to you about today was how to be continuous, and even more  so than how to be continuous, how to be not just with delivery, but with everything you  do when it comes to mobile development.  But first, I think I would like to tell you a story, which is about a mobile startup that  I did called Ravid a few years ago.  After I left Microsoft, me and a co-founder of mine back in Boston at the time, we decided  to make a mobile video messaging app that would allow somebody to open up their phone,  choose someone in their contact list, record a video, and send it to them.  And we felt this was probably 2009, 2012, or sorry, 2010.  We felt this was something that was kind of missing in the world.  MMS wasn't really capable of doing this, and so we thought, you know, this is an idea.  This is a startup idea.  We can go strong with it.  Unfortunately, or fortunately, I should say, we were very agile.  There was two of us to begin with.  We added some more later, and we were able to code really fast.  We had a nice Kanban board.  We did great customer development.  So we had no trouble writing our features.  And we started with Android.  We were kind of an Android first startup, which was rare at the time.  Most people were focusing on iOS, but we felt it was particularly useful to start with Android.  So we were very proud of ourselves.  We were very agile.  We were writing lots of code, but we were also slow.  And the reason we were slow is because of this pipeline of development.  As you can see, we could code quite quickly.  But after it was time to, after someone would check in something, then it would come time  to make a build.  And to begin with, I'm sure many of you have seen this before, me or my co-founder, we  opened up our MacBook, and we would make a build.  And then we would test it.  And when we tested it, we didn't have automated tests.  We had unit tests, but we really didn't have things that would test the UI.  We would hold up the phone, and we would start playing with it.  And then if we thought everything was good, we'd submit it to the store.  And of course, that took quite a while.  And with the Android, with the Play Store, it was fairly quick.  But then we had to let people know about it.  And ultimately, we would start seeing through analytics or through crash reporting what people  thought of it.  But this entire process was very error-prone, and it really took us a long time to do it.  The constraint for testing, though, remained even after we implemented continuous integration.  We stood up a box that would take pull requests and automatically make builds.  It would mail us out those builds.  But we still had to manually go and interact with our mobile devices to make sure that  the functionality was good and that we weren't regressing anybody.  Now, what even worse for us would be, we would hear bug reports from our users.  And in the bug reports, it would be, hey, I have a Samsung S3, or hey, I have an HTC  Evolve, and I'm having this crash.  Or hey, the camera doesn't work.  And we didn't have all of those devices.  So what did we do?  We would actually get up.  We'd go to the mall.  And in the United States, malls have these kiosks in the middle where you can buy phones.  And we would go and say, hey, we'd like to buy a Samsung phone.  Can I play with it for a second?  Make, you know, see if I like it?  And then the person would be like, yeah, sure.  And then we'd secretly install our app, and we'd reproduce the bug.  And then we'd be like, OK, now I know what the problem is.  And then we'd go back home, and we'd go fix the bug.  As you can imagine, that is not very quick.  That's not very speedy.  And so we ended up with this huge, huge constraint in our pipeline of testing.  So we thought, let's speed it up.  We just won't test anymore.  That'll work.  We'll just wait for the crash reports to go in.  We'll just live at the mall.  I think you can imagine how well this worked out.  It worked out real well for us.  We got lots of one-star reviews, lots of comments like, hey, I upgraded your app.  Doesn't work.  Uninstalling now.  This was just terrible for us.  We needed to test.  But we didn't know how to do it quickly enough.  This was really the result.  This is an artistic rendition of how popular our app was and then how quickly it became  very unpopular.  Even worse, we weren't learning anything from our customers.  And that's really at the heart of it.  We had this mobile application that we could use to send videos.  And you could send any size video.  There were users who were sending two-hour long videos across multiple continents with  our technology.  At its core, it was brilliant technology, reliable messaging with great ubiquitous video encoding.  But we didn't know how users wanted to use it.  We didn't know if our login screen was good.  We didn't know if our message composition screen was good.  But our competition, they did.  Our competition could release faster than us.  They could release higher quality applications than us, and they were able to determine what  users wanted much, much more quickly than we were.  So how do we solve this problem?  How does anybody solve the problem of releasing their software in a way that is high quality  enough but in a way that is fast enough so that they can learn what their users need  and then deliver that actual solution?  Well, the answer isn't just agile development.  The answer is actually DevOps.  And that can be surprising to people because they think, what, developers doing operations?  How does that work?  Well, it's not at its core what DevOps is about.  Now, I have a logical syllogism for you.  Validated learning, which I will define later, is equal to continuous delivery.  If you do continuous delivery, you can have validated learning.  And if you want to do continuous deployment and continuous delivery, you need DevOps.  That is how you do it.  And so that means if you have DevOps, you can be learning from your customers.  So let's dive into this a little bit more.  I want to unpack in more detail what DevOps is.  And at its core, DevOps is not just tooling.  DevOps is not just using CI in an automated fashion.  DevOps is actually a culture of how you compose your team, how you organize, and how you look  at the problem of software engineering and delivery.  But the core of this culture are three pillars that a very smart man named Gene Kim has come  up with.  And these pillars are very useful a useful way to think about any problem that you have  in engineering in a DevOps-like manner.  Now the first way of DevOps is what Gene Kim calls systems thinking.  That is, you can't really treat your coding separately from the problem of how you're  going to monitor your users, separate from how you're going to test your software, separate  from how you're going to build.  No, with DevOps, you need to have a systematic look at the entire thing.  You need to understand where your actual bottlenecks are in delivering your software and solve  those bottlenecks versus just looking for the easiest thing to make faster.  Going back to my account earlier, we were very fast at coding, but it didn't matter  how fast we were because we were slow at everything else.  And the slowest thing we had was testing.  You need to optimize the most important bottlenecks.  Now the second way of DevOps is to have feedback loops.  I'm not going to get a chance to go into this in much detail with today's talk, but the  important thing to think about here is your pipeline, how you deliver software and the  stages it goes through, that is a queue.  And if you've ever used queues in programming and you've connected multiple queues together,  one of the things that you'll soon discover is that if one queue is full, you don't want  the next queue to be popping things off and trying to shove them into the next one.  You need what's called back pressure.  You need systematic feedback throughout your pipeline.  Finally, the third way of DevOps is the one that I was talking about earlier.  That is, at its core, DevOps helps you to learn about your technology and about your  users and how they're going to use your application and system.  What do I mean by validated learning?  And how is that different from just plain old learning?  Validated learning is really about shipping the right thing to the right customer at the  right time and doing that through experiments.  That's a very key thing to think about.  It's shipping the right thing.  It's the process of how you get to the place where you've shipped the right thing to the  right customer.  So what do I mean by the right thing?  Well, there's three kind of main components there, main ideas in the right thing.  One is that it has to function correctly.  If you ship software to somebody that doesn't work, they start the app and it crashes, it  just doesn't matter.  Secondly, you have to actually be solving someone's problem.  If you're not solving the user's problem, it's not the right thing because they're not  going to use it.  It doesn't matter how perfect your quality is, it just doesn't matter.  And finally, even if you've made something that doesn't crash, that works the way you  specced it out, and even if it actually solves someone's problem, if it's not usable, if  it's not easy to use and responsive and quick, discoverable in your user interface, then,  again, you haven't shipped the right thing.  So how do you do this?  Well, as I mentioned before, you do it with experiments.  You have to be a scientist and not just an engineer.  And so what do I mean by that?  Well, validated learning is really a four-step process.  First you're going to code something, and really you're coding something because you've  made a hypothesis.  So for instance, we had a hypothesis that customers wanted videos of any length.  So how did we validate that hypothesis?  Well, we wrote code, and then we gave it to users, and then we monitored those users.  We looked at their behavior to understand if, given this opportunity to make videos of  any length, how long a video would they make?  And over and over again.  This loop of learning, hypothesizing, creating an experiment, deploying it out there, learning  from it.  This is at the core of how you improve software.  And this is at the core of how you end up shipping the right thing to the right customer.  And as you can imagine, if you can't do this very quickly, you're not going to ship many  great features.  So let's imagine for a second, two companies.  The A, that was my startup.  We took weeks and weeks to form a single hypothesis, code it up, get it out there, and learn from  it.  Company B, that was our competition.  They were able to, in a much smaller span of time, create smaller bets, but equally  interesting ones, form that hypothesis, get it out there, learn from the customer.  So in the time it took us to learn one thing, they learned three things or four things.  Now many times what they learned is people don't want this or people don't want that.  But that's just as important.  We spent months and months finding out that people actually didn't want really long videos.  Our competition found out very quickly, people love to send very short videos, 10 seconds,  30 seconds, 30 seconds, not one hour and two hours, and they were able to leap ahead of  us in the marketplace over and over again.  This was our kind of curve of learning.  As you gain knowledge at compounds like interest in a bank account, and we were learning more  and more, but they were learning even faster and faster, and when you plot our kind of  trajectories together, you can see they were just able to outperform us because they were  able to out learn what we were doing.  So continuous integration, as I mentioned before, it leads to something, right?  It leads to continuous quality.  And continuous quality leads to continuous deployment, which then leads to this validated  learning.  And that is why they were able to do so much better than us.  Because when you deliver continuously, you are actually learning continuously.  And if you are learning continuously, you will be successful, much more successful than  someone like me five, six years ago who was not.  Let's take another step back to DevOps.  I talked about culture, and again, culture at the end, it's not about what software they  were using, right?  It wasn't about what framework they used to build their app, what framework they used  to test their app.  My competition had a culture of that continuous learning, but we didn't.  Now tooling, though, is important.  Without the right tools, you can't have that culture.  And then you need to use those tools correctly.  So the tools you use can actually enable cultural change with your company, with your team.  Without those tools, you end up having to write everything from scratch and build those  tools.  You can't make many tools out there today that you can use.  So I'm just going to take one example, because I'm very familiar with it, and that is of  quality.  You want continuous quality.  You want a world where you're checking in your code, it's automatically getting built.  It's automatically getting tested.  It's automatically getting deployed.  And you're automatically monitoring your users.  You don't want to have to have human intervention here.  But with quality, that is, knowing that your software is functioning correctly, that can  be very, very difficult.  And it's difficult on Android because of all the form factors and all the various types  of hardware.  There's literally thousands of models.  And while it is an utter joy, at least for me, to write software for Android, being able  to make sure that everything is functioning correctly could be quite difficult.  And back again in 2009, 2010, this was particularly an issue for me with the camera and with video  playback and with certain kinds of rendering.  So but you know, Android isn't the only place that actually is a problem.  iOS, actually, is very fragmented as well.  Oftentimes people don't think of it that way, but it really is.  There's a variety of form factors.  There's a variety of operating systems.  There's a variety of ways people interact with your app, enter your app, input data,  all of those kinds of things.  So with all that fragmentation, you end up with this constraint.  I've seen this with many, many development teams who are automated with their continuous  integration system, who even have automated deployment and monitoring solutions.  But it doesn't really matter that they have automated deployment solutions because they're  still in this place, this constraint of testing, where if they actually want to validate that  they've built something that functions correctly on their user's actual devices, well, they  have to do that manually.  Or like me, they have to, I guess, go to the mall.  So functional testing that is automating that your software works on the types of devices  that your users have, that is key to removing that constraint.  Now again, if we want to remove that bottleneck, that does mean, though, that we need our results  quickly.  We need our results that we know, hey, this works on these HTC models.  It works on these Samsung models.  We need to know that as quickly as we can make a build with CI.  And we need to be sure that we're actually validating the right things.  So my one kind of product pitch to you today is I will tell you about Test Cloud, which  is our solution, our tool to enable this piece of the culture that we built at Xamarin.  And Test Cloud allows you to build automated tests, run them on actual real devices, and  hook that up into your continuous integration system.  And by doing that, all of a sudden, you can see I can code, I can build, I can test, and  that constraint goes away.  And then you can worry about the next thing, which is how am I going to deploy?  How am I going to monitor?  You want this entire chain to be as automatic as possible.  Well that was the end of what I was here to discuss with you, so I'd like to actually  take questions now.  Thank you.  Who of you still has the power to ask a question?  Maybe, well, you don't let me down.  I'm proud of you.  You didn't go into any specific spot.  Like what kind of test Xamarin actually supports, does it support only Xamarin apps or native  apps and those details?  Great question.  Yeah, I didn't want to do too much of a product pitch.  I wanted to kind of focus on the philosophy here.  And also, I just had 20 minutes.  But it is a good question.  And just to repeat, what kind of test, what kind of scenarios does Test Cloud support?  Basically any kind of application can be tested in Test Cloud.  So for instance, and this is public, Slack, their mobile apps which are native, those  are tested in Test Cloud.  And obviously there are many applications built with Xamarin's platform, that is the  C Sharp for iOS and Android technology that are also tested.  You can use Ruby to write your test, you can use C Sharp to write your test.  Within just a small period of time, the summer, you'll be able to use Java with Appium to  write your test.  So in terms of platforms and the languages you want to use, it's fairly broad and will  continue to make it very broad.  It's a very complex topic as to how you should test, for instance, what you should mock,  what you should not mock, what scenarios are most important, which are entire topics of  their own.  But we do have various blogs and documentation on some of these things.  Thank you for the question.  One topic with Android is that version 4.4 is most popular for games and that even for  this conference, the developers like to use version 5 and now version 6 and so on.  And what is the testing strategy to find out that it doesn't work with a certain, that  it, at least after the testing works, this all must have supported versions.  So how to, okay, you can set up your emulators for 5, 6 and so on, but it doesn't help if  you just test the OS and then the SDK level and all this stuff.  So there must be a test matrix.  This is suggested to implement such this test matrix, which is known from totally different  testing qualities also from the engineering.  This is a very good point.  You can't just test on the latest operating system.  You have to test on the devices that your users actually have.  In fact, one of the things we do in Test Cloud is when you create a test to submit, you can  select, okay, I want devices that map to certain market share.  I want the top 20 most popular devices, but not just the devices.  What we're actually going to give you is the top 20 model plus operating systems that are  popular.  Of course, if your audience, if your users are using something different, you can select  those.  So we don't stick to just the most recent operating systems.  If you want to test on various popular Android models with Android 4.4, for instance, you  can do that to ensure you're seeing that.  And very quickly, in fact, what you'll see those differences.  And I think it's a very common kind of testing scenario we've seen with people where they  started by just trying the latest with emulators.  And it's only after using real devices with real world operating systems that people actually  use not just the latest, that they discover a lot of those bugs that they're giving them  one-stars, right?  Yeah, one more question over here.  Thank you for a great talk.  On the previous slide, you showed like this Xamarin Continuous, maybe one, or this one,  whatever, like this Xamarin Continuous Integration Stack.  So the question is, is it possible to replace one of the steps with some third party tool  or platform, whatever?  Like for instance, if I want to take this cloud test, is it possible to replace particular  this module with Firebase Test Cloud, for instance, or it's going to be like isolated  structure?  That's a good question.  So in fact, this diagram isn't supposed to represent the software and services that we  have built at Xamarin.  In fact, we do not have a continuous integration service with Xamarin.  This is just the pipeline of how people build software.  We have tooling for testing, and we have tooling for monitoring, but we actually don't have  tooling for all of these things today.  Microsoft certainly does things like Visual Studio Team Services let you do CI.  But Test Cloud itself specifically works with any continuous integration service, Jenkins,  Team City, Bitrise, CircleCI, all of those will work with Test Cloud.  And conversely, if you're doing continuous integration with something like Visual Studio  Team Services or Bitrise or whatever, you can just as easily use, like you said, Firebase  Test Cloud instead of Xamarin Test Cloud.  The only thing you really have to watch out for as you mix and match tooling is just to  make sure that you have somewhat of a consistent experience.  So for instance, if you decide to use Calabash and Ruby with Test Cloud, you'll want to make  sure that another testing, if you want to move to another testing vendor, that they  also support Calabash with Ruby so that you don't have to replace all of your, all the  work that you've done.  But in general, I think most of the top performing teams in the world, they choose the best in  class service for each one of these pieces.  And that doesn't necessarily mean they get everything from the same vendor.  They get the thing that's best for them, and then they make them work together.  And I think that interoperability, that's like a UNIX-like philosophy, right?  Have a tool that does one thing really well.  That's very important, I think, for high performing teams.  And important to take care of.  Thank you.  Thank you, Keith.  I don't see any more questions.  One more warm hand for Keith, please.  Thank you so much.  Thank you for being with us.
_droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_0_131.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_132_443.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_444_563.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_564_935.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_936_2327.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_2328_2399.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_2400_4763.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_4764_5471.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_5472_5699.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_5700_5759.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_5760_7055.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_7056_10415.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_10416_10559.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_10560_15983.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_15984_17003.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_17004_17435.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_17436_17531.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_17532_18479.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_18480_19703.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_19704_20831.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_20832_21035.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_21036_21299.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_21300_26243.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_26244_29615.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_29616_29735.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_29736_30035.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_30036_30419.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_30420_30779.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_30780_30899.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_30900_31259.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_31260_31571.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_31572_31787.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_31788_31955.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_31956_32207.png _droidconDE__Keith_Ballinger___Continuous_Everythi_6NOJQ-E_YRM_32208_38801.png
Thank you very much. So yes, my name is Keith Ballinger. I was the VP of product at Xamarin where we made various tooling and services for mobile developers. And as you know, very recently, we were purchased by Microsoft. And so now at Microsoft, I lead a team who's focusing on mobile DevOps and continuous delivery, continuous testing, these kinds of things for mobile developers. And so what I wanted to speak to you about today was how to be continuous, and even more so than how to be continuous, how to be not just with delivery, but with everything you do when it comes to mobile development. But first, I think I would like to tell you a story, which is about a mobile startup that I did called Ravid a few years ago. After I left Microsoft, me and a co-founder of mine back in Boston at the time, we decided to make a mobile video messaging app that would allow somebody to open up their phone, choose someone in their contact list, record a video, and send it to them. And we felt this was probably 2009, 2012, or sorry, 2010. We felt this was something that was kind of missing in the world. MMS wasn't really capable of doing this, and so we thought, you know, this is an idea. This is a startup idea. We can go strong with it. Unfortunately, or fortunately, I should say, we were very agile. There was two of us to begin with. We added some more later, and we were able to code really fast. We had a nice Kanban board. We did great customer development. So we had no trouble writing our features. And we started with Android. We were kind of an Android first startup, which was rare at the time. Most people were focusing on iOS, but we felt it was particularly useful to start with Android. So we were very proud of ourselves. We were very agile. We were writing lots of code, but we were also slow. And the reason we were slow is because of this pipeline of development. As you can see, we could code quite quickly. But after it was time to, after someone would check in something, then it would come time to make a build. And to begin with, I'm sure many of you have seen this before, me or my co-founder, we opened up our MacBook, and we would make a build. And then we would test it. And when we tested it, we didn't have automated tests. We had unit tests, but we really didn't have things that would test the UI. We would hold up the phone, and we would start playing with it. And then if we thought everything was good, we'd submit it to the store. And of course, that took quite a while. And with the Android, with the Play Store, it was fairly quick. But then we had to let people know about it. And ultimately, we would start seeing through analytics or through crash reporting what people thought of it. But this entire process was very error-prone, and it really took us a long time to do it. The constraint for testing, though, remained even after we implemented continuous integration. We stood up a box that would take pull requests and automatically make builds. It would mail us out those builds. But we still had to manually go and interact with our mobile devices to make sure that the functionality was good and that we weren't regressing anybody. Now, what even worse for us would be, we would hear bug reports from our users. And in the bug reports, it would be, hey, I have a Samsung S3, or hey, I have an HTC Evolve, and I'm having this crash. Or hey, the camera doesn't work. And we didn't have all of those devices. So what did we do? We would actually get up. We'd go to the mall. And in the United States, malls have these kiosks in the middle where you can buy phones. And we would go and say, hey, we'd like to buy a Samsung phone. Can I play with it for a second? Make, you know, see if I like it? And then the person would be like, yeah, sure. And then we'd secretly install our app, and we'd reproduce the bug. And then we'd be like, OK, now I know what the problem is. And then we'd go back home, and we'd go fix the bug. As you can imagine, that is not very quick. That's not very speedy. And so we ended up with this huge, huge constraint in our pipeline of testing. So we thought, let's speed it up. We just won't test anymore. That'll work. We'll just wait for the crash reports to go in. We'll just live at the mall. I think you can imagine how well this worked out. It worked out real well for us. We got lots of one-star reviews, lots of comments like, hey, I upgraded your app. Doesn't work. Uninstalling now. This was just terrible for us. We needed to test. But we didn't know how to do it quickly enough. This was really the result. This is an artistic rendition of how popular our app was and then how quickly it became very unpopular. Even worse, we weren't learning anything from our customers. And that's really at the heart of it. We had this mobile application that we could use to send videos. And you could send any size video. There were users who were sending two-hour long videos across multiple continents with our technology. At its core, it was brilliant technology, reliable messaging with great ubiquitous video encoding. But we didn't know how users wanted to use it. We didn't know if our login screen was good. We didn't know if our message composition screen was good. But our competition, they did. Our competition could release faster than us. They could release higher quality applications than us, and they were able to determine what users wanted much, much more quickly than we were. So how do we solve this problem? How does anybody solve the problem of releasing their software in a way that is high quality enough but in a way that is fast enough so that they can learn what their users need and then deliver that actual solution? Well, the answer isn't just agile development. The answer is actually DevOps. And that can be surprising to people because they think, what, developers doing operations? How does that work? Well, it's not at its core what DevOps is about. Now, I have a logical syllogism for you. Validated learning, which I will define later, is equal to continuous delivery. If you do continuous delivery, you can have validated learning. And if you want to do continuous deployment and continuous delivery, you need DevOps. That is how you do it. And so that means if you have DevOps, you can be learning from your customers. So let's dive into this a little bit more. I want to unpack in more detail what DevOps is. And at its core, DevOps is not just tooling. DevOps is not just using CI in an automated fashion. DevOps is actually a culture of how you compose your team, how you organize, and how you look at the problem of software engineering and delivery. But the core of this culture are three pillars that a very smart man named Gene Kim has come up with. And these pillars are very useful a useful way to think about any problem that you have in engineering in a DevOps-like manner. Now the first way of DevOps is what Gene Kim calls systems thinking. That is, you can't really treat your coding separately from the problem of how you're going to monitor your users, separate from how you're going to test your software, separate from how you're going to build. No, with DevOps, you need to have a systematic look at the entire thing. You need to understand where your actual bottlenecks are in delivering your software and solve those bottlenecks versus just looking for the easiest thing to make faster. Going back to my account earlier, we were very fast at coding, but it didn't matter how fast we were because we were slow at everything else. And the slowest thing we had was testing. You need to optimize the most important bottlenecks. Now the second way of DevOps is to have feedback loops. I'm not going to get a chance to go into this in much detail with today's talk, but the important thing to think about here is your pipeline, how you deliver software and the stages it goes through, that is a queue. And if you've ever used queues in programming and you've connected multiple queues together, one of the things that you'll soon discover is that if one queue is full, you don't want the next queue to be popping things off and trying to shove them into the next one. You need what's called back pressure. You need systematic feedback throughout your pipeline. Finally, the third way of DevOps is the one that I was talking about earlier. That is, at its core, DevOps helps you to learn about your technology and about your users and how they're going to use your application and system. What do I mean by validated learning? And how is that different from just plain old learning? Validated learning is really about shipping the right thing to the right customer at the right time and doing that through experiments. That's a very key thing to think about. It's shipping the right thing. It's the process of how you get to the place where you've shipped the right thing to the right customer. So what do I mean by the right thing? Well, there's three kind of main components there, main ideas in the right thing. One is that it has to function correctly. If you ship software to somebody that doesn't work, they start the app and it crashes, it just doesn't matter. Secondly, you have to actually be solving someone's problem. If you're not solving the user's problem, it's not the right thing because they're not going to use it. It doesn't matter how perfect your quality is, it just doesn't matter. And finally, even if you've made something that doesn't crash, that works the way you specced it out, and even if it actually solves someone's problem, if it's not usable, if it's not easy to use and responsive and quick, discoverable in your user interface, then, again, you haven't shipped the right thing. So how do you do this? Well, as I mentioned before, you do it with experiments. You have to be a scientist and not just an engineer. And so what do I mean by that? Well, validated learning is really a four-step process. First you're going to code something, and really you're coding something because you've made a hypothesis. So for instance, we had a hypothesis that customers wanted videos of any length. So how did we validate that hypothesis? Well, we wrote code, and then we gave it to users, and then we monitored those users. We looked at their behavior to understand if, given this opportunity to make videos of any length, how long a video would they make? And over and over again. This loop of learning, hypothesizing, creating an experiment, deploying it out there, learning from it. This is at the core of how you improve software. And this is at the core of how you end up shipping the right thing to the right customer. And as you can imagine, if you can't do this very quickly, you're not going to ship many great features. So let's imagine for a second, two companies. The A, that was my startup. We took weeks and weeks to form a single hypothesis, code it up, get it out there, and learn from it. Company B, that was our competition. They were able to, in a much smaller span of time, create smaller bets, but equally interesting ones, form that hypothesis, get it out there, learn from the customer. So in the time it took us to learn one thing, they learned three things or four things. Now many times what they learned is people don't want this or people don't want that. But that's just as important. We spent months and months finding out that people actually didn't want really long videos. Our competition found out very quickly, people love to send very short videos, 10 seconds, 30 seconds, 30 seconds, not one hour and two hours, and they were able to leap ahead of us in the marketplace over and over again. This was our kind of curve of learning. As you gain knowledge at compounds like interest in a bank account, and we were learning more and more, but they were learning even faster and faster, and when you plot our kind of trajectories together, you can see they were just able to outperform us because they were able to out learn what we were doing. So continuous integration, as I mentioned before, it leads to something, right? It leads to continuous quality. And continuous quality leads to continuous deployment, which then leads to this validated learning. And that is why they were able to do so much better than us. Because when you deliver continuously, you are actually learning continuously. And if you are learning continuously, you will be successful, much more successful than someone like me five, six years ago who was not. Let's take another step back to DevOps. I talked about culture, and again, culture at the end, it's not about what software they were using, right? It wasn't about what framework they used to build their app, what framework they used to test their app. My competition had a culture of that continuous learning, but we didn't. Now tooling, though, is important. Without the right tools, you can't have that culture. And then you need to use those tools correctly. So the tools you use can actually enable cultural change with your company, with your team. Without those tools, you end up having to write everything from scratch and build those tools. You can't make many tools out there today that you can use. So I'm just going to take one example, because I'm very familiar with it, and that is of quality. You want continuous quality. You want a world where you're checking in your code, it's automatically getting built. It's automatically getting tested. It's automatically getting deployed. And you're automatically monitoring your users. You don't want to have to have human intervention here. But with quality, that is, knowing that your software is functioning correctly, that can be very, very difficult. And it's difficult on Android because of all the form factors and all the various types of hardware. There's literally thousands of models. And while it is an utter joy, at least for me, to write software for Android, being able to make sure that everything is functioning correctly could be quite difficult. And back again in 2009, 2010, this was particularly an issue for me with the camera and with video playback and with certain kinds of rendering. So but you know, Android isn't the only place that actually is a problem. iOS, actually, is very fragmented as well. Oftentimes people don't think of it that way, but it really is. There's a variety of form factors. There's a variety of operating systems. There's a variety of ways people interact with your app, enter your app, input data, all of those kinds of things. So with all that fragmentation, you end up with this constraint. I've seen this with many, many development teams who are automated with their continuous integration system, who even have automated deployment and monitoring solutions. But it doesn't really matter that they have automated deployment solutions because they're still in this place, this constraint of testing, where if they actually want to validate that they've built something that functions correctly on their user's actual devices, well, they have to do that manually. Or like me, they have to, I guess, go to the mall. So functional testing that is automating that your software works on the types of devices that your users have, that is key to removing that constraint. Now again, if we want to remove that bottleneck, that does mean, though, that we need our results quickly. We need our results that we know, hey, this works on these HTC models. It works on these Samsung models. We need to know that as quickly as we can make a build with CI. And we need to be sure that we're actually validating the right things. So my one kind of product pitch to you today is I will tell you about Test Cloud, which is our solution, our tool to enable this piece of the culture that we built at Xamarin. And Test Cloud allows you to build automated tests, run them on actual real devices, and hook that up into your continuous integration system. And by doing that, all of a sudden, you can see I can code, I can build, I can test, and that constraint goes away. And then you can worry about the next thing, which is how am I going to deploy? How am I going to monitor? You want this entire chain to be as automatic as possible. Well that was the end of what I was here to discuss with you, so I'd like to actually take questions now. Thank you. Who of you still has the power to ask a question? Maybe, well, you don't let me down. I'm proud of you. You didn't go into any specific spot. Like what kind of test Xamarin actually supports, does it support only Xamarin apps or native apps and those details? Great question. Yeah, I didn't want to do too much of a product pitch. I wanted to kind of focus on the philosophy here. And also, I just had 20 minutes. But it is a good question. And just to repeat, what kind of test, what kind of scenarios does Test Cloud support? Basically any kind of application can be tested in Test Cloud. So for instance, and this is public, Slack, their mobile apps which are native, those are tested in Test Cloud. And obviously there are many applications built with Xamarin's platform, that is the C Sharp for iOS and Android technology that are also tested. You can use Ruby to write your test, you can use C Sharp to write your test. Within just a small period of time, the summer, you'll be able to use Java with Appium to write your test. So in terms of platforms and the languages you want to use, it's fairly broad and will continue to make it very broad. It's a very complex topic as to how you should test, for instance, what you should mock, what you should not mock, what scenarios are most important, which are entire topics of their own. But we do have various blogs and documentation on some of these things. Thank you for the question. One topic with Android is that version 4.4 is most popular for games and that even for this conference, the developers like to use version 5 and now version 6 and so on. And what is the testing strategy to find out that it doesn't work with a certain, that it, at least after the testing works, this all must have supported versions. So how to, okay, you can set up your emulators for 5, 6 and so on, but it doesn't help if you just test the OS and then the SDK level and all this stuff. So there must be a test matrix. This is suggested to implement such this test matrix, which is known from totally different testing qualities also from the engineering. This is a very good point. You can't just test on the latest operating system. You have to test on the devices that your users actually have. In fact, one of the things we do in Test Cloud is when you create a test to submit, you can select, okay, I want devices that map to certain market share. I want the top 20 most popular devices, but not just the devices. What we're actually going to give you is the top 20 model plus operating systems that are popular. Of course, if your audience, if your users are using something different, you can select those. So we don't stick to just the most recent operating systems. If you want to test on various popular Android models with Android 4.4, for instance, you can do that to ensure you're seeing that. And very quickly, in fact, what you'll see those differences. And I think it's a very common kind of testing scenario we've seen with people where they started by just trying the latest with emulators. And it's only after using real devices with real world operating systems that people actually use not just the latest, that they discover a lot of those bugs that they're giving them one-stars, right? Yeah, one more question over here. Thank you for a great talk. On the previous slide, you showed like this Xamarin Continuous, maybe one, or this one, whatever, like this Xamarin Continuous Integration Stack. So the question is, is it possible to replace one of the steps with some third party tool or platform, whatever? Like for instance, if I want to take this cloud test, is it possible to replace particular this module with Firebase Test Cloud, for instance, or it's going to be like isolated structure? That's a good question. So in fact, this diagram isn't supposed to represent the software and services that we have built at Xamarin. In fact, we do not have a continuous integration service with Xamarin. This is just the pipeline of how people build software. We have tooling for testing, and we have tooling for monitoring, but we actually don't have tooling for all of these things today. Microsoft certainly does things like Visual Studio Team Services let you do CI. But Test Cloud itself specifically works with any continuous integration service, Jenkins, Team City, Bitrise, CircleCI, all of those will work with Test Cloud. And conversely, if you're doing continuous integration with something like Visual Studio Team Services or Bitrise or whatever, you can just as easily use, like you said, Firebase Test Cloud instead of Xamarin Test Cloud. The only thing you really have to watch out for as you mix and match tooling is just to make sure that you have somewhat of a consistent experience. So for instance, if you decide to use Calabash and Ruby with Test Cloud, you'll want to make sure that another testing, if you want to move to another testing vendor, that they also support Calabash with Ruby so that you don't have to replace all of your, all the work that you've done. But in general, I think most of the top performing teams in the world, they choose the best in class service for each one of these pieces. And that doesn't necessarily mean they get everything from the same vendor. They get the thing that's best for them, and then they make them work together. And I think that interoperability, that's like a UNIX-like philosophy, right? Have a tool that does one thing really well. That's very important, I think, for high performing teams. And important to take care of. Thank you. Thank you, Keith. I don't see any more questions. One more warm hand for Keith, please. Thank you so much. Thank you for being with us.
testing, these kinds of things for mobile think, what, developers doing operations? How deployment and continuous delivery, you need thing. You need to understand where your actual start the app and it crashes, it just doesn't framework they used to build their app, what engineering. This is a very good point. You can't Of course, if your audience, if your users are for all of these things today. Microsoft in the world, they choose the best in class