Shazam mobile apps - Data driven project management
Kustrzynski, Tomasz
Shazam has been growing very fast recently. A lot of this growth happened in the engineering department at London HQ where our mobile apps are being developed. This talk will go into technicalities of project management techniques we use to keep 100+ MAU a month happy while maintain agility and respond to rapidly changing market situation. We will talk about data driven Kanban, flow, visual standups, changing requirements and structure of our teams. We will show what metrics we care about, how we measure them and what we do with the results. You should expect some observations about development of state of art apps useful from product/project manager’s and developer’s perspective.
 Okay, it's quite a small session. I blame late hour and also the fact that it's not very technical, so it's not the mainstream on this conference. So shall we do this? If you have a question, just raise your hand as I speak and I'll try to elaborate. So my name is Tomasz Kostuniski. I work for Shazam as a technical project manager and I wanted to talk about how we do software at Shazam and what techniques do we use. There's a good chance you are familiar with Shazam as it's fairly popular app. We have more than 500 million downloads globally across iOS and Android. It's been founded in 1999 and launched in the UK in 2002. So in the beginning in 1999, 2002 when we actually went public, this was the method of Shazamming things. You had to call this special number to 580, listen to the music for a while and then wait patiently for the SMS to come back to you after a couple of minutes and you had to pay for this as well, some 20 or 50 pence. Can remember now. Currently, these are obviously two, well more than two, but mobile apps and web service. Last year we got integrated with iOS 8. So if you ask Siri what music is being played, it is now recognized and served by Shazam. This year we just introduced visual Shazam so you can also recognize some items like billboards and QR codes and they give you some extra information about the thing that you Shazam. We also released wearable apps. Shazam is quite a popular app and it's part of the culture, especially in the US where you have a lot of TV commercials which are Shazamable. So we get a lot of tweets like this one for example where someone wanted to search for reception and wait for that she is Shazamming and try to be very helpful by providing actually the wrong information because that get back is by the Beatles not Paul McCartney. But he tried. So today I wanted to talk about the organization as such and how we deal with frequent changes. What ceremonies do we use and perhaps most interestingly how we use data to steer the development, sorry, development process. So let's start with the organizational kind of chart. So this is maybe boring but important for the things that I'll be talking about later. So you can see this vertical boxes which are basically our teams. We have teams per platform so you can see so that would be say iOS server, Android, etc. Each team has a set of developers, testers which work closely with them but testers themselves form a team on its own. So they exchange information on their own. They don't report to Tech Lead, etc. But they are part of the team. They sit with developers. On top of that chart there you can see product managers. These are the guys who are responsible for the product. Every team has more or less dedicated product manager but some product managers are responsible for cross-platform functionalities. For example, all the social related things or all the streaming services would belong to one product manager which makes it easy if you have any question related to this functionality. And then at the bottom there is a team of technical product managers and part of the team. So we are also linked to teams. I work mostly with iOS and Android teams but some of us work across projects which span multiple teams. And our role is to coach, to improve processes and unblock any blockers and resolve all the issues, basically make things happen. And the way we work is we are trying to use a scientific method. Yeah, of course. Yeah. So I actually said that we have, so one product manager as a person would be normally responsible for a specific team like Android but also he would have or he would have responsibilities across the functionality as well. So that if an Android team has a platform specific question, for example, should we use toast or should we display a pop-up dialogue, they can go to their own product manager. But then again, if they have questions specific to a given functionality, for example, streaming music, they would know that the guy who knows everything about music streaming is someone else. So that's the setup that we have. But yes, we have product managers who are across functionalities. Yeah. But we also feel that each team should have one person to go to at all times, to keep consistency of the platform because that's important as well. So the way project managers work is we try to be very scientific and we come up with hypotheses of what is perhaps what could be improved. We perform experiments by changing the process or whatever needs changing, analyze the data and draw conclusions and improve in this way. I put the word kaizen there and that's because we use Kanban. If you are familiar with Kanban, kaizen is Japanese word that means continuous improvement basically. So yeah, we use Kanban for those who are not familiar with it. This means that we visualize the whole workflow on a board. We then limit working progress for each part of the workflow. So we manage the flow by unblocking any problems that occur in any of the stages of the process, define explicit policies so that everyone knows what it means to be done, or def complete, etc. and improve continuously in this way. So the reason why we use Kanban, is that our work is complex and it changes actually so often that any fixed scope iterations basically didn't work for us, even if it's crumb and it's two weeks. So the scope tends to change even more frequently than that. And using Kanban we can accommodate for that. We can be flexible in how often we release stuff for example. We tend to release mobile apps every month, but it happens sometimes that we need to release two times a week. And Kanban allows us to do this. And the third thing which is perhaps, well for me, this is the most interesting thing because visibility and metrics, we see exactly what's happening at any given time and we can calculate some interesting statistics and I'll show you what we actually do to monitor the teams and improve them as well. So I'm not going to dive deep into the theory of Kanban or whatever processes we use because there's dedicated talks for that. While limiting work in progress works, well, you, what matters, the delivery rate is the rate at which you finish things, not start things. So it's all related. Slack is good for efficiency. Well, you can find sources that say that not having ever run busy at 100% of time actually allows the team to react faster, handle unexpected items or events much more effectively. And some people show the busy traffic jammed motorway versus like, you know, easy road with not so many cars. Obviously, you can go faster with on the road which doesn't have traffic. So, yeah, I'm not going to talk too much about theory. I wanted to show you what we actually do. And, yeah, the first thing is how we deal with change. We, so the most important thing that we care about is our users, our customers. We have quite a lot of them. It's currently, I think, about 120 million active users a month. And that means that whenever something goes wrong or there's a need for improvements to a feature which turns out not to perform as well as we hoped for, or even introduce new feature, we need to react very quickly. And the impact of the delay is relatively big. That's why we need to be really agile with responding. And the types of work that we deal with normally is either new functionalities which we develop in-house. So, we have our own schedule. We release them when we think that we are ready when the quality is basically good enough. And we can iterate on them. We have in-house user testing and that gives us feedback so that we can improve the features before releasing. On the other hand, we have these commercial deals which normally would come with a deadline, although quality obviously still matters for this. And they normally come with defined scope, although the practice shows that the interpretation of the scope is often flexible. So, actually, in both of these cases, we have a lot of changes, a lot of requirements that change and can change in the last moment. There's a little digression here. There's a third time of work that we do which is we call it hack time. It's the kind of free time the developers can do whatever they please with. And there's a couple of examples of projects that actually came out of these type of work. We released this Shazam for OSX last year. It was started by a single engineer and then released into, well, productionized within a month. Yeah, it did pretty well in the app store on OSX. And the second example which we released literally two or three weeks ago is this Shazam quiz. There's a URL there you can try. So it plays with the music and you have four options you have to answer. It's quite fun. And it was also idea of just a single person. So this is the third type of work. Right. So this is what changes can, or rapid changes at the last stage of the process can do with developers mind. This is a gift that I found on our developers chat after we failed to release as we planned because of the last minute changes. Right. So last minute changes can be quite destructive. From developers perspective, it creates, it probably creates a rework. They have to redo something that they've already done. You introduce a risk of invalidating your regression tests that perhaps you've already performed. It increases technical debt as last minute work often means introducing some hacks in the project and it lowers morale of the team. So last minute changes are not very good, but they happen. This is the timeline of the project that the gift comes from. I created this graph where you see the light blue boxes, which mean additions to the backlog. And this black, actually I should have said bars, which mean remove us from the backlog. So as you see before the development of this iteration started, we already had the backlog of stuff. We added even more stuff. And then we realized that actually it's too much. We won't do all of this. We removed even more and even more and also added some stories at like really last minute. So what's the lesson from this? To get to the lesson, I'll have another digression. Real options model. So in financial world, this comes from the financial world and it's a model of calculating risk or cost of an option as understood in the financial market. So you have to put and call options. This is the real thing. Black and Sholes got the Nobel Prize for this in 1996. And it's all easy because you can put numbers of this and calculate the price of what is the price of me deferring the decision until the next month, for example. Unfortunately, it's not that easy in software development world. You can't come up with the formula. But this advanced mathematics gives us the three points that we can actually use in software development. So the first thing is options have value. You don't have to make a decision now. People often think that there's just two ways you can make decision. It's either the right decision or wrong decision. But sometimes the third option is to not make a decision now. Waiting gives you the option to do stuff better. Option is expire. There's a time at which you won't be able to do what you want anymore. For example, it may be some law-related requirements. You have a deadline that by the 30th of June you have to do it. Or it may be other things. You won't have any more time to implement something. And finally, never commit early unless you know why. Yeah, as I said, there are cases where you need to commit early because you know that something will happen in the future. But often having an option to not do something is the best option. It's the best way to do things. So in practice what this means is, and that's what we try to use now after this unfortunate iteration which I showed, to not have long queues. This includes back log or bugs, et cetera. Because long queues mean that you've already made the decision to do something. But next week it may not be true anymore. So you wasted your time by making this decision. We design and specify stories and functionalities as we go. So we start implementation usually with rough specification and it gets more detailed as we progress. AB testing, obviously, sometimes it evolves into ABCDE testing. But it's a very valuable thing to have. Totally differs the decision on which way to go. Try to shorten release cycles and irreversible commitments. Once we start doing something, we finish it. So that at least removes the option of not, of canceling what we already started. So in general we have this commitment. Once the developer team started doing something, they have to finish. And we have statistics around things that have been pulled out of development. We try to keep them to zero. Just a short word about ceremonies that we have. We don't have many of them actually. We have many meetings which are at Hock. What we do as ceremony is morning stand-ups where everyone gets synchronized, what is going to happen on this day. We do have retrospectives after iterations. And we have something that we call visual reviews. I'll speak about it in a minute. What we don't do is back-to-rash meetings. Bucks. We don't have cues of you, Bucks. We just fix them or not fix them. Not on the spot, but within very short time. We don't have detailed planning meetings and estimation meetings. I'll tell you how we do estimations in a minute. So this is the visual reviews area. Every day at 5 p.m., developers show what they've done. Because in Shazam, most of the things that we do have some visual impact either on the web or in our mobile apps. So you can see the stuff as it happens. And this gives us a very tight feedback on where we are with implementation. This doesn't happen every day. There are days where developers literally don't have anything to show. But it usually does happen. And the arrangement and colors of the seeds kind of reminded us of terminal. So we call the area terminal 6. We are close to Heathrow and there's just five terminals at Heathrow. So ours is number 6. Right now, what do we measure? Just a couple of quotes from kind of important people. Deming is the guy behind statistical process control and also plan to study act quadrant if you're familiar with the thing. In God we trust all others must bring data. And Drucker as a founder of modern management techniques, what's measured improves. And that's what we try to do. What we try to avoid is hippos. I took this image from Google's how Google works. But it's relevant for us. So hippo is highest paid person's opinion. Hippos also happen to be one of the most dangerous animals in Africa. And everyone knows that. But hippos can run up to almost 50 kilometers per hour. You have no chance. And this is one of the examples of many information radiators that we have across the office. It's kind of taking the terminal 6 metaphor even further. And we have this flight board which shows every epic in the company across all teams and shows whether the epic is in flight or you're already good. Released, et cetera. But we track more things than this. We have a lot of these radiators. I took a couple of snapshots. Every team tracks different data. A server team may be interested in performance of database and servers. The recognition team may be, well, is interested in the rate of recognition that Shazam is getting. Not sure this information is public. So I'm not saying what kind of rates of recognition we are getting. Project managers in particular are interested in teams. So how teams perform. And I'll talk about that. Estimation and forecasting. It's a, it has been a hot topic for a while. Even today I attended one talk where someone was talking about estimation. So we don't do this. And that's pretty much the main reason. Once you say that something may be done by around the week, it, you know, it will be taken for granted and people will stick to this. So even though sometimes we know the dates, we don't publish them internally within the company. What we do do though, this kind of estimation scale that we use, you might have seen this card. They were invented by Lunar Logic. It's a company in Poland. So there are these three cards. That's a very simplified agile poker, right? So you have one. This means the story or item, which is well understood and the right size to get implemented. Then you have two freaking big. We are not going to do this because it's obviously too big. And we, we don't have freaking clue what it is. That's the third option. This thing also doesn't go into development. The fourth optional thing there is just give me a minute. I will do it. That's, you know, a different scale of things to do. So that's what we use. But still, as a project manager, I get asked this question. So yeah, so when this is going to be completed. And it would be kind of rude to say, I'm not telling you. I work in England, so we are very polite. So that's what we do. We came up with this, this name Disney times. So in Disneyland, when you queue for a ride, you are presented with the time that it will take you to finish the ride. Hence the name Disney time. It's also called cycle times in Kanban terminology, for example. So the cycle time is basically a time for a story to get implemented from the start to the end. And having lots of these numbers, these samples, we can come up with some conclusions statistically. That's a timeline where you see several, let's say, developers or tracks of developments. If you use per programming or mop programming, then this will not number of these lines won't be equal to number of developers, obviously. Each story starts implementation and gets finished at some time. And we collect the lens of this development. So these are our cycle times. Once we have a lot of these samples, we can then draw this nice histogram. So the Y axis on this histogram obviously shows the buckets of lens of time. So let's say the first bucket is one day, two days, et cetera. And the Y axis shows the number of stories that got finished within this time. So you can see that the most stories got finished within. Let's assume this is two days. Having this histogram and cumulative density function, we can then say that say 50, so this is 50th percentile, which means that 50% of items got finished before this bucket. That's 85% of items got finished before this time. Having this histogram, I can then come back to a product manager and say, we'll get this story done within that many days with 85% probability. Or if you want to get it within a week, then this is the probability you will get it. Another digression here, which shows our scientific methods in action. The graph that you see there is the median. So this is a 50% out of 50% probability that the item will get finished. And this is tracked on a daily basis. So every day, we calculate how long does it take in average to, or with 50% probability to finish an item. And over time, you can see the kind of performance of the team. If it goes down, that it means that we are suddenly tackling items much faster. And vice versa. So we have this graph, and our hypothesis was that limiting working progress will shorten our cycle times. You familiar with little slow? Okay. So this is another Kanban thing. There's a lot of maths behind this. It can be proven that in a stable system, on average, the cycle time depends on working progress. So basically, how much stuff you are working on at the moment. And it's inverse proportional to throughput. And this, in a simpler words, this means that if you have less items on your board, then in average, it will probably take less time for each of these items to get finished. If you have 100 items, and a given size of the team, then every item in this pack, on average, will spend more time than if you had, say, 10 items. It's kind of simple logic, but it's mathematically also a back tap. So we did that, and we lowered the working progress for one of the teams and said to them, the maximum number of items you can have in development at any time is x, x being, say, 3. And the next day, this is what we saw. Really nice correlated results. This is again the graph of how fast of cycle times, basically, average cycle times. It went down a lot. So that was kind of kind of interesting result, good one. And here's another interesting graph, also nice correlation. You have a consumption of ice creams on the x-axis. This is not in Shazam, by the way. And that's by drowning on the y-axis. There's a nice correlation between these two values, but it doesn't mean that one causes another. So in our case, it was just correlation. It turned out it's just a glitch in the system, which coincidentally manifested itself a day after we introduced the change to the process. So we have to be careful with data and not trust anything blindly. And you have to prove that something is caused by something else. We are pretty sure that our experiment was successful because we tracked it for one more month. And our cycle times actually went down. And we were able to replicate the same in another teams. So we're pretty sure that it was actually a causation eventually. Another question. So as a product manager, I ask you about this one story, but now we have the whole project to implement. So how fast can you deliver that? Again, we use some statistical analysis for that. However, this parallel work would be difficult to model using cycle times. So if our average time of implementing one issue is five days, it does not mean that 10 issues or 10 stories will be implemented in 50 days, right? Because work happens in parallel. So what we use instead of cycle times is called tag times. Tag time is a time distance between finishing off stories in our case. So these are these green time distances at the bottom. It's a rate at which we finish things. So again, we collect many tag times and perform so mathematical magic there. So we have this tag times over there. We calculate average tag time by choosing random tag times from the set on the left side and perform this many times. So we actually not end up with many average tag times, not just one. And this many average tag times can again be presented on a histogram. So if this is getting a bit complicated, then just show. Then having the set of average tag times, we can choose one of these average tag times multiplied by n, where the n is number of issues that we have in the project, for example, 15 stories. Perform this many times again, every time choosing different average tag time. And we end up with another histogram. This time, this histogram shows us the probabilities of expected delivery times. So again, we normally use 85th percentile. So we are happy with 85 percent probability of finishing. And this is basically our answer. The important thing here, obviously, is to have stories or whatever you're estimating or forecasting to be more or less the same size that is difficult to achieve sometimes. So you have to be careful with it again. But once you have a stable team that worked on more or less similar projects in the past and you are sure that you don't have like very big stories in the backlog, then this can work. That's how we answer our questions anyway. You can get these stories or these projects within that many days with this probability. But if you want them within the week, then this is your probability, 0.5, for example. So without explicit estimation, as in Edge, poker and stuff, we can come up with fairly accurate predictions for a project. And they are actually accurate. On some projects, I've got like really, the accuracy of this was down to a day, literally. But again, you have to have proper environment for that. There's some more questions that we get asked. These are all kind of similar questions. So why did we end up with this large number of stories to test at the last minute? Or is there any type of items that take longer time than others? Are the teams more productive in certain times? These kind of questions are around patterns. Being aware of these patterns can be beneficial because you can change them if they are not welcome, or enhance them in other teams if you learn something useful in different teams. So this means in practice that you can introduce different policies. And the example that I gave there, the UI assets for the story have to be ready before the development starts. This is actually one of these policies that we introduced at some point by seeing the patterns. People were basically waiting for stuff, for assets which weren't ready. So this is how we recognize patterns by using control chart. Again, the x-axis is a timeline. The y-axis shows the cycle time. So they hire the bubble on the chart. The longer it took to finish it from the start of the development until ready for production. And the size of the bubble shows the development time itself. Just the development time. Without waiting time, without testing, without testing, pure development. So if you have a small bubble high on the chart, it means that it took long to finish this thing, but it was very simple to implement. Which usually means that there was something wrong in the process. It waited too long. Or testing was very difficult, which may lead to other questions like, why? How can we? So small bubbles high on the chart, we don't like them. And we also don't like big bubbles high on the chart. And that's because this means the story was basically big. Right. So this comparison to one of the motorways, which is very busy in the morning. So the time, the place where the bubble occurs is the time of delivery of this bubble. So you can see that a lot of bubbles or stories got delivered here. Then nothing happens for a while. The more bubbles got delivered here. And we had this stops and, you know, stop and rush, stop and rush. That is a pattern in itself, which is not very healthy for testers, for example, who have to deal with sudden surge of new items. So this is kind of a negative example. I have a better one here as well. You can even visually see that the bubbles are kind of more uniform. Okay. You have a lot of small things, but we don't care about small things that much. They were just fast. Maybe these were some bugs that we quickly fixed or some small tasks. The important thing is that this, there's no like really big bubbles. And they are delivered kind of uniformly. You have these gaps here, but they can be easily explained. We don't expect teams to deliver during weekends. These were literally weekends. Okay. And the last thing they wanted to talk about is dependencies and blockers and how we handle this. So the question here is, as a project manager, I come in the morning, what should I do next? How can I help the team? Just as an illustration, this is five or six boards, six of our Kanban boards. Each board belongs to a different team. And these blue links show dependencies between issues. This is just to illustrate that there's a lot of dependencies. And there may be like all sorts of reasons for this. For example, we're waiting on definitions of requirements. We're waiting for external companies under APIs or SDKs or some commercial stuff or a mobile team, mobile apps team is waiting for server team, et cetera. A lot of this stuff. So what we do with this is we visualize our blockers and dependencies. We prioritize them, act on them, and track how we are doing. And this is how we do it. We have one of these information radios show blockers. So we have this team, team A, B, C, D. You can see that team B has four blockers at the moment. The red color means that it's been blocked more than seven days ago. The green means that it's a recent block within the last two days, I think. And these colorful graphs show you the history. So this is maybe three weeks of history. You can see that team A had some blockers for a short time, but they always resolved them quickly. They don't have any at the moment. Team B always struggled with some red, like, longstanding blockers. So just one look at this shows us where we are. And obviously, we have to draw another histogram. It's here. And it shows durations of blockages. So again, you can read here that about eight items got unblocked within this 20 hours. There was one blocked item which got unblocked within 11 days. So this is bad. And from here, we can read that 80% of items got unblocked within three days. So we have internal standards as to how fast we should react to these blockers. Not all of the blockers are our response, well, it's always our responsibility to try to unblock them, but our influence may be very indirect. For example, we are waiting for the new version of the SDK from Google or something. We can talk to them, but it's outside of our team. So this is all what I wanted to talk about today. Just a quick summary. So we use Kanban because it's flexible. And other methods didn't really work in the past. We embrace the change, but we try to be smart about this and not make decisions too soon, but make them at the last responsible moment. Not too late as well, because this is bad and leads to late changes. We have this daily visual review meeting, which is very useful to see how developers are doing. We make decisions based on data. We use some statistics to come up with estimates and we visualize blockers and try to handle them as soon as possible. So this is all from me. Thank you very much. And if you have any questions, then I think we still have a few minutes. Thank you. Thank you. So using these buttons there.
 Okay, it's quite a small session. I blame late hour and also the fact that it's not very  technical, so it's not the mainstream on this conference. So shall we do this? If you have  a question, just raise your hand as I speak and I'll try to elaborate. So my name is Tomasz  Kostuniski. I work for Shazam as a technical project manager and I wanted to talk about how we do  software at Shazam and what techniques do we use. There's a good chance you are familiar with Shazam  as it's fairly popular app. We have more than 500 million downloads globally across iOS and Android.  It's been founded in 1999 and launched in the UK in 2002. So in the beginning in 1999, 2002 when  we actually went public, this was the method of Shazamming things. You had to call this special  number to 580, listen to the music for a while and then wait patiently for the SMS to come back  to you after a couple of minutes and you had to pay for this as well, some 20 or 50 pence.  Can remember now. Currently, these are obviously two, well more than two, but mobile apps and  web service. Last year we got integrated with iOS 8. So if you ask Siri what music is being played,  it is now recognized and served by Shazam. This year we just introduced visual Shazam so you can  also recognize some items like billboards and QR codes and they give you some extra information  about the thing that you Shazam. We also released wearable apps. Shazam is quite a popular app and  it's part of the culture, especially in the US where you have a lot of TV commercials which are  Shazamable. So we get a lot of tweets like this one for example where someone wanted to  search for reception and wait for that she is Shazamming and try to be very helpful by providing  actually the wrong information because that get back is by the Beatles not Paul McCartney.  But he tried. So today I wanted to talk about the organization as such and how we deal with  frequent changes. What ceremonies do we use and perhaps most interestingly how we use data to  steer the development, sorry, development process. So let's start with the organizational kind of  chart. So this is maybe boring but important for the things that I'll be talking about later.  So you can see this vertical boxes which are basically our teams. We have teams per platform  so you can see so that would be say iOS server, Android, etc. Each team has a set of developers,  testers which work closely with them but testers themselves form a team on its own. So they exchange  information on their own. They don't report to Tech Lead, etc. But they are part of the team.  They sit with developers. On top of that chart there you can see product managers. These are the  guys who are responsible for the product. Every team has more or less dedicated product manager  but some product managers are responsible for cross-platform functionalities. For example,  all the social related things or all the streaming services would belong to one  product manager which makes it easy if you have any question related to this functionality.  And then at the bottom there is a team of technical product managers and part of the team.  So we are also linked to teams. I work mostly with iOS and Android teams  but some of us work across projects which span multiple teams.  And our role is to coach, to improve processes and unblock any blockers and resolve all the issues,  basically make things happen. And the way we work is we are trying to use a scientific method.  Yeah, of course. Yeah. So I actually said that we have,  so one product manager as a person would be normally responsible for a specific team like  Android but also he would have or he would have responsibilities across the functionality as well.  So that if an Android team has a platform specific question, for example, should we use  toast or should we display a pop-up dialogue, they can go to their own product manager.  But then again, if they have questions specific to a given functionality, for example,  streaming music, they would know that the guy who knows everything about music streaming is someone  else. So that's the setup that we have. But yes, we have product managers who are across functionalities.  Yeah. But we also feel that each team should have one person to go to at all times,  to keep consistency of the platform because that's important as well.  So the way project managers work is we try to be very scientific and we come up with  hypotheses of what is perhaps what could be improved. We perform experiments by changing  the process or whatever needs changing, analyze the data and draw conclusions  and improve in this way. I put the word kaizen there and that's because we use Kanban. If you  are familiar with Kanban, kaizen is Japanese word that means continuous improvement basically.  So yeah, we use Kanban for those who are not familiar with it. This means that we visualize  the whole workflow on a board. We then limit working progress for each part of the workflow.  So we manage the flow by unblocking any problems that occur in any of the stages of the process,  define explicit policies so that everyone knows what it means to be done,  or def complete, etc. and improve continuously in this way. So the reason why we use Kanban,  is that our work is complex and it changes actually so often that any fixed scope iterations  basically didn't work for us, even if it's crumb and it's two weeks. So the scope tends to change  even more frequently than that. And using Kanban we can accommodate for that. We can be flexible  in how often we release stuff for example. We tend to release mobile apps every month,  but it happens sometimes that we need to release two times a week. And Kanban allows us to do this.  And the third thing which is perhaps, well for me, this is the most interesting thing because  visibility and metrics, we see exactly what's happening at any given time and we can calculate  some interesting statistics and I'll show you what we actually do to monitor the teams and improve  them as well. So I'm not going to dive deep into the theory of Kanban or whatever processes we use  because there's dedicated talks for that. While limiting work in progress works, well,  you, what matters, the delivery rate is the rate at which you finish things, not start things. So  it's all related. Slack is good for efficiency. Well, you can find sources that say that  not having ever run busy at 100% of time actually allows the team to react faster, handle  unexpected items or events much more effectively. And some people show the busy traffic jammed  motorway versus like, you know, easy road with not so many cars. Obviously, you can go faster  with on the road which doesn't have traffic. So, yeah, I'm not going to talk too much about  theory. I wanted to show you what we actually do. And, yeah, the first thing is how we deal with change.  We, so the most important thing that we care about is our users, our customers. We have quite a lot  of them. It's currently, I think, about 120 million active users a month. And that means that  whenever something goes wrong or there's a need for improvements to a feature which turns out  not to perform as well as we hoped for, or even introduce new feature, we need to react very  quickly. And the impact of the delay is relatively big. That's why we need to be really agile with  responding. And the types of work that we deal with normally is either new functionalities which  we develop in-house. So, we have our own schedule. We release them when we think that we are ready  when the quality is basically good enough. And we can iterate on them. We have in-house user  testing and that gives us feedback so that we can improve the features before releasing. On the other  hand, we have these commercial deals which normally would come with a deadline, although quality  obviously still matters for this. And they normally come with defined scope, although the practice  shows that the interpretation of the scope is often flexible. So, actually, in both of these cases,  we have a lot of changes, a lot of requirements that change and can change in the last moment.  There's a little digression here. There's a third time of work that we do which is we call it hack  time. It's the kind of free time the developers can do whatever they please with. And there's a  couple of examples of projects that actually came out of these type of work. We released this  Shazam for OSX last year. It was started by a single engineer and then released into, well,  productionized within a month. Yeah, it did pretty well in the app store on OSX.  And the second example which we released literally two or three weeks ago is this Shazam quiz.  There's a URL there you can try. So it plays with the music and you have four options you have to  answer. It's quite fun. And it was also idea of just a single person. So this is the third type of  work. Right. So this is what changes can, or rapid changes at the last stage of the process can do  with developers mind. This is a gift that I found on our developers chat after we failed to release  as we planned because of the last minute changes. Right. So last minute changes can be quite  destructive. From developers perspective, it creates, it probably creates a rework. They have  to redo something that they've already done. You introduce a risk of invalidating your regression  tests that perhaps you've already performed. It increases technical debt as last minute work  often means introducing some hacks in the project and it lowers morale of the team.  So last minute changes are not very good, but they happen. This is the timeline of the project  that the gift comes from. I created this graph where you see the light blue boxes,  which mean additions to the backlog. And this black, actually I should have said bars,  which mean remove us from the backlog. So as you see before the development of this iteration  started, we already had the backlog of stuff. We added even more stuff. And then we realized that  actually it's too much. We won't do all of this. We removed even more and even more and also added  some stories at like really last minute. So what's the lesson from this? To get to the lesson,  I'll have another digression. Real options model. So in financial world, this comes from the  financial world and it's a model of calculating risk or cost of an option as understood in the  financial market. So you have to put and call options. This is the real thing. Black and  Sholes got the Nobel Prize for this in 1996. And it's all easy because you can put numbers of this  and calculate the price of what is the price of me deferring the decision until the next month,  for example. Unfortunately, it's not that easy in software development world. You can't come up  with the formula. But this advanced mathematics gives us the three points that we can actually  use in software development. So the first thing is options have value. You don't have to make a  decision now. People often think that there's just two ways you can make decision. It's either  the right decision or wrong decision. But sometimes the third option is to not make a decision now.  Waiting gives you the option to do stuff better. Option is expire. There's a time at which you  won't be able to do what you want anymore. For example, it may be some law-related requirements.  You have a deadline that by the 30th of June you have to do it. Or it may be other things.  You won't have any more time to implement something. And finally, never commit early unless you know  why. Yeah, as I said, there are cases where you need to commit early because you know that something  will happen in the future. But often having an option to not do something is the best option.  It's the best way to do things. So in practice what this means is, and that's what we try to use  now after this unfortunate iteration which I showed, to not have long queues. This includes  back log or bugs, et cetera. Because long queues mean that you've already made the decision to do  something. But next week it may not be true anymore. So you wasted your time by making this decision.  We design and specify stories and functionalities as we go. So we start implementation usually  with rough specification and it gets more detailed as we progress. AB testing, obviously, sometimes  it evolves into ABCDE testing. But it's a very valuable thing to have. Totally differs the decision  on which way to go. Try to shorten release cycles and irreversible commitments. Once we start doing  something, we finish it. So that at least removes the option of not, of canceling what we already  started. So in general we have this commitment. Once the developer team started doing something,  they have to finish. And we have statistics around things that have been pulled out of  development. We try to keep them to zero. Just a short word about ceremonies that we  have. We don't have many of them actually. We have many meetings which are at Hock.  What we do as ceremony is morning stand-ups where everyone gets synchronized, what is going to  happen on this day. We do have retrospectives after iterations. And we have something that  we call visual reviews. I'll speak about it in a minute. What we don't do  is back-to-rash meetings. Bucks. We don't have cues of you, Bucks. We just fix them or not fix them.  Not on the spot, but within very short time. We don't have detailed planning meetings and  estimation meetings. I'll tell you how we do estimations in a minute. So this is the visual  reviews area. Every day at 5 p.m., developers show what they've done. Because in Shazam,  most of the things that we do have some visual impact either on the web or in our mobile apps.  So you can see the stuff as it happens. And this gives us a very tight feedback on where we are  with implementation. This doesn't happen every day. There are days where developers literally  don't have anything to show. But it usually does happen. And the arrangement and colors of the  seeds kind of reminded us of terminal. So we call the area terminal 6. We are close to  Heathrow and there's just five terminals at Heathrow. So ours is number 6. Right now, what do we  measure? Just a couple of quotes from kind of important people. Deming is the guy behind  statistical process control and also plan to study act quadrant if you're familiar with the thing.  In God we trust all others must bring data. And Drucker as a founder of modern management  techniques, what's measured improves. And that's what we try to do. What we try to avoid is hippos.  I took this image from Google's how Google works. But it's relevant for us. So hippo is highest  paid person's opinion. Hippos also happen to be one of the most dangerous animals in Africa.  And everyone knows that. But hippos can run up to almost 50 kilometers per hour. You have no chance.  And this is one of the examples of many information radiators that we have across the office.  It's kind of taking the terminal 6 metaphor even further. And we have this flight board which shows  every epic in the company across all teams and shows whether the epic is in flight or you're  already good. Released, et cetera. But we track more things than this. We have a lot of these  radiators. I took a couple of snapshots. Every team tracks different data. A server team may be  interested in performance of database and servers. The recognition team may be, well, is interested  in the rate of recognition that Shazam is getting. Not sure this information is public. So I'm not  saying what kind of rates of recognition we are getting. Project managers in particular are  interested in teams. So how teams perform. And I'll talk about that. Estimation and forecasting.  It's a, it has been a hot topic for a while. Even today I attended one talk where  someone was talking about estimation. So we don't do this. And that's pretty much the main reason.  Once you say that something may be done by around the week, it, you know, it will be taken for  granted and people will stick to this. So even though sometimes we know the dates, we don't  publish them internally within the company. What we do do though, this kind of estimation scale  that we use, you might have seen this card. They were invented by Lunar Logic. It's a company in  Poland. So there are these three cards. That's a very simplified agile poker, right? So you have  one. This means the story or item, which is well understood and the right size to get implemented.  Then you have two freaking big. We are not going to do this because it's obviously too big. And  we, we don't have freaking clue what it is. That's the third option. This thing also doesn't go into  development. The fourth optional thing there is just give me a minute. I will do it. That's, you  know, a different scale of things to do. So that's what we use. But still, as a project manager,  I get asked this question. So yeah, so when this is going to be completed. And it would be kind of  rude to say, I'm not telling you. I work in England, so we are very polite. So that's what we do.  We came up with this, this name Disney times. So in Disneyland, when you queue for a ride,  you are presented with the time that it will take you to finish the ride. Hence the name  Disney time. It's also called cycle times in Kanban terminology, for example. So the cycle  time is basically a time for a story to get implemented from the start to the end.  And having lots of these numbers, these samples, we can come up with some conclusions statistically.  That's a timeline where you see several, let's say, developers or tracks of developments. If you  use per programming or mop programming, then this will not number of these lines won't be equal to  number of developers, obviously. Each story starts implementation and gets finished at some time.  And we collect the lens of this development. So these are our cycle times. Once we have a lot of  these samples, we can then draw this nice histogram. So the Y axis on this histogram  obviously shows the buckets of lens of time. So let's say the first bucket is one day, two days,  et cetera. And the Y axis shows the number of stories that got finished within this time.  So you can see that the most stories got finished within. Let's assume this is two days.  Having this histogram and cumulative density function, we can then say that say 50,  so this is 50th percentile, which means that 50% of items got finished before this bucket.  That's 85% of items got finished before this time. Having this histogram, I can then come back to a  product manager and say, we'll get this story done within that many days with 85% probability.  Or if you want to get it within a week, then this is the probability you will get it.  Another digression here, which shows our scientific methods in action. The graph that you see there  is the median. So this is a 50% out of 50% probability that the item will get finished.  And this is tracked on a daily basis. So every day, we calculate how long does it take in average  to, or with 50% probability to finish an item. And over time, you can see the kind of performance  of the team. If it goes down, that it means that we are suddenly tackling items much faster.  And vice versa. So we have this graph, and our hypothesis was that limiting  working progress will shorten our cycle times. You familiar with little slow?  Okay. So this is another Kanban thing. There's a lot of maths behind this. It can be proven  that in a stable system, on average, the cycle time depends on working progress. So basically,  how much stuff you are working on at the moment. And it's inverse proportional to throughput.  And this, in a simpler words, this means that if you have less items on your board,  then in average, it will probably take less time for each of these items to get finished.  If you have 100 items, and a given size of the team, then every item in this pack, on average,  will spend more time than if you had, say, 10 items. It's kind of simple logic, but it's mathematically  also a back tap. So we did that, and we lowered the working progress for one of the teams and said  to them, the maximum number of items you can have in development at any time is x, x being, say, 3.  And the next day, this is what we saw. Really nice correlated results. This is again the graph of  how fast of cycle times, basically, average cycle times. It went down a lot. So that was kind of  kind of interesting result, good one. And here's another interesting graph, also nice correlation.  You have a consumption of ice creams on the x-axis. This is not in Shazam, by the way. And  that's by drowning on the y-axis. There's a nice correlation between these two values,  but it doesn't mean that one causes another. So in our case, it was just correlation. It turned out  it's just a glitch in the system, which coincidentally manifested itself a day after we introduced  the change to the process. So we have to be careful with data and not trust anything blindly. And you  have to prove that something is caused by something else. We are pretty sure that our experiment was  successful because we tracked it for one more month. And our cycle times actually went down.  And we were able to replicate the same in another teams. So we're pretty sure that it was actually  a causation eventually. Another question. So as a product manager, I ask you about this one story,  but now we have the whole project to implement. So how fast can you deliver that? Again,  we use some statistical analysis for that. However, this parallel work would be difficult to model  using cycle times. So if our average time of implementing one issue is five days, it does  not mean that 10 issues or 10 stories will be implemented in 50 days, right? Because work happens  in parallel. So what we use instead of cycle times is called tag times. Tag time is a time  distance between finishing off stories in our case. So these are these green time distances  at the bottom. It's a rate at which we finish things. So again, we collect many  tag times and perform so mathematical magic there. So we have this tag times over there.  We calculate average tag time by choosing random tag times from the set on the left side  and perform this many times. So we actually not end up with many average tag times, not just one.  And this many average tag times can again be presented on a histogram.  So if this is getting a bit complicated, then just show. Then having the set of average tag times,  we can choose one of these average tag times multiplied by n, where the n is number of issues  that we have in the project, for example, 15 stories. Perform this many times again,  every time choosing different average tag time. And we end up with another histogram.  This time, this histogram shows us the probabilities of expected delivery times.  So again, we normally use 85th percentile. So we are happy with 85 percent probability  of finishing. And this is basically our answer. The important thing here, obviously, is to  have stories or whatever you're estimating or forecasting to be more or less the same size  that is difficult to achieve sometimes. So you have to be careful with it again.  But once you have a stable team that worked on more or less similar projects in the past  and you are sure that you don't have like very big stories in the backlog, then this can work.  That's how we answer our questions anyway. You can get these stories or these projects  within that many days with this probability. But if you want them within the week,  then this is your probability, 0.5, for example. So without explicit estimation,  as in Edge, poker and stuff, we can come up with fairly accurate predictions for a project.  And they are actually accurate. On some projects, I've got like really,  the accuracy of this was down to a day, literally. But again, you have to have proper environment  for that. There's some more questions that we get asked. These are all kind of similar questions.  So why did we end up with this large number of stories to test at the last minute? Or is there  any type of items that take longer time than others? Are the teams more productive in certain times?  These kind of questions are around patterns. Being aware of these patterns  can be beneficial because you can change them if they are not welcome,  or enhance them in other teams if you learn something useful in different teams.  So this means in practice that you can introduce different policies. And the example that I gave  there, the UI assets for the story have to be ready before the development starts.  This is actually one of these policies that we introduced at some point by seeing the patterns.  People were basically waiting for stuff, for assets which weren't ready.  So this is how we recognize patterns by using control chart. Again, the x-axis is a timeline.  The y-axis shows the cycle time. So they hire the bubble on the chart. The longer it took to finish  it from the start of the development until ready for production. And the size of the bubble shows  the development time itself. Just the development time. Without waiting time, without testing,  without testing, pure development. So if you have a small bubble high on the chart,  it means that it took long to finish this thing, but it was very simple to implement.  Which usually means that there was something wrong in the process. It waited too long.  Or testing was very difficult, which may lead to other questions like, why? How can we?  So small bubbles high on the chart, we don't like them. And we also don't like big bubbles  high on the chart. And that's because this means the story was basically big.  Right. So this comparison to one of the motorways, which is very busy in the morning.  So the time, the place where the bubble occurs is the time of delivery of this bubble.  So you can see that a lot of bubbles or stories got delivered here.  Then nothing happens for a while. The more bubbles got delivered here. And we had this  stops and, you know, stop and rush, stop and rush. That is a pattern in itself,  which is not very healthy for testers, for example, who have to deal with sudden surge of  new items. So this is kind of a negative example. I have a better one here as well.  You can even visually see that the bubbles are kind of more uniform. Okay. You have a lot of  small things, but we don't care about small things that much. They were just fast. Maybe  these were some bugs that we quickly fixed or some small tasks. The important thing is that  this, there's no like really big bubbles. And they are delivered kind of uniformly. You have  these gaps here, but they can be easily explained. We don't expect teams to deliver during weekends.  These were literally weekends. Okay. And the last thing they wanted to talk about  is dependencies and blockers and how we handle this. So the question here is, as a project manager,  I come in the morning, what should I do next? How can I help the team? Just as an illustration,  this is five or six boards, six of our Kanban boards. Each board belongs to a different team.  And these blue links show dependencies between issues. This is just to illustrate that there's  a lot of dependencies. And there may be like all sorts of reasons for this. For example,  we're waiting on definitions of requirements. We're waiting for external companies under APIs  or SDKs or some commercial stuff or a mobile team, mobile apps team is waiting for server  team, et cetera. A lot of this stuff. So what we do with this is we visualize our blockers  and dependencies. We prioritize them, act on them, and track how we are doing.  And this is how we do it. We have one of these information radios show blockers. So we have  this team, team A, B, C, D. You can see that team B has four blockers at the moment.  The red color means that it's been blocked more than seven days ago. The green means that it's  a recent block within the last two days, I think. And these colorful graphs show you the history.  So this is maybe three weeks of history. You can see that team A had some blockers for a short  time, but they always resolved them quickly. They don't have any at the moment. Team B always  struggled with some red, like, longstanding blockers. So just one look at this shows us where we are.  And obviously, we have to draw another histogram. It's here. And it shows  durations of blockages. So again, you can read here that about eight items got unblocked within  this 20 hours. There was one blocked item which got unblocked within 11 days.  So this is bad. And from here, we can read that 80% of items got unblocked within three days.  So we have internal standards as to how fast we should react to these blockers. Not all of the  blockers are our response, well, it's always our responsibility to try to unblock them, but  our influence may be very indirect. For example, we are waiting for the new version of the SDK from  Google or something. We can talk to them, but it's outside of our team. So this is all what I  wanted to talk about today. Just a quick summary. So we use Kanban because it's flexible.  And other methods didn't really work in the past. We embrace the change, but we try to be smart  about this and not make decisions too soon, but make them at the last responsible moment. Not too  late as well, because this is bad and leads to late changes. We have this daily visual review  meeting, which is very useful to see how developers are doing. We make decisions based on data.  We use some statistics to come up with estimates and we visualize blockers  and try to handle them as soon as possible. So this is all from me. Thank you very much.  And if you have any questions, then I think we still have a few minutes.  Thank you.  Thank you.  So using these buttons there.
Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_0_959.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_960_1511.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_1512_2315.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_2316_3347.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_3348_4139.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_4140_4667.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_4668_7655.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_7656_8819.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_8820_10391.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_10392_11243.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_11244_12419.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_12420_14087.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_14088_15839.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_15840_15971.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_15972_17039.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_17040_18323.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_18324_18503.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_18504_18935.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_18936_19403.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_19404_19523.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_19524_20099.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_20100_20231.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_20232_20399.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_20400_20735.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_20736_20903.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_20904_21071.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_21072_21251.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_21252_21419.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_21420_21683.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_21684_23075.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_23076_24359.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_24360_26255.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_26256_28703.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_28704_28751.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_28752_30059.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_30060_31391.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_31392_31475.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_31476_32243.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_32244_33107.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_33108_34019.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_34020_34955.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_34956_35411.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_35412_36107.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_36108_37703.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_37704_38243.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_38244_39431.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_39432_40403.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_40404_41879.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_41880_42347.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_42348_43607.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_43608_45599.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_45600_46535.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_46536_47219.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_47220_48359.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_48360_48719.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_48720_50279.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_50280_50999.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_51000_51371.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_51372_53723.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_53724_54827.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_54828_55751.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_55752_56927.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_56928_60191.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_60192_61379.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_61380_61847.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_61848_61955.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_61956_62051.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62052_62303.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62304_62399.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62400_62495.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62496_62579.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62580_62651.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62652_62747.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62748_62855.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62856_62927.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_62928_62999.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_63000_63095.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_63096_63491.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_63492_64919.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_64920_66899.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_66900_68267.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_68268_69203.png Shazam_mobile_apps_-_data_driven_project_management_-_Tomasz_Kustrzynski_69204_69356.png
Okay, it's quite a small session. I blame late hour and also the fact that it's not very technical, so it's not the mainstream on this conference. So shall we do this? If you have a question, just raise your hand as I speak and I'll try to elaborate. So my name is Tomasz Kostuniski. I work for Shazam as a technical project manager and I wanted to talk about how we do software at Shazam and what techniques do we use. There's a good chance you are familiar with Shazam as it's fairly popular app. We have more than 500 million downloads globally across iOS and Android. It's been founded in 1999 and launched in the UK in 2002. So in the beginning in 1999, 2002 when we actually went public, this was the method of Shazamming things. You had to call this special number to 580, listen to the music for a while and then wait patiently for the SMS to come back to you after a couple of minutes and you had to pay for this as well, some 20 or 50 pence. Can remember now. Currently, these are obviously two, well more than two, but mobile apps and web service. Last year we got integrated with iOS 8. So if you ask Siri what music is being played, it is now recognized and served by Shazam. This year we just introduced visual Shazam so you can also recognize some items like billboards and QR codes and they give you some extra information about the thing that you Shazam. We also released wearable apps. Shazam is quite a popular app and it's part of the culture, especially in the US where you have a lot of TV commercials which are Shazamable. So we get a lot of tweets like this one for example where someone wanted to search for reception and wait for that she is Shazamming and try to be very helpful by providing actually the wrong information because that get back is by the Beatles not Paul McCartney. But he tried. So today I wanted to talk about the organization as such and how we deal with frequent changes. What ceremonies do we use and perhaps most interestingly how we use data to steer the development, sorry, development process. So let's start with the organizational kind of chart. So this is maybe boring but important for the things that I'll be talking about later. So you can see this vertical boxes which are basically our teams. We have teams per platform so you can see so that would be say iOS server, Android, etc. Each team has a set of developers, testers which work closely with them but testers themselves form a team on its own. So they exchange information on their own. They don't report to Tech Lead, etc. But they are part of the team. They sit with developers. On top of that chart there you can see product managers. These are the guys who are responsible for the product. Every team has more or less dedicated product manager but some product managers are responsible for cross-platform functionalities. For example, all the social related things or all the streaming services would belong to one product manager which makes it easy if you have any question related to this functionality. And then at the bottom there is a team of technical product managers and part of the team. So we are also linked to teams. I work mostly with iOS and Android teams but some of us work across projects which span multiple teams. And our role is to coach, to improve processes and unblock any blockers and resolve all the issues, basically make things happen. And the way we work is we are trying to use a scientific method. Yeah, of course. Yeah. So I actually said that we have, so one product manager as a person would be normally responsible for a specific team like Android but also he would have or he would have responsibilities across the functionality as well. So that if an Android team has a platform specific question, for example, should we use toast or should we display a pop-up dialogue, they can go to their own product manager. But then again, if they have questions specific to a given functionality, for example, streaming music, they would know that the guy who knows everything about music streaming is someone else. So that's the setup that we have. But yes, we have product managers who are across functionalities. Yeah. But we also feel that each team should have one person to go to at all times, to keep consistency of the platform because that's important as well. So the way project managers work is we try to be very scientific and we come up with hypotheses of what is perhaps what could be improved. We perform experiments by changing the process or whatever needs changing, analyze the data and draw conclusions and improve in this way. I put the word kaizen there and that's because we use Kanban. If you are familiar with Kanban, kaizen is Japanese word that means continuous improvement basically. So yeah, we use Kanban for those who are not familiar with it. This means that we visualize the whole workflow on a board. We then limit working progress for each part of the workflow. So we manage the flow by unblocking any problems that occur in any of the stages of the process, define explicit policies so that everyone knows what it means to be done, or def complete, etc. and improve continuously in this way. So the reason why we use Kanban, is that our work is complex and it changes actually so often that any fixed scope iterations basically didn't work for us, even if it's crumb and it's two weeks. So the scope tends to change even more frequently than that. And using Kanban we can accommodate for that. We can be flexible in how often we release stuff for example. We tend to release mobile apps every month, but it happens sometimes that we need to release two times a week. And Kanban allows us to do this. And the third thing which is perhaps, well for me, this is the most interesting thing because visibility and metrics, we see exactly what's happening at any given time and we can calculate some interesting statistics and I'll show you what we actually do to monitor the teams and improve them as well. So I'm not going to dive deep into the theory of Kanban or whatever processes we use because there's dedicated talks for that. While limiting work in progress works, well, you, what matters, the delivery rate is the rate at which you finish things, not start things. So it's all related. Slack is good for efficiency. Well, you can find sources that say that not having ever run busy at 100% of time actually allows the team to react faster, handle unexpected items or events much more effectively. And some people show the busy traffic jammed motorway versus like, you know, easy road with not so many cars. Obviously, you can go faster with on the road which doesn't have traffic. So, yeah, I'm not going to talk too much about theory. I wanted to show you what we actually do. And, yeah, the first thing is how we deal with change. We, so the most important thing that we care about is our users, our customers. We have quite a lot of them. It's currently, I think, about 120 million active users a month. And that means that whenever something goes wrong or there's a need for improvements to a feature which turns out not to perform as well as we hoped for, or even introduce new feature, we need to react very quickly. And the impact of the delay is relatively big. That's why we need to be really agile with responding. And the types of work that we deal with normally is either new functionalities which we develop in-house. So, we have our own schedule. We release them when we think that we are ready when the quality is basically good enough. And we can iterate on them. We have in-house user testing and that gives us feedback so that we can improve the features before releasing. On the other hand, we have these commercial deals which normally would come with a deadline, although quality obviously still matters for this. And they normally come with defined scope, although the practice shows that the interpretation of the scope is often flexible. So, actually, in both of these cases, we have a lot of changes, a lot of requirements that change and can change in the last moment. There's a little digression here. There's a third time of work that we do which is we call it hack time. It's the kind of free time the developers can do whatever they please with. And there's a couple of examples of projects that actually came out of these type of work. We released this Shazam for OSX last year. It was started by a single engineer and then released into, well, productionized within a month. Yeah, it did pretty well in the app store on OSX. And the second example which we released literally two or three weeks ago is this Shazam quiz. There's a URL there you can try. So it plays with the music and you have four options you have to answer. It's quite fun. And it was also idea of just a single person. So this is the third type of work. Right. So this is what changes can, or rapid changes at the last stage of the process can do with developers mind. This is a gift that I found on our developers chat after we failed to release as we planned because of the last minute changes. Right. So last minute changes can be quite destructive. From developers perspective, it creates, it probably creates a rework. They have to redo something that they've already done. You introduce a risk of invalidating your regression tests that perhaps you've already performed. It increases technical debt as last minute work often means introducing some hacks in the project and it lowers morale of the team. So last minute changes are not very good, but they happen. This is the timeline of the project that the gift comes from. I created this graph where you see the light blue boxes, which mean additions to the backlog. And this black, actually I should have said bars, which mean remove us from the backlog. So as you see before the development of this iteration started, we already had the backlog of stuff. We added even more stuff. And then we realized that actually it's too much. We won't do all of this. We removed even more and even more and also added some stories at like really last minute. So what's the lesson from this? To get to the lesson, I'll have another digression. Real options model. So in financial world, this comes from the financial world and it's a model of calculating risk or cost of an option as understood in the financial market. So you have to put and call options. This is the real thing. Black and Sholes got the Nobel Prize for this in 1996. And it's all easy because you can put numbers of this and calculate the price of what is the price of me deferring the decision until the next month, for example. Unfortunately, it's not that easy in software development world. You can't come up with the formula. But this advanced mathematics gives us the three points that we can actually use in software development. So the first thing is options have value. You don't have to make a decision now. People often think that there's just two ways you can make decision. It's either the right decision or wrong decision. But sometimes the third option is to not make a decision now. Waiting gives you the option to do stuff better. Option is expire. There's a time at which you won't be able to do what you want anymore. For example, it may be some law-related requirements. You have a deadline that by the 30th of June you have to do it. Or it may be other things. You won't have any more time to implement something. And finally, never commit early unless you know why. Yeah, as I said, there are cases where you need to commit early because you know that something will happen in the future. But often having an option to not do something is the best option. It's the best way to do things. So in practice what this means is, and that's what we try to use now after this unfortunate iteration which I showed, to not have long queues. This includes back log or bugs, et cetera. Because long queues mean that you've already made the decision to do something. But next week it may not be true anymore. So you wasted your time by making this decision. We design and specify stories and functionalities as we go. So we start implementation usually with rough specification and it gets more detailed as we progress. AB testing, obviously, sometimes it evolves into ABCDE testing. But it's a very valuable thing to have. Totally differs the decision on which way to go. Try to shorten release cycles and irreversible commitments. Once we start doing something, we finish it. So that at least removes the option of not, of canceling what we already started. So in general we have this commitment. Once the developer team started doing something, they have to finish. And we have statistics around things that have been pulled out of development. We try to keep them to zero. Just a short word about ceremonies that we have. We don't have many of them actually. We have many meetings which are at Hock. What we do as ceremony is morning stand-ups where everyone gets synchronized, what is going to happen on this day. We do have retrospectives after iterations. And we have something that we call visual reviews. I'll speak about it in a minute. What we don't do is back-to-rash meetings. Bucks. We don't have cues of you, Bucks. We just fix them or not fix them. Not on the spot, but within very short time. We don't have detailed planning meetings and estimation meetings. I'll tell you how we do estimations in a minute. So this is the visual reviews area. Every day at 5 p.m., developers show what they've done. Because in Shazam, most of the things that we do have some visual impact either on the web or in our mobile apps. So you can see the stuff as it happens. And this gives us a very tight feedback on where we are with implementation. This doesn't happen every day. There are days where developers literally don't have anything to show. But it usually does happen. And the arrangement and colors of the seeds kind of reminded us of terminal. So we call the area terminal 6. We are close to Heathrow and there's just five terminals at Heathrow. So ours is number 6. Right now, what do we measure? Just a couple of quotes from kind of important people. Deming is the guy behind statistical process control and also plan to study act quadrant if you're familiar with the thing. In God we trust all others must bring data. And Drucker as a founder of modern management techniques, what's measured improves. And that's what we try to do. What we try to avoid is hippos. I took this image from Google's how Google works. But it's relevant for us. So hippo is highest paid person's opinion. Hippos also happen to be one of the most dangerous animals in Africa. And everyone knows that. But hippos can run up to almost 50 kilometers per hour. You have no chance. And this is one of the examples of many information radiators that we have across the office. It's kind of taking the terminal 6 metaphor even further. And we have this flight board which shows every epic in the company across all teams and shows whether the epic is in flight or you're already good. Released, et cetera. But we track more things than this. We have a lot of these radiators. I took a couple of snapshots. Every team tracks different data. A server team may be interested in performance of database and servers. The recognition team may be, well, is interested in the rate of recognition that Shazam is getting. Not sure this information is public. So I'm not saying what kind of rates of recognition we are getting. Project managers in particular are interested in teams. So how teams perform. And I'll talk about that. Estimation and forecasting. It's a, it has been a hot topic for a while. Even today I attended one talk where someone was talking about estimation. So we don't do this. And that's pretty much the main reason. Once you say that something may be done by around the week, it, you know, it will be taken for granted and people will stick to this. So even though sometimes we know the dates, we don't publish them internally within the company. What we do do though, this kind of estimation scale that we use, you might have seen this card. They were invented by Lunar Logic. It's a company in Poland. So there are these three cards. That's a very simplified agile poker, right? So you have one. This means the story or item, which is well understood and the right size to get implemented. Then you have two freaking big. We are not going to do this because it's obviously too big. And we, we don't have freaking clue what it is. That's the third option. This thing also doesn't go into development. The fourth optional thing there is just give me a minute. I will do it. That's, you know, a different scale of things to do. So that's what we use. But still, as a project manager, I get asked this question. So yeah, so when this is going to be completed. And it would be kind of rude to say, I'm not telling you. I work in England, so we are very polite. So that's what we do. We came up with this, this name Disney times. So in Disneyland, when you queue for a ride, you are presented with the time that it will take you to finish the ride. Hence the name Disney time. It's also called cycle times in Kanban terminology, for example. So the cycle time is basically a time for a story to get implemented from the start to the end. And having lots of these numbers, these samples, we can come up with some conclusions statistically. That's a timeline where you see several, let's say, developers or tracks of developments. If you use per programming or mop programming, then this will not number of these lines won't be equal to number of developers, obviously. Each story starts implementation and gets finished at some time. And we collect the lens of this development. So these are our cycle times. Once we have a lot of these samples, we can then draw this nice histogram. So the Y axis on this histogram obviously shows the buckets of lens of time. So let's say the first bucket is one day, two days, et cetera. And the Y axis shows the number of stories that got finished within this time. So you can see that the most stories got finished within. Let's assume this is two days. Having this histogram and cumulative density function, we can then say that say 50, so this is 50th percentile, which means that 50% of items got finished before this bucket. That's 85% of items got finished before this time. Having this histogram, I can then come back to a product manager and say, we'll get this story done within that many days with 85% probability. Or if you want to get it within a week, then this is the probability you will get it. Another digression here, which shows our scientific methods in action. The graph that you see there is the median. So this is a 50% out of 50% probability that the item will get finished. And this is tracked on a daily basis. So every day, we calculate how long does it take in average to, or with 50% probability to finish an item. And over time, you can see the kind of performance of the team. If it goes down, that it means that we are suddenly tackling items much faster. And vice versa. So we have this graph, and our hypothesis was that limiting working progress will shorten our cycle times. You familiar with little slow? Okay. So this is another Kanban thing. There's a lot of maths behind this. It can be proven that in a stable system, on average, the cycle time depends on working progress. So basically, how much stuff you are working on at the moment. And it's inverse proportional to throughput. And this, in a simpler words, this means that if you have less items on your board, then in average, it will probably take less time for each of these items to get finished. If you have 100 items, and a given size of the team, then every item in this pack, on average, will spend more time than if you had, say, 10 items. It's kind of simple logic, but it's mathematically also a back tap. So we did that, and we lowered the working progress for one of the teams and said to them, the maximum number of items you can have in development at any time is x, x being, say, 3. And the next day, this is what we saw. Really nice correlated results. This is again the graph of how fast of cycle times, basically, average cycle times. It went down a lot. So that was kind of kind of interesting result, good one. And here's another interesting graph, also nice correlation. You have a consumption of ice creams on the x-axis. This is not in Shazam, by the way. And that's by drowning on the y-axis. There's a nice correlation between these two values, but it doesn't mean that one causes another. So in our case, it was just correlation. It turned out it's just a glitch in the system, which coincidentally manifested itself a day after we introduced the change to the process. So we have to be careful with data and not trust anything blindly. And you have to prove that something is caused by something else. We are pretty sure that our experiment was successful because we tracked it for one more month. And our cycle times actually went down. And we were able to replicate the same in another teams. So we're pretty sure that it was actually a causation eventually. Another question. So as a product manager, I ask you about this one story, but now we have the whole project to implement. So how fast can you deliver that? Again, we use some statistical analysis for that. However, this parallel work would be difficult to model using cycle times. So if our average time of implementing one issue is five days, it does not mean that 10 issues or 10 stories will be implemented in 50 days, right? Because work happens in parallel. So what we use instead of cycle times is called tag times. Tag time is a time distance between finishing off stories in our case. So these are these green time distances at the bottom. It's a rate at which we finish things. So again, we collect many tag times and perform so mathematical magic there. So we have this tag times over there. We calculate average tag time by choosing random tag times from the set on the left side and perform this many times. So we actually not end up with many average tag times, not just one. And this many average tag times can again be presented on a histogram. So if this is getting a bit complicated, then just show. Then having the set of average tag times, we can choose one of these average tag times multiplied by n, where the n is number of issues that we have in the project, for example, 15 stories. Perform this many times again, every time choosing different average tag time. And we end up with another histogram. This time, this histogram shows us the probabilities of expected delivery times. So again, we normally use 85th percentile. So we are happy with 85 percent probability of finishing. And this is basically our answer. The important thing here, obviously, is to have stories or whatever you're estimating or forecasting to be more or less the same size that is difficult to achieve sometimes. So you have to be careful with it again. But once you have a stable team that worked on more or less similar projects in the past and you are sure that you don't have like very big stories in the backlog, then this can work. That's how we answer our questions anyway. You can get these stories or these projects within that many days with this probability. But if you want them within the week, then this is your probability, 0.5, for example. So without explicit estimation, as in Edge, poker and stuff, we can come up with fairly accurate predictions for a project. And they are actually accurate. On some projects, I've got like really, the accuracy of this was down to a day, literally. But again, you have to have proper environment for that. There's some more questions that we get asked. These are all kind of similar questions. So why did we end up with this large number of stories to test at the last minute? Or is there any type of items that take longer time than others? Are the teams more productive in certain times? These kind of questions are around patterns. Being aware of these patterns can be beneficial because you can change them if they are not welcome, or enhance them in other teams if you learn something useful in different teams. So this means in practice that you can introduce different policies. And the example that I gave there, the UI assets for the story have to be ready before the development starts. This is actually one of these policies that we introduced at some point by seeing the patterns. People were basically waiting for stuff, for assets which weren't ready. So this is how we recognize patterns by using control chart. Again, the x-axis is a timeline. The y-axis shows the cycle time. So they hire the bubble on the chart. The longer it took to finish it from the start of the development until ready for production. And the size of the bubble shows the development time itself. Just the development time. Without waiting time, without testing, without testing, pure development. So if you have a small bubble high on the chart, it means that it took long to finish this thing, but it was very simple to implement. Which usually means that there was something wrong in the process. It waited too long. Or testing was very difficult, which may lead to other questions like, why? How can we? So small bubbles high on the chart, we don't like them. And we also don't like big bubbles high on the chart. And that's because this means the story was basically big. Right. So this comparison to one of the motorways, which is very busy in the morning. So the time, the place where the bubble occurs is the time of delivery of this bubble. So you can see that a lot of bubbles or stories got delivered here. Then nothing happens for a while. The more bubbles got delivered here. And we had this stops and, you know, stop and rush, stop and rush. That is a pattern in itself, which is not very healthy for testers, for example, who have to deal with sudden surge of new items. So this is kind of a negative example. I have a better one here as well. You can even visually see that the bubbles are kind of more uniform. Okay. You have a lot of small things, but we don't care about small things that much. They were just fast. Maybe these were some bugs that we quickly fixed or some small tasks. The important thing is that this, there's no like really big bubbles. And they are delivered kind of uniformly. You have these gaps here, but they can be easily explained. We don't expect teams to deliver during weekends. These were literally weekends. Okay. And the last thing they wanted to talk about is dependencies and blockers and how we handle this. So the question here is, as a project manager, I come in the morning, what should I do next? How can I help the team? Just as an illustration, this is five or six boards, six of our Kanban boards. Each board belongs to a different team. And these blue links show dependencies between issues. This is just to illustrate that there's a lot of dependencies. And there may be like all sorts of reasons for this. For example, we're waiting on definitions of requirements. We're waiting for external companies under APIs or SDKs or some commercial stuff or a mobile team, mobile apps team is waiting for server team, et cetera. A lot of this stuff. So what we do with this is we visualize our blockers and dependencies. We prioritize them, act on them, and track how we are doing. And this is how we do it. We have one of these information radios show blockers. So we have this team, team A, B, C, D. You can see that team B has four blockers at the moment. The red color means that it's been blocked more than seven days ago. The green means that it's a recent block within the last two days, I think. And these colorful graphs show you the history. So this is maybe three weeks of history. You can see that team A had some blockers for a short time, but they always resolved them quickly. They don't have any at the moment. Team B always struggled with some red, like, longstanding blockers. So just one look at this shows us where we are. And obviously, we have to draw another histogram. It's here. And it shows durations of blockages. So again, you can read here that about eight items got unblocked within this 20 hours. There was one blocked item which got unblocked within 11 days. So this is bad. And from here, we can read that 80% of items got unblocked within three days. So we have internal standards as to how fast we should react to these blockers. Not all of the blockers are our response, well, it's always our responsibility to try to unblock them, but our influence may be very indirect. For example, we are waiting for the new version of the SDK from Google or something. We can talk to them, but it's outside of our team. So this is all what I wanted to talk about today. Just a quick summary. So we use Kanban because it's flexible. And other methods didn't really work in the past. We embrace the change, but we try to be smart about this and not make decisions too soon, but make them at the last responsible moment. Not too late as well, because this is bad and leads to late changes. We have this daily visual review meeting, which is very useful to see how developers are doing. We make decisions based on data. We use some statistics to come up with estimates and we visualize blockers and try to handle them as soon as possible. So this is all from me. Thank you very much. And if you have any questions, then I think we still have a few minutes. Thank you. Thank you. So using these buttons there.
project manager and I wanted to talk about how we do software at Shazam and what techniques do we especially in the US where you have a lot of TV which are basically our teams. We have teams per product manager but some product managers are improvement basically. So yeah, we use Kanban for release mobile apps every month, but it happens the most important thing that we care about is destructive. From developers perspective, it requirements. You have a deadline that by the out of development. We try to keep them to zero. do we measure? Just a couple of quotes from kind It's a, it has been a hot topic for a while. Even go into development. The fourth optional thing stuff. So what we do with this is we visualize