Making Komplett big by going small
Singh Saund, Pavneet
Ekeli, Tomas
How we're scaling the architecture, ecommerce platform and our business in one of Europe’s largest e-commerce providers & changing with the business as we go. Growth problems are good, it means your business is doing well. How can we build an organisation and architecture that gives us room to grow and change, while still keeping our customers happy? At Komplett Group we are going through growth as an organisation in size and scope. In this talk we will dive into how the development team at Komplett are breaking up a 10 year old platform. By making every mistake in the book we’re building a platform for our future! We’d like to share our experiences in scaling the architecture and development team. We are breaking a monolith into micro-services to match change within our organization. At the same time we have scaled our web development team from 5 to 50 , and we're not stopping here!
 So, hi everyone. Welcome. Thanks for being here at the very last talk and our talk. Are we ready to start? Yeah, I think so. So, I want to start with us. Okay, so thanks for coming to our talk titled Making Complete Big by Going Small. Who are we? So, I'm Pav Neet. I'm a web developer at Komplett. I'm also a squad lead. We'll get into that a little later. And my name is Thomas. I'm a lead software architect on the web team in Komplett. And what are we going to talk about? Feedback? We are going to talk about how we've scaled our team and our group as we've changed. We've gone from a small group to a rather large one. And we've had to make changes both to our architecture and our organization in this process. And that's what we're going to talk about today. So, it's kind of an experience talk. Right. So, Komplett Group. For those of you who aren't familiar with us, we are a group of web stores. We have 16 web shops active now. We have around 7.3 billion crowns revenue. Just under a billion dollars. 1.8 million active customers, 800 employees and around 20 million uniques. We suspect that means unique sessions. In 2015. So, we, these are some of our stores. It may not be that easy to see. But we're traditionally an e-commerce electronic store. But you may be able to see that we've now branched off into travel, insurance, car parts, baby equipment. Amongst other things. And just last week we started the mobile phone company as well. So, things are changing. But we've not always been as big as we were yesterday or today. We started in the mid 90s. And then we had a very simple architecture. So, you, we had an online catalog. And as a customer, you contacted the sales department. So, you actually phoned them. I know it sounds quaint, but that's how it was done. You phoned them and they called out into the office and said everybody get off the system because I have a customer. It was a one user system where we placed orders directly into the system when we had the customer on the line. That didn't scale very well. So, we went on and created what would become our future web platform, Chrome. We called it Chrome in the early 2000s. So, we were before the browser. If anybody mentions Chrome, it's us. It's our name. Okay? At least for this talk. So, the Chrome 2B, it wasn't called Chrome at that point. But it was one of the first e-commerce stores in Europe at least that actually showed you an online stock status. So, you could go to it and see if we had the product in stock. The customer connected it directly and the orders were placed in the order database. Still a quite simple architecture. Then, we got a bit over ourselves and we bought an ERP system, a German three letter acronym. Shout at you. So, now we had to, from this point on, it's actually called Chrome. So, now we had to have a system that kept up better off time than this German system because apparently such systems go down. So, now we had to have a message queue talking to the system. And Chrome placed orders through messages and the ERP system got them from there and populated our database directly. So, that was our main architecture up until, well now really. Pretty recently. So, we're really still here but we have done some changes and we are working to change it. But what Chrome did at this time was, and this will blow your minds, we connect to the database, get an XML back and then we read that in and we do an XSLT transformation on that database, producing HTML and DTD format. So, this is actual running production code from May 2008, not today. It's much better today, right? No, no, it's not. So, this does three things that I want to point out. First, we have a static method returning a string. This string will go directly onto the response. No controllers, no nothing. It's very efficient. And what it does is that it goes directly to the database. There you go. Yeah, it goes directly to the database, populating parameters or arguments. I don't know, I think the ones on the left are parameters and the ones on the right are arguments. Well, as you can see, it shouts at the database because the database talks SAP. And we populate the stored procedure called directly. We actually make an XML header down there. And what we get back, if you go to the next one, is then run through an XSL transformation and that's put directly onto the response. This is how you create a million-dollar industry people. This is high-tech. Okay? So, if you were wondering, we weren't really very happy with this. It's nasty and it's error-prone and it's global state all over the place. And every time we did something, we ended up breaking something else somewhere on our side. Usually gift cards, they always break. So, we'd add a button somewhere and gift card stops working and we remove a button and then gift card stops working. It's horrible, horrible. So, what do we do? Well, like Thomas mentioned, we are at a bad place here. What does that mean, a bad place? This platform has served us for many, many years. The team, if you look at the team, we were five developers at this point. We were sitting in each of our corners in our room. Yes, we had a pentagonal room, yeah. Doing each of our own things based on business needs, which just sounds fair enough, right? Because we prioritized things based on the person that shouted the loudest and whoever shouted loudest got their things through. So, as we did things, based on what you just saw, we started to grind to a halt in regards to our speed and our velocity. So, we stopped delivering features, or slowed down delivering features, and started living more bug fixes. Usually on gift cards. And when he didn't fix gift cards, we fixed something else and broke it. So, if you ever get a gift card, you can hope that it works. Sometimes it does. It's been fine, hasn't it? This is fine, right? So, basically, we were struggling. We were at a point where we needed to do something. We weren't really feeling inspired. And along came a business need. Yeah, so this was when mobile phones started to get popular. I think you may remember that time, the heady days of the first iPhone. So, we needed to be able to show a mobile site that worked on a mobile phone. So, either we could produce an app, or we could produce a website that actually worked on mobile phones. But we needed a mobile presence. It was a growing market, and business came to us with this need. And we said, sure, we can do that. We just need to rewrite everything. Because if you've ever done XSLT transformations, taking XML and making DTD for HTML out of it, it's not really conducive to mobile phones rendering. It's nasty. It's challenging. So, what did we do? Well, then let's talk about Project K2. So, the case for complete, two is for two. Yes, we are good at naming. There's more on that later. So, Project K2 is our response to the business need. And the business said, we need a mobile presence. We translate adapt to be, we need a mobile first web store. So, at this time, responsive web, or adaptive web as we like to call it, but we lost that battle a long time ago. Responsive web was in its, I wouldn't say infancy, but it was starting up. We didn't have cross-browser support, and it was a little unsure. Another approach at this time was building standalone websites, which is what we went for. And that's also what we proposed. It was a good thing because that meant we didn't have to touch the old code. So, we had a... That's a really good thing. Yeah. Which meant we could do a file new project. And we thought, okay, this is great. Now, we can finally focus on the good stuff. Do it right. Do it right. Yeah. Buzzwords. Anyone recognize these? We did all of them. Yes. So, that's it. We went for code quality. We went for the solar principles. We were going to be test driven. We pulled in CQRS, DDD, event sourcing. And then there's the technical side. Then obviously on the process side, we introduced Agile and Scrum. We did pair programming. And with all this came new admin pages. A new infrastructure, a new architecture, as you see. And obviously a new deployment pipeline. It will be easy. Yeah. And what this meant was we're taking our old monolith, turning around, and creating a new one. And it was actually pretty good. We learned a lot. We learned to work as a team. We learned to write better code. We learned a lot about Agile processes. And we delivered the M-complet NO, the first mobile version of our store, within around six months. We made it. We did it. And we made it work. It was good. But what did this mean? Six months. But it didn't have gift cards. Oh, yeah. So we couldn't break anything. That's why it worked. So we had basically, all five of us had gone over to be, were working 100% on this platform, which meant six months without any new features on the existing desktop platform. Or bug figures to gift cards. Which meant that the business owners had to wait, right? So they were fine. Let's wait for this. We delivered this. And then they were eager. Cool. Now we can focus on fixing gift card. Couldn't leave gift card. It's nightmare. And we thought, okay, we got the feature request in. We started looking at what was involved. And we realized we didn't want to go back to this XSLT. We don't want to go back there. So we thought, we've more or less created a new web store. Let's do a little more. Let's create the desktop version of this on the new platform. Should be easy. Should be easy. And what happened is the business never asked for this. But we got a natural accept. We said it'll be fast. We'll see what it did in six months. We'll get used to it in six months and we were ready. And we ended up, well, building out. What we have to do is we need to get gather requirements because we didn't know what's happening in the old system. And the thing is neither did the business. They didn't really know all the quirks. I mean, when you have a system running for 10 years, there's so many small quirks. So the spec is the old system. Make it like that one. And that's what we did. And that's what they told us. So how should this work? We can do this. We can do something nice in Chinese. We can not implement gift cards. No, that was an option. But they said, look at the old system. Do what that does. And this led us to a feature parity that much. Yeah. So we spent, I think, one and a half years trying to reimplement the entire Chrome platform in K2. Just building functionality we already had. And remember, while we were doing this, the old desktop that actually was earning us money, the one that was important, it had now not been touched in two years. And the mobile platform that some very, very few users actually used because they don't like to go to M sites for some reason. And it had very, very few features. That also wasn't being worked on because we were working on the new shiny stuff that hadn't been deployed and hadn't been activated in any way yet. So after two years of this, the organization came to us and they asked, hey, guys, we kind of need to do stuff. Are you finished soon? And we said, we don't know. We didn't even know which features we were missing because nobody knew. So they did the only same thing. They killed it. Yeah. So that's like two or three years of your life down the drain. So if you learn one thing from this, rewrites are bad. I'm not saying that they can't work, but it's really, really hard. And if you can find a different way of doing it, please consider it. So this was a very dark time for us in the web team at Komplett. We had five developers when the project was killed and we lost half-ish of those. So one of the largest e-commerce sites in Norway suddenly didn't have a lot of web developers. We'd lost our self-confidence. We'd lost our respect in the organization. The organization then embarked on a year-long search for some platform. We still want to be a web e-commerce thing. There must be some platform that we can buy. And there are. There are platforms out there. Most of them have three-letter acronyms. So we launched a rather large project to figure out which one is best for us. We got proposals in where we kind of played them against each other. And we saw who had the features we needed, who had the support we needed. And after a year of this, we had, I think it was three finalists. And the web team was asked to just submit a proposal of our own for our Chrome platform, just kind of to have a baseline, because we knew that nothing could be worse than that. So we had to have something above that. And then on, I think it was a Wednesday, we actually got the call that we won. So we have it on great authority that we are the best in the world at this. Do you remember what time it was? I don't remember the exact time. So what does that mean? What that meant was they still trusted in us, or they put their trust in us rather. But what that also meant is there are new business domains that they wanted to go into. They wanted to do new things. Well, actually not day, we. And we still had a great heap of code. We hadn't touched this in actually three years now, right? And we still didn't have a mobile platform. So we're three years behind the rest of the field. And we're asking ourselves, can we do this? I mean, can we do this? We were actually really unsure. Three years earlier we just said we couldn't do it on Chrome. And now we had to. So how could the same people get trust and deliver? That's what we have to ask ourselves. So they trust us, meaning the business and the organization trust us. But do we trust us? Can we trust in ourselves? Can we get to a place where we can make this work? So they're saying that culture eats process for breakfast. So that's where we started. We started with culture. We started with values. We started with defining who we are as developers. And what it meant to be a developer in Scandinavia or Northern Europe's largest e-commerce provider. So we did this. We spent time at this one of our workshops. And we end up with 10, 11, I can't count. A number of values. These values are things that we believed in. Things that we embodied. Things we felt that we already have, some of them. And other things were things we aspired to be. This is for ourselves. And this is also for new developers coming on. Because this is the point, right? We can't scale with us five. We needed to increase and we need to expand. So? Yeah. So then we did the next thing. We just added lots of developers, right? So business came to us with needs. So the entire organization had needs. Our customers had needs. And we weren't able to, we didn't have the manpower, the muscle to do it. So we needed more people. And we hired a lot of more people. The good thing was that we'd already thought about cultures. So we didn't kind of just fall apart on that. But just adding people or developers has its own problems. So at this time, we were like five developers. And we had a QA guy and we had two ops. And that was the web team. So pretty small, right? And we needed to scale. So we looked around. And if you've ever searched about scaling agile, there's a lot of talk about that. But there's a kind of local firm that seems to do this pretty well. So we looked at them. They're not Norwegian. They're Swedish. Spotify. You might have heard of them. So around this time, they published about their Spotify model. Have you heard of it? Raise of hands. Yeah, good, good. Then we can go quickly through this. Yeah. So this is a part of the Spotify model we've tried to cut out. It's just to go quickly through this. It's basically built around autonomous squads who deliver to a specific business need with their dedicated product owner. They're autonomous, meaning they have control over their build pipeline. They have control over everything, deployment, all the testing, and direct access to whoever is the representative on the business side. These squads are then grouped together in tribes. And these tribes usually represent a logical business area, so crossing different business domain. So across squads, we then have something called chapters. Chapters basically meaning you could be interests, right? Front-end interests. It could be older DevOps. It could be something domain specific. And then we can have multiple tribes. This is how Spotify did it. One of the core features or one of the most important aspects of the Spotify model is that each team is co-located, right? So each team has to be at the same area, same place, same office, and work together. So we took this, and this is awesome. This is a blueprint. This is something we could use for ourselves, right? And we take this, and we start with our major project called the Minion Project, which is a... We can get to that. So this is Bob. He's in our office. Yeah. He runs around all the time. Fun guy. So the Minion Project is to not do a big rewrite. It is to change Chrome while it is living and make it mobile-friendly. It's nearing its end now, actually. So if you go to our sites, they're actually quite good, I think, on mobile. That's what the Minion Project is. And we had said we couldn't do this years earlier, and then we couldn't. But that was three years ago, which is an eternity, an internet years. So now we actually could. Now we had the technology. Browsers were sufficiently advanced, and we might have learned a thing or two. But still, we needed to produce at a rate that we couldn't as a few. And with the Spotify model, we thought we were able to scale our team. Ramp up. So we started ramping up. This is an illustration of how we ramped up over time. The time area right now is beginning mid-2014. So we started scaling against multiple offices. And so we scaled by adding consultants, basically, because it's really hard to get good developers locally. And added smart developers, good people that became part of the team. But something happens when you throw many people into a monolith. So even with the Spotify model and starting to split up people into squads and talking about business domains, we still had one code base. And now we had many, many, many developers working in that code base. And if you've ever worked in a shared code base, I know Facebook can do it, and I know Google can do it. But really, we are not Facebook or Google. We're not that clever. So what happens is that one developer does something to make some new feature or fix some bug and then give car breaks. And another developer does something else, and then that doesn't merge with his thing. And suddenly you have long-living feature branches that are weeks out of master, and you need to rebase them, and you can't rebase them because things have changed horribly. And it's just not working at all. So what do we do? What do we do about this? Well, our problem was that when we actually managed to merge something and deliver some deployable thing, and we put it on our test systems and it never worked, gift card always breaks, and everything else always breaks. So we had things living in our test systems for weeks before we could almost never, but we could dare to put it in production and then figure out what breaks only in production. It's a bad place to be in because with such a large monolith with many people working on it, actually stabilizing it is a non-trivial task. So we figured out that we need to change. We can't work like this. We need to, just because we're not good enough, I guess, to make such a thing work, we need to split it up into smaller deployables, just because we need to get something to production quickly. And we sat down and we made an architectural vision for COMPLET, which is its COMPLET vision. I'm not saying this is what everyone has to do, but we believe that for our domain and our organization, we need to have a landscape of services. We need to have many services that live in the environments, that connect to each other. We need those services to be stable in their contracts, so we are not allowed to do breaking changes on services. We actually go as far as to say, if you are breaking your contract, then you are making a new service. You should not override the old service. You should deploy a new service beside it and tell your clients to start using that instead. We needed a consistent deployment, and we had to do this while our system was running. So how do you do that? Well, you take whatever new part of the monolith you need to split out and you make some service for it, just because you can't do it in the old one, because gift card will break, and you put a feature switch around it. So you deploy your new service and you make sure it works, and you can actually make it work because it's smaller, so you are able to understand it. Once we're happy with it, we've seen that it actually works in the development environment, then we turn it on by the feature switch in the test environment and check that it works and fix whatever doesn't work in test, because there's always something. And then we do the same in production. So we separate the actual deployment of the services from the activation of the services through feature switches. So it's a strategic choice. It is done at a different level than the technical level. As developers, we don't decide when something turns on. That's up to the guys who are responsible for that part of the site. So we have people who are responsible for the checkout, and they decide when we turn on and off stuff in the checkout. So that's how we decided to do it, and it's been working relatively good. But the problem is, to create a new service, it sounds easy, right? But it's not, because you need to make a new repository, you need to set up the build jobs, the deployment pipeline, you need to manage the servers and make sure they're up and running and ready to be deployed to you. You may even need to create databases and all of that stuff. And that meant that when we told the developers to please don't make this in the old one, make a new thing, it's like, I don't want to do that, because then I can't start actually solving the problem for a week or two, because it takes me that long to actually get it out and running. And often they kind of short-cutted it, so they got it running in the development environment, and nobody really cared about test, because it's not going live for a month, right? And then you have to take it live and it doesn't work in test. So what we had to do is we had to make something that can make a service. So we debated the naming of this. One time it was the service maker, service maker, but it's now just the service maker. I think we also called it foundry at one time. Well, it doesn't matter. So it's an interesting little thing that's been very, very important for us. So we have a developer who needs he or she, needs to create a service, so they go to the service maker. And the service maker is actually a website on our internal network where the developer enters the name of the service and whether or not that service is going to need a database. So they go to the service maker, and they press the create button, and the service maker goes out, connects to Bitbucket in our case and creates the Git repo. It connects to our build machine, which is Jenkins for now, and it connects to our deployment machines, which is Octopus for now, and it sets it up on all of those. It also goes to the database service and creates new databases, and the build service is allowed to use another service's database. So if you need the database, you get a new one. Yeah. Once it's done that, it actually then commits a template of a service to the Git repo. That's picked up by the build server, which builds the service, because that's what build servers do. Then it pushes that to our deployment server, Octopus, and that then deploys that service to all environments. So after about 30 seconds from the create, they have an actual running service all the way. This has made it immensely easier to convince developers that you should create a new service instead of doing it in the old one. So what we've learned is that if you are going to have a service architecture, creating services must be cheap. So we've talked about how we do this technically, but how do we actually do this from a process and business standpoint? How do we actually push work through our system? So we mentioned that the Spotify model brought us this far. But we were feeling pain, right? And being strictly location-based didn't really work for us. We didn't have that many developers, or it just didn't work for us. It's not how our organizations grew together. So we went back and looked at our model, and we thought we don't like those squares or things, we created circles. Circles are much better. This is what we would like to call the complete model. It's circular. This is actually a snapshot of how the team is basically screwed together today. And just to kind of walk you through this, each circle is then a person. Each small circle. Each small circle is a person, each white circle. Each color circle is a squad. And each squad has a color which identifies them. Not a name, not a feature, not a business domain. Because you're not guaranteed to work in the same thing forever. So each squad has a color, and we keep them as stable as possible. You want the same people working on the same part of the domain. So a squad was built up of four to six developers with their own dedicated QA. So someone who's expert at testing. How they solve QA within the squad, that's up to them. But that's what we have. We have two dedicated DevOps, and they're outside the squads. And then we have the orange squad up there, which is our infrastructure team. Each of these squads are actually focusing on delivering business value. And they were working within a feature set. So, as an example, the red squad here is solely responsible for the checkout and nothing else. The black squad is solely responsible for the shopping cart and what's involved there. And these are now different parts of the code. This is something we've split up. We split up the code into a different deployment unit, and that squad is responsible for delivering the value there. And each of these groupings, these larger circles, they are a logical grouping across multiple domains. So in the Spotify model, this would be similar to a tribe. Within this tribe, we then have one or more UX responsible people who speak to the business experts and gather requirements and work with them directly. And we also have a delivery manager. So this is basically how we're set up now. And the infrastructure squad is responsible for making the lives of each other squad a lot easier. So we make the service maker. I'm in the orange squad. We make the service maker and stuff like that. So you're the service maker maker. Oh, yes. So what we've really done here is we like to call it the reverse Conway. Because Conway's law says that any architecture will kind of reflect the organizations and the petitions in the organization. So we've tried to petition the organization the way we want the architecture to be. If that makes sense. But again, we've talked about people, we've talked about scaling, we've talked about everything, but how do we actually work? So the real key here is basically over how we've learned experience is communication. And none of these squads are co-located. So each squad has always got members from at least two locations. And one, I think, actually has four, which is almost as many as there are people in the squad. Not counting people working from home. Yeah. And this is where we really break from the Spotify model. There's reason for this, right? The reason is bringing aboard new members, especially when you're in remote locations, is then it's really hard for them to actually get the domain, get the culture. And by having a squad which is spread with representatives in the main office in Norway and with other offices, then it's easier to actually spread knowledge, bring people aboard, and the onboarding process becomes easier for everyone. But communication is still hard, right? We still need to have some way of bringing new developers that aren't nearby, near to us, I mean closer to us. So what we have done, one of the things we've done is basically set up a window into each office. And we have something called a pyramid for this. That's me. And that's Thomas. Yeah, so we have multiple of these screens where we can peer into each other offices. We're not kind of watching what people do. This is just about glancing and seeing who's there, seeing what's happening. Are they having cake? And that's a lot of fun having cake. Because then we're standing on the other side and looking at the camera. We want cake, please. Send cake here. We dance sometimes and yeah, we have a lot of fun. So this gives us kind of a window into each other. It's easier just to say, you know, we're there all the time. And beginning was awkward, but now it's natural. And again, continuing on communication, we have something called Lodok. It's a lot like Slack, except this works. So, for us, for us, please remember green, green notes only. It's just the major difference between this and anything else is it's not as fancy, but we have something called threads or flows, which makes communication a lot easier. This is a shot from our water cooler. You see a very beautiful man over there. He's known as... Pretty Thomas. I'm the other one. Thank you. And if you... I'm not sure if you can see this, but if you see the rooms on the side, the rooms are basically whatever we need in regard to squads and structure. We have other rooms like interest rooms, like the checkout room over there, and then we have something called code upon L. So this is the code panel. It's something we do on YouTube. Then we have something called KDC, which is our complete developer conference. This is where we gather together yearly and do things together. So basically we create channels based on whatever we need, right? And we have J4s, yes, I said J4s, on the water cooler. We also use a lot of Skype and TeamViewer for pair programming. So the main learning here is... I mean, it should be obvious, but organization matters. But we tend to not care sometimes, right? So it just emphasizes organization really does matters. So we're making a landscape of services. That should be easy, right? Each service is a small thing. It's not hard to make a service. We've made sure of that. And then let's just do it, right? Well, it turns out there are different problems with services than with monoliths. So how do we make sure that our services are actually up? How do we make them perform? How do we make them scale in ways that are relevant to these services? Well, each service has an SLA to its clients. We use a lot of caching to make sure that they perform at that level. And we are very, very into performance testing our solutions. So whenever we launch something new, it has been performance tested quite strenuously. But we also need to have the services have some common characteristics. And we make that easier through the service maker, but they need to do stuff like they need to have known health endpoints so that our infrastructure can make sure that they're up and say that it's safe to actually send traffic there because services go up and down all the time. And we need them to log to a centralized logging service in a specific way so that we can recognize requests throughout different services. So if you actually go to our frontend and do something, that might be five services, ten services, involved in serving your request in some way. In a monolith, that's quite easy to log because you just log and it's all from the same place. But when you are in a service landscape, it gets really hard to figure out what actually happened and why didn't it work. So we've instituted standards on that, ways we log, and to make this easier, we've made that a library that we have in our internal nugget server that's actually put into each new service by the service maker. So we use the service maker kind of to drive the service behavior that we want. If you make a new service now, it already has the endpoints that we expected to have, and it already has all the logging mechanisms. You can just log as normal and it will log everything that we expect to find. So we've had to do quite a few of these things, and that's what the Orange Squad does. We figure out what turns out to be the best practice that we want our systems to follow, and then we pull that out of the projects that started doing it, usually as a nugget package or something, and we put that in the service maker so that from now on all new services do that. So the learning here is that services aren't monolith, they are different durr, and they have different needs, and they have different characteristics, and you need to be aware of that. So where do we go from here? One thing we haven't really spoken about is what we do within each squad technologically wise. We are kind of connected to an existing technology stack. We are used to working with.NET, we used to working with knockout, is what we have in our main store now. But because we now started splitting things into different autonomous units, we are now available to have the opportunity to work on completely new technologies. So, which is what you touched on, right? So usually learnings come through squads doing new things within their domain, within their solutions, and talking to other squads saying that's a good idea. Why we also need it? And they might start implementing it, and then this is what Orange short comes in, right? So what would be cool though is having fully mandated squads, or completely autonomous squads, where we actually then pull in our operations, we pull in the business users, there's no people in between. So we are really interested in trying to have complete business units, including developers. There will be more microservices, but how small they will be is something we are unsure of. Because we are doing our best to stick to the business domains in regard to what we are spread out. We think about business context, we think about our domain, right? But we have to take some baby steps along the way. So we are not religious about the size of our microservices. Size shouldn't matter, right? But we need to have them smaller than our huge monolith, that we know. So we are also looking into exposing some more APIs, right? We want to expose public APIs. I hope we get there soon, but we should get there. We don't think we can make everything, but we have a platform of lots of things, and it will be really cool if someone made Android apps or iOS apps even. What other way around? I like Android. And obviously more business areas. So just to summarize, I mean, scaling is hard. And we have to find our own way. We are still finding our own way. We are not there yet, but we are getting there. Culture is crucial, right? This is number one in regards to getting people on board and actually building a team. So we also want to really embrace and learn from failures. This is really important, especially across cultures. Failing should be something everyone does by default. It should be the default way of approaching things. And that's actually really hard when you are working in several different cultures, because different cultures have different views on failure. So that's been one of the things we had to work on. So we now call them fabulous failures, just to try to make it a bit better. It's a good thing to fail, when you've learned something. And we're going to continue failing. And hopefully, we're also going to continue sharing as we go. We started doing things in public. This is one of the things. And we'll continue as much as we can. But one of the greatest success factors, which you can feel on a day-to-day basis, is are your developers or is everyone happy? It's because happy people make great stuff. So make them happy. And then they will probably make good stuff. So, let Denise take a picture. I can send you out first. So we'd like to get feedback on this. We want to hear more from you guys. We want to hear if this thing makes sense, right? Yeah, contact us on Twitter. Talk to us now. You can also follow us on the CodePanela, which is a YouTube stream. We try to do it at least once a month, perhaps twice a month. It's in Norwegian, sorry, for our non-Norwegian friends. You really should learn it. It's a great language. So easy. But we try to interview people and do stuff there. And Pavel has a blog that you really should read. It's not Norwegian. That's in English. Yeah. And I'm just on Twitter. So I hope you guys have learned something or got some ideas or inspired to do something. Or have feedback for us. That'd be awesome. We want to go more into detail moving forward. And we hope you guys leave green notes over there or press the ring button as many as possible. And if nobody wants to go home, we can have lots of questions today. So... It's in the tens. It's not in the hundreds yet. But we believe we will reach at least 100 by the end of the year. All written in the same language? Mostly. We are, as I said, we have this service maker which makes it really easy to make services with the technology that we usually use. We have a few Java services that we are phasing out, actually. Because C-sharp is cool. And I have a bet going on. So I want to be the first one to actually produce an F-sharp service. There is no requirement on that. So you're open for a new API? Yes, we are. But the easy path right now is.NET. So that's the thing most of our developers do. You're not too worried about getting the whole set of languages? We are moving towards deploying our services to service fabric. That's what we're now looking at. Not quite sure whether it will work or not. But in that you can run anything you want, as long as it's executable. It can run, they say. That is the thing that's further down the line. I'm sorry? I'm just thinking further down the line. When you have first-person people, you can have a switch, then you can start. Yeah. That's one of the things with smaller services. One of the kind of goals is that they should be so small that if there's a bug, you can delete it and start a new. And that's not a huge cost. I'm not sure that many of our services are that small yet. But it's certainly something we're hoping to get towards. But as I said, we don't think there's a magical, this is the right size for a microservice. What are the examples of services you have? We have, of course, a catalog service. That is our catalog. We have services for actually placing orders. It's for getting users, for doing all of these small things. We have some services that are more like verticals. So as I said, our checkout is a separate site. It looks like you're still on the same site, but actually you're not. And in these days, we are... My page, or your customer page, is also a separate site? Yeah, with separate services underneath that, that handle the storage. Right now, we are actually in the process of deploying a new checkout for our BTB customers. That is on a different technology from our other checkouts. We have two different checkouts on different technologies. Not very different, but different. Different from a second. Yeah. And that's kind of what we want. We want to be able to test out news things. How many versions of these services do you have on the network? Yeah. How do you name them? Well, we named them... Sorry. So the question is, how many different services, different versions of services do we have deployed? Since we say that we don't want to deploy... I'm sorry. We don't want to deploy over. We want to deploy side by side. So this is not something we are very good at yet. So we are working on that. And it's also something that becomes more important as things stabilize. It's in effect when people are moving quickly. They are usually communicating on both sides of this communication. And then it's not that important to us. So the things that are stable are the things... And in use are the things we don't overwrite. But we have some things that are in different versions, but not a lot of them. Because we go into different kinds of problems there with the data storage, for instance. They are not allowed to use the same data. It's not as easy as it sounds. I think our time is running out. We have more questions though. So we can take one over here now. You, on your diagram, about how you are organized, you have squads and in a larger group of squads, you have two squads. So the question is, if I understand correctly, if this is representative of how we are organized today. We have two tribes, which represent the two areas of our domain. How we see it now. Based on how the business is also structured. We have a before buy. Everything about getting a customer into the store. It's where we do a lot of scaling, a lot of work on presenting content. And we have a kind of during slash after buy. Where we do a lot of interactive pages. We do the checkout, you do your cart, you do your MyPage, that type of stuff. And yeah, the orange squad is something... You have seven people on the each squad. So in a way, we have a number of people on each squad. We have around between four and seven, basically, depending on how we do things. And we do, when you start something new, we do a task force approach. Where you may split off two people, three people. They start working something, just to get the service up and running. And then the rest of the squad just finishes up and joins them. They stay within their tribe and do different... Each squad has at least one dedicated QA. And the rest are developers. There's one lead who is more responsible for the delivery. But he or she is usually a developer. And how they solve this world-wise is up to the squad. So the question is, how do microservices communicate with each other? Mainly by rest. So they call each other. This is one of the things we are working on changing now. Because we think to be able to scale, we need to push information out instead of having them get it when the customer is actually coming in. So we are reworking to get a push architecture where when something happens somewhere in the system, a message is sent. And then the services that are interested in that, they do their work. So that when the customer actually kind of knocks on the door, we don't have to build the store. It's ready for them. This is also one of the things we are now discovering as we move to a service landscape. Because if you do a lot of requests, then you end up in a situation where you deploy something new. And if one service is down or unstable, then that kind of bubbles all the way out. So instead of that, we want it to be so that when a service, when some event happens in the system, it propagates it all the way out. So that if something goes down in that pipeline, it should still be up as far as the user discovers. You might not get the latest updates. So you might see somewhat stale data. But at least we will have data. Okay. How much is left of the monolith? The question is how much is left of the monolith? I should have checked that. Quite a lot. Quite a bit. Quite a lot. We still have XSLT. Actually, there are pages where you will be served by the XSLT. But mostly you can avoid them, right? Most of the shopping experiences actually been moved over. Yeah. It's things like the feeds and stuff that's not really important. But the main things that a normal user encounters, those are, well, on the new technology. So, yeah, we have one more. When a squad develops a service, do they have to keep maintaining it during the lifeline? Or does that responsibility transfer to another squad? Yeah. So the question was, when a squad develops a service, do they have to maintain it for the lifetime of that service? Or does it move to a different squad? And the answer is that it's there. So there are certain cases where, for instance, if one squad has an area of responsibility and they don't have time to develop something that has to be developed, then a different squad can do that and then transfer it. But usually they own the service as long as it's alive. But since we've elected to do it by business area, it might be that the people who are responsible for master data will have to maintain something someone else made. Okay. Thank you, guys. Thanks so much for being here. Have a nice weekend. Green.
 So, hi everyone. Welcome. Thanks for being here at the very last talk and our talk.  Are we ready to start?  Yeah, I think so. So, I want to start with us.  Okay, so thanks for coming to our talk titled Making Complete Big by Going Small.  Who are we?  So, I'm Pav Neet. I'm a web developer at Komplett. I'm also a squad lead. We'll get into that a little later.  And my name is Thomas. I'm a lead software architect on the web team in Komplett.  And what are we going to talk about? Feedback?  We are going to talk about how we've scaled our team and our group as we've changed.  We've gone from a small group to a rather large one. And we've had to make changes both to our architecture and our organization in this process.  And that's what we're going to talk about today. So, it's kind of an experience talk.  Right. So, Komplett Group. For those of you who aren't familiar with us, we are a group of web stores.  We have 16 web shops active now. We have around 7.3 billion crowns revenue. Just under a billion dollars.  1.8 million active customers, 800 employees and around 20 million uniques.  We suspect that means unique sessions. In 2015.  So, we, these are some of our stores. It may not be that easy to see.  But we're traditionally an e-commerce electronic store.  But you may be able to see that we've now branched off into travel, insurance, car parts, baby equipment.  Amongst other things.  And just last week we started the mobile phone company as well. So, things are changing.  But we've not always been as big as we were yesterday or today.  We started in the mid 90s. And then we had a very simple architecture.  So, you, we had an online catalog. And as a customer, you contacted the sales department.  So, you actually phoned them. I know it sounds quaint, but that's how it was done.  You phoned them and they called out into the office and said everybody get off the system because I have a customer.  It was a one user system where we placed orders directly into the system when we had the customer on the line.  That didn't scale very well. So, we went on and created what would become our future web platform, Chrome.  We called it Chrome in the early 2000s. So, we were before the browser.  If anybody mentions Chrome, it's us. It's our name. Okay? At least for this talk.  So, the Chrome 2B, it wasn't called Chrome at that point.  But it was one of the first e-commerce stores in Europe at least that actually showed you an online stock status.  So, you could go to it and see if we had the product in stock.  The customer connected it directly and the orders were placed in the order database.  Still a quite simple architecture.  Then, we got a bit over ourselves and we bought an ERP system, a German three letter acronym.  Shout at you.  So, now we had to, from this point on, it's actually called Chrome.  So, now we had to have a system that kept up better off time than this German system because apparently such systems go down.  So, now we had to have a message queue talking to the system.  And Chrome placed orders through messages and the ERP system got them from there and populated our database directly.  So, that was our main architecture up until, well now really.  Pretty recently.  So, we're really still here but we have done some changes and we are working to change it.  But what Chrome did at this time was, and this will blow your minds,  we connect to the database, get an XML back and then we read that in and we do an XSLT transformation on that database,  producing HTML and DTD format.  So, this is actual running production code from May 2008, not today.  It's much better today, right?  No, no, it's not. So, this does three things that I want to point out.  First, we have a static method returning a string.  This string will go directly onto the response.  No controllers, no nothing. It's very efficient.  And what it does is that it goes directly to the database.  There you go.  Yeah, it goes directly to the database, populating parameters or arguments.  I don't know, I think the ones on the left are parameters and the ones on the right are arguments.  Well, as you can see, it shouts at the database because the database talks SAP.  And we populate the stored procedure called directly.  We actually make an XML header down there.  And what we get back, if you go to the next one, is then run through an XSL transformation  and that's put directly onto the response.  This is how you create a million-dollar industry people.  This is high-tech.  Okay? So, if you were wondering,  we weren't really very happy with this.  It's nasty and it's error-prone and it's global state all over the place.  And every time we did something, we ended up breaking something else somewhere on our side.  Usually gift cards, they always break.  So, we'd add a button somewhere and gift card stops working  and we remove a button and then gift card stops working. It's horrible, horrible.  So, what do we do?  Well, like Thomas mentioned, we are at a bad place here.  What does that mean, a bad place?  This platform has served us for many, many years.  The team, if you look at the team, we were five developers at this point.  We were sitting in each of our corners in our room.  Yes, we had a pentagonal room, yeah.  Doing each of our own things based on business needs, which just sounds fair enough, right?  Because we prioritized things based on the person that shouted the loudest  and whoever shouted loudest got their things through.  So, as we did things, based on what you just saw,  we started to grind to a halt in regards to our speed and our velocity.  So, we stopped delivering features, or slowed down delivering features, and started living more bug fixes.  Usually on gift cards.  And when he didn't fix gift cards, we fixed something else and broke it.  So, if you ever get a gift card, you can hope that it works. Sometimes it does.  It's been fine, hasn't it? This is fine, right?  So, basically, we were struggling.  We were at a point where we needed to do something.  We weren't really feeling inspired.  And along came a business need.  Yeah, so this was when mobile phones started to get popular.  I think you may remember that time, the heady days of the first iPhone.  So, we needed to be able to show a mobile site that worked on a mobile phone.  So, either we could produce an app, or we could produce a website that actually worked on mobile phones.  But we needed a mobile presence.  It was a growing market, and business came to us with this need.  And we said, sure, we can do that.  We just need to rewrite everything.  Because if you've ever done XSLT transformations, taking XML and making DTD for HTML out of it,  it's not really conducive to mobile phones rendering.  It's nasty.  It's challenging.  So, what did we do?  Well, then let's talk about Project K2.  So, the case for complete, two is for two.  Yes, we are good at naming.  There's more on that later.  So, Project K2 is our response to the business need.  And the business said, we need a mobile presence.  We translate adapt to be, we need a mobile first web store.  So, at this time, responsive web, or adaptive web as we like to call it,  but we lost that battle a long time ago.  Responsive web was in its, I wouldn't say infancy, but it was starting up.  We didn't have cross-browser support, and it was a little unsure.  Another approach at this time was building standalone websites,  which is what we went for.  And that's also what we proposed.  It was a good thing because that meant we didn't have to touch the old code.  So, we had a...  That's a really good thing.  Yeah.  Which meant we could do a file new project.  And we thought, okay, this is great.  Now, we can finally focus on the good stuff.  Do it right.  Do it right.  Yeah.  Buzzwords.  Anyone recognize these?  We did all of them.  Yes.  So, that's it.  We went for code quality.  We went for the solar principles.  We were going to be test driven.  We pulled in CQRS, DDD, event sourcing.  And then there's the technical side.  Then obviously on the process side, we introduced Agile and Scrum.  We did pair programming.  And with all this came new admin pages.  A new infrastructure, a new architecture, as you see.  And obviously a new deployment pipeline.  It will be easy.  Yeah.  And what this meant was we're taking our old monolith, turning around, and creating a new one.  And it was actually pretty good.  We learned a lot.  We learned to work as a team.  We learned to write better code.  We learned a lot about Agile processes.  And we delivered the M-complet NO, the first mobile version of our store, within around six months.  We made it.  We did it.  And we made it work.  It was good.  But what did this mean?  Six months.  But it didn't have gift cards.  Oh, yeah.  So we couldn't break anything.  That's why it worked.  So we had basically, all five of us had gone over to be, were working 100% on this platform,  which meant six months without any new features on the existing desktop platform.  Or bug figures to gift cards.  Which meant that the business owners had to wait, right?  So they were fine.  Let's wait for this.  We delivered this.  And then they were eager.  Cool.  Now we can focus on fixing gift card.  Couldn't leave gift card.  It's nightmare.  And we thought, okay, we got the feature request in.  We started looking at what was involved.  And we realized we didn't want to go back to this XSLT.  We don't want to go back there.  So we thought, we've more or less created a new web store.  Let's do a little more.  Let's create the desktop version of this on the new platform.  Should be easy.  Should be easy.  And what happened is the business never asked for this.  But we got a natural accept.  We said it'll be fast.  We'll see what it did in six months.  We'll get used to it in six months and we were ready.  And we ended up, well, building out.  What we have to do is we need to get gather requirements because we didn't know what's happening in the old system.  And the thing is neither did the business.  They didn't really know all the quirks.  I mean, when you have a system running for 10 years, there's so many small quirks.  So the spec is the old system.  Make it like that one.  And that's what we did.  And that's what they told us.  So how should this work?  We can do this.  We can do something nice in Chinese.  We can not implement gift cards.  No, that was an option.  But they said, look at the old system.  Do what that does.  And this led us to a feature parity that much.  Yeah.  So we spent, I think, one and a half years trying to reimplement the entire Chrome platform in K2.  Just building functionality we already had.  And remember, while we were doing this, the old desktop that actually was earning us money, the one that was important,  it had now not been touched in two years.  And the mobile platform that some very, very few users actually used because they don't like to go to M sites for some reason.  And it had very, very few features.  That also wasn't being worked on because we were working on the new shiny stuff that hadn't been deployed and hadn't been activated in any way yet.  So after two years of this, the organization came to us and they asked, hey, guys, we kind of need to do stuff.  Are you finished soon?  And we said, we don't know.  We didn't even know which features we were missing because nobody knew.  So they did the only same thing.  They killed it.  Yeah.  So that's like two or three years of your life down the drain.  So if you learn one thing from this, rewrites are bad.  I'm not saying that they can't work, but it's really, really hard.  And if you can find a different way of doing it, please consider it.  So this was a very dark time for us in the web team at Komplett.  We had five developers when the project was killed and we lost half-ish of those.  So one of the largest e-commerce sites in Norway suddenly didn't have a lot of web developers.  We'd lost our self-confidence.  We'd lost our respect in the organization.  The organization then embarked on a year-long search for some platform.  We still want to be a web e-commerce thing.  There must be some platform that we can buy.  And there are. There are platforms out there.  Most of them have three-letter acronyms.  So we launched a rather large project to figure out which one is best for us.  We got proposals in where we kind of played them against each other.  And we saw who had the features we needed, who had the support we needed.  And after a year of this, we had, I think it was three finalists.  And the web team was asked to just submit a proposal of our own for our Chrome platform,  just kind of to have a baseline, because we knew that nothing could be worse than that.  So we had to have something above that.  And then on, I think it was a Wednesday, we actually got the call that we won.  So we have it on great authority that we are the best in the world at this.  Do you remember what time it was?  I don't remember the exact time.  So what does that mean?  What that meant was they still trusted in us, or they put their trust in us rather.  But what that also meant is there are new business domains that they wanted to go into.  They wanted to do new things.  Well, actually not day, we.  And we still had a great heap of code.  We hadn't touched this in actually three years now, right?  And we still didn't have a mobile platform.  So we're three years behind the rest of the field.  And we're asking ourselves, can we do this?  I mean, can we do this?  We were actually really unsure.  Three years earlier we just said we couldn't do it on Chrome.  And now we had to.  So how could the same people get trust and deliver?  That's what we have to ask ourselves.  So they trust us, meaning the business and the organization trust us.  But do we trust us?  Can we trust in ourselves?  Can we get to a place where we can make this work?  So they're saying that culture eats process for breakfast.  So that's where we started.  We started with culture.  We started with values.  We started with defining who we are as developers.  And what it meant to be a developer in Scandinavia or Northern Europe's largest e-commerce provider.  So we did this.  We spent time at this one of our workshops.  And we end up with 10, 11, I can't count.  A number of values.  These values are things that we believed in.  Things that we embodied.  Things we felt that we already have, some of them.  And other things were things we aspired to be.  This is for ourselves.  And this is also for new developers coming on.  Because this is the point, right?  We can't scale with us five.  We needed to increase and we need to expand.  So?  Yeah.  So then we did the next thing.  We just added lots of developers, right?  So business came to us with needs.  So the entire organization had needs.  Our customers had needs.  And we weren't able to, we didn't have the manpower, the muscle to do it.  So we needed more people.  And we hired a lot of more people.  The good thing was that we'd already thought about cultures.  So we didn't kind of just fall apart on that.  But just adding people or developers has its own problems.  So at this time, we were like five developers.  And we had a QA guy and we had two ops.  And that was the web team.  So pretty small, right?  And we needed to scale.  So we looked around.  And if you've ever searched about scaling agile, there's a lot of talk about that.  But there's a kind of local firm that seems to do this pretty well.  So we looked at them.  They're not Norwegian.  They're Swedish.  Spotify.  You might have heard of them.  So around this time, they published about their Spotify model.  Have you heard of it?  Raise of hands.  Yeah, good, good.  Then we can go quickly through this.  Yeah.  So this is a part of the Spotify model we've tried to cut out.  It's just to go quickly through this.  It's basically built around autonomous squads who deliver to a specific business need  with their dedicated product owner.  They're autonomous, meaning they have control over their build pipeline.  They have control over everything, deployment, all the testing,  and direct access to whoever is the representative on the business side.  These squads are then grouped together in tribes.  And these tribes usually represent a logical business area,  so crossing different business domain.  So across squads, we then have something called chapters.  Chapters basically meaning you could be interests, right?  Front-end interests.  It could be older DevOps.  It could be something domain specific.  And then we can have multiple tribes.  This is how Spotify did it.  One of the core features or one of the most important aspects of the Spotify model  is that each team is co-located, right?  So each team has to be at the same area, same place, same office, and work together.  So we took this, and this is awesome.  This is a blueprint.  This is something we could use for ourselves, right?  And we take this, and we start with our major project called the Minion Project,  which is a...  We can get to that.  So this is Bob.  He's in our office.  Yeah.  He runs around all the time.  Fun guy.  So the Minion Project is to not do a big rewrite.  It is to change Chrome while it is living and make it mobile-friendly.  It's nearing its end now, actually.  So if you go to our sites, they're actually quite good, I think, on mobile.  That's what the Minion Project is.  And we had said we couldn't do this years earlier, and then we couldn't.  But that was three years ago, which is an eternity, an internet years.  So now we actually could.  Now we had the technology.  Browsers were sufficiently advanced, and we might have learned a thing or two.  But still, we needed to produce at a rate that we couldn't as a few.  And with the Spotify model, we thought we were able to scale our team.  Ramp up.  So we started ramping up.  This is an illustration of how we ramped up over time.  The time area right now is beginning mid-2014.  So we started scaling against multiple offices.  And so we scaled by adding consultants, basically, because it's really hard to get good developers locally.  And added smart developers, good people that became part of the team.  But something happens when you throw many people into a monolith.  So even with the Spotify model and starting to split up people into squads and talking about business domains,  we still had one code base.  And now we had many, many, many developers working in that code base.  And if you've ever worked in a shared code base, I know Facebook can do it, and I know Google can do it.  But really, we are not Facebook or Google. We're not that clever.  So what happens is that one developer does something to make some new feature or fix some bug and then give car breaks.  And another developer does something else, and then that doesn't merge with his thing.  And suddenly you have long-living feature branches that are weeks out of master, and you need to rebase them,  and you can't rebase them because things have changed horribly.  And it's just not working at all.  So what do we do? What do we do about this?  Well, our problem was that when we actually managed to merge something and deliver some deployable thing,  and we put it on our test systems and it never worked, gift card always breaks, and everything else always breaks.  So we had things living in our test systems for weeks before we could almost never,  but we could dare to put it in production and then figure out what breaks only in production.  It's a bad place to be in because with such a large monolith with many people working on it,  actually stabilizing it is a non-trivial task.  So we figured out that we need to change. We can't work like this.  We need to, just because we're not good enough, I guess, to make such a thing work,  we need to split it up into smaller deployables, just because we need to get something to production quickly.  And we sat down and we made an architectural vision for COMPLET, which is its COMPLET vision.  I'm not saying this is what everyone has to do, but we believe that for our domain and our organization,  we need to have a landscape of services. We need to have many services that live in the environments,  that connect to each other. We need those services to be stable in their contracts,  so we are not allowed to do breaking changes on services.  We actually go as far as to say, if you are breaking your contract, then you are making a new service.  You should not override the old service. You should deploy a new service beside it  and tell your clients to start using that instead.  We needed a consistent deployment, and we had to do this while our system was running.  So how do you do that? Well, you take whatever new part of the monolith you need to split out  and you make some service for it, just because you can't do it in the old one,  because gift card will break, and you put a feature switch around it.  So you deploy your new service and you make sure it works, and you can actually make it work  because it's smaller, so you are able to understand it.  Once we're happy with it, we've seen that it actually works in the development environment,  then we turn it on by the feature switch in the test environment and check that it works  and fix whatever doesn't work in test, because there's always something.  And then we do the same in production.  So we separate the actual deployment of the services from the activation of the services through feature switches.  So it's a strategic choice. It is done at a different level than the technical level.  As developers, we don't decide when something turns on.  That's up to the guys who are responsible for that part of the site.  So we have people who are responsible for the checkout,  and they decide when we turn on and off stuff in the checkout.  So that's how we decided to do it, and it's been working relatively good.  But the problem is, to create a new service, it sounds easy, right?  But it's not, because you need to make a new repository, you need to set up the build jobs,  the deployment pipeline, you need to manage the servers and make sure they're up and running  and ready to be deployed to you.  You may even need to create databases and all of that stuff.  And that meant that when we told the developers to please don't make this in the old one,  make a new thing, it's like, I don't want to do that,  because then I can't start actually solving the problem for a week or two,  because it takes me that long to actually get it out and running.  And often they kind of short-cutted it, so they got it running in the development environment,  and nobody really cared about test, because it's not going live for a month, right?  And then you have to take it live and it doesn't work in test.  So what we had to do is we had to make something that can make a service.  So we debated the naming of this.  One time it was the service maker, service maker, but it's now just the service maker.  I think we also called it foundry at one time.  Well, it doesn't matter.  So it's an interesting little thing that's been very, very important for us.  So we have a developer who needs he or she, needs to create a service,  so they go to the service maker.  And the service maker is actually a website on our internal network  where the developer enters the name of the service  and whether or not that service is going to need a database.  So they go to the service maker, and they press the create button,  and the service maker goes out, connects to Bitbucket in our case  and creates the Git repo.  It connects to our build machine, which is Jenkins for now,  and it connects to our deployment machines, which is Octopus for now,  and it sets it up on all of those.  It also goes to the database service and creates new databases,  and the build service is allowed to use another service's database.  So if you need the database, you get a new one.  Yeah.  Once it's done that, it actually then commits a template of a service to the Git repo.  That's picked up by the build server, which builds the service,  because that's what build servers do.  Then it pushes that to our deployment server, Octopus,  and that then deploys that service to all environments.  So after about 30 seconds from the create, they have an actual running service all the way.  This has made it immensely easier to convince developers  that you should create a new service instead of doing it in the old one.  So what we've learned is that if you are going to have a service architecture,  creating services must be cheap.  So we've talked about how we do this technically,  but how do we actually do this from a process and business standpoint?  How do we actually push work through our system?  So we mentioned that the Spotify model brought us this far.  But we were feeling pain, right?  And being strictly location-based didn't really work for us.  We didn't have that many developers, or it just didn't work for us.  It's not how our organizations grew together.  So we went back and looked at our model,  and we thought we don't like those squares or things, we created circles.  Circles are much better.  This is what we would like to call the complete model.  It's circular.  This is actually a snapshot of how the team is basically screwed together today.  And just to kind of walk you through this, each circle is then a person.  Each small circle.  Each small circle is a person, each white circle.  Each color circle is a squad.  And each squad has a color which identifies them.  Not a name, not a feature, not a business domain.  Because you're not guaranteed to work in the same thing forever.  So each squad has a color, and we keep them as stable as possible.  You want the same people working on the same part of the domain.  So a squad was built up of four to six developers with their own dedicated QA.  So someone who's expert at testing.  How they solve QA within the squad, that's up to them.  But that's what we have.  We have two dedicated DevOps, and they're outside the squads.  And then we have the orange squad up there, which is our infrastructure team.  Each of these squads are actually focusing on delivering business value.  And they were working within a feature set.  So, as an example, the red squad here is solely responsible for the checkout and nothing else.  The black squad is solely responsible for the shopping cart and what's involved there.  And these are now different parts of the code.  This is something we've split up.  We split up the code into a different deployment unit,  and that squad is responsible for delivering the value there.  And each of these groupings, these larger circles,  they are a logical grouping across multiple domains.  So in the Spotify model, this would be similar to a tribe.  Within this tribe, we then have one or more UX responsible people who speak to the business experts  and gather requirements and work with them directly.  And we also have a delivery manager.  So this is basically how we're set up now.  And the infrastructure squad is responsible for making the lives of each other squad a lot easier.  So we make the service maker. I'm in the orange squad.  We make the service maker and stuff like that.  So you're the service maker maker.  Oh, yes.  So what we've really done here is we like to call it the reverse Conway.  Because Conway's law says that any architecture will kind of reflect the organizations and the petitions in the organization.  So we've tried to petition the organization the way we want the architecture to be.  If that makes sense.  But again, we've talked about people, we've talked about scaling, we've talked about everything, but how do we actually work?  So the real key here is basically over how we've learned experience is communication.  And none of these squads are co-located.  So each squad has always got members from at least two locations.  And one, I think, actually has four, which is almost as many as there are people in the squad.  Not counting people working from home.  Yeah.  And this is where we really break from the Spotify model.  There's reason for this, right?  The reason is bringing aboard new members, especially when you're in remote locations, is then it's really hard for them to actually get the domain, get the culture.  And by having a squad which is spread with representatives in the main office in Norway and with other offices,  then it's easier to actually spread knowledge, bring people aboard, and the onboarding process becomes easier for everyone.  But communication is still hard, right?  We still need to have some way of bringing new developers that aren't nearby, near to us, I mean closer to us.  So what we have done, one of the things we've done is basically set up a window into each office.  And we have something called a pyramid for this.  That's me.  And that's Thomas.  Yeah, so we have multiple of these screens where we can peer into each other offices.  We're not kind of watching what people do.  This is just about glancing and seeing who's there, seeing what's happening.  Are they having cake?  And that's a lot of fun having cake.  Because then we're standing on the other side and looking at the camera. We want cake, please. Send cake here.  We dance sometimes and yeah, we have a lot of fun.  So this gives us kind of a window into each other.  It's easier just to say, you know, we're there all the time.  And beginning was awkward, but now it's natural.  And again, continuing on communication, we have something called Lodok.  It's a lot like Slack, except this works.  So, for us, for us, please remember green, green notes only.  It's just the major difference between this and anything else is it's not as fancy, but we have something called threads or flows, which makes communication a lot easier.  This is a shot from our water cooler. You see a very beautiful man over there.  He's known as...  Pretty Thomas. I'm the other one.  Thank you.  And if you... I'm not sure if you can see this, but if you see the rooms on the side, the rooms are basically whatever we need in regard to squads and structure.  We have other rooms like interest rooms, like the checkout room over there, and then we have something called code upon L.  So this is the code panel. It's something we do on YouTube.  Then we have something called KDC, which is our complete developer conference.  This is where we gather together yearly and do things together.  So basically we create channels based on whatever we need, right?  And we have J4s, yes, I said J4s, on the water cooler.  We also use a lot of Skype and TeamViewer for pair programming.  So the main learning here is... I mean, it should be obvious, but organization matters.  But we tend to not care sometimes, right?  So it just emphasizes organization really does matters.  So we're making a landscape of services.  That should be easy, right? Each service is a small thing.  It's not hard to make a service. We've made sure of that.  And then let's just do it, right?  Well, it turns out there are different problems with services than with monoliths.  So how do we make sure that our services are actually up?  How do we make them perform?  How do we make them scale in ways that are relevant to these services?  Well, each service has an SLA to its clients.  We use a lot of caching to make sure that they perform at that level.  And we are very, very into performance testing our solutions.  So whenever we launch something new, it has been performance tested quite strenuously.  But we also need to have the services have some common characteristics.  And we make that easier through the service maker,  but they need to do stuff like they need to have known health endpoints  so that our infrastructure can make sure that they're up  and say that it's safe to actually send traffic there  because services go up and down all the time.  And we need them to log to a centralized logging service in a specific way  so that we can recognize requests throughout different services.  So if you actually go to our frontend and do something,  that might be five services, ten services,  involved in serving your request in some way.  In a monolith, that's quite easy to log because you just log  and it's all from the same place.  But when you are in a service landscape,  it gets really hard to figure out what actually happened  and why didn't it work.  So we've instituted standards on that, ways we log,  and to make this easier, we've made that a library that we have  in our internal nugget server that's actually put into each new service by the service maker.  So we use the service maker kind of to drive the service behavior that we want.  If you make a new service now, it already has the endpoints that we expected to have,  and it already has all the logging mechanisms.  You can just log as normal and it will log everything that we expect to find.  So we've had to do quite a few of these things,  and that's what the Orange Squad does.  We figure out what turns out to be the best practice that we want our systems to follow,  and then we pull that out of the projects that started doing it,  usually as a nugget package or something,  and we put that in the service maker so that from now on all new services do that.  So the learning here is that services aren't monolith,  they are different durr, and they have different needs,  and they have different characteristics, and you need to be aware of that.  So where do we go from here?  One thing we haven't really spoken about is what we do within each squad technologically wise.  We are kind of connected to an existing technology stack.  We are used to working with.NET, we used to working with knockout,  is what we have in our main store now.  But because we now started splitting things into different autonomous units,  we are now available to have the opportunity to work on completely new technologies.  So, which is what you touched on, right?  So usually learnings come through squads doing new things within their domain,  within their solutions, and talking to other squads saying that's a good idea.  Why we also need it?  And they might start implementing it, and then this is what Orange short comes in, right?  So what would be cool though is having fully mandated squads,  or completely autonomous squads, where we actually then pull in our operations,  we pull in the business users, there's no people in between.  So we are really interested in trying to have complete business units, including developers.  There will be more microservices, but how small they will be is something we are unsure of.  Because we are doing our best to stick to the business domains in regard to what we are spread out.  We think about business context, we think about our domain, right?  But we have to take some baby steps along the way.  So we are not religious about the size of our microservices.  Size shouldn't matter, right?  But we need to have them smaller than our huge monolith, that we know.  So we are also looking into exposing some more APIs, right?  We want to expose public APIs.  I hope we get there soon, but we should get there.  We don't think we can make everything, but we have a platform of lots of things,  and it will be really cool if someone made Android apps or iOS apps even.  What other way around?  I like Android.  And obviously more business areas.  So just to summarize, I mean, scaling is hard.  And we have to find our own way. We are still finding our own way.  We are not there yet, but we are getting there.  Culture is crucial, right?  This is number one in regards to getting people on board and actually building a team.  So we also want to really embrace and learn from failures.  This is really important, especially across cultures.  Failing should be something everyone does by default.  It should be the default way of approaching things.  And that's actually really hard when you are working in several different cultures,  because different cultures have different views on failure.  So that's been one of the things we had to work on.  So we now call them fabulous failures, just to try to make it a bit better.  It's a good thing to fail, when you've learned something.  And we're going to continue failing.  And hopefully, we're also going to continue sharing as we go.  We started doing things in public. This is one of the things.  And we'll continue as much as we can.  But one of the greatest success factors, which you can feel on a day-to-day basis,  is are your developers or is everyone happy?  It's because happy people make great stuff.  So make them happy.  And then they will probably make good stuff.  So, let Denise take a picture.  I can send you out first.  So we'd like to get feedback on this.  We want to hear more from you guys.  We want to hear if this thing makes sense, right?  Yeah, contact us on Twitter. Talk to us now.  You can also follow us on the CodePanela, which is a YouTube stream.  We try to do it at least once a month, perhaps twice a month.  It's in Norwegian, sorry, for our non-Norwegian friends.  You really should learn it. It's a great language.  So easy.  But we try to interview people and do stuff there.  And Pavel has a blog that you really should read.  It's not Norwegian. That's in English.  Yeah. And I'm just on Twitter.  So I hope you guys have learned something or got some ideas or inspired to do something.  Or have feedback for us. That'd be awesome.  We want to go more into detail moving forward.  And we hope you guys leave green notes over there or press the ring button as many as possible.  And if nobody wants to go home, we can have lots of questions today.  So...  It's in the tens. It's not in the hundreds yet.  But we believe we will reach at least 100 by the end of the year.  All written in the same language?  Mostly.  We are, as I said, we have this service maker which makes it really easy to make services with the technology that we usually use.  We have a few Java services that we are phasing out, actually.  Because C-sharp is cool.  And I have a bet going on.  So I want to be the first one to actually produce an F-sharp service.  There is no requirement on that.  So you're open for a new API?  Yes, we are.  But the easy path right now is.NET.  So that's the thing most of our developers do.  You're not too worried about getting the whole set of languages?  We are moving towards deploying our services to service fabric.  That's what we're now looking at.  Not quite sure whether it will work or not.  But in that you can run anything you want, as long as it's executable.  It can run, they say.  That is the thing that's further down the line.  I'm sorry?  I'm just thinking further down the line.  When you have first-person people, you can have a switch, then you can start.  Yeah.  That's one of the things with smaller services.  One of the kind of goals is that they should be so small that if there's a bug, you can delete it and start a new.  And that's not a huge cost.  I'm not sure that many of our services are that small yet.  But it's certainly something we're hoping to get towards.  But as I said, we don't think there's a magical, this is the right size for a microservice.  What are the examples of services you have?  We have, of course, a catalog service.  That is our catalog.  We have services for actually placing orders.  It's for getting users, for doing all of these small things.  We have some services that are more like verticals.  So as I said, our checkout is a separate site.  It looks like you're still on the same site, but actually you're not.  And in these days, we are...  My page, or your customer page, is also a separate site?  Yeah, with separate services underneath that, that handle the storage.  Right now, we are actually in the process of deploying a new checkout for our BTB customers.  That is on a different technology from our other checkouts.  We have two different checkouts on different technologies.  Not very different, but different.  Different from a second.  Yeah.  And that's kind of what we want.  We want to be able to test out news things.  How many versions of these services do you have on the network?  Yeah.  How do you name them?  Well, we named them...  Sorry.  So the question is, how many different services, different versions of services do we have deployed?  Since we say that we don't want to deploy...  I'm sorry.  We don't want to deploy over.  We want to deploy side by side.  So this is not something we are very good at yet.  So we are working on that.  And it's also something that becomes more important as things stabilize.  It's in effect when people are moving quickly.  They are usually communicating on both sides of this communication.  And then it's not that important to us.  So the things that are stable are the things...  And in use are the things we don't overwrite.  But we have some things that are in different versions, but not a lot of them.  Because we go into different kinds of problems there with the data storage, for instance.  They are not allowed to use the same data.  It's not as easy as it sounds.  I think our time is running out.  We have more questions though.  So we can take one over here now.  You, on your diagram, about how you are organized, you have squads and in a larger group of squads, you have two squads.  So the question is, if I understand correctly, if this is representative of how we are organized today.  We have two tribes, which represent the two areas of our domain.  How we see it now.  Based on how the business is also structured.  We have a before buy.  Everything about getting a customer into the store.  It's where we do a lot of scaling, a lot of work on presenting content.  And we have a kind of during slash after buy.  Where we do a lot of interactive pages.  We do the checkout, you do your cart, you do your MyPage, that type of stuff.  And yeah, the orange squad is something...  You have seven people on the each squad.  So in a way, we have a number of people on each squad.  We have around between four and seven, basically, depending on how we do things.  And we do, when you start something new, we do a task force approach.  Where you may split off two people, three people.  They start working something, just to get the service up and running.  And then the rest of the squad just finishes up and joins them.  They stay within their tribe and do different...  Each squad has at least one dedicated QA.  And the rest are developers.  There's one lead who is more responsible for the delivery.  But he or she is usually a developer.  And how they solve this world-wise is up to the squad.  So the question is, how do microservices communicate with each other?  Mainly by rest.  So they call each other.  This is one of the things we are working on changing now.  Because we think to be able to scale, we need to push information out  instead of having them get it when the customer is actually coming in.  So we are reworking to get a push architecture where when something happens  somewhere in the system, a message is sent.  And then the services that are interested in that, they do their work.  So that when the customer actually kind of knocks on the door,  we don't have to build the store.  It's ready for them.  This is also one of the things we are now discovering  as we move to a service landscape.  Because if you do a lot of requests, then you end up in a situation  where you deploy something new.  And if one service is down or unstable, then that kind of bubbles all the way out.  So instead of that, we want it to be so that when a service,  when some event happens in the system, it propagates it all the way out.  So that if something goes down in that pipeline, it should still be up  as far as the user discovers.  You might not get the latest updates.  So you might see somewhat stale data.  But at least we will have data.  Okay.  How much is left of the monolith?  The question is how much is left of the monolith?  I should have checked that.  Quite a lot.  Quite a bit.  Quite a lot.  We still have XSLT.  Actually, there are pages where you will be served by the XSLT.  But mostly you can avoid them, right?  Most of the shopping experiences actually been moved over.  Yeah.  It's things like the feeds and stuff that's not really important.  But the main things that a normal user encounters,  those are, well, on the new technology.  So, yeah, we have one more.  When a squad develops a service, do they have to keep maintaining it during the  lifeline?  Or does that responsibility transfer to another squad?  Yeah.  So the question was, when a squad develops a service, do they have to maintain it  for the lifetime of that service?  Or does it move to a different squad?  And the answer is that it's there.  So there are certain cases where, for instance, if one squad has an area of  responsibility and they don't have time to develop something that has to be  developed, then a different squad can do that and then transfer it.  But usually they own the service as long as it's alive.  But since we've elected to do it by business area, it might be that the  people who are responsible for master data will have to maintain something  someone else made.  Okay.  Thank you, guys.  Thanks so much for being here.  Have a nice weekend.  Green.
Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_0_59.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_60_779.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_780_1235.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_1236_2003.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_2004_2867.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_2868_3743.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_3744_6155.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_6156_8075.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_8076_11315.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_11316_13979.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_13980_14495.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_14496_15047.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15048_15167.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15168_15275.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15276_15323.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15324_15371.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15372_15467.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15468_15515.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15516_15623.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_15624_17675.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_17676_22883.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_22884_24995.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_24996_25067.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_25068_25295.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_25296_25343.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_25344_25415.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_25416_25523.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_25524_25595.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_25596_28607.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_28608_30215.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_30216_31475.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_31476_32471.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_32472_32543.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_32544_32639.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_32640_32795.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_32796_32843.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_32844_32891.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_32892_33071.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_33072_33323.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_33324_33395.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_33396_33443.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_33444_33587.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_33588_33731.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_33732_34763.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_34764_37307.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_37308_38915.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_38916_40307.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_40308_40907.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_40908_41315.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_41316_42923.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_42924_48263.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_48264_48695.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_48696_49475.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_49476_50855.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_50856_51359.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_51360_51467.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_51468_51515.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_51516_51647.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_51648_52907.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_52908_60371.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_60372_61535.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_61536_63779.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_63780_63827.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_63828_63875.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_63876_64067.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64068_64139.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64140_64367.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64368_64439.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64440_64499.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64500_64583.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64584_64667.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64668_64715.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_64716_69215.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_69216_69299.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_69300_69347.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_69348_69527.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_69528_69647.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_69648_73271.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_73272_75443.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_75444_76031.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_76032_85679.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_85680_85727.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_85728_93803.png Making_Komplett_big_by_going_small_-_Pavneet_Singh_Saund_Tomas_Ekeli_93804_94023.png
So, hi everyone. Welcome. Thanks for being here at the very last talk and our talk. Are we ready to start? Yeah, I think so. So, I want to start with us. Okay, so thanks for coming to our talk titled Making Complete Big by Going Small. Who are we? So, I'm Pav Neet. I'm a web developer at Komplett. I'm also a squad lead. We'll get into that a little later. And my name is Thomas. I'm a lead software architect on the web team in Komplett. And what are we going to talk about? Feedback? We are going to talk about how we've scaled our team and our group as we've changed. We've gone from a small group to a rather large one. And we've had to make changes both to our architecture and our organization in this process. And that's what we're going to talk about today. So, it's kind of an experience talk. Right. So, Komplett Group. For those of you who aren't familiar with us, we are a group of web stores. We have 16 web shops active now. We have around 7.3 billion crowns revenue. Just under a billion dollars. 1.8 million active customers, 800 employees and around 20 million uniques. We suspect that means unique sessions. In 2015. So, we, these are some of our stores. It may not be that easy to see. But we're traditionally an e-commerce electronic store. But you may be able to see that we've now branched off into travel, insurance, car parts, baby equipment. Amongst other things. And just last week we started the mobile phone company as well. So, things are changing. But we've not always been as big as we were yesterday or today. We started in the mid 90s. And then we had a very simple architecture. So, you, we had an online catalog. And as a customer, you contacted the sales department. So, you actually phoned them. I know it sounds quaint, but that's how it was done. You phoned them and they called out into the office and said everybody get off the system because I have a customer. It was a one user system where we placed orders directly into the system when we had the customer on the line. That didn't scale very well. So, we went on and created what would become our future web platform, Chrome. We called it Chrome in the early 2000s. So, we were before the browser. If anybody mentions Chrome, it's us. It's our name. Okay? At least for this talk. So, the Chrome 2B, it wasn't called Chrome at that point. But it was one of the first e-commerce stores in Europe at least that actually showed you an online stock status. So, you could go to it and see if we had the product in stock. The customer connected it directly and the orders were placed in the order database. Still a quite simple architecture. Then, we got a bit over ourselves and we bought an ERP system, a German three letter acronym. Shout at you. So, now we had to, from this point on, it's actually called Chrome. So, now we had to have a system that kept up better off time than this German system because apparently such systems go down. So, now we had to have a message queue talking to the system. And Chrome placed orders through messages and the ERP system got them from there and populated our database directly. So, that was our main architecture up until, well now really. Pretty recently. So, we're really still here but we have done some changes and we are working to change it. But what Chrome did at this time was, and this will blow your minds, we connect to the database, get an XML back and then we read that in and we do an XSLT transformation on that database, producing HTML and DTD format. So, this is actual running production code from May 2008, not today. It's much better today, right? No, no, it's not. So, this does three things that I want to point out. First, we have a static method returning a string. This string will go directly onto the response. No controllers, no nothing. It's very efficient. And what it does is that it goes directly to the database. There you go. Yeah, it goes directly to the database, populating parameters or arguments. I don't know, I think the ones on the left are parameters and the ones on the right are arguments. Well, as you can see, it shouts at the database because the database talks SAP. And we populate the stored procedure called directly. We actually make an XML header down there. And what we get back, if you go to the next one, is then run through an XSL transformation and that's put directly onto the response. This is how you create a million-dollar industry people. This is high-tech. Okay? So, if you were wondering, we weren't really very happy with this. It's nasty and it's error-prone and it's global state all over the place. And every time we did something, we ended up breaking something else somewhere on our side. Usually gift cards, they always break. So, we'd add a button somewhere and gift card stops working and we remove a button and then gift card stops working. It's horrible, horrible. So, what do we do? Well, like Thomas mentioned, we are at a bad place here. What does that mean, a bad place? This platform has served us for many, many years. The team, if you look at the team, we were five developers at this point. We were sitting in each of our corners in our room. Yes, we had a pentagonal room, yeah. Doing each of our own things based on business needs, which just sounds fair enough, right? Because we prioritized things based on the person that shouted the loudest and whoever shouted loudest got their things through. So, as we did things, based on what you just saw, we started to grind to a halt in regards to our speed and our velocity. So, we stopped delivering features, or slowed down delivering features, and started living more bug fixes. Usually on gift cards. And when he didn't fix gift cards, we fixed something else and broke it. So, if you ever get a gift card, you can hope that it works. Sometimes it does. It's been fine, hasn't it? This is fine, right? So, basically, we were struggling. We were at a point where we needed to do something. We weren't really feeling inspired. And along came a business need. Yeah, so this was when mobile phones started to get popular. I think you may remember that time, the heady days of the first iPhone. So, we needed to be able to show a mobile site that worked on a mobile phone. So, either we could produce an app, or we could produce a website that actually worked on mobile phones. But we needed a mobile presence. It was a growing market, and business came to us with this need. And we said, sure, we can do that. We just need to rewrite everything. Because if you've ever done XSLT transformations, taking XML and making DTD for HTML out of it, it's not really conducive to mobile phones rendering. It's nasty. It's challenging. So, what did we do? Well, then let's talk about Project K2. So, the case for complete, two is for two. Yes, we are good at naming. There's more on that later. So, Project K2 is our response to the business need. And the business said, we need a mobile presence. We translate adapt to be, we need a mobile first web store. So, at this time, responsive web, or adaptive web as we like to call it, but we lost that battle a long time ago. Responsive web was in its, I wouldn't say infancy, but it was starting up. We didn't have cross-browser support, and it was a little unsure. Another approach at this time was building standalone websites, which is what we went for. And that's also what we proposed. It was a good thing because that meant we didn't have to touch the old code. So, we had a... That's a really good thing. Yeah. Which meant we could do a file new project. And we thought, okay, this is great. Now, we can finally focus on the good stuff. Do it right. Do it right. Yeah. Buzzwords. Anyone recognize these? We did all of them. Yes. So, that's it. We went for code quality. We went for the solar principles. We were going to be test driven. We pulled in CQRS, DDD, event sourcing. And then there's the technical side. Then obviously on the process side, we introduced Agile and Scrum. We did pair programming. And with all this came new admin pages. A new infrastructure, a new architecture, as you see. And obviously a new deployment pipeline. It will be easy. Yeah. And what this meant was we're taking our old monolith, turning around, and creating a new one. And it was actually pretty good. We learned a lot. We learned to work as a team. We learned to write better code. We learned a lot about Agile processes. And we delivered the M-complet NO, the first mobile version of our store, within around six months. We made it. We did it. And we made it work. It was good. But what did this mean? Six months. But it didn't have gift cards. Oh, yeah. So we couldn't break anything. That's why it worked. So we had basically, all five of us had gone over to be, were working 100% on this platform, which meant six months without any new features on the existing desktop platform. Or bug figures to gift cards. Which meant that the business owners had to wait, right? So they were fine. Let's wait for this. We delivered this. And then they were eager. Cool. Now we can focus on fixing gift card. Couldn't leave gift card. It's nightmare. And we thought, okay, we got the feature request in. We started looking at what was involved. And we realized we didn't want to go back to this XSLT. We don't want to go back there. So we thought, we've more or less created a new web store. Let's do a little more. Let's create the desktop version of this on the new platform. Should be easy. Should be easy. And what happened is the business never asked for this. But we got a natural accept. We said it'll be fast. We'll see what it did in six months. We'll get used to it in six months and we were ready. And we ended up, well, building out. What we have to do is we need to get gather requirements because we didn't know what's happening in the old system. And the thing is neither did the business. They didn't really know all the quirks. I mean, when you have a system running for 10 years, there's so many small quirks. So the spec is the old system. Make it like that one. And that's what we did. And that's what they told us. So how should this work? We can do this. We can do something nice in Chinese. We can not implement gift cards. No, that was an option. But they said, look at the old system. Do what that does. And this led us to a feature parity that much. Yeah. So we spent, I think, one and a half years trying to reimplement the entire Chrome platform in K2. Just building functionality we already had. And remember, while we were doing this, the old desktop that actually was earning us money, the one that was important, it had now not been touched in two years. And the mobile platform that some very, very few users actually used because they don't like to go to M sites for some reason. And it had very, very few features. That also wasn't being worked on because we were working on the new shiny stuff that hadn't been deployed and hadn't been activated in any way yet. So after two years of this, the organization came to us and they asked, hey, guys, we kind of need to do stuff. Are you finished soon? And we said, we don't know. We didn't even know which features we were missing because nobody knew. So they did the only same thing. They killed it. Yeah. So that's like two or three years of your life down the drain. So if you learn one thing from this, rewrites are bad. I'm not saying that they can't work, but it's really, really hard. And if you can find a different way of doing it, please consider it. So this was a very dark time for us in the web team at Komplett. We had five developers when the project was killed and we lost half-ish of those. So one of the largest e-commerce sites in Norway suddenly didn't have a lot of web developers. We'd lost our self-confidence. We'd lost our respect in the organization. The organization then embarked on a year-long search for some platform. We still want to be a web e-commerce thing. There must be some platform that we can buy. And there are. There are platforms out there. Most of them have three-letter acronyms. So we launched a rather large project to figure out which one is best for us. We got proposals in where we kind of played them against each other. And we saw who had the features we needed, who had the support we needed. And after a year of this, we had, I think it was three finalists. And the web team was asked to just submit a proposal of our own for our Chrome platform, just kind of to have a baseline, because we knew that nothing could be worse than that. So we had to have something above that. And then on, I think it was a Wednesday, we actually got the call that we won. So we have it on great authority that we are the best in the world at this. Do you remember what time it was? I don't remember the exact time. So what does that mean? What that meant was they still trusted in us, or they put their trust in us rather. But what that also meant is there are new business domains that they wanted to go into. They wanted to do new things. Well, actually not day, we. And we still had a great heap of code. We hadn't touched this in actually three years now, right? And we still didn't have a mobile platform. So we're three years behind the rest of the field. And we're asking ourselves, can we do this? I mean, can we do this? We were actually really unsure. Three years earlier we just said we couldn't do it on Chrome. And now we had to. So how could the same people get trust and deliver? That's what we have to ask ourselves. So they trust us, meaning the business and the organization trust us. But do we trust us? Can we trust in ourselves? Can we get to a place where we can make this work? So they're saying that culture eats process for breakfast. So that's where we started. We started with culture. We started with values. We started with defining who we are as developers. And what it meant to be a developer in Scandinavia or Northern Europe's largest e-commerce provider. So we did this. We spent time at this one of our workshops. And we end up with 10, 11, I can't count. A number of values. These values are things that we believed in. Things that we embodied. Things we felt that we already have, some of them. And other things were things we aspired to be. This is for ourselves. And this is also for new developers coming on. Because this is the point, right? We can't scale with us five. We needed to increase and we need to expand. So? Yeah. So then we did the next thing. We just added lots of developers, right? So business came to us with needs. So the entire organization had needs. Our customers had needs. And we weren't able to, we didn't have the manpower, the muscle to do it. So we needed more people. And we hired a lot of more people. The good thing was that we'd already thought about cultures. So we didn't kind of just fall apart on that. But just adding people or developers has its own problems. So at this time, we were like five developers. And we had a QA guy and we had two ops. And that was the web team. So pretty small, right? And we needed to scale. So we looked around. And if you've ever searched about scaling agile, there's a lot of talk about that. But there's a kind of local firm that seems to do this pretty well. So we looked at them. They're not Norwegian. They're Swedish. Spotify. You might have heard of them. So around this time, they published about their Spotify model. Have you heard of it? Raise of hands. Yeah, good, good. Then we can go quickly through this. Yeah. So this is a part of the Spotify model we've tried to cut out. It's just to go quickly through this. It's basically built around autonomous squads who deliver to a specific business need with their dedicated product owner. They're autonomous, meaning they have control over their build pipeline. They have control over everything, deployment, all the testing, and direct access to whoever is the representative on the business side. These squads are then grouped together in tribes. And these tribes usually represent a logical business area, so crossing different business domain. So across squads, we then have something called chapters. Chapters basically meaning you could be interests, right? Front-end interests. It could be older DevOps. It could be something domain specific. And then we can have multiple tribes. This is how Spotify did it. One of the core features or one of the most important aspects of the Spotify model is that each team is co-located, right? So each team has to be at the same area, same place, same office, and work together. So we took this, and this is awesome. This is a blueprint. This is something we could use for ourselves, right? And we take this, and we start with our major project called the Minion Project, which is a... We can get to that. So this is Bob. He's in our office. Yeah. He runs around all the time. Fun guy. So the Minion Project is to not do a big rewrite. It is to change Chrome while it is living and make it mobile-friendly. It's nearing its end now, actually. So if you go to our sites, they're actually quite good, I think, on mobile. That's what the Minion Project is. And we had said we couldn't do this years earlier, and then we couldn't. But that was three years ago, which is an eternity, an internet years. So now we actually could. Now we had the technology. Browsers were sufficiently advanced, and we might have learned a thing or two. But still, we needed to produce at a rate that we couldn't as a few. And with the Spotify model, we thought we were able to scale our team. Ramp up. So we started ramping up. This is an illustration of how we ramped up over time. The time area right now is beginning mid-2014. So we started scaling against multiple offices. And so we scaled by adding consultants, basically, because it's really hard to get good developers locally. And added smart developers, good people that became part of the team. But something happens when you throw many people into a monolith. So even with the Spotify model and starting to split up people into squads and talking about business domains, we still had one code base. And now we had many, many, many developers working in that code base. And if you've ever worked in a shared code base, I know Facebook can do it, and I know Google can do it. But really, we are not Facebook or Google. We're not that clever. So what happens is that one developer does something to make some new feature or fix some bug and then give car breaks. And another developer does something else, and then that doesn't merge with his thing. And suddenly you have long-living feature branches that are weeks out of master, and you need to rebase them, and you can't rebase them because things have changed horribly. And it's just not working at all. So what do we do? What do we do about this? Well, our problem was that when we actually managed to merge something and deliver some deployable thing, and we put it on our test systems and it never worked, gift card always breaks, and everything else always breaks. So we had things living in our test systems for weeks before we could almost never, but we could dare to put it in production and then figure out what breaks only in production. It's a bad place to be in because with such a large monolith with many people working on it, actually stabilizing it is a non-trivial task. So we figured out that we need to change. We can't work like this. We need to, just because we're not good enough, I guess, to make such a thing work, we need to split it up into smaller deployables, just because we need to get something to production quickly. And we sat down and we made an architectural vision for COMPLET, which is its COMPLET vision. I'm not saying this is what everyone has to do, but we believe that for our domain and our organization, we need to have a landscape of services. We need to have many services that live in the environments, that connect to each other. We need those services to be stable in their contracts, so we are not allowed to do breaking changes on services. We actually go as far as to say, if you are breaking your contract, then you are making a new service. You should not override the old service. You should deploy a new service beside it and tell your clients to start using that instead. We needed a consistent deployment, and we had to do this while our system was running. So how do you do that? Well, you take whatever new part of the monolith you need to split out and you make some service for it, just because you can't do it in the old one, because gift card will break, and you put a feature switch around it. So you deploy your new service and you make sure it works, and you can actually make it work because it's smaller, so you are able to understand it. Once we're happy with it, we've seen that it actually works in the development environment, then we turn it on by the feature switch in the test environment and check that it works and fix whatever doesn't work in test, because there's always something. And then we do the same in production. So we separate the actual deployment of the services from the activation of the services through feature switches. So it's a strategic choice. It is done at a different level than the technical level. As developers, we don't decide when something turns on. That's up to the guys who are responsible for that part of the site. So we have people who are responsible for the checkout, and they decide when we turn on and off stuff in the checkout. So that's how we decided to do it, and it's been working relatively good. But the problem is, to create a new service, it sounds easy, right? But it's not, because you need to make a new repository, you need to set up the build jobs, the deployment pipeline, you need to manage the servers and make sure they're up and running and ready to be deployed to you. You may even need to create databases and all of that stuff. And that meant that when we told the developers to please don't make this in the old one, make a new thing, it's like, I don't want to do that, because then I can't start actually solving the problem for a week or two, because it takes me that long to actually get it out and running. And often they kind of short-cutted it, so they got it running in the development environment, and nobody really cared about test, because it's not going live for a month, right? And then you have to take it live and it doesn't work in test. So what we had to do is we had to make something that can make a service. So we debated the naming of this. One time it was the service maker, service maker, but it's now just the service maker. I think we also called it foundry at one time. Well, it doesn't matter. So it's an interesting little thing that's been very, very important for us. So we have a developer who needs he or she, needs to create a service, so they go to the service maker. And the service maker is actually a website on our internal network where the developer enters the name of the service and whether or not that service is going to need a database. So they go to the service maker, and they press the create button, and the service maker goes out, connects to Bitbucket in our case and creates the Git repo. It connects to our build machine, which is Jenkins for now, and it connects to our deployment machines, which is Octopus for now, and it sets it up on all of those. It also goes to the database service and creates new databases, and the build service is allowed to use another service's database. So if you need the database, you get a new one. Yeah. Once it's done that, it actually then commits a template of a service to the Git repo. That's picked up by the build server, which builds the service, because that's what build servers do. Then it pushes that to our deployment server, Octopus, and that then deploys that service to all environments. So after about 30 seconds from the create, they have an actual running service all the way. This has made it immensely easier to convince developers that you should create a new service instead of doing it in the old one. So what we've learned is that if you are going to have a service architecture, creating services must be cheap. So we've talked about how we do this technically, but how do we actually do this from a process and business standpoint? How do we actually push work through our system? So we mentioned that the Spotify model brought us this far. But we were feeling pain, right? And being strictly location-based didn't really work for us. We didn't have that many developers, or it just didn't work for us. It's not how our organizations grew together. So we went back and looked at our model, and we thought we don't like those squares or things, we created circles. Circles are much better. This is what we would like to call the complete model. It's circular. This is actually a snapshot of how the team is basically screwed together today. And just to kind of walk you through this, each circle is then a person. Each small circle. Each small circle is a person, each white circle. Each color circle is a squad. And each squad has a color which identifies them. Not a name, not a feature, not a business domain. Because you're not guaranteed to work in the same thing forever. So each squad has a color, and we keep them as stable as possible. You want the same people working on the same part of the domain. So a squad was built up of four to six developers with their own dedicated QA. So someone who's expert at testing. How they solve QA within the squad, that's up to them. But that's what we have. We have two dedicated DevOps, and they're outside the squads. And then we have the orange squad up there, which is our infrastructure team. Each of these squads are actually focusing on delivering business value. And they were working within a feature set. So, as an example, the red squad here is solely responsible for the checkout and nothing else. The black squad is solely responsible for the shopping cart and what's involved there. And these are now different parts of the code. This is something we've split up. We split up the code into a different deployment unit, and that squad is responsible for delivering the value there. And each of these groupings, these larger circles, they are a logical grouping across multiple domains. So in the Spotify model, this would be similar to a tribe. Within this tribe, we then have one or more UX responsible people who speak to the business experts and gather requirements and work with them directly. And we also have a delivery manager. So this is basically how we're set up now. And the infrastructure squad is responsible for making the lives of each other squad a lot easier. So we make the service maker. I'm in the orange squad. We make the service maker and stuff like that. So you're the service maker maker. Oh, yes. So what we've really done here is we like to call it the reverse Conway. Because Conway's law says that any architecture will kind of reflect the organizations and the petitions in the organization. So we've tried to petition the organization the way we want the architecture to be. If that makes sense. But again, we've talked about people, we've talked about scaling, we've talked about everything, but how do we actually work? So the real key here is basically over how we've learned experience is communication. And none of these squads are co-located. So each squad has always got members from at least two locations. And one, I think, actually has four, which is almost as many as there are people in the squad. Not counting people working from home. Yeah. And this is where we really break from the Spotify model. There's reason for this, right? The reason is bringing aboard new members, especially when you're in remote locations, is then it's really hard for them to actually get the domain, get the culture. And by having a squad which is spread with representatives in the main office in Norway and with other offices, then it's easier to actually spread knowledge, bring people aboard, and the onboarding process becomes easier for everyone. But communication is still hard, right? We still need to have some way of bringing new developers that aren't nearby, near to us, I mean closer to us. So what we have done, one of the things we've done is basically set up a window into each office. And we have something called a pyramid for this. That's me. And that's Thomas. Yeah, so we have multiple of these screens where we can peer into each other offices. We're not kind of watching what people do. This is just about glancing and seeing who's there, seeing what's happening. Are they having cake? And that's a lot of fun having cake. Because then we're standing on the other side and looking at the camera. We want cake, please. Send cake here. We dance sometimes and yeah, we have a lot of fun. So this gives us kind of a window into each other. It's easier just to say, you know, we're there all the time. And beginning was awkward, but now it's natural. And again, continuing on communication, we have something called Lodok. It's a lot like Slack, except this works. So, for us, for us, please remember green, green notes only. It's just the major difference between this and anything else is it's not as fancy, but we have something called threads or flows, which makes communication a lot easier. This is a shot from our water cooler. You see a very beautiful man over there. He's known as... Pretty Thomas. I'm the other one. Thank you. And if you... I'm not sure if you can see this, but if you see the rooms on the side, the rooms are basically whatever we need in regard to squads and structure. We have other rooms like interest rooms, like the checkout room over there, and then we have something called code upon L. So this is the code panel. It's something we do on YouTube. Then we have something called KDC, which is our complete developer conference. This is where we gather together yearly and do things together. So basically we create channels based on whatever we need, right? And we have J4s, yes, I said J4s, on the water cooler. We also use a lot of Skype and TeamViewer for pair programming. So the main learning here is... I mean, it should be obvious, but organization matters. But we tend to not care sometimes, right? So it just emphasizes organization really does matters. So we're making a landscape of services. That should be easy, right? Each service is a small thing. It's not hard to make a service. We've made sure of that. And then let's just do it, right? Well, it turns out there are different problems with services than with monoliths. So how do we make sure that our services are actually up? How do we make them perform? How do we make them scale in ways that are relevant to these services? Well, each service has an SLA to its clients. We use a lot of caching to make sure that they perform at that level. And we are very, very into performance testing our solutions. So whenever we launch something new, it has been performance tested quite strenuously. But we also need to have the services have some common characteristics. And we make that easier through the service maker, but they need to do stuff like they need to have known health endpoints so that our infrastructure can make sure that they're up and say that it's safe to actually send traffic there because services go up and down all the time. And we need them to log to a centralized logging service in a specific way so that we can recognize requests throughout different services. So if you actually go to our frontend and do something, that might be five services, ten services, involved in serving your request in some way. In a monolith, that's quite easy to log because you just log and it's all from the same place. But when you are in a service landscape, it gets really hard to figure out what actually happened and why didn't it work. So we've instituted standards on that, ways we log, and to make this easier, we've made that a library that we have in our internal nugget server that's actually put into each new service by the service maker. So we use the service maker kind of to drive the service behavior that we want. If you make a new service now, it already has the endpoints that we expected to have, and it already has all the logging mechanisms. You can just log as normal and it will log everything that we expect to find. So we've had to do quite a few of these things, and that's what the Orange Squad does. We figure out what turns out to be the best practice that we want our systems to follow, and then we pull that out of the projects that started doing it, usually as a nugget package or something, and we put that in the service maker so that from now on all new services do that. So the learning here is that services aren't monolith, they are different durr, and they have different needs, and they have different characteristics, and you need to be aware of that. So where do we go from here? One thing we haven't really spoken about is what we do within each squad technologically wise. We are kind of connected to an existing technology stack. We are used to working with.NET, we used to working with knockout, is what we have in our main store now. But because we now started splitting things into different autonomous units, we are now available to have the opportunity to work on completely new technologies. So, which is what you touched on, right? So usually learnings come through squads doing new things within their domain, within their solutions, and talking to other squads saying that's a good idea. Why we also need it? And they might start implementing it, and then this is what Orange short comes in, right? So what would be cool though is having fully mandated squads, or completely autonomous squads, where we actually then pull in our operations, we pull in the business users, there's no people in between. So we are really interested in trying to have complete business units, including developers. There will be more microservices, but how small they will be is something we are unsure of. Because we are doing our best to stick to the business domains in regard to what we are spread out. We think about business context, we think about our domain, right? But we have to take some baby steps along the way. So we are not religious about the size of our microservices. Size shouldn't matter, right? But we need to have them smaller than our huge monolith, that we know. So we are also looking into exposing some more APIs, right? We want to expose public APIs. I hope we get there soon, but we should get there. We don't think we can make everything, but we have a platform of lots of things, and it will be really cool if someone made Android apps or iOS apps even. What other way around? I like Android. And obviously more business areas. So just to summarize, I mean, scaling is hard. And we have to find our own way. We are still finding our own way. We are not there yet, but we are getting there. Culture is crucial, right? This is number one in regards to getting people on board and actually building a team. So we also want to really embrace and learn from failures. This is really important, especially across cultures. Failing should be something everyone does by default. It should be the default way of approaching things. And that's actually really hard when you are working in several different cultures, because different cultures have different views on failure. So that's been one of the things we had to work on. So we now call them fabulous failures, just to try to make it a bit better. It's a good thing to fail, when you've learned something. And we're going to continue failing. And hopefully, we're also going to continue sharing as we go. We started doing things in public. This is one of the things. And we'll continue as much as we can. But one of the greatest success factors, which you can feel on a day-to-day basis, is are your developers or is everyone happy? It's because happy people make great stuff. So make them happy. And then they will probably make good stuff. So, let Denise take a picture. I can send you out first. So we'd like to get feedback on this. We want to hear more from you guys. We want to hear if this thing makes sense, right? Yeah, contact us on Twitter. Talk to us now. You can also follow us on the CodePanela, which is a YouTube stream. We try to do it at least once a month, perhaps twice a month. It's in Norwegian, sorry, for our non-Norwegian friends. You really should learn it. It's a great language. So easy. But we try to interview people and do stuff there. And Pavel has a blog that you really should read. It's not Norwegian. That's in English. Yeah. And I'm just on Twitter. So I hope you guys have learned something or got some ideas or inspired to do something. Or have feedback for us. That'd be awesome. We want to go more into detail moving forward. And we hope you guys leave green notes over there or press the ring button as many as possible. And if nobody wants to go home, we can have lots of questions today. So... It's in the tens. It's not in the hundreds yet. But we believe we will reach at least 100 by the end of the year. All written in the same language? Mostly. We are, as I said, we have this service maker which makes it really easy to make services with the technology that we usually use. We have a few Java services that we are phasing out, actually. Because C-sharp is cool. And I have a bet going on. So I want to be the first one to actually produce an F-sharp service. There is no requirement on that. So you're open for a new API? Yes, we are. But the easy path right now is.NET. So that's the thing most of our developers do. You're not too worried about getting the whole set of languages? We are moving towards deploying our services to service fabric. That's what we're now looking at. Not quite sure whether it will work or not. But in that you can run anything you want, as long as it's executable. It can run, they say. That is the thing that's further down the line. I'm sorry? I'm just thinking further down the line. When you have first-person people, you can have a switch, then you can start. Yeah. That's one of the things with smaller services. One of the kind of goals is that they should be so small that if there's a bug, you can delete it and start a new. And that's not a huge cost. I'm not sure that many of our services are that small yet. But it's certainly something we're hoping to get towards. But as I said, we don't think there's a magical, this is the right size for a microservice. What are the examples of services you have? We have, of course, a catalog service. That is our catalog. We have services for actually placing orders. It's for getting users, for doing all of these small things. We have some services that are more like verticals. So as I said, our checkout is a separate site. It looks like you're still on the same site, but actually you're not. And in these days, we are... My page, or your customer page, is also a separate site? Yeah, with separate services underneath that, that handle the storage. Right now, we are actually in the process of deploying a new checkout for our BTB customers. That is on a different technology from our other checkouts. We have two different checkouts on different technologies. Not very different, but different. Different from a second. Yeah. And that's kind of what we want. We want to be able to test out news things. How many versions of these services do you have on the network? Yeah. How do you name them? Well, we named them... Sorry. So the question is, how many different services, different versions of services do we have deployed? Since we say that we don't want to deploy... I'm sorry. We don't want to deploy over. We want to deploy side by side. So this is not something we are very good at yet. So we are working on that. And it's also something that becomes more important as things stabilize. It's in effect when people are moving quickly. They are usually communicating on both sides of this communication. And then it's not that important to us. So the things that are stable are the things... And in use are the things we don't overwrite. But we have some things that are in different versions, but not a lot of them. Because we go into different kinds of problems there with the data storage, for instance. They are not allowed to use the same data. It's not as easy as it sounds. I think our time is running out. We have more questions though. So we can take one over here now. You, on your diagram, about how you are organized, you have squads and in a larger group of squads, you have two squads. So the question is, if I understand correctly, if this is representative of how we are organized today. We have two tribes, which represent the two areas of our domain. How we see it now. Based on how the business is also structured. We have a before buy. Everything about getting a customer into the store. It's where we do a lot of scaling, a lot of work on presenting content. And we have a kind of during slash after buy. Where we do a lot of interactive pages. We do the checkout, you do your cart, you do your MyPage, that type of stuff. And yeah, the orange squad is something... You have seven people on the each squad. So in a way, we have a number of people on each squad. We have around between four and seven, basically, depending on how we do things. And we do, when you start something new, we do a task force approach. Where you may split off two people, three people. They start working something, just to get the service up and running. And then the rest of the squad just finishes up and joins them. They stay within their tribe and do different... Each squad has at least one dedicated QA. And the rest are developers. There's one lead who is more responsible for the delivery. But he or she is usually a developer. And how they solve this world-wise is up to the squad. So the question is, how do microservices communicate with each other? Mainly by rest. So they call each other. This is one of the things we are working on changing now. Because we think to be able to scale, we need to push information out instead of having them get it when the customer is actually coming in. So we are reworking to get a push architecture where when something happens somewhere in the system, a message is sent. And then the services that are interested in that, they do their work. So that when the customer actually kind of knocks on the door, we don't have to build the store. It's ready for them. This is also one of the things we are now discovering as we move to a service landscape. Because if you do a lot of requests, then you end up in a situation where you deploy something new. And if one service is down or unstable, then that kind of bubbles all the way out. So instead of that, we want it to be so that when a service, when some event happens in the system, it propagates it all the way out. So that if something goes down in that pipeline, it should still be up as far as the user discovers. You might not get the latest updates. So you might see somewhat stale data. But at least we will have data. Okay. How much is left of the monolith? The question is how much is left of the monolith? I should have checked that. Quite a lot. Quite a bit. Quite a lot. We still have XSLT. Actually, there are pages where you will be served by the XSLT. But mostly you can avoid them, right? Most of the shopping experiences actually been moved over. Yeah. It's things like the feeds and stuff that's not really important. But the main things that a normal user encounters, those are, well, on the new technology. So, yeah, we have one more. When a squad develops a service, do they have to keep maintaining it during the lifeline? Or does that responsibility transfer to another squad? Yeah. So the question was, when a squad develops a service, do they have to maintain it for the lifetime of that service? Or does it move to a different squad? And the answer is that it's there. So there are certain cases where, for instance, if one squad has an area of responsibility and they don't have time to develop something that has to be developed, then a different squad can do that and then transfer it. But usually they own the service as long as it's alive. But since we've elected to do it by business area, it might be that the people who are responsible for master data will have to maintain something someone else made. Okay. Thank you, guys. Thanks so much for being here. Have a nice weekend. Green.
Feedback? We are going to talk about how we've scaled our team and our group as we've changed. architecture and our organization in this Right. So, Komplett Group. For those of you who become our future web platform, Chrome. We called two is for two. Yes, we are good at naming. ready. And we ended up, well, building out. What ourselves. So they trust us, meaning the business Northern Europe's largest e-commerce provider. So this time, we were like five developers. And we to change Chrome while it is living and make it people into a monolith. So even with the Spotify are breaking your contract, then you are making a seen that it actually works in the development service instead of doing it in the old one. So be aware of that. So where do we go from here? microservices, but how small they will be is can make everything, but we have a platform of still on the same site, but actually you're not.