(Re-)architecting for Continuous Delivery
Humble, Jez
Continuous delivery can be considered a cross-functional requirement, rather like testability and maintainability. In the first part of this talk I'll explore the implications of continuous delivery for your systems architecture. The second part will present patterns that successful organizations have used to enable continuous delivery. In the final part, I'll discuss the architectural and organizational obstacles presented by by typical legacy "big ball of mud" systems, and what can be done to incrementally move away from them.
 Okay, so my name is Jess Humble. I work for ThoughtWorks. We do consulting and delivery for software and we have tools to help you with that. Mingle for agile project management, Go for agile release management and Twist for agile testing. I co-authored this book called Delivery and I'm here today to talk about architecting and re-architecting and continuous delivery and the relationship between these things. So it's not going to be a detailed kind of lots of code on the whiteboard talk. So if you want something very detailed and codey, now would be a good time to go and see one of the other things that is more detailed and codey. I'm not at all offended if people decide to walk out and go and see something else instead. If that would be the best use of your time, please go ahead and do that. Don't feel bad at all. Also there's voting at the end. So at the end, please choose a green card, a yellow card and a red card depending on how you feel about the talk. Remember that in China red is lucky but in Norway not so much. I've been having a great time in Norway. It's my first time here. I went to Bergen on Monday and I got sunburn which was unexpected but very nice. So architecture. Architecture is a term obviously that comes from building buildings and if you're actually building, if you're doing construction, doing civil engineering projects, one of the important things as an architect you have to care about is what's the nature of the materials you're using. Iron, steel, wood, plastic, all these things have different characteristics and that affects the way you think about what you're building. And so obviously if you're building, if you're architecting things that have to stand up for a long time like a bridge, you have to think carefully about the materials that you're using. Now project management and architecture are focused around building things like this or buildings which have a certain set of characteristics. So once you build a bridge or a building, normally it doesn't fall down. That's bad normally. That means you've done a bad job. Once you build it, it doesn't change a lot. So we don't build a bridge and go, well that isn't really right and that isn't what we wanted. We're going to totally change the way that works now because our users didn't like that. Running a bridge requires different skills from building the bridge, completely different set of practices. Once we have built the, once we've designed the architecture for the bridge and thought about how we want it to behave, actually then building it will not give us lots of new information that makes us think again about the architecture. That's not supposed to happen when you're building bridges and buildings. And then most importantly when you're building a bridge, you have to finish it before you can start using it and before you can determine the return on investment that you will get from having built it, the value that it's delivered. So you'll notice that all of these statements that I've made about bridges do not apply to software. And what that means is that architecture and project management techniques that we use for building bridges are actually really bad for building software. We should not be using them because they drive the wrong kinds of behaviors. Do we want our software once we've released it to users? Do we want to make sure that it doesn't change that much? No. We want to get feedback from users and build something that's valuable to them. Most of the time when you build software for someone, it's not what they want. 19 out of 20 startups fail because they've built something that wasn't valuable to customers. So, you know, if 19 out of 20 bridges that we built turn out not to be useful, that would be really bad. Right? So software is fundamentally different from civil engineering. And that affects the way we should think about architecture. In software, we can start delivering value to our customers from early on in the project before the thing is finished in terms of having all the features that we would ultimately like. People might derive, hopefully people can derive value from early on before we finish building all the features. We can give some people a subset of the features and they will find those useful. It's cheap to prototype and experiment with software. We can spike things out. We can prototype things very, very cheaply. That's not true with bridges. Once you've delivered some software, compared to a bridge, it's relatively simple to make changes, even large scale changes with software. And then finally, with a bridge, to build a bridge, you don't need highly paid architects. You need a bunch of construction workers. That's not true for software. And unfortunately, this idea that the highly paid architects will design the software and we can hire a bunch of cheap people who aren't very smart to actually build the software is one of the biggest problems in the software industry today. Because when you actually build the software, you make loads and loads of discoveries about how the thing will actually work because software is a complex system. Software systems are complex and they have dynamic complexity. That's not true of bridges. You can model bridges and model the forces on them analytically. You cannot model the way users will interact with your software using analytic models. They are complex systems. And when you build complex systems, you discover a lot of information in the course of building them that affects your decisions or that gives you feedback about your decisions on your design. Building software is harder than thinking about the design. So, what would a system built to leverage these characteristics, which are characteristics of software, if we built a system that leveraged these, if we actually used our materials and played to the strengths of these materials that we're using, software, what might that system look like if we actually played to the strengths of software rather than applied civil engineering techniques? Well, you might find something like this. Amazon.com, this was at Velocity last year, you can see that Amazon.com, they were making changes to production every 11 point seconds. They were making, doing 1,079 deployments to production per hour. Up to 10,000 boxes in production at the receiving end of a deployment, oh, sorry, on average, and 30,000 in total. So, people sometimes ask me, you know, Jess, can continuous delivery scale? And I say, have you heard of a website called Amazon? It's quite big. They are doing this and they have to worry about Sabane's Oxley and PCI DSS and all the regulations and things that make this stuff difficult. So, why did they do this? I mean, this is hard. Continuous delivery is hard. Doing all this stuff is hard. Why do they care about it? Primarily so they can run lots and lots of experiments to try and find out how to deliver value to users. When you design, when you come up with an idea for a software product or a feature, what you have is a hypothesis. And what you want to do is try and test that hypothesis. And the most expensive way to test the hypothesis is actually to build the software. That's the most expensive and inefficient thing that you can possibly do. You know, and if you're building bridges, you kind of have to do that, right, or buildings, but which is why in 2008, the construction industry took a total nose dive because suddenly we had all these things that were unfinished that we couldn't make money from. So, the reason they do this is so they can run experiments. So, they can come up with ideas for features and run an experiment and A-B test in production to find out if it will actually change people's habits. If people, they will make more money from people if we have the buy button here rather than here, if we present books that other customers like here on the page rather than here. How can we improve the experience of customers so they spend more money? And we do that by running loads and loads of experiments doing A-B testing. That's why they do this. It's so they can make things better for their customers by testing hypotheses really, really fast and getting feedback and establishing a close connection with their customers so they can deliver value. So, that's the main value of continuous delivery. And by the way, as a side effect of continuous delivery, you also reduce the risk of release and make deployments much easier and less risky, which is nice. Otherwise, you couldn't do this, right? So, the point of continuous delivery is to reduce the cost, the time and the risk of delivering incremental changes to users. It's so we can make incremental changes to our software all the time really, really cheaply and at low risk. That's what it is. How do you know when you're doing it? The idea is that your software should always be releasable on demand. From the very first feature you build, you should be able to press a button and release it. Releasing should be boring. If we're doing continuous delivery right, releases are boring. You're not going in at the weekend. You're not staying late at night. It's a boring thing you can do during peak hours by pressing a button. No big deal. We prioritize keeping the system releasable over new functionality. If at any point we discover the performance is dropped below a certain threshold or the acceptance tests are failing or there's a security problem or the unit tests are broken, we stop and fix that problem before we do new work. So, we're not, oh, you know, we must cut. The acceptance tests aren't working but those failures aren't important. We'll just carry on delivering features. No, you shouldn't be doing that. In order to make that possible, people need to get fast feedback. If I write some code or make a configuration change and it takes me a week to find out whether that's going to degrade performance, that makes it very hard for me to actually be fast at delivering code because I'm getting feedback from what I was doing a week ago on whether that thing actually affected performance. I don't know about you guys. I can barely remember what I was doing yesterday, let alone a week ago. So, I get a bug report from a week ago. I have to try and triage it, work out what was going on. It's an absolute pain. By the way, people sitting here, there are seats up here. So, please feel free to grab seats and bother these people. And if people at the end could move down so that these people can move in, that would be super. Thank you. Cool. Thanks very much, everyone. And this fundamentally changes the way we do product delivery. Instead of having these projects where it's very waterfully, what we should be doing is you start with an inception. We've got a new idea for a product. And we're going to get together as a team and work out what's the vision we're trying to achieve? What's the measurable customer outcome we want to achieve? And then design an experiment to test if we can achieve that measurable customer outcome using the proposed software program. So, we design experiments and then we run the experiment. It's called the minimum viable products, but it's an experiment. And most of the time, that will produce some results that tell us that users don't want what we built. If we're innovating, innovation is high risk. We want potentially high returns. When you want a high return, you are going to take risks. I mean, this is investment 101, right? And so naturally, if you're taking risks, most of the time, what you want, what you build is not actually going to be valuable. So, most of the time, we'll fail with this. And what we produce is not something that people want. And we either pivot, change our business plan, or we can keep delivering increments and trying to move closer towards what customers want. But the first thing we build will not be what our customers want if we're innovating. So, we need to find ways to find that out cheaply and then to iterate or pivot based on that. So, what are the implications of this for architecture? So, fundamentally, what architecture is about is cross functional requirements, performance, availability, scalability, all the other illities. That's what architecture is for. Architecture is what makes sure that our software will meet its cross functional requirements as opposed to the design of the features which is about meeting the functional requirements. So, architecture is what allows you to do this stuff. What is important for continuous delivery? First of all, we want to make sure that any increment that we build is testable at all levels. So, I should be able to run acceptance tests on my workstation. I should be able to get, find 80% of the bugs in my software by running it on my workstation. If I have to get an integrated environment and stand up 20 different services and try and get them to all work together before I can run my acceptance tests, I have a problem because it's expensive to do that and it's painful to do that. And so, I'm going to get that feedback very, very infrequently because it's such a pain to actually do that. So, that's an architectural thing. If I can't run the software in a non-integrated environment on my workstation and get the feedback, that means I have an architectural problem. We need to take account of that in the architecture, actually making sure that I can run tests on my workstation. If I need continuous delivery, I need to be able to do incremental development on trunk. That means continuous integration means everybody is checking into trunk at least once a day. We are not working on long-lived feature branches. If you're using feature branches and you're not integrating them into trunk once a day, then you can't do continuous delivery because you can't get feedback on whether your code actually works. You know, working on my workstation is meaningless. Works in production integrated with the rest of the code, with realistic data sets and realistic loads, then maybe I'll consider thinking about saying it's done. People use feature branches because they want to optimize for working, for getting done with their feature, but that's kind of pointless. So our architecture needs to enable us to be able to do that, to be able to work incrementally on trunk, which means we need to follow things like the solid principles, make sure that our software is well encapsulated and that it's componentized so that if I make a change, it doesn't splatter all over the code base. Deployability is an architectural concern. How easy is it to deploy the software? So people talk about deployment automation. We should take the deployment process and automate it. If the deployment process is really complex and flaky and I automate it, what happens is I turn a complex error-prone manual process into a complex error-prone automated process, which is no value. We need to simplify first, make sure that it's simple to deploy and then automate that. And if the architect, and that's an architectural concern, if it's really hard to deploy the app because I need to stand out a bunch of different things and then apply a bunch of different configuration settings from a bunch of different repositories and it's really difficult to actually stand up or to deploy some new part of that, then I can't do that. We need to focus on making deployment simple so that we can automate it effectively. Also, if we're going to deploy, you know, frequently, then we need to make sure that maybe we have availability constraints, maybe we have an SLA that says our service can't be down. So if I need to take the service down in order to deploy, maybe that's a problem. Maybe I won't meet up my SLAs. So you're thinking about things like, well, if I keep a bunch of state in the cache and I need to flush the cache every time I do a deployment and I can't do that, that's a problem. This is one of the reasons that Ruby on Rails has at least some nice characteristics. All the user state is in the database, so I don't need to worry about session caches and maintaining session caches across deployments. These are things that you need to think about if you're thinking about continuous delivery and by keeping state in session caches, availability and security and all these things during deployment. So architecture for continuous delivery, there's two things. There's firstly, making sure that we can test our cross functional characteristics all the time against any change. And this is an important characteristic of complex systems is that I make a change, I cannot predict the consequences of that change. This is the problem with code inspection, with code reviews. Code reviews will help you find some problems, but we are building complex systems. And so I make a change and I can perhaps in my head predict some of the consequences of that, but it's going to be really hard for me to predict things like what's the impact on performance. Nobody sets out to degrade performance. People degrade performance by mistake because they're building complex systems and we make changes at degrade performance and we didn't realize that was going to happen. And the person reviewing our code didn't realize that was going to happen either. So we need to be able to find those things and validate them by testing. The other part of architecture is designing for resilience. When you build complex systems, failure is a normal part of the operation of complex systems. It's not something exceptional, failure is a normal part of the operation of systems. This is what people doing disaster analysis for Three Mile Island and other problems discovered. When you build complex systems, you can't prevent failure, but you need to design your system to be resilient to failure. So that's the other part, the second part of architecture. So we're going to talk about testing first. If you have questions at any point, stick your hands up. I'll try and leave time for questions at the end as well, but if you ask, if you want clarifications, stick your hand up during. If you have a long question, keep it to the end, but feel free to argue with me if you think I'm talking crap. I appreciate that. So I'm going to talk about testing. There's this quadrant diagram created by Brian Merrick where he divides testing up along two axes, whether the test support programming or whether they critique the project, whether the tests are technology facing or whether they're business facing. And so down on the bottom left here, unit tests, component tests, these are tests that test one small part of the system in isolation, a single method, a single function, and then groups larger than that. And these should all be automated. And you should be using, who here uses TDD, who writes tests before they write the code that makes the test pass? That is fabulous. That's about a third to nearly a half of you. That's the best result I've ever had. So good job. TDD is, in my experience, the only way to create maintainable suites of unit tests. But that stuff should always be automated. Up on the top right here is the stuff that you use your expensive valuable human beings for. It showcases usability testing, exploratory testing. These are things that only human beings can do. On the top left here are the functional acceptance tests, end-to-end tests that run in a production-like environment that demonstrate that the software delivers the expected business value to users. And then here are the illities. These are the things that you're validating your architecture. Will the system scale? Will it meet its SLAs in terms of availability? Are there security problems? Is it deployable? Is it testable? So these are the things that are validating your architecture. And some of those things are automatable. So performance testing, if you've done your acceptance tests right, you can reuse them as a basis for load testing. Some kinds of security tests are automatable, like penetration testing. You can do some of that in an automated way. You can use static analysis to look for buffer overrun, SQL injection, attack, and all these kinds of things. But some of it can't be automated. So some parts of security testing are very hard to automate. What do we do once we have all these tests? What we do is we create a deployment pipeline. And I want to give you an example of a deployment pipeline in the context of an interesting domain which is firmware. So who here is working in embedded software? Any embedded software people? Okay, one person. Who's building websites or web services? Whoa, okay, almost everyone. Anyone building user-installed products? Okay, a smattering of you, maybe a quarter of you building user-installed products. So what I'm going to talk about applies to user-installed products as well. But it also is very important for building web services. A lot of the same lesson supply. So HP LaserJet team was building, I mean, these are the people who built the firmware for LaserJets. And in 2008, they had a problem. And their problem was they were too slow. And the business came to them and said, you're too slow, we need you to fix it. And they went and looked at what they were doing. And they found that most of their time was not spent writing new functionality. They had, for example, a different branch for every group of printers and scanners and multifunction devices they were creating. So every time they came out with a new product line, they created a new branch inversion control. And so they were spending 25% of their time porting code between branches. I had a new feature or make a bug fix to this line of printers. I want the same thing in another line of printers. I need to merge it between branches. And they were spending 25% of their time on current product support, 15% of their time on manual testing, detailed planning, code integration. You subtract that from 100, you're left with 5%. Which they claim was time spent innovating rather than lying down with aspirin in their body because they were totally exhausted. So they had to do something. And what they did was pretty radical. They re-architected the system from scratch. And their main, the main criteria for the re-architecture was the ability to check in everything to trunk. Not to have branches, everything was going to be on mainline. And the way they did that was they created feature toggles, basically. So there was one build that was created that was deployed to all the different devices. And when you start the device, the firmware boots, it looks to see the device that it's on and it switches features on or off depending on the device. So the ability for the developers to check in on trunk was an architectural concern. And they made sure that architecture supported that so they could get rid of this 25% of their time they were spending porting code between branches. And what they were then able to do was to create a deployment pipeline. Where they had, so HP LaserJet team, 400 people distributed across the US, Brazil, and India. Anytime anyone makes a change, we run about 45 minutes that so people check in to get to a branch, to a queue. And when they check in, we automatically run 45 minutes worth of tests against that queue. If they fail, you get an email which tells you what went wrong and they actually send you a binary of the build so you can run that on your workstation and reproduce that problem on your workstation. So you automatically get an email which tells you what the problem was and gives you a build so you can press a button and reproduce that problem on your workstation. Again, architectural concern. Any change that passes those tests, they batch them up and run about 45 minutes of tests again against the batched up changes. And if those changes pass that set of tests, then it gets checked automatically into trunk. So unless you pass these two gates, you can't get into trunk. But they were getting 10 to 15 good builds a day out of this process. So every day there'd be enough check-ins. There was about 100 to 150 check-ins every day. About 100,000 lines of code were changing every day. So it's a 10 million line code base, 100,000 lines of code changing every day, 100 to 150 check-ins, and they were getting 10 to 15 good builds a day out of this process, off trunk. Any build that comes out of stage two goes into level two. Level two is automated acceptance tests, about two hours worth of automated acceptance tests that run against the software running in a virtual machine that simulates being on a printer. If the build passes that, it goes to level three. Level three is a bunch of logic boards. And so the logic boards power on, they download the latest build that comes out of level two, and then automated acceptance tests that run against that build running on physical electronic logic boards to test. And then if you pass that, every night we run regression tests. They had about 30,000 hours worth of regression tests that run in parallel on a massive grid every night, which was level four. And what they did is they were constantly moving the tests between the different stages. So if you have a test that failed down here, we push it up into stage two. Any tests that are passing all the time in stage two, we push them down. So they were constantly moving tests between these stages to make sure they got fast feedback that they found out here that there was a problem, not here. And one of their goals at the program level was to make sure they were getting 95 to 98 percent pass rate out of level two and level three. So most of 95 to 98 percent of the problems were being found up here when it was fast. So it took a lot of investment for them to build this. They had to invest in building the automated tests. They had to invest in building this deployment pipeline. They had to invest in re-architecting the system to support development on trunk and to support being able to run. So anytime something fails in level three, same thing, you get an email with the bug details and a binary that you can deploy on your workstation to reproduce the problem. So they invested a lot of effort into that and they re-architected the system to support the ability to do this. And what they got as a result of that was that they knew within 24 hours whether there was a serious problem. And what that meant was they could introduce new features after code freeze. So they had a code freeze but it was basically irrelevant. You could still change the software and make new changes to the system even after code freeze right up to days before the new release because you knew you were always working off a good foundation. That your software was deployable and they prioritized keeping the software deployable over doing new work. And what that meant was they could go to the business and the business likes one year plans. And they said, okay, we're not going to do that anymore. In return, you can change your mind at any time about the features and we will just change direction right there and then. And that was the trade-off they were able to make which got rid of the time spent doing detail planning. So they didn't do detail planning anymore because the business can change their mind at any time and that's fine. And instead what they were able to do is get rid of a lot of this waste in the process. People think lean is about cutting cost. Lean is not about cutting cost. This is expensive to invest in. What it does is it removes waste and then that leads to cost reduction. And in particular they are able to spend much more time innovating and that drove down the cost of actually building new software. So continuous integration is probably the first thing that you need to be practicing as a way to drive this out. So who's doing continuous integration? Put your hands up. Keep your hands up if you're doing continuous integration. Keep your hands up. Put your hands down unless everyone is checking into trunk at least once a day. If you're working on feature branches that live for more than a day, put your hand down. If your build is broken for more than 10 minutes at a time before being fixed, put your hand down. If every check-in does not result in build and tests being run, put your hands down. Okay. So the people in the audience with their hands up are the people who are actually doing continuous integration. Congratulations to you. Continuous integration is not running Jenkins against your feature branches. It's a practice. You can do that practice without any tools at all. James Shaw has a paper, Continuous Integration on a Dollar a Day, that talks about doing continuous integration with an old workstation, a rubber chicken and a bell. No CI server. It's a practice. It's a mindset. Keeping the software releasable over doing new work. Always working off trunk. You know, here's 400 people asking, you know, does continuous integration scale? So exhibit A, 400 people distributed across three continents doing 100,000 lines of change every day to a 10 million line code base getting 10 to 15 good builds out of that. You scale this by throwing hardware at it. That's the definition of a scalable process that you can throw hardware at it and solve it. Using feature branches is not a scalable process because the longer the feature branches live and the more people there are on the team, the pain of integrating scales in a nonlinear way as you increase those numbers so it does not scale. Whereas continuous integration does scale. Exhibit B, this is from Google. Google have a continuous integration system that runs, there's 200,000 test suites in the code base. They're running 10 million test suites per day off their CI system in Google. They run about 60 million individual test cases per day. About 4,000 continuous integration builds every day. So continuous integration scales. But what continuous integration requires is that we work in small batches, that we make all changes. Any big change, we break into a series of small changes that keeps the software releasable. And that is the difficult bit. Having developers understand how to break every change into small bits. So if I was going to hypnotize you and have you and slip something into your unconscious, the most important thing I would have you take away from this is the importance of working in small batches. Whether that is, you know, if that's code changes or organizational change or any other kind of change that you're doing, working small batches, that's the thing I want you to take away. Whiskey, beer, working small batches, budgeting, anything you do, find ways to work in small batches, that's the important thing. So I'm going to talk now about patterns for continuous delivery in addition to the kind of the practices that you should be doing. So there's a great quote from this guy. This is Jesse Robbins. He was in charge of availability at Amazon. So all of Amazon's sites, he was the head of availability for all those sites. He's now, he was co-founder of Ops Code, who makes Chef. He's also a volunteer firefighter in his spare time, so an enormous overachiever, basically. So this is what he says about web architecture. Success on the web depends on continuous deployment of reliable software to an unreliable platform that scales horizontally. So that is one of the best statements of web architecture that I've ever found, because it covers so many things in such a dense single quote. So unreliable platform. If you're using Amazon, AWS, any of these different systems, that's an unreliable platform. Do you guys have Netflix in Norway? Yeah, okay. So if you were using Netflix on Christmas Eve last year, you would have noticed that it didn't work. So there was me, Christmas Eve, I have a four-year-old daughter, and she was wanting to watch Disney films on Christmas Eve, and she couldn't because Netflix was down and we have Apple TV and not a proper TV. So that was a problem. I was quite upset about this. And it turns out this was because of a load balancing failure in one of the Amazon zones that caused this problem. Now, people were very quick to blame Amazon about this. This was not Amazon's problem. Amazon were within their SLA. Netflix was not able to meet its SLA. Actually, I'm not sure if that's true because they probably don't even have an SLA because they're sensible. But in as much as they were to have an SLA, if their SLA was higher than the SLA of the infrastructure, that's not Amazon's problem. That's Netflix's problem. So think about the SLA for the software you're building. If that has to be more reliable than the platform that it's built on, that's an architectural problem. You need to think about how to architect your way around that. And that's where the scales horizontally thing comes from. The way you make a, one of the ways you make a resilient system is by making sure that system is resilient to failures of the underlying infrastructure, which means that you should make sure you can move transparently between different availability zones. So Google and Amazon and Etsy and other high performing companies, they actually test this. And they test this by actually bringing data centers down. They do things called game days. There's a great article on game days if you look in ACM where they actually will just, they will say, okay, in six months we're going to run an exercise and they actually just turn off the fiber to a data center. And then they'll ping the person and say, have you, and they'll let them know they're going to do this. They're like, we're going to run a test on your SLA. And then they'll say, okay, we're going to bring down the data center. And they physically switch it off. And they're like, are you within your SLA? And then the site reliability engineer will actually check to see if they're in their SLA. And then they'll bring down a different data center because the SLA says you have to meet, be able to meet your SLA even if there's two data centers down. So then they brought down the European data center and made sure that they were still within their SLA's. So at Google, they test this by actually causing disasters and seeing if the system will actually still meet its SLA's. And obviously they make sure that if it doesn't, they can back out the disaster. They don't actually like burn down data centers. Right? I mean, they just switch off the fiber to make sure that you're okay and then they can switch it back on again if you're not. So that's how you test it by actually causing real problems. And if that makes your compliance people start to freak out, then you know, you've got a problem. But the problem is not with you, it's with your compliance people. So I'm going to talk a bit about the continuous deployment part of this, patterns for continuous deployment. So number one thing is that low risk releases are incremental. So I'm going to present a number of different patterns for making releases low risk and boring. Pattern number one is called expand contract. So you have a system, it's probably composed of a bunch of different bits. You know, you have some static content, you have your app, you have a bunch of services it depends on, probably has a database. Who's building a system that doesn't have a database? Okay, maybe one dude, maybe. Okay. Yeah, all right, fair enough. So what we don't want to do is deploy, you know, every time you make a bunch of changes, you have to deploy all of those, what you don't want to do is deploy changes to all the system at once. Instead what you do is you put out the new version of the static content in a different directory before the release. And then what you do is you can roll forward and roll back this without having to change the static content. The same is true of services. If you depend on an upstream service, you don't deploy the new version at the same time you deploy the app, you deploy the new version beforehand and smoke test it and make sure that it works before you deploy the new version of the app that depends on it. This is something that Amazon Web Services does, you know, if AWS releases a new version of EC2, they don't just change the API and sorry you can't use the old API anymore, that would make people very angry. Instead, you provide the API version you want to use as a query string parameter so that as a consumer of the service, you can choose when you want to upgrade. So we do that first again. We can also do that with the database. So Amazon, Facebook, Etsy, these companies are not deploying database schema changes a thousand times an hour. That's not happening. Etsy has database Thursday when they deploy their database schema changes. So what that means is in Facebook, there's one enormous production database and the developers do not get to choose what version of the database is running at the time their software is deployed. They have to defensively code against whatever version of the database happens to be there. And so what that forces you to do is make database changes incrementally using expand contract. So for example, say you're changing your address column to address line one and address line two. What do you do? Do you delete this column and add these new columns at the same time you do the deployment? No. What you do is first of all, you add the two new columns and you keep the old column there. And then what happens is, so you can do this before you deploy the app because this doesn't affect the existing application, right? We've just added some new database objects. That application is going to ignore those. They're set to default null, right? And then you deploy the new version of the application and the new version of the application will try and read from these columns. If they're null or the columns don't exist, then it will read from the old column. But it will always write to the new columns, yeah, the new columns, so and the old ones. So we write to all the columns, but reading, we try and read from the new ones and if we can't, we fall back to the old ones. And in this way, the application will lazily migrate the data. So the new version of the app will lazily migrate the data from here to here. But if we need to roll back the app, we've got the new data there. Rolling back the app doesn't require rolling back the database because the old version of the app can read the new data from the old data structure. And then much later, we can come back and delete the old columns if we want to. So that's the essence of Expand Contract. We expand, we add new stuff first, new database schema changes, new services, new static content. Then we deploy the new version of the app and then we come back and contract to remove the old stuff later. And again, that's expensive. You have to invest in that. What you're doing is you're adding complexity to your software in order to reduce the risk of deployments. It's a trade-off. It's not a silver bullet, it's a trade-off. And there's a fabulous book called Release It by Mike Nygaard where this pattern and a bunch of other patterns for creating resilient systems. It's a great book. So how about making changes to the application in an incremental way? How do we deploy the application in an incremental way? Well, we use a pattern called blue-green deployments. So what this means is, real story, I was working on a system in 2005 where we could only get one production set of hardware. We were not going to get another set of hardware because it was too expensive. And so we didn't have a staging environment. That was scary. So what we did is we created what we called a green environment, which was a new instance of Apache and WebLogic and a new green database, which sat on the same boxes, but they were installed on a different place in the file system and they listened on different ports. So what we did is we deployed the new version to our green environment. And then that's deployment. Release is just changing the router to point to the green environment ports. Rollback is changing the router back to point at the blue environment ports. So in this way, we were able to do deployment and the deployment took about an hour, but we were able to do roll forward and roll back in less than a second, super safely, by using this technique of separating out deployment and having two environments. And then when we deployed version 1.3, that would go into the blue environment and 1.4 would go into the green environment. So we did this by having multiple things on the same box. You can also do this in several other different ways. By virtualization, if you have two data centers, a main data center and a backup data center, your main data center is blue, your backup data center is green. So you've got your current version running in your main data center, you deploy the next version to your backup data center. And then release is switching DNS to point to the new data center or your load balancer. And then your backup data center becomes your main data center. Your main data center is now your backup data center and you've tested your business continuity process for free at the same time. And this brings us to a really important principle. Release and deployment are two different things. Deployment is taking a build and putting it into an environment. Release is making a feature available to users. And we normally achieve release through deployment. But they're actually two completely different things. Facebook's release manager, Chuck Rossi, has a video where he says all major features in Facebook are already in production. For the next, all features they're going to release in the next six months are already in production. You just can't see them yet. So they're deploying this stuff constantly into production, but it's not user-visible. They have a software called Gatekeeper that controls which users can see which features. And they can change that dynamically, which lets them roll a small feature experiment out, test it with a small set of their users, get feedback, iterate, and then when they're happy, then they roll it out to a larger and larger group of users. But we should be deploying all the time. Release is when we turn that knob up to 100. And that should be a boring, low-risk activity that the business can perform without even talking to IT. And it's low-risk because that software has already been in production for months. And we know that it works. And we've done all the testing on it in production before we actually release it to all the users. Another technique for reducing the risk of release is called canary releasing. And again, this is something that many large, high-performing organizations do. Facebook in particular, they have a cluster called A1, which is a few thousand boxes. New releases go out to that first. Facebook employees, when they go to facebook.com, they don't actually go to facebook.com, which is this big cluster. They're automatically re-rooted to this cluster, which is called inyour.facebook.com. So Facebook employees go to this. And so basically what they do, I mean, people call this dog-fooding, which is weird because who wants to eat dog food? So they call it drinking your own champagne. So they drink their own champagne. They test it here before it goes out here. And then if the build is good here, then it goes here, which is maybe a data center. And a very small set of users get sent to that. And then if it's good, then they send it out to the rest of the world. And so it's like a canary in a coal mine. In the old days, people used to take a canary into the coal mine. And the canary, if there was dangerous gas in the coal mine, the canary would die. And then everyone would run really fast out of the coal mine. And these days, they have chemical detectors, which means that no birds are killed, which is good. So no animals were harmed in the course of this presentation. So this is the canary. We make sure the software works here. And we're monitoring business metrics. We're not monitoring like CPU or disk space or IO. We're measuring how many users are actually sending messages through Facebook chat. How many people are actually posting onto their timeline measuring actual user activities? So if you're doing green field development, you can design with these patterns upfront. You can design for testability. You can architect for deployability. What happens, who here has an existing system that is definitely not designed for continuous delivery, but they want to get from A to B? Okay, many of you. So that's why I just want to spend 10 minutes on re-architecting. So I want to talk briefly about Amazon. Amazon in 2001 had a big ball of mud in production, a big monolithic app called Obidos. And in 2001, Jeff Bezos, the CEO of Amazon, sent a memo to his entire technical staff saying, we are going to re-architect around a service-oriented architecture. And this is before anyone was even talking about service-oriented architecture. And we're going to have a team for each service. So they invented this term, two pizza teams. So no team can be bigger than you can feed with two pizzas. Now, this is Seattle, so the pizzas are quite big. But still, that's about 10 to 12 people per team. So they re-architected the system so that every, they broke it into services. The service was maintained through its entire life by a team of people who was responsible for that particular service. And the rule is, you're not allowed to talk to another service's database. You have to talk to other services through their service layer, through their API. And they hired an US Army Ranger. And the job of the US Army Ranger was to go around the different teams, and if he saw someone talking to another service through the database, he would shout at them a lot. And if they carried on doing that, the rumor has it that you would get fired. So I'm not recommending people get fired, but rumor has it that that, you know, is a very serious offense not to actually go through the service layer when you're talking to other services. And so over the course of a few years, they did this. And so this is Werner Vogels, the CTO of Amazon. I'm going to put words into his mouth. I love doing this. So this is in 2006, 2007, I think. If you hear Amazon.com, the app calls more than 100 services to render the front page. Now, that causes, I mean, that is non-trivial, and it causes other problems, like trying to, if something goes wrong with that, how do you debug that when you've got 100 services collaborating to produce that problem? Well, you need to do really good monitoring of the way calls go through that graph of services. You need to do lots of caching to make that actually performant. But again, it scales. It's a horizontally scalable architecture that doesn't have a single point of failure. The reason they had to do this was because they kept scaling their single big database bigger and bigger until finally the UI couldn't handle it. So they had to re-architect around to create a reliable platform that based, sorry, a reliable service based on an unreliable platform that scaled horizontally. And that's the upshot of this. But, you know, how do we do this? I mean, and they did this in a somewhat incremental way. So there's still parts of Albedos that are live today, I'm told. I don't know if that's true. It's really hard to find out what's really going on at Amazon because they don't like to talk about it. So how do we do this incrementally? Well, there's a pattern called strangler applications. So if you've seen Tomb Raider, the film, okay, many of you. So lessons from Tomb Raider. If you've seen Tomb Raider, you would have seen this temple in Cambodia where there's this tree that's grown on top of the temple. And what's grown on top of the tree is there's a kind of a fig called a strangler fig. And the seeds are on the tree and the fig grows down and the fig grows up around the tree. And eventually the fig will strangle the tree. And all that's left is the fig vine that's grown up around it. So this is a metaphor for what you should be doing with architecture. I'm very happy that I was able to write our architectural lessons from Tomb Raider. Super. So customers, big ball of mud, big fat database. What we do is when new requirements come in, what we do is we find ways to implement those new requirements using a new module. So maybe this module serves just one particular web page as part of your whole site. And the stuff that needs to do that is in its own database. But everything else gets delegated through to the original app. So we delegate most of the stuff here and just this new page or even a new part of a page. So for example, Amazon, the recommendations engine is a service. And so the part of the page that the recommendations engine is done by this service. And maybe you have a new requirement for your website, we need recommendations. So you don't implement that in the existing system. You implement that as a new service. And maybe what we do is we have an include statement that pulls from this particular service to render that part of the page. But all new functionality, all new requirements you implement using new modules, new services that you build. And gradually over time, what you end up with is a bunch of new services. And eventually, finally, this thing gets strangled. We strangle the original application. So this is how we incrementally move from big monolithic ball of mud that's not suitable for continuous delivery to a new architecture that's service oriented that is designed for continuous delivery. And again, this happens incrementally over time. The problem is, you know, our organizations don't set up to do this. There's an annual budgeting cycle. We start a huge new project. Who's experienced the enormous rebuilds? You know, the big bang re-system re-architecture. We're going to re-architect everything from scratch. So hands up if that's actually worked for you. Yeah, I thought so. So yeah, someone who has worked for it, it does sometimes work. When that happens, I'm like, wow, you guys are really amazing. Or you have great senior management politicians to make it look like it worked even though it didn't. But starting a new project that's going to take a long time to make sure that it's definitely going to work this time. So it does sometimes work. It's very, very, very risky. And so what I saw, I mean, people object to this because they say, this is going to take too long. But my reply to this is it's better for the journey to take a long time and to arrive at the destination than for the journey to take a shorter amount of time but you never actually get there. So I want to talk about how we do this in the small. So this is big scale architectural change. How do we make smaller scale architectural changes, particularly on trunk? How do you change a particular module or service on trunk and make architectural changes? Well, this is a practical branch by abstraction. And what you do is this. We employ, there's a joke in computer science. There's no problem you can't solve with an additional layer of abstraction. So, or in direction. So what we do is we have a library we want to change or a component we want to change. What we do is we put an abstraction layer above it and make everything go through that. And then we incrementally move methods or API calls from one implementation to another one. So I used to work on a product called Go for release management. We changed the back end from Ibarthus to Hibernate and we changed the front end from Velocity to JRubion Rails and we carried on releasing the software during those two rearchitectures. So I just want to make sure I don't run out of time. I'm going to talk briefly about the front end. The abstraction layer was the servler API. We basically said if the URI had slash new, it would go through to JRubion Rails. Otherwise it went through to the old Velocity stack. Any new requirement that came in that added a new page or changed an existing page, we copied that page over to JRubion Rails. And then we did just that one page. And then what happened was when that page was ready, we changed the rest of the app to point to the slash new version here. But the rest of the app was still using Velocity. So there's no way as a user for you to tell if you are going through the Velocity stack or the JRubion Rails stack unless you looked at the URI. Same CSS, same HTML, but which stack it was being rendered through, you had no way of telling unless you looked at the URI to see which route it was going through. And so there's still some pages in Velocity right now. It's just that, so you can keep the old stuff for as long as you want. If you're a bit anal, you know, a bit OCD, then maybe you want to get rid of the old implementation eventually. And, you know, if you're really OCD, you might want to get rid of the interface as well. But it turns out you don't really need to do that. And again, as a trade-off, you're adding complexity in order to manage the ability to, this architectural cross-functional requirement of deployability. The final point I want to make is that continuous delivery, it's not about changing your architecture to do CD. Doing CD exerts a force on your architecture. Architecture is easy to change when you're starting work. When you're a long way into your work, it's hard to change. By continuously deploying throughout, you're constantly validating your architecture. By constantly doing performance testing, by constantly deploying, by constantly doing acceptance testing, what you're doing is creating a force on your architecture to make it testable, to make it deployable, and to make sure that we're actually meeting our cross-functional requirements so that we can find if there's a problem with our architecture early when it's cheap to fix, not late when it's expensive and painful to fix. So, Dan North has a pattern called Dancing Skeleton, which is based on Alistair Cobain's walking skeleton. And the idea is with a dancing skeleton, you know, you get a few features in production and test them with a realistic load and data set. And you actually have an API and make sure you can really exercise that. And that's why it's dancing, because you're really using it. And again, as I talked about, you're testing things like availability, security, disaster recovery, business continuity planning by having game days, actually creating real disasters to make sure that your software meets its cross-functional requirements. So the final take-aways, testability and deployability are architectural constraints. You need to architect with these things in mind. We want to be able to develop functionality on trunks so we can get fast feedback on whether what we're doing really actually works and whether the work we're doing actually produces meets our SLAs with the architecture we're using. You need to design for continuous deployment and incremental deployment. You can do rearchitecture and redesign in an incremental way as well. And the way to make sure that you're actually going to, your architecture is actually going to work and meet your cross-functional constraints is by doing continuous delivery, by actually practicing that thing all the time. That's how you verify it's really going to work. So we have four minutes for questions. What questions do you have? Yes? At the beginning you mentioned that you would be able to run your integration test on your development machine. How do you manage that when you have to be like Android on? Right. So any serious system, there's going to be like a bunch of different things and there's going to be the platform as a service, infrastructure it runs on, these other things. What you basically need to do is create mocks of those other services. So you should be able to mock out that service and you don't necessarily need to do it for every service because that becomes hard to maintain. But you find the big boundaries. So maybe you can stand up two of these services in your development machine at once and you just stand those things up. But then there's some other, so you group the services up and then stand up a group of them and then look at the other things and you create an abstraction layer between those things which is just good API design in any case and you create mocks of those things. And what you can do, there's a technique that you can use where you basically create a proxy and you stand up the whole system and record the messages and then you just store those and then you can stand up part of the system and you just use the recording of the messages, the can responses in order to be able to mock out the other systems. I mean, that's how you solve it if you have an existing system that you need to redesign. I mean, the other way you do it is just by making sure that you design the system so you can find 80% of the bugs without even having to talk to the other systems at all. So you just abstract that away. So if the other systems aren't there, my service will still work but with a degraded quality of service. So using patterns, Mike Nygaard has a pattern called circuit breaker which basically means if I can't contact the remote service, I will break the circuit for a certain period of time and my service will still carry on working but without that extra stuff that the external service brings you. So that's just good design is that I should be able to operate my service even if the other ones aren't there and so I should be able to do most of my testing with that being the case. And then for the small part of your functionality where that's not true, then we use mocks. Does that answer your question? Yes. So that's an excellent question. There's always some sharp person who spots that in the database part of BLEG green deployments. The question is, you know, if I do this, if I roll forward to version 1.2 and I need to roll back, there's new stuff in this database, right? What do I do about that? Well, the first thing you do is, I mean, maybe you don't care about that. You need to start with what's my recovery point objective? What's my recovery time objective? What are my actual SLAs around this? Maybe it's okay for me to lose the data. If that's true, that's super because you can just, I mean, literally what you do is before you deploy, you back up 1.1 database and you restore to the green 1.2 and you switch over. Then anything that happens between that and roll back, you just lose. That may be fine. If it's not, you can do things like putting the app into read-only modes for a certain period of time. If that's not acceptable, then you're going to have to use expand contract. So expand contract the whole point.
 Okay, so my name is Jess Humble. I work for ThoughtWorks. We do consulting and delivery  for software and we have tools to help you with that. Mingle for agile project management,  Go for agile release management and Twist for agile testing. I co-authored this book  called Delivery and I'm here today to talk about architecting and re-architecting and  continuous delivery and the relationship between these things. So it's not going to be a detailed  kind of lots of code on the whiteboard talk. So if you want something very detailed and  codey, now would be a good time to go and see one of the other things that is more detailed  and codey. I'm not at all offended if people decide to walk out and go and see something  else instead. If that would be the best use of your time, please go ahead and do that.  Don't feel bad at all. Also there's voting at the end. So at the end, please choose a  green card, a yellow card and a red card depending on how you feel about the talk. Remember that  in China red is lucky but in Norway not so much. I've been having a great time in Norway.  It's my first time here. I went to Bergen on Monday and I got sunburn which was unexpected  but very nice. So architecture. Architecture is a term obviously that comes from building  buildings and if you're actually building, if you're doing construction, doing civil  engineering projects, one of the important things as an architect you have to care about  is what's the nature of the materials you're using. Iron, steel, wood, plastic, all these  things have different characteristics and that affects the way you think about what  you're building. And so obviously if you're building, if you're architecting things that  have to stand up for a long time like a bridge, you have to think carefully about the materials  that you're using. Now project management and architecture are focused around building  things like this or buildings which have a certain set of characteristics. So once you  build a bridge or a building, normally it doesn't fall down. That's bad normally. That  means you've done a bad job. Once you build it, it doesn't change a lot. So we don't build  a bridge and go, well that isn't really right and that isn't what we wanted. We're going  to totally change the way that works now because our users didn't like that. Running  a bridge requires different skills from building the bridge, completely different set of practices.  Once we have built the, once we've designed the architecture for the bridge and thought  about how we want it to behave, actually then building it will not give us lots of new information  that makes us think again about the architecture. That's not supposed to happen when you're  building bridges and buildings. And then most importantly when you're building a bridge,  you have to finish it before you can start using it and before you can determine the  return on investment that you will get from having built it, the value that it's delivered.  So you'll notice that all of these statements that I've made about bridges do not apply  to software. And what that means is that architecture and project management techniques  that we use for building bridges are actually really bad for building software. We should  not be using them because they drive the wrong kinds of behaviors. Do we want our software  once we've released it to users? Do we want to make sure that it doesn't change that much?  No. We want to get feedback from users and build something that's valuable to them. Most  of the time when you build software for someone, it's not what they want. 19 out of 20 startups  fail because they've built something that wasn't valuable to customers. So, you know,  if 19 out of 20 bridges that we built turn out not to be useful, that would be really  bad. Right? So software is fundamentally different from civil engineering. And that affects the  way we should think about architecture. In software, we can start delivering value to  our customers from early on in the project before the thing is finished in terms of having  all the features that we would ultimately like. People might derive, hopefully people  can derive value from early on before we finish building all the features. We can give some  people a subset of the features and they will find those useful. It's cheap to prototype  and experiment with software. We can spike things out. We can prototype things very,  very cheaply. That's not true with bridges. Once you've delivered some software, compared  to a bridge, it's relatively simple to make changes, even large scale changes with software.  And then finally, with a bridge, to build a bridge, you don't need highly paid architects.  You need a bunch of construction workers. That's not true for software. And unfortunately,  this idea that the highly paid architects will design the software and we can hire a  bunch of cheap people who aren't very smart to actually build the software is one of the  biggest problems in the software industry today. Because when you actually build the  software, you make loads and loads of discoveries about how the thing will actually work because  software is a complex system. Software systems are complex and they have dynamic complexity.  That's not true of bridges. You can model bridges and model the forces on them analytically.  You cannot model the way users will interact with your software using analytic models.  They are complex systems. And when you build complex systems, you discover a lot of information  in the course of building them that affects your decisions or that gives you feedback  about your decisions on your design. Building software is harder than thinking about the  design. So, what would a system built to leverage these characteristics, which are characteristics  of software, if we built a system that leveraged these, if we actually used our materials and  played to the strengths of these materials that we're using, software, what might that  system look like if we actually played to the strengths of software rather than applied  civil engineering techniques? Well, you might find something like this. Amazon.com, this  was at Velocity last year, you can see that Amazon.com, they were making changes to production  every 11 point seconds. They were making, doing 1,079 deployments to production per  hour. Up to 10,000 boxes in production at the receiving end of a deployment, oh, sorry,  on average, and 30,000 in total. So, people sometimes ask me, you know, Jess, can continuous  delivery scale? And I say, have you heard of a website called Amazon? It's quite big.  They are doing this and they have to worry about Sabane's Oxley and PCI DSS and all the  regulations and things that make this stuff difficult. So, why did they do this? I mean,  this is hard. Continuous delivery is hard. Doing all this stuff is hard. Why do they  care about it? Primarily so they can run lots and lots of experiments to try and find out  how to deliver value to users. When you design, when you come up with an idea for a software  product or a feature, what you have is a hypothesis. And what you want to do is try and test that  hypothesis. And the most expensive way to test the hypothesis is actually to build  the software. That's the most expensive and inefficient thing that you can possibly do.  You know, and if you're building bridges, you kind of have to do that, right, or buildings,  but which is why in 2008, the construction industry took a total nose dive because suddenly  we had all these things that were unfinished that we couldn't make money from. So, the  reason they do this is so they can run experiments. So, they can come up with ideas for features  and run an experiment and A-B test in production to find out if it will actually change people's  habits. If people, they will make more money from people if we have the buy button here  rather than here, if we present books that other customers like here on the page rather  than here. How can we improve the experience of customers so they spend more money? And  we do that by running loads and loads of experiments doing A-B testing. That's why they do this.  It's so they can make things better for their customers by testing hypotheses really, really  fast and getting feedback and establishing a close connection with their customers so  they can deliver value. So, that's the main value of continuous delivery. And by the way,  as a side effect of continuous delivery, you also reduce the risk of release and make deployments  much easier and less risky, which is nice. Otherwise, you couldn't do this, right? So,  the point of continuous delivery is to reduce the cost, the time and the risk of delivering  incremental changes to users. It's so we can make incremental changes to our software all  the time really, really cheaply and at low risk. That's what it is. How do you know when  you're doing it? The idea is that your software should always be releasable on demand. From  the very first feature you build, you should be able to press a button and release it.  Releasing should be boring. If we're doing continuous delivery right, releases are boring.  You're not going in at the weekend. You're not staying late at night. It's a boring thing  you can do during peak hours by pressing a button. No big deal. We prioritize keeping  the system releasable over new functionality. If at any point we discover the performance  is dropped below a certain threshold or the acceptance tests are failing or there's a  security problem or the unit tests are broken, we stop and fix that problem before we do  new work. So, we're not, oh, you know, we must cut. The acceptance tests aren't working  but those failures aren't important. We'll just carry on delivering features. No, you  shouldn't be doing that. In order to make that possible, people need to get fast feedback.  If I write some code or make a configuration change and it takes me a week to find out  whether that's going to degrade performance, that makes it very hard for me to actually  be fast at delivering code because I'm getting feedback from what I was doing a week ago  on whether that thing actually affected performance. I don't know about you guys. I can barely  remember what I was doing yesterday, let alone a week ago. So, I get a bug report from a  week ago. I have to try and triage it, work out what was going on. It's an absolute pain.  By the way, people sitting here, there are seats up here. So, please feel free to grab  seats and bother these people. And if people at the end could move down so that these people  can move in, that would be super. Thank you. Cool. Thanks very much, everyone.  And this fundamentally changes the way we do product delivery. Instead of having these  projects where it's very waterfully, what we should be doing is you start with an inception.  We've got a new idea for a product. And we're going to get together as a team and work out  what's the vision we're trying to achieve? What's the measurable customer outcome we  want to achieve? And then design an experiment to test if we can achieve that measurable customer  outcome using the proposed software program. So, we design experiments and then we run  the experiment. It's called the minimum viable products, but it's an experiment. And most  of the time, that will produce some results that tell us that users don't want what we  built. If we're innovating, innovation is high risk. We want potentially high returns.  When you want a high return, you are going to take risks. I mean, this is investment  101, right? And so naturally, if you're taking risks, most of the time, what you want, what  you build is not actually going to be valuable. So, most of the time, we'll fail with this.  And what we produce is not something that people want. And we either pivot, change our  business plan, or we can keep delivering increments and trying to move closer towards what customers  want. But the first thing we build will not be what our customers want if we're innovating.  So, we need to find ways to find that out cheaply and then to iterate or pivot based  on that. So, what are the implications of this for architecture?  So, fundamentally, what architecture is about is cross functional requirements, performance,  availability, scalability, all the other illities. That's what architecture is for. Architecture  is what makes sure that our software will meet its cross functional requirements as  opposed to the design of the features which is about meeting the functional requirements.  So, architecture is what allows you to do this stuff.  What is important for continuous delivery? First of all, we want to make sure that any  increment that we build is testable at all levels. So, I should be able to run acceptance  tests on my workstation. I should be able to get, find 80% of the bugs in my software  by running it on my workstation. If I have to get an integrated environment and stand  up 20 different services and try and get them to all work together before I can run my acceptance  tests, I have a problem because it's expensive to do that and it's painful to do that. And  so, I'm going to get that feedback very, very infrequently because it's such a pain to actually  do that. So, that's an architectural thing. If I can't run the software in a non-integrated  environment on my workstation and get the feedback, that means I have an architectural  problem. We need to take account of that in the architecture, actually making sure that  I can run tests on my workstation. If I need continuous delivery, I need to be able to  do incremental development on trunk. That means continuous integration means everybody  is checking into trunk at least once a day. We are not working on long-lived feature branches.  If you're using feature branches and you're not integrating them into trunk once a day,  then you can't do continuous delivery because you can't get feedback on whether your code  actually works. You know, working on my workstation is meaningless. Works in production integrated  with the rest of the code, with realistic data sets and realistic loads, then maybe I'll  consider thinking about saying it's done. People use feature branches because they want to  optimize for working, for getting done with their feature, but that's kind of pointless.  So our architecture needs to enable us to be able to do that, to be able to work incrementally  on trunk, which means we need to follow things like the solid principles, make sure that our  software is well encapsulated and that it's componentized so that if I make a change,  it doesn't splatter all over the code base. Deployability is an architectural concern.  How easy is it to deploy the software? So people talk about deployment automation. We  should take the deployment process and automate it. If the deployment process is really complex  and flaky and I automate it, what happens is I turn a complex error-prone manual process  into a complex error-prone automated process, which is no value. We need to simplify first,  make sure that it's simple to deploy and then automate that. And if the architect, and that's  an architectural concern, if it's really hard to deploy the app because I need to stand  out a bunch of different things and then apply a bunch of different configuration settings  from a bunch of different repositories and it's really difficult to actually stand up  or to deploy some new part of that, then I can't do that. We need to focus on making  deployment simple so that we can automate it effectively. Also, if we're going to deploy,  you know, frequently, then we need to make sure that maybe we have availability constraints,  maybe we have an SLA that says our service can't be down. So if I need to take the service  down in order to deploy, maybe that's a problem. Maybe I won't meet up my SLAs. So you're thinking  about things like, well, if I keep a bunch of state in the cache and I need to flush  the cache every time I do a deployment and I can't do that, that's a problem. This is  one of the reasons that Ruby on Rails has at least some nice characteristics. All the  user state is in the database, so I don't need to worry about session caches and maintaining  session caches across deployments. These are things that you need to think about if you're  thinking about continuous delivery and by keeping state in session caches, availability  and security and all these things during deployment.  So architecture for continuous delivery, there's two things. There's firstly, making  sure that we can test our cross functional characteristics all the time against any change.  And this is an important characteristic of complex systems is that I make a change, I  cannot predict the consequences of that change. This is the problem with code inspection,  with code reviews. Code reviews will help you find some problems, but we are building  complex systems. And so I make a change and I can perhaps in my head predict some of the  consequences of that, but it's going to be really hard for me to predict things like  what's the impact on performance. Nobody sets out to degrade performance. People degrade  performance by mistake because they're building complex systems and we make changes at degrade  performance and we didn't realize that was going to happen. And the person reviewing  our code didn't realize that was going to happen either. So we need to be able to find  those things and validate them by testing. The other part of architecture is designing  for resilience. When you build complex systems, failure is a normal part of the operation  of complex systems. It's not something exceptional, failure is a normal part of the operation  of systems. This is what people doing disaster analysis for Three Mile Island and other problems  discovered. When you build complex systems, you can't prevent failure, but you need to  design your system to be resilient to failure. So that's the other part, the second part  of architecture. So we're going to talk about testing first. If you have questions at any  point, stick your hands up. I'll try and leave time for questions at the end as well, but  if you ask, if you want clarifications, stick your hand up during. If you have a long question,  keep it to the end, but feel free to argue with me if you think I'm talking crap. I appreciate  that. So I'm going to talk about testing. There's this quadrant diagram created by Brian  Merrick where he divides testing up along two axes, whether the test support programming  or whether they critique the project, whether the tests are technology facing or whether  they're business facing. And so down on the bottom left here, unit tests, component tests,  these are tests that test one small part of the system in isolation, a single method,  a single function, and then groups larger than that. And these should all be automated.  And you should be using, who here uses TDD, who writes tests before they write the code  that makes the test pass? That is fabulous. That's about a third to nearly a half of you.  That's the best result I've ever had. So good job. TDD is, in my experience, the only way  to create maintainable suites of unit tests. But that stuff should always be automated.  Up on the top right here is the stuff that you use your expensive valuable human beings  for. It showcases usability testing, exploratory testing. These are things that only human  beings can do. On the top left here are the functional acceptance tests, end-to-end tests  that run in a production-like environment that demonstrate that the software delivers  the expected business value to users. And then here are the illities. These are the  things that you're validating your architecture. Will the system scale? Will it meet its SLAs  in terms of availability? Are there security problems? Is it deployable? Is it testable?  So these are the things that are validating your architecture. And some of those things  are automatable. So performance testing, if you've done your acceptance tests right, you  can reuse them as a basis for load testing. Some kinds of security tests are automatable,  like penetration testing. You can do some of that in an automated way. You can use static  analysis to look for buffer overrun, SQL injection, attack, and all these kinds of things. But  some of it can't be automated. So some parts of security testing are very hard to automate.  What do we do once we have all these tests? What we do is we create a deployment pipeline.  And I want to give you an example of a deployment pipeline in the context of an interesting  domain which is firmware. So who here is working in embedded software? Any embedded software  people? Okay, one person. Who's building websites or web services? Whoa, okay, almost everyone.  Anyone building user-installed products? Okay, a smattering of you, maybe a quarter of you  building user-installed products. So what I'm going to talk about applies to user-installed  products as well. But it also is very important for building web services. A lot of the same  lesson supply. So HP LaserJet team was building, I mean, these are the people who built the  firmware for LaserJets. And in 2008, they had a problem. And their problem was they  were too slow. And the business came to them and said, you're too slow, we need you to  fix it. And they went and looked at what they were doing. And they found that most of their  time was not spent writing new functionality. They had, for example, a different branch  for every group of printers and scanners and multifunction devices they were creating.  So every time they came out with a new product line, they created a new branch inversion  control. And so they were spending 25% of their time porting code between branches. I had  a new feature or make a bug fix to this line of printers. I want the same thing in another  line of printers. I need to merge it between branches. And they were spending 25% of their  time on current product support, 15% of their time on manual testing, detailed planning,  code integration. You subtract that from 100, you're left with 5%. Which they claim was  time spent innovating rather than lying down with aspirin in their body because they were  totally exhausted. So they had to do something. And what they did was pretty radical. They  re-architected the system from scratch. And their main, the main criteria for the re-architecture  was the ability to check in everything to trunk. Not to have branches, everything was  going to be on mainline. And the way they did that was they created feature toggles,  basically. So there was one build that was created that was deployed to all the different  devices. And when you start the device, the firmware boots, it looks to see the device  that it's on and it switches features on or off depending on the device. So the ability  for the developers to check in on trunk was an architectural concern. And they made sure  that architecture supported that so they could get rid of this 25% of their time they were  spending porting code between branches. And what they were then able to do was to create  a deployment pipeline. Where they had, so HP LaserJet team, 400 people distributed across  the US, Brazil, and India. Anytime anyone makes a change, we run about 45 minutes that  so people check in to get to a branch, to a queue. And when they check in, we automatically  run 45 minutes worth of tests against that queue. If they fail, you get an email which  tells you what went wrong and they actually send you a binary of the build so you can  run that on your workstation and reproduce that problem on your workstation. So you automatically  get an email which tells you what the problem was and gives you a build so you can press  a button and reproduce that problem on your workstation. Again, architectural concern.  Any change that passes those tests, they batch them up and run about 45 minutes of tests again  against the batched up changes. And if those changes pass that set of tests, then it gets  checked automatically into trunk. So unless you pass these two gates, you can't get into  trunk. But they were getting 10 to 15 good builds a day out of this process. So every  day there'd be enough check-ins. There was about 100 to 150 check-ins every day. About  100,000 lines of code were changing every day. So it's a 10 million line code base, 100,000  lines of code changing every day, 100 to 150 check-ins, and they were getting 10 to 15  good builds a day out of this process, off trunk. Any build that comes out of stage two  goes into level two. Level two is automated acceptance tests, about two hours worth of  automated acceptance tests that run against the software running in a virtual machine  that simulates being on a printer. If the build passes that, it goes to level three. Level  three is a bunch of logic boards. And so the logic boards power on, they download the latest  build that comes out of level two, and then automated acceptance tests that run against  that build running on physical electronic logic boards to test. And then if you pass  that, every night we run regression tests. They had about 30,000 hours worth of regression  tests that run in parallel on a massive grid every night, which was level four. And what  they did is they were constantly moving the tests between the different stages. So if  you have a test that failed down here, we push it up into stage two. Any tests that  are passing all the time in stage two, we push them down. So they were constantly moving  tests between these stages to make sure they got fast feedback that they found out here  that there was a problem, not here. And one of their goals at the program level was to  make sure they were getting 95 to 98 percent pass rate out of level two and level three.  So most of 95 to 98 percent of the problems were being found up here when it was fast.  So it took a lot of investment for them to build this. They had to invest in building  the automated tests. They had to invest in building this deployment pipeline. They had  to invest in re-architecting the system to support development on trunk and to support  being able to run. So anytime something fails in level three, same thing, you get an email  with the bug details and a binary that you can deploy on your workstation to reproduce  the problem. So they invested a lot of effort into that and they re-architected the system  to support the ability to do this. And what they got as a result of that was that they  knew within 24 hours whether there was a serious problem. And what that meant was they could  introduce new features after code freeze. So they had a code freeze but it was basically  irrelevant. You could still change the software and make new changes to the system even after  code freeze right up to days before the new release because you knew you were always working  off a good foundation. That your software was deployable and they prioritized keeping  the software deployable over doing new work. And what that meant was they could go to the  business and the business likes one year plans. And they said, okay, we're not going to do  that anymore. In return, you can change your mind at any time about the features and we  will just change direction right there and then. And that was the trade-off they were  able to make which got rid of the time spent doing detail planning. So they didn't do detail  planning anymore because the business can change their mind at any time and that's fine. And  instead what they were able to do is get rid of a lot of this waste in the process. People  think lean is about cutting cost. Lean is not about cutting cost. This is expensive  to invest in. What it does is it removes waste and then that leads to cost reduction. And  in particular they are able to spend much more time innovating and that drove down the  cost of actually building new software. So continuous integration is probably the first  thing that you need to be practicing as a way to drive this out. So who's doing continuous  integration? Put your hands up. Keep your hands up if you're doing continuous integration.  Keep your hands up. Put your hands down unless everyone is checking into trunk at least once  a day. If you're working on feature branches that live for more than a day, put your hand  down. If your build is broken for more than 10 minutes at a time before being fixed, put  your hand down. If every check-in does not result in build and tests being run, put your  hands down. Okay. So the people in the audience with their hands up are the people who are  actually doing continuous integration. Congratulations to you. Continuous integration is not running  Jenkins against your feature branches. It's a practice. You can do that practice without  any tools at all. James Shaw has a paper, Continuous Integration on a Dollar a Day,  that talks about doing continuous integration with an old workstation, a rubber chicken  and a bell. No CI server. It's a practice. It's a mindset. Keeping the software releasable  over doing new work. Always working off trunk. You know, here's 400 people asking, you know,  does continuous integration scale? So exhibit A, 400 people distributed across three continents  doing 100,000 lines of change every day to a 10 million line code base getting 10 to  15 good builds out of that. You scale this by throwing hardware at it. That's the definition  of a scalable process that you can throw hardware at it and solve it. Using feature branches  is not a scalable process because the longer the feature branches live and the more people  there are on the team, the pain of integrating scales in a nonlinear way as you increase  those numbers so it does not scale. Whereas continuous integration does scale. Exhibit  B, this is from Google. Google have a continuous integration system that runs, there's 200,000  test suites in the code base. They're running 10 million test suites per day off their CI  system in Google. They run about 60 million individual test cases per day. About 4,000  continuous integration builds every day. So continuous integration scales. But what continuous  integration requires is that we work in small batches, that we make all changes. Any big  change, we break into a series of small changes that keeps the software releasable. And that  is the difficult bit. Having developers understand how to break every change into small bits.  So if I was going to hypnotize you and have you and slip something into your unconscious,  the most important thing I would have you take away from this is the importance of working  in small batches. Whether that is, you know, if that's code changes or organizational change  or any other kind of change that you're doing, working small batches, that's the thing I  want you to take away. Whiskey, beer, working small batches, budgeting, anything you do,  find ways to work in small batches, that's the important thing.  So I'm going to talk now about patterns for continuous delivery in addition to the kind  of the practices that you should be doing. So there's a great quote from this guy. This  is Jesse Robbins. He was in charge of availability at Amazon. So all of Amazon's sites, he was  the head of availability for all those sites. He's now, he was co-founder of Ops Code, who  makes Chef. He's also a volunteer firefighter in his spare time, so an enormous overachiever,  basically. So this is what he says about web architecture.  Success on the web depends on continuous deployment of reliable software to an unreliable platform  that scales horizontally. So that is one of the best statements of web architecture that  I've ever found, because it covers so many things in such a dense single quote.  So unreliable platform. If you're using Amazon, AWS, any of these different systems, that's  an unreliable platform. Do you guys have Netflix in Norway? Yeah, okay. So if you were using  Netflix on Christmas Eve last year, you would have noticed that it didn't work. So there  was me, Christmas Eve, I have a four-year-old daughter, and she was wanting to watch Disney  films on Christmas Eve, and she couldn't because Netflix was down and we have Apple TV and  not a proper TV. So that was a problem. I was quite upset about this. And it turns out  this was because of a load balancing failure in one of the Amazon zones that caused this  problem. Now, people were very quick to blame Amazon about this. This was not Amazon's  problem. Amazon were within their SLA. Netflix was not able to meet its SLA. Actually, I'm  not sure if that's true because they probably don't even have an SLA because they're sensible.  But in as much as they were to have an SLA, if their SLA was higher than the SLA of the  infrastructure, that's not Amazon's problem. That's Netflix's problem. So think about the  SLA for the software you're building. If that has to be more reliable than the platform  that it's built on, that's an architectural problem. You need to think about how to architect  your way around that. And that's where the scales horizontally thing comes from. The  way you make a, one of the ways you make a resilient system is by making sure that system  is resilient to failures of the underlying infrastructure, which means that you should  make sure you can move transparently between different availability zones. So Google and  Amazon and Etsy and other high performing companies, they actually test this. And they  test this by actually bringing data centers down. They do things called game days. There's  a great article on game days if you look in ACM where they actually will just, they will  say, okay, in six months we're going to run an exercise and they actually just turn off  the fiber to a data center. And then they'll ping the person and say, have you, and they'll  let them know they're going to do this. They're like, we're going to run a test on your SLA.  And then they'll say, okay, we're going to bring down the data center. And they physically  switch it off. And they're like, are you within your SLA? And then the site reliability engineer  will actually check to see if they're in their SLA. And then they'll bring down a different  data center because the SLA says you have to meet, be able to meet your SLA even if there's  two data centers down. So then they brought down the European data center and made sure  that they were still within their SLA's. So at Google, they test this by actually causing  disasters and seeing if the system will actually still meet its SLA's. And obviously they make  sure that if it doesn't, they can back out the disaster. They don't actually like burn  down data centers. Right? I mean, they just switch off the fiber to make sure that you're  okay and then they can switch it back on again if you're not. So that's how you test it by  actually causing real problems. And if that makes your compliance people start to freak  out, then you know, you've got a problem. But the problem is not with you, it's with  your compliance people. So I'm going to talk a bit about the continuous deployment part  of this, patterns for continuous deployment. So number one thing is that low risk releases  are incremental. So I'm going to present a number of different patterns for making releases  low risk and boring. Pattern number one is called expand contract. So you have a system,  it's probably composed of a bunch of different bits. You know, you have some static content,  you have your app, you have a bunch of services it depends on, probably has a database. Who's  building a system that doesn't have a database? Okay, maybe one dude, maybe. Okay. Yeah, all  right, fair enough. So what we don't want to do is deploy, you know, every time you  make a bunch of changes, you have to deploy all of those, what you don't want to do is  deploy changes to all the system at once. Instead what you do is you put out the new  version of the static content in a different directory before the release. And then what  you do is you can roll forward and roll back this without having to change the static content.  The same is true of services. If you depend on an upstream service, you don't deploy the  new version at the same time you deploy the app, you deploy the new version beforehand  and smoke test it and make sure that it works before you deploy the new version of the app  that depends on it. This is something that Amazon Web Services does, you know, if AWS  releases a new version of EC2, they don't just change the API and sorry you can't use  the old API anymore, that would make people very angry. Instead, you provide the API version  you want to use as a query string parameter so that as a consumer of the service, you  can choose when you want to upgrade. So we do that first again. We can also do that with  the database. So Amazon, Facebook, Etsy, these companies are not deploying database  schema changes a thousand times an hour. That's not happening. Etsy has database Thursday  when they deploy their database schema changes. So what that means is in Facebook, there's  one enormous production database and the developers do not get to choose what version of the database  is running at the time their software is deployed. They have to defensively code against whatever  version of the database happens to be there. And so what that forces you to do is make  database changes incrementally using expand contract. So for example, say you're changing  your address column to address line one and address line two. What do you do? Do you delete  this column and add these new columns at the same time you do the deployment? No. What  you do is first of all, you add the two new columns and you keep the old column there.  And then what happens is, so you can do this before you deploy the app because this doesn't  affect the existing application, right? We've just added some new database objects. That  application is going to ignore those. They're set to default null, right? And then you deploy  the new version of the application and the new version of the application will try and  read from these columns. If they're null or the columns don't exist, then it will read  from the old column. But it will always write to the new columns, yeah, the new columns,  so and the old ones. So we write to all the columns, but reading, we try and read from  the new ones and if we can't, we fall back to the old ones. And in this way, the application  will lazily migrate the data. So the new version of the app will lazily migrate the data from  here to here. But if we need to roll back the app, we've got the new data there. Rolling  back the app doesn't require rolling back the database because the old version of the app  can read the new data from the old data structure. And then much later, we can come back and  delete the old columns if we want to. So that's the essence of Expand Contract. We expand,  we add new stuff first, new database schema changes, new services, new static content.  Then we deploy the new version of the app and then we come back and contract to remove  the old stuff later. And again, that's expensive. You have to invest in that. What you're doing  is you're adding complexity to your software in order to reduce the risk of deployments.  It's a trade-off. It's not a silver bullet, it's a trade-off. And there's a fabulous  book called Release It by Mike Nygaard where this pattern and a bunch of other patterns  for creating resilient systems. It's a great book.  So how about making changes to the application in an incremental way? How do we deploy the  application in an incremental way? Well, we use a pattern called blue-green deployments.  So what this means is, real story, I was working on a system in 2005 where we could only get  one production set of hardware. We were not going to get another set of hardware because  it was too expensive. And so we didn't have a staging environment. That was scary. So  what we did is we created what we called a green environment, which was a new instance  of Apache and WebLogic and a new green database, which sat on the same boxes, but they were  installed on a different place in the file system and they listened on different ports.  So what we did is we deployed the new version to our green environment. And then that's  deployment. Release is just changing the router to point to the green environment ports.  Rollback is changing the router back to point at the blue environment ports. So in this  way, we were able to do deployment and the deployment took about an hour, but we were  able to do roll forward and roll back in less than a second, super safely, by using this  technique of separating out deployment and having two environments. And then when we  deployed version 1.3, that would go into the blue environment and 1.4 would go into the  green environment. So we did this by having multiple things on the same box. You can also  do this in several other different ways. By virtualization, if you have two data centers,  a main data center and a backup data center, your main data center is blue, your backup  data center is green. So you've got your current version running in your main data center,  you deploy the next version to your backup data center. And then release is switching  DNS to point to the new data center or your load balancer. And then your backup data center  becomes your main data center. Your main data center is now your backup data center and  you've tested your business continuity process for free at the same time. And this brings  us to a really important principle. Release and deployment are two different things. Deployment  is taking a build and putting it into an environment. Release is making a feature available to users.  And we normally achieve release through deployment. But they're actually two completely different  things. Facebook's release manager, Chuck Rossi, has a video where he says all major features  in Facebook are already in production. For the next, all features they're going to release  in the next six months are already in production. You just can't see them yet. So they're deploying  this stuff constantly into production, but it's not user-visible. They have a software  called Gatekeeper that controls which users can see which features. And they can change  that dynamically, which lets them roll a small feature experiment out, test it with a small  set of their users, get feedback, iterate, and then when they're happy, then they roll  it out to a larger and larger group of users. But we should be deploying all the time. Release  is when we turn that knob up to 100. And that should be a boring, low-risk activity that  the business can perform without even talking to IT. And it's low-risk because that software  has already been in production for months. And we know that it works. And we've done  all the testing on it in production before we actually release it to all the users.  Another technique for reducing the risk of release is called canary releasing. And again,  this is something that many large, high-performing organizations do. Facebook in particular,  they have a cluster called A1, which is a few thousand boxes. New releases go out to  that first. Facebook employees, when they go to facebook.com, they don't actually go  to facebook.com, which is this big cluster. They're automatically re-rooted to this cluster,  which is called inyour.facebook.com. So Facebook employees go to this. And so basically what  they do, I mean, people call this dog-fooding, which is weird because who wants to eat dog  food? So they call it drinking your own champagne. So they drink their own champagne. They test  it here before it goes out here. And then if the build is good here, then it goes here,  which is maybe a data center. And a very small set of users get sent to that. And then if  it's good, then they send it out to the rest of the world. And so it's like a canary in  a coal mine. In the old days, people used to take a canary into the coal mine. And the  canary, if there was dangerous gas in the coal mine, the canary would die. And then everyone  would run really fast out of the coal mine. And these days, they have chemical detectors,  which means that no birds are killed, which is good. So no animals were harmed in the  course of this presentation. So this is the canary. We make sure the software works here.  And we're monitoring business metrics. We're not monitoring like CPU or disk space or IO.  We're measuring how many users are actually sending messages through Facebook chat. How  many people are actually posting onto their timeline measuring actual user activities?  So if you're doing green field development, you can design with these patterns upfront.  You can design for testability. You can architect for deployability. What happens, who here has  an existing system that is definitely not designed for continuous delivery, but they  want to get from A to B? Okay, many of you. So that's why I just want to spend 10 minutes  on re-architecting. So I want to talk briefly about Amazon. Amazon in 2001 had a big ball  of mud in production, a big monolithic app called Obidos. And in 2001, Jeff Bezos, the  CEO of Amazon, sent a memo to his entire technical staff saying, we are going to re-architect  around a service-oriented architecture. And this is before anyone was even talking about  service-oriented architecture. And we're going to have a team for each service. So they  invented this term, two pizza teams. So no team can be bigger than you can feed with  two pizzas. Now, this is Seattle, so the pizzas are quite big. But still, that's about 10  to 12 people per team. So they re-architected the system so that every, they broke it into  services. The service was maintained through its entire life by a team of people who was  responsible for that particular service. And the rule is, you're not allowed to talk to  another service's database. You have to talk to other services through their service layer,  through their API. And they hired an US Army Ranger. And the job of the US Army Ranger was  to go around the different teams, and if he saw someone talking to another service through  the database, he would shout at them a lot. And if they carried on doing that, the rumor  has it that you would get fired. So I'm not recommending people get fired, but rumor has  it that that, you know, is a very serious offense not to actually go through the service layer  when you're talking to other services. And so over the course of a few years, they did  this. And so this is Werner Vogels, the CTO of Amazon. I'm going to put words into his  mouth. I love doing this. So this is in 2006, 2007, I think. If you hear Amazon.com, the  app calls more than 100 services to render the front page. Now, that causes, I mean,  that is non-trivial, and it causes other problems, like trying to, if something goes wrong with  that, how do you debug that when you've got 100 services collaborating to produce that  problem? Well, you need to do really good monitoring of the way calls go through that  graph of services. You need to do lots of caching to make that actually performant. But again,  it scales. It's a horizontally scalable architecture that doesn't have a single point of failure.  The reason they had to do this was because they kept scaling their single big database  bigger and bigger until finally the UI couldn't handle it. So they had to re-architect around  to create a reliable platform that based, sorry, a reliable service based on an unreliable  platform that scaled horizontally. And that's the upshot of this. But, you know, how do we  do this? I mean, and they did this in a somewhat incremental way. So there's still parts of  Albedos that are live today, I'm told. I don't know if that's true. It's really hard to find  out what's really going on at Amazon because they don't like to talk about it. So how do  we do this incrementally? Well, there's a pattern called strangler applications. So if  you've seen Tomb Raider, the film, okay, many of you. So lessons from Tomb Raider. If you've  seen Tomb Raider, you would have seen this temple in Cambodia where there's this tree  that's grown on top of the temple. And what's grown on top of the tree is there's a kind  of a fig called a strangler fig. And the seeds are on the tree and the fig grows down and  the fig grows up around the tree. And eventually the fig will strangle the tree. And all that's  left is the fig vine that's grown up around it. So this is a metaphor for what you should  be doing with architecture. I'm very happy that I was able to write our architectural  lessons from Tomb Raider. Super. So customers, big ball of mud, big fat database. What we  do is when new requirements come in, what we do is we find ways to implement those new  requirements using a new module. So maybe this module serves just one particular web  page as part of your whole site. And the stuff that needs to do that is in its own database.  But everything else gets delegated through to the original app. So we delegate most of  the stuff here and just this new page or even a new part of a page. So for example, Amazon,  the recommendations engine is a service. And so the part of the page that the recommendations  engine is done by this service. And maybe you have a new requirement for your website,  we need recommendations. So you don't implement that in the existing system. You implement  that as a new service. And maybe what we do is we have an include statement that pulls  from this particular service to render that part of the page. But all new functionality,  all new requirements you implement using new modules, new services that you build. And  gradually over time, what you end up with is a bunch of new services. And eventually,  finally, this thing gets strangled. We strangle the original application. So this is how we  incrementally move from big monolithic ball of mud that's not suitable for continuous  delivery to a new architecture that's service oriented that is designed for continuous delivery.  And again, this happens incrementally over time. The problem is, you know, our organizations  don't set up to do this. There's an annual budgeting cycle. We start a huge new project.  Who's experienced the enormous rebuilds? You know, the big bang re-system re-architecture.  We're going to re-architect everything from scratch. So hands up if that's actually worked  for you. Yeah, I thought so. So yeah, someone who has worked for it, it does sometimes work.  When that happens, I'm like, wow, you guys are really amazing. Or you have great senior  management politicians to make it look like it worked even though it didn't. But starting  a new project that's going to take a long time to make sure that it's definitely going  to work this time. So it does sometimes work. It's very, very, very risky. And so what I  saw, I mean, people object to this because they say, this is going to take too long.  But my reply to this is it's better for the journey to take a long time and to arrive  at the destination than for the journey to take a shorter amount of time but you never  actually get there. So I want to talk about how we do this in the small. So this is big  scale architectural change. How do we make smaller scale architectural changes, particularly  on trunk? How do you change a particular module or service on trunk and make architectural  changes? Well, this is a practical branch by abstraction. And what you do is this. We  employ, there's a joke in computer science. There's no problem you can't solve with an  additional layer of abstraction. So, or in direction. So what we do is we have a library  we want to change or a component we want to change. What we do is we put an abstraction  layer above it and make everything go through that. And then we incrementally move methods  or API calls from one implementation to another one. So I used to work on a product called  Go for release management. We changed the back end from Ibarthus to Hibernate and we  changed the front end from Velocity to JRubion Rails and we carried on releasing the software  during those two rearchitectures. So I just want to make sure I don't run out of time.  I'm going to talk briefly about the front end. The abstraction layer was the servler  API. We basically said if the URI had slash new, it would go through to JRubion Rails.  Otherwise it went through to the old Velocity stack. Any new requirement that came in that  added a new page or changed an existing page, we copied that page over to JRubion Rails.  And then we did just that one page. And then what happened was when that page was ready,  we changed the rest of the app to point to the slash new version here. But the rest of  the app was still using Velocity. So there's no way as a user for you to tell if you are  going through the Velocity stack or the JRubion Rails stack unless you looked at the URI.  Same CSS, same HTML, but which stack it was being rendered through, you had no way of  telling unless you looked at the URI to see which route it was going through. And so there's  still some pages in Velocity right now. It's just that, so you can keep the old stuff for  as long as you want. If you're a bit anal, you know, a bit OCD, then maybe you want to  get rid of the old implementation eventually. And, you know, if you're really OCD, you might  want to get rid of the interface as well. But it turns out you don't really need to  do that. And again, as a trade-off, you're adding complexity in order to manage the ability  to, this architectural cross-functional requirement of deployability. The final point I want to  make is that continuous delivery, it's not about changing your architecture to do CD.  Doing CD exerts a force on your architecture. Architecture is easy to change when you're  starting work. When you're a long way into your work, it's hard to change. By continuously  deploying throughout, you're constantly validating your architecture. By constantly doing performance  testing, by constantly deploying, by constantly doing acceptance testing, what you're doing  is creating a force on your architecture to make it testable, to make it deployable, and  to make sure that we're actually meeting our cross-functional requirements so that we can  find if there's a problem with our architecture early when it's cheap to fix, not late when  it's expensive and painful to fix. So, Dan North has a pattern called Dancing Skeleton,  which is based on Alistair Cobain's walking skeleton. And the idea is with a dancing  skeleton, you know, you get a few features in production and test them with a realistic  load and data set. And you actually have an API and make sure you can really exercise  that. And that's why it's dancing, because you're really using it. And again, as I talked  about, you're testing things like availability, security, disaster recovery, business continuity  planning by having game days, actually creating real disasters to make sure that your software  meets its cross-functional requirements. So the final take-aways, testability and deployability  are architectural constraints. You need to architect with these things in mind. We want  to be able to develop functionality on trunks so we can get fast feedback on whether what  we're doing really actually works and whether the work we're doing actually produces meets  our SLAs with the architecture we're using. You need to design for continuous deployment  and incremental deployment. You can do rearchitecture and redesign in an incremental way as well.  And the way to make sure that you're actually going to, your architecture is actually going  to work and meet your cross-functional constraints is by doing continuous delivery, by actually  practicing that thing all the time. That's how you verify it's really going to work.  So we have four minutes for questions. What questions do you have? Yes?  At the beginning you mentioned that you would be able to run your integration test on your  development machine. How do you manage that when you have to be like Android on?  Right. So any serious system, there's going to be like a bunch of different things and  there's going to be the platform as a service, infrastructure it runs on, these other things.  What you basically need to do is create mocks of those other services. So you should be  able to mock out that service and you don't necessarily need to do it for every service  because that becomes hard to maintain. But you find the big boundaries. So maybe you  can stand up two of these services in your development machine at once and you just stand  those things up. But then there's some other, so you group the services up and then stand  up a group of them and then look at the other things and you create an abstraction layer  between those things which is just good API design in any case and you create mocks of  those things. And what you can do, there's a technique that you can use where you basically  create a proxy and you stand up the whole system and record the messages and then you  just store those and then you can stand up part of the system and you just use the recording  of the messages, the can responses in order to be able to mock out the other systems.  I mean, that's how you solve it if you have an existing system that you need to redesign.  I mean, the other way you do it is just by making sure that you design the system so  you can find 80% of the bugs without even having to talk to the other systems at all.  So you just abstract that away. So if the other systems aren't there, my service will  still work but with a degraded quality of service. So using patterns, Mike Nygaard has  a pattern called circuit breaker which basically means if I can't contact the remote service,  I will break the circuit for a certain period of time and my service will still carry on  working but without that extra stuff that the external service brings you. So that's  just good design is that I should be able to operate my service even if the other ones  aren't there and so I should be able to do most of my testing with that being the case.  And then for the small part of your functionality where that's not true, then we use mocks.  Does that answer your question? Yes.  So that's an excellent question. There's always some sharp person who spots that in  the database part of BLEG green deployments. The question is, you know, if I do this, if  I roll forward to version 1.2 and I need to roll back, there's new stuff in this database,  right? What do I do about that? Well, the first thing you do is, I mean, maybe you  don't care about that. You need to start with what's my recovery point objective? What's  my recovery time objective? What are my actual SLAs around this? Maybe it's okay for me to  lose the data. If that's true, that's super because you can just, I mean, literally what  you do is before you deploy, you back up 1.1 database and you restore to the green 1.2  and you switch over. Then anything that happens between that and roll back, you just lose.  That may be fine. If it's not, you can do things like putting the app into read-only  modes for a certain period of time. If that's not acceptable, then you're going to have  to use expand contract. So expand contract the whole point.
Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_0_59.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_60_107.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_108_2195.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_2196_2639.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_2640_5675.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_5676_6359.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_6360_7415.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_7416_9047.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_9048_12875.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_12876_13319.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_13320_13427.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_13428_18635.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_18636_19391.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_19392_20783.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_20784_27947.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_27948_31787.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_31788_35039.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_35040_40835.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_40836_41639.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_41640_43391.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_43392_44303.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_44304_45287.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_45288_46727.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_46728_46775.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_46776_46847.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_46848_46919.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_46920_46967.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_46968_47015.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_47016_47735.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_47736_48311.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_48312_51599.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_51600_51647.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_51648_53099.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_53100_53771.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_53772_54107.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_54108_54155.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_54156_57155.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_57156_57491.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_57492_59375.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_59376_60527.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_60528_60575.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_60576_60635.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_60636_61139.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_61140_63491.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_63492_63551.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_63552_65699.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_65700_65867.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_65868_68591.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_68592_71183.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_71184_73211.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_73212_73535.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_73536_74363.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_74364_74711.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_74712_76019.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_76020_78731.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_78732_78947.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_78948_82043.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_82044_82511.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_82512_84143.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_84144_84743.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_84744_85331.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_85332_85835.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_85836_89243.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_89244_90695.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_90696_90755.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_90756_90803.png Jez_humble_-__Re-_architecting_for_Continuous_Delivery-68226813_90804_90832.png
Okay, so my name is Jess Humble. I work for ThoughtWorks. We do consulting and delivery for software and we have tools to help you with that. Mingle for agile project management, Go for agile release management and Twist for agile testing. I co-authored this book called Delivery and I'm here today to talk about architecting and re-architecting and continuous delivery and the relationship between these things. So it's not going to be a detailed kind of lots of code on the whiteboard talk. So if you want something very detailed and codey, now would be a good time to go and see one of the other things that is more detailed and codey. I'm not at all offended if people decide to walk out and go and see something else instead. If that would be the best use of your time, please go ahead and do that. Don't feel bad at all. Also there's voting at the end. So at the end, please choose a green card, a yellow card and a red card depending on how you feel about the talk. Remember that in China red is lucky but in Norway not so much. I've been having a great time in Norway. It's my first time here. I went to Bergen on Monday and I got sunburn which was unexpected but very nice. So architecture. Architecture is a term obviously that comes from building buildings and if you're actually building, if you're doing construction, doing civil engineering projects, one of the important things as an architect you have to care about is what's the nature of the materials you're using. Iron, steel, wood, plastic, all these things have different characteristics and that affects the way you think about what you're building. And so obviously if you're building, if you're architecting things that have to stand up for a long time like a bridge, you have to think carefully about the materials that you're using. Now project management and architecture are focused around building things like this or buildings which have a certain set of characteristics. So once you build a bridge or a building, normally it doesn't fall down. That's bad normally. That means you've done a bad job. Once you build it, it doesn't change a lot. So we don't build a bridge and go, well that isn't really right and that isn't what we wanted. We're going to totally change the way that works now because our users didn't like that. Running a bridge requires different skills from building the bridge, completely different set of practices. Once we have built the, once we've designed the architecture for the bridge and thought about how we want it to behave, actually then building it will not give us lots of new information that makes us think again about the architecture. That's not supposed to happen when you're building bridges and buildings. And then most importantly when you're building a bridge, you have to finish it before you can start using it and before you can determine the return on investment that you will get from having built it, the value that it's delivered. So you'll notice that all of these statements that I've made about bridges do not apply to software. And what that means is that architecture and project management techniques that we use for building bridges are actually really bad for building software. We should not be using them because they drive the wrong kinds of behaviors. Do we want our software once we've released it to users? Do we want to make sure that it doesn't change that much? No. We want to get feedback from users and build something that's valuable to them. Most of the time when you build software for someone, it's not what they want. 19 out of 20 startups fail because they've built something that wasn't valuable to customers. So, you know, if 19 out of 20 bridges that we built turn out not to be useful, that would be really bad. Right? So software is fundamentally different from civil engineering. And that affects the way we should think about architecture. In software, we can start delivering value to our customers from early on in the project before the thing is finished in terms of having all the features that we would ultimately like. People might derive, hopefully people can derive value from early on before we finish building all the features. We can give some people a subset of the features and they will find those useful. It's cheap to prototype and experiment with software. We can spike things out. We can prototype things very, very cheaply. That's not true with bridges. Once you've delivered some software, compared to a bridge, it's relatively simple to make changes, even large scale changes with software. And then finally, with a bridge, to build a bridge, you don't need highly paid architects. You need a bunch of construction workers. That's not true for software. And unfortunately, this idea that the highly paid architects will design the software and we can hire a bunch of cheap people who aren't very smart to actually build the software is one of the biggest problems in the software industry today. Because when you actually build the software, you make loads and loads of discoveries about how the thing will actually work because software is a complex system. Software systems are complex and they have dynamic complexity. That's not true of bridges. You can model bridges and model the forces on them analytically. You cannot model the way users will interact with your software using analytic models. They are complex systems. And when you build complex systems, you discover a lot of information in the course of building them that affects your decisions or that gives you feedback about your decisions on your design. Building software is harder than thinking about the design. So, what would a system built to leverage these characteristics, which are characteristics of software, if we built a system that leveraged these, if we actually used our materials and played to the strengths of these materials that we're using, software, what might that system look like if we actually played to the strengths of software rather than applied civil engineering techniques? Well, you might find something like this. Amazon.com, this was at Velocity last year, you can see that Amazon.com, they were making changes to production every 11 point seconds. They were making, doing 1,079 deployments to production per hour. Up to 10,000 boxes in production at the receiving end of a deployment, oh, sorry, on average, and 30,000 in total. So, people sometimes ask me, you know, Jess, can continuous delivery scale? And I say, have you heard of a website called Amazon? It's quite big. They are doing this and they have to worry about Sabane's Oxley and PCI DSS and all the regulations and things that make this stuff difficult. So, why did they do this? I mean, this is hard. Continuous delivery is hard. Doing all this stuff is hard. Why do they care about it? Primarily so they can run lots and lots of experiments to try and find out how to deliver value to users. When you design, when you come up with an idea for a software product or a feature, what you have is a hypothesis. And what you want to do is try and test that hypothesis. And the most expensive way to test the hypothesis is actually to build the software. That's the most expensive and inefficient thing that you can possibly do. You know, and if you're building bridges, you kind of have to do that, right, or buildings, but which is why in 2008, the construction industry took a total nose dive because suddenly we had all these things that were unfinished that we couldn't make money from. So, the reason they do this is so they can run experiments. So, they can come up with ideas for features and run an experiment and A-B test in production to find out if it will actually change people's habits. If people, they will make more money from people if we have the buy button here rather than here, if we present books that other customers like here on the page rather than here. How can we improve the experience of customers so they spend more money? And we do that by running loads and loads of experiments doing A-B testing. That's why they do this. It's so they can make things better for their customers by testing hypotheses really, really fast and getting feedback and establishing a close connection with their customers so they can deliver value. So, that's the main value of continuous delivery. And by the way, as a side effect of continuous delivery, you also reduce the risk of release and make deployments much easier and less risky, which is nice. Otherwise, you couldn't do this, right? So, the point of continuous delivery is to reduce the cost, the time and the risk of delivering incremental changes to users. It's so we can make incremental changes to our software all the time really, really cheaply and at low risk. That's what it is. How do you know when you're doing it? The idea is that your software should always be releasable on demand. From the very first feature you build, you should be able to press a button and release it. Releasing should be boring. If we're doing continuous delivery right, releases are boring. You're not going in at the weekend. You're not staying late at night. It's a boring thing you can do during peak hours by pressing a button. No big deal. We prioritize keeping the system releasable over new functionality. If at any point we discover the performance is dropped below a certain threshold or the acceptance tests are failing or there's a security problem or the unit tests are broken, we stop and fix that problem before we do new work. So, we're not, oh, you know, we must cut. The acceptance tests aren't working but those failures aren't important. We'll just carry on delivering features. No, you shouldn't be doing that. In order to make that possible, people need to get fast feedback. If I write some code or make a configuration change and it takes me a week to find out whether that's going to degrade performance, that makes it very hard for me to actually be fast at delivering code because I'm getting feedback from what I was doing a week ago on whether that thing actually affected performance. I don't know about you guys. I can barely remember what I was doing yesterday, let alone a week ago. So, I get a bug report from a week ago. I have to try and triage it, work out what was going on. It's an absolute pain. By the way, people sitting here, there are seats up here. So, please feel free to grab seats and bother these people. And if people at the end could move down so that these people can move in, that would be super. Thank you. Cool. Thanks very much, everyone. And this fundamentally changes the way we do product delivery. Instead of having these projects where it's very waterfully, what we should be doing is you start with an inception. We've got a new idea for a product. And we're going to get together as a team and work out what's the vision we're trying to achieve? What's the measurable customer outcome we want to achieve? And then design an experiment to test if we can achieve that measurable customer outcome using the proposed software program. So, we design experiments and then we run the experiment. It's called the minimum viable products, but it's an experiment. And most of the time, that will produce some results that tell us that users don't want what we built. If we're innovating, innovation is high risk. We want potentially high returns. When you want a high return, you are going to take risks. I mean, this is investment 101, right? And so naturally, if you're taking risks, most of the time, what you want, what you build is not actually going to be valuable. So, most of the time, we'll fail with this. And what we produce is not something that people want. And we either pivot, change our business plan, or we can keep delivering increments and trying to move closer towards what customers want. But the first thing we build will not be what our customers want if we're innovating. So, we need to find ways to find that out cheaply and then to iterate or pivot based on that. So, what are the implications of this for architecture? So, fundamentally, what architecture is about is cross functional requirements, performance, availability, scalability, all the other illities. That's what architecture is for. Architecture is what makes sure that our software will meet its cross functional requirements as opposed to the design of the features which is about meeting the functional requirements. So, architecture is what allows you to do this stuff. What is important for continuous delivery? First of all, we want to make sure that any increment that we build is testable at all levels. So, I should be able to run acceptance tests on my workstation. I should be able to get, find 80% of the bugs in my software by running it on my workstation. If I have to get an integrated environment and stand up 20 different services and try and get them to all work together before I can run my acceptance tests, I have a problem because it's expensive to do that and it's painful to do that. And so, I'm going to get that feedback very, very infrequently because it's such a pain to actually do that. So, that's an architectural thing. If I can't run the software in a non-integrated environment on my workstation and get the feedback, that means I have an architectural problem. We need to take account of that in the architecture, actually making sure that I can run tests on my workstation. If I need continuous delivery, I need to be able to do incremental development on trunk. That means continuous integration means everybody is checking into trunk at least once a day. We are not working on long-lived feature branches. If you're using feature branches and you're not integrating them into trunk once a day, then you can't do continuous delivery because you can't get feedback on whether your code actually works. You know, working on my workstation is meaningless. Works in production integrated with the rest of the code, with realistic data sets and realistic loads, then maybe I'll consider thinking about saying it's done. People use feature branches because they want to optimize for working, for getting done with their feature, but that's kind of pointless. So our architecture needs to enable us to be able to do that, to be able to work incrementally on trunk, which means we need to follow things like the solid principles, make sure that our software is well encapsulated and that it's componentized so that if I make a change, it doesn't splatter all over the code base. Deployability is an architectural concern. How easy is it to deploy the software? So people talk about deployment automation. We should take the deployment process and automate it. If the deployment process is really complex and flaky and I automate it, what happens is I turn a complex error-prone manual process into a complex error-prone automated process, which is no value. We need to simplify first, make sure that it's simple to deploy and then automate that. And if the architect, and that's an architectural concern, if it's really hard to deploy the app because I need to stand out a bunch of different things and then apply a bunch of different configuration settings from a bunch of different repositories and it's really difficult to actually stand up or to deploy some new part of that, then I can't do that. We need to focus on making deployment simple so that we can automate it effectively. Also, if we're going to deploy, you know, frequently, then we need to make sure that maybe we have availability constraints, maybe we have an SLA that says our service can't be down. So if I need to take the service down in order to deploy, maybe that's a problem. Maybe I won't meet up my SLAs. So you're thinking about things like, well, if I keep a bunch of state in the cache and I need to flush the cache every time I do a deployment and I can't do that, that's a problem. This is one of the reasons that Ruby on Rails has at least some nice characteristics. All the user state is in the database, so I don't need to worry about session caches and maintaining session caches across deployments. These are things that you need to think about if you're thinking about continuous delivery and by keeping state in session caches, availability and security and all these things during deployment. So architecture for continuous delivery, there's two things. There's firstly, making sure that we can test our cross functional characteristics all the time against any change. And this is an important characteristic of complex systems is that I make a change, I cannot predict the consequences of that change. This is the problem with code inspection, with code reviews. Code reviews will help you find some problems, but we are building complex systems. And so I make a change and I can perhaps in my head predict some of the consequences of that, but it's going to be really hard for me to predict things like what's the impact on performance. Nobody sets out to degrade performance. People degrade performance by mistake because they're building complex systems and we make changes at degrade performance and we didn't realize that was going to happen. And the person reviewing our code didn't realize that was going to happen either. So we need to be able to find those things and validate them by testing. The other part of architecture is designing for resilience. When you build complex systems, failure is a normal part of the operation of complex systems. It's not something exceptional, failure is a normal part of the operation of systems. This is what people doing disaster analysis for Three Mile Island and other problems discovered. When you build complex systems, you can't prevent failure, but you need to design your system to be resilient to failure. So that's the other part, the second part of architecture. So we're going to talk about testing first. If you have questions at any point, stick your hands up. I'll try and leave time for questions at the end as well, but if you ask, if you want clarifications, stick your hand up during. If you have a long question, keep it to the end, but feel free to argue with me if you think I'm talking crap. I appreciate that. So I'm going to talk about testing. There's this quadrant diagram created by Brian Merrick where he divides testing up along two axes, whether the test support programming or whether they critique the project, whether the tests are technology facing or whether they're business facing. And so down on the bottom left here, unit tests, component tests, these are tests that test one small part of the system in isolation, a single method, a single function, and then groups larger than that. And these should all be automated. And you should be using, who here uses TDD, who writes tests before they write the code that makes the test pass? That is fabulous. That's about a third to nearly a half of you. That's the best result I've ever had. So good job. TDD is, in my experience, the only way to create maintainable suites of unit tests. But that stuff should always be automated. Up on the top right here is the stuff that you use your expensive valuable human beings for. It showcases usability testing, exploratory testing. These are things that only human beings can do. On the top left here are the functional acceptance tests, end-to-end tests that run in a production-like environment that demonstrate that the software delivers the expected business value to users. And then here are the illities. These are the things that you're validating your architecture. Will the system scale? Will it meet its SLAs in terms of availability? Are there security problems? Is it deployable? Is it testable? So these are the things that are validating your architecture. And some of those things are automatable. So performance testing, if you've done your acceptance tests right, you can reuse them as a basis for load testing. Some kinds of security tests are automatable, like penetration testing. You can do some of that in an automated way. You can use static analysis to look for buffer overrun, SQL injection, attack, and all these kinds of things. But some of it can't be automated. So some parts of security testing are very hard to automate. What do we do once we have all these tests? What we do is we create a deployment pipeline. And I want to give you an example of a deployment pipeline in the context of an interesting domain which is firmware. So who here is working in embedded software? Any embedded software people? Okay, one person. Who's building websites or web services? Whoa, okay, almost everyone. Anyone building user-installed products? Okay, a smattering of you, maybe a quarter of you building user-installed products. So what I'm going to talk about applies to user-installed products as well. But it also is very important for building web services. A lot of the same lesson supply. So HP LaserJet team was building, I mean, these are the people who built the firmware for LaserJets. And in 2008, they had a problem. And their problem was they were too slow. And the business came to them and said, you're too slow, we need you to fix it. And they went and looked at what they were doing. And they found that most of their time was not spent writing new functionality. They had, for example, a different branch for every group of printers and scanners and multifunction devices they were creating. So every time they came out with a new product line, they created a new branch inversion control. And so they were spending 25% of their time porting code between branches. I had a new feature or make a bug fix to this line of printers. I want the same thing in another line of printers. I need to merge it between branches. And they were spending 25% of their time on current product support, 15% of their time on manual testing, detailed planning, code integration. You subtract that from 100, you're left with 5%. Which they claim was time spent innovating rather than lying down with aspirin in their body because they were totally exhausted. So they had to do something. And what they did was pretty radical. They re-architected the system from scratch. And their main, the main criteria for the re-architecture was the ability to check in everything to trunk. Not to have branches, everything was going to be on mainline. And the way they did that was they created feature toggles, basically. So there was one build that was created that was deployed to all the different devices. And when you start the device, the firmware boots, it looks to see the device that it's on and it switches features on or off depending on the device. So the ability for the developers to check in on trunk was an architectural concern. And they made sure that architecture supported that so they could get rid of this 25% of their time they were spending porting code between branches. And what they were then able to do was to create a deployment pipeline. Where they had, so HP LaserJet team, 400 people distributed across the US, Brazil, and India. Anytime anyone makes a change, we run about 45 minutes that so people check in to get to a branch, to a queue. And when they check in, we automatically run 45 minutes worth of tests against that queue. If they fail, you get an email which tells you what went wrong and they actually send you a binary of the build so you can run that on your workstation and reproduce that problem on your workstation. So you automatically get an email which tells you what the problem was and gives you a build so you can press a button and reproduce that problem on your workstation. Again, architectural concern. Any change that passes those tests, they batch them up and run about 45 minutes of tests again against the batched up changes. And if those changes pass that set of tests, then it gets checked automatically into trunk. So unless you pass these two gates, you can't get into trunk. But they were getting 10 to 15 good builds a day out of this process. So every day there'd be enough check-ins. There was about 100 to 150 check-ins every day. About 100,000 lines of code were changing every day. So it's a 10 million line code base, 100,000 lines of code changing every day, 100 to 150 check-ins, and they were getting 10 to 15 good builds a day out of this process, off trunk. Any build that comes out of stage two goes into level two. Level two is automated acceptance tests, about two hours worth of automated acceptance tests that run against the software running in a virtual machine that simulates being on a printer. If the build passes that, it goes to level three. Level three is a bunch of logic boards. And so the logic boards power on, they download the latest build that comes out of level two, and then automated acceptance tests that run against that build running on physical electronic logic boards to test. And then if you pass that, every night we run regression tests. They had about 30,000 hours worth of regression tests that run in parallel on a massive grid every night, which was level four. And what they did is they were constantly moving the tests between the different stages. So if you have a test that failed down here, we push it up into stage two. Any tests that are passing all the time in stage two, we push them down. So they were constantly moving tests between these stages to make sure they got fast feedback that they found out here that there was a problem, not here. And one of their goals at the program level was to make sure they were getting 95 to 98 percent pass rate out of level two and level three. So most of 95 to 98 percent of the problems were being found up here when it was fast. So it took a lot of investment for them to build this. They had to invest in building the automated tests. They had to invest in building this deployment pipeline. They had to invest in re-architecting the system to support development on trunk and to support being able to run. So anytime something fails in level three, same thing, you get an email with the bug details and a binary that you can deploy on your workstation to reproduce the problem. So they invested a lot of effort into that and they re-architected the system to support the ability to do this. And what they got as a result of that was that they knew within 24 hours whether there was a serious problem. And what that meant was they could introduce new features after code freeze. So they had a code freeze but it was basically irrelevant. You could still change the software and make new changes to the system even after code freeze right up to days before the new release because you knew you were always working off a good foundation. That your software was deployable and they prioritized keeping the software deployable over doing new work. And what that meant was they could go to the business and the business likes one year plans. And they said, okay, we're not going to do that anymore. In return, you can change your mind at any time about the features and we will just change direction right there and then. And that was the trade-off they were able to make which got rid of the time spent doing detail planning. So they didn't do detail planning anymore because the business can change their mind at any time and that's fine. And instead what they were able to do is get rid of a lot of this waste in the process. People think lean is about cutting cost. Lean is not about cutting cost. This is expensive to invest in. What it does is it removes waste and then that leads to cost reduction. And in particular they are able to spend much more time innovating and that drove down the cost of actually building new software. So continuous integration is probably the first thing that you need to be practicing as a way to drive this out. So who's doing continuous integration? Put your hands up. Keep your hands up if you're doing continuous integration. Keep your hands up. Put your hands down unless everyone is checking into trunk at least once a day. If you're working on feature branches that live for more than a day, put your hand down. If your build is broken for more than 10 minutes at a time before being fixed, put your hand down. If every check-in does not result in build and tests being run, put your hands down. Okay. So the people in the audience with their hands up are the people who are actually doing continuous integration. Congratulations to you. Continuous integration is not running Jenkins against your feature branches. It's a practice. You can do that practice without any tools at all. James Shaw has a paper, Continuous Integration on a Dollar a Day, that talks about doing continuous integration with an old workstation, a rubber chicken and a bell. No CI server. It's a practice. It's a mindset. Keeping the software releasable over doing new work. Always working off trunk. You know, here's 400 people asking, you know, does continuous integration scale? So exhibit A, 400 people distributed across three continents doing 100,000 lines of change every day to a 10 million line code base getting 10 to 15 good builds out of that. You scale this by throwing hardware at it. That's the definition of a scalable process that you can throw hardware at it and solve it. Using feature branches is not a scalable process because the longer the feature branches live and the more people there are on the team, the pain of integrating scales in a nonlinear way as you increase those numbers so it does not scale. Whereas continuous integration does scale. Exhibit B, this is from Google. Google have a continuous integration system that runs, there's 200,000 test suites in the code base. They're running 10 million test suites per day off their CI system in Google. They run about 60 million individual test cases per day. About 4,000 continuous integration builds every day. So continuous integration scales. But what continuous integration requires is that we work in small batches, that we make all changes. Any big change, we break into a series of small changes that keeps the software releasable. And that is the difficult bit. Having developers understand how to break every change into small bits. So if I was going to hypnotize you and have you and slip something into your unconscious, the most important thing I would have you take away from this is the importance of working in small batches. Whether that is, you know, if that's code changes or organizational change or any other kind of change that you're doing, working small batches, that's the thing I want you to take away. Whiskey, beer, working small batches, budgeting, anything you do, find ways to work in small batches, that's the important thing. So I'm going to talk now about patterns for continuous delivery in addition to the kind of the practices that you should be doing. So there's a great quote from this guy. This is Jesse Robbins. He was in charge of availability at Amazon. So all of Amazon's sites, he was the head of availability for all those sites. He's now, he was co-founder of Ops Code, who makes Chef. He's also a volunteer firefighter in his spare time, so an enormous overachiever, basically. So this is what he says about web architecture. Success on the web depends on continuous deployment of reliable software to an unreliable platform that scales horizontally. So that is one of the best statements of web architecture that I've ever found, because it covers so many things in such a dense single quote. So unreliable platform. If you're using Amazon, AWS, any of these different systems, that's an unreliable platform. Do you guys have Netflix in Norway? Yeah, okay. So if you were using Netflix on Christmas Eve last year, you would have noticed that it didn't work. So there was me, Christmas Eve, I have a four-year-old daughter, and she was wanting to watch Disney films on Christmas Eve, and she couldn't because Netflix was down and we have Apple TV and not a proper TV. So that was a problem. I was quite upset about this. And it turns out this was because of a load balancing failure in one of the Amazon zones that caused this problem. Now, people were very quick to blame Amazon about this. This was not Amazon's problem. Amazon were within their SLA. Netflix was not able to meet its SLA. Actually, I'm not sure if that's true because they probably don't even have an SLA because they're sensible. But in as much as they were to have an SLA, if their SLA was higher than the SLA of the infrastructure, that's not Amazon's problem. That's Netflix's problem. So think about the SLA for the software you're building. If that has to be more reliable than the platform that it's built on, that's an architectural problem. You need to think about how to architect your way around that. And that's where the scales horizontally thing comes from. The way you make a, one of the ways you make a resilient system is by making sure that system is resilient to failures of the underlying infrastructure, which means that you should make sure you can move transparently between different availability zones. So Google and Amazon and Etsy and other high performing companies, they actually test this. And they test this by actually bringing data centers down. They do things called game days. There's a great article on game days if you look in ACM where they actually will just, they will say, okay, in six months we're going to run an exercise and they actually just turn off the fiber to a data center. And then they'll ping the person and say, have you, and they'll let them know they're going to do this. They're like, we're going to run a test on your SLA. And then they'll say, okay, we're going to bring down the data center. And they physically switch it off. And they're like, are you within your SLA? And then the site reliability engineer will actually check to see if they're in their SLA. And then they'll bring down a different data center because the SLA says you have to meet, be able to meet your SLA even if there's two data centers down. So then they brought down the European data center and made sure that they were still within their SLA's. So at Google, they test this by actually causing disasters and seeing if the system will actually still meet its SLA's. And obviously they make sure that if it doesn't, they can back out the disaster. They don't actually like burn down data centers. Right? I mean, they just switch off the fiber to make sure that you're okay and then they can switch it back on again if you're not. So that's how you test it by actually causing real problems. And if that makes your compliance people start to freak out, then you know, you've got a problem. But the problem is not with you, it's with your compliance people. So I'm going to talk a bit about the continuous deployment part of this, patterns for continuous deployment. So number one thing is that low risk releases are incremental. So I'm going to present a number of different patterns for making releases low risk and boring. Pattern number one is called expand contract. So you have a system, it's probably composed of a bunch of different bits. You know, you have some static content, you have your app, you have a bunch of services it depends on, probably has a database. Who's building a system that doesn't have a database? Okay, maybe one dude, maybe. Okay. Yeah, all right, fair enough. So what we don't want to do is deploy, you know, every time you make a bunch of changes, you have to deploy all of those, what you don't want to do is deploy changes to all the system at once. Instead what you do is you put out the new version of the static content in a different directory before the release. And then what you do is you can roll forward and roll back this without having to change the static content. The same is true of services. If you depend on an upstream service, you don't deploy the new version at the same time you deploy the app, you deploy the new version beforehand and smoke test it and make sure that it works before you deploy the new version of the app that depends on it. This is something that Amazon Web Services does, you know, if AWS releases a new version of EC2, they don't just change the API and sorry you can't use the old API anymore, that would make people very angry. Instead, you provide the API version you want to use as a query string parameter so that as a consumer of the service, you can choose when you want to upgrade. So we do that first again. We can also do that with the database. So Amazon, Facebook, Etsy, these companies are not deploying database schema changes a thousand times an hour. That's not happening. Etsy has database Thursday when they deploy their database schema changes. So what that means is in Facebook, there's one enormous production database and the developers do not get to choose what version of the database is running at the time their software is deployed. They have to defensively code against whatever version of the database happens to be there. And so what that forces you to do is make database changes incrementally using expand contract. So for example, say you're changing your address column to address line one and address line two. What do you do? Do you delete this column and add these new columns at the same time you do the deployment? No. What you do is first of all, you add the two new columns and you keep the old column there. And then what happens is, so you can do this before you deploy the app because this doesn't affect the existing application, right? We've just added some new database objects. That application is going to ignore those. They're set to default null, right? And then you deploy the new version of the application and the new version of the application will try and read from these columns. If they're null or the columns don't exist, then it will read from the old column. But it will always write to the new columns, yeah, the new columns, so and the old ones. So we write to all the columns, but reading, we try and read from the new ones and if we can't, we fall back to the old ones. And in this way, the application will lazily migrate the data. So the new version of the app will lazily migrate the data from here to here. But if we need to roll back the app, we've got the new data there. Rolling back the app doesn't require rolling back the database because the old version of the app can read the new data from the old data structure. And then much later, we can come back and delete the old columns if we want to. So that's the essence of Expand Contract. We expand, we add new stuff first, new database schema changes, new services, new static content. Then we deploy the new version of the app and then we come back and contract to remove the old stuff later. And again, that's expensive. You have to invest in that. What you're doing is you're adding complexity to your software in order to reduce the risk of deployments. It's a trade-off. It's not a silver bullet, it's a trade-off. And there's a fabulous book called Release It by Mike Nygaard where this pattern and a bunch of other patterns for creating resilient systems. It's a great book. So how about making changes to the application in an incremental way? How do we deploy the application in an incremental way? Well, we use a pattern called blue-green deployments. So what this means is, real story, I was working on a system in 2005 where we could only get one production set of hardware. We were not going to get another set of hardware because it was too expensive. And so we didn't have a staging environment. That was scary. So what we did is we created what we called a green environment, which was a new instance of Apache and WebLogic and a new green database, which sat on the same boxes, but they were installed on a different place in the file system and they listened on different ports. So what we did is we deployed the new version to our green environment. And then that's deployment. Release is just changing the router to point to the green environment ports. Rollback is changing the router back to point at the blue environment ports. So in this way, we were able to do deployment and the deployment took about an hour, but we were able to do roll forward and roll back in less than a second, super safely, by using this technique of separating out deployment and having two environments. And then when we deployed version 1.3, that would go into the blue environment and 1.4 would go into the green environment. So we did this by having multiple things on the same box. You can also do this in several other different ways. By virtualization, if you have two data centers, a main data center and a backup data center, your main data center is blue, your backup data center is green. So you've got your current version running in your main data center, you deploy the next version to your backup data center. And then release is switching DNS to point to the new data center or your load balancer. And then your backup data center becomes your main data center. Your main data center is now your backup data center and you've tested your business continuity process for free at the same time. And this brings us to a really important principle. Release and deployment are two different things. Deployment is taking a build and putting it into an environment. Release is making a feature available to users. And we normally achieve release through deployment. But they're actually two completely different things. Facebook's release manager, Chuck Rossi, has a video where he says all major features in Facebook are already in production. For the next, all features they're going to release in the next six months are already in production. You just can't see them yet. So they're deploying this stuff constantly into production, but it's not user-visible. They have a software called Gatekeeper that controls which users can see which features. And they can change that dynamically, which lets them roll a small feature experiment out, test it with a small set of their users, get feedback, iterate, and then when they're happy, then they roll it out to a larger and larger group of users. But we should be deploying all the time. Release is when we turn that knob up to 100. And that should be a boring, low-risk activity that the business can perform without even talking to IT. And it's low-risk because that software has already been in production for months. And we know that it works. And we've done all the testing on it in production before we actually release it to all the users. Another technique for reducing the risk of release is called canary releasing. And again, this is something that many large, high-performing organizations do. Facebook in particular, they have a cluster called A1, which is a few thousand boxes. New releases go out to that first. Facebook employees, when they go to facebook.com, they don't actually go to facebook.com, which is this big cluster. They're automatically re-rooted to this cluster, which is called inyour.facebook.com. So Facebook employees go to this. And so basically what they do, I mean, people call this dog-fooding, which is weird because who wants to eat dog food? So they call it drinking your own champagne. So they drink their own champagne. They test it here before it goes out here. And then if the build is good here, then it goes here, which is maybe a data center. And a very small set of users get sent to that. And then if it's good, then they send it out to the rest of the world. And so it's like a canary in a coal mine. In the old days, people used to take a canary into the coal mine. And the canary, if there was dangerous gas in the coal mine, the canary would die. And then everyone would run really fast out of the coal mine. And these days, they have chemical detectors, which means that no birds are killed, which is good. So no animals were harmed in the course of this presentation. So this is the canary. We make sure the software works here. And we're monitoring business metrics. We're not monitoring like CPU or disk space or IO. We're measuring how many users are actually sending messages through Facebook chat. How many people are actually posting onto their timeline measuring actual user activities? So if you're doing green field development, you can design with these patterns upfront. You can design for testability. You can architect for deployability. What happens, who here has an existing system that is definitely not designed for continuous delivery, but they want to get from A to B? Okay, many of you. So that's why I just want to spend 10 minutes on re-architecting. So I want to talk briefly about Amazon. Amazon in 2001 had a big ball of mud in production, a big monolithic app called Obidos. And in 2001, Jeff Bezos, the CEO of Amazon, sent a memo to his entire technical staff saying, we are going to re-architect around a service-oriented architecture. And this is before anyone was even talking about service-oriented architecture. And we're going to have a team for each service. So they invented this term, two pizza teams. So no team can be bigger than you can feed with two pizzas. Now, this is Seattle, so the pizzas are quite big. But still, that's about 10 to 12 people per team. So they re-architected the system so that every, they broke it into services. The service was maintained through its entire life by a team of people who was responsible for that particular service. And the rule is, you're not allowed to talk to another service's database. You have to talk to other services through their service layer, through their API. And they hired an US Army Ranger. And the job of the US Army Ranger was to go around the different teams, and if he saw someone talking to another service through the database, he would shout at them a lot. And if they carried on doing that, the rumor has it that you would get fired. So I'm not recommending people get fired, but rumor has it that that, you know, is a very serious offense not to actually go through the service layer when you're talking to other services. And so over the course of a few years, they did this. And so this is Werner Vogels, the CTO of Amazon. I'm going to put words into his mouth. I love doing this. So this is in 2006, 2007, I think. If you hear Amazon.com, the app calls more than 100 services to render the front page. Now, that causes, I mean, that is non-trivial, and it causes other problems, like trying to, if something goes wrong with that, how do you debug that when you've got 100 services collaborating to produce that problem? Well, you need to do really good monitoring of the way calls go through that graph of services. You need to do lots of caching to make that actually performant. But again, it scales. It's a horizontally scalable architecture that doesn't have a single point of failure. The reason they had to do this was because they kept scaling their single big database bigger and bigger until finally the UI couldn't handle it. So they had to re-architect around to create a reliable platform that based, sorry, a reliable service based on an unreliable platform that scaled horizontally. And that's the upshot of this. But, you know, how do we do this? I mean, and they did this in a somewhat incremental way. So there's still parts of Albedos that are live today, I'm told. I don't know if that's true. It's really hard to find out what's really going on at Amazon because they don't like to talk about it. So how do we do this incrementally? Well, there's a pattern called strangler applications. So if you've seen Tomb Raider, the film, okay, many of you. So lessons from Tomb Raider. If you've seen Tomb Raider, you would have seen this temple in Cambodia where there's this tree that's grown on top of the temple. And what's grown on top of the tree is there's a kind of a fig called a strangler fig. And the seeds are on the tree and the fig grows down and the fig grows up around the tree. And eventually the fig will strangle the tree. And all that's left is the fig vine that's grown up around it. So this is a metaphor for what you should be doing with architecture. I'm very happy that I was able to write our architectural lessons from Tomb Raider. Super. So customers, big ball of mud, big fat database. What we do is when new requirements come in, what we do is we find ways to implement those new requirements using a new module. So maybe this module serves just one particular web page as part of your whole site. And the stuff that needs to do that is in its own database. But everything else gets delegated through to the original app. So we delegate most of the stuff here and just this new page or even a new part of a page. So for example, Amazon, the recommendations engine is a service. And so the part of the page that the recommendations engine is done by this service. And maybe you have a new requirement for your website, we need recommendations. So you don't implement that in the existing system. You implement that as a new service. And maybe what we do is we have an include statement that pulls from this particular service to render that part of the page. But all new functionality, all new requirements you implement using new modules, new services that you build. And gradually over time, what you end up with is a bunch of new services. And eventually, finally, this thing gets strangled. We strangle the original application. So this is how we incrementally move from big monolithic ball of mud that's not suitable for continuous delivery to a new architecture that's service oriented that is designed for continuous delivery. And again, this happens incrementally over time. The problem is, you know, our organizations don't set up to do this. There's an annual budgeting cycle. We start a huge new project. Who's experienced the enormous rebuilds? You know, the big bang re-system re-architecture. We're going to re-architect everything from scratch. So hands up if that's actually worked for you. Yeah, I thought so. So yeah, someone who has worked for it, it does sometimes work. When that happens, I'm like, wow, you guys are really amazing. Or you have great senior management politicians to make it look like it worked even though it didn't. But starting a new project that's going to take a long time to make sure that it's definitely going to work this time. So it does sometimes work. It's very, very, very risky. And so what I saw, I mean, people object to this because they say, this is going to take too long. But my reply to this is it's better for the journey to take a long time and to arrive at the destination than for the journey to take a shorter amount of time but you never actually get there. So I want to talk about how we do this in the small. So this is big scale architectural change. How do we make smaller scale architectural changes, particularly on trunk? How do you change a particular module or service on trunk and make architectural changes? Well, this is a practical branch by abstraction. And what you do is this. We employ, there's a joke in computer science. There's no problem you can't solve with an additional layer of abstraction. So, or in direction. So what we do is we have a library we want to change or a component we want to change. What we do is we put an abstraction layer above it and make everything go through that. And then we incrementally move methods or API calls from one implementation to another one. So I used to work on a product called Go for release management. We changed the back end from Ibarthus to Hibernate and we changed the front end from Velocity to JRubion Rails and we carried on releasing the software during those two rearchitectures. So I just want to make sure I don't run out of time. I'm going to talk briefly about the front end. The abstraction layer was the servler API. We basically said if the URI had slash new, it would go through to JRubion Rails. Otherwise it went through to the old Velocity stack. Any new requirement that came in that added a new page or changed an existing page, we copied that page over to JRubion Rails. And then we did just that one page. And then what happened was when that page was ready, we changed the rest of the app to point to the slash new version here. But the rest of the app was still using Velocity. So there's no way as a user for you to tell if you are going through the Velocity stack or the JRubion Rails stack unless you looked at the URI. Same CSS, same HTML, but which stack it was being rendered through, you had no way of telling unless you looked at the URI to see which route it was going through. And so there's still some pages in Velocity right now. It's just that, so you can keep the old stuff for as long as you want. If you're a bit anal, you know, a bit OCD, then maybe you want to get rid of the old implementation eventually. And, you know, if you're really OCD, you might want to get rid of the interface as well. But it turns out you don't really need to do that. And again, as a trade-off, you're adding complexity in order to manage the ability to, this architectural cross-functional requirement of deployability. The final point I want to make is that continuous delivery, it's not about changing your architecture to do CD. Doing CD exerts a force on your architecture. Architecture is easy to change when you're starting work. When you're a long way into your work, it's hard to change. By continuously deploying throughout, you're constantly validating your architecture. By constantly doing performance testing, by constantly deploying, by constantly doing acceptance testing, what you're doing is creating a force on your architecture to make it testable, to make it deployable, and to make sure that we're actually meeting our cross-functional requirements so that we can find if there's a problem with our architecture early when it's cheap to fix, not late when it's expensive and painful to fix. So, Dan North has a pattern called Dancing Skeleton, which is based on Alistair Cobain's walking skeleton. And the idea is with a dancing skeleton, you know, you get a few features in production and test them with a realistic load and data set. And you actually have an API and make sure you can really exercise that. And that's why it's dancing, because you're really using it. And again, as I talked about, you're testing things like availability, security, disaster recovery, business continuity planning by having game days, actually creating real disasters to make sure that your software meets its cross-functional requirements. So the final take-aways, testability and deployability are architectural constraints. You need to architect with these things in mind. We want to be able to develop functionality on trunks so we can get fast feedback on whether what we're doing really actually works and whether the work we're doing actually produces meets our SLAs with the architecture we're using. You need to design for continuous deployment and incremental deployment. You can do rearchitecture and redesign in an incremental way as well. And the way to make sure that you're actually going to, your architecture is actually going to work and meet your cross-functional constraints is by doing continuous delivery, by actually practicing that thing all the time. That's how you verify it's really going to work. So we have four minutes for questions. What questions do you have? Yes? At the beginning you mentioned that you would be able to run your integration test on your development machine. How do you manage that when you have to be like Android on? Right. So any serious system, there's going to be like a bunch of different things and there's going to be the platform as a service, infrastructure it runs on, these other things. What you basically need to do is create mocks of those other services. So you should be able to mock out that service and you don't necessarily need to do it for every service because that becomes hard to maintain. But you find the big boundaries. So maybe you can stand up two of these services in your development machine at once and you just stand those things up. But then there's some other, so you group the services up and then stand up a group of them and then look at the other things and you create an abstraction layer between those things which is just good API design in any case and you create mocks of those things. And what you can do, there's a technique that you can use where you basically create a proxy and you stand up the whole system and record the messages and then you just store those and then you can stand up part of the system and you just use the recording of the messages, the can responses in order to be able to mock out the other systems. I mean, that's how you solve it if you have an existing system that you need to redesign. I mean, the other way you do it is just by making sure that you design the system so you can find 80% of the bugs without even having to talk to the other systems at all. So you just abstract that away. So if the other systems aren't there, my service will still work but with a degraded quality of service. So using patterns, Mike Nygaard has a pattern called circuit breaker which basically means if I can't contact the remote service, I will break the circuit for a certain period of time and my service will still carry on working but without that extra stuff that the external service brings you. So that's just good design is that I should be able to operate my service even if the other ones aren't there and so I should be able to do most of my testing with that being the case. And then for the small part of your functionality where that's not true, then we use mocks. Does that answer your question? Yes. So that's an excellent question. There's always some sharp person who spots that in the database part of BLEG green deployments. The question is, you know, if I do this, if I roll forward to version 1.2 and I need to roll back, there's new stuff in this database, right? What do I do about that? Well, the first thing you do is, I mean, maybe you don't care about that. You need to start with what's my recovery point objective? What's my recovery time objective? What are my actual SLAs around this? Maybe it's okay for me to lose the data. If that's true, that's super because you can just, I mean, literally what you do is before you deploy, you back up 1.1 database and you restore to the green 1.2 and you switch over. Then anything that happens between that and roll back, you just lose. That may be fine. If it's not, you can do things like putting the app into read-only modes for a certain period of time. If that's not acceptable, then you're going to have to use expand contract. So expand contract the whole point.
is hard. Continuous delivery is hard. Doing all right? So, the point of continuous delivery is to customers want. But the first thing we build will that. So, what are the implications of this for of pointless. So our architecture needs to enable the other part, the second part of architecture. big ball of mud, big fat database. What we do is recommendations engine is done by this service. And maybe you have a new requirement for your So this is how we incrementally move from big cross-functional requirement of deployability. final take-aways, testability and deployability