Using Git Hooks to Help Your Engineering Teams Work Autonomously
Santos, João
João Santos - Using Git Hooks to Help Your Engineering Teams Work Autonomously In this talk, Software Engineer Joao Santos will describe how the engineering team at Zalando has been migrating to local Git hooks to ensure that engineers can work autonomously and flexibly. Zalando- Europe’s leading online fashion platform for men, women and children- began shifting from SVN to Git in late 2013. Santos and his colleagues used Python to create a Git update hook that enabled the team to reject changes to a branch while still allowing changes to other branches. He’ll explain why his team chose Python for this job instead of a bash script, point out mistakes made during the process (and solutions his team used to fix them), and the benefits generated by this migration. He’ll also talk about turnstile: a set of open-source, configurable, optional local Git hooks, created by the Zalando team, that enables engineers to abide by internal rules for committing code while following their own coding style and workflow preferences.
 So first to present myself, I'm Jun Santu. I'm a software engineer at Salando as part of the continuous delivery and deployment team. And today I'm going to present the part we took at Salando from the dark cases of subversion to the enlightened age of autonomous work with EGIT. So some quick facts about Salando. So you understand the context of my work. We are, we have our headquarters in Berlin. We have also other offices in other places like Dublin, Helsinki, Dortmund, airport. We have more than 700 actually nowadays almost 1000 engineers. And we do lots of every almost everything we do is done in house. So we do lots of development. Okay. So the first question when talking about GITUX is why do we need GITUX in or subversion groups or other kinds of commit so in the first place? The answer to this is to enforce rules. This raises the second question. Why do we need rules? Any large group organization needs rules. Some rules are internal to the groups, some are external. And they are very important to achieve an efficient workflow to ensure quality and transparency to stakeholders. And in case of Salando, stakeholders go from the colleague at next desk to the customer that as trusted as his credit card to the investor who spent lots of his money and he trusted and believed in us. Second question is how we enforce rules. Not our rules are enforced in GITUX. Some rules are enforced before deployments. Some rules are only monitored are not enforced. And so you have to look at your workflow and see what rules can be enforced a priori or with checks and what rules can be checked after you have moved on your workflow. And it's vital to keep a proper balance between two types of enforcement because if you have too many blockers on your workflow, you prevent your developers from being productive. But if you have to, few checks and blockers, you risk that bad software goes live and you break the trust all stakeholders that are in you. So now starting point. So this is a version. This was the place where we were when I arrived at Salando two years ago. We had one batch script with some basic checks. A precommit took. Then we were growing as a company and we needed more rules because it was not possible for someone to keep track of everything that was happening. It was not possible to keep a batch script anymore. It was not efficient. It was not maintainable. So we ported the GITUX subversion from Pesh to Python. Around December 2013, we started moving from GIT because the subversion was not good enough for guys anymore and we had to do something. So we started moving to GIT. We still had same rules in places before and we knew we would still keep subversion for some time because we already had lots of teams and some teams wanted to move faster and others slower. Not everyone wanted to move to GIT so we had to convince everyone and we decided to do migration in waves. So the first decision we took was to adapt the SVN commit took to work on both SVN and GIT. So we implemented GITUX as a per-gC book because it's invoked at the same point on the workflow as commit took on subversion. This is when user sense changes to the remote server and because it allowed us to reject the reference or branch without rejecting full push. The other alternative would be an update took but in that one if you return an error code it will block the full push even if most of it is following the rules. Doing this had some problems. This approach had some problems. The first problem in this plan was that GIT is not subversion. They are different in fundamental ways and trying to support both would mean support for both of them. The other problem is that being distributed GIT is more flexible than subversion which means that people use GIT in the same way they use subversion and don't use GIT in the same way that other people use GIT. The first things that I learned to migrate to GIT start experimenting different workflows and this was a good thing because they were very different projects and different projects have very different requirements. So it makes sense for them to have a different set of best practices and slightly different rules and different pricing strategy but this also forces us to rethink how we check the rules and how we implement the rules and we had to rethink the GIT rules. So we moved to plan B where before we had one common look for GIT and subversion. Now we decided to have two different looks. In practice this meant we forked to look into...the subversion commit look was aggregated so no more changes there because we don't want to really support the subversion anymore and we continued to work on to get priority look because different teams wanted to use different workflows. So we moved away from one size fits all strategy and allowed teams to specify set rules for themselves. So this is the way we allowed teams to set rules for themselves. We have a configuration format based on YAML where Stash projects and repositories could mesh several times by name. This is similar to CSS so for example you can see there that one team could have one general rule for themselves for their project but for a specific repository they could change part of the rules and not the whole set and a second team could have a completely different rule set because they have different needs. The problem with this second approach and this was a reflection we did after using the GIT looks for one year was that we had a centralized configuration which became a bottleneck because it could only be changed by a small number of people and sometimes we were all on vacation and it was really bad. Another more technical problem we had is that we were trying to check all the commits on a push because some rules apply to every single commit. For example we checked that one of the rules we had at Zalendo was that every commit message has to have a ticket and for this we have to go through all the commits and this was problematic for a number of reasons. For example GIT's history is not linear so if the user merges or rebases from a branch and we try to check all the messages in the commit range for ticket ID and see if it matches to a branch, a commit set came from other branches by the rebase or the merge would fail the check. In the end we tried to several solutions, some worked way better than others, in the end the solution for this was filtering out commits that were already in a branch because this meant they were already checked before and were not originated on this branch. The second problem we had is that when you merge to or from a long-lived branch, some teams had some very long-lived projects and they had branches for it and sometimes they deferred from master for two, three months and when they merged from master or to master or when they rebased they had to check. Because of some of our checks, sometimes we had pushes that took more than half an hour to check and this was not a good experience for our developers. Related to this, one of the slowest checks we had is checking code style, especially for Java code because we use Jallopee to validate code style and it runs on JVM and the architecture of our code validator meant we had to spawn one JVM for every changed file. So if someone pushed 200 changed files we would spawn up 200 JVMs and other people were pushing at the same time and it was constantly freezing and slowing down our server. We were able to mitigate this problem by using NailGant that is a kind of server that keeps JVM alive and you can run JVM programs there and we could run Jallopee there but we had to do some work around to run Jallopee in parallel and we still had some performance issues for some reason there were more pushes than usual. The last problem of this is that the system was inflexible. Sometimes people have very good reasons to ignore rules. Sometimes they pushed something that they didn't want to push and they want to do a forced push and we didn't allow that. Sometimes they, sometimes some push was accepted by mistake and it was a bag and it has to be removed because it should not be there. Sometimes there's a bag and we reject it. Something that we should have accepted and because this was on our server and people didn't have access to the server, they had no way to go around this. The other way we were inflexible is that because it's a remote Github and it's living on the server, we only supported our internal Github server. For example, you could not use our Github for our projects on github.com and so our open source projects were completely unsupported. In the end, I could summarize all these problems as we tried to centralize Github. While Github is distributed, we still saw our source control management system as centralized because we came from subversion and we were still thinking like subversion. But it was clear that we would have to rethink our approach. This process of thinking about the Github's coincided with the use change that's the ZLANDU as part of something we call ready calligility. ZLANDU was getting rid of all the error keys and giving teams more autonomy and one of the new motors was autonomy instead of control. And this presented a clear path for us and in the end was the solution that we are still implementing. So we decided to move away from remote Github's to a set of local Github's. One advantage of this is that Github's see one commit at a time. So we avoided the issues where we had a set, a list of commits to check and we didn't know what came from a merge or a rebase and what was created in the branch. And it's better adjusted to the distributed nature of Github because they are distributed by themselves and people run it on their local machines. We decided that they should be optional because we don't trust people. This also means that Github's are no longer responsible for enforcing the rules. Instead, we are creating mechanisms to ensure that a level of development follows the rules before the code goes live but we don't block anyone from pushing anything right now. We did this because we believe Github's should be seen as a tool to help autonomous developers and not as a barrier that makes their work harder. We decided to make them extensible because as I said before, one size doesn't fit all different teams have different needs and sometimes we only have a small group of developers working on this more or less full time but sometimes people have other ideas and they can implement new stuff for themselves. Sometimes they don't want one of our checks and they don't want to install the dependencies of our checks so by being extensible, they can opt out of those dependencies. And we decided to make it open source because by only being and thinking open source, it could fully support autonomous teams and this also allowed us to support our teams that want to open source their code and this also includes my team. So our new set of hooks that we call turnstile are available on Github and by PI so everyone is free to use them. So if you want to install it on your machine, you can just use pip install turnstile core. This will just install core without any of our extensions. We also have some extensions on Zalendo's Github account. So now you want to use it on your repository. What do you have to use? Well, the first step is add a turnstile.eml file in the root of your repository. If you use something like Travis or something similar, the same process you just added a file to your repository and it looks something like this. You have a list of checks you want to use. So you just name them and you can give options to all checks that they support by themselves. And every check, even if you are implementing your own checks as an extension, you can read this file and you have access to it. So and because the way local Github works, you need to see link to on your.kit folder inside the repository and to make this easy, turnstile supports a subcomment that's turnstile installed that will automatically see link to looks for you. And because I was already adding support for subcomments, I added several more. We have to config subcomments that right now the only thing it does is set, allow you to set verbosity. So if you want only to see errors, you can only see the errors. If you want to see everything that turnstile is doing behind since you can also do it, you have to install command to add to looks to a specific repository. We have to remove if you decide to, you don't want to looks anymore for the repository. The specification subcomment is used for check if all your projects contain a valid specification of all your commits contain a valid specification. And right now this means if they start with a new URL, but I'm planning to support more stuff in the near future. The upgrades of command will check by PI to see if turnstile and all your extensions are updated and if they are not, it will offer to update everything for you. And version just brings the version and it's not that interesting. So now if you want to create extensions to turnstile, how can you do it? The way you do it is with set of tools and entry points. So you just have, if you want to add a command, you just go to the end, use the entry point turnstile.comments. You give the name of your subcomment and this will be used to call it from the command line and you just provide the module where your subcomment is and you can see more on documentation later. It's saving things for checks. So if you want to check something with the precommit hook, you also give it a name that will be used on the configuration and you provide the module that is going to be used. So you have three entry points right now, one for commands, one for commit message hook, the checks commit message and the other for precommit hook that we use for checking code style, for example. So what did we learn from this process? First, don't, and this is the most important, don't get stuck in the past. It's very easy to make mistakes because it was always done like this. What was the best solution yesterday can be the wrong solution today and this will be mostly, will mostly likely be true if you change technologies because if you change technologies, you will have different limitations and different opportunities and you should rethink what you're doing and adapt to it. One other thing we learned was developing the open because by being more transparent about what we wanted, where we were going, we avoided assuming too much and we avoided backing ourselves into a corner and we got early feedback that also avoided lots of issues. We also learned that we should build tools, not barriers. Software is meant to help people and to be more productive. If your software is making people lose time instead of saving time, you should rethink it. By rethinking our GIT hooks, we were able to move from a position where sometimes our software was entering the productivity of our engineering teams to a position where they are useful tools to help them work autonomously. So, any questions? Thank you for the talk. My question is about the cost and the effort of any change. We're talking about your company changing. Can you speak louder? I cannot hear you. Sorry. No? Yes. Yes, thank you. I was asking about the effort of changing in each step because sometimes, okay, leaving subversion may be a good thing, but changing for centralized hooks to local hooks and everything, out the effort. Yes. Yes, it's an art process and we are still working on it. Sometimes, and actually, we are doing things where we are disabling checks on the remote hook in phases and telling people, now start using this and we are trying to make it compatible. So, I think that's not working in the same way on the remote GIT hooks and local GIT hooks specification because before, we only allowed people to use gyro tickets and now we are autonomous. We want to allow them to use whatever solution they want. So, now we support more than that. Fortunately, at Zalendo, we are already used to always question everything and it's one of our philosophy. If you are doing the same thing all the time, you should question yourself and see if there's no better way to do it. It's part of our corporate culture and that doesn't mean that people don't complain that we are changing again and that we have to manage it. But at least in relationship to this, most feedback I have is that people are happy to be able to control when they use the hooks, how they use it and they are very happy that they don't have to rely on me and my team to change the step for them. You can provide any samples of the kind of rules you are enforcing with that? Yes. So, for example, this is actually from one of our open source repositories. I don't remember exactly what. This is a very minimum set. We have more than that. But for example, we have the specification check. What it does right now is checking if your commit message starts with a URL and you have the option allowed schemes right now and I'm saying that I only accept HTTPS URLs because on this case, all specifications are done using GitHub issues. In the near future, I want also to support GitHub references so you can just put the text that I support by GitHub and also the Jira ticket IDs because we still use it at Solando and we wanted to make it easy for people to use it. We have the branch release check that in this case uses that regex expression on the bottom to check if a branch that starts with release slash something, the second part has too much that one. In this case, it has to start with a V and follow some rules that's format to use at Solando but you can use whatever you want and to protect master branch in this case forbids you to commit directly to master because I want to enforce pull requests. Any other questions? Okay, just one more thing. If you want to find more information about Solando, we have a tech blog and tech.solando.com. We have our GitHub page, GitHub.com slash Solando. We have lots of open source projects there and we are going to have much more in the future. We also have Twitter, Instagram account and a jobs page and one of my colleagues will give a presentation in the recruiting session later today. Thank you.
 So first to present myself, I'm Jun Santu.  I'm a software engineer at Salando as part of the continuous delivery and deployment team.  And today I'm going to present the part we took at Salando from the dark cases of subversion to the  enlightened age of autonomous work with EGIT.  So some quick facts about Salando.  So you understand the context of my work.  We are, we have our headquarters in Berlin.  We have also other offices in other places like Dublin, Helsinki, Dortmund,  airport.  We have more than 700 actually nowadays almost 1000 engineers.  And we do lots of every almost everything we do is done in house.  So we do lots of development.  Okay.  So the first question when talking about GITUX is why do we need GITUX in or subversion  groups or other kinds of commit so in the first place?  The answer to this is to enforce rules.  This raises the second question.  Why do we need rules?  Any large group organization needs rules.  Some rules are internal to the groups, some are external.  And they are very important to achieve an efficient workflow to ensure quality and transparency to  stakeholders. And in case of Salando, stakeholders go from the colleague at next desk to the  customer that as trusted as his credit card to the investor who spent lots of his money  and he trusted and believed in us.  Second question is how we enforce rules.  Not our rules are enforced in GITUX.  Some rules are enforced before deployments.  Some rules are only monitored are not enforced.  And so you have to look at your workflow and see what rules can be enforced a priori or with  checks and what rules can be checked after you have moved on your workflow.  And it's vital to keep a proper balance between two types of enforcement because if you have too many  blockers on your workflow, you prevent your developers from being productive.  But if you have to, few checks and blockers, you risk that bad software goes live and you break the  trust all stakeholders that are in you.  So now starting point.  So this is a version. This was the place where we were when I arrived at Salando two years ago.  We had one batch script with some basic checks.  A precommit took.  Then we were growing as a company and we needed more rules because it was not possible for someone to  keep track of everything that was happening.  It was not possible to keep a batch script anymore.  It was not efficient. It was not maintainable.  So we ported the GITUX subversion from Pesh to Python.  Around December 2013, we started moving from GIT because the subversion was not good enough for  guys anymore and we had to do something.  So we started moving to GIT.  We still had same rules in places before and we knew we would still keep subversion for some time  because we already had lots of teams and some teams wanted to move faster and others slower.  Not everyone wanted to move to GIT so we had to convince everyone and we decided to do migration in  waves.  So the first decision we took was to adapt the SVN commit took to work on both SVN and GIT.  So we implemented GITUX as a per-gC book because it's invoked at the same point on the workflow as  commit took on subversion.  This is when user sense changes to the remote server and because it allowed us to reject the reference  or branch without rejecting full push.  The other alternative would be an update took but in that one if you return an error code it will  block the full push even if most of it is following the rules.  Doing this had some problems.  This approach had some problems.  The first problem in this plan was that GIT is not subversion.  They are different in fundamental ways and trying to support both would mean support for both of them.  The other problem is that being distributed GIT is more flexible than subversion which means that people  use GIT in the same way they use subversion and don't use GIT in the same way that other people use GIT.  The first things that I learned to migrate to GIT start experimenting different workflows and this was a good  thing because they were very different projects and different projects have very different requirements.  So it makes sense for them to have a different set of best practices and slightly different rules and  different pricing strategy but this also forces us to rethink how we check the rules and how we implement  the rules and we had to rethink the GIT rules.  So we moved to plan B where before we had one common look for GIT and subversion.  Now we decided to have two different looks.  In practice this meant we forked to look into...the subversion commit look was aggregated so no more  changes there because we don't want to really support the subversion anymore and we continued to work on  to get priority look because different teams wanted to use different workflows.  So we moved away from one size fits all strategy and allowed teams to specify set rules for themselves.  So this is the way we allowed teams to set rules for themselves.  We have a configuration format based on YAML where Stash projects and repositories could mesh several  times by name. This is similar to CSS so for example you can see there that one team could have one  general rule for themselves for their project but for a specific repository they could change part of  the rules and not the whole set and a second team could have a completely different rule set because  they have different needs.  The problem with this second approach and this was a reflection we did after using the GIT looks for one year  was that we had a centralized configuration which became a bottleneck because it could only be changed  by a small number of people and sometimes we were all on vacation and it was really bad.  Another more technical problem we had is that we were trying to check all the commits on a push  because some rules apply to every single commit. For example we checked that one of the rules we had at  Zalendo was that every commit message has to have a ticket and for this we have to go through all  the commits and this was problematic for a number of reasons. For example GIT's history is not linear  so if the user merges or rebases from a branch and we try to check all the messages in the commit range  for ticket ID and see if it matches to a branch, a commit set came from other branches by the rebase  or the merge would fail the check. In the end we tried to several solutions, some worked way better  than others, in the end the solution for this was filtering out commits that were already in a branch  because this meant they were already checked before and were not originated on this branch.  The second problem we had is that when you merge to or from a long-lived branch, some teams had some  very long-lived projects and they had branches for it and sometimes they deferred from master for  two, three months and when they merged from master or to master or when they rebased they had  to check. Because of some of our checks, sometimes we had pushes that took more than half an hour to  check and this was not a good experience for our developers. Related to this, one of the slowest  checks we had is checking code style, especially for Java code because we use Jallopee to  validate code style and it runs on JVM and the architecture of our code validator meant we had to  spawn one JVM for every changed file. So if someone pushed 200 changed files we would spawn up 200  JVMs and other people were pushing at the same time and it was constantly freezing and slowing down  our server. We were able to mitigate this problem by using NailGant that is a kind of server that  keeps JVM alive and you can run JVM programs there and we could run Jallopee there but we had to  do some work around to run Jallopee in parallel and we still had some performance issues for some  reason there were more pushes than usual. The last problem of this is that the system was  inflexible. Sometimes people have very good reasons to ignore rules. Sometimes they pushed  something that they didn't want to push and they want to do a forced push and we didn't allow that.  Sometimes they, sometimes some push was accepted by mistake and it was a bag and it has to be removed  because it should not be there. Sometimes there's a bag and we reject it. Something that we should  have accepted and because this was on our server and people didn't have access to the server, they  had no way to go around this. The other way we were inflexible is that because it's a remote  Github and it's living on the server, we only supported our internal Github server. For example,  you could not use our Github for our projects on github.com and so our open source projects were  completely unsupported. In the end, I could summarize all these problems as we tried to  centralize Github. While Github is distributed, we still saw our source control management system  as centralized because we came from subversion and we were still thinking like subversion. But it  was clear that we would have to rethink our approach. This process of thinking about the  Github's coincided with the use change that's the ZLANDU as part of something we call ready  calligility. ZLANDU was getting rid of all the error keys and giving teams more autonomy and one of  the new motors was autonomy instead of control. And this presented a clear path for us and in the  end was the solution that we are still implementing. So we decided to move away from remote  Github's to a set of local Github's. One advantage of this is that Github's see one commit at a time.  So we avoided the issues where we had a set, a list of commits to check and we didn't know what came  from a merge or a rebase and what was created in the branch. And it's better adjusted to the  distributed nature of Github because they are distributed by themselves and people run it on  their local machines. We decided that they should be optional because we don't trust people. This  also means that Github's are no longer responsible for enforcing the rules. Instead, we are creating  mechanisms to ensure that a level of development follows the rules before the code goes live but we  don't block anyone from pushing anything right now. We did this because we believe Github's should be  seen as a tool to help autonomous developers and not as a barrier that makes their work harder. We  decided to make them extensible because as I said before, one size doesn't fit all different teams  have different needs and sometimes we only have a small group of developers working on this more or less full time  but sometimes people have other ideas and they can implement new stuff for themselves. Sometimes they don't  want one of our checks and they don't want to install the dependencies of our checks so by being  extensible, they can opt out of those dependencies. And we decided to make it open source because by  only being and thinking open source, it could fully support autonomous teams and this also allowed us to  support our teams that want to open source their code and this also includes my team. So our new set of  hooks that we call turnstile are available on Github and by PI so everyone is free to use them. So if you  want to install it on your machine, you can just use pip install turnstile core. This will just install  core without any of our extensions. We also have some extensions on Zalendo's Github account. So now you  want to use it on your repository. What do you have to use? Well, the first step is add a turnstile .eml file in the root of your repository. If you use something like Travis or something similar, the same  process you just added a file to your repository and it looks something like this. You have a list of  checks you want to use. So you just name them and you can give options to all checks that they support  by themselves. And every check, even if you are implementing your own checks as an extension, you can  read this file and you have access to it. So and because the way local Github works, you need to  see link to on your.kit folder inside the repository and to make this easy, turnstile supports a  subcomment that's turnstile installed that will automatically see link to looks for you. And because  I was already adding support for subcomments, I added several more. We have to config subcomments that  right now the only thing it does is set, allow you to set verbosity. So if you want only to see errors,  you can only see the errors. If you want to see everything that turnstile is doing behind since you  can also do it, you have to install command to add to looks to a specific repository. We have to remove  if you decide to, you don't want to looks anymore for the repository. The specification subcomment is  used for check if all your projects contain a valid specification of all your commits contain a  valid specification. And right now this means if they start with a new URL, but I'm planning to support  more stuff in the near future. The upgrades of command will check by PI to see if turnstile and all  your extensions are updated and if they are not, it will offer to update everything for you. And  version just brings the version and it's not that interesting. So now if you want to create extensions  to turnstile, how can you do it? The way you do it is with set of tools and entry points. So you  just have, if you want to add a command, you just go to the end, use the entry point turnstile.comments.  You give the name of your subcomment and this will be used to call it from the command line and you  just provide the module where your subcomment is and you can see more on documentation later. It's  saving things for checks. So if you want to check something with the precommit hook, you also give  it a name that will be used on the configuration and you provide the module that is going to be used.  So you have three entry points right now, one for commands, one for commit message hook, the  checks commit message and the other for precommit hook that we use for checking code style, for  example. So what did we learn from this process? First, don't, and this is the most important,  don't get stuck in the past. It's very easy to make mistakes because it was always done like this.  What was the best solution yesterday can be the wrong solution today and this will be mostly,  will mostly likely be true if you change technologies because if you change technologies, you will  have different limitations and different opportunities and you should rethink what you're doing  and adapt to it. One other thing we learned was developing the open because by being more  transparent about what we wanted, where we were going, we avoided assuming too much and we avoided  backing ourselves into a corner and we got early feedback that also avoided lots of issues.  We also learned that we should build tools, not barriers. Software is meant to help people and  to be more productive. If your software is making people lose time instead of saving time, you  should rethink it. By rethinking our GIT hooks, we were able to move from a position where  sometimes our software was entering the productivity of our engineering teams to a position where  they are useful tools to help them work autonomously. So, any questions?  Thank you for the talk. My question is about the cost and the effort of any change.  We're talking about your company changing.  Can you speak louder? I cannot hear you.  Sorry. No?  Yes.  Yes, thank you. I was asking about the effort of changing in each step because sometimes,  okay, leaving subversion may be a good thing, but changing for centralized hooks to local  hooks and everything, out the effort.  Yes. Yes, it's an art process and we are still working on it.  Sometimes, and actually, we are doing things where we are disabling checks on the remote  hook in phases and telling people, now start using this and we are trying to make it compatible.  So, I think that's not working in the same way on the remote GIT hooks and local GIT hooks  specification because before, we only allowed people to use gyro tickets and now we are  autonomous. We want to allow them to use whatever solution they want. So, now we support more  than that. Fortunately, at Zalendo, we are already used to always question everything  and it's one of our philosophy. If you are doing the same thing all the time, you should  question yourself and see if there's no better way to do it.  It's part of our corporate culture and that doesn't mean that people don't complain that  we are changing again and that we have to manage it. But at least in relationship to  this, most feedback I have is that people are happy to be able to control when they use  the hooks, how they use it and they are very happy that they don't have to rely on me and  my team to change the step for them.  You can provide any samples of the kind of rules you are enforcing with that?  Yes. So, for example, this is actually from one of our open source repositories. I don't  remember exactly what. This is a very minimum set. We have more than that. But for example,  we have the specification check. What it does right now is checking if your commit message  starts with a URL and you have the option allowed schemes right now and I'm saying that I only  accept HTTPS URLs because on this case, all specifications are done using GitHub issues.  In the near future, I want also to support GitHub references so you can just put the  text that I support by GitHub and also the Jira ticket IDs because we still use it at  Solando and we wanted to make it easy for people to use it. We have the branch release  check that in this case uses that regex expression on the bottom to check if a branch that starts  with release slash something, the second part has too much that one. In this case, it has  to start with a V and follow some rules that's format to use at Solando but you can use  whatever you want and to protect master branch in this case forbids you to commit directly  to master because I want to enforce pull requests.  Any other questions?  Okay, just one more thing. If you want to find more information about Solando, we have a  tech blog and tech.solando.com. We have our GitHub page, GitHub.com slash Solando. We have  lots of open source projects there and we are going to have much more in the future. We  also have Twitter, Instagram account and a jobs page and one of my colleagues will give  a presentation in the recruiting session later today.  Thank you.
Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_0_1259.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_1260_4811.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_4812_4979.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_4980_5879.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_5880_6863.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_6864_9143.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_9144_9359.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_9360_11267.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_11268_17807.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_17808_22703.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_22704_25511.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_25512_27395.png Joao_Santos_-_Using_Git_Hooks_to_Help_Your_Engineering_Teams_Work_Autonomously_27396_38124.png
So first to present myself, I'm Jun Santu. I'm a software engineer at Salando as part of the continuous delivery and deployment team. And today I'm going to present the part we took at Salando from the dark cases of subversion to the enlightened age of autonomous work with EGIT. So some quick facts about Salando. So you understand the context of my work. We are, we have our headquarters in Berlin. We have also other offices in other places like Dublin, Helsinki, Dortmund, airport. We have more than 700 actually nowadays almost 1000 engineers. And we do lots of every almost everything we do is done in house. So we do lots of development. Okay. So the first question when talking about GITUX is why do we need GITUX in or subversion groups or other kinds of commit so in the first place? The answer to this is to enforce rules. This raises the second question. Why do we need rules? Any large group organization needs rules. Some rules are internal to the groups, some are external. And they are very important to achieve an efficient workflow to ensure quality and transparency to stakeholders. And in case of Salando, stakeholders go from the colleague at next desk to the customer that as trusted as his credit card to the investor who spent lots of his money and he trusted and believed in us. Second question is how we enforce rules. Not our rules are enforced in GITUX. Some rules are enforced before deployments. Some rules are only monitored are not enforced. And so you have to look at your workflow and see what rules can be enforced a priori or with checks and what rules can be checked after you have moved on your workflow. And it's vital to keep a proper balance between two types of enforcement because if you have too many blockers on your workflow, you prevent your developers from being productive. But if you have to, few checks and blockers, you risk that bad software goes live and you break the trust all stakeholders that are in you. So now starting point. So this is a version. This was the place where we were when I arrived at Salando two years ago. We had one batch script with some basic checks. A precommit took. Then we were growing as a company and we needed more rules because it was not possible for someone to keep track of everything that was happening. It was not possible to keep a batch script anymore. It was not efficient. It was not maintainable. So we ported the GITUX subversion from Pesh to Python. Around December 2013, we started moving from GIT because the subversion was not good enough for guys anymore and we had to do something. So we started moving to GIT. We still had same rules in places before and we knew we would still keep subversion for some time because we already had lots of teams and some teams wanted to move faster and others slower. Not everyone wanted to move to GIT so we had to convince everyone and we decided to do migration in waves. So the first decision we took was to adapt the SVN commit took to work on both SVN and GIT. So we implemented GITUX as a per-gC book because it's invoked at the same point on the workflow as commit took on subversion. This is when user sense changes to the remote server and because it allowed us to reject the reference or branch without rejecting full push. The other alternative would be an update took but in that one if you return an error code it will block the full push even if most of it is following the rules. Doing this had some problems. This approach had some problems. The first problem in this plan was that GIT is not subversion. They are different in fundamental ways and trying to support both would mean support for both of them. The other problem is that being distributed GIT is more flexible than subversion which means that people use GIT in the same way they use subversion and don't use GIT in the same way that other people use GIT. The first things that I learned to migrate to GIT start experimenting different workflows and this was a good thing because they were very different projects and different projects have very different requirements. So it makes sense for them to have a different set of best practices and slightly different rules and different pricing strategy but this also forces us to rethink how we check the rules and how we implement the rules and we had to rethink the GIT rules. So we moved to plan B where before we had one common look for GIT and subversion. Now we decided to have two different looks. In practice this meant we forked to look into...the subversion commit look was aggregated so no more changes there because we don't want to really support the subversion anymore and we continued to work on to get priority look because different teams wanted to use different workflows. So we moved away from one size fits all strategy and allowed teams to specify set rules for themselves. So this is the way we allowed teams to set rules for themselves. We have a configuration format based on YAML where Stash projects and repositories could mesh several times by name. This is similar to CSS so for example you can see there that one team could have one general rule for themselves for their project but for a specific repository they could change part of the rules and not the whole set and a second team could have a completely different rule set because they have different needs. The problem with this second approach and this was a reflection we did after using the GIT looks for one year was that we had a centralized configuration which became a bottleneck because it could only be changed by a small number of people and sometimes we were all on vacation and it was really bad. Another more technical problem we had is that we were trying to check all the commits on a push because some rules apply to every single commit. For example we checked that one of the rules we had at Zalendo was that every commit message has to have a ticket and for this we have to go through all the commits and this was problematic for a number of reasons. For example GIT's history is not linear so if the user merges or rebases from a branch and we try to check all the messages in the commit range for ticket ID and see if it matches to a branch, a commit set came from other branches by the rebase or the merge would fail the check. In the end we tried to several solutions, some worked way better than others, in the end the solution for this was filtering out commits that were already in a branch because this meant they were already checked before and were not originated on this branch. The second problem we had is that when you merge to or from a long-lived branch, some teams had some very long-lived projects and they had branches for it and sometimes they deferred from master for two, three months and when they merged from master or to master or when they rebased they had to check. Because of some of our checks, sometimes we had pushes that took more than half an hour to check and this was not a good experience for our developers. Related to this, one of the slowest checks we had is checking code style, especially for Java code because we use Jallopee to validate code style and it runs on JVM and the architecture of our code validator meant we had to spawn one JVM for every changed file. So if someone pushed 200 changed files we would spawn up 200 JVMs and other people were pushing at the same time and it was constantly freezing and slowing down our server. We were able to mitigate this problem by using NailGant that is a kind of server that keeps JVM alive and you can run JVM programs there and we could run Jallopee there but we had to do some work around to run Jallopee in parallel and we still had some performance issues for some reason there were more pushes than usual. The last problem of this is that the system was inflexible. Sometimes people have very good reasons to ignore rules. Sometimes they pushed something that they didn't want to push and they want to do a forced push and we didn't allow that. Sometimes they, sometimes some push was accepted by mistake and it was a bag and it has to be removed because it should not be there. Sometimes there's a bag and we reject it. Something that we should have accepted and because this was on our server and people didn't have access to the server, they had no way to go around this. The other way we were inflexible is that because it's a remote Github and it's living on the server, we only supported our internal Github server. For example, you could not use our Github for our projects on github.com and so our open source projects were completely unsupported. In the end, I could summarize all these problems as we tried to centralize Github. While Github is distributed, we still saw our source control management system as centralized because we came from subversion and we were still thinking like subversion. But it was clear that we would have to rethink our approach. This process of thinking about the Github's coincided with the use change that's the ZLANDU as part of something we call ready calligility. ZLANDU was getting rid of all the error keys and giving teams more autonomy and one of the new motors was autonomy instead of control. And this presented a clear path for us and in the end was the solution that we are still implementing. So we decided to move away from remote Github's to a set of local Github's. One advantage of this is that Github's see one commit at a time. So we avoided the issues where we had a set, a list of commits to check and we didn't know what came from a merge or a rebase and what was created in the branch. And it's better adjusted to the distributed nature of Github because they are distributed by themselves and people run it on their local machines. We decided that they should be optional because we don't trust people. This also means that Github's are no longer responsible for enforcing the rules. Instead, we are creating mechanisms to ensure that a level of development follows the rules before the code goes live but we don't block anyone from pushing anything right now. We did this because we believe Github's should be seen as a tool to help autonomous developers and not as a barrier that makes their work harder. We decided to make them extensible because as I said before, one size doesn't fit all different teams have different needs and sometimes we only have a small group of developers working on this more or less full time but sometimes people have other ideas and they can implement new stuff for themselves. Sometimes they don't want one of our checks and they don't want to install the dependencies of our checks so by being extensible, they can opt out of those dependencies. And we decided to make it open source because by only being and thinking open source, it could fully support autonomous teams and this also allowed us to support our teams that want to open source their code and this also includes my team. So our new set of hooks that we call turnstile are available on Github and by PI so everyone is free to use them. So if you want to install it on your machine, you can just use pip install turnstile core. This will just install core without any of our extensions. We also have some extensions on Zalendo's Github account. So now you want to use it on your repository. What do you have to use? Well, the first step is add a turnstile.eml file in the root of your repository. If you use something like Travis or something similar, the same process you just added a file to your repository and it looks something like this. You have a list of checks you want to use. So you just name them and you can give options to all checks that they support by themselves. And every check, even if you are implementing your own checks as an extension, you can read this file and you have access to it. So and because the way local Github works, you need to see link to on your.kit folder inside the repository and to make this easy, turnstile supports a subcomment that's turnstile installed that will automatically see link to looks for you. And because I was already adding support for subcomments, I added several more. We have to config subcomments that right now the only thing it does is set, allow you to set verbosity. So if you want only to see errors, you can only see the errors. If you want to see everything that turnstile is doing behind since you can also do it, you have to install command to add to looks to a specific repository. We have to remove if you decide to, you don't want to looks anymore for the repository. The specification subcomment is used for check if all your projects contain a valid specification of all your commits contain a valid specification. And right now this means if they start with a new URL, but I'm planning to support more stuff in the near future. The upgrades of command will check by PI to see if turnstile and all your extensions are updated and if they are not, it will offer to update everything for you. And version just brings the version and it's not that interesting. So now if you want to create extensions to turnstile, how can you do it? The way you do it is with set of tools and entry points. So you just have, if you want to add a command, you just go to the end, use the entry point turnstile.comments. You give the name of your subcomment and this will be used to call it from the command line and you just provide the module where your subcomment is and you can see more on documentation later. It's saving things for checks. So if you want to check something with the precommit hook, you also give it a name that will be used on the configuration and you provide the module that is going to be used. So you have three entry points right now, one for commands, one for commit message hook, the checks commit message and the other for precommit hook that we use for checking code style, for example. So what did we learn from this process? First, don't, and this is the most important, don't get stuck in the past. It's very easy to make mistakes because it was always done like this. What was the best solution yesterday can be the wrong solution today and this will be mostly, will mostly likely be true if you change technologies because if you change technologies, you will have different limitations and different opportunities and you should rethink what you're doing and adapt to it. One other thing we learned was developing the open because by being more transparent about what we wanted, where we were going, we avoided assuming too much and we avoided backing ourselves into a corner and we got early feedback that also avoided lots of issues. We also learned that we should build tools, not barriers. Software is meant to help people and to be more productive. If your software is making people lose time instead of saving time, you should rethink it. By rethinking our GIT hooks, we were able to move from a position where sometimes our software was entering the productivity of our engineering teams to a position where they are useful tools to help them work autonomously. So, any questions? Thank you for the talk. My question is about the cost and the effort of any change. We're talking about your company changing. Can you speak louder? I cannot hear you. Sorry. No? Yes. Yes, thank you. I was asking about the effort of changing in each step because sometimes, okay, leaving subversion may be a good thing, but changing for centralized hooks to local hooks and everything, out the effort. Yes. Yes, it's an art process and we are still working on it. Sometimes, and actually, we are doing things where we are disabling checks on the remote hook in phases and telling people, now start using this and we are trying to make it compatible. So, I think that's not working in the same way on the remote GIT hooks and local GIT hooks specification because before, we only allowed people to use gyro tickets and now we are autonomous. We want to allow them to use whatever solution they want. So, now we support more than that. Fortunately, at Zalendo, we are already used to always question everything and it's one of our philosophy. If you are doing the same thing all the time, you should question yourself and see if there's no better way to do it. It's part of our corporate culture and that doesn't mean that people don't complain that we are changing again and that we have to manage it. But at least in relationship to this, most feedback I have is that people are happy to be able to control when they use the hooks, how they use it and they are very happy that they don't have to rely on me and my team to change the step for them. You can provide any samples of the kind of rules you are enforcing with that? Yes. So, for example, this is actually from one of our open source repositories. I don't remember exactly what. This is a very minimum set. We have more than that. But for example, we have the specification check. What it does right now is checking if your commit message starts with a URL and you have the option allowed schemes right now and I'm saying that I only accept HTTPS URLs because on this case, all specifications are done using GitHub issues. In the near future, I want also to support GitHub references so you can just put the text that I support by GitHub and also the Jira ticket IDs because we still use it at Solando and we wanted to make it easy for people to use it. We have the branch release check that in this case uses that regex expression on the bottom to check if a branch that starts with release slash something, the second part has too much that one. In this case, it has to start with a V and follow some rules that's format to use at Solando but you can use whatever you want and to protect master branch in this case forbids you to commit directly to master because I want to enforce pull requests. Any other questions? Okay, just one more thing. If you want to find more information about Solando, we have a tech blog and tech.solando.com. We have our GitHub page, GitHub.com slash Solando. We have lots of open source projects there and we are going to have much more in the future. We also have Twitter, Instagram account and a jobs page and one of my colleagues will give a presentation in the recruiting session later today. Thank you.
question when talking about GITUX is why do we to ensure quality and transparency to ported the GITUX subversion from Pesh to Python. the same point on the workflow as commit took on subversion. This is when user sense changes to to set rules for themselves. We have a commit message has to have a ticket and for this commit set came from other branches by the rebase one of the new motors was autonomy instead of remote Github's to a set of local Github's. One was created in the branch. And it's better that want to open source their code and this also you are implementing your own checks as an will offer to update everything for you. And interesting. So now if you want to create extensions to turnstile, how can you do it? The your subcomment and this will be used to call it commit message and the other for precommit hook to help them work autonomously. So, any local hooks and everything, out the effort. Yes. Yes, it's an art process and we are still working much that one. In this case, it has to start with colleagues will give a presentation in the