Who watches the watchers? A Jenkins journey
Martinez, Victor
Calvo, Ivan Fernandez
When Everything as Code converges to automate/test your processes, in this talk we would like to discuss further our journey and our vision to handle our automation programmatically. When we started to use Jenkins at Elastic, there was already a team providing the service, similar to the SaaS you could see nowadays, so teams were encouraged to handle their automation using the Configuration as Code paradigm with the Jenkins Job Builder tool, but at a certain moment of time this particular approach didn’t scale much, and that’s when we started to think about a more robust, testable and automated CI/CD ecosystem. Then our journey started by applying the below principles and practices: - Everything as Code - Everything is Tested - Documentation as Code - Functional testing - Continuous Improvement - Continuous Deployment All the above concepts are the ones that helped us in this journey and that’s the reason for this talk to share our experience and vision. Besides gathering from you your opinions and feedback.
 Good morning, good afternoon, good evening. This is Who Watches the Watches, I am Yengi Juni, my name is Ivan Fernandez. I am a software engineer at Elastic for two years. I am a member of the productivity team inside the OXORA Leading Team and I was working with CIICD with Jenkins for 10 years or so. I maintain a couple of plugins of Jenkins also. My name is Victor Martinez. I am also a software engineer at Elastic. I work with Ivan in the same team and I've been using any things related with CIICD similar to Jenkins and other tools for over 10 years. We are here to talk a little bit as well with the company. So our company is Elastic. It's an open source company and contributes by different meanings to the community, to the open source community. In this particular case, we would like to highlight the way we use Jenkins for CIICD is by using the H versions of the weekly releases and also the latest plugin versions of the plugin that we need. This leads us to contribute back to the community when we detect any issues in any particular plugin or weekly releases. It doesn't happen often and when it does, either we raise a particular issue or Jira or whatever tracking system, they want that particular project with what we found and then we just also help them to develop features and other things. Why? Because we like to be on the edge and get the benefit of the latest version and this contributes back in the way like it gives more maturity in the quality of the Jenkins ecosystem. Okay, let's start explaining a little bit what is vulnerability. Surveillance or vulnerability or O11Y. Try to join your application logs, your metrics and your application monitoring data in one single point. In our case, elastic search. To show that data in dashboards or anything similar, Kibana for example, and analyze that data we are much in learning and other techniques to detect misbehavior and generate a life as related with that misbehavior. In elastic, in elastic, we use elastic search, Kibana and bits. But what is elastic search, Kibana and bits? Elastic search is a distributed open source, search and analyze and generate. For all types of data, Kibana provides search and data visualization and bits provide data ingestion from many sources of that. For example, logs with five bits, metrics with metric bits, monitoring with hard bits, and cloud data with functional bits, network traffic with packet bits, Windows 7 with long bits, and application performer monitoring with APM that collects performance information about responding times and or incoming requests, database queries, call to the catchees, external agit bit request, and so on. All that data, we analyze with machine learning to generate alerts and show the data in dashboards to try to find misbehavior in that data. But the question here is, we can use the same approach that we use for applications in the CI CD. We can try to send data to elastic search and the misbehavior in the CI CD. That's what we want to watch and we want to do in elastic. So after this introduction, let's talk about what's the problem statement, what we're trying to solve. Let's talk about precisely what we do or the complexity of it. So a little bit of context. The projects that we maintain is about 18 different GitHub repositories. There are different programming languages. We just say 10 to be roughly that or even more. Every programming language might have a different build tool system, or even any or none. There is also a different package, a distribution we need to provide. In order to provide this, there are different versions of the operating system that have to be supported. And different architectures. Again, different iterations. So, so probably as we mentioned earlier, helps you to detect and to observe how your system interacts with databases, front-end or cloud provider. So that's why we need to support different iterations. So this is the entire ecosystem that we need to maintain. From the continuous integration and continuous delivery or deployment, we need to basically be sure like how we can go complete that particular aim of having a high quality in what we deliver. So then we're going to discuss a little bit further. What are the solutions to solve this particular problem statement? Okay. First of all, we have to set a little number of rules before to start to achieve the problem statement. We have to try to standardize some things. Not all, but a little bit of things. So the first thing that we have to standardize is the tool that we use across all our projects. In our case, we decide to use Jenkins. The second thing that we have to standardize is how we want to build the workflows related to other pipelines related with the projects. In our case, we choose the clarifying pipeline because it's an opinionated way to define our flow that keeps in the pipeline. Only the orchestration logic, so it's simple. We don't want to end with a weird logic in those pipelines. So this is the best approach in our opinion. For the jobs, we use multivariance pipeline because it allows us to have only one job per repository. Because the multivariance pipeline allows us to manage tasks and per request in only one job. All the things related with every branch, pipeline, task and branch is created automatically. For the tools that we use to build the projects, we try to delegate as much as possible in build tools, maybe in gradle, whatever tool we have in the programming language that we have. That allows us to have the same workflow locally. So we can reproduce the same things that we do in the HCI. We can do exactly the same in local machines. The last thing is to try to have for all the projects a single release with one click. So we click in one place and we generate the binaries of that. So another aspect in the solution is don't repeat yourself. This is an extension of the standardization. When you really work with many different projects, you are probably repeating certain common patterns such as a checkout of the source code, the different notifications to the different GitHub or different SCM tools that you use, how to generate secrets from both on a similar tooling, how to compress a file, how to generate the test output and trend the output to be able to visualize it easily in the UI or install certain tools that you need. So by identifying this common process in the different projects and have this particular definition somewhere else, it helps us as well to be able to provide the same framework to a bunch of different projects. So the idea is to convert these particular processes in pipeline steps. In this case, it's quite precisely about how we use Jenkins and the clarity pipeline. Put all these steps in a cell library. In our case, we use the Jenkins cell library. This will help us to simplify the workflows easily and therefore people can contribute to them rather than having a really messy scripting logic. In our case, the cell library that we use is about 150 different steps to integrate with Vault or to integrate with GitHub or AWS, Google Cloud or the kind of cloud provider we use. So it's quite, yeah, it's there. So you can have a look later. So we want to have every theme that we have in our environments and all SCICD in a way that we can repeat things. So we can repeat the same process and obtain the same workflows. To do that, we try to achieve this but putting everything in code. Our infrastructure or jobs, panel lines, definitions, test environments and in general, everything that we can define as code is defined as code. So our infra team provides a standard way to provision in Jenkins systems that is defined as code in Ansible. Our items, the same. We have the same process in Ansible to provisioning and generate the image that we use to launch those items is defined as code in Ansible. Our job definitions, we use a Jenkins job builder and job DSL to define our jobs and generate the jobs from that code. For our local environments, we use a Docker and Jenkins configuration as code to launch local instance that can run in Bangram definitions. We can reproduce the same behaviors that we have in the SCICD locally for local testing. We also provide a terminal environment for integration and test that we use in our jobs that are defined as code. Our permanent environments for testing and dog fooding also are defined as code. We have documentation that uses Ansible and other things to deploy environments in a repeatable manner. And also the dogs. Everything that we have in documentation is generated from code. So test them all is part of the next solution. In order to do that, we really need everything as code. That's what we talked about earlier. What do we do with this particular testing? Testing should happen before all the different systems we use. One of them is the Jenkins set library. This is where we delegate all the responsibility for certain complex logic to be easy to reach and use by the consumer. In this case, the pipelines or workflows that we define in every single repository. So the logic can be used from different, it can be consumed by different repositories or pipelines or workflows. It's going to be defined there. So the way we test this is using unit testing and functional testing. With this, we can be sure that any release we do of this particular pipeline, so library, Jenkins library, can be consumed with the certainty that it will work. It's always more compatible. And the library itself is tested itself with itself. So it sounds a bit redundant. But the idea is we have a process of branches and tags that help us to be sure any upcoming release is tested beforehand to avoid any kind of surprises. Regarding the infrastructure, we were talking like if we use Antibole, we use Docker, Parker images, all of them they are tested before creating new virtual machines or Docker containers, sorry, Docker images. We also test the Ansible deployment and the Helm charts. The other thing that is important to say is about how we can ensure certain validation before committing them in any Git repository. Some build system they provided because it's built in the build system itself. Others, they don't have that particular maturity. And for such, we use pre-commit in the Git repository. Pre-commit is one of the tools that you can use. You basically define certain validations before allowing people to commit that in their repo. It's not just for local development, it's also something we use in our CI CD. So it's a mandatory sanity check. Other things that are important is that how you test the functionality, but also to scan the security as well. For such, we use sync, false output, to detect any kind of security issues. Third-party libraries dependences that can be compromised also to apply these practices. Communication. Communication is trying to send the right information to the right people. We try to facilitate the communication regarding the CI CD status only on one place. Why? To avoid the communication overflow. Such, send to my emails, your ERC message, issues, and we have checks. Also, help people that is not familiar with the CI CD, not all the people know how to share something in Jenkins. We decided to provide this, the gested information about build, test, deployment status, and other highlights in the Github PRs in no case. The benefits of that, people don't need to access to Jenkins log by a file, but in Blue Oaks and contains specific laws if need more details about the issue. They have the laws in Github. In the next slide, I will provide an example of how we digest that information in Github pull requests. We send a message to the Github repository, and in that message, we have the links to the Jenkins UI and the gested information related with the failure. For example, in a PR, we go to the PR, we will show the message of a build that was successful, and we show that the build is successful. We can access to the pylite test and another link to the Jenkins UI. We can show the number of tests that we launched and the results of those tests. And also, we can check those pull requests that are failed. In those cases, we have a similar message, but with different information. Let's see. So, if we go to a PR that fails, we show that the build failed. And we have the test errors, and we can navigate to the log. We can show a brief of the log. And also, we can show the steps of fail, and we can navigate to the exact log of that fail. And there we have the error. So, people that don't know anything about the CI CD tool can work in the same way that people that know how Jenkins or other tools work. That's the goal. Another important part of the solution we provide is the centralized knowledge. To observe the status of the CI CD in different ways, we ingest all the key assets for any build, test deployment to elastic search, and use Kibana to visualize and get a better understanding of what's going on. Some of the samples could be to centralize the task for, in a way like all the data is in one single place, as it's digested, and then the information will be easy to reach by anyone. I'm going to show you different kind of real life examples of how we do the CI analysis. This particular case, I want to show you, like, we do have a centralized dashboard in a markdown, and we navigate there. We click on that particular test that it failed, and from there, we can access exactly what is the build in Jenkins that fail, have a more deeper understanding of the logs, and so on. Again, this is the view. Just click on the dashboard, and in the dashboard, you will be able to see another view of how it looks like, the builds, and so on for a particular branch or a particular project, and go to the failure test, and link from there to the build in Jenkins that fail. Another thing that we can do is have other flagging, right? In this example, we do have an issue with the label CI reported, and from there, we can see that there has been a flagging test. We're clicking one of these GitHub issues from that particular project. We're going to see certain information that has been generated automatically, such as the name of the test, such as the build that it failed, such as the PR that caused that issue on the commit, certain error details, the stack trace for that particular test failure, and a link as well to the PR that caused that particular failure. This issue happens eight days ago, so we navigate to the build comment that we mentioned earlier. From the communication, you will see a flaggy test report message. We go to the particular history of eight days ago, we will see the flaggy test report detected at that time, two test failures that were related to probably some flakiness. Just to clarify, the way we tag a particular test failure as flaky is something we delegate to elastic search. There is another process in place that all the DST test failures are analyzed, and if they happen with certain, with an algorithm, is trying to see the previous build history, then tag them with a certain likelihood that they are failure as flaky test. It's going to be that information. The CI itself, after running any test, will evaluate any matches, and then the information will be always in GitHub, as I said, GitHub issue. This is the idea of centralizing the knowledge, to just in one place to understand when the build failed for any test, if they are flaky, and then start a conversation between the different people that they own the project in GitHub. Okay, monitoring demo. What we are trying to achieve is to have in science our CI CD and the test problems. So to do that, we ingest all the data related with the status of our CI CD. We have monitoring data related to the health of the CI CD environments, data related with laws, metrics, and everything is in our search and everything is ingested with our tools. So we are doing the food in for free about our products. What information we did just is exactly. We send monitoring data about the CI workers, all the Jenkins instance, reverse proxies, trip persistence, like GitHub, and this kind of stuff. We pull in the logs for the Jenkins, for the builds, and we also monitor the status of the other branches. We have a lot of data when something bad happens. We can go to the, with the, to Kibana and search the data, and localize the error, and try to resolve it. We go to the worker that is failing, because there is not enough memory or whatever, and we are solving all that. Okay. Let's see an example of the data that we collect, and how we manage to do things with that. For example, this is data related with the reverse proxy that we have for the Jenkins instance. We collect a lot of data, and on that data, for example, we can filter for a region or a country, and we can solve the request that we have from that country. In this case, we solve some patterns of some requests, and for example, let's go to the 404 request and see what happens. We can be some kind of a tag or something like that, and we can detect it more in a way. We saw that there is a bunch of requests that fail requesting something, and we can check the request, the kind of request that we have, and we can run the scans of the CIC to try to find some security issues or whatever. So it's an example of how we use that. Another aspect in this particular set of solutions we are talking about is the continuous deployment. We don't have the continuous integration of the foundation there. In some cases, we do continuous delivery, and now we are going to talk about the next aspect, how we deploy continuously for some of the projects. For us, every successful merge to any of the principal branches will generate binaries. This allows us to have a first snapshot to be able to ready to be deployed and test in some of the environments we have. As we talked earlier, in the AgriFina code, part of the solution we provide, we are able to generate a femoral and permanent environment for testing and development. These permanent environments are massive. They are used for demos, real testing or real testing in the sense of functional testing or manual testing, and help developers to have the closer version with massive data. One of the examples is about the data that is generated, and some of these, the environment is about four terabytes of data, and they are created in the same manner of any production environment. Our release cycle is based on a timeline, so there is a feature freeze that happens, and for that particular period of time, there are certain testing done, and we produce release build candidates, and anytime there is a new build candidate or snapshot, it is updated automatically in these environments. We also have a specific edge environment. This edge environment is matching the principal branch, basically the main branch, or most of the things, what's the terminology you use. The idea with this is to be able to test any changes, even though they might break, we will detect things faster, because incompatibility or any kind of other dependencies. The idea is not just to be able to detect this, but also dog food or elastic cloud platform, which is a SaaS service. This give value not only for our team, but also for other teams. This is a way we can post the collaboration with all the teams as well. In the next slide, the one that we are seeing now, is just giving you the way we not just use our product itself, or just validating the work as expected, but it's also how we dog food. So we can see the monitoring, the uptime, the logs of all the applications that are running in this particular demo environment or edge environment. This is pretty much an example of, yes, we are doing continuous deployment and continuously verifying that it behaves as expected. Cance engineering is about cancing incidents in producing environments, and see how the system is resilient to that incident. In our case, our production environments are over the environments that we talked in the previous slides. We are interested to detect those incidents are reported. So for us, caching engineering is another source of data for tuning all our tools. So we generate random incidents, and we test specific scenarios, and we analyze the data, and we try to detect those incidents that we define as experiments. So future, we just finished to talk about the solution we put in place to apply CI-CD, but there are still different areas that we would like to improve, or just today, get some kind of that. So we highlighted three major topics. The very first one is using much of learning to analyze why there was a failure. It could be related to tests, it could be related to an environmental issue, or a genuine one. So based on this, it will help us to optimize the resources that we use for building testing or deploying, and it will also help us to have some self-healing. So the system itself will be able to retry a number of times if there are certain flakiness, or if there was a genuine error or environmental issue. Other things is, as we talk about in one of the solutions about communication, what we do have is at the moment, is a kit-hop request comment with the status of the bill, deployment, or test. We like to leverage these to have more information that has been digested even more. It will help us to avoid any UI interactions. The sooner we get there, the better for everyone, because we will help the onboarding to anyone with any of the projects we have. So this is our aim, to don't have any UI interaction. And the last thing is about to standardize everything we have as a solution for the CI. We're trying to move further with the way we use Jenkins. There are examples of certain initiatives about monitoring, which is open telemetry as a standard or convention. It's one of the areas we are also dedicating and investing some time to work on that. So that's all. Thank you. You have a care, a couple of things. One is about the pipeline library that we use in Jenkins. You have the limitation there, and you can find what we do with Jenkins. It's open source. You can use it. And now, so a block of elastic search with all the details about the products that we use and how to use it. Thank you. If you have some questions, there is five minutes for questions. So write in the chat and we will answer your questions. Thank you. Thank you.
 Good morning, good afternoon, good evening. This is Who Watches the Watches, I am Yengi  Juni, my name is Ivan Fernandez. I am a software engineer at Elastic for two years. I am a  member of the productivity team inside the OXORA Leading Team and I was working with  CIICD with Jenkins for 10 years or so. I maintain a couple of plugins of Jenkins also.  My name is Victor Martinez. I am also a software engineer at Elastic. I work with Ivan in the  same team and I've been using any things related with CIICD similar to Jenkins and other tools  for over 10 years. We are here to talk a little bit as well with the company. So our company is  Elastic. It's an open source company and contributes by different meanings to the community, to the  open source community. In this particular case, we would like to highlight the way we use Jenkins  for CIICD is by using the H versions of the weekly releases and also the latest plugin versions of  the plugin that we need. This leads us to contribute back to the community when we detect any issues  in any particular plugin or weekly releases. It doesn't happen often and when it does, either we  raise a particular issue or Jira or whatever tracking system, they want that particular project  with what we found and then we just also help them to develop features and other things. Why?  Because we like to be on the edge and get the benefit of the latest version and this contributes back  in the way like it gives more maturity in the quality of the Jenkins ecosystem.  Okay, let's start explaining a little bit what is vulnerability. Surveillance or vulnerability  or O11Y. Try to join your application logs, your metrics and your application monitoring data  in one single point. In our case, elastic search. To show that data in dashboards or anything similar,  Kibana for example, and analyze that data we are much in learning and other techniques to detect  misbehavior and generate a life as related with that misbehavior. In elastic, in elastic, we use  elastic search, Kibana and bits. But what is elastic search, Kibana and bits? Elastic search is a  distributed open source, search and analyze and generate. For all types of data, Kibana provides  search and data visualization and bits provide data ingestion from many sources of that. For  example, logs with five bits, metrics with metric bits, monitoring with hard bits,  and cloud data with functional bits, network traffic with packet bits, Windows 7 with long bits,  and application performer monitoring with APM that collects performance information  about responding times and or incoming requests, database queries, call to the catchees,  external agit bit request, and so on. All that data, we analyze with machine learning to generate  alerts and show the data in dashboards to try to find misbehavior in that data. But the question here  is, we can use the same approach that we use for applications in the CI CD. We can try to send  data to elastic search and the misbehavior in the CI CD. That's what we want to watch and we want to  do in elastic. So after this introduction, let's talk about what's the problem statement, what we're  trying to solve. Let's talk about precisely what we do or the complexity of it. So a little bit of  context. The projects that we maintain is about 18 different GitHub repositories. There are different  programming languages. We just say 10 to be roughly that or even more. Every programming language  might have a different build tool system, or even any or none. There is also a different package,  a distribution we need to provide. In order to provide this, there are different versions of  the operating system that have to be supported. And different architectures. Again, different  iterations. So, so probably as we mentioned earlier, helps you to detect and to observe how  your system interacts with databases, front-end or cloud provider. So that's why we need to  support different iterations. So this is the entire ecosystem that we need to maintain.  From the continuous integration and continuous delivery or deployment, we need to basically  be sure like how we can go complete that particular aim of having a high quality  in what we deliver. So then we're going to discuss a little bit further. What are the  solutions to solve this particular problem statement? Okay. First of all, we have to set  a little number of rules before to start to achieve the problem statement. We have to try to  standardize some things. Not all, but a little bit of things. So the first thing that we have to  standardize is the tool that we use across all our projects. In our case, we decide to use Jenkins.  The second thing that we have to standardize is how we want to build the workflows related to  other pipelines related with the projects. In our case, we choose the clarifying  pipeline because it's an opinionated way to define our flow that keeps in the pipeline.  Only the orchestration logic, so it's simple. We don't want to end with a weird logic in those  pipelines. So this is the best approach in our opinion. For the jobs, we use multivariance  pipeline because it allows us to have only one job per repository. Because the multivariance  pipeline allows us to manage tasks and per request in only one job. All the things related  with every branch, pipeline, task and branch is created automatically. For the tools that we use  to build the projects, we try to delegate as much as possible in build tools, maybe in gradle,  whatever tool we have in the programming language that we have. That allows us to have the same  workflow locally. So we can reproduce the same things that we do in the HCI. We can do exactly  the same in local machines. The last thing is to try to have for all the projects a single  release with one click. So we click in one place and we generate the binaries of that.  So another aspect in the solution is don't repeat yourself. This is an extension of the  standardization. When you really work with many different projects, you are probably repeating  certain common patterns such as a checkout of the source code, the different notifications to the  different GitHub or different SCM tools that you use, how to generate secrets from both on  a similar tooling, how to compress a file, how to generate the test output and trend the output  to be able to visualize it easily in the UI or install certain tools that you need.  So by identifying this common process in the different projects and have this particular  definition somewhere else, it helps us as well to be able to provide the same framework to a  bunch of different projects. So the idea is to convert these particular processes in pipeline  steps. In this case, it's quite precisely about how we use Jenkins and the clarity pipeline.  Put all these steps in a cell library. In our case, we use the Jenkins cell library.  This will help us to simplify the workflows easily and therefore people can contribute to them  rather than having a really messy scripting logic. In our case, the cell library that we use  is about 150 different steps to integrate with Vault or to integrate with GitHub or AWS,  Google Cloud or the kind of cloud provider we use. So it's quite, yeah, it's there. So you can have a  look later. So we want to have every theme that we have in our environments and all SCICD  in a way that we can repeat things. So we can repeat the same process and obtain the same  workflows. To do that, we try to achieve this but putting everything in code.  Our infrastructure or jobs, panel lines, definitions, test environments and in general,  everything that we can define as code is defined as code. So our infra team provides  a standard way to provision in Jenkins systems that is defined as code in Ansible.  Our items, the same. We have the same process in Ansible to provisioning and  generate the image that we use to launch those items is defined as code in Ansible.  Our job definitions, we use a Jenkins job builder and job DSL to define our jobs and generate the  jobs from that code. For our local environments, we use a Docker and Jenkins configuration as code  to launch local instance that can run in Bangram definitions. We can reproduce the same  behaviors that we have in the SCICD locally for local testing. We also provide  a terminal environment for integration and test that we use in our jobs that are defined as code.  Our permanent environments for testing and dog fooding also are defined as code. We have  documentation that uses Ansible and other things to deploy environments in a repeatable manner.  And also the dogs. Everything that we have in documentation is generated from code.  So test them all is part of the next solution. In order to do that, we really need everything  as code. That's what we talked about earlier. What do we do with this particular testing?  Testing should happen before all the different systems we use. One of them is the Jenkins  set library. This is where we delegate all the responsibility for certain complex logic to be  easy to reach and use by the consumer. In this case, the pipelines or workflows that we define  in every single repository. So the logic can be used from different, it can be consumed by different  repositories or pipelines or workflows. It's going to be defined there. So the way we test this is  using unit testing and functional testing. With this, we can be sure that any release we do of  this particular pipeline, so library, Jenkins library, can be consumed with the certainty  that it will work. It's always more compatible. And the library itself is tested itself with  itself. So it sounds a bit redundant. But the idea is we have a process of branches and tags that  help us to be sure any upcoming release is tested beforehand to avoid any kind of surprises.  Regarding the infrastructure, we were talking like if we use Antibole, we use Docker,  Parker images, all of them they are tested before creating new virtual machines or Docker  containers, sorry, Docker images. We also test the Ansible deployment and the Helm charts.  The other thing that is important to say is about how we can ensure certain validation before  committing them in any Git repository. Some build system they provided because it's built in the  build system itself. Others, they don't have that particular maturity. And for such, we use pre-commit  in the Git repository. Pre-commit is one of the tools that you can use. You basically define  certain validations before allowing people to commit that in their repo. It's not just for  local development, it's also something we use in our CI CD. So it's a mandatory sanity check.  Other things that are important is that how you test the functionality, but also to scan the  security as well. For such, we use sync, false output, to detect any kind of security issues.  Third-party libraries dependences that can be compromised also to apply these practices.  Communication. Communication is trying to send the right information to the right people.  We try to facilitate the communication regarding the CI CD status only on one place.  Why? To avoid the communication overflow. Such, send to my emails,  your ERC message, issues, and we have checks. Also, help people that is not familiar with  the CI CD, not all the people know how to share something in Jenkins. We decided to provide this,  the gested information about build, test, deployment status, and other highlights in the Github  PRs in no case. The benefits of that, people don't need to access to Jenkins log by a file, but  in Blue Oaks and contains specific laws if need more details about the  issue. They have the laws in Github. In the next slide, I will provide an example of how we  digest that information in Github pull requests. We send a message to the Github repository,  and in that message, we have the links to the Jenkins UI and the gested information related  with the failure. For example, in a PR, we go to the PR, we will show the message of a build that  was successful, and we show that the build is successful. We can access to the pylite test and  another link to the Jenkins UI. We can show the number of tests that we launched and the results  of those tests. And also, we can check those pull requests that are failed. In those cases,  we have a similar message, but with different information. Let's see. So, if we go to a PR that  fails, we show that the build failed. And we have the test errors, and we can navigate to the  log. We can show a brief of the log. And also, we can show the steps of fail, and we can navigate  to the exact log of that fail. And there we have the error. So, people that don't know anything  about the CI CD tool can work in the same way that people that know how Jenkins or other tools work.  That's the goal.  Another important part of the solution we provide is the centralized knowledge.  To observe the status of the CI CD in different ways, we ingest all the key assets for any build,  test deployment to elastic search, and use Kibana to visualize and get a better understanding  of what's going on. Some of the samples could be to centralize the task for, in a way like all the  data is in one single place, as it's digested, and then the information will be easy to reach by  anyone. I'm going to show you different kind of real life examples of how we do the CI analysis.  This particular case, I want to show you, like, we do have a centralized dashboard in a markdown,  and we navigate there. We click on that particular test that it failed, and from there, we can access  exactly what is the build in Jenkins that fail, have a more deeper understanding of the logs,  and so on. Again, this is the view. Just click on the dashboard, and in the dashboard, you will be able  to see another view of how it looks like, the builds, and so on for a particular branch or a  particular project, and go to the failure test, and link from there to the build in Jenkins that fail.  Another thing that we can do is have other flagging, right? In this example, we do have an issue  with the label CI reported, and from there, we can see that there has been a flagging test.  We're clicking one of these GitHub issues from that particular project. We're going to see  certain information that has been generated automatically, such as the name of the test,  such as the build that it failed, such as the PR that caused that issue on the commit,  certain error details, the stack trace for that particular test failure, and a link as well to  the PR that caused that particular failure. This issue happens eight days ago, so we navigate  to the build comment that we mentioned earlier. From the communication, you will see a flaggy test  report message. We go to the particular history of eight days ago, we will see the flaggy test report  detected at that time, two test failures that were related to probably some flakiness. Just to  clarify, the way we tag a particular test failure as flaky is something we delegate to elastic search.  There is another process in place that all the DST test failures are analyzed, and if they happen  with certain, with an algorithm, is trying to see the previous build history, then tag them  with a certain likelihood that they are failure as flaky test. It's going to be that information.  The CI itself, after running any test, will evaluate any matches, and then the information  will be always in GitHub, as I said, GitHub issue. This is the idea of centralizing the knowledge,  to just in one place to understand when the build failed for any test, if they are flaky,  and then start a conversation between the different people that they own the project in GitHub.  Okay, monitoring demo. What we are trying to achieve is to have in science our CI CD and the  test problems. So to do that, we ingest all the data related with the status of our CI CD. We have  monitoring data related to the health of the CI CD environments, data related with laws,  metrics, and everything is in our search and everything is ingested with our tools. So we are  doing the food in for free about our products. What information we did just is exactly. We  send monitoring data about the CI workers, all the Jenkins instance, reverse proxies,  trip persistence, like GitHub, and this kind of stuff. We pull in the logs for the Jenkins,  for the builds, and we also monitor the status of the other branches. We have a lot of data when  something bad happens. We can go to the, with the, to Kibana and search the data,  and localize the error, and try to resolve it. We go to the worker that is failing,  because there is not enough memory or whatever, and we are solving all that.  Okay. Let's see an example of the data that we collect, and how we manage to do things with  that. For example, this is data related with the reverse proxy that we have for the Jenkins instance.  We collect a lot of data, and on that data, for example, we can filter for a region or a country,  and we can solve the request that we have from that country. In this case, we solve some patterns  of some requests, and for example, let's go to the 404 request and see what happens. We can be  some kind of a tag or something like that, and we can detect it more in a way. We saw that there is  a bunch of requests that fail requesting something, and we can check the request, the kind of request  that we have, and we can run the scans of the CIC to try to find some security issues or whatever.  So it's an example of how we use that.  Another aspect in this particular set of solutions we are talking about is the continuous deployment.  We don't have the continuous integration of the foundation there. In some cases, we do continuous  delivery, and now we are going to talk about the next aspect, how we deploy continuously  for some of the projects. For us, every successful merge to any of the principal branches  will generate binaries. This allows us to have a first snapshot to be able to ready to be deployed  and test in some of the environments we have. As we talked earlier, in the AgriFina code,  part of the solution we provide, we are able to generate a femoral and permanent environment  for testing and development. These permanent environments are massive. They are used for  demos, real testing or real testing in the sense of functional testing or manual testing, and help  developers to have the closer version with massive data. One of the examples is about the data that  is generated, and some of these, the environment is about four terabytes of data, and they are  created in the same manner of any production environment. Our release cycle is based on a  timeline, so there is a feature freeze that happens, and for that particular period of time,  there are certain testing done, and we produce release build candidates, and anytime there is a  new build candidate or snapshot, it is updated automatically in these environments.  We also have a specific edge environment. This edge environment is matching the principal branch,  basically the main branch, or most of the things, what's the terminology you use. The idea with this  is to be able to test any changes, even though they might break, we will detect things faster,  because incompatibility or any kind of other dependencies. The idea is not just to be able  to detect this, but also dog food or elastic cloud platform, which is a SaaS service. This  give value not only for our team, but also for other teams. This is a way we can  post the collaboration with all the teams as well.  In the next slide, the one that we are seeing now, is just giving you the way we not just use  our product itself, or just validating the work as expected, but it's also how we dog food.  So we can see the monitoring, the uptime, the logs of all the applications that are running in this  particular demo environment or edge environment. This is pretty much an example of, yes, we are  doing continuous deployment and continuously verifying that it behaves as expected.  Cance engineering is about cancing incidents in producing environments, and see how the system  is resilient to that incident. In our case, our production environments are over the environments  that we talked in the previous slides. We are interested to detect those incidents are reported.  So for us, caching engineering is another source of data for tuning all our tools.  So we generate random incidents, and we test specific scenarios, and we analyze the data,  and we try to detect those incidents that we define as experiments.  So future, we just finished to talk about the solution we put in place to apply CI-CD,  but there are still different areas that we would like to improve, or just today,  get some kind of that. So we highlighted three major topics. The very first one is using  much of learning to analyze why there was a failure. It could be related to tests, it could be  related to an environmental issue, or a genuine one. So based on this, it will help us to optimize  the resources that we use for building testing or deploying, and it will also help us to have some  self-healing. So the system itself will be able to retry a number of times if there are certain  flakiness, or if there was a genuine error or environmental issue. Other things is, as we  talk about in one of the solutions about communication, what we do have is at the moment,  is a kit-hop request comment with the status of the bill, deployment, or test. We like to  leverage these to have more information that has been digested even more. It will help us to avoid  any UI interactions. The sooner we get there, the better for everyone, because we will help  the onboarding to anyone with any of the projects we have. So this is our aim, to don't have any  UI interaction. And the last thing is about to standardize everything we have as a solution for  the CI. We're trying to move further with the way we use Jenkins. There are examples of certain  initiatives about monitoring, which is open telemetry as a standard or convention. It's one  of the areas we are also dedicating and investing some time to work on that.  So that's all. Thank you. You have a care, a couple of things. One is about the pipeline  library that we use in Jenkins. You have the limitation there, and you can find what we do  with Jenkins. It's open source. You can use it. And now, so a block of elastic search with all  the details about the products that we use and how to use it. Thank you. If you have some questions,  there is five minutes for questions. So write in the chat and we will answer your questions. Thank you.  Thank you.
look_ma_no_hands_jenkins_testability_and_monitoring_0_179.png look_ma_no_hands_jenkins_testability_and_monitoring_180_1271.png look_ma_no_hands_jenkins_testability_and_monitoring_1272_2819.png look_ma_no_hands_jenkins_testability_and_monitoring_2820_5987.png look_ma_no_hands_jenkins_testability_and_monitoring_5988_6119.png look_ma_no_hands_jenkins_testability_and_monitoring_6120_6227.png look_ma_no_hands_jenkins_testability_and_monitoring_6228_22559.png look_ma_no_hands_jenkins_testability_and_monitoring_22560_24887.png look_ma_no_hands_jenkins_testability_and_monitoring_24888_25331.png look_ma_no_hands_jenkins_testability_and_monitoring_25332_25727.png look_ma_no_hands_jenkins_testability_and_monitoring_25728_25835.png look_ma_no_hands_jenkins_testability_and_monitoring_25836_25895.png look_ma_no_hands_jenkins_testability_and_monitoring_25896_26807.png look_ma_no_hands_jenkins_testability_and_monitoring_26808_27035.png look_ma_no_hands_jenkins_testability_and_monitoring_27036_28223.png look_ma_no_hands_jenkins_testability_and_monitoring_28224_28427.png look_ma_no_hands_jenkins_testability_and_monitoring_28428_28607.png look_ma_no_hands_jenkins_testability_and_monitoring_28608_28751.png look_ma_no_hands_jenkins_testability_and_monitoring_28752_28895.png look_ma_no_hands_jenkins_testability_and_monitoring_28896_29219.png look_ma_no_hands_jenkins_testability_and_monitoring_29220_30407.png look_ma_no_hands_jenkins_testability_and_monitoring_30408_30647.png look_ma_no_hands_jenkins_testability_and_monitoring_30648_31799.png look_ma_no_hands_jenkins_testability_and_monitoring_31800_32291.png look_ma_no_hands_jenkins_testability_and_monitoring_32292_34559.png look_ma_no_hands_jenkins_testability_and_monitoring_34560_35603.png look_ma_no_hands_jenkins_testability_and_monitoring_35604_35903.png look_ma_no_hands_jenkins_testability_and_monitoring_35904_36059.png look_ma_no_hands_jenkins_testability_and_monitoring_36060_41315.png look_ma_no_hands_jenkins_testability_and_monitoring_41316_42347.png look_ma_no_hands_jenkins_testability_and_monitoring_42348_42659.png look_ma_no_hands_jenkins_testability_and_monitoring_42660_45047.png look_ma_no_hands_jenkins_testability_and_monitoring_45048_46515.png
Good morning, good afternoon, good evening. This is Who Watches the Watches, I am Yengi Juni, my name is Ivan Fernandez. I am a software engineer at Elastic for two years. I am a member of the productivity team inside the OXORA Leading Team and I was working with CIICD with Jenkins for 10 years or so. I maintain a couple of plugins of Jenkins also. My name is Victor Martinez. I am also a software engineer at Elastic. I work with Ivan in the same team and I've been using any things related with CIICD similar to Jenkins and other tools for over 10 years. We are here to talk a little bit as well with the company. So our company is Elastic. It's an open source company and contributes by different meanings to the community, to the open source community. In this particular case, we would like to highlight the way we use Jenkins for CIICD is by using the H versions of the weekly releases and also the latest plugin versions of the plugin that we need. This leads us to contribute back to the community when we detect any issues in any particular plugin or weekly releases. It doesn't happen often and when it does, either we raise a particular issue or Jira or whatever tracking system, they want that particular project with what we found and then we just also help them to develop features and other things. Why? Because we like to be on the edge and get the benefit of the latest version and this contributes back in the way like it gives more maturity in the quality of the Jenkins ecosystem. Okay, let's start explaining a little bit what is vulnerability. Surveillance or vulnerability or O11Y. Try to join your application logs, your metrics and your application monitoring data in one single point. In our case, elastic search. To show that data in dashboards or anything similar, Kibana for example, and analyze that data we are much in learning and other techniques to detect misbehavior and generate a life as related with that misbehavior. In elastic, in elastic, we use elastic search, Kibana and bits. But what is elastic search, Kibana and bits? Elastic search is a distributed open source, search and analyze and generate. For all types of data, Kibana provides search and data visualization and bits provide data ingestion from many sources of that. For example, logs with five bits, metrics with metric bits, monitoring with hard bits, and cloud data with functional bits, network traffic with packet bits, Windows 7 with long bits, and application performer monitoring with APM that collects performance information about responding times and or incoming requests, database queries, call to the catchees, external agit bit request, and so on. All that data, we analyze with machine learning to generate alerts and show the data in dashboards to try to find misbehavior in that data. But the question here is, we can use the same approach that we use for applications in the CI CD. We can try to send data to elastic search and the misbehavior in the CI CD. That's what we want to watch and we want to do in elastic. So after this introduction, let's talk about what's the problem statement, what we're trying to solve. Let's talk about precisely what we do or the complexity of it. So a little bit of context. The projects that we maintain is about 18 different GitHub repositories. There are different programming languages. We just say 10 to be roughly that or even more. Every programming language might have a different build tool system, or even any or none. There is also a different package, a distribution we need to provide. In order to provide this, there are different versions of the operating system that have to be supported. And different architectures. Again, different iterations. So, so probably as we mentioned earlier, helps you to detect and to observe how your system interacts with databases, front-end or cloud provider. So that's why we need to support different iterations. So this is the entire ecosystem that we need to maintain. From the continuous integration and continuous delivery or deployment, we need to basically be sure like how we can go complete that particular aim of having a high quality in what we deliver. So then we're going to discuss a little bit further. What are the solutions to solve this particular problem statement? Okay. First of all, we have to set a little number of rules before to start to achieve the problem statement. We have to try to standardize some things. Not all, but a little bit of things. So the first thing that we have to standardize is the tool that we use across all our projects. In our case, we decide to use Jenkins. The second thing that we have to standardize is how we want to build the workflows related to other pipelines related with the projects. In our case, we choose the clarifying pipeline because it's an opinionated way to define our flow that keeps in the pipeline. Only the orchestration logic, so it's simple. We don't want to end with a weird logic in those pipelines. So this is the best approach in our opinion. For the jobs, we use multivariance pipeline because it allows us to have only one job per repository. Because the multivariance pipeline allows us to manage tasks and per request in only one job. All the things related with every branch, pipeline, task and branch is created automatically. For the tools that we use to build the projects, we try to delegate as much as possible in build tools, maybe in gradle, whatever tool we have in the programming language that we have. That allows us to have the same workflow locally. So we can reproduce the same things that we do in the HCI. We can do exactly the same in local machines. The last thing is to try to have for all the projects a single release with one click. So we click in one place and we generate the binaries of that. So another aspect in the solution is don't repeat yourself. This is an extension of the standardization. When you really work with many different projects, you are probably repeating certain common patterns such as a checkout of the source code, the different notifications to the different GitHub or different SCM tools that you use, how to generate secrets from both on a similar tooling, how to compress a file, how to generate the test output and trend the output to be able to visualize it easily in the UI or install certain tools that you need. So by identifying this common process in the different projects and have this particular definition somewhere else, it helps us as well to be able to provide the same framework to a bunch of different projects. So the idea is to convert these particular processes in pipeline steps. In this case, it's quite precisely about how we use Jenkins and the clarity pipeline. Put all these steps in a cell library. In our case, we use the Jenkins cell library. This will help us to simplify the workflows easily and therefore people can contribute to them rather than having a really messy scripting logic. In our case, the cell library that we use is about 150 different steps to integrate with Vault or to integrate with GitHub or AWS, Google Cloud or the kind of cloud provider we use. So it's quite, yeah, it's there. So you can have a look later. So we want to have every theme that we have in our environments and all SCICD in a way that we can repeat things. So we can repeat the same process and obtain the same workflows. To do that, we try to achieve this but putting everything in code. Our infrastructure or jobs, panel lines, definitions, test environments and in general, everything that we can define as code is defined as code. So our infra team provides a standard way to provision in Jenkins systems that is defined as code in Ansible. Our items, the same. We have the same process in Ansible to provisioning and generate the image that we use to launch those items is defined as code in Ansible. Our job definitions, we use a Jenkins job builder and job DSL to define our jobs and generate the jobs from that code. For our local environments, we use a Docker and Jenkins configuration as code to launch local instance that can run in Bangram definitions. We can reproduce the same behaviors that we have in the SCICD locally for local testing. We also provide a terminal environment for integration and test that we use in our jobs that are defined as code. Our permanent environments for testing and dog fooding also are defined as code. We have documentation that uses Ansible and other things to deploy environments in a repeatable manner. And also the dogs. Everything that we have in documentation is generated from code. So test them all is part of the next solution. In order to do that, we really need everything as code. That's what we talked about earlier. What do we do with this particular testing? Testing should happen before all the different systems we use. One of them is the Jenkins set library. This is where we delegate all the responsibility for certain complex logic to be easy to reach and use by the consumer. In this case, the pipelines or workflows that we define in every single repository. So the logic can be used from different, it can be consumed by different repositories or pipelines or workflows. It's going to be defined there. So the way we test this is using unit testing and functional testing. With this, we can be sure that any release we do of this particular pipeline, so library, Jenkins library, can be consumed with the certainty that it will work. It's always more compatible. And the library itself is tested itself with itself. So it sounds a bit redundant. But the idea is we have a process of branches and tags that help us to be sure any upcoming release is tested beforehand to avoid any kind of surprises. Regarding the infrastructure, we were talking like if we use Antibole, we use Docker, Parker images, all of them they are tested before creating new virtual machines or Docker containers, sorry, Docker images. We also test the Ansible deployment and the Helm charts. The other thing that is important to say is about how we can ensure certain validation before committing them in any Git repository. Some build system they provided because it's built in the build system itself. Others, they don't have that particular maturity. And for such, we use pre-commit in the Git repository. Pre-commit is one of the tools that you can use. You basically define certain validations before allowing people to commit that in their repo. It's not just for local development, it's also something we use in our CI CD. So it's a mandatory sanity check. Other things that are important is that how you test the functionality, but also to scan the security as well. For such, we use sync, false output, to detect any kind of security issues. Third-party libraries dependences that can be compromised also to apply these practices. Communication. Communication is trying to send the right information to the right people. We try to facilitate the communication regarding the CI CD status only on one place. Why? To avoid the communication overflow. Such, send to my emails, your ERC message, issues, and we have checks. Also, help people that is not familiar with the CI CD, not all the people know how to share something in Jenkins. We decided to provide this, the gested information about build, test, deployment status, and other highlights in the Github PRs in no case. The benefits of that, people don't need to access to Jenkins log by a file, but in Blue Oaks and contains specific laws if need more details about the issue. They have the laws in Github. In the next slide, I will provide an example of how we digest that information in Github pull requests. We send a message to the Github repository, and in that message, we have the links to the Jenkins UI and the gested information related with the failure. For example, in a PR, we go to the PR, we will show the message of a build that was successful, and we show that the build is successful. We can access to the pylite test and another link to the Jenkins UI. We can show the number of tests that we launched and the results of those tests. And also, we can check those pull requests that are failed. In those cases, we have a similar message, but with different information. Let's see. So, if we go to a PR that fails, we show that the build failed. And we have the test errors, and we can navigate to the log. We can show a brief of the log. And also, we can show the steps of fail, and we can navigate to the exact log of that fail. And there we have the error. So, people that don't know anything about the CI CD tool can work in the same way that people that know how Jenkins or other tools work. That's the goal. Another important part of the solution we provide is the centralized knowledge. To observe the status of the CI CD in different ways, we ingest all the key assets for any build, test deployment to elastic search, and use Kibana to visualize and get a better understanding of what's going on. Some of the samples could be to centralize the task for, in a way like all the data is in one single place, as it's digested, and then the information will be easy to reach by anyone. I'm going to show you different kind of real life examples of how we do the CI analysis. This particular case, I want to show you, like, we do have a centralized dashboard in a markdown, and we navigate there. We click on that particular test that it failed, and from there, we can access exactly what is the build in Jenkins that fail, have a more deeper understanding of the logs, and so on. Again, this is the view. Just click on the dashboard, and in the dashboard, you will be able to see another view of how it looks like, the builds, and so on for a particular branch or a particular project, and go to the failure test, and link from there to the build in Jenkins that fail. Another thing that we can do is have other flagging, right? In this example, we do have an issue with the label CI reported, and from there, we can see that there has been a flagging test. We're clicking one of these GitHub issues from that particular project. We're going to see certain information that has been generated automatically, such as the name of the test, such as the build that it failed, such as the PR that caused that issue on the commit, certain error details, the stack trace for that particular test failure, and a link as well to the PR that caused that particular failure. This issue happens eight days ago, so we navigate to the build comment that we mentioned earlier. From the communication, you will see a flaggy test report message. We go to the particular history of eight days ago, we will see the flaggy test report detected at that time, two test failures that were related to probably some flakiness. Just to clarify, the way we tag a particular test failure as flaky is something we delegate to elastic search. There is another process in place that all the DST test failures are analyzed, and if they happen with certain, with an algorithm, is trying to see the previous build history, then tag them with a certain likelihood that they are failure as flaky test. It's going to be that information. The CI itself, after running any test, will evaluate any matches, and then the information will be always in GitHub, as I said, GitHub issue. This is the idea of centralizing the knowledge, to just in one place to understand when the build failed for any test, if they are flaky, and then start a conversation between the different people that they own the project in GitHub. Okay, monitoring demo. What we are trying to achieve is to have in science our CI CD and the test problems. So to do that, we ingest all the data related with the status of our CI CD. We have monitoring data related to the health of the CI CD environments, data related with laws, metrics, and everything is in our search and everything is ingested with our tools. So we are doing the food in for free about our products. What information we did just is exactly. We send monitoring data about the CI workers, all the Jenkins instance, reverse proxies, trip persistence, like GitHub, and this kind of stuff. We pull in the logs for the Jenkins, for the builds, and we also monitor the status of the other branches. We have a lot of data when something bad happens. We can go to the, with the, to Kibana and search the data, and localize the error, and try to resolve it. We go to the worker that is failing, because there is not enough memory or whatever, and we are solving all that. Okay. Let's see an example of the data that we collect, and how we manage to do things with that. For example, this is data related with the reverse proxy that we have for the Jenkins instance. We collect a lot of data, and on that data, for example, we can filter for a region or a country, and we can solve the request that we have from that country. In this case, we solve some patterns of some requests, and for example, let's go to the 404 request and see what happens. We can be some kind of a tag or something like that, and we can detect it more in a way. We saw that there is a bunch of requests that fail requesting something, and we can check the request, the kind of request that we have, and we can run the scans of the CIC to try to find some security issues or whatever. So it's an example of how we use that. Another aspect in this particular set of solutions we are talking about is the continuous deployment. We don't have the continuous integration of the foundation there. In some cases, we do continuous delivery, and now we are going to talk about the next aspect, how we deploy continuously for some of the projects. For us, every successful merge to any of the principal branches will generate binaries. This allows us to have a first snapshot to be able to ready to be deployed and test in some of the environments we have. As we talked earlier, in the AgriFina code, part of the solution we provide, we are able to generate a femoral and permanent environment for testing and development. These permanent environments are massive. They are used for demos, real testing or real testing in the sense of functional testing or manual testing, and help developers to have the closer version with massive data. One of the examples is about the data that is generated, and some of these, the environment is about four terabytes of data, and they are created in the same manner of any production environment. Our release cycle is based on a timeline, so there is a feature freeze that happens, and for that particular period of time, there are certain testing done, and we produce release build candidates, and anytime there is a new build candidate or snapshot, it is updated automatically in these environments. We also have a specific edge environment. This edge environment is matching the principal branch, basically the main branch, or most of the things, what's the terminology you use. The idea with this is to be able to test any changes, even though they might break, we will detect things faster, because incompatibility or any kind of other dependencies. The idea is not just to be able to detect this, but also dog food or elastic cloud platform, which is a SaaS service. This give value not only for our team, but also for other teams. This is a way we can post the collaboration with all the teams as well. In the next slide, the one that we are seeing now, is just giving you the way we not just use our product itself, or just validating the work as expected, but it's also how we dog food. So we can see the monitoring, the uptime, the logs of all the applications that are running in this particular demo environment or edge environment. This is pretty much an example of, yes, we are doing continuous deployment and continuously verifying that it behaves as expected. Cance engineering is about cancing incidents in producing environments, and see how the system is resilient to that incident. In our case, our production environments are over the environments that we talked in the previous slides. We are interested to detect those incidents are reported. So for us, caching engineering is another source of data for tuning all our tools. So we generate random incidents, and we test specific scenarios, and we analyze the data, and we try to detect those incidents that we define as experiments. So future, we just finished to talk about the solution we put in place to apply CI-CD, but there are still different areas that we would like to improve, or just today, get some kind of that. So we highlighted three major topics. The very first one is using much of learning to analyze why there was a failure. It could be related to tests, it could be related to an environmental issue, or a genuine one. So based on this, it will help us to optimize the resources that we use for building testing or deploying, and it will also help us to have some self-healing. So the system itself will be able to retry a number of times if there are certain flakiness, or if there was a genuine error or environmental issue. Other things is, as we talk about in one of the solutions about communication, what we do have is at the moment, is a kit-hop request comment with the status of the bill, deployment, or test. We like to leverage these to have more information that has been digested even more. It will help us to avoid any UI interactions. The sooner we get there, the better for everyone, because we will help the onboarding to anyone with any of the projects we have. So this is our aim, to don't have any UI interaction. And the last thing is about to standardize everything we have as a solution for the CI. We're trying to move further with the way we use Jenkins. There are examples of certain initiatives about monitoring, which is open telemetry as a standard or convention. It's one of the areas we are also dedicating and investing some time to work on that. So that's all. Thank you. You have a care, a couple of things. One is about the pipeline library that we use in Jenkins. You have the limitation there, and you can find what we do with Jenkins. It's open source. You can use it. And now, so a block of elastic search with all the details about the products that we use and how to use it. Thank you. If you have some questions, there is five minutes for questions. So write in the chat and we will answer your questions. Thank you. Thank you.
also a software engineer at Elastic. I work with things related with CIICD similar to Jenkins and talk a little bit as well with the company. So this particular case, we would like to highlight the way we use Jenkins for CIICD is by using the need. This leads us to contribute back to the community when we detect any issues in any quality of the Jenkins ecosystem. Okay, let's same approach that we use for applications in the going to discuss a little bit further. What are request in only one job. All the things related to convert these particular processes in pipeline CI CD, not all the people know how to share the CI CD tool can work in the same way that Jenkins that fail, have a more deeper CI reported, and from there, we can see that particular period of time, there are certain cloud platform, which is a SaaS service. This give value not only for our team, but also for all the applications that are running in this about in one of the solutions about you.