Standardize Testing in Python
Gabor, Bernat
In this talk, Bernat will introduce tox, an open source tool with the bold vision of standardizing testing. From a CIs point of view, testing contains much more than just unit and/or integration tests; other things like code style checks, packaging, testing under various versions of Python, and checking that documentation still generates, are just as important. Things quickly start to spiral out of control once you add into the mix that there are many tools and ways to accomplish each of these tasks. In this talk, we will learn how tox tries to abstract away all this complexity, how to easily run CI tests on your local machine, and how one can use this tool inside your CI frameworks to ensure high quality and easily maintainable packages.
 Thank you guys for coming for this talk. So today we're going to learn something about talks mainly, but before I learn it, it's actually a tool which has the bold aim to actually standardize testing inside Python. And we already have tests, you may say. Everyone heard about PyTest, but if you like a lot of the, one of the GitHub repositories, for example, for the PyTest tools, but it's if you see that down there, there is a talk that in a file, meaning it PyTest itself uses talks, or you can go it, if similarly does that or there are other tools such as, for example, request also does similarly use the talks. So if you come and start one to contribute to this project, you might get the question, so why is this talks all around and what actually this purpose is? Hopefully by the end of this session, you will have a good understanding of what talks tries to do, why it's needed, and what can it do for you, how can you use it, how it works. Okay? So let's start with the first question, why it is needed. Yeah, just to brighten up the mood ever so often, I put in a few bugs here and there, so that's for that. Okay, so let's start with like, generally, you have your Python application, what kind of parts Python application nowadays has. So first and foremost, you have your business logic, this basically boils down to your actual source files, this is what your manager writes for you to actually, or pays for you to do. Okay? Then we have the test, which are basically for yourself, so you can actually ensure that whatever your manager asks, it's actually doing whatever you wrote. Then you have some packaging, because usually whenever you create an application or even like a library, you actually want to package it and deploy it to the, where it will run or to the other users. So you need to have packaging, then you need to also have documentation, because someone else when it comes back, still need to understand the project. Then you have nowadays also quite popular type checks, for example, my pipe, I try to actually check if anything's okay or not in your source code. And then you have also like set it code analysis, this is think of tools such as Flake or Pep 8 or Piling. And finally, usually when, especially if you're writing a library, this is true, you want to support that library for more than one Python versions, so that everyone can be happy and start using it. And therefore, you actually want to test that whatever you're creating works under multiple Python socares. And then there's another dimension, which is mostly used, for example, the Django world, where you have multiple versions of Django. So if you write a Django plugin, you want to ensure that it runs with all types of Django versions, okay? So these are basically complications, which means that instead of the like beforehand, you maybe had to write one test. Now you're that one test instead now you have like basically 10 tests or 12 tests or similar, a lot more tests, which you want to basically run. And additional thing which is usually nice whenever someone comes to your project and they want to like quickly get started with the project, they don't want to read two pages of how to get started developing. They usually would be nice if they can just have a single comment which they can use to actually set up the entire developer environment, so to say. And obviously you want to ensure that all these kind of configurations and all these kind of steps and all these kind of checks in your application actually pass after each commit. So the next people who come and start contributing something to your project, it actually can start with a clean slate and doesn't have to first repair whatever is broken in the project. Luckily we live inside the great ecosystem of Python. This means that we have a lot of people who are eager to open source their solutions for various solutions, for various problems. And what this means that for example, we obviously have a tool for everything. For test we have pie test, nose test for packaging. Now this is quite for libraries that is set up to split or poetry, but for applications like Pax or Ser or things which are nowadays quite popular or the documentation, both things and make docs is widely used inside the Python ecosystem. As I said before, and static analysis, style checks like black, piling, flint also make it. Then we have all the ways we can create parts of a virtual environment which basically helps you test multiple Python versions and then you also have, you can use this also to test the virtual dependency types. For example, you want to test again Django one and Django two in the same time. Okay. So now basically you can set up for your project, use one of these tools and make sure and you use one of these tools and basically once you use that tool, everyone can ensure after every commit that everything runs. Okay. But the downside is that now basically you introduced your project 10 tools and everyone who wants to basically just maybe contribute even a little might need to actually learn all those 10 tools and understand how they need to be used to actually be able to work with the project. Okay. So this is not really a good place to be because basically now you mean that whoever comes to your project, you will have a really steep learning curve to actually join the project and actually understand why all this test are needed and basically you will have some really sad users when they want to contribute to the project because they will not actually understand why something, for example, my fail, some checks my failing in the CI for which he locally did not fail. For example, a quite common example is that for example, he messes up the formatting or some static checking and even to locally test past, it will still mean that the CI will broke on the remote and these generally leaves contributors needing to come back all the time to whatever they started to create. So the idea would be that it would be nice if you could somehow bring down the bridge or build the bridge between the CI and the local development environment and somehow it could basically be possible to run the same thing that we run the CI run also locally. Okay? So as I said beforehand, GitHub advertises creating the contributing file which basically means someone has to read it and understand it and then actually use it. Then you have the option you may probably at one point you will get fed up with copying or uncomment so you will start writing shell scripts to actually contain all this kind of test or if you're really hard core, you go and write a make file which basically does the same, it's more like an advanced shell script. Okay? So here's just a good example how one of the shell scripts could look. So you basically, you want to basically start with just checking that if anything fails, stop, don't continue and then basically start packaging your project, installing it, creating a virtual environment, running your test and if everything, once all these comments are run, you probably everything will also pass on the CI side. Okay? Now, as I said beforehand, this, in this example, it's only for one Python which is like a system Python would be nice to add virtual environments because what if you need to support both for Python 3.7 and 3.6? So also, you want to run all checks regardless and actually print maybe at the end of the report because here I did not end, I did not actually address the point of maybe it would be nice in the end, we would print out all the things which failed and because that's what the usual actually would need to address and it would also be nice if you could actually run this in parallel because basically many of these tools can run in parallel because they're not depending on each other. Okay? So as far as going to actually testing multiple Python, you can use the virtual and for the VM which is added to Python core libraries and basically you can create your own lightweight side directory and then keep installing your package into it. We, this you can use basically by activating but you don't have to actually activate it. If you really want, you can, and you can spell out the full path, you can use the full path to actually run things inside the given virtual environment. Okay? Obviously, this now requires you to actually remember what is the path for your virtual environment or this basically, once we add this just for one virtual environment, this kind of our best script now becomes the stand-lines which makes all, still does not actually do any reporting when also just hand there's one single Python version, so you probably want more than that. So one can see that this point that this kind of writing this kind of magic script which run every test basically, every kind of tools starts to be hard to actually maintain. It also looks error-prone because if you can do anywhere in type of the error if you want to add another environment, it's like you have to type a lot of other, it probably will incorporate a lot of copying around. So this sort of seems like there must be a better way with this. And what we want is happy users. We want them to be jumping around in our project, so let's try to do something like that. And this is the Toxas Dream. Toxas actually started out by the same developers who started out PyTest. They envisioned the testing, the test run or more or less to be Tox and PyTest to be actually the test frame, the testing framework. But you can, they, they, they didn't those to be the tool which actually drives which people start interacting with. Okay. So how can we achieve merging the CI and Shell-based testing? Okay. So the idea is that we should be able to run locally everything which we run on the CI. This is all especially important because in anything breaks on the CI, you want to be able to reproduce it locally because that's how you, that's where you can actually, for example, start debugging interaction and understand what's happening. Okay. So Toxas basically allows for Python project to define all tools in one place, meaning that all kind of tools that you may use, you may add checks for it. You can define it in one single configuration file with their actual configurations. Furthermore, you also have a central way of invoking them. When we talk about central way of invoking them, think of it like make targets. That's probably the best approximation or generally just build targets. And finally, you should be ensuring, you should be creating an environment which is isolated and reproducible, so to say. So when you run it, then it should be. And by isolation, we don't just necessarily mean virtual environment. We also mean, for example, shielding you from whatever you have in your user operating system environment. Okay. So it is basically a CLI tool, meaning if you actually, to actually use Toxas, you don't need anything. You just need Python pip and then just install it. And then you can invoke it with, for example, Python minus and Tox. And that, if the user should work it and should be able to run all the testing without actually remembering what are the comments. Okay. And it basically manages, creates and runs the reproducible environment. So this is what Tox actually does for you. It basically will ensure it has a look at your configuration and will try to recreate your own, your, recreate a testing environment for a given target exactly as you defined it. Okay. So let's see how it actually works. This is like a flow diagram which shows what Tox actually does. It initially has like a simple configuration phase. This is basically just putting together everything you have in your configuration file and whatever you're passing from the command line plus whatever may be inside the operating system environment. Then there is this, some of the steps are optional. So you don't have to do all the steps. For example, the packaging is optional. If you have a library, you also want to, for example, package your library because when you want to test it, you actually want to test the package version, whatever the end user would use, not what your local file system contains. Okay. And, but as I said, this is basically packaging. So you can also use it for your applications. So once the packaging is done, it will, for each of your targets, it will define a virtual environment. And this virtual environment will face over three phases. It will have an environment creation, an install phase, and finally it will have a run comments phase. And once all the comments have run, it will have a reporting. Okay. So this is, for example, a simple configuration. I'm not sure how good it is. I'm going to try to zoom in. It shows a bit better. So for example, here is a simple project. I defined three environments. For example, one of my environments is basically just running the test under Python 2.7, for example, here. And then I have a linting, a documenting, and a publishing target. Also, this target, for example, the linting, because you use a spiling 2.0 requires Python 3 plus. So I specify to you like Python 3.7. My documentation does the same because maybe inside my conf py, I actually want to use the fancy f strings and all that things. So it is possible to specify that for the documentation, it's a different Python environment or a different version. And then I also defined in the end a publishing. For example, I don't have to actually remember how to publish everything I just defined there that for me, publish basically means have an environment which has both set up tools and one. And then just basically run the set up py as this baby skill and then use twine to actually publish it. Okay? So this means that now basically whenever I need to do publish, I don't have to any more remember whatever I just call the publish target and it will work and it will create for me and make sure that I have the correct twine versions and that kind of things. Okay? The first thing which I have there in like the first time when I see emblist, that is basically the emblist which will run always. So if the user doesn't specify what should run, it will run those environments in dent order more or less. But as I can see, things which I don't need to run always, I can define it as additional one and they are for me to be run as on demand. For example, the publishing is something we probably don't want to do it after every commit. So that's something that is not actually defined inside the default environment. So to say which are run by default. Okay? So, yeah. As you can see, each of them are like separately defined. They have their own section where you can define not just the dependencies. You can also define their actual arguments that you want to pass on to those kind of configurations. And this allows you, for example, you can now have all your configuration in one file because most of the tools provide command line environment flex and you don't no longer necessarily need to have custom configuration files laying around like five, four in your project for each different kind of project. Okay? For example, this will be the actual documentation building and you can see it even generates a unique build directory for it. The user can actually then see the output if they want to see it. And yeah. Now let me zoom back. Okay. So let's go to the next one, the packaging. So the packaging is basically, it creates an installable package. This is the only in the 20 you have a library. The build dependencies at the moment are not actually insured, meaning that from wherever the environment you're actually invoking to needs to have all the build dependencies. Now this is a temporary thing because I wrote here long term but probably is more like in three months or so. The plan is that once PIP will have PEP 517 and 518 support, we will actually be able to specify our project or build dependencies inside those files and we'll be able to actually insure also that you can actually install, like people can insure that you have all your build dependencies. Like build dependencies are, for example, the setup tools that you want to use but also like additional ones which are actually doing the build but are required to do the build. For example, setup tools SCM is one of the popular which you can use to like automatically version your project. This is also like build dependency but it's not actually a build tool. Okay? So the next step, environment creation. This basically wants to be optimized meaning that if the environment already exists, you already run the comment beforehand. It will not be created from scratch. It will reuse it. Now as I, as you can remember beforehand, like here, I had here at the start section where I defined what kind of Python I want. So this means that there must be a way somehow the application or the tool needs to figure out what Python's do you have on your computer and what actually interpreter where it lives. So for this one, we actually use the system path on Linux. So basically, we try to see if there is a Python 3.7 executable in your system. Okay? Obviously, this doesn't work on Windows. On Windows you need to use the PyExe. For this kind of interpreter discovery, there are two perhaps which is basically detail how Python discovery should work on different platform basically like a unique something in those and the tool follows basically those two ideologies. And we use virtual and the create virtual environments for the given version. And then at the end of it, there is an install dependencies meaning that you basically we just whatever you specify. You can specify explicitly the dependency. One of the things we install the dependency is specify here. This supports whatever PEP440 supports meaning that you can also specify like requirements to the XT file. They basically just get forwarded to your installer which by default is PIP. But for example, someone can use PEPAMP or whatever other they might want to use. And then there are also other dependencies after we install the project dependencies or the ones that specify this for the tool. Then there's another set of dependencies to the itself. And this also means everything specified inside the install requires you'll also be installed in the second phase. Okay? Now there is a slight, yeah, the default is PIP but can be changed real tools. Okay? So once you installed your project, you have it available in a fresh environment just as it would be installed basically at the user end. If they would use it, you then the following things will actually run your comments. Okay? So for this kind of comments, for example, here we do T things or three things. There are T comments to be run. The first comment is basically let's just build the documentation. The second thing is basically let's see if all the links are valid inside the documentation. And then the final one is just print out for the user where you can find the documentation and view it. Okay? One of the important things that we do for running the comments is that we basically strip away most of your operating system environment variables. And this is quite helpful in a lot of the cases, especially when you're relying on some environment variables or your project has some environment variables which drives its functionality. It is often the case that you set it locally but then forget it that on the other users machine that not may be set. So this ensures that you can already fail on this kind of environments not set case locally and not have to actually wait until other users reported that by the way on my system that doesn't work. Okay? And basically the chain of comments will run it one after another and whenever one fails, we stop. And if any of the comments fail, that environment is marked as being a failure. Okay? And we stream basically whatever your tool puts out on the output. This can be configured basically but by default we stream out. This allows you to actually see what the tool output was so you can then take action based on that. Okay. So, and the final thing is we just print a nice report. For example, in this case you can see that two environments failed. Basically the test and the linking but the documentation generation finished its success. Okay? So basically this is like the workflow of Tux. This is what it does you. It basically ensures you that you have a virtual environment which is as specified inside your configuration. It automatically sets its up. It will package your project. It will in the end allow you to actually print out whatever happened when you ran it. The idea is that for example, if some new user joins the project, they don't have to say like, so how do I run the test? Basically they can just invoke Tux and that will print out and that will do everything for them and basically have a local CI if they need it. Okay. Let's see some use cases or how it actually works in practice. Okay? So for example, this is a default and you can see that this actual call might take some time because this is creating it on a fresh state. You can see that we have three targets. We have basically the same, the testing, the linking and the documentation phase and we will start creating this phase as one by one once we just invoke the Tux and it will invoke Python with the correct argument. It will ensure that the environment has been created and whenever any error happens, we immediately display it for the user so they can act on it. Okay? Now, the thing is this usually takes a lot faster. The reason why it now it is a bit slower is because basically we have create operations there and creating a virtual environment and installing dependencies because network is involved and I was involved is not as fast as many of the things. So at first time, it takes longer but for example, when you invoke it the second time, then basically we don't have to do anything then just re-invoke the comments that you have because all the dependencies that already installed and are in place. Okay? So it is obviously possible that you can run also just a subset of environments. For example, the cases you might only want to test what actually modified the source code. For example, documentation generation, you don't really care what you're actually developing or for example, another popular use case of this is that you want to run your, which you have for example, testing against two Python environment and you want to ensure that for example, you run the test for Python 2.7 and 3.6 and then you want to have another step after it which basically calculates the aggregated coverage reports that you can do with this kind of way because you can specify that run this to Python environments and then display an aggregated coverage for those two. Okay? Okay, another thing that is quite nice is that you can actually document whatever the targets are so the users don't have to actually guess. So you can add descriptions to it and then the user when, for example, this is for our own toxin if I basically when someone says, okay, what can I do with this project? They can just list whatever the possible targets are and they can read if they want to run any of them or not. Okay? You can see that there are default environments which are always run and then additional ones which are on demand basically. Another option is that you support the post-arts. I'm sure. So basically if you testify for your comment that you want to post our Gantt's end, this means that if you use the double dash and after a double dash, you can basically forward arguments to your underlying tool. This is very helpful. For example, in this case, I wanted to run the test on the Python 2.7 and if anything fails, I wanted to enter into the debugging, for example, PDB. Okay? So I did not, I could basically just pass them on to the invocation after the double dash as additional arguments and they automatically get forwarded. This is also a nice way where, for example, you can have more verbose test reports and all those kind of things which are basically more like you want to understand more deeply what something broke and not just run the default test with so to say. Okay? Another option which we have there is basically, we have another project which is like detox. This basically allows you to run things in parallel. For example, it is possible to run all the environments in parallel and then display an aggregated output. Now, obviously the limitation of this is that we can no longer stream the output in real time because if we would do, basically, you would mean that five environment, five tools run in parallel and all five tools would print the character and the screen would be quite hard to actually see what's on the output. Okay? So another quite, you can actually create a developer environment. Why don't I say developer environment and this is especially true for libraries. It means that we don't actually install your package as the user would install it but we install your package using PPS in developer mode which basically means that people not actually install your package by putting it into the site package but creates a same link and says that the source code lives in your wherever you're developing and then the whenever Python is running, it will actually pick up your local files and not the remote, not the site package files. Okay? Okay. Now, just one thing, once you start creating this kind of environment or generally, it is quite healthy for example, in your environment, you can quickly select one of the environments to be the one for developer ecosystem to use. For example, in this case, for Python, there is this PyMVmanage plugin which is quite nice which allows you to basically just right click on any of the environments, be made talk for any other virtual environment and say, but by then I'll change the interpreter package to this Python which is inside this folder. Okay? So, another thing which is quite popular is like obviously, especially like big projects care about 45 Python versions. You might not have all the Python versions locally on your machine so it doesn't make sense to force failing those environments. You don't have Python. You can specify that only you run environments which for which we can actually find the correct Python version. For example, don't try to run Python 3.5 if that's not on the system because it will fail but it's not actually a failure that the test will be just that we could not, we have dependently missing for it. Okay? Another helpful thing is this is especially helpful for like bigger projects or like Django project is that Django projects historically have like a lot of versions and you want to check a lot of combinations and ensure that all the combinations actually work. It allows you to basically specify and you can see in three lines all that kind of Django combinations and there, for example, you can see that, for example, the Django master actually can specify even like GitHub links and it will install the latest from GitHub and try to test it again. Okay? So, as you can see, basically, we just tend like so of configuration. Now you can test against like 12 combination of dependently that your users might run into. Okay? Now obviously, you might not necessarily want to run all of this locally when you develop but this can work something in the CI and whenever something works in the CI, you locally, you don't have to manually recreate the environment. You can just say, by the way, on the CI, I saw that the Py37 Django master failed. Let me just recreate that and run the test only in that and try to debug what actually is the underlying issue for that. Okay? So, CI integration. As I said, the whole concept of it is that we would prefer to not be CI agnostic, meaning that we want the CI to basically don't have to actually specify all these. The CI should basically just invoke the same thing as the user would locally basically just invoke docs. And this means that your CI configuration, if you use docs, beside the user actually having the option to easily set up locally any kind of environment. Now the CI, you don't have to use the property CI environment but you can basically just say that, oh, by the way, make sure that I have these pythons in that CI and then finally install them with, install, for example, my docs and then just run docs and that's the entire CI configuration. Okay? So, let me sum up a bit what I tried to present here today and what we learned. So, the first thing is obviously there are weaknesses of the tools. The biggest weaknesses of what people often get confused is that dependency updates. The way PIP works, if you say to PIP that install PIP 11 and unless you specify that, by the way, please make sure that you upgrade if you have any things, the package is not there, PIP will actually not install it. You will say, I have PIP, it doesn't even check what version it is. It says, I have my version, PIP is already there. I do not actually evaluate that you wanted PIP 11. So, then if I need to specify the dash capital U, which basically means upgrade and then it will force make sure that it upgrades whenever it installs that PIP version. Another thing which is, but this is not the true, for example, dependencies. For example, if you install a library and the library's dependency is already there, PIP will not actually upgrade it. And because we use PIP, we have this kind of mistake on our side or like this kind of vulnerability on our side, which bottom line falls down to that whenever you add new dependencies to your project, talks will not actually upgrade your or pull in your new version of the dependency. It will add new dependencies, but will not actually, if you just bomb the version requirement, it will not actually ensure that the bigger version is installed. Okay? So, the workaround for now is actually just force the recreating the environment. You can pass in the dash R flag and this basically means, by the way, recreate the environment from scratch and that ensures it. The medium plan goal for that is that once PIP will actually support installing or like having dependency graphs and evaluating that kind of things, this should be also solvable. Okay. Another thing is that the build dependency assets beforehand, this probably should have some solution in the next few months. And, but let's see what strengths we have. So, what we can use. Basically, you can make sure that you can add new tools to your project and users can run those projects locally without actually needing to teach them the new tools, only if they actually want to run those or like they actually want to debug those new tools then they have to learn about it. It allows you to run your test sheet on your multiple Python versions without having to manually create all the virtual environments for the Python versions. Plus, it allows you to run locally exactly what would run on the CI. Okay? So, this means that if, and as I said before, like, if anything breaks on the CI, it's easy to reproduce it locally. And finally, it is just a Python package meaning it runs on any environment. So, for example, your contributor or user maybe it's easy for them to get on board because now they, for example, compared to make, they don't have to install something which only works on unit system. Okay? And it has a, hopefully, a readable configuration instead of some ugly, ugly configuration. Okay. So, finally, one thing which we have like similar as PyTest has, we also have a plug-in system. So, if you want to do anything which doesn't work out of BoxingTux, you can do it by having a plug-in. For example, you can see that there are plug-ins out there and quite popular which basically swap out the Python setup, PyTing, for example, to using CMake to build in the project or use the running Docker as a tool, as an environment where you can have virtual environments. So, it's also kind of like a virtual environment. So, it's easily to plug in those kind of things but there are a lot of other things. For example, there's also in development a package which will allow you to use Kanda if you're more like a Kanda developer and use the environment from there. Okay. Thank you very much. And now I'll take your question. Thank you for talking. We have about 50 minutes. I have a question. Who has questions about talks? What tools do you have any thoughts about SQL integration or integration with databases? Or is it just a purely Python specific thing? Can you just write your own tool to go off and populate your Postgres database with test data, et cetera? And how do you extend it, I guess, is the question, is the broad question? Depends on what you're writing. I would say probably the easiest what you're speaking about is basically, I suppose, database dependencies. I would say probably what you're looking for is you can set three different kind of fixtures which set a different kind of databases and then you can specify, for example, that different environment generated, use different kind of fixtures which set up the database. And that could, I think, fairly reasonably work and you can do like that, for example, if you want to test both with Oracle and MySQL, that would make it possible. I'm not sure that you could probably also do it without having the texting direction. You could probably also just have parameterized stress in PyTest. I think it's personal preference if you prefer. All right. Thanks for the talk. Would you know if during the environment creation stage, would you be able to specify requirements.txt file or something like that? Yeah. As I said, like in the dependencies, you can actually specify both the requirements takes the or constraint takes the, which is another thing which we introduced. As opposed to what's in setup.txt. Sorry? As opposed to what's specified in setup.txt. So setup.txt is only required if you have a library. Usually for libraries, I think the best practice is to use setup.txt to specify dependencies. Requirements takes is more popular for applications. Like the requirements takes were developed for applications. If you have an application, then you don't have a build process. You only have the dependent, so the environment creation, in that case, you can specify the requirements takes the easily just pass it on as instead of dependently. I think at one point you mentioned running things in parallel, doing multiple environments at once. I don't remember seeing talks do that before. How do you? It's a, so at the moment it's not built in, meaning that you need to run detox, which is also lives under the talks organization, but it basically is just a driver which allows you to run multiple text environments in parallel. It's an higher abstraction. So yeah, it's not built in. But if you look up detox, should be what you want. Will detox go as far as running a couple projects in parallel? Or is it still strictly one project at a time? Mostly because your different projects might have different configurations. That being said, there is no real reason why we could not like have parallel configuration parsing and running. I don't think anyone actually raised the need for that. It was so strongly that they actually wanted to develop it. Thank you for that. Just a question you mentioned, you can make a variation on the jungle dependency in your project there. In our case, we have dozens of dependencies. Usually we ship with every dependency has been fixed to a specific version. And what is interesting is to see for many of those, can we upgrade because they're fixed in time and we realize three months or six months later, maybe we should upgrade to the latest and not be stuck in the past. And we want to explore the versions we can now use. How could we leverage that kind of tool to explore how we can upgrade our dependencies without breaking completely? Because sometimes they collide and it doesn't work anymore. So in my view, that would be a requirement that is fixed to specific versions and one other requirement that would be more flexible and say, okay, I just... Yeah, I mean, if you can see this example, for example, you can see that Django 1.8 is actually fixed to Django 1.8. But for example, the master is actually master. I could easily just specify Django latest, which would be like... What have... Without any dependency specification, right? We need to install the latest. That's one. How do you cope with 80 dependencies? Oh. I mean, technically speaking, you could do it with similar... Like, this kind of factor expressions that... Because we call them factor expressions, they're at the top. They are put there even like the dependencies you can put it in. So it allows you to do even more deeper kind of these kind of specifications. But just a word of caution, if you have like 10 dependencies, you're talking about factorial of 10. So I'm not sure you would actually run that in real time. Probably usually, like in practice, what we saw is usually people want a few major things and maybe the latest. And then when that breaks, according to that, you may be altering your... Like, for example, specifying that, by the way, our package no longer works with... Because it broke on the old... When I installed all the latest, it broke with this package's latest. So I'm just pinning that to a given version until someone takes up the task of fixing what, why it doesn't work now with the latest. Because otherwise, you just have too many... The space is too big. And even if your test run five seconds, you're already looking at like probably next year when one of the running finishes. Okay. Any more questions? So we don't want to wait until next year. So we have break now and have lunch. Enjoy lunch. And thank you very much. Thank you very much. Thank you.
 Thank you guys for coming for this talk. So today we're going to learn something about  talks mainly, but before I learn it, it's actually a tool which has the bold aim to actually  standardize testing inside Python. And we already have tests, you may say. Everyone  heard about PyTest, but if you like a lot of the, one of the GitHub repositories, for  example, for the PyTest tools, but it's if you see that down there, there is a talk  that in a file, meaning it PyTest itself uses talks, or you can go it, if similarly does  that or there are other tools such as, for example, request also does similarly use the  talks. So if you come and start one to contribute to this project, you might get the question,  so why is this talks all around and what actually this purpose is? Hopefully by the  end of this session, you will have a good understanding of what talks tries to do, why  it's needed, and what can it do for you, how can you use it, how it works. Okay? So let's  start with the first question, why it is needed. Yeah, just to brighten up the mood  ever so often, I put in a few bugs here and there, so that's for that. Okay, so let's  start with like, generally, you have your Python application, what kind of parts Python  application nowadays has. So first and foremost, you have your business logic, this basically  boils down to your actual source files, this is what your manager writes for you to actually,  or pays for you to do. Okay? Then we have the test, which are basically for yourself,  so you can actually ensure that whatever your manager asks, it's actually doing whatever  you wrote. Then you have some packaging, because usually whenever you create an application  or even like a library, you actually want to package it and deploy it to the, where it  will run or to the other users. So you need to have packaging, then you need to also have  documentation, because someone else when it comes back, still need to understand the project.  Then you have nowadays also quite popular type checks, for example, my pipe, I try to  actually check if anything's okay or not in your source code. And then you have also like  set it code analysis, this is think of tools such as Flake or Pep 8 or Piling. And finally,  usually when, especially if you're writing a library, this is true, you want to support  that library for more than one Python versions, so that everyone can be happy and start using  it. And therefore, you actually want to test that whatever you're creating works under  multiple Python socares. And then there's another dimension, which is mostly used, for  example, the Django world, where you have multiple versions of Django. So if you write  a Django plugin, you want to ensure that it runs with all types of Django versions, okay?  So these are basically complications, which means that instead of the like beforehand,  you maybe had to write one test. Now you're that one test instead now you have like basically  10 tests or 12 tests or similar, a lot more tests, which you want to basically run. And  additional thing which is usually nice whenever someone comes to your project and they want  to like quickly get started with the project, they don't want to read two pages of how to  get started developing. They usually would be nice if they can just have a single comment  which they can use to actually set up the entire developer environment, so to say. And  obviously you want to ensure that all these kind of configurations and all these kind  of steps and all these kind of checks in your application actually pass after each commit.  So the next people who come and start contributing something to your project, it actually can  start with a clean slate and doesn't have to first repair whatever is broken in the project.  Luckily we live inside the great ecosystem of Python. This means that we have a lot of  people who are eager to open source their solutions for various solutions, for various  problems. And what this means that for example, we obviously have a tool for everything. For  test we have pie test, nose test for packaging. Now this is quite for libraries that is set  up to split or poetry, but for applications like Pax or Ser or things which are nowadays  quite popular or the documentation, both things and make docs is widely used inside  the Python ecosystem. As I said before, and static analysis, style checks like black,  piling, flint also make it. Then we have all the ways we can create parts of a virtual  environment which basically helps you test multiple Python versions and then you also  have, you can use this also to test the virtual dependency types. For example, you want to  test again Django one and Django two in the same time. Okay. So now basically you can  set up for your project, use one of these tools and make sure and you use one of these  tools and basically once you use that tool, everyone can ensure after every commit that  everything runs. Okay. But the downside is that now basically you introduced your project  10 tools and everyone who wants to basically just maybe contribute even a little might need  to actually learn all those 10 tools and understand how they need to be used to actually  be able to work with the project. Okay. So this is not really a good place to be because  basically now you mean that whoever comes to your project, you will have a really steep  learning curve to actually join the project and actually understand why all this test  are needed and basically you will have some really sad users when they want to contribute  to the project because they will not actually understand why something, for example, my  fail, some checks my failing in the CI for which he locally did not fail. For example,  a quite common example is that for example, he messes up the formatting or some static  checking and even to locally test past, it will still mean that the CI will broke on  the remote and these generally leaves contributors needing to come back all the time to whatever  they started to create. So the idea would be that it would be nice if you could somehow  bring down the bridge or build the bridge between the CI and the local development environment  and somehow it could basically be possible to run the same thing that we run the CI run  also locally. Okay? So as I said beforehand, GitHub advertises creating the contributing  file which basically means someone has to read it and understand it and then actually  use it. Then you have the option you may probably at one point you will get fed up with copying  or uncomment so you will start writing shell scripts to actually contain all this kind  of test or if you're really hard core, you go and write a make file which basically does  the same, it's more like an advanced shell script. Okay? So here's just a good example  how one of the shell scripts could look. So you basically, you want to basically start  with just checking that if anything fails, stop, don't continue and then basically start  packaging your project, installing it, creating a virtual environment, running your test and  if everything, once all these comments are run, you probably everything will also pass  on the CI side. Okay? Now, as I said beforehand, this, in this example, it's only for one  Python which is like a system Python would be nice to add virtual environments because  what if you need to support both for Python 3.7 and 3.6? So also, you want to run all  checks regardless and actually print maybe at the end of the report because here I did  not end, I did not actually address the point of maybe it would be nice in the end, we would  print out all the things which failed and because that's what the usual actually would  need to address and it would also be nice if you could actually run this in parallel  because basically many of these tools can run in parallel because they're not depending  on each other. Okay? So as far as going to actually testing multiple Python, you can  use the virtual and for the VM which is added to Python core libraries and basically you  can create your own lightweight side directory and then keep installing your package into  it. We, this you can use basically by activating but you don't have to actually activate it.  If you really want, you can, and you can spell out the full path, you can use the full path  to actually run things inside the given virtual environment. Okay? Obviously, this now requires  you to actually remember what is the path for your virtual environment or this basically,  once we add this just for one virtual environment, this kind of our best script now becomes the  stand-lines which makes all, still does not actually do any reporting when also just hand  there's one single Python version, so you probably want more than that. So one can see  that this point that this kind of writing this kind of magic script which run every  test basically, every kind of tools starts to be hard to actually maintain. It also looks  error-prone because if you can do anywhere in type of the error if you want to add another  environment, it's like you have to type a lot of other, it probably will incorporate  a lot of copying around. So this sort of seems like there must be a better way with  this. And what we want is happy users. We want them to be jumping around in our project,  so let's try to do something like that. And this is the Toxas Dream. Toxas actually started  out by the same developers who started out PyTest. They envisioned the testing, the test  run or more or less to be Tox and PyTest to be actually the test frame, the testing framework.  But you can, they, they, they didn't those to be the tool which actually drives which  people start interacting with. Okay. So how can we achieve merging the CI and Shell-based  testing? Okay. So the idea is that we should be able to run locally everything which we  run on the CI. This is all especially important because in anything breaks on the CI, you  want to be able to reproduce it locally because that's how you, that's where you can actually,  for example, start debugging interaction and understand what's happening. Okay. So Toxas  basically allows for Python project to define all tools in one place, meaning that all kind  of tools that you may use, you may add checks for it. You can define it in one single configuration  file with their actual configurations. Furthermore, you also have a central way of invoking them.  When we talk about central way of invoking them, think of it like make targets. That's  probably the best approximation or generally just build targets. And finally, you should  be ensuring, you should be creating an environment which is isolated and reproducible, so to  say. So when you run it, then it should be. And by isolation, we don't just necessarily  mean virtual environment. We also mean, for example, shielding you from whatever you have  in your user operating system environment. Okay. So it is basically a CLI tool, meaning  if you actually, to actually use Toxas, you don't need anything. You just need Python  pip and then just install it. And then you can invoke it with, for example, Python minus  and Tox. And that, if the user should work it and should be able to run all the testing  without actually remembering what are the comments. Okay. And it basically manages, creates and  runs the reproducible environment. So this is what Tox actually does for you. It basically  will ensure it has a look at your configuration and will try to recreate your own, your, recreate  a testing environment for a given target exactly as you defined it. Okay. So let's see how  it actually works. This is like a flow diagram which shows what Tox actually does. It initially  has like a simple configuration phase. This is basically just putting together everything  you have in your configuration file and whatever you're passing from the command line plus  whatever may be inside the operating system environment. Then there is this, some of the  steps are optional. So you don't have to do all the steps. For example, the packaging  is optional. If you have a library, you also want to, for example, package your library  because when you want to test it, you actually want to test the package version, whatever  the end user would use, not what your local file system contains. Okay. And, but as I  said, this is basically packaging. So you can also use it for your applications. So  once the packaging is done, it will, for each of your targets, it will define a virtual  environment. And this virtual environment will face over three phases. It will have an environment  creation, an install phase, and finally it will have a run comments phase. And once all  the comments have run, it will have a reporting. Okay. So this is, for example, a simple configuration.  I'm not sure how good it is. I'm going to try to zoom in. It shows a bit better. So  for example, here is a simple project. I defined three environments. For example, one of my  environments is basically just running the test under Python 2.7, for example, here.  And then I have a linting, a documenting, and a publishing target. Also, this target,  for example, the linting, because you use a spiling 2.0 requires Python 3 plus. So I  specify to you like Python 3.7. My documentation does the same because maybe inside my conf  py, I actually want to use the fancy f strings and all that things. So it is possible to  specify that for the documentation, it's a different Python environment or a different  version. And then I also defined in the end a publishing. For example, I don't have to  actually remember how to publish everything I just defined there that for me, publish  basically means have an environment which has both set up tools and one. And then just  basically run the set up py as this baby skill and then use twine to actually publish it.  Okay? So this means that now basically whenever I need to do publish, I don't have to any  more remember whatever I just call the publish target and it will work and it will create  for me and make sure that I have the correct twine versions and that kind of things. Okay?  The first thing which I have there in like the first time when I see emblist, that is  basically the emblist which will run always. So if the user doesn't specify what should  run, it will run those environments in dent order more or less. But as I can see, things  which I don't need to run always, I can define it as additional one and they are for me to  be run as on demand. For example, the publishing is something we probably don't want to do  it after every commit. So that's something that is not actually defined inside the default  environment. So to say which are run by default. Okay? So, yeah. As you can see, each of them  are like separately defined. They have their own section where you can define not just  the dependencies. You can also define their actual arguments that you want to pass on  to those kind of configurations. And this allows you, for example, you can now have  all your configuration in one file because most of the tools provide command line environment  flex and you don't no longer necessarily need to have custom configuration files laying  around like five, four in your project for each different kind of project. Okay? For  example, this will be the actual documentation building and you can see it even generates  a unique build directory for it. The user can actually then see the output if they want  to see it. And yeah. Now let me zoom back. Okay. So let's go to the next one, the packaging.  So the packaging is basically, it creates an installable package. This is the only  in the 20 you have a library. The build dependencies at the moment are not actually insured, meaning  that from wherever the environment you're actually invoking to needs to have all the  build dependencies. Now this is a temporary thing because I wrote here long term but probably  is more like in three months or so. The plan is that once PIP will have PEP 517 and 518  support, we will actually be able to specify our project or build dependencies inside those  files and we'll be able to actually insure also that you can actually install, like people  can insure that you have all your build dependencies. Like build dependencies are, for example,  the setup tools that you want to use but also like additional ones which are actually doing  the build but are required to do the build. For example, setup tools SCM is one of the  popular which you can use to like automatically version your project. This is also like build  dependency but it's not actually a build tool. Okay? So the next step, environment creation.  This basically wants to be optimized meaning that if the environment already exists, you  already run the comment beforehand. It will not be created from scratch. It will reuse  it. Now as I, as you can remember beforehand, like here, I had here at the start section  where I defined what kind of Python I want. So this means that there must be a way somehow  the application or the tool needs to figure out what Python's do you have on your computer  and what actually interpreter where it lives. So for this one, we actually use the system  path on Linux. So basically, we try to see if there is a Python 3.7 executable in your  system. Okay? Obviously, this doesn't work on Windows. On Windows you need to use the  PyExe. For this kind of interpreter discovery, there are two perhaps which is basically detail  how Python discovery should work on different platform basically like a unique something  in those and the tool follows basically those two ideologies. And we use virtual and the  create virtual environments for the given version. And then at the end of it, there is  an install dependencies meaning that you basically we just whatever you specify. You can specify  explicitly the dependency. One of the things we install the dependency is specify here.  This supports whatever PEP440 supports meaning that you can also specify like requirements  to the XT file. They basically just get forwarded to your installer which by default is PIP.  But for example, someone can use PEPAMP or whatever other they might want to use. And  then there are also other dependencies after we install the project dependencies or the  ones that specify this for the tool. Then there's another set of dependencies to the  itself. And this also means everything specified inside the install requires you'll also be  installed in the second phase. Okay? Now there is a slight, yeah, the default is  PIP but can be changed real tools. Okay? So once you installed your project, you have  it available in a fresh environment just as it would be installed basically at the user  end. If they would use it, you then the following things will actually run your comments. Okay?  So for this kind of comments, for example, here we do T things or three things. There  are T comments to be run. The first comment is basically let's just build the documentation.  The second thing is basically let's see if all the links are valid inside the documentation.  And then the final one is just print out for the user where you can find the documentation  and view it. Okay? One of the important things that we do for running the comments is that  we basically strip away most of your operating system environment variables. And this is  quite helpful in a lot of the cases, especially when you're relying on some environment variables  or your project has some environment variables which drives its functionality. It is often  the case that you set it locally but then forget it that on the other users machine  that not may be set. So this ensures that you can already fail on this kind of environments  not set case locally and not have to actually wait until other users reported that by the  way on my system that doesn't work. Okay? And basically the chain of comments will run  it one after another and whenever one fails, we stop. And if any of the comments fail,  that environment is marked as being a failure. Okay? And we stream basically whatever your  tool puts out on the output. This can be configured basically but by default we stream out. This  allows you to actually see what the tool output was so you can then take action based on that.  Okay. So, and the final thing is we just print a nice report. For example, in this case you  can see that two environments failed. Basically the test and the linking but the documentation  generation finished its success. Okay? So basically this is like the workflow of Tux.  This is what it does you. It basically ensures you that you have a virtual environment which  is as specified inside your configuration. It automatically sets its up. It will package  your project. It will in the end allow you to actually print out whatever happened when  you ran it. The idea is that for example, if some new user joins the project, they don't  have to say like, so how do I run the test? Basically they can just invoke Tux and that  will print out and that will do everything for them and basically have a local CI if  they need it. Okay. Let's see some use cases or how it actually works in practice. Okay?  So for example, this is a default and you can see that this actual call might take some  time because this is creating it on a fresh state. You can see that we have three targets.  We have basically the same, the testing, the linking and the documentation phase and we  will start creating this phase as one by one once we just invoke the Tux and it will invoke  Python with the correct argument. It will ensure that the environment has been created  and whenever any error happens, we immediately display it for the user so they can act on  it. Okay? Now, the thing is this usually takes a lot faster. The reason why it now it is  a bit slower is because basically we have create operations there and creating a virtual  environment and installing dependencies because network is involved and I was involved is  not as fast as many of the things. So at first time, it takes longer but for example, when  you invoke it the second time, then basically we don't have to do anything then just re-invoke  the comments that you have because all the dependencies that already installed and are  in place. Okay? So it is obviously possible that you can run also just a subset of environments.  For example, the cases you might only want to test what actually modified the source  code. For example, documentation generation, you don't really care what you're actually  developing or for example, another popular use case of this is that you want to run your,  which you have for example, testing against two Python environment and you want to ensure  that for example, you run the test for Python 2.7 and 3.6 and then you want to have another  step after it which basically calculates the aggregated coverage reports that you can do  with this kind of way because you can specify that run this to Python environments and then  display an aggregated coverage for those two. Okay? Okay, another thing that is quite nice  is that you can actually document whatever the targets are so the users don't have to  actually guess. So you can add descriptions to it and then the user when, for example,  this is for our own toxin if I basically when someone says, okay, what can I do with this  project? They can just list whatever the possible targets are and they can read if they want  to run any of them or not. Okay? You can see that there are default environments which  are always run and then additional ones which are on demand basically. Another option is  that you support the post-arts. I'm sure. So basically if you testify for your comment  that you want to post our Gantt's end, this means that if you use the double dash and  after a double dash, you can basically forward arguments to your underlying tool. This is  very helpful. For example, in this case, I wanted to run the test on the Python 2.7 and  if anything fails, I wanted to enter into the debugging, for example, PDB. Okay? So  I did not, I could basically just pass them on to the invocation after the double dash  as additional arguments and they automatically get forwarded. This is also a nice way where,  for example, you can have more verbose test reports and all those kind of things which  are basically more like you want to understand more deeply what something broke and not just  run the default test with so to say. Okay? Another option which we have there is basically,  we have another project which is like detox. This basically allows you to run things in  parallel. For example, it is possible to run all the environments in parallel and then  display an aggregated output. Now, obviously the limitation of this is that we can no longer  stream the output in real time because if we would do, basically, you would mean that  five environment, five tools run in parallel and all five tools would print the character  and the screen would be quite hard to actually see what's on the output. Okay? So another  quite, you can actually create a developer environment. Why don't I say developer environment  and this is especially true for libraries. It means that we don't actually install your  package as the user would install it but we install your package using PPS in developer  mode which basically means that people not actually install your package by putting it  into the site package but creates a same link and says that the source code lives in your  wherever you're developing and then the whenever Python is running, it will actually pick up  your local files and not the remote, not the site package files. Okay? Okay. Now, just  one thing, once you start creating this kind of environment or generally, it is quite  healthy for example, in your environment, you can quickly select one of the environments  to be the one for developer ecosystem to use. For example, in this case, for Python, there  is this PyMVmanage plugin which is quite nice which allows you to basically just right click  on any of the environments, be made talk for any other virtual environment and say, but  by then I'll change the interpreter package to this Python which is inside this folder.  Okay? So, another thing which is quite popular is like obviously, especially like big projects  care about 45 Python versions. You might not have all the Python versions locally on your  machine so it doesn't make sense to force failing those environments. You don't have  Python. You can specify that only you run environments which for which we can actually  find the correct Python version. For example, don't try to run Python 3.5 if that's not  on the system because it will fail but it's not actually a failure that the test will  be just that we could not, we have dependently missing for it. Okay? Another helpful thing  is this is especially helpful for like bigger projects or like Django project is that Django  projects historically have like a lot of versions and you want to check a lot of combinations  and ensure that all the combinations actually work. It allows you to basically specify and  you can see in three lines all that kind of Django combinations and there, for example,  you can see that, for example, the Django master actually can specify even like GitHub  links and it will install the latest from GitHub and try to test it again. Okay? So,  as you can see, basically, we just tend like so of configuration. Now you can test against  like 12 combination of dependently that your users might run into. Okay? Now obviously,  you might not necessarily want to run all of this locally when you develop but this  can work something in the CI and whenever something works in the CI, you locally, you  don't have to manually recreate the environment. You can just say, by the way, on the CI, I  saw that the Py37 Django master failed. Let me just recreate that and run the test only  in that and try to debug what actually is the underlying issue for that. Okay? So, CI  integration. As I said, the whole concept of it is that we would prefer to not be CI  agnostic, meaning that we want the CI to basically don't have to actually specify all these.  The CI should basically just invoke the same thing as the user would locally basically  just invoke docs. And this means that your CI configuration, if you use docs, beside  the user actually having the option to easily set up locally any kind of environment. Now  the CI, you don't have to use the property CI environment but you can basically just  say that, oh, by the way, make sure that I have these pythons in that CI and then finally  install them with, install, for example, my docs and then just run docs and that's the  entire CI configuration. Okay? So, let me sum up a bit what I tried to present here today  and what we learned. So, the first thing is obviously there are weaknesses of the tools.  The biggest weaknesses of what people often get confused is that dependency updates. The  way PIP works, if you say to PIP that install PIP 11 and unless you specify that, by the  way, please make sure that you upgrade if you have any things, the package is not there,  PIP will actually not install it. You will say, I have PIP, it doesn't even check what  version it is. It says, I have my version, PIP is already there. I do not actually evaluate  that you wanted PIP 11. So, then if I need to specify the dash capital U, which basically  means upgrade and then it will force make sure that it upgrades whenever it installs  that PIP version. Another thing which is, but this is not the true, for example, dependencies.  For example, if you install a library and the library's dependency is already there,  PIP will not actually upgrade it. And because we use PIP, we have this kind of mistake on  our side or like this kind of vulnerability on our side, which bottom line falls down  to that whenever you add new dependencies to your project, talks will not actually upgrade  your or pull in your new version of the dependency. It will add new dependencies, but will not  actually, if you just bomb the version requirement, it will not actually ensure that the bigger  version is installed. Okay? So, the workaround for now is actually just  force the recreating the environment. You can pass in the dash R flag and this basically  means, by the way, recreate the environment from scratch and that ensures it. The medium  plan goal for that is that once PIP will actually support installing or like having dependency  graphs and evaluating that kind of things, this should be also solvable. Okay. Another  thing is that the build dependency assets beforehand, this probably should have some  solution in the next few months. And, but let's see what strengths we have. So, what  we can use. Basically, you can make sure that you can add new tools to your project and  users can run those projects locally without actually needing to teach them the new tools,  only if they actually want to run those or like they actually want to debug those new  tools then they have to learn about it. It allows you to run your test sheet on your  multiple Python versions without having to manually create all the virtual environments  for the Python versions. Plus, it allows you to run locally exactly what would run on the  CI. Okay? So, this means that if, and as I said before, like, if anything breaks on  the CI, it's easy to reproduce it locally. And finally, it is just a Python package meaning  it runs on any environment. So, for example, your contributor or user maybe it's easy for  them to get on board because now they, for example, compared to make, they don't have  to install something which only works on unit system. Okay? And it has a, hopefully, a readable  configuration instead of some ugly, ugly configuration. Okay. So, finally, one thing  which we have like similar as PyTest has, we also have a plug-in system. So, if you want  to do anything which doesn't work out of BoxingTux, you can do it by having a plug-in. For example,  you can see that there are plug-ins out there and quite popular which basically swap out  the Python setup, PyTing, for example, to using CMake to build in the project or use  the running Docker as a tool, as an environment where you can have virtual environments. So,  it's also kind of like a virtual environment. So, it's easily to plug in those kind of things  but there are a lot of other things. For example, there's also in development a package which  will allow you to use Kanda if you're more like a Kanda developer and use the environment  from there. Okay. Thank you very much. And now I'll take your question.  Thank you for talking. We have about 50 minutes. I have a question. Who has questions about  talks? What tools do you have any thoughts about SQL integration or integration with  databases? Or is it just a purely Python specific thing? Can you just write your own  tool to go off and populate your Postgres database with test data, et cetera? And how  do you extend it, I guess, is the question, is the broad question?  Depends on what you're writing. I would say probably the easiest what you're speaking  about is basically, I suppose, database dependencies. I would say probably what you're looking for  is you can set three different kind of fixtures which set a different kind of databases and  then you can specify, for example, that different environment generated, use different kind  of fixtures which set up the database. And that could, I think, fairly reasonably work  and you can do like that, for example, if you want to test both with Oracle and MySQL,  that would make it possible. I'm not sure that you could probably also do it without  having the texting direction. You could probably also just have parameterized stress in PyTest.  I think it's personal preference if you prefer.  All right. Thanks for the talk. Would you know if during the environment creation stage,  would you be able to specify requirements.txt file or something like that?  Yeah. As I said, like in the dependencies, you can actually specify both the requirements  takes the or constraint takes the, which is another thing which we introduced.  As opposed to what's in setup.txt.  Sorry?  As opposed to what's specified in setup.txt.  So setup.txt is only required if you have a library. Usually for libraries, I think  the best practice is to use setup.txt to specify dependencies. Requirements takes  is more popular for applications. Like the requirements takes were developed for applications.  If you have an application, then you don't have a build process. You only have the dependent,  so the environment creation, in that case, you can specify the requirements takes the  easily just pass it on as instead of dependently.  I think at one point you mentioned running things in parallel, doing multiple environments  at once. I don't remember seeing talks do that before. How do you?  It's a, so at the moment it's not built in, meaning that you need to run detox, which  is also lives under the talks organization, but it basically is just a driver which allows  you to run multiple text environments in parallel. It's an higher abstraction. So yeah, it's  not built in. But if you look up detox, should be what you want.  Will detox go as far as running a couple projects in parallel? Or is it still strictly  one project at a time?  Mostly because your different projects might have different configurations. That being  said, there is no real reason why we could not like have parallel configuration parsing  and running. I don't think anyone actually raised the need for that. It was so strongly  that they actually wanted to develop it.  Thank you for that. Just a question you mentioned, you can make a variation on the jungle dependency  in your project there. In our case, we have dozens of dependencies. Usually we ship with  every dependency has been fixed to a specific version. And what is interesting is to see  for many of those, can we upgrade because they're fixed in time and we realize three  months or six months later, maybe we should upgrade to the latest and not be stuck in  the past. And we want to explore the versions we can now use. How could we leverage that  kind of tool to explore how we can upgrade our dependencies without breaking completely?  Because sometimes they collide and it doesn't work anymore.  So in my view, that would be a requirement that is fixed to specific versions and one  other requirement that would be more flexible and say, okay, I just...  Yeah, I mean, if you can see this example, for example, you can see that Django 1.8 is  actually fixed to Django 1.8. But for example, the master is actually master. I could easily  just specify Django latest, which would be like...  What have... Without any dependency specification, right? We need to install the latest.  That's one. How do you cope with 80 dependencies?  Oh. I mean, technically speaking, you could do it with similar... Like, this kind of factor  expressions that... Because we call them factor expressions, they're at the top. They are put  there even like the dependencies you can put it in. So it allows you to do even more deeper  kind of these kind of specifications. But just a word of caution, if you have like 10  dependencies, you're talking about factorial of 10. So I'm not sure you would actually  run that in real time. Probably usually, like in practice, what we saw is usually people  want a few major things and maybe the latest. And then when that breaks, according to that,  you may be altering your... Like, for example, specifying that, by the way, our package no  longer works with... Because it broke on the old... When I installed all the latest, it  broke with this package's latest. So I'm just pinning that to a given version until someone  takes up the task of fixing what, why it doesn't work now with the latest. Because otherwise,  you just have too many... The space is too big. And even if your test run five seconds,  you're already looking at like probably next year when one of the running finishes.  Okay. Any more questions? So we don't want to wait until next year. So we have break  now and have lunch. Enjoy lunch. And thank you very much.  Thank you very much.  Thank you.
Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_0_143.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_144_659.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_660_719.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_720_1535.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_1536_1811.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_1812_7667.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_7668_12419.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_12420_12551.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_12552_13703.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_13704_14543.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_14544_17375.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_17376_17483.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_17484_19355.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_19356_23075.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_23076_23123.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_23124_23279.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_23280_24839.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_24840_24935.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_24936_25691.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_25692_26639.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_26640_26747.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_26748_27983.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_27984_28055.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_28056_28415.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_28416_30707.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_30708_30767.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_30768_30827.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_30828_31151.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_31152_32123.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_32124_34967.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_34968_35711.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_35712_37763.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_37764_38771.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_38772_39611.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_39612_40559.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_40560_40751.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_40752_42359.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_42360_43619.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_43620_43667.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_43668_43811.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_43812_43859.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_43860_43919.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_43920_48323.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_48324_49643.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_49644_56015.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56016_56063.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56064_56111.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56112_56171.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56172_56219.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56220_56267.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56268_56387.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_56388_59855.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_59856_59903.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_59904_59951.png Bernat_Gabor_-_Standardize_Testing_in_Python_SFqna5ilqig_59952_60056.png
Thank you guys for coming for this talk. So today we're going to learn something about talks mainly, but before I learn it, it's actually a tool which has the bold aim to actually standardize testing inside Python. And we already have tests, you may say. Everyone heard about PyTest, but if you like a lot of the, one of the GitHub repositories, for example, for the PyTest tools, but it's if you see that down there, there is a talk that in a file, meaning it PyTest itself uses talks, or you can go it, if similarly does that or there are other tools such as, for example, request also does similarly use the talks. So if you come and start one to contribute to this project, you might get the question, so why is this talks all around and what actually this purpose is? Hopefully by the end of this session, you will have a good understanding of what talks tries to do, why it's needed, and what can it do for you, how can you use it, how it works. Okay? So let's start with the first question, why it is needed. Yeah, just to brighten up the mood ever so often, I put in a few bugs here and there, so that's for that. Okay, so let's start with like, generally, you have your Python application, what kind of parts Python application nowadays has. So first and foremost, you have your business logic, this basically boils down to your actual source files, this is what your manager writes for you to actually, or pays for you to do. Okay? Then we have the test, which are basically for yourself, so you can actually ensure that whatever your manager asks, it's actually doing whatever you wrote. Then you have some packaging, because usually whenever you create an application or even like a library, you actually want to package it and deploy it to the, where it will run or to the other users. So you need to have packaging, then you need to also have documentation, because someone else when it comes back, still need to understand the project. Then you have nowadays also quite popular type checks, for example, my pipe, I try to actually check if anything's okay or not in your source code. And then you have also like set it code analysis, this is think of tools such as Flake or Pep 8 or Piling. And finally, usually when, especially if you're writing a library, this is true, you want to support that library for more than one Python versions, so that everyone can be happy and start using it. And therefore, you actually want to test that whatever you're creating works under multiple Python socares. And then there's another dimension, which is mostly used, for example, the Django world, where you have multiple versions of Django. So if you write a Django plugin, you want to ensure that it runs with all types of Django versions, okay? So these are basically complications, which means that instead of the like beforehand, you maybe had to write one test. Now you're that one test instead now you have like basically 10 tests or 12 tests or similar, a lot more tests, which you want to basically run. And additional thing which is usually nice whenever someone comes to your project and they want to like quickly get started with the project, they don't want to read two pages of how to get started developing. They usually would be nice if they can just have a single comment which they can use to actually set up the entire developer environment, so to say. And obviously you want to ensure that all these kind of configurations and all these kind of steps and all these kind of checks in your application actually pass after each commit. So the next people who come and start contributing something to your project, it actually can start with a clean slate and doesn't have to first repair whatever is broken in the project. Luckily we live inside the great ecosystem of Python. This means that we have a lot of people who are eager to open source their solutions for various solutions, for various problems. And what this means that for example, we obviously have a tool for everything. For test we have pie test, nose test for packaging. Now this is quite for libraries that is set up to split or poetry, but for applications like Pax or Ser or things which are nowadays quite popular or the documentation, both things and make docs is widely used inside the Python ecosystem. As I said before, and static analysis, style checks like black, piling, flint also make it. Then we have all the ways we can create parts of a virtual environment which basically helps you test multiple Python versions and then you also have, you can use this also to test the virtual dependency types. For example, you want to test again Django one and Django two in the same time. Okay. So now basically you can set up for your project, use one of these tools and make sure and you use one of these tools and basically once you use that tool, everyone can ensure after every commit that everything runs. Okay. But the downside is that now basically you introduced your project 10 tools and everyone who wants to basically just maybe contribute even a little might need to actually learn all those 10 tools and understand how they need to be used to actually be able to work with the project. Okay. So this is not really a good place to be because basically now you mean that whoever comes to your project, you will have a really steep learning curve to actually join the project and actually understand why all this test are needed and basically you will have some really sad users when they want to contribute to the project because they will not actually understand why something, for example, my fail, some checks my failing in the CI for which he locally did not fail. For example, a quite common example is that for example, he messes up the formatting or some static checking and even to locally test past, it will still mean that the CI will broke on the remote and these generally leaves contributors needing to come back all the time to whatever they started to create. So the idea would be that it would be nice if you could somehow bring down the bridge or build the bridge between the CI and the local development environment and somehow it could basically be possible to run the same thing that we run the CI run also locally. Okay? So as I said beforehand, GitHub advertises creating the contributing file which basically means someone has to read it and understand it and then actually use it. Then you have the option you may probably at one point you will get fed up with copying or uncomment so you will start writing shell scripts to actually contain all this kind of test or if you're really hard core, you go and write a make file which basically does the same, it's more like an advanced shell script. Okay? So here's just a good example how one of the shell scripts could look. So you basically, you want to basically start with just checking that if anything fails, stop, don't continue and then basically start packaging your project, installing it, creating a virtual environment, running your test and if everything, once all these comments are run, you probably everything will also pass on the CI side. Okay? Now, as I said beforehand, this, in this example, it's only for one Python which is like a system Python would be nice to add virtual environments because what if you need to support both for Python 3.7 and 3.6? So also, you want to run all checks regardless and actually print maybe at the end of the report because here I did not end, I did not actually address the point of maybe it would be nice in the end, we would print out all the things which failed and because that's what the usual actually would need to address and it would also be nice if you could actually run this in parallel because basically many of these tools can run in parallel because they're not depending on each other. Okay? So as far as going to actually testing multiple Python, you can use the virtual and for the VM which is added to Python core libraries and basically you can create your own lightweight side directory and then keep installing your package into it. We, this you can use basically by activating but you don't have to actually activate it. If you really want, you can, and you can spell out the full path, you can use the full path to actually run things inside the given virtual environment. Okay? Obviously, this now requires you to actually remember what is the path for your virtual environment or this basically, once we add this just for one virtual environment, this kind of our best script now becomes the stand-lines which makes all, still does not actually do any reporting when also just hand there's one single Python version, so you probably want more than that. So one can see that this point that this kind of writing this kind of magic script which run every test basically, every kind of tools starts to be hard to actually maintain. It also looks error-prone because if you can do anywhere in type of the error if you want to add another environment, it's like you have to type a lot of other, it probably will incorporate a lot of copying around. So this sort of seems like there must be a better way with this. And what we want is happy users. We want them to be jumping around in our project, so let's try to do something like that. And this is the Toxas Dream. Toxas actually started out by the same developers who started out PyTest. They envisioned the testing, the test run or more or less to be Tox and PyTest to be actually the test frame, the testing framework. But you can, they, they, they didn't those to be the tool which actually drives which people start interacting with. Okay. So how can we achieve merging the CI and Shell-based testing? Okay. So the idea is that we should be able to run locally everything which we run on the CI. This is all especially important because in anything breaks on the CI, you want to be able to reproduce it locally because that's how you, that's where you can actually, for example, start debugging interaction and understand what's happening. Okay. So Toxas basically allows for Python project to define all tools in one place, meaning that all kind of tools that you may use, you may add checks for it. You can define it in one single configuration file with their actual configurations. Furthermore, you also have a central way of invoking them. When we talk about central way of invoking them, think of it like make targets. That's probably the best approximation or generally just build targets. And finally, you should be ensuring, you should be creating an environment which is isolated and reproducible, so to say. So when you run it, then it should be. And by isolation, we don't just necessarily mean virtual environment. We also mean, for example, shielding you from whatever you have in your user operating system environment. Okay. So it is basically a CLI tool, meaning if you actually, to actually use Toxas, you don't need anything. You just need Python pip and then just install it. And then you can invoke it with, for example, Python minus and Tox. And that, if the user should work it and should be able to run all the testing without actually remembering what are the comments. Okay. And it basically manages, creates and runs the reproducible environment. So this is what Tox actually does for you. It basically will ensure it has a look at your configuration and will try to recreate your own, your, recreate a testing environment for a given target exactly as you defined it. Okay. So let's see how it actually works. This is like a flow diagram which shows what Tox actually does. It initially has like a simple configuration phase. This is basically just putting together everything you have in your configuration file and whatever you're passing from the command line plus whatever may be inside the operating system environment. Then there is this, some of the steps are optional. So you don't have to do all the steps. For example, the packaging is optional. If you have a library, you also want to, for example, package your library because when you want to test it, you actually want to test the package version, whatever the end user would use, not what your local file system contains. Okay. And, but as I said, this is basically packaging. So you can also use it for your applications. So once the packaging is done, it will, for each of your targets, it will define a virtual environment. And this virtual environment will face over three phases. It will have an environment creation, an install phase, and finally it will have a run comments phase. And once all the comments have run, it will have a reporting. Okay. So this is, for example, a simple configuration. I'm not sure how good it is. I'm going to try to zoom in. It shows a bit better. So for example, here is a simple project. I defined three environments. For example, one of my environments is basically just running the test under Python 2.7, for example, here. And then I have a linting, a documenting, and a publishing target. Also, this target, for example, the linting, because you use a spiling 2.0 requires Python 3 plus. So I specify to you like Python 3.7. My documentation does the same because maybe inside my conf py, I actually want to use the fancy f strings and all that things. So it is possible to specify that for the documentation, it's a different Python environment or a different version. And then I also defined in the end a publishing. For example, I don't have to actually remember how to publish everything I just defined there that for me, publish basically means have an environment which has both set up tools and one. And then just basically run the set up py as this baby skill and then use twine to actually publish it. Okay? So this means that now basically whenever I need to do publish, I don't have to any more remember whatever I just call the publish target and it will work and it will create for me and make sure that I have the correct twine versions and that kind of things. Okay? The first thing which I have there in like the first time when I see emblist, that is basically the emblist which will run always. So if the user doesn't specify what should run, it will run those environments in dent order more or less. But as I can see, things which I don't need to run always, I can define it as additional one and they are for me to be run as on demand. For example, the publishing is something we probably don't want to do it after every commit. So that's something that is not actually defined inside the default environment. So to say which are run by default. Okay? So, yeah. As you can see, each of them are like separately defined. They have their own section where you can define not just the dependencies. You can also define their actual arguments that you want to pass on to those kind of configurations. And this allows you, for example, you can now have all your configuration in one file because most of the tools provide command line environment flex and you don't no longer necessarily need to have custom configuration files laying around like five, four in your project for each different kind of project. Okay? For example, this will be the actual documentation building and you can see it even generates a unique build directory for it. The user can actually then see the output if they want to see it. And yeah. Now let me zoom back. Okay. So let's go to the next one, the packaging. So the packaging is basically, it creates an installable package. This is the only in the 20 you have a library. The build dependencies at the moment are not actually insured, meaning that from wherever the environment you're actually invoking to needs to have all the build dependencies. Now this is a temporary thing because I wrote here long term but probably is more like in three months or so. The plan is that once PIP will have PEP 517 and 518 support, we will actually be able to specify our project or build dependencies inside those files and we'll be able to actually insure also that you can actually install, like people can insure that you have all your build dependencies. Like build dependencies are, for example, the setup tools that you want to use but also like additional ones which are actually doing the build but are required to do the build. For example, setup tools SCM is one of the popular which you can use to like automatically version your project. This is also like build dependency but it's not actually a build tool. Okay? So the next step, environment creation. This basically wants to be optimized meaning that if the environment already exists, you already run the comment beforehand. It will not be created from scratch. It will reuse it. Now as I, as you can remember beforehand, like here, I had here at the start section where I defined what kind of Python I want. So this means that there must be a way somehow the application or the tool needs to figure out what Python's do you have on your computer and what actually interpreter where it lives. So for this one, we actually use the system path on Linux. So basically, we try to see if there is a Python 3.7 executable in your system. Okay? Obviously, this doesn't work on Windows. On Windows you need to use the PyExe. For this kind of interpreter discovery, there are two perhaps which is basically detail how Python discovery should work on different platform basically like a unique something in those and the tool follows basically those two ideologies. And we use virtual and the create virtual environments for the given version. And then at the end of it, there is an install dependencies meaning that you basically we just whatever you specify. You can specify explicitly the dependency. One of the things we install the dependency is specify here. This supports whatever PEP440 supports meaning that you can also specify like requirements to the XT file. They basically just get forwarded to your installer which by default is PIP. But for example, someone can use PEPAMP or whatever other they might want to use. And then there are also other dependencies after we install the project dependencies or the ones that specify this for the tool. Then there's another set of dependencies to the itself. And this also means everything specified inside the install requires you'll also be installed in the second phase. Okay? Now there is a slight, yeah, the default is PIP but can be changed real tools. Okay? So once you installed your project, you have it available in a fresh environment just as it would be installed basically at the user end. If they would use it, you then the following things will actually run your comments. Okay? So for this kind of comments, for example, here we do T things or three things. There are T comments to be run. The first comment is basically let's just build the documentation. The second thing is basically let's see if all the links are valid inside the documentation. And then the final one is just print out for the user where you can find the documentation and view it. Okay? One of the important things that we do for running the comments is that we basically strip away most of your operating system environment variables. And this is quite helpful in a lot of the cases, especially when you're relying on some environment variables or your project has some environment variables which drives its functionality. It is often the case that you set it locally but then forget it that on the other users machine that not may be set. So this ensures that you can already fail on this kind of environments not set case locally and not have to actually wait until other users reported that by the way on my system that doesn't work. Okay? And basically the chain of comments will run it one after another and whenever one fails, we stop. And if any of the comments fail, that environment is marked as being a failure. Okay? And we stream basically whatever your tool puts out on the output. This can be configured basically but by default we stream out. This allows you to actually see what the tool output was so you can then take action based on that. Okay. So, and the final thing is we just print a nice report. For example, in this case you can see that two environments failed. Basically the test and the linking but the documentation generation finished its success. Okay? So basically this is like the workflow of Tux. This is what it does you. It basically ensures you that you have a virtual environment which is as specified inside your configuration. It automatically sets its up. It will package your project. It will in the end allow you to actually print out whatever happened when you ran it. The idea is that for example, if some new user joins the project, they don't have to say like, so how do I run the test? Basically they can just invoke Tux and that will print out and that will do everything for them and basically have a local CI if they need it. Okay. Let's see some use cases or how it actually works in practice. Okay? So for example, this is a default and you can see that this actual call might take some time because this is creating it on a fresh state. You can see that we have three targets. We have basically the same, the testing, the linking and the documentation phase and we will start creating this phase as one by one once we just invoke the Tux and it will invoke Python with the correct argument. It will ensure that the environment has been created and whenever any error happens, we immediately display it for the user so they can act on it. Okay? Now, the thing is this usually takes a lot faster. The reason why it now it is a bit slower is because basically we have create operations there and creating a virtual environment and installing dependencies because network is involved and I was involved is not as fast as many of the things. So at first time, it takes longer but for example, when you invoke it the second time, then basically we don't have to do anything then just re-invoke the comments that you have because all the dependencies that already installed and are in place. Okay? So it is obviously possible that you can run also just a subset of environments. For example, the cases you might only want to test what actually modified the source code. For example, documentation generation, you don't really care what you're actually developing or for example, another popular use case of this is that you want to run your, which you have for example, testing against two Python environment and you want to ensure that for example, you run the test for Python 2.7 and 3.6 and then you want to have another step after it which basically calculates the aggregated coverage reports that you can do with this kind of way because you can specify that run this to Python environments and then display an aggregated coverage for those two. Okay? Okay, another thing that is quite nice is that you can actually document whatever the targets are so the users don't have to actually guess. So you can add descriptions to it and then the user when, for example, this is for our own toxin if I basically when someone says, okay, what can I do with this project? They can just list whatever the possible targets are and they can read if they want to run any of them or not. Okay? You can see that there are default environments which are always run and then additional ones which are on demand basically. Another option is that you support the post-arts. I'm sure. So basically if you testify for your comment that you want to post our Gantt's end, this means that if you use the double dash and after a double dash, you can basically forward arguments to your underlying tool. This is very helpful. For example, in this case, I wanted to run the test on the Python 2.7 and if anything fails, I wanted to enter into the debugging, for example, PDB. Okay? So I did not, I could basically just pass them on to the invocation after the double dash as additional arguments and they automatically get forwarded. This is also a nice way where, for example, you can have more verbose test reports and all those kind of things which are basically more like you want to understand more deeply what something broke and not just run the default test with so to say. Okay? Another option which we have there is basically, we have another project which is like detox. This basically allows you to run things in parallel. For example, it is possible to run all the environments in parallel and then display an aggregated output. Now, obviously the limitation of this is that we can no longer stream the output in real time because if we would do, basically, you would mean that five environment, five tools run in parallel and all five tools would print the character and the screen would be quite hard to actually see what's on the output. Okay? So another quite, you can actually create a developer environment. Why don't I say developer environment and this is especially true for libraries. It means that we don't actually install your package as the user would install it but we install your package using PPS in developer mode which basically means that people not actually install your package by putting it into the site package but creates a same link and says that the source code lives in your wherever you're developing and then the whenever Python is running, it will actually pick up your local files and not the remote, not the site package files. Okay? Okay. Now, just one thing, once you start creating this kind of environment or generally, it is quite healthy for example, in your environment, you can quickly select one of the environments to be the one for developer ecosystem to use. For example, in this case, for Python, there is this PyMVmanage plugin which is quite nice which allows you to basically just right click on any of the environments, be made talk for any other virtual environment and say, but by then I'll change the interpreter package to this Python which is inside this folder. Okay? So, another thing which is quite popular is like obviously, especially like big projects care about 45 Python versions. You might not have all the Python versions locally on your machine so it doesn't make sense to force failing those environments. You don't have Python. You can specify that only you run environments which for which we can actually find the correct Python version. For example, don't try to run Python 3.5 if that's not on the system because it will fail but it's not actually a failure that the test will be just that we could not, we have dependently missing for it. Okay? Another helpful thing is this is especially helpful for like bigger projects or like Django project is that Django projects historically have like a lot of versions and you want to check a lot of combinations and ensure that all the combinations actually work. It allows you to basically specify and you can see in three lines all that kind of Django combinations and there, for example, you can see that, for example, the Django master actually can specify even like GitHub links and it will install the latest from GitHub and try to test it again. Okay? So, as you can see, basically, we just tend like so of configuration. Now you can test against like 12 combination of dependently that your users might run into. Okay? Now obviously, you might not necessarily want to run all of this locally when you develop but this can work something in the CI and whenever something works in the CI, you locally, you don't have to manually recreate the environment. You can just say, by the way, on the CI, I saw that the Py37 Django master failed. Let me just recreate that and run the test only in that and try to debug what actually is the underlying issue for that. Okay? So, CI integration. As I said, the whole concept of it is that we would prefer to not be CI agnostic, meaning that we want the CI to basically don't have to actually specify all these. The CI should basically just invoke the same thing as the user would locally basically just invoke docs. And this means that your CI configuration, if you use docs, beside the user actually having the option to easily set up locally any kind of environment. Now the CI, you don't have to use the property CI environment but you can basically just say that, oh, by the way, make sure that I have these pythons in that CI and then finally install them with, install, for example, my docs and then just run docs and that's the entire CI configuration. Okay? So, let me sum up a bit what I tried to present here today and what we learned. So, the first thing is obviously there are weaknesses of the tools. The biggest weaknesses of what people often get confused is that dependency updates. The way PIP works, if you say to PIP that install PIP 11 and unless you specify that, by the way, please make sure that you upgrade if you have any things, the package is not there, PIP will actually not install it. You will say, I have PIP, it doesn't even check what version it is. It says, I have my version, PIP is already there. I do not actually evaluate that you wanted PIP 11. So, then if I need to specify the dash capital U, which basically means upgrade and then it will force make sure that it upgrades whenever it installs that PIP version. Another thing which is, but this is not the true, for example, dependencies. For example, if you install a library and the library's dependency is already there, PIP will not actually upgrade it. And because we use PIP, we have this kind of mistake on our side or like this kind of vulnerability on our side, which bottom line falls down to that whenever you add new dependencies to your project, talks will not actually upgrade your or pull in your new version of the dependency. It will add new dependencies, but will not actually, if you just bomb the version requirement, it will not actually ensure that the bigger version is installed. Okay? So, the workaround for now is actually just force the recreating the environment. You can pass in the dash R flag and this basically means, by the way, recreate the environment from scratch and that ensures it. The medium plan goal for that is that once PIP will actually support installing or like having dependency graphs and evaluating that kind of things, this should be also solvable. Okay. Another thing is that the build dependency assets beforehand, this probably should have some solution in the next few months. And, but let's see what strengths we have. So, what we can use. Basically, you can make sure that you can add new tools to your project and users can run those projects locally without actually needing to teach them the new tools, only if they actually want to run those or like they actually want to debug those new tools then they have to learn about it. It allows you to run your test sheet on your multiple Python versions without having to manually create all the virtual environments for the Python versions. Plus, it allows you to run locally exactly what would run on the CI. Okay? So, this means that if, and as I said before, like, if anything breaks on the CI, it's easy to reproduce it locally. And finally, it is just a Python package meaning it runs on any environment. So, for example, your contributor or user maybe it's easy for them to get on board because now they, for example, compared to make, they don't have to install something which only works on unit system. Okay? And it has a, hopefully, a readable configuration instead of some ugly, ugly configuration. Okay. So, finally, one thing which we have like similar as PyTest has, we also have a plug-in system. So, if you want to do anything which doesn't work out of BoxingTux, you can do it by having a plug-in. For example, you can see that there are plug-ins out there and quite popular which basically swap out the Python setup, PyTing, for example, to using CMake to build in the project or use the running Docker as a tool, as an environment where you can have virtual environments. So, it's also kind of like a virtual environment. So, it's easily to plug in those kind of things but there are a lot of other things. For example, there's also in development a package which will allow you to use Kanda if you're more like a Kanda developer and use the environment from there. Okay. Thank you very much. And now I'll take your question. Thank you for talking. We have about 50 minutes. I have a question. Who has questions about talks? What tools do you have any thoughts about SQL integration or integration with databases? Or is it just a purely Python specific thing? Can you just write your own tool to go off and populate your Postgres database with test data, et cetera? And how do you extend it, I guess, is the question, is the broad question? Depends on what you're writing. I would say probably the easiest what you're speaking about is basically, I suppose, database dependencies. I would say probably what you're looking for is you can set three different kind of fixtures which set a different kind of databases and then you can specify, for example, that different environment generated, use different kind of fixtures which set up the database. And that could, I think, fairly reasonably work and you can do like that, for example, if you want to test both with Oracle and MySQL, that would make it possible. I'm not sure that you could probably also do it without having the texting direction. You could probably also just have parameterized stress in PyTest. I think it's personal preference if you prefer. All right. Thanks for the talk. Would you know if during the environment creation stage, would you be able to specify requirements.txt file or something like that? Yeah. As I said, like in the dependencies, you can actually specify both the requirements takes the or constraint takes the, which is another thing which we introduced. As opposed to what's in setup.txt. Sorry? As opposed to what's specified in setup.txt. So setup.txt is only required if you have a library. Usually for libraries, I think the best practice is to use setup.txt to specify dependencies. Requirements takes is more popular for applications. Like the requirements takes were developed for applications. If you have an application, then you don't have a build process. You only have the dependent, so the environment creation, in that case, you can specify the requirements takes the easily just pass it on as instead of dependently. I think at one point you mentioned running things in parallel, doing multiple environments at once. I don't remember seeing talks do that before. How do you? It's a, so at the moment it's not built in, meaning that you need to run detox, which is also lives under the talks organization, but it basically is just a driver which allows you to run multiple text environments in parallel. It's an higher abstraction. So yeah, it's not built in. But if you look up detox, should be what you want. Will detox go as far as running a couple projects in parallel? Or is it still strictly one project at a time? Mostly because your different projects might have different configurations. That being said, there is no real reason why we could not like have parallel configuration parsing and running. I don't think anyone actually raised the need for that. It was so strongly that they actually wanted to develop it. Thank you for that. Just a question you mentioned, you can make a variation on the jungle dependency in your project there. In our case, we have dozens of dependencies. Usually we ship with every dependency has been fixed to a specific version. And what is interesting is to see for many of those, can we upgrade because they're fixed in time and we realize three months or six months later, maybe we should upgrade to the latest and not be stuck in the past. And we want to explore the versions we can now use. How could we leverage that kind of tool to explore how we can upgrade our dependencies without breaking completely? Because sometimes they collide and it doesn't work anymore. So in my view, that would be a requirement that is fixed to specific versions and one other requirement that would be more flexible and say, okay, I just... Yeah, I mean, if you can see this example, for example, you can see that Django 1.8 is actually fixed to Django 1.8. But for example, the master is actually master. I could easily just specify Django latest, which would be like... What have... Without any dependency specification, right? We need to install the latest. That's one. How do you cope with 80 dependencies? Oh. I mean, technically speaking, you could do it with similar... Like, this kind of factor expressions that... Because we call them factor expressions, they're at the top. They are put there even like the dependencies you can put it in. So it allows you to do even more deeper kind of these kind of specifications. But just a word of caution, if you have like 10 dependencies, you're talking about factorial of 10. So I'm not sure you would actually run that in real time. Probably usually, like in practice, what we saw is usually people want a few major things and maybe the latest. And then when that breaks, according to that, you may be altering your... Like, for example, specifying that, by the way, our package no longer works with... Because it broke on the old... When I installed all the latest, it broke with this package's latest. So I'm just pinning that to a given version until someone takes up the task of fixing what, why it doesn't work now with the latest. Because otherwise, you just have too many... The space is too big. And even if your test run five seconds, you're already looking at like probably next year when one of the running finishes. Okay. Any more questions? So we don't want to wait until next year. So we have break now and have lunch. Enjoy lunch. And thank you very much. Thank you very much. Thank you.
want to like quickly get started with the you want to ensure that all these kind of to open source their solutions for various static analysis, style checks like black, piling, and make sure and you use one of these tools and basically start with just checking that if actually testing multiple Python, you can use the use the full path to actually run things inside basically, once we add this just for one virtual probably want more than that. So one can see that done, it will, for each of your targets, it will the documentation and view it. Okay? One of the talk for any other virtual environment and say, the Python versions locally on your machine so it example, you can see that there are plug-ins out Docker as a tool, as an environment where you can