status quo of virtual environments
Gabor, Bernat
Python is easy to learn and use programming language; however, managing dependencies and package versions for it are nowhere as pleasant. One of the basic building block created to help with this is virtual environments. Join me in understanding how virtual environments work from within (by one of the project maintainers); also, to find out if the good old virtualenv project has any place left, now that Python 3.4+ contains venv. This talk is aimed to be a bit more technical in its first parts, presenting in technical details what a virtualenv is. The target audience is anyone who used virtual environments and wants to understand how they tick from within. I’ll also emphasise diversity and inclusion at Python interpreter level by highlighting other interpreters than CPython: Jython, PyPy or Iron Python. A concise outline goes as follows: What is a virtual environment? - why we need it - what we use it for - demo - virtualenv vs system env How do we build a virtual environment (CPython) - technical workflow of venv creation - activation - bash - powershell - cmd.bat Other interpreters - why other than CPython? - PyPy - Jython - virtualenv - all Python support - extra activation - xonosh Summary and q/a
 Thank you. So yeah, today we're going to learn about virtual environment and I have to make a confession. This talk is marked as a beginner and which is true that I'm not expecting from you any kind of previous knowledge, but we're going to dive in quite deep into the Python interpreter and its implementation details. So it's going to be a bit dense. To make it a bit more sweet, I did brought a few nice pictures of my Yo-Shite-Area puppy. He's silky, he's just a five-month-old, and this is basically his just after he got home, his first picture with a tennis ball in his mouth. So similar to him, we'll get on to a course to actually discover how virtual environments work underneath. Okay? So just a quick intro. Who am I? I'm the maintainer of the virtual amp tool, which hopefully would make me quite competent to talk about this subject. I'm a member of the Python Packaging Authority, and I'm quite active in the open-sport space, and you can find my contributions under this GitHub handle, Gabber Burnett. And I'm also maintainer of the Tox tool, which heavily uses virtual amps. This is one of the main reasons I became maintainer of the virtual amp tool. A lot of Tox tools, ToxBox, are kind of like virtual amp box. So my test path is there. And I'm also a software engineer at Bloomberg working mostly at Data Interest and Pipeline inside the company. So what is a virtual environment? Luckily, I don't have to come up with the definition. The PEP405 defines it very clearly. It's an isolated Python environment that has access to its own site package, shared success to the host building standard library, option isolated from the system site packages, and this is the point where you're like, whoa, what are all these kind of random definitions that you're throwing out? And why? What do we want to do here? And in order to understand this better, let's take one step back and actually cover, so how do we actually get hold of a Python module? Python is this famous language in which you can do pretty much everything by some package that is implemented, and you can get hold of some module. But how do you actually get hold of that module? And the answer is basically, in this case, the slide which basically means if you want to basically use anything, all you need to do is basically import it like that. So the thing is, the way you get something is by importing it. So how does the importing system work? Well pretty much you just specify import, you specify what you want to import, and then in the next line you can already magically use it. But the question is, what actually happens under the hood when you say that import.cl? And another kind of like complication around this is that not all modules are equal. For example, if I look at the cish, I can see that even if I print the representation, this is a built-in module. If I look at the os module, you can see that that's kind of like it has a file inside that optpy in 3.7 blah blah blah dash os.py. Similarly the.csv, this is more interesting because it's actually, you can see this implementation is an esof file, which is kind of like a DLL for on the windows. Then you have pytumble, which is a different location, or I can have like your py5.7, which is even a different location. So as you can see, just by the fact that they're living in different places, they are considered different type of modules. Like this one is a built-in module, the OS one is a standard library module. This is the module that comes if you install it, if you just Python, built into the standard library. And then we have also standard library module by dynamic module. These are modules that are, would be slow to actually implement it within Python. So what the core developer team does, they implement the actual underlying implementation inside C. And then they just create a slight front proxy to that. That's why in this case you can see this is actually underscore csv and not csv, because the csv is basically just a slight proxy. The actual implementation, the performance implementation is the underscore one, and that's available under some actual binary code. And then we have this import pyTaml, and this is a global site package, and it differs from the user site package in the sense that this is actually available on the system path, while this other one is basically available on the user site path. Like you can see, it's users burnout.local, which is kind of like users home. So the next section is, okay, now that we know that we import things, how does the import work, and this is something that has evolved throughout the Python's life, but this is how it works in 3.7. It works similarly on Python too, but the only difference is that there you could not modify this, it's not as easily exposed, so we can actually inspect it. But the idea is that Python doesn't know if something is importable or not. It just basically goes with the principle that try first, ask for forgiveness later. So it tries to import all the actual packages, and it has a list of importers, if knows, and it just asks in order each importer, hey, can you give me this sys, if yes, it's going to return that, it's not going to follow our importer to the first one, the first one can return it in the order, that's the one that Vincent actually provides that module for you. So we have the built-in importer, this is the first one, built-in importer imports elements which are literally compiled within the C Python binary, or like the Python binary, and in this case it's actually, such example is the sys itself. You're not going to find a sys.py anywhere, it's literally available already inside the binary itself. Then we have the frozen importer. By default, everything can be put straight into your Python binary, and the way you do it is you freeze it, and the freezing means I take your Python code, I compile it into the something machine code, and I literally attach it into the Python binary, and because this is no longer live anywhere on the file system, it's not on the no namespace of the built-in, it has this frozen importer which doesn't know at which location inside the binary I'm going to be able to find that element. Or my final option, at least in the default, is the pathfinder, and the pathfinder is named kind of like a in-chat, it basically looks at the operating system file system, and we'll try to find it somewhere there. So let's investigate it later. The pathfinder is basically just give me somewhere on the file system, and it doesn't operate, it has its own configuration, and the main configuration for this is going to be actually the information that you find in the sys.py. And the sys.py is basically a list of locations where this importer will try to look and find if that package is there. If in any of, and this again, we're going to do it in the order, and the first one is going to find it, that's what's going to be returned to you. So we have, you can have multiple elements, but this is like a list of defaults that I get on my Macbook, for example. You can see this third one is basically a zip file, like we can even import from zip itself. This is the first place it's going to look, then it's going to look in this sender library directory, then we have this libdial which is basically where we store the C extensions for the sender library, or we have the user size package or global site package. So now that we know all this, we can actually interpret this definition. So a virtual environment is just an isolated Python environment that has access to its own site package, so it must have something else besides what is there. But the system basically has access, while it has access to the sender library pass plus the user and the global site package, the virtual amp may or may not actually have access to the global one. So like in this case, you can see it has the first three, they are the sender library locations and the fourth element is basically its own site package. Or if you configure it to also have access to the global site package, it's going to append to end of this list, the user and the global site package. So virtual environment is nothing else, just a Python executable that whenever it starts up, it's magically going to make available its own site package folder and may or may not disable the global one. Quite simple. So the question is why do we want to do this, and the reason for why is because this will allow you to actually install packages to the system administrator right? Because we saw everything else other than user site package was available, only modified by the system Python, so if you don't have admin right, you might not be installing it. You also have the option that you can create easily and reproduce test environments. You don't have to reinstall your entire operating system just to test your package, you can just create a virtual environment and it's going to be exactly the same as you would have done that. And it's also easier dependency management and the way this manifests is that in practice, you would have multiple dependencies and if two tools, if you would install everything in a single global site package, you may end up in a situation when two of the site packages have conflicting dependency requirements and this would mean that basically by just installing a new package into that environment, you break another one which was already there. And it also just allows you, for as more like as a testing, the ability to easily test against multiple dependencies easily without even having to go too hard about it. So let's see how this actually works. So in the next we're going to try to look into, I told you already that basically what going to determine what is a module is going to be this sys.pat where something is going to be loaded is the sys.pat. Now this is that pat, the question one could ask, okay, but who comes up with this sys? Like this is just magically made up or and what more if you start up in a different platform, it may well be the case that not well may be. For sure it's going to be the case that you're going to have a totally system, totally different type of layout here or you may have more or less element. So the question to this is before, at the moment at least, there is different location where the standard library is identified and there's a different location where the other site packages are inside. All the standard library discovery happens straight into the Python binary. So this is kind of like a built-in logic. And then inside whatever customizes the global site packages is the side of pi. This is at least the most of the case. You can disable the side of pi. We're not going to cover that case too much, but in that case the built-in interpreter is going to fall back into some built-in logic, but this is very rare in practice. You're most often what's going to happen is that on your machine in the standard library there's going to be this side of pi which is going to magically customize and come up with the global and user site packages. So if you can look at it, you see there's some commonality in this. All those other packages which are outside of the virtual and its own site package and the user home site package. All these other ones start with this magic prefix called optpython 3.7. And this is a good reason because the way this list is generated is basically by using some prefixes. There is these two variables that you can change, but they're ultimately going to have some same defaults. This prefix and this is basically what's going to determine where the Python is going to try to find these packages. Now, obviously, Python doesn't just magically come up with, yeah, it's right here. It's like a very hard-coded version. It's actually dynamic. And what ends up happening is that there's a different algorithm that comes up with this logic. And this is actually available mostly in get.c. This has changed quite a few times as the Cpython code team, for example, started optimizing startup and they needed to move around to make things faster. But the rule of thumb is that if you set the Python home, we're going to use that for this prefix. So this is a great thing if you want to break someone's environment. You just set the Python home environment variable to something that doesn't exist and everything is going to be broken. But otherwise, if this environment variable doesn't exist, we actually are looking for a landmark file. And the landmark file chosen is actually this os.py. So the Python is going to start from its own executable. Here the Python executable itself is going to start going up and at each level, you're going to see, can I find this libpython.x.epsilon.os.py file? If I can find it, that is what six-dot prefix will be. And otherwise, it's going to be like a built-in hard-coded value, but this more like a fallback most likely will not work because this is whatever the compile, whoever compile your Python set is going to be exactly that one. And there is a different logic for the exact prefix, which is basically the C extension modules. The only difference is that in this case, it's not going to look for the os.py file. Instead it's going to look for the libdial node element, okay, the folder. And wherever it finds the folder, that's what the exact prefix is going to be set to. So that's more or less it. We only need, if you want to create a virtual environment, like roll your own, almost, the only thing you have to do is modify these six exact prefix and six prefix. If you're done with this, all your power is in your hand. So how we created this environment, in order to have shared access to the host, we just need to make sure that whenever Python starts up, it's going to resolve the standard library element by having available those two landmark files, the file and the folder, okay? If you want to actually create, we want to have the system side package added or not, this is just a conditional logic in the side.py. The side.py automatically gets loaded before any code gets executed at the startup of your folder. So if we do any changes here, whenever the user will get the interpreter, we'll already have modified sys.py that's going to work for him. And similarly, adding our own virtual environment side packages is building just some logic that we can tossing into the side.py, okay? So what are the changes needed to actually have a virtual environment? I just take a Python executable and I want to not use the system one. Well, we just need to keep track of the host so I can actually find this global standard library. We just need to know where that was available so I can set the sys prefix according to that and just some flex into the side.py for anything. Now according to the path I will say, this configuration information is actually sorting this pyvm.cfg and this is something that Python 3.3 or later actually already will parse and read and the way it's going to set the standard libraries for the standard library sys.profic is going to be basically going to look at whatever that home value is, okay? If that home value points to something from where it can find those landmark files, you're going to be happy, it's going to resolve the correct system side packages. And otherwise, just the fact that should we add the include system side packages is just this second argument that includes system side packages equals false and the fact that should we add the virtual environment side.py or not is basically just done by, is this file exist? Then yes, if not, we should not bother with that, okay? So quite simple and straightforward in that sense. Now there's one more thing, if you create a virtual environment, it's not going to be much of a use if you can't install packages in it. So the first thing that you'll have to do once you create a virtual environment, you want to be able to put a pip into it. Now this is kind of catch 22 because usually when I was someone who said install a package, you just go, yeah, I use pip to install it, but in this case we don't actually have at this point any pip in that package yet. So how do we install pip without pip? And the way we resolve this question 22 is basically we put, we have a wheel of the whatever we want to install or a zip file of the pip and we can actually put this onto the sys.pad and this basically is going to mean that for that invocation, pip is going to be available, it's going to be lookable by the importing system, so it's going to be able to start up and then we're going to ask pip, can you please install yourself by using this wheel that I just pointed to you at? And pretty much this is what in your self-installing itself and pretty much what ensure pip more or less does. It does a bit more magic, but this is the core gist of it, okay? So now we have virtual environment, it has a lovely pip in it, we're kind of like done, yeah? Well, then let's have a quick look at implementations. So the implementations out there available, the one that's recommended for use, so to say, is the VM. The VM is a standard library module, it's built in within the interpreter and it works only Python 3.3 only, but hey, Python 2 is going to be deprecated next year, so who cares? It's going to be also available of the standard library, well, mostly, because like Debian derivatives decide not to install it. So and the only main scope is going to be able only to target itself. So if you want to create a Python 3.4 or 3.6 virtual environment, you must have available Python 3.6 or 3.4 with virtual environment within it to be able to do it, okay? Now there's another thing called out here and this is for what I'm the maintainer is this virtual amp thing and how virtual amp differs from that built in VM, should this project even exist anymore or can we just drop it? And in order to answer this question, let's quickly go over what virtual amp offers on top of VM, okay? So the first thing out is that it's a third-party package, which means that you can upgrade it out of the band. So you no longer have to wait for the operating system to upgrade your Python, you can literally get the latest virtual amp creation by just installing this package, okay? So in that case, it's a bit more feature proof, it's a bit more easier to have the latest of it with the latest features. It also targets to support both Python 2 and 3 and it supports C Python and PyPy. We have some jiton implementation, iron Python is something that may be working progress, but if you have interpreter and you want to have a virtual amp for it and maybe you don't necessarily want it within your sender library, just submit a pull request and if it looks good, we're going to accept it. It's also cross Python support, meaning that you can actually, if you install virtual amp on Python 3.4, it's going to be able to create for you virtual amp for all supported pythons, okay? So even if the VM package is not there for the 3.6, it's still going to be able to create a virtual amp environment for that 3.6. This makes it very useful for tools such as stocks, PPM, all these tools, or Ppx, all these tools, because now they can actually don't have to check if the virtual amp is correctly set up or not, they can actually just have a look at it and they can just use this tool and this is guaranteed to be work for all available Python X YouTubers, okay? Another thing is cross-pattern support, obviously, and it has built and interpreted the discovery, meaning that this is especially handy on the Windows. You don't even, you automatically discover all pythons available or try to do our best, all pythons available in your operating system, so you don't have to figure out, hey, is my Python 3.4 on the path? Is Python 3.7 on the path, especially on Windows? For example, the path file for a loves on Windows tools to actually register their Python existence within the Windows registry and the virtual amp will automatically load this and it will be able to, you can just specify to it that dash p, I want the Python 3.4 and we're going to discover it and create it using that. And similarly, we're going to use a path discovery method by using path 394. This basically means that we're going to be able to create, you're going to have a single virtual amp for creator for all the pythons and you don't have to use, have different ones for different Python versions, okay? So and it also has its own bundle Pipset R sweep to the bootstrap, meaning that it can work in offline mode, but it also available to use index servers, so it's possible to install the latest one, rather than whatever we distributed the last package with, okay? So you may have like a newer Pipset up to the wheel and this in turn causes that you have less time, the awkward case when something doesn't work, you go to Stackoverfl and they did you upgrade your PIP? Okay, okay, now it works. So let's have a bit of, the other thing is though, because we need to support Python 2, virtual and virtual is a bit different than VM. Everything I told you in the beginning of the VM, that pyvm.dcfg is not a thing in virtual amp. So let's go over how virtual amp does actually create a virtual environment. So we have to basically do the same on alter the standard library logic discovery and alter the side of py logic to be able to create a virtual amp, but without the interpreter having any knowledge of any configuration files. And the way we do this is basically the first thing is we always copy the Python 2 at least the executable and we put it in a folder that looks like a Python virtual environment, okay, under bin and in this macOS. Then we create the two landmark files. We just simulink this to the operating system one, so they actually contain the same thing as the operating system one. Then we add a few configurations. We have a configuration flag if we should add the system Python or not, what is the prefix from where the Python that we created with is available and we inject our own side of py. Now the reason we need to inject our own side of py is because the side of py that is shipped by operating system is not aware of this magical oryc prefix or no global prefix. So if you would use the side of py that is shipped with the operating system, that would magically ignore all these configurations and you would not get an actual virtual amp. So for this reason, we have to like whenever you use a virtual amp, we have to like basically lie to you and have our own side of py that works for all platforms out there. Now you can imagine if this works for all platforms and all pythons, it's not too easy to maintain. So there's one more thing. The side of py is going to take care of adding everything on the path, but there's a caveat. It seems the side of py doesn't work on its own in magic. It actually works by using Python standard library modules, but at this point, at the point the side of py gets loaded, those are not yet on the path. So okay, this means that in order to be able to load or start up the side of py file and that's going to do the modification, we first need to be able to modify the, to have available all those packages that side of py happens to use. Now this is a short list of what we need basically. And again, this is kind of like platform dependent and each version dependent. So yeah, you can see that we create a lot more links or copy them literally over the standard library just so that our customization logic is able to run. And yeah, it's few I meant 21 in this case. I don't know, quoting a recent show, not great, not terrible. Could be 60 or 60, so 25 maybe. Let's work with it. And then we just have to do some fix up basically, add some activation scripts, link the include, fix up the distutils, because turns out the distutils uses its own kind of site package discovery. So we have to basically ship a different distutils than your operating system. And then we just fix all other bugs that are available. Long list of it like being able to discover Tlk platforms, having Python, the documentation run. You can see there's a long list of fixes that we have to do for this to work. And just, yeah, that's more or less this. Now, just to have a bigger thing, in this virtual environment, the only thing that we actually carry is that lonely side of packages. Everything else is just there to make things work. And just for a comparison, this is what VM has. It looks a lot slimmer, no? So and the reason why it can get away with having this slimmer package, this slimmer directory is because the interpreter itself is able to know about virtual environments and the interpreter itself is able to basically load these configurations from that pyvm that does CFG across it. So yeah, next time you create a virtual environment and you see this on your hard drive, don't despair, it's just how it is. Obviously, this is not a good way to have things. And because we basically say before we patch both the side of py and the disk tutorials, as opposed to what your distribution ships with, we have a lot of bug reports around ID doesn't work on this magic random platform because basically that has a customized disk tutorial, so the side of py, and various features start breaking. And because this is kind of like hard to maintain, I start doing a rewrite, and this is my proposal. Basically the idea is that it's going to still be offered as a third package, so this is what we're going to differentiate from VM. It's going to also have all Python implementations welcomed. This means including Python 2 and 3, like PyPy, even to the C Python, it's going to be deprecated next year. PyPy is going to be supported, it's two versions are going to support it for a foreseeable future, so we still need to be able to create virtual environments for that interpreter. So we're still going to support that. We're also going to have a two-year grace period, meaning that, well, I put here two years, but maybe I'm going to make a year and a half. The idea is that we're going to maintain our virtual AMP to be compatible with a Python that has drop support for its implementation a bit earlier, and the reason we want to do this is that you're most likely, unless you live in a perfect world, you're not able to migrate to the latest fanciest Python immediately, and we want to make your life a bit easier. You still can use virtual AMP to actually create VMs for your favorite deprecated Python. So it's also going to be able to select target pythons. It can even do fancy things such as, for example, request a given subversion or request a given architecture on Windows. Again, this is kind of like maybe not as interesting for the end user, but it's something that tools definitely want, and it's very handy. And one thing in order to minimize the bug request, we actually are going to, what we're going to do is we're going to ask, if possible, we're going to delegate the work to the VM in the spirit of why work if others can work for you, but we're basically just going to make, add a few optimizations on top of it. And we're going to have a broad shell activation support. This is something that differentiates for us, for example, also from the VMs, that we actually support a lot of other shells than whatever the standard library does. For example, we support the Sonnage Python shell interpreter, and also we're able to move much faster. For example, when PowerShell will suddenly become available on the POSIX operating system 2, we immediately added that now we generate that activation script on that platform 2. And we're going to keep doing that, and we're going to make sure that it features work similarly. And we have a three-layered configuration system, which basically allows you to have some global settings customize how virtual apps are created on your hard drive. And the kind of things that you can, this can be useful, is for example, whenever P releases a new version, and for some reason it's broke, you can set it at operating system level that I actually want to use an older PIP and create all my virtual environments with an older PIP, not a latest, but still the less known working version or something like that. And we might also have a plugin system for extension. This is mostly something that people are able to create their virtual environment implementations for the pythons without necessarily having to get a poor request to us. You can basically load and test it on your own, and then if it's stable, we can upstream it within the virtual environment itself. So yeah, that's going to be all of my talk. I think I might have a question left or so. So yeah, thank you. So thank you very much for this rare insight. There is time for one quick question and any other questions, you will be around afterwards, so feel free to do that after the talk. So is there any quick question? Oh, sorry, go on, yes. Thank you for that. It was really interesting to see the internals. You said you were planning on doing a rewrite which would include support to select any Python version. Yes. Obviously, say you were doing front end, you can use NVM at the minute to practically select any node version. Right now, what is your go-to for downloading and managing Python versions as opposed to the virtual end? For Python versions, I either install it from upstream or I install, for example, using PyVM or other things which are sometimes I use Konda which has. But like this, for example, would allow us to even, you'd be able to say that you want a Konda environment. And if you have Konda inside the system, it would be Konda would actually create its own specific environment. And that could be still managed by the single front end of the virtual end instead of having to use multiple tools. Virtual end would kind of act like a unified interface which is very nice for tools especially. Yeah, thank you. One last thank you and welcome for this talk. And then that's it. Thank you guys. Thank you guys.
 Thank you.  So yeah, today we're going to learn about virtual environment and I have to make a confession.  This talk is marked as a beginner and which is true that I'm not expecting from you any  kind of previous knowledge, but we're going to dive in quite deep into the Python interpreter  and its implementation details.  So it's going to be a bit dense.  To make it a bit more sweet, I did brought a few nice pictures of my Yo-Shite-Area puppy.  He's silky, he's just a five-month-old, and this is basically his just after he got home,  his first picture with a tennis ball in his mouth.  So similar to him, we'll get on to a course to actually discover how virtual environments  work underneath.  Okay?  So just a quick intro.  Who am I?  I'm the maintainer of the virtual amp tool, which hopefully would make me quite competent  to talk about this subject.  I'm a member of the Python Packaging Authority, and I'm quite active in the open-sport space,  and you can find my contributions under this GitHub handle, Gabber Burnett.  And I'm also maintainer of the Tox tool, which heavily uses virtual amps.  This is one of the main reasons I became maintainer of the virtual amp tool.  A lot of Tox tools, ToxBox, are kind of like virtual amp box.  So my test path is there.  And I'm also a software engineer at Bloomberg working mostly at Data Interest and Pipeline  inside the company.  So what is a virtual environment?  Luckily, I don't have to come up with the definition.  The PEP405 defines it very clearly.  It's an isolated Python environment that has access to its own site package, shared success  to the host building standard library, option isolated from the system site packages, and  this is the point where you're like, whoa, what are all these kind of random definitions  that you're throwing out?  And why?  What do we want to do here?  And in order to understand this better, let's take one step back and actually cover, so  how do we actually get hold of a Python module?  Python is this famous language in which you can do pretty much everything by some package  that is implemented, and you can get hold of some module.  But how do you actually get hold of that module?  And the answer is basically, in this case, the slide which basically means if you want  to basically use anything, all you need to do is basically import it like that.  So the thing is, the way you get something is by importing it.  So how does the importing system work?  Well pretty much you just specify import, you specify what you want to import, and then  in the next line you can already magically use it.  But the question is, what actually happens under the hood when you say that import.cl?  And another kind of like complication around this is that not all modules are equal.  For example, if I look at the cish, I can see that even if I print the representation,  this is a built-in module.  If I look at the os module, you can see that that's kind of like it has a file inside that  optpy in 3.7 blah blah blah dash os.py.  Similarly the.csv, this is more interesting because it's actually, you can see this implementation  is an esof file, which is kind of like a DLL for on the windows.  Then you have pytumble, which is a different location, or I can have like your py5.7, which  is even a different location.  So as you can see, just by the fact that they're living in different places, they are considered  different type of modules.  Like this one is a built-in module, the OS one is a standard library module.  This is the module that comes if you install it, if you just Python, built into the standard  library.  And then we have also standard library module by dynamic module.  These are modules that are, would be slow to actually implement it within Python.  So what the core developer team does, they implement the actual underlying implementation  inside C.  And then they just create a slight front proxy to that.  That's why in this case you can see this is actually underscore csv and not csv, because  the csv is basically just a slight proxy.  The actual implementation, the performance implementation is the underscore one, and  that's available under some actual binary code.  And then we have this import pyTaml, and this is a global site package, and it differs from  the user site package in the sense that this is actually available on the system path, while  this other one is basically available on the user site path.  Like you can see, it's users burnout.local, which is kind of like users home.  So the next section is, okay, now that we know that we import things, how does the  import work, and this is something that has evolved throughout the Python's life, but  this is how it works in 3.7.  It works similarly on Python too, but the only difference is that there you could not  modify this, it's not as easily exposed, so we can actually inspect it.  But the idea is that Python doesn't know if something is importable or not.  It just basically goes with the principle that try first, ask for forgiveness later.  So it tries to import all the actual packages, and it has a list of importers, if knows,  and it just asks in order each importer, hey, can you give me this sys, if yes, it's  going to return that, it's not going to follow our importer to the first one, the first one  can return it in the order, that's the one that Vincent actually provides that module  for you.  So we have the built-in importer, this is the first one, built-in importer imports  elements which are literally compiled within the C Python binary, or like the Python binary,  and in this case it's actually, such example is the sys itself.  You're not going to find a sys.py anywhere, it's literally available already inside the  binary itself.  Then we have the frozen importer.  By default, everything can be put straight into your Python binary, and the way you do  it is you freeze it, and the freezing means I take your Python code, I compile it into  the something machine code, and I literally attach it into the Python binary, and because  this is no longer live anywhere on the file system, it's not on the no namespace of the  built-in, it has this frozen importer which doesn't know at which location inside the  binary I'm going to be able to find that element.  Or my final option, at least in the default, is the pathfinder, and the pathfinder is named  kind of like a in-chat, it basically looks at the operating system file system, and we'll  try to find it somewhere there.  So let's investigate it later.  The pathfinder is basically just give me somewhere on the file system, and it doesn't  operate, it has its own configuration, and the main configuration for this is going to  be actually the information that you find in the sys.py.  And the sys.py is basically a list of locations where this importer will try to look and find  if that package is there.  If in any of, and this again, we're going to do it in the order, and the first one is  going to find it, that's what's going to be returned to you.  So we have, you can have multiple elements, but this is like a list of defaults that I  get on my Macbook, for example.  You can see this third one is basically a zip file, like we can even import from zip  itself.  This is the first place it's going to look, then it's going to look in this sender library  directory, then we have this libdial which is basically where we store the C extensions  for the sender library, or we have the user size package or global site package.  So now that we know all this, we can actually interpret this definition.  So a virtual environment is just an isolated Python environment that has access to its  own site package, so it must have something else besides what is there.  But the system basically has access, while it has access to the sender library pass  plus the user and the global site package, the virtual amp may or may not actually have  access to the global one.  So like in this case, you can see it has the first three, they are the sender library locations  and the fourth element is basically its own site package.  Or if you configure it to also have access to the global site package, it's going to  append to end of this list, the user and the global site package.  So virtual environment is nothing else, just a Python executable that whenever it starts  up, it's magically going to make available its own site package folder and may or may  not disable the global one.  Quite simple.  So the question is why do we want to do this, and the reason for why is because this will  allow you to actually install packages to the system administrator right?  Because we saw everything else other than user site package was available, only modified  by the system Python, so if you don't have admin right, you might not be installing it.  You also have the option that you can create easily and reproduce test environments.  You don't have to reinstall your entire operating system just to test your package, you can  just create a virtual environment and it's going to be exactly the same as you would  have done that.  And it's also easier dependency management and the way this manifests is that in practice,  you would have multiple dependencies and if two tools, if you would install everything  in a single global site package, you may end up in a situation when two of the site packages  have conflicting dependency requirements and this would mean that basically by just installing  a new package into that environment, you break another one which was already there.  And it also just allows you, for as more like as a testing, the ability to easily test against  multiple dependencies easily without even having to go too hard about it.  So let's see how this actually works.  So in the next we're going to try to look into, I told you already that basically what  going to determine what is a module is going to be this sys.pat where something is going  to be loaded is the sys.pat.  Now this is that pat, the question one could ask, okay, but who comes up with this sys?  Like this is just magically made up or and what more if you start up in a different platform,  it may well be the case that not well may be.  For sure it's going to be the case that you're going to have a totally system, totally different  type of layout here or you may have more or less element.  So the question to this is before, at the moment at least, there is different location  where the standard library is identified and there's a different location where the other  site packages are inside.  All the standard library discovery happens straight into the Python binary.  So this is kind of like a built-in logic.  And then inside whatever customizes the global site packages is the side of pi.  This is at least the most of the case.  You can disable the side of pi.  We're not going to cover that case too much, but in that case the built-in interpreter is  going to fall back into some built-in logic, but this is very rare in practice.  You're most often what's going to happen is that on your machine in the standard library  there's going to be this side of pi which is going to magically customize and come up  with the global and user site packages.  So if you can look at it, you see there's some commonality in this.  All those other packages which are outside of the virtual and its own site package and  the user home site package.  All these other ones start with this magic prefix called optpython 3.7.  And this is a good reason because the way this list is generated is basically by using  some prefixes.  There is these two variables that you can change, but they're ultimately going to have  some same defaults.  This prefix and this is basically what's going to determine where the Python is going to  try to find these packages.  Now, obviously, Python doesn't just magically come up with, yeah, it's right here.  It's like a very hard-coded version.  It's actually dynamic.  And what ends up happening is that there's a different algorithm that comes up with this  logic.  And this is actually available mostly in get.c.  This has changed quite a few times as the Cpython code team, for example, started optimizing  startup and they needed to move around to make things faster.  But the rule of thumb is that if you set the Python home, we're going to use that for  this prefix.  So this is a great thing if you want to break someone's environment.  You just set the Python home environment variable to something that doesn't exist and everything  is going to be broken.  But otherwise, if this environment variable doesn't exist, we actually are looking for  a landmark file.  And the landmark file chosen is actually this os.py.  So the Python is going to start from its own executable.  Here the Python executable itself is going to start going up and at each level, you're  going to see, can I find this libpython.x.epsilon.os.py file?  If I can find it, that is what six-dot prefix will be.  And otherwise, it's going to be like a built-in hard-coded value, but this more like a fallback  most likely will not work because this is whatever the compile, whoever compile your  Python set is going to be exactly that one.  And there is a different logic for the exact prefix, which is basically the C extension  modules.  The only difference is that in this case, it's not going to look for the os.py file.  Instead it's going to look for the libdial node element, okay, the folder.  And wherever it finds the folder, that's what the exact prefix is going to be set to.  So that's more or less it.  We only need, if you want to create a virtual environment, like roll your own, almost, the  only thing you have to do is modify these six exact prefix and six prefix.  If you're done with this, all your power is in your hand.  So how we created this environment, in order to have shared access to the host, we just  need to make sure that whenever Python starts up, it's going to resolve the standard library  element by having available those two landmark files, the file and the folder, okay?  If you want to actually create, we want to have the system side package added or not,  this is just a conditional logic in the side.py.  The side.py automatically gets loaded before any code gets executed at the startup of your  folder.  So if we do any changes here, whenever the user will get the interpreter, we'll already  have modified sys.py that's going to work for him.  And similarly, adding our own virtual environment side packages is building just some logic  that we can tossing into the side.py, okay?  So what are the changes needed to actually have a virtual environment?  I just take a Python executable and I want to not use the system one.  Well, we just need to keep track of the host so I can actually find this global standard  library.  We just need to know where that was available so I can set the sys prefix according to that  and just some flex into the side.py for anything.  Now according to the path I will say, this configuration information is actually sorting  this pyvm.cfg and this is something that Python 3.3 or later actually already will parse  and read and the way it's going to set the standard libraries for the standard library  sys.profic is going to be basically going to look at whatever that home value is, okay?  If that home value points to something from where it can find those landmark files, you're  going to be happy, it's going to resolve the correct system side packages.  And otherwise, just the fact that should we add the include system side packages is just  this second argument that includes system side packages equals false and the fact that  should we add the virtual environment side.py or not is basically just done by, is this  file exist?  Then yes, if not, we should not bother with that, okay?  So quite simple and straightforward in that sense.  Now there's one more thing, if you create a virtual environment, it's not going to be  much of a use if you can't install packages in it.  So the first thing that you'll have to do once you create a virtual environment, you  want to be able to put a pip into it.  Now this is kind of catch 22 because usually when I was someone who said install a package,  you just go, yeah, I use pip to install it, but in this case we don't actually have at  this point any pip in that package yet.  So how do we install pip without pip?  And the way we resolve this question 22 is basically we put, we have a wheel of the whatever  we want to install or a zip file of the pip and we can actually put this onto the sys.pad  and this basically is going to mean that for that invocation, pip is going to be available,  it's going to be lookable by the importing system, so it's going to be able to start  up and then we're going to ask pip, can you please install yourself by using this wheel  that I just pointed to you at?  And pretty much this is what in your self-installing itself and pretty much what ensure pip more  or less does.  It does a bit more magic, but this is the core gist of it, okay?  So now we have virtual environment, it has a lovely pip in it, we're kind of like done,  yeah?  Well, then let's have a quick look at implementations.  So the implementations out there available, the one that's recommended for use, so to  say, is the VM.  The VM is a standard library module, it's built in within the interpreter and it works  only Python 3.3 only, but hey, Python 2 is going to be deprecated next year, so who cares?  It's going to be also available of the standard library, well, mostly, because like Debian  derivatives decide not to install it.  So and the only main scope is going to be able only to target itself.  So if you want to create a Python 3.4 or 3.6 virtual environment, you must have available  Python 3.6 or 3.4 with virtual environment within it to be able to do it, okay?  Now there's another thing called out here and this is for what I'm the maintainer is  this virtual amp thing and how virtual amp differs from that built in VM, should this  project even exist anymore or can we just drop it?  And in order to answer this question, let's quickly go over what virtual amp offers on  top of VM, okay?  So the first thing out is that it's a third-party package, which means that you can upgrade it  out of the band.  So you no longer have to wait for the operating system to upgrade your Python, you can literally  get the latest virtual amp creation by just installing this package, okay?  So in that case, it's a bit more feature proof, it's a bit more easier to have the latest  of it with the latest features.  It also targets to support both Python 2 and 3 and it supports C Python and PyPy.  We have some jiton implementation, iron Python is something that may be working progress,  but if you have interpreter and you want to have a virtual amp for it and maybe you don't  necessarily want it within your sender library, just submit a pull request and if it looks  good, we're going to accept it.  It's also cross Python support, meaning that you can actually, if you install virtual amp  on Python 3.4, it's going to be able to create for you virtual amp for all supported pythons,  okay?  So even if the VM package is not there for the 3.6, it's still going to be able to create  a virtual amp environment for that 3.6.  This makes it very useful for tools such as stocks, PPM, all these tools, or Ppx, all  these tools, because now they can actually don't have to check if the virtual amp is  correctly set up or not, they can actually just have a look at it and they can just use  this tool and this is guaranteed to be work for all available Python X YouTubers, okay?  Another thing is cross-pattern support, obviously, and it has built and interpreted the discovery,  meaning that this is especially handy on the Windows.  You don't even, you automatically discover all pythons available or try to do our best,  all pythons available in your operating system, so you don't have to figure out, hey, is my  Python 3.4 on the path?  Is Python 3.7 on the path, especially on Windows?  For example, the path file for a loves on Windows tools to actually register their Python existence  within the Windows registry and the virtual amp will automatically load this and it will  be able to, you can just specify to it that dash p, I want the Python 3.4 and we're going  to discover it and create it using that.  And similarly, we're going to use a path discovery method by using path 394.  This basically means that we're going to be able to create, you're going to have a single  virtual amp for creator for all the pythons and you don't have to use, have different  ones for different Python versions, okay?  So and it also has its own bundle Pipset R sweep to the bootstrap, meaning that it can  work in offline mode, but it also available to use index servers, so it's possible to  install the latest one, rather than whatever we distributed the last package with, okay?  So you may have like a newer Pipset up to the wheel and this in turn causes that you  have less time, the awkward case when something doesn't work, you go to Stackoverfl and they  did you upgrade your PIP?  Okay, okay, now it works.  So let's have a bit of, the other thing is though, because we need to support Python 2,  virtual and virtual is a bit different than VM.  Everything I told you in the beginning of the VM, that pyvm.dcfg is not a thing in  virtual amp.  So let's go over how virtual amp does actually create a virtual environment.  So we have to basically do the same on alter the standard library logic discovery and alter  the side of py logic to be able to create a virtual amp, but without the interpreter  having any knowledge of any configuration files.  And the way we do this is basically the first thing is we always copy the Python 2 at least  the executable and we put it in a folder that looks like a Python virtual environment,  okay, under bin and in this macOS.  Then we create the two landmark files.  We just simulink this to the operating system one, so they actually contain the same thing  as the operating system one.  Then we add a few configurations.  We have a configuration flag if we should add the system Python or not, what is the  prefix from where the Python that we created with is available and we inject our own side  of py.  Now the reason we need to inject our own side of py is because the side of py that is shipped  by operating system is not aware of this magical oryc prefix or no global prefix.  So if you would use the side of py that is shipped with the operating system, that would  magically ignore all these configurations and you would not get an actual virtual amp.  So for this reason, we have to like whenever you use a virtual amp, we have to like basically  lie to you and have our own side of py that works for all platforms out there.  Now you can imagine if this works for all platforms and all pythons, it's not too easy  to maintain.  So there's one more thing.  The side of py is going to take care of adding everything on the path, but there's a caveat.  It seems the side of py doesn't work on its own in magic.  It actually works by using Python standard library modules, but at this point, at the  point the side of py gets loaded, those are not yet on the path.  So okay, this means that in order to be able to load or start up the side of py file and  that's going to do the modification, we first need to be able to modify the, to have available  all those packages that side of py happens to use.  Now this is a short list of what we need basically.  And again, this is kind of like platform dependent and each version dependent.  So yeah, you can see that we create a lot more links or copy them literally over the  standard library just so that our customization logic is able to run.  And yeah, it's few I meant 21 in this case.  I don't know, quoting a recent show, not great, not terrible.  Could be 60 or 60, so 25 maybe.  Let's work with it.  And then we just have to do some fix up basically, add some activation scripts, link the include,  fix up the distutils, because turns out the distutils uses its own kind of site package  discovery.  So we have to basically ship a different distutils than your operating system.  And then we just fix all other bugs that are available.  Long list of it like being able to discover Tlk platforms, having Python, the documentation  run.  You can see there's a long list of fixes that we have to do for this to work.  And just, yeah, that's more or less this.  Now, just to have a bigger thing, in this virtual environment, the only thing that we  actually carry is that lonely side of packages.  Everything else is just there to make things work.  And just for a comparison, this is what VM has.  It looks a lot slimmer, no?  So and the reason why it can get away with having this slimmer package, this slimmer  directory is because the interpreter itself is able to know about virtual environments  and the interpreter itself is able to basically load these configurations from that pyvm  that does CFG across it.  So yeah, next time you create a virtual environment and you see this on your hard drive, don't  despair, it's just how it is.  Obviously, this is not a good way to have things.  And because we basically say before we patch both the side of py and the disk tutorials,  as opposed to what your distribution ships with, we have a lot of bug reports around  ID doesn't work on this magic random platform because basically that has a customized disk  tutorial, so the side of py, and various features start breaking.  And because this is kind of like hard to maintain, I start doing a rewrite, and this is my proposal.  Basically the idea is that it's going to still be offered as a third package, so this is  what we're going to differentiate from VM.  It's going to also have all Python implementations welcomed.  This means including Python 2 and 3, like PyPy, even to the C Python, it's going to be deprecated  next year.  PyPy is going to be supported, it's two versions are going to support it for a foreseeable future,  so we still need to be able to create virtual environments for that interpreter.  So we're still going to support that.  We're also going to have a two-year grace period, meaning that, well, I put here two  years, but maybe I'm going to make a year and a half.  The idea is that we're going to maintain our virtual AMP to be compatible with a Python  that has drop support for its implementation a bit earlier, and the reason we want to do  this is that you're most likely, unless you live in a perfect world, you're not able to  migrate to the latest fanciest Python immediately, and we want to make your life a bit easier.  You still can use virtual AMP to actually create VMs for your favorite deprecated Python.  So it's also going to be able to select target pythons.  It can even do fancy things such as, for example, request a given subversion or request a given  architecture on Windows.  Again, this is kind of like maybe not as interesting for the end user, but it's something that  tools definitely want, and it's very handy.  And one thing in order to minimize the bug request, we actually are going to, what we're  going to do is we're going to ask, if possible, we're going to delegate the work to the VM  in the spirit of why work if others can work for you, but we're basically just going to  make, add a few optimizations on top of it.  And we're going to have a broad shell activation support.  This is something that differentiates for us, for example, also from the VMs, that we  actually support a lot of other shells than whatever the standard library does.  For example, we support the Sonnage Python shell interpreter, and also we're able to  move much faster.  For example, when PowerShell will suddenly become available on the POSIX operating system  2, we immediately added that now we generate that activation script on that platform 2.  And we're going to keep doing that, and we're going to make sure that it features work similarly.  And we have a three-layered configuration system, which basically allows you to have  some global settings customize how virtual apps are created on your hard drive.  And the kind of things that you can, this can be useful, is for example, whenever P  releases a new version, and for some reason it's broke, you can set it at operating system  level that I actually want to use an older PIP and create all my virtual environments  with an older PIP, not a latest, but still the less known working version or something  like that.  And we might also have a plugin system for extension.  This is mostly something that people are able to create their virtual environment implementations  for the pythons without necessarily having to get a poor request to us.  You can basically load and test it on your own, and then if it's stable, we can upstream  it within the virtual environment itself.  So yeah, that's going to be all of my talk.  I think I might have a question left or so.  So yeah, thank you.  So thank you very much for this rare insight.  There is time for one quick question and any other questions, you will be around afterwards,  so feel free to do that after the talk.  So is there any quick question?  Oh, sorry, go on, yes.  Thank you for that.  It was really interesting to see the internals.  You said you were planning on doing a rewrite which would include support to select any  Python version.  Yes.  Obviously, say you were doing front end, you can use NVM at the minute to practically  select any node version.  Right now, what is your go-to for downloading and managing Python versions as opposed to  the virtual end?  For Python versions, I either install it from upstream or I install, for example, using  PyVM or other things which are sometimes I use Konda which has.  But like this, for example, would allow us to even, you'd be able to say that you want  a Konda environment.  And if you have Konda inside the system, it would be Konda would actually create its own  specific environment.  And that could be still managed by the single front end of the virtual end instead of having  to use multiple tools.  Virtual end would kind of act like a unified interface which is very nice for tools especially.  Yeah, thank you.  One last thank you and welcome for this talk.  And then that's it.  Thank you guys.  Thank you guys.
Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_0_635.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_636_683.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_684_1319.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_1320_1367.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_1368_2471.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_2472_2519.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_2520_3011.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_3012_3587.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_3588_3647.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_3648_19895.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_19896_20123.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_20124_20219.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_20220_26399.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_26400_30527.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_30528_30599.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_30600_30683.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_30684_30767.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_30768_30839.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_30840_30935.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_30936_42443.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_42444_42527.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_42528_42779.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_42780_42839.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_42840_42935.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_42936_43007.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_43008_43667.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_43668_43991.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_43992_44171.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_44172_44591.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_44592_44651.png Bernat_Gabor_-_status_quo_of_virtual_environments_o1Vue9CWRxU_44652_44882.png
Thank you. So yeah, today we're going to learn about virtual environment and I have to make a confession. This talk is marked as a beginner and which is true that I'm not expecting from you any kind of previous knowledge, but we're going to dive in quite deep into the Python interpreter and its implementation details. So it's going to be a bit dense. To make it a bit more sweet, I did brought a few nice pictures of my Yo-Shite-Area puppy. He's silky, he's just a five-month-old, and this is basically his just after he got home, his first picture with a tennis ball in his mouth. So similar to him, we'll get on to a course to actually discover how virtual environments work underneath. Okay? So just a quick intro. Who am I? I'm the maintainer of the virtual amp tool, which hopefully would make me quite competent to talk about this subject. I'm a member of the Python Packaging Authority, and I'm quite active in the open-sport space, and you can find my contributions under this GitHub handle, Gabber Burnett. And I'm also maintainer of the Tox tool, which heavily uses virtual amps. This is one of the main reasons I became maintainer of the virtual amp tool. A lot of Tox tools, ToxBox, are kind of like virtual amp box. So my test path is there. And I'm also a software engineer at Bloomberg working mostly at Data Interest and Pipeline inside the company. So what is a virtual environment? Luckily, I don't have to come up with the definition. The PEP405 defines it very clearly. It's an isolated Python environment that has access to its own site package, shared success to the host building standard library, option isolated from the system site packages, and this is the point where you're like, whoa, what are all these kind of random definitions that you're throwing out? And why? What do we want to do here? And in order to understand this better, let's take one step back and actually cover, so how do we actually get hold of a Python module? Python is this famous language in which you can do pretty much everything by some package that is implemented, and you can get hold of some module. But how do you actually get hold of that module? And the answer is basically, in this case, the slide which basically means if you want to basically use anything, all you need to do is basically import it like that. So the thing is, the way you get something is by importing it. So how does the importing system work? Well pretty much you just specify import, you specify what you want to import, and then in the next line you can already magically use it. But the question is, what actually happens under the hood when you say that import.cl? And another kind of like complication around this is that not all modules are equal. For example, if I look at the cish, I can see that even if I print the representation, this is a built-in module. If I look at the os module, you can see that that's kind of like it has a file inside that optpy in 3.7 blah blah blah dash os.py. Similarly the.csv, this is more interesting because it's actually, you can see this implementation is an esof file, which is kind of like a DLL for on the windows. Then you have pytumble, which is a different location, or I can have like your py5.7, which is even a different location. So as you can see, just by the fact that they're living in different places, they are considered different type of modules. Like this one is a built-in module, the OS one is a standard library module. This is the module that comes if you install it, if you just Python, built into the standard library. And then we have also standard library module by dynamic module. These are modules that are, would be slow to actually implement it within Python. So what the core developer team does, they implement the actual underlying implementation inside C. And then they just create a slight front proxy to that. That's why in this case you can see this is actually underscore csv and not csv, because the csv is basically just a slight proxy. The actual implementation, the performance implementation is the underscore one, and that's available under some actual binary code. And then we have this import pyTaml, and this is a global site package, and it differs from the user site package in the sense that this is actually available on the system path, while this other one is basically available on the user site path. Like you can see, it's users burnout.local, which is kind of like users home. So the next section is, okay, now that we know that we import things, how does the import work, and this is something that has evolved throughout the Python's life, but this is how it works in 3.7. It works similarly on Python too, but the only difference is that there you could not modify this, it's not as easily exposed, so we can actually inspect it. But the idea is that Python doesn't know if something is importable or not. It just basically goes with the principle that try first, ask for forgiveness later. So it tries to import all the actual packages, and it has a list of importers, if knows, and it just asks in order each importer, hey, can you give me this sys, if yes, it's going to return that, it's not going to follow our importer to the first one, the first one can return it in the order, that's the one that Vincent actually provides that module for you. So we have the built-in importer, this is the first one, built-in importer imports elements which are literally compiled within the C Python binary, or like the Python binary, and in this case it's actually, such example is the sys itself. You're not going to find a sys.py anywhere, it's literally available already inside the binary itself. Then we have the frozen importer. By default, everything can be put straight into your Python binary, and the way you do it is you freeze it, and the freezing means I take your Python code, I compile it into the something machine code, and I literally attach it into the Python binary, and because this is no longer live anywhere on the file system, it's not on the no namespace of the built-in, it has this frozen importer which doesn't know at which location inside the binary I'm going to be able to find that element. Or my final option, at least in the default, is the pathfinder, and the pathfinder is named kind of like a in-chat, it basically looks at the operating system file system, and we'll try to find it somewhere there. So let's investigate it later. The pathfinder is basically just give me somewhere on the file system, and it doesn't operate, it has its own configuration, and the main configuration for this is going to be actually the information that you find in the sys.py. And the sys.py is basically a list of locations where this importer will try to look and find if that package is there. If in any of, and this again, we're going to do it in the order, and the first one is going to find it, that's what's going to be returned to you. So we have, you can have multiple elements, but this is like a list of defaults that I get on my Macbook, for example. You can see this third one is basically a zip file, like we can even import from zip itself. This is the first place it's going to look, then it's going to look in this sender library directory, then we have this libdial which is basically where we store the C extensions for the sender library, or we have the user size package or global site package. So now that we know all this, we can actually interpret this definition. So a virtual environment is just an isolated Python environment that has access to its own site package, so it must have something else besides what is there. But the system basically has access, while it has access to the sender library pass plus the user and the global site package, the virtual amp may or may not actually have access to the global one. So like in this case, you can see it has the first three, they are the sender library locations and the fourth element is basically its own site package. Or if you configure it to also have access to the global site package, it's going to append to end of this list, the user and the global site package. So virtual environment is nothing else, just a Python executable that whenever it starts up, it's magically going to make available its own site package folder and may or may not disable the global one. Quite simple. So the question is why do we want to do this, and the reason for why is because this will allow you to actually install packages to the system administrator right? Because we saw everything else other than user site package was available, only modified by the system Python, so if you don't have admin right, you might not be installing it. You also have the option that you can create easily and reproduce test environments. You don't have to reinstall your entire operating system just to test your package, you can just create a virtual environment and it's going to be exactly the same as you would have done that. And it's also easier dependency management and the way this manifests is that in practice, you would have multiple dependencies and if two tools, if you would install everything in a single global site package, you may end up in a situation when two of the site packages have conflicting dependency requirements and this would mean that basically by just installing a new package into that environment, you break another one which was already there. And it also just allows you, for as more like as a testing, the ability to easily test against multiple dependencies easily without even having to go too hard about it. So let's see how this actually works. So in the next we're going to try to look into, I told you already that basically what going to determine what is a module is going to be this sys.pat where something is going to be loaded is the sys.pat. Now this is that pat, the question one could ask, okay, but who comes up with this sys? Like this is just magically made up or and what more if you start up in a different platform, it may well be the case that not well may be. For sure it's going to be the case that you're going to have a totally system, totally different type of layout here or you may have more or less element. So the question to this is before, at the moment at least, there is different location where the standard library is identified and there's a different location where the other site packages are inside. All the standard library discovery happens straight into the Python binary. So this is kind of like a built-in logic. And then inside whatever customizes the global site packages is the side of pi. This is at least the most of the case. You can disable the side of pi. We're not going to cover that case too much, but in that case the built-in interpreter is going to fall back into some built-in logic, but this is very rare in practice. You're most often what's going to happen is that on your machine in the standard library there's going to be this side of pi which is going to magically customize and come up with the global and user site packages. So if you can look at it, you see there's some commonality in this. All those other packages which are outside of the virtual and its own site package and the user home site package. All these other ones start with this magic prefix called optpython 3.7. And this is a good reason because the way this list is generated is basically by using some prefixes. There is these two variables that you can change, but they're ultimately going to have some same defaults. This prefix and this is basically what's going to determine where the Python is going to try to find these packages. Now, obviously, Python doesn't just magically come up with, yeah, it's right here. It's like a very hard-coded version. It's actually dynamic. And what ends up happening is that there's a different algorithm that comes up with this logic. And this is actually available mostly in get.c. This has changed quite a few times as the Cpython code team, for example, started optimizing startup and they needed to move around to make things faster. But the rule of thumb is that if you set the Python home, we're going to use that for this prefix. So this is a great thing if you want to break someone's environment. You just set the Python home environment variable to something that doesn't exist and everything is going to be broken. But otherwise, if this environment variable doesn't exist, we actually are looking for a landmark file. And the landmark file chosen is actually this os.py. So the Python is going to start from its own executable. Here the Python executable itself is going to start going up and at each level, you're going to see, can I find this libpython.x.epsilon.os.py file? If I can find it, that is what six-dot prefix will be. And otherwise, it's going to be like a built-in hard-coded value, but this more like a fallback most likely will not work because this is whatever the compile, whoever compile your Python set is going to be exactly that one. And there is a different logic for the exact prefix, which is basically the C extension modules. The only difference is that in this case, it's not going to look for the os.py file. Instead it's going to look for the libdial node element, okay, the folder. And wherever it finds the folder, that's what the exact prefix is going to be set to. So that's more or less it. We only need, if you want to create a virtual environment, like roll your own, almost, the only thing you have to do is modify these six exact prefix and six prefix. If you're done with this, all your power is in your hand. So how we created this environment, in order to have shared access to the host, we just need to make sure that whenever Python starts up, it's going to resolve the standard library element by having available those two landmark files, the file and the folder, okay? If you want to actually create, we want to have the system side package added or not, this is just a conditional logic in the side.py. The side.py automatically gets loaded before any code gets executed at the startup of your folder. So if we do any changes here, whenever the user will get the interpreter, we'll already have modified sys.py that's going to work for him. And similarly, adding our own virtual environment side packages is building just some logic that we can tossing into the side.py, okay? So what are the changes needed to actually have a virtual environment? I just take a Python executable and I want to not use the system one. Well, we just need to keep track of the host so I can actually find this global standard library. We just need to know where that was available so I can set the sys prefix according to that and just some flex into the side.py for anything. Now according to the path I will say, this configuration information is actually sorting this pyvm.cfg and this is something that Python 3.3 or later actually already will parse and read and the way it's going to set the standard libraries for the standard library sys.profic is going to be basically going to look at whatever that home value is, okay? If that home value points to something from where it can find those landmark files, you're going to be happy, it's going to resolve the correct system side packages. And otherwise, just the fact that should we add the include system side packages is just this second argument that includes system side packages equals false and the fact that should we add the virtual environment side.py or not is basically just done by, is this file exist? Then yes, if not, we should not bother with that, okay? So quite simple and straightforward in that sense. Now there's one more thing, if you create a virtual environment, it's not going to be much of a use if you can't install packages in it. So the first thing that you'll have to do once you create a virtual environment, you want to be able to put a pip into it. Now this is kind of catch 22 because usually when I was someone who said install a package, you just go, yeah, I use pip to install it, but in this case we don't actually have at this point any pip in that package yet. So how do we install pip without pip? And the way we resolve this question 22 is basically we put, we have a wheel of the whatever we want to install or a zip file of the pip and we can actually put this onto the sys.pad and this basically is going to mean that for that invocation, pip is going to be available, it's going to be lookable by the importing system, so it's going to be able to start up and then we're going to ask pip, can you please install yourself by using this wheel that I just pointed to you at? And pretty much this is what in your self-installing itself and pretty much what ensure pip more or less does. It does a bit more magic, but this is the core gist of it, okay? So now we have virtual environment, it has a lovely pip in it, we're kind of like done, yeah? Well, then let's have a quick look at implementations. So the implementations out there available, the one that's recommended for use, so to say, is the VM. The VM is a standard library module, it's built in within the interpreter and it works only Python 3.3 only, but hey, Python 2 is going to be deprecated next year, so who cares? It's going to be also available of the standard library, well, mostly, because like Debian derivatives decide not to install it. So and the only main scope is going to be able only to target itself. So if you want to create a Python 3.4 or 3.6 virtual environment, you must have available Python 3.6 or 3.4 with virtual environment within it to be able to do it, okay? Now there's another thing called out here and this is for what I'm the maintainer is this virtual amp thing and how virtual amp differs from that built in VM, should this project even exist anymore or can we just drop it? And in order to answer this question, let's quickly go over what virtual amp offers on top of VM, okay? So the first thing out is that it's a third-party package, which means that you can upgrade it out of the band. So you no longer have to wait for the operating system to upgrade your Python, you can literally get the latest virtual amp creation by just installing this package, okay? So in that case, it's a bit more feature proof, it's a bit more easier to have the latest of it with the latest features. It also targets to support both Python 2 and 3 and it supports C Python and PyPy. We have some jiton implementation, iron Python is something that may be working progress, but if you have interpreter and you want to have a virtual amp for it and maybe you don't necessarily want it within your sender library, just submit a pull request and if it looks good, we're going to accept it. It's also cross Python support, meaning that you can actually, if you install virtual amp on Python 3.4, it's going to be able to create for you virtual amp for all supported pythons, okay? So even if the VM package is not there for the 3.6, it's still going to be able to create a virtual amp environment for that 3.6. This makes it very useful for tools such as stocks, PPM, all these tools, or Ppx, all these tools, because now they can actually don't have to check if the virtual amp is correctly set up or not, they can actually just have a look at it and they can just use this tool and this is guaranteed to be work for all available Python X YouTubers, okay? Another thing is cross-pattern support, obviously, and it has built and interpreted the discovery, meaning that this is especially handy on the Windows. You don't even, you automatically discover all pythons available or try to do our best, all pythons available in your operating system, so you don't have to figure out, hey, is my Python 3.4 on the path? Is Python 3.7 on the path, especially on Windows? For example, the path file for a loves on Windows tools to actually register their Python existence within the Windows registry and the virtual amp will automatically load this and it will be able to, you can just specify to it that dash p, I want the Python 3.4 and we're going to discover it and create it using that. And similarly, we're going to use a path discovery method by using path 394. This basically means that we're going to be able to create, you're going to have a single virtual amp for creator for all the pythons and you don't have to use, have different ones for different Python versions, okay? So and it also has its own bundle Pipset R sweep to the bootstrap, meaning that it can work in offline mode, but it also available to use index servers, so it's possible to install the latest one, rather than whatever we distributed the last package with, okay? So you may have like a newer Pipset up to the wheel and this in turn causes that you have less time, the awkward case when something doesn't work, you go to Stackoverfl and they did you upgrade your PIP? Okay, okay, now it works. So let's have a bit of, the other thing is though, because we need to support Python 2, virtual and virtual is a bit different than VM. Everything I told you in the beginning of the VM, that pyvm.dcfg is not a thing in virtual amp. So let's go over how virtual amp does actually create a virtual environment. So we have to basically do the same on alter the standard library logic discovery and alter the side of py logic to be able to create a virtual amp, but without the interpreter having any knowledge of any configuration files. And the way we do this is basically the first thing is we always copy the Python 2 at least the executable and we put it in a folder that looks like a Python virtual environment, okay, under bin and in this macOS. Then we create the two landmark files. We just simulink this to the operating system one, so they actually contain the same thing as the operating system one. Then we add a few configurations. We have a configuration flag if we should add the system Python or not, what is the prefix from where the Python that we created with is available and we inject our own side of py. Now the reason we need to inject our own side of py is because the side of py that is shipped by operating system is not aware of this magical oryc prefix or no global prefix. So if you would use the side of py that is shipped with the operating system, that would magically ignore all these configurations and you would not get an actual virtual amp. So for this reason, we have to like whenever you use a virtual amp, we have to like basically lie to you and have our own side of py that works for all platforms out there. Now you can imagine if this works for all platforms and all pythons, it's not too easy to maintain. So there's one more thing. The side of py is going to take care of adding everything on the path, but there's a caveat. It seems the side of py doesn't work on its own in magic. It actually works by using Python standard library modules, but at this point, at the point the side of py gets loaded, those are not yet on the path. So okay, this means that in order to be able to load or start up the side of py file and that's going to do the modification, we first need to be able to modify the, to have available all those packages that side of py happens to use. Now this is a short list of what we need basically. And again, this is kind of like platform dependent and each version dependent. So yeah, you can see that we create a lot more links or copy them literally over the standard library just so that our customization logic is able to run. And yeah, it's few I meant 21 in this case. I don't know, quoting a recent show, not great, not terrible. Could be 60 or 60, so 25 maybe. Let's work with it. And then we just have to do some fix up basically, add some activation scripts, link the include, fix up the distutils, because turns out the distutils uses its own kind of site package discovery. So we have to basically ship a different distutils than your operating system. And then we just fix all other bugs that are available. Long list of it like being able to discover Tlk platforms, having Python, the documentation run. You can see there's a long list of fixes that we have to do for this to work. And just, yeah, that's more or less this. Now, just to have a bigger thing, in this virtual environment, the only thing that we actually carry is that lonely side of packages. Everything else is just there to make things work. And just for a comparison, this is what VM has. It looks a lot slimmer, no? So and the reason why it can get away with having this slimmer package, this slimmer directory is because the interpreter itself is able to know about virtual environments and the interpreter itself is able to basically load these configurations from that pyvm that does CFG across it. So yeah, next time you create a virtual environment and you see this on your hard drive, don't despair, it's just how it is. Obviously, this is not a good way to have things. And because we basically say before we patch both the side of py and the disk tutorials, as opposed to what your distribution ships with, we have a lot of bug reports around ID doesn't work on this magic random platform because basically that has a customized disk tutorial, so the side of py, and various features start breaking. And because this is kind of like hard to maintain, I start doing a rewrite, and this is my proposal. Basically the idea is that it's going to still be offered as a third package, so this is what we're going to differentiate from VM. It's going to also have all Python implementations welcomed. This means including Python 2 and 3, like PyPy, even to the C Python, it's going to be deprecated next year. PyPy is going to be supported, it's two versions are going to support it for a foreseeable future, so we still need to be able to create virtual environments for that interpreter. So we're still going to support that. We're also going to have a two-year grace period, meaning that, well, I put here two years, but maybe I'm going to make a year and a half. The idea is that we're going to maintain our virtual AMP to be compatible with a Python that has drop support for its implementation a bit earlier, and the reason we want to do this is that you're most likely, unless you live in a perfect world, you're not able to migrate to the latest fanciest Python immediately, and we want to make your life a bit easier. You still can use virtual AMP to actually create VMs for your favorite deprecated Python. So it's also going to be able to select target pythons. It can even do fancy things such as, for example, request a given subversion or request a given architecture on Windows. Again, this is kind of like maybe not as interesting for the end user, but it's something that tools definitely want, and it's very handy. And one thing in order to minimize the bug request, we actually are going to, what we're going to do is we're going to ask, if possible, we're going to delegate the work to the VM in the spirit of why work if others can work for you, but we're basically just going to make, add a few optimizations on top of it. And we're going to have a broad shell activation support. This is something that differentiates for us, for example, also from the VMs, that we actually support a lot of other shells than whatever the standard library does. For example, we support the Sonnage Python shell interpreter, and also we're able to move much faster. For example, when PowerShell will suddenly become available on the POSIX operating system 2, we immediately added that now we generate that activation script on that platform 2. And we're going to keep doing that, and we're going to make sure that it features work similarly. And we have a three-layered configuration system, which basically allows you to have some global settings customize how virtual apps are created on your hard drive. And the kind of things that you can, this can be useful, is for example, whenever P releases a new version, and for some reason it's broke, you can set it at operating system level that I actually want to use an older PIP and create all my virtual environments with an older PIP, not a latest, but still the less known working version or something like that. And we might also have a plugin system for extension. This is mostly something that people are able to create their virtual environment implementations for the pythons without necessarily having to get a poor request to us. You can basically load and test it on your own, and then if it's stable, we can upstream it within the virtual environment itself. So yeah, that's going to be all of my talk. I think I might have a question left or so. So yeah, thank you. So thank you very much for this rare insight. There is time for one quick question and any other questions, you will be around afterwards, so feel free to do that after the talk. So is there any quick question? Oh, sorry, go on, yes. Thank you for that. It was really interesting to see the internals. You said you were planning on doing a rewrite which would include support to select any Python version. Yes. Obviously, say you were doing front end, you can use NVM at the minute to practically select any node version. Right now, what is your go-to for downloading and managing Python versions as opposed to the virtual end? For Python versions, I either install it from upstream or I install, for example, using PyVM or other things which are sometimes I use Konda which has. But like this, for example, would allow us to even, you'd be able to say that you want a Konda environment. And if you have Konda inside the system, it would be Konda would actually create its own specific environment. And that could be still managed by the single front end of the virtual end instead of having to use multiple tools. Virtual end would kind of act like a unified interface which is very nice for tools especially. Yeah, thank you. One last thank you and welcome for this talk. And then that's it. Thank you guys. Thank you guys.
confession. This talk is marked as a beginner and and its implementation details. So it's going to be a bit dense. To make it a bit more sweet, I virtual environments work underneath. Okay? So virtual amps. This is one of the main reasons I system path, while this other one is basically and find if that package is there. If in any of, from zip itself. This is the first place it's question is why do we want to do this, and the package, you can just create a virtual dependencies and if two tools, if you would okay, but who comes up with this sys? Like this Python is going to try to find these packages. this pyvm.cfg and this is something that Python okay? So now we have virtual environment, it has implementations. So the implementations out there to target itself. So if you want to create a Python 3.4 or 3.6 virtual environment, you must environment within it to be able to do it, okay? by just installing this package, okay? So in that just have a look at it and they can just use this is a short list of what we need basically. And and the interpreter itself is able to basically also have all Python implementations welcomed. two versions are going to support it for a customize how virtual apps are created on your for downloading and managing Python versions as