Managing Python Environments with pypi2nix
Jordan, Sebastian
This talk wants to give an overview over the usage and development of pypi2nix, a tool to generate and maintain python environments with nix. pypi2nix is already a viable tool to generate and manage python environments with their respective dependencies inside of nix. The first part of this talk will be focused on the current functionality of it. This includes features and shortcomings. The second part of the talk is about nixpkgs-python and how we use pypi2nix to maintain semi-automated self-updating sets of python packages. Finally we will discuss how pypi2nix could be used to maintain python package sets inside of nixpkgs in the future.
 I'm going to present to you Sebastian Jordan. He contributes to Next packages occasionally. And he's the maintainer and code developer to Pipeye to Next. And in his day to day, he watches way too many cartoon. Give it up to Sebastian Jordan. Good morning. Yeah, I want to talk about Pipeye to Next. And Next packages Python. And the goal of this talk is basically to give you an overview about what these tools are doing. And also ask for help a little bit, but more on that later. What is Pipeye to begin with? The official way to install Python packages, basically, probably everybody who used Python software at one point said, pip install package x. And that's how it was done until now. And we want to basically change that to use nix instead. And there are a lot of packages on Pipeye as of the day before yesterday. Yeah, 120,000. We probably won't be able to support all of them. But shoot for the stars, right? Python packaging has some issues. I don't know. If you've ever written a Python package, you probably know that you have to write the setup.py to make your Python code into a package. And yeah, this setup.py is just a Python script. So yeah, you can put any code in there. And a lot of people do that. Like if it's Windows or I don't know if certain check for certain C libraries and do something else if they are not there and stuff like that. So there's actually no way in general to get, for example, dependency information before you run the setup script. But give some meta information about installed dependencies when you build a package. There's real metadata. We, by the way, as a binary format of Python packages. And in there, you can find some information about what was done, what was installed. And we will see later that we pass exactly that kind of information. And there's another problem there. Because it's not a compiled language, there's no configure phase or anything. So if we want to build our Python environments, we also have to make sure that when we call our executable or import certain modules that the linker path is correctly set. And as I've already said, we pass the real metadata. And because it's not such a simple step as, for example, with, I don't know, kabai to nicks, if anybody knows that probably. Or no to nicks. We have to do a lot more stuff to get where we want. We do that in three stages. And the first stage is basically just pip install. We built an environment where pip is available and also your C dependencies that you need for installing your Python packages. And in this environment, which we create just with nicks. Yeah, we run pip install. That's it. Stage two is there to extract the metadata from the wheel, like license or dependencies, author, and that kind of stuff. And in stage two, we also calculate the checksums that we put in our to be generated nicks expression. Yeah, that's stage two. And stage three is just generated nicks expression from that. Yeah, our nicks expression that we generate with pipy to nicks have three parts. Maybe, of course, we have the machinery in place to build Python packages with the tooling from nicks packages. But mostly, it is just the generated expressions for downloading and building Python packages as you would find them in nicks packages, basically, because we use just the Python package from nicks packages. And there's an overrides part, which is similar to an overlay. It uses the overrides. First of all, the overrides are there to give the user the ability to insert special information that pipy to nicks couldn't find. For example, dependencies that pipy to nicks wasn't able to pick up because they were not declared properly in the Python package or special flags that are not flags, special environment variables that should be set on build time. And we use the fixed point calculation similar to the overlays. And I will later show how this might look. Yeah, this is just to show you this stuff in special. Like, this is part of the generated package set for CFFI, and it just looks like you would expect it in nicks packages. This is an example for overrides. Fortunately, this is not an overlay, so we can do stuff like packages and Python there in the top, because our self and super attributes in this case are only the Python packages and not nicks packages. But it's a little bit cut off, but you get the idea. We just overwrite the derivation to remove some stuff that we don't want to have in our setup.py. For example, version pinnings, because we want to take care of that. But nothing special, I guess, if you've ever overwritten any derivation. Yeah, now comes the fun part, the live demo. Yeah, I prepared a make file, and let's first look into that. Is this big enough to read? Yeah, most people who work with Python probably know about requirements.txt, but just to summarize, it's basically a format, like the requirements.txt file is usually there to specify what should go into, for example, the development environment as py dependencies or even in the production environment. And the format is you just list the packages you line separated, basically. You can put more than just package names in there, also version pinnings, but we won't do that in this example. Or you could reference git repositories. And we also have to install pyp2nix. This part. And then we generate our next expression with pyp2nix. And this is basically the example invocation for this case. I will quickly go over the flags. Pyp2nix supports Python 2 and Python 3. And the first flag, like the minus v3, just says, use Python 3. The minus r flag is similar to the pip minus r flag. It specifies the requirements txt file that should be used to generate the next expression. The dash e flag lets the user specify nix packages that should be available at build time or at pip install time. And minus lowercase v is just verbose output. And let's do that. So nothing in here. And now we want to make our requirements txt file. Nothing special. Just, yeah, there's everything in there. Now let's generate the requirements.nix file. Yeah, this takes, I don't know, 30 seconds or a minute because it has to compile some stuff. In this case, we built LXML, which is just one of the more famous XML parsing libraries for Python. And I chose this example to demonstrate that we can handle C dependencies. Yeah, and here you can see, yeah, we just run pip. Yeah. I don't know. Maybe we have to wait a little bit. Maybe I don't know. There's somebody questioned so far. Maybe we can do that in the meantime. Sorry? I'm really sorry. I didn't get the first part. If we can reuse overwrites. Like that you created for a specific environment. Yes, there is a way to, there is a way to, let's just show this. No. Result pipetune. There is a flag that lets you specify additional overwrites. I don't know. On the screen it's still there. Anyway, there is a dash dash overwrites flag that lets you specify any override file that you previously created. You can even get your overwrites file from URLs or Git repositories with the exact revision that is currently at master or you can even specify another revision. So there is a way to reuse overwrites and I will talk about that later. So this LXML library also depends on libXML too, right? Yeah. How can the generator find that dependency? Do you have to specify it manually? Yeah, that's a little bit dirty secret of pipetune. When you specify, you need libXML2 and XSAT, it will just be there for every Python package because we cannot in general get information about which package exactly uses the C dependency. So we just give it as a build input to all the Python packages in the whole set that we just generated. The Python packages can be derived from the... Yeah, the Python dependencies are derived from the pip build output. Okay, I will continue now with the demo. We just generated the requirements.nix file. And the... Like this screen which is also connected to the computer shows everything constantly. I don't know. Here's the generated expression for LXML. And in the... Okay. And I mean in this case LXML has no Python dependency, so in the propagated build inputs, it's empty, it's an empty list. And here is the dirty hacker I mentioned earlier. The common build inputs are now libXML2 and XSAT. And this would be true for all the Python packages that we would have put in there. And just to show that it works, let's make the interpreter... Oh, yeah. This is instantly because I previously tested it on this computer, so you have to believe me that... Nix ran. But yeah, let's quickly start up the Python interpreter and import LXML. Yeah. Works. Okay, let's go on with the presentation. Okay, so now you see what works. You've seen that we can generate Python environments and that we can use them, but there's a lot of stuff that doesn't work for now. And this is especially tests. And this is a big issue because we really have to support tests to make the survival option maybe to use the Nix packages. And even not, like, this is just necessary if you want to use production. We don't support multiple PyPy instances, so if you have a private PyPy server, you cannot use PyPy to Nix to, like, selectively pull from official PyPy and your custom PyPy server. And this is due to, at least to my knowledge, PyP has no way of telling the user from where it got the packages. Like, of course, we could pass it from standard out, but yeah, we don't want to do that because it would break every time PyP updates. We don't want to rely on the front end of PyP to generate our Nix expressions. And if anybody wants to do that, feel free to contact Rockormin. Yeah, how much time do I have? Ten minutes. Okay. Now I want to quickly talk about Nix packages Python. Nix packages Python is a collection of Python packages that are generated with PyPy to Nix. And there is a GitHub repository called Nix packages Python, which Rock owns. And yeah, it's semi-automated, self-updating. What that means, I will explain that later. It offers binary substitutions because we push build artifacts to the binary cache by SSH. And it also serves as a library for common overrides. Yeah, semi-automated, self-updating sets of Python packages. Yeah, we automated this with Travis CI. And there's basically a cron job running every night which executes PyPy to Nix and generates expressions, which can be used by the user. And it commits its changes that it made maybe because of a Python package update, upstream or whatever, to this repository back. There are multiple Python sets, like all the great stuff you know or use, but yeah, by far not everything. Like for example Django, I think it's only Django. It's no plugins, no extensions by now, but we plan on expanding that. And they're organized in groups that make sense. Like yeah, you would put, for example, PyTest with its extensions in one package set because you have to make sure that these work together and this is how we organize it for now, especially to mitigate really long build times. Yeah. And yeah, different versions for Python 2 and 3. I mean, this won't be an issue for too long, but for now it is. Yeah, it's a library of common overwrites. That means like there are already predefined overwrites that you can use and PyPy offers the dash, default, overwrites flag that puts automatically the overwrites from Nix packages Python in your Nix expression. This is also like, it depends on the revision of that was current when you executed this command. So if you generate Nix expression, it won't break in the future. Even if we update something that might break you, your old expressions would still work. This is not the default behavior, but it is planned to be the default behavior soon. Yeah. What we want to do is with Nix packages Python, we want to provide an easy way to install your Python packages and for that we would like to use overlays, maybe like as intended. And there is already an overlay definition in the repo, but that you could use, but as we saw yesterday, it still needs some overhauling. And we could generate the static PyPy from our expression. That would be also really nice to have with generation of Docker images or whatever, because if you have PyPy repository with only the versions you really need for the job, you don't run into trouble with version fagups or whatever. Yeah. To enable tests, we need to rework the Python infrastructure a little bit. Yeah, if that's possible at least. And there's currently no way to differentiate between build inputs for package like test inputs, especially like py test and runtime inputs from the way we generate our expression, so we have to figure out something there. Yeah. So now, I don't know, the future. We would very much like to use an automated tool in X packages to generate Python packages for basically end user consumption or I don't know, for packages that need these as dependencies. I don't know. But we definitely don't support this now. Like, yeah, don't expect us to integrate real with your next packages needs for now because we are just not there yet. We would like to support that, but yeah, for the reasons I already explained, it's a little bit difficult. So what we could do instead is generate a set of overlays and offer that to next users. This would probably be like the least intrusive way to get people to know the tool and how it works. Yeah. Building pull requests on Hydra would be nice, but that's probably what a lot of people think. Okay. Yeah. That was my talk. Any questions? We don't have much time. Any questions? So there is a new tool called the pip-m by Kenneth Reed, I think. So it's using pip-file and set of requirement.nix, requirement.txt, and generate pip-file.log, which have like the dependency, the version, and the hash of the package. So does that make sense to be used in pip-pytonix? It's officially recommended to be used by a Python organization. Okay. Yeah, thanks for the hint. We look into that. Actually, I didn't know about this pip-m command. Last question. So I was just wondering how do you select which packages to put into an expected Python? What do you only build dependencies of these three packages? The workflow until now was, okay, I need some Python packages for my work, so I put it in expected Python. But it was basically derived from what I and Rock needed for our purposes until now. So there is no formal specification on what goes into what kind of package set in expected Python until now. But we are open for proposals, of course. Okay. Thank you.
 I'm going to present to you Sebastian Jordan.  He contributes to Next packages occasionally.  And he's the maintainer and code developer to Pipeye to  Next.  And in his day to day, he watches way too many cartoon.  Give it up to Sebastian Jordan.  Good morning.  Yeah, I want to talk about Pipeye to Next.  And Next packages Python.  And the goal of this talk is basically to give you an  overview about what these tools are doing.  And also ask for help a little bit, but more on that later.  What is Pipeye to begin with?  The official way to install Python packages, basically,  probably everybody who used Python software at one point  said, pip install package x.  And that's how it was done until now.  And we want to basically change that to use nix instead.  And there are a lot of packages on Pipeye as of the day  before yesterday.  Yeah, 120,000.  We probably won't be able to support all of them.  But shoot for the stars, right?  Python packaging has some issues.  I don't know.  If you've ever written a Python package, you probably  know that you have to write the setup.py to make your  Python code into a package.  And yeah, this setup.py is just a Python script.  So yeah, you can put any code in there.  And a lot of people do that.  Like if it's Windows or I don't know if certain check for  certain C libraries and do something else if they are not  there and stuff like that.  So there's actually no way in general to get, for example,  dependency information before you run the setup script.  But give some meta information about installed dependencies  when you build a package.  There's real metadata.  We, by the way, as a binary format of Python packages.  And in there, you can find some information about what was  done, what was installed.  And we will see later that we pass exactly that kind of  information.  And there's another problem there.  Because it's not a compiled language, there's no  configure phase or anything.  So if we want to build our Python environments, we also  have to make sure that when we call our executable or  import certain modules that the linker path is  correctly set.  And as I've already said, we pass the real metadata.  And because it's not such a simple step as, for example,  with, I don't know, kabai to nicks, if anybody knows  that probably.  Or no to nicks.  We have to do a lot more stuff to get where we want.  We do that in three stages.  And the first stage is basically just pip install.  We built an environment where pip is available and also your  C dependencies that you need for installing your Python  packages.  And in this environment, which we create just with  nicks.  Yeah, we run pip install.  That's it.  Stage two is there to extract the metadata from the  wheel, like license or dependencies, author, and  that kind of stuff.  And in stage two, we also calculate the checksums that  we put in our to be generated nicks expression.  Yeah, that's stage two.  And stage three is just generated nicks expression  from that.  Yeah, our nicks expression that we generate with pipy to  nicks have three parts.  Maybe, of course, we have the machinery in place to build  Python packages with the tooling from nicks packages.  But mostly, it is just the generated expressions for  downloading and building Python packages as you would find  them in nicks packages, basically, because we use  just the Python package from nicks packages.  And there's an overrides part, which is similar to an  overlay.  It uses the overrides.  First of all, the overrides are there to give the user the  ability to insert special information that pipy to nicks  couldn't find.  For example, dependencies that pipy to nicks wasn't able to  pick up because they were not declared properly in the Python  package or special flags that are not flags, special  environment variables that should be set on build time.  And we use the fixed point calculation similar to the  overlays.  And I will later show how this might look.  Yeah, this is just to show you this stuff in special.  Like, this is part of the generated package set for  CFFI, and it just looks like you would expect it in nicks  packages.  This is an example for overrides.  Fortunately, this is not an overlay, so we can do stuff  like packages and Python there in the top, because our  self and super attributes in this case are only the Python  packages and not nicks packages.  But it's a little bit cut off, but you get the idea.  We just overwrite the derivation to remove some stuff  that we don't want to have in our setup.py.  For example, version pinnings, because we want to take  care of that.  But nothing special, I guess, if you've ever  overwritten any derivation.  Yeah, now comes the fun part, the live demo.  Yeah, I prepared a make file, and let's first look into that.  Is this big enough to read?  Yeah, most people who work with Python probably know about  requirements.txt, but just to summarize, it's basically a  format, like the requirements.txt file is usually there to  specify what should go into, for example, the development  environment as py dependencies or even in the  production environment.  And the format is you just list the packages you line  separated, basically.  You can put more than just package names in there, also  version pinnings, but we won't do that in this example.  Or you could reference git repositories.  And we also have to install pyp2nix.  This part.  And then we generate our next expression with pyp2nix.  And this is basically the example  invocation for this case.  I will quickly go over the flags.  Pyp2nix supports Python 2 and Python 3.  And the first flag, like the minus v3, just says, use  Python 3.  The minus r flag is similar to the pip minus r flag.  It specifies the requirements txt file that should be used  to generate the next expression.  The dash e flag lets the user specify nix packages that  should be available at build time or at pip install time.  And minus lowercase v is just verbose output.  And let's do that.  So nothing in here.  And now we want to make our requirements txt file.  Nothing special.  Just, yeah, there's everything in there.  Now let's generate the requirements.nix file.  Yeah, this takes, I don't know, 30 seconds or a minute  because it has to compile some stuff.  In this case, we built LXML, which is just one of the more  famous XML parsing libraries for Python.  And I chose this example to demonstrate that we can handle  C dependencies.  Yeah, and here you can see, yeah, we just run pip.  Yeah.  I don't know.  Maybe we have to wait a little bit.  Maybe I don't know.  There's somebody questioned so far.  Maybe we can do that in the meantime.  Sorry?  I'm really sorry.  I didn't get the first part.  If we can reuse overwrites.  Like that you created for a specific environment.  Yes, there is a way to, there is a way to,  let's just show this.  No.  Result pipetune.  There is a flag that lets you specify additional overwrites.  I don't know.  On the screen it's still there.  Anyway, there is a dash dash overwrites flag that lets you  specify any override file that you previously created.  You can even get your overwrites file from URLs or Git  repositories with the exact revision that is currently at  master or you can even specify another revision.  So there is a way to reuse overwrites and I will talk about  that later.  So this LXML library also depends on libXML too, right?  Yeah.  How can the generator find that dependency?  Do you have to specify it manually?  Yeah, that's a little bit dirty secret of pipetune.  When you specify, you need libXML2 and XSAT,  it will just be there for every Python package because we  cannot in general get information about which package  exactly uses the C dependency.  So we just give it as a build input to all the Python  packages in the whole set that we just generated.  The Python packages can be derived from the...  Yeah, the Python dependencies are derived from the pip build  output.  Okay, I will continue now with the demo.  We just generated the requirements.nix file.  And the...  Like this screen which is also connected to the computer  shows everything constantly.  I don't know.  Here's the generated expression for LXML.  And in the...  Okay.  And I mean in this case LXML has no Python dependency,  so in the propagated build inputs,  it's empty, it's an empty list.  And here is the dirty hacker I mentioned earlier.  The common build inputs are now libXML2 and XSAT.  And this would be true for all the Python packages  that we would have put in there.  And just to show that it works,  let's make the interpreter...  Oh, yeah.  This is instantly because I previously tested it on this computer,  so you have to believe me that...  Nix ran.  But yeah, let's quickly start up the Python interpreter  and import LXML.  Yeah.  Works.  Okay, let's go on with the presentation.  Okay, so now you see what works.  You've seen that we can generate Python environments  and that we can use them,  but there's a lot of stuff that doesn't work for now.  And this is especially tests.  And this is a big issue because we really have to support tests  to make the survival option maybe to use the Nix packages.  And even not, like, this is just necessary  if you want to use production.  We don't support multiple PyPy instances,  so if you have a private PyPy server,  you cannot use PyPy to Nix to, like, selectively pull from  official PyPy and your custom PyPy server.  And this is due to, at least to my knowledge,  PyP has no way of telling the user from where it got the packages.  Like, of course, we could pass it from standard out,  but yeah, we don't want to do that  because it would break every time PyP updates.  We don't want to rely on the front end of PyP  to generate our Nix expressions.  And if anybody wants to do that,  feel free to contact Rockormin.  Yeah, how much time do I have?  Ten minutes.  Okay.  Now I want to quickly talk about Nix packages Python.  Nix packages Python is a collection of Python packages  that are generated with PyPy to Nix.  And there is a GitHub repository called Nix packages Python,  which Rock owns.  And yeah, it's semi-automated, self-updating.  What that means, I will explain that later.  It offers binary substitutions  because we push build artifacts to the binary cache by SSH.  And it also serves as a library for common overrides.  Yeah, semi-automated, self-updating sets of Python packages.  Yeah, we automated this with Travis CI.  And there's basically a cron job running every night  which executes PyPy to Nix and generates expressions,  which can be used by the user.  And it commits its changes that it made  maybe because of a Python package update, upstream or whatever,  to this repository back.  There are multiple Python sets,  like all the great stuff you know or use,  but yeah, by far not everything.  Like for example Django, I think it's only Django.  It's no plugins, no extensions by now,  but we plan on expanding that.  And they're organized in groups that make sense.  Like yeah, you would put, for example,  PyTest with its extensions in one package set  because you have to make sure that these work together  and this is how we organize it for now,  especially to mitigate really long build times.  Yeah.  And yeah, different versions for Python 2 and 3.  I mean, this won't be an issue for too long,  but for now it is.  Yeah, it's a library of common overwrites.  That means like there are already predefined overwrites  that you can use and PyPy offers  the dash, default, overwrites flag  that puts automatically the overwrites  from Nix packages Python in your Nix expression.  This is also like, it depends on the revision  of that was current when you executed this command.  So if you generate Nix expression,  it won't break in the future.  Even if we update something that might break you,  your old expressions would still work.  This is not the default behavior,  but it is planned to be the default behavior soon.  Yeah.  What we want to do is with Nix packages Python,  we want to provide an easy way to install your Python packages  and for that we would like to use overlays,  maybe like as intended.  And there is already an overlay definition in the repo,  but that you could use, but as we saw yesterday,  it still needs some overhauling.  And we could generate the static PyPy from our expression.  That would be also really nice to have  with generation of Docker images or whatever,  because if you have PyPy repository with only the versions  you really need for the job,  you don't run into trouble with version fagups or whatever.  Yeah.  To enable tests, we need to rework the Python infrastructure  a little bit.  Yeah, if that's possible at least.  And there's currently no way to differentiate  between build inputs for package like test inputs,  especially like py test and runtime inputs  from the way we generate our expression,  so we have to figure out something there.  Yeah.  So now, I don't know, the future.  We would very much like to use an automated tool  in X packages to generate Python packages  for basically end user consumption  or I don't know, for packages that need these as dependencies.  I don't know.  But we definitely don't support this now.  Like, yeah, don't expect us to integrate  real with your next packages needs for now  because we are just not there yet.  We would like to support that,  but yeah, for the reasons I already explained,  it's a little bit difficult.  So what we could do instead is generate a set of overlays  and offer that to next users.  This would probably be like the least intrusive way  to get people to know the tool and how it works.  Yeah.  Building pull requests on Hydra would be nice,  but that's probably what a lot of people think.  Okay.  Yeah.  That was my talk.  Any questions?  We don't have much time.  Any questions?  So there is a new tool called the pip-m  by Kenneth Reed, I think.  So it's using pip-file and set of requirement.nix,  requirement.txt, and generate pip-file.log,  which have like the dependency, the version,  and the hash of the package.  So does that make sense to be used in pip-pytonix?  It's officially recommended to be used by a Python organization.  Okay.  Yeah, thanks for the hint.  We look into that.  Actually, I didn't know about this pip-m  command.  Last question.  So I was just wondering how do you select which packages to put into an  expected Python?  What do you only build dependencies of these three packages?  The workflow until now was, okay, I need some Python packages for my work,  so I put it in expected Python.  But it was basically derived from what I and Rock needed for our purposes  until now.  So there is no formal specification on what goes into what kind of package  set in expected Python until now.  But we are open for proposals, of course.  Okay.  Thank you.
Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_0_635.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_636_5063.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_5064_10679.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_10680_10979.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_10980_12479.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_12480_15923.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_15924_16727.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_16728_18287.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_18288_22511.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_22512_22643.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_22644_22979.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_22980_23543.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_23544_24767.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_24768_26171.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_26172_26399.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_26400_28823.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_28824_38495.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_38496_39791.png Managing_Python_Environments_with_pypi2nix_by_Sebastian_Jordan_ezb6a1_dtzQ_39792_41505.png
I'm going to present to you Sebastian Jordan. He contributes to Next packages occasionally. And he's the maintainer and code developer to Pipeye to Next. And in his day to day, he watches way too many cartoon. Give it up to Sebastian Jordan. Good morning. Yeah, I want to talk about Pipeye to Next. And Next packages Python. And the goal of this talk is basically to give you an overview about what these tools are doing. And also ask for help a little bit, but more on that later. What is Pipeye to begin with? The official way to install Python packages, basically, probably everybody who used Python software at one point said, pip install package x. And that's how it was done until now. And we want to basically change that to use nix instead. And there are a lot of packages on Pipeye as of the day before yesterday. Yeah, 120,000. We probably won't be able to support all of them. But shoot for the stars, right? Python packaging has some issues. I don't know. If you've ever written a Python package, you probably know that you have to write the setup.py to make your Python code into a package. And yeah, this setup.py is just a Python script. So yeah, you can put any code in there. And a lot of people do that. Like if it's Windows or I don't know if certain check for certain C libraries and do something else if they are not there and stuff like that. So there's actually no way in general to get, for example, dependency information before you run the setup script. But give some meta information about installed dependencies when you build a package. There's real metadata. We, by the way, as a binary format of Python packages. And in there, you can find some information about what was done, what was installed. And we will see later that we pass exactly that kind of information. And there's another problem there. Because it's not a compiled language, there's no configure phase or anything. So if we want to build our Python environments, we also have to make sure that when we call our executable or import certain modules that the linker path is correctly set. And as I've already said, we pass the real metadata. And because it's not such a simple step as, for example, with, I don't know, kabai to nicks, if anybody knows that probably. Or no to nicks. We have to do a lot more stuff to get where we want. We do that in three stages. And the first stage is basically just pip install. We built an environment where pip is available and also your C dependencies that you need for installing your Python packages. And in this environment, which we create just with nicks. Yeah, we run pip install. That's it. Stage two is there to extract the metadata from the wheel, like license or dependencies, author, and that kind of stuff. And in stage two, we also calculate the checksums that we put in our to be generated nicks expression. Yeah, that's stage two. And stage three is just generated nicks expression from that. Yeah, our nicks expression that we generate with pipy to nicks have three parts. Maybe, of course, we have the machinery in place to build Python packages with the tooling from nicks packages. But mostly, it is just the generated expressions for downloading and building Python packages as you would find them in nicks packages, basically, because we use just the Python package from nicks packages. And there's an overrides part, which is similar to an overlay. It uses the overrides. First of all, the overrides are there to give the user the ability to insert special information that pipy to nicks couldn't find. For example, dependencies that pipy to nicks wasn't able to pick up because they were not declared properly in the Python package or special flags that are not flags, special environment variables that should be set on build time. And we use the fixed point calculation similar to the overlays. And I will later show how this might look. Yeah, this is just to show you this stuff in special. Like, this is part of the generated package set for CFFI, and it just looks like you would expect it in nicks packages. This is an example for overrides. Fortunately, this is not an overlay, so we can do stuff like packages and Python there in the top, because our self and super attributes in this case are only the Python packages and not nicks packages. But it's a little bit cut off, but you get the idea. We just overwrite the derivation to remove some stuff that we don't want to have in our setup.py. For example, version pinnings, because we want to take care of that. But nothing special, I guess, if you've ever overwritten any derivation. Yeah, now comes the fun part, the live demo. Yeah, I prepared a make file, and let's first look into that. Is this big enough to read? Yeah, most people who work with Python probably know about requirements.txt, but just to summarize, it's basically a format, like the requirements.txt file is usually there to specify what should go into, for example, the development environment as py dependencies or even in the production environment. And the format is you just list the packages you line separated, basically. You can put more than just package names in there, also version pinnings, but we won't do that in this example. Or you could reference git repositories. And we also have to install pyp2nix. This part. And then we generate our next expression with pyp2nix. And this is basically the example invocation for this case. I will quickly go over the flags. Pyp2nix supports Python 2 and Python 3. And the first flag, like the minus v3, just says, use Python 3. The minus r flag is similar to the pip minus r flag. It specifies the requirements txt file that should be used to generate the next expression. The dash e flag lets the user specify nix packages that should be available at build time or at pip install time. And minus lowercase v is just verbose output. And let's do that. So nothing in here. And now we want to make our requirements txt file. Nothing special. Just, yeah, there's everything in there. Now let's generate the requirements.nix file. Yeah, this takes, I don't know, 30 seconds or a minute because it has to compile some stuff. In this case, we built LXML, which is just one of the more famous XML parsing libraries for Python. And I chose this example to demonstrate that we can handle C dependencies. Yeah, and here you can see, yeah, we just run pip. Yeah. I don't know. Maybe we have to wait a little bit. Maybe I don't know. There's somebody questioned so far. Maybe we can do that in the meantime. Sorry? I'm really sorry. I didn't get the first part. If we can reuse overwrites. Like that you created for a specific environment. Yes, there is a way to, there is a way to, let's just show this. No. Result pipetune. There is a flag that lets you specify additional overwrites. I don't know. On the screen it's still there. Anyway, there is a dash dash overwrites flag that lets you specify any override file that you previously created. You can even get your overwrites file from URLs or Git repositories with the exact revision that is currently at master or you can even specify another revision. So there is a way to reuse overwrites and I will talk about that later. So this LXML library also depends on libXML too, right? Yeah. How can the generator find that dependency? Do you have to specify it manually? Yeah, that's a little bit dirty secret of pipetune. When you specify, you need libXML2 and XSAT, it will just be there for every Python package because we cannot in general get information about which package exactly uses the C dependency. So we just give it as a build input to all the Python packages in the whole set that we just generated. The Python packages can be derived from the... Yeah, the Python dependencies are derived from the pip build output. Okay, I will continue now with the demo. We just generated the requirements.nix file. And the... Like this screen which is also connected to the computer shows everything constantly. I don't know. Here's the generated expression for LXML. And in the... Okay. And I mean in this case LXML has no Python dependency, so in the propagated build inputs, it's empty, it's an empty list. And here is the dirty hacker I mentioned earlier. The common build inputs are now libXML2 and XSAT. And this would be true for all the Python packages that we would have put in there. And just to show that it works, let's make the interpreter... Oh, yeah. This is instantly because I previously tested it on this computer, so you have to believe me that... Nix ran. But yeah, let's quickly start up the Python interpreter and import LXML. Yeah. Works. Okay, let's go on with the presentation. Okay, so now you see what works. You've seen that we can generate Python environments and that we can use them, but there's a lot of stuff that doesn't work for now. And this is especially tests. And this is a big issue because we really have to support tests to make the survival option maybe to use the Nix packages. And even not, like, this is just necessary if you want to use production. We don't support multiple PyPy instances, so if you have a private PyPy server, you cannot use PyPy to Nix to, like, selectively pull from official PyPy and your custom PyPy server. And this is due to, at least to my knowledge, PyP has no way of telling the user from where it got the packages. Like, of course, we could pass it from standard out, but yeah, we don't want to do that because it would break every time PyP updates. We don't want to rely on the front end of PyP to generate our Nix expressions. And if anybody wants to do that, feel free to contact Rockormin. Yeah, how much time do I have? Ten minutes. Okay. Now I want to quickly talk about Nix packages Python. Nix packages Python is a collection of Python packages that are generated with PyPy to Nix. And there is a GitHub repository called Nix packages Python, which Rock owns. And yeah, it's semi-automated, self-updating. What that means, I will explain that later. It offers binary substitutions because we push build artifacts to the binary cache by SSH. And it also serves as a library for common overrides. Yeah, semi-automated, self-updating sets of Python packages. Yeah, we automated this with Travis CI. And there's basically a cron job running every night which executes PyPy to Nix and generates expressions, which can be used by the user. And it commits its changes that it made maybe because of a Python package update, upstream or whatever, to this repository back. There are multiple Python sets, like all the great stuff you know or use, but yeah, by far not everything. Like for example Django, I think it's only Django. It's no plugins, no extensions by now, but we plan on expanding that. And they're organized in groups that make sense. Like yeah, you would put, for example, PyTest with its extensions in one package set because you have to make sure that these work together and this is how we organize it for now, especially to mitigate really long build times. Yeah. And yeah, different versions for Python 2 and 3. I mean, this won't be an issue for too long, but for now it is. Yeah, it's a library of common overwrites. That means like there are already predefined overwrites that you can use and PyPy offers the dash, default, overwrites flag that puts automatically the overwrites from Nix packages Python in your Nix expression. This is also like, it depends on the revision of that was current when you executed this command. So if you generate Nix expression, it won't break in the future. Even if we update something that might break you, your old expressions would still work. This is not the default behavior, but it is planned to be the default behavior soon. Yeah. What we want to do is with Nix packages Python, we want to provide an easy way to install your Python packages and for that we would like to use overlays, maybe like as intended. And there is already an overlay definition in the repo, but that you could use, but as we saw yesterday, it still needs some overhauling. And we could generate the static PyPy from our expression. That would be also really nice to have with generation of Docker images or whatever, because if you have PyPy repository with only the versions you really need for the job, you don't run into trouble with version fagups or whatever. Yeah. To enable tests, we need to rework the Python infrastructure a little bit. Yeah, if that's possible at least. And there's currently no way to differentiate between build inputs for package like test inputs, especially like py test and runtime inputs from the way we generate our expression, so we have to figure out something there. Yeah. So now, I don't know, the future. We would very much like to use an automated tool in X packages to generate Python packages for basically end user consumption or I don't know, for packages that need these as dependencies. I don't know. But we definitely don't support this now. Like, yeah, don't expect us to integrate real with your next packages needs for now because we are just not there yet. We would like to support that, but yeah, for the reasons I already explained, it's a little bit difficult. So what we could do instead is generate a set of overlays and offer that to next users. This would probably be like the least intrusive way to get people to know the tool and how it works. Yeah. Building pull requests on Hydra would be nice, but that's probably what a lot of people think. Okay. Yeah. That was my talk. Any questions? We don't have much time. Any questions? So there is a new tool called the pip-m by Kenneth Reed, I think. So it's using pip-file and set of requirement.nix, requirement.txt, and generate pip-file.log, which have like the dependency, the version, and the hash of the package. So does that make sense to be used in pip-pytonix? It's officially recommended to be used by a Python organization. Okay. Yeah, thanks for the hint. We look into that. Actually, I didn't know about this pip-m command. Last question. So I was just wondering how do you select which packages to put into an expected Python? What do you only build dependencies of these three packages? The workflow until now was, okay, I need some Python packages for my work, so I put it in expected Python. But it was basically derived from what I and Rock needed for our purposes until now. So there is no formal specification on what goes into what kind of package set in expected Python until now. But we are open for proposals, of course. Okay. Thank you.
of this talk is basically to give you an overview packages, basically, because we use just the you this stuff in special. Like, this is part of people who work with Python probably know about semi-automated, self-updating sets of Python expression. This is also like, it depends on the it won't break in the future. Even if we update X packages to generate Python packages for people to know the tool and how it works. Yeah. officially recommended to be used by a Python dependencies of these three packages? The