Conda - Easier Installs and Simpler Builds
Müller, Mike
Mike Müller - Conda - Easier Installs and Simpler Builds Learn about `conda`, the package installer from the scientific community. It offers very interesting features that can improve your installation experience considerably. The talk gives an overview of the basic usage of `conda`. It covers the topics installation and building of packages. `conda` can be combined with `pip` to use all PyPi packages. Its cross-platform and multi-languages features combined with power environments can help to improve your productivity. - The BSD license `conda` is a package installer for Python and other languages. While it originates form the scientific Python community, it can be really useful for all Python programmers. Installation of Python packages has become much simpler over the last years. The use of `pip` and `virtualenv` simplify the installation of Python packages a lot. However, they are specific to Python. The Python-agnostic `conda` has advantages for packages with C or Fortran extension that are very common for scientific libraries. `conda` is cross-platform. According to different statistics, the most Python users work on Windows. Often is especially complicate to get extensions with many dependencies installer on this platform. `conda` facilities the installation for Windows considerably. This talk introduces the basic usage of `conda` to install packages. This includes the basic commands for searching and installing of packages. Furthermore, the talk demonstrates the creation of environments for different Python versions and combinations of packages. The building of a packages is simple. The talk shows how to build recipes that contain declarations of dependencies . `conda` can work together with `pip`. This allows to use all packages from the Python Package Index ( PyPI). The talk explains the concept of channels that allow to get packages from different sources.
 Okay, everyone give Mike Miller a big hand. He's gonna talk to us about condor Okay, thank you very much for the great introduction I Would like to talk about condor easy installs and simple builds just a real quick raise of hands Who of you knows what condor is? Third and who is actually using anac are using condor Just about 20% thank you very much condor is Quite a few things actually it's an installer similar to pip, but I think a bit better It's an environment manager similar to virtual and but still ever a bit better. It's cross-platform. There works on Windows Linux and Mac It's not limited to Python so you can use it for different languages. Actually. There are people using for Ruby and follow languages And actually it comes from the scientific community, but I think it can be useful for every Python programmer So it's PC licensed and typically you install it either install me condor or anaconda I would talk about this different condors here So there's what's called mini condor and it's a kind of a small bootstrap like version includes Python itself condor and a bunch of dependencies like pip and wheel and some other tools and It provides access to many hundred libraries. You can install a lot of libraries with condor maybe even thousands and You just need to say condor install either talk about this process a little bit and there's something called anaconda which is Different thing it's a distribution similar to like if you like a Linux distribution So you get Python packaged up with a bunch of scientific applications and And this includes of course Python condor condor install and is about more or less like scientific 200 scientific factors you could pop off here So supposed to be there but scientific package about 200 or so and if you install it it takes about two gigs of your hot drive But then I think a lot of scientists need a lot of these tools and you don't have to fiddle around installing the old separately They'll come in one thing and then then it barely pay a little bit price was some dis space but this space is cheap and Programmers time is expensive And there's something called channel so when you install something this condor condor looks for a channel and chon channel is a place somewhere Somewhere on the internet was maybe even on your own machine. That doesn't it's also possible That looks for packages and The default is anaconda server, so we don't say anything typically looks there But there's also something called condor forge, which is a channel There's a lot of open-source packages by the pricing community But you can also have private channels so you can have a channel with your own name These are company name whatever and then you can install from this channel by and say say install dash see My channel my channel is a name of the channel and then it will grab the package from this channel will be included There's also a lot of configuration options which channel has precedence and which channels look first You can configure this in a configuration file Okay, a few basic tasks. I want to introduce you to what you can do with it That's by no means a comprehensive introduction, but gives you some feeling what you can do so you can install packages you can create and Administer environments and you also can create packages yourself so Writing is that a pie as you know and doing a lot of steps to create a package a First thing you can search for something so you say condor search and you type in the name of the Package you're looking for I use pandas pandas is very much used package and scientific library and then corner goes out there and is searching Places so I added condor added a few channels you see every dot you see they're fetching packages dot dot dot dot And every dot means looking for a channel so I have by a bunch of channels It's looking at and then gives you a long display I just cut out of a few things when you see the three dots there It's cut out and it lists you those then the package so everything this search looks everything has pandas in its name and shows you the version that's available Though all these versions are available is a normal one dot a dot here means it is available And it's downloaded though it when you're down when you install something it puts it in a local repository and You install from there. You don't have to download it every time and the one with the star here was a straight That's the one I have installed right now here and you see if it had to dependency on umpire was a scientific focus tells you have numpy one dot 10 and 35 and build zero and it comes from default switches for my naconda But there's also a version that comes from conda force. It gives you the channel here that comes from But it's pretty interesting can be very long list and but it shows you what you have So it's easy a bit longer here and by the cut out quite a few it can be several pages long easily Okay, the search can be it more fancy so That's a problem. I have a pit when you search for something for church for jungle. It just gives you I don't know how much very very very long listing this everything has a good jungle inside here You can say for the full name and now it displays only the package It's called pandas not all the packages have pandas in their name and now I get only pandas It's still pretty long list, but it's much shorter ready And I know it that's pandas and that that's a good feature and there's more so you can also Limits a search or say I want to have only Even though I did it on my Mac I can search for all the available packages for windows in 32 and I also can specify the version I want now shows me all the available packages for Versions 0.181 and it tells me okay here now it offers everything starting This is numpy 110 Python 27 up to one pie numpy 111 Python 35 So there's no dots and last race because obviously if I want Mac at thousand You need to download it here, so that's not there But it would be available and would download it and they all come from defaults and there's more options if you want There's quite a few switches you can use and can make this search even more specific pretty interesting feature Installing the package is very easy You just say condo install and you name the package and it goes through your channels and you can at the conflict finds it The channel comes first attacks on this channel see now it provides me with a name I want to install this package and typically tells I'm going to download here doesn't say because I've downloaded ready Typicalist, okay I want you to download these packages and then I install these ones like mkl's mask on a library all the dependencies typically if I have a properly packaged Application in a conduct package and all the dependencies are there installed all the dependencies and everything's binary So everything especially is scientific field. There's a lot of C and Fortron extensions and And compiling these things typically works in Linux it might work on OS X and is right like it doesn't work on Windows So that's why it's and still I think most people still Windows users So the most users of Python I think on Windows as According to all the statistics I've heard about and a lot of scientists and a lot of engineers So I teach a lot of engineers and most of the engineering companies are mostly Some of them have using Linux But by far the biggest majority in Windows so need to support Windows users and it's a pretty good way everything's binary and you can Do this okay? Create an environment so this is the first thing installing something but very often we don't want to install in your what's called the root environment You want to create environment? That's what is this command up here. You can see conduct create Dash n or dash dash name so I create a new environment which I call my pi There are three five and I can specify the Python version and I say I want to have Python 3.5 And you can also have if you wanted to could have a 3.4 or 2.7 or even 2.6 so far it supports 2.6 2.7 I'm not sure 3.3 but 3.4 3.5 and We are pretty quick as we know 3.5 is out. I won't take along 3.6 3.6 3.6 We'll be supported typically a new numpy version is supported within 24 hours or something pretty fast And I tell you what it's going to do. There's it never does something this out of asking And it tells me okay. I need to download these packages and I need to install these packages or the dependencies And then if you say yes, which I did then it starts creating this package and gives you this The car cut off here, but it shows you how long it takes and how fast is doing stuff Okay, now you created environment to in to actually activate Now I can actually list environments. I created environments and I don't have any one I've just a small selection of the environments I have you say content and list for instance It's one of the commands that shows you what environments you have and you see an environment is nothing But it directory in your home folder in my my user folder under conda under conda amps So if you install mini conda would be mini conda amps and here's another conda amps It's just a different directory and these directories contain all the information about environment and you see now The root environment is a It's the one that's active right now. It has this Okay, you see I gave the trailer the pie data Berlin here Then I made a new post tutorial. I made a new environment and I don't mess up with my route I can install what I like different versions and everything is isolated Okay, activate an environment so depending if you're on a unix kind of system Linux in OS X Then you have to say source activate name of my environment on windows is just activate so typing source activate is pretty long. So I just had defined a short alias But that's to be how we how we activate an environment and just type deactivated deactivated again And then the prompt changes and then you see prompt and now when you list it again, you see now the environment will be Activated you always can see which one's active Okay, and then you can say list and Then it gives you a list of all the packages you have installed you see it tells me again the name of the package the version Here this dependencies the numpy or not the Channel and this one is red because this when you see it's installed this pip So you want you will find maybe a few hundred packages there But you can install this conda directly if not you can always install this pip and conda will recognize this one So it's not a contradiction to work. I love this conda pip. We can use both. That's what I typically do It's not available this conda. I use pip and install this pip, but conda raises these packages And that's why I say it's installed this pip Yeah, so you have the version is installed this pip and you can also install deals. There's no problem So there's no contradiction in this and you have this pip installed there also Okay This is if you're kind of a consumer of packages you can install it can contain environments There are way more options. I just show a few but now the next thing is building a package So you want to build a package and there are essentially two ways you can build a package you can just take a package from pipe I and Converted into a conda package with a few commands or you can start from scratch Bring it with your own package Okay, building from pipe. I is pretty easy. There's a skeleton first. You need to install Conda build so conda builds if you have an account that's there already if you have mini con there need to install conda build which is Command line tool that helps you to build a package and then you say conda skeleton pipe. Hi pipe. I might package So this one goes to pipe. I fetches The package we have there and then you say conda build my package. That's pretty much it So it's not very difficult and it's built a package and the package is a is a tar ball It's a car PC to in this case. It's just one file with everything inside And that's it and now we can install it and per default it creates it for the for the platform You're on so if I do this as you can see here's now score as X And You can now install it and there are actually two different ways installing it right now I can say conda install use local my package and you just will go to the place because I know that you thought place We have put it and installed it but you can also if you wanted to if you had somewhere else You could just specify the full path. We'll just file name. This is exactly the same here But it could be a different way just if you download one of these you can always install it is conda install from the command line You can also specify a Python version So even if you have a different Python you can say okay build this one for Python 3 4 and it builds it for 3 4 Yeah, so even if it's pure Python it always has a Python version inside and this is especially useful if you build extensions and see or Fortran which is very common in scientific field and This one builds everything for this You can also convert to other platforms you see I can say conda convert platform all for instance and just specify my My file and say output directory and if it's a plain Python version It will create a version for each platform for Windows Linux and Mac If it's this extensions is not that easy then you need actually need the operating system because you need to compile it So then I'm I don't think it's there's a cross-compile at least I didn't try and I don't know but So so far I compiled on Windows and Mac and I think in Linux is the same To be Linux is easy to compile it anyway So if you compile extensions it works and I even compiled Fortran extensions Which is typically more involved than C-SIMC and the rules nicely so far so you can create this extension source and then if you like to you can upload and You need the Anaconda client and now I'm uploaded to the Anaconda And you can also make your own channel if you like so I don't have it here But you can also create a channel and you just say an icon upload and just give it full pass of this file and it uploads This thing and a condor and other people can install it just by saying condor install name of your package and If you put it in different channels, they have to specify the channel when they install it Okay, this is building From a PI PI package you can also do from scratch So if you want you can customize things and you need a few things you need a meta yaml so a yaml file that Declare to the flea specifies what you want to do and Then you need a build file and there's two different ones if you're in Windows you have it build dot BAT that file a batch file and then Linux and Mac you have a shell file and You just use a normal setup dot py so you probably familiar with it So you just use the normal setup dot py and then you have additional information in the meta yaml Which also has dependencies in it Let's have a look at this file So the meta yaml looks like this is just the yaml file and has a lot of different Sections in there. You don't need all of them So you have something about met information about the package then you have the sources so you can say it comes from some It's URL something like this you can have requirements And there are different requirements your violence for building things and provides for running For instance, if you use syson you might need syson to build it But you might indeed install on the client machine because you just have this to be the extensions. You have different dependencies for the tests what you do with the tests and then you have normal information about who the also is what the licenses Okay, this is just a short part of an Event I am I used so I used just my package name here and you say okay the sources just Depends where all two directories up there can be a relative path for instance. It doesn't have to be URL Requirements of Python and set up tools and then for running I need Jupiter and see here This is something you can specify this one. I Need it only for Windows and the same way you can specify Linux and Mac. So this will be only installed For the Windows version but not for the for the version of other operating systems. That's as well Very important and you can use the same syntax for and other places. So you make it dependent on the platform with one file Of course, and you have this MIT license, whatever you want to put in there Then you have this build files and they're typically if there's a skeleton They just you set it up and you create files that look like this and they are just you see You just call Python and set up to py install and they just call your setup to be right The setup device just a normal one as you know from the normal installation process But you can also add other commands in here, which is very interesting So if something is not available as a content package in here You can do a pip install something and this will install the package and this will go into this top ball So everything you pip install the you will deliver the same package. I had this problem I had the library that was not available as a content and difficult just Instorted and she's good distributed alongside with your package. There's quite a few things you can do here in this build file You can customize things or something doesn't work typically have a way to do it in build file some way or the other Okay, and then just build it is a conduct with my package and that's it And there are all the other steps are exactly the same as with the skeleton So you can do upload and all kind of stuff install from from the local machine These kind of operations Good That's it already so conclusions I think conda is a very interesting package and it's very much used in scientific field But can be really useful for all Python programs because it's a pretty good installer in a very nice package manager You can manage environments and you can also build conduct packages It works together with pip so it's not a contradiction You can still use it if conda doesn't supply the package, but as you can see you can always convert a pip installer package into Conda package if you like that might make the installation more smooth for it try for your users then you might actually Just convert pip package to a conda Very well known in scientific community if you go to a sci-pi your sci-pi conference pretty much everybody will know about it I think but it can be really useful for For all Python programmers and I encourage you to give it give it a try Thank you very much, and I'm here for questions Because and ask them for me right first hand went up here Thanks I understand I can use conda for distribution and platform independent package installation, which is great now if I Have I am a maintainer of a tool and I have lots of different dependencies Outside pipi and inside pipi and with npm and whatever and it would be nice to be able to just Use conda install whatever for me Would I have to package all those npm pipi packages to conda manually or Is it possible to I don't know say conda just install this and install it from pip because you don't have the package yet Depends there's just some things you can do so if you install a packages that that is conda Then this package will install its dependencies So if you and say I say I depend on pandas pandas has a bunch of dependencies and install numpy for me and so on To some degree you can do pip so far you have to as I think you have to write it in the In the in this build script you have to say pip install pip install pip install But probably the this meta yaml will be improved That's what I just saw a ticket that you can then even say okay This is supposed to become from conda. This is supposed to be pip and this might be a different channel So it's still evolving and not everything works perfectly. There are some Problems, there's no problem. There's a perfect but so far you can always have a walk around by putting it in the build script I think So what would be your suggestion for this kind of situation? I'd put it in the build script. I had the same dependency thing and I put it put a pip install in the build script in this build SH and it worked for me so that was was the easiest one the other thing is just you can always convert the pip Packages into contact packages and then you can list them in a matter yaml and just upload them as long as they're BST MIT lesson shouldn't be a problem and anything just upload them to Contact out and that's it. It is just you can automate it You just need to list all the packages and then you can update it if it's in a version Just check if the new version and just build new packages. Okay. Thanks. Thanks Thanks for the talk so one of the selling point at the beginning that is that not limited Not only to Python right but all the examples have seen all the pattern packages So what do you mean by not limited to Python? So you can use this everything is declarative so people use it for Ruby and other languages So you can use the same thing and you can also have Dependencies that are not necessary Python for instance one package is called pie tables as the interface to htf5 And this depends not only a numpy but also in the htf5 library There has to be in a different version and all the things can be packaged in there So it you can have depend ELLs char libraries You can include them in your thing and that makes it much easier if you have a good if you have a package of the at the end then the user doesn't have to do anything about fiddling around this include parsing all kind of stuff suppose Compiling htf5 is not the simplest thing. So I tried this once before condor And you can easily take two days out of your life and you still haven't got anywhere All right, thanks Hello, thanks for your talk very to the point Given that recently we have been adopting on PyPI binary wheels They seem to solve most of the issues that condor seems to solve Other than the fact that you have a single command to manage your install packages in virtual environments. That's good It's nice, but what's the Big advantage now that binary wheels are taking off You write wheels is a great step forward as I say that for things not everything is in a real format yet and you might not have the right version and the right platform in the wheel and Condor can install wheels no problem. So you say people install and install the wheels. So that's not a contradiction this way This at this as far as I know the story goes like this Travis Elephant. He's a CEO of the continuum which is behind this it's open everything's open source But there's a company behind it and he asked Peter van Rossum a while a few years ago about this packaging and because I just go Ahead and do it. That's what it did. So it's kind of a little bit a parallel to some degree a little bit parallel to pip and wheels something like this, but it's I think it's more comprehensive and It's better and many has a lot of nicer features and pipping wheels for instance the search for name I couldn't I don't know maybe I'm not smart enough I couldn't get pipped into just showing me this package it shows me everything shows me hundreds of packages for Django And this is not really useful to have to run it through grab and do your own thing. Why not just say I wanted Many just customized search and those things aren't this the condo the condo is nice But you as I said you can use everything that pip and wheels and pipe your eyes You don't exclude it you can still use it and I do it very often because it does only if you have the packages Or maybe now a thousand out there, but not eighty thousand or ninety thousand whatever it's on All right any more interesting questions for Mike here we go. Yes Best question yet Okay, so is there like any Standardization effort like are you working with Python packaging authority or something or is it going anywhere? Or will it be forever separate like conda is great supposedly but only in the scientific community and like everyone else This is bad Great question, but I don't know the answer Would be very nice. I would I would like to see it this way that both but actually everything's open source and both communities could use the code from the other Easily no no problem. I think everything is BSD or MIT license I shouldn't be a big problem to use code and the I don't I'm not really sure why this is not a bit more coordinated efforts And I don't know it would be very nice to have one stop and use join forces and do the gathered for For these things and I don't know why it's the case. There might be some reasons and I haven't really found out yet More questions more questions keep the dream alive everyone I I Have a question so like a two-parter so presumably if I install anaconda It's gonna come with its own versions of Python as well as my system pythons that I might have installed separately And I once had this problem where or you know I don't know if anybody else seen the problem where you upgrade your operating system and you upgrade your system Python and all your Virtual ms are now broken because something something something which I don't really understand And you have to delete all your old virtual ms and recreate them is anaconda gonna magically fix that problem for me or sub question If I upgrade, you know, if there's a Python 3.5.1 and a Python 3.5.2 How does anaconda manage the upgrade process there? Does it need to rebuild a bunch of things? There should be a problem so if you work with the environments those environments are pretty much isolated under totally and Isolated sometimes if you use command line tools and they're installed like in the scripts directory is from there if not it goes outside So if you say pile in something and you haven't put installed by linden your environment It might take the wrong version and gives you a lot of syntax errors because it's a pile in for two seven and you and a three five environment But if you install everything it puts everything in the directory and there shouldn't be a problem and you update if you work with environments and You will see if you work for while you will focus environments anyway all the time you will rare Actually, maybe the best approach just have the root just as a helper and then work in environments So isolated of course you can always if you can always reproduce the environment you can freeze the environment It just just specify what you have installed You get a big list of installation and then you can give this this file to somebody else and the other person can say okay Install from the file and then store exactly the same versions of the ones you can reproduce the environments and they should solve the problem I Don't know if that could be some you can you only find it out if you test it really and do it But theoretically there shouldn't be a problem practically there might be one. I don't know Today is that only me that's ever had that problem of changing the system Python and all the virtual ends are broken So so I guess we're thinking that anacondas like kind of even more isolated from system pythons than than virtual end managers if It never has that problem Yeah, there's some voodoo there further research required any more questions. Yes. Yes fantastic All right. No, he wants to hear my voice any longer and absolutely necessary Are the environment Table or is there any plan for that? Can you repeat a question that didn't get at the environment Relocate a ball. So I Installed somewhere. Let's say I caved the environment and and Compress it somewhere else like another Computer. Yeah, so I just said it you don't actually copy the environment You just copy all this average all the packages are installed as a version. So but I showed you this listings here So one of this earlier things shows you this Whatever is installed you have this List somewhere do I have a list? Yeah, something like this. So you can actually Freeze no, that's a search add a somewhere you can tell tell okay export all the versions I have in a file and you just get a file with all the names equal The version and then you can give this file to somebody else and somebody else can recreate exactly the same environment exactly the same versions I'm not speaking about Recreating the environment, but just taking the environment and put it somewhere else You should work. It's just a directory The problem is not everything is copied though if like I have this pandas version I have may I use it in 10 different environments, but it's only once in my in my Repository and not directly in the files just a sim link and you have to make sure that that you copied it along with Is it but I think there should be an option to copy everything just sip the whole directory input to some belts It should work as long as the same operating system But I no guarantee but theoretically should work can try it out But maybe recreating is it's a safe of thing So if you recreate you can also say okay, it freezes everything with except the same version But you can also go in the file and edit is okay It shouldn't be exact the same version should be at least the version or even update the version so you can go in the file and Customize it and then still everybody who has this file would create and this nice thing is would work I can freeze on my Mac and somebody else can create the same environment or windows It shouldn't be a problem if I sip the file and would only be for the same operating system All right any more for anymore it is now or there is one okay You have to make it a question you talk very very fast and Mike will talk very very fast Okay, so let's say that I have a package Python package on Linux that has that needs separate installed C library and Let's say that this C library expects another C library very so what will come that it will Download this library and all of its dependencies dependencies and put it in the condo repository locally or What will that different ways I had the same problem typically what I did I just included in the file so I had this You can you can specify the dependencies so if you compile something I had Fortran dependencies some shared libraries somewhere and you included so And I'm not read myself. I'm not sure how to the best way to do it, but that different ways including Shared libraries and you can distribute the shared libraries along and they will put the shared libraries in this environment They wouldn't clash with your shared libraries because you might have to share the maybe different versions Then it works for while and until it doesn't work anymore because this is a different version something is different So you can distribute shared libraries with it. There's different ways of doing it It had you have you still have to look at there's a lot of recipes out there how to do it and probably you will find one that serves your purpose Okay, let's give Mike a mother big hand. Thank you very much for a great talking during the break
 Okay, everyone give Mike Miller a big hand. He's gonna talk to us about condor  Okay, thank you very much for the great introduction I  Would like to talk about condor easy installs and simple builds just a real quick raise of hands  Who of you knows what condor is?  Third and who is actually using anac are using condor  Just about 20% thank you very much  condor is  Quite a few things actually it's an installer similar to pip, but I think a bit better  It's an environment manager similar to virtual and but still ever a bit better. It's cross-platform. There works on Windows  Linux and Mac  It's not limited to Python so you can use it for different languages. Actually. There are people using for Ruby and follow languages  And actually it comes from the scientific community, but I think it can be useful for every Python programmer  So it's PC licensed and typically you install it either install me condor or anaconda  I would talk about this different condors here  So there's what's called mini condor and it's a kind of a small bootstrap like version  includes Python itself condor and a bunch of dependencies  like pip and wheel and some other tools and  It provides access to many hundred libraries. You can install a lot of libraries with condor maybe even thousands and  You just need to say condor install either talk about this process a little bit and there's something called anaconda which is  Different thing it's a distribution similar to like if you like a Linux distribution  So you get Python packaged up with a bunch of scientific applications and  And this includes of course Python condor condor install and is about more or less like scientific  200 scientific factors you could pop off here  So supposed to be there but scientific package about 200 or so and if you install it it takes about two gigs of your hot drive  But then I think a lot of scientists need a lot of these tools and you don't have to fiddle around installing the old separately  They'll come in one thing and then then it barely pay a little bit price was some dis space but this space is cheap and  Programmers time is expensive  And there's something called channel so when you install something this condor condor looks for a channel and chon channel is a place somewhere  Somewhere on the internet was maybe even on your own machine. That doesn't it's also possible  That looks for packages and  The default is anaconda server, so we don't say anything typically looks there  But there's also something called condor forge, which is a channel  There's a lot of open-source packages by the pricing community  But you can also have private channels so you can have a channel with your own name  These are company name whatever and then you can install from this channel by and say say install dash see  My channel my channel is a name of the channel and then it will grab the package from this channel will be included  There's also a lot of configuration options which channel has precedence and which channels look first  You can configure this in a configuration file  Okay, a few basic tasks. I want to introduce you to what you can do with it  That's by no means a comprehensive introduction, but gives you some feeling what you can do so you can install packages  you can create and  Administer environments and you also can create packages yourself so  Writing is that a pie as you know and doing a lot of steps to create a package a  First thing you can search for something  so you say condor search and you type in the name of the  Package you're looking for I use pandas pandas is very much used package and scientific library and then corner goes out there and is searching  Places so I added condor added a few channels you see every dot you see they're fetching packages dot dot dot dot  And every dot means looking for a channel so I have by a bunch of channels  It's looking at and then gives you a long display  I just cut out of a few things when you see the three dots there  It's cut out and it lists you those then the package so everything this search looks everything has pandas  in its name and shows you the version that's available  Though all these versions are available is a normal one dot a dot here means it is available  And it's downloaded though it when you're down when you install something it puts it in a local repository and  You install from there. You don't have to download it every time and the one with the star here was a straight  That's the one I have installed right now here and you see if it had to dependency on umpire was a scientific focus tells you  have numpy one dot  10 and  35 and build zero and it comes from default switches for my naconda  But there's also a version that comes from conda force. It gives you the channel here that comes from  But it's pretty interesting can be very long list and but it shows you what you have  So it's easy a bit longer here and by the cut out quite a few it can be several pages long easily  Okay, the search can be it more fancy so  That's a problem. I have a pit when you search for something for church for jungle. It just gives you  I don't know how much very very very long listing this everything has a good jungle inside here  You can say for the full name and now it displays only the package  It's called pandas not all the packages have pandas in their name and now I get only pandas  It's still pretty long list, but it's much shorter ready  And I know it that's pandas and that that's a good feature and there's more so you can also  Limits a search or say I want to have only  Even though I did it on my Mac  I can search for all the available packages for windows in 32 and I also can specify the version  I want now shows me all the available  packages for  Versions 0.181 and it tells me okay here now it offers everything starting  This is numpy 110 Python 27 up to one pie numpy 111 Python 35  So there's no dots and last race because obviously if I want Mac at thousand  You need to download it here, so that's not there  But it would be available and would download it and they all come from defaults and there's more options if you want  There's quite a few switches you can use and can make this search even more specific pretty interesting feature  Installing the package is very easy  You just say condo install and you name the package and it goes through your channels and you can at the conflict finds it  The channel comes first attacks on this channel see now it provides me with a name  I want to install this package and typically tells I'm going to download here doesn't say because I've downloaded ready  Typicalist, okay  I want you to download these packages and then I install these ones like mkl's mask on a library all the dependencies  typically if I have a  properly packaged  Application in a conduct package and all the dependencies are there installed all the dependencies and everything's binary  So everything especially is scientific field. There's a lot of C and Fortron extensions and  And compiling these things typically works in Linux it might work on OS X and is right like it doesn't work on Windows  So that's why it's and still I think most people still Windows users  So the most users of Python I think on Windows as  According to all the statistics I've heard about and a lot of scientists and a lot of engineers  So I teach a lot of engineers and most of the engineering companies are mostly  Some of them have using Linux  But by far the biggest majority in Windows so need to support Windows users and it's a pretty good way everything's binary and you can  Do this okay?  Create an environment so this is the first thing installing something but very often we don't want to install in your what's called the root environment  You want to create environment? That's what is this command up here. You can see conduct create  Dash n or dash dash name so I create a new environment which I call my pi  There are three five and I can specify the Python version and I say I want to have Python 3.5  And you can also have if you wanted to could have a 3.4 or 2.7 or even 2.6 so far it supports 2.6 2.7  I'm not sure 3.3 but 3.4 3.5 and  We are pretty quick as we know 3.5 is out. I won't take along 3.6 3.6 3.6  We'll be supported typically a new numpy version is supported within 24 hours or something pretty fast  And I tell you what it's going to do. There's it never does something this out of asking  And it tells me okay. I need to download these packages and I need to install these packages or the dependencies  And then if you say yes, which I did then it starts creating this package and gives you this  The car cut off here, but it shows you how long it takes and how fast is doing stuff  Okay, now you created environment to in to actually  activate  Now I can actually list environments. I created environments and I don't have any one  I've just a small selection of the environments I have you say content and list for instance  It's one of the commands that shows you what environments you have and you see an environment is nothing  But it directory in your home folder in my my user folder under conda under conda amps  So if you install mini conda would be mini conda amps and here's another conda amps  It's just a different directory and these directories contain all the information about environment and you see now  The root environment is a  It's the one that's active right now. It has this  Okay, you see I gave the trailer the pie data Berlin here  Then I made a new post tutorial. I made a new environment and I don't mess up with my route  I can install what I like different versions and everything is isolated  Okay, activate an environment so depending if you're on a unix kind of system Linux in OS X  Then you have to say source activate name of my environment on windows is just activate so  typing source activate is pretty long. So I just had defined a short alias  But that's to be how we how we activate an environment and just type deactivated deactivated again  And then the prompt changes and then you see  prompt and now when you list it again, you see now the environment will be  Activated you always can see which one's active  Okay, and then you can say list and  Then it gives you a list of all the packages you have installed you see it tells me again the name of the package the version  Here this dependencies the numpy or not the  Channel and this one is red because this when you see it's installed this pip  So you want you will find maybe a few hundred packages there  But you can install this conda directly if not you can always install this pip and conda will recognize this one  So it's not a contradiction to work. I love this conda pip. We can use both. That's what I typically do  It's not available this conda. I use pip and install this pip, but conda raises these packages  And that's why I say it's installed this pip  Yeah, so you have the version is installed this pip and you can also install deals. There's no problem  So there's no contradiction in this and you have this pip installed there also  Okay  This is if you're kind of a consumer of packages you can install it can contain environments  There are way more options. I just show a few but now the next thing is building a package  So you want to build a package and there are essentially two ways you can build a package  you can just take a package from pipe I and  Converted into a conda package with a few commands or you can start from scratch  Bring it with your own  package  Okay, building from pipe. I is pretty easy. There's a skeleton first. You need to install  Conda build so conda builds if you have an account that's there already if you have mini con there need to install conda build  which is  Command line tool that helps you to build a package and then you say conda skeleton pipe. Hi pipe. I might package  So this one goes to pipe. I fetches  The package we have there and then you say conda build my package. That's pretty much it  So it's not very difficult and it's built a package and the package is a is a tar ball  It's a car PC to in this case. It's just one file with everything inside  And that's it and now we can install it and per default it creates it for the for the platform  You're on so if I do this as you can see here's now score as X  And  You can now install it and there are actually two different ways installing it right now  I can say conda install use local my package and you just will go to the place because I know that you thought place  We have put it and installed it but you can also if you wanted to if you had somewhere else  You could just specify the full path. We'll just file name. This is exactly the same here  But it could be a different way just if you download one of these you can always install it is conda install from the command line  You can also specify a Python version  So even if you have a different Python you can say okay build this one for Python 3 4 and it builds it for 3 4  Yeah, so even if it's pure Python it always has a Python version inside and this is especially useful if you build  extensions and see or Fortran which is very common in scientific field and  This one builds everything for this  You can also convert to other platforms you see I can say conda convert platform all for instance and just specify my  My file and say output directory and if it's a plain Python version  It will create a version for each platform for Windows Linux and Mac  If it's this extensions is not that easy then you need actually need the operating system because you need to compile it  So then I'm I don't think it's there's a cross-compile at least I didn't try and I don't know but  So so far I compiled on Windows and Mac and I think in Linux is the same  To be Linux is easy to compile it anyway  So if you compile extensions it works and I even compiled Fortran extensions  Which is typically more involved than C-SIMC and the rules nicely so far so you can create this extension source  and then if you like to you can upload and  You need the Anaconda client and now I'm uploaded to the Anaconda  And you can also make your own channel if you like so I don't have it here  But you can also create a channel and you just say an icon upload and just give it full pass of this  file and it uploads  This thing and a condor and other people can install it just by saying condor install  name of your package and  If you put it in different channels, they have to specify the channel when they install it  Okay, this is  building  From a PI PI package you can also do from scratch  So if you want you can customize things and you need a few things you need a meta yaml so a yaml file that  Declare to the flea specifies what you want to do and  Then you need a build file and there's two different ones if you're in Windows you have it  build dot BAT that file a batch file and then Linux and Mac you have a shell file and  You just use a normal setup dot py so you probably familiar with it  So you just use the normal setup dot py and then you have additional information in the meta yaml  Which also has dependencies in it  Let's have a look at this file  So the meta yaml looks like this is just the yaml file and has a lot of different  Sections in there. You don't need all of them  So you have something about met information about the package then you have the sources so you can say it comes from some  It's URL something like this you can have requirements  And there are different requirements your violence for building things and provides for running  For instance, if you use syson you might need syson to build it  But you might indeed install on the client machine because you just have this to be the extensions. You have different  dependencies for the tests what you do with the tests and then you have normal information about who the also is what the licenses  Okay, this is just a short  part of an  Event I am I used so I used just my package name here and you say okay the sources just  Depends where all two directories up there can be a relative path for instance. It doesn't have to be  URL  Requirements of Python and set up tools and then for running I need Jupiter and see here  This is something you can specify this one. I  Need it only for Windows and the same way you can specify Linux and Mac. So this will be only installed  For the Windows version but not for the for the version of other operating systems. That's as well  Very important and you can use the same syntax for and other places. So you make it dependent on the platform with one file  Of course, and you have this MIT license, whatever you want to put in there  Then you have this build files and they're typically if there's a skeleton  They just you set it up and you create files that look like this and they are just you see  You just call Python and set up to py install and they just call your setup to be right  The setup device just a normal one as you know from the normal installation process  But you can also add other commands in here, which is very interesting  So if something is not available as a content package in here  You can do a pip install something and this will install the package and this will go into this top ball  So everything you pip install the you will deliver the same package. I had this problem  I had the library that was not available as a content and difficult just  Instorted and she's good distributed alongside with your package. There's quite a few things you can do here in this build file  You can customize things or something doesn't work typically have a way to do it in build file some way or the other  Okay, and then just build it is a conduct with my package and that's it  And there are all the other steps are exactly the same as with the skeleton  So you can do upload and all kind of stuff install from from the local machine  These kind of operations  Good  That's it already so conclusions  I think conda is a very interesting package and it's very much used in scientific field  But can be really useful for all Python programs because it's a pretty good installer in a very nice package manager  You can manage environments and you can also build conduct packages  It works together with pip so it's not a contradiction  You can still use it if conda doesn't supply the package, but as you can see you can always convert a pip installer package into  Conda package if you like that might make the installation more smooth for it try for your users then you might actually  Just convert pip package to a conda  Very well known in scientific community if you go to a sci-pi your sci-pi conference pretty much everybody will know about it  I think  but it can be really useful for  For all Python programmers and I encourage you to give it give it a try  Thank you very much, and I'm here for questions  Because and ask them for me right first hand went up here  Thanks  I understand I can use conda for distribution and platform independent package installation, which is great now if I  Have I am a maintainer of a tool and I have lots of different dependencies  Outside pipi and inside pipi and with npm and whatever and it would be nice to be able to just  Use conda install whatever for me  Would I have to package all those npm pipi packages to conda manually or  Is it possible to I don't know say conda just install this and install it from pip because you don't have the package yet  Depends there's just some things you can do so if you install a packages that that is conda  Then this package will install its dependencies  So if you and say I say I depend on pandas pandas has a bunch of dependencies and install numpy for me and so on  To some degree you can do pip so far you have to as I think you have to write it in the  In the in this build script you have to say pip install pip install pip install  But probably the this meta yaml will be improved  That's what I just saw a ticket that you can then even say okay  This is supposed to become from conda. This is supposed to be pip and this might be a different channel  So it's still evolving and not everything works perfectly. There are some  Problems, there's no problem. There's a perfect but so far you can always have a walk around by putting it in the build script  I think  So what would be your suggestion for this kind of situation?  I'd put it in the build script. I had the same dependency thing and I put it put a pip install in the build script in this  build  SH and it worked for me so that was was the easiest one the other thing is just you can always convert the pip  Packages into contact packages and then you can list them in a matter yaml and just upload them as long as they're  BST MIT lesson shouldn't be a problem and anything just upload them to  Contact out and that's it. It is just you can automate it  You just need to list all the packages and then you can update it if it's in a version  Just check if the new version and just build new packages. Okay. Thanks. Thanks  Thanks for the talk so one of the selling point at the beginning that is that not limited  Not only to Python right but all the examples have seen all the pattern packages  So what do you mean by not limited to Python?  So you can use this everything is declarative so people use it for Ruby and other languages  So you can use the same thing and you can also have  Dependencies that are not necessary Python for instance one package is called pie tables as the interface to htf5  And this depends not only a numpy but also in the htf5 library  There has to be in a different version and all the things can be packaged in there  So it you can have depend ELLs char libraries  You can include them in your thing and that makes it much easier if you have a good if you have a package of the  at the end then the user doesn't have to do anything about fiddling around this  include parsing all kind of stuff suppose  Compiling htf5 is not the simplest thing. So I tried this once before condor  And you can easily take two days out of your life and you still haven't got anywhere  All right, thanks  Hello, thanks for your talk very to the point  Given that recently we have been adopting on PyPI binary wheels  They seem to solve most of the issues that condor seems to solve  Other than the fact that you have a single command to manage your install packages in virtual environments. That's good  It's nice, but what's the  Big advantage now that binary wheels are taking off  You write wheels is a great step forward as I say that for things not everything is in a real format yet  and you might not have the right version and the right platform in the wheel and  Condor can install wheels no problem. So you say people install and install the wheels. So that's not a contradiction this way  This at this as far as I know the story goes like this Travis Elephant. He's a  CEO of the continuum which is behind this it's open everything's open source  But there's a company behind it and he asked Peter van Rossum a while a few years ago about this packaging and because I just go  Ahead and do it. That's what it did. So it's kind of a little bit a parallel to some degree a little bit parallel to pip and wheels  something like this, but it's I think it's more comprehensive and  It's better and many has a lot of nicer features and pipping wheels for instance the search for name  I couldn't I don't know maybe I'm not smart enough  I couldn't get pipped into just showing me this package it shows me everything shows me hundreds of packages for Django  And this is not really useful to have to run it through grab and do your own thing. Why not just say I wanted  Many just customized search and those things aren't this the condo the condo is nice  But you as I said you can use everything that pip and wheels and pipe your eyes  You don't exclude it you can still use it and I do it very often because it does only if you have the packages  Or maybe now a thousand out there, but not eighty thousand or ninety thousand whatever it's on  All right any more interesting questions for Mike here we go. Yes  Best question yet  Okay, so is there like any  Standardization effort like are you working with Python packaging authority or something or is it going anywhere?  Or will it be forever separate like conda is great supposedly but only in the scientific community and like everyone else  This is bad  Great question, but I don't know the answer  Would be very nice. I would I would like to see it this way that both but actually everything's open source and both communities could use  the code from the other  Easily no no problem. I think everything is BSD or MIT license  I shouldn't be a big problem to use code and the I don't I'm not really sure why this is not a bit more coordinated efforts  And I don't know it would be very nice to have one stop and use  join forces and do the gathered for  For these things and I don't know why it's the case. There might be some reasons and I haven't really found out yet  More questions more questions keep the dream alive everyone I  I  Have a question so like a two-parter so presumably if I install anaconda  It's gonna come with its own versions of Python as well as my system pythons that I might have installed separately  And I once had this problem where or you know  I don't know if anybody else seen the problem where you upgrade your operating system and you upgrade your system Python and all your  Virtual ms are now broken because something something something which I don't really understand  And you have to delete all your old virtual ms and recreate them is anaconda gonna magically fix that problem for me or sub question  If I upgrade, you know, if there's a Python 3.5.1 and a Python 3.5.2  How does anaconda manage the upgrade process there? Does it need to rebuild a bunch of things?  There should be a problem so if you work with the environments those environments are pretty much isolated under totally and  Isolated sometimes if you use command line tools and they're installed like in the scripts directory is from there if not it goes outside  So if you say pile in something and you haven't put installed by linden your environment  It might take the wrong version and gives you a lot of syntax errors because it's a pile in for two seven and you and a three five  environment  But if you install everything it puts everything in the directory and there shouldn't be a problem and you update if you work with  environments and  You will see if you work for while you will focus environments anyway all the time you will rare  Actually, maybe the best approach just have the root just as a helper and then work in environments  So isolated of course you can always if you can always reproduce the environment you can freeze the environment  It just just specify what you have installed  You get a big list of installation and then you can give this this file to somebody else and the other person can say okay  Install from the file and then store exactly the same versions of the ones you can reproduce the environments and they should solve the problem  I  Don't know if that could be some you can you only find it out if you test it really and do it  But theoretically there shouldn't be a problem practically there might be one. I don't know  Today is that only me that's ever had that problem of changing the system Python and all the virtual ends are broken  So so I guess we're thinking that anacondas like kind of even more isolated from system pythons than  than virtual end managers if  It never has that problem  Yeah, there's some voodoo there further research required any more questions. Yes. Yes fantastic  All right. No, he wants to hear my voice any longer and absolutely necessary  Are the environment  Table or is there any plan for that?  Can you repeat a question that didn't get at the environment  Relocate a ball. So I  Installed somewhere. Let's say I caved the environment and  and  Compress it somewhere else like another  Computer. Yeah, so I just said it you don't actually copy the environment  You just copy all this average all the packages are installed as a version. So but I showed you this listings here  So one of this earlier things shows you this  Whatever is installed you have this  List somewhere do I have a list? Yeah, something like this. So you can actually  Freeze no, that's a search add a somewhere you can tell tell okay export all the versions  I have in a file and you just get a file with all the names equal  The version and then you can give this file to somebody else and somebody else can recreate exactly the same environment exactly the same versions  I'm not speaking about  Recreating the environment, but just taking the environment and put it somewhere else  You should work. It's just a directory  The problem is not everything is copied though if like I have this pandas version I have may I use it in 10 different  environments, but it's only once in my  in my  Repository and not directly in the files just a sim link and you have to make sure that that you copied it along with  Is it but I think there should be an option to copy everything just sip the whole directory input to some belts  It should work as long as the same operating system  But I no guarantee but theoretically should work can try it out  But maybe recreating is it's a safe of thing  So if you recreate you can also say okay, it freezes everything with except the same version  But you can also go in the file and edit is okay  It shouldn't be exact the same version should be at least the version or even update the version so you can go in the file and  Customize it and then still everybody who has this file would create and this nice thing is would work  I can freeze on my Mac and somebody else can create the same environment or windows  It shouldn't be a problem if I sip the file and would only be for the same operating system  All right any more for anymore it is now or there is one okay  You have to make it a question you talk very very fast and Mike will talk very very fast  Okay, so let's say that I have a package Python package on Linux that has  that needs separate  installed C library and  Let's say that this C library expects another C library very so what will come that it will  Download this library and all of its dependencies dependencies and put it in the condo repository locally or  What will that different ways I had the same problem typically what I did I just included in the file so I had this  You can you can specify the dependencies so if you compile something I had  Fortran dependencies some shared libraries somewhere and you included so  And I'm not read myself. I'm not sure how to the best way to do it, but that different ways including  Shared libraries and you can distribute the shared libraries along and they will put the shared libraries in this environment  They wouldn't clash with your shared libraries because you might have to share the maybe different versions  Then it works for while and until it doesn't work anymore because this is a different version something is different  So you can distribute shared libraries with it. There's different ways of doing it  It had you have you still have to look at there's a lot of recipes out there how to do it and probably you will find one that  serves your purpose  Okay, let's give Mike a mother big hand. Thank you very much for a great talking during the break
Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_0_647.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_648_839.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_840_947.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_948_1667.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_1668_1739.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_1740_1799.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_1800_4079.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_4080_4127.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_4128_4175.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_4176_4223.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_4224_4355.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_4356_5171.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_5172_5219.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_5220_8291.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_8292_12515.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_12516_12839.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_12840_13355.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_13356_14855.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_14856_15383.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_15384_16823.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_16824_16907.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_16908_17555.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_17556_17651.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_17652_17807.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_17808_18203.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_18204_18251.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_18252_19727.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_19728_20723.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_20724_21131.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_21132_22751.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_22752_23303.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_23304_24227.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_24228_25259.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_25260_25355.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_25356_26831.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_26832_41255.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_41256_43811.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_43812_44291.png Mike_Muller_-_Conda_-_Easier_Installs_and_Simpler_Builds_44292_48299.png
Okay, everyone give Mike Miller a big hand. He's gonna talk to us about condor Okay, thank you very much for the great introduction I Would like to talk about condor easy installs and simple builds just a real quick raise of hands Who of you knows what condor is? Third and who is actually using anac are using condor Just about 20% thank you very much condor is Quite a few things actually it's an installer similar to pip, but I think a bit better It's an environment manager similar to virtual and but still ever a bit better. It's cross-platform. There works on Windows Linux and Mac It's not limited to Python so you can use it for different languages. Actually. There are people using for Ruby and follow languages And actually it comes from the scientific community, but I think it can be useful for every Python programmer So it's PC licensed and typically you install it either install me condor or anaconda I would talk about this different condors here So there's what's called mini condor and it's a kind of a small bootstrap like version includes Python itself condor and a bunch of dependencies like pip and wheel and some other tools and It provides access to many hundred libraries. You can install a lot of libraries with condor maybe even thousands and You just need to say condor install either talk about this process a little bit and there's something called anaconda which is Different thing it's a distribution similar to like if you like a Linux distribution So you get Python packaged up with a bunch of scientific applications and And this includes of course Python condor condor install and is about more or less like scientific 200 scientific factors you could pop off here So supposed to be there but scientific package about 200 or so and if you install it it takes about two gigs of your hot drive But then I think a lot of scientists need a lot of these tools and you don't have to fiddle around installing the old separately They'll come in one thing and then then it barely pay a little bit price was some dis space but this space is cheap and Programmers time is expensive And there's something called channel so when you install something this condor condor looks for a channel and chon channel is a place somewhere Somewhere on the internet was maybe even on your own machine. That doesn't it's also possible That looks for packages and The default is anaconda server, so we don't say anything typically looks there But there's also something called condor forge, which is a channel There's a lot of open-source packages by the pricing community But you can also have private channels so you can have a channel with your own name These are company name whatever and then you can install from this channel by and say say install dash see My channel my channel is a name of the channel and then it will grab the package from this channel will be included There's also a lot of configuration options which channel has precedence and which channels look first You can configure this in a configuration file Okay, a few basic tasks. I want to introduce you to what you can do with it That's by no means a comprehensive introduction, but gives you some feeling what you can do so you can install packages you can create and Administer environments and you also can create packages yourself so Writing is that a pie as you know and doing a lot of steps to create a package a First thing you can search for something so you say condor search and you type in the name of the Package you're looking for I use pandas pandas is very much used package and scientific library and then corner goes out there and is searching Places so I added condor added a few channels you see every dot you see they're fetching packages dot dot dot dot And every dot means looking for a channel so I have by a bunch of channels It's looking at and then gives you a long display I just cut out of a few things when you see the three dots there It's cut out and it lists you those then the package so everything this search looks everything has pandas in its name and shows you the version that's available Though all these versions are available is a normal one dot a dot here means it is available And it's downloaded though it when you're down when you install something it puts it in a local repository and You install from there. You don't have to download it every time and the one with the star here was a straight That's the one I have installed right now here and you see if it had to dependency on umpire was a scientific focus tells you have numpy one dot 10 and 35 and build zero and it comes from default switches for my naconda But there's also a version that comes from conda force. It gives you the channel here that comes from But it's pretty interesting can be very long list and but it shows you what you have So it's easy a bit longer here and by the cut out quite a few it can be several pages long easily Okay, the search can be it more fancy so That's a problem. I have a pit when you search for something for church for jungle. It just gives you I don't know how much very very very long listing this everything has a good jungle inside here You can say for the full name and now it displays only the package It's called pandas not all the packages have pandas in their name and now I get only pandas It's still pretty long list, but it's much shorter ready And I know it that's pandas and that that's a good feature and there's more so you can also Limits a search or say I want to have only Even though I did it on my Mac I can search for all the available packages for windows in 32 and I also can specify the version I want now shows me all the available packages for Versions 0.181 and it tells me okay here now it offers everything starting This is numpy 110 Python 27 up to one pie numpy 111 Python 35 So there's no dots and last race because obviously if I want Mac at thousand You need to download it here, so that's not there But it would be available and would download it and they all come from defaults and there's more options if you want There's quite a few switches you can use and can make this search even more specific pretty interesting feature Installing the package is very easy You just say condo install and you name the package and it goes through your channels and you can at the conflict finds it The channel comes first attacks on this channel see now it provides me with a name I want to install this package and typically tells I'm going to download here doesn't say because I've downloaded ready Typicalist, okay I want you to download these packages and then I install these ones like mkl's mask on a library all the dependencies typically if I have a properly packaged Application in a conduct package and all the dependencies are there installed all the dependencies and everything's binary So everything especially is scientific field. There's a lot of C and Fortron extensions and And compiling these things typically works in Linux it might work on OS X and is right like it doesn't work on Windows So that's why it's and still I think most people still Windows users So the most users of Python I think on Windows as According to all the statistics I've heard about and a lot of scientists and a lot of engineers So I teach a lot of engineers and most of the engineering companies are mostly Some of them have using Linux But by far the biggest majority in Windows so need to support Windows users and it's a pretty good way everything's binary and you can Do this okay? Create an environment so this is the first thing installing something but very often we don't want to install in your what's called the root environment You want to create environment? That's what is this command up here. You can see conduct create Dash n or dash dash name so I create a new environment which I call my pi There are three five and I can specify the Python version and I say I want to have Python 3.5 And you can also have if you wanted to could have a 3.4 or 2.7 or even 2.6 so far it supports 2.6 2.7 I'm not sure 3.3 but 3.4 3.5 and We are pretty quick as we know 3.5 is out. I won't take along 3.6 3.6 3.6 We'll be supported typically a new numpy version is supported within 24 hours or something pretty fast And I tell you what it's going to do. There's it never does something this out of asking And it tells me okay. I need to download these packages and I need to install these packages or the dependencies And then if you say yes, which I did then it starts creating this package and gives you this The car cut off here, but it shows you how long it takes and how fast is doing stuff Okay, now you created environment to in to actually activate Now I can actually list environments. I created environments and I don't have any one I've just a small selection of the environments I have you say content and list for instance It's one of the commands that shows you what environments you have and you see an environment is nothing But it directory in your home folder in my my user folder under conda under conda amps So if you install mini conda would be mini conda amps and here's another conda amps It's just a different directory and these directories contain all the information about environment and you see now The root environment is a It's the one that's active right now. It has this Okay, you see I gave the trailer the pie data Berlin here Then I made a new post tutorial. I made a new environment and I don't mess up with my route I can install what I like different versions and everything is isolated Okay, activate an environment so depending if you're on a unix kind of system Linux in OS X Then you have to say source activate name of my environment on windows is just activate so typing source activate is pretty long. So I just had defined a short alias But that's to be how we how we activate an environment and just type deactivated deactivated again And then the prompt changes and then you see prompt and now when you list it again, you see now the environment will be Activated you always can see which one's active Okay, and then you can say list and Then it gives you a list of all the packages you have installed you see it tells me again the name of the package the version Here this dependencies the numpy or not the Channel and this one is red because this when you see it's installed this pip So you want you will find maybe a few hundred packages there But you can install this conda directly if not you can always install this pip and conda will recognize this one So it's not a contradiction to work. I love this conda pip. We can use both. That's what I typically do It's not available this conda. I use pip and install this pip, but conda raises these packages And that's why I say it's installed this pip Yeah, so you have the version is installed this pip and you can also install deals. There's no problem So there's no contradiction in this and you have this pip installed there also Okay This is if you're kind of a consumer of packages you can install it can contain environments There are way more options. I just show a few but now the next thing is building a package So you want to build a package and there are essentially two ways you can build a package you can just take a package from pipe I and Converted into a conda package with a few commands or you can start from scratch Bring it with your own package Okay, building from pipe. I is pretty easy. There's a skeleton first. You need to install Conda build so conda builds if you have an account that's there already if you have mini con there need to install conda build which is Command line tool that helps you to build a package and then you say conda skeleton pipe. Hi pipe. I might package So this one goes to pipe. I fetches The package we have there and then you say conda build my package. That's pretty much it So it's not very difficult and it's built a package and the package is a is a tar ball It's a car PC to in this case. It's just one file with everything inside And that's it and now we can install it and per default it creates it for the for the platform You're on so if I do this as you can see here's now score as X And You can now install it and there are actually two different ways installing it right now I can say conda install use local my package and you just will go to the place because I know that you thought place We have put it and installed it but you can also if you wanted to if you had somewhere else You could just specify the full path. We'll just file name. This is exactly the same here But it could be a different way just if you download one of these you can always install it is conda install from the command line You can also specify a Python version So even if you have a different Python you can say okay build this one for Python 3 4 and it builds it for 3 4 Yeah, so even if it's pure Python it always has a Python version inside and this is especially useful if you build extensions and see or Fortran which is very common in scientific field and This one builds everything for this You can also convert to other platforms you see I can say conda convert platform all for instance and just specify my My file and say output directory and if it's a plain Python version It will create a version for each platform for Windows Linux and Mac If it's this extensions is not that easy then you need actually need the operating system because you need to compile it So then I'm I don't think it's there's a cross-compile at least I didn't try and I don't know but So so far I compiled on Windows and Mac and I think in Linux is the same To be Linux is easy to compile it anyway So if you compile extensions it works and I even compiled Fortran extensions Which is typically more involved than C-SIMC and the rules nicely so far so you can create this extension source and then if you like to you can upload and You need the Anaconda client and now I'm uploaded to the Anaconda And you can also make your own channel if you like so I don't have it here But you can also create a channel and you just say an icon upload and just give it full pass of this file and it uploads This thing and a condor and other people can install it just by saying condor install name of your package and If you put it in different channels, they have to specify the channel when they install it Okay, this is building From a PI PI package you can also do from scratch So if you want you can customize things and you need a few things you need a meta yaml so a yaml file that Declare to the flea specifies what you want to do and Then you need a build file and there's two different ones if you're in Windows you have it build dot BAT that file a batch file and then Linux and Mac you have a shell file and You just use a normal setup dot py so you probably familiar with it So you just use the normal setup dot py and then you have additional information in the meta yaml Which also has dependencies in it Let's have a look at this file So the meta yaml looks like this is just the yaml file and has a lot of different Sections in there. You don't need all of them So you have something about met information about the package then you have the sources so you can say it comes from some It's URL something like this you can have requirements And there are different requirements your violence for building things and provides for running For instance, if you use syson you might need syson to build it But you might indeed install on the client machine because you just have this to be the extensions. You have different dependencies for the tests what you do with the tests and then you have normal information about who the also is what the licenses Okay, this is just a short part of an Event I am I used so I used just my package name here and you say okay the sources just Depends where all two directories up there can be a relative path for instance. It doesn't have to be URL Requirements of Python and set up tools and then for running I need Jupiter and see here This is something you can specify this one. I Need it only for Windows and the same way you can specify Linux and Mac. So this will be only installed For the Windows version but not for the for the version of other operating systems. That's as well Very important and you can use the same syntax for and other places. So you make it dependent on the platform with one file Of course, and you have this MIT license, whatever you want to put in there Then you have this build files and they're typically if there's a skeleton They just you set it up and you create files that look like this and they are just you see You just call Python and set up to py install and they just call your setup to be right The setup device just a normal one as you know from the normal installation process But you can also add other commands in here, which is very interesting So if something is not available as a content package in here You can do a pip install something and this will install the package and this will go into this top ball So everything you pip install the you will deliver the same package. I had this problem I had the library that was not available as a content and difficult just Instorted and she's good distributed alongside with your package. There's quite a few things you can do here in this build file You can customize things or something doesn't work typically have a way to do it in build file some way or the other Okay, and then just build it is a conduct with my package and that's it And there are all the other steps are exactly the same as with the skeleton So you can do upload and all kind of stuff install from from the local machine These kind of operations Good That's it already so conclusions I think conda is a very interesting package and it's very much used in scientific field But can be really useful for all Python programs because it's a pretty good installer in a very nice package manager You can manage environments and you can also build conduct packages It works together with pip so it's not a contradiction You can still use it if conda doesn't supply the package, but as you can see you can always convert a pip installer package into Conda package if you like that might make the installation more smooth for it try for your users then you might actually Just convert pip package to a conda Very well known in scientific community if you go to a sci-pi your sci-pi conference pretty much everybody will know about it I think but it can be really useful for For all Python programmers and I encourage you to give it give it a try Thank you very much, and I'm here for questions Because and ask them for me right first hand went up here Thanks I understand I can use conda for distribution and platform independent package installation, which is great now if I Have I am a maintainer of a tool and I have lots of different dependencies Outside pipi and inside pipi and with npm and whatever and it would be nice to be able to just Use conda install whatever for me Would I have to package all those npm pipi packages to conda manually or Is it possible to I don't know say conda just install this and install it from pip because you don't have the package yet Depends there's just some things you can do so if you install a packages that that is conda Then this package will install its dependencies So if you and say I say I depend on pandas pandas has a bunch of dependencies and install numpy for me and so on To some degree you can do pip so far you have to as I think you have to write it in the In the in this build script you have to say pip install pip install pip install But probably the this meta yaml will be improved That's what I just saw a ticket that you can then even say okay This is supposed to become from conda. This is supposed to be pip and this might be a different channel So it's still evolving and not everything works perfectly. There are some Problems, there's no problem. There's a perfect but so far you can always have a walk around by putting it in the build script I think So what would be your suggestion for this kind of situation? I'd put it in the build script. I had the same dependency thing and I put it put a pip install in the build script in this build SH and it worked for me so that was was the easiest one the other thing is just you can always convert the pip Packages into contact packages and then you can list them in a matter yaml and just upload them as long as they're BST MIT lesson shouldn't be a problem and anything just upload them to Contact out and that's it. It is just you can automate it You just need to list all the packages and then you can update it if it's in a version Just check if the new version and just build new packages. Okay. Thanks. Thanks Thanks for the talk so one of the selling point at the beginning that is that not limited Not only to Python right but all the examples have seen all the pattern packages So what do you mean by not limited to Python? So you can use this everything is declarative so people use it for Ruby and other languages So you can use the same thing and you can also have Dependencies that are not necessary Python for instance one package is called pie tables as the interface to htf5 And this depends not only a numpy but also in the htf5 library There has to be in a different version and all the things can be packaged in there So it you can have depend ELLs char libraries You can include them in your thing and that makes it much easier if you have a good if you have a package of the at the end then the user doesn't have to do anything about fiddling around this include parsing all kind of stuff suppose Compiling htf5 is not the simplest thing. So I tried this once before condor And you can easily take two days out of your life and you still haven't got anywhere All right, thanks Hello, thanks for your talk very to the point Given that recently we have been adopting on PyPI binary wheels They seem to solve most of the issues that condor seems to solve Other than the fact that you have a single command to manage your install packages in virtual environments. That's good It's nice, but what's the Big advantage now that binary wheels are taking off You write wheels is a great step forward as I say that for things not everything is in a real format yet and you might not have the right version and the right platform in the wheel and Condor can install wheels no problem. So you say people install and install the wheels. So that's not a contradiction this way This at this as far as I know the story goes like this Travis Elephant. He's a CEO of the continuum which is behind this it's open everything's open source But there's a company behind it and he asked Peter van Rossum a while a few years ago about this packaging and because I just go Ahead and do it. That's what it did. So it's kind of a little bit a parallel to some degree a little bit parallel to pip and wheels something like this, but it's I think it's more comprehensive and It's better and many has a lot of nicer features and pipping wheels for instance the search for name I couldn't I don't know maybe I'm not smart enough I couldn't get pipped into just showing me this package it shows me everything shows me hundreds of packages for Django And this is not really useful to have to run it through grab and do your own thing. Why not just say I wanted Many just customized search and those things aren't this the condo the condo is nice But you as I said you can use everything that pip and wheels and pipe your eyes You don't exclude it you can still use it and I do it very often because it does only if you have the packages Or maybe now a thousand out there, but not eighty thousand or ninety thousand whatever it's on All right any more interesting questions for Mike here we go. Yes Best question yet Okay, so is there like any Standardization effort like are you working with Python packaging authority or something or is it going anywhere? Or will it be forever separate like conda is great supposedly but only in the scientific community and like everyone else This is bad Great question, but I don't know the answer Would be very nice. I would I would like to see it this way that both but actually everything's open source and both communities could use the code from the other Easily no no problem. I think everything is BSD or MIT license I shouldn't be a big problem to use code and the I don't I'm not really sure why this is not a bit more coordinated efforts And I don't know it would be very nice to have one stop and use join forces and do the gathered for For these things and I don't know why it's the case. There might be some reasons and I haven't really found out yet More questions more questions keep the dream alive everyone I I Have a question so like a two-parter so presumably if I install anaconda It's gonna come with its own versions of Python as well as my system pythons that I might have installed separately And I once had this problem where or you know I don't know if anybody else seen the problem where you upgrade your operating system and you upgrade your system Python and all your Virtual ms are now broken because something something something which I don't really understand And you have to delete all your old virtual ms and recreate them is anaconda gonna magically fix that problem for me or sub question If I upgrade, you know, if there's a Python 3.5.1 and a Python 3.5.2 How does anaconda manage the upgrade process there? Does it need to rebuild a bunch of things? There should be a problem so if you work with the environments those environments are pretty much isolated under totally and Isolated sometimes if you use command line tools and they're installed like in the scripts directory is from there if not it goes outside So if you say pile in something and you haven't put installed by linden your environment It might take the wrong version and gives you a lot of syntax errors because it's a pile in for two seven and you and a three five environment But if you install everything it puts everything in the directory and there shouldn't be a problem and you update if you work with environments and You will see if you work for while you will focus environments anyway all the time you will rare Actually, maybe the best approach just have the root just as a helper and then work in environments So isolated of course you can always if you can always reproduce the environment you can freeze the environment It just just specify what you have installed You get a big list of installation and then you can give this this file to somebody else and the other person can say okay Install from the file and then store exactly the same versions of the ones you can reproduce the environments and they should solve the problem I Don't know if that could be some you can you only find it out if you test it really and do it But theoretically there shouldn't be a problem practically there might be one. I don't know Today is that only me that's ever had that problem of changing the system Python and all the virtual ends are broken So so I guess we're thinking that anacondas like kind of even more isolated from system pythons than than virtual end managers if It never has that problem Yeah, there's some voodoo there further research required any more questions. Yes. Yes fantastic All right. No, he wants to hear my voice any longer and absolutely necessary Are the environment Table or is there any plan for that? Can you repeat a question that didn't get at the environment Relocate a ball. So I Installed somewhere. Let's say I caved the environment and and Compress it somewhere else like another Computer. Yeah, so I just said it you don't actually copy the environment You just copy all this average all the packages are installed as a version. So but I showed you this listings here So one of this earlier things shows you this Whatever is installed you have this List somewhere do I have a list? Yeah, something like this. So you can actually Freeze no, that's a search add a somewhere you can tell tell okay export all the versions I have in a file and you just get a file with all the names equal The version and then you can give this file to somebody else and somebody else can recreate exactly the same environment exactly the same versions I'm not speaking about Recreating the environment, but just taking the environment and put it somewhere else You should work. It's just a directory The problem is not everything is copied though if like I have this pandas version I have may I use it in 10 different environments, but it's only once in my in my Repository and not directly in the files just a sim link and you have to make sure that that you copied it along with Is it but I think there should be an option to copy everything just sip the whole directory input to some belts It should work as long as the same operating system But I no guarantee but theoretically should work can try it out But maybe recreating is it's a safe of thing So if you recreate you can also say okay, it freezes everything with except the same version But you can also go in the file and edit is okay It shouldn't be exact the same version should be at least the version or even update the version so you can go in the file and Customize it and then still everybody who has this file would create and this nice thing is would work I can freeze on my Mac and somebody else can create the same environment or windows It shouldn't be a problem if I sip the file and would only be for the same operating system All right any more for anymore it is now or there is one okay You have to make it a question you talk very very fast and Mike will talk very very fast Okay, so let's say that I have a package Python package on Linux that has that needs separate installed C library and Let's say that this C library expects another C library very so what will come that it will Download this library and all of its dependencies dependencies and put it in the condo repository locally or What will that different ways I had the same problem typically what I did I just included in the file so I had this You can you can specify the dependencies so if you compile something I had Fortran dependencies some shared libraries somewhere and you included so And I'm not read myself. I'm not sure how to the best way to do it, but that different ways including Shared libraries and you can distribute the shared libraries along and they will put the shared libraries in this environment They wouldn't clash with your shared libraries because you might have to share the maybe different versions Then it works for while and until it doesn't work anymore because this is a different version something is different So you can distribute shared libraries with it. There's different ways of doing it It had you have you still have to look at there's a lot of recipes out there how to do it and probably you will find one that serves your purpose Okay, let's give Mike a mother big hand. Thank you very much for a great talking during the break
to talk about condor easy installs and simple but I think a bit better It's an environment bit better. It's cross-platform. There works on so you can use it for different languages. scientific community, but I think it can be condor and a bunch of dependencies like pip and to many hundred libraries. You can install a lot channel so I have by a bunch of channels It's force. It gives you the channel here that comes and is right like it doesn't work on Windows So still Windows users So the most users of Python I think on Windows as According to all the commands that shows you what environments you like different versions and everything is of a consumer of packages you can install it can building a package So you want to build a package Python version inside and this is especially useful if you build extensions and see or Fortran which is very common in scientific field and This the package then you have the sources so you can look like this and they are just you see You just I think conda is a very interesting package and be really useful for all Python programs because together with pip so it's not a contradiction You convert a pip installer package into Conda all Python programmers and I encourage you to platform independent package installation, which This is supposed to become from conda. This is people use it for Ruby and other languages So you your install packages in virtual environments. better and many has a lot of nicer features and everything is BSD or MIT license I shouldn't be a installation and then you can give this this file just get a file with all the names equal The lot of recipes out there how to do it and