Writing Awesome Command-Line Programs in Python
Smith, Mark
Mark Smith - Writing Awesome Command-Line Programs in Python Command-Line programs can have a lot to them - usually more than you think, yet often suffer from a lack of thought. This is a tour through how to structure your code, tools in the standard library and some 3rd party libraries. Take your command-line programs to the next level! - Python is a great language for writing command-line tools - which is why so much of Linux is secretly written in Python these days. Unfortunately, what starts as a simple script can quickly get out of hand as more features are added and more people start using it! The talk will consist of a tour through various useful libraries and practical code showing how each can be used, and include advice on how to best structure simple and complex command-line tools. Things to consider when writing command-line apps: * Single-file vs Multiple-file * Standard library only vs. 3rd party requirements * Installation - setup.py vs. native packaging The different parts of a command-line program: * Option Parsing: * Libraries: getopt, optparse, argparse, docopt * Sub-commands * Configuration: * Formats: Ini file, JSON, YAML * Where should it be stored (cross-platform); * Having multiple configuration files, and allowing user config to override global config * Output: * Colour - colorama * Formatting output for the user * Formatting output for other programs * How do you know when your output is being piped to another program? * Managing logging and verbosity * Managing streamed input * Exit values: What are the conventions? * Interactive apps - REPL * Structuring a bunch of programs/commands around a shared codebase. * Command-line frameworks: clint, compago & cliff * Testing command-line apps * Writing command-line tools in Python 3 vs Python 2
 So our next speaker is Mark Smith, who's a developer and trainer at Fandule in Edinburgh. And apart from that, Mark also runs Edinburgh Python user group. And his talk is writing awesome command line programs in Python. Hi. Thanks very much for coming to my talk. I'm going to spend the next 40 minutes or so talking to you about the things you can do to take your command line programs to the next level. And some of the modules that are in the standard library and are available on the net to help you do that. So just quickly to talk about my favourite subject, me. My name is Mark Smith. I work for Fandule. And we're hiring. We're looking for smart people. So if you're interested in that, come and talk to me afterwards. And I'm pretty much called Judy on most social networks for reasons that aren't as interesting as they sound. So I wouldn't normally put the word awesome in a talk title, but this talk is vaguely based on a book that I read about this time last year called Build Awesome Command Line Applications in Ruby. So I was stuck building a command line application in Ruby. And I'm not a Ruby developer. So I kind of needed to look around. And I came across this book. It really is excellent. So if you're stuck in the same situation, I highly recommend it. But what really became apparent to me while I was reading it was that a lot of the libraries that are out there in Ruby, they're similar libraries in Python. Python is just as good if not better for writing this kind of thing. So that's really what led to this talk. So let's just talk quickly about why you might want to write command line programs, because I think this is the only talk on the topic at this conference. So it seems like kind of a minor concern. And yet I'd be surprised if there are any developers at this conference who don't write command line programs regularly to simplify their job. I mean, we're developers. We like writing code. They can't all be web applications. So I would say that the reason to write a command line application is that, in many cases, a command line application is the best application for the job. We have a habit of thinking of them as inferior to web applications. And I would argue that's not necessarily the case. There are definitely some applications that benefit from Windows and buttons and menus. But as developers, we all know that we can automate processes. Programs are run by programs. And if we have something like Bash or even other scripting languages, other programming languages, we can glue applications together in really powerful ways. And besides, it's much easier to write command line applications than a full web application. So why would we want to spend any time writing a good command line application? Why not just chuck a main function in our script and be done with it? And that's because I would say that you want to think about this, do this properly, because scripts are never thrown away. They never get thrown away. You write them, you think you're going to run them once, and then you need to run them again in a week's time. Then you need to read the source code to work out whether it really did what you want. You might need to make some modifications. So I'd say if you start from the beginning with the idea that this is a reusable application, then when somebody comes up to you and says, hey, you've got that cool script that does something I need, then it's already documented. They can understand the output. They don't need to actually modify the file to run it. So let's just talk quickly about what makes up a command line application. So I suspect most people just kind of fell into using and then writing command line applications without really thinking about what makes one. More importantly, what makes a good, easy-to-use, reusable command line application. So this is your program. It's just a program. It's just a bundle of functions, maybe some classes. This is the thing that actually does the useful stuff. But without an interface, it's not any use. So let's start sticking a command line interface on top of it. So the first thing you probably think of are the command line arguments, the flags, the things that begin with minus or minus minus and data that you pass on the command line to run the application. So I've been talking to people about the topic of my talk during the week, and I've been getting this look as if to say, you're doing a talk on arg pass, a 40-minute talk on arg pass. But there's a lot more to it than that. We've got standard in, standard out, standard error. And it kind of looks like we finished, but we haven't finished. We've got signals. So can I have a raise of hands if anybody who's actually caught a signal in a command line application? Wow, that's way more than I expected. I'm not going to go into great detail on signals. I'm going to kind of skim over that because I've got far too many slides. And then we've got configuration. So configuration is not specific to command line applications. Obviously, we have that in GUI applications as well. But I think it's more important. So in GUI applications, the user tends not to see the configuration. It tends to be hidden behind a preference, his pain or something like that. Whereas with a command line application, if you're providing configuration, your user's probably going to have to write that. They're going to have to know where to find it. They're going to have to know how to modify it. And so the decisions you make are more important. And then finally, we've got the exit code, which is the number that's returned when the program's finished to say whether it did what it was supposed to or whether it encountered an error. So I'll start with command line passing. There's a bunch of options. There are three in the standard library. And you'll find that most people who aren't experienced Python developers pick getopt. Partly because actually, it's not alphabetically at the beginning anymore. It's the first one they see, and they've probably written some C, or they've encountered essentially the same interface in other languages. And so they just kind of go, oh, that's what I'm looking for. But getopt's horrible. It's really horrible to program. It's not really Pythonic. And it doesn't validate your arguments very well. And I'm not going to talk about opt-pass a lot because it's really been superseded by argpass. Argpass offers all the same functionality as far as I'm aware, plus some extra functionality. So I'm going to talk about argpass briefly. In the third-party domain, we have a few options as well. Dockopt is interesting because it takes the opposite approach to anything else I've ever used. Dockopt, you actually write your help text. So the kind of text you'd expect to be printed out when you ran your script with minus, minus help. And then Dockopt will pass that, work out what arguments your program should expect, and then it builds you an object that will pass those command line arguments. So it's kind of clever. Having used it, it's kind of fiddly to get the syntax right when you're writing the help. And I found it just actually took a lot longer, and it wasn't really easier in the end. But I quite like the idea. Plus, it's cross-platform, so it's been implemented in more than one language. So if you, for some reason, are writing a script that is implemented in multiple languages, that may well be a good option for you. Clint is much more than a command line parser. Clint is a project by Kenneth Wrights, and it's a whole framework for building command line applications. So I highly recommend it. It covers a whole cross-section of stuff that we're going to cover in this talk. I admit that the one thing I don't like about it particularly is the command line parser. But it's a good option. If you're already using Clint, then you're probably going to want to use the command line parser that comes with it. And as Clint, it's been creating a bit of a buzz recently. It was released by Armin Ronacker a couple of months ago. I haven't really used it. It looks pretty powerful. Armin's very good at these kinds of libraries. And I believe he solved a lot of localization encoding issues. So if you're going to be running under lots of different locales, then Clint is probably quite a good option for you. And then finally, we have Compago, which I don't think is a hugely popular library, but I like the approach that it took. And I think it's simpler for certain things than argparser. I'm going to quickly cover that with one slide after talking about argparser. So I'm sure we've all used argparser in this room. So it's not going to spend a lot of time on this. But you instantiate an argument parser. And then you tell it about all the arguments you expect. Some of them will be compulsory. Some will be optional. And then when you finish doing that, you actually call parse args. It will go talk to sys argv, extract the arguments, work out which flags correspond to which values, and then hand you back a namespace object, which contains the extracted values. If it fails, if somebody is parsing in valid value, it will essentially, it will print out the help text and then finish with an error code of one. So here we've got an optional variable called, optional parameter called name. That's what the n args question mark means. And so if it's not provided, I'm just replacing it with word world. This is just a silly hello world script. And then it prints hello and the value of name. And that's what happens when you run it. So because it's optional, we can run it without. It uses world. We can run it with Douglas. And it will print hello, Douglas. Sorry, skipped ahead. So we get the same output as we would if we had just extracted that single variable from argv. So you might say, why did we bother? It turns out that that was only one line more than the code that just extracted it from the first item of argv. But you get this when you run minus, minus help. So although we didn't specify minus, minus help, and we didn't actually provide any help text anywhere in there, we didn't say what name was. But instantly we can give the user a picture of how they run the application. I think that's really powerful. I now no longer write any scripts that don't use arg pass or something similar. Especially because when somebody comes to me and says, that script you gave me, how exactly do I run it? Have you got documentation somewhere? You can say, run it with minus, minus help. It will tell you what you should be doing. And then come back if that's not enough. So here's a more complex example, just covering some more of the details of arg pass at random. So here with the first one, we've decided there should be a Boolean value. There's no value associated with verbose, really. It's just on or off. And so we say store true, and now it will store it. It's true if it's available, false if it's not available. And then the second argument, number. We've got this type parameter. We're passing an int. So int is just the standard Python int type. But you'll remember, hopefully, that int is also a function that takes a string and returns an integer value. And that's actually all you can pass in any function like that for ad argument under the type parameter. So anything that any function that takes a string and returns the actual type you want, you can pass in. So then when your namespace will actually contain the types that you're after rather than you having to constantly convert things, it's nicely reusable. Yes, sorry. So arg pass also supports subcommands. So if you're writing a much more complicated application without the traditional Unix single use specification, then if you want to write something like git, where you've got subcommands like ad, push, pull, fetch, remove, sit, stay, you can do this. It tends to be very verbose. It works very well, and it's kind of worth doing. You have to be quite careful about how you factor out all the parsing for the different commands so that you don't just end up with one huge block of code. In some ways it's configuration more than code. So this is where CompoCo is actually quite good. CompoCo takes a much more lightweight approach. And in some ways it's not so powerful, but if you're writing this kind of application, you just want to kind of get it done quickly. CompoCo can be a really good option. So I have no idea whether it's supposed to be pronounced CompoCo, by the way. I don't really know what that means. So here you instantiate an application. That's essentially your parser, and that provides you with a decorator. And then you just use that decorator for any function you want to be an entry point into your application. So in this case, we've given our application a greet subcommand, which takes an optional parameter of two. So it just parses this out of the function. And the same with un greet, just so I added another subcommand. Now we can call this program with un greet, and again, we can choose to call it with a parameter or not. If you don't provide a default value for the parameter, then it becomes a compulsory parameter, and you have to provide it if you're running the un greet command. And then when you run it, it looks like this. So it's not quite so nice. It doesn't have positional parameters as far as I'm aware. But it's really, really lightweight. It's really easy to write. It doesn't take up a lot of space in your code. So if you're already using third party libraries, then Comprehco's probably a good option. So that was command line parsing, quick well-binned tour. Let's talk about input and output. So IO is often an afterthought. It's just kind of you think, oh yeah, I'll dump some print statements into my script. But good IO can really transform a command line application from something that is just a hacky script. And it doesn't take a huge amount of work. It just takes a little bit of thought. And possibly some libraries that are out there that can really add some power. So just before we move on, it's worth talking about standard out and standard error, because I suspect most people don't really think about them. Standard out is for the output of your program. So if you think of your program as being a pipe, and it takes some input and it spits out some output, that's kind of the data that you're producing. You might, it's perfectly valid to have a script that doesn't produce output. It's perfectly valid to have a script that doesn't take input. But if your program produces data in some form, standard out is where it should go. Standard error is probably the worst name it could be given as a stream. Standard error is not an error stream. You can use it for errors, but standard error is really for updates on how the program is running. It's kind of information on what's going on. You can assume that the user is probably going to see standard error, whereas if they're piping the output of the program to another program or a file, they're not going to see that. So they get no feedback on how the program is running. So the way I think about it is that standard out can be piped, piped to a file or another program, whereas standard error can be logged. So at the least, it goes to the screen, or you can save it to a log file. That's really what it's for. So let's talk about logging. Logging is written the right way. It logs to standard error by default because it's ultimately an informational thing. Logging is a really powerful library. I wish I had time to kind of go through it. But this is an idiom that I use in command line programs quite a bit. And again, it's not a huge amount of code, and it makes a big difference in the rest of your program in terms of making it easy to provide information to your user. So I call basic config to just set up logging. I set it to warning by default, so anything that's info or debug won't be spat out to the user. The reason I do this is because programs should be relatively quiet by default, and then you can provide a flag to allow the program to be more verbose. So that's what we've done slightly further down. We've just checked. If they want more information, I've set it to levels down to debug. You can set it down to info. It all depends on how you like to use logging levels, it's up to you. And then later, the last four lines are just how you then use that through your program. But the nice thing here is that logging just takes care of how much information is spat out to the user after that. It knows the level that should be spat out. So it doesn't invade your actual code. You can use logging as you would in any other kind of program, and it will just do the right thing after running this block. So another thing that's worth thinking about that we often don't think about is, are you talking to a user? So I think we often assume when we're writing the program that actually standard out is just going to go to a console, and input is probably going to be fed to us from a file, I don't know. So it's just slightly, slightly weird that we have that sort of different approach to the two streams. And it's really easy to do in Python. The main file types, standard in, standard out, standard error, all support is a TTY as a module, and that just tells you that that stream is attached to a screen somewhere. So the thing you want to think about as a result of this is, so I was just taking a picture. So this is the result for running that program. So if you run it on its own, it just says none of its pipes to anywhere, it's just coming to the screen. If you pipe it to another program, in this case cat, then it says the output is piped to cat, but obviously not standard error. And finally, we're just piping input and output. So it's just testing that all works. But the thing you really want to think about with this is you can change the format of your output based on whether it's going to file or another program. So if your information is being printed to the screen, your main concern is that the user can read it. So you can think about formatting stuff in aligned columns, you can think of color, and you can think about the type of information they want to see, how they want to read it. If you're piping to a file or another program, you want something that's easily passable. So you could switch your output to JSON, CSV, or something that's easy for another program to pass. It becomes data rather than kind of information. So I just mentioned color. Color is also surprisingly easy. And weirdly, the Ruby guys do this all the time. We don't tend to see so much color output from Python programs. You can just use standard ANSI escape characters. But I don't advise that. I recommend using a third-party library called Colorama. So Colorama provides some constants for changing the stream. And all it really does is spit out a character to standard out. And so here I'm printing out red, and I print out some text. The standard out will continue to be printing red text until I send the reset token. So the next line actually adds a background color of green. So you then get red text with green background. And that's kind of what this looks like when you run it. And you can see how this would make a huge difference to a program. If you've got errors that are coming out in red, and you've got warnings coming out in yellow, and you've got normal text coming out in white, it just means that as your reams of information stream past, you can kind of see that something happened that's actually interesting rather than just information. So it makes a big difference with long-running programs, and it makes a big difference with programs that spit out a lot of information. The problem with this is that with that script, if I pipe the output to a file, it adds the it keeps the tokens in. And you don't really want that. You want to store plain text when you're saving it in a text file. And the way that you do this, and this isn't something that's really promoted in documentation, is you call Colorama.init and you pass in strip equals true. And that just means, in this case, we're just saying if it's going to a file or another program, strip out all the characters. Colorama, one of the neat features is it wraps standard out and standard error when you initialize it. And it means that if you print out those ANSI color codes that are only supported on Unix, not on Windows, if you're on Windows, if the program is being run on Windows, it will automatically convert them to the Windows equivalent. So just with kind of a single line of code, you've essentially made the colorization at least of your application cross platform, which can be kind of nice. User credentials is something that's supported in the standard Python library, batteries included. And I'm just going to quickly cover it. It's got two functions, get pass. Get pass prints out password to the screen, and then it allows the user to type in a password when they hit return. It returns that as a string. And it doesn't actually print the password to the screen. Get user is quite different. Get user is not an interactive function. So it's slightly confusing. Get user will actually talk to the operating system and try and obtain the username of the user that's currently logged in. So in this case, I was logged in as Mark, and so it's just picked that up. And again, these can be kind of useful if your script is communicating with a database. You don't want to have your password stored in the script that's fundamentally insecure. You could end up committing that to version control or something. So if you want it to be a professional application, you should be asking the user what the password is. Or perhaps you can store it in a configuration file. Progress is another one. So if you've got a long-running task, I can't count the number of times I've run a program that doesn't spit out very much information. And you kind of think, has it died? Should I cancel it? Should I start again? Is it a network problem? Is it just a temporary thing? Will it carry on working? So I mean, progress is a really nice, easy way of just saying, you know, we're in the middle of something, and it's still going. You can see progress. You can get an idea for how long it's going to take to complete as well. So there's a library called Progress Bar, which has been around for a while and I think is really, really needs some love. And it's been forked. So it's received some love, but just in a different place. This is Progress Bar 2. If you pip install Progress Bar 2, you get a module called Progress Bar, confusingly. And then you can instantiate this. And you can use it in a couple of ways. So in this way, I'm wrapping an iterator or an iterable and just each time through, it knows that it's an extra 80th through the process. And it will just print out this Progress Bar on the line with a percentage in front of it. You can do much more complicated things if you want to. So here, it has this idea of a list of widgets that make up your Progress Bar line. This is a little bit messy, but it demonstrates a few things. So you can have strings and you can have what they call widgets, which are kind of updated each time. So in this case, it's going to print out the string loading at the start. You can see that in the comment at the bottom. And then it's going to print out the percentage that it is through. And then it's got space. It's going to print out Progress Bar. And then the last two are kind of interesting. So ETA just keeps running a calculation of when the process is going to finish. So that would be something that would be a pain to develop yourself. File transfer speed assumes that every time you're updating, you're actually updating with the number of bytes that you've read. And so it keeps a running tally of the number of bytes per second, which, again, is kind of a professional level of update. So if you're writing a program that's streaming stuff from one place to another, that's quite nice information for the user. So with IO, you want to think about adding a flag to specify output. So I said you can automatically decide, and you probably should, but always let the user override that because they might not want that JSON written to a file. They might want the output that they're used to on the screen for some reason. And the same with verbosity and quietness. Add a flag to allow them to choose how much information they see. And also be responsive. So tell the user that things are still going on. Don't just leave them with a blinking cursor on a black screen. That would be bad. So let's talk about configuration. We have a bunch of options for configuration. And we have way more just out of the box in the standard library than I thought until I went looking. I kind of thought of any files in JSON. I suppose I thought about CSV. But yeah, there's a whole bunch of stuff that we got out of the box. So you can use environment variables. We can use JSON out of the box. You can use CSV. We've got some rudimentary XML parsers in Python. Not fantastically performant but fine for config. But I don't think anybody wants to write XML for configuration. We've got Apple P list built in on all platforms, which also I don't recommend. Or obviously we could write our own parser using regular expressions or something horrible to end again. Just don't do that. The third part we've got YAML. I've been working with Ansible recently. YAML turns out to be quite a nice format to write. It's got all the same data types as JSON, but it's actually kind of nice to write. And then we've got Java properties, which you probably only want to consider if you're interacting with Java applications. But I'm going to talk about any files. And they're kind of not exciting. And they have a lot of us are Unix developers and we're probably thinking, isn't that a Windows thing? But I think they're hugely powerful and I think people underestimate the power of them. Possibly just don't use them in the right way. So I wanted to cover it. So the config parser module has a few different config parser objects in there. I recommend you use safe config parser because that way your application does some security holes in some of the others. They're there for historic reasons more than anything else, as far as I can tell. And the way that you use it is you instantiate your config parser at which point you will have an empty configuration object. And then you call read. And read doesn't take a single file name. It takes a list of file names, which is counterintuitive until you understand really what it's designed for. And what it does is it goes through that list of paths. And if the file that that path points to is there, it will read it in. And it will merge whatever data is in there with whatever is in your config object. And then it will go on to the next file. And if that's there, then it will read it in and merge it in. And if it's not there, it will just skip it. There are no errors if it doesn't find any of these files. It expects to begin a list of effectively optional files, files with optional existence. So the way that you use this, I was talking about these paths. So the first one is a default file that one assumes is provided with the script. It's expected to be in the same directory. It could be in slash, et cetera, or something like that. But it's essentially a default configuration. And then on top of that, we've got the per user configuration. So we're picking tool.ini from the home directory. And then on top of that, we've got config.ini, which is in our current working directory. We haven't specified an absolute path for that. So that's just kind of where we are. So that's your project configuration. And these might look like this. So starting at the bottom. So the first one that's loaded is defaults.ini. And that assumes normally when you run this script, whatever it does, it's going to be talking to local host on port 8080. And then I'll talk about URL in a second. And then what it does is it loads tool.ini. If it's not there, it obviously doesn't load it. It doesn't complain. But if it is there, it loads it. And it overrides port. So this is like inheritance with objects. So now we have localhost and port 5000. And then we load in the project configuration. For this project, we're pushing everything to www.ninjarockstar.guru. Love those new domain names. And so now we've got that host. We've got port 5000. But we still have this URL value that's generated with placeholders. So it's built up from host and port. So if we print these out, we've got the host that I said, the port that I said. So you'll notice that also we can call get in. So it does some basic casting for us, some parsing if we want to. It's not a huge benefit, but it's kind of nice. And then we've got URL, which is our generated value. And you'll notice that it's generated on demand. So it's got the values that we inserted in later files. So Ninja Rockstar wasn't around in the file that URL was defined in, but it still pulls it in anyway. And that may not seem like a huge benefit. I mean, it's a bit ugly as well. But what it means is that if the user wants to use an HTTPS URL, they don't need to change any code. They don't need to go into that defaults file and change it. They can set URL in their personal config file or their project config file. And then it will use that. And again, they can use these patterns. They can use host and port in there. So it doesn't need to start from scratch again. So it's kind of powerful. That's config. So I'm going to kind of skim through signals a bit. Talking really fast, actually. I hope everybody can follow me. Cool. Signals. So it's not nice code, but you can set up a callback, essentially. So when a signal is received from the operating system, you can get it to call your function. What that first line is doing is it's overriding the default behavior. So when your function returns, this is SIGINFO, which is a non-standard flag. I can't actually remember how you pass it to your program. But it can be kind of useful if your program is not spitting out any output and the user still wants to get some progress. DD does this. You can pass in a SIGINFO using a control letter sequence. And then it will just print out its progress report, which is kind of nice. But the truth is, probably most of us aren't going to do that very often. We're usually interested in SIGINT, which is keyboard interrupt. So when somebody hits Control C to finish your program, and that is actually caught by Python anyway. So that isn't ignored, and it doesn't, by default, just quit your program. It raises a keyboard interrupt. And this is why in all my command line programs, I wrap everything with the keyboard interrupt. Because the default behavior is to print out a stack trace and exit with an error, which isn't really what happens. So you generally want to handle that and not really do anything. I'm not really sure whether the correct behavior would be to make sure your exit code is zero when this happens, because it's not an error. So I'm just going to run very quickly through code structure and packaging, because I didn't think I had enough time to do more. So all projects are different. And a lot of the stuff I work on are actually single file scripts. And then there's not a lot to it. You could add a setup.py to make it installable. But you don't need to do much. And often, to be honest, they just end up sent by email, which is not great, because any of these versioning can use the ability for people to update, things like that. But there's nothing magic about command line scripts. I recommend you use setup tools rather than distu-tils, because that handles dependencies, installing any dependencies you might have. And it also includes some extra features specifically for command line, writing command line programs. So here we've got the setup.py that we've written. I've got an example on the next slide. And we've got our actual script, which is my tool. And that should be tiny. That should be really, really small. So we should be writing all our code to be reusable. We may be writing a great, awesome command line application. But one day, we might want to take that functionality and stick a web interface on it. We might want to take it and stick a GUI on top of it. When we might want all three at the same time, depending on how the user starts the application. So really, you want all your functionality in the library as much as you can. So my tool should really just call out to something inside my tool lib. And you can put that in a couple of places. I added the double underscore main file this morning. So I know that a few people haven't really come across it. I think it was added with Python 2.7, obviously Python 3. You never used to be able to run packages. You could run modules. So if your library was distributed as a single file, you could run that. But it had no way to run modules. Now if you run my tool lib with Python minus M, my tool lib, then it will look for that double underscore main file and execute that. So it's not, I suppose you could import it if you really wanted to, but it's not designed to be ever imported. It's only designed to be run. It's the entry point into your program. So I think I'd be tempted to put my command line interface in that file, or at least some of it. And then we've just got some other files that actually have functionality in there. It's that gray circle is my tool.py and utils.py. That's a reasonable way of structuring a relatively small script. And then the set op.py. If anybody hasn't written one of these before, they look a little bit overwhelming. They're a huge amount of just data. And it's a function call as well. It's not a data structure. And so they're just a bit scary. But it turns out when you write them, they're not that scary. I recommend kind of, it's a bit of a cargo cult, but I recommend finding some of their command line scripts that you've installed with pip, where you respect the developer in some way, where the script has been well developed. I recommend you're going to look at theirs. There's a whole host of options, depending on how you want to install these things. So in this case, the important lines are really the two up, one up from the bottom. So you tell it what package you want to install, and you tell it there's a script called myTool. And you could install a series of scripts. You could have different entry points into your application. And they'd just be one-liners calling out to functions in myToollib. And there's other options with setup tools as well, because this is technically the setup tools setup function. There's such a thing called an entry point, where you don't write your script. You write a function as your entry point. You tell setup tools what functions you want to be, entry points you tell them what you want those scripts to be called. And when somebody pip installs, it will install those scripts for you. And the nice thing about that is on Windows, it will create an executable. So it'll create a binary. So it looks much more like a native application to the user. And they won't have.py on the end. So they're more native on Unix as well. Exit codes. There's not a huge amount to exit codes. By default, Python does some sensible stuff for you. If your program just exits, because it got to the end of the main function or the end of the script that it was running, it exits with zero. And if you've got an uncaught exception, it will exit with one. But it's not a great amount of information. So if you exited because the user provided bad data, you could provide a different exit code. If you had network problems, you could provide a different exit code. There's any number of options. You can just, it's a very small token of information that's easy for somebody else to then understand. They don't have to pass any information. They're just handed back a number. You should always consider that your program is being run by another program. Because that's going to make it more reusable and make it more useful. And there's no specific standard for what these numbers should be. GNU, the GNU project recommendations, they recommend that you set the high bit if it's a serious error. But I'm not really sure of any projects that are actually doing that. And on BSD, they have a sys exit header file that lists a whole load of what they consider to be standard exit codes. So that's probably a good place to look. I mean, it doesn't hurt if there's a list of exit codes for things and you are exiting with that condition. It doesn't hurt to reuse somebody else's standard. So the stuff I haven't had a chance to cover really are the command line frameworks. So people have been thinking about this. A lot of the stuff here, you can see it's kind of boilerplate code. And it would be nice for somebody to handle this stuff for you, like where is configuration stored on Windows versus Mac versus Linux. And Cliff and Clint both provide some subset of that. I haven't really worked with Cliff. Cliff is designed really for multi entry point projects and it kind of integrates with set up tools. And Clint is really a whole bag of stuff. It does colorization and progress and command line passing and it will load configuration from sensible options. It's a really cool library. It's not fantastically documented. Something I'm planning to fork. Well, I have forked it. I'm planning to add some documentation to that at some point and see whether it's accepted. And I haven't covered cross-platform considerations. So I just kind of skimmed over that when I was talking about configuration file locations. And it's just for lack of time. There's a huge amount to this topic and I'm going to write up some blog posts on it. Hopefully that will be useful to some people. So I do have five minutes left. So I wasn't going to do questions. But if anybody has any questions. Hey, what are the tricks to get these command line programs to run fast, especially if it's a short-lived command line program? I would say minimize your dependencies. So with a short-lived program, I guess it's not really doing very much. So you're looking at interpreter startup time and kind of the time it takes to load in libraries. So, yeah, I would say stick to the smaller. None of these are big libraries. None of them take a long time to load. But, yeah, I guess if you're loading a library that has a large data file behind it or something like that, that's going to take some time. But, I mean, the nice thing about Python is it has a really fast startup time anyway. So it's one of the things that makes Python such a good language for writing command line tools because they often are short-lived. There's a syntax error on that file, but ignoring that. So if you change the behavior when you see a terminal versus when you see a pipe, how do you go about testing that? Because obviously sometimes if you, programs that change their behavior in a test environment can be a bit annoying at times. Yeah, yeah, I can see that. I guess testing that I would mock out the standard error object with one that lied about whether it's a TTY or not. And then you could use a, you could sort the results in a string IO object and just then just check it against what you're expecting. Do you have any recommendation for interactive programs on the command line? I got this question last time and I totally forgot to write the slide up on it. So I have used end curses briefly. It can be a little bit awkward to get running on non-linux platforms. But no, I haven't done a huge amount of it, so I can't really recommend anything specifically. There's the built-in module CMD, which enables you to create stuff like that, but it gets kind of awkward once it gets very big. Yes. But yeah, in that case, you don't know any alternative. CMD is a very cool module for anybody who hasn't used it. So it's for creating your own shell, essentially your own dedicated shell. So you allow it, when you run your program, you then get the blinking cursor and then you can then type in commands. So it works quite well with programs with subcommands, for example, so you don't have to type in the git part of git at each time. You're just within your own contained environment. I've used it quite well with Agpass before. So it'll call you a function that you've declared as being one of your commands. And then anything else that's on the command line just kind of gets passed in, and it works quite well with Agpass. But you're right, they're kind of difficult to manage. And it's just a question of trying to make sure that everything's extracted out into their own functions, and it doesn't just become one big mess. It kind of provides an event loop for a command line program. So very cool. So awesome, that means basically that it has to work, right? So about testing, are there any specific tools that you would recommend, like, you know, Cram, for example, comes to mind? Are there other tools that you would suggest for testing the command line parts of the application? I tend to use unit test just because we're dealing with text, or at least data, and we're not dealing with kind of events and things like that. I find that actually unit test tends to have most of the functionality you want. I mean, obviously there's some add-ons to unit test as well. I haven't used Cram. Sorry, I think we're back. But if you're unit testing, sorry, you're going to mock the send in, send out, so we can get callers as well, and stuff. So it can get callers, sorry. Yeah, but if you're really unit testing that, and you want to see the callers and stuff, that's going to be pain, right? Yeah, it's going to be a pain. I mean, mocking is a way to pain. Sorry. Yeah, sorry. If you're going to test the actual app on the command line, PXSpect works well. Oh, okay, thank you. I think that's going to have to be it for questions. Yeah, but I think we're out of time, I'm afraid, so thank you very much.
 So our next speaker is Mark Smith, who's a developer and trainer at Fandule in Edinburgh.  And apart from that, Mark also runs Edinburgh Python user group.  And his talk is writing awesome command line programs in Python.  Hi.  Thanks very much for coming to my talk.  I'm going to spend the next 40 minutes or so talking to you about the things you can  do to take your command line programs to the next level.  And some of the modules that are in the standard library and are available on the net to help  you do that.  So just quickly to talk about my favourite subject, me.  My name is Mark Smith.  I work for Fandule.  And we're hiring.  We're looking for smart people.  So if you're interested in that, come and talk to me afterwards.  And I'm pretty much called Judy on most social networks for reasons that aren't as interesting  as they sound.  So I wouldn't normally put the word awesome in a talk title, but this talk is vaguely  based on a book that I read about this time last year called Build Awesome Command Line  Applications in Ruby.  So I was stuck building a command line application in Ruby.  And I'm not a Ruby developer.  So I kind of needed to look around.  And I came across this book.  It really is excellent.  So if you're stuck in the same situation, I highly recommend it.  But what really became apparent to me while I was reading it was that a lot of the libraries  that are out there in Ruby, they're similar libraries in Python.  Python is just as good if not better for writing this kind of thing.  So that's really what led to this talk.  So let's just talk quickly about why you might want to write command line programs, because  I think this is the only talk on the topic at this conference.  So it seems like kind of a minor concern.  And yet I'd be surprised if there are any developers at this conference who don't write  command line programs regularly to simplify their job.  I mean, we're developers.  We like writing code.  They can't all be web applications.  So I would say that the reason to write a command line application is that, in many  cases, a command line application is the best application for the job.  We have a habit of thinking of them as inferior to web applications.  And I would argue that's not necessarily the case.  There are definitely some applications that benefit from Windows and buttons and menus.  But as developers, we all know that we can automate processes.  Programs are run by programs.  And if we have something like Bash or even other scripting languages, other programming  languages, we can glue applications together in really powerful ways.  And besides, it's much easier to write command line applications than a full web application.  So why would we want to spend any time writing a good command line application?  Why not just chuck a main function in our script and be done with it?  And that's because I would say that you want to think about this, do this properly, because  scripts are never thrown away.  They never get thrown away.  You write them, you think you're going to run them once, and then you need to run them  again in a week's time.  Then you need to read the source code to work out whether it really did what you want.  You might need to make some modifications.  So I'd say if you start from the beginning with the idea that this is a reusable application,  then when somebody comes up to you and says, hey, you've got that cool script that does  something I need, then it's already documented.  They can understand the output.  They don't need to actually modify the file to run it.  So let's just talk quickly about what makes up a command line application.  So I suspect most people just kind of fell into using and then writing command line applications  without really thinking about what makes one.  More importantly, what makes a good, easy-to-use, reusable command line application.  So this is your program.  It's just a program.  It's just a bundle of functions, maybe some classes.  This is the thing that actually does the useful stuff.  But without an interface, it's not any use.  So let's start sticking a command line interface on top of it.  So the first thing you probably think of are the command line arguments, the flags, the  things that begin with minus or minus minus and data that you pass on the command line  to run the application.  So I've been talking to people about the topic of my talk during the week, and I've been  getting this look as if to say, you're doing a talk on arg pass, a 40-minute talk on arg  pass.  But there's a lot more to it than that.  We've got standard in, standard out, standard error.  And it kind of looks like we finished, but we haven't finished.  We've got signals.  So can I have a raise of hands if anybody who's actually caught a signal in a command  line application?  Wow, that's way more than I expected.  I'm not going to go into great detail on signals.  I'm going to kind of skim over that because I've got far too many slides.  And then we've got configuration.  So configuration is not specific to command line applications.  Obviously, we have that in GUI applications as well.  But I think it's more important.  So in GUI applications, the user tends not to see the configuration.  It tends to be hidden behind a preference, his pain or something like that.  Whereas with a command line application, if you're providing configuration, your user's  probably going to have to write that.  They're going to have to know where to find it.  They're going to have to know how to modify it.  And so the decisions you make are more important.  And then finally, we've got the exit code, which is the number that's returned when the  program's finished to say whether it did what it was supposed to or whether it encountered  an error.  So I'll start with command line passing.  There's a bunch of options.  There are three in the standard library.  And you'll find that most people who aren't experienced Python developers pick getopt.  Partly because actually, it's not alphabetically at the beginning anymore.  It's the first one they see, and they've probably written some C, or they've encountered essentially  the same interface in other languages.  And so they just kind of go, oh, that's what I'm looking for.  But getopt's horrible.  It's really horrible to program.  It's not really Pythonic.  And it doesn't validate your arguments very well.  And I'm not going to talk about opt-pass a lot because it's really been superseded by  argpass.  Argpass offers all the same functionality as far as I'm aware, plus some extra functionality.  So I'm going to talk about argpass briefly.  In the third-party domain, we have a few options as well.  Dockopt is interesting because it takes the opposite approach to anything else I've ever  used.  Dockopt, you actually write your help text.  So the kind of text you'd expect to be printed out when you ran your script with minus, minus  help.  And then Dockopt will pass that, work out what arguments your program should expect, and  then it builds you an object that will pass those command line arguments.  So it's kind of clever.  Having used it, it's kind of fiddly to get the syntax right when you're writing the help.  And I found it just actually took a lot longer, and it wasn't really easier in the end.  But I quite like the idea.  Plus, it's cross-platform, so it's been implemented in more than one language.  So if you, for some reason, are writing a script that is implemented in multiple languages,  that may well be a good option for you.  Clint is much more than a command line parser.  Clint is a project by Kenneth Wrights, and it's a whole framework for building command  line applications.  So I highly recommend it.  It covers a whole cross-section of stuff that we're going to cover in this talk.  I admit that the one thing I don't like about it particularly is the command line parser.  But it's a good option.  If you're already using Clint, then you're probably going to want to use the command  line parser that comes with it.  And as Clint, it's been creating a bit of a buzz recently.  It was released by Armin Ronacker a couple of months ago.  I haven't really used it.  It looks pretty powerful.  Armin's very good at these kinds of libraries.  And I believe he solved a lot of localization encoding issues.  So if you're going to be running under lots of different locales, then Clint is probably  quite a good option for you.  And then finally, we have Compago, which I don't think is a hugely popular library,  but I like the approach that it took.  And I think it's simpler for certain things than argparser.  I'm going to quickly cover that with one slide after talking about argparser.  So I'm sure we've all used argparser in this room.  So it's not going to spend a lot of time on this.  But you instantiate an argument parser.  And then you tell it about all the arguments you expect.  Some of them will be compulsory.  Some will be optional.  And then when you finish doing that, you actually call parse args.  It will go talk to sys argv, extract the arguments, work out which flags correspond to which values,  and then hand you back a namespace object, which contains the extracted values.  If it fails, if somebody is parsing in valid value, it will essentially, it will print  out the help text and then finish with an error code of one.  So here we've got an optional variable called, optional parameter called name.  That's what the n args question mark means.  And so if it's not provided, I'm just replacing it with word world.  This is just a silly hello world script.  And then it prints hello and the value of name.  And that's what happens when you run it.  So because it's optional, we can run it without.  It uses world.  We can run it with Douglas.  And it will print hello, Douglas.  Sorry, skipped ahead.  So we get the same output as we would if we had just extracted that single variable from  argv.  So you might say, why did we bother?  It turns out that that was only one line more than the code that just extracted it from  the first item of argv.  But you get this when you run minus, minus help.  So although we didn't specify minus, minus help, and we didn't actually provide any help  text anywhere in there, we didn't say what name was.  But instantly we can give the user a picture of how they run the application.  I think that's really powerful.  I now no longer write any scripts that don't use arg pass or something similar.  Especially because when somebody comes to me and says, that script you gave me, how exactly  do I run it?  Have you got documentation somewhere?  You can say, run it with minus, minus help.  It will tell you what you should be doing.  And then come back if that's not enough.  So here's a more complex example, just covering some more of the details of arg pass at random.  So here with the first one, we've decided there should be a Boolean value.  There's no value associated with verbose, really.  It's just on or off.  And so we say store true, and now it will store it.  It's true if it's available, false if it's not available.  And then the second argument, number.  We've got this type parameter.  We're passing an int.  So int is just the standard Python int type.  But you'll remember, hopefully, that int is also a function that takes a string and returns  an integer value.  And that's actually all you can pass in any function like that for ad argument under the  type parameter.  So anything that any function that takes a string and returns the actual type you want,  you can pass in.  So then when your namespace will actually contain the types that you're after rather  than you having to constantly convert things, it's nicely reusable.  Yes, sorry.  So arg pass also supports subcommands.  So if you're writing a much more complicated application without the traditional Unix single  use specification, then if you want to write something like git, where you've got subcommands  like ad, push, pull, fetch, remove, sit, stay, you can do this.  It tends to be very verbose.  It works very well, and it's kind of worth doing.  You have to be quite careful about how you factor out all the parsing for the different  commands so that you don't just end up with one huge block of code.  In some ways it's configuration more than code.  So this is where CompoCo is actually quite good.  CompoCo takes a much more lightweight approach.  And in some ways it's not so powerful, but if you're writing this kind of application,  you just want to kind of get it done quickly.  CompoCo can be a really good option.  So I have no idea whether it's supposed to be pronounced CompoCo, by the way.  I don't really know what that means.  So here you instantiate an application.  That's essentially your parser, and that provides you with a decorator.  And then you just use that decorator for any function you want to be an entry point into  your application.  So in this case, we've given our application a greet subcommand, which takes an optional  parameter of two.  So it just parses this out of the function.  And the same with un greet, just so I added another subcommand.  Now we can call this program with un greet, and again, we can choose to call it with a  parameter or not.  If you don't provide a default value for the parameter, then it becomes a compulsory parameter,  and you have to provide it if you're running the un greet command.  And then when you run it, it looks like this.  So it's not quite so nice.  It doesn't have positional parameters as far as I'm aware.  But it's really, really lightweight.  It's really easy to write.  It doesn't take up a lot of space in your code.  So if you're already using third party libraries, then Comprehco's probably a good option.  So that was command line parsing, quick well-binned tour.  Let's talk about input and output.  So IO is often an afterthought.  It's just kind of you think, oh yeah, I'll dump some print statements into my script.  But good IO can really transform a command line application from something that is just  a hacky script.  And it doesn't take a huge amount of work.  It just takes a little bit of thought.  And possibly some libraries that are out there that can really add some power.  So just before we move on, it's worth talking about standard out and standard error, because  I suspect most people don't really think about them.  Standard out is for the output of your program.  So if you think of your program as being a pipe, and it takes some input and it spits  out some output, that's kind of the data that you're producing.  You might, it's perfectly valid to have a script that doesn't produce output.  It's perfectly valid to have a script that doesn't take input.  But if your program produces data in some form, standard out is where it should go.  Standard error is probably the worst name it could be given as a stream.  Standard error is not an error stream.  You can use it for errors, but standard error is really for updates on how the program is  running.  It's kind of information on what's going on.  You can assume that the user is probably going to see standard error, whereas if they're  piping the output of the program to another program or a file, they're not going to see  that.  So they get no feedback on how the program is running.  So the way I think about it is that standard out can be piped, piped to a file or another  program, whereas standard error can be logged.  So at the least, it goes to the screen, or you can save it to a log file.  That's really what it's for.  So let's talk about logging.  Logging is written the right way.  It logs to standard error by default because it's ultimately an informational thing.  Logging is a really powerful library.  I wish I had time to kind of go through it.  But this is an idiom that I use in command line programs quite a bit.  And again, it's not a huge amount of code, and it makes a big difference in the rest  of your program in terms of making it easy to provide information to your user.  So I call basic config to just set up logging.  I set it to warning by default, so anything that's info or debug won't be spat out to  the user.  The reason I do this is because programs should be relatively quiet by default, and then you  can provide a flag to allow the program to be more verbose.  So that's what we've done slightly further down.  We've just checked.  If they want more information, I've set it to levels down to debug.  You can set it down to info.  It all depends on how you like to use logging levels, it's up to you.  And then later, the last four lines are just how you then use that through your program.  But the nice thing here is that logging just takes care of how much information is spat  out to the user after that.  It knows the level that should be spat out.  So it doesn't invade your actual code.  You can use logging as you would in any other kind of program, and it will just do the right  thing after running this block.  So another thing that's worth thinking about that we often don't think about is, are you  talking to a user?  So I think we often assume when we're writing the program that actually standard out is just  going to go to a console, and input is probably going to be fed to us from a file, I don't  know.  So it's just slightly, slightly weird that we have that sort of different approach to  the two streams.  And it's really easy to do in Python.  The main file types, standard in, standard out, standard error, all support is a TTY  as a module, and that just tells you that that stream is attached to a screen somewhere.  So the thing you want to think about as a result of this is, so I was just taking a  picture.  So this is the result for running that program.  So if you run it on its own, it just says none of its pipes to anywhere, it's just coming  to the screen.  If you pipe it to another program, in this case cat, then it says the output is piped  to cat, but obviously not standard error.  And finally, we're just piping input and output.  So it's just testing that all works.  But the thing you really want to think about with this is you can change the format of  your output based on whether it's going to file or another program.  So if your information is being printed to the screen, your main concern is that the  user can read it.  So you can think about formatting stuff in aligned columns, you can think of color, and  you can think about the type of information they want to see, how they want to read it.  If you're piping to a file or another program, you want something that's easily passable.  So you could switch your output to JSON, CSV, or something that's easy for another program  to pass.  It becomes data rather than kind of information.  So I just mentioned color.  Color is also surprisingly easy.  And weirdly, the Ruby guys do this all the time.  We don't tend to see so much color output from Python programs.  You can just use standard ANSI escape characters.  But I don't advise that.  I recommend using a third-party library called Colorama.  So Colorama provides some constants for changing the stream.  And all it really does is spit out a character to standard out.  And so here I'm printing out red, and I print out some text.  The standard out will continue to be printing red text until I send the reset token.  So the next line actually adds a background color of green.  So you then get red text with green background.  And that's kind of what this looks like when you run it.  And you can see how this would make a huge difference to a program.  If you've got errors that are coming out in red, and you've got warnings coming out in  yellow, and you've got normal text coming out in white, it just means that as your reams  of information stream past, you can kind of see that something happened that's actually  interesting rather than just information.  So it makes a big difference with long-running programs, and it makes a big difference with  programs that spit out a lot of information.  The problem with this is that with that script, if I pipe the output to a file, it adds the  it keeps the tokens in.  And you don't really want that.  You want to store plain text when you're saving it in a text file.  And the way that you do this, and this isn't something that's really promoted in documentation,  is you call Colorama.init and you pass in strip equals true.  And that just means, in this case, we're just saying if it's going to a file or another  program, strip out all the characters.  Colorama, one of the neat features is it wraps standard out and standard error when you initialize  it.  And it means that if you print out those ANSI color codes that are only supported on Unix,  not on Windows, if you're on Windows, if the program is being run on Windows, it will automatically  convert them to the Windows equivalent.  So just with kind of a single line of code, you've essentially made the colorization at  least of your application cross platform, which can be kind of nice.  User credentials is something that's supported in the standard Python library, batteries  included.  And I'm just going to quickly cover it.  It's got two functions, get pass.  Get pass prints out password to the screen, and then it allows the user to type in a password  when they hit return.  It returns that as a string.  And it doesn't actually print the password to the screen.  Get user is quite different.  Get user is not an interactive function.  So it's slightly confusing.  Get user will actually talk to the operating system and try and obtain the username of  the user that's currently logged in.  So in this case, I was logged in as Mark, and so it's just picked that up.  And again, these can be kind of useful if your script is communicating with a database.  You don't want to have your password stored in the script that's fundamentally insecure.  You could end up committing that to version control or something.  So if you want it to be a professional application, you should be asking the user what the password  is.  Or perhaps you can store it in a configuration file.  Progress is another one.  So if you've got a long-running task, I can't count the number of times I've run a program  that doesn't spit out very much information.  And you kind of think, has it died?  Should I cancel it?  Should I start again?  Is it a network problem?  Is it just a temporary thing?  Will it carry on working?  So I mean, progress is a really nice, easy way of just saying, you know, we're in the  middle of something, and it's still going.  You can see progress.  You can get an idea for how long it's going to take to complete as well.  So there's a library called Progress Bar, which has been around for a while and I think  is really, really needs some love.  And it's been forked.  So it's received some love, but just in a different place.  This is Progress Bar 2.  If you pip install Progress Bar 2, you get a module called Progress Bar, confusingly.  And then you can instantiate this.  And you can use it in a couple of ways.  So in this way, I'm wrapping an iterator or an iterable and just each time through,  it knows that it's an extra 80th through the process.  And it will just print out this Progress Bar on the line with a percentage in front of  it.  You can do much more complicated things if you want to.  So here, it has this idea of a list of widgets that make up your Progress Bar line.  This is a little bit messy, but it demonstrates a few things.  So you can have strings and you can have what they call widgets, which are kind of updated  each time.  So in this case, it's going to print out the string loading at the start.  You can see that in the comment at the bottom.  And then it's going to print out the percentage that it is through.  And then it's got space.  It's going to print out Progress Bar.  And then the last two are kind of interesting.  So ETA just keeps running a calculation of when the process is going to finish.  So that would be something that would be a pain to develop yourself.  File transfer speed assumes that every time you're updating, you're actually updating  with the number of bytes that you've read.  And so it keeps a running tally of the number of bytes per second, which, again, is kind  of a professional level of update.  So if you're writing a program that's streaming stuff from one place to another, that's quite  nice information for the user.  So with IO, you want to think about adding a flag to specify output.  So I said you can automatically decide, and you probably should, but always let the user  override that because they might not want that JSON written to a file.  They might want the output that they're used to on the screen for some reason.  And the same with verbosity and quietness.  Add a flag to allow them to choose how much information they see.  And also be responsive.  So tell the user that things are still going on.  Don't just leave them with a blinking cursor on a black screen.  That would be bad.  So let's talk about configuration.  We have a bunch of options for configuration.  And we have way more just out of the box in the standard library than I thought until  I went looking.  I kind of thought of any files in JSON.  I suppose I thought about CSV.  But yeah, there's a whole bunch of stuff that we got out of the box.  So you can use environment variables.  We can use JSON out of the box.  You can use CSV.  We've got some rudimentary XML parsers in Python.  Not fantastically performant but fine for config.  But I don't think anybody wants to write XML for configuration.  We've got Apple P list built in on all platforms, which also I don't recommend.  Or obviously we could write our own parser using regular expressions or something horrible  to end again.  Just don't do that.  The third part we've got YAML.  I've been working with Ansible recently.  YAML turns out to be quite a nice format to write.  It's got all the same data types as JSON, but it's actually kind of nice to write.  And then we've got Java properties, which you probably only want to consider if you're  interacting with Java applications.  But I'm going to talk about any files.  And they're kind of not exciting.  And they have a lot of us are Unix developers and we're probably thinking, isn't that a  Windows thing?  But I think they're hugely powerful and I think people underestimate the power of them.  Possibly just don't use them in the right way.  So I wanted to cover it.  So the config parser module has a few different config parser objects in there.  I recommend you use safe config parser because that way your application does some security  holes in some of the others.  They're there for historic reasons more than anything else, as far as I can tell.  And the way that you use it is you instantiate your config parser at which point you will  have an empty configuration object.  And then you call read.  And read doesn't take a single file name.  It takes a list of file names, which is counterintuitive until you understand really what it's designed  for.  And what it does is it goes through that list of paths.  And if the file that that path points to is there, it will read it in.  And it will merge whatever data is in there with whatever is in your config object.  And then it will go on to the next file.  And if that's there, then it will read it in and merge it in.  And if it's not there, it will just skip it.  There are no errors if it doesn't find any of these files.  It expects to begin a list of effectively optional files, files with optional existence.  So the way that you use this, I was talking about these paths.  So the first one is a default file that one assumes is provided with the script.  It's expected to be in the same directory.  It could be in slash, et cetera, or something like that.  But it's essentially a default configuration.  And then on top of that, we've got the per user configuration.  So we're picking tool.ini from the home directory.  And then on top of that, we've got config.ini, which is in our current working directory.  We haven't specified an absolute path for that.  So that's just kind of where we are.  So that's your project configuration.  And these might look like this.  So starting at the bottom.  So the first one that's loaded is defaults.ini.  And that assumes normally when you run this script, whatever it does, it's going to be  talking to local host on port 8080.  And then I'll talk about URL in a second.  And then what it does is it loads tool.ini.  If it's not there, it obviously doesn't load it.  It doesn't complain.  But if it is there, it loads it.  And it overrides port.  So this is like inheritance with objects.  So now we have localhost and port 5000.  And then we load in the project configuration.  For this project, we're pushing everything to www.ninjarockstar.guru.  Love those new domain names.  And so now we've got that host.  We've got port 5000.  But we still have this URL value that's generated with placeholders.  So it's built up from host and port.  So if we print these out, we've got the host that I said, the port that I said.  So you'll notice that also we can call get in.  So it does some basic casting for us, some parsing if we want to.  It's not a huge benefit, but it's kind of nice.  And then we've got URL, which is our generated value.  And you'll notice that it's generated on demand.  So it's got the values that we inserted in later files.  So Ninja Rockstar wasn't around in the file that URL was defined in, but it still pulls  it in anyway.  And that may not seem like a huge benefit.  I mean, it's a bit ugly as well.  But what it means is that if the user wants to use an HTTPS URL, they don't need to change  any code.  They don't need to go into that defaults file and change it.  They can set URL in their personal config file or their project config file.  And then it will use that.  And again, they can use these patterns.  They can use host and port in there.  So it doesn't need to start from scratch again.  So it's kind of powerful.  That's config.  So I'm going to kind of skim through signals a bit.  Talking really fast, actually.  I hope everybody can follow me.  Cool.  Signals.  So it's not nice code, but you can set up a callback, essentially.  So when a signal is received from the operating system, you can get it to call your function.  What that first line is doing is it's overriding the default behavior.  So when your function returns, this is SIGINFO, which is a non-standard flag.  I can't actually remember how you pass it to your program.  But it can be kind of useful if your program is not spitting out any output and the user  still wants to get some progress.  DD does this.  You can pass in a SIGINFO using a control letter sequence.  And then it will just print out its progress report, which is kind of nice.  But the truth is, probably most of us aren't going to do that very often.  We're usually interested in SIGINT, which is keyboard interrupt.  So when somebody hits Control C to finish your program, and that is actually caught  by Python anyway.  So that isn't ignored, and it doesn't, by default, just quit your program.  It raises a keyboard interrupt.  And this is why in all my command line programs, I wrap everything with the keyboard interrupt.  Because the default behavior is to print out a stack trace and exit with an error, which  isn't really what happens.  So you generally want to handle that and not really do anything.  I'm not really sure whether the correct behavior would be to make sure your exit code is zero  when this happens, because it's not an error.  So I'm just going to run very quickly through code structure and packaging, because I didn't  think I had enough time to do more.  So all projects are different.  And a lot of the stuff I work on are actually single file scripts.  And then there's not a lot to it.  You could add a setup.py to make it installable.  But you don't need to do much.  And often, to be honest, they just end up sent by email, which is not great, because  any of these versioning can use the ability for people to update, things like that.  But there's nothing magic about command line scripts.  I recommend you use setup tools rather than distu-tils, because that handles dependencies,  installing any dependencies you might have.  And it also includes some extra features specifically for command line, writing command  line programs.  So here we've got the setup.py that we've written.  I've got an example on the next slide.  And we've got our actual script, which is my tool.  And that should be tiny.  That should be really, really small.  So we should be writing all our code to be reusable.  We may be writing a great, awesome command line application.  But one day, we might want to take that functionality and stick a web interface on it.  We might want to take it and stick a GUI on top of it.  When we might want all three at the same time, depending on how the user starts the application.  So really, you want all your functionality in the library as much as you can.  So my tool should really just call out to something inside my tool lib.  And you can put that in a couple of places.  I added the double underscore main file this morning.  So I know that a few people haven't really come across it.  I think it was added with Python 2.7, obviously Python 3.  You never used to be able to run packages.  You could run modules.  So if your library was distributed as a single file, you could run that.  But it had no way to run modules.  Now if you run my tool lib with Python minus M, my tool lib, then it will look for that  double underscore main file and execute that.  So it's not, I suppose you could import it if you really wanted to, but it's not designed  to be ever imported.  It's only designed to be run.  It's the entry point into your program.  So I think I'd be tempted to put my command line interface in that file, or at least some  of it.  And then we've just got some other files that actually have functionality in there.  It's that gray circle is my tool.py and utils.py.  That's a reasonable way of structuring a relatively small script.  And then the set op.py.  If anybody hasn't written one of these before, they look a little bit overwhelming.  They're a huge amount of just data.  And it's a function call as well.  It's not a data structure.  And so they're just a bit scary.  But it turns out when you write them, they're not that scary.  I recommend kind of, it's a bit of a cargo cult, but I recommend finding some of their  command line scripts that you've installed with pip, where you respect the developer  in some way, where the script has been well developed.  I recommend you're going to look at theirs.  There's a whole host of options, depending on how you want to install these things.  So in this case, the important lines are really the two up, one up from the bottom.  So you tell it what package you want to install, and you tell it there's a script called myTool.  And you could install a series of scripts.  You could have different entry points into your application.  And they'd just be one-liners calling out to functions in myToollib.  And there's other options with setup tools as well, because this is technically the setup  tools setup function.  There's such a thing called an entry point, where you don't write your script.  You write a function as your entry point.  You tell setup tools what functions you want to be, entry points you tell them what you  want those scripts to be called.  And when somebody pip installs, it will install those scripts for you.  And the nice thing about that is on Windows, it will create an executable.  So it'll create a binary.  So it looks much more like a native application to the user.  And they won't have.py on the end.  So they're more native on Unix as well.  Exit codes.  There's not a huge amount to exit codes.  By default, Python does some sensible stuff for you.  If your program just exits, because it got to the end of the main function or the end  of the script that it was running, it exits with zero.  And if you've got an uncaught exception, it will exit with one.  But it's not a great amount of information.  So if you exited because the user provided bad data, you could provide a different exit  code.  If you had network problems, you could provide a different exit code.  There's any number of options.  You can just, it's a very small token of information that's easy for somebody else to then understand.  They don't have to pass any information.  They're just handed back a number.  You should always consider that your program is being run by another program.  Because that's going to make it more reusable and make it more useful.  And there's no specific standard for what these numbers should be.  GNU, the GNU project recommendations, they recommend that you set the high bit if it's  a serious error.  But I'm not really sure of any projects that are actually doing that.  And on BSD, they have a sys exit header file that lists a whole load of what they consider  to be standard exit codes.  So that's probably a good place to look.  I mean, it doesn't hurt if there's a list of exit codes for things and you are exiting  with that condition.  It doesn't hurt to reuse somebody else's standard.  So the stuff I haven't had a chance to cover really are the command line frameworks.  So people have been thinking about this.  A lot of the stuff here, you can see it's kind of boilerplate code.  And it would be nice for somebody to handle this stuff for you, like where is configuration  stored on Windows versus Mac versus Linux.  And Cliff and Clint both provide some subset of that.  I haven't really worked with Cliff.  Cliff is designed really for multi entry point projects and it kind of integrates with set  up tools.  And Clint is really a whole bag of stuff.  It does colorization and progress and command line passing and it will load configuration  from sensible options.  It's a really cool library.  It's not fantastically documented.  Something I'm planning to fork.  Well, I have forked it.  I'm planning to add some documentation to that at some point and see whether it's accepted.  And I haven't covered cross-platform considerations.  So I just kind of skimmed over that when I was talking about configuration file locations.  And it's just for lack of time.  There's a huge amount to this topic and I'm going to write up some blog posts on it.  Hopefully that will be useful to some people.  So I do have five minutes left.  So I wasn't going to do questions.  But if anybody has any questions.  Hey, what are the tricks to get these command line programs to run fast, especially if it's  a short-lived command line program?  I would say minimize your dependencies.  So with a short-lived program, I guess it's not really doing very much.  So you're looking at interpreter startup time and kind of the time it takes to load in libraries.  So, yeah, I would say stick to the smaller.  None of these are big libraries.  None of them take a long time to load.  But, yeah, I guess if you're loading a library that has a large data file behind it or something  like that, that's going to take some time.  But, I mean, the nice thing about Python is it has a really fast startup time anyway.  So it's one of the things that makes Python such a good language for writing command line  tools because they often are short-lived.  There's a syntax error on that file, but ignoring that.  So if you change the behavior when you see a terminal versus when you see a pipe, how  do you go about testing that?  Because obviously sometimes if you, programs that change their behavior in a test environment  can be a bit annoying at times.  Yeah, yeah, I can see that.  I guess testing that I would mock out the standard error object with one that lied about  whether it's a TTY or not.  And then you could use a, you could sort the results in a string IO object and just then  just check it against what you're expecting.  Do you have any recommendation for interactive programs on the command line?  I got this question last time and I totally forgot to write the slide up on it.  So I have used end curses briefly.  It can be a little bit awkward to get running on non-linux platforms.  But no, I haven't done a huge amount of it, so I can't really recommend anything specifically.  There's the built-in module CMD, which enables you to create stuff like that, but it gets  kind of awkward once it gets very big.  Yes.  But yeah, in that case, you don't know any alternative.  CMD is a very cool module for anybody who hasn't used it.  So it's for creating your own shell, essentially your own dedicated shell.  So you allow it, when you run your program, you then get the blinking cursor and then  you can then type in commands.  So it works quite well with programs with subcommands, for example, so you don't have  to type in the git part of git at each time.  You're just within your own contained environment.  I've used it quite well with Agpass before.  So it'll call you a function that you've declared as being one of your commands.  And then anything else that's on the command line just kind of gets passed in, and it  works quite well with Agpass.  But you're right, they're kind of difficult to manage.  And it's just a question of trying to make sure that everything's extracted out into  their own functions, and it doesn't just become one big mess.  It kind of provides an event loop for a command line program.  So very cool.  So awesome, that means basically that it has to work, right?  So about testing, are there any specific tools that you would recommend, like, you know,  Cram, for example, comes to mind?  Are there other tools that you would suggest for testing the command line parts of the  application?  I tend to use unit test just because we're dealing with text, or at least data, and we're  not dealing with kind of events and things like that.  I find that actually unit test tends to have most of the functionality you want.  I mean, obviously there's some add-ons to unit test as well.  I haven't used Cram.  Sorry, I think we're back.  But if you're unit testing, sorry, you're going to mock the send in, send out, so we  can get callers as well, and stuff.  So it can get callers, sorry.  Yeah, but if you're really unit testing that, and you want to see the callers and stuff,  that's going to be pain, right?  Yeah, it's going to be a pain.  I mean, mocking is a way to pain.  Sorry.  Yeah, sorry.  If you're going to test the actual app on the command line, PXSpect works well.  Oh, okay, thank you.  I think that's going to have to be it for questions.  Yeah, but I think we're out of time, I'm afraid, so thank you very much.
Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_0_971.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_972_1079.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_1080_2111.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_2112_2219.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_2220_2711.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_2712_2891.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_2892_3623.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_3624_4595.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_4596_4991.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_4992_5327.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_5328_12683.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_12684_17267.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_17268_18179.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_18180_18911.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_18912_19007.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_19008_19115.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_19116_19451.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_19452_19931.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_19932_20771.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_20772_20819.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_20820_20879.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_20880_21071.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_21072_25739.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_25740_27047.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_27048_27107.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_27108_27455.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_27456_27827.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_27828_27983.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_27984_28535.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_28536_29459.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_29460_30311.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_30312_30695.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_30696_36191.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_36192_43871.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_43872_44219.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_44220_46019.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_46020_47039.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_47040_47423.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_47424_48155.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_48156_48251.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_48252_49379.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_49380_51239.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_51240_55007.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_55008_55355.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_55356_56111.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_56112_61979.png Mark_Smith_-_Writing_Awesome_Command-Line_Programs_in_Python_61980_62109.png
So our next speaker is Mark Smith, who's a developer and trainer at Fandule in Edinburgh. And apart from that, Mark also runs Edinburgh Python user group. And his talk is writing awesome command line programs in Python. Hi. Thanks very much for coming to my talk. I'm going to spend the next 40 minutes or so talking to you about the things you can do to take your command line programs to the next level. And some of the modules that are in the standard library and are available on the net to help you do that. So just quickly to talk about my favourite subject, me. My name is Mark Smith. I work for Fandule. And we're hiring. We're looking for smart people. So if you're interested in that, come and talk to me afterwards. And I'm pretty much called Judy on most social networks for reasons that aren't as interesting as they sound. So I wouldn't normally put the word awesome in a talk title, but this talk is vaguely based on a book that I read about this time last year called Build Awesome Command Line Applications in Ruby. So I was stuck building a command line application in Ruby. And I'm not a Ruby developer. So I kind of needed to look around. And I came across this book. It really is excellent. So if you're stuck in the same situation, I highly recommend it. But what really became apparent to me while I was reading it was that a lot of the libraries that are out there in Ruby, they're similar libraries in Python. Python is just as good if not better for writing this kind of thing. So that's really what led to this talk. So let's just talk quickly about why you might want to write command line programs, because I think this is the only talk on the topic at this conference. So it seems like kind of a minor concern. And yet I'd be surprised if there are any developers at this conference who don't write command line programs regularly to simplify their job. I mean, we're developers. We like writing code. They can't all be web applications. So I would say that the reason to write a command line application is that, in many cases, a command line application is the best application for the job. We have a habit of thinking of them as inferior to web applications. And I would argue that's not necessarily the case. There are definitely some applications that benefit from Windows and buttons and menus. But as developers, we all know that we can automate processes. Programs are run by programs. And if we have something like Bash or even other scripting languages, other programming languages, we can glue applications together in really powerful ways. And besides, it's much easier to write command line applications than a full web application. So why would we want to spend any time writing a good command line application? Why not just chuck a main function in our script and be done with it? And that's because I would say that you want to think about this, do this properly, because scripts are never thrown away. They never get thrown away. You write them, you think you're going to run them once, and then you need to run them again in a week's time. Then you need to read the source code to work out whether it really did what you want. You might need to make some modifications. So I'd say if you start from the beginning with the idea that this is a reusable application, then when somebody comes up to you and says, hey, you've got that cool script that does something I need, then it's already documented. They can understand the output. They don't need to actually modify the file to run it. So let's just talk quickly about what makes up a command line application. So I suspect most people just kind of fell into using and then writing command line applications without really thinking about what makes one. More importantly, what makes a good, easy-to-use, reusable command line application. So this is your program. It's just a program. It's just a bundle of functions, maybe some classes. This is the thing that actually does the useful stuff. But without an interface, it's not any use. So let's start sticking a command line interface on top of it. So the first thing you probably think of are the command line arguments, the flags, the things that begin with minus or minus minus and data that you pass on the command line to run the application. So I've been talking to people about the topic of my talk during the week, and I've been getting this look as if to say, you're doing a talk on arg pass, a 40-minute talk on arg pass. But there's a lot more to it than that. We've got standard in, standard out, standard error. And it kind of looks like we finished, but we haven't finished. We've got signals. So can I have a raise of hands if anybody who's actually caught a signal in a command line application? Wow, that's way more than I expected. I'm not going to go into great detail on signals. I'm going to kind of skim over that because I've got far too many slides. And then we've got configuration. So configuration is not specific to command line applications. Obviously, we have that in GUI applications as well. But I think it's more important. So in GUI applications, the user tends not to see the configuration. It tends to be hidden behind a preference, his pain or something like that. Whereas with a command line application, if you're providing configuration, your user's probably going to have to write that. They're going to have to know where to find it. They're going to have to know how to modify it. And so the decisions you make are more important. And then finally, we've got the exit code, which is the number that's returned when the program's finished to say whether it did what it was supposed to or whether it encountered an error. So I'll start with command line passing. There's a bunch of options. There are three in the standard library. And you'll find that most people who aren't experienced Python developers pick getopt. Partly because actually, it's not alphabetically at the beginning anymore. It's the first one they see, and they've probably written some C, or they've encountered essentially the same interface in other languages. And so they just kind of go, oh, that's what I'm looking for. But getopt's horrible. It's really horrible to program. It's not really Pythonic. And it doesn't validate your arguments very well. And I'm not going to talk about opt-pass a lot because it's really been superseded by argpass. Argpass offers all the same functionality as far as I'm aware, plus some extra functionality. So I'm going to talk about argpass briefly. In the third-party domain, we have a few options as well. Dockopt is interesting because it takes the opposite approach to anything else I've ever used. Dockopt, you actually write your help text. So the kind of text you'd expect to be printed out when you ran your script with minus, minus help. And then Dockopt will pass that, work out what arguments your program should expect, and then it builds you an object that will pass those command line arguments. So it's kind of clever. Having used it, it's kind of fiddly to get the syntax right when you're writing the help. And I found it just actually took a lot longer, and it wasn't really easier in the end. But I quite like the idea. Plus, it's cross-platform, so it's been implemented in more than one language. So if you, for some reason, are writing a script that is implemented in multiple languages, that may well be a good option for you. Clint is much more than a command line parser. Clint is a project by Kenneth Wrights, and it's a whole framework for building command line applications. So I highly recommend it. It covers a whole cross-section of stuff that we're going to cover in this talk. I admit that the one thing I don't like about it particularly is the command line parser. But it's a good option. If you're already using Clint, then you're probably going to want to use the command line parser that comes with it. And as Clint, it's been creating a bit of a buzz recently. It was released by Armin Ronacker a couple of months ago. I haven't really used it. It looks pretty powerful. Armin's very good at these kinds of libraries. And I believe he solved a lot of localization encoding issues. So if you're going to be running under lots of different locales, then Clint is probably quite a good option for you. And then finally, we have Compago, which I don't think is a hugely popular library, but I like the approach that it took. And I think it's simpler for certain things than argparser. I'm going to quickly cover that with one slide after talking about argparser. So I'm sure we've all used argparser in this room. So it's not going to spend a lot of time on this. But you instantiate an argument parser. And then you tell it about all the arguments you expect. Some of them will be compulsory. Some will be optional. And then when you finish doing that, you actually call parse args. It will go talk to sys argv, extract the arguments, work out which flags correspond to which values, and then hand you back a namespace object, which contains the extracted values. If it fails, if somebody is parsing in valid value, it will essentially, it will print out the help text and then finish with an error code of one. So here we've got an optional variable called, optional parameter called name. That's what the n args question mark means. And so if it's not provided, I'm just replacing it with word world. This is just a silly hello world script. And then it prints hello and the value of name. And that's what happens when you run it. So because it's optional, we can run it without. It uses world. We can run it with Douglas. And it will print hello, Douglas. Sorry, skipped ahead. So we get the same output as we would if we had just extracted that single variable from argv. So you might say, why did we bother? It turns out that that was only one line more than the code that just extracted it from the first item of argv. But you get this when you run minus, minus help. So although we didn't specify minus, minus help, and we didn't actually provide any help text anywhere in there, we didn't say what name was. But instantly we can give the user a picture of how they run the application. I think that's really powerful. I now no longer write any scripts that don't use arg pass or something similar. Especially because when somebody comes to me and says, that script you gave me, how exactly do I run it? Have you got documentation somewhere? You can say, run it with minus, minus help. It will tell you what you should be doing. And then come back if that's not enough. So here's a more complex example, just covering some more of the details of arg pass at random. So here with the first one, we've decided there should be a Boolean value. There's no value associated with verbose, really. It's just on or off. And so we say store true, and now it will store it. It's true if it's available, false if it's not available. And then the second argument, number. We've got this type parameter. We're passing an int. So int is just the standard Python int type. But you'll remember, hopefully, that int is also a function that takes a string and returns an integer value. And that's actually all you can pass in any function like that for ad argument under the type parameter. So anything that any function that takes a string and returns the actual type you want, you can pass in. So then when your namespace will actually contain the types that you're after rather than you having to constantly convert things, it's nicely reusable. Yes, sorry. So arg pass also supports subcommands. So if you're writing a much more complicated application without the traditional Unix single use specification, then if you want to write something like git, where you've got subcommands like ad, push, pull, fetch, remove, sit, stay, you can do this. It tends to be very verbose. It works very well, and it's kind of worth doing. You have to be quite careful about how you factor out all the parsing for the different commands so that you don't just end up with one huge block of code. In some ways it's configuration more than code. So this is where CompoCo is actually quite good. CompoCo takes a much more lightweight approach. And in some ways it's not so powerful, but if you're writing this kind of application, you just want to kind of get it done quickly. CompoCo can be a really good option. So I have no idea whether it's supposed to be pronounced CompoCo, by the way. I don't really know what that means. So here you instantiate an application. That's essentially your parser, and that provides you with a decorator. And then you just use that decorator for any function you want to be an entry point into your application. So in this case, we've given our application a greet subcommand, which takes an optional parameter of two. So it just parses this out of the function. And the same with un greet, just so I added another subcommand. Now we can call this program with un greet, and again, we can choose to call it with a parameter or not. If you don't provide a default value for the parameter, then it becomes a compulsory parameter, and you have to provide it if you're running the un greet command. And then when you run it, it looks like this. So it's not quite so nice. It doesn't have positional parameters as far as I'm aware. But it's really, really lightweight. It's really easy to write. It doesn't take up a lot of space in your code. So if you're already using third party libraries, then Comprehco's probably a good option. So that was command line parsing, quick well-binned tour. Let's talk about input and output. So IO is often an afterthought. It's just kind of you think, oh yeah, I'll dump some print statements into my script. But good IO can really transform a command line application from something that is just a hacky script. And it doesn't take a huge amount of work. It just takes a little bit of thought. And possibly some libraries that are out there that can really add some power. So just before we move on, it's worth talking about standard out and standard error, because I suspect most people don't really think about them. Standard out is for the output of your program. So if you think of your program as being a pipe, and it takes some input and it spits out some output, that's kind of the data that you're producing. You might, it's perfectly valid to have a script that doesn't produce output. It's perfectly valid to have a script that doesn't take input. But if your program produces data in some form, standard out is where it should go. Standard error is probably the worst name it could be given as a stream. Standard error is not an error stream. You can use it for errors, but standard error is really for updates on how the program is running. It's kind of information on what's going on. You can assume that the user is probably going to see standard error, whereas if they're piping the output of the program to another program or a file, they're not going to see that. So they get no feedback on how the program is running. So the way I think about it is that standard out can be piped, piped to a file or another program, whereas standard error can be logged. So at the least, it goes to the screen, or you can save it to a log file. That's really what it's for. So let's talk about logging. Logging is written the right way. It logs to standard error by default because it's ultimately an informational thing. Logging is a really powerful library. I wish I had time to kind of go through it. But this is an idiom that I use in command line programs quite a bit. And again, it's not a huge amount of code, and it makes a big difference in the rest of your program in terms of making it easy to provide information to your user. So I call basic config to just set up logging. I set it to warning by default, so anything that's info or debug won't be spat out to the user. The reason I do this is because programs should be relatively quiet by default, and then you can provide a flag to allow the program to be more verbose. So that's what we've done slightly further down. We've just checked. If they want more information, I've set it to levels down to debug. You can set it down to info. It all depends on how you like to use logging levels, it's up to you. And then later, the last four lines are just how you then use that through your program. But the nice thing here is that logging just takes care of how much information is spat out to the user after that. It knows the level that should be spat out. So it doesn't invade your actual code. You can use logging as you would in any other kind of program, and it will just do the right thing after running this block. So another thing that's worth thinking about that we often don't think about is, are you talking to a user? So I think we often assume when we're writing the program that actually standard out is just going to go to a console, and input is probably going to be fed to us from a file, I don't know. So it's just slightly, slightly weird that we have that sort of different approach to the two streams. And it's really easy to do in Python. The main file types, standard in, standard out, standard error, all support is a TTY as a module, and that just tells you that that stream is attached to a screen somewhere. So the thing you want to think about as a result of this is, so I was just taking a picture. So this is the result for running that program. So if you run it on its own, it just says none of its pipes to anywhere, it's just coming to the screen. If you pipe it to another program, in this case cat, then it says the output is piped to cat, but obviously not standard error. And finally, we're just piping input and output. So it's just testing that all works. But the thing you really want to think about with this is you can change the format of your output based on whether it's going to file or another program. So if your information is being printed to the screen, your main concern is that the user can read it. So you can think about formatting stuff in aligned columns, you can think of color, and you can think about the type of information they want to see, how they want to read it. If you're piping to a file or another program, you want something that's easily passable. So you could switch your output to JSON, CSV, or something that's easy for another program to pass. It becomes data rather than kind of information. So I just mentioned color. Color is also surprisingly easy. And weirdly, the Ruby guys do this all the time. We don't tend to see so much color output from Python programs. You can just use standard ANSI escape characters. But I don't advise that. I recommend using a third-party library called Colorama. So Colorama provides some constants for changing the stream. And all it really does is spit out a character to standard out. And so here I'm printing out red, and I print out some text. The standard out will continue to be printing red text until I send the reset token. So the next line actually adds a background color of green. So you then get red text with green background. And that's kind of what this looks like when you run it. And you can see how this would make a huge difference to a program. If you've got errors that are coming out in red, and you've got warnings coming out in yellow, and you've got normal text coming out in white, it just means that as your reams of information stream past, you can kind of see that something happened that's actually interesting rather than just information. So it makes a big difference with long-running programs, and it makes a big difference with programs that spit out a lot of information. The problem with this is that with that script, if I pipe the output to a file, it adds the it keeps the tokens in. And you don't really want that. You want to store plain text when you're saving it in a text file. And the way that you do this, and this isn't something that's really promoted in documentation, is you call Colorama.init and you pass in strip equals true. And that just means, in this case, we're just saying if it's going to a file or another program, strip out all the characters. Colorama, one of the neat features is it wraps standard out and standard error when you initialize it. And it means that if you print out those ANSI color codes that are only supported on Unix, not on Windows, if you're on Windows, if the program is being run on Windows, it will automatically convert them to the Windows equivalent. So just with kind of a single line of code, you've essentially made the colorization at least of your application cross platform, which can be kind of nice. User credentials is something that's supported in the standard Python library, batteries included. And I'm just going to quickly cover it. It's got two functions, get pass. Get pass prints out password to the screen, and then it allows the user to type in a password when they hit return. It returns that as a string. And it doesn't actually print the password to the screen. Get user is quite different. Get user is not an interactive function. So it's slightly confusing. Get user will actually talk to the operating system and try and obtain the username of the user that's currently logged in. So in this case, I was logged in as Mark, and so it's just picked that up. And again, these can be kind of useful if your script is communicating with a database. You don't want to have your password stored in the script that's fundamentally insecure. You could end up committing that to version control or something. So if you want it to be a professional application, you should be asking the user what the password is. Or perhaps you can store it in a configuration file. Progress is another one. So if you've got a long-running task, I can't count the number of times I've run a program that doesn't spit out very much information. And you kind of think, has it died? Should I cancel it? Should I start again? Is it a network problem? Is it just a temporary thing? Will it carry on working? So I mean, progress is a really nice, easy way of just saying, you know, we're in the middle of something, and it's still going. You can see progress. You can get an idea for how long it's going to take to complete as well. So there's a library called Progress Bar, which has been around for a while and I think is really, really needs some love. And it's been forked. So it's received some love, but just in a different place. This is Progress Bar 2. If you pip install Progress Bar 2, you get a module called Progress Bar, confusingly. And then you can instantiate this. And you can use it in a couple of ways. So in this way, I'm wrapping an iterator or an iterable and just each time through, it knows that it's an extra 80th through the process. And it will just print out this Progress Bar on the line with a percentage in front of it. You can do much more complicated things if you want to. So here, it has this idea of a list of widgets that make up your Progress Bar line. This is a little bit messy, but it demonstrates a few things. So you can have strings and you can have what they call widgets, which are kind of updated each time. So in this case, it's going to print out the string loading at the start. You can see that in the comment at the bottom. And then it's going to print out the percentage that it is through. And then it's got space. It's going to print out Progress Bar. And then the last two are kind of interesting. So ETA just keeps running a calculation of when the process is going to finish. So that would be something that would be a pain to develop yourself. File transfer speed assumes that every time you're updating, you're actually updating with the number of bytes that you've read. And so it keeps a running tally of the number of bytes per second, which, again, is kind of a professional level of update. So if you're writing a program that's streaming stuff from one place to another, that's quite nice information for the user. So with IO, you want to think about adding a flag to specify output. So I said you can automatically decide, and you probably should, but always let the user override that because they might not want that JSON written to a file. They might want the output that they're used to on the screen for some reason. And the same with verbosity and quietness. Add a flag to allow them to choose how much information they see. And also be responsive. So tell the user that things are still going on. Don't just leave them with a blinking cursor on a black screen. That would be bad. So let's talk about configuration. We have a bunch of options for configuration. And we have way more just out of the box in the standard library than I thought until I went looking. I kind of thought of any files in JSON. I suppose I thought about CSV. But yeah, there's a whole bunch of stuff that we got out of the box. So you can use environment variables. We can use JSON out of the box. You can use CSV. We've got some rudimentary XML parsers in Python. Not fantastically performant but fine for config. But I don't think anybody wants to write XML for configuration. We've got Apple P list built in on all platforms, which also I don't recommend. Or obviously we could write our own parser using regular expressions or something horrible to end again. Just don't do that. The third part we've got YAML. I've been working with Ansible recently. YAML turns out to be quite a nice format to write. It's got all the same data types as JSON, but it's actually kind of nice to write. And then we've got Java properties, which you probably only want to consider if you're interacting with Java applications. But I'm going to talk about any files. And they're kind of not exciting. And they have a lot of us are Unix developers and we're probably thinking, isn't that a Windows thing? But I think they're hugely powerful and I think people underestimate the power of them. Possibly just don't use them in the right way. So I wanted to cover it. So the config parser module has a few different config parser objects in there. I recommend you use safe config parser because that way your application does some security holes in some of the others. They're there for historic reasons more than anything else, as far as I can tell. And the way that you use it is you instantiate your config parser at which point you will have an empty configuration object. And then you call read. And read doesn't take a single file name. It takes a list of file names, which is counterintuitive until you understand really what it's designed for. And what it does is it goes through that list of paths. And if the file that that path points to is there, it will read it in. And it will merge whatever data is in there with whatever is in your config object. And then it will go on to the next file. And if that's there, then it will read it in and merge it in. And if it's not there, it will just skip it. There are no errors if it doesn't find any of these files. It expects to begin a list of effectively optional files, files with optional existence. So the way that you use this, I was talking about these paths. So the first one is a default file that one assumes is provided with the script. It's expected to be in the same directory. It could be in slash, et cetera, or something like that. But it's essentially a default configuration. And then on top of that, we've got the per user configuration. So we're picking tool.ini from the home directory. And then on top of that, we've got config.ini, which is in our current working directory. We haven't specified an absolute path for that. So that's just kind of where we are. So that's your project configuration. And these might look like this. So starting at the bottom. So the first one that's loaded is defaults.ini. And that assumes normally when you run this script, whatever it does, it's going to be talking to local host on port 8080. And then I'll talk about URL in a second. And then what it does is it loads tool.ini. If it's not there, it obviously doesn't load it. It doesn't complain. But if it is there, it loads it. And it overrides port. So this is like inheritance with objects. So now we have localhost and port 5000. And then we load in the project configuration. For this project, we're pushing everything to www.ninjarockstar.guru. Love those new domain names. And so now we've got that host. We've got port 5000. But we still have this URL value that's generated with placeholders. So it's built up from host and port. So if we print these out, we've got the host that I said, the port that I said. So you'll notice that also we can call get in. So it does some basic casting for us, some parsing if we want to. It's not a huge benefit, but it's kind of nice. And then we've got URL, which is our generated value. And you'll notice that it's generated on demand. So it's got the values that we inserted in later files. So Ninja Rockstar wasn't around in the file that URL was defined in, but it still pulls it in anyway. And that may not seem like a huge benefit. I mean, it's a bit ugly as well. But what it means is that if the user wants to use an HTTPS URL, they don't need to change any code. They don't need to go into that defaults file and change it. They can set URL in their personal config file or their project config file. And then it will use that. And again, they can use these patterns. They can use host and port in there. So it doesn't need to start from scratch again. So it's kind of powerful. That's config. So I'm going to kind of skim through signals a bit. Talking really fast, actually. I hope everybody can follow me. Cool. Signals. So it's not nice code, but you can set up a callback, essentially. So when a signal is received from the operating system, you can get it to call your function. What that first line is doing is it's overriding the default behavior. So when your function returns, this is SIGINFO, which is a non-standard flag. I can't actually remember how you pass it to your program. But it can be kind of useful if your program is not spitting out any output and the user still wants to get some progress. DD does this. You can pass in a SIGINFO using a control letter sequence. And then it will just print out its progress report, which is kind of nice. But the truth is, probably most of us aren't going to do that very often. We're usually interested in SIGINT, which is keyboard interrupt. So when somebody hits Control C to finish your program, and that is actually caught by Python anyway. So that isn't ignored, and it doesn't, by default, just quit your program. It raises a keyboard interrupt. And this is why in all my command line programs, I wrap everything with the keyboard interrupt. Because the default behavior is to print out a stack trace and exit with an error, which isn't really what happens. So you generally want to handle that and not really do anything. I'm not really sure whether the correct behavior would be to make sure your exit code is zero when this happens, because it's not an error. So I'm just going to run very quickly through code structure and packaging, because I didn't think I had enough time to do more. So all projects are different. And a lot of the stuff I work on are actually single file scripts. And then there's not a lot to it. You could add a setup.py to make it installable. But you don't need to do much. And often, to be honest, they just end up sent by email, which is not great, because any of these versioning can use the ability for people to update, things like that. But there's nothing magic about command line scripts. I recommend you use setup tools rather than distu-tils, because that handles dependencies, installing any dependencies you might have. And it also includes some extra features specifically for command line, writing command line programs. So here we've got the setup.py that we've written. I've got an example on the next slide. And we've got our actual script, which is my tool. And that should be tiny. That should be really, really small. So we should be writing all our code to be reusable. We may be writing a great, awesome command line application. But one day, we might want to take that functionality and stick a web interface on it. We might want to take it and stick a GUI on top of it. When we might want all three at the same time, depending on how the user starts the application. So really, you want all your functionality in the library as much as you can. So my tool should really just call out to something inside my tool lib. And you can put that in a couple of places. I added the double underscore main file this morning. So I know that a few people haven't really come across it. I think it was added with Python 2.7, obviously Python 3. You never used to be able to run packages. You could run modules. So if your library was distributed as a single file, you could run that. But it had no way to run modules. Now if you run my tool lib with Python minus M, my tool lib, then it will look for that double underscore main file and execute that. So it's not, I suppose you could import it if you really wanted to, but it's not designed to be ever imported. It's only designed to be run. It's the entry point into your program. So I think I'd be tempted to put my command line interface in that file, or at least some of it. And then we've just got some other files that actually have functionality in there. It's that gray circle is my tool.py and utils.py. That's a reasonable way of structuring a relatively small script. And then the set op.py. If anybody hasn't written one of these before, they look a little bit overwhelming. They're a huge amount of just data. And it's a function call as well. It's not a data structure. And so they're just a bit scary. But it turns out when you write them, they're not that scary. I recommend kind of, it's a bit of a cargo cult, but I recommend finding some of their command line scripts that you've installed with pip, where you respect the developer in some way, where the script has been well developed. I recommend you're going to look at theirs. There's a whole host of options, depending on how you want to install these things. So in this case, the important lines are really the two up, one up from the bottom. So you tell it what package you want to install, and you tell it there's a script called myTool. And you could install a series of scripts. You could have different entry points into your application. And they'd just be one-liners calling out to functions in myToollib. And there's other options with setup tools as well, because this is technically the setup tools setup function. There's such a thing called an entry point, where you don't write your script. You write a function as your entry point. You tell setup tools what functions you want to be, entry points you tell them what you want those scripts to be called. And when somebody pip installs, it will install those scripts for you. And the nice thing about that is on Windows, it will create an executable. So it'll create a binary. So it looks much more like a native application to the user. And they won't have.py on the end. So they're more native on Unix as well. Exit codes. There's not a huge amount to exit codes. By default, Python does some sensible stuff for you. If your program just exits, because it got to the end of the main function or the end of the script that it was running, it exits with zero. And if you've got an uncaught exception, it will exit with one. But it's not a great amount of information. So if you exited because the user provided bad data, you could provide a different exit code. If you had network problems, you could provide a different exit code. There's any number of options. You can just, it's a very small token of information that's easy for somebody else to then understand. They don't have to pass any information. They're just handed back a number. You should always consider that your program is being run by another program. Because that's going to make it more reusable and make it more useful. And there's no specific standard for what these numbers should be. GNU, the GNU project recommendations, they recommend that you set the high bit if it's a serious error. But I'm not really sure of any projects that are actually doing that. And on BSD, they have a sys exit header file that lists a whole load of what they consider to be standard exit codes. So that's probably a good place to look. I mean, it doesn't hurt if there's a list of exit codes for things and you are exiting with that condition. It doesn't hurt to reuse somebody else's standard. So the stuff I haven't had a chance to cover really are the command line frameworks. So people have been thinking about this. A lot of the stuff here, you can see it's kind of boilerplate code. And it would be nice for somebody to handle this stuff for you, like where is configuration stored on Windows versus Mac versus Linux. And Cliff and Clint both provide some subset of that. I haven't really worked with Cliff. Cliff is designed really for multi entry point projects and it kind of integrates with set up tools. And Clint is really a whole bag of stuff. It does colorization and progress and command line passing and it will load configuration from sensible options. It's a really cool library. It's not fantastically documented. Something I'm planning to fork. Well, I have forked it. I'm planning to add some documentation to that at some point and see whether it's accepted. And I haven't covered cross-platform considerations. So I just kind of skimmed over that when I was talking about configuration file locations. And it's just for lack of time. There's a huge amount to this topic and I'm going to write up some blog posts on it. Hopefully that will be useful to some people. So I do have five minutes left. So I wasn't going to do questions. But if anybody has any questions. Hey, what are the tricks to get these command line programs to run fast, especially if it's a short-lived command line program? I would say minimize your dependencies. So with a short-lived program, I guess it's not really doing very much. So you're looking at interpreter startup time and kind of the time it takes to load in libraries. So, yeah, I would say stick to the smaller. None of these are big libraries. None of them take a long time to load. But, yeah, I guess if you're loading a library that has a large data file behind it or something like that, that's going to take some time. But, I mean, the nice thing about Python is it has a really fast startup time anyway. So it's one of the things that makes Python such a good language for writing command line tools because they often are short-lived. There's a syntax error on that file, but ignoring that. So if you change the behavior when you see a terminal versus when you see a pipe, how do you go about testing that? Because obviously sometimes if you, programs that change their behavior in a test environment can be a bit annoying at times. Yeah, yeah, I can see that. I guess testing that I would mock out the standard error object with one that lied about whether it's a TTY or not. And then you could use a, you could sort the results in a string IO object and just then just check it against what you're expecting. Do you have any recommendation for interactive programs on the command line? I got this question last time and I totally forgot to write the slide up on it. So I have used end curses briefly. It can be a little bit awkward to get running on non-linux platforms. But no, I haven't done a huge amount of it, so I can't really recommend anything specifically. There's the built-in module CMD, which enables you to create stuff like that, but it gets kind of awkward once it gets very big. Yes. But yeah, in that case, you don't know any alternative. CMD is a very cool module for anybody who hasn't used it. So it's for creating your own shell, essentially your own dedicated shell. So you allow it, when you run your program, you then get the blinking cursor and then you can then type in commands. So it works quite well with programs with subcommands, for example, so you don't have to type in the git part of git at each time. You're just within your own contained environment. I've used it quite well with Agpass before. So it'll call you a function that you've declared as being one of your commands. And then anything else that's on the command line just kind of gets passed in, and it works quite well with Agpass. But you're right, they're kind of difficult to manage. And it's just a question of trying to make sure that everything's extracted out into their own functions, and it doesn't just become one big mess. It kind of provides an event loop for a command line program. So very cool. So awesome, that means basically that it has to work, right? So about testing, are there any specific tools that you would recommend, like, you know, Cram, for example, comes to mind? Are there other tools that you would suggest for testing the command line parts of the application? I tend to use unit test just because we're dealing with text, or at least data, and we're not dealing with kind of events and things like that. I find that actually unit test tends to have most of the functionality you want. I mean, obviously there's some add-ons to unit test as well. I haven't used Cram. Sorry, I think we're back. But if you're unit testing, sorry, you're going to mock the send in, send out, so we can get callers as well, and stuff. So it can get callers, sorry. Yeah, but if you're really unit testing that, and you want to see the callers and stuff, that's going to be pain, right? Yeah, it's going to be a pain. I mean, mocking is a way to pain. Sorry. Yeah, sorry. If you're going to test the actual app on the command line, PXSpect works well. Oh, okay, thank you. I think that's going to have to be it for questions. Yeah, but I think we're out of time, I'm afraid, so thank you very much.
So our next speaker is Mark Smith, who's a line programs to the next level. And some of the modules that are in the standard library and are cases, a command line application is the best processes. Programs are run by programs. And if They're going to have to know how to modify it. a bunch of options. There are three in the flags correspond to which values, and then hand then when your namespace will actually contain configuration more than code. So this is where if you're already using third party libraries, Let's talk about input and output. So IO is often them. Standard out is for the output of your program. So if you think of your program as being perfectly valid to have a script that doesn't is that standard out can be piped, piped to a logging as you would in any other kind of it's really easy to do in Python. The main file or another program. So if your information is so much color output from Python programs. You application, you should be asking the user what Progress Bar 2, you get a module called Progress make up your Progress Bar line. This is a little kind of thought of any files in JSON. I suppose I probably only want to consider if you're can be kind of useful if your program is not is why in all my command line programs, I wrap going to run very quickly through code structure different. And a lot of the stuff I work on are not a lot to it. You could add a setup.py to make email, which is not great, because any of these starts the application. So really, you want all Python 3. You never used to be able to run consider that your program is being run by really are the command line frameworks. So people stuff for you, like where is configuration stored Python such a good language for writing command testing the command line parts of the