Automated spell-checking in Django projects
Schnell, Jakob
I'm aiming to show how avoid spelling errors by showing ways to implement automated spell-checking. Nearly all Django applications have two main textual bodies that users come in touch with: First, any text in the application and its translation, second the documentation. Since both are usually written by humans, they will contain spelling errors. This is considered harmful and can from time to time hinder the user trying to understand what to do. Therefore, an automated spell-checking tool should be a part of any CI-cyle. For spell-checking documentation, I will give a short demonstration on how to use the "sphinxcontrib-spelling"-tool written by Doug Hellmann, the problems we had and how we overcame them. For spell-checking text in the application and its translations that are usually found in .po-files, I have implemented a small tool called "potypo" (name and development in progress). I will present this tool and show challenges and problems on the way to implementing automated spell checking for .po-files.
 Our next speaker will be Jakob Schnell and he will be talking about automated spell checking. Hello everybody, my name is Jakob, some of you might know me by my nickname, I am a math and computer science student currently on an Erasmus year in Milan in Italy but originally studying at the University of Heidelberg. This is my first talk at any DjangoCon or PyCon or whatsoever, so I thought as an introduction it might be fair to just have a few words about why I am here. And it all starts back in October last year. I was sitting in a database class I think and compared to Katie's talk from earlier it was boring at least to me. And so I decided to do something worthy of my time. And a friend of me, Rafael that has just introduced me has a project called Pre-Tix that you might know that you all bought your ticket from and so I decided to start hacking on Pre-Tix. And I fired up the program and after a few seconds I discovered a typo somewhere in the code and I was like, wow, great, cool. I can just search for this wrong word and I can fix it and submit a PR and it will be great and I will be doing something useful and I will be doing good stuff with my time. And so I did that and that worked fairly well and then I figured maybe there are more typos than just this one and so I fired up a spellchecker, I think a spell and spoiler alert, there were more typos and so I fixed them all or all of them I could find and I asked whether maybe automated spell checking would be something that Pre-Tix would like or that the Django world could use and Rafael's answer was yes, of course, please do so. And now you might be wondering, well, but I am typing my text and I am typing good and perfectly and here are some examples of wrong words that were fixed that are to be found in the Pre-Tix project and I think since all the text that we write is written by us, by humans, we can all agree on the fact that it will contain errors and in front of mind even goes as far as saying whenever you publish anything, be it a book or a thesis or an article or whatsoever, once it is published, once you handed it in, you can open your thesis, your book at whatever page and the first thing you will find is a typo and this will never change. Oh yeah, the audience agrees, cool. And for most of these typos, it's not too bad because most of us are just so used to reading all the time that they will just skip those typos. But if you are not an English native speaker or if you have trouble with the language altogether, then you might stumble across some of these typos and every now and then there might be typos that wouldn't even be, well, let's say, code of conduct compliant. For example, I encountered a typo in the word account where the second C and the O were switched and that's why it's not on the slide. And yes, so obviously we need spell checking because we can't find those errors all by ourselves. Some of them will slip past us and computers are just better at doing so. In general, there are two places where we need spell checking in a Django project. One, first is the documentation and the second is the code itself or the user interface that we provide. And the code is the much more complicated part and I will come to that in about five minutes, I think. So I first want to talk about documentation. For documentation, spell checking is rather easy. That is because all of your documentation is usually in one place. You have one folder called docs and it contains all of your documentation. It's usually large text files that are easily checkable. Usually you only provide documentation in one language so you only need one spell checker for that language. And in the Python world, documentation is usually done with Sphinx, with a project package, whatever. And for Sphinx, fortunately, spell checking already does exist. And it's a package called Sphinx contra.spelling. And the usage is, it was written by Dachalman, the usage is quite simple. You fire up Sphinx minus build and with the directive, oh, nice, I can look here, not there. You fire it up via Sphinx minus build, minus be spelling, then you put all your other options that you need and then you specify in what folder your output should be. And usually that will be something like underscore build slash spelling. The requirements are fairly easy. You can just install Sphinx contra minus spelling via pip and it needs pyenchant, which I will talk about in a few minutes. Those are readily available. You can just add those two lines to your requirements.txt file or whatever. If you are using Travis, you can easily implement the Sphinx build thingy by just adding the enchant package that is needed for pyenchant in your Travis setup. And you can generally easily integrate it into your CI by just checking whether after the build is done, the file output.txt in the directory underscore build slash spelling does exist. And if it does, then you have some errors and you should go fix them. And if it does not, then everything is fine and you can just move on. So this is fairly straightforward. This is nice. It works just fine. It has been working for pre-tics for, I don't know, let's say four or five months now. And it's great. It needs a bit of configuration, of course. First of all, your Sphinx config in constatpy needs to have Sphinx-contribute spelling as an extension. Then you should specify a spelling language which I guess will be en underscore us for most of our applications. Since your application will probably contain some words that are valid for your application, but not for the English language in general. For example, in the pre-tics project, this is pre-tics as a word itself. It will obviously come up in the documentation, but it is not a word to be found in the English dictionary, I think. You need a so-called word list that just lists all these words that are not found anywhere in a dictionary, but in your documentation and that are correctly spelled. And you can just specify a word list via the spelling underscore word underscore list underscore filename directive. And if you want, you can ask Sphinx-contribute spelling to show suggestions for things that you misspelled, which makes it even easier for you to fix all those spellings. And this is easy and nice and works great. For code, everything gets a little bit more complicated. This is because in general, wherever you interact, the second part where you interact with your users, apart from documentation or manuals or handbooks or whatsoever, is your user interface, either usually in some kind of graphical user interface or text user interface, and actually most GUIs contain a lot of text. And if you want to fix all the errors that are there, then you would have to parse Python code, you would have to parse HTML code, you would have to parse CSS probably and extract all the strings and then it's unclear what string should be translated and what strings shouldn't be translated. And to be fair, I have no clue on how you would do this. But there is one thing that makes it very, very easy for us as developers to indeed check all those strings. And that thing is translation. Once your project scales beyond a certain point, you probably want to translate it in another language. For example, for myself, I am German, so English is not my mother tongue. So I would probably develop my application in English since it should be accessible to more than just the German and German-speaking population. But I would probably want to have a German translation because it is easier for me and probably for my use cases to work with the product. It is easier for me to sell it to other people. And so I want that translation. Even if you are an English native speaker, then you might still want translation because you want to sell your product to other people. You might want to translate it into Spanish, which is a language that many, many people speak. And once we have translation, it makes it quite easy for us to automate our spell checking because translation gathers all those strings and there is a system that does this for us that I will present in a minute. And since we now have all of our strings gathered in one place, we can check them for spelling errors rather easily. And basically what the project I want to present to you today is it combines the PO library for translation and Pian Shan for spell checking into one program called Potypo. And of course, it is not just those two components, but it is also a lot of sweat and tears and work and glue code wrapped around stuff to bring this all together. So let's talk about the two components that go together. The first is GetText. GetText is a localization and internationalization system. That is, I think, the standard for translation for internationalization, for localization of things that handle text. And it is based on so-called.po files. This is an example of such a file. It has some metadata in the top, like the creation date, the data for last revision, whoever was the last translator, and most importantly, what language we are translating into in the language tag that is written up there, and then some more metadata in the top. And then it has a bunch of entries that all basically look like the last three lines here. The first is where does this text appear in my project? For example, in this sample.po file, the first message would appear in the file Django.con.slash.talk.intro.pi.inline1. And every entry has a message ID that is the string that is there in your code, which would be hello, Django.con. I am so happy to be here. I truly am. And the second would be the message string, which is the translation of that code into the language that is given above. And the usage of this system in general is fairly simple as well. You can just import pget text and uget text from Django.utils.translation, not t9n, but otherwise it wouldn't fit. And you would, for example, have the two statements, print underscore open brackets. My name is name, closing brackets. And the underscore is the call to uget text. And you would have, for example, a call like print pget text, storeback sizes, venti. And if you now let get text do its thing on the files and gather all those message IDs for you, then this would render as something like the following.po file. You see we have the file Django.con.slash.talk.potato.py. And in the third line we have the my name is name in curly brackets, message ID. And the comment that we gave before this, which is leave name as is, the code will handle it, gets automatically drawn into the.po file. It adds that this is the Python brace format, which is a thing that tells get text that this is special or the thing in curly brackets is special. It shouldn't handle this. And you can add a message string or translation for this. In German this would be heise name. And what the pget text thing does is the first string is a context for your application. So usually a translator wouldn't, I don't think he would be able to handle the word venti, but once you add the context, oh, it's about starbucks sizes, then of course the translation, the meaning is large and it translates to growth in German. So this is fairly easy or fairly simple to handle. We can use the pio library to extract the message IDs and the message strings and then we can check them separately for errors. And since we have a base language given, we know what our message ID, ID's language is and since we have a language tag in our.po file, we know what the message string language is and we can check for errors in both of these languages quite easily. The second part that goes into.po typo is a pie and chant and if you're working with spell checking, then you will probably have encountered some of those words and I will try to give a short overview of what they are. And it all starts with is spell, which is a spell checker that is really old. It was written in 1971 if I'm not mistaken. Originally for the English language, it works quite well. Even today, it is sort of the de facto standard, but along came UTF-8 and with that, well, ispell wasn't really able to handle that, so they, so ispell was created. And to this day, I think ispell is the best spell checker for the English language. When the open office project came along, they implemented their own spell checker, myspell, as part of their abby word word processor and that has replaced ispell as a spell checker in open office and later, hunspell was developed originally for the Hungarian language and it has by now replaced myspell as a spell checker. And from my point of view, ispell is the spell checker that you want if you spell check English text and hunspell is the spell checker that you want to use whenever you check other European languages and for non-European languages, I'm very sorry, but I have no idea. I don't encounter them frequently. In general, those mass of spell checkers produces kind of a problem because we would have to implement them all or we would have to handle them all and they all work sort of the same because they all take text and then they tell you what is wrong in the text, but they also all work sort of differently and that is why the enchant project exists. The live enchant is a library that wraps all of those spell checkers and provides you as a developer and API with which you can easily check all the things and it handles all the specifications of the spell checkers for you and even if they don't implement some functionality but others do, enchant will try to emulate the functionality that other spell checkers have for you and it is very convenient as a developer to just have one framework that you need to know and you don't need to interface with a spell or hunspell or finish spell checkers or Hspell for the Hebrew language or whatnot. There is a list on Wikipedia. Go look it up. It's huge. And there exists a pie enchant written by Ryan Kelly which is currently unfortunately unmaintained which wraps the live enchant and gives you Python binding. So it's quite nice to interface with pie enchant. All right. These are the two main components that go into Potypo and I will now talk about how you would use Potypo in your project and how you would set this up but first I will have a short drink with some water. All right. The usage of Potypo is fairly simple. You just fire up a Potypo in your directory where your setup.cfg lives. I will talk about this in a second. The requirements are fairly simple. Potypo itself is installable via pip install Potypo and you of course need a pie enchant and maybe the Peel library as well. You will then have packages for every language that you want to check your spelling in. So for a German and English project you would install something like A spell minus en for English dictionaries for A spell and you would install something like my spell minus de for the German language. Or actually you would install a spell but I haven't checked whether it is as good as my spell. It should be but I don't know. So again it is fairly easy to use. And the configuration is compliant with the setup.cfg file format. So you have one part where you configure Potypo and your setup.cfg where you also configure flake 8 and other Django and Python things. And you first of course specify a default language which is the language that everything will be translated from. So it is basically the language of your message IDs in your.po file. Then you specify where your local, where your.po files live. Usually they will live in a local directory like Django minus project slash locale or somewhere. And Potypo handles the finding of all those.po files by itself. Currently it just assumes that you follow the structure that is given in the left that I will explain in a second. But it may add some magic to find your.po files automatically. Then of course you can specify languages for which you do not want Potypo to fail or to report any errors. For example if you are just in the process of translating your application into Danish then you might of course want Potypo to report what errors there are in the translation into Danish but you do not want these errors to break your continuous integration process or to break your Travis build. This is why the no fail directory exists. And then of course as before we need a word list for words that are present in our application that are not present in an English dictionary. And you have basically two ways of handling these, of implementing this. The first is you can put them all in a word list directory. If you do so you should specify the WL underscore variable and you should give the directory that your word list live in. And if you have that then they should be named language tag.txt which is the example on the right here. You would have in your Django project you would have a folder word list and then for every language you would have a file d.txt, en.txt and so on and so forth. If you do not like this then you can also put them in various places in the local directory. And usually in your local directory you have one directory for every language that you are translating into, one directory for Danish, one directory for German and so on and so forth. And in those directories you have directory lc underscore messages and in that directory live your.po files. And you can just put your word list.txt in this case called word list.txt in this lc messages folder or in the folder above in the folder for German or for Danish or for whatsoever. And for your base language, for your default language you would just put a file called word list.txt into your local folder. Those are the two ways that word lists are currently handled. If you have any other ideas or if you think, wow, this is not as nice as I wanted, I want this another way, please come talk to me and we can figure something out. But for now this works quite well. Now there are a few exceptions to simply using word lists or to just spell checking. And the first are what I call edge case words. And these are words that contain punctuation or that contain a mix of numbers and letters. This is because of the way that a client works. It basically splits your text by a white space and then strips punctuation from your text. And so the word add ons would be split into a part add and a part ons. And well add is an English word, ons is not, not really. And it would report an error for that. Or for example for translate.pretics.eu it would report an error because it would split it into translate pretics. And it probably won't be able to handle those. And so all those words that contain punctuation, all the words that contain a mix of numbers and letters like for example 214th which is a part of an address I think, you can specify them and then it will just skip those words across all of your text. And you would not want to add them to your word list because again they are not, the parts, the single parts are not really words. MT940 is not really an English word so you would not want it in your English word list. But it's correct. It should be spelled that way. And so we just tell a prototype or please skip this word. And the next thing that is somewhat different is what I called phrases. And a phrase is something that is present in your text although it doesn't technically belong. For example, ticketing powered by is a text that might be used in a German application. And German people probably would understand it because a ticket in English is a ticket in German so ticketing works and powered by, well by in the German word by are quite similar and also powered would be understandable. So you can use the phrase ticketing powered by in a German text. But neither were ticketing nor the word powered nor the word by. By are German words. So you would have to put them in your word list if you do not want them to pop up as errors. Which is bad because by might be a misspelling of the German word by which is written B-E-I because you are typing English and German parallel. And so you want this to be an error but you do not want the phrase ticketing powered by itself to be an error. So this is also handled by phrases. Those are the two main things that you have to filter out when you are checking for your code. If you encounter any other things where neither edge case words nor phrases are enough for you to spell check your application in the way that you want to, I am very happy to see more edge cases and to find a way to work around them. And then last but not least, you might have some HTML code in your code somewhere that will be chunked out by an HTML chunker which is provided by the Enchant project. And then you have some filters that filter the Python brace format from before or that filter URLs or that filter HTML from your code. And those are fairly easy to use. You might write your own. It is quite nice. And this is the complete feature set of Po typo to this point. The project is not that old. It is quite new. It is still a working project, working progress. But it is already used in one application which is Pre-Tix. One is done organizing this conference and gets around merging pull requests which I totally cannot blame him for. If you have any wishes on features that you think you might use, if you want to use this project, you are very welcome to open issues and we can discuss about it. If you think, oh, wow, this is something that I can use, then please do so. Please come to me. Please talk to me. I will very gladly help you set all of this up. As I said, this is still work in progress. There might be things that change. This is my contact information also where you can find the slides and the GitHub repository. Before I finish, I want to thank three entities. First and foremost, I want to thank Rafael for his many, many times that he has provided help for me while going through the process of writing a Python project and publishing it and so on and so forth. He has been super, super helpful and is in general an awesome person. Second, I would like to thank Matthias Vogelgesang for the Metropolis Beamer theme that I have used to create this presentation. And third, I want to thank you as my audience for your time and for your attention. If you have any feedback, please go to the pre-talk system, click on this talk, click on this talk and give me some feedback. If you have questions, please go. Thank you. Okay. We've got a question of the microphone in the middle. Hello. I have one question regarding the documentation. Do you have actually an example of what kind of documentation we can validate and run the library to just spell, to run the spell checking? I don't know if I got your question right, but any kind of documentation basically that is using Sphinx to document its stuff, you can use the Sphinx.spelling project to spell check your documentation. Does this answer your question? Yeah, kind of. Okay. And another one is you showed actually how to spell check the strings representation that we show to the user. Can we use this on the specific code pieces? As I said, if you do not have a translation, so if all of the strings that should be spelled checked are scattered throughout your project in Python files and HTML files and CSS files, then I don't know how to spell check those. If you want to translate your project into any other languages, then you would probably use the get text system and in that case you have a.po file and you can use Potypo to spell check. But in the other case, unfortunately I have no idea on how to do this. Okay. Thank you. Next question from the microphone on the back. Yeah, so thank you for the talk and for the project as my spelling is notoriously bad and one of the biggest source of frustrations for my reviewers. I think I will have plenty of need for that. But one of the things I'm a bit unclear is doc strings. So if you use it for automated documentation of your code, clearly all your class names, your function names, all this stuff is not spelled according to regular syntax rules. Would I have to maintain all these things in the edge case list or is there a different procedure to handle doc strings and function names and class names and the weird spelling? Well, class names and function names are spelled correctly by default, I would say, because you really cannot, I don't know, spell them wrong. If you know what I mean. For example, one of the projects I had, I had the misfortune to name the controller with one L, which is quite annoying. And of course, this was something that was not just a mistake I made when naming the class, but all the documentation that I wrote referring to this very class. So obviously, once spotted, it's easy to rectify, but it would, so how would I approach this particular problem? Okay. These things,.spelling project does not, it only checks for errors in the pros that you write to document your process and not in function names or anything else. If you use the word controller, for example, with only one L in pros to refer to a controller, then it will, of course, tell you this is wrong, but it doesn't check function names or methods or whatever. Thank you. Welcome. Wonderful. If we don't have any further questions, thank you very much. Jacob for the talk. Thank you.
 Our next speaker will be Jakob Schnell and he will be talking about automated spell checking.  Hello everybody, my name is Jakob, some of you might know me by my nickname, I am a math  and computer science student currently on an Erasmus year in Milan in Italy but originally  studying at the University of Heidelberg. This is my first talk at any DjangoCon or  PyCon or whatsoever, so I thought as an introduction it might be fair to just have a few words  about why I am here. And it all starts back in October last year. I was sitting in a database  class I think and compared to Katie's talk from earlier it was boring at least to me.  And so I decided to do something worthy of my time. And a friend of me, Rafael that has  just introduced me has a project called Pre-Tix that you might know that you all bought your  ticket from and so I decided to start hacking on Pre-Tix. And I fired up the program and  after a few seconds I discovered a typo somewhere in the code and I was like, wow, great, cool.  I can just search for this wrong word and I can fix it and submit a PR and it will be  great and I will be doing something useful and I will be doing good stuff with my time.  And so I did that and that worked fairly well and then I figured maybe there are more typos  than just this one and so I fired up a spellchecker, I think a spell and spoiler alert, there were  more typos and so I fixed them all or all of them I could find and I asked whether maybe  automated spell checking would be something that Pre-Tix would like or that the Django  world could use and Rafael's answer was yes, of course, please do so. And now you might  be wondering, well, but I am typing my text and I am typing good and perfectly and here  are some examples of wrong words that were fixed that are to be found in the Pre-Tix  project and I think since all the text that we write is written by us, by humans, we can  all agree on the fact that it will contain errors and in front of mind even goes as far  as saying whenever you publish anything, be it a book or a thesis or an article or whatsoever,  once it is published, once you handed it in, you can open your thesis, your book at whatever  page and the first thing you will find is a typo and this will never change. Oh yeah,  the audience agrees, cool. And for most of these typos, it's not too bad because most  of us are just so used to reading all the time that they will just skip those typos.  But if you are not an English native speaker or if you have trouble with the language altogether,  then you might stumble across some of these typos and every now and then there might be  typos that wouldn't even be, well, let's say, code of conduct compliant. For example, I  encountered a typo in the word account where the second C and the O were switched and that's  why it's not on the slide. And yes, so obviously we need spell checking because we can't find  those errors all by ourselves. Some of them will slip past us and computers are just better  at doing so. In general, there are two places where we need spell checking in a Django project.  One, first is the documentation and the second is the code itself or the user interface that  we provide. And the code is the much more complicated part and I will come to that in  about five minutes, I think. So I first want to talk about documentation. For documentation,  spell checking is rather easy. That is because all of your documentation is usually in one  place. You have one folder called docs and it contains all of your documentation. It's  usually large text files that are easily checkable. Usually you only provide documentation in one  language so you only need one spell checker for that language. And in the Python world,  documentation is usually done with Sphinx, with a project package, whatever. And for  Sphinx, fortunately, spell checking already does exist. And it's a package called Sphinx  contra.spelling. And the usage is, it was written by Dachalman, the usage is quite simple. You  fire up Sphinx minus build and with the directive, oh, nice, I can look here, not there. You  fire it up via Sphinx minus build, minus be spelling, then you put all your other options  that you need and then you specify in what folder your output should be. And usually  that will be something like underscore build slash spelling. The requirements are fairly  easy. You can just install Sphinx contra minus spelling via pip and it needs pyenchant, which  I will talk about in a few minutes. Those are readily available. You can just add those  two lines to your requirements.txt file or whatever. If you are using Travis, you can  easily implement the Sphinx build thingy by just adding the enchant package that is needed  for pyenchant in your Travis setup. And you can generally easily integrate it into your  CI by just checking whether after the build is done, the file output.txt in the directory  underscore build slash spelling does exist. And if it does, then you have some errors  and you should go fix them. And if it does not, then everything is fine and you can just  move on. So this is fairly straightforward. This is nice. It works just fine. It has been  working for pre-tics for, I don't know, let's say four or five months now. And it's great.  It needs a bit of configuration, of course. First of all, your Sphinx config in constatpy  needs to have Sphinx-contribute spelling as an extension. Then you should specify a spelling  language which I guess will be en underscore us for most of our applications. Since your  application will probably contain some words that are valid for your application, but not  for the English language in general. For example, in the pre-tics project, this is pre-tics  as a word itself. It will obviously come up in the documentation, but it is not a word  to be found in the English dictionary, I think. You need a so-called word list that just lists  all these words that are not found anywhere in a dictionary, but in your documentation  and that are correctly spelled. And you can just specify a word list via the spelling  underscore word underscore list underscore filename directive. And if you want, you can  ask Sphinx-contribute spelling to show suggestions for things that you misspelled, which makes  it even easier for you to fix all those spellings. And this is easy and nice and works great.  For code, everything gets a little bit more complicated. This is because in general, wherever  you interact, the second part where you interact with your users, apart from documentation  or manuals or handbooks or whatsoever, is your user interface, either usually in some  kind of graphical user interface or text user interface, and actually most GUIs contain  a lot of text. And if you want to fix all the errors that are there, then you would  have to parse Python code, you would have to parse HTML code, you would have to parse  CSS probably and extract all the strings and then it's unclear what string should be translated  and what strings shouldn't be translated. And to be fair, I have no clue on how you  would do this. But there is one thing that makes it very, very easy for us as developers  to indeed check all those strings. And that thing is translation. Once your project scales  beyond a certain point, you probably want to translate it in another language. For example,  for myself, I am German, so English is not my mother tongue. So I would probably develop  my application in English since it should be accessible to more than just the German  and German-speaking population. But I would probably want to have a German translation  because it is easier for me and probably for my use cases to work with the product. It  is easier for me to sell it to other people. And so I want that translation. Even if you  are an English native speaker, then you might still want translation because you want to  sell your product to other people. You might want to translate it into Spanish, which is  a language that many, many people speak. And once we have translation, it makes it quite  easy for us to automate our spell checking because translation gathers all those strings  and there is a system that does this for us that I will present in a minute. And since  we now have all of our strings gathered in one place, we can check them for spelling  errors rather easily. And basically what the project I want to present to you today is  it combines the PO library for translation and Pian Shan for spell checking into one  program called Potypo. And of course, it is not just those two components, but it is also  a lot of sweat and tears and work and glue code wrapped around stuff to bring this all  together. So let's talk about the two components that go together. The first is GetText. GetText  is a localization and internationalization system. That is, I think, the standard for  translation for internationalization, for localization of things that handle text. And  it is based on so-called.po files. This is an example of such a file. It has some metadata  in the top, like the creation date, the data for last revision, whoever was the last translator,  and most importantly, what language we are translating into in the language tag that  is written up there, and then some more metadata in the top. And then it has a bunch of entries  that all basically look like the last three lines here. The first is where does this text  appear in my project? For example, in this sample.po file, the first message would appear  in the file Django.con.slash.talk.intro.pi.inline1. And every entry has a message ID that is the  string that is there in your code, which would be hello, Django.con. I am so happy to be  here. I truly am. And the second would be the message string, which is the translation  of that code into the language that is given above. And the usage of this system in general  is fairly simple as well. You can just import pget text and uget text from Django.utils.translation,  not t9n, but otherwise it wouldn't fit. And you would, for example, have the two statements,  print underscore open brackets. My name is name, closing brackets. And the underscore  is the call to uget text. And you would have, for example, a call like print pget text,  storeback sizes, venti. And if you now let get text do its thing on the files and gather  all those message IDs for you, then this would render as something like the following.po  file. You see we have the file Django.con.slash.talk.potato.py. And in the third line we have the my name  is name in curly brackets, message ID. And the comment that we gave before this, which  is leave name as is, the code will handle it, gets automatically drawn into the.po file.  It adds that this is the Python brace format, which is a thing that tells get text that  this is special or the thing in curly brackets is special. It shouldn't handle this. And  you can add a message string or translation for this. In German this would be heise name.  And what the pget text thing does is the first string is a context for your application.  So usually a translator wouldn't, I don't think he would be able to handle the word  venti, but once you add the context, oh, it's about starbucks sizes, then of course the  translation, the meaning is large and it translates to growth in German. So this is fairly easy  or fairly simple to handle. We can use the pio library to extract the message IDs and  the message strings and then we can check them separately for errors. And since we have  a base language given, we know what our message ID, ID's language is and since we have a language  tag in our.po file, we know what the message string language is and we can check for errors  in both of these languages quite easily. The second part that goes into.po typo is a  pie and chant and if you're working with spell checking, then you will probably have encountered  some of those words and I will try to give a short overview of what they are. And it  all starts with is spell, which is a spell checker that is really old. It was written  in 1971 if I'm not mistaken. Originally for the English language, it works quite well.  Even today, it is sort of the de facto standard, but along came UTF-8 and with that, well,  ispell wasn't really able to handle that, so they, so ispell was created. And to this  day, I think ispell is the best spell checker for the English language. When the open office  project came along, they implemented their own spell checker, myspell, as part of their  abby word word processor and that has replaced ispell as a spell checker in open office and  later, hunspell was developed originally for the Hungarian language and it has by now replaced  myspell as a spell checker. And from my point of view, ispell is the spell checker that  you want if you spell check English text and hunspell is the spell checker that you want  to use whenever you check other European languages and for non-European languages, I'm very sorry,  but I have no idea. I don't encounter them frequently. In general, those mass of spell  checkers produces kind of a problem because we would have to implement them all or we  would have to handle them all and they all work sort of the same because they all take  text and then they tell you what is wrong in the text, but they also all work sort of  differently and that is why the enchant project exists. The live enchant is a library that  wraps all of those spell checkers and provides you as a developer and API with which you  can easily check all the things and it handles all the specifications of the spell checkers  for you and even if they don't implement some functionality but others do, enchant will  try to emulate the functionality that other spell checkers have for you and it is very  convenient as a developer to just have one framework that you need to know and you don't  need to interface with a spell or hunspell or finish spell checkers or Hspell for the  Hebrew language or whatnot. There is a list on Wikipedia. Go look it up. It's huge. And  there exists a pie enchant written by Ryan Kelly which is currently unfortunately unmaintained  which wraps the live enchant and gives you Python binding. So it's quite nice to interface  with pie enchant. All right. These are the two main components that go into Potypo and  I will now talk about how you would use Potypo in your project and how you would set this  up but first I will have a short drink with some water. All right. The usage of Potypo  is fairly simple. You just fire up a Potypo in your directory where your setup.cfg lives.  I will talk about this in a second. The requirements are fairly simple. Potypo itself is installable  via pip install Potypo and you of course need a pie enchant and maybe the Peel library as  well. You will then have packages for every language that you want to check your spelling  in. So for a German and English project you would install something like A spell minus  en for English dictionaries for A spell and you would install something like my spell  minus de for the German language. Or actually you would install a spell but I haven't checked  whether it is as good as my spell. It should be but I don't know. So again it is fairly  easy to use. And the configuration is compliant with the setup.cfg file format. So you have  one part where you configure Potypo and your setup.cfg where you also configure flake 8  and other Django and Python things. And you first of course specify a default language  which is the language that everything will be translated from. So it is basically the  language of your message IDs in your.po file. Then you specify where your local, where your .po files live. Usually they will live in a local directory like Django minus project  slash locale or somewhere. And Potypo handles the finding of all those.po files by itself.  Currently it just assumes that you follow the structure that is given in the left that  I will explain in a second. But it may add some magic to find your.po files automatically.  Then of course you can specify languages for which you do not want Potypo to fail or to  report any errors. For example if you are just in the process of translating your application  into Danish then you might of course want Potypo to report what errors there are in  the translation into Danish but you do not want these errors to break your continuous  integration process or to break your Travis build. This is why the no fail directory exists.  And then of course as before we need a word list for words that are present in our application  that are not present in an English dictionary. And you have basically two ways of handling  these, of implementing this. The first is you can put them all in a word list directory.  If you do so you should specify the WL underscore variable and you should give the directory  that your word list live in. And if you have that then they should be named language tag.txt  which is the example on the right here. You would have in your Django project you would  have a folder word list and then for every language you would have a file d.txt, en.txt  and so on and so forth. If you do not like this then you can also put them in various  places in the local directory. And usually in your local directory you have one directory  for every language that you are translating into, one directory for Danish, one directory  for German and so on and so forth. And in those directories you have directory lc underscore  messages and in that directory live your.po files. And you can just put your word list.txt  in this case called word list.txt in this lc messages folder or in the folder above  in the folder for German or for Danish or for whatsoever. And for your base language,  for your default language you would just put a file called word list.txt into your local  folder. Those are the two ways that word lists are currently handled. If you have any other  ideas or if you think, wow, this is not as nice as I wanted, I want this another way,  please come talk to me and we can figure something out. But for now this works quite  well. Now there are a few exceptions to simply using word lists or to just spell checking.  And the first are what I call edge case words. And these are words that contain punctuation  or that contain a mix of numbers and letters. This is because of the way that a client works.  It basically splits your text by a white space and then strips punctuation from your text.  And so the word add ons would be split into a part add and a part ons. And well add is  an English word, ons is not, not really. And it would report an error for that. Or for  example for translate.pretics.eu it would report an error because it would split it  into translate pretics. And it probably won't be able to handle those. And so all those  words that contain punctuation, all the words that contain a mix of numbers and letters  like for example 214th which is a part of an address I think, you can specify them and  then it will just skip those words across all of your text. And you would not want to  add them to your word list because again they are not, the parts, the single parts are not  really words. MT940 is not really an English word so you would not want it in your English  word list. But it's correct. It should be spelled that way. And so we just tell a prototype  or please skip this word. And the next thing that is somewhat different is what I called  phrases. And a phrase is something that is present in your text although it doesn't technically  belong. For example, ticketing powered by is a text that might be used in a German application.  And German people probably would understand it because a ticket in English is a ticket  in German so ticketing works and powered by, well by in the German word by are quite similar  and also powered would be understandable. So you can use the phrase ticketing powered  by in a German text. But neither were ticketing nor the word powered nor the word by.  By are German words. So you would have to put them in your word list if you do not want  them to pop up as errors. Which is bad because by might be a misspelling of the German word  by which is written B-E-I because you are typing English and German parallel. And so  you want this to be an error but you do not want the phrase ticketing powered by itself  to be an error. So this is also handled by phrases. Those are the two main things that  you have to filter out when you are checking for your code. If you encounter any other  things where neither edge case words nor phrases are enough for you to spell check your application  in the way that you want to, I am very happy to see more edge cases and to find a way to  work around them. And then last but not least, you might have some HTML code in your code  somewhere that will be chunked out by an HTML chunker which is provided by the Enchant  project. And then you have some filters that filter the Python brace format from before  or that filter URLs or that filter HTML from your code. And those are fairly easy to use.  You might write your own. It is quite nice. And this is the complete feature set of Po  typo to this point. The project is not that old. It is quite new. It is still a working  project, working progress. But it is already used in one application which is Pre-Tix.  One is done organizing this conference and gets around merging pull requests which I  totally cannot blame him for. If you have any wishes on features that you think you  might use, if you want to use this project, you are very welcome to open issues and we  can discuss about it. If you think, oh, wow, this is something that I can use, then please  do so. Please come to me. Please talk to me. I will very gladly help you set all of this  up. As I said, this is still work in progress. There might be things that change. This is  my contact information also where you can find the slides and the GitHub repository.  Before I finish, I want to thank three entities. First and foremost, I want to thank Rafael  for his many, many times that he has provided help for me while going through the process  of writing a Python project and publishing it and so on and so forth. He has been super,  super helpful and is in general an awesome person. Second, I would like to thank Matthias  Vogelgesang for the Metropolis Beamer theme that I have used to create this presentation.  And third, I want to thank you as my audience for your time and for your attention. If you  have any feedback, please go to the pre-talk system, click on this talk, click on this  talk and give me some feedback. If you have questions, please go. Thank you.  Okay. We've got a question of the microphone in the middle.  Hello. I have one question regarding the documentation. Do you have actually an example of what kind  of documentation we can validate and run the library to just spell, to run the spell checking?  I don't know if I got your question right, but any kind of documentation basically that  is using Sphinx to document its stuff, you can use the Sphinx.spelling project to spell  check your documentation. Does this answer your question?  Yeah, kind of. Okay. And another one is you showed actually how to spell check the strings  representation that we show to the user. Can we use this on the specific code pieces?  As I said, if you do not have a translation, so if all of the strings that should be spelled  checked are scattered throughout your project in Python files and HTML files and CSS files,  then I don't know how to spell check those. If you want to translate your project into  any other languages, then you would probably use the get text system and in that case you  have a.po file and you can use Potypo to spell check. But in the other case, unfortunately  I have no idea on how to do this. Okay. Thank you.  Next question from the microphone on the back. Yeah, so thank you for the talk and for the  project as my spelling is notoriously bad and one of the biggest source of frustrations  for my reviewers. I think I will have plenty of need for that. But one of the things I'm  a bit unclear is doc strings. So if you use it for automated documentation of your code,  clearly all your class names, your function names, all this stuff is not spelled according  to regular syntax rules. Would I have to maintain all these things in the edge case list or  is there a different procedure to handle doc strings and function names and class names  and the weird spelling? Well, class names and function names are spelled  correctly by default, I would say, because you really cannot, I don't know, spell them  wrong. If you know what I mean. For example, one of the projects I had, I had the misfortune  to name the controller with one L, which is quite annoying. And of course, this was something  that was not just a mistake I made when naming the class, but all the documentation that  I wrote referring to this very class. So obviously, once spotted, it's easy to rectify, but it  would, so how would I approach this particular problem?  Okay. These things,.spelling project does not, it only checks for errors in the pros  that you write to document your process and not in function names or anything else. If  you use the word controller, for example, with only one L in pros to refer to a controller,  then it will, of course, tell you this is wrong, but it doesn't check function names  or methods or whatever. Thank you. Welcome. Wonderful. If we don't have any further questions,  thank you very much. Jacob for the talk. Thank you.
DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_0_371.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_372_7955.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_7956_10463.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_10464_14999.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_15000_15827.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_15828_16943.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_16944_18971.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_18972_26243.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_26244_26411.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_26412_26519.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_26520_32327.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_32328_32699.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_32700_32855.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_32856_33047.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_33048_39467.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_39468_39803.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_39804_39875.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_39876_39935.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_39936_41327.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_41328_43499.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_43500_44327.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_44328_44507.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_44508_44579.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_44580_44711.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_44712_44951.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_44952_47363.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_47364_47411.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_47412_47531.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_47532_47663.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_47664_48035.png DjangoCon_2018_-_Automated_spell-checking_in_Django_projects_48036_48081.png
Our next speaker will be Jakob Schnell and he will be talking about automated spell checking. Hello everybody, my name is Jakob, some of you might know me by my nickname, I am a math and computer science student currently on an Erasmus year in Milan in Italy but originally studying at the University of Heidelberg. This is my first talk at any DjangoCon or PyCon or whatsoever, so I thought as an introduction it might be fair to just have a few words about why I am here. And it all starts back in October last year. I was sitting in a database class I think and compared to Katie's talk from earlier it was boring at least to me. And so I decided to do something worthy of my time. And a friend of me, Rafael that has just introduced me has a project called Pre-Tix that you might know that you all bought your ticket from and so I decided to start hacking on Pre-Tix. And I fired up the program and after a few seconds I discovered a typo somewhere in the code and I was like, wow, great, cool. I can just search for this wrong word and I can fix it and submit a PR and it will be great and I will be doing something useful and I will be doing good stuff with my time. And so I did that and that worked fairly well and then I figured maybe there are more typos than just this one and so I fired up a spellchecker, I think a spell and spoiler alert, there were more typos and so I fixed them all or all of them I could find and I asked whether maybe automated spell checking would be something that Pre-Tix would like or that the Django world could use and Rafael's answer was yes, of course, please do so. And now you might be wondering, well, but I am typing my text and I am typing good and perfectly and here are some examples of wrong words that were fixed that are to be found in the Pre-Tix project and I think since all the text that we write is written by us, by humans, we can all agree on the fact that it will contain errors and in front of mind even goes as far as saying whenever you publish anything, be it a book or a thesis or an article or whatsoever, once it is published, once you handed it in, you can open your thesis, your book at whatever page and the first thing you will find is a typo and this will never change. Oh yeah, the audience agrees, cool. And for most of these typos, it's not too bad because most of us are just so used to reading all the time that they will just skip those typos. But if you are not an English native speaker or if you have trouble with the language altogether, then you might stumble across some of these typos and every now and then there might be typos that wouldn't even be, well, let's say, code of conduct compliant. For example, I encountered a typo in the word account where the second C and the O were switched and that's why it's not on the slide. And yes, so obviously we need spell checking because we can't find those errors all by ourselves. Some of them will slip past us and computers are just better at doing so. In general, there are two places where we need spell checking in a Django project. One, first is the documentation and the second is the code itself or the user interface that we provide. And the code is the much more complicated part and I will come to that in about five minutes, I think. So I first want to talk about documentation. For documentation, spell checking is rather easy. That is because all of your documentation is usually in one place. You have one folder called docs and it contains all of your documentation. It's usually large text files that are easily checkable. Usually you only provide documentation in one language so you only need one spell checker for that language. And in the Python world, documentation is usually done with Sphinx, with a project package, whatever. And for Sphinx, fortunately, spell checking already does exist. And it's a package called Sphinx contra.spelling. And the usage is, it was written by Dachalman, the usage is quite simple. You fire up Sphinx minus build and with the directive, oh, nice, I can look here, not there. You fire it up via Sphinx minus build, minus be spelling, then you put all your other options that you need and then you specify in what folder your output should be. And usually that will be something like underscore build slash spelling. The requirements are fairly easy. You can just install Sphinx contra minus spelling via pip and it needs pyenchant, which I will talk about in a few minutes. Those are readily available. You can just add those two lines to your requirements.txt file or whatever. If you are using Travis, you can easily implement the Sphinx build thingy by just adding the enchant package that is needed for pyenchant in your Travis setup. And you can generally easily integrate it into your CI by just checking whether after the build is done, the file output.txt in the directory underscore build slash spelling does exist. And if it does, then you have some errors and you should go fix them. And if it does not, then everything is fine and you can just move on. So this is fairly straightforward. This is nice. It works just fine. It has been working for pre-tics for, I don't know, let's say four or five months now. And it's great. It needs a bit of configuration, of course. First of all, your Sphinx config in constatpy needs to have Sphinx-contribute spelling as an extension. Then you should specify a spelling language which I guess will be en underscore us for most of our applications. Since your application will probably contain some words that are valid for your application, but not for the English language in general. For example, in the pre-tics project, this is pre-tics as a word itself. It will obviously come up in the documentation, but it is not a word to be found in the English dictionary, I think. You need a so-called word list that just lists all these words that are not found anywhere in a dictionary, but in your documentation and that are correctly spelled. And you can just specify a word list via the spelling underscore word underscore list underscore filename directive. And if you want, you can ask Sphinx-contribute spelling to show suggestions for things that you misspelled, which makes it even easier for you to fix all those spellings. And this is easy and nice and works great. For code, everything gets a little bit more complicated. This is because in general, wherever you interact, the second part where you interact with your users, apart from documentation or manuals or handbooks or whatsoever, is your user interface, either usually in some kind of graphical user interface or text user interface, and actually most GUIs contain a lot of text. And if you want to fix all the errors that are there, then you would have to parse Python code, you would have to parse HTML code, you would have to parse CSS probably and extract all the strings and then it's unclear what string should be translated and what strings shouldn't be translated. And to be fair, I have no clue on how you would do this. But there is one thing that makes it very, very easy for us as developers to indeed check all those strings. And that thing is translation. Once your project scales beyond a certain point, you probably want to translate it in another language. For example, for myself, I am German, so English is not my mother tongue. So I would probably develop my application in English since it should be accessible to more than just the German and German-speaking population. But I would probably want to have a German translation because it is easier for me and probably for my use cases to work with the product. It is easier for me to sell it to other people. And so I want that translation. Even if you are an English native speaker, then you might still want translation because you want to sell your product to other people. You might want to translate it into Spanish, which is a language that many, many people speak. And once we have translation, it makes it quite easy for us to automate our spell checking because translation gathers all those strings and there is a system that does this for us that I will present in a minute. And since we now have all of our strings gathered in one place, we can check them for spelling errors rather easily. And basically what the project I want to present to you today is it combines the PO library for translation and Pian Shan for spell checking into one program called Potypo. And of course, it is not just those two components, but it is also a lot of sweat and tears and work and glue code wrapped around stuff to bring this all together. So let's talk about the two components that go together. The first is GetText. GetText is a localization and internationalization system. That is, I think, the standard for translation for internationalization, for localization of things that handle text. And it is based on so-called.po files. This is an example of such a file. It has some metadata in the top, like the creation date, the data for last revision, whoever was the last translator, and most importantly, what language we are translating into in the language tag that is written up there, and then some more metadata in the top. And then it has a bunch of entries that all basically look like the last three lines here. The first is where does this text appear in my project? For example, in this sample.po file, the first message would appear in the file Django.con.slash.talk.intro.pi.inline1. And every entry has a message ID that is the string that is there in your code, which would be hello, Django.con. I am so happy to be here. I truly am. And the second would be the message string, which is the translation of that code into the language that is given above. And the usage of this system in general is fairly simple as well. You can just import pget text and uget text from Django.utils.translation, not t9n, but otherwise it wouldn't fit. And you would, for example, have the two statements, print underscore open brackets. My name is name, closing brackets. And the underscore is the call to uget text. And you would have, for example, a call like print pget text, storeback sizes, venti. And if you now let get text do its thing on the files and gather all those message IDs for you, then this would render as something like the following.po file. You see we have the file Django.con.slash.talk.potato.py. And in the third line we have the my name is name in curly brackets, message ID. And the comment that we gave before this, which is leave name as is, the code will handle it, gets automatically drawn into the.po file. It adds that this is the Python brace format, which is a thing that tells get text that this is special or the thing in curly brackets is special. It shouldn't handle this. And you can add a message string or translation for this. In German this would be heise name. And what the pget text thing does is the first string is a context for your application. So usually a translator wouldn't, I don't think he would be able to handle the word venti, but once you add the context, oh, it's about starbucks sizes, then of course the translation, the meaning is large and it translates to growth in German. So this is fairly easy or fairly simple to handle. We can use the pio library to extract the message IDs and the message strings and then we can check them separately for errors. And since we have a base language given, we know what our message ID, ID's language is and since we have a language tag in our.po file, we know what the message string language is and we can check for errors in both of these languages quite easily. The second part that goes into.po typo is a pie and chant and if you're working with spell checking, then you will probably have encountered some of those words and I will try to give a short overview of what they are. And it all starts with is spell, which is a spell checker that is really old. It was written in 1971 if I'm not mistaken. Originally for the English language, it works quite well. Even today, it is sort of the de facto standard, but along came UTF-8 and with that, well, ispell wasn't really able to handle that, so they, so ispell was created. And to this day, I think ispell is the best spell checker for the English language. When the open office project came along, they implemented their own spell checker, myspell, as part of their abby word word processor and that has replaced ispell as a spell checker in open office and later, hunspell was developed originally for the Hungarian language and it has by now replaced myspell as a spell checker. And from my point of view, ispell is the spell checker that you want if you spell check English text and hunspell is the spell checker that you want to use whenever you check other European languages and for non-European languages, I'm very sorry, but I have no idea. I don't encounter them frequently. In general, those mass of spell checkers produces kind of a problem because we would have to implement them all or we would have to handle them all and they all work sort of the same because they all take text and then they tell you what is wrong in the text, but they also all work sort of differently and that is why the enchant project exists. The live enchant is a library that wraps all of those spell checkers and provides you as a developer and API with which you can easily check all the things and it handles all the specifications of the spell checkers for you and even if they don't implement some functionality but others do, enchant will try to emulate the functionality that other spell checkers have for you and it is very convenient as a developer to just have one framework that you need to know and you don't need to interface with a spell or hunspell or finish spell checkers or Hspell for the Hebrew language or whatnot. There is a list on Wikipedia. Go look it up. It's huge. And there exists a pie enchant written by Ryan Kelly which is currently unfortunately unmaintained which wraps the live enchant and gives you Python binding. So it's quite nice to interface with pie enchant. All right. These are the two main components that go into Potypo and I will now talk about how you would use Potypo in your project and how you would set this up but first I will have a short drink with some water. All right. The usage of Potypo is fairly simple. You just fire up a Potypo in your directory where your setup.cfg lives. I will talk about this in a second. The requirements are fairly simple. Potypo itself is installable via pip install Potypo and you of course need a pie enchant and maybe the Peel library as well. You will then have packages for every language that you want to check your spelling in. So for a German and English project you would install something like A spell minus en for English dictionaries for A spell and you would install something like my spell minus de for the German language. Or actually you would install a spell but I haven't checked whether it is as good as my spell. It should be but I don't know. So again it is fairly easy to use. And the configuration is compliant with the setup.cfg file format. So you have one part where you configure Potypo and your setup.cfg where you also configure flake 8 and other Django and Python things. And you first of course specify a default language which is the language that everything will be translated from. So it is basically the language of your message IDs in your.po file. Then you specify where your local, where your.po files live. Usually they will live in a local directory like Django minus project slash locale or somewhere. And Potypo handles the finding of all those.po files by itself. Currently it just assumes that you follow the structure that is given in the left that I will explain in a second. But it may add some magic to find your.po files automatically. Then of course you can specify languages for which you do not want Potypo to fail or to report any errors. For example if you are just in the process of translating your application into Danish then you might of course want Potypo to report what errors there are in the translation into Danish but you do not want these errors to break your continuous integration process or to break your Travis build. This is why the no fail directory exists. And then of course as before we need a word list for words that are present in our application that are not present in an English dictionary. And you have basically two ways of handling these, of implementing this. The first is you can put them all in a word list directory. If you do so you should specify the WL underscore variable and you should give the directory that your word list live in. And if you have that then they should be named language tag.txt which is the example on the right here. You would have in your Django project you would have a folder word list and then for every language you would have a file d.txt, en.txt and so on and so forth. If you do not like this then you can also put them in various places in the local directory. And usually in your local directory you have one directory for every language that you are translating into, one directory for Danish, one directory for German and so on and so forth. And in those directories you have directory lc underscore messages and in that directory live your.po files. And you can just put your word list.txt in this case called word list.txt in this lc messages folder or in the folder above in the folder for German or for Danish or for whatsoever. And for your base language, for your default language you would just put a file called word list.txt into your local folder. Those are the two ways that word lists are currently handled. If you have any other ideas or if you think, wow, this is not as nice as I wanted, I want this another way, please come talk to me and we can figure something out. But for now this works quite well. Now there are a few exceptions to simply using word lists or to just spell checking. And the first are what I call edge case words. And these are words that contain punctuation or that contain a mix of numbers and letters. This is because of the way that a client works. It basically splits your text by a white space and then strips punctuation from your text. And so the word add ons would be split into a part add and a part ons. And well add is an English word, ons is not, not really. And it would report an error for that. Or for example for translate.pretics.eu it would report an error because it would split it into translate pretics. And it probably won't be able to handle those. And so all those words that contain punctuation, all the words that contain a mix of numbers and letters like for example 214th which is a part of an address I think, you can specify them and then it will just skip those words across all of your text. And you would not want to add them to your word list because again they are not, the parts, the single parts are not really words. MT940 is not really an English word so you would not want it in your English word list. But it's correct. It should be spelled that way. And so we just tell a prototype or please skip this word. And the next thing that is somewhat different is what I called phrases. And a phrase is something that is present in your text although it doesn't technically belong. For example, ticketing powered by is a text that might be used in a German application. And German people probably would understand it because a ticket in English is a ticket in German so ticketing works and powered by, well by in the German word by are quite similar and also powered would be understandable. So you can use the phrase ticketing powered by in a German text. But neither were ticketing nor the word powered nor the word by. By are German words. So you would have to put them in your word list if you do not want them to pop up as errors. Which is bad because by might be a misspelling of the German word by which is written B-E-I because you are typing English and German parallel. And so you want this to be an error but you do not want the phrase ticketing powered by itself to be an error. So this is also handled by phrases. Those are the two main things that you have to filter out when you are checking for your code. If you encounter any other things where neither edge case words nor phrases are enough for you to spell check your application in the way that you want to, I am very happy to see more edge cases and to find a way to work around them. And then last but not least, you might have some HTML code in your code somewhere that will be chunked out by an HTML chunker which is provided by the Enchant project. And then you have some filters that filter the Python brace format from before or that filter URLs or that filter HTML from your code. And those are fairly easy to use. You might write your own. It is quite nice. And this is the complete feature set of Po typo to this point. The project is not that old. It is quite new. It is still a working project, working progress. But it is already used in one application which is Pre-Tix. One is done organizing this conference and gets around merging pull requests which I totally cannot blame him for. If you have any wishes on features that you think you might use, if you want to use this project, you are very welcome to open issues and we can discuss about it. If you think, oh, wow, this is something that I can use, then please do so. Please come to me. Please talk to me. I will very gladly help you set all of this up. As I said, this is still work in progress. There might be things that change. This is my contact information also where you can find the slides and the GitHub repository. Before I finish, I want to thank three entities. First and foremost, I want to thank Rafael for his many, many times that he has provided help for me while going through the process of writing a Python project and publishing it and so on and so forth. He has been super, super helpful and is in general an awesome person. Second, I would like to thank Matthias Vogelgesang for the Metropolis Beamer theme that I have used to create this presentation. And third, I want to thank you as my audience for your time and for your attention. If you have any feedback, please go to the pre-talk system, click on this talk, click on this talk and give me some feedback. If you have questions, please go. Thank you. Okay. We've got a question of the microphone in the middle. Hello. I have one question regarding the documentation. Do you have actually an example of what kind of documentation we can validate and run the library to just spell, to run the spell checking? I don't know if I got your question right, but any kind of documentation basically that is using Sphinx to document its stuff, you can use the Sphinx.spelling project to spell check your documentation. Does this answer your question? Yeah, kind of. Okay. And another one is you showed actually how to spell check the strings representation that we show to the user. Can we use this on the specific code pieces? As I said, if you do not have a translation, so if all of the strings that should be spelled checked are scattered throughout your project in Python files and HTML files and CSS files, then I don't know how to spell check those. If you want to translate your project into any other languages, then you would probably use the get text system and in that case you have a.po file and you can use Potypo to spell check. But in the other case, unfortunately I have no idea on how to do this. Okay. Thank you. Next question from the microphone on the back. Yeah, so thank you for the talk and for the project as my spelling is notoriously bad and one of the biggest source of frustrations for my reviewers. I think I will have plenty of need for that. But one of the things I'm a bit unclear is doc strings. So if you use it for automated documentation of your code, clearly all your class names, your function names, all this stuff is not spelled according to regular syntax rules. Would I have to maintain all these things in the edge case list or is there a different procedure to handle doc strings and function names and class names and the weird spelling? Well, class names and function names are spelled correctly by default, I would say, because you really cannot, I don't know, spell them wrong. If you know what I mean. For example, one of the projects I had, I had the misfortune to name the controller with one L, which is quite annoying. And of course, this was something that was not just a mistake I made when naming the class, but all the documentation that I wrote referring to this very class. So obviously, once spotted, it's easy to rectify, but it would, so how would I approach this particular problem? Okay. These things,.spelling project does not, it only checks for errors in the pros that you write to document your process and not in function names or anything else. If you use the word controller, for example, with only one L in pros to refer to a controller, then it will, of course, tell you this is wrong, but it doesn't check function names or methods or whatever. Thank you. Welcome. Wonderful. If we don't have any further questions, thank you very much. Jacob for the talk. Thank you.
will be talking about automated spell checking. were fixed that are to be found in the Pre-Tix write is written by us, by humans, we can all agree on the fact that it will contain errors and first is the documentation and the second is the about documentation. For documentation, spell your output should be. And usually that will be underscore us for most of our applications. Since where you interact with your users, apart from place, we can check them for spelling errors the standard for translation for I will try to give a short overview of what they problem because we would have to implement them our application that are not present in an ways of handling these, of implementing this. The numbers and letters. This is because of the way is a part of an address I think, you can specify are the two main things that you have to filter in progress. There might be things that change. strings representation that we show to the user. then you would probably use the get text system unfortunately I have no idea on how to do this.