Plone + Python 3: How to make the switch
Glick, David
2020 is coming fast, and the journey toward Python 3 compatibility is almost complete — for Plone core itself. But what will it take to migrate an actual Plone website? I’ll review the internal changes that have happened to make Plone run on Python 3, and offer advice on how to approach the upgrade for add-ons and Plone-based projects. Including: - What has changed in Zope 4, - Running Plone under WSGI, - How to migrate your ZODB, - Suggestions of areas that will need careful attention.
 Yeah, so there's a lot of things that you could blame me for, as Kim would say, and get degrees a lot of time. So welcome to my talk on Plone and Python 3, how to make the switch. I'm going to sort of take over where Philip left off yesterday and talk about, he talked about the process of how we got to the release yesterday. I'm going to tell you what I know about how you can take your site on just inversion of Plone Python 2 and get it running on Python 3. So yeah, I'm David Glick. This is a picture. I've developed this habit of going to sprints in Europe and then going to visit Philip afterwards and continuing to sprint. So this was a picture he took just after the Alpine City Sprint in Innsbruck 2017. So yeah, I've been involved for, I think, 2008, was my first conference in D.C. I haven't made it to quite all of them since then, but most of them. I started working for Salesforce a few months ago, so I'm no longer using Plone in my day-to-day job, but in some ways, when I don't use Plone during the day, it feels a little bit more relaxing to do it in the evening. So I've been trying to help Philip finish up the Python 3 work and so forth. Sorry, my throat is a little bit dry. So migrating Python 2 to Python 3. I'm going to talk about... Sorry, my notes aren't showing up here, so I'll have to wing it. I'm going to talk about a little bit of why you would want to do that, what are some of the nice things you'll get in the end. I'm going to talk about how you migrate your database to work in Python 3. And I'm going to talk a little bit about how you would update your code and your add-ons or custom code that you have. So why do you want to be on Python 3? Well, the big one is the stick rather than the carrot, right? It's Python 2 support is ending. You've heard this. The core team is no longer going to support it as of the beginning of 2020. So the time to start working on this is now. And as of a month ago, we have all the tests passing. As of yesterday, we have an alpha release. So things are moving along. We've come a long way from five years ago when we had the Plone 2020 discussion and started scheming how to do this. And there's dozens and dozens of people who have worked on making this happen, so I want to give thanks to all of them. So a big change in Plone 5.2 is that we're now running on Zope 4. And Zope 4 is in some ways not such a big change from Zope 2, even though it's a major version increase. There are probably more changes in between Zope 2.10 and 2.12 than there are from 2.13 to 4, which is the jump we're making here. There are some significant things that got removed in order for us to have less code that needed to be ported to Python 3. So there are a couple of things that are worth mentioning. One is goodnight Z server. If you're running on 5.2, if you're on Python 2, you can still use the Z server, have things running like they were before. But if you're running on Python 3, you're going to be running under whiskey. And we're trying to make this as smooth as possible, so you won't need to change very much of anything. I'll show you one change to make in your build out. But mostly this should be a fairly transparent change. But it's the reason for the major increase in part in the version. Another exciting thing is that we finally have a slightly more modern looking ZMI. It's based on Bootstrap. Still based on DTML, but it looks a little bit more modern. Most of the functionality that was there before is still there. Just moved around a little bit. Control panel link and add menu are up there in the corner now. One thing that's pretty neat is that you can now use Unicode characters in paths, in object IDs. I mean, that's a change that happened in Python for Python 3. And since the Zope graph is a graph of objects, that means we automatically sort of have the ability to do that now. I cheated a little bit for this screenshot because in Plone, we still have normalizers. So you create something, you type the title, and it creates a normalized version of that. So I actually made this and then I renamed it in ZMI. But it should be pretty easy if you want to create a custom normalizer that actually preserves things as they are. Let's talk about some things I like in Python 3, language features. There's way too many to enumerate all of them. So I'll just list a few of my favorites. Chained exceptions. So if you've got a block of code like this where you've got an error in the first statement, it's divide by zero, you can't do that. And then we catch that and then there's another error in the exception handler trying to get some attribute that doesn't exist. In Python 2, you would only see the second exception and you'd look at it and you'd be like, well, where did that come from? In Python 3, it actually shows you, we hit this thing and then while handling it, we hit this other thing. So that's pretty nice. F-strings are a nice new way to format text as if we didn't have enough of those in Python already. But F-strings are cool. So instead of using the percent operator or doing the format method like that where you have to repeat name three times just to get it in there, an F-string, you put an F at the beginning of the string and then you can just put whatever expression you want in curly brackets and it will get evaluated as though it were Python code outside the string. So I haven't actually gotten to use them yet because I'm mostly writing code that is trying to run on both Python 2 and Python 3, but for your projects, if you start moving to Python 3, you might be able to start using them. Pathlib is a neat thing in the standard library for working with file system paths. In Python 2, we would use os.path.join and that would take care of figuring out what the separator character is for your operating system and putting some things together with that. In Python 3, you import path and you create a path instance, defaults to the current directory and then you can actually use the division operator, the slash, so it sort of looks like you're building up a directory path. But when you format it as a string, it will actually format it using the correct separator for your operating system. So neat little thing. And finally, I just wanted to give a shout out to performance and security improvements in Python 3. This is a little bit nebulous to talk about, it's like any performance thing, you got to actually measure it. But just in general, the move to Python 3 means that whatever the core Python team is coming up with in terms of improvements to these areas, we're going to get. So that's nice. So how can we have the nice things? Let's talk about how you upgrade Ploan. This is sort of the classic steps that we're all used to going through, right? You update your build out, rerun it with pulling in the new version pins, then you start up Ploan, you run your upgrade steps. Maybe before that you had to update some of your code like if some imports moved around or something. But if it's a small enough upgrade, you don't even have to do that. This is what the process is going to be like for moving to Python 3, so add a little bit. You update your build out, you're also going to have to add a line to turn on WISG, so that's a small change. You have to convert the database, so all the pickles that are stored that Python 2 knows how to load are not going to necessarily work on Python 3 unless you go through a step to convert it, so I'll talk about that. And then starting it up, running the upgrade steps, that will be pretty much like normal. Updating your code will be necessary, and I'll give some tips about how to do that for Python 3. So when you update your build out, you're going to change the version, so you're extending Ploan 5.2's known good set of versions. You're going to bootstrap your build out using Python 3. I guarantee you'll probably forget that like I have before. And then in the section of the build out that actually installs your Ploan instance, you just put WISG equals on, and that's going to tell it to generate a bin instance script that will run the WISG server instead of running Z server. You might actually be able to get rid of that step, but right now you have to do that. Okay, so converting your database. This is kind of the heart of the talk, the part that is the most still, that we're in the middle of figuring out still. So let's talk about pickles. What's a pickle? It's a delicious fermented food. It's a convenient way of storing objects in your database. We also have this phrase in English. I'm in a bit of a pickle, which sort of means you've gotten yourself into a situation that you aren't quite sure how to get yourself out of. And I think that's a little bit of an apt saying for database migration here. Let me explain why. So I'm going to teach you to be like Jim and read your database at the byte level, just for a little bit if you'll indulge me. This is what we're going to do, and then I'll show you how this looks as a pickle in Python 3 and in Python 2. So we've got some data. It's a list. The list has some bytes. It's got some Unicode, some text. And the third element is a string. So if this is Python 2, then that means it's a byte string. If it's Python 3, that would be a Unicode string. So what we're going to do is we're going to dump that into a pickle and print out the bytes of that pickle. And then we're going to also use pickleTools.dis, which disassembles the pickle and shows you what the different op codes are in the pickle language. So let's do this. I'm actually going to start with Python 3 to show you how it does this. So top part is sort of the raw bytes you get, and then the bottom part is the disassembly. So the first bytes are saying this is prickleProtocol3, which is the new protocol that Python 3 uses. We create an empty list, put it on the stack, and then we add onto the stack the short bin bytes means we're going to put some bytes on there. Remember the bytes. BinUnicodes says we're going to put some Unicode on there, which is what we saw in the code 2. And then the third one also ends up as binUnicode. So Python 3 is interpreting the string as Unicode. So we have these explicit op codes for bytes and Unicode that Python 3 is using. Everything is clear. That's great. Python 2, I'll just hop back and forth here for a little bit, and then I'll point out the differences. So Python 2, where we used to have bin bytes, we now have bin string because string is bytes in Python 2. Okay, that's the first element. Second element is still binUnicode. Unicode is still pickled explicitly as Unicode in Python 2. And then the third one, the ambiguous one, the string becomes also a string. So we have binString for first and the third element. So if you unpickle this in Python 2, like it was pickled, you get bytes, Unicode bytes. If you unpickle this in Python 3, the big question is how does Python 3 deal with this opcode string, binString, that wasn't there before, right? So back here in Python 3, we had bytes and we had Unicode. Everything was explicit. Now we've got these strings. Well, what Python 3 is going to do is it's going to try to turn it into a Python 3 string. Right? So it's going to take the bytes that are stored in the pickle and it's going to decode them to get a Unicode text for Python 3. Now if the... How does it decode it? It decodes it using the default codec, which is Latin 1 in Python, right? So as long as you have non-ASCII characters stored there, that's fine. It gets turned into Unicode and works fine. But if you have something that was stored as UTF-8 encoded string, got pickled as UTF-8 encoded string and then it tries to get loaded by Python 3 as Latin 1, decoded that way, it's going to break. So the result is you've got pickles in your database that can't be loaded without a Unicode decoder. You're kind of stuck. So that's sort of the problem why we need to convert things. But it's not obvious how to convert things because we look at these... Think about creating a tool to walk through the database and update things. It's going to find these bin string opcodes and it doesn't know what it's supposed to become, right? Is it supposed to become... Get decoded and become text? Or is it a binary file that's supposed to stay as bytes and still be treated as bytes in Python 3? So that's sort of the core problem. Sort of an example of a problem you can have with duck typing, right? Duck typing is what we have in Python, right, where you sort of... You don't care what type it is, but as long as it provides the right interface, we know how to work with it. Well, what if you got something that looks like a duck, walks like a duck, but it also looks like a rabbit and walks like a rabbit? You don't know which one it's supposed to become. That's the situation we've got here. And of course, I have to throw in my favorite quote from Trace. Persistence means always having to say, I'm sorry, because that is sort of how I feel about this. So how are we going to deal with this? Well, don't use zodb.py3 migrate, which is one tool that you might run across if you search. It was sort of a start at an attempt to create a tool to do this, but there's just lots of cases it doesn't handle. It didn't quite get far enough, at least at this point. The tool that I have had some success with, and I'll talk about where it succeeds and where it doesn't, is zodb update. This was a tool that's been around for a long time. It was created originally for the use case where you have one class that's stored in your zodb, and then you move the code to a different module, and suddenly you have to update all your pickles to also refer to the new module. So zodb update does that. Sylvan Violant updated it and added a feature to also use it for converting your pickles to Python 3. And apparently they've done that successfully with a big site that wasn't blown. So what does zodb update do? You run it in Python 2, right? So it walks through the database, loads every record in Python 2, you know, so it loads like normal. And then it looks for things that it knows should be binary or things that it knows should be decoded, and it explicitly decodes them or marks them as binary and repickles them, commits a new transaction, voila, you've got your database with Python 3 style pickles. It also writes to the very first bytes at the beginning of your zodb file storage just to say, hey, this is compatible with Python 3 now. How does it pickle something explicitly as bytes in Python 2? Because Python 2 doesn't have a bytes opcode in the pickle protocol. Well, this is what it does. There's in zodb pickle, which is a package, there's a class called binary that's just a simple subclass of bytes. So what this means is that it's going to behave exactly like a bytes. But when it gets pickled, instead of using bin string like normal, Python is actually going to pickle it as a reference to the zodb binary class. So if you load it in Python 2, you get bytes. If you load it in Python 3, you get bytes. So that's sort of the technique that zodb update uses. And where does it use this? It uses it automatically for object IDs. If you've got one record in the zodb that's pointing at the ID of another object, and it uses it automatically for dates and date times, which also get pickled with a byte representation of their value. So what about everything else? You can customize zodb update with entry points. So each package that has a persistent class can do this to say how zodb update should deal with its objects. So you put an entry point in setup.py that says, hey, here's where my mapping is. And then this is what the actual mapping looks like. So this is the example from Zope for the OFS.image module. So it's got the module. It's got a class name file. It's got an attribute name data that says, OK, this should stay binary. And then for the title, on the other hand, it says it should get decoded as UDF8 and become Unicode. So you can sort of go through and do this, maybe a little bit tedious to find all the places in your application that need to have this happen. But it's fairly straightforward to do. But there are some cases that are harder to deal with. Limitations of zodb update are you can't run it in Python 3. So you sort of have to do an initial step of creating your clone 5.2 build out, converting your database, then creating another build out on Python 3 and moving the database over just a little bit of a hassle. Like I said, it only converts the attributes that you tell it about. So if you have something like a custom dexterity type that has something that's supposed to be binary, like you can put that in the mapping, but it's not going to happen automatically. And the really big problem is that zodb update is just working. All it knows about is the pickle that it's looking at one at a time as it goes through the database. So when you've got something like a B tree that has several different persistent records in them, you've got the tree, you've got a sub tree, you've got different buckets, zodb update is going to find one of those buckets and you're going to, it's going to look at it and it's not going to know whether it's a B tree bucket in the catalog or a B tree bucket somewhere else. So it's hard to know, it's hard to target a specific B tree for updates. So we have a few places in Ploen where this is a problem. There's in the catalog, you've got a text index, one of the attributes of a text index is a B tree that stores a mapping of a document ID to this like encoded string of all the words in that document. And that thing is, it's a string in Python 2, it should become actually Unicode in Python 3, but it needs to get decoded using Latin 1 rather than UTF-8. So we sort of need to tell zodb update to deal with it specifically and because it's in a B tree, there's no real good way to do that. So I've done a few experiments over the past couple months trying to like solve some of these edge cases in migrating things and just improve the process a little bit. I'd say we're maybe 70, 80% of the way towards having a good process and there's still some things to figure out. One of the things I did was I made a script which actually got merged early this week into CMF Ploen that lets you run bin instance verifyDB and what that will do is it'll load every record in your database just to make sure that it can do that. So if you've started with the database in Python 2 and you ran zodb update to migrate it to Python 3, then you can run this script and it'll tell you like are there pickles in there that Python 3 won't be able to load because, you know, things are still using the bin string opcode but are actually encoded in some other encoding. So this is a way you can sort of check whether you've added all of the decoder mappings that you need to or if there's other issues. I also have an experiment trying to make zodb update possible to run in Python 3. It's not going to work exactly the same as it does in Python 2 because when you unpickle strings in Python 3 you need to know which encoding to use, right? So I added an option to it. This is in a branch of zodb update right now. I added an option so you can specify the default encoding to use. In Plone we use utf8 in a lot of different places so that's like a pretty good default. So I made it so you can run binzodb update and pass in utf8 and it'll try to decode everything as utf8 and if it fails it'll load it as bytes and just keep it as bytes. So that actually works pretty well for like 95% of the time but it can still have some problems. So you've got something that is supposed to be bytes but you haven't set up a decoder mapping for it and so using the default decoding, if it happens to be bytes that are valid utf8 it'll actually get decoded and stored as unicode when it's not supposed to be unicode. And similarly if you've got something that is supposed to be unicode but decoded in a different encoding than utf8 then that will not end up correct and you'll be in a situation where you can run the verify script and it'll tell you like, oh yeah, sure, I know how to load all those pickles but once it actually gets into the application it'll blow up because it got bytes where it was supposed to have unicode or vice versa. So that doesn't quite solve all the problems either and there's some debate as to whether we should recommend that option or not. Third experiment I did was running ZODB update as part of replication. So if you're using ZRS and you have your site running in Python 2, the idea is you could set up another site in Python 3, set up ZRS so you're replicating from the Python 2 site to the Python 3 site and sort of as pickles get transferred over it would automatically run the ZODB update on them so that you would be able to probably do the migration without as much downtime. I've done this with a sort of, you know, mostly empty site and it sort of worked. I sort of had to hack up ZRS a little bit but it looks like it might be a feasible option if you're using the branch of ZODB update that works in Python 3 and that's sort of the big caveat there. Although I suppose you could also replicate to a Python 2 secondary running ZODB update and then convert it and move the database over to Python 3 after that. So as I said, there's still research underway. We need to sort of figure out how do we deal with the edge cases that don't get handled correctly by ZODB update? And we have like another script or another step that gets run at the start that sort of targets the specific B trees that we know need to be updated and converts them before ZODB update sort of does its thing at the pickle level. Lots of room for trying things out and seeing what works what doesn't. We also really need to figure out like how does this perform? How long does it take to run ZODB update on a real site? I really wanted to try this and have something to report to you at this talk but we just had the release yesterday so I haven't really had a chance and also since I'm not working with Plo and I don't have a really good representative site and also a lot of the sites that we were working with were still had archetypes. So if anybody has a site that is not using archetypes, doesn't have a lot of customizations, might be a really good candidate for trying some of this stuff out in the near future. So wrapping that will wrap up the section on converting your database. That's sort of what I know at this point. I want to talk a little bit about how you update your code for Python 3. Again I can't tell you everything because there's lots of little things, details that you might discover as you go along but I'll at least give some tips. So with the Internet Archetypes, archetypes does not run in Python 3. We haven't spent time trying to make it run. If you need archetypes to run on Python 3, you're welcome to do that but it might be a good opportunity to finally move to dexterity which is what the core team is supporting. There's a tool called 2-3 which is included with Python that you can run on your code. It'll take code for Python 2, convert it to code for Python 3. There's also another tool called Python Modernize that Philip mentioned which sort of does what 2-3 does but also more things. So this is sort of a starting point. You can run that and then look at its changes and see whether they make sense or not. There's some things it does that don't quite make sense like in Python 3 if you've got a dictionary, its methods like items now return an iterator. So 2-3 like hopefully turns it into a list by adding the list around it just to make sure that your any code that was expecting a list will still work. But it does that even in cases where you're clearly just iterating over it and it doesn't really need to do that. So it's not perfect. Use the 6th library to smooth over differences between Python 2 and Python 3 if you're working on an add-on where you're trying to support both. Go read the documentation about the details but some examples of some things it can do. It gives you 6.text type and 6.binary type so if there's places in your code where you were explicitly checking for Unicode or base string or string to see like what type of thing do I have, you can use these instead and it'll work in both for you. So it's a version of Python. It gives you 6.py 2 and 6.py 3 just as, you know, constants you can check to see which one you're in if you need to do things different one or the other. And then the 6.moves module is sort of aliases. If you need to import something that moved in the Python standard library between Python 2 and Python 3, you can import it from 6.moves and it'll get it from the correct location on both pythons. You're going to want to test your add-ons on both Python 2 and Python 3 to make sure that they keep working. This is an area where talks may be helpful. It'll run your tests in a couple different environments but there's probably a little bit of work needed to figure out what's the right pattern for doing that with clone add-ons because talks doesn't really play nice with build out. And that's maybe an area for some work at the sprint. All these hints for the Zope component architecture like implements and adapts and provides need to be updated. There was a change in Python where the top thing used to work but there's a change to how classes get loaded that means we have to switch to a decorator form. You're going to want to just do this everywhere and it works on Python 2 also so it's a pretty easy change to make. I mentioned this one, the change from dict methods returning lists to returning iterators. So there's cases where you might need to make this change. This is sort of a weird gotcha. You can't compare instances of different types in Python 3. So in Python 2 you could create a B tree and you could throw a string key in there and you could throw some other key in there and it would sort of work. In Python 3 it would work except for when it doesn't. So it wasn't necessarily a great idea but in Python 3 it'll explicitly blow up. So there's, you may run into cases where you were doing that without realizing it and you have to think more carefully about what you're putting into a B tree. We have this bad habit of doing open something.read just to get the contents out of a file and then it stays open until the file gets garbage collected. If that happens in Python 3 it'll actually show a warning which is not a big deal but it might get annoying. So if that happens you can use the context manager form with open and then just work with the file in that and it'll automatically close it when you leave the context manager. And save the best for last differences between text and bytes and native string are the ones that require the most thought because none of the automated tools will do this for you. The general principles are use text for most of your internal processing, use bytes for file contents and for input output, use native string if you're like storing the name of some Python object because Python names are strings in Python 2 and they're Python 3 strings in Python 3. But also if you're using the standard library there's some weird exceptions so go read the docs, figure out what the function that you're using is expecting. And then we've got these helpers implements. If you have something that you know is a string but you're not sure if it's like encoded as bytes or if it's Unicode you can pass it in here and get out text or bytes or native string if that's what you want in a particular case. All right. I think I have time for live demo so we'll try to do this. Let's see if I can make this bigger. So here I am, actually let me go to my Python 3 instance and I'm going to copy in. So I made a site just create a new clone site in Japanese. I didn't put anything special in it. I'm going to copy that in here, just the data file. Okay. So that's there. I'm going to run ZODB update, convert pi 3. I'm going to tell it to use the UDF 8 encoding because I'm doing this with my branch of ZODB update that works in Python 3 so need to know that and then I'm going to point out the file. And it'll run along loading those decoder mappings running through the pickles updating them. Okay, then I'm going to run my verified DB script just to make sure that all the pickles can be loaded. Actually takes longer than doing the conversion. Okay. And I'll start things up. Here we have clone site in Japanese, migrated from Python 2 to Python 3. Without any content in it. And in theory I should even be able to like log in and stuff. Edit a page. The great thing about clone is that I can use it even in Japanese because things are color coded. And that's all I've got. Happy to take any questions. Thanks, David. Fantastic talk. Question. Have you ever tried or thought about using the ZODB convert methods of the Red Storage package because it's also for transferring data between two ZODB instances. It should be possible that if you use it with one ZO server running on Python 2, one on Python 3 to get the data transferred directly. I haven't tried it but I assume it would be possible similar to how the ZRS thing that I tried would work. Thank you, David, for the talk and all the work that you did there. Thanks a lot. Can you give an example how much, if which kinds of add-ons would need how many items in their dict to do this, how to call, I can't remember how to call it, in the dictionary that's referenced in the setup py. And who would do that, would need to do that. So are there add-ons that don't need it and which add-ons needed? So hard to speak from experience here because I haven't really tried doing it but you're going to want to look specifically at your persistent classes for any attributes that are Unicode or any attributes that are bytes and add them to the mapping to be explicit. It was sort of looking at that and realizing like, oh, that's going to be like a lot of things to find and easy to miss things. That was sort of why I started looking at the Python 3 version of the conversion and having a default encoding but neither one is really perfect. So yeah, it'll be interesting to try it and see what happens. I was wondering if you have a strong opinion on migration. I mean, it's clear that we have to, that we need a migration path within like the database from with CDB update but there's also transmogrifier, right? Do you think that like when it comes to Python 3, do you have any like strong opinion or recommendation which path to go or is it the usual trade-off that we had before? I mean like always the answer is it depends. I think, yeah, so that's a good point. I wanted to point out that there are other options if you don't feel like staring at pickles all day, you can use transmogrifier, you can use point-out-rest API to just get things out at the application level and put them into a new site. And for something that is a small to medium-sized site that may well be the most straightforward way to approach it, especially if you don't have a lot of legacy in the site that you need to keep around or it's a fairly simple thing. I think there are going to be bigger sites, sites that have been around for a while and who knows what's in there where I guess maybe this is the chance to figure out what's in there. But yeah, certainly doing a database level conversion at least in the short term is going to take a lot more like tinkering. Thanks. Any other questions? All right. So yeah. Thank you, David.
 Yeah, so there's a lot of things that you could blame me for, as Kim would say, and  get degrees a lot of time.  So welcome to my talk on Plone and Python 3, how to make the switch.  I'm going to sort of take over where Philip left off yesterday and talk about, he talked  about the process of how we got to the release yesterday.  I'm going to tell you what I know about how you can take your site on just inversion of  Plone Python 2 and get it running on Python 3.  So yeah, I'm David Glick.  This is a picture.  I've developed this habit of going to sprints in Europe and then going to visit Philip afterwards  and continuing to sprint.  So this was a picture he took just after the Alpine City Sprint in Innsbruck 2017.  So yeah, I've been involved for, I think, 2008, was my first conference in D.C.  I haven't made it to quite all of them since then, but most of them.  I started working for Salesforce a few months ago, so I'm no longer using Plone in my day-to-day  job, but in some ways, when I don't use Plone during the day, it feels a little bit more  relaxing to do it in the evening.  So I've been trying to help Philip finish up the Python 3 work and so forth.  Sorry, my throat is a little bit dry.  So migrating Python 2 to Python 3.  I'm going to talk about...  Sorry, my notes aren't showing up here, so I'll have to wing it.  I'm going to talk about a little bit of why you would want to do that, what are some of  the nice things you'll get in the end.  I'm going to talk about how you migrate your database to work in Python 3.  And I'm going to talk a little bit about how you would update your code and your add-ons  or custom code that you have.  So why do you want to be on Python 3?  Well, the big one is the stick rather than the carrot, right?  It's Python 2 support is ending.  You've heard this.  The core team is no longer going to support it as of the beginning of 2020.  So the time to start working on this is now.  And as of a month ago, we have all the tests passing.  As of yesterday, we have an alpha release.  So things are moving along.  We've come a long way from five years ago when we had the Plone 2020 discussion and  started scheming how to do this.  And there's dozens and dozens of people who have worked on making this happen, so I want  to give thanks to all of them.  So a big change in Plone 5.2 is that we're now running on Zope 4.  And Zope 4 is in some ways not such a big change from Zope 2, even though it's a major  version increase.  There are probably more changes in between Zope 2.10 and 2.12 than there are from 2.13  to 4, which is the jump we're making here.  There are some significant things that got removed in order for us to have less code  that needed to be ported to Python 3.  So there are a couple of things that are worth mentioning.  One is goodnight Z server.  If you're running on 5.2, if you're on Python 2, you can still use the Z server, have things  running like they were before.  But if you're running on Python 3, you're going to be running under whiskey.  And we're trying to make this as smooth as possible, so you won't need to change very  much of anything.  I'll show you one change to make in your build out.  But mostly this should be a fairly transparent change.  But it's the reason for the major increase in part in the version.  Another exciting thing is that we finally have a slightly more modern looking ZMI.  It's based on Bootstrap.  Still based on DTML, but it looks a little bit more modern.  Most of the functionality that was there before is still there.  Just moved around a little bit.  Control panel link and add menu are up there in the corner now.  One thing that's pretty neat is that you can now use Unicode characters in paths, in object  IDs.  I mean, that's a change that happened in Python for Python 3.  And since the Zope graph is a graph of objects, that means we automatically sort of have the  ability to do that now.  I cheated a little bit for this screenshot because in Plone, we still have normalizers.  So you create something, you type the title, and it creates a normalized version of that.  So I actually made this and then I renamed it in ZMI.  But it should be pretty easy if you want to create a custom normalizer that actually preserves  things as they are.  Let's talk about some things I like in Python 3, language features.  There's way too many to enumerate all of them.  So I'll just list a few of my favorites.  Chained exceptions.  So if you've got a block of code like this where you've got an error in the first statement,  it's divide by zero, you can't do that.  And then we catch that and then there's another error in the exception handler trying to get  some attribute that doesn't exist.  In Python 2, you would only see the second exception and you'd look at it and you'd be  like, well, where did that come from?  In Python 3, it actually shows you, we hit this thing and then while handling it, we  hit this other thing.  So that's pretty nice.  F-strings are a nice new way to format text as if we didn't have enough of those in Python  already.  But F-strings are cool.  So instead of using the percent operator or doing the format method like that where you  have to repeat name three times just to get it in there, an F-string, you put an F at  the beginning of the string and then you can just put whatever expression you want in curly  brackets and it will get evaluated as though it were Python code outside the string.  So I haven't actually gotten to use them yet because I'm mostly writing code that is trying  to run on both Python 2 and Python 3, but for your projects, if you start moving to  Python 3, you might be able to start using them.  Pathlib is a neat thing in the standard library for working with file system paths.  In Python 2, we would use os.path.join and that would take care of figuring out what  the separator character is for your operating system and putting some things together with  that.  In Python 3, you import path and you create a path instance, defaults to the current directory  and then you can actually use the division operator, the slash, so it sort of looks like  you're building up a directory path.  But when you format it as a string, it will actually format it using the correct separator  for your operating system.  So neat little thing.  And finally, I just wanted to give a shout out to performance and security improvements  in Python 3.  This is a little bit nebulous to talk about, it's like any performance thing, you got to  actually measure it.  But just in general, the move to Python 3 means that whatever the core Python team is  coming up with in terms of improvements to these areas, we're going to get.  So that's nice.  So how can we have the nice things?  Let's talk about how you upgrade Ploan.  This is sort of the classic steps that we're all used to going through, right?  You update your build out, rerun it with pulling in the new version pins, then you start up  Ploan, you run your upgrade steps.  Maybe before that you had to update some of your code like if some imports moved around  or something.  But if it's a small enough upgrade, you don't even have to do that.  This is what the process is going to be like for moving to Python 3, so add a little bit.  You update your build out, you're also going to have to add a line to turn on WISG, so  that's a small change.  You have to convert the database, so all the pickles that are stored that Python 2 knows  how to load are not going to necessarily work on Python 3 unless you go through a step to  convert it, so I'll talk about that.  And then starting it up, running the upgrade steps, that will be pretty much like normal.  Updating your code will be necessary, and I'll give some tips about how to do that for  Python 3.  So when you update your build out, you're going to change the version, so you're extending  Ploan 5.2's known good set of versions.  You're going to bootstrap your build out using Python 3.  I guarantee you'll probably forget that like I have before.  And then in the section of the build out that actually installs your Ploan instance, you  just put WISG equals on, and that's going to tell it to generate a bin instance script  that will run the WISG server instead of running Z server.  You might actually be able to get rid of that step, but right now you have to do that.  Okay, so converting your database.  This is kind of the heart of the talk, the part that is the most still, that we're in  the middle of figuring out still.  So let's talk about pickles.  What's a pickle?  It's a delicious fermented food.  It's a convenient way of storing objects in your database.  We also have this phrase in English.  I'm in a bit of a pickle, which sort of means you've gotten yourself into a situation that  you aren't quite sure how to get yourself out of.  And I think that's a little bit of an apt saying for database migration here.  Let me explain why.  So I'm going to teach you to be like Jim and read your database at the byte level, just  for a little bit if you'll indulge me.  This is what we're going to do, and then I'll show you how this looks as a pickle in Python  3 and in Python 2.  So we've got some data.  It's a list.  The list has some bytes.  It's got some Unicode, some text.  And the third element is a string.  So if this is Python 2, then that means it's a byte string.  If it's Python 3, that would be a Unicode string.  So what we're going to do is we're going to dump that into a pickle and print out the  bytes of that pickle.  And then we're going to also use pickleTools.dis, which disassembles the pickle and shows you  what the different op codes are in the pickle language.  So let's do this.  I'm actually going to start with Python 3 to show you how it does this.  So top part is sort of the raw bytes you get, and then the bottom part is the disassembly.  So the first bytes are saying this is prickleProtocol3, which is the new protocol that Python 3 uses.  We create an empty list, put it on the stack, and then we add onto the stack the short bin  bytes means we're going to put some bytes on there.  Remember the bytes.  BinUnicodes says we're going to put some Unicode on there, which is what we saw in the code  2.  And then the third one also ends up as binUnicode.  So Python 3 is interpreting the string as Unicode.  So we have these explicit op codes for bytes and Unicode that Python 3 is using.  Everything is clear.  That's great.  Python 2, I'll just hop back and forth here for a little bit, and then I'll point out  the differences.  So Python 2, where we used to have bin bytes, we now have bin string because string is bytes  in Python 2.  Okay, that's the first element.  Second element is still binUnicode.  Unicode is still pickled explicitly as Unicode in Python 2.  And then the third one, the ambiguous one, the string becomes also a string.  So we have binString for first and the third element.  So if you unpickle this in Python 2, like it was pickled, you get bytes, Unicode bytes.  If you unpickle this in Python 3, the big question is how does Python 3 deal with this  opcode string, binString, that wasn't there before, right?  So back here in Python 3, we had bytes and we had Unicode.  Everything was explicit.  Now we've got these strings.  Well, what Python 3 is going to do is it's going to try to turn it into a Python 3 string.  Right?  So it's going to take the bytes that are stored in the pickle and it's going to decode them  to get a Unicode text for Python 3.  Now if the...  How does it decode it?  It decodes it using the default codec, which is Latin 1 in Python, right?  So as long as you have non-ASCII characters stored there, that's fine.  It gets turned into Unicode and works fine.  But if you have something that was stored as UTF-8 encoded string, got pickled as UTF-8  encoded string and then it tries to get loaded by Python 3 as Latin 1, decoded that way,  it's going to break.  So the result is you've got pickles in your database that can't be loaded without a Unicode  decoder.  You're kind of stuck.  So that's sort of the problem why we need to convert things.  But it's not obvious how to convert things because we look at these...  Think about creating a tool to walk through the database and update things.  It's going to find these bin string opcodes and it doesn't know what it's supposed to  become, right?  Is it supposed to become...  Get decoded and become text?  Or is it a binary file that's supposed to stay as bytes and still be treated as bytes  in Python 3?  So that's sort of the core problem.  Sort of an example of a problem you can have with duck typing, right?  Duck typing is what we have in Python, right, where you sort of...  You don't care what type it is, but as long as it provides the right interface, we know  how to work with it.  Well, what if you got something that looks like a duck, walks like a duck, but it also  looks like a rabbit and walks like a rabbit?  You don't know which one it's supposed to become.  That's the situation we've got here.  And of course, I have to throw in my favorite quote from Trace.  Persistence means always having to say, I'm sorry, because that is sort of how I feel  about this.  So how are we going to deal with this?  Well, don't use zodb.py3 migrate, which is one tool that you might run across if you  search.  It was sort of a start at an attempt to create a tool to do this, but there's just lots of  cases it doesn't handle.  It didn't quite get far enough, at least at this point.  The tool that I have had some success with, and I'll talk about where it succeeds and  where it doesn't, is zodb update.  This was a tool that's been around for a long time.  It was created originally for the use case where you have one class that's stored in  your zodb, and then you move the code to a different module, and suddenly you have to  update all your pickles to also refer to the new module.  So zodb update does that.  Sylvan Violant updated it and added a feature to also use it for converting your pickles  to Python 3.  And apparently they've done that successfully with a big site that wasn't blown.  So what does zodb update do?  You run it in Python 2, right?  So it walks through the database, loads every record in Python 2, you know, so it loads  like normal.  And then it looks for things that it knows should be binary or things that it knows should  be decoded, and it explicitly decodes them or marks them as binary and repickles them,  commits a new transaction, voila, you've got your database with Python 3 style pickles.  It also writes to the very first bytes at the beginning of your zodb file storage just  to say, hey, this is compatible with Python 3 now.  How does it pickle something explicitly as bytes in Python 2?  Because Python 2 doesn't have a bytes opcode in the pickle protocol.  Well, this is what it does.  There's in zodb pickle, which is a package, there's a class called binary that's just  a simple subclass of bytes.  So what this means is that it's going to behave exactly like a bytes.  But when it gets pickled, instead of using bin string like normal, Python is actually  going to pickle it as a reference to the zodb binary class.  So if you load it in Python 2, you get bytes.  If you load it in Python 3, you get bytes.  So that's sort of the technique that zodb update uses.  And where does it use this?  It uses it automatically for object IDs.  If you've got one record in the zodb that's pointing at the ID of another object, and  it uses it automatically for dates and date times, which also get pickled with a byte  representation of their value.  So what about everything else?  You can customize zodb update with entry points.  So each package that has a persistent class can do this to say how zodb update should  deal with its objects.  So you put an entry point in setup.py that says, hey, here's where my mapping is.  And then this is what the actual mapping looks like.  So this is the example from Zope for the OFS.image module.  So it's got the module.  It's got a class name file.  It's got an attribute name data that says, OK, this should stay binary.  And then for the title, on the other hand, it says it should get decoded as UDF8 and  become Unicode.  So you can sort of go through and do this, maybe a little bit tedious to find all the  places in your application that need to have this happen.  But it's fairly straightforward to do.  But there are some cases that are harder to deal with.  Limitations of zodb update are you can't run it in Python 3.  So you sort of have to do an initial step of creating your clone 5.2 build out, converting  your database, then creating another build out on Python 3 and moving the database over  just a little bit of a hassle.  Like I said, it only converts the attributes that you tell it about.  So if you have something like a custom dexterity type that has something that's supposed to  be binary, like you can put that in the mapping, but it's not going to happen automatically.  And the really big problem is that zodb update is just working.  All it knows about is the pickle that it's looking at one at a time as it goes through  the database.  So when you've got something like a B tree that has several different persistent records  in them, you've got the tree, you've got a sub tree, you've got different buckets, zodb  update is going to find one of those buckets and you're going to, it's going to look at  it and it's not going to know whether it's a B tree bucket in the catalog or a B tree  bucket somewhere else.  So it's hard to know, it's hard to target a specific B tree for updates.  So we have a few places in Ploen where this is a problem.  There's in the catalog, you've got a text index, one of the attributes of a text index  is a B tree that stores a mapping of a document ID to this like encoded string of all the  words in that document.  And that thing is, it's a string in Python 2, it should become actually Unicode in Python  3, but it needs to get decoded using Latin 1 rather than UTF-8.  So we sort of need to tell zodb update to deal with it specifically and because it's  in a B tree, there's no real good way to do that.  So I've done a few experiments over the past couple months trying to like solve some of  these edge cases in migrating things and just improve the process a little bit.  I'd say we're maybe 70, 80% of the way towards having a good process and there's still some  things to figure out.  One of the things I did was I made a script which actually got merged early this week  into CMF Ploen that lets you run bin instance verifyDB and what that will do is it'll load  every record in your database just to make sure that it can do that.  So if you've started with the database in Python 2 and you ran zodb update to migrate  it to Python 3, then you can run this script and it'll tell you like are there pickles  in there that Python 3 won't be able to load because, you know, things are still using  the bin string opcode but are actually encoded in some other encoding.  So this is a way you can sort of check whether you've added all of the decoder mappings that  you need to or if there's other issues.  I also have an experiment trying to make zodb update possible to run in Python 3.  It's not going to work exactly the same as it does in Python 2 because when you unpickle  strings in Python 3 you need to know which encoding to use, right?  So I added an option to it.  This is in a branch of zodb update right now.  I added an option so you can specify the default encoding to use.  In Plone we use utf8 in a lot of different places so that's like a pretty good default.  So I made it so you can run binzodb update and pass in utf8 and it'll try to decode everything  as utf8 and if it fails it'll load it as bytes and just keep it as bytes.  So that actually works pretty well for like 95% of the time but it can still have some  problems.  So you've got something that is supposed to be bytes but you haven't set up a decoder  mapping for it and so using the default decoding, if it happens to be bytes that are valid utf8  it'll actually get decoded and stored as unicode when it's not supposed to be unicode.  And similarly if you've got something that is supposed to be unicode but decoded in a  different encoding than utf8 then that will not end up correct and you'll be in a situation  where you can run the verify script and it'll tell you like, oh yeah, sure, I know how to  load all those pickles but once it actually gets into the application it'll blow up because  it got bytes where it was supposed to have unicode or vice versa.  So that doesn't quite solve all the problems either and there's some debate as to whether  we should recommend that option or not.  Third experiment I did was running ZODB update as part of replication.  So if you're using ZRS and you have your site running in Python 2, the idea is you could  set up another site in Python 3, set up ZRS so you're replicating from the Python 2 site  to the Python 3 site and sort of as pickles get transferred over it would automatically  run the ZODB update on them so that you would be able to probably do the migration without  as much downtime.  I've done this with a sort of, you know, mostly empty site and it sort of worked.  I sort of had to hack up ZRS a little bit but it looks like it might be a feasible option  if you're using the branch of ZODB update that works in Python 3 and that's sort of  the big caveat there.  Although I suppose you could also replicate to a Python 2 secondary running ZODB update  and then convert it and move the database over to Python 3 after that.  So as I said, there's still research underway.  We need to sort of figure out how do we deal with the edge cases that don't get handled  correctly by ZODB update?  And we have like another script or another step that gets run at the start that sort  of targets the specific B trees that we know need to be updated and converts them before  ZODB update sort of does its thing at the pickle level.  Lots of room for trying things out and seeing what works what doesn't.  We also really need to figure out like how does this perform?  How long does it take to run ZODB update on a real site?  I really wanted to try this and have something to report to you at this talk but we just  had the release yesterday so I haven't really had a chance and also since I'm not working  with Plo and I don't have a really good representative site and also a lot of the sites that we were  working with were still had archetypes.  So if anybody has a site that is not using archetypes, doesn't have a lot of customizations,  might be a really good candidate for trying some of this stuff out in the near future.  So wrapping that will wrap up the section on converting your database.  That's sort of what I know at this point.  I want to talk a little bit about how you update your code for Python 3.  Again I can't tell you everything because there's lots of little things, details that  you might discover as you go along but I'll at least give some tips.  So with the Internet Archetypes, archetypes does not run in Python 3.  We haven't spent time trying to make it run.  If you need archetypes to run on Python 3, you're welcome to do that but it might be  a good opportunity to finally move to dexterity which is what the core team is supporting.  There's a tool called 2-3 which is included with Python that you can run on your code.  It'll take code for Python 2, convert it to code for Python 3.  There's also another tool called Python Modernize that Philip mentioned which sort of does what  2-3 does but also more things.  So this is sort of a starting point.  You can run that and then look at its changes and see whether they make sense or not.  There's some things it does that don't quite make sense like in Python 3 if you've got  a dictionary, its methods like items now return an iterator.  So 2-3 like hopefully turns it into a list by adding the list around it just to make  sure that your any code that was expecting a list will still work.  But it does that even in cases where you're clearly just iterating over it and it doesn't  really need to do that.  So it's not perfect.  Use the 6th library to smooth over differences between Python 2 and Python 3 if you're working  on an add-on where you're trying to support both.  Go read the documentation about the details but some examples of some things it can do.  It gives you 6.text type and 6.binary type so if there's places in your code where you  were explicitly checking for Unicode or base string or string to see like what type of  thing do I have, you can use these instead and it'll work in both for you.  So it's a version of Python.  It gives you 6.py 2 and 6.py 3 just as, you know, constants you can check to see which  one you're in if you need to do things different one or the other.  And then the 6.moves module is sort of aliases.  If you need to import something that moved in the Python standard library between Python  2 and Python 3, you can import it from 6.moves and it'll get it from the correct location  on both pythons.  You're going to want to test your add-ons on both Python 2 and Python 3 to make sure  that they keep working.  This is an area where talks may be helpful.  It'll run your tests in a couple different environments but there's probably a little  bit of work needed to figure out what's the right pattern for doing that with clone add-ons  because talks doesn't really play nice with build out.  And that's maybe an area for some work at the sprint.  All these hints for the Zope component architecture like implements and adapts and provides need  to be updated.  There was a change in Python where the top thing used to work but there's a change to  how classes get loaded that means we have to switch to a decorator form.  You're going to want to just do this everywhere and it works on Python 2 also so it's a pretty  easy change to make.  I mentioned this one, the change from dict methods returning lists to returning iterators.  So there's cases where you might need to make this change.  This is sort of a weird gotcha.  You can't compare instances of different types in Python 3.  So in Python 2 you could create a B tree and you could throw a string key in there and  you could throw some other key in there and it would sort of work.  In Python 3 it would work except for when it doesn't.  So it wasn't necessarily a great idea but in Python 3 it'll explicitly blow up.  So there's, you may run into cases where you were doing that without realizing it and  you have to think more carefully about what you're putting into a B tree.  We have this bad habit of doing open something.read just to get the contents out of a file and  then it stays open until the file gets garbage collected.  If that happens in Python 3 it'll actually show a warning which is not a big deal but  it might get annoying.  So if that happens you can use the context manager form with open and then just work  with the file in that and it'll automatically close it when you leave the context manager.  And save the best for last differences between text and bytes and native string are the ones  that require the most thought because none of the automated tools will do this for you.  The general principles are use text for most of your internal processing, use bytes for  file contents and for input output, use native string if you're like storing the name of  some Python object because Python names are strings in Python 2 and they're Python 3 strings  in Python 3.  But also if you're using the standard library there's some weird exceptions so go read the  docs, figure out what the function that you're using is expecting.  And then we've got these helpers implements.  If you have something that you know is a string but you're not sure if it's like encoded as  bytes or if it's Unicode you can pass it in here and get out text or bytes or native string  if that's what you want in a particular case.  All right.  I think I have time for live demo so we'll try to do this.  Let's see if I can make this bigger.  So here I am, actually let me go to my Python 3 instance and I'm going to copy in.  So I made a site just create a new clone site in Japanese.  I didn't put anything special in it.  I'm going to copy that in here, just the data file.  Okay.  So that's there.  I'm going to run ZODB update, convert pi 3.  I'm going to tell it to use the UDF 8 encoding because I'm doing this with my branch of ZODB  update that works in Python 3 so need to know that and then I'm going to point out the file.  And it'll run along loading those decoder mappings running through the pickles updating  them.  Okay, then I'm going to run my verified DB script just to make sure that all the pickles  can be loaded.  Actually takes longer than doing the conversion.  Okay.  And I'll start things up.  Here we have clone site in Japanese, migrated from Python 2 to Python 3.  Without any content in it.  And in theory I should even be able to like log in and stuff.  Edit a page.  The great thing about clone is that I can use it even in Japanese because things are  color coded.  And that's all I've got.  Happy to take any questions.  Thanks, David.  Fantastic talk.  Question.  Have you ever tried or thought about using the ZODB convert methods of the Red Storage  package because it's also for transferring data between two ZODB instances.  It should be possible that if you use it with one ZO server running on Python 2, one on  Python 3 to get the data transferred directly.  I haven't tried it but I assume it would be possible similar to how the ZRS thing that  I tried would work.  Thank you, David, for the talk and all the work that you did there.  Thanks a lot.  Can you give an example how much, if which kinds of add-ons would need how many items  in their dict to do this, how to call, I can't remember how to call it, in the dictionary  that's referenced in the setup py.  And who would do that, would need to do that.  So are there add-ons that don't need it and which add-ons needed?  So hard to speak from experience here because I haven't really tried doing it but you're  going to want to look specifically at your persistent classes for any attributes that  are Unicode or any attributes that are bytes and add them to the mapping to be explicit.  It was sort of looking at that and realizing like, oh, that's going to be like a lot of  things to find and easy to miss things.  That was sort of why I started looking at the Python 3 version of the conversion and  having a default encoding but neither one is really perfect.  So yeah, it'll be interesting to try it and see what happens.  I was wondering if you have a strong opinion on migration.  I mean, it's clear that we have to, that we need a migration path within like the database  from with CDB update but there's also transmogrifier, right?  Do you think that like when it comes to Python 3, do you have any like strong opinion or  recommendation which path to go or is it the usual trade-off that we had before?  I mean like always the answer is it depends.  I think, yeah, so that's a good point.  I wanted to point out that there are other options if you don't feel like staring at  pickles all day, you can use transmogrifier, you can use point-out-rest API to just get  things out at the application level and put them into a new site.  And for something that is a small to medium-sized site that may well be the most straightforward  way to approach it, especially if you don't have a lot of legacy in the site that you  need to keep around or it's a fairly simple thing.  I think there are going to be bigger sites, sites that have been around for a while and  who knows what's in there where I guess maybe this is the chance to figure out what's in  there.  But yeah, certainly doing a database level conversion at least in the short term is going  to take a lot more like tinkering.  Thanks.  Any other questions?  All right.  So yeah.  Thank you, David.
2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_0_203.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_204_755.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_756_1895.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_1896_3239.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_3240_3851.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_3852_5579.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_5580_6215.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_6216_7175.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_7176_8171.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_8172_9203.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_9204_10799.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_10800_15263.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_15264_20639.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_20640_20915.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_20916_23447.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_23448_33803.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_33804_37331.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_37332_37883.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_37884_38483.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_38484_41567.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_41568_44915.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_44916_46535.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_46536_46727.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_46728_46811.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_46812_46859.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_46860_46907.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_46908_46955.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_46956_47015.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47016_47063.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47064_47111.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47112_47195.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47196_47255.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47256_47351.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47352_47411.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47412_47519.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47520_47579.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47580_47639.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47640_47747.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47748_47795.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47796_47843.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47844_47891.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47892_47975.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_47976_48299.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_48300_48359.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_48360_49187.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_49188_49235.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_49236_49679.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_49680_49799.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_49800_49931.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_49932_50039.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_50040_50111.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_50112_50783.png 2018_Day3_Plone_on_Python_3__How_to_make_the_switc_qKh_3DtJZpk_50784_56183.png
Yeah, so there's a lot of things that you could blame me for, as Kim would say, and get degrees a lot of time. So welcome to my talk on Plone and Python 3, how to make the switch. I'm going to sort of take over where Philip left off yesterday and talk about, he talked about the process of how we got to the release yesterday. I'm going to tell you what I know about how you can take your site on just inversion of Plone Python 2 and get it running on Python 3. So yeah, I'm David Glick. This is a picture. I've developed this habit of going to sprints in Europe and then going to visit Philip afterwards and continuing to sprint. So this was a picture he took just after the Alpine City Sprint in Innsbruck 2017. So yeah, I've been involved for, I think, 2008, was my first conference in D.C. I haven't made it to quite all of them since then, but most of them. I started working for Salesforce a few months ago, so I'm no longer using Plone in my day-to-day job, but in some ways, when I don't use Plone during the day, it feels a little bit more relaxing to do it in the evening. So I've been trying to help Philip finish up the Python 3 work and so forth. Sorry, my throat is a little bit dry. So migrating Python 2 to Python 3. I'm going to talk about... Sorry, my notes aren't showing up here, so I'll have to wing it. I'm going to talk about a little bit of why you would want to do that, what are some of the nice things you'll get in the end. I'm going to talk about how you migrate your database to work in Python 3. And I'm going to talk a little bit about how you would update your code and your add-ons or custom code that you have. So why do you want to be on Python 3? Well, the big one is the stick rather than the carrot, right? It's Python 2 support is ending. You've heard this. The core team is no longer going to support it as of the beginning of 2020. So the time to start working on this is now. And as of a month ago, we have all the tests passing. As of yesterday, we have an alpha release. So things are moving along. We've come a long way from five years ago when we had the Plone 2020 discussion and started scheming how to do this. And there's dozens and dozens of people who have worked on making this happen, so I want to give thanks to all of them. So a big change in Plone 5.2 is that we're now running on Zope 4. And Zope 4 is in some ways not such a big change from Zope 2, even though it's a major version increase. There are probably more changes in between Zope 2.10 and 2.12 than there are from 2.13 to 4, which is the jump we're making here. There are some significant things that got removed in order for us to have less code that needed to be ported to Python 3. So there are a couple of things that are worth mentioning. One is goodnight Z server. If you're running on 5.2, if you're on Python 2, you can still use the Z server, have things running like they were before. But if you're running on Python 3, you're going to be running under whiskey. And we're trying to make this as smooth as possible, so you won't need to change very much of anything. I'll show you one change to make in your build out. But mostly this should be a fairly transparent change. But it's the reason for the major increase in part in the version. Another exciting thing is that we finally have a slightly more modern looking ZMI. It's based on Bootstrap. Still based on DTML, but it looks a little bit more modern. Most of the functionality that was there before is still there. Just moved around a little bit. Control panel link and add menu are up there in the corner now. One thing that's pretty neat is that you can now use Unicode characters in paths, in object IDs. I mean, that's a change that happened in Python for Python 3. And since the Zope graph is a graph of objects, that means we automatically sort of have the ability to do that now. I cheated a little bit for this screenshot because in Plone, we still have normalizers. So you create something, you type the title, and it creates a normalized version of that. So I actually made this and then I renamed it in ZMI. But it should be pretty easy if you want to create a custom normalizer that actually preserves things as they are. Let's talk about some things I like in Python 3, language features. There's way too many to enumerate all of them. So I'll just list a few of my favorites. Chained exceptions. So if you've got a block of code like this where you've got an error in the first statement, it's divide by zero, you can't do that. And then we catch that and then there's another error in the exception handler trying to get some attribute that doesn't exist. In Python 2, you would only see the second exception and you'd look at it and you'd be like, well, where did that come from? In Python 3, it actually shows you, we hit this thing and then while handling it, we hit this other thing. So that's pretty nice. F-strings are a nice new way to format text as if we didn't have enough of those in Python already. But F-strings are cool. So instead of using the percent operator or doing the format method like that where you have to repeat name three times just to get it in there, an F-string, you put an F at the beginning of the string and then you can just put whatever expression you want in curly brackets and it will get evaluated as though it were Python code outside the string. So I haven't actually gotten to use them yet because I'm mostly writing code that is trying to run on both Python 2 and Python 3, but for your projects, if you start moving to Python 3, you might be able to start using them. Pathlib is a neat thing in the standard library for working with file system paths. In Python 2, we would use os.path.join and that would take care of figuring out what the separator character is for your operating system and putting some things together with that. In Python 3, you import path and you create a path instance, defaults to the current directory and then you can actually use the division operator, the slash, so it sort of looks like you're building up a directory path. But when you format it as a string, it will actually format it using the correct separator for your operating system. So neat little thing. And finally, I just wanted to give a shout out to performance and security improvements in Python 3. This is a little bit nebulous to talk about, it's like any performance thing, you got to actually measure it. But just in general, the move to Python 3 means that whatever the core Python team is coming up with in terms of improvements to these areas, we're going to get. So that's nice. So how can we have the nice things? Let's talk about how you upgrade Ploan. This is sort of the classic steps that we're all used to going through, right? You update your build out, rerun it with pulling in the new version pins, then you start up Ploan, you run your upgrade steps. Maybe before that you had to update some of your code like if some imports moved around or something. But if it's a small enough upgrade, you don't even have to do that. This is what the process is going to be like for moving to Python 3, so add a little bit. You update your build out, you're also going to have to add a line to turn on WISG, so that's a small change. You have to convert the database, so all the pickles that are stored that Python 2 knows how to load are not going to necessarily work on Python 3 unless you go through a step to convert it, so I'll talk about that. And then starting it up, running the upgrade steps, that will be pretty much like normal. Updating your code will be necessary, and I'll give some tips about how to do that for Python 3. So when you update your build out, you're going to change the version, so you're extending Ploan 5.2's known good set of versions. You're going to bootstrap your build out using Python 3. I guarantee you'll probably forget that like I have before. And then in the section of the build out that actually installs your Ploan instance, you just put WISG equals on, and that's going to tell it to generate a bin instance script that will run the WISG server instead of running Z server. You might actually be able to get rid of that step, but right now you have to do that. Okay, so converting your database. This is kind of the heart of the talk, the part that is the most still, that we're in the middle of figuring out still. So let's talk about pickles. What's a pickle? It's a delicious fermented food. It's a convenient way of storing objects in your database. We also have this phrase in English. I'm in a bit of a pickle, which sort of means you've gotten yourself into a situation that you aren't quite sure how to get yourself out of. And I think that's a little bit of an apt saying for database migration here. Let me explain why. So I'm going to teach you to be like Jim and read your database at the byte level, just for a little bit if you'll indulge me. This is what we're going to do, and then I'll show you how this looks as a pickle in Python 3 and in Python 2. So we've got some data. It's a list. The list has some bytes. It's got some Unicode, some text. And the third element is a string. So if this is Python 2, then that means it's a byte string. If it's Python 3, that would be a Unicode string. So what we're going to do is we're going to dump that into a pickle and print out the bytes of that pickle. And then we're going to also use pickleTools.dis, which disassembles the pickle and shows you what the different op codes are in the pickle language. So let's do this. I'm actually going to start with Python 3 to show you how it does this. So top part is sort of the raw bytes you get, and then the bottom part is the disassembly. So the first bytes are saying this is prickleProtocol3, which is the new protocol that Python 3 uses. We create an empty list, put it on the stack, and then we add onto the stack the short bin bytes means we're going to put some bytes on there. Remember the bytes. BinUnicodes says we're going to put some Unicode on there, which is what we saw in the code 2. And then the third one also ends up as binUnicode. So Python 3 is interpreting the string as Unicode. So we have these explicit op codes for bytes and Unicode that Python 3 is using. Everything is clear. That's great. Python 2, I'll just hop back and forth here for a little bit, and then I'll point out the differences. So Python 2, where we used to have bin bytes, we now have bin string because string is bytes in Python 2. Okay, that's the first element. Second element is still binUnicode. Unicode is still pickled explicitly as Unicode in Python 2. And then the third one, the ambiguous one, the string becomes also a string. So we have binString for first and the third element. So if you unpickle this in Python 2, like it was pickled, you get bytes, Unicode bytes. If you unpickle this in Python 3, the big question is how does Python 3 deal with this opcode string, binString, that wasn't there before, right? So back here in Python 3, we had bytes and we had Unicode. Everything was explicit. Now we've got these strings. Well, what Python 3 is going to do is it's going to try to turn it into a Python 3 string. Right? So it's going to take the bytes that are stored in the pickle and it's going to decode them to get a Unicode text for Python 3. Now if the... How does it decode it? It decodes it using the default codec, which is Latin 1 in Python, right? So as long as you have non-ASCII characters stored there, that's fine. It gets turned into Unicode and works fine. But if you have something that was stored as UTF-8 encoded string, got pickled as UTF-8 encoded string and then it tries to get loaded by Python 3 as Latin 1, decoded that way, it's going to break. So the result is you've got pickles in your database that can't be loaded without a Unicode decoder. You're kind of stuck. So that's sort of the problem why we need to convert things. But it's not obvious how to convert things because we look at these... Think about creating a tool to walk through the database and update things. It's going to find these bin string opcodes and it doesn't know what it's supposed to become, right? Is it supposed to become... Get decoded and become text? Or is it a binary file that's supposed to stay as bytes and still be treated as bytes in Python 3? So that's sort of the core problem. Sort of an example of a problem you can have with duck typing, right? Duck typing is what we have in Python, right, where you sort of... You don't care what type it is, but as long as it provides the right interface, we know how to work with it. Well, what if you got something that looks like a duck, walks like a duck, but it also looks like a rabbit and walks like a rabbit? You don't know which one it's supposed to become. That's the situation we've got here. And of course, I have to throw in my favorite quote from Trace. Persistence means always having to say, I'm sorry, because that is sort of how I feel about this. So how are we going to deal with this? Well, don't use zodb.py3 migrate, which is one tool that you might run across if you search. It was sort of a start at an attempt to create a tool to do this, but there's just lots of cases it doesn't handle. It didn't quite get far enough, at least at this point. The tool that I have had some success with, and I'll talk about where it succeeds and where it doesn't, is zodb update. This was a tool that's been around for a long time. It was created originally for the use case where you have one class that's stored in your zodb, and then you move the code to a different module, and suddenly you have to update all your pickles to also refer to the new module. So zodb update does that. Sylvan Violant updated it and added a feature to also use it for converting your pickles to Python 3. And apparently they've done that successfully with a big site that wasn't blown. So what does zodb update do? You run it in Python 2, right? So it walks through the database, loads every record in Python 2, you know, so it loads like normal. And then it looks for things that it knows should be binary or things that it knows should be decoded, and it explicitly decodes them or marks them as binary and repickles them, commits a new transaction, voila, you've got your database with Python 3 style pickles. It also writes to the very first bytes at the beginning of your zodb file storage just to say, hey, this is compatible with Python 3 now. How does it pickle something explicitly as bytes in Python 2? Because Python 2 doesn't have a bytes opcode in the pickle protocol. Well, this is what it does. There's in zodb pickle, which is a package, there's a class called binary that's just a simple subclass of bytes. So what this means is that it's going to behave exactly like a bytes. But when it gets pickled, instead of using bin string like normal, Python is actually going to pickle it as a reference to the zodb binary class. So if you load it in Python 2, you get bytes. If you load it in Python 3, you get bytes. So that's sort of the technique that zodb update uses. And where does it use this? It uses it automatically for object IDs. If you've got one record in the zodb that's pointing at the ID of another object, and it uses it automatically for dates and date times, which also get pickled with a byte representation of their value. So what about everything else? You can customize zodb update with entry points. So each package that has a persistent class can do this to say how zodb update should deal with its objects. So you put an entry point in setup.py that says, hey, here's where my mapping is. And then this is what the actual mapping looks like. So this is the example from Zope for the OFS.image module. So it's got the module. It's got a class name file. It's got an attribute name data that says, OK, this should stay binary. And then for the title, on the other hand, it says it should get decoded as UDF8 and become Unicode. So you can sort of go through and do this, maybe a little bit tedious to find all the places in your application that need to have this happen. But it's fairly straightforward to do. But there are some cases that are harder to deal with. Limitations of zodb update are you can't run it in Python 3. So you sort of have to do an initial step of creating your clone 5.2 build out, converting your database, then creating another build out on Python 3 and moving the database over just a little bit of a hassle. Like I said, it only converts the attributes that you tell it about. So if you have something like a custom dexterity type that has something that's supposed to be binary, like you can put that in the mapping, but it's not going to happen automatically. And the really big problem is that zodb update is just working. All it knows about is the pickle that it's looking at one at a time as it goes through the database. So when you've got something like a B tree that has several different persistent records in them, you've got the tree, you've got a sub tree, you've got different buckets, zodb update is going to find one of those buckets and you're going to, it's going to look at it and it's not going to know whether it's a B tree bucket in the catalog or a B tree bucket somewhere else. So it's hard to know, it's hard to target a specific B tree for updates. So we have a few places in Ploen where this is a problem. There's in the catalog, you've got a text index, one of the attributes of a text index is a B tree that stores a mapping of a document ID to this like encoded string of all the words in that document. And that thing is, it's a string in Python 2, it should become actually Unicode in Python 3, but it needs to get decoded using Latin 1 rather than UTF-8. So we sort of need to tell zodb update to deal with it specifically and because it's in a B tree, there's no real good way to do that. So I've done a few experiments over the past couple months trying to like solve some of these edge cases in migrating things and just improve the process a little bit. I'd say we're maybe 70, 80% of the way towards having a good process and there's still some things to figure out. One of the things I did was I made a script which actually got merged early this week into CMF Ploen that lets you run bin instance verifyDB and what that will do is it'll load every record in your database just to make sure that it can do that. So if you've started with the database in Python 2 and you ran zodb update to migrate it to Python 3, then you can run this script and it'll tell you like are there pickles in there that Python 3 won't be able to load because, you know, things are still using the bin string opcode but are actually encoded in some other encoding. So this is a way you can sort of check whether you've added all of the decoder mappings that you need to or if there's other issues. I also have an experiment trying to make zodb update possible to run in Python 3. It's not going to work exactly the same as it does in Python 2 because when you unpickle strings in Python 3 you need to know which encoding to use, right? So I added an option to it. This is in a branch of zodb update right now. I added an option so you can specify the default encoding to use. In Plone we use utf8 in a lot of different places so that's like a pretty good default. So I made it so you can run binzodb update and pass in utf8 and it'll try to decode everything as utf8 and if it fails it'll load it as bytes and just keep it as bytes. So that actually works pretty well for like 95% of the time but it can still have some problems. So you've got something that is supposed to be bytes but you haven't set up a decoder mapping for it and so using the default decoding, if it happens to be bytes that are valid utf8 it'll actually get decoded and stored as unicode when it's not supposed to be unicode. And similarly if you've got something that is supposed to be unicode but decoded in a different encoding than utf8 then that will not end up correct and you'll be in a situation where you can run the verify script and it'll tell you like, oh yeah, sure, I know how to load all those pickles but once it actually gets into the application it'll blow up because it got bytes where it was supposed to have unicode or vice versa. So that doesn't quite solve all the problems either and there's some debate as to whether we should recommend that option or not. Third experiment I did was running ZODB update as part of replication. So if you're using ZRS and you have your site running in Python 2, the idea is you could set up another site in Python 3, set up ZRS so you're replicating from the Python 2 site to the Python 3 site and sort of as pickles get transferred over it would automatically run the ZODB update on them so that you would be able to probably do the migration without as much downtime. I've done this with a sort of, you know, mostly empty site and it sort of worked. I sort of had to hack up ZRS a little bit but it looks like it might be a feasible option if you're using the branch of ZODB update that works in Python 3 and that's sort of the big caveat there. Although I suppose you could also replicate to a Python 2 secondary running ZODB update and then convert it and move the database over to Python 3 after that. So as I said, there's still research underway. We need to sort of figure out how do we deal with the edge cases that don't get handled correctly by ZODB update? And we have like another script or another step that gets run at the start that sort of targets the specific B trees that we know need to be updated and converts them before ZODB update sort of does its thing at the pickle level. Lots of room for trying things out and seeing what works what doesn't. We also really need to figure out like how does this perform? How long does it take to run ZODB update on a real site? I really wanted to try this and have something to report to you at this talk but we just had the release yesterday so I haven't really had a chance and also since I'm not working with Plo and I don't have a really good representative site and also a lot of the sites that we were working with were still had archetypes. So if anybody has a site that is not using archetypes, doesn't have a lot of customizations, might be a really good candidate for trying some of this stuff out in the near future. So wrapping that will wrap up the section on converting your database. That's sort of what I know at this point. I want to talk a little bit about how you update your code for Python 3. Again I can't tell you everything because there's lots of little things, details that you might discover as you go along but I'll at least give some tips. So with the Internet Archetypes, archetypes does not run in Python 3. We haven't spent time trying to make it run. If you need archetypes to run on Python 3, you're welcome to do that but it might be a good opportunity to finally move to dexterity which is what the core team is supporting. There's a tool called 2-3 which is included with Python that you can run on your code. It'll take code for Python 2, convert it to code for Python 3. There's also another tool called Python Modernize that Philip mentioned which sort of does what 2-3 does but also more things. So this is sort of a starting point. You can run that and then look at its changes and see whether they make sense or not. There's some things it does that don't quite make sense like in Python 3 if you've got a dictionary, its methods like items now return an iterator. So 2-3 like hopefully turns it into a list by adding the list around it just to make sure that your any code that was expecting a list will still work. But it does that even in cases where you're clearly just iterating over it and it doesn't really need to do that. So it's not perfect. Use the 6th library to smooth over differences between Python 2 and Python 3 if you're working on an add-on where you're trying to support both. Go read the documentation about the details but some examples of some things it can do. It gives you 6.text type and 6.binary type so if there's places in your code where you were explicitly checking for Unicode or base string or string to see like what type of thing do I have, you can use these instead and it'll work in both for you. So it's a version of Python. It gives you 6.py 2 and 6.py 3 just as, you know, constants you can check to see which one you're in if you need to do things different one or the other. And then the 6.moves module is sort of aliases. If you need to import something that moved in the Python standard library between Python 2 and Python 3, you can import it from 6.moves and it'll get it from the correct location on both pythons. You're going to want to test your add-ons on both Python 2 and Python 3 to make sure that they keep working. This is an area where talks may be helpful. It'll run your tests in a couple different environments but there's probably a little bit of work needed to figure out what's the right pattern for doing that with clone add-ons because talks doesn't really play nice with build out. And that's maybe an area for some work at the sprint. All these hints for the Zope component architecture like implements and adapts and provides need to be updated. There was a change in Python where the top thing used to work but there's a change to how classes get loaded that means we have to switch to a decorator form. You're going to want to just do this everywhere and it works on Python 2 also so it's a pretty easy change to make. I mentioned this one, the change from dict methods returning lists to returning iterators. So there's cases where you might need to make this change. This is sort of a weird gotcha. You can't compare instances of different types in Python 3. So in Python 2 you could create a B tree and you could throw a string key in there and you could throw some other key in there and it would sort of work. In Python 3 it would work except for when it doesn't. So it wasn't necessarily a great idea but in Python 3 it'll explicitly blow up. So there's, you may run into cases where you were doing that without realizing it and you have to think more carefully about what you're putting into a B tree. We have this bad habit of doing open something.read just to get the contents out of a file and then it stays open until the file gets garbage collected. If that happens in Python 3 it'll actually show a warning which is not a big deal but it might get annoying. So if that happens you can use the context manager form with open and then just work with the file in that and it'll automatically close it when you leave the context manager. And save the best for last differences between text and bytes and native string are the ones that require the most thought because none of the automated tools will do this for you. The general principles are use text for most of your internal processing, use bytes for file contents and for input output, use native string if you're like storing the name of some Python object because Python names are strings in Python 2 and they're Python 3 strings in Python 3. But also if you're using the standard library there's some weird exceptions so go read the docs, figure out what the function that you're using is expecting. And then we've got these helpers implements. If you have something that you know is a string but you're not sure if it's like encoded as bytes or if it's Unicode you can pass it in here and get out text or bytes or native string if that's what you want in a particular case. All right. I think I have time for live demo so we'll try to do this. Let's see if I can make this bigger. So here I am, actually let me go to my Python 3 instance and I'm going to copy in. So I made a site just create a new clone site in Japanese. I didn't put anything special in it. I'm going to copy that in here, just the data file. Okay. So that's there. I'm going to run ZODB update, convert pi 3. I'm going to tell it to use the UDF 8 encoding because I'm doing this with my branch of ZODB update that works in Python 3 so need to know that and then I'm going to point out the file. And it'll run along loading those decoder mappings running through the pickles updating them. Okay, then I'm going to run my verified DB script just to make sure that all the pickles can be loaded. Actually takes longer than doing the conversion. Okay. And I'll start things up. Here we have clone site in Japanese, migrated from Python 2 to Python 3. Without any content in it. And in theory I should even be able to like log in and stuff. Edit a page. The great thing about clone is that I can use it even in Japanese because things are color coded. And that's all I've got. Happy to take any questions. Thanks, David. Fantastic talk. Question. Have you ever tried or thought about using the ZODB convert methods of the Red Storage package because it's also for transferring data between two ZODB instances. It should be possible that if you use it with one ZO server running on Python 2, one on Python 3 to get the data transferred directly. I haven't tried it but I assume it would be possible similar to how the ZRS thing that I tried would work. Thank you, David, for the talk and all the work that you did there. Thanks a lot. Can you give an example how much, if which kinds of add-ons would need how many items in their dict to do this, how to call, I can't remember how to call it, in the dictionary that's referenced in the setup py. And who would do that, would need to do that. So are there add-ons that don't need it and which add-ons needed? So hard to speak from experience here because I haven't really tried doing it but you're going to want to look specifically at your persistent classes for any attributes that are Unicode or any attributes that are bytes and add them to the mapping to be explicit. It was sort of looking at that and realizing like, oh, that's going to be like a lot of things to find and easy to miss things. That was sort of why I started looking at the Python 3 version of the conversion and having a default encoding but neither one is really perfect. So yeah, it'll be interesting to try it and see what happens. I was wondering if you have a strong opinion on migration. I mean, it's clear that we have to, that we need a migration path within like the database from with CDB update but there's also transmogrifier, right? Do you think that like when it comes to Python 3, do you have any like strong opinion or recommendation which path to go or is it the usual trade-off that we had before? I mean like always the answer is it depends. I think, yeah, so that's a good point. I wanted to point out that there are other options if you don't feel like staring at pickles all day, you can use transmogrifier, you can use point-out-rest API to just get things out at the application level and put them into a new site. And for something that is a small to medium-sized site that may well be the most straightforward way to approach it, especially if you don't have a lot of legacy in the site that you need to keep around or it's a fairly simple thing. I think there are going to be bigger sites, sites that have been around for a while and who knows what's in there where I guess maybe this is the chance to figure out what's in there. But yeah, certainly doing a database level conversion at least in the short term is going to take a lot more like tinkering. Thanks. Any other questions? All right. So yeah. Thank you, David.
Python 3, how to make the switch. I'm going to migrate your database to work in Python 3. And Plone 5.2 is that we're now running on Zope 4. whatever the core Python team is coming up with in terms of improvements to these areas, we're starting it up, running the upgrade steps, that string. So we have binString for first and the like how does this perform? How long does it take you need archetypes to run on Python 3, you're will still work. But it does that even in cases the most straightforward way to approach it,