Data internationalization in Django
Michel, Raphael
DjangoCon US 2018 - Data internationalization in Django by Raphel Michel There is a multitude of options to translate database data in Django, for example django-parler, django-modeltranslation, django-nece, django-hvad, and django-i18nfield (which is my own). The interesting thing is that these libraries are not multiple implementations of the same thing, but they are all radically different in their design and there are good reasons for every one of them. The sometimes subtle differences might not be obvious to a beginner in the Django world. This talk will help them navigate through different solutions and make an informed decision.
 Hello, good afternoon from my side as well. My name is Raphael, I'm a software developer from Heidelberg in Germany and I've been the co-chair of DjangoCon Europe this year and as this conference is coming to close shortly, I will want to take the opportunity to thank the organizers for putting up this conference because once you've done something like this, you get to really appreciate not having to do it and just sitting here and enjoy the wonderful conference that they created for us. Thank you so much for bringing me here to see this. Now let's get to the topic I want to be talking about which is data internationalization in Django and before we do that, let's recap shortly what internationalization in Django without the data could mean and we've had a talk, I think yesterday or the day before on the subject before, Django has a set of features that allows you to translate your application into other languages. This means that you can use certain functions to mark strings in your code or in your templates to say, okay, this is a string that is different when my application is not run in English or not used in English but in another language and then you can use the GetTex toolkit which is like the standard translation toolkit in the whole Unix world to extract all of those strings, create a file with all the translations, send it out to translators, collect the translations again, compile it and use it and that's great but sometimes it's not enough. One of the open source projects I'm maintaining is Pre-Tix which is an open source ticket shop for events just like this one and it allows you to create a shop that speaks to a multilingual audience. So if you have attendees coming from different countries speaking different languages, you might want to present your shop in multiple languages at the same time and that means it's not sufficient if only the application is translated, you need to translate data as well and by data I mean things that are entered by the administrator for the shop, for example, the names of the products or the description on how to get to the event and so on. That's all something that needs to be entered in multiple languages and then in the output must be shown in the language of the respective user. So we need forms or some kind of input method that allows us to store data in multiple languages. So it's not really suitable to use getText for that because we would need to, like every time someone changed something we would need to generate such files and the translators and so on but that just doesn't work and so we cannot really use the tools provided by Django. Surely there is a third party library that we just need to install and that will solve this problem for us. I've got good news and bad news for you. The good news is there is such a library. The bad news is I counted 23 of them until I stopped. So who in this room has ever used one of these libraries? Okay, that's not that much. Is there someone here who has written one of these libraries? Okay, that was more people at Django Con Europe last year. So disclaimer, I'm the author of Django I18N field. I put those seven at the top who appear to be actively maintained. By that I mean they at least have a development branch that is compatible to Django 2.1 and they had commits like in the last 6 to 12 months. So I will be focusing on giving you a short brief overview over those top seven libraries because the problem is there is not a single best library for this use case because they are fundamentally different in their approach and fundamentally different appropriate for the different use cases. If I get something wrong about one of these libraries, please feel free to correct me later on Slack, I haven't used most of them actually in an actual product but I've played around with all of them. So to compare these we want to look at different categories. We want to look at how the data is stored in the database. We want to know how their Python API looks and how easy it is to work with them. What other features they might provide, for example integration with Django admin, integration with forms and so on. And we are interested in if they have a significant performance impact and how large that is. So to have an example to work with, let's use a model where we can store a list of movies and for every movie we want to store the title of the movie and the year the movie was released. Finally the year is something that is not really local dependent although it might be but the title is certainly something that is different everywhere in the world even though it's the same movie. So let's look at how different libraries try to represent that in a database. And the first approach that we see for example in Django well in Django parlay is to have a separate table that contains the translated strings. So in our main table movies we have just the untranslated attributes with the idea of the movie and the year and then we have a second table where every row references an object in the main table and then says okay for English this is the name of the movie and for Italian this is the name of the movie. So this is in terms of relational databases this is a very clean approach. It kind of fits the normalize until it hurts that we learned yesterday. And if we are interested to like build our shop front and we want to have a movie list and we want to have list of movies and we want to have the Italian movie, Italian title for every movie then this is also very efficient because modern databases are very good at performing joins. However, for example in the back end where we want to have a list of the movies where we want to show every language per movie this gets really expensive because we need to do either a lot of queries or we need to work with the query data a lot. A separate approach seen in Django model translation or Django translated fields is to just have separate columns per language. This way you don't need any joins and it's very cheap to get all languages at the same time. However, every time you add a new language you need to do a database migration which can be very annoying. And the third style that we see used in Django 18 end field Django model trans is to use a JSON like field. This is less clean in terms of database normalization but we don't need to do any joins. We don't need to do any changes to our schema when we add languages and it's all contained in one field as we had it before. If we're on Postgres and if we use Postgres native JSON data type we can still retain the functionality of filtering by or searching the name in a specific language or sorting by name. If we're not on Postgres we kind of lose the functionality to index or query that data. That might be a problem or might be totally fine for your use case. The model trans is a bit different than the other two. It uses not one JSON column per translated column but only one JSON column for the whole table no matter how many fields you translate. But apart from that there's a pretty similar niche and model trans only work in Postgres. The 18 end field drops the indexing and filtering possibilities but works on all databases. I'll be working at the sprints on something that uses the Postgres data type when you are on Postgres and gracefully falls back to your text field on all other databases. Next we want to look at how you define your models. There are a couple of different styles. For example in Django well, Django palet and Django Nechi you have a custom base class that you inherit your model from and they will change your query manager and change a lot of things and how your model works to as automatically as possible build those joins for you or translate your queries for you. Sometimes you need to wrap your fields in some wrapper object to tell the library which once you translate, sometimes you have an additional meta option but in the end it's the same thing. The other style is that you have a custom field type and do not change the way the model in itself works at all. For example in Django I18n field or Django translated fields you just have a custom type that is a translated character field or translated text field. Whereas in Django model trends you have per model you have one field that is called I18n or whatever you want to call it that stores the translation for all other fields. The third style is to decouple it from the model definition process completely and have like a separate registry where you register those options. This is like the most, I would call it the most unclean style of doing it because it's kind of not obvious where your code lives. On the other hand this allows you to enable translations for models that are not in the code that you control which might be something you need. And the third thing I want to look at in detail is how to interact with your model objects. In some of the libraries you can only interact with one language at a time. This is mostly because of this joins that they are performing. If you pull the object from the database it will pull the information for one language like you need to specify that within your query and then the title attribute of the object will be populated with the Italian title. And then if you want to access the English title you need to change the language and it will perform a new query or depending on the implementation or in the case of Nietzsche it will not but it will change like the state and title will now contain the English title. The other option is to be able to access all properties at once. This comes naturally to the libraries where you have separate columns per language because you have your main attribute title that usually evaluates lazily to the value of the currently active locale and you can directly access every other language by just using title underscore and the language code because that's just either because that's just the field that the library creates or because it virtually creates it for you. In Django I18 end field it's a bit different. The title attribute will always contain a special data type, lazy internationalized string which is in some ways like what you get in return from yougettext underscore lazy. It will, whenever you cast it to a string it will coerce to the currently active locale but it is a special data object that contains the information on all language. So you can pass that around as one object. So to recap and to add the other features we have a couple of different database layouts that are in use. We have the version where we have everything in its own table like we have multiple tables to store our model. We have the version where we have a very wide table with multiple columns for every language that we have and we have the embedded version within one column. We have database support for most of the libraries for all databases that Django supports but in the case of Django NG and Django model trends they only run on PostgreSQL. We have different levels of support for filtering the objects for example in those that use normalize database layout it is really easy although it might be computationally expensive. Whereas in those that use the PostgreSQL JSON field it should be easy. They don't provide any utilities for you to make it even easier like querying in the currently default language. You need to do that on your own but it is conceptually possible whereas in IIT in field it is currently not really possible. Like searching somehow works but ordering or indexing is not possible. We have the separate styles of defining the model either by defining a base class or by registration or by custom field type and we have the separate styles of object operation where you can either access one language at a time or all of the language at once. I didn't talk in detail about form support, some of them provide, for most of them form support comes naturally by just like when they generate your model fields that have a separate column per language then it will just when you use a model form it will just automatically generate that number of fields. So in some of them you will get in a model form you will get a field that just allows you to edit the currently active language which I don't think is useful in very many use cases. In some you will get different form fields for each one form field per language. NHA doesn't have form support at all it just gives you a text widget where you can edit the JSON block and in IIT in end field you will get a special form field type with a special widget that works like the compound datum widget that just has compound input fields within one widget to ask for the different languages. Some of them have very elaborate support for the Django admin and do that very nicely. Others just present you with the JSON block or just different fields below. I did a small benchmark to have a look at their performance the benchmark is of course not representative for real life application it just stores a lot of objects into the database pulls them out again and tries to access the attributes in various languages. That obviously is rather slow on those that need to do joints and need to refetch although at least Django can make use of caching to reduce this while the others quite unsurprisingly the post-resql JSON field back implementations are very, very fast and the others also reasonably fast. I've created a demo app that uses all of the seven libraries and also contains the benchmark code in case you're interested in that and with that I'm a bit faster than I expected at the end of my talk and I would be happy if you have any questions on that subject. That was a great talk thank you. Just one question where did you get the inspiration to do the emoji feature comparison? I don't remember I've seen a lot of emojis at the Django conferences I've attended so it's maybe Katie's fault but. Hi, great thanks for the presentation. Right to left support how easy is that to implement and how do the libraries succeed? In context of storing the data I haven't tried I don't see any problems ahead there because in the end those libraries are just about how to store these Unicode strings that uses input and that we output at the later stage. So I think on this on the model level it's not that important it might get more interesting if you have like if you're rendering a form and you need to render some of the widgets with left to right CSS and others with right to left CSS that might get interesting but on the database level I don't think it should be a problem. Let's give a big round to Rafael.
 Hello, good afternoon from my side as well.  My name is Raphael, I'm a software developer from Heidelberg in Germany and I've been the  co-chair of DjangoCon Europe this year and as this conference is coming to close shortly,  I will want to take the opportunity to thank the organizers for putting up this conference  because once you've done something like this, you get to really appreciate not having to  do it and just sitting here and enjoy the wonderful conference that they created for  us.  Thank you so much for bringing me here to see this.  Now let's get to the topic I want to be talking about which is data internationalization in  Django and before we do that, let's recap shortly what internationalization in Django  without the data could mean and we've had a talk, I think yesterday or the day before  on the subject before, Django has a set of features that allows you to translate your  application into other languages.  This means that you can use certain functions to mark strings in your code or in your templates  to say, okay, this is a string that is different when my application is not run in English or  not used in English but in another language and then you can use the GetTex toolkit which  is like the standard translation toolkit in the whole Unix world to extract all of those  strings, create a file with all the translations, send it out to translators, collect the translations  again, compile it and use it and that's great but sometimes it's not enough.  One of the open source projects I'm maintaining is Pre-Tix which is an open source ticket  shop for events just like this one and it allows you to create a shop that speaks to  a multilingual audience.  So if you have attendees coming from different countries speaking different languages, you  might want to present your shop in multiple languages at the same time and that means  it's not sufficient if only the application is translated, you need to translate data  as well and by data I mean things that are entered by the administrator for the shop,  for example, the names of the products or the description on how to get to the event  and so on.  That's all something that needs to be entered in multiple languages and then in the output  must be shown in the language of the respective user.  So we need forms or some kind of input method that allows us to store data in multiple languages.  So it's not really suitable to use getText for that because we would need to, like every  time someone changed something we would need to generate such files and the translators  and so on but that just doesn't work and so we cannot really use the tools provided  by Django.  Surely there is a third party library that we just need to install and that will solve  this problem for us.  I've got good news and bad news for you.  The good news is there is such a library.  The bad news is I counted 23 of them until I stopped.  So who in this room has ever used one of these libraries?  Okay, that's not that much.  Is there someone here who has written one of these libraries?  Okay, that was more people at Django Con Europe last year.  So disclaimer, I'm the author of Django I18N field.  I put those seven at the top who appear to be actively maintained.  By that I mean they at least have a development branch that is compatible to Django 2.1 and  they had commits like in the last 6 to 12 months.  So I will be focusing on giving you a short brief overview over those top seven libraries  because the problem is there is not a single best library for this use case because they  are fundamentally different in their approach and fundamentally different appropriate for  the different use cases.  If I get something wrong about one of these libraries, please feel free to correct me later  on Slack, I haven't used most of them actually in an actual product but I've played around  with all of them.  So to compare these we want to look at different categories.  We want to look at how the data is stored in the database.  We want to know how their Python API looks and how easy it is to work with them.  What other features they might provide, for example integration with Django admin, integration  with forms and so on.  And we are interested in if they have a significant performance impact and how large that is.  So to have an example to work with, let's use a model where we can store a list of movies  and for every movie we want to store the title of the movie and the year the movie was released.  Finally the year is something that is not really local dependent although it might be  but the title is certainly something that is different everywhere in the world even  though it's the same movie.  So let's look at how different libraries try to represent that in a database.  And the first approach that we see for example in Django well in Django parlay is to have  a separate table that contains the translated strings.  So in our main table movies we have just the untranslated attributes with the idea of the  movie and the year and then we have a second table where every row references an object  in the main table and then says okay for English this is the name of the movie and for Italian  this is the name of the movie.  So this is in terms of relational databases this is a very clean approach.  It kind of fits the normalize until it hurts that we learned yesterday.  And if we are interested to like build our shop front and we want to have a movie list  and we want to have list of movies and we want to have the Italian movie, Italian title  for every movie then this is also very efficient because modern databases are very good at  performing joins.  However, for example in the back end where we want to have a list of the movies where  we want to show every language per movie this gets really expensive because we need to do  either a lot of queries or we need to work with the query data a lot.  A separate approach seen in Django model translation or Django translated fields is to just have  separate columns per language.  This way you don't need any joins and it's very cheap to get all languages at the same  time.  However, every time you add a new language you need to do a database migration which  can be very annoying.  And the third style that we see used in Django 18 end field Django model trans is to use  a JSON like field.  This is less clean in terms of database normalization but we don't need to do any joins.  We don't need to do any changes to our schema when we add languages and it's all contained  in one field as we had it before.  If we're on Postgres and if we use Postgres native JSON data type we can still retain  the functionality of filtering by or searching the name in a specific language or sorting  by name.  If we're not on Postgres we kind of lose the functionality to index or query that data.  That might be a problem or might be totally fine for your use case.  The model trans is a bit different than the other two.  It uses not one JSON column per translated column but only one JSON column for the whole  table no matter how many fields you translate.  But apart from that there's a pretty similar niche and model trans only work in Postgres.  The 18 end field drops the indexing and filtering possibilities but works on all databases.  I'll be working at the sprints on something that uses the Postgres data type when you  are on Postgres and gracefully falls back to your text field on all other databases.  Next we want to look at how you define your models.  There are a couple of different styles.  For example in Django well, Django palet and Django Nechi you have a custom base class that  you inherit your model from and they will change your query manager and change a lot  of things and how your model works to as automatically as possible build those joins for you or translate  your queries for you.  Sometimes you need to wrap your fields in some wrapper object to tell the library which once  you translate, sometimes you have an additional meta option but in the end it's the same thing.  The other style is that you have a custom field type and do not change the way the model  in itself works at all.  For example in Django I18n field or Django translated fields you just have a custom type  that is a translated character field or translated text field.  Whereas in Django model trends you have per model you have one field that is called I18n  or whatever you want to call it that stores the translation for all other fields.  The third style is to decouple it from the model definition process completely and have  like a separate registry where you register those options.  This is like the most, I would call it the most unclean style of doing it because it's  kind of not obvious where your code lives.  On the other hand this allows you to enable translations for models that are not in the  code that you control which might be something you need.  And the third thing I want to look at in detail is how to interact with your model objects.  In some of the libraries you can only interact with one language at a time.  This is mostly because of this joins that they are performing.  If you pull the object from the database it will pull the information for one language  like you need to specify that within your query and then the title attribute of the  object will be populated with the Italian title.  And then if you want to access the English title you need to change the language and  it will perform a new query or depending on the implementation or in the case of Nietzsche  it will not but it will change like the state and title will now contain the English title.  The other option is to be able to access all properties at once.  This comes naturally to the libraries where you have separate columns per language because  you have your main attribute title that usually evaluates lazily to the value of the currently  active locale and you can directly access every other language by just using title underscore  and the language code because that's just either because that's just the field that  the library creates or because it virtually creates it for you.  In Django I18 end field it's a bit different.  The title attribute will always contain a special data type, lazy internationalized string  which is in some ways like what you get in return from yougettext underscore lazy.  It will, whenever you cast it to a string it will coerce to the currently active locale  but it is a special data object that contains the information on all language.  So you can pass that around as one object.  So to recap and to add the other features we have a couple of different database layouts  that are in use.  We have the version where we have everything in its own table like we have multiple tables  to store our model.  We have the version where we have a very wide table with multiple columns for every language  that we have and we have the embedded version within one column.  We have database support for most of the libraries for all databases that Django supports  but in the case of Django NG and Django model trends they only run on PostgreSQL.  We have different levels of support for filtering the objects for example in those that use  normalize database layout it is really easy although it might be computationally expensive.  Whereas in those that use the PostgreSQL JSON field it should be easy.  They don't provide any utilities for you to make it even easier like querying in the currently  default language.  You need to do that on your own but it is conceptually possible whereas in IIT in field  it is currently not really possible.  Like searching somehow works but ordering or indexing is not possible.  We have the separate styles of defining the model either by defining a base class or by  registration or by custom field type and we have the separate styles of object operation  where you can either access one language at a time or all of the language at once.  I didn't talk in detail about form support, some of them provide, for most of them form  support comes naturally by just like when they generate your model fields that have  a separate column per language then it will just when you use a model form it will just  automatically generate that number of fields.  So in some of them you will get in a model form you will get a field that just allows  you to edit the currently active language which I don't think is useful in very many  use cases.  In some you will get different form fields for each one form field per language.  NHA doesn't have form support at all it just gives you a text widget where you can edit  the JSON block and in IIT in end field you will get a special form field type with a  special widget that works like the compound datum widget that just has compound input  fields within one widget to ask for the different languages.  Some of them have very elaborate support for the Django admin and do that very nicely.  Others just present you with the JSON block or just different fields below.  I did a small benchmark to have a look at their performance the benchmark is of course  not representative for real life application it just stores a lot of objects into the database  pulls them out again and tries to access the attributes in various languages.  That obviously is rather slow on those that need to do joints and need to refetch although  at least Django can make use of caching to reduce this while the others quite unsurprisingly  the post-resql JSON field back implementations are very, very fast and the others also reasonably  fast.  I've created a demo app that uses all of the seven libraries and also contains the benchmark  code in case you're interested in that and with that I'm a bit faster than I expected  at the end of my talk and I would be happy if you have any questions on that subject.  That was a great talk thank you.  Just one question where did you get the inspiration to do the emoji feature comparison?  I don't remember I've seen a lot of emojis at the Django conferences I've attended so  it's maybe Katie's fault but.  Hi, great thanks for the presentation.  Right to left support how easy is that to implement and how do the libraries succeed?  In context of storing the data I haven't tried I don't see any problems ahead there because  in the end those libraries are just about how to store these Unicode strings that uses  input and that we output at the later stage.  So I think on this on the model level it's not that important it might get more interesting  if you have like if you're rendering a form and you need to render some of the widgets  with left to right CSS and others with right to left CSS that might get interesting but  on the database level I don't think it should be a problem.  Let's give a big round to Rafael.
DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_0_71.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_72_119.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_120_251.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_252_335.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_336_407.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_408_2003.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_2004_2999.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_3000_4259.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_4260_4955.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_4956_5255.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_5256_7151.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_7152_8051.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_8052_8867.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_8868_10715.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_10716_13703.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_13704_13907.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_13908_14855.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_14856_15695.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_15696_16715.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_16716_16943.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_16944_20207.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_20208_20903.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_20904_26339.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_26340_26987.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_26988_27479.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_27480_28367.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_28368_28511.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_28512_28559.png DjangoCon_US_2018_-_Data_internationalization_in_Django_by_Raphel_Michel_4WEMXydnQs8_28560_28627.png
Hello, good afternoon from my side as well. My name is Raphael, I'm a software developer from Heidelberg in Germany and I've been the co-chair of DjangoCon Europe this year and as this conference is coming to close shortly, I will want to take the opportunity to thank the organizers for putting up this conference because once you've done something like this, you get to really appreciate not having to do it and just sitting here and enjoy the wonderful conference that they created for us. Thank you so much for bringing me here to see this. Now let's get to the topic I want to be talking about which is data internationalization in Django and before we do that, let's recap shortly what internationalization in Django without the data could mean and we've had a talk, I think yesterday or the day before on the subject before, Django has a set of features that allows you to translate your application into other languages. This means that you can use certain functions to mark strings in your code or in your templates to say, okay, this is a string that is different when my application is not run in English or not used in English but in another language and then you can use the GetTex toolkit which is like the standard translation toolkit in the whole Unix world to extract all of those strings, create a file with all the translations, send it out to translators, collect the translations again, compile it and use it and that's great but sometimes it's not enough. One of the open source projects I'm maintaining is Pre-Tix which is an open source ticket shop for events just like this one and it allows you to create a shop that speaks to a multilingual audience. So if you have attendees coming from different countries speaking different languages, you might want to present your shop in multiple languages at the same time and that means it's not sufficient if only the application is translated, you need to translate data as well and by data I mean things that are entered by the administrator for the shop, for example, the names of the products or the description on how to get to the event and so on. That's all something that needs to be entered in multiple languages and then in the output must be shown in the language of the respective user. So we need forms or some kind of input method that allows us to store data in multiple languages. So it's not really suitable to use getText for that because we would need to, like every time someone changed something we would need to generate such files and the translators and so on but that just doesn't work and so we cannot really use the tools provided by Django. Surely there is a third party library that we just need to install and that will solve this problem for us. I've got good news and bad news for you. The good news is there is such a library. The bad news is I counted 23 of them until I stopped. So who in this room has ever used one of these libraries? Okay, that's not that much. Is there someone here who has written one of these libraries? Okay, that was more people at Django Con Europe last year. So disclaimer, I'm the author of Django I18N field. I put those seven at the top who appear to be actively maintained. By that I mean they at least have a development branch that is compatible to Django 2.1 and they had commits like in the last 6 to 12 months. So I will be focusing on giving you a short brief overview over those top seven libraries because the problem is there is not a single best library for this use case because they are fundamentally different in their approach and fundamentally different appropriate for the different use cases. If I get something wrong about one of these libraries, please feel free to correct me later on Slack, I haven't used most of them actually in an actual product but I've played around with all of them. So to compare these we want to look at different categories. We want to look at how the data is stored in the database. We want to know how their Python API looks and how easy it is to work with them. What other features they might provide, for example integration with Django admin, integration with forms and so on. And we are interested in if they have a significant performance impact and how large that is. So to have an example to work with, let's use a model where we can store a list of movies and for every movie we want to store the title of the movie and the year the movie was released. Finally the year is something that is not really local dependent although it might be but the title is certainly something that is different everywhere in the world even though it's the same movie. So let's look at how different libraries try to represent that in a database. And the first approach that we see for example in Django well in Django parlay is to have a separate table that contains the translated strings. So in our main table movies we have just the untranslated attributes with the idea of the movie and the year and then we have a second table where every row references an object in the main table and then says okay for English this is the name of the movie and for Italian this is the name of the movie. So this is in terms of relational databases this is a very clean approach. It kind of fits the normalize until it hurts that we learned yesterday. And if we are interested to like build our shop front and we want to have a movie list and we want to have list of movies and we want to have the Italian movie, Italian title for every movie then this is also very efficient because modern databases are very good at performing joins. However, for example in the back end where we want to have a list of the movies where we want to show every language per movie this gets really expensive because we need to do either a lot of queries or we need to work with the query data a lot. A separate approach seen in Django model translation or Django translated fields is to just have separate columns per language. This way you don't need any joins and it's very cheap to get all languages at the same time. However, every time you add a new language you need to do a database migration which can be very annoying. And the third style that we see used in Django 18 end field Django model trans is to use a JSON like field. This is less clean in terms of database normalization but we don't need to do any joins. We don't need to do any changes to our schema when we add languages and it's all contained in one field as we had it before. If we're on Postgres and if we use Postgres native JSON data type we can still retain the functionality of filtering by or searching the name in a specific language or sorting by name. If we're not on Postgres we kind of lose the functionality to index or query that data. That might be a problem or might be totally fine for your use case. The model trans is a bit different than the other two. It uses not one JSON column per translated column but only one JSON column for the whole table no matter how many fields you translate. But apart from that there's a pretty similar niche and model trans only work in Postgres. The 18 end field drops the indexing and filtering possibilities but works on all databases. I'll be working at the sprints on something that uses the Postgres data type when you are on Postgres and gracefully falls back to your text field on all other databases. Next we want to look at how you define your models. There are a couple of different styles. For example in Django well, Django palet and Django Nechi you have a custom base class that you inherit your model from and they will change your query manager and change a lot of things and how your model works to as automatically as possible build those joins for you or translate your queries for you. Sometimes you need to wrap your fields in some wrapper object to tell the library which once you translate, sometimes you have an additional meta option but in the end it's the same thing. The other style is that you have a custom field type and do not change the way the model in itself works at all. For example in Django I18n field or Django translated fields you just have a custom type that is a translated character field or translated text field. Whereas in Django model trends you have per model you have one field that is called I18n or whatever you want to call it that stores the translation for all other fields. The third style is to decouple it from the model definition process completely and have like a separate registry where you register those options. This is like the most, I would call it the most unclean style of doing it because it's kind of not obvious where your code lives. On the other hand this allows you to enable translations for models that are not in the code that you control which might be something you need. And the third thing I want to look at in detail is how to interact with your model objects. In some of the libraries you can only interact with one language at a time. This is mostly because of this joins that they are performing. If you pull the object from the database it will pull the information for one language like you need to specify that within your query and then the title attribute of the object will be populated with the Italian title. And then if you want to access the English title you need to change the language and it will perform a new query or depending on the implementation or in the case of Nietzsche it will not but it will change like the state and title will now contain the English title. The other option is to be able to access all properties at once. This comes naturally to the libraries where you have separate columns per language because you have your main attribute title that usually evaluates lazily to the value of the currently active locale and you can directly access every other language by just using title underscore and the language code because that's just either because that's just the field that the library creates or because it virtually creates it for you. In Django I18 end field it's a bit different. The title attribute will always contain a special data type, lazy internationalized string which is in some ways like what you get in return from yougettext underscore lazy. It will, whenever you cast it to a string it will coerce to the currently active locale but it is a special data object that contains the information on all language. So you can pass that around as one object. So to recap and to add the other features we have a couple of different database layouts that are in use. We have the version where we have everything in its own table like we have multiple tables to store our model. We have the version where we have a very wide table with multiple columns for every language that we have and we have the embedded version within one column. We have database support for most of the libraries for all databases that Django supports but in the case of Django NG and Django model trends they only run on PostgreSQL. We have different levels of support for filtering the objects for example in those that use normalize database layout it is really easy although it might be computationally expensive. Whereas in those that use the PostgreSQL JSON field it should be easy. They don't provide any utilities for you to make it even easier like querying in the currently default language. You need to do that on your own but it is conceptually possible whereas in IIT in field it is currently not really possible. Like searching somehow works but ordering or indexing is not possible. We have the separate styles of defining the model either by defining a base class or by registration or by custom field type and we have the separate styles of object operation where you can either access one language at a time or all of the language at once. I didn't talk in detail about form support, some of them provide, for most of them form support comes naturally by just like when they generate your model fields that have a separate column per language then it will just when you use a model form it will just automatically generate that number of fields. So in some of them you will get in a model form you will get a field that just allows you to edit the currently active language which I don't think is useful in very many use cases. In some you will get different form fields for each one form field per language. NHA doesn't have form support at all it just gives you a text widget where you can edit the JSON block and in IIT in end field you will get a special form field type with a special widget that works like the compound datum widget that just has compound input fields within one widget to ask for the different languages. Some of them have very elaborate support for the Django admin and do that very nicely. Others just present you with the JSON block or just different fields below. I did a small benchmark to have a look at their performance the benchmark is of course not representative for real life application it just stores a lot of objects into the database pulls them out again and tries to access the attributes in various languages. That obviously is rather slow on those that need to do joints and need to refetch although at least Django can make use of caching to reduce this while the others quite unsurprisingly the post-resql JSON field back implementations are very, very fast and the others also reasonably fast. I've created a demo app that uses all of the seven libraries and also contains the benchmark code in case you're interested in that and with that I'm a bit faster than I expected at the end of my talk and I would be happy if you have any questions on that subject. That was a great talk thank you. Just one question where did you get the inspiration to do the emoji feature comparison? I don't remember I've seen a lot of emojis at the Django conferences I've attended so it's maybe Katie's fault but. Hi, great thanks for the presentation. Right to left support how easy is that to implement and how do the libraries succeed? In context of storing the data I haven't tried I don't see any problems ahead there because in the end those libraries are just about how to store these Unicode strings that uses input and that we output at the later stage. So I think on this on the model level it's not that important it might get more interesting if you have like if you're rendering a form and you need to render some of the widgets with left to right CSS and others with right to left CSS that might get interesting but on the database level I don't think it should be a problem. Let's give a big round to Rafael.
internationalization in Django without the data something that needs to be entered in multiple tools provided by Django. Surely there is a third who has written one of these libraries? Okay, different in their approach and fundamentally features they might provide, for example Italian movie, Italian title for every movie then once you translate, sometimes you have an same thing. The other style is that you have a are not in the code that you control which might support for most of the libraries for all some of them provide, for most of them form might get interesting but on the database level I