TransCoder: Unsupervised Translation of Programming Languages (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper
Code migration between languages is an expensive and laborious task. To translate from one language to the other, one needs to be an expert at both. Current automatic tools often produce illegible and complicated code. This paper applies unsupervised neural machine translation to source code of Python, C++, and Java and is able to translate between them, without ever being trained in a supervised fashion.

OUTLINE:
0:00 - Intro & Overview
1:15 - The Transcompiling Problem
5:55 - Neural Machine Translation
8:45 - Unsupervised NMT
12:55 - Shared Embeddings via Token Overlap
20:45 - MLM Objective
25:30 - Denoising Objective
30:10 - Back-Translation Objective
33:00 - Evaluation Dataset
37:25 - Results
41:45 - Tokenization
42:40 - Shared Embeddings
43:30 - Human-Aware Translation
47:25 - Failure Cases
48:05 - Conclusion

Paper: https://arxiv.org/abs/2006.03511

Abstract:
A transcompiler, also known as source-to-source translator, is a system that converts source code from a high-level programming language (such as C++ or Python) to another. Transcompilers are primarily used for interoperability, and to port codebases written in an obsolete or deprecated language (e.g. COBOL, Python 2) to a modern one. They typically rely on handcrafted rewrite rules, applied to the source code abstract syntax tree. Unfortunately, the resulting translations often lack readability, fail to respect the target language conventions, and require manual modifications in order to work properly. The overall translation process is timeconsuming and requires expertise in both the source and target languages, making code-translation projects expensive. Although neural models significantly outperform their rule-based counterparts in the context of natural language translation, their applications to transcompilation have been limited due to the scarcity of parallel data in this domain. In this paper, we propose to leverage recent approaches in unsupervised machine translation to train a fully unsupervised neural transcompiler. We train our model on source code from open source GitHub projects, and show that it can translate functions between C++, Java, and Python with high accuracy. Our method relies exclusively on monolingual source code, requires no expertise in the source or target languages, and can easily be generalized to other programming languages. We also build and release a test set composed of 852 parallel functions, along with unit tests to check the correctness of translations. We show that our model outperforms rule-based commercial baselines by a significant margin.

Authors: Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, Guillaume Lample

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! So the paper we're looking at today can take the code on the left, which is written in Python, and can output the code on the right, which is written in C++. Now the point here is that the code on the right does the same thing as the code on the left, so it is implementing the same function. The surprising thing here is that this model that takes the Python as an input has never been explicitly trained to output C++. So this is an unsupervised translation model. And the cool thing about this paper is that by having no target, having no supervised signal at translating source code languages into one another, it can perform pretty well at the task nonetheless. So we're going to look at this paper. It's called Unsupervised Translation of Programming Languages by Marie-Anne Lachaud, Baptiste Rosier, Loïc Chanusso and Jérôme Lomple at Facebook AI Research. As always, if you like content like this, consider sharing it out and leaving a like and also leaving a comment if you have something to say about it. They say a trans compiler, also known as a source-to-source translator, is a system that converts source code from a high-level programming language such as C++ or Python to another. They say trans compilers are primarily used for interoperability and to port code bases written in an obsolete or deprecated language such as COBOL or Python 2 to a modern one. So for Python 2, you might know this tool that's called 2-to-3. So 2-to-3 is a tool that ships with the Python 3 standard library, I believe, that allows you to take Python 2 code and produce Python 3 code. And that is to kind of push people to convert their old code bases of Python 2 to the modern Python 3. Now 2-to-3 is a handwritten program. It has specific rules built in that the programmers know if we modify Python 2 like this, Python 3 gets out. For example, the print statement in Python 2 requires no brackets, so we make a rule that whenever there's a print statement with no brackets, we'll add the brackets such that it's Python 3 compliant. Most of this code will transform the source code first into an abstract syntax tree, modify that, apply specific rules to that, and then output the language from the abstract syntax tree. Now the problem here is many-fold. First of all, there can only be so much translation as there are rules. So every one of these rules has to be coded as a modification to the abstract syntax tree, and every one of these rules is handcrafted and therefore needs sort of human ingenuity. Humans need to go and write these rules of how to transform one language into another. And often times, even though you can write these rules, often times whatever comes out is sort of a bit of a cryptic source code, because you kind of have to make sure that your rules cover all the possible things, and the source code that comes out is oftentimes very cryptic and a bit hard to understand, because it's been sort of expanded and formalized to make sure that it still does the same thing as the original source code. Now for Python 2 to Python 3, this is still easy, right? These languages are extremely similar, because it's not that big of a step to Python 3, except if you use very low level language constructs or language features which have been obsoleted with Python 3. On the other hand, if there's something like Cobol, so a lot of this old banking code or insurance code or government agency code whatnot is written in these really old programming languages, and they've been kept alive by these old-school programmers that are slowly but surely all retiring now, and there are just not many new programmers around that can support these languages, and the languages themselves aren't really updated that much anymore, so you would like to transform Cobalt into something like... I don't want to call Java modern programming language, but it is used in modern times. I'd rather not call it modern per se. Java itself is a beast that's been sort of supported since forever, but in any case, you would like to transform something like Cobalt to something like Java, where you have a lot of programmers that can develop and further develop your code. This is much harder. Cobalt and Java are much more away from each other than are Python 2 and Python 3. So what you would like to do is you would like to have a tool that is like 2 to 3, but humans... because now if you want a tool like this, you need someone that's really proficient in Cobalt and Java in order to write a tool like this, and you need lots of them, and they need to invest lots of time. What you would rather like to do is you would like to learn a system that translates from one language into another, such that the meaning is conserved. And this of course is exactly the domain of natural language machine translation, except it's source code. So we all know that we've all realized in the last years that things like Google Translate have become extremely good at translating. So they say right here, although neural models significantly outperform their rule-based counterparts in the context of natural language translation, which is Google Translate is all learned right now, they say their applications to trans compilation have been limited due to the scarcity of parallel data in this domain. So what's the problem with just going and saying, oh, we can build really good neural machine translation models, let's just apply them to source code. The problem is if you build a neural machine translation model, say something that transforms English to German, so you have the word hello and you output hello. You can do it not with just one word, but with entire sentences and so on. These models, they usually, or in the classical sense, what they need are parallel corpora, which means that you have documents that are written in many languages and you can guarantee that they mean the same thing. So this is a supervised signal. One example of this is, let's say press releases of the United Nations. So the United Nations will make some press release and they will then have professional translators translate that press release into all of the different or into many different languages. And so you can pretty much guarantee that these mean the same thing. So these pairs of documents or triplets or whatnot, they are supervised training data for a machine translation model that translates from one language into the other. And the neural machine translation models rely heavily on these parallel corpora. For source code, you just don't have that as much. You don't have big code bases in great numbers where the exact same thing is implemented in one language and in the other language. There's just not that much data available. It is the case that sometimes, sorry, that sometimes, let's say in the case of Torch, it started as Lua and then it went to PyTorch and the developers had to translate the code from Torch to Python. But in the same in the same step, they've also made improvements. They sort of re-engineered and reinvented the framework and made it better. And so you can't really say these are the same things. And likewise, there's not a lot of code available where the same thing is implemented in two languages. So we just don't have these parallel corpora for the source code translation. So rather what this paper does is this paper goes into unsupervised machine translation. Now what does unsupervised machine translation mean? Unsupervised machine translation, you imagine I have just a big database of documents. And these documents, I know they're all in English. And I have this other big database and I know that they are all in German. I just know their documents are in German. But they don't correspond to each other. They're just German documents. And over here, they're just English documents. They don't, I don't say that these two here are somehow the same. No, I just have a bunch of German, a bunch of English. They don't even have to correspond. They're just text. And now what I want to do is I want to learn a shared embedding space. I sort of want to learn a shared space of embeddings for these two languages, such that similar things are mapped to the similar place. So if these two documents just happen to talk about the same thing, I want them to be mapped to similar spaces in this shared embedding space. So I'm gonna have one model, a single model, where I input the text and it goes into this shared embedding space. Okay, now this is unusual because usually in machine translation, if you translate from here from English to German, then you'll have your dedicated model that takes as English as an input and German as an output. And that would be a different model than one that takes even German as an input and English as an output or French as an input and German as an output. In this case, we have this process right here and this process right here is the same model. And then the decoder that translates this, of course, now we have the encoder embedding, and then the decoder that actually translates to a language is also going to be the same model. So it's the same model that translates to English, then it translates to German. So first of all, how do we make the same model? Let's say we have the perfect encoder, right? This is E, the encoder, the same encoder for all languages. Let's say we have the perfect encoder and it can map the... whenever a sentence means the same thing in different languages, we can completely map it to the same point in embedding space, irrespective of the language it comes from. Now, how do we tell the decoder, which is also the same model, like how does it know what to do? This is a little trick where you basically take this embedding, so you take the input, you put it into your model, I don't even know how you do this, and then you're output is going to be autoregressive, right? So you decode one token at a time. So you decode this token and then you feed it back into the model, and then you decode this token, you feed that back into the model, and so on. This is an autoregressive language model. And the trick here is that the very first token is a special token that describes the language you want to output. So here you say, I want German, and then you let the model decode its thing. And by conditioning on this token right here, it knows it should now produce German. During training, you will simply put the token here, and if it produces something other than German, that's a loss, right? So it will learn to produce German after you produce this tag. Alright, so what we need is an encoder and a decoder, such that in the encoder we can put in any language text, and it will map to the same space the things that mean the same things, and we need a decoder that can produce any language given this first thing. Now the decoder should be fairly easy, right? If we have a shared vocabulary between the languages, and we always put this token, the decoder is not a problem. You can just learn the decoder in a straightforward way, but the encoder is going to be a problem. So how does the encoder map the different languages to the same space, such that the same things are ending up in the same place? It seems a bit counterintuitive, right? Because it doesn't know which things correspond to which things. Now the first thing you need is a shared vocabulary here. Since we are in a shared space right here, what you need is a shared vocabulary. So you tokenize all of the text with a shared vocabulary. And this vocabulary is going to consist of sort of word pieces. Now if you don't know what word pieces are, in a word piece tokenization what you would do is you would split words into so-called word pieces. So for example hello right here might be split into two word pieces. The first word piece might be he, and the second word piece might be LLO. There is usually some kind of indicator here that this is the end of a word and so on, but we'll simplify. And hello right here would be a HA for the first token and then LLO for the second token. And these kind of word piece encodings, since the smallest units are going to be the characters themselves, they ensure that you always have everything is in vocabulary. You have no out-of-vocabulary tokens. But here you can already see that if we tokenize the languages like this and then we use the same encoder, so the same encoder will pop them into this shared space, that means that to the model this and this looks like the same thing. It is the same thing, right? It's the same input token in different languages. Now as you can see this comes from the same word, the LLO at the end in English and the LLO at the end in German. It comes from the same word. So you know it's fair to assume that since it's the same input it's going to be mapped into a same embedding space right here. Or since these things are usually context-dependent we can say in a similar embedding space or a close embedding space, but certainly the initial vectors are the same. That is already half the task, right? So by tokenizing in this way we have already mapped part of our languages, even though they're just the different languages, we have mapped the same word to the same space. And this relies on the fact that in this case for example English and German and for example French they have significant overlap in their words as such. So the word hello and the word hollow, they are almost the same word as letters, as word pieces. And these shared embedding techniques abuse sort of the fact that these languages are close. They'll say there will be some word pieces that are going to be the same in these languages and naturally because they're the same they'll end up in the same place in embedding space. And because that now you have the... so what these embedding techniques do is they simply figure out the statistical relations between the word pieces. So if two things appear together often in the same context they'll be mapped into the same space as well. So it would realize there is a lot of times where ha and he appear in front of this low thing. So I should probably map the ha and the he to the same location in embedding space. So the same relation to the low. So they would end up at the same place. So now you see even though these word pieces are different, like they get different IDs, they'll be mapped to the same place in embedding space because their relation to the low is the same and the low themselves are being mapped to the same place because they are actually the same. So you can see that this partial overlap between word pieces in the different languages combined with the shared embedding pre-training results in these token across the languages results in an alignment of the embeddings. So naturally the things that mean the same things are going to be in the same places in embedding space, either because they are the same or because their statistical relation to the things that are the same is the same. It is sort of like these ha and the he are like synonyms in this shared language. So if you jumble all the English and German text together, the model thinks ha and he are synonyms and therefore maps it to the same space. This happens exactly the same in if you only have one language, two true synonyms. So this would exactly be the same thing in this case. Alright, so now we have different languages. We have a single encoder where we can input any of those languages mapping it to the shared space. The decoder can be trained by simply giving it this indicator token right here to decode the appropriate language. So now the question is how exactly do we train this such that this happens? There's one caveat in programming languages of course we still have to check whether that's the same or not and we know that in programming languages a lot of programming languages for example the word if is the same right so if you tokenize Java or Python or C++ the word if is the same and likewise there is a lot of overlap between the different programming languages and that exactly is this correspondence to here. These models use the parts that are overlapping either tokens themselves or this can also be grammatical constructs and so on they can also be overlapping and therefore map to the same space so if a construct is used in the same way this can be in higher layers this can induce the same effect and they'll use that to map the and sorry they'll use that the result will be that the similar things in the same in these languages will be mapped to the similar spaces in embedding space. Now this makes this example a bit weaker because that means this method would work exceptionally well for something like Python 2 to Python 3 because they of course have like a lot of overlap of syntax and keywords and constructs whereas something like cobalt to Java it's more let's say doubtable that they will work so well. In this paper here they've chosen C++ Python and Java which do have significant overlap but especially something like Python to Java of course there is a lot of a difference or Python to C++ as well Python is not typed and so you can see a bit of the difficulty is already in the paper here but you have to be aware that this works less and less the less this shared overlap is given. Alright so how do you train these models and remember we don't have parallel corpora we are simply reliant on having databases from having big repositories of Python code C++ code and Java code and they don't correspond to each other so the first thing you do and as I understand it you can do these things in parallel but there are three different objectives that achieve three different things in these models. The first objective is the cross-lingual masked language pre-training. Now the models here are going to be transformer models with encoders and decoders and that's comes from the attention is all you need papers and various other papers like this I've done videos on those if you want to see that. This masked language model pre-training however is from the BERT paper so BERT if you don't know what that is I've also done a video on that. This simply trains the encoder so this is to train the encoder. What you would do is you would input code so usually in masked language model if you train the encoder you input code with tokens like hello there you would then so this is your input you would then mask some of the tokens for example here the low and maybe the entire word there you would scrap that you would put it through your encoder which is this transformer model like BERT and then BERT is supposed to reconstruct hello there. BERT is supposed to reconstruct these two tokens like it doesn't see them and you ask it what did I cross out and it needs to reconstruct that so you train the model to reconstruct these masked tokens and the research on BERT and other things has shown that if you train with this objective the encoder sort of learns about the structure of code it learns about it learns which tokens and which constructs often appear together and therefore it learns something about the structure of the input and that means it can create whatever is up here is a good and meaningful embedding for these things that tells you something about the statistical coexistence of tokens and of course since we're doing this with all the languages so the Python goes in here C++ goes in here without telling the model what it is you just throw it in there right Java goes in there by tokenizing it and you see an example right here so if this right here is C++ but in Python this would also be if and since it needs to learn a single encoder for all of these languages and since the tokens overlap partially it is going to result in exactly what we want namely a shared embedding space where even though the input comes from different languages it is mapped similar things are mapped to similar places in the embedding space right so the mask language model pre training very quickly as you take a piece of code like here on the left you mask out some of the tokens here you can see them in this mask and you simply ask the model the encoder to reconstruct those things okay so this is just for the encoder as far as I understand it the encoder doesn't it doesn't see the thing back here it simply sees this and you tell it please reconstruct please tell me which words or which tokens I I clipped out here and it's supposed to tell you okay the first one is if the second one is int and the third one is the I now if you consider what the encoder has to do here so if you were to see this then that pretty clearly you know you could you could guess that that is an if of course it's not a hundred percent but this is just pre training right so you train it to output if here now here you have to do a little bit more inference maybe you've seen this for construct a bunch of times and you can see that this is compared here and this is added so probably it's an integer and then in the last thing this is even more complicated if you don't see that the eye is here you somehow have to guess that what it is it's not clear right but you can guess that okay there's a local variable I right here and probably it's going to be used somewhere in this block now this here isn't I and I don't see I anywhere else so probably I goes in here which makes sense because it's an integer and prime is an array and and it integers index arrays so on okay so this is what the model does first the second thing is we need to train the decoder somehow how do we train the decoder in a very similar way we make the decoder do denoising auto encoding now before we just had single tokens we just asked the encoder to reconstruct tokens so the encoder is this box right here this colors this box is the encoder and the actual part that's going to predict these words is going to be one sort of one classification layer on top that is going to predict for each position the individual word now did this is just for pre-training after the pre-training you scrap that and you attach it to a decoder so you attach whatever you got out of the encoder to a decoder and the decoder will output in an autoregressive way one token after another I did I put a it output a token right here and it feed that token back into the decoder saying okay here's what I've produced now produce the next thing you produce the next token and so on so it would produce token after token the output and now as I said I'm not exactly sure I think they're doing all of these things at the same time so this would still be here but the information would just be routed in two different ways or maybe they do it one after another it doesn't really matter but what matters is in this thing here you now train the decoder I mean you train it jointly with the encoder but you also involve the decoder and you do this by doing something very similar you corrupt a piece of code and you get corrupted code now you can see part of this corruption is masking like you did before but also part of the corruption is like here you scramble some of the tokens right this was it was this over here you just jumble some of them around a bit and then you here you also drop a token as you can see that the one is dropped and you simply so you don't show this to the encoder or the decoder you input this corrupted code into the decode into the first the encoder and then you ask the decoder to give you back the original code without showing it the original code so the the task for the decoder for the encoder decoder for the entire model here is if you're if you see this here is corrupt the code I have corrupted it in various ways please tell me what I originally had now it can the masking it does the same as before it sort of infers it this thing here it says well probably I probably this isn't really correct you don't even tell it where the errors are right before with the masking you at least told it where the errors are now you don't even tell it where the where the errors are so it needs to recognize this here is probably correct this isn't this I'm gonna rewrite this to that okay and it does this one token at a time so it first goes into the dirt and it needs to output the correct thing this is I hope the difference is clear to the masked language modeling which involved the involved the decoder and here also is the first time where in the encoder you you prepend this Java token now this as you can see it still goes from the same language to the same language but this is where you train the decoder to output a given language so here with the token again this is the same decoder for all the three languages the only difference here is every time you simply provided with the special token at the beginning to tell it which language it should decode right now so this this now we have an encoder that maps all the languages to a shared space and we have a decoder that conditioned on a token like this can output a valid code in that thing assuming this here was corrupted code now since the encoder is shared it should map the same kind of corrupted code of the different languages to the same place in the embedding space and therefore this would also this would already be enough to have this model that we desire we can input some code it doesn't actually have to be corrupted right we can just input some code in one language and ask the decoder to output the other language and this works but it doesn't work super well and here the authors go for another idea from the unsupervised machine translation literature which is back translation so back translation is a technique where you can tune an unsupervised machine translation model in a way that you would tune a supervised one but of course you don't have supervised data so what's the plan you will produce the data yourself using your own model so the plan is pretty simple it's actually contained in the back translation name so if you have a piece of code what you would do is you would first use your model to translate this to another language any of your choice now you have no clue whether this thing here is correct or not you have no clue and you have no way of assessing it because you don't have ground truth what you can do is use your model again or actually use a second model that you train in parallel now I believe in this case they could use the same model but you can that could be instable and so on but in any case you can use your system again to translate it back to your original language your system can do that right and here whatever you get as an output you know the ground truth it's whatever you started with so now you can compare what comes out to what you started with the difficulty of course is if there is a mistake you don't know which of the two models made a mistake and you so it could be could be that your original translation model made a mistake and or it could be that your back translation model made a mistake and you have to find a loss function that kind of punishes both equally or you simply keep one sort of constant and loss free and train the other one because there there's going to be a sample where you have C++ as an input and then the intermediate language is Python so all of the models sort of get trained once as an as a source to target translator and once as a target to source translator but I hope the the objective is clear from the back translation so now with the back translation you actually you train the models to go from one language to another language okay and that's the that's the final goal even though you do it without supervised data you now have a model that can encode things into a shared space that can decode into a language and that is attuned to translating from one language to another language so that's that's it how this is all how does this work now for evaluation the question is of course how do you evaluate models like this for evaluation they go to this website called geeks for geeks and this is a an online platform with computer science and programming articles it gathers many coding problems and presents solution in several programming languages okay so this is a website that teaches you to code and it will have like an exercise please do this and then it will provide solutions in the different languages now why is that cool and they have an example they have an example right here why is that cool because not only can you be relatively sure that these different functions that you have here do the same thing but you can also relatively be relatively sure that they are implemented in the similar way right because you what this website is trying to do is it's trying to teach the people how to how to code up an algorithm that they think up in their head and therefore not only is the solution correct and the same it is implemented in the in the same way as you can see here the construct there's this if construct is everywhere the else if is everywhere so even though some of the languages might have specialty things for implementing some algorithms these are really the same algorithmic the same expression of algorithmic thought in the different languages so that is a perfect parallel data set the problem of course is that there is not that many so it is good enough as a test set it is not good enough as a training set but given that it's a test set you can just have these as test set and then you can input the C++ and see whether or not the Java comes out the problem here of course is that even though this is very clear there are still you know sort of many variations of how you can implement that to even express the same algorithmic thought so metrics from natural language processing like blue just aren't going to be very good because they look at n-gram overlap and you can write this function with very different n-grams and still be very very valid and correct and also exact match is not going to be really the the gold standard here so what they do is they create a set of unit tests where for each of these functions they go they check their input types they randomly generate input randomly generate a set of inputs look whatever comes out and if the same thing comes out in all of their test functions that they consider this a good unit test for that function so whenever you your model now produces let's say you input Python it produces a C++ you simply put these unit tests through the C++ function you produce and if they produce the same output as the Python the original Python function when on the same inputs then you consider the unit test to succeed and you consider the function to be correct that this of course this isn't this isn't the super duper gold standard especially with random inputs because usually what you want to do is test sort of corner cases but it's better than anything else so far I've been a long dis-advocate of unit tests honestly because I think whenever a human writes a unit test then they're probably since they have already implemented the function itself they're probably going to make the same mistakes or they're probably just going to replicate the code and thinking of the function in the unit test itself and therefore it doesn't really get you anything I guess in large organizations you write unit tests so that someone else doesn't screw up your code but in this case it would actually be cool because now as a human you could simply write a bunch of unit tests and then let your let your trans compiler do the heavy lifting and you simply check whether or not the output is good alright so how does this do here you can see they have some baselines the C++ to Java as I understand it is a commercial system and the Java to Python is an open source system both are human experts that make up these rule-based systems on how to trans on how to translate code into other languages now the if you do what they have here is trans coder beam one which means a beam size of one so if you don't know a beam search is very shortly beam search is like if you decode from your language model you can either always take the next token that has the highest probability this would be greedy decoding or a beam size of one or you can sort of always keep the top n hypotheses of what the of what the most likely output is as you can keep that as a you can keep the top five in memory and always decode these five on sort of like you have a mini batch of five sequences and you always keep the top five in memory so at the end of the decoding you're going to have five different variants of the same sentence or of the same decoded output and you can then decide which one you like best and usually what you do is you then output the one that has the highest probability which is not the same as the greedy because sometimes the next token will be will look one next token will look very good in a greedy way but you'd better take the second most likely because the next to next token is going to sort of make up for that to make the entire sequence even more likely so more beam size basically means you can keep more hypotheses of the output in memory until the end so if you just do the greedy decoding you see you already get fairly close to these baselines it's very very cool very interesting and if you up the beam size you surpass these baselines now the way they up the beam size here I find to be a bit let's call it a bit cheaty because when they say beam five what they mean is they keep the five hypotheses and then at the end I as I understand it if any of the five hypotheses passes all the unit tests or the most they keep it right so basically they give themselves the freedom to say whichever one of the five we output is the best that's the one we count and of course that's not really a match to the commercial or to the baseline system because it can output one thing now it is maybe a good practical application to give the human that you know you input a function you give the human five options to choose from and it can choose and thereby decide which one the human likes best but it is sort of it is a wonky what I like more is this here the beam 10 top one this is what you would actually do so we could keep 10 hypotheses during decoding and that the end output the top one the top likely one and as you can see that is better than greedy but it is worse than where you you know give yourself the freedom to output multiple ones of course though they say that most of the errors that this top one makes come from compilation errors when the target language is Java or C++ it suggests that the beam and top one metric could easily be improved we leave this to future work which this again I find valid right so if you if your method is I'm going to keep the top 10 hypothesis until the end and then I'm going from the top and I simply compile them and I output the first one that compiles that that's not cheating right that's a valid thing again yeah so in that way I can I can understand what they're saying right here okay so they give some examples some of which I find very interesting so the first thing here is that oh yeah by the way I've said in the I've said that the tokenizer between the natural languages is shared they make a little tweak here in that they tokenize the different languages with their language respective tokenizers which will still end up tokenizing pretty much you know this the print statement in C++ or in Java no actually the print statement in Python is print and in Java it's println and so on but it will still like the all the if statements it will still tokenize into the same into the same word but it's simply not viable to to parse Python with a C++ parser okay so we have looked at this the results this is one of the results they look at their shared embedding space and this is a t-sneak plot so a 2d projection of this shared embedding space and you can see that this is actually happening so the different so null null and none are mapped to similar locations println and cout are mapped to similar locations in this space so this is exactly what we want this is sort of a verification that this method of embedding the different languages into the same space really turns out such that whatever means the same thing is mapped to the same place you can see here catch and accept two very different tokens are mapped to the same place simply because they're used in the same sort of constructs across the languages very cool one of these examples here is quite impressive and kind of shows the difference between this and and rule-based translation in this function right here you have a C plus plus function that takes a character pointer to that is called str str in as an input now in C++ strings are at least in old versions of C++ strings are handled as character arrays so a string is indistinguishable from a character array and in this case usually what you do is you don't input the array because that will cause a copy you input a pointer to the first to you input a pointer to the array and that would define the string okay so if you translate this again this the type of this is simply character array if you translate this with this transcoder system that they've built into Java in Java there is a type called string right there's a native type called string and is that true I think oh yeah that's and then that's handled really weirdly in the JVM I think yes so there is it at least there is a type called string so it would map that it would recognize are you mean a string therefore I'm going to put a string here and it uses all the string method like string length string character at and so on where in C++ this is just an array and you just have array accesses now they take this same C++ function and only change one thing they change the name of the parameter everything else is the same but now the character array is called our okay and they put it through the same system and that system now outputs a function that takes in a character array called our instead of a string and it uses you know here the property length it uses array access instead of this car character at method so simply by changing the name and this is something where I believe the rule-based systems can this can be an advantage over rule-based system because what this here does is it simply says oh I've seen a lot of humans in my code base that use this use like stir as a as a variable name and that usually means that the constructs here are like the constructs in Java where people use strings and I've seen other places where people use you know names like this right here and usually that is used in the same context as in Java people use character arrays right so it in programming it's not only important what the the code actually does but a lot of programming goes via naming of things like other programmers will read your code and by reading stir right here they will sort of assume that this is a string whereas if they read our right here they will assume you're a pirate and you are referring to a character array and they will treat the code the code means something different and these systems right here these neural machine translation systems can actually understand that part because they do statistical inference on code that humans wrote if you change this back to say input then again it goes back to a string and uses all the string functions so that's fairly impressive in my mind and it yeah definitely an advantage over rule-based systems of course the disadvantage over rule-based systems is that in rule-based systems you can almost get on like sometimes you can even guarantee that the code does the same thing here you can't they give some examples of failed translations where so now you get you run into this problem where the min function in Python is overloaded it can either give you the minimum of a sequence or it can give you the minimum of two things now this is translated to Java right here and math dot min is not overloaded in Java it only gives you the minimum of two things and not the minimum of an array and it still outputs that now given enough data probably could learn because these things are all context dependent but this is one of the this is one of the failure cases of these models of course all right so this was this paper I I've read that the code of this and the unit tests will be output will be put online at some times they are not right now if I if I hear about it I can link to it or let you know about it let me know what you think of this paper in the comments share it out and subscribe if you haven't yet and bye bye
 Hi there! So the paper we're looking at today can take the code on the left,
 which is written in Python, and can output the code on the right, which is
 written in C++. Now the point here is that the code on the right does the same
 thing as the code on the left, so it is implementing the same function. The
 surprising thing here is that this model that takes the Python as an input has
 never been explicitly trained to output C++. So this is an unsupervised
 translation model. And the cool thing about this paper is that by
 having no target, having no supervised signal at translating source code
 languages into one another, it can perform pretty well at the task
 nonetheless. So we're going to look at this paper. It's called
 Unsupervised Translation of Programming Languages by Marie-Anne Lachaud, Baptiste
 Rosier, Loïc Chanusso and Jérôme Lomple at Facebook AI Research. As always, if you
 like content like this, consider sharing it out and leaving a like and also
 leaving a comment if you have something to say about it. They say a trans
 compiler, also known as a source-to-source translator, is a system
 that converts source code from a high-level programming language such as
 C++ or Python to another. They say trans compilers are primarily used for
 interoperability and to port code bases written in an obsolete or deprecated
 language such as COBOL or Python 2 to a modern one. So for Python 2, you might
 know this tool that's called 2-to-3. So 2-to-3 is a tool that ships with the
 Python 3 standard library, I believe, that allows you to take Python 2 code
 and produce Python 3 code. And that is to kind of push people to convert their
 old code bases of Python 2 to the modern Python 3. Now 2-to-3 is a
 handwritten program. It has specific rules built in that the programmers know
 if we modify Python 2 like this, Python 3 gets out. For example, the print
 statement in Python 2 requires no brackets, so we make a rule that whenever
 there's a print statement with no brackets, we'll add the brackets such that
 it's Python 3 compliant. Most of this code will transform the
 source code first into an abstract syntax tree, modify that, apply specific
 rules to that, and then output the language from the abstract syntax tree.
 Now the problem here is many-fold. First of all, there can only be so much
 translation as there are rules. So every one of these rules has to be coded as a
 modification to the abstract syntax tree, and every one of these rules is
 handcrafted and therefore needs sort of human ingenuity. Humans need to go and
 write these rules of how to transform one language into another. And often
 times, even though you can write these rules,
 often times whatever comes out is sort of a bit of a cryptic source code, because
 you kind of have to make sure that your rules cover all the possible things, and
 the source code that comes out is oftentimes very cryptic and a bit hard
 to understand, because it's been sort of expanded and formalized to make sure
 that it still does the same thing as the original source code. Now for Python 2 to
 Python 3, this is still easy, right? These languages are extremely similar,
 because it's not that big of a step to Python 3, except if you use very low
 level language constructs or language features which have been
 obsoleted with Python 3. On the other hand, if there's something like Cobol, so a
 lot of this old banking code or insurance code or government agency code
 whatnot is written in these really old programming languages, and they've been
 kept alive by these old-school programmers that are slowly but surely
 all retiring now, and there are just not many new programmers around that can
 support these languages, and the languages themselves aren't really
 updated that much anymore, so you would like to transform Cobalt into something
 like... I don't want to call Java modern programming language, but it is used in
 modern times. I'd rather not call it modern per se. Java itself is a beast
 that's been sort of supported since forever, but in any case, you would like
 to transform something like Cobalt to something like Java, where you have
 a lot of programmers that can develop and further develop your code. This is
 much harder. Cobalt and Java are much more away from each other than are Python 2
 and Python 3. So what you would like to do is you would like to have a tool that
 is like 2 to 3, but humans... because now if you want a tool like this, you need
 someone that's really proficient in Cobalt and Java in order to write a tool
 like this, and you need lots of them, and they need to invest lots of time. What
 you would rather like to do is you would like to learn a system that translates
 from one language into another, such that the meaning is conserved. And this of
 course is exactly the domain of natural language machine translation, except it's
 source code. So we all know that we've all realized in the last years that
 things like Google Translate have become extremely good at translating. So they
 say right here, although neural models significantly outperform their rule-based
 counterparts in the context of natural language translation, which is Google
 Translate is all learned right now, they say their applications to trans
 compilation have been limited due to the scarcity of parallel data in this domain.
 So what's the problem with just going and saying, oh, we can build
 really good neural machine translation models, let's just apply them to source
 code. The problem is if you build a neural machine translation model, say
 something that transforms English to German, so you have the word hello and
 you output hello. You can do it not with just one word, but with entire sentences
 and so on. These models, they usually, or in the classical sense, what they need
 are parallel corpora, which means that you have documents that are
 written in many languages and you can guarantee that they mean the same thing.
 So this is a supervised signal. One example of this is, let's say press
 releases of the United Nations. So the United Nations will make some press
 release and they will then have professional translators translate that
 press release into all of the different or into many different languages. And so
 you can pretty much guarantee that these mean the same thing. So these pairs of
 documents or triplets or whatnot, they are supervised training data for a
 machine translation model that translates from one language into the
 other. And the neural machine translation models rely heavily on these parallel
 corpora. For source code, you just don't have that as much. You don't have big
 code bases in great numbers where the exact same thing is implemented in one
 language and in the other language. There's just not that much data
 available. It is the case that sometimes, sorry, that sometimes, let's say in the
 case of Torch, it started as Lua and then it went to PyTorch and the developers
 had to translate the code from Torch to Python. But in the same in the same step,
 they've also made improvements. They sort of re-engineered and reinvented the
 framework and made it better. And so you can't really say these are the same
 things. And likewise, there's not a lot of code available where the same thing is
 implemented in two languages. So we just don't have these parallel corpora for
 the source code translation. So rather what this paper does is this paper goes
 into unsupervised machine translation. Now what does unsupervised machine
 translation mean? Unsupervised machine translation, you imagine I have just a
 big database of documents. And these documents, I know they're all in
 English. And I have this other big database and I know that they are all
 in German. I just know their documents are in German. But they don't correspond
 to each other. They're just German documents. And over here, they're just
 English documents. They don't, I don't say that these two here are somehow the
 same. No, I just have a bunch of German, a bunch of English. They don't even have
 to correspond. They're just text. And now what I want to do is I want to learn a
 shared embedding space. I sort of want to learn a shared space of embeddings
 for these two languages, such that similar things are mapped to the similar
 place. So if these two documents just happen to talk about the same thing, I
 want them to be mapped to similar spaces in this shared embedding space. So I'm
 gonna have one model, a single model, where I input the text and it goes into
 this shared embedding space. Okay, now this is unusual because usually in
 machine translation, if you translate from here from English to German, then
 you'll have your dedicated model that takes as English as an input and German
 as an output. And that would be a different model than one that takes even
 German as an input and English as an output or French as an input and German
 as an output. In this case, we have this process right here and this process
 right here is the same model. And then the decoder that translates this, of
 course, now we have the encoder embedding, and then the decoder that actually
 translates to a language is also going to be the same model. So it's the same
 model that translates to English, then it translates to German. So first of all, how
 do we make the same model? Let's say we have the perfect encoder, right?
 This is E, the encoder, the same encoder for all languages. Let's say we have the
 perfect encoder and it can map the... whenever a sentence means the same thing
 in different languages, we can completely map it to the same point in
 embedding space, irrespective of the language it comes from. Now, how do we tell
 the decoder, which is also the same model, like how does it know what to do? This is
 a little trick where you basically take this embedding, so you take the input,
 you put it into your model, I don't even know how you do this, and then you're
 output is going to be autoregressive, right? So you decode one token at a time.
 So you decode this token and then you feed it back into the model, and then you
 decode this token, you feed that back into the model, and so on. This is an
 autoregressive language model. And the trick here is that the very
 first token is a special token that describes the language you want to
 output. So here you say, I want German, and then you let the model decode its
 thing. And by conditioning on this token right here, it knows it should
 now produce German. During training, you will simply put the token here,
 and if it produces something other than German, that's a loss, right? So it will
 learn to produce German after you produce this tag. Alright, so what we need
 is an encoder and a decoder, such that in the encoder we can put in any language
 text, and it will map to the same space the things that mean the same things, and
 we need a decoder that can produce any language given this first thing.
 Now the decoder should be fairly easy, right? If we have a shared vocabulary
 between the languages, and we always put this token, the decoder is not a
 problem. You can just learn the decoder in a straightforward way, but the encoder
 is going to be a problem. So how does the encoder map the different languages to
 the same space, such that the same things are ending up in the same place? It seems
 a bit counterintuitive, right? Because it doesn't know which things
 correspond to which things. Now the first thing you need is a shared vocabulary
 here. Since we are in a shared space right here, what you need is a
 shared vocabulary. So you tokenize all of the text with a shared vocabulary. And
 this vocabulary is going to consist of sort of word pieces. Now if you don't
 know what word pieces are, in a word piece tokenization what you would
 do is you would split words into so-called word pieces. So for example
 hello right here might be split into two word pieces. The first word piece might
 be he, and the second word piece might be LLO. There is usually some kind of
 indicator here that this is the end of a word and so on, but we'll simplify. And
 hello right here would be a HA for the first token and then LLO for the second
 token. And these kind of word piece encodings, since the smallest units are
 going to be the characters themselves, they ensure that you always have
 everything is in vocabulary. You have no out-of-vocabulary tokens. But here you
 can already see that if we tokenize the languages like this and then we use the
 same encoder, so the same encoder will pop them into this shared space, that
 means that to the model this and this looks like the same thing. It is the same
 thing, right? It's the same input token in different languages. Now as you can see
 this comes from the same word, the LLO at the end in English and the LLO at the
 end in German. It comes from the same word. So you know it's fair to assume
 that since it's the same input it's going to be mapped into a same embedding
 space right here. Or since these things are usually context-dependent we can say
 in a similar embedding space or a close embedding space, but certainly the
 initial vectors are the same. That is already half the task, right? So by
 tokenizing in this way we have already mapped part of our languages, even though
 they're just the different languages, we have mapped the same word to the same
 space. And this relies on the fact that in this case for example English and
 German and for example French they have significant overlap in their words
 as such. So the word hello and the word hollow, they are almost the same
 word as letters, as word pieces. And these shared embedding techniques abuse
 sort of the fact that these languages are close. They'll say there will be
 some word pieces that are going to be the same in these languages and
 naturally because they're the same they'll end up in the same place in
 embedding space. And because that now you have the... so what these
 embedding techniques do is they simply figure out the statistical relations
 between the word pieces. So if two things appear together often in the same
 context they'll be mapped into the same space as well. So it would realize there
 is a lot of times where ha and he appear in front of this low thing. So I should
 probably map the ha and the he to the same location in embedding space. So the
 same relation to the low. So they would end up at the same place. So now you see
 even though these word pieces are different, like they get different
 IDs, they'll be mapped to the same place in embedding space because their
 relation to the low is the same and the low themselves are being mapped to the
 same place because they are actually the same. So you can see that this
 partial overlap between word pieces in the different languages combined with
 the shared embedding pre-training results in these token across the
 languages results in an alignment of the embeddings. So naturally the things that
 mean the same things are going to be in the same places in embedding space, either
 because they are the same or because their statistical relation to the things
 that are the same is the same. It is sort of like these ha and the he are
 like synonyms in this shared language. So if you jumble all the English and
 German text together, the model thinks ha and he are synonyms and therefore
 maps it to the same space. This happens exactly the same in if you only have one
 language, two true synonyms. So this would exactly be the same thing in
 this case. Alright, so now we have different languages. We have a single
 encoder where we can input any of those languages mapping it to the shared space.
 The decoder can be trained by simply giving it this indicator token right
 here to decode the appropriate language. So now the question is how exactly do we
 train this such that this happens? There's one caveat in
 programming languages of course we still have to check whether that's the same or
 not and we know that in programming languages a lot of programming languages
 for example the word if is the same right so if you tokenize Java or Python
 or C++ the word if is the same and likewise there is a lot of overlap
 between the different programming languages and that exactly is this
 correspondence to here. These models use the parts that are overlapping either
 tokens themselves or this can also be grammatical constructs and so on they
 can also be overlapping and therefore map to the same space so if a construct
 is used in the same way this can be in higher layers this can induce the same
 effect and they'll use that to map the and sorry they'll use that the result
 will be that the similar things in the same in these languages will be
 mapped to the similar spaces in embedding space. Now this makes this
 example a bit weaker because that means this method would work exceptionally
 well for something like Python 2 to Python 3 because they of course have
 like a lot of overlap of syntax and keywords and constructs whereas
 something like cobalt to Java it's more let's say doubtable that they will
 work so well. In this paper here they've chosen C++ Python and Java which
 do have significant overlap but especially something like Python to Java
 of course there is a lot of a difference or Python to C++ as well
 Python is not typed and so you can see a bit of the difficulty is already in the
 paper here but you have to be aware that this works less and less the less this
 shared overlap is given. Alright so how do you train these models and remember
 we don't have parallel corpora we are simply reliant on having databases from
 having big repositories of Python code C++ code and Java code and they don't
 correspond to each other so the first thing you do and as I understand it you
 can do these things in parallel but there are three different objectives
 that achieve three different things in these models. The first objective is the
 cross-lingual masked language pre-training. Now the models here are
 going to be transformer models with encoders and decoders and that's comes
 from the attention is all you need papers and various other papers like
 this I've done videos on those if you want to see that. This masked language
 model pre-training however is from the BERT paper so BERT if you don't know
 what that is I've also done a video on that. This simply trains the encoder so
 this is to train the encoder. What you would do is you would input code so
 usually in masked language model if you train the encoder you input code with
 tokens like hello there you would then so this is your input you would then mask
 some of the tokens for example here the low and maybe the entire word there you
 would scrap that you would put it through your encoder which is this
 transformer model like BERT and then BERT is supposed to reconstruct hello
 there. BERT is supposed to reconstruct these two tokens like it doesn't see
 them and you ask it what did I cross out and it needs to reconstruct that so you
 train the model to reconstruct these masked tokens and the research on BERT
 and other things has shown that if you train with this objective the encoder
 sort of learns about the structure of code it learns about it learns which
 tokens and which constructs often appear together and therefore it learns
 something about the structure of the input and that means it can create
 whatever is up here is a good and meaningful embedding for these things
 that tells you something about the statistical coexistence of tokens and of
 course since we're doing this with all the languages so the Python goes in here
 C++ goes in here without telling the model what it is you just throw it in
 there right Java goes in there by tokenizing it and you see an example
 right here so if this right here is C++ but in Python this would also be if and
 since it needs to learn a single encoder for all of these languages and since the
 tokens overlap partially it is going to result in exactly what we want namely a
 shared embedding space where even though the input comes from different
 languages it is mapped similar things are mapped to similar places in the
 embedding space right so the mask language model pre training very quickly
 as you take a piece of code like here on the left you mask out some of the tokens
 here you can see them in this mask and you simply ask the model the encoder to
 reconstruct those things okay so this is just for the encoder as far as I
 understand it the encoder doesn't it doesn't see the thing back here it
 simply sees this and you tell it please reconstruct please tell me which words
 or which tokens I I clipped out here and it's supposed to tell you okay the first
 one is if the second one is int and the third one is the I now if you consider
 what the encoder has to do here so if you were to see this then that pretty
 clearly you know you could you could guess that that is an if of course it's
 not a hundred percent but this is just pre training right so you train it to
 output if here now here you have to do a little bit more inference maybe you've
 seen this for construct a bunch of times and you can see that this is compared
 here and this is added so probably it's an integer and then in the last thing
 this is even more complicated if you don't see that the eye is here you
 somehow have to guess that what it is it's not clear right but you can guess
 that okay there's a local variable I right here and probably it's going to be
 used somewhere in this block now this here isn't I and I don't see I anywhere
 else so probably I goes in here which makes sense because it's an integer and
 prime is an array and and it integers index arrays so on okay so this is what
 the model does first the second thing is we need to train the decoder somehow
 how do we train the decoder in a very similar way we make the decoder do
 denoising auto encoding now before we just had single tokens we just asked the
 encoder to reconstruct tokens so the encoder is this box right here this
 colors this box is the encoder and the actual part that's going to predict
 these words is going to be one sort of one classification layer on top that is
 going to predict for each position the individual word now did this is just for
 pre-training after the pre-training you scrap that and you attach it to a
 decoder so you attach whatever you got out of the encoder to a decoder and the
 decoder will output in an autoregressive way one token after another I did I put
 a it output a token right here and it feed that token back into the decoder
 saying okay here's what I've produced now produce the next thing you produce
 the next token and so on so it would produce token after token the output and
 now as I said I'm not exactly sure I think they're doing all of these things
 at the same time so this would still be here but the information would just be
 routed in two different ways or maybe they do it one after another it doesn't
 really matter but what matters is in this thing here you now train the decoder
 I mean you train it jointly with the encoder but you also involve the decoder
 and you do this by doing something very similar you corrupt a piece of code and
 you get corrupted code now you can see part of this corruption is masking like
 you did before but also part of the corruption is like here you scramble
 some of the tokens right this was it was this over here you just jumble some of
 them around a bit and then you here you also drop a token as you can see that
 the one is dropped and you simply so you don't show this to the encoder or the
 decoder you input this corrupted code into the decode into the first the
 encoder and then you ask the decoder to give you back the original code without
 showing it the original code so the the task for the decoder for the encoder
 decoder for the entire model here is if you're if you see this here is corrupt
 the code I have corrupted it in various ways please tell me what I originally
 had now it can the masking it does the same as before it sort of infers it this
 thing here it says well probably I probably this isn't really correct you
 don't even tell it where the errors are right before with the masking you at
 least told it where the errors are now you don't even tell it where the where
 the errors are so it needs to recognize this here is probably correct this isn't
 this I'm gonna rewrite this to that okay and it does this one token at a time so
 it first goes into the dirt and it needs to output the correct thing this is I
 hope the difference is clear to the masked language modeling which involved
 the involved the decoder and here also is the first time where in the encoder
 you you prepend this Java token now this as you can see it still goes from the
 same language to the same language but this is where you train the decoder to
 output a given language so here with the token again this is the same decoder for
 all the three languages the only difference here is every time you simply
 provided with the special token at the beginning to tell it which language it
 should decode right now so this this now we have an encoder that maps all the
 languages to a shared space and we have a decoder that conditioned on a token
 like this can output a valid code in that thing assuming this here was
 corrupted code now since the encoder is shared it should map the same kind of
 corrupted code of the different languages to the same place in the
 embedding space and therefore this would also this would already be enough to
 have this model that we desire we can input some code it doesn't actually have
 to be corrupted right we can just input some code in one language and ask the
 decoder to output the other language and this works but it doesn't work super
 well and here the authors go for another idea from the unsupervised machine
 translation literature which is back translation so back translation is a
 technique where you can tune an unsupervised machine translation model
 in a way that you would tune a supervised one but of course you don't
 have supervised data so what's the plan you will produce the data yourself using
 your own model so the plan is pretty simple it's actually contained in the
 back translation name so if you have a piece of code what you would do is you
 would first use your model to translate this to another language any of your
 choice now you have no clue whether this thing here is correct or not you have no
 clue and you have no way of assessing it because you don't have ground truth
 what you can do is use your model again or actually use a second model that you
 train in parallel now I believe in this case they could use the same model but
 you can that could be instable and so on but in any case you can use your system
 again to translate it back to your original language your system can do
 that right and here whatever you get as an output you know the ground truth it's
 whatever you started with so now you can compare what comes out to what you
 started with the difficulty of course is if there is a mistake you don't know
 which of the two models made a mistake and you so it could be could be that
 your original translation model made a mistake and or it could be that your
 back translation model made a mistake and you have to find a loss function that
 kind of punishes both equally or you simply keep one sort of constant and
 loss free and train the other one because there there's going to be a
 sample where you have C++ as an input and then the intermediate language is
 Python so all of the models sort of get trained once as an as a source to target
 translator and once as a target to source translator but I hope the the
 objective is clear from the back translation so now with the back
 translation you actually you train the models to go from one language to
 another language okay and that's the that's the final goal even though you do
 it without supervised data you now have a model that can encode things into a
 shared space that can decode into a language and that is attuned to
 translating from one language to another language so that's that's it how this is
 all how does this work now for evaluation the question is of course how
 do you evaluate models like this for evaluation they go to this website
 called geeks for geeks and this is a an online platform with computer science
 and programming articles it gathers many coding problems and presents solution in
 several programming languages okay so this is a website that teaches you to
 code and it will have like an exercise please do this and then it will provide
 solutions in the different languages now why is that cool and they have an
 example they have an example right here why is that cool because not only can
 you be relatively sure that these different functions that you have here
 do the same thing but you can also relatively be relatively sure that they
 are implemented in the similar way right because you what this website is trying
 to do is it's trying to teach the people how to how to code up an algorithm that
 they think up in their head and therefore not only is the solution
 correct and the same it is implemented in the in the same way as you can see
 here the construct there's this if construct is everywhere the else if is
 everywhere so even though some of the languages might have specialty things
 for implementing some algorithms these are really the same algorithmic the same
 expression of algorithmic thought in the different languages so that is a perfect
 parallel data set the problem of course is that there is not that many so it is
 good enough as a test set it is not good enough as a training set but given that
 it's a test set you can just have these as test set and then you can input the
 C++ and see whether or not the Java comes out the problem here of course is
 that even though this is very clear there are still you know sort of many
 variations of how you can implement that to even express the same algorithmic
 thought so metrics from natural language processing like blue just aren't going
 to be very good because they look at n-gram overlap and you can write this
 function with very different n-grams and still be very very valid and correct and
 also exact match is not going to be really the the gold standard here so
 what they do is they create a set of unit tests where for each of these
 functions they go they check their input types they randomly generate input
 randomly generate a set of inputs look whatever comes out and if the same thing
 comes out in all of their test functions that they consider this a good unit test
 for that function so whenever you your model now produces let's say you input
 Python it produces a C++ you simply put these unit tests through the C++
 function you produce and if they produce the same output as the Python the
 original Python function when on the same inputs then you consider the unit
 test to succeed and you consider the function to be correct that this of
 course this isn't this isn't the super duper gold standard especially with
 random inputs because usually what you want to do is test sort of corner cases
 but it's better than anything else so far I've been a long
 dis-advocate of unit tests honestly because I think whenever a human writes
 a unit test then they're probably since they have already implemented the
 function itself they're probably going to make the same mistakes or they're
 probably just going to replicate the code and thinking of the function in the
 unit test itself and therefore it doesn't really get you anything I guess
 in large organizations you write unit tests so that someone else doesn't screw
 up your code but in this case it would actually be cool because now as a human
 you could simply write a bunch of unit tests and then let your let your trans
 compiler do the heavy lifting and you simply check whether or not the
 output is good alright so how does this do here you can see they have some
 baselines the C++ to Java as I understand it is a commercial system and
 the Java to Python is an open source system both are human experts that make
 up these rule-based systems on how to trans on how to translate code into
 other languages now the if you do what they have here is trans coder beam one
 which means a beam size of one so if you don't know a beam search is very shortly
 beam search is like if you decode from your language model you can either
 always take the next token that has the highest probability this would be greedy
 decoding or a beam size of one or you can sort of always keep the top n
 hypotheses of what the of what the most likely output is as you can keep that as
 a you can keep the top five in memory and always decode these five on sort of
 like you have a mini batch of five sequences and you always keep the top
 five in memory so at the end of the decoding you're going to have five
 different variants of the same sentence or of the same decoded output and you
 can then decide which one you like best and usually what you do is you then
 output the one that has the highest probability which is not the same as the
 greedy because sometimes the next token will be will look one next token will
 look very good in a greedy way but you'd better take the second most likely
 because the next to next token is going to sort of make up for that to make the
 entire sequence even more likely so more beam size basically means you can keep
 more hypotheses of the output in memory until the end so if you just do the
 greedy decoding you see you already get fairly close to these baselines it's
 very very cool very interesting and if you up the beam size you surpass these
 baselines now the way they up the beam size here I find to be a bit let's call
 it a bit cheaty because when they say beam five what they mean is they keep
 the five hypotheses and then at the end I as I understand it if any of the five
 hypotheses passes all the unit tests or the most they keep it right so basically
 they give themselves the freedom to say whichever one of the five we output is
 the best that's the one we count and of course that's not really a match to the
 commercial or to the baseline system because it can output one thing now
 it is maybe a good practical application to give the human that you know you input
 a function you give the human five options to choose from and it can choose
 and thereby decide which one the human likes best but it is sort of it is a
 wonky what I like more is this here the beam 10 top one this is what you would
 actually do so we could keep 10 hypotheses during decoding and that the
 end output the top one the top likely one and as you can see that is better
 than greedy but it is worse than where you you know give yourself the freedom
 to output multiple ones of course though they say that most of the errors that
 this top one makes come from compilation errors when the target language is Java
 or C++ it suggests that the beam and top one metric could easily be improved we
 leave this to future work which this again I find valid right so if you if
 your method is I'm going to keep the top 10 hypothesis until the end and
 then I'm going from the top and I simply compile them and I output the first one
 that compiles that that's not cheating right that's a valid thing again yeah so
 in that way I can I can understand what they're saying right here okay so they
 give some examples some of which I find very interesting so the first thing here
 is that oh yeah by the way I've said in the I've said that the tokenizer between
 the natural languages is shared they make a little tweak here in that they
 tokenize the different languages with their language respective tokenizers
 which will still end up tokenizing pretty much you know this the print
 statement in C++ or in Java no actually the print statement in Python is print
 and in Java it's println and so on but it will still like the all the if
 statements it will still tokenize into the same into the same word but it's
 simply not viable to to parse Python with a C++ parser okay so we have looked
 at this the results this is one of the results they look at their shared
 embedding space and this is a t-sneak plot so a 2d projection of this shared
 embedding space and you can see that this is actually happening so the
 different so null null and none are mapped to similar locations println
 and cout are mapped to similar locations in this space so this is exactly what we
 want this is sort of a verification that this method of embedding the different
 languages into the same space really turns out such that whatever means the
 same thing is mapped to the same place you can see here catch and accept two
 very different tokens are mapped to the same place simply because they're used
 in the same sort of constructs across the languages very cool one of these
 examples here is quite impressive and kind of shows the difference between
 this and and rule-based translation in this function right here you have a C
 plus plus function that takes a character pointer to that is called str
 str in as an input now in C++ strings are at least in old versions of C++
 strings are handled as character arrays so a string is indistinguishable from a
 character array and in this case usually what you do is you don't input the array
 because that will cause a copy you input a pointer to the first to you input a
 pointer to the array and that would define the string okay so if you
 translate this again this the type of this is simply character array if you
 translate this with this transcoder system that they've built into Java in
 Java there is a type called string right there's a native type called string and
 is that true I think oh yeah that's and then that's handled really weirdly in
 the JVM I think yes so there is it at least there is a type called string so
 it would map that it would recognize are you mean a string therefore I'm going to
 put a string here and it uses all the string method like string length string
 character at and so on where in C++ this is just an array and you just have array
 accesses now they take this same C++ function and only change one thing they
 change the name of the parameter everything else is the same but now the
 character array is called our okay and they put it through the same system and
 that system now outputs a function that takes in a character array called our
 instead of a string and it uses you know here the property length it uses array
 access instead of this car character at method so simply by changing the name
 and this is something where I believe the rule-based systems can this can be
 an advantage over rule-based system because what this here does is it simply
 says oh I've seen a lot of humans in my code base that use this use like stir as
 a as a variable name and that usually means that the constructs here are like
 the constructs in Java where people use strings and I've seen other places where
 people use you know names like this right here and usually that is used in
 the same context as in Java people use character arrays right so it in
 programming it's not only important what the the code actually does but a lot of
 programming goes via naming of things like other programmers will read your
 code and by reading stir right here they will sort of assume that this is a
 string whereas if they read our right here they will assume you're a pirate
 and you are referring to a character array and they will treat the code the
 code means something different and these systems right here these neural machine
 translation systems can actually understand that part because they do
 statistical inference on code that humans wrote if you change this back to
 say input then again it goes back to a string and uses all the string functions
 so that's fairly impressive in my mind and it yeah definitely an advantage over
 rule-based systems of course the disadvantage over rule-based systems is
 that in rule-based systems you can almost get on like sometimes you can
 even guarantee that the code does the same thing here you can't they give some
 examples of failed translations where so now you get you run into this problem
 where the min function in Python is overloaded it can either give you the
 minimum of a sequence or it can give you the minimum of two things now this is
 translated to Java right here and math dot min is not overloaded in Java it
 only gives you the minimum of two things and not the minimum of an array and it
 still outputs that now given enough data probably could learn because these
 things are all context dependent but this is one of the this is one of the
 failure cases of these models of course all right so this was this paper I
 I've read that the code of this and the unit tests will be output will be put
 online at some times they are not right now if I if I hear about it I can link to
 it or let you know about it let me know what you think of this paper in the
 comments share it out and subscribe if you haven't yet and bye bye